From tfheen at projects.linpro.no Mon Jan 4 12:50:52 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Mon, 4 Jan 2010 13:50:52 +0100 (CET) Subject: r4420 - trunk/varnish-cache Message-ID: <20100104125052.D2D1F1F751A@projects.linpro.no> Author: tfheen Date: 2010-01-04 13:50:52 +0100 (Mon, 04 Jan 2010) New Revision: 4420 Modified: trunk/varnish-cache/INSTALL Log: Document need for ldconfig on Linux On Linux, ldconfig needs to be run after installing shared libraries to system paths (typically /usr/local/lib, /usr/lib and /lib). Document this. Fixes #586 Modified: trunk/varnish-cache/INSTALL =================================================================== --- trunk/varnish-cache/INSTALL 2009-12-16 15:46:44 UTC (rev 4419) +++ trunk/varnish-cache/INSTALL 2010-01-04 12:50:52 UTC (rev 4420) @@ -2,7 +2,8 @@ Varnish uses the GNU autotools. To build and install Varnish, simply run the 'configure' script in the top-level directory, then run 'make' -and 'make install'. +and 'make install'. On Linux, you need to run 'ldconfig' as root +afterwards in order to update the shared library cache. If you obtained the sources directly from the Subversion repository, you will need to run autogen.sh first to create the configure script. From tfheen at projects.linpro.no Mon Jan 4 21:36:00 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Mon, 4 Jan 2010 22:36:00 +0100 (CET) Subject: r4421 - in trunk/varnish-cache: bin/varnishd man Message-ID: <20100104213600.6907E1F747C@projects.linpro.no> Author: tfheen Date: 2010-01-04 22:36:00 +0100 (Mon, 04 Jan 2010) New Revision: 4421 Modified: trunk/varnish-cache/bin/varnishd/varnishd.1 trunk/varnish-cache/man/vcl.7so Log: Document server.identity and server.hostname (and -i to varnishd) Fixes #602 Modified: trunk/varnish-cache/bin/varnishd/varnishd.1 =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.1 2010-01-04 12:50:52 UTC (rev 4420) +++ trunk/varnish-cache/bin/varnishd/varnishd.1 2010-01-04 21:36:00 UTC (rev 4421) @@ -43,6 +43,7 @@ .Op Fl f Ar config .Op Fl g Ar group .Op Fl h Ar type Ns Op , Ns Ar options +.Op Fl i Ar identity .Op Fl l Ar shmlogsize .Op Fl n Ar name .Op Fl P Ar file @@ -138,6 +139,11 @@ See .Sx Hash Algorithms for a list of supported algorithms. +.It Fl i Ar identity +Specify the identity of the varnish server. This can be accessed +using +.Va server.identity +from VCL .It Fl l Ar shmlogsize Specify size of shmlog file. Scaling suffixes like 'k', 'm' can be used up to (e)tabytes. Modified: trunk/varnish-cache/man/vcl.7so =================================================================== --- trunk/varnish-cache/man/vcl.7so 2010-01-04 12:50:52 UTC (rev 4420) +++ trunk/varnish-cache/man/vcl.7so 2010-01-04 21:36:00 UTC (rev 4421) @@ -489,6 +489,20 @@ .Bl -tag -width 4n .It Va client.ip The client's IP address. +.It Va server.hostname +The host name of the server. +.It Va server.identity +The identity of the server, as set by the +.Fl i +parameter. +If the +.Fl i +parameter is not passed to +.Nm varnishd , +.Va server.identity +will be set to the name of the instance, as specified by the +.Fl n +parameter. .It Va server.ip The IP address of the socket on which the client connection was received. From tfheen at projects.linpro.no Wed Jan 6 09:42:34 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 6 Jan 2010 10:42:34 +0100 (CET) Subject: r4422 - trunk/varnish-cache Message-ID: <20100106094234.EF63A1F7BA6@projects.linpro.no> Author: tfheen Date: 2010-01-06 10:42:34 +0100 (Wed, 06 Jan 2010) New Revision: 4422 Modified: trunk/varnish-cache/configure.ac Log: Increase default max number of headers People have been running into the maximum number of headers limitation more and more lately, so increase this a little bit. It is a per-thread overhead, so it should not make any real difference when it comes to memory consumption. Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2010-01-04 21:36:00 UTC (rev 4421) +++ trunk/varnish-cache/configure.ac 2010-01-06 09:42:34 UTC (rev 4422) @@ -417,9 +417,9 @@ # Define HTTP_HDR_MAX_VAL AC_ARG_WITH(max-header-fields, AS_HELP_STRING([--with-max-header-fields=NUM], - [How many header fields to support (default=32)]), + [How many header fields to support (default=64)]), [], - [with_max_header_fields=32]) + [with_max_header_fields=64]) AC_DEFINE_UNQUOTED(HTTP_HDR_MAX_VAL, $with_max_header_fields, [Define maximum number of header fields supported by varnish ]) From phk at projects.linpro.no Wed Jan 6 15:50:42 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 6 Jan 2010 16:50:42 +0100 (CET) Subject: r4423 - trunk/varnish-cache/bin/varnishd Message-ID: <20100106155042.9DF371F7BDD@projects.linpro.no> Author: phk Date: 2010-01-06 16:50:42 +0100 (Wed, 06 Jan 2010) New Revision: 4423 Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c Log: Write the ESI includ URL into the correct workspace. NB: This fixes bad ESI-breakage I introduced in r4351 on 2009-11-16 12:44:03 NB: This breakage has not affected the 2.x branch. Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-06 09:42:34 UTC (rev 4422) +++ trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-06 15:50:42 UTC (rev 4423) @@ -435,13 +435,13 @@ tag.e = q + 1; u = WS_Reserve(ew->sp->wrk->ws, 0); - v = snprintf(ws->f, u - 1, "%.*s%.*s", + v = snprintf(ew->sp->wrk->ws->f, u - 1, "%.*s%.*s", pdiff(tag.b, tag.e), tag.b, pdiff(val.b, val.e), val.b); v++; xxxassert(v < u); - eb->include.b = ws->f; - eb->include.e = ws->f + v; + eb->include.b = ew->sp->wrk->ws->f; + eb->include.e = ew->sp->wrk->ws->f + v; WS_Release(ew->sp->wrk->ws, v); } if (eb->include.b != NULL) From phk at projects.linpro.no Wed Jan 6 15:51:06 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 6 Jan 2010 16:51:06 +0100 (CET) Subject: r4424 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100106155106.1E7961F7BE6@projects.linpro.no> Author: phk Date: 2010-01-06 16:51:05 +0100 (Wed, 06 Jan 2010) New Revision: 4424 Added: trunk/varnish-cache/bin/varnishtest/tests/r00612.vtc Log: Add testcase for #612. Fixes: #612 Added: trunk/varnish-cache/bin/varnishtest/tests/r00612.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/r00612.vtc (rev 0) +++ trunk/varnish-cache/bin/varnishtest/tests/r00612.vtc 2010-01-06 15:51:05 UTC (rev 4424) @@ -0,0 +1,49 @@ +# $Id$ + +test "Url workspace gets overwritten/reused" + +server s1 { + rxreq + expect req.url == "/" + txresp -body { + + + + + + } + + rxreq + expect req.url == "/1" + txresp -body "11111" + + rxreq + expect req.url == "/2" + txresp -body "22222" + + rxreq + expect req.url == "/3" + txresp -body "33333" + + rxreq + expect req.url == "/4" + txresp -body "44444" + + rxreq + expect req.url == "/5" + txresp -body "55555" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } + sub vcl_fetch { + esi; + } +} -start + +client c1 { + txreq + rxresp +} -run From phk at projects.linpro.no Wed Jan 6 16:49:54 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 6 Jan 2010 17:49:54 +0100 (CET) Subject: r4425 - in trunk/varnish-cache/bin/varnishtest: . tests Message-ID: <20100106164954.1667328129@projects.linpro.no> Author: phk Date: 2010-01-06 17:49:53 +0100 (Wed, 06 Jan 2010) New Revision: 4425 Modified: trunk/varnish-cache/bin/varnishtest/tests/r00416.vtc trunk/varnish-cache/bin/varnishtest/vtc.c Log: Unbreak the "too many HTTP headers" test after the default was raised from 32 to 64. Modified: trunk/varnish-cache/bin/varnishtest/tests/r00416.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/r00416.vtc 2010-01-06 15:51:05 UTC (rev 4424) +++ trunk/varnish-cache/bin/varnishtest/tests/r00416.vtc 2010-01-06 16:49:53 UTC (rev 4425) @@ -45,6 +45,36 @@ -hdr hdr37=37 \ -hdr hdr38=38 \ -hdr hdr39=39 \ + -hdr hdr40=40 \ + -hdr hdr41=41 \ + -hdr hdr42=42 \ + -hdr hdr43=43 \ + -hdr hdr44=44 \ + -hdr hdr45=45 \ + -hdr hdr46=46 \ + -hdr hdr47=47 \ + -hdr hdr48=48 \ + -hdr hdr49=49 \ + -hdr hdr50=50 \ + -hdr hdr51=51 \ + -hdr hdr52=52 \ + -hdr hdr53=53 \ + -hdr hdr54=54 \ + -hdr hdr55=55 \ + -hdr hdr56=56 \ + -hdr hdr57=57 \ + -hdr hdr58=58 \ + -hdr hdr59=59 \ + -hdr hdr60=60 \ + -hdr hdr61=61 \ + -hdr hdr62=62 \ + -hdr hdr63=63 \ + -hdr hdr64=64 \ + -hdr hdr65=65 \ + -hdr hdr66=66 \ + -hdr hdr67=67 \ + -hdr hdr68=68 \ + -hdr hdr69=69 \ -body "foo" } -start @@ -91,7 +121,37 @@ -hdr hdr36=36 \ -hdr hdr37=37 \ -hdr hdr38=38 \ - -hdr hdr39=39 + -hdr hdr39=39 \ + -hdr hdr40=40 \ + -hdr hdr41=41 \ + -hdr hdr42=42 \ + -hdr hdr43=43 \ + -hdr hdr44=44 \ + -hdr hdr45=45 \ + -hdr hdr46=46 \ + -hdr hdr47=47 \ + -hdr hdr48=48 \ + -hdr hdr49=49 \ + -hdr hdr50=50 \ + -hdr hdr51=51 \ + -hdr hdr52=52 \ + -hdr hdr53=53 \ + -hdr hdr54=54 \ + -hdr hdr55=55 \ + -hdr hdr56=56 \ + -hdr hdr57=57 \ + -hdr hdr58=58 \ + -hdr hdr59=59 \ + -hdr hdr60=60 \ + -hdr hdr61=61 \ + -hdr hdr62=62 \ + -hdr hdr63=63 \ + -hdr hdr64=64 \ + -hdr hdr65=65 \ + -hdr hdr66=66 \ + -hdr hdr67=67 \ + -hdr hdr68=68 \ + -hdr hdr69=69 rxresp } -run Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-06 15:51:05 UTC (rev 4424) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-06 16:49:53 UTC (rev 4425) @@ -49,7 +49,7 @@ #include "vtc.h" #define MAX_FILESIZE (1024 * 1024) -#define MAX_TOKENS 100 +#define MAX_TOKENS 200 const char *vtc_file; char *vtc_desc; From phk at projects.linpro.no Wed Jan 6 17:06:57 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 6 Jan 2010 18:06:57 +0100 (CET) Subject: r4426 - in trunk/varnish-cache/bin: varnishd varnishtest/tests Message-ID: <20100106170657.39E1D38CDA@projects.linpro.no> Author: phk Date: 2010-01-06 18:06:56 +0100 (Wed, 06 Jan 2010) New Revision: 4426 Added: trunk/varnish-cache/bin/varnishtest/tests/e00018.vtc Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c Log: Add code to replace the five mandatory XML 1.0 entity references. Inspired by: patch submitted in #607 Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-06 16:49:53 UTC (rev 4425) +++ trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-06 17:06:56 UTC (rev 4426) @@ -153,6 +153,37 @@ /*-------------------------------------------------------------------- + * Replace the mandatory XML 1.0 entity references, in place. + */ + +static void +XMLentity(txt *t) +{ + char *s, *d; + + for (s = d = t->b; s < t->e; ) { + if (*s == '&') { +#define R(l,f,r) \ + if (s + l <= t->e && !memcmp(s, f, l)) { \ + *d++ = r; \ + s += l; \ + continue; \ + } + R(6, "'", '\''); + R(6, """, '"'); + R(4, "<", '<'); + R(4, ">", '>'); + R(5, "&", '&'); + } +#undef R + *d++ = *s++; + } + t->e = d; + t->e[0] = '\0'; +} + + +/*-------------------------------------------------------------------- * Report a parsing error * * XXX: The "at xxx" count is usually the tail of the sequence. Since we @@ -385,7 +416,7 @@ WS_Assert(ws); s = 0; - if ( val.b != val.e ) { + if (val.b != val.e) { s = Tlen(val) + 1; c = WS_Alloc(ew->sp->wrk->ws, s); XXXAN(c); @@ -395,6 +426,9 @@ val.e[-1] = '\0'; } + if (strchr(val.b, '&')) + XMLentity(&val); + if (Tlen(val) > 7 && !memcmp(val.b, "http://", 7)) { /* Rewrite to Host: header inplace */ eb->host.b = val.b; Added: trunk/varnish-cache/bin/varnishtest/tests/e00018.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/e00018.vtc (rev 0) +++ trunk/varnish-cache/bin/varnishtest/tests/e00018.vtc 2010-01-06 17:06:56 UTC (rev 4426) @@ -0,0 +1,51 @@ +# $Id$ + +test "Test XML 1.0 entity references" + +server s1 { + rxreq + expect req.url == "/" + txresp -body { + + + + + + } + + rxreq + expect req.url == "/&" + txresp -body "1" + + rxreq + expect req.url == "/<" + txresp -body "22" + + rxreq + expect req.url == "/>" + txresp -body "333" + + rxreq + expect req.url == {/'} + txresp -body "4444" + + rxreq + expect req.url == {/"} + txresp -body "55555" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } + sub vcl_fetch { + esi; + } +} -start + +client c1 { + txreq + rxresp + expect resp.status == 200 + expect resp.bodylen == 32 +} -run From tfheen at projects.linpro.no Wed Jan 6 17:38:53 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 6 Jan 2010 18:38:53 +0100 (CET) Subject: r4427 - in trunk/varnish-cache: . lib/libvarnish Message-ID: <20100106173853.B4BB238B61@projects.linpro.no> Author: tfheen Date: 2010-01-06 18:38:53 +0100 (Wed, 06 Jan 2010) New Revision: 4427 Modified: trunk/varnish-cache/configure.ac trunk/varnish-cache/lib/libvarnish/time.c Log: Use gethrtimer rather than clock_gettime if available gethrtimer does not use a syscall on Solaris and is therefore faster. Prefer gethrtimer if it exists Fixes #609 Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2010-01-06 17:06:56 UTC (rev 4426) +++ trunk/varnish-cache/configure.ac 2010-01-06 17:38:53 UTC (rev 4427) @@ -232,6 +232,7 @@ save_LIBS="${LIBS}" LIBS="${LIBS} ${RT_LIBS}" AC_CHECK_FUNCS([clock_gettime]) +AC_CHECK_FUNCS([gethrtime]) LIBS="${save_LIBS}" # Check which mechanism to use for the acceptor. We look for kqueue Modified: trunk/varnish-cache/lib/libvarnish/time.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/time.c 2010-01-06 17:06:56 UTC (rev 4426) +++ trunk/varnish-cache/lib/libvarnish/time.c 2010-01-06 17:38:53 UTC (rev 4427) @@ -59,10 +59,18 @@ #include "libvarnish.h" +/* + * Note on Solaris: for some reason, clock_gettime(CLOCK_MONOTONIC, &ts) is not + * implemented in assembly, but falls into a syscall, while gethrtime() doesn't, + * so we save a syscall by using gethrtime() if it is defined. + */ + double TIM_mono(void) { -#ifdef HAVE_CLOCK_GETTIME +#ifdef HAVE_GETHRTIME + return (gethrtime() * 1e-9); +#elif HAVE_CLOCK_GETTIME struct timespec ts; assert(clock_gettime(CLOCK_MONOTONIC, &ts) == 0); @@ -177,6 +185,8 @@ /* * Compile with: * cc -o foo -DTEST_DRIVER -I../.. -I../../include time.c assert.c + * (Solaris) + * cc -o foo -DTEST_DRIVER -I../.. -I../../include -lm time.c assert.c * Test with: * env TZ=UTC ./foo * env TZ=CET ./foo @@ -198,6 +208,49 @@ } } +static int +tst_delta_check(const char *name, double begin, double end, double ref) +{ + const double tol_max = 1.1; + const double tol_min = 1; + + printf("%s delta for %fs sleep: %f\n", name, ref, (end - begin)); + + if ((end - begin) > tol_max * ref) { + printf("%s delta above tolerance: ((%f - %f) = %f) > %f\n", + name, end, begin, (end - begin), tol_max); + return (1); + } else if ((end - begin) < tol_min * ref) { + printf("%s delta below tolerance: ((%f - %f) = %f) < %f\n", + name, end, begin, (end - begin), tol_min); + return (1); + } + return (0); +} + +static void +tst_delta() +{ + double m_begin, m_end; + double r_begin, r_end; + const double ref = 1; + int err = 0; + + r_begin = TIM_real(); + m_begin = TIM_mono(); + TIM_sleep(ref); + r_end = TIM_real(); + m_end = TIM_mono(); + + err += tst_delta_check("TIM_mono", m_begin, m_end, ref); + err += tst_delta_check("TIM_real", r_begin, r_end, ref); + + if (err) { + printf("%d time delta test errrors\n", err); + exit (2); + } +} + int main(int argc, char **argv) { @@ -214,6 +267,8 @@ tst("Sunday, 06-Nov-94 08:49:37 GMT", 784111777); tst("Sun Nov 6 08:49:37 1994", 784111777); + tst_delta(); + return (0); } #endif From tfheen at projects.linpro.no Wed Jan 6 17:38:59 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 6 Jan 2010 18:38:59 +0100 (CET) Subject: r4428 - in trunk/varnish-cache: bin/varnishd bin/varnishtest/tests include lib/libvarnish lib/libvcl Message-ID: <20100106173859.B991A97D32@projects.linpro.no> Author: tfheen Date: 2010-01-06 18:38:59 +0100 (Wed, 06 Jan 2010) New Revision: 4428 Added: trunk/varnish-cache/bin/varnishtest/tests/b00030.vtc Modified: trunk/varnish-cache/bin/varnishd/cache_vrt.c trunk/varnish-cache/bin/varnishtest/tests/v00018.vtc trunk/varnish-cache/include/libvarnish.h trunk/varnish-cache/include/vrt.h trunk/varnish-cache/lib/libvarnish/time.c trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c trunk/varnish-cache/lib/libvcl/vcc_gen_fixed_token.tcl trunk/varnish-cache/lib/libvcl/vcc_obj.c trunk/varnish-cache/lib/libvcl/vcc_parse.c trunk/varnish-cache/lib/libvcl/vcc_string.c Log: Add string representations of TIME and RTIME Make it possible to get useful string representations of TIME and RTIME out. Also change the definition of types that should be RTIME from TIME. Modified: trunk/varnish-cache/bin/varnishd/cache_vrt.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/bin/varnishd/cache_vrt.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -840,6 +840,16 @@ return (p); } +char * +VRT_time_string(const struct sess *sp, double t) +{ + char *p; + + AN(p = WS_Alloc(sp->http->ws, TIM_FORMAT_SIZE)); + TIM_format(t, p); + return p; +} + const char * VRT_backend_string(struct sess *sp) { Added: trunk/varnish-cache/bin/varnishtest/tests/b00030.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00030.vtc (rev 0) +++ trunk/varnish-cache/bin/varnishtest/tests/b00030.vtc 2010-01-06 17:38:59 UTC (rev 4428) @@ -0,0 +1,20 @@ +# $Id$ + +test "Test formatting of timestamps" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + error 200 req.grace; + } +} -start + +client c1 { + txreq + rxresp + expect resp.msg == "10.000" +} -run Modified: trunk/varnish-cache/bin/varnishtest/tests/v00018.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00018.vtc 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/bin/varnishtest/tests/v00018.vtc 2010-01-06 17:38:59 UTC (rev 4428) @@ -27,11 +27,6 @@ varnish v1 -badvcl { backend b { .host = "127.0.0.1"; } - sub vcl_fetch { error 404 req.grace; } -} - -varnish v1 -badvcl { - backend b { .host = "127.0.0.1"; } sub vcl_miss { set server.port = 1000; } } Modified: trunk/varnish-cache/include/libvarnish.h =================================================================== --- trunk/varnish-cache/include/libvarnish.h 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/include/libvarnish.h 2010-01-06 17:38:59 UTC (rev 4428) @@ -75,6 +75,7 @@ #endif /* from libvarnish/time.c */ +#define TIM_FORMAT_SIZE 30 void TIM_format(double t, char *p); time_t TIM_parse(const char *p); double TIM_mono(void); Modified: trunk/varnish-cache/include/vrt.h =================================================================== --- trunk/varnish-cache/include/vrt.h 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/include/vrt.h 2010-01-06 17:38:59 UTC (rev 4428) @@ -168,6 +168,7 @@ char *VRT_IP_string(const struct sess *sp, const struct sockaddr *sa); char *VRT_int_string(const struct sess *sp, int); char *VRT_double_string(const struct sess *sp, double); +char *VRT_time_string(const struct sess *sp, double); const char *VRT_backend_string(struct sess *sp); #define VRT_done(sp, hand) \ Modified: trunk/varnish-cache/lib/libvarnish/time.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/time.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvarnish/time.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -107,7 +107,7 @@ tt = (time_t) t; (void)gmtime_r(&tt, &tm); - AN(strftime(p, 30, "%a, %d %b %Y %T GMT", &tm)); + AN(strftime(p, TIM_FORMAT_SIZE, "%a, %d %b %Y %T GMT", &tm)); } /* XXX: add statistics ? */ Modified: trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -1,5 +1,5 @@ /* - * $Id: vcc_gen_fixed_token.tcl 4376 2009-11-27 11:06:32Z phk $ + * $Id$ * * NB: This file is machine generated, DO NOT EDIT! * @@ -159,10 +159,9 @@ /* ../../include/vcl.h */ - vsb_cat(sb, "/*\n * $Id: vcc_gen_fixed_token.tcl 4376 2009-11-27 11"); - vsb_cat(sb, ":06:32Z phk $\n *\n * NB: This file is machine genera"); - vsb_cat(sb, "ted, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_t"); - vsb_cat(sb, "oken.tcl instead\n */\n\nstruct sess;\n"); + vsb_cat(sb, "/*\n * $Id$\n *\n * NB: This file is machine generate"); + vsb_cat(sb, "d, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_tok"); + vsb_cat(sb, "en.tcl instead\n */\n\nstruct sess;\n"); vsb_cat(sb, "struct cli;\n\ntypedef void vcl_init_f(struct cli *);\n"); vsb_cat(sb, "typedef void vcl_fini_f(struct cli *);\n"); vsb_cat(sb, "typedef int vcl_func_f(struct sess *sp);\n"); @@ -227,16 +226,16 @@ vsb_cat(sb, " * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWI"); vsb_cat(sb, "SE) ARISING IN ANY WAY\n * OUT OF THE USE OF THIS SOFT"); vsb_cat(sb, "WARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n"); - vsb_cat(sb, " * SUCH DAMAGE.\n *\n * $Id: vrt.h 4405 2009-12-16 10:"); - vsb_cat(sb, "45:39Z phk $\n *\n * Runtime support for compiled VCL "); - vsb_cat(sb, "programs.\n *\n * XXX: When this file is changed, lib/"); - vsb_cat(sb, "libvcl/vcc_gen_fixed_token.tcl\n"); - vsb_cat(sb, " * XXX: *MUST* be rerun.\n */\n"); - vsb_cat(sb, "\nstruct sess;\nstruct vsb;\nstruct cli;\n"); - vsb_cat(sb, "struct director;\nstruct VCL_conf;\n"); - vsb_cat(sb, "struct sockaddr;\n\n/*\n * A backend probe specificati"); - vsb_cat(sb, "on\n */\n\nextern const void * const vrt_magic_string_"); - vsb_cat(sb, "end;\n\nstruct vrt_backend_probe {\n"); + vsb_cat(sb, " * SUCH DAMAGE.\n *\n * $Id$\n *\n"); + vsb_cat(sb, " * Runtime support for compiled VCL programs.\n"); + vsb_cat(sb, " *\n * XXX: When this file is changed, lib/libvcl/vcc_"); + vsb_cat(sb, "gen_fixed_token.tcl\n * XXX: *MUST* be rerun.\n"); + vsb_cat(sb, " */\n\nstruct sess;\nstruct vsb;\n"); + vsb_cat(sb, "struct cli;\nstruct director;\n"); + vsb_cat(sb, "struct VCL_conf;\nstruct sockaddr;\n"); + vsb_cat(sb, "\n/*\n * A backend probe specification\n"); + vsb_cat(sb, " */\n\nextern const void * const vrt_magic_string_end;"); + vsb_cat(sb, "\n\nstruct vrt_backend_probe {\n"); vsb_cat(sb, "\tconst char\t*url;\n\tconst char\t*request;\n"); vsb_cat(sb, "\tdouble\t\ttimeout;\n\tdouble\t\tinterval;\n"); vsb_cat(sb, "\tunsigned\texp_status;\n\tunsigned\twindow;\n"); @@ -304,33 +303,33 @@ vsb_cat(sb, "\nchar *VRT_IP_string(const struct sess *sp, const str"); vsb_cat(sb, "uct sockaddr *sa);\nchar *VRT_int_string(const struct "); vsb_cat(sb, "sess *sp, int);\nchar *VRT_double_string(const struct "); - vsb_cat(sb, "sess *sp, double);\nconst char *VRT_backend_string(str"); - vsb_cat(sb, "uct sess *sp);\n\n#define VRT_done(sp, hand)\t\t\t\\\n"); + vsb_cat(sb, "sess *sp, double);\nchar *VRT_time_string(const struct"); + vsb_cat(sb, " sess *sp, double);\nconst char *VRT_backend_string(st"); + vsb_cat(sb, "ruct sess *sp);\n\n#define VRT_done(sp, hand)\t\t\t\\\n"); vsb_cat(sb, "\tdo {\t\t\t\t\t\\\n\t\tVRT_handling(sp, hand);\t\t\\\n"); vsb_cat(sb, "\t\treturn (1);\t\t\t\\\n\t} while (0)\n"); /* ../../include/vrt_obj.h */ - vsb_cat(sb, "/*\n * $Id: vcc_gen_fixed_token.tcl 4376 2009-11-27 11"); - vsb_cat(sb, ":06:32Z phk $\n *\n * NB: This file is machine genera"); - vsb_cat(sb, "ted, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_t"); - vsb_cat(sb, "oken.tcl instead\n */\n\nstruct sockaddr * VRT_r_clien"); - vsb_cat(sb, "t_ip(const struct sess *);\nstruct sockaddr * VRT_r_se"); - vsb_cat(sb, "rver_ip(struct sess *);\nconst char * VRT_r_server_hos"); - vsb_cat(sb, "tname(struct sess *);\nconst char * VRT_r_server_ident"); - vsb_cat(sb, "ity(struct sess *);\nint VRT_r_server_port(struct sess"); - vsb_cat(sb, " *);\nconst char * VRT_r_req_request(const struct sess"); - vsb_cat(sb, " *);\nvoid VRT_l_req_request(const struct sess *, cons"); - vsb_cat(sb, "t char *, ...);\nconst char * VRT_r_req_url(const stru"); - vsb_cat(sb, "ct sess *);\nvoid VRT_l_req_url(const struct sess *, c"); - vsb_cat(sb, "onst char *, ...);\nconst char * VRT_r_req_proto(const"); - vsb_cat(sb, " struct sess *);\nvoid VRT_l_req_proto(const struct se"); - vsb_cat(sb, "ss *, const char *, ...);\nvoid VRT_l_req_hash(struct "); - vsb_cat(sb, "sess *, const char *);\nstruct director * VRT_r_req_ba"); - vsb_cat(sb, "ckend(struct sess *);\nvoid VRT_l_req_backend(struct s"); - vsb_cat(sb, "ess *, struct director *);\nint VRT_r_req_restarts(con"); - vsb_cat(sb, "st struct sess *);\ndouble VRT_r_req_grace(struct sess"); - vsb_cat(sb, " *);\nvoid VRT_l_req_grace(struct sess *, double);\n"); + vsb_cat(sb, "/*\n * $Id$\n *\n * NB: This file is machine generate"); + vsb_cat(sb, "d, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_tok"); + vsb_cat(sb, "en.tcl instead\n */\n\nstruct sockaddr * VRT_r_client_"); + vsb_cat(sb, "ip(const struct sess *);\nstruct sockaddr * VRT_r_serv"); + vsb_cat(sb, "er_ip(struct sess *);\nconst char * VRT_r_server_hostn"); + vsb_cat(sb, "ame(struct sess *);\nconst char * VRT_r_server_identit"); + vsb_cat(sb, "y(struct sess *);\nint VRT_r_server_port(struct sess *"); + vsb_cat(sb, ");\nconst char * VRT_r_req_request(const struct sess *"); + vsb_cat(sb, ");\nvoid VRT_l_req_request(const struct sess *, const "); + vsb_cat(sb, "char *, ...);\nconst char * VRT_r_req_url(const struct"); + vsb_cat(sb, " sess *);\nvoid VRT_l_req_url(const struct sess *, con"); + vsb_cat(sb, "st char *, ...);\nconst char * VRT_r_req_proto(const s"); + vsb_cat(sb, "truct sess *);\nvoid VRT_l_req_proto(const struct sess"); + vsb_cat(sb, " *, const char *, ...);\nvoid VRT_l_req_hash(struct se"); + vsb_cat(sb, "ss *, const char *);\nstruct director * VRT_r_req_back"); + vsb_cat(sb, "end(struct sess *);\nvoid VRT_l_req_backend(struct ses"); + vsb_cat(sb, "s *, struct director *);\nint VRT_r_req_restarts(const"); + vsb_cat(sb, " struct sess *);\ndouble VRT_r_req_grace(struct sess *"); + vsb_cat(sb, ");\nvoid VRT_l_req_grace(struct sess *, double);\n"); vsb_cat(sb, "const char * VRT_r_req_xid(struct sess *);\n"); vsb_cat(sb, "unsigned VRT_r_req_esi(struct sess *);\n"); vsb_cat(sb, "void VRT_l_req_esi(struct sess *, unsigned);\n"); Modified: trunk/varnish-cache/lib/libvcl/vcc_gen_fixed_token.tcl =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_gen_fixed_token.tcl 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvcl/vcc_gen_fixed_token.tcl 2010-01-06 17:38:59 UTC (rev 4428) @@ -150,7 +150,7 @@ all "const struct sess *" } - { req.grace TIME + { req.grace RTIME RW all "struct sess *" @@ -196,17 +196,17 @@ { pipe pass miss fetch } "const struct sess *" } - { bereq.connect_timeout TIME + { bereq.connect_timeout RTIME RW { pass miss } "struct sess *" } - { bereq.first_byte_timeout TIME + { bereq.first_byte_timeout RTIME RW { pass miss } "struct sess *" } - { bereq.between_bytes_timeout TIME + { bereq.between_bytes_timeout RTIME RW { pass miss } "struct sess *" @@ -219,7 +219,7 @@ { fetch } "const struct sess *" } - { beresp.saintmode TIME + { beresp.saintmode RTIME WO { fetch } "const struct sess *" @@ -244,12 +244,12 @@ { fetch } "const struct sess *" } - { beresp.ttl TIME + { beresp.ttl RTIME RW { fetch } "const struct sess *" } - { beresp.grace TIME + { beresp.grace RTIME RW { fetch } "const struct sess *" @@ -288,17 +288,17 @@ { hit } "const struct sess *" } - { obj.ttl TIME + { obj.ttl RTIME RW { hit error } "const struct sess *" } - { obj.grace TIME + { obj.grace RTIME RW { hit error } "const struct sess *" } - { obj.lastuse TIME + { obj.lastuse RTIME RO { hit deliver error } "const struct sess *" Modified: trunk/varnish-cache/lib/libvcl/vcc_obj.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_obj.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvcl/vcc_obj.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -1,5 +1,5 @@ /* - * $Id: vcc_gen_fixed_token.tcl 4376 2009-11-27 11:06:32Z phk $ + * $Id$ * * NB: This file is machine generated, DO NOT EDIT! * @@ -105,7 +105,7 @@ | VCL_MET_MISS | VCL_MET_HIT | VCL_MET_FETCH | VCL_MET_DELIVER | VCL_MET_ERROR }, - { "req.grace", TIME, 9, + { "req.grace", RTIME, 9, "VRT_r_req_grace(sp)", "VRT_l_req_grace(sp, ", V_RW, 0, @@ -159,19 +159,19 @@ V_RW, "HDR_BEREQ", VCL_MET_PIPE | VCL_MET_PASS | VCL_MET_MISS | VCL_MET_FETCH }, - { "bereq.connect_timeout", TIME, 21, + { "bereq.connect_timeout", RTIME, 21, "VRT_r_bereq_connect_timeout(sp)", "VRT_l_bereq_connect_timeout(sp, ", V_RW, 0, VCL_MET_PASS | VCL_MET_MISS }, - { "bereq.first_byte_timeout", TIME, 24, + { "bereq.first_byte_timeout", RTIME, 24, "VRT_r_bereq_first_byte_timeout(sp)", "VRT_l_bereq_first_byte_timeout(sp, ", V_RW, 0, VCL_MET_PASS | VCL_MET_MISS }, - { "bereq.between_bytes_timeout", TIME, 27, + { "bereq.between_bytes_timeout", RTIME, 27, "VRT_r_bereq_between_bytes_timeout(sp)", "VRT_l_bereq_between_bytes_timeout(sp, ", V_RW, 0, @@ -183,7 +183,7 @@ V_RW, 0, VCL_MET_FETCH }, - { "beresp.saintmode", TIME, 16, + { "beresp.saintmode", RTIME, 16, NULL, "VRT_l_beresp_saintmode(sp, ", V_WO, 0, @@ -213,13 +213,13 @@ V_RW, 0, VCL_MET_FETCH }, - { "beresp.ttl", TIME, 10, + { "beresp.ttl", RTIME, 10, "VRT_r_beresp_ttl(sp)", "VRT_l_beresp_ttl(sp, ", V_RW, 0, VCL_MET_FETCH }, - { "beresp.grace", TIME, 12, + { "beresp.grace", RTIME, 12, "VRT_r_beresp_grace(sp)", "VRT_l_beresp_grace(sp, ", V_RW, 0, @@ -261,19 +261,19 @@ V_RW, 0, VCL_MET_HIT }, - { "obj.ttl", TIME, 7, + { "obj.ttl", RTIME, 7, "VRT_r_obj_ttl(sp)", "VRT_l_obj_ttl(sp, ", V_RW, 0, VCL_MET_HIT | VCL_MET_ERROR }, - { "obj.grace", TIME, 9, + { "obj.grace", RTIME, 9, "VRT_r_obj_grace(sp)", "VRT_l_obj_grace(sp, ", V_RW, 0, VCL_MET_HIT | VCL_MET_ERROR }, - { "obj.lastuse", TIME, 11, + { "obj.lastuse", RTIME, 11, "VRT_r_obj_lastuse(sp)", NULL, V_RO, 0, Modified: trunk/varnish-cache/lib/libvcl/vcc_parse.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_parse.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvcl/vcc_parse.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -265,6 +265,9 @@ case TIME: vcc_TimeVal(tl); break; + case RTIME: + vcc_RTimeVal(tl); + break; case INT: ExpectErr(tl, CNUM); Fb(tl, 0, "%.*s ", PF(tl->t)); @@ -355,6 +358,7 @@ case IP: L(tl, vcc_Cond_Ip(vp, tl)); break; case STRING: L(tl, Cond_String(vp, tl)); break; case TIME: L(tl, Cond_Int(vp, tl)); break; + case RTIME: L(tl, Cond_Int(vp, tl)); break; case BACKEND: L(tl, Cond_Backend(vp, tl)); break; default: vsb_printf(tl->sb, Modified: trunk/varnish-cache/lib/libvcl/vcc_string.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_string.c 2010-01-06 17:38:53 UTC (rev 4427) +++ trunk/varnish-cache/lib/libvcl/vcc_string.c 2010-01-06 17:38:59 UTC (rev 4428) @@ -171,6 +171,12 @@ case FLOAT: Fb(tl, 0, "VRT_double_string(sp, %s)", vp->rname); break; + case TIME: + Fb(tl, 0, "VRT_time_string(sp, %s)", vp->rname); + break; + case RTIME: + Fb(tl, 0, "VRT_double_string(sp, %s)", vp->rname); + break; case BACKEND: Fb(tl, 0, "VRT_backend_string(sp)"); break; From tfheen at projects.linpro.no Wed Jan 6 18:49:40 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 6 Jan 2010 19:49:40 +0100 (CET) Subject: r4429 - trunk/varnish-cache/include Message-ID: <20100106184940.40C5D383D9@projects.linpro.no> Author: tfheen Date: 2010-01-06 19:49:39 +0100 (Wed, 06 Jan 2010) New Revision: 4429 Modified: trunk/varnish-cache/include/Makefile.am Log: Make sure to distribute vre.h too Modified: trunk/varnish-cache/include/Makefile.am =================================================================== --- trunk/varnish-cache/include/Makefile.am 2010-01-06 17:38:59 UTC (rev 4428) +++ trunk/varnish-cache/include/Makefile.am 2010-01-06 18:49:39 UTC (rev 4429) @@ -40,6 +40,7 @@ vev.h \ vlu.h \ vbm.h \ + vre.h \ vrt.h \ vrt_obj.h \ vss.h From tfheen at projects.linpro.no Wed Jan 6 18:50:18 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 6 Jan 2010 19:50:18 +0100 (CET) Subject: r4430 - trunk/varnish-cache/lib/libvcl Message-ID: <20100106185018.547D438868@projects.linpro.no> Author: tfheen Date: 2010-01-06 19:50:18 +0100 (Wed, 06 Jan 2010) New Revision: 4430 Modified: trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c trunk/varnish-cache/lib/libvcl/vcc_obj.c Log: Updated generated files Modified: trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c 2010-01-06 18:49:39 UTC (rev 4429) +++ trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c 2010-01-06 18:50:18 UTC (rev 4430) @@ -1,5 +1,5 @@ /* - * $Id$ + * $Id: vcc_gen_fixed_token.tcl 4428 2010-01-06 17:38:59Z tfheen $ * * NB: This file is machine generated, DO NOT EDIT! * @@ -159,9 +159,10 @@ /* ../../include/vcl.h */ - vsb_cat(sb, "/*\n * $Id$\n *\n * NB: This file is machine generate"); - vsb_cat(sb, "d, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_tok"); - vsb_cat(sb, "en.tcl instead\n */\n\nstruct sess;\n"); + vsb_cat(sb, "/*\n * $Id: vcc_gen_fixed_token.tcl 4428 2010-01-06 17"); + vsb_cat(sb, ":38:59Z tfheen $\n *\n * NB: This file is machine gen"); + vsb_cat(sb, "erated, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixe"); + vsb_cat(sb, "d_token.tcl instead\n */\n\nstruct sess;\n"); vsb_cat(sb, "struct cli;\n\ntypedef void vcl_init_f(struct cli *);\n"); vsb_cat(sb, "typedef void vcl_fini_f(struct cli *);\n"); vsb_cat(sb, "typedef int vcl_func_f(struct sess *sp);\n"); @@ -226,16 +227,16 @@ vsb_cat(sb, " * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWI"); vsb_cat(sb, "SE) ARISING IN ANY WAY\n * OUT OF THE USE OF THIS SOFT"); vsb_cat(sb, "WARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n"); - vsb_cat(sb, " * SUCH DAMAGE.\n *\n * $Id$\n *\n"); - vsb_cat(sb, " * Runtime support for compiled VCL programs.\n"); - vsb_cat(sb, " *\n * XXX: When this file is changed, lib/libvcl/vcc_"); - vsb_cat(sb, "gen_fixed_token.tcl\n * XXX: *MUST* be rerun.\n"); - vsb_cat(sb, " */\n\nstruct sess;\nstruct vsb;\n"); - vsb_cat(sb, "struct cli;\nstruct director;\n"); - vsb_cat(sb, "struct VCL_conf;\nstruct sockaddr;\n"); - vsb_cat(sb, "\n/*\n * A backend probe specification\n"); - vsb_cat(sb, " */\n\nextern const void * const vrt_magic_string_end;"); - vsb_cat(sb, "\n\nstruct vrt_backend_probe {\n"); + vsb_cat(sb, " * SUCH DAMAGE.\n *\n * $Id: vrt.h 4428 2010-01-06 17:"); + vsb_cat(sb, "38:59Z tfheen $\n *\n * Runtime support for compiled V"); + vsb_cat(sb, "CL programs.\n *\n * XXX: When this file is changed, l"); + vsb_cat(sb, "ib/libvcl/vcc_gen_fixed_token.tcl\n"); + vsb_cat(sb, " * XXX: *MUST* be rerun.\n */\n"); + vsb_cat(sb, "\nstruct sess;\nstruct vsb;\nstruct cli;\n"); + vsb_cat(sb, "struct director;\nstruct VCL_conf;\n"); + vsb_cat(sb, "struct sockaddr;\n\n/*\n * A backend probe specificati"); + vsb_cat(sb, "on\n */\n\nextern const void * const vrt_magic_string_"); + vsb_cat(sb, "end;\n\nstruct vrt_backend_probe {\n"); vsb_cat(sb, "\tconst char\t*url;\n\tconst char\t*request;\n"); vsb_cat(sb, "\tdouble\t\ttimeout;\n\tdouble\t\tinterval;\n"); vsb_cat(sb, "\tunsigned\texp_status;\n\tunsigned\twindow;\n"); @@ -311,25 +312,27 @@ /* ../../include/vrt_obj.h */ - vsb_cat(sb, "/*\n * $Id$\n *\n * NB: This file is machine generate"); - vsb_cat(sb, "d, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixed_tok"); - vsb_cat(sb, "en.tcl instead\n */\n\nstruct sockaddr * VRT_r_client_"); - vsb_cat(sb, "ip(const struct sess *);\nstruct sockaddr * VRT_r_serv"); - vsb_cat(sb, "er_ip(struct sess *);\nconst char * VRT_r_server_hostn"); - vsb_cat(sb, "ame(struct sess *);\nconst char * VRT_r_server_identit"); - vsb_cat(sb, "y(struct sess *);\nint VRT_r_server_port(struct sess *"); - vsb_cat(sb, ");\nconst char * VRT_r_req_request(const struct sess *"); - vsb_cat(sb, ");\nvoid VRT_l_req_request(const struct sess *, const "); - vsb_cat(sb, "char *, ...);\nconst char * VRT_r_req_url(const struct"); - vsb_cat(sb, " sess *);\nvoid VRT_l_req_url(const struct sess *, con"); - vsb_cat(sb, "st char *, ...);\nconst char * VRT_r_req_proto(const s"); - vsb_cat(sb, "truct sess *);\nvoid VRT_l_req_proto(const struct sess"); - vsb_cat(sb, " *, const char *, ...);\nvoid VRT_l_req_hash(struct se"); - vsb_cat(sb, "ss *, const char *);\nstruct director * VRT_r_req_back"); - vsb_cat(sb, "end(struct sess *);\nvoid VRT_l_req_backend(struct ses"); - vsb_cat(sb, "s *, struct director *);\nint VRT_r_req_restarts(const"); - vsb_cat(sb, " struct sess *);\ndouble VRT_r_req_grace(struct sess *"); - vsb_cat(sb, ");\nvoid VRT_l_req_grace(struct sess *, double);\n"); + vsb_cat(sb, "/*\n * $Id: vcc_gen_fixed_token.tcl 4428 2010-01-06 17"); + vsb_cat(sb, ":38:59Z tfheen $\n *\n * NB: This file is machine gen"); + vsb_cat(sb, "erated, DO NOT EDIT!\n *\n * Edit and run vcc_gen_fixe"); + vsb_cat(sb, "d_token.tcl instead\n */\n\nstruct sockaddr * VRT_r_cl"); + vsb_cat(sb, "ient_ip(const struct sess *);\n"); + vsb_cat(sb, "struct sockaddr * VRT_r_server_ip(struct sess *);\n"); + vsb_cat(sb, "const char * VRT_r_server_hostname(struct sess *);\n"); + vsb_cat(sb, "const char * VRT_r_server_identity(struct sess *);\n"); + vsb_cat(sb, "int VRT_r_server_port(struct sess *);\n"); + vsb_cat(sb, "const char * VRT_r_req_request(const struct sess *);\n"); + vsb_cat(sb, "void VRT_l_req_request(const struct sess *, const char"); + vsb_cat(sb, " *, ...);\nconst char * VRT_r_req_url(const struct ses"); + vsb_cat(sb, "s *);\nvoid VRT_l_req_url(const struct sess *, const c"); + vsb_cat(sb, "har *, ...);\nconst char * VRT_r_req_proto(const struc"); + vsb_cat(sb, "t sess *);\nvoid VRT_l_req_proto(const struct sess *, "); + vsb_cat(sb, "const char *, ...);\nvoid VRT_l_req_hash(struct sess *"); + vsb_cat(sb, ", const char *);\nstruct director * VRT_r_req_backend("); + vsb_cat(sb, "struct sess *);\nvoid VRT_l_req_backend(struct sess *,"); + vsb_cat(sb, " struct director *);\nint VRT_r_req_restarts(const str"); + vsb_cat(sb, "uct sess *);\ndouble VRT_r_req_grace(struct sess *);\n"); + vsb_cat(sb, "void VRT_l_req_grace(struct sess *, double);\n"); vsb_cat(sb, "const char * VRT_r_req_xid(struct sess *);\n"); vsb_cat(sb, "unsigned VRT_r_req_esi(struct sess *);\n"); vsb_cat(sb, "void VRT_l_req_esi(struct sess *, unsigned);\n"); Modified: trunk/varnish-cache/lib/libvcl/vcc_obj.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_obj.c 2010-01-06 18:49:39 UTC (rev 4429) +++ trunk/varnish-cache/lib/libvcl/vcc_obj.c 2010-01-06 18:50:18 UTC (rev 4430) @@ -1,5 +1,5 @@ /* - * $Id$ + * $Id: vcc_gen_fixed_token.tcl 4428 2010-01-06 17:38:59Z tfheen $ * * NB: This file is machine generated, DO NOT EDIT! * From phk at projects.linpro.no Wed Jan 6 20:45:13 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 6 Jan 2010 21:45:13 +0100 (CET) Subject: r4431 - in trunk/varnish-cache: include lib/libvarnish Message-ID: <20100106204513.292761F7BA6@projects.linpro.no> Author: phk Date: 2010-01-06 21:45:12 +0100 (Wed, 06 Jan 2010) New Revision: 4431 Modified: trunk/varnish-cache/include/vct.h trunk/varnish-cache/lib/libvarnish/vct.c Log: Add ability to tell if a char is valid "XML Name" or "XML Name Start". Modified: trunk/varnish-cache/include/vct.h =================================================================== --- trunk/varnish-cache/include/vct.h 2010-01-06 18:50:18 UTC (rev 4430) +++ trunk/varnish-cache/include/vct.h 2010-01-06 20:45:12 UTC (rev 4431) @@ -31,16 +31,18 @@ /* from libvarnish/vct.c */ -#define VCT_SP (1<<0) -#define VCT_CRLF (1<<1) -#define VCT_LWS (VCT_CRLF | VCT_SP) -#define VCT_CTL (1<<2) -#define VCT_ALPHA (1<<3) -#define VCT_SEPARATOR (1<<4) -#define VCT_DIGIT (1<<5) -#define VCT_HEX (1<<6) +#define VCT_SP (1<<0) +#define VCT_CRLF (1<<1) +#define VCT_LWS (VCT_CRLF | VCT_SP) +#define VCT_CTL (1<<2) +#define VCT_ALPHA (1<<3) +#define VCT_SEPARATOR (1<<4) +#define VCT_DIGIT (1<<5) +#define VCT_HEX (1<<6) +#define VCT_XMLNAMESTART (1<<7) +#define VCT_XMLNAME (1<<8) -extern const unsigned char vct_typtab[256]; +extern const uint16_t vct_typtab[256]; static inline int vct_is(unsigned char x, unsigned char y) @@ -56,6 +58,8 @@ #define vct_isalpha(x) vct_is(x, VCT_ALPHA) #define vct_issep(x) vct_is(x, VCT_SEPARATOR) #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL) +#define vct_isxmlnamestart(x) vct_is(x, VCT_XMLNAMES) +#define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNM) /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */ #define vct_skipcrlf(p) (p[0] == 0x0d && p[1] == 0x0a ? 2 : 1) Modified: trunk/varnish-cache/lib/libvarnish/vct.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/vct.c 2010-01-06 18:50:18 UTC (rev 4430) +++ trunk/varnish-cache/lib/libvarnish/vct.c 2010-01-06 20:45:12 UTC (rev 4431) @@ -33,6 +33,7 @@ #include "svnid.h" SVNID("$Id$") +#include #include /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */ @@ -40,7 +41,7 @@ #define VCT_UPALPHA VCT_ALPHA #define VCT_LOALPHA VCT_ALPHA -unsigned const char vct_typtab[256] = { +const uint16_t vct_typtab[256] = { [0x00] = VCT_CTL, [0x01] = VCT_CTL, [0x02] = VCT_CTL, @@ -78,80 +79,146 @@ [0x28] = VCT_SEPARATOR, [0x29] = VCT_SEPARATOR, [0x2c] = VCT_SEPARATOR, + [0x2d] = VCT_XMLNAME, + [0x2e] = VCT_XMLNAME, [0x2f] = VCT_SEPARATOR, - [0x30] = VCT_DIGIT | VCT_HEX, - [0x31] = VCT_DIGIT | VCT_HEX, - [0x32] = VCT_DIGIT | VCT_HEX, - [0x33] = VCT_DIGIT | VCT_HEX, - [0x34] = VCT_DIGIT | VCT_HEX, - [0x35] = VCT_DIGIT | VCT_HEX, - [0x36] = VCT_DIGIT | VCT_HEX, - [0x37] = VCT_DIGIT | VCT_HEX, - [0x38] = VCT_DIGIT | VCT_HEX, - [0x39] = VCT_DIGIT | VCT_HEX, - [0x3a] = VCT_SEPARATOR, + [0x30] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x31] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x32] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x33] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x34] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x35] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x36] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x37] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x38] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x39] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x3a] = VCT_SEPARATOR | VCT_XMLNAMESTART, [0x3b] = VCT_SEPARATOR, [0x3c] = VCT_SEPARATOR, [0x3d] = VCT_SEPARATOR, [0x3e] = VCT_SEPARATOR, [0x3f] = VCT_SEPARATOR, [0x40] = VCT_SEPARATOR, - [0x41] = VCT_UPALPHA | VCT_HEX, - [0x42] = VCT_UPALPHA | VCT_HEX, - [0x43] = VCT_UPALPHA | VCT_HEX, - [0x44] = VCT_UPALPHA | VCT_HEX, - [0x45] = VCT_UPALPHA | VCT_HEX, - [0x46] = VCT_UPALPHA | VCT_HEX, - [0x47] = VCT_UPALPHA, - [0x48] = VCT_UPALPHA, - [0x49] = VCT_UPALPHA, - [0x4a] = VCT_UPALPHA, - [0x4b] = VCT_UPALPHA, - [0x4c] = VCT_UPALPHA, - [0x4d] = VCT_UPALPHA, - [0x4e] = VCT_UPALPHA, - [0x4f] = VCT_UPALPHA, - [0x50] = VCT_UPALPHA, - [0x51] = VCT_UPALPHA, - [0x52] = VCT_UPALPHA, - [0x53] = VCT_UPALPHA, - [0x54] = VCT_UPALPHA, - [0x55] = VCT_UPALPHA, - [0x56] = VCT_UPALPHA, - [0x57] = VCT_UPALPHA, - [0x58] = VCT_UPALPHA, - [0x59] = VCT_UPALPHA, - [0x5a] = VCT_UPALPHA, + [0x41] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x42] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x43] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x44] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x45] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x46] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x47] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x48] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x49] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4a] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4b] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4c] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4d] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4e] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x4f] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x50] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x51] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x52] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x53] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x54] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x55] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x56] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x57] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x58] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x59] = VCT_UPALPHA | VCT_XMLNAMESTART, + [0x5a] = VCT_UPALPHA | VCT_XMLNAMESTART, [0x5b] = VCT_SEPARATOR, [0x5c] = VCT_SEPARATOR, [0x5d] = VCT_SEPARATOR, - [0x61] = VCT_LOALPHA | VCT_HEX, - [0x62] = VCT_LOALPHA | VCT_HEX, - [0x63] = VCT_LOALPHA | VCT_HEX, - [0x64] = VCT_LOALPHA | VCT_HEX, - [0x65] = VCT_LOALPHA | VCT_HEX, - [0x66] = VCT_LOALPHA | VCT_HEX, - [0x67] = VCT_LOALPHA, - [0x68] = VCT_LOALPHA, - [0x69] = VCT_LOALPHA, - [0x6a] = VCT_LOALPHA, - [0x6b] = VCT_LOALPHA, - [0x6c] = VCT_LOALPHA, - [0x6d] = VCT_LOALPHA, - [0x6e] = VCT_LOALPHA, - [0x6f] = VCT_LOALPHA, - [0x70] = VCT_LOALPHA, - [0x71] = VCT_LOALPHA, - [0x72] = VCT_LOALPHA, - [0x73] = VCT_LOALPHA, - [0x74] = VCT_LOALPHA, - [0x75] = VCT_LOALPHA, - [0x76] = VCT_LOALPHA, - [0x77] = VCT_LOALPHA, - [0x78] = VCT_LOALPHA, - [0x79] = VCT_LOALPHA, - [0x7a] = VCT_LOALPHA, + [0x5f] = VCT_XMLNAMESTART, + [0x61] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x62] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x63] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x64] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x65] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x66] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, + [0x67] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x68] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x69] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6a] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6b] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6c] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6d] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6e] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x6f] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x70] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x71] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x72] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x73] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x74] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x75] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x76] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x77] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x78] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x79] = VCT_LOALPHA | VCT_XMLNAMESTART, + [0x7a] = VCT_LOALPHA | VCT_XMLNAMESTART, [0x7b] = VCT_SEPARATOR, [0x7d] = VCT_SEPARATOR, [0x7f] = VCT_CTL, + [0xb7] = VCT_XMLNAME, + [0xc0] = VCT_XMLNAMESTART, + [0xc1] = VCT_XMLNAMESTART, + [0xc2] = VCT_XMLNAMESTART, + [0xc3] = VCT_XMLNAMESTART, + [0xc4] = VCT_XMLNAMESTART, + [0xc5] = VCT_XMLNAMESTART, + [0xc6] = VCT_XMLNAMESTART, + [0xc7] = VCT_XMLNAMESTART, + [0xc8] = VCT_XMLNAMESTART, + [0xc9] = VCT_XMLNAMESTART, + [0xca] = VCT_XMLNAMESTART, + [0xcb] = VCT_XMLNAMESTART, + [0xcc] = VCT_XMLNAMESTART, + [0xcd] = VCT_XMLNAMESTART, + [0xce] = VCT_XMLNAMESTART, + [0xcf] = VCT_XMLNAMESTART, + [0xd0] = VCT_XMLNAMESTART, + [0xd1] = VCT_XMLNAMESTART, + [0xd2] = VCT_XMLNAMESTART, + [0xd3] = VCT_XMLNAMESTART, + [0xd4] = VCT_XMLNAMESTART, + [0xd5] = VCT_XMLNAMESTART, + [0xd6] = VCT_XMLNAMESTART, + [0xd8] = VCT_XMLNAMESTART, + [0xd9] = VCT_XMLNAMESTART, + [0xda] = VCT_XMLNAMESTART, + [0xdb] = VCT_XMLNAMESTART, + [0xdc] = VCT_XMLNAMESTART, + [0xdd] = VCT_XMLNAMESTART, + [0xde] = VCT_XMLNAMESTART, + [0xdf] = VCT_XMLNAMESTART, + [0xe0] = VCT_XMLNAMESTART, + [0xe1] = VCT_XMLNAMESTART, + [0xe2] = VCT_XMLNAMESTART, + [0xe3] = VCT_XMLNAMESTART, + [0xe4] = VCT_XMLNAMESTART, + [0xe5] = VCT_XMLNAMESTART, + [0xe6] = VCT_XMLNAMESTART, + [0xe7] = VCT_XMLNAMESTART, + [0xe8] = VCT_XMLNAMESTART, + [0xe9] = VCT_XMLNAMESTART, + [0xea] = VCT_XMLNAMESTART, + [0xeb] = VCT_XMLNAMESTART, + [0xec] = VCT_XMLNAMESTART, + [0xed] = VCT_XMLNAMESTART, + [0xee] = VCT_XMLNAMESTART, + [0xef] = VCT_XMLNAMESTART, + [0xf0] = VCT_XMLNAMESTART, + [0xf1] = VCT_XMLNAMESTART, + [0xf2] = VCT_XMLNAMESTART, + [0xf3] = VCT_XMLNAMESTART, + [0xf4] = VCT_XMLNAMESTART, + [0xf5] = VCT_XMLNAMESTART, + [0xf6] = VCT_XMLNAMESTART, + [0xf8] = VCT_XMLNAMESTART, + [0xf9] = VCT_XMLNAMESTART, + [0xfa] = VCT_XMLNAMESTART, + [0xfb] = VCT_XMLNAMESTART, + [0xfc] = VCT_XMLNAMESTART, + [0xfd] = VCT_XMLNAMESTART, + [0xfe] = VCT_XMLNAMESTART, + [0xff] = VCT_XMLNAMESTART, }; From phk at projects.linpro.no Thu Jan 7 11:28:24 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 7 Jan 2010 12:28:24 +0100 (CET) Subject: r4432 - trunk/varnish-cache/include Message-ID: <20100107112824.EACE01F7BA6@projects.linpro.no> Author: phk Date: 2010-01-07 12:28:24 +0100 (Thu, 07 Jan 2010) New Revision: 4432 Modified: trunk/varnish-cache/include/vct.h Log: Finish yesterdays work, adding XML charset rules to VCT. Modified: trunk/varnish-cache/include/vct.h =================================================================== --- trunk/varnish-cache/include/vct.h 2010-01-06 20:45:12 UTC (rev 4431) +++ trunk/varnish-cache/include/vct.h 2010-01-07 11:28:24 UTC (rev 4432) @@ -45,7 +45,7 @@ extern const uint16_t vct_typtab[256]; static inline int -vct_is(unsigned char x, unsigned char y) +vct_is(unsigned char x, uint16_t y) { return (vct_typtab[x] & (y)); @@ -58,8 +58,8 @@ #define vct_isalpha(x) vct_is(x, VCT_ALPHA) #define vct_issep(x) vct_is(x, VCT_SEPARATOR) #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL) -#define vct_isxmlnamestart(x) vct_is(x, VCT_XMLNAMES) -#define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNM) +#define vct_isxmlnamestart(x) vct_is(x, VCT_XMLNAMESTART) +#define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNAME) /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */ #define vct_skipcrlf(p) (p[0] == 0x0d && p[1] == 0x0a ? 2 : 1) From phk at projects.linpro.no Thu Jan 7 11:29:28 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 7 Jan 2010 12:29:28 +0100 (CET) Subject: r4433 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100107112928.94C241F7BAA@projects.linpro.no> Author: phk Date: 2010-01-07 12:29:28 +0100 (Thu, 07 Jan 2010) New Revision: 4433 Modified: trunk/varnish-cache/bin/varnishtest/tests/e00008.vtc Log: As a matter of fact, a period is a valid character in a XML 1.0 attribute name. A question mark is not. Modified: trunk/varnish-cache/bin/varnishtest/tests/e00008.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/e00008.vtc 2010-01-07 11:28:24 UTC (rev 4432) +++ trunk/varnish-cache/bin/varnishtest/tests/e00008.vtc 2010-01-07 11:29:28 UTC (rev 4433) @@ -34,7 +34,7 @@ > 25 26 27 - 28 + 28 29 30 31 From phk at projects.linpro.no Thu Jan 7 11:29:57 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 7 Jan 2010 12:29:57 +0100 (CET) Subject: r4434 - trunk/varnish-cache/bin/varnishd Message-ID: <20100107112957.2441A1F7BB1@projects.linpro.no> Author: phk Date: 2010-01-07 12:29:56 +0100 (Thu, 07 Jan 2010) New Revision: 4434 Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c Log: Correctly check the XML 1.0 names for illegal characters. Fixes #207 Modified: trunk/varnish-cache/bin/varnishd/cache_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-07 11:29:28 UTC (rev 4433) +++ trunk/varnish-cache/bin/varnishd/cache_esi.c 2010-01-07 11:29:56 UTC (rev 4434) @@ -53,6 +53,7 @@ #include "shmlog.h" #include "vrt.h" #include "vcl.h" +#include "vct.h" #include "cache.h" #include "stevedore.h" @@ -314,16 +315,17 @@ if (in->b >= in->e) return (0); - if (!isalpha(*in->b)) { + if (!vct_isxmlnamestart(*in->b)) { /* XXX error */ - esi_error(ew, in->b, 1, "XML 1.0 Illegal attribute character"); + esi_error(ew, in->b, 1, + "XML 1.0 Illegal attribute start character"); return (-1); } /* Attribute name until '=' or space */ *attrib = *in; while(in->b < in->e && *in->b != '=' && !isspace(*in->b)) { - if (!isalnum(*in->b)) { + if (!vct_isxmlname(*in->b)) { esi_error(ew, attrib->b, 1 + (in->b - attrib->b), "XML 1.0 Illegal attribute character"); return (-1); From phk at phk.freebsd.dk Thu Jan 7 11:32:35 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 07 Jan 2010 11:32:35 +0000 Subject: r4434 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Thu, 07 Jan 2010 12:29:57 +0100." <20100107112957.2441A1F7BB1@projects.linpro.no> Message-ID: <9177.1262863955@critter.freebsd.dk> In message <20100107112957.2441A1F7BB1 at projects.linpro.no>, phk at projects.linpro .no writes: >Author: phk >Date: 2010-01-07 12:29:56 +0100 (Thu, 07 Jan 2010) >New Revision: 4434 > >Modified: > trunk/varnish-cache/bin/varnishd/cache_esi.c >Log: >Correctly check the XML 1.0 names for illegal characters. > >Fixes #207 That was a a typo: meant #607 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ssm at projects.linpro.no Fri Jan 8 09:13:17 2010 From: ssm at projects.linpro.no (ssm at projects.linpro.no) Date: Fri, 8 Jan 2010 10:13:17 +0100 (CET) Subject: r4435 - trunk/varnish-cache/man Message-ID: <20100108091317.8DE751F7BD2@projects.linpro.no> Author: ssm Date: 2010-01-08 10:13:17 +0100 (Fri, 08 Jan 2010) New Revision: 4435 Modified: trunk/varnish-cache/man/vcl.7so Log: Document multiple subroutine definitions a bit more Modified: trunk/varnish-cache/man/vcl.7so =================================================================== --- trunk/varnish-cache/man/vcl.7so 2010-01-07 11:29:56 UTC (rev 4434) +++ trunk/varnish-cache/man/vcl.7so 2010-01-08 09:13:17 UTC (rev 4435) @@ -282,9 +282,6 @@ .Pp Subroutines in VCL do not take arguments, nor do they return values. .Pp -If multiple subroutines with the same name are defined, they are -concatenated in the order in which the appear in the source. -.Pp To call a subroutine, use the .Cm call keyword followed by the subroutine's name: @@ -467,6 +464,36 @@ See the .Sx EXAMPLES section for a listing of the default code. +.Ss Multiple subroutines +If multiple subroutines with the same name are defined, they are +concatenated in the order in which the appear in the source. +.Pp +Example: +.Bd -literal -offset 4n +# in file "main.vcl" +include "backends.vcl"; +include "purge.vcl"; + +# in file "backends.vcl" +sub vcl_recv { + if (req.http.host ~ "example.com") { + set req.backend = foo; + } elsif (req.http.host ~ "example.org") { + set req.backend = bar; + } +} + +# in file "purge.vcl" +sub vcl_recv { + if (client.ip ~ admin_network) { + if (req.http.Cache-Control ~ "no-cache") { + purge_url(req.url); + } + } +} +.Ed +.Pp +The builtin default subroutines are implicitly appended in this way. .Ss Variables Although subroutines take no arguments, the necessary information is made available to the handler subroutines through global variables. From tfheen at projects.linpro.no Fri Jan 8 09:36:17 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Fri, 8 Jan 2010 10:36:17 +0100 (CET) Subject: r4436 - in trunk/varnish-cache: bin/varnishd bin/varnishtest lib/libvcl Message-ID: <20100108093617.D26C91F7BDA@projects.linpro.no> Author: tfheen Date: 2010-01-08 10:36:17 +0100 (Fri, 08 Jan 2010) New Revision: 4436 Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/bin/varnishd/mgt_vcc.c trunk/varnish-cache/bin/varnishd/stevedore_utils.c trunk/varnish-cache/bin/varnishd/storage_file.c trunk/varnish-cache/bin/varnishd/varnishd.c trunk/varnish-cache/bin/varnishtest/vtc.c trunk/varnish-cache/bin/varnishtest/vtc_varnish.c trunk/varnish-cache/lib/libvcl/vcc_acl.c trunk/varnish-cache/lib/libvcl/vcc_var.c Log: asprintf cleanup glibc and FreeBSD libc differ in how they handle asprintf allocation failures. Make sure we handle both cases properly and clean up the includes a bit. Fixes #334 Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -46,9 +46,7 @@ #include #include -#ifndef HAVE_VASPRINTF #include "compat/vasprintf.h" -#endif #ifndef HAVE_SRANDOMDEV #include "compat/srandomdev.h" @@ -593,7 +591,7 @@ TCP_myname(ev->fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1); TCP_name((void*)&addr, addrlen, abuf2, sizeof abuf2, pbuf2, sizeof pbuf2); - asprintf(&p, "telnet %s:%s %s:%s", abuf2, pbuf2, abuf1, pbuf1); + assert(asprintf(&p, "telnet %s:%s %s:%s", abuf2, pbuf2, abuf1, pbuf1) > 0); XXXAN(p); (void)telnet_new(i); Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -44,9 +44,7 @@ #include #include -#ifndef HAVE_ASPRINTF #include "compat/asprintf.h" -#endif #include "vsb.h" #include "libvcl.h" Modified: trunk/varnish-cache/bin/varnishd/stevedore_utils.c =================================================================== --- trunk/varnish-cache/bin/varnishd/stevedore_utils.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishd/stevedore_utils.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -55,6 +55,7 @@ #include #endif +#include "compat/asprintf.h" #include "mgt.h" #include "stevedore.h" @@ -106,7 +107,7 @@ ctx, fn); if (S_ISDIR(st.st_mode)) { - asprintf(&q, "%s/varnish.XXXXXX", fn); + xxxassert(asprintf(&q, "%s/varnish.XXXXXX", fn) > 0); XXXAN(q); fd = mkstemp(q); if (fd < 0) Modified: trunk/varnish-cache/bin/varnishd/storage_file.c =================================================================== --- trunk/varnish-cache/bin/varnishd/storage_file.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishd/storage_file.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -46,10 +46,6 @@ #include #include -#ifndef HAVE_ASPRINTF -#include "compat/asprintf.h" -#endif - #include "shmlog.h" #include "cache.h" #include "stevedore.h" Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -48,6 +48,7 @@ #include #include +#include "compat/asprintf.h" #include "compat/daemon.h" #ifndef HAVE_STRLCPY @@ -435,10 +436,11 @@ FILE *fi; uintptr_t a; struct symbols *s; + int i; p = NULL; - asprintf(&p, "nm -an %s 2>/dev/null", a0); - if (p == NULL) + i = asprintf(&p, "nm -an %s 2>/dev/null", a0); + if (i < 0 || p == NULL) return; fi = popen(p, "r"); free(p); Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -40,6 +40,7 @@ #include #include #include +#include "compat/asprintf.h" #include "libvarnish.h" #include "vsb.h" @@ -106,7 +107,7 @@ va_start(ap, fmt); free(m->val); m->val = NULL; - (void)vasprintf(&m->val, fmt, ap); + assert(vasprintf(&m->val, fmt, ap) >= 0); va_end(ap); AN(m->val); vtc_log(vl, 4, "macro def %s=%s", name, m->val); Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -46,6 +46,7 @@ #include #include +#include "compat/asprintf.h" #include "vqueue.h" #include "miniobj.h" #include "libvarnish.h" @@ -153,12 +154,13 @@ REPLACE(v->name, name); if (getuid() == 0) - (void)asprintf(&v->workdir, "/tmp/__%s", name); + assert(asprintf(&v->workdir, "/tmp/__%s", name) >= 0); else - (void)asprintf(&v->workdir, "/tmp/__%s.%d", name, getuid()); + assert(asprintf(&v->workdir, "/tmp/__%s.%d", name, getuid()) >= 0); AN(v->workdir); - (void)asprintf(&c, "rm -rf %s ; mkdir -p %s", v->workdir, v->workdir); + assert(asprintf(&c, "rm -rf %s ; mkdir -p %s", v->workdir, v->workdir) >= 0); + AN(c); AZ(system(c)); v->vl = vtc_logopen(name); Modified: trunk/varnish-cache/lib/libvcl/vcc_acl.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_acl.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/lib/libvcl/vcc_acl.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -47,6 +47,7 @@ #include "vcc_priv.h" #include "vcc_compile.h" #include "libvarnish.h" +#include "compat/vasprintf.h" struct acl_e { VTAILQ_ENTRY(acl_e) list; @@ -472,8 +473,8 @@ VTAILQ_INIT(&tl->acl); tcond = tl->t->tok; vcc_NextToken(tl); - asprintf(&acln, "%u", tl->cnt); - assert(acln != NULL); + assert(asprintf(&acln, "%u", tl->cnt) > 0); + AN(acln); vcc_acl_entry(tl); vcc_acl_emit(tl, acln, 1); Fb(tl, 1, "%smatch_acl_anon_%s(sp, %s)\n", @@ -504,8 +505,8 @@ vcc_NextToken(tl); vcc_AddDef(tl, an, R_ACL); - asprintf(&acln, "%.*s", PF(an)); - assert(acln != NULL); + assert(asprintf(&acln, "%.*s", PF(an)) > 0); + AN(acln); ExpectErr(tl, '{'); vcc_NextToken(tl); Modified: trunk/varnish-cache/lib/libvcl/vcc_var.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_var.c 2010-01-08 09:13:17 UTC (rev 4435) +++ trunk/varnish-cache/lib/libvcl/vcc_var.c 2010-01-08 09:36:17 UTC (rev 4436) @@ -40,6 +40,7 @@ #include "vcc_priv.h" #include "vcc_compile.h" #include "libvarnish.h" +#include "compat/vasprintf.h" /*--------------------------------------------------------------------*/ @@ -64,13 +65,13 @@ v->fmt = STRING; v->hdr = vh->hdr; v->methods = vh->methods; - asprintf(&p, "VRT_GetHdr(sp, %s, \"\\%03o%s:\")", v->hdr, - (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len); + assert(asprintf(&p, "VRT_GetHdr(sp, %s, \"\\%03o%s:\")", v->hdr, + (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len) > 0); AN(p); TlFree(tl, p); v->rname = p; - asprintf(&p, "VRT_SetHdr(sp, %s, \"\\%03o%s:\", ", v->hdr, - (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len); + assert(asprintf(&p, "VRT_SetHdr(sp, %s, \"\\%03o%s:\", ", v->hdr, + (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len) > 0); AN(p); TlFree(tl, p); v->lname = p; From phk at projects.linpro.no Fri Jan 8 10:17:32 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 8 Jan 2010 11:17:32 +0100 (CET) Subject: r4437 - trunk/varnish-cache/bin/varnishd Message-ID: <20100108101732.7CCDB1F7BDD@projects.linpro.no> Author: phk Date: 2010-01-08 11:17:32 +0100 (Fri, 08 Jan 2010) New Revision: 4437 Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c Log: Break down and lock the n_vbe_conn counter to avoid over & underflows. Fixes #604 Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-08 09:36:17 UTC (rev 4436) +++ trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-08 10:17:32 UTC (rev 4437) @@ -84,7 +84,9 @@ VSL_stats->backend_unused++; Lck_Unlock(&VBE_mtx); } else { + Lck_Lock(&VBE_mtx); VSL_stats->n_vbe_conn--; + Lck_Unlock(&VBE_mtx); free(vc); } } @@ -211,7 +213,9 @@ XXXAN(vc); vc->magic = VBE_CONN_MAGIC; vc->fd = -1; + Lck_Lock(&VBE_mtx); VSL_stats->n_vbe_conn++; + Lck_Unlock(&VBE_mtx); return (vc); } From knan at projects.linpro.no Fri Jan 8 12:27:50 2010 From: knan at projects.linpro.no (knan at projects.linpro.no) Date: Fri, 8 Jan 2010 13:27:50 +0100 (CET) Subject: r4438 - trunk/varnish-tools/munin Message-ID: <20100108122750.DF4C61F7BF1@projects.linpro.no> Author: knan Date: 2010-01-08 13:27:50 +0100 (Fri, 08 Jan 2010) New Revision: 4438 Modified: trunk/varnish-tools/munin/varnish_ Log: munin/varnish_: backend_unhealthy is a DERIVE value Modified: trunk/varnish-tools/munin/varnish_ =================================================================== --- trunk/varnish-tools/munin/varnish_ 2010-01-08 10:17:32 UTC (rev 4437) +++ trunk/varnish-tools/munin/varnish_ 2010-01-08 12:27:50 UTC (rev 4438) @@ -181,8 +181,9 @@ 'min' => '0' }, 'backend_unhealthy' => { - 'colour' => 'FF55FF', - 'type' => 'GAUGE' + 'type' => 'DERIVE', + 'min' => '0', + 'colour' => 'FF55FF' }, 's_pipe' => { 'type' => 'DERIVE', @@ -237,7 +238,7 @@ 'min' => '0' }, 'backend_unhealthy' => { - 'type' => 'GAUGE', + 'type' => 'DERIVE', 'min' => '0', 'warning' => ':1' }, @@ -692,7 +693,7 @@ # Returns false if non-existant. # # Output is formatted for plugins if arg4 is blank, otherwise arg4 is used -# as the title/name of the field (ie: arg4=graph_titel). +# as the title/name of the field (ie: arg4=graph_title). sub print_if_exist { my %values = %{$_[0]}; From knan at projects.linpro.no Fri Jan 8 12:27:53 2010 From: knan at projects.linpro.no (knan at projects.linpro.no) Date: Fri, 8 Jan 2010 13:27:53 +0100 (CET) Subject: r4439 - trunk/varnish-tools/munin Message-ID: <20100108122753.065FA1F7BFB@projects.linpro.no> Author: knan Date: 2010-01-08 13:27:52 +0100 (Fri, 08 Jan 2010) New Revision: 4439 Modified: trunk/varnish-tools/munin/varnish_ Log: munin/varnish_: note that canonical upstream is the varnish project Modified: trunk/varnish-tools/munin/varnish_ =================================================================== --- trunk/varnish-tools/munin/varnish_ 2010-01-08 12:27:50 UTC (rev 4438) +++ trunk/varnish-tools/munin/varnish_ 2010-01-08 12:27:52 UTC (rev 4439) @@ -53,7 +53,7 @@ You can link them yourself with something like this: ln -s /foo/varnish_ /etc/munin/plugins/varnish_data_structures -=head1 INTERPERTATION +=head1 INTERPRETATION Each graph uses data from varnishstat. @@ -71,6 +71,13 @@ The hit_rate graph requires munin r2040 or newer to display correctly. +=head1 PATCHES-TO + +The varnish-tools repository is the canonical upstream for this plugin. +Please send patches to Kristian Lyngstol +and/or varnish-misc at projects.linpro.no for inclusion before adding to +munin trunk. + =head1 AUTHOR Kristian Lyngstol @@ -81,9 +88,7 @@ =cut -# FIXME: License? - use strict; # Set to 1 to enable output when a variable is defined in a graph but From phk at projects.linpro.no Fri Jan 8 16:53:38 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 8 Jan 2010 17:53:38 +0100 (CET) Subject: r4440 - in trunk/varnish-cache: bin/varnishd lib/libvarnish lib/libvcl Message-ID: <20100108165338.691FD383D9@projects.linpro.no> Author: phk Date: 2010-01-08 17:53:38 +0100 (Fri, 08 Jan 2010) New Revision: 4440 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_vary.c trunk/varnish-cache/bin/varnishd/flint.lnt trunk/varnish-cache/lib/libvarnish/time.c trunk/varnish-cache/lib/libvcl/vcc_backend.c Log: Various flexelint nits. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-08 12:27:52 UTC (rev 4439) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-08 16:53:38 UTC (rev 4440) @@ -654,7 +654,7 @@ void RES_WriteObj(struct sess *sp); /* cache_vary.c */ -struct vsb *VRY_Create(const struct sess *sp, struct http *hp); +struct vsb *VRY_Create(const struct sess *sp, const struct http *hp); int VRY_Match(const struct sess *sp, const unsigned char *vary); /* cache_vcl.c */ Modified: trunk/varnish-cache/bin/varnishd/cache_vary.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vary.c 2010-01-08 12:27:52 UTC (rev 4439) +++ trunk/varnish-cache/bin/varnishd/cache_vary.c 2010-01-08 16:53:38 UTC (rev 4440) @@ -64,7 +64,7 @@ #include "cache.h" struct vsb * -VRY_Create(const struct sess *sp, struct http *hp) +VRY_Create(const struct sess *sp, const struct http *hp) { char *v, *p, *q, *h, *e; struct vsb *sb, *sbh; Modified: trunk/varnish-cache/bin/varnishd/flint.lnt =================================================================== --- trunk/varnish-cache/bin/varnishd/flint.lnt 2010-01-08 12:27:52 UTC (rev 4439) +++ trunk/varnish-cache/bin/varnishd/flint.lnt 2010-01-08 16:53:38 UTC (rev 4440) @@ -203,6 +203,8 @@ -esym(528, vcl_handlingname) // Not referenced -e641 // Converting enum 'cli_status_e' to int +-e441 // for clause irregularity: loop variable '___' not found in 2nd for expression + // Review all below this line /////////////////////////////////////////////// -e732 // 183 Loss of sign (___) (___ to ___) Modified: trunk/varnish-cache/lib/libvarnish/time.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/time.c 2010-01-08 12:27:52 UTC (rev 4439) +++ trunk/varnish-cache/lib/libvarnish/time.c 2010-01-08 16:53:38 UTC (rev 4440) @@ -163,8 +163,8 @@ { struct timespec ts; - ts.tv_sec = floor(t); - ts.tv_nsec = floor((t - ts.tv_sec) * 1e9); + ts.tv_sec = (time_t)floor(t); + ts.tv_nsec = (long)floor((t - ts.tv_sec) * 1e9); #ifdef HAVE_NANOSLEEP (void)nanosleep(&ts, NULL); Modified: trunk/varnish-cache/lib/libvcl/vcc_backend.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_backend.c 2010-01-08 12:27:52 UTC (rev 4439) +++ trunk/varnish-cache/lib/libvcl/vcc_backend.c 2010-01-08 16:53:38 UTC (rev 4440) @@ -100,7 +100,7 @@ Emit_Sockaddr(struct tokenlist *tl, const struct token *t_host, const char *port) { - struct addrinfo *res, *res0, hint; + struct addrinfo *res, *res0, *res1, hint; int n4, n6, len, error, retval; const char *emit, *multiple; unsigned char *u; @@ -138,9 +138,9 @@ "Please specify which exact address " "you want to use, we found these:\n", PF(t_host), multiple); - for (res = res0; res != NULL; res = res->ai_next) { - error = getnameinfo(res->ai_addr, - res->ai_addrlen, hbuf, sizeof hbuf, + for (res1 = res0; res1 != NULL; res1 = res1->ai_next) { + error = getnameinfo(res1->ai_addr, + res1->ai_addrlen, hbuf, sizeof hbuf, NULL, 0, NI_NUMERICHOST); AZ(error); vsb_printf(tl->sb, "\t%s\n", hbuf); @@ -190,7 +190,7 @@ */ static void -vcc_EmitBeIdent(struct tokenlist *tl, struct vsb *v, +vcc_EmitBeIdent(const struct tokenlist *tl, struct vsb *v, int serial, const struct token *first, const struct token *last) { From phk at projects.linpro.no Fri Jan 8 17:13:10 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 8 Jan 2010 18:13:10 +0100 (CET) Subject: r4441 - trunk/varnish-cache/bin/varnishd Message-ID: <20100108171310.C98A71F7501@projects.linpro.no> Author: phk Date: 2010-01-08 18:13:10 +0100 (Fri, 08 Jan 2010) New Revision: 4441 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_backend.c trunk/varnish-cache/bin/varnishd/cache_backend.h trunk/varnish-cache/bin/varnishd/cache_center.c trunk/varnish-cache/bin/varnishd/cache_dir_random.c trunk/varnish-cache/bin/varnishd/cache_dir_round_robin.c Log: More flexelint silencing Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-08 17:13:10 UTC (rev 4441) @@ -456,8 +456,8 @@ /* cache_backend.c */ -struct vbe_conn *VBE_GetFd(struct director *, struct sess *sp); -int VBE_Healthy(struct director *, const struct sess *sp); +struct vbe_conn *VBE_GetFd(const struct director *, struct sess *sp); +int VBE_Healthy(const struct director *, const struct sess *sp); void VBE_ClosedFd(struct sess *sp); void VBE_RecycleFd(struct sess *sp); void VBE_AddHostHeader(const struct sess *sp); Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-08 17:13:10 UTC (rev 4441) @@ -402,7 +402,7 @@ /* Get a connection --------------------------------------------------*/ struct vbe_conn * -VBE_GetFd(struct director *d, struct sess *sp) +VBE_GetFd(const struct director *d, struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); @@ -415,7 +415,7 @@ /* Cheack health -----------------------------------------------------*/ int -VBE_Healthy(struct director *d, const struct sess *sp) +VBE_Healthy(const struct director *d, const struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); @@ -439,7 +439,7 @@ }; static struct vbe_conn * -vdi_simple_getfd(struct director *d, struct sess *sp) +vdi_simple_getfd(const struct director *d, struct sess *sp) { struct vdi_simple *vs; @@ -450,7 +450,7 @@ } static unsigned -vdi_simple_healthy(struct director *d, const struct sess *sp) +vdi_simple_healthy(const struct director *d, const struct sess *sp) { struct vdi_simple *vs; Modified: trunk/varnish-cache/bin/varnishd/cache_backend.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.h 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache_backend.h 2010-01-08 17:13:10 UTC (rev 4441) @@ -77,9 +77,9 @@ * backends to use. */ -typedef struct vbe_conn *vdi_getfd_f(struct director *, struct sess *sp); +typedef struct vbe_conn *vdi_getfd_f(const struct director *, struct sess *sp); typedef void vdi_fini_f(struct director *); -typedef unsigned vdi_healthy(struct director *, const struct sess *sp); +typedef unsigned vdi_healthy(const struct director *, const struct sess *sp); struct director { unsigned magic; Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-08 17:13:10 UTC (rev 4441) @@ -422,7 +422,8 @@ struct http *hp, *hp2; char *b; unsigned handling, l; - struct vsb *vary; + int varyl = 0; + struct vsb *vary = NULL; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC); @@ -519,16 +520,19 @@ CHECK_OBJ_NOTNULL(sp->objhead, OBJHEAD_MAGIC); CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC); vary = VRY_Create(sp, sp->wrk->beresp); + if (vary != NULL) { + varyl = vsb_len(vary); + assert(varyl > 0); + } } else { AZ(sp->objhead); AZ(sp->objcore); - vary = NULL; } l = http_EstimateWS(sp->wrk->beresp, HTTPH_A_INS); if (vary != NULL) - l += vsb_len(vary); + l += varyl; /* Space for producing a Content-Length: header */ l += 30; @@ -553,9 +557,9 @@ if (vary != NULL) { sp->obj->vary = - (void *)WS_Alloc(sp->obj->http->ws, vsb_len(vary)); + (void *)WS_Alloc(sp->obj->http->ws, varyl); AN(sp->obj->vary); - memcpy(sp->obj->vary, vsb_data(vary), vsb_len(vary)); + memcpy(sp->obj->vary, vsb_data(vary), varyl); vsb_delete(vary); vary = NULL; } Modified: trunk/varnish-cache/bin/varnishd/cache_dir_random.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-08 17:13:10 UTC (rev 4441) @@ -78,7 +78,7 @@ }; static struct vbe_conn * -vdi_random_getfd(struct director *d, struct sess *sp) +vdi_random_getfd(const struct director *d, struct sess *sp) { int i, k; struct vdi_random *vs; @@ -158,7 +158,7 @@ } static unsigned -vdi_random_healthy(struct director *d, const struct sess *sp) +vdi_random_healthy(const struct director *d, const struct sess *sp) { struct vdi_random *vs; int i; Modified: trunk/varnish-cache/bin/varnishd/cache_dir_round_robin.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_dir_round_robin.c 2010-01-08 16:53:38 UTC (rev 4440) +++ trunk/varnish-cache/bin/varnishd/cache_dir_round_robin.c 2010-01-08 17:13:10 UTC (rev 4441) @@ -61,7 +61,7 @@ }; static struct vbe_conn * -vdi_round_robin_getfd(struct director *d, struct sess *sp) +vdi_round_robin_getfd(const struct director *d, struct sess *sp) { int i; struct vdi_round_robin *vs; @@ -86,7 +86,7 @@ } static unsigned -vdi_round_robin_healthy(struct director *d, const struct sess *sp) +vdi_round_robin_healthy(const struct director *d, const struct sess *sp) { struct vdi_round_robin *vs; struct director *backend; From phk at projects.linpro.no Fri Jan 8 18:26:09 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 8 Jan 2010 19:26:09 +0100 (CET) Subject: r4442 - trunk/varnish-cache/bin/varnishd Message-ID: <20100108182609.9276238868@projects.linpro.no> Author: phk Date: 2010-01-08 19:26:09 +0100 (Fri, 08 Jan 2010) New Revision: 4442 Modified: trunk/varnish-cache/bin/varnishd/cache_dir_random.c Log: Be hash-endianess-agnostic for the benefit of varnishtest on BE archs. Modified: trunk/varnish-cache/bin/varnishd/cache_dir_random.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-08 17:13:10 UTC (rev 4441) +++ trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-08 18:26:09 UTC (rev 4442) @@ -97,7 +97,10 @@ * amongst the good set. */ if (vs->use_hash) { - memcpy(&u, sp->digest, sizeof u); + u = sp->digest[3] << 24; + u |= sp->digest[2] << 16; + u |= sp->digest[1] << 8; + u |= sp->digest[0] << 0; r = u / 4294967296.0; r *= vs->tot_weight; s1 = 0.0; @@ -130,7 +133,10 @@ return (NULL); if (vs->use_hash) { - memcpy(&u, sp->digest, sizeof u); + u = sp->digest[3] << 24; + u |= sp->digest[2] << 16; + u |= sp->digest[1] << 8; + u |= sp->digest[0] << 0; r = u / 4294967296.0; } else { /* Pick a random threshold in that interval */ From tfheen at projects.linpro.no Fri Jan 8 19:28:58 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Fri, 8 Jan 2010 20:28:58 +0100 (CET) Subject: r4443 - trunk/varnish-cache/include/compat Message-ID: <20100108192858.C9EB638B61@projects.linpro.no> Author: tfheen Date: 2010-01-08 20:28:58 +0100 (Fri, 08 Jan 2010) New Revision: 4443 Modified: trunk/varnish-cache/include/compat/vasprintf.h Log: Include stdarg.h in compat/vasprintf.h Ensure we have va_list defined before trying to use it, fixes compilation problem on Solaris. Modified: trunk/varnish-cache/include/compat/vasprintf.h =================================================================== --- trunk/varnish-cache/include/compat/vasprintf.h 2010-01-08 18:26:09 UTC (rev 4442) +++ trunk/varnish-cache/include/compat/vasprintf.h 2010-01-08 19:28:58 UTC (rev 4443) @@ -32,6 +32,8 @@ #ifndef COMPAT_VASPRINTF_H_INCLUDED #define COMPAT_VASPRINTF_H_INCLUDED +#include + #ifndef HAVE_VASPRINTF int vasprintf(char **strp, const char *fmt, va_list ap); #endif From ssm at projects.linpro.no Mon Jan 11 06:48:33 2010 From: ssm at projects.linpro.no (ssm at projects.linpro.no) Date: Mon, 11 Jan 2010 07:48:33 +0100 (CET) Subject: r4444 - trunk/varnish-cache/bin/varnishd Message-ID: <20100111064833.2F9EC1F7BDC@projects.linpro.no> Author: ssm Date: 2010-01-11 07:48:32 +0100 (Mon, 11 Jan 2010) New Revision: 4444 Modified: trunk/varnish-cache/bin/varnishd/varnishd.1 Log: Update thread pool run-time parameters in varnishd man page This should address the issue reported in #614 Modified: trunk/varnish-cache/bin/varnishd/varnishd.1 =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.1 2010-01-08 19:28:58 UTC (rev 4443) +++ trunk/varnish-cache/bin/varnishd/varnishd.1 2010-01-11 06:48:32 UTC (rev 4444) @@ -446,7 +446,14 @@ use when they arrived. .El .Ss Run-Time Parameters -.\" Keep in synch with parspec[] in mgt_param.c +.\" Keep in synch with parspec[] in mgt_param.c and WRK_parspec[] in + \" mgt_pool.c +.Pp +Note that if an run-time parameter is marked +.Bq experimental , +we do not know yet if it is a good idea to change this parameter, or +if the default value is even sensible. Caution is advised, and +feedback is most welcome. .Bl -tag -width 4n .It Va auto_restart Whether to automatically restart the child process if it dies. @@ -561,6 +568,15 @@ The time to wait before dropping an idle pipe mode connection. .Pp The default is 60 seconds. +.It Va rush_exponent +.Bq experimental +.Pp +How many parked request we start for each completed request on the object. +.Pp +NB: Even with the implict delay of delivery, this parameter controls +an exponential increase in number of worker threads. +.Pp +The default is 3 requests per request. .It Va saintmode_threshold The number of items allowed on the saintmode list for a backend before it is considered sick. A value of 0 disables saintmode and any associated @@ -596,7 +612,51 @@ .It Va srcaddr_ttl The length of time to keep per-client accounting records. Setting this to 0 will disable per-client accounting. +.It Va thread_pool_add_delay +.Bq experimental +.Pp +Wait at least many miliseconds between creating threads. +.Pp +Setting this too long results in insuffient worker threads. +.Pp +Settings this too high increase the risk for worker thread pile-up. +.Pp +The default is 20 miliseconds. +.It Va thread_pool_add_threshold +.Bq experimental +.Pp +Overflow threshold in number of requests for worker thread creation. +.Pp +Setting this too low, will result in excess worker threads, which is +generally a bad idea. +.Pp +Setting it too high results in insuffient worker threads. +.Pp +The default is 2 requests +.It Va thread_pool_fail_delay +.Bq experimental +.Pp +Wait at least this many miliseconds after a failed thread creation +before trying to create another thread. +.Pp +Failure to create a worker thread is often a sign that the end is +near, because the process is running out of RAM resources for thread +stacks. This delay tries to not rush it on needlessly. +.Pp +If thread creation failures are a problem, check that +.Va thread_pool_max +is not too high. +.Pp +It may also help to increase +.Va thread_pool_timeout +and +.Va thread_pool_min , +to reduce the rate at which treads are destroyed and later recreated. +.Pp +The default is 200 miliseconds .It Va thread_pool_max +.Bq experimental +.Pp The maximum total number of worker threads. If the number of concurrent requests rises beyond this number, requests will start queueing up waiting for a worker thread to pick @@ -606,12 +666,38 @@ .Pp The default is 1000. .It Va thread_pool_min +.Bq experimental +.Pp The minimum total number of worker threads. Higher values may allow .Nm to respond faster to a sudden increase in traffic. .Pp The default is 5. +.It Va thread_pool_purge_delay +.Bq experimental +.Pp +Wait this many miliseconds between purging threads. +.Pp +This controls the decay of thread pool when idle(-ish). +.Pp +The minimum is 100 miliseconds, the default is 1000 miliseconds. +.It Va thread_pool_stack +.Bq experimental +.Pp +Worker thread stack size. The number used is the amount of bytes used +as thread stack size. +.Pp +On 32bit systems in particular you may need to tweak this down to fit +many threads into the limited address space. +.It Va thread_pool_timeout +.Bq experimental +.Pp +The amount of time a worker thread can be idle before it is killed, +when the number of worker threads exceeds +.Va thread_pool_min . +.Pp +The default is 120 seconds. .It Va thread_pools The number of worker thread pools. Higher values reduce lock contention but increase pressure on the @@ -620,12 +706,15 @@ restart. .Pp The default is 2. -.It Va thread_pool_timeout -The amount of time a worker thread can be idle before it is killed, -when the number of worker threads exceeds -.Va thread_pool_min . +.It Va thread_stats_rate +.Bq experimental .Pp -The default is 120 seconds. +Worker threads accumulate statistics, and dump these into the global +stats counters if the lock is free when they finish a request. +.Pp +This parameters defines the maximum number of requests a worker thread +may handle, before it is forced to dump its accumulated stats into the +global counters. .It Va user The name of an unprivileged user to which the child process should switch before it starts accepting connections. From phk at projects.linpro.no Mon Jan 11 14:56:52 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 11 Jan 2010 15:56:52 +0100 (CET) Subject: r4445 - trunk/varnish-cache/bin/varnishd Message-ID: <20100111145652.79EF81F7B68@projects.linpro.no> Author: phk Date: 2010-01-11 15:56:52 +0100 (Mon, 11 Jan 2010) New Revision: 4445 Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c Log: Don't recycle sessions one by one, batch them up for efficiency. Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-11 06:48:32 UTC (rev 4444) +++ trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-11 14:56:52 UTC (rev 4445) @@ -44,6 +44,8 @@ #include "cache.h" #include "cache_waiter.h" +#define NEEV 128 + static pthread_t vca_poll_thread; static struct pollfd *pollfd; static unsigned npoll, hpoll; @@ -106,9 +108,9 @@ vca_main(void *arg) { unsigned v; - struct sess *sp, *sp2; + struct sess *ss[NEEV], *sp, *sp2; double deadline; - int i, fd; + int i, j, fd; THR_SetName("cache-poll"); (void)arg; @@ -119,11 +121,14 @@ v = poll(pollfd, hpoll + 1, 100); if (v && pollfd[vca_pipes[0]].revents) { v--; - i = read(vca_pipes[0], &sp, sizeof sp); - assert(i == sizeof sp); - CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - VTAILQ_INSERT_TAIL(&sesshead, sp, list); - vca_poll(sp->fd); + i = read(vca_pipes[0], ss, sizeof ss); + assert(i >= 0); + assert((i % sizeof ss[0]) == 0); + for (j = 0; j * sizeof ss[0] < i; j += sizeof ss[0]) { + CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC); + VTAILQ_INSERT_TAIL(&sesshead, ss[j], list); + vca_poll(ss[j]->fd); + } } deadline = TIM_real() - params->sess_timeout; VTAILQ_FOREACH_SAFE(sp, &sesshead, list, sp2) { From phk at projects.linpro.no Tue Jan 12 10:03:38 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 12 Jan 2010 11:03:38 +0100 (CET) Subject: r4446 - trunk/varnish-cache/include Message-ID: <20100112100338.029BC1F7C24@projects.linpro.no> Author: phk Date: 2010-01-12 11:03:37 +0100 (Tue, 12 Jan 2010) New Revision: 4446 Modified: trunk/varnish-cache/include/vbm.h Log: Use assert(foo != NULL) rather than AN() to make this usable outside varnishd. Modified: trunk/varnish-cache/include/vbm.h =================================================================== --- trunk/varnish-cache/include/vbm.h 2010-01-11 14:56:52 UTC (rev 4445) +++ trunk/varnish-cache/include/vbm.h 2010-01-12 10:03:37 UTC (rev 4446) @@ -56,7 +56,7 @@ bit += VBITMAP_LUMP - 1; bit -= (bit % VBITMAP_LUMP); p = realloc(vb->bits, bit / 8); - AN(p); + assert(p != NULL); memset(p + vb->nbits / 8, 0, (bit - vb->nbits) / 8); vb->bits = (void*)p; vb->nbits = bit; @@ -68,7 +68,7 @@ struct vbitmap *vb; vb = calloc(sizeof *vb, 1); - AN(vb); + assert(vb != NULL); if (initial == 0) initial = VBITMAP_LUMP; vbit_expand(vb, initial); From phk at projects.linpro.no Tue Jan 12 10:07:21 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 12 Jan 2010 11:07:21 +0100 (CET) Subject: r4447 - trunk/varnish-cache/lib/libvarnishapi Message-ID: <20100112100721.EA1FD1F7C24@projects.linpro.no> Author: phk Date: 2010-01-12 11:07:21 +0100 (Tue, 12 Jan 2010) New Revision: 4447 Modified: trunk/varnish-cache/lib/libvarnishapi/shmlog.c Log: Catch up with changes in shmlog format: handle records longer than 256 char and id's higher than 64k, by employing variable size bitmaps and dynamic buffers. Modified: trunk/varnish-cache/lib/libvarnishapi/shmlog.c =================================================================== --- trunk/varnish-cache/lib/libvarnishapi/shmlog.c 2010-01-12 10:03:37 UTC (rev 4446) +++ trunk/varnish-cache/lib/libvarnishapi/shmlog.c 2010-01-12 10:07:21 UTC (rev 4447) @@ -47,6 +47,7 @@ #include "shmlog.h" #include "vre.h" +#include "vbm.h" #include "miniobj.h" #include "varnishapi.h" @@ -54,8 +55,6 @@ #define SLEEP_USEC (50*1000) #define TIMEOUT_USEC (5*1000*1000) -#define NFD (256 * 256) - struct VSL_data { unsigned magic; #define VSL_MAGIC 0x6e3bd69b @@ -67,7 +66,8 @@ /* for -r option */ int fd; - unsigned char rbuf[SHMLOG_NEXTTAG + 255 + 1]; + unsigned rbuflen; + unsigned char *rbuf; int b_opt; int c_opt; @@ -77,12 +77,23 @@ #define F_SEEN_IX (1 << 0) #define F_NON_BLOCKING (1 << 1) - unsigned char map[NFD]; -#define M_CLIENT (1 << 0) -#define M_BACKEND (1 << 1) -#define M_SUPPRESS (1 << 2) -#define M_SELECT (1 << 3) + /* + * These two bitmaps mark fd's as belonging to client or backend + * transactions respectively. + */ + struct vbitmap *vbm_client; + struct vbitmap *vbm_backend; + /* + * Bit map of programatically selected tags, that cannot be suppressed. + * This way programs can make sure they will see certain tags, even + * if the user tries to supress them with -x/-X + */ + struct vbitmap *vbm_select; /* index: tag */ + + /* Bit map of tags selected/supressed with -[iIxX] options */ + struct vbitmap *vbm_supress; /* index: tag */ + int regflags; vre_t *regincl; vre_t *regexcl; @@ -166,13 +177,18 @@ { struct VSL_data *vd; - assert(VSL_S_CLIENT == M_CLIENT); - assert(VSL_S_BACKEND == M_BACKEND); vd = calloc(sizeof *vd, 1); assert(vd != NULL); vd->regflags = 0; vd->magic = VSL_MAGIC; vd->fd = -1; + vd->vbm_client = vbit_init(4096); + vd->vbm_backend = vbit_init(4096); + vd->vbm_supress = vbit_init(256); + vd->vbm_select = vbit_init(256); + vd->rbuflen = SHMLOG_NEXTTAG + 256; + vd->rbuf = malloc(vd->rbuflen); + assert(vd->rbuf != NULL); return (vd); } @@ -183,7 +199,7 @@ { CHECK_OBJ_NOTNULL(vd, VSL_MAGIC); - vd->map[tag] |= M_SELECT; + vbit_set(vd->vbm_select, tag); } /*--------------------------------------------------------------------*/ @@ -235,12 +251,20 @@ CHECK_OBJ_NOTNULL(vd, VSL_MAGIC); if (vd->fd != -1) { + assert(vd->rbuflen >= SHMLOG_DATA); i = read(vd->fd, vd->rbuf, SHMLOG_DATA); if (i != SHMLOG_DATA) return (-1); - i = read(vd->fd, vd->rbuf + SHMLOG_DATA, - SHMLOG_LEN(vd->rbuf) + 1); - if (i != SHMLOG_LEN(vd->rbuf) + 1) + l = SHMLOG_LEN(vd->rbuf) + SHMLOG_NEXTTAG; + if (vd->rbuflen < l) { + l += 200; + vd->rbuf = realloc(vd->rbuf, l); + assert(vd->rbuf != NULL); + vd->rbuflen = l; + } + l = SHMLOG_LEN(vd->rbuf) + 1; + i = read(vd->fd, vd->rbuf + SHMLOG_DATA, l); + if (i != l) return (-1); *pp = vd->rbuf; return (1); @@ -271,7 +295,7 @@ int VSL_NextLog(struct VSL_data *vd, unsigned char **pp) { - unsigned char *p; + unsigned char *p, t; unsigned u, l; int i; @@ -285,13 +309,13 @@ switch(p[SHMLOG_TAG]) { case SLT_SessionOpen: case SLT_ReqStart: - vd->map[u] |= M_CLIENT; - vd->map[u] &= ~M_BACKEND; + vbit_set(vd->vbm_client, u); + vbit_clr(vd->vbm_backend, u); break; case SLT_BackendOpen: case SLT_BackendXID: - vd->map[u] |= M_BACKEND; - vd->map[u] &= ~M_CLIENT; + vbit_clr(vd->vbm_client, u); + vbit_set(vd->vbm_backend, u); break; default: break; @@ -303,15 +327,16 @@ if (--vd->keep == 0) return (-1); } - if (vd->map[p[SHMLOG_TAG]] & M_SELECT) { + t = p[SHMLOG_TAG]; + if (vbit_test(vd->vbm_select, t)) { *pp = p; return (1); } - if (vd->map[p[SHMLOG_TAG]] & M_SUPPRESS) + if (vbit_test(vd->vbm_supress, t)) continue; - if (vd->b_opt && !(vd->map[u] & M_BACKEND)) + if (vd->b_opt && !vbit_test(vd->vbm_backend, u)) continue; - if (vd->c_opt && !(vd->map[u] & M_CLIENT)) + if (vd->c_opt && !vbit_test(vd->vbm_client, u)) continue; if (vd->regincl != NULL) { i = VRE_exec(vd->regincl, @@ -340,7 +365,7 @@ VSL_Dispatch(struct VSL_data *vd, vsl_handler *func, void *priv) { int i; - unsigned u, l; + unsigned u, l, s; unsigned char *p; CHECK_OBJ_NOTNULL(vd, VSL_MAGIC); @@ -350,10 +375,13 @@ return (i); u = SHMLOG_ID(p); l = SHMLOG_LEN(p); + s = 0; + if (vbit_test(vd->vbm_backend, u)) + s |= VSL_S_BACKEND; + if (vbit_test(vd->vbm_client, u)) + s |= VSL_S_CLIENT; if (func(priv, - p[SHMLOG_TAG], u, l, - vd->map[u] & (VSL_S_CLIENT|VSL_S_BACKEND), - (char *)p + SHMLOG_DATA)) + p[SHMLOG_TAG], u, l, s, (char *)p + SHMLOG_DATA)) return (1); } } @@ -444,7 +472,7 @@ /* If first option is 'i', set all bits for supression */ if (arg == 'i' && !(vd->flags & F_SEEN_IX)) for (i = 0; i < 256; i++) - vd->map[i] |= M_SUPPRESS; + vbit_set(vd->vbm_supress, i); vd->flags |= F_SEEN_IX; for (b = opt; *b; b = e) { @@ -470,9 +498,9 @@ continue; if (arg == 'x') - vd->map[i] |= M_SUPPRESS; + vbit_set(vd->vbm_supress, i); else - vd->map[i] &= ~M_SUPPRESS; + vbit_clr(vd->vbm_supress, i); break; } if (i == 256) { From phk at projects.linpro.no Wed Jan 13 09:19:50 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 10:19:50 +0100 (CET) Subject: r4448 - trunk/varnish-cache/bin/varnishd Message-ID: <20100113091950.0AD231F7B67@projects.linpro.no> Author: phk Date: 2010-01-13 10:19:49 +0100 (Wed, 13 Jan 2010) New Revision: 4448 Modified: trunk/varnish-cache/bin/varnishd/mgt.h trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/bin/varnishd/varnishd.c Log: Give mgt_cli_setup() a explict close callback argument, to call when we reach EOF on the input filedescriptor. List all cli connections on a VTAILQ, and add a function for closing them all. Use close callback for stdin and telnet CLI connections. Modified: trunk/varnish-cache/bin/varnishd/mgt.h =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-12 10:07:21 UTC (rev 4447) +++ trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-13 09:19:49 UTC (rev 4448) @@ -49,12 +49,14 @@ /* mgt_cli.c */ -void mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident); +typedef void mgt_cli_close_f(void *priv); +void mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident, mgt_cli_close_f *close_func, void *priv); int mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...); void mgt_cli_start_child(int fdi, int fdo); void mgt_cli_stop_child(void); void mgt_cli_telnet(const char *T_arg); void mgt_cli_secret(const char *S_arg); +void mgt_cli_close_all(void); /* mgt_param.c */ void MCF_ParamSync(void); Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-12 10:07:21 UTC (rev 4447) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-13 09:19:49 UTC (rev 4448) @@ -71,14 +71,16 @@ static const char *secret_file; struct telnet { + unsigned magic; +#define TELNET_MAGIC 0x53ec3ac0 int fd; struct vev *ev; - VTAILQ_ENTRY(telnet) list; }; struct cli_port { unsigned magic; #define CLI_PORT_MAGIC 0x5791079f + VTAILQ_ENTRY(cli_port) list; struct vev *ev; int fdi; int fdo; @@ -87,11 +89,11 @@ struct cli cli[1]; char *name; char challenge[34]; + mgt_cli_close_f *closefunc; + void *priv; }; -static VTAILQ_HEAD(,telnet) telnets = VTAILQ_HEAD_INITIALIZER(telnets); -static void telnet_close_all(void); -static void telnet_close_one(int fd); +static VTAILQ_HEAD(,cli_port) clilist = VTAILQ_HEAD_INITIALIZER(clilist); /*--------------------------------------------------------------------*/ @@ -436,31 +438,35 @@ CHECK_OBJ_NOTNULL(cp, CLI_PORT_MAGIC); syslog(LOG_NOTICE, "CLI %d closed", cp->fdi); + /* + * Remove from list, so that if closefunc calls mgt_cli_close_all + * it will not try to remove this one too. + */ + VTAILQ_REMOVE(&clilist, cp, list); + free(cp->name); vsb_delete(cp->cli->sb); VLU_Destroy(cp->vlu); - (void)close(cp->fdi); - if (cp->fdo != cp->fdi) - (void)close(cp->fdo); + cp->closefunc(cp->priv); + FREE_OBJ(cp); + return (1); +} - /* Special case for stdin/out/err */ - if (cp->fdi == 0) { - assert(open("/dev/null", O_RDONLY) == 0); - assert(open("/dev/null", O_WRONLY) == 1); - (void)close(2); - assert(open("/dev/null", O_WRONLY) == 2); +/*-------------------------------------------------------------------- + * Get rid of all CLI sessions + */ - if (d_flag == 2) { - mgt_stop_child(); - telnet_close_all(); - } - } else { - telnet_close_one(cp->fdi); - } +void +mgt_cli_close_all(void) +{ + struct cli_port *cp; - free(cp); - return (1); + while (!VTAILQ_EMPTY(&clilist)) { + cp = VTAILQ_FIRST(&clilist); + vev_del(mgt_evb, cp->ev); + (void)mgt_cli_close(cp); + } } /*-------------------------------------------------------------------- @@ -485,7 +491,7 @@ /*--------------------------------------------------------------------*/ void -mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident) +mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident, mgt_cli_close_f *closefunc, void *priv) { struct cli_port *cp; @@ -502,6 +508,9 @@ cp->fdo = fdo; cp->verbose = verbose; + cp->closefunc = closefunc; + cp->priv = priv; + cp->cli->sb = vsb_newauto(); XXXAN(cp->cli->sb); cp->cli->priv = cp; @@ -520,53 +529,36 @@ } else if (params->cli_banner) (void)VLU_Data("banner\n", -1, cp->vlu); - cp->ev = calloc(sizeof *cp->ev, 1); + cp->ev = vev_new(); cp->ev->name = cp->name; cp->ev->fd = fdi; cp->ev->fd_flags = EV_RD; cp->ev->callback = mgt_cli_callback; cp->ev->priv = cp; + VTAILQ_INSERT_TAIL(&clilist, cp, list); AZ(vev_add(mgt_evb, cp->ev)); } /*--------------------------------------------------------------------*/ static void -telnet_close_one(int fd) +telnet_close(void *priv) { - struct telnet *tn, *tn2; + struct telnet *tn; - VTAILQ_FOREACH_SAFE(tn, &telnets, list, tn2) { - if (tn->fd != fd) - continue; - VTAILQ_REMOVE(&telnets, tn, list); - (void)close(tn->fd); - free(tn); - break; - } + CAST_OBJ_NOTNULL(tn, priv, TELNET_MAGIC); + (void)close(tn->fd); + FREE_OBJ(tn); } -static void -telnet_close_all() -{ - struct telnet *tn, *tn2; - - VTAILQ_FOREACH_SAFE(tn, &telnets, list, tn2) { - VTAILQ_REMOVE(&telnets, tn, list); - AZ(close(tn->fd)); - free(tn); - } -} - static struct telnet * telnet_new(int fd) { struct telnet *tn; - tn = calloc(sizeof *tn, 1); + ALLOC_OBJ(tn, TELNET_MAGIC); AN(tn); tn->fd = fd; - VTAILQ_INSERT_TAIL(&telnets, tn, list); return (tn); } @@ -575,6 +567,7 @@ { struct sockaddr_storage addr; socklen_t addrlen; + struct telnet *tn; int i; char abuf1[TCP_ADDRBUFSIZE], abuf2[TCP_ADDRBUFSIZE]; char pbuf1[TCP_PORTBUFSIZE], pbuf2[TCP_PORTBUFSIZE]; @@ -594,9 +587,9 @@ assert(asprintf(&p, "telnet %s:%s %s:%s", abuf2, pbuf2, abuf1, pbuf1) > 0); XXXAN(p); - (void)telnet_new(i); + tn = telnet_new(i); - mgt_cli_setup(i, i, 0, p); + mgt_cli_setup(i, i, 0, p, telnet_close, tn); free(p); return (0); } Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-12 10:07:21 UTC (rev 4447) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-13 09:19:49 UTC (rev 4448) @@ -38,6 +38,7 @@ #include #include +#include #include #include #include @@ -477,6 +478,29 @@ (void)pclose(fi); } +/*-------------------------------------------------------------------- + * This function is called when the CLI on stdin is closed. + */ + +static void +cli_stdin_close(void *priv) +{ + + (void)priv; + (void)close(0); + (void)close(1); + (void)close(2); + assert(open("/dev/null", O_RDONLY) == 0); + assert(open("/dev/null", O_WRONLY) == 1); + assert(open("/dev/null", O_WRONLY) == 2); + + if (d_flag == 2) { + mgt_stop_child(); + mgt_cli_close_all(); + exit(0); + } +} + /*--------------------------------------------------------------------*/ int @@ -728,7 +752,7 @@ XXXAN(mgt_evb); if (d_flag) - mgt_cli_setup(0, 1, 1, "debug"); + mgt_cli_setup(0, 1, 1, "debug", cli_stdin_close, NULL); if (S_arg != NULL) mgt_cli_secret(S_arg); if (T_arg != NULL) From phk at projects.linpro.no Wed Jan 13 10:58:24 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 11:58:24 +0100 (CET) Subject: r4449 - in trunk/varnish-cache: include lib/libvarnish Message-ID: <20100113105824.6BBC91F7C24@projects.linpro.no> Author: phk Date: 2010-01-13 11:58:23 +0100 (Wed, 13 Jan 2010) New Revision: 4449 Modified: trunk/varnish-cache/include/libvarnish.h trunk/varnish-cache/lib/libvarnish/tcp.c Log: Add TCP_hisname() function Modified: trunk/varnish-cache/include/libvarnish.h =================================================================== --- trunk/varnish-cache/include/libvarnish.h 2010-01-13 09:19:49 UTC (rev 4448) +++ trunk/varnish-cache/include/libvarnish.h 2010-01-13 10:58:23 UTC (rev 4449) @@ -61,6 +61,7 @@ #define TCP_PORTBUFSIZE 16 void TCP_myname(int sock, char *abuf, unsigned alen, char *pbuf, unsigned plen); +void TCP_hisname(int sock, char *abuf, unsigned alen, char *pbuf, unsigned plen); int TCP_filter_http(int sock); void TCP_blocking(int sock); void TCP_nonblocking(int sock); Modified: trunk/varnish-cache/lib/libvarnish/tcp.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/tcp.c 2010-01-13 09:19:49 UTC (rev 4448) +++ trunk/varnish-cache/lib/libvarnish/tcp.c 2010-01-13 10:58:23 UTC (rev 4449) @@ -98,7 +98,24 @@ AZ(getsockname(sock, addr, &l)); TCP_name(addr, l, abuf, alen, pbuf, plen); } +/*--------------------------------------------------------------------*/ +void +TCP_hisname(int sock, char *abuf, unsigned alen, char *pbuf, unsigned plen) +{ + struct sockaddr_storage addr_s; + struct sockaddr *addr = (void*)&addr_s; + socklen_t l; + + l = sizeof addr_s; + if (!getpeername(sock, addr, &l)) + TCP_name(addr, l, abuf, alen, pbuf, plen); + else { + strlcpy(abuf, "", alen); + strlcpy(pbuf, "", plen); + } +} + /*--------------------------------------------------------------------*/ int From phk at projects.linpro.no Wed Jan 13 11:08:00 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 12:08:00 +0100 (CET) Subject: r4450 - in trunk/varnish-cache: bin/varnishadm bin/varnishreplay include lib/libvarnish Message-ID: <20100113110800.778E61F7B67@projects.linpro.no> Author: phk Date: 2010-01-13 12:08:00 +0100 (Wed, 13 Jan 2010) New Revision: 4450 Modified: trunk/varnish-cache/bin/varnishadm/varnishadm.c trunk/varnish-cache/bin/varnishreplay/varnishreplay.c trunk/varnish-cache/include/vss.h trunk/varnish-cache/lib/libvarnish/vss.c Log: Make it possible to use VSS_connect() for nonblocking connects. Modified: trunk/varnish-cache/bin/varnishadm/varnishadm.c =================================================================== --- trunk/varnish-cache/bin/varnishadm/varnishadm.c 2010-01-13 10:58:23 UTC (rev 4449) +++ trunk/varnish-cache/bin/varnishadm/varnishadm.c 2010-01-13 11:08:00 UTC (rev 4450) @@ -104,7 +104,7 @@ exit(2); } - sock = VSS_connect(ta[0]); + sock = VSS_connect(ta[0], 0); for (i = 0; i < n; ++i) { free(ta[i]); Modified: trunk/varnish-cache/bin/varnishreplay/varnishreplay.c =================================================================== --- trunk/varnish-cache/bin/varnishreplay/varnishreplay.c 2010-01-13 10:58:23 UTC (rev 4449) +++ trunk/varnish-cache/bin/varnishreplay/varnishreplay.c 2010-01-13 11:08:00 UTC (rev 4450) @@ -581,7 +581,8 @@ for (;;) { thread_log(1, 0, "sleeping before connect..."); usleep(1000 * (thr->fd % 3001)); - if ((thr->sock = VSS_connect(addr_info)) >= 0) + thr->sock = VSS_connect(addr_info, 0); + if (thr->sock >= 0) break; thread_log(0, errno, "connect failed"); } Modified: trunk/varnish-cache/include/vss.h =================================================================== --- trunk/varnish-cache/include/vss.h 2010-01-13 10:58:23 UTC (rev 4449) +++ trunk/varnish-cache/include/vss.h 2010-01-13 11:08:00 UTC (rev 4450) @@ -34,5 +34,5 @@ int VSS_resolve(const char *addr, const char *port, struct vss_addr ***ta); int VSS_bind(const struct vss_addr *addr); int VSS_listen(const struct vss_addr *addr, int depth); -int VSS_connect(const struct vss_addr *addr); +int VSS_connect(const struct vss_addr *addr, int nonblock); int VSS_open(const char *str); Modified: trunk/varnish-cache/lib/libvarnish/vss.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/vss.c 2010-01-13 10:58:23 UTC (rev 4449) +++ trunk/varnish-cache/lib/libvarnish/vss.c 2010-01-13 11:08:00 UTC (rev 4450) @@ -239,9 +239,9 @@ * Return the socket. */ int -VSS_connect(const struct vss_addr *va) +VSS_connect(const struct vss_addr *va, int nonblock) { - int sd; + int sd, i; sd = socket(va->va_family, va->va_socktype, va->va_protocol); if (sd < 0) { @@ -249,12 +249,14 @@ perror("socket()"); return (-1); } - if (connect(sd, &va->va_addr.sa, va->va_addrlen) != 0) { - perror("connect()"); - (void)close(sd); - return (-1); - } - return (sd); + if (nonblock) + TCP_nonblocking(sd); + i = connect(sd, &va->va_addr.sa, va->va_addrlen); + if (i == 0 || (nonblock && errno == EINPROGRESS)) + return (sd); + perror("connect()"); + (void)close(sd); + return (-1); } /* @@ -279,7 +281,7 @@ return (-1); } for (n = 0; n < nvaddr; n++) { - retval = VSS_connect(vaddr[n]); + retval = VSS_connect(vaddr[n], 0); if (retval >= 0) break; } From phk at projects.linpro.no Wed Jan 13 11:25:32 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 12:25:32 +0100 (CET) Subject: r4451 - trunk/varnish-cache/bin/varnishd Message-ID: <20100113112532.B53EC1F7C24@projects.linpro.no> Author: phk Date: 2010-01-13 12:25:32 +0100 (Wed, 13 Jan 2010) New Revision: 4451 Modified: trunk/varnish-cache/bin/varnishd/mgt.h trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/bin/varnishd/varnishd.c Log: Add -M addr:port argument, for connecting to a source of CLI commands, such as the mythical "cluster-controller". Modified: trunk/varnish-cache/bin/varnishd/mgt.h =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-13 11:08:00 UTC (rev 4450) +++ trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-13 11:25:32 UTC (rev 4451) @@ -55,6 +55,7 @@ void mgt_cli_start_child(int fdi, int fdo); void mgt_cli_stop_child(void); void mgt_cli_telnet(const char *T_arg); +void mgt_cli_master(const char *M_arg); void mgt_cli_secret(const char *S_arg); void mgt_cli_close_all(void); Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-13 11:08:00 UTC (rev 4450) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-13 11:25:32 UTC (rev 4451) @@ -46,8 +46,6 @@ #include #include -#include "compat/vasprintf.h" - #ifndef HAVE_SRANDOMDEV #include "compat/srandomdev.h" #endif @@ -501,7 +499,7 @@ cp->name = strdup(ident); XXXAN(cp->name); - syslog(LOG_NOTICE, "CLI %d open from %s", fdi, cp->name); + syslog(LOG_NOTICE, "CLI %d open %s", fdi, cp->name); cp->magic = CLI_PORT_MAGIC; cp->fdi = fdi; @@ -541,6 +539,26 @@ /*--------------------------------------------------------------------*/ +static struct vsb * +sock_id(const char *pfx, int fd) +{ + struct vsb *vsb; + + char abuf1[TCP_ADDRBUFSIZE], abuf2[TCP_ADDRBUFSIZE]; + char pbuf1[TCP_PORTBUFSIZE], pbuf2[TCP_PORTBUFSIZE]; + + vsb = vsb_newauto(); + AN(vsb); + TCP_myname(fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1); + TCP_hisname(fd, abuf2, sizeof abuf2, pbuf2, sizeof pbuf2); + vsb_printf(vsb, "%s %s:%s %s:%s", pfx, abuf2, pbuf2, abuf1, pbuf1); + vsb_finish(vsb); + AZ(vsb_overflowed(vsb)); + return (vsb); +} + +/*--------------------------------------------------------------------*/ + static void telnet_close(void *priv) { @@ -565,13 +583,11 @@ static int telnet_accept(const struct vev *ev, int what) { + struct vsb *vsb; struct sockaddr_storage addr; socklen_t addrlen; struct telnet *tn; int i; - char abuf1[TCP_ADDRBUFSIZE], abuf2[TCP_ADDRBUFSIZE]; - char pbuf1[TCP_PORTBUFSIZE], pbuf2[TCP_PORTBUFSIZE]; - char *p; (void)what; addrlen = sizeof addr; @@ -581,16 +597,10 @@ if (i < 0) return (0); - TCP_myname(ev->fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1); - TCP_name((void*)&addr, addrlen, abuf2, sizeof abuf2, - pbuf2, sizeof pbuf2); - assert(asprintf(&p, "telnet %s:%s %s:%s", abuf2, pbuf2, abuf1, pbuf1) > 0); - XXXAN(p); - tn = telnet_new(i); - - mgt_cli_setup(i, i, 0, p, telnet_close, tn); - free(p); + vsb = sock_id("telnet", ev->fd); + mgt_cli_setup(i, i, 0, vsb_data(vsb), telnet_close, tn); + vsb_delete(vsb); return (0); } @@ -658,3 +668,100 @@ free(addr); free(port); } + +/* Reverse CLI ("Master") connections --------------------------------*/ + +static int M_fd = -1; +static struct vev *M_poker, *M_conn; +static char *M_addr, *M_port; +static struct vss_addr **M_ta; +static int M_nta, M_nxt; +static double M_poll = 0.1; + +static void +Marg_closer(void *priv) +{ + + (void)priv; + (void)close(M_fd); + M_fd = -1; +} + +static int +Marg_poker(const struct vev *e, int what) +{ + struct vsb *vsb; + int s, k; + socklen_t l; + + (void)what; /* XXX: ??? */ + + if (e == M_conn) { + /* Our connect(2) returned, check result */ + l = sizeof k; + AZ(getsockopt(M_fd, SOL_SOCKET, SO_ERROR, &k, &l)); + if (k) { + errno = k; + syslog(LOG_INFO, "Could not connect to CLI-master: %m"); + (void)close(M_fd); + M_fd = -1; + /* Try next address */ + if (++M_nxt >= M_nta) { + M_nxt = 0; + if (M_poll < 10) + M_poll *= 2; + } + return (1); + } + vsb = sock_id("master", M_fd); + mgt_cli_setup(M_fd, M_fd, 0, vsb_data(vsb), Marg_closer, NULL); + vsb_delete(vsb); + M_poll = 1; + return (1); + } + + assert(e == M_poker); + + M_poker->timeout = M_poll; /* XXX nasty ? */ + if (M_fd >= 0) + return (0); + + /* Try to connect asynchronously */ + s = VSS_connect(M_ta[M_nxt], 1); + if (s < 0) + return (0); + + M_conn = vev_new(); + AN(M_conn); + M_conn->callback = Marg_poker; + M_conn->name = "-M connector"; + M_conn->fd_flags = EV_WR; + M_conn->fd = s; + M_fd = s; + AZ(vev_add(mgt_evb, M_conn)); + return (0); +} + +void +mgt_cli_master(const char *M_arg) +{ + (void)M_arg; + + if (VSS_parse(M_arg, &M_addr, &M_port) || M_port == NULL) { + fprintf(stderr, "Could not parse -M argument\n"); + exit (1); + } + M_nta = VSS_resolve(M_addr, M_port, &M_ta); + if (M_nta <= 0) { + fprintf(stderr, "Could resolve -M argument to address\n"); + exit (1); + } + M_nxt = 0; + AZ(M_poker); + M_poker = vev_new(); + AN(M_poker); + M_poker->timeout = M_poll; + M_poker->callback = Marg_poker; + M_poker->name = "-M poker"; + AZ(vev_add(mgt_evb, M_poker)); +} Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-13 11:08:00 UTC (rev 4450) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-13 11:25:32 UTC (rev 4451) @@ -192,6 +192,7 @@ fprintf(stderr, FMT, "", " -h classic,"); fprintf(stderr, FMT, "-i identity", "Identity of varnish instance"); fprintf(stderr, FMT, "-l bytesize", "Size of shared memory log"); + fprintf(stderr, FMT, "-M address:port", "CLI-master to connect to."); fprintf(stderr, FMT, "-n dir", "varnishd working directory"); fprintf(stderr, FMT, "-P file", "PID file"); fprintf(stderr, FMT, "-p param=value", "set parameter"); @@ -516,6 +517,7 @@ uintmax_t l_size; const char *q; const char *h_arg = "classic"; + const char *M_arg = NULL; const char *n_arg = NULL; const char *P_arg = NULL; const char *S_arg = NULL; @@ -565,7 +567,7 @@ cli_check(cli); while ((o = getopt(argc, argv, - "a:b:Cdf:Fg:h:i:l:n:P:p:S:s:T:t:u:Vw:")) != -1) + "a:b:Cdf:Fg:h:i:l:M:n:P:p:S:s:T:t:u:Vw:")) != -1) switch (o) { case 'a': MCF_ParamSet(cli, "listen_address", optarg); @@ -598,6 +600,9 @@ case 'l': l_arg = optarg; break; + case 'M': + M_arg = optarg; + break; case 'n': n_arg = optarg; break; @@ -755,6 +760,8 @@ mgt_cli_setup(0, 1, 1, "debug", cli_stdin_close, NULL); if (S_arg != NULL) mgt_cli_secret(S_arg); + if (M_arg != NULL) + mgt_cli_master(M_arg); if (T_arg != NULL) mgt_cli_telnet(T_arg); From ingvar at projects.linpro.no Wed Jan 13 12:32:04 2010 From: ingvar at projects.linpro.no (ingvar at projects.linpro.no) Date: Wed, 13 Jan 2010 13:32:04 +0100 (CET) Subject: r4452 - trunk/varnish-cache/redhat Message-ID: <20100113123204.EB0181F7B7A@projects.linpro.no> Author: ingvar Date: 2010-01-13 13:32:04 +0100 (Wed, 13 Jan 2010) New Revision: 4452 Modified: trunk/varnish-cache/redhat/varnish.logrotate Log: Wrong path to pidfiles in redhat/varnish.logrotate Modified: trunk/varnish-cache/redhat/varnish.logrotate =================================================================== --- trunk/varnish-cache/redhat/varnish.logrotate 2010-01-13 11:25:32 UTC (rev 4451) +++ trunk/varnish-cache/redhat/varnish.logrotate 2010-01-13 12:32:04 UTC (rev 4452) @@ -3,7 +3,7 @@ notifempty sharedscripts postrotate - /bin/kill -HUP `cat /var/run/varnish/varnishlog.pid 2>/dev/null` 2> /dev/null || true - /bin/kill -HUP `cat /var/run/varnish/varnishncsa.pid 2>/dev/null` 2> /dev/null || true + /bin/kill -HUP `cat /var/run/varnishlog.pid 2>/dev/null` 2> /dev/null || true + /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true endscript } From ingvar at projects.linpro.no Wed Jan 13 12:34:48 2010 From: ingvar at projects.linpro.no (ingvar at projects.linpro.no) Date: Wed, 13 Jan 2010 13:34:48 +0100 (CET) Subject: r4453 - trunk/varnish-cache/redhat Message-ID: <20100113123448.2A5771F7B7A@projects.linpro.no> Author: ingvar Date: 2010-01-13 13:34:47 +0100 (Wed, 13 Jan 2010) New Revision: 4453 Modified: trunk/varnish-cache/redhat/varnish.logrotate Log: Added delaycompress to redhat/varnish.logrotate, fixes BZ #554745 Modified: trunk/varnish-cache/redhat/varnish.logrotate =================================================================== --- trunk/varnish-cache/redhat/varnish.logrotate 2010-01-13 12:32:04 UTC (rev 4452) +++ trunk/varnish-cache/redhat/varnish.logrotate 2010-01-13 12:34:47 UTC (rev 4453) @@ -2,6 +2,7 @@ missingok notifempty sharedscripts + delaycompress postrotate /bin/kill -HUP `cat /var/run/varnishlog.pid 2>/dev/null` 2> /dev/null || true /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true From phk at projects.linpro.no Wed Jan 13 13:52:25 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 14:52:25 +0100 (CET) Subject: r4454 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100113135225.072121F7B80@projects.linpro.no> Author: phk Date: 2010-01-13 14:52:24 +0100 (Wed, 13 Jan 2010) New Revision: 4454 Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c Log: Use the new varnishd -M argument to turn the CLI connection around, so we can use anonymous TCP ports for it. Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-13 12:34:47 UTC (rev 4453) +++ trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-13 13:52:24 UTC (rev 4454) @@ -71,8 +71,6 @@ struct vsb *args; int fds[4]; pid_t pid; - const char *telnet; - const char *accept; pthread_t tp; @@ -173,8 +171,6 @@ vtc_log(v->vl, 0, "Varnish name must start with 'v'"); v->args = vsb_newauto(); - v->telnet = "127.0.0.1:9001"; - v->accept = "127.0.0.1:0"; v->cli_fd = -1; VTAILQ_INSERT_TAIL(&varnishes, v, list); @@ -247,8 +243,17 @@ varnish_launch(struct varnish *v) { struct vsb *vsb; - int i; + int i, nfd, nap; + struct vss_addr **ap; + char abuf[128],pbuf[128]; + struct pollfd fd; + /* Create listener socket */ + nap = VSS_resolve("127.0.0.1", "0", &ap); + AN(nap); + v->cli_fd = VSS_listen(ap[0], 1); + TCP_myname(v->cli_fd, abuf, sizeof abuf, pbuf, sizeof pbuf); + vsb_finish(v->args); AZ(vsb_overflowed(v->args)); vtc_log(v->vl, 2, "Launch"); @@ -258,7 +263,8 @@ vsb_printf(vsb, " ./varnishd -d -d -n %s", v->workdir); vsb_printf(vsb, " -p cli_banner=off"); vsb_printf(vsb, " -p auto_restart=off"); - vsb_printf(vsb, " -a '%s' -T %s", v->accept, v->telnet); + vsb_printf(vsb, " -a '%s'", "127.0.0.1:0"); + vsb_printf(vsb, " -M %s:%s", abuf, pbuf); vsb_printf(vsb, " -P %s/varnishd.pid", v->workdir); vsb_printf(vsb, " %s", vsb_data(v->args)); vsb_finish(vsb); @@ -288,19 +294,27 @@ vsb_delete(vsb); AZ(pthread_create(&v->tp, NULL, varnish_thread, v)); - vtc_log(v->vl, 3, "opening CLI connection"); - for (i = 0; i < 30; i++) { - (void)usleep(200000); - v->cli_fd = VSS_open(v->telnet); - if (v->cli_fd >= 0) - break; + /* Wait for the varnish to call home */ + fd.fd = v->cli_fd; + fd.events = POLLIN; + i = poll(&fd, 1, 6000); + if (i != 1) { + AZ(close(v->fds[1])); + (void)kill(v->pid, SIGKILL); + vtc_log(v->vl, 0, "FAIL timeout waiting for CLI connection"); + return; } - if (v->cli_fd < 0) { + nfd = accept(v->cli_fd, NULL, NULL); + if (nfd < 0) { AZ(close(v->fds[1])); (void)kill(v->pid, SIGKILL); - vtc_log(v->vl, 0, "FAIL no CLI connection"); + vtc_log(v->vl, 0, "FAIL no CLI connection accepted"); return; } + + AZ(close(v->cli_fd)); + v->cli_fd = nfd; + vtc_log(v->vl, 3, "CLI connection fd = %d", v->cli_fd); assert(v->cli_fd >= 0); if (v->stats != NULL) @@ -611,18 +625,6 @@ for (; *av != NULL; av++) { if (vtc_error) break; - if (!strcmp(*av, "-telnet")) { - AN(av[1]); - v->telnet = av[1]; - av++; - continue; - } - if (!strcmp(*av, "-accept")) { - AN(av[1]); - v->accept = av[1]; - av++; - continue; - } if (!strcmp(*av, "-arg")) { AN(av[1]); vsb_cat(v->args, " "); @@ -649,10 +651,6 @@ av += 2; continue; } - if (!strcmp(*av, "-launch")) { - varnish_launch(v); - continue; - } if (!strcmp(*av, "-start")) { varnish_start(v); continue; From tfheen at projects.linpro.no Wed Jan 13 14:55:46 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Wed, 13 Jan 2010 15:55:46 +0100 (CET) Subject: r4455 - in trunk/varnish-cache/bin/varnishtest: . tests Message-ID: <20100113145546.8706C1F7B80@projects.linpro.no> Author: tfheen Date: 2010-01-13 15:55:46 +0100 (Wed, 13 Jan 2010) New Revision: 4455 Modified: trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc trunk/varnish-cache/bin/varnishtest/tests/b00014.vtc trunk/varnish-cache/bin/varnishtest/tests/p00000.vtc trunk/varnish-cache/bin/varnishtest/tests/p00001.vtc trunk/varnish-cache/bin/varnishtest/tests/p00002.vtc trunk/varnish-cache/bin/varnishtest/tests/p00003.vtc trunk/varnish-cache/bin/varnishtest/tests/p00004.vtc trunk/varnish-cache/bin/varnishtest/tests/p00005.vtc trunk/varnish-cache/bin/varnishtest/tests/p00006.vtc trunk/varnish-cache/bin/varnishtest/tests/v00016.vtc trunk/varnish-cache/bin/varnishtest/vtc.c Log: Add tmpdir macro to varnishtest Make a tmpdir macro available that points at a temporary directory which will be cleaned up after the test has run. Adjust test cases to use this rather than hard-coding paths in /tmp. Modified: trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,4 +2,4 @@ test "See that the VCL compiler works" -shell "cd ../varnishd && ./varnishd -b 127.0.0.1:80 -C -n /tmp/__v1" +shell "cd ../varnishd && ./varnishd -b 127.0.0.1:80 -C -n ${tmpdir}" Modified: trunk/varnish-cache/bin/varnishtest/tests/b00014.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00014.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/b00014.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -12,19 +12,17 @@ txresp -body "bar" } -start -shell "echo 'backend foo { .host = \"${s1_addr}\"; .port = \"${s1_port}\"; }' > /tmp/_b00014.vcl" -varnish v1 -arg "-f /tmp/_b00014.vcl" -start +shell "echo 'backend foo { .host = \"${s1_addr}\"; .port = \"${s1_port}\"; }' > ${tmpdir}/_b00014.vcl" +varnish v1 -arg "-f ${tmpdir}/_b00014.vcl" -start client c1 { txreq -url /foo rxresp } -run -varnish v1 -cliok "vcl.load foo /tmp/_b00014.vcl" -cliok "vcl.use foo" +varnish v1 -cliok "vcl.load foo ${tmpdir}/_b00014.vcl" -cliok "vcl.use foo" client c1 { txreq -url /bar rxresp } -run - -shell "rm -f /tmp/_b00014.vcl" Modified: trunk/varnish-cache/bin/varnishtest/tests/p00000.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00000.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00000.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -7,11 +7,11 @@ txresp } -start -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" varnish v1 \ -arg "-pdiag_bitmap=0x20000" \ - -arg "-spersistent,/tmp/__v1/_.per,10m" -vcl+backend { } -start + -arg "-spersistent,${tmpdir}/_.per,10m" -vcl+backend { } -start varnish v1 -stop Modified: trunk/varnish-cache/bin/varnishtest/tests/p00001.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00001.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00001.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Change TTL on persistent object" -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" server s1 { rxreq @@ -11,7 +11,7 @@ varnish v1 \ -arg "-pdiag_bitmap=0x20000" \ - -arg "-spersistent,/tmp/__v1/_.per,10m" -vcl+backend { } -start + -arg "-spersistent,${tmpdir}/_.per,10m" -vcl+backend { } -start client c1 { txreq -url "/" Modified: trunk/varnish-cache/bin/varnishtest/tests/p00002.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00002.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00002.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Ban a persistent object" -shell "rm -f /tmp/__v1/_.per[12]" +shell "rm -f ${tmpdir}/_.per[12]" server s1 { rxreq @@ -11,8 +11,8 @@ varnish v1 \ -arg "-pdiag_bitmap=0x20000" \ - -arg "-spersistent,/tmp/__v1/_.per1,10m" \ - -arg "-spersistent,/tmp/__v1/_.per2,10m" \ + -arg "-spersistent,${tmpdir}/_.per1,10m" \ + -arg "-spersistent,${tmpdir}/_.per2,10m" \ -vcl+backend { } -start client c1 { Modified: trunk/varnish-cache/bin/varnishtest/tests/p00003.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00003.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00003.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Ban a persistent object" -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" server s1 { rxreq @@ -11,7 +11,7 @@ varnish v1 \ -arg "-pdiag_bitmap=0x20000" \ - -arg "-spersistent,/tmp/__v1/_.per,10m" \ + -arg "-spersistent,${tmpdir}/_.per,10m" \ -vcl+backend { } -start varnish v1 -cliok purge.list Modified: trunk/varnish-cache/bin/varnishtest/tests/p00004.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00004.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00004.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Check object references" -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" server s1 { rxreq @@ -13,7 +13,7 @@ varnish v1 \ -arg "-pdiag_bitmap=0x20000" \ - -arg "-spersistent,/tmp/__v1/_.per,10m" \ + -arg "-spersistent,${tmpdir}/_.per,10m" \ -vcl+backend { } -start client c1 { Modified: trunk/varnish-cache/bin/varnishtest/tests/p00005.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00005.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00005.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Check expiry of non-instantiated object" -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" server s1 { rxreq @@ -11,7 +11,7 @@ varnish v1 \ -arg "-pdiag_bitmap=0x30000" \ - -arg "-spersistent,/tmp/__v1/_.per,10m" \ + -arg "-spersistent,${tmpdir}/_.per,10m" \ -vcl+backend { sub vcl_fetch { set beresp.ttl = 3s; Modified: trunk/varnish-cache/bin/varnishtest/tests/p00006.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/p00006.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/p00006.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,7 +2,7 @@ test "Check that Vary headers are stored" -shell "rm -f /tmp/__v1/_.per" +shell "rm -f ${tmpdir}/_.per" server s1 { rxreq @@ -13,7 +13,7 @@ varnish v1 \ - -arg "-spersistent,/tmp/__v1/_.per,10m" \ + -arg "-spersistent,${tmpdir}/_.per,10m" \ -vcl+backend { } -start client c1 { Modified: trunk/varnish-cache/bin/varnishtest/tests/v00016.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00016.vtc 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/tests/v00016.vtc 2010-01-13 14:55:46 UTC (rev 4455) @@ -2,19 +2,19 @@ test "Various VCL compiler coverage tests" -shell "true > /tmp/_varnishtest_empty_file" +shell "true > ${tmpdir}/_varnishtest_empty_file" varnish v1 -vcl { backend b { .host = "127.0.0.1"; } - include "/tmp/_varnishtest_empty_file" ; + include "${tmpdir}/_varnishtest_empty_file" ; } varnish v1 -badvcl { backend b { .host = "127.0.0.1"; } - include "/tmp/_varnishtest_empty_file" | + include "${tmpdir}/_varnishtest_empty_file" | } -shell "rm -f /tmp/_varnishtest_empty_file" +shell "rm -f ${tmpdir}/_varnishtest_empty_file" varnish v1 -badvcl { backend b { .host = "127.0.0.1"; } Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-13 13:52:24 UTC (rev 4454) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-13 14:55:46 UTC (rev 4455) @@ -40,6 +40,8 @@ #include #include #include +#include +#include #include "compat/asprintf.h" #include "libvarnish.h" @@ -509,6 +511,7 @@ static struct vtclog *vl; double tmax, t0, t00; const char *nmax; + char *tmpdir, *cmd; setbuf(stdout, NULL); setbuf(stderr, NULL); @@ -538,6 +541,10 @@ init_macro(); init_sema(); + tmpdir = tempnam(NULL, "vtc"); + AN(tmpdir); + mkdir(tmpdir, 0700); + macro_def(vl, NULL, "tmpdir", tmpdir); vtc_thread = pthread_self(); tmax = 0; nmax = NULL; @@ -558,6 +565,13 @@ break; } + /* XXX this will always remove the tmpdir even on failures. + * Maybe we should keep it in that case? */ + assert(asprintf(&cmd, "rm -rf %s", tmpdir) > 0); + AZ(system(cmd)); + free(tmpdir); + free(cmd); + if (vtc_error) return (2); From phk at projects.linpro.no Wed Jan 13 22:04:15 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 13 Jan 2010 23:04:15 +0100 (CET) Subject: r4456 - in trunk/varnish-cache: . bin/varnishd bin/varnishtest/tests Message-ID: <20100113220415.91AB11F7C33@projects.linpro.no> Author: phk Date: 2010-01-13 23:04:15 +0100 (Wed, 13 Jan 2010) New Revision: 4456 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_center.c trunk/varnish-cache/bin/varnishd/cache_http.c trunk/varnish-cache/bin/varnishd/cache_pool.c trunk/varnish-cache/bin/varnishd/cache_session.c trunk/varnish-cache/bin/varnishd/cache_wrw.c trunk/varnish-cache/bin/varnishd/heritage.h trunk/varnish-cache/bin/varnishd/mgt_param.c trunk/varnish-cache/bin/varnishd/stevedore.c trunk/varnish-cache/bin/varnishd/stevedore.h trunk/varnish-cache/bin/varnishtest/tests/b00000.vtc trunk/varnish-cache/configure.ac Log: Make the number of HTTP headers a parameter, and allocate only as many as we need for the objects. This passes the tests, but performance tests have not been performed and there may be alignment issues on platforms with strict alignment. If this works out, the benefits will include: A) no need to recompile if you need more than 64 HTTP headers. B) You do not waste a kilobyte per object, on space for headers you do not have. Reports very welcome! Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-13 22:04:15 UTC (rev 4456) @@ -71,16 +71,8 @@ HTTP_HDR_RESPONSE, /* HTTP header lines */ HTTP_HDR_FIRST, - HTTP_HDR_MAX = HTTP_HDR_MAX_VAL }; -/* Note: intentionally not IOV_MAX unless it has to be */ -#if (IOV_MAX < (HTTP_HDR_MAX_VAL * 2)) -# define MAX_IOVS IOV_MAX -#else -# define MAX_IOVS (HTTP_HDR_MAX_VAL * 2) -#endif - struct cli; struct vsb; struct sess; @@ -154,11 +146,12 @@ int status; double protover; - txt hd[HTTP_HDR_MAX]; - unsigned char hdf[HTTP_HDR_MAX]; + unsigned shd; /* Size of hd space */ + txt *hd; + unsigned char *hdf; #define HDF_FILTER (1 << 0) /* Filtered by Connection */ #define HDF_COPY (1 << 1) /* Copy this field */ - unsigned nhd; + unsigned nhd; /* Next free hd */ }; /*-------------------------------------------------------------------- @@ -214,8 +207,9 @@ int *wfd; unsigned werr; /* valid after WRK_Flush() */ - struct iovec iov[MAX_IOVS]; - int niov; + struct iovec *iov; + unsigned siov; + unsigned niov; ssize_t liov; struct VCL_conf *vcl; @@ -227,7 +221,7 @@ struct http_conn htc[1]; struct ws ws[1]; - struct http http[3]; + struct http *http[3]; struct http *bereq; struct http *beresp1; struct http *beresp; @@ -341,7 +335,7 @@ double last_modified; double last_lru; - struct http http[1]; + struct http *http; VTAILQ_HEAD(, storage) store; @@ -513,8 +507,10 @@ void Fetch_Init(void); /* cache_http.c */ +unsigned HTTP_estimate(unsigned nhttp); +struct http *HTTP_create(void *p, unsigned nhttp); const char *http_StatusMessage(unsigned); -unsigned http_EstimateWS(const struct http *fm, unsigned how); +unsigned http_EstimateWS(const struct http *fm, unsigned how, unsigned *nhd); void HTTP_Init(void); void http_ClrHeader(struct http *to); unsigned http_Write(struct worker *w, const struct http *hp, int resp); Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -184,7 +184,7 @@ sp->obj->last_lru = sp->t_resp; /* XXX: locking ? */ sp->obj->last_use = sp->t_resp; /* XXX: locking ? */ } - sp->wrk->resp = &sp->wrk->http[2]; + sp->wrk->resp = sp->wrk->http[2]; http_Setup(sp->wrk->resp, sp->wrk->ws); RES_BuildHttp(sp); VCL_deliver_method(sp); @@ -336,7 +336,7 @@ if (sp->obj == NULL) { HSH_Prealloc(sp); sp->wrk->cacheable = 0; - sp->obj = STV_NewObject(sp, 0, 0); + sp->obj = STV_NewObject(sp, 0, 0, params->http_headers); sp->obj->xid = sp->xid; sp->obj->entered = sp->t_req; } else { @@ -421,7 +421,7 @@ int i; struct http *hp, *hp2; char *b; - unsigned handling, l; + unsigned handling, l, nhttp; int varyl = 0; struct vsb *vary = NULL; @@ -432,7 +432,7 @@ AZ(sp->vbe); /* sp->wrk->http[0] is (still) bereq */ - sp->wrk->beresp = &sp->wrk->http[1]; + sp->wrk->beresp = sp->wrk->http[1]; http_Setup(sp->wrk->beresp, sp->wrk->ws); i = FetchHdr(sp); @@ -441,7 +441,7 @@ * Save a copy before it might get mangled in VCL. When it comes to * dealing with the body, we want to see the unadultered headers. */ - sp->wrk->beresp1 = &sp->wrk->http[2]; + sp->wrk->beresp1 = sp->wrk->http[2]; *sp->wrk->beresp1 = *sp->wrk->beresp; if (i) { @@ -529,7 +529,7 @@ AZ(sp->objcore); } - l = http_EstimateWS(sp->wrk->beresp, HTTPH_A_INS); + l = http_EstimateWS(sp->wrk->beresp, HTTPH_A_INS, &nhttp); if (vary != NULL) l += varyl; @@ -542,7 +542,7 @@ * XXX: also. */ - sp->obj = STV_NewObject(sp, l, sp->wrk->ttl); + sp->obj = STV_NewObject(sp, l, sp->wrk->ttl, nhttp); if (sp->objhead != NULL) { CHECK_OBJ_NOTNULL(sp->objhead, OBJHEAD_MAGIC); @@ -861,7 +861,7 @@ AN(sp->objcore); AN(sp->objhead); WS_Reset(sp->wrk->ws, NULL); - sp->wrk->bereq = &sp->wrk->http[0]; + sp->wrk->bereq = sp->wrk->http[0]; http_Setup(sp->wrk->bereq, sp->wrk->ws); http_FilterHeader(sp, HTTPH_R_FETCH); VCL_miss_method(sp); @@ -926,7 +926,7 @@ AZ(sp->obj); WS_Reset(sp->wrk->ws, NULL); - sp->wrk->bereq = &sp->wrk->http[0]; + sp->wrk->bereq = sp->wrk->http[0]; http_Setup(sp->wrk->bereq, sp->wrk->ws); http_FilterHeader(sp, HTTPH_R_PASS); @@ -976,7 +976,7 @@ sp->acct_req.pipe++; WS_Reset(sp->wrk->ws, NULL); - sp->wrk->bereq = &sp->wrk->http[0]; + sp->wrk->bereq = sp->wrk->http[0]; http_Setup(sp->wrk->bereq, sp->wrk->ws); http_FilterHeader(sp, HTTPH_R_PIPE); Modified: trunk/varnish-cache/bin/varnishd/cache_http.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -117,14 +117,49 @@ /*--------------------------------------------------------------------*/ +unsigned +HTTP_estimate(unsigned nhttp) +{ + + /* XXX: We trust the structs to size-aligned as necessary */ + return (sizeof (struct http) + (sizeof (txt) + 1) * nhttp); +} + +struct http * +HTTP_create(void *p, unsigned nhttp) +{ + struct http *hp; + + hp = p; + hp->magic = HTTP_MAGIC; + hp->hd = (void*)(hp + 1); + hp->shd = nhttp; + hp->hdf = (void*)(hp->hd + nhttp); + return (hp); +} + +/*--------------------------------------------------------------------*/ + void http_Setup(struct http *hp, struct ws *ws) { + unsigned shd; + txt *hd; + unsigned char *hdf; + /* XXX: This is not elegant, is it efficient ? */ + shd = hp->shd; + hd = hp->hd; + hdf = hp->hdf; memset(hp, 0, sizeof *hp); + memset(hd, 0, sizeof *hd * shd); + memset(hdf, 0, sizeof *hdf * shd); hp->magic = HTTP_MAGIC; hp->ws = ws; hp->nhd = HTTP_HDR_FIRST; + hp->shd = shd; + hp->hd = hd; + hp->hdf = hdf; } /*--------------------------------------------------------------------*/ @@ -367,7 +402,7 @@ q--; *q = '\0'; - if (hp->nhd < HTTP_HDR_MAX) { + if (hp->nhd < hp->shd) { hp->hdf[hp->nhd] = 0; hp->hd[hp->nhd].b = p; hp->hd[hp->nhd].e = q; @@ -556,7 +591,7 @@ http_SetH(struct http *to, unsigned n, const char *fm) { - assert(n < HTTP_HDR_MAX); + assert(n < to->shd); AN(fm); to->hd[n].b = TRUST_ME(fm); to->hd[n].e = strchr(to->hd[n].b, '\0'); @@ -567,7 +602,7 @@ http_copyh(struct http *to, const struct http *fm, unsigned n) { - assert(n < HTTP_HDR_MAX); + assert(n < to->shd); Tcheck(fm->hd[n]); to->hd[n] = fm->hd[n]; to->hdf[n] = fm->hdf[n]; @@ -612,9 +647,9 @@ CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC); CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - assert(n < HTTP_HDR_MAX); + assert(n < to->shd); Tcheck(fm->hd[n]); - if (to->nhd < HTTP_HDR_MAX) { + if (to->nhd < to->shd) { to->hd[to->nhd] = fm->hd[n]; to->hdf[to->nhd] = 0; to->nhd++; @@ -630,11 +665,12 @@ */ unsigned -http_EstimateWS(const struct http *fm, unsigned how) +http_EstimateWS(const struct http *fm, unsigned how, unsigned *nhd) { unsigned u, l; l = 0; + *nhd = HTTP_HDR_FIRST; CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC); for (u = 0; u < fm->nhd; u++) { if (fm->hd[u].b == NULL) @@ -647,6 +683,7 @@ #include "http_headers.h" #undef HTTPH l += Tlen(fm->hd[u]) + 1; + (*nhd)++; // fm->hdf[u] |= HDF_COPY; } return (l); @@ -755,7 +792,7 @@ { CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - if (to->nhd >= HTTP_HDR_MAX) { + if (to->nhd >= to->shd) { VSL_stats->losthdr++; WSL(w, SLT_LostHeader, fd, "%s", hdr); return; @@ -827,7 +864,7 @@ va_start(ap, fmt); n = vsnprintf(to->ws->f, l, fmt, ap); va_end(ap); - if (n + 1 >= l || to->nhd >= HTTP_HDR_MAX) { + if (n + 1 >= l || to->nhd >= to->shd) { VSL_stats->losthdr++; WSL(w, SLT_LostHeader, fd, "%s", to->ws->f); WS_Release(to->ws, 0); Modified: trunk/varnish-cache/bin/varnishd/cache_pool.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -117,11 +117,16 @@ /*--------------------------------------------------------------------*/ static void * -wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace) +wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace, + unsigned nhttp, unsigned http_space, unsigned siov) { struct worker *w, ww; unsigned char wlog[shm_workspace]; unsigned char ws[sess_workspace]; + unsigned char http0[http_space]; + unsigned char http1[http_space]; + unsigned char http2[http_space]; + struct iovec iov[siov]; struct SHA256Context sha256; int stats_clean; @@ -133,6 +138,11 @@ w->wlb = w->wlp = wlog; w->wle = wlog + sizeof wlog; w->sha256ctx = &sha256; + w->http[0] = HTTP_create(http0, nhttp); + w->http[1] = HTTP_create(http1, nhttp); + w->http[2] = HTTP_create(http2, nhttp); + w->iov = iov; + w->siov = siov; AZ(pthread_cond_init(&w->cond, NULL)); WS_Init(w->ws, "wrk", ws, sess_workspace); @@ -202,12 +212,19 @@ wrk_thread(void *priv) { struct wq *qp; + volatile unsigned nhttp; + unsigned siov; CAST_OBJ_NOTNULL(qp, priv, WQ_MAGIC); /* We need to snapshot these two for consistency */ + nhttp = params->http_headers; + siov = nhttp * 2; /* XXX param ? */ + if (siov > IOV_MAX) + siov = IOV_MAX; return (wrk_thread_real(qp, params->shm_workspace, - params->sess_workspace)); + params->sess_workspace, + nhttp, HTTP_estimate(nhttp), siov)); } /*-------------------------------------------------------------------- Modified: trunk/varnish-cache/bin/varnishd/cache_session.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -60,8 +60,9 @@ #define SESSMEM_MAGIC 0x555859c5 struct sess sess; - struct http http[2]; unsigned workspace; + void *wsp; + struct http *http[2]; VTAILQ_ENTRY(sessmem) list; struct sockaddr_storage sockaddr[2]; }; @@ -99,25 +100,40 @@ ses_setup(struct sessmem *sm, const struct sockaddr *addr, unsigned len) { struct sess *sp; - volatile unsigned u; + unsigned char *p; + volatile unsigned nws; + volatile unsigned nhttp; + unsigned l, hl; if (sm == NULL) { if (VSL_stats->n_sess_mem >= params->max_sess) return (NULL); /* - * It is not necessary to lock mem_workspace, but we - * need to cache it locally, to make sure we get a - * consistent view of it. + * It is not necessary to lock these, but we need to + * cache them locally, to make sure we get a consistent + * view of the value. */ - u = params->sess_workspace; - sm = malloc(sizeof *sm + u); - if (sm == NULL) + nws = params->sess_workspace; + nhttp = params->http_headers; + hl = HTTP_estimate(nhttp); + l = sizeof *sm + nws + 2 * hl; + p = malloc(l); + if (p == NULL) return (NULL); + /* Don't waste time zeroing the workspace */ - memset(sm, 0, sizeof *sm); + memset(p, 0, l - nws); + + sm = (void*)p; + p += sizeof *sm; sm->magic = SESSMEM_MAGIC; - sm->workspace = u; + sm->workspace = nws; VSL_stats->n_sess_mem++; + sm->http[0] = HTTP_create(p, nhttp); + p += hl; + sm->http[1] = HTTP_create(p, nhttp); + p += hl; + sm->wsp = p; } CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC); VSL_stats->n_sess++; @@ -142,9 +158,9 @@ sp->sockaddrlen = len; } - WS_Init(sp->ws, "sess", (void *)(sm + 1), sm->workspace); - sp->http = &sm->http[0]; - sp->http0 = &sm->http[1]; + WS_Init(sp->ws, "sess", sm->wsp, sm->workspace); + sp->http = sm->http[0]; + sp->http0 = sm->http[1]; SES_ResetBackendTimeouts(sp); @@ -193,7 +209,7 @@ { struct acct *b = &sp->acct; struct sessmem *sm; - unsigned workspace; + // unsigned workspace; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); sm = sp->mem; @@ -213,10 +229,10 @@ free(sm); } else { /* Clean and prepare for reuse */ - workspace = sm->workspace; - memset(sm, 0, sizeof *sm); - sm->magic = SESSMEM_MAGIC; - sm->workspace = workspace; + // workspace = sm->workspace; + // memset(sm, 0, sizeof *sm); + // sm->magic = SESSMEM_MAGIC; + // sm->workspace = workspace; Lck_Lock(&ses_mem_mtx); VTAILQ_INSERT_HEAD(&ses_free_mem[1 - ses_qp], sm, list); Modified: trunk/varnish-cache/bin/varnishd/cache_wrw.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_wrw.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/cache_wrw.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -153,7 +153,7 @@ return (0); if (len == -1) len = strlen(ptr); - if (w->niov == MAX_IOVS) + if (w->niov == w->siov) (void)WRW_Flush(w); w->iov[w->niov].iov_base = TRUST_ME(ptr); w->iov[w->niov].iov_len = len; @@ -193,7 +193,7 @@ } while (0); #elif defined(__sun) && defined(HAVE_SENDFILEV) do { - sendfilevec_t svvec[HTTP_HDR_MAX * 2 + 1]; + sendfilevec_t svvec[params->http_headers * 2 + 1]; size_t xferred = 0, expected = 0; int i; for (i = 0; i < w->niov; i++) { Modified: trunk/varnish-cache/bin/varnishd/heritage.h =================================================================== --- trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-13 22:04:15 UTC (rev 4456) @@ -101,6 +101,7 @@ unsigned sess_workspace; unsigned obj_workspace; unsigned shm_workspace; + unsigned http_headers; unsigned shm_reclen; Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -493,6 +493,12 @@ "Minimum is 1024 bytes.", DELAYED_EFFECT, "16384", "bytes" }, + { "http_headers", tweak_uint, &master.http_headers, 32, UINT_MAX, + "Maximum number of HTTP headers we will deal with.\n" + "This space is preallocated in sessions and workthreads only " + "objects allocate only space for the headers they store.\n", + 0, + "64", "header lines" }, { "obj_workspace", tweak_uint, &master.obj_workspace, 0, UINT_MAX, "Bytes of HTTP protocol workspace allocated for objects. " "This space must be big enough for the entire HTTP protocol " Modified: trunk/varnish-cache/bin/varnishd/stevedore.c =================================================================== --- trunk/varnish-cache/bin/varnishd/stevedore.c 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/stevedore.c 2010-01-13 22:04:15 UTC (rev 4456) @@ -88,13 +88,15 @@ /*********************************************************************/ static void -STV_InitObj(struct sess *sp, struct object *o, unsigned wsl) +STV_InitObj(struct sess *sp, struct object *o, unsigned wsl, unsigned lhttp, + unsigned nhttp) { memset(o, 0, sizeof *o); o->magic = OBJECT_MAGIC; - WS_Init(o->ws_o, "obj", (o + 1), wsl); + o->http = HTTP_create(o + 1, nhttp); + WS_Init(o->ws_o, "obj", (char *)(o + 1) + lhttp, wsl); WS_Assert(o->ws_o); http_Setup(o->http, o->ws_o); @@ -109,10 +111,11 @@ /*********************************************************************/ struct object * -STV_NewObject(struct sess *sp, unsigned l, double ttl) +STV_NewObject(struct sess *sp, unsigned l, double ttl, unsigned nhttp) { struct object *o; struct storage *st; + unsigned lh; (void)ttl; if (l == 0) @@ -120,21 +123,23 @@ if (params->obj_workspace > 0 && params->obj_workspace > l) l = params->obj_workspace; + lh = HTTP_estimate(nhttp); + if (!sp->wrk->cacheable) { - o = malloc(sizeof *o + l); + o = malloc(sizeof *o + l + lh); XXXAN(o); - STV_InitObj(sp, o, l); + STV_InitObj(sp, o, l, lh, nhttp); return (o); } - st = STV_alloc(sp, sizeof *o + l, sp->objcore); + st = STV_alloc(sp, sizeof *o + l + lh, sp->objcore); XXXAN(st); - xxxassert(st->space >= (sizeof *o + l)); + xxxassert(st->space >= (sizeof *o + l + lh)); st->len = st->space; o = (void *)st->ptr; /* XXX: align ? */ - STV_InitObj(sp, o, st->space - sizeof *o); + STV_InitObj(sp, o, st->space - (sizeof *o + lh), lh, nhttp); o->objstore = st; return (o); } Modified: trunk/varnish-cache/bin/varnishd/stevedore.h =================================================================== --- trunk/varnish-cache/bin/varnishd/stevedore.h 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishd/stevedore.h 2010-01-13 22:04:15 UTC (rev 4456) @@ -68,7 +68,8 @@ VTAILQ_ENTRY(stevedore) list; }; -struct object *STV_NewObject(struct sess *sp, unsigned len, double ttl); +struct object *STV_NewObject(struct sess *sp, unsigned len, double ttl, + unsigned nhttp); struct storage *STV_alloc(struct sess *sp, size_t size, struct objcore *oc); void STV_trim(struct storage *st, size_t size); void STV_free(struct storage *st); Modified: trunk/varnish-cache/bin/varnishtest/tests/b00000.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00000.vtc 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/bin/varnishtest/tests/b00000.vtc 2010-01-13 22:04:15 UTC (rev 4456) @@ -9,7 +9,7 @@ txresp -body "012345\n" } -start -varnish v1 -vcl+backend {} -start +varnish v1 -arg "-smalloc,1m" -vcl+backend {} -start varnish v1 -expect n_object == 0 varnish v1 -expect client_conn == 0 Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2010-01-13 14:55:46 UTC (rev 4455) +++ trunk/varnish-cache/configure.ac 2010-01-13 22:04:15 UTC (rev 4456) @@ -415,13 +415,6 @@ fi AC_DEFINE_UNQUOTED([VCC_CC],"$VCC_CC",[C compiler command line for VCL code]) -# Define HTTP_HDR_MAX_VAL -AC_ARG_WITH(max-header-fields, - AS_HELP_STRING([--with-max-header-fields=NUM], - [How many header fields to support (default=64)]), - [], - [with_max_header_fields=64]) - AC_DEFINE_UNQUOTED(HTTP_HDR_MAX_VAL, $with_max_header_fields, [Define maximum number of header fields supported by varnish ]) # Use jemalloc on Linux From ssm at projects.linpro.no Thu Jan 14 10:01:07 2010 From: ssm at projects.linpro.no (ssm at projects.linpro.no) Date: Thu, 14 Jan 2010 11:01:07 +0100 (CET) Subject: r4457 - trunk/varnish-cache/lib/libjemalloc Message-ID: <20100114100107.D0FAE1F72B4@projects.linpro.no> Author: ssm Date: 2010-01-14 11:01:07 +0100 (Thu, 14 Jan 2010) New Revision: 4457 Modified: trunk/varnish-cache/lib/libjemalloc/jemalloc_linux.c trunk/varnish-cache/lib/libjemalloc/malloc.c Log: Add jemalloc information for the SH architecture Thanks to Nobuhiro Iwamatsu for the patch, at http://bugs.debian.org/565239 Modified: trunk/varnish-cache/lib/libjemalloc/jemalloc_linux.c =================================================================== --- trunk/varnish-cache/lib/libjemalloc/jemalloc_linux.c 2010-01-13 22:04:15 UTC (rev 4456) +++ trunk/varnish-cache/lib/libjemalloc/jemalloc_linux.c 2010-01-14 10:01:07 UTC (rev 4457) @@ -278,6 +278,12 @@ # define QUANTUM_2POW 4 # define SIZEOF_PTR_2POW 3 #endif +#ifdef __sh__ +# define PAGESIZE_2POW 12 +# define QUANTUM_2POW 3 +# define SIZEOF_PTR_2POW 2 +# define NO_TLS +#endif #define QUANTUM ((size_t)(1U << QUANTUM_2POW)) #define QUANTUM_MASK (QUANTUM - 1) Modified: trunk/varnish-cache/lib/libjemalloc/malloc.c =================================================================== --- trunk/varnish-cache/lib/libjemalloc/malloc.c 2010-01-13 22:04:15 UTC (rev 4456) +++ trunk/varnish-cache/lib/libjemalloc/malloc.c 2010-01-14 10:01:07 UTC (rev 4457) @@ -264,6 +264,12 @@ # define QUANTUM_2POW 4 # define SIZEOF_PTR_2POW 2 #endif +#ifdef __sh__ +# define PAGESIZE_2POW 12 +# define QUANTUM_2POW 3 +# define SIZEOF_PTR_2POW 2 +# define NO_TLS +#endif #define QUANTUM ((size_t)(1U << QUANTUM_2POW)) #define QUANTUM_MASK (QUANTUM - 1) From phk at projects.linpro.no Thu Jan 14 11:58:48 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 14 Jan 2010 12:58:48 +0100 (CET) Subject: r4458 - trunk/varnish-cache/include Message-ID: <20100114115848.8BB121F73FF@projects.linpro.no> Author: phk Date: 2010-01-14 12:58:48 +0100 (Thu, 14 Jan 2010) New Revision: 4458 Modified: trunk/varnish-cache/include/libvarnish.h Log: Add a couple of macros for safely (assertedly) printing into a static sized buffer. Modified: trunk/varnish-cache/include/libvarnish.h =================================================================== --- trunk/varnish-cache/include/libvarnish.h 2010-01-14 10:01:07 UTC (rev 4457) +++ trunk/varnish-cache/include/libvarnish.h 2010-01-14 11:58:48 UTC (rev 4458) @@ -133,3 +133,17 @@ abort(); \ } while (0) const char* svn_version(void); + +/* Safe printf into a fixed-size buffer */ +#define bprintf(buf, fmt, ...) \ + do { \ + assert(snprintf(buf, sizeof buf, fmt, __VA_ARGS__) \ + < sizeof buf); \ + } while (0) + +/* Safe printf into a fixed-size buffer */ +#define vbprintf(buf, fmt, ap) \ + do { \ + assert(vsnprintf(buf, sizeof buf, fmt, ap) \ + < sizeof buf); \ + } while (0) From phk at projects.linpro.no Thu Jan 14 11:59:57 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 14 Jan 2010 12:59:57 +0100 (CET) Subject: r4459 - in trunk/varnish-cache: bin/varnishtest lib/libvcl Message-ID: <20100114115957.AE6161F750D@projects.linpro.no> Author: phk Date: 2010-01-14 12:59:57 +0100 (Thu, 14 Jan 2010) New Revision: 4459 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c trunk/varnish-cache/lib/libvcl/vcc_acl.c trunk/varnish-cache/lib/libvcl/vcc_var.c Log: Use the new bprintf() macros to avoid some (v)asprintf() calls. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-14 11:58:48 UTC (rev 4458) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-14 11:59:57 UTC (rev 4459) @@ -89,8 +89,7 @@ va_list ap; if (instance != NULL) { - assert (snprintf(buf, sizeof buf, "%s_%s", instance, name) - < sizeof buf); + bprintf(buf, "%s_%s", instance, name); name = buf; } @@ -109,8 +108,9 @@ va_start(ap, fmt); free(m->val); m->val = NULL; - assert(vasprintf(&m->val, fmt, ap) >= 0); + vbprintf(buf, fmt, ap); va_end(ap); + m->val = strdup(buf); AN(m->val); vtc_log(vl, 4, "macro def %s=%s", name, m->val); } else if (m != NULL) { Modified: trunk/varnish-cache/lib/libvcl/vcc_acl.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_acl.c 2010-01-14 11:58:48 UTC (rev 4458) +++ trunk/varnish-cache/lib/libvcl/vcc_acl.c 2010-01-14 11:59:57 UTC (rev 4459) @@ -47,7 +47,6 @@ #include "vcc_priv.h" #include "vcc_compile.h" #include "libvarnish.h" -#include "compat/vasprintf.h" struct acl_e { VTAILQ_ENTRY(acl_e) list; @@ -456,7 +455,7 @@ vcc_Cond_Ip(const struct var *vp, struct tokenlist *tl) { unsigned tcond; - char *acln; + char acln[32]; switch (tl->t->tok) { case '~': @@ -473,13 +472,11 @@ VTAILQ_INIT(&tl->acl); tcond = tl->t->tok; vcc_NextToken(tl); - assert(asprintf(&acln, "%u", tl->cnt) > 0); - AN(acln); + bprintf(acln, "%u", tl->cnt); vcc_acl_entry(tl); vcc_acl_emit(tl, acln, 1); Fb(tl, 1, "%smatch_acl_anon_%s(sp, %s)\n", (tcond == T_NEQ ? "!" : ""), acln, vp->rname); - free(acln); break; default: vsb_printf(tl->sb, "Invalid condition "); @@ -495,7 +492,7 @@ vcc_Acl(struct tokenlist *tl) { struct token *an; - char *acln; + char acln[1024]; vcc_NextToken(tl); VTAILQ_INIT(&tl->acl); @@ -505,8 +502,7 @@ vcc_NextToken(tl); vcc_AddDef(tl, an, R_ACL); - assert(asprintf(&acln, "%.*s", PF(an)) > 0); - AN(acln); + bprintf(acln, "%.*s", PF(an)); ExpectErr(tl, '{'); vcc_NextToken(tl); @@ -521,6 +517,4 @@ vcc_NextToken(tl); vcc_acl_emit(tl, acln, 0); - - free(acln); } Modified: trunk/varnish-cache/lib/libvcl/vcc_var.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_var.c 2010-01-14 11:58:48 UTC (rev 4458) +++ trunk/varnish-cache/lib/libvcl/vcc_var.c 2010-01-14 11:59:57 UTC (rev 4459) @@ -40,7 +40,6 @@ #include "vcc_priv.h" #include "vcc_compile.h" #include "libvarnish.h" -#include "compat/vasprintf.h" /*--------------------------------------------------------------------*/ @@ -49,7 +48,8 @@ { char *p; struct var *v; - int i; + int i, l; + char buf[258]; (void)tl; @@ -65,16 +65,22 @@ v->fmt = STRING; v->hdr = vh->hdr; v->methods = vh->methods; - assert(asprintf(&p, "VRT_GetHdr(sp, %s, \"\\%03o%s:\")", v->hdr, - (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len) > 0); - AN(p); - TlFree(tl, p); + l = strlen(v->name + vh->len) + 1; + + bprintf(buf, "VRT_GetHdr(sp, %s, \"\\%03o%s:\")", + v->hdr, (unsigned)l, v->name + vh->len); + i = strlen(buf); + p = TlAlloc(tl, i + 1); + memcpy(p, buf, i + 1); v->rname = p; - assert(asprintf(&p, "VRT_SetHdr(sp, %s, \"\\%03o%s:\", ", v->hdr, - (unsigned)(strlen(v->name + vh->len) + 1), v->name + vh->len) > 0); - AN(p); - TlFree(tl, p); + + bprintf(buf, "VRT_SetHdr(sp, %s, \"\\%03o%s:\", ", + v->hdr, (unsigned)l, v->name + vh->len); + i = strlen(buf); + p = TlAlloc(tl, i + 1); + memcpy(p, buf, i + 1); v->lname = p; + return (v); } From phk at projects.linpro.no Thu Jan 14 13:35:56 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 14 Jan 2010 14:35:56 +0100 (CET) Subject: r4460 - trunk/varnish-cache/bin/varnishd Message-ID: <20100114133556.952C01F7B7A@projects.linpro.no> Author: phk Date: 2010-01-14 14:35:56 +0100 (Thu, 14 Jan 2010) New Revision: 4460 Modified: trunk/varnish-cache/bin/varnishd/cache_backend.h trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c trunk/varnish-cache/bin/varnishd/cache_dir_random.c Log: Experimental "client" director, which chooses backend based on client-ip. This does _not_ take X-Forwarded-For: into account. Feedback welcome. Modified: trunk/varnish-cache/bin/varnishd/cache_backend.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.h 2010-01-14 11:59:57 UTC (rev 4459) +++ trunk/varnish-cache/bin/varnishd/cache_backend.h 2010-01-14 13:35:56 UTC (rev 4460) @@ -160,3 +160,4 @@ dir_init_f VRT_init_dir_hash; dir_init_f VRT_init_dir_random; dir_init_f VRT_init_dir_round_robin; +dir_init_f VRT_init_dir_client; Modified: trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c 2010-01-14 11:59:57 UTC (rev 4459) +++ trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c 2010-01-14 13:35:56 UTC (rev 4460) @@ -275,6 +275,8 @@ VRT_init_dir_random(cli, dir, idx, priv); else if (!strcmp(name, "round-robin")) VRT_init_dir_round_robin(cli, dir, idx, priv); + else if (!strcmp(name, "client")) + VRT_init_dir_client(cli, dir, idx, priv); else INCOMPL(); } Modified: trunk/varnish-cache/bin/varnishd/cache_dir_random.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-14 11:59:57 UTC (rev 4459) +++ trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-14 13:35:56 UTC (rev 4460) @@ -57,6 +57,7 @@ #include "cache.h" #include "cache_backend.h" #include "vrt.h" +#include "vsha256.h" /*--------------------------------------------------------------------*/ @@ -65,12 +66,14 @@ double weight; }; +enum crit_e {c_random, c_hash, c_client}; + struct vdi_random { unsigned magic; #define VDI_RANDOM_MAGIC 0x3771ae23 struct director dir; - unsigned use_hash; + enum crit_e criteria; unsigned retries; double tot_weight; struct vdi_random_host *hosts; @@ -86,22 +89,47 @@ unsigned u; struct vbe_conn *vbe; struct director *d2; + struct SHA256Context ctx; + unsigned char sign[SHA256_LEN], *hp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC); + if (vs->criteria == c_client) { + /* + * Hash the client IP# ascii representation, rather than + * rely on the raw IP# being a good hash distributor, since + * experience shows this not to be the case. + * We do not hash the port number, to make everybody behind + * a given NAT gateway fetch from the same backend. + */ + SHA256_Init(&ctx); + AN(sp->addr); + SHA256_Update(&ctx, sp->addr, strlen(sp->addr)); + SHA256_Final(sign, &ctx); + hp = sign; + } + if (vs->criteria == c_hash) { + /* + * Reuse the hash-string, the objective here is to fetch the + * same object on the same backend all the time + */ + hp = sp->digest; + } + /* * If we are hashing, first try to hit our "canonical backend" * If that fails, we fall through, and select a weighted backend - * amongst the good set. + * amongst the healthy set. */ - if (vs->use_hash) { - u = sp->digest[3] << 24; - u |= sp->digest[2] << 16; - u |= sp->digest[1] << 8; - u |= sp->digest[0] << 0; + if (vs->criteria != c_random) { + u = hp[3] << 24; + u |= hp[2] << 16; + u |= hp[1] << 8; + u |= hp[0] << 0; r = u / 4294967296.0; + assert(r >= 0.0 && r < 1.0); r *= vs->tot_weight; s1 = 0.0; for (i = 0; i < vs->nhosts; i++) { @@ -119,7 +147,6 @@ } for (k = 0; k < vs->retries; ) { - /* Sum up the weights of healty backends */ s1 = 0.0; for (i = 0; i < vs->nhosts; i++) { @@ -132,11 +159,7 @@ if (s1 == 0.0) return (NULL); - if (vs->use_hash) { - u = sp->digest[3] << 24; - u |= sp->digest[2] << 16; - u |= sp->digest[1] << 8; - u |= sp->digest[0] << 0; + if (vs->criteria != c_random) { r = u / 4294967296.0; } else { /* Pick a random threshold in that interval */ @@ -197,7 +220,7 @@ static void vrt_init(struct cli *cli, struct director **bp, int idx, - const void *priv, int use_hash) + const void *priv, enum crit_e criteria) { const struct vrt_dir_random *t; struct vdi_random *vs; @@ -222,7 +245,7 @@ vs->dir.fini = vdi_random_fini; vs->dir.healthy = vdi_random_healthy; - vs->use_hash = use_hash; + vs->criteria = criteria; vs->retries = t->retries; if (vs->retries == 0) vs->retries = t->nmember; @@ -244,12 +267,19 @@ VRT_init_dir_random(struct cli *cli, struct director **bp, int idx, const void *priv) { - vrt_init(cli, bp, idx, priv, 0); + vrt_init(cli, bp, idx, priv, c_random); } void VRT_init_dir_hash(struct cli *cli, struct director **bp, int idx, const void *priv) { - vrt_init(cli, bp, idx, priv, 1); + vrt_init(cli, bp, idx, priv, c_hash); } + +void +VRT_init_dir_client(struct cli *cli, struct director **bp, int idx, + const void *priv) +{ + vrt_init(cli, bp, idx, priv, c_client); +} From phk at projects.linpro.no Thu Jan 14 13:44:59 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 14 Jan 2010 14:44:59 +0100 (CET) Subject: r4461 - trunk/varnish-cache/lib/libvcl Message-ID: <20100114134459.472571F7B7A@projects.linpro.no> Author: phk Date: 2010-01-14 14:44:58 +0100 (Thu, 14 Jan 2010) New Revision: 4461 Modified: trunk/varnish-cache/lib/libvcl/vcc_backend.c Log: Add the VCC magic to recognize the "client" director Modified: trunk/varnish-cache/lib/libvcl/vcc_backend.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_backend.c 2010-01-14 13:35:56 UTC (rev 4460) +++ trunk/varnish-cache/lib/libvcl/vcc_backend.c 2010-01-14 13:44:58 UTC (rev 4461) @@ -778,6 +778,7 @@ } dirlist[] = { { "hash", vcc_ParseRandomDirector }, { "random", vcc_ParseRandomDirector }, + { "client", vcc_ParseRandomDirector }, { "round-robin", vcc_ParseRoundRobinDirector }, { NULL, NULL } }; From phk at projects.linpro.no Fri Jan 15 13:22:01 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 15 Jan 2010 14:22:01 +0100 (CET) Subject: r4462 - trunk/varnish-cache/bin/varnishd Message-ID: <20100115132201.BC77B38587@projects.linpro.no> Author: phk Date: 2010-01-15 14:22:01 +0100 (Fri, 15 Jan 2010) New Revision: 4462 Modified: trunk/varnish-cache/bin/varnishd/mgt.h trunk/varnish-cache/bin/varnishd/mgt_vcc.c trunk/varnish-cache/bin/varnishd/varnishd.c Log: Make add info to the boot VCL about where it came from, -b or -f argument. Modified: trunk/varnish-cache/bin/varnishd/mgt.h =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-14 13:44:58 UTC (rev 4461) +++ trunk/varnish-cache/bin/varnishd/mgt.h 2010-01-15 13:22:01 UTC (rev 4462) @@ -66,7 +66,7 @@ /* mgt_vcc.c */ void mgt_vcc_init(void); -int mgt_vcc_default(const char *bflag, char *vcl, int Cflag); +int mgt_vcc_default(const char *bflag, const char *f_arg, char *vcl, int Cflag); int mgt_push_vcls_and_start(unsigned *status, char **p); int mgt_has_vcl(void); extern char *mgt_cc_cmd; Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-14 13:44:58 UTC (rev 4461) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-15 13:22:01 UTC (rev 4462) @@ -325,12 +325,13 @@ /*--------------------------------------------------------------------*/ int -mgt_vcc_default(const char *b_arg, char *vcl, int C_flag) +mgt_vcc_default(const char *b_arg, const char *f_arg, char *vcl, int C_flag) { char *addr, *port; char *vf; struct vsb *sb; struct vclprog *vp; + char buf[BUFSIZ]; if (b_arg != NULL) { AZ(vcl); @@ -363,6 +364,9 @@ free(addr); free(port); AN(vcl); + bprintf(buf, "boot (-b %s)", b_arg); + } else { + brintf(buf, "boot (-f %s)", f_arg); } vf = mgt_VccCompile(&sb, vcl, C_flag); @@ -379,7 +383,7 @@ fprintf(stderr, "\nVCL compilation failed\n"); return (1); } - vp = mgt_vcc_add("boot", vf); + vp = mgt_vcc_add(buf, vf); vp->active = 1; return (0); } Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-14 13:44:58 UTC (rev 4461) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-15 13:22:01 UTC (rev 4462) @@ -727,7 +727,7 @@ } if (b_arg != NULL || f_arg != NULL) - if (mgt_vcc_default(b_arg, vcl, C_flag)) + if (mgt_vcc_default(b_arg, f_arg, vcl, C_flag)) exit (2); if (C_flag) From phk at projects.linpro.no Fri Jan 15 13:23:22 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 15 Jan 2010 14:23:22 +0100 (CET) Subject: r4463 - trunk/varnish-cache/bin/varnishd Message-ID: <20100115132322.BFA403858A@projects.linpro.no> Author: phk Date: 2010-01-15 14:23:22 +0100 (Fri, 15 Jan 2010) New Revision: 4463 Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c Log: typo. Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-15 13:22:01 UTC (rev 4462) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-15 13:23:22 UTC (rev 4463) @@ -366,7 +366,7 @@ AN(vcl); bprintf(buf, "boot (-b %s)", b_arg); } else { - brintf(buf, "boot (-f %s)", f_arg); + bprintf(buf, "boot (-f %s)", f_arg); } vf = mgt_VccCompile(&sb, vcl, C_flag); From phk at projects.linpro.no Fri Jan 15 13:46:23 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 15 Jan 2010 14:46:23 +0100 (CET) Subject: r4464 - trunk/varnish-cache/bin/varnishd Message-ID: <20100115134623.C01CE3873A@projects.linpro.no> Author: phk Date: 2010-01-15 14:46:23 +0100 (Fri, 15 Jan 2010) New Revision: 4464 Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c Log: Add necessary quoting. Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-15 13:23:22 UTC (rev 4463) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-15 13:46:23 UTC (rev 4464) @@ -406,13 +406,13 @@ VTAILQ_FOREACH(vp, &vclhead, list) { if (mgt_cli_askchild(status, p, - "vcl.load %s %s\n", vp->name, vp->fname)) + "vcl.load \"%s\" %s\n", vp->name, vp->fname)) return (1); free(*p); if (!vp->active) continue; if (mgt_cli_askchild(status, p, - "vcl.use %s\n", vp->name)) + "vcl.use \"%s\"\n", vp->name)) return (1); free(*p); } From phk at projects.linpro.no Fri Jan 15 13:46:46 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 15 Jan 2010 14:46:46 +0100 (CET) Subject: r4465 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100115134646.7474738CD9@projects.linpro.no> Author: phk Date: 2010-01-15 14:46:46 +0100 (Fri, 15 Jan 2010) New Revision: 4465 Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c Log: Use proper tests instead of asserts. Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-15 13:46:23 UTC (rev 4464) +++ trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-15 13:46:46 UTC (rev 4465) @@ -337,18 +337,25 @@ if (vtc_error) return; vtc_log(v->vl, 2, "Start"); - u = varnish_ask_cli(v, "start", NULL); + u = varnish_ask_cli(v, "start", &resp); if (vtc_error) return; - assert(u == CLIS_OK); - u = varnish_ask_cli(v, "debug.xid 1000", NULL); + if (u != CLIS_OK) + vtc_log(v->vl, 0, "CLI start command failed: %u %s", u, resp); + free(resp); + u = varnish_ask_cli(v, "debug.xid 1000", &resp); if (vtc_error) return; - assert(u == CLIS_OK); + if (u != CLIS_OK) + vtc_log(v->vl, 0, "CLI debug.xid command failed: %u %s", + u, resp); + free(resp); u = varnish_ask_cli(v, "debug.listen_address", &resp); if (vtc_error) return; - assert(u == CLIS_OK); + if (u != CLIS_OK) + vtc_log(v->vl, 0, + "CLI debug.listen_address command failed: %u %s", u, resp); h = resp; p = strchr(h, ' '); AN(p); From phk at projects.linpro.no Fri Jan 15 15:11:08 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 15 Jan 2010 16:11:08 +0100 (CET) Subject: r4466 - trunk/varnish-cache/bin/varnishd Message-ID: <20100115151108.5E4572811B@projects.linpro.no> Author: phk Date: 2010-01-15 16:11:07 +0100 (Fri, 15 Jan 2010) New Revision: 4466 Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c Log: Report correct remote address for telnet CLI connections Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-15 13:46:46 UTC (rev 4465) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-15 15:11:07 UTC (rev 4466) @@ -598,7 +598,7 @@ return (0); tn = telnet_new(i); - vsb = sock_id("telnet", ev->fd); + vsb = sock_id("telnet", i); mgt_cli_setup(i, i, 0, vsb_data(vsb), telnet_close, tn); vsb_delete(vsb); return (0); From phk at projects.linpro.no Mon Jan 18 13:30:47 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 18 Jan 2010 14:30:47 +0100 (CET) Subject: r4467 - in trunk/varnish-cache/bin: varnishd varnishtest/tests Message-ID: <20100118133047.E6C571F73C0@projects.linpro.no> Author: phk Date: 2010-01-18 14:30:47 +0100 (Mon, 18 Jan 2010) New Revision: 4467 Added: trunk/varnish-cache/bin/varnishtest/tests/b00031.vtc Modified: trunk/varnish-cache/bin/varnishd/cache_http.c trunk/varnish-cache/bin/varnishd/default.vcl Log: Produce the X-Forwarded-For: header in vcl_recv, so people can tweak as they want. Append to already existing header if possible. NB: If you return early from your own vcl_recv, without pasting these lines in top of your vcl_recv, your backend gets no X-F-F header. Fixes #601 Fixes #540 Modified: trunk/varnish-cache/bin/varnishd/cache_http.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-15 15:11:07 UTC (rev 4466) +++ trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-18 13:30:47 UTC (rev 4467) @@ -734,8 +734,6 @@ http_copyh(hp, sp->http, HTTP_HDR_PROTO); http_FilterFields(sp->wrk, sp->fd, hp, sp->http, how); http_PrintfHeader(sp->wrk, sp->fd, hp, "X-Varnish: %u", sp->xid); - http_PrintfHeader(sp->wrk, sp->fd, hp, - "X-Forwarded-For: %s", sp->addr); } /*-------------------------------------------------------------------- Modified: trunk/varnish-cache/bin/varnishd/default.vcl =================================================================== --- trunk/varnish-cache/bin/varnishd/default.vcl 2010-01-15 15:11:07 UTC (rev 4466) +++ trunk/varnish-cache/bin/varnishd/default.vcl 2010-01-18 13:30:47 UTC (rev 4467) @@ -40,6 +40,12 @@ */ sub vcl_recv { + if (req.http.x-forwarded-for) { + set req.http.X-Forwarded-For = + req.http.X-Forwarded-For ", " client.ip; + } else { + set req.http.X-Forwarded-For = client.ip; + } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && Added: trunk/varnish-cache/bin/varnishtest/tests/b00031.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00031.vtc (rev 0) +++ trunk/varnish-cache/bin/varnishtest/tests/b00031.vtc 2010-01-18 13:30:47 UTC (rev 4467) @@ -0,0 +1,22 @@ +# $Id$ + +test "Test X-Forward-For headers" + +server s1 { + rxreq + expect req.http.X-Forwarded-For == "127.0.0.1" + txresp + rxreq + expect req.http.X-Forwarded-For == "1.2.3.4, 127.0.0.1" + txresp +} -start + +varnish v1 -vcl+backend { +} -start + +client c1 { + txreq -url /1 + rxresp + txreq -url /2 -hdr "X-forwarded-for: 1.2.3.4" + rxresp +} -run From tfheen at projects.linpro.no Mon Jan 18 14:58:10 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Mon, 18 Jan 2010 15:58:10 +0100 (CET) Subject: r4468 - branches/2.0/varnish-cache/bin/varnishd Message-ID: <20100118145810.14AD338587@projects.linpro.no> Author: tfheen Date: 2010-01-18 15:58:09 +0100 (Mon, 18 Jan 2010) New Revision: 4468 Modified: branches/2.0/varnish-cache/bin/varnishd/cache_center.c Log: Avoid repeated pointless rushing This should address #610 Modified: branches/2.0/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- branches/2.0/varnish-cache/bin/varnishd/cache_center.c 2010-01-18 13:30:47 UTC (rev 4467) +++ branches/2.0/varnish-cache/bin/varnishd/cache_center.c 2010-01-18 14:58:09 UTC (rev 4468) @@ -594,7 +594,7 @@ CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC); - if (sp->obj == NULL) { + if (sp->objhead == NULL) { HSH_Prepare(sp, sp->vcl->nhashcount); VCL_hash_method(sp); assert(sp->handling == VCL_RET_HASH); From phk at projects.linpro.no Mon Jan 18 20:33:42 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 18 Jan 2010 21:33:42 +0100 (CET) Subject: r4469 - in trunk/varnish-cache: include lib/libvarnish Message-ID: <20100118203342.B5B381F72D4@projects.linpro.no> Author: phk Date: 2010-01-18 21:33:42 +0100 (Mon, 18 Jan 2010) New Revision: 4469 Added: trunk/varnish-cache/include/cli_serve.h trunk/varnish-cache/lib/libvarnish/cli_serve.c Modified: trunk/varnish-cache/include/Makefile.am trunk/varnish-cache/lib/libvarnish/Makefile.am Log: First attempt at unified, general "serve CLI requests" code. (not used yet) Modified: trunk/varnish-cache/include/Makefile.am =================================================================== --- trunk/varnish-cache/include/Makefile.am 2010-01-18 14:58:09 UTC (rev 4468) +++ trunk/varnish-cache/include/Makefile.am 2010-01-18 20:33:42 UTC (rev 4469) @@ -12,6 +12,7 @@ cli.h \ cli_common.h \ cli_priv.h \ + cli_serve.h \ compat/asprintf.h \ compat/daemon.h \ compat/execinfo.h \ Added: trunk/varnish-cache/include/cli_serve.h =================================================================== --- trunk/varnish-cache/include/cli_serve.h (rev 0) +++ trunk/varnish-cache/include/cli_serve.h 2010-01-18 20:33:42 UTC (rev 4469) @@ -0,0 +1,38 @@ +/*- + * Copyright (c) 2006-2009 Linpro AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $Id$ + * + */ + +struct cls; +typedef void cls_cb_f(void *priv); +struct cls *CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv); +int CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, + void *priv); +int CLS_AddFunc(struct cls *cs, struct cli_proto *clp); +int CLS_Poll(struct cls *cs, int timeout); Modified: trunk/varnish-cache/lib/libvarnish/Makefile.am =================================================================== --- trunk/varnish-cache/lib/libvarnish/Makefile.am 2010-01-18 14:58:09 UTC (rev 4468) +++ trunk/varnish-cache/lib/libvarnish/Makefile.am 2010-01-18 20:33:42 UTC (rev 4469) @@ -13,6 +13,7 @@ subproc.c \ cli.c \ cli_common.c \ + cli_serve.c \ flopen.c \ num.c \ svn_version.c \ Added: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c (rev 0) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 20:33:42 UTC (rev 4469) @@ -0,0 +1,228 @@ +/*- + * Copyright (c) 2006 Verdens Gang AS + * Copyright (c) 2006-2009 Linpro AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * Stuff for handling the CLI protocol + */ + +#include "config.h" + +#include "svnid.h" +SVNID("$Id: cli.c 4235 2009-09-11 13:06:15Z phk $") + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct cls_func { + unsigned magic; +#define CLS_FUNC_MAGIC 0x7d280c9b + VTAILQ_ENTRY(cls_func) list; + struct cli_proto *clp; +}; + +struct cls_fd { + unsigned magic; +#define CLS_FD_MAGIC 0x010dbd1e + VTAILQ_ENTRY(cls_fd) list; + int fdi, fdo; + struct vlu *vlu; + struct cls *cls; + struct cli *cli, clis; + cls_cb_f *closefunc; + void *priv; +}; + +struct cls { + unsigned magic; +#define CLS_MAGIC 0x60f044a3 + VTAILQ_HEAD(,cls_fd) fds; + unsigned nfd; + VTAILQ_HEAD(,cls_func) funcs; + cls_cb_f *before, *after; + void *priv; +}; + +static int +cls_vlu(void *priv, const char *p) +{ + struct cls_fd *cfd; + struct cls *cs; + struct cls_func *cfn; + + CAST_OBJ_NOTNULL(cfd, priv, CLS_FD_MAGIC); + cs = cfd->cls; + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + + if (cs->before != NULL) + cs->before(cs->priv); + VTAILQ_FOREACH(cfn, &cs->funcs, list) { + vsb_clear(cfd->cli->sb); + cfd->cli->result = CLIS_OK; + cli_dispatch(cfd->cli, cfn->clp, p); + if (cfd->cli->result != CLIS_UNKNOWN) + break; + } + if (cs->after != NULL) + cs->after(cs->priv); + vsb_finish(cfd->cli->sb); + AZ(vsb_overflowed(cfd->cli->sb)); + if (cli_writeres(cfd->fdo, cfd->cli) || cfd->cli->result == CLIS_CLOSE) + return (1); + return (0); +} + +struct cls * +CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv) +{ + struct cls *cs; + + ALLOC_OBJ(cs, CLS_MAGIC); + AN(cs); + VTAILQ_INIT(&cs->fds); + VTAILQ_INIT(&cs->funcs); + cs->before = before; + cs->after = after; + cs->priv = priv; + return (cs); +} + +/* XXX close call back */ +int +CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv) +{ + struct cls_fd *cfd; + + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + assert(fdi >= 0); + assert(fdo >= 0); + ALLOC_OBJ(cfd, CLS_FD_MAGIC); + AN(cfd); + cfd->cls = cs; + cfd->fdi = fdi; + cfd->fdo = fdo; + cfd->vlu = VLU_New(cfd, cls_vlu, 65536); /* XXX */ + cfd->cli = &cfd->clis; + cfd->cli->sb = vsb_newauto(); + cfd->closefunc = closefunc; + cfd->priv = priv; + AN(cfd->cli->sb); + VTAILQ_INSERT_TAIL(&cs->fds, cfd, list); + cs->nfd++; + return (0); +} + +static void +cls_close_fd(struct cls *cs, struct cls_fd *cfd) +{ + + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + CHECK_OBJ_NOTNULL(cfd, CLS_FD_MAGIC); + + VTAILQ_REMOVE(&cs->fds, cfd, list); + cs->nfd--; + VLU_Destroy(cfd->vlu); + vsb_delete(cfd->cli->sb); + if (cfd->closefunc == NULL) { + (void)close(cfd->fdi); + if (cfd->fdo != cfd->fdi) + (void)close(cfd->fdo); + } else { + cfd->closefunc(cfd->priv); + } + FREE_OBJ(cfd); +} + + +int +CLS_AddFunc(struct cls *cs, struct cli_proto *clp) +{ + struct cls_func *cfn; + + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + ALLOC_OBJ(cfn, CLS_FUNC_MAGIC); + AN(cfn); + cfn->clp = clp; + VTAILQ_INSERT_TAIL(&cs->funcs, cfn, list); + return (0); +} + +int +CLS_Poll(struct cls *cs, int timeout) +{ + struct cls_fd *cfd, *cfd2; + int i, j, k; + + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + if (cs->nfd == 0) { + errno = 0; + return (-1); + } + assert(cs->nfd > 0); + { + struct pollfd pfd[cs->nfd]; + + i = 0; + VTAILQ_FOREACH(cfd, &cs->fds, list) { + pfd[i].fd = cfd->fdi; + pfd[i].events = POLLIN; + pfd[i].revents = 0; + i++; + } + assert(i == cs->nfd); + + j = poll(pfd, cs->nfd, timeout); + if (j <= 0) + return (j); + i = 0; + VTAILQ_FOREACH_SAFE(cfd, &cs->fds, list, cfd2) { + assert(pfd[i].fd == cfd->fdi); + if (pfd[i].revents & POLLHUP) + k = 1; + else + k = VLU_Fd(cfd->fdi, cfd->vlu); + if (k) + cls_close_fd(cs, cfd); + i++; + } + assert(i == j); + } + return (j); +} From phk at projects.linpro.no Mon Jan 18 21:07:41 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 18 Jan 2010 22:07:41 +0100 (CET) Subject: r4470 - in trunk/varnish-cache: bin/varnishd include lib/libvarnish Message-ID: <20100118210741.4C6931F72D4@projects.linpro.no> Author: phk Date: 2010-01-18 22:07:41 +0100 (Mon, 18 Jan 2010) New Revision: 4470 Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c trunk/varnish-cache/include/cli_serve.h trunk/varnish-cache/lib/libvarnish/cli_serve.c Log: Have the child process use the new "CLI_serve()" functions to hand cli requests from the master. Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 20:33:42 UTC (rev 4469) +++ trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 21:07:41 UTC (rev 4470) @@ -50,13 +50,14 @@ #include "cli.h" #include "cli_priv.h" #include "cli_common.h" +#include "cli_serve.h" #include "cache.h" -#include "vlu.h" #include "vsb.h" #include "hash_slinger.h" pthread_t cli_thread; static struct lock cli_mtx; +static int add_check; /* * The CLI commandlist is split in three: @@ -77,6 +78,7 @@ { struct cli_proto *c, **cp; + AZ(add_check); switch (which) { case MASTER_CLI: cp = &ccf_master_cli; break; case PUBLIC_CLI: cp = &ccf_public_cli; break; @@ -91,86 +93,44 @@ Lck_Unlock(&cli_mtx); } -/*-------------------------------------------------------------------- - * Called when we have a full line, look through all three command - * lists to find it. - */ - -static int -cli_vlu(void *priv, const char *p) +static void +cli_cb_before(void *priv) { - struct cli *cli; - int i; - cli = priv; - VSL(SLT_CLI, 0, "Rd %s", p); + VSL(SLT_CLI, 0, "Rd %s", priv); VCL_Poll(); VBE_Poll(); - vsb_clear(cli->sb); Lck_Lock(&cli_mtx); - cli_dispatch(cli, ccf_master_cli, p); - if (cli->result == CLIS_UNKNOWN) { - vsb_clear(cli->sb); - cli->result = CLIS_OK; - cli_dispatch(cli, ccf_public_cli, p); - } - if (cli->result == CLIS_UNKNOWN) { - vsb_clear(cli->sb); - cli->result = CLIS_OK; - cli_dispatch(cli, ccf_debug_cli, p); - } - Lck_Unlock(&cli_mtx); - vsb_finish(cli->sb); - AZ(vsb_overflowed(cli->sb)); - i = cli_writeres(heritage.cli_out, cli); - if (i) - VSL(SLT_Error, 0, "CLI write failed (errno=%d)", errno); - else - VSL(SLT_CLI, 0, "Wr %d %d %s", - i, cli->result, vsb_data(cli->sb)); - return (0); } -/*-------------------------------------------------------------------- - * Run CLI on cli pipes from manager - */ +static void +cli_cb_after(void *priv) +{ + struct cli *cli; + Lck_Unlock(&cli_mtx); + cli = priv; + VSL(SLT_CLI, 0, "Wr %03u %s", cli->result, vsb_data(cli->sb)); +} + void CLI_Run(void) { - struct pollfd pfd[1]; - struct cli *cli, clis; - struct vlu *vlu; + struct cls *cls; int i; - cli = &clis; - memset(cli, 0, sizeof *cli); + add_check = 1; - cli->sb = vsb_newauto(); - XXXAN(cli->sb); - vlu = VLU_New(cli, cli_vlu, params->cli_buffer); - XXXAN(vlu); - printf("Ready\n"); - while (1) { - pfd[0].fd = heritage.cli_in; - pfd[0].events = POLLIN; - i = poll(pfd, 1, -1); - if (i == -1 && errno == EINTR) - continue; - assert(i == 1); - if (pfd[0].revents & POLLHUP) { - VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops"); - break; - } - i = VLU_Fd(heritage.cli_in, vlu); - if (i) { - fprintf(stderr, - "Error on CLI connection, exiting " - "(VLU_Fd %d ev: %x)\n", - i, pfd[0].revents); - break; - } - } + cls = CLS_New(cli_cb_before, cli_cb_after, NULL, params->cli_buffer); + AZ(CLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL)); + AZ(CLS_AddFunc(cls, ccf_master_cli)); + AZ(CLS_AddFunc(cls, ccf_public_cli)); + AZ(CLS_AddFunc(cls, ccf_debug_cli)); + + do { + i = CLS_Poll(cls, -1); + } while(i > 0); + VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops"); } /*--------------------------------------------------------------------*/ Modified: trunk/varnish-cache/include/cli_serve.h =================================================================== --- trunk/varnish-cache/include/cli_serve.h 2010-01-18 20:33:42 UTC (rev 4469) +++ trunk/varnish-cache/include/cli_serve.h 2010-01-18 21:07:41 UTC (rev 4470) @@ -31,7 +31,7 @@ struct cls; typedef void cls_cb_f(void *priv); -struct cls *CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv); +struct cls *CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv, unsigned maxlen); int CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv); int CLS_AddFunc(struct cls *cs, struct cli_proto *clp); Modified: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 20:33:42 UTC (rev 4469) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 21:07:41 UTC (rev 4470) @@ -78,6 +78,7 @@ VTAILQ_HEAD(,cls_func) funcs; cls_cb_f *before, *after; void *priv; + unsigned maxlen; }; static int @@ -92,7 +93,7 @@ CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); if (cs->before != NULL) - cs->before(cs->priv); + cs->before(cs->priv != NULL ? cs->priv : (void*)(uintptr_t)p); VTAILQ_FOREACH(cfn, &cs->funcs, list) { vsb_clear(cfd->cli->sb); cfd->cli->result = CLIS_OK; @@ -100,17 +101,17 @@ if (cfd->cli->result != CLIS_UNKNOWN) break; } - if (cs->after != NULL) - cs->after(cs->priv); vsb_finish(cfd->cli->sb); AZ(vsb_overflowed(cfd->cli->sb)); + if (cs->after != NULL) + cs->after(cs->priv != NULL ? cs->priv : cfd->cli); if (cli_writeres(cfd->fdo, cfd->cli) || cfd->cli->result == CLIS_CLOSE) return (1); return (0); } struct cls * -CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv) +CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv, unsigned maxlen) { struct cls *cs; @@ -121,10 +122,10 @@ cs->before = before; cs->after = after; cs->priv = priv; + cs->maxlen = maxlen; return (cs); } -/* XXX close call back */ int CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv) { @@ -138,7 +139,7 @@ cfd->cls = cs; cfd->fdi = fdi; cfd->fdo = fdo; - cfd->vlu = VLU_New(cfd, cls_vlu, 65536); /* XXX */ + cfd->vlu = VLU_New(cfd, cls_vlu, cs->maxlen); cfd->cli = &cfd->clis; cfd->cli->sb = vsb_newauto(); cfd->closefunc = closefunc; From phk at projects.linpro.no Mon Jan 18 21:28:20 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 18 Jan 2010 22:28:20 +0100 (CET) Subject: r4471 - in trunk/varnish-cache: bin/varnishd include lib/libvarnish Message-ID: <20100118212820.57A151F72D4@projects.linpro.no> Author: phk Date: 2010-01-18 22:28:19 +0100 (Mon, 18 Jan 2010) New Revision: 4471 Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c trunk/varnish-cache/include/cli_common.h trunk/varnish-cache/lib/libvarnish/cli_serve.c Log: Add the unadultered cli command to struct cli, and make the default "before" callback arg be the cli struct like the "after" callback. Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 21:07:41 UTC (rev 4470) +++ trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 21:28:19 UTC (rev 4471) @@ -96,8 +96,10 @@ static void cli_cb_before(void *priv) { + struct cli *cli; - VSL(SLT_CLI, 0, "Rd %s", priv); + cli = priv; + VSL(SLT_CLI, 0, "Rd %s", cli->cmd); VCL_Poll(); VBE_Poll(); Lck_Lock(&cli_mtx); Modified: trunk/varnish-cache/include/cli_common.h =================================================================== --- trunk/varnish-cache/include/cli_common.h 2010-01-18 21:07:41 UTC (rev 4470) +++ trunk/varnish-cache/include/cli_common.h 2010-01-18 21:28:19 UTC (rev 4471) @@ -34,6 +34,7 @@ struct vsb *sb; enum cli_status_e result; void *priv; + const char *cmd; }; int cli_writeres(int fd, const struct cli *cli); Modified: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 21:07:41 UTC (rev 4470) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 21:28:19 UTC (rev 4471) @@ -92,8 +92,9 @@ cs = cfd->cls; CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + cfd->cli->cmd = p; if (cs->before != NULL) - cs->before(cs->priv != NULL ? cs->priv : (void*)(uintptr_t)p); + cs->before(cs->priv != NULL ? cs->priv : cfd->cli); VTAILQ_FOREACH(cfn, &cs->funcs, list) { vsb_clear(cfd->cli->sb); cfd->cli->result = CLIS_OK; @@ -107,6 +108,7 @@ cs->after(cs->priv != NULL ? cs->priv : cfd->cli); if (cli_writeres(cfd->fdo, cfd->cli) || cfd->cli->result == CLIS_CLOSE) return (1); + cfd->cli->cmd = NULL; return (0); } From phk at projects.linpro.no Mon Jan 18 21:53:58 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 18 Jan 2010 22:53:58 +0100 (CET) Subject: r4472 - in trunk/varnish-cache: bin/varnishd lib/libvarnish Message-ID: <20100118215358.8C6331F72D4@projects.linpro.no> Author: phk Date: 2010-01-18 22:53:58 +0100 (Mon, 18 Jan 2010) New Revision: 4472 Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/lib/libvarnish/cli.c Log: Add a wild-card CLI hack, and use it for passing cli to the child process. Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-18 21:28:19 UTC (rev 4471) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-18 21:53:58 UTC (rev 4472) @@ -203,6 +203,47 @@ { NULL } }; +/*--------------------------------------------------------------------*/ + +static void +mcf_askchild(struct cli *cli, const char * const *av, void *priv) +{ + int i; + char *q; + unsigned u; + + (void)av; + (void)priv; + /* + * Command not recognized in master, try cacher if it is + * running. + */ + if (cli_o <= 0) { + cli_result(cli, CLIS_UNKNOWN); + cli_out(cli, + "Unknown request in manager process " + "(child not running).\n" + "Type 'help' for more info."); + return; + } + AN(cli->cmd); + i = write(cli_o, cli->cmd, strlen(cli->cmd)); + xxxassert(i == strlen(cli->cmd)); + i = write(cli_o, "\n", 1); + xxxassert(i == 1); + (void)cli_readres(cli_i, + &u, &q, params->cli_timeout); + cli_result(cli, u); + cli_out(cli, "%s", q); + free(q); +} + +static struct cli_proto cli_askchild[] = { + { "*", "", "\t\n", + 0, 9999, mcf_askchild, NULL}, + { NULL } +}; + /*-------------------------------------------------------------------- * Ask the child something over CLI, return zero only if everything is * happy happy. @@ -359,9 +400,6 @@ mgt_cli_vlu(void *priv, const char *p) { struct cli_port *cp; - char *q; - unsigned u; - int i; CAST_OBJ_NOTNULL(cp, priv, CLI_PORT_MAGIC); vsb_clear(cp->cli->sb); @@ -374,6 +412,7 @@ if (*p == '\0') return (0); + cp->cli->cmd = p; if (secret_file != NULL && cp->challenge[0] != '\0') { /* Authentication not yet passed */ cli_dispatch(cp->cli, cli_auth, p); @@ -381,37 +420,20 @@ mgt_cli_challenge(cp); } else { cli_dispatch(cp->cli, cli_proto, p); - if (cp->cli->result == CLIS_UNKNOWN) + if (cp->cli->result == CLIS_UNKNOWN) { + vsb_clear(cp->cli->sb); cli_dispatch(cp->cli, cli_debug, p); + } if (cp->cli->result == CLIS_UNKNOWN) { - /* - * Command not recognized in master, try cacher if it is - * running. - */ vsb_clear(cp->cli->sb); - cp->cli->result = CLIS_OK; - if (cli_o <= 0) { - cli_result(cp->cli, CLIS_UNKNOWN); - cli_out(cp->cli, - "Unknown request in manager process " - "(child not running).\n" - "Type 'help' for more info."); - } else { - i = write(cli_o, p, strlen(p)); - xxxassert(i == strlen(p)); - i = write(cli_o, "\n", 1); - xxxassert(i == 1); - (void)cli_readres(cli_i, - &u, &q, params->cli_timeout); - cli_result(cp->cli, u); - cli_out(cp->cli, "%s", q); - free(q); - } + cli_dispatch(cp->cli, cli_askchild, p); } } vsb_finish(cp->cli->sb); AZ(vsb_overflowed(cp->cli->sb)); + cp->cli->cmd = NULL; + /* send the result back */ syslog(LOG_INFO, "CLI %d result %d \"%s\"", cp->fdi, cp->cli->result, p); Modified: trunk/varnish-cache/lib/libvarnish/cli.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-18 21:28:19 UTC (rev 4471) +++ trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-18 21:53:58 UTC (rev 4472) @@ -97,9 +97,12 @@ cli_result(cli, CLIS_UNKNOWN); break; } - for (cp = clp; cp->request != NULL; cp++) + for (cp = clp; cp->request != NULL; cp++) { if (!strcmp(av[1], cp->request)) break; + if (!strcmp("*", cp->request)) + break; + } if (cp->request == NULL) { cli_out(cli, "Unknown request.\nType 'help' for more info.\n"); From phk at projects.linpro.no Mon Jan 18 23:53:48 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 19 Jan 2010 00:53:48 +0100 (CET) Subject: r4473 - in trunk/varnish-cache: bin/varnishd bin/varnishtest include lib/libvarnish Message-ID: <20100118235348.DC3191F73B6@projects.linpro.no> Author: phk Date: 2010-01-19 00:53:48 +0100 (Tue, 19 Jan 2010) New Revision: 4473 Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c trunk/varnish-cache/bin/varnishd/heritage.h trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/bin/varnishd/mgt_param.c trunk/varnish-cache/bin/varnishtest/vtc_varnish.c trunk/varnish-cache/include/cli_common.h trunk/varnish-cache/include/cli_serve.h trunk/varnish-cache/lib/libvarnish/cli_serve.c Log: CLI reworking in the manager process. Use the new cli_serve stuff, as ammended. Give cli's an "auth level" and only allow the commands with lower auth levels than what the cli has collected. Use this to implement the -S handling. In the future, we can also use it to do "R/O" vs. "R/W" command separation. Add a param (syslog_cli_traffic) to control if all CLI traffic is syslog'ed. Everything should work the same, as far as I know. Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 23:53:48 UTC (rev 4473) @@ -94,11 +94,9 @@ } static void -cli_cb_before(void *priv) +cli_cb_before(struct cli *cli) { - struct cli *cli; - cli = priv; VSL(SLT_CLI, 0, "Rd %s", cli->cmd); VCL_Poll(); VBE_Poll(); @@ -106,12 +104,9 @@ } static void -cli_cb_after(void *priv) +cli_cb_after(struct cli *cli) { - struct cli *cli; - Lck_Unlock(&cli_mtx); - cli = priv; VSL(SLT_CLI, 0, "Wr %03u %s", cli->result, vsb_data(cli->sb)); } @@ -123,11 +118,12 @@ add_check = 1; - cls = CLS_New(cli_cb_before, cli_cb_after, NULL, params->cli_buffer); - AZ(CLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL)); - AZ(CLS_AddFunc(cls, ccf_master_cli)); - AZ(CLS_AddFunc(cls, ccf_public_cli)); - AZ(CLS_AddFunc(cls, ccf_debug_cli)); + cls = CLS_New(cli_cb_before, cli_cb_after, params->cli_buffer); + AN(cls); + AN(CLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL)); + AZ(CLS_AddFunc(cls, 0, ccf_master_cli)); + AZ(CLS_AddFunc(cls, 0, ccf_public_cli)); + AZ(CLS_AddFunc(cls, 0, ccf_debug_cli)); do { i = CLS_Poll(cls, -1); Modified: trunk/varnish-cache/bin/varnishd/heritage.h =================================================================== --- trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-18 23:53:48 UTC (rev 4473) @@ -193,14 +193,13 @@ /* Get rid of duplicate purges */ unsigned purge_dups; - /* CLI banner */ - unsigned cli_banner; - /* How long time does the ban lurker sleep */ double ban_lurker_sleep; /* Max size of the saintmode list. 0 == no saint mode. */ unsigned saintmode_threshold; + + unsigned syslog_cli_traffic; }; extern volatile struct params *params; Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-18 23:53:48 UTC (rev 4473) @@ -38,7 +38,6 @@ #include #include -#include #include #include #include @@ -54,6 +53,7 @@ #include "cli.h" #include "vsb.h" #include "cli_common.h" +#include "cli_serve.h" #include "vev.h" #include "vsha256.h" #include "shmlog.h" @@ -66,32 +66,14 @@ #include "mgt_cli.h" static int cli_i = -1, cli_o = -1; +static struct cls *cls; static const char *secret_file; -struct telnet { - unsigned magic; -#define TELNET_MAGIC 0x53ec3ac0 - int fd; - struct vev *ev; -}; +#define MCF_NOAUTH 0 +#define MCF_AUTH 16 -struct cli_port { - unsigned magic; -#define CLI_PORT_MAGIC 0x5791079f - VTAILQ_ENTRY(cli_port) list; - struct vev *ev; - int fdi; - int fdo; - int verbose; - struct vlu *vlu; - struct cli cli[1]; - char *name; - char challenge[34]; - mgt_cli_close_f *closefunc; - void *priv; -}; +static void mcf_help(struct cli *cli, const char * const *av, void *priv); -static VTAILQ_HEAD(,cli_port) clilist = VTAILQ_HEAD_INITIALIZER(clilist); /*--------------------------------------------------------------------*/ @@ -110,26 +92,7 @@ #undef MAC_STAT } -/*--------------------------------------------------------------------*/ -static void -mcf_help(struct cli *cli, const char * const *av, void *priv) -{ - unsigned u; - char *p; - - cli_func_help(cli, av, priv); - if (cli_o >= 0 && (av[2] == NULL || *av[2] == '-')) { - p = NULL; - if (!mgt_cli_askchild(&u, &p, - "help %s\n", av[2] != NULL ? av[2] : "")) { - cli_out(cli, "%s", p); - cli_result(cli, u); - } - free(p); - } -} - /*--------------------------------------------------------------------*/ static void @@ -164,7 +127,6 @@ /* XXX: what order should this list be in ? */ static struct cli_proto cli_proto[] = { - { CLI_HELP, mcf_help, cli_proto }, { CLI_BANNER, mcf_banner, NULL }, { CLI_PING, cli_func_ping }, { CLI_SERVER_STATUS, mcf_server_status, NULL }, @@ -309,6 +271,24 @@ } /*-------------------------------------------------------------------- + * Generate a random challenge + */ + +static void +mgt_cli_challenge(struct cli *cli) +{ + int i; + + for (i = 0; i + 2L < sizeof cli->challenge; i++) + cli->challenge[i] = (random() % 26) + 'a'; + cli->challenge[i++] = '\n'; + cli->challenge[i] = '\0'; + cli_out(cli, "%s", cli->challenge); + cli_out(cli, "\nAuthentication required.\n"); + cli_result(cli, CLIS_AUTH); +} + +/*-------------------------------------------------------------------- * Validate the authentication */ @@ -319,10 +299,8 @@ int i, fd; struct SHA256Context sha256ctx; unsigned char digest[SHA256_LEN]; - struct cli_port *cp; AN(av[2]); - CAST_OBJ_NOTNULL(cp, cli->priv, CLI_PORT_MAGIC); (void)priv; AN(secret_file); fd = open(secret_file, O_RDONLY); @@ -353,20 +331,20 @@ buf[i] = '\0'; AZ(close(fd)); SHA256_Init(&sha256ctx); - SHA256_Update(&sha256ctx, cp->challenge, strlen(cp->challenge)); + SHA256_Update(&sha256ctx, cli->challenge, strlen(cli->challenge)); SHA256_Update(&sha256ctx, buf, i); - SHA256_Update(&sha256ctx, cp->challenge, strlen(cp->challenge)); + SHA256_Update(&sha256ctx, cli->challenge, strlen(cli->challenge)); SHA256_Final(digest, &sha256ctx); for (i = 0; i < SHA256_LEN; i++) sprintf(buf + i + i, "%02x", digest[i]); if (strcasecmp(buf, av[2])) { - cli_result(cli, CLIS_UNKNOWN); + mgt_cli_challenge(cli); return; } - cp->challenge[0] = '\0'; + cli->auth = MCF_AUTH; + memset(cli->challenge, 0, sizeof cli->challenge); cli_result(cli, CLIS_OK); - if (params->cli_banner) - mcf_banner(cli, av, priv); + mcf_banner(cli, av, priv); } static struct cli_proto cli_auth[] = { @@ -376,101 +354,61 @@ { NULL } }; -/*-------------------------------------------------------------------- - * Generate a random challenge - */ +/*--------------------------------------------------------------------*/ static void -mgt_cli_challenge(struct cli_port *cp) +mcf_help(struct cli *cli, const char * const *av, void *priv) { - int i; + unsigned u; + char *p; - for (i = 0; i + 2L < sizeof cp->challenge; i++) - cp->challenge[i] = (random() % 26) + 'a'; - cp->challenge[i++] = '\n'; - cp->challenge[i] = '\0'; - cli_out(cp->cli, "%s", cp->challenge); - cli_out(cp->cli, "\nAuthentication required.\n"); - cli_result(cp->cli, CLIS_AUTH); + (void)priv; + cli_func_help(cli, av, cli_auth); + if (cli->auth == MCF_NOAUTH) + return; + cli_func_help(cli, av, cli_proto); + if (cli_o >= 0 && (av[2] == NULL || *av[2] == '-')) { + p = NULL; + if (!mgt_cli_askchild(&u, &p, + "help %s\n", av[2] != NULL ? av[2] : "")) { + cli_out(cli, "%s", p); + cli_result(cli, u); + } + free(p); + } } -/*--------------------------------------------------------------------*/ -static int -mgt_cli_vlu(void *priv, const char *p) +/*--------------------------------------------------------------------*/ +static void +mgt_cli_cb_before(struct cli *cli) { - struct cli_port *cp; - CAST_OBJ_NOTNULL(cp, priv, CLI_PORT_MAGIC); - vsb_clear(cp->cli->sb); + if (params->syslog_cli_traffic) + syslog(LOG_NOTICE, "CLI %s Rd %s", cli->ident, cli->cmd); +} - /* Skip whitespace */ - for (; isspace(*p); p++) - continue; +static void +mgt_cli_cb_after(struct cli *cli) +{ - /* Ignore empty lines */ - if (*p == '\0') - return (0); - - cp->cli->cmd = p; - if (secret_file != NULL && cp->challenge[0] != '\0') { - /* Authentication not yet passed */ - cli_dispatch(cp->cli, cli_auth, p); - if (cp->cli->result == CLIS_UNKNOWN) - mgt_cli_challenge(cp); - } else { - cli_dispatch(cp->cli, cli_proto, p); - if (cp->cli->result == CLIS_UNKNOWN) { - vsb_clear(cp->cli->sb); - cli_dispatch(cp->cli, cli_debug, p); - } - if (cp->cli->result == CLIS_UNKNOWN) { - vsb_clear(cp->cli->sb); - cli_dispatch(cp->cli, cli_askchild, p); - } - } - vsb_finish(cp->cli->sb); - AZ(vsb_overflowed(cp->cli->sb)); - - cp->cli->cmd = NULL; - - /* send the result back */ - syslog(LOG_INFO, "CLI %d result %d \"%s\"", - cp->fdi, cp->cli->result, p); - if (cli_writeres(cp->fdo, cp->cli) || cp->cli->result == CLIS_CLOSE) - return (1); - return (0); + if (params->syslog_cli_traffic) + syslog(LOG_NOTICE, "CLI %s Wr %03u %s", + cli->ident, cli->result, vsb_data(cli->sb)); } -/*-------------------------------------------------------------------- - * Get rid of a CLI session. - * - * Always and only called from mgt_cli_callback(). - * - * We must get rid of everything but the event, which gets GC'ed by - * ev_schdule_one() when mgt_cli_callback, through our return value - * returns non-zero. - */ +/*--------------------------------------------------------------------*/ -static int -mgt_cli_close(struct cli_port *cp) +static void +mgt_cli_init_cls(void) { - CHECK_OBJ_NOTNULL(cp, CLI_PORT_MAGIC); - syslog(LOG_NOTICE, "CLI %d closed", cp->fdi); - /* - * Remove from list, so that if closefunc calls mgt_cli_close_all - * it will not try to remove this one too. - */ - VTAILQ_REMOVE(&clilist, cp, list); - - free(cp->name); - vsb_delete(cp->cli->sb); - VLU_Destroy(cp->vlu); - - cp->closefunc(cp->priv); - FREE_OBJ(cp); - return (1); + cls = CLS_New(mgt_cli_cb_before, mgt_cli_cb_after, params->cli_buffer); + AN(cls); + AZ(CLS_AddFunc(cls, MCF_NOAUTH, cli_auth)); + AZ(CLS_AddFunc(cls, MCF_AUTH, cli_proto)); + AZ(CLS_AddFunc(cls, MCF_AUTH, cli_debug)); + AZ(CLS_AddFunc(cls, MCF_AUTH, cli_askchild)); } /*-------------------------------------------------------------------- @@ -480,13 +418,8 @@ void mgt_cli_close_all(void) { - struct cli_port *cp; - while (!VTAILQ_EMPTY(&clilist)) { - cp = VTAILQ_FIRST(&clilist); - vev_del(mgt_evb, cp->ev); - (void)mgt_cli_close(cp); - } + CLS_Destroy(&cls); } /*-------------------------------------------------------------------- @@ -494,18 +427,14 @@ */ static int -mgt_cli_callback(const struct vev *e, int what) +mgt_cli_callback2(const struct vev *e, int what) { - struct cli_port *cp; + int i; - CAST_OBJ_NOTNULL(cp, e->priv, CLI_PORT_MAGIC); - - if (what & (EV_ERR | EV_HUP | EV_GONE)) - return (mgt_cli_close(cp)); - - if (VLU_Fd(cp->fdi, cp->vlu)) - return (mgt_cli_close(cp)); - return (0); + (void)e; + (void)what; + i = CLS_PollFd(cls, e->fd, 0); + return (i); } /*--------------------------------------------------------------------*/ @@ -513,50 +442,41 @@ void mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident, mgt_cli_close_f *closefunc, void *priv) { - struct cli_port *cp; + struct cli *cli; + struct vev *ev; - cp = calloc(sizeof *cp, 1); - XXXAN(cp); - cp->vlu = VLU_New(cp, mgt_cli_vlu, params->cli_buffer); + (void)ident; + (void)verbose; + if (cls == NULL) + mgt_cli_init_cls(); - cp->name = strdup(ident); - XXXAN(cp->name); - syslog(LOG_NOTICE, "CLI %d open %s", fdi, cp->name); - cp->magic = CLI_PORT_MAGIC; + cli = CLS_AddFd(cls, fdi, fdo, closefunc, priv); - cp->fdi = fdi; - cp->fdo = fdo; - cp->verbose = verbose; + cli->ident = strdup(ident); - cp->closefunc = closefunc; - cp->priv = priv; - - cp->cli->sb = vsb_newauto(); - XXXAN(cp->cli->sb); - cp->cli->priv = cp; - /* Deal with TELNET options */ - if (cp->fdi != 0) - VLU_SetTelnet(cp->vlu, cp->fdo); + if (fdi != 0) + VLU_SetTelnet(cli->vlu, fdo); - /* - * If we have a secret file authenticate all CLI connections - * except the stdin/stdout debug port. - */ - if (cp->fdi != 0 && secret_file != NULL) { - mgt_cli_challenge(cp); - (void)VLU_Data("auth -\n", -1, cp->vlu); - } else if (params->cli_banner) - (void)VLU_Data("banner\n", -1, cp->vlu); + if (fdi != 0 && secret_file != NULL) { + cli->auth = MCF_NOAUTH; + mgt_cli_challenge(cli); + } else { + cli->auth = MCF_AUTH; + mcf_banner(cli, NULL, NULL); + } + vsb_finish(cli->sb); + (void)cli_writeres(fdo, cli); - cp->ev = vev_new(); - cp->ev->name = cp->name; - cp->ev->fd = fdi; - cp->ev->fd_flags = EV_RD; - cp->ev->callback = mgt_cli_callback; - cp->ev->priv = cp; - VTAILQ_INSERT_TAIL(&clilist, cp, list); - AZ(vev_add(mgt_evb, cp->ev)); + + ev = vev_new(); + AN(ev); + ev->name = cli->ident; + ev->fd = fdi; + ev->fd_flags = EV_RD; + ev->callback = mgt_cli_callback2; + ev->priv = cli; + AZ(vev_add(mgt_evb, ev)); } /*--------------------------------------------------------------------*/ @@ -581,6 +501,13 @@ /*--------------------------------------------------------------------*/ +struct telnet { + unsigned magic; +#define TELNET_MAGIC 0x53ec3ac0 + int fd; + struct vev *ev; +}; + static void telnet_close(void *priv) { Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-18 23:53:48 UTC (rev 4473) @@ -756,9 +756,8 @@ "Detect and eliminate duplicate purges.\n", 0, "on", "bool" }, - { "cli_banner", tweak_bool, &master.cli_banner, 0, 0, - "Emit CLI banner on connect.\n" - "Set to off for compatibility with pre 2.1 versions.\n", + { "syslog_cli_traffic", tweak_bool, &master.syslog_cli_traffic, 0, 0, + "Log all CLI traffic to syslog(LOG_INFO).\n", 0, "on", "bool" }, { "ban_lurker_sleep", tweak_timeout_double, Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-18 23:53:48 UTC (rev 4473) @@ -247,6 +247,8 @@ struct vss_addr **ap; char abuf[128],pbuf[128]; struct pollfd fd; + unsigned retval; + char *r; /* Create listener socket */ nap = VSS_resolve("127.0.0.1", "0", &ap); @@ -261,8 +263,8 @@ AN(vsb); vsb_printf(vsb, "cd ../varnishd &&"); vsb_printf(vsb, " ./varnishd -d -d -n %s", v->workdir); - vsb_printf(vsb, " -p cli_banner=off"); vsb_printf(vsb, " -p auto_restart=off"); + vsb_printf(vsb, " -p syslog_cli_traffic=off"); vsb_printf(vsb, " -a '%s'", "127.0.0.1:0"); vsb_printf(vsb, " -M %s:%s", abuf, pbuf); vsb_printf(vsb, " -P %s/varnishd.pid", v->workdir); @@ -317,6 +319,13 @@ vtc_log(v->vl, 3, "CLI connection fd = %d", v->cli_fd); assert(v->cli_fd >= 0); + + i = cli_readres(v->cli_fd, &retval, &r, 20.0); + if (i != 0 || retval != CLIS_OK) + vtc_log(v->vl, 0, "CLI banner fail", v->cli_fd); + vtc_log(v->vl, 4, "CLI banner %03u", retval); + free(r); + if (v->stats != NULL) VSL_Close(); v->stats = VSL_OpenStats(v->workdir); Modified: trunk/varnish-cache/include/cli_common.h =================================================================== --- trunk/varnish-cache/include/cli_common.h 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/include/cli_common.h 2010-01-18 23:53:48 UTC (rev 4473) @@ -29,12 +29,17 @@ * $Id$ */ +struct vlu; + struct cli { /* XXX: should be MINI_OBJ */ struct vsb *sb; enum cli_status_e result; - void *priv; const char *cmd; + unsigned auth; + char challenge[34]; + char *ident; + struct vlu *vlu; }; int cli_writeres(int fd, const struct cli *cli); Modified: trunk/varnish-cache/include/cli_serve.h =================================================================== --- trunk/varnish-cache/include/cli_serve.h 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/include/cli_serve.h 2010-01-18 23:53:48 UTC (rev 4473) @@ -31,8 +31,12 @@ struct cls; typedef void cls_cb_f(void *priv); -struct cls *CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv, unsigned maxlen); -int CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, +typedef void cls_cbc_f(struct cli*); +struct cls *CLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen); +struct cli *CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv); -int CLS_AddFunc(struct cls *cs, struct cli_proto *clp); +int CLS_AddFunc(struct cls *cs, unsigned auth, struct cli_proto *clp); int CLS_Poll(struct cls *cs, int timeout); +int CLS_PollFd(struct cls *cs, int fd, int timeout); +void CLS_Destroy(struct cls **); + Modified: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 21:53:58 UTC (rev 4472) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 23:53:48 UTC (rev 4473) @@ -55,6 +55,7 @@ unsigned magic; #define CLS_FUNC_MAGIC 0x7d280c9b VTAILQ_ENTRY(cls_func) list; + unsigned auth; struct cli_proto *clp; }; @@ -63,7 +64,6 @@ #define CLS_FD_MAGIC 0x010dbd1e VTAILQ_ENTRY(cls_fd) list; int fdi, fdo; - struct vlu *vlu; struct cls *cls; struct cli *cli, clis; cls_cb_f *closefunc; @@ -76,8 +76,7 @@ VTAILQ_HEAD(,cls_fd) fds; unsigned nfd; VTAILQ_HEAD(,cls_func) funcs; - cls_cb_f *before, *after; - void *priv; + cls_cbc_f *before, *after; unsigned maxlen; }; @@ -92,10 +91,22 @@ cs = cfd->cls; CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + /* Skip whitespace */ + for (; isspace(*p); p++) + continue; + + /* Ignore empty lines */ + if (*p == '\0') + return (0); + cfd->cli->cmd = p; if (cs->before != NULL) - cs->before(cs->priv != NULL ? cs->priv : cfd->cli); + cs->before(cfd->cli); + vsb_clear(cfd->cli->sb); + cfd->cli->result = CLIS_UNKNOWN; VTAILQ_FOREACH(cfn, &cs->funcs, list) { + if (cfn->auth > cfd->cli->auth) + continue; vsb_clear(cfd->cli->sb); cfd->cli->result = CLIS_OK; cli_dispatch(cfd->cli, cfn->clp, p); @@ -105,7 +116,7 @@ vsb_finish(cfd->cli->sb); AZ(vsb_overflowed(cfd->cli->sb)); if (cs->after != NULL) - cs->after(cs->priv != NULL ? cs->priv : cfd->cli); + cs->after(cfd->cli); if (cli_writeres(cfd->fdo, cfd->cli) || cfd->cli->result == CLIS_CLOSE) return (1); cfd->cli->cmd = NULL; @@ -113,7 +124,7 @@ } struct cls * -CLS_New(cls_cb_f *before, cls_cb_f *after, void *priv, unsigned maxlen) +CLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen) { struct cls *cs; @@ -123,12 +134,11 @@ VTAILQ_INIT(&cs->funcs); cs->before = before; cs->after = after; - cs->priv = priv; cs->maxlen = maxlen; return (cs); } -int +struct cli * CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv) { struct cls_fd *cfd; @@ -141,15 +151,15 @@ cfd->cls = cs; cfd->fdi = fdi; cfd->fdo = fdo; - cfd->vlu = VLU_New(cfd, cls_vlu, cs->maxlen); cfd->cli = &cfd->clis; + cfd->cli->vlu = VLU_New(cfd, cls_vlu, cs->maxlen); cfd->cli->sb = vsb_newauto(); cfd->closefunc = closefunc; cfd->priv = priv; AN(cfd->cli->sb); VTAILQ_INSERT_TAIL(&cs->fds, cfd, list); cs->nfd++; - return (0); + return (cfd->cli); } static void @@ -161,7 +171,7 @@ VTAILQ_REMOVE(&cs->fds, cfd, list); cs->nfd--; - VLU_Destroy(cfd->vlu); + VLU_Destroy(cfd->cli->vlu); vsb_delete(cfd->cli->sb); if (cfd->closefunc == NULL) { (void)close(cfd->fdi); @@ -170,12 +180,14 @@ } else { cfd->closefunc(cfd->priv); } + if (cfd->cli->ident != NULL) + free(cfd->cli->ident); FREE_OBJ(cfd); } int -CLS_AddFunc(struct cls *cs, struct cli_proto *clp) +CLS_AddFunc(struct cls *cs, unsigned auth, struct cli_proto *clp) { struct cls_func *cfn; @@ -183,11 +195,51 @@ ALLOC_OBJ(cfn, CLS_FUNC_MAGIC); AN(cfn); cfn->clp = clp; + cfn->auth = auth; VTAILQ_INSERT_TAIL(&cs->funcs, cfn, list); return (0); } int +CLS_PollFd(struct cls *cs, int fd, int timeout) +{ + struct cls_fd *cfd; + struct pollfd pfd[1]; + int i, j, k; + + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + if (cs->nfd == 0) { + errno = 0; + return (-1); + } + assert(cs->nfd > 0); + + i = 0; + VTAILQ_FOREACH(cfd, &cs->fds, list) { + if (cfd->fdi != fd) + continue; + pfd[i].fd = cfd->fdi; + pfd[i].events = POLLIN; + pfd[i].revents = 0; + i++; + break; + } + assert(i == 1); + CHECK_OBJ_NOTNULL(cfd, CLS_FD_MAGIC); + + j = poll(pfd, 1, timeout); + if (j <= 0) + return (j); + if (pfd[0].revents & POLLHUP) + k = 1; + else + k = VLU_Fd(cfd->fdi, cfd->cli->vlu); + if (k) + cls_close_fd(cs, cfd); + return (k); +} + +int CLS_Poll(struct cls *cs, int timeout) { struct cls_fd *cfd, *cfd2; @@ -220,7 +272,7 @@ if (pfd[i].revents & POLLHUP) k = 1; else - k = VLU_Fd(cfd->fdi, cfd->vlu); + k = VLU_Fd(cfd->fdi, cfd->cli->vlu); if (k) cls_close_fd(cs, cfd); i++; @@ -229,3 +281,25 @@ } return (j); } + +void +CLS_Destroy(struct cls **csp) +{ + struct cls *cs; + struct cls_fd *cfd, *cfd2; + struct cls_func *cfn; + + cs = *csp; + *csp = NULL; + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + VTAILQ_FOREACH_SAFE(cfd, &cs->fds, list, cfd2) + cls_close_fd(cs, cfd); + + while (!VTAILQ_EMPTY(&cs->funcs)) { + cfn = VTAILQ_FIRST(&cs->funcs); + VTAILQ_REMOVE(&cs->funcs, cfn, list); + FREE_OBJ(cfn); + } + FREE_OBJ(cs); +} + From phk at projects.linpro.no Wed Jan 20 11:22:52 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 20 Jan 2010 12:22:52 +0100 (CET) Subject: r4474 - in trunk/varnish-cache: bin/varnishd include lib/libvarnish Message-ID: <20100120112252.637141F7C2D@projects.linpro.no> Author: phk Date: 2010-01-20 12:22:51 +0100 (Wed, 20 Jan 2010) New Revision: 4474 Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/include/cli_common.h trunk/varnish-cache/include/cli_priv.h trunk/varnish-cache/include/cli_serve.h trunk/varnish-cache/lib/libvarnish/cli.c trunk/varnish-cache/lib/libvarnish/cli_serve.c Log: Move cli_dispatch into cli_serve.c and unroll stuff that only needs to be done once. Finally make the cli structure a miniobj. Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-20 11:22:51 UTC (rev 4474) @@ -94,7 +94,7 @@ } static void -cli_cb_before(struct cli *cli) +cli_cb_before(const struct cli *cli) { VSL(SLT_CLI, 0, "Rd %s", cli->cmd); @@ -104,7 +104,7 @@ } static void -cli_cb_after(struct cli *cli) +cli_cb_after(const struct cli *cli) { Lck_Unlock(&cli_mtx); VSL(SLT_CLI, 0, "Wr %03u %s", cli->result, vsb_data(cli->sb)); Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-20 11:22:51 UTC (rev 4474) @@ -381,7 +381,7 @@ /*--------------------------------------------------------------------*/ static void -mgt_cli_cb_before(struct cli *cli) +mgt_cli_cb_before(const struct cli *cli) { if (params->syslog_cli_traffic) @@ -389,7 +389,7 @@ } static void -mgt_cli_cb_after(struct cli *cli) +mgt_cli_cb_after(const struct cli *cli) { if (params->syslog_cli_traffic) Modified: trunk/varnish-cache/include/cli_common.h =================================================================== --- trunk/varnish-cache/include/cli_common.h 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/include/cli_common.h 2010-01-20 11:22:51 UTC (rev 4474) @@ -32,7 +32,8 @@ struct vlu; struct cli { - /* XXX: should be MINI_OBJ */ + unsigned magic; +#define CLI_MAGIC 0x4038d570 struct vsb *sb; enum cli_status_e result; const char *cmd; Modified: trunk/varnish-cache/include/cli_priv.h =================================================================== --- trunk/varnish-cache/include/cli_priv.h 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/include/cli_priv.h 2010-01-20 11:22:51 UTC (rev 4474) @@ -58,7 +58,6 @@ void cli_result(struct cli *cli, unsigned r); /* From libvarnish/cli.c */ -void cli_dispatch(struct cli *cli, struct cli_proto *clp, const char *line); cli_func_t cli_func_help; cli_func_t cli_func_ping; struct cli_proto *cli_concat(struct cli_proto *, struct cli_proto *); Modified: trunk/varnish-cache/include/cli_serve.h =================================================================== --- trunk/varnish-cache/include/cli_serve.h 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/include/cli_serve.h 2010-01-20 11:22:51 UTC (rev 4474) @@ -31,7 +31,7 @@ struct cls; typedef void cls_cb_f(void *priv); -typedef void cls_cbc_f(struct cli*); +typedef void cls_cbc_f(const struct cli*); struct cls *CLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen); struct cli *CLS_AddFd(struct cls *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv); Modified: trunk/varnish-cache/lib/libvarnish/cli.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-20 11:22:51 UTC (rev 4474) @@ -71,75 +71,6 @@ cli_out(cli, "Unknown request.\nType 'help' for more info.\n"); cli_result(cli, CLIS_UNKNOWN); } - -void -cli_dispatch(struct cli *cli, struct cli_proto *clp, const char *line) -{ - char **av; - unsigned u; - struct cli_proto *cp; - - cli_result(cli, CLIS_OK); - /* XXX: syslog commands */ - av = ParseArgv(line, 0); - AN(av); - do { - if (av[0] != NULL) { - cli_out(cli, "Syntax Error: %s\n", av[0]); - cli_result(cli, CLIS_SYNTAX); - break; - } - if (av[1] == NULL) - break; - if (isupper(av[1][0])) { - cli_out(cli, - "all commands are in lower-case.\n"); - cli_result(cli, CLIS_UNKNOWN); - break; - } - for (cp = clp; cp->request != NULL; cp++) { - if (!strcmp(av[1], cp->request)) - break; - if (!strcmp("*", cp->request)) - break; - } - if (cp->request == NULL) { - cli_out(cli, - "Unknown request.\nType 'help' for more info.\n"); - cli_result(cli, CLIS_UNKNOWN); - break; - } - - if (cp->func == NULL) { - cli_out(cli, "Unimplemented\n"); - cli_result(cli, CLIS_UNIMPL); - break; - } - - for (u = 0; u <= cp->minarg; u++) { - if (av[u + 1] != NULL) - continue; - cli_out(cli, "Too few parameters\n"); - cli_result(cli, CLIS_TOOFEW); - break; - } - if (u <= cp->minarg) - break; - for (; u <= cp->maxarg; u++) - if (av[u + 1] == NULL) - break; - if (av[u + 1] != NULL) { - cli_out(cli, "Too many parameters\n"); - cli_result(cli, CLIS_TOOMANY); - break; - } - - cp->func(cli, (const char * const *)av, cp->priv); - - } while (0); - FreeArgv(av); -} - struct cli_proto * cli_concat(struct cli_proto *c1, struct cli_proto *c2) { Modified: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-18 23:53:48 UTC (rev 4473) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-20 11:22:51 UTC (rev 4474) @@ -80,46 +80,132 @@ unsigned maxlen; }; +/*-------------------------------------------------------------------- + * Look for a CLI command to execute + */ + static int +cls_dispatch(struct cli *cli, struct cli_proto *clp, char * const * av, + unsigned ac) +{ + struct cli_proto *cp; + + AN(av); + for (cp = clp; cp->request != NULL; cp++) { + if (!strcmp(av[1], cp->request)) + break; + if (!strcmp("*", cp->request)) + break; + } + if (cp->request == NULL) + return (0); + + if (cp->func == NULL) { + cli_out(cli, "Unimplemented\n"); + cli_result(cli, CLIS_UNIMPL); + return(1); + } + + if (ac - 1 < cp->minarg) { + cli_out(cli, "Too few parameters\n"); + cli_result(cli, CLIS_TOOFEW); + return(1); + } + + if (ac - 1> cp->maxarg) { + cli_out(cli, "Too many parameters\n"); + cli_result(cli, CLIS_TOOMANY); + return(1); + } + + cli->result = CLIS_OK; + vsb_clear(cli->sb); + cp->func(cli, (const char * const *)av, cp->priv); + return (1); +} + +/*-------------------------------------------------------------------- + * We have collected a full cli line, parse it and execute, if possible. + */ + +static int cls_vlu(void *priv, const char *p) { struct cls_fd *cfd; struct cls *cs; struct cls_func *cfn; + struct cli *cli; + char * * av; + unsigned na; CAST_OBJ_NOTNULL(cfd, priv, CLS_FD_MAGIC); cs = cfd->cls; CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); - /* Skip whitespace */ + cli = cfd->cli; + CHECK_OBJ_NOTNULL(cli, CLI_MAGIC); + AZ(cli->cmd); + + /* + * Lines with only whitespace are simply ignored, in order to not + * complicate CLI-client side scripts and TELNET users + */ for (; isspace(*p); p++) continue; - - /* Ignore empty lines */ if (*p == '\0') return (0); - cfd->cli->cmd = p; + cli->cmd = p; + + av = ParseArgv(p, 0); + AN(av); + + cli->result = CLIS_UNKNOWN; + vsb_clear(cli->sb); + cli_out(cli, "Unknown request.\nType 'help' for more info.\n"); + if (cs->before != NULL) - cs->before(cfd->cli); - vsb_clear(cfd->cli->sb); - cfd->cli->result = CLIS_UNKNOWN; - VTAILQ_FOREACH(cfn, &cs->funcs, list) { - if (cfn->auth > cfd->cli->auth) + cs->before(cli); + + do { + if (av[0] != NULL) { + cli_out(cli, "Syntax Error: %s\n", av[0]); + cli_result(cli, CLIS_SYNTAX); + break; + } + + if (isupper(av[1][0])) { + cli_out(cli, "all commands are in lower-case.\n"); + cli_result(cli, CLIS_UNKNOWN); + break; + } + + if (!islower(av[1][0])) + break; + + for (na = 0; av[na + 1] != NULL; na++) continue; - vsb_clear(cfd->cli->sb); - cfd->cli->result = CLIS_OK; - cli_dispatch(cfd->cli, cfn->clp, p); - if (cfd->cli->result != CLIS_UNKNOWN) - break; - } - vsb_finish(cfd->cli->sb); - AZ(vsb_overflowed(cfd->cli->sb)); + + VTAILQ_FOREACH(cfn, &cs->funcs, list) { + if (cfn->auth > cli->auth) + continue; + if (cls_dispatch(cli, cfn->clp, av, na)) + break; + } + } while (0); + + vsb_finish(cli->sb); + AZ(vsb_overflowed(cli->sb)); + if (cs->after != NULL) - cs->after(cfd->cli); - if (cli_writeres(cfd->fdo, cfd->cli) || cfd->cli->result == CLIS_CLOSE) + cs->after(cli); + + cli->cmd = NULL; + FreeArgv(av); + + if (cli_writeres(cfd->fdo, cli) || cli->result == CLIS_CLOSE) return (1); - cfd->cli->cmd = NULL; + return (0); } @@ -152,6 +238,7 @@ cfd->fdi = fdi; cfd->fdo = fdo; cfd->cli = &cfd->clis; + cfd->cli->magic = CLI_MAGIC; cfd->cli->vlu = VLU_New(cfd, cls_vlu, cs->maxlen); cfd->cli->sb = vsb_newauto(); cfd->closefunc = closefunc; From phk at projects.linpro.no Wed Jan 20 12:26:30 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 20 Jan 2010 13:26:30 +0100 (CET) Subject: r4475 - in trunk/varnish-cache: bin/varnishd include lib/libvcl Message-ID: <20100120122630.341301F7B53@projects.linpro.no> Author: phk Date: 2010-01-20 13:26:29 +0100 (Wed, 20 Jan 2010) New Revision: 4475 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_dir_random.c trunk/varnish-cache/bin/varnishd/cache_http.c trunk/varnish-cache/bin/varnishd/flint.lnt trunk/varnish-cache/bin/varnishd/mgt_vcc.c trunk/varnish-cache/bin/varnishd/stevedore.h trunk/varnish-cache/bin/varnishd/stevedore_utils.c trunk/varnish-cache/bin/varnishd/varnishd.c trunk/varnish-cache/include/vbm.h trunk/varnish-cache/lib/libvcl/vcc_compile.c trunk/varnish-cache/lib/libvcl/vcc_compile.h Log: Various flexelint silencings. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-20 12:26:29 UTC (rev 4475) @@ -514,22 +514,22 @@ void HTTP_Init(void); void http_ClrHeader(struct http *to); unsigned http_Write(struct worker *w, const struct http *hp, int resp); -void http_CopyResp(struct http *to, const struct http *fm); -void http_SetResp(struct http *to, const char *proto, const char *status, +void http_CopyResp(const struct http *to, const struct http *fm); +void http_SetResp(const struct http *to, const char *proto, const char *status, const char *response); void http_FilterFields(struct worker *w, int fd, struct http *to, const struct http *fm, unsigned how); void http_FilterHeader(const struct sess *sp, unsigned how); -void http_PutProtocol(struct worker *w, int fd, struct http *to, +void http_PutProtocol(struct worker *w, int fd, const struct http *to, const char *protocol); void http_PutStatus(struct worker *w, int fd, struct http *to, int status); -void http_PutResponse(struct worker *w, int fd, struct http *to, +void http_PutResponse(struct worker *w, int fd, const struct http *to, const char *response); void http_PrintfHeader(struct worker *w, int fd, struct http *to, const char *fmt, ...); void http_SetHeader(struct worker *w, int fd, struct http *to, const char *hdr); -void http_SetH(struct http *to, unsigned n, const char *fm); -void http_ForceGet(struct http *to); +void http_SetH(const struct http *to, unsigned n, const char *fm); +void http_ForceGet(const struct http *to); void http_Setup(struct http *ht, struct ws *ws); int http_GetHdr(const struct http *hp, const char *hdr, char **ptr); int http_GetHdrField(const struct http *hp, const char *hdr, @@ -540,8 +540,8 @@ int http_DissectRequest(struct sess *sp); int http_DissectResponse(struct worker *w, const struct http_conn *htc, struct http *sp); -const char *http_DoConnection(struct http *hp); -void http_CopyHome(struct worker *w, int fd, struct http *hp); +const char *http_DoConnection(const struct http *hp); +void http_CopyHome(struct worker *w, int fd, const struct http *hp); void http_Unset(struct http *hp, const char *hdr); /* cache_httpconn.c */ Modified: trunk/varnish-cache/bin/varnishd/cache_dir_random.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/cache_dir_random.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -86,7 +86,7 @@ int i, k; struct vdi_random *vs; double r, s1; - unsigned u; + unsigned u = 0; struct vbe_conn *vbe; struct director *d2; struct SHA256Context ctx; Modified: trunk/varnish-cache/bin/varnishd/cache_http.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/cache_http.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -284,7 +284,7 @@ */ const char * -http_DoConnection(struct http *hp) +http_DoConnection(const struct http *hp) { char *p, *q; const char *ret; @@ -588,7 +588,7 @@ /*--------------------------------------------------------------------*/ void -http_SetH(struct http *to, unsigned n, const char *fm) +http_SetH(const struct http *to, unsigned n, const char *fm) { assert(n < to->shd); @@ -599,7 +599,7 @@ } static void -http_copyh(struct http *to, const struct http *fm, unsigned n) +http_copyh(const struct http *to, const struct http *fm, unsigned n) { assert(n < to->shd); @@ -609,14 +609,14 @@ } void -http_ForceGet(struct http *to) +http_ForceGet(const struct http *to) { if (strcmp(http_GetReq(to), "GET")) http_SetH(to, HTTP_HDR_REQ, "GET"); } void -http_CopyResp(struct http *to, const struct http *fm) +http_CopyResp(const struct http *to, const struct http *fm) { CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC); @@ -630,7 +630,7 @@ } void -http_SetResp(struct http *to, const char *proto, const char *status, +http_SetResp(const struct http *to, const char *proto, const char *status, const char *response) { @@ -742,7 +742,7 @@ */ void -http_CopyHome(struct worker *w, int fd, struct http *hp) +http_CopyHome(struct worker *w, int fd, const struct http *hp) { unsigned u, l; char *p; @@ -801,7 +801,7 @@ /*--------------------------------------------------------------------*/ static void -http_PutField(struct worker *w, int fd, struct http *to, int field, +http_PutField(struct worker *w, int fd, const struct http *to, int field, const char *string) { char *p; @@ -824,7 +824,7 @@ } void -http_PutProtocol(struct worker *w, int fd, struct http *to, +http_PutProtocol(struct worker *w, int fd, const struct http *to, const char *protocol) { @@ -843,7 +843,7 @@ } void -http_PutResponse(struct worker *w, int fd, struct http *to, +http_PutResponse(struct worker *w, int fd, const struct http *to, const char *response) { Modified: trunk/varnish-cache/bin/varnishd/flint.lnt =================================================================== --- trunk/varnish-cache/bin/varnishd/flint.lnt 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/flint.lnt 2010-01-20 12:26:29 UTC (rev 4475) @@ -39,6 +39,9 @@ -esym(755, VSTAILQ_*) -esym(755, VTAILQ_*) +// XXX: I think this is a flexelint bug: +-esym(522, vbit_clr) + // Stuff used outside varnishd -esym(759, BackSlash) -esym(765, BackSlash) Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -44,7 +44,6 @@ #include #include -#include "compat/asprintf.h" #include "vsb.h" #include "libvcl.h" @@ -356,13 +355,14 @@ return (1); } - asprintf(&vcl, + bprintf(buf, "backend default {\n" " .host = \"%s\";\n" " .port = \"%s\";\n" "}\n", addr, port ? port : "http"); free(addr); free(port); + vcl = strdup(buf); AN(vcl); bprintf(buf, "boot (-b %s)", b_arg); } else { Modified: trunk/varnish-cache/bin/varnishd/stevedore.h =================================================================== --- trunk/varnish-cache/bin/varnishd/stevedore.h 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/stevedore.h 2010-01-20 12:26:29 UTC (rev 4475) @@ -39,8 +39,6 @@ typedef void storage_open_f(const struct stevedore *); typedef struct storage *storage_alloc_f(struct stevedore *, size_t size, struct objcore *); -typedef struct object *storage_alloc_obj_f(struct stevedore *, size_t size, - double ttl); typedef void storage_trim_f(struct storage *, size_t size); typedef void storage_free_f(struct storage *); typedef void storage_object_f(const struct sess *sp); @@ -54,7 +52,6 @@ storage_init_f *init; /* called by mgt process */ storage_open_f *open; /* called by cache process */ storage_alloc_f *alloc; - storage_alloc_obj_f *allocobj; storage_trim_f *trim; storage_free_f *free; storage_object_f *object; Modified: trunk/varnish-cache/bin/varnishd/stevedore_utils.c =================================================================== --- trunk/varnish-cache/bin/varnishd/stevedore_utils.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/stevedore_utils.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -55,7 +55,6 @@ #include #endif -#include "compat/asprintf.h" #include "mgt.h" #include "stevedore.h" @@ -86,6 +85,7 @@ struct stat st; char *q; int retval = 1; + char buf[FILENAME_MAX]; AN(fn); AN(fnp); @@ -107,13 +107,13 @@ ctx, fn); if (S_ISDIR(st.st_mode)) { - xxxassert(asprintf(&q, "%s/varnish.XXXXXX", fn) > 0); - XXXAN(q); - fd = mkstemp(q); + bprintf(buf, "%s/varnish.XXXXXX", fn); + fd = mkstemp(buf); if (fd < 0) ARGV_ERR("(%s) \"%s\" mkstemp(%s) failed (%s)\n", ctx, fn, q, strerror(errno)); - *fnp = q; + *fnp = strdup(buf); + AN(*fnp); retval = 2; } else if (S_ISREG(st.st_mode)) { fd = open(fn, O_RDWR | O_LARGEFILE); Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -49,7 +49,6 @@ #include #include -#include "compat/asprintf.h" #include "compat/daemon.h" #ifndef HAVE_STRLCPY @@ -438,14 +437,9 @@ FILE *fi; uintptr_t a; struct symbols *s; - int i; - p = NULL; - i = asprintf(&p, "nm -an %s 2>/dev/null", a0); - if (i < 0 || p == NULL) - return; - fi = popen(p, "r"); - free(p); + bprintf(buf, "nm -an %s 2>/dev/null", a0); + fi = popen(buf, "r"); if (fi == NULL) return; while (fgets(buf, sizeof buf, fi)) { Modified: trunk/varnish-cache/include/vbm.h =================================================================== --- trunk/varnish-cache/include/vbm.h 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/include/vbm.h 2010-01-20 12:26:29 UTC (rev 4475) @@ -85,7 +85,7 @@ } static inline void -vbit_clr(struct vbitmap *vb, unsigned bit) +vbit_clr(const struct vbitmap *vb, unsigned bit) { if (bit < vb->nbits) @@ -93,7 +93,7 @@ } static inline int -vbit_test(struct vbitmap *vb, unsigned bit) +vbit_test(const struct vbitmap *vb, unsigned bit) { if (bit >= vb->nbits) Modified: trunk/varnish-cache/lib/libvcl/vcc_compile.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_compile.c 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/lib/libvcl/vcc_compile.c 2010-01-20 12:26:29 UTC (rev 4475) @@ -96,8 +96,8 @@ /*--------------------------------------------------------------------*/ -void -TlFree(struct tokenlist *tl, void *p) +static void +TlDoFree(struct tokenlist *tl, void *p) { struct membit *mb; @@ -115,7 +115,7 @@ p = calloc(len, 1); assert(p != NULL); - TlFree(tl, p); + TlDoFree(tl, p); return (p); } Modified: trunk/varnish-cache/lib/libvcl/vcc_compile.h =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_compile.h 2010-01-20 11:22:51 UTC (rev 4474) +++ trunk/varnish-cache/lib/libvcl/vcc_compile.h 2010-01-20 12:26:29 UTC (rev 4475) @@ -176,7 +176,6 @@ void Ff(const struct tokenlist *tl, int indent, const char *fmt, ...); void EncToken(struct vsb *sb, const struct token *t); int IsMethod(const struct token *t); -void TlFree(struct tokenlist *tl, void *p); void *TlAlloc(struct tokenlist *tl, unsigned len); /* vcc_dir_random.c */ From kristian at projects.linpro.no Wed Jan 20 18:12:19 2010 From: kristian at projects.linpro.no (kristian at projects.linpro.no) Date: Wed, 20 Jan 2010 19:12:19 +0100 (CET) Subject: r4476 - trunk/varnish-cache/bin/varnishd Message-ID: <20100120181219.E2B1B1F7C29@projects.linpro.no> Author: kristian Date: 2010-01-20 19:12:19 +0100 (Wed, 20 Jan 2010) New Revision: 4476 Modified: trunk/varnish-cache/bin/varnishd/cache_session.c Log: Zero the epoll data.ptr on SES_Delete As of r4456, the sm-structure isn't zeroed. This confused the epoll-waiter which relied on ev.data.ptr to be NULL. Modified: trunk/varnish-cache/bin/varnishd/cache_session.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-20 12:26:29 UTC (rev 4475) +++ trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-20 18:12:19 UTC (rev 4476) @@ -234,6 +234,15 @@ // sm->magic = SESSMEM_MAGIC; // sm->workspace = workspace; + /* XXX: cache_waiter_epoll.c evaluates data.ptr. If it's + * XXX: not nulled, things go wrong during load. + * XXX: Should probably find a better method to deal with + * XXX: this scenario... + */ +#if defined(HAVE_EPOLL_CTL) + sm->sess.ev.data.ptr = NULL; +#endif + Lck_Lock(&ses_mem_mtx); VTAILQ_INSERT_HEAD(&ses_free_mem[1 - ses_qp], sm, list); Lck_Unlock(&ses_mem_mtx); From phk at projects.linpro.no Thu Jan 21 10:26:39 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 21 Jan 2010 11:26:39 +0100 (CET) Subject: r4477 - in trunk/varnish-cache: bin/varnishd include lib/libvarnish Message-ID: <20100121102639.AA7C01F70A2@projects.linpro.no> Author: phk Date: 2010-01-21 11:26:39 +0100 (Thu, 21 Jan 2010) New Revision: 4477 Removed: trunk/varnish-cache/lib/libvarnish/cli.c Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c trunk/varnish-cache/bin/varnishd/cache_backend_poll.c trunk/varnish-cache/bin/varnishd/cache_ban.c trunk/varnish-cache/bin/varnishd/cache_center.c trunk/varnish-cache/bin/varnishd/cache_cli.c trunk/varnish-cache/bin/varnishd/cache_fetch.c trunk/varnish-cache/bin/varnishd/cache_vcl.c trunk/varnish-cache/bin/varnishd/hash_critbit.c trunk/varnish-cache/bin/varnishd/mgt_cli.c trunk/varnish-cache/include/cli_common.h trunk/varnish-cache/include/cli_priv.h trunk/varnish-cache/include/cli_serve.h trunk/varnish-cache/lib/libvarnish/Makefile.am trunk/varnish-cache/lib/libvarnish/cli_common.c trunk/varnish-cache/lib/libvarnish/cli_serve.c Log: Take another trip over the CLI code, this time the help function. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-21 10:26:39 UTC (rev 4477) @@ -487,8 +487,7 @@ /* cache_cli.c [CLI] */ void CLI_Init(void); void CLI_Run(void); -enum cli_set_e {MASTER_CLI, PUBLIC_CLI, DEBUG_CLI}; -void CLI_AddFuncs(enum cli_set_e which, struct cli_proto *p); +void CLI_AddFuncs(struct cli_proto *p); extern pthread_t cli_thread; #define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0) Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -375,15 +375,11 @@ /*--------------------------------------------------------------------*/ static struct cli_proto vca_cmds[] = { - { CLI_SERVER_START, ccf_start }, - { NULL } -}; - -static struct cli_proto vca_debug_cmds[] = { + { CLI_SERVER_START, "i", ccf_start }, { "debug.listen_address", "debug.listen_address", "Report the actual listen address\n", 0, 0, - ccf_listen_address, NULL }, + "d", ccf_listen_address, NULL }, { NULL } }; @@ -391,8 +387,7 @@ VCA_Init(void) { - CLI_AddFuncs(MASTER_CLI, vca_cmds); - CLI_AddFuncs(DEBUG_CLI, vca_debug_cmds); + CLI_AddFuncs(vca_cmds); } void Modified: trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_backend_cfg.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -311,7 +311,7 @@ static struct cli_proto debug_cmds[] = { { "debug.backend", "debug.backend", - "\tExamine Backend internals\n", 0, 0, cli_debug_backend }, + "\tExamine Backend internals\n", 0, 0, "d", cli_debug_backend }, { NULL } }; @@ -322,5 +322,5 @@ { Lck_New(&VBE_mtx); - CLI_AddFuncs(DEBUG_CLI, debug_cmds); + CLI_AddFuncs(debug_cmds); } Modified: trunk/varnish-cache/bin/varnishd/cache_backend_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend_poll.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_backend_poll.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -409,7 +409,7 @@ static struct cli_proto debug_cmds[] = { { "debug.health", "debug.health", "\tDump backend health stuff\n", - 0, 0, vbp_health }, + 0, 0, "d", vbp_health }, { NULL } }; @@ -494,5 +494,5 @@ VBP_Init(void) { - CLI_AddFuncs(DEBUG_CLI, debug_cmds); + CLI_AddFuncs(debug_cmds); } Modified: trunk/varnish-cache/bin/varnishd/cache_ban.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_ban.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_ban.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -794,13 +794,13 @@ * XXX: COMPAT: Retain these two entries for entire 2.x series * XXX: COMPAT: to stay compatible with 1.x series syntax. */ - { CLI_HIDDEN("url.purge", 1, 1) ccf_purge_url }, - { CLI_HIDDEN("hash.purge", 0, 1) ccf_purge_hash }, - { CLI_HIDDEN("purge.hash", 0, 1) ccf_purge_hash }, + { CLI_HIDDEN("url.purge", 1, 1) "h", ccf_purge_url }, + { CLI_HIDDEN("hash.purge", 0, 1) "h", ccf_purge_hash }, + { CLI_HIDDEN("purge.hash", 0, 1) "h", ccf_purge_hash }, - { CLI_PURGE_URL, ccf_purge_url }, - { CLI_PURGE, ccf_purge }, - { CLI_PURGE_LIST, ccf_purge_list }, + { CLI_PURGE_URL, "", ccf_purge_url }, + { CLI_PURGE, "", ccf_purge }, + { CLI_PURGE_LIST, "", ccf_purge_list }, { NULL } }; @@ -809,7 +809,7 @@ { Lck_New(&ban_mtx); - CLI_AddFuncs(PUBLIC_CLI, ban_cmds); + CLI_AddFuncs(ban_cmds); ban_magic = BAN_New(); AN(ban_magic); Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -1278,9 +1278,9 @@ static struct cli_proto debug_cmds[] = { { "debug.xid", "debug.xid", - "\tExamine or set XID\n", 0, 1, cli_debug_xid }, + "\tExamine or set XID\n", 0, 1, "d", cli_debug_xid }, { "debug.srandom", "debug.srandom", - "\tSeed the random(3) function\n", 0, 1, cli_debug_srandom }, + "\tSeed the random(3) function\n", 0, 1, "d", cli_debug_srandom }, { NULL } }; @@ -1294,7 +1294,7 @@ srandomdev(); xids = random(); - CLI_AddFuncs(DEBUG_CLI, debug_cmds); + CLI_AddFuncs(debug_cmds); } Modified: trunk/varnish-cache/bin/varnishd/cache_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_cli.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -58,6 +58,7 @@ pthread_t cli_thread; static struct lock cli_mtx; static int add_check; +static struct cls *cls; /* * The CLI commandlist is split in three: @@ -67,29 +68,17 @@ * - Undocumented debug commands, show in undocumented "help -d" */ -static struct cli_proto *ccf_master_cli, *ccf_public_cli, *ccf_debug_cli; - /*-------------------------------------------------------------------- * Add CLI functions to the appropriate command set */ void -CLI_AddFuncs(enum cli_set_e which, struct cli_proto *p) +CLI_AddFuncs(struct cli_proto *p) { - struct cli_proto *c, **cp; AZ(add_check); - switch (which) { - case MASTER_CLI: cp = &ccf_master_cli; break; - case PUBLIC_CLI: cp = &ccf_public_cli; break; - case DEBUG_CLI: cp = &ccf_debug_cli; break; - default: INCOMPL(); - } Lck_Lock(&cli_mtx); - c = cli_concat(*cp, p); - AN(c); - free(*cp); - *cp = c; + AZ(CLS_AddFunc(cls, 0, p)); Lck_Unlock(&cli_mtx); } @@ -113,17 +102,11 @@ void CLI_Run(void) { - struct cls *cls; int i; add_check = 1; - cls = CLS_New(cli_cb_before, cli_cb_after, params->cli_buffer); - AN(cls); AN(CLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL)); - AZ(CLS_AddFunc(cls, 0, ccf_master_cli)); - AZ(CLS_AddFunc(cls, 0, ccf_public_cli)); - AZ(CLS_AddFunc(cls, 0, ccf_debug_cli)); do { i = CLS_Poll(cls, -1); @@ -160,27 +143,6 @@ /*--------------------------------------------------------------------*/ static void -ccf_help(struct cli *cli, const char * const *av, void *priv) -{ - - (void)priv; - cli_func_help(cli, av, ccf_public_cli); - - if (av[2] != NULL && !strcmp(av[2], "-d")) { - /* Also list undocumented commands */ - cli_out(cli, "\nDebugging commands:\n"); - cli_func_help(cli, av, ccf_debug_cli); - } else if (cli->result == CLIS_UNKNOWN) { - /* Otherwise, try the undocumented list */ - vsb_clear(cli->sb); - cli->result = CLIS_OK; - cli_func_help(cli, av, ccf_debug_cli); - } -} - -/*--------------------------------------------------------------------*/ - -static void ccf_panic(struct cli *cli, const char * const *av, void *priv) { @@ -193,18 +155,14 @@ /*--------------------------------------------------------------------*/ static struct cli_proto master_cmds[] = { - { CLI_PING, cli_func_ping }, - { CLI_HELP, ccf_help, NULL }, - { NULL } -}; - -static struct cli_proto debug_cmds[] = { + { CLI_PING, "i", CLS_func_ping }, + { CLI_HELP, "i", CLS_func_help }, { "debug.sizeof", "debug.sizeof", "\tDump sizeof various data structures\n", - 0, 0, cli_debug_sizeof }, + 0, 0, "d", cli_debug_sizeof }, { "debug.panic.worker", "debug.panic.worker", "\tPanic the worker process.\n", - 0, 0, ccf_panic }, + 0, 0, "d", ccf_panic }, { NULL } }; @@ -220,7 +178,9 @@ Lck_New(&cli_mtx); cli_thread = pthread_self(); - CLI_AddFuncs(MASTER_CLI, master_cmds); - CLI_AddFuncs(DEBUG_CLI, debug_cmds); + cls = CLS_New(cli_cb_before, cli_cb_after, params->cli_buffer); + AN(cls); + + CLI_AddFuncs(master_cmds); } Modified: trunk/varnish-cache/bin/varnishd/cache_fetch.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_fetch.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_fetch.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -544,7 +544,7 @@ static struct cli_proto debug_cmds[] = { { "debug.fragfetch", "debug.fragfetch", - "\tEnable fetch fragmentation\n", 1, 1, debug_fragfetch }, + "\tEnable fetch fragmentation\n", 1, 1, "d", debug_fragfetch }, { NULL } }; @@ -556,5 +556,5 @@ Fetch_Init(void) { - CLI_AddFuncs(DEBUG_CLI, debug_cmds); + CLI_AddFuncs(debug_cmds); } Modified: trunk/varnish-cache/bin/varnishd/cache_vcl.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vcl.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/cache_vcl.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -328,10 +328,10 @@ /*--------------------------------------------------------------------*/ static struct cli_proto vcl_cmds[] = { - { CLI_VCL_LOAD, ccf_config_load }, - { CLI_VCL_LIST, ccf_config_list }, - { CLI_VCL_DISCARD, ccf_config_discard }, - { CLI_VCL_USE, ccf_config_use }, + { CLI_VCL_LOAD, "i", ccf_config_load }, + { CLI_VCL_LIST, "i", ccf_config_list }, + { CLI_VCL_DISCARD, "i", ccf_config_discard }, + { CLI_VCL_USE, "i", ccf_config_use }, { NULL } }; @@ -339,6 +339,6 @@ VCL_Init() { - CLI_AddFuncs(MASTER_CLI, vcl_cmds); + CLI_AddFuncs(vcl_cmds); Lck_New(&vcl_mtx); } Modified: trunk/varnish-cache/bin/varnishd/hash_critbit.c =================================================================== --- trunk/varnish-cache/bin/varnishd/hash_critbit.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/hash_critbit.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -333,7 +333,7 @@ } static struct cli_proto hcb_cmds[] = { - { "hcb.dump", "hcb.dump", "dump HCB tree\n", 0, 0, hcb_dump }, + { "hcb.dump", "hcb.dump", "dump HCB tree\n", 0, 0, "d", hcb_dump }, { NULL } }; @@ -384,7 +384,7 @@ pthread_t tp; (void)oh; - CLI_AddFuncs(DEBUG_CLI, hcb_cmds); + CLI_AddFuncs(hcb_cmds); AZ(pthread_create(&tp, NULL, hcb_cleaner, NULL)); assert(sizeof(struct hcb_y) <= sizeof(oh->u)); memset(&hcb_root, 0, sizeof hcb_root); Modified: trunk/varnish-cache/bin/varnishd/mgt_cli.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/bin/varnishd/mgt_cli.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -72,9 +72,6 @@ #define MCF_NOAUTH 0 #define MCF_AUTH 16 -static void mcf_help(struct cli *cli, const char * const *av, void *priv); - - /*--------------------------------------------------------------------*/ static void @@ -92,22 +89,9 @@ #undef MAC_STAT } - /*--------------------------------------------------------------------*/ static void -mcf_close(struct cli *cli, const char *const *av, void *priv) -{ - - (void)av; - (void)priv; - cli_out(cli, "Closing CLI connection"); - cli_result(cli, CLIS_CLOSE); -} - -/*--------------------------------------------------------------------*/ - -static void mcf_banner(struct cli *cli, const char *const *av, void *priv) { @@ -127,22 +111,19 @@ /* XXX: what order should this list be in ? */ static struct cli_proto cli_proto[] = { - { CLI_BANNER, mcf_banner, NULL }, - { CLI_PING, cli_func_ping }, - { CLI_SERVER_STATUS, mcf_server_status, NULL }, - { CLI_SERVER_START, mcf_server_startstop, NULL }, - { CLI_SERVER_STOP, mcf_server_startstop, cli_proto }, - { CLI_STATS, mcf_stats, NULL }, - { CLI_VCL_LOAD, mcf_config_load, NULL }, - { CLI_VCL_INLINE, mcf_config_inline, NULL }, - { CLI_VCL_USE, mcf_config_use, NULL }, - { CLI_VCL_DISCARD, mcf_config_discard, NULL }, - { CLI_VCL_LIST, mcf_config_list, NULL }, - { CLI_VCL_SHOW, mcf_config_show, NULL }, - { CLI_PARAM_SHOW, mcf_param_show, NULL }, - { CLI_PARAM_SET, mcf_param_set, NULL }, - - { CLI_QUIT, mcf_close, NULL}, + { CLI_BANNER, "", mcf_banner, NULL }, + { CLI_SERVER_STATUS, "", mcf_server_status, NULL }, + { CLI_SERVER_START, "", mcf_server_startstop, NULL }, + { CLI_SERVER_STOP, "", mcf_server_startstop, cli_proto }, + { CLI_STATS, "", mcf_stats, NULL }, + { CLI_VCL_LOAD, "", mcf_config_load, NULL }, + { CLI_VCL_INLINE, "", mcf_config_inline, NULL }, + { CLI_VCL_USE, "", mcf_config_use, NULL }, + { CLI_VCL_DISCARD, "", mcf_config_discard, NULL }, + { CLI_VCL_LIST, "", mcf_config_list, NULL }, + { CLI_VCL_SHOW, "", mcf_config_show, NULL }, + { CLI_PARAM_SHOW, "", mcf_param_show, NULL }, + { CLI_PARAM_SET, "", mcf_param_set, NULL }, { NULL } }; @@ -161,7 +142,7 @@ static struct cli_proto cli_debug[] = { { "debug.panic.master", "debug.panic.master", "\tPanic the master process.\n", - 0, 0, mcf_panic, NULL}, + 0, 0, "d", mcf_panic, NULL}, { NULL } }; @@ -181,6 +162,10 @@ * running. */ if (cli_o <= 0) { + if (!strcmp(av[1], "help")) { + cli_out(cli, "No help from child, (not running).\n"); + return; + } cli_result(cli, CLIS_UNKNOWN); cli_out(cli, "Unknown request in manager process " @@ -202,7 +187,7 @@ static struct cli_proto cli_askchild[] = { { "*", "", "\t\n", - 0, 9999, mcf_askchild, NULL}, + 0, 9999, "h*", mcf_askchild, NULL}, { NULL } }; @@ -348,39 +333,15 @@ } static struct cli_proto cli_auth[] = { - { CLI_HELP, mcf_help, cli_auth }, - { CLI_AUTH, mcf_auth, NULL }, - { CLI_QUIT, mcf_close, NULL}, + { CLI_HELP, "", CLS_func_help, NULL }, + { CLI_PING, "", CLS_func_ping }, + { CLI_AUTH, "", mcf_auth, NULL }, + { CLI_QUIT, "", CLS_func_close, NULL}, { NULL } }; /*--------------------------------------------------------------------*/ - static void -mcf_help(struct cli *cli, const char * const *av, void *priv) -{ - unsigned u; - char *p; - - (void)priv; - cli_func_help(cli, av, cli_auth); - if (cli->auth == MCF_NOAUTH) - return; - cli_func_help(cli, av, cli_proto); - if (cli_o >= 0 && (av[2] == NULL || *av[2] == '-')) { - p = NULL; - if (!mgt_cli_askchild(&u, &p, - "help %s\n", av[2] != NULL ? av[2] : "")) { - cli_out(cli, "%s", p); - cli_result(cli, u); - } - free(p); - } -} - - -/*--------------------------------------------------------------------*/ -static void mgt_cli_cb_before(const struct cli *cli) { Modified: trunk/varnish-cache/include/cli_common.h =================================================================== --- trunk/varnish-cache/include/cli_common.h 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/include/cli_common.h 2010-01-21 10:26:39 UTC (rev 4477) @@ -30,6 +30,7 @@ */ struct vlu; +struct cls; struct cli { unsigned magic; @@ -41,6 +42,7 @@ char challenge[34]; char *ident; struct vlu *vlu; + struct cls *cls; }; int cli_writeres(int fd, const struct cli *cli); Modified: trunk/varnish-cache/include/cli_priv.h =================================================================== --- trunk/varnish-cache/include/cli_priv.h 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/include/cli_priv.h 2010-01-21 10:26:39 UTC (rev 4477) @@ -46,6 +46,7 @@ const char *help; unsigned minarg; unsigned maxarg; + char flags[4]; /* Dispatch information */ cli_func_t *func; @@ -56,8 +57,3 @@ void cli_out(struct cli *cli, const char *fmt, ...); void cli_quote(struct cli *cli, const char *str); void cli_result(struct cli *cli, unsigned r); - -/* From libvarnish/cli.c */ -cli_func_t cli_func_help; -cli_func_t cli_func_ping; -struct cli_proto *cli_concat(struct cli_proto *, struct cli_proto *); Modified: trunk/varnish-cache/include/cli_serve.h =================================================================== --- trunk/varnish-cache/include/cli_serve.h 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/include/cli_serve.h 2010-01-21 10:26:39 UTC (rev 4477) @@ -40,3 +40,8 @@ int CLS_PollFd(struct cls *cs, int fd, int timeout); void CLS_Destroy(struct cls **); +/* From libvarnish/cli.c */ +cli_func_t CLS_func_close; +cli_func_t CLS_func_help; +cli_func_t CLS_func_ping; + Modified: trunk/varnish-cache/lib/libvarnish/Makefile.am =================================================================== --- trunk/varnish-cache/lib/libvarnish/Makefile.am 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/lib/libvarnish/Makefile.am 2010-01-21 10:26:39 UTC (rev 4477) @@ -11,7 +11,6 @@ assert.c \ binary_heap.c \ subproc.c \ - cli.c \ cli_common.c \ cli_serve.c \ flopen.c \ Deleted: trunk/varnish-cache/lib/libvarnish/cli.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/lib/libvarnish/cli.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -1,97 +0,0 @@ -/*- - * Copyright (c) 2006 Verdens Gang AS - * Copyright (c) 2006-2009 Linpro AS - * All rights reserved. - * - * Author: Poul-Henning Kamp - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * - * Stuff for handling the CLI protocol - */ - -#include "config.h" - -#include "svnid.h" -SVNID("$Id$") - -#include -#include -#include -#include - -#include -#include -#include - -/* - * Generic help function. - * - * priv must point to cli_proto array - */ - -void -cli_func_help(struct cli *cli, const char * const *av, void *priv) -{ - struct cli_proto *cp; - - if (av[2] == NULL || *av[2] == '-') { - for (cp = priv; cp->request != NULL; cp++) - if (cp->syntax != NULL) - cli_out(cli, "%s\n", cp->syntax); - return; - } - for (cp = priv; cp->request != NULL; cp++) { - if (cp->syntax == NULL) - continue; - if (!strcmp(cp->request, av[2])) { - cli_out(cli, "%s\n%s\n", cp->syntax, cp->help); - return; - } - } - cli_out(cli, "Unknown request.\nType 'help' for more info.\n"); - cli_result(cli, CLIS_UNKNOWN); -} -struct cli_proto * -cli_concat(struct cli_proto *c1, struct cli_proto *c2) -{ - struct cli_proto *c; - int i1, i2; - - i1 = 0; - for(c = c1; c != NULL && c->request != NULL; c++) - i1++; - i2 = 0; - for(c = c2; c != NULL && c->request != NULL; c++) - i2++; - - c = malloc(sizeof(*c) * (1L + i1 + i2)); - if (c == NULL) - return (c); - if (c1 != NULL) - memcpy(c, c1, sizeof(*c1) * i1); - if (c2 != NULL) - memcpy(c + i1, c2, sizeof(*c2) * i2); - memset(c + i1 + i2, 0, sizeof(*c)); - return (c); -} - Modified: trunk/varnish-cache/lib/libvarnish/cli_common.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_common.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/lib/libvarnish/cli_common.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -184,16 +184,3 @@ *ptr = p; return (0); } - -/*--------------------------------------------------------------------*/ - -void -cli_func_ping(struct cli *cli, const char * const *av, void *priv) -{ - time_t t; - - (void)priv; - (void)av; - t = time(NULL); - cli_out(cli, "PONG %ld 1.0", t); -} Modified: trunk/varnish-cache/lib/libvarnish/cli_serve.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-20 18:12:19 UTC (rev 4476) +++ trunk/varnish-cache/lib/libvarnish/cli_serve.c 2010-01-21 10:26:39 UTC (rev 4477) @@ -80,6 +80,107 @@ unsigned maxlen; }; +/*--------------------------------------------------------------------*/ + +void +CLS_func_close(struct cli *cli, const char *const *av, void *priv) +{ + + (void)av; + (void)priv; + cli_out(cli, "Closing CLI connection"); + cli_result(cli, CLIS_CLOSE); +} + +/*--------------------------------------------------------------------*/ + +void +CLS_func_ping(struct cli *cli, const char * const *av, void *priv) +{ + time_t t; + + (void)priv; + (void)av; + t = time(NULL); + cli_out(cli, "PONG %ld 1.0", t); +} + +/*--------------------------------------------------------------------*/ + +void +CLS_func_help(struct cli *cli, const char * const *av, void *priv) +{ + struct cli_proto *cp; + struct cls_func *cfn; + unsigned all, debug, u, d, h, i, wc; + struct cls *cs; + + (void)priv; + cs = cli->cls; + CHECK_OBJ_NOTNULL(cs, CLS_MAGIC); + + if (av[2] == NULL) { + all = debug = 0; + } else if (!strcmp(av[2], "-a")) { + all = 1; + debug = 0; + } else if (!strcmp(av[2], "-d")) { + all = 0; + debug = 1; + } else { + VTAILQ_FOREACH(cfn, &cs->funcs, list) { + for (cp = cfn->clp; cp->request != NULL; cp++) { + if (!strcmp(cp->request, av[2])) { + cli_out(cli, "%s\n%s\n", + cp->syntax, cp->help); + return; + } + for (u = 0; u < sizeof cp->flags; u++) { + if (cp->flags[u] == '*') { + cp->func(cli,av,priv); + return; + } + } + } + } + cli_out(cli, "Unknown request.\nType 'help' for more info.\n"); + cli_result(cli, CLIS_UNKNOWN); + return; + } + VTAILQ_FOREACH(cfn, &cs->funcs, list) { + for (cp = cfn->clp; cp->request != NULL; cp++) { + d = 0; + h = 0; + i = 0; + wc = 0; + for (u = 0; u < sizeof cp->flags; u++) { + if (cp->flags[u] == '\0') + continue; + if (cp->flags[u] == 'd') + d = 1; + if (cp->flags[u] == 'h') + h = 1; + if (cp->flags[u] == 'i') + i = 1; + if (cp->flags[u] == '*') + wc = 1; + } + if (i) + continue; + if (wc) { + cp->func(cli, av, priv); + continue; + } + if (debug != d) + continue; + if (h && !all) + continue; + if (cp->syntax != NULL) + cli_out(cli, "%s\n", cp->syntax); + } + } +} + /*-------------------------------------------------------------------- * Look for a CLI command to execute */ @@ -156,6 +257,7 @@ return (0); cli->cmd = p; + cli->cls = cs; av = ParseArgv(p, 0); AN(av); @@ -201,6 +303,7 @@ cs->after(cli); cli->cmd = NULL; + cli->cls = NULL; FreeArgv(av); if (cli_writeres(cfd->fdo, cli) || cli->result == CLIS_CLOSE) From phk at projects.linpro.no Fri Jan 22 08:25:26 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 22 Jan 2010 09:25:26 +0100 (CET) Subject: r4478 - trunk/varnish-cache/bin/varnishd Message-ID: <20100122082526.52FDB28157@projects.linpro.no> Author: phk Date: 2010-01-22 09:25:26 +0100 (Fri, 22 Jan 2010) New Revision: 4478 Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c Log: Fix the poll-waiter after I botched it in r4445. Fixes #625 Submitted by: slink Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-21 10:26:39 UTC (rev 4477) +++ trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-22 08:25:26 UTC (rev 4478) @@ -124,7 +124,7 @@ i = read(vca_pipes[0], ss, sizeof ss); assert(i >= 0); assert((i % sizeof ss[0]) == 0); - for (j = 0; j * sizeof ss[0] < i; j += sizeof ss[0]) { + for (j = 0; j * sizeof ss[0] < i; j++) { CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC); VTAILQ_INSERT_TAIL(&sesshead, ss[j], list); vca_poll(ss[j]->fd); From phk at projects.linpro.no Mon Jan 25 11:36:43 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 12:36:43 +0100 (CET) Subject: r4479 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125113643.71D6E1F7C61@projects.linpro.no> Author: phk Date: 2010-01-25 12:36:43 +0100 (Mon, 25 Jan 2010) New Revision: 4479 Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c Log: Don't mess up the vcl name of the VCL created with -f or -b, need to add proper metadata/comment field. Spotted by: Kristian Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-22 08:25:26 UTC (rev 4478) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2010-01-25 11:36:43 UTC (rev 4479) @@ -332,6 +332,8 @@ struct vclprog *vp; char buf[BUFSIZ]; + /* XXX: annotate vcl with -b/-f arg so people know where it came from */ + (void)f_arg; if (b_arg != NULL) { AZ(vcl); /* @@ -364,10 +366,8 @@ free(port); vcl = strdup(buf); AN(vcl); - bprintf(buf, "boot (-b %s)", b_arg); - } else { - bprintf(buf, "boot (-f %s)", f_arg); } + strcpy(buf, "boot"); vf = mgt_VccCompile(&sb, vcl, C_flag); free(vcl); From kristian at projects.linpro.no Mon Jan 25 12:19:22 2010 From: kristian at projects.linpro.no (kristian at projects.linpro.no) Date: Mon, 25 Jan 2010 13:19:22 +0100 (CET) Subject: r4480 - trunk/varnish-cache/bin/varnishncsa Message-ID: <20100125121922.9C4631F7C7E@projects.linpro.no> Author: kristian Date: 2010-01-25 13:19:22 +0100 (Mon, 25 Jan 2010) New Revision: 4480 Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c Log: Only handle client requests (-c) for varnishncsa Fixes #617 Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c =================================================================== --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2010-01-25 11:36:43 UTC (rev 4479) +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2010-01-25 12:19:22 UTC (rev 4480) @@ -553,6 +553,13 @@ case 'w': w_arg = optarg; break; + case 'b': + fprintf(stderr, "-b is not valid for varnishncsa\n"); + exit(1); + break; + case 'c': + /* XXX: Silently ignored: it's required anyway */ + break; default: if (VSL_Arg(vd, c, optarg) > 0) break; @@ -560,6 +567,8 @@ } } + VSL_Arg(vd, 'c', optarg); + if (VSL_OpenLog(vd, n_arg)) exit(1); From phk at projects.linpro.no Mon Jan 25 13:42:04 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 14:42:04 +0100 (CET) Subject: r4481 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125134204.CDD7D1F7C96@projects.linpro.no> Author: phk Date: 2010-01-25 14:42:04 +0100 (Mon, 25 Jan 2010) New Revision: 4481 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/bin/varnishd/cache_panic.c trunk/varnish-cache/bin/varnishd/common.h trunk/varnish-cache/bin/varnishd/varnishd.c Log: Add an ident-string to panic output, which reveals arch,os,rev and which subsystems are used (storage/hash/waiter) but not the options given to these. This info is often missing in bugreports. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-25 12:19:22 UTC (rev 4480) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-25 13:42:04 UTC (rev 4481) @@ -446,6 +446,7 @@ void vca_close_session(struct sess *sp, const char *why); void VCA_Prep(struct sess *sp); void VCA_Init(void); +const char *VCA_waiter_name(void); extern pthread_t VCA_thread; /* cache_backend.c */ Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-25 12:19:22 UTC (rev 4480) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-25 13:42:04 UTC (rev 4481) @@ -70,7 +70,22 @@ static struct timeval tv_sndtimeo; static struct timeval tv_rcvtimeo; -/* +/*-------------------------------------------------------------------- + * Report waiter name to panics + */ + +const char * +VCA_waiter_name(void) +{ + + if (vca_act != NULL) + return (vca_act->name); + else + return ("no_waiter"); +} + + +/*-------------------------------------------------------------------- * We want to get out of any kind of touble-hit TCP connections as fast * as absolutely possible, so we set them LINGER enabled with zero timeout, * so that even if there are outstanding write data on the socket, a close(2) @@ -395,6 +410,8 @@ { int i; + ASSERT_MGT(); + if (arg == NULL) { if (vca_act == NULL) cli_out(cli, "default"); Modified: trunk/varnish-cache/bin/varnishd/cache_panic.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_panic.c 2010-01-25 12:19:22 UTC (rev 4480) +++ trunk/varnish-cache/bin/varnishd/cache_panic.c 2010-01-25 13:42:04 UTC (rev 4481) @@ -326,6 +326,9 @@ if (q != NULL) vsb_printf(vsp, "thread = (%s)\n", q); + vsb_printf(vsp, "ident = %s,%s\n", + vsb_data(vident) + 1, VCA_waiter_name()); + pan_backtrace(); if (!(params->diag_bitmap & 0x2000)) { Modified: trunk/varnish-cache/bin/varnishd/common.h =================================================================== --- trunk/varnish-cache/bin/varnishd/common.h 2010-01-25 12:19:22 UTC (rev 4480) +++ trunk/varnish-cache/bin/varnishd/common.h 2010-01-25 13:42:04 UTC (rev 4481) @@ -31,6 +31,7 @@ struct cli; struct sockaddr; + extern pid_t mgt_pid; #define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0) @@ -46,6 +47,7 @@ /* varnishd.c */ struct vsb; +extern struct vsb *vident; int Symbol_Lookup(struct vsb *vsb, void *ptr); #define TRUST_ME(ptr) ((void*)(uintptr_t)(ptr)) Modified: trunk/varnish-cache/bin/varnishd/varnishd.c =================================================================== --- trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-25 12:19:22 UTC (rev 4480) +++ trunk/varnish-cache/bin/varnishd/varnishd.c 2010-01-25 13:42:04 UTC (rev 4481) @@ -49,6 +49,8 @@ #include #include +#include + #include "compat/daemon.h" #ifndef HAVE_STRLCPY @@ -81,7 +83,22 @@ pid_t mgt_pid; struct vev_base *mgt_evb; int exit_status = 0; +struct vsb *vident; +static void +build_vident(void) +{ + struct utsname uts; + + vident = vsb_newauto(); + AN(vident); + if (!uname(&uts)) { + vsb_printf(vident, ",%s", uts.sysname); + vsb_printf(vident, ",%s", uts.release); + vsb_printf(vident, ",%s", uts.machine); + } +} + /*--------------------------------------------------------------------*/ static void * @@ -132,6 +149,7 @@ priv = pick(STV_choice, av[1], "storage"); AN(priv); + vsb_printf(vident, ",-s%s", av[1]); STV_add(priv, ac, av + 2); @@ -161,6 +179,7 @@ hp = pick(hsh_choice, av[1], "hash"); CHECK_OBJ_NOTNULL(hp, SLINGER_MAGIC); + vsb_printf(vident, ",-h%s", av[1]); heritage.hash = hp; if (hp->init != NULL) hp->init(ac, av + 2); @@ -526,6 +545,8 @@ setbuf(stdout, NULL); setbuf(stderr, NULL); + build_vident(); + Symbol_hack(argv[0]); /* for ASSERT_MGT() */ @@ -734,6 +755,9 @@ VSL_MgtInit(SHMLOG_FILENAME, l_size); + vsb_finish(vident); + AZ(vsb_overflowed(vident)); + if (d_flag == 1) DebugStunt(); if (d_flag < 2 && !F_flag) @@ -744,6 +768,9 @@ if (pfh != NULL && vpf_write(pfh)) fprintf(stderr, "NOTE: Could not write PID file\n"); + if (d_flag > 0) + fprintf(stderr, "Varnish on %s\n", vsb_data(vident) + 1); + /* Do this again after debugstunt and daemon has run */ mgt_pid = getpid(); From tfheen at projects.linpro.no Mon Jan 25 14:47:29 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Mon, 25 Jan 2010 15:47:29 +0100 (CET) Subject: r4482 - in trunk/varnish-cache: . bin/varnishd Message-ID: <20100125144729.4AFEB1F7E33@projects.linpro.no> Author: tfheen Date: 2010-01-25 15:47:29 +0100 (Mon, 25 Jan 2010) New Revision: 4482 Modified: trunk/varnish-cache/bin/varnishd/mgt_child.c trunk/varnish-cache/configure.ac Log: Drop all privileges in worker children on Solaris Fixes #628 Modified: trunk/varnish-cache/bin/varnishd/mgt_child.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_child.c 2010-01-25 13:42:04 UTC (rev 4481) +++ trunk/varnish-cache/bin/varnishd/mgt_child.c 2010-01-25 14:47:29 UTC (rev 4482) @@ -46,6 +46,10 @@ #include #include +#ifdef HAVE_PRIV_H +#include +#endif + #ifndef HAVE_SETPROCTITLE #include "compat/setproctitle.h" #endif @@ -228,6 +232,36 @@ /*--------------------------------------------------------------------*/ +/* Waive all privileges in the child, it does not need any */ + +static inline void +waive_privileges(void) +{ +#ifdef HAVE_SETPPRIV + priv_set_t *empty; + + if (!(empty = priv_allocset())) { + perror("priv_allocset_failed"); + return; + } + priv_emptyset(empty); + +#define SETPPRIV(which, set) \ + if (setppriv(PRIV_SET, which, set)) \ + perror("Waiving privileges failed on " #which) + + SETPPRIV(PRIV_LIMIT, empty); + SETPPRIV(PRIV_INHERITABLE, empty); + SETPPRIV(PRIV_PERMITTED, empty); /* implies PRIV_EFFECTIVE */ + + priv_freeset(empty); +#else + return; +#endif +} + +/*--------------------------------------------------------------------*/ + static void start_child(struct cli *cli) { @@ -312,6 +346,8 @@ } printf("\n"); + waive_privileges(); + setproctitle("Varnish-Chld %s", heritage.name); (void)signal(SIGINT, SIG_DFL); Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2010-01-25 13:42:04 UTC (rev 4481) +++ trunk/varnish-cache/configure.ac 2010-01-25 14:47:29 UTC (rev 4482) @@ -126,6 +126,7 @@ AC_CHECK_HEADERS([stddef.h]) AC_CHECK_HEADERS([stdlib.h]) AC_CHECK_HEADERS([unistd.h]) +AC_CHECK_HEADERS([priv.h]) # Checks for typedefs, structures, and compiler characteristics. AC_C_CONST @@ -150,6 +151,7 @@ AC_CHECK_FUNCS([abort2]) AC_CHECK_FUNCS([timegm]) AC_CHECK_FUNCS([nanosleep]) +AC_CHECK_FUNCS([setppriv]) save_LIBS="${LIBS}" LIBS="${PTHREAD_LIBS}" From phk at projects.linpro.no Mon Jan 25 14:51:33 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 15:51:33 +0100 (CET) Subject: r4483 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125145133.4B20F1F7C96@projects.linpro.no> Author: phk Date: 2010-01-25 15:51:32 +0100 (Mon, 25 Jan 2010) New Revision: 4483 Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c Log: Go over the poll-waiter with a fine tooth comb, just in case. Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 14:47:29 UTC (rev 4482) +++ trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 14:51:32 UTC (rev 4483) @@ -63,15 +63,18 @@ if (fd < npoll) return; newnpoll = npoll; + if (newnpoll == 0) + newnpoll = 1; while (fd >= newnpoll) - newnpoll = newnpoll * 2 + 1; + newnpoll = newnpoll * 2; VSL(SLT_Debug, 0, "Acceptor poll space increased to %u", newnpoll); newpollfd = realloc(newpollfd, newnpoll * sizeof *newpollfd); - XXXAN(newpollfd); /* close offending fd */ + XXXAN(newpollfd); memset(newpollfd + npoll, 0, (newnpoll - npoll) * sizeof *newpollfd); pollfd = newpollfd; while (npoll < newnpoll) pollfd[npoll++].fd = -1; + assert(fd < npoll); } /*--------------------------------------------------------------------*/ @@ -82,6 +85,7 @@ assert(fd >= 0); vca_pollspace((unsigned)fd); + assert(fd < npoll); if (hpoll < fd) hpoll = fd; pollfd[fd].fd = fd; @@ -92,14 +96,11 @@ vca_unpoll(int fd) { + assert(fd < npoll); assert(fd >= 0); vca_pollspace((unsigned)fd); pollfd[fd].fd = -1; pollfd[fd].events = 0; - if (hpoll == fd) { - while (pollfd[--hpoll].fd == -1) - continue; - } } /*--------------------------------------------------------------------*/ @@ -107,7 +108,7 @@ static void * vca_main(void *arg) { - unsigned v; + int v; struct sess *ss[NEEV], *sp, *sp2; double deadline; int i, j, fd; @@ -118,14 +119,19 @@ vca_poll(vca_pipes[0]); while (1) { + assert(hpoll < npoll); + while (hpoll > 0 && pollfd[hpoll].fd == -1) + hpoll--; v = poll(pollfd, hpoll + 1, 100); + assert(v >= 0); if (v && pollfd[vca_pipes[0]].revents) { v--; i = read(vca_pipes[0], ss, sizeof ss); assert(i >= 0); - assert((i % sizeof ss[0]) == 0); + assert(((unsigned)i % sizeof ss[0]) == 0); for (j = 0; j * sizeof ss[0] < i; j++) { CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC); + assert(ss[j]->fd >= 0); VTAILQ_INSERT_TAIL(&sesshead, ss[j], list); vca_poll(ss[j]->fd); } @@ -136,25 +142,25 @@ break; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); fd = sp->fd; + assert(pollfd[fd].fd == fd); if (pollfd[fd].revents) { v--; i = HTC_Rx(sp->htc); VTAILQ_REMOVE(&sesshead, sp, list); if (i == 0) { + /* Mov to front of list for speed */ VTAILQ_INSERT_HEAD(&sesshead, sp, list); - continue; + } else { + vca_unpoll(fd); + vca_handover(sp, i); } + } else if (sp->t_open <= deadline) { + VTAILQ_REMOVE(&sesshead, sp, list); vca_unpoll(fd); - vca_handover(sp, i); - continue; + TCP_linger(sp->fd, 0); + vca_close_session(sp, "timeout"); + SES_Delete(sp); } - if (sp->t_open > deadline) - continue; - VTAILQ_REMOVE(&sesshead, sp, list); - vca_unpoll(fd); - TCP_linger(sp->fd, 0); - vca_close_session(sp, "timeout"); - SES_Delete(sp); } } NEEDLESS_RETURN(NULL); @@ -166,6 +172,7 @@ vca_poll_init(void) { + vca_pollspace(256); AZ(pthread_create(&vca_poll_thread, NULL, vca_main, NULL)); } From phk at projects.linpro.no Mon Jan 25 19:00:24 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 20:00:24 +0100 (CET) Subject: r4484 - in trunk/varnish-cache/bin: varnishd varnishtest/tests Message-ID: <20100125190024.2D9471F7E36@projects.linpro.no> Author: phk Date: 2010-01-25 20:00:23 +0100 (Mon, 25 Jan 2010) New Revision: 4484 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c trunk/varnish-cache/bin/varnishtest/tests/b00009.vtc Log: Add more paranoia to the poll waiter. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-25 14:51:32 UTC (rev 4483) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-25 19:00:23 UTC (rev 4484) @@ -97,7 +97,7 @@ static unsigned char need_sndtimeo, need_rcvtimeo, need_linger, need_test; -int vca_pipes[2]; +int vca_pipes[2] = { -1, -1 }; static void sock_test(int fd) @@ -344,7 +344,6 @@ vca_act->pass(sp); } - /*--------------------------------------------------------------------*/ static void Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 14:51:32 UTC (rev 4483) +++ trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 19:00:23 UTC (rev 4484) @@ -122,9 +122,17 @@ assert(hpoll < npoll); while (hpoll > 0 && pollfd[hpoll].fd == -1) hpoll--; + assert(vca_pipes[0] <= hpoll); + assert(pollfd[vca_pipes[0]].fd = vca_pipes[0]); + assert(pollfd[vca_pipes[1]].fd = -1); v = poll(pollfd, hpoll + 1, 100); assert(v >= 0); if (v && pollfd[vca_pipes[0]].revents) { + + if (pollfd[vca_pipes[0]].revents != POLLIN) + VSL(SLT_Debug, 0, "pipe.revents= 0x%x", + pollfd[vca_pipes[0]].revents); + assert(pollfd[vca_pipes[0]].revents == POLLIN); v--; i = read(vca_pipes[0], ss, sizeof ss); assert(i >= 0); Modified: trunk/varnish-cache/bin/varnishtest/tests/b00009.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00009.vtc 2010-01-25 14:51:32 UTC (rev 4483) +++ trunk/varnish-cache/bin/varnishtest/tests/b00009.vtc 2010-01-25 19:00:23 UTC (rev 4484) @@ -13,4 +13,8 @@ txreq -url "/" rxresp expect resp.status == 200 + delay .1 + txreq -url "/" + rxresp + expect resp.status == 200 } -run From phk at projects.linpro.no Mon Jan 25 19:44:57 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 20:44:57 +0100 (CET) Subject: r4485 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100125194457.C35471F7E36@projects.linpro.no> Author: phk Date: 2010-01-25 20:44:57 +0100 (Mon, 25 Jan 2010) New Revision: 4485 Modified: trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc Log: Don't spit the compiled VCL on stdout, /dev/null it. Modified: trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc 2010-01-25 19:00:23 UTC (rev 4484) +++ trunk/varnish-cache/bin/varnishtest/tests/a00009.vtc 2010-01-25 19:44:57 UTC (rev 4485) @@ -2,4 +2,4 @@ test "See that the VCL compiler works" -shell "cd ../varnishd && ./varnishd -b 127.0.0.1:80 -C -n ${tmpdir}" +shell "cd ../varnishd && ./varnishd -b 127.0.0.1:80 -C -n ${tmpdir} > /dev/null" From phk at projects.linpro.no Mon Jan 25 20:04:24 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 21:04:24 +0100 (CET) Subject: r4486 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100125200424.C5B351F7E37@projects.linpro.no> Author: phk Date: 2010-01-25 21:04:24 +0100 (Mon, 25 Jan 2010) New Revision: 4486 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c trunk/varnish-cache/bin/varnishtest/vtc.h trunk/varnish-cache/bin/varnishtest/vtc_log.c Log: Make verbose=0 the default. At verbose=0, only spit out a single line if a test passes, but dump the full show if it fails. Should make "make check" less tedious to read through. Fix a race situation related to vsb's while shutting down. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-25 19:44:57 UTC (rev 4485) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-25 20:04:24 UTC (rev 4486) @@ -336,7 +336,7 @@ return; assert(!strcmp(av[0], "test")); - printf("# TEST %s\n", av[1]); + vtc_log(vl, 1, "TEST %s", av[1]); AZ(av[2]); vtc_desc = strdup(av[1]); } @@ -460,12 +460,14 @@ { NULL, NULL } }; -static void +static double exec_file(const char *fn, struct vtclog *vl) { char *buf; + double t0; unsigned old_err; + t0 = TIM_mono(); vtc_stop = 0; vtc_file = fn; vtc_desc = NULL; @@ -480,12 +482,24 @@ vtc_log(vl, 1, "RESETTING after %s", fn); reset_cmds(cmds); vtc_error = old_err; + if (vtc_error) vtc_log(vl, 1, "TEST %s FAILED", fn); - else + else { vtc_log(vl, 1, "TEST %s completed", fn); + vtc_logreset(); + } + + t0 = TIM_mono() - t0; + + if (vtc_error && vtc_verbosity == 0) + printf("%s", vtc_logfull()); + else if (vtc_verbosity == 0) + printf("# top TEST %s passed (%.3fs)\n", fn, t0); + vtc_file = NULL; free(vtc_desc); + return (t0); } /********************************************************************** @@ -506,7 +520,7 @@ int main(int argc, char * const *argv) { - int ch, i, ntest = 1; + int ch, i, ntest = 1, ncheck = 0; FILE *fok; static struct vtclog *vl; double tmax, t0, t00; @@ -515,6 +529,7 @@ setbuf(stdout, NULL); setbuf(stderr, NULL); + vtc_loginit(); vl = vtc_logopen("top"); AN(vl); while ((ch = getopt(argc, argv, "n:qv")) != -1) { @@ -551,9 +566,8 @@ t00 = TIM_mono(); for (i = 0; i < ntest; i++) { for (ch = 0; ch < argc; ch++) { - t0 = TIM_mono(); - exec_file(argv[ch], vl); - t0 = TIM_mono() - t0; + t0 = exec_file(argv[ch], vl); + ncheck++; if (t0 > tmax) { tmax = t0; nmax = argv[ch]; @@ -576,9 +590,11 @@ return (2); t00 = TIM_mono() - t00; - if (tmax > 0 && nmax != NULL) - vtc_log(vl, 1, "Slowest test: %s %.3fs", nmax, tmax); - vtc_log(vl, 1, "Total duration: %.3fs", t00); + if (ncheck > 1) { + printf("# top Slowest test: %s %.3fs\n", nmax, tmax); + printf("# top Total tests run: %d\n", ncheck); + printf("# top Total duration: %.3fs\n", t00); + } fok = fopen("_.ok", "w"); if (fok != NULL) Modified: trunk/varnish-cache/bin/varnishtest/vtc.h =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-25 19:44:57 UTC (rev 4485) +++ trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-25 20:04:24 UTC (rev 4486) @@ -67,6 +67,9 @@ void cmd_server_genvcl(struct vsb *vsb); +void vtc_loginit(void); +void vtc_logreset(void); +const char *vtc_logfull(void); struct vtclog *vtc_logopen(const char *id); void vtc_logclose(struct vtclog *vl); void vtc_log(struct vtclog *vl, unsigned lvl, const char *fmt, ...); Modified: trunk/varnish-cache/bin/varnishtest/vtc_log.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_log.c 2010-01-25 19:44:57 UTC (rev 4485) +++ trunk/varnish-cache/bin/varnishtest/vtc_log.c 2010-01-25 20:04:24 UTC (rev 4486) @@ -45,15 +45,48 @@ #include "vtc.h" -int vtc_verbosity = 3; +int vtc_verbosity = 0; +static struct vsb *vtclog_full; +static pthread_mutex_t vtclog_mtx; + struct vtclog { unsigned magic; #define VTCLOG_MAGIC 0x82731202 const char *id; struct vsb *vsb; + pthread_mutex_t mtx; }; +/**********************************************************************/ + +void +vtc_loginit() +{ + + vtclog_full = vsb_newauto(); + AN(vtclog_full); + AZ(pthread_mutex_init(&vtclog_mtx, NULL)); +} + +void +vtc_logreset() +{ + + vsb_clear(vtclog_full); +} + +const char * +vtc_logfull(void) +{ + vsb_finish(vtclog_full); + AZ(vsb_overflowed(vtclog_full)); + return (vsb_data(vtclog_full)); +} + +/**********************************************************************/ + + struct vtclog * vtc_logopen(const char *id) { @@ -63,6 +96,7 @@ AN(vl); vl->id = id; vl->vsb = vsb_newauto(); + AZ(pthread_mutex_init(&vl->mtx, NULL)); return (vl); } @@ -72,6 +106,7 @@ CHECK_OBJ_NOTNULL(vl, VTCLOG_MAGIC); vsb_delete(vl->vsb); + AZ(pthread_mutex_destroy(&vl->mtx)); FREE_OBJ(vl); } @@ -91,9 +126,8 @@ { CHECK_OBJ_NOTNULL(vl, VTCLOG_MAGIC); + AZ(pthread_mutex_lock(&vl->mtx)); assert(lvl < NLEAD); - if (lvl > vtc_verbosity) - return; vsb_clear(vl->vsb); vsb_printf(vl->vsb, "%s %-4s ", lead[lvl], vl->id); va_list ap; @@ -103,11 +137,16 @@ vsb_putc(vl->vsb, '\n'); vsb_finish(vl->vsb); AZ(vsb_overflowed(vl->vsb)); - (void)fputs(vsb_data(vl->vsb), stdout); + + AZ(pthread_mutex_lock(&vtclog_mtx)); + vsb_cat(vtclog_full, vsb_data(vl->vsb)); + AZ(pthread_mutex_unlock(&vtclog_mtx)); + + if (lvl > 0 && lvl <= vtc_verbosity) + (void)fputs(vsb_data(vl->vsb), stdout); vsb_clear(vl->vsb); + AZ(pthread_mutex_unlock(&vl->mtx)); if (lvl == 0) { - printf("---- TEST FILE: %s\n", vtc_file); - printf("---- TEST DESCRIPTION: %s\n", vtc_desc); vtc_error = 1; if (pthread_self() != vtc_thread) pthread_exit(NULL); From phk at projects.linpro.no Mon Jan 25 20:46:49 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 21:46:49 +0100 (CET) Subject: r4487 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100125204649.B40CF1F7E37@projects.linpro.no> Author: phk Date: 2010-01-25 21:46:49 +0100 (Mon, 25 Jan 2010) New Revision: 4487 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c trunk/varnish-cache/bin/varnishtest/vtc.h trunk/varnish-cache/bin/varnishtest/vtc_varnish.c Log: Go further with the uniquely named tmpdir thing and put the varnishd workdir below there as well. Do not remove the tmpdir if we have verbosity and failure, otherwise do. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-25 20:04:24 UTC (rev 4486) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-25 20:46:49 UTC (rev 4487) @@ -42,7 +42,6 @@ #include #include #include -#include "compat/asprintf.h" #include "libvarnish.h" #include "vsb.h" @@ -59,6 +58,7 @@ int vtc_error; /* Error encountered */ int vtc_stop; /* Stops current test without error */ pthread_t vtc_thread; +char *vtc_tmpdir; /********************************************************************** * Macro facility @@ -140,7 +140,7 @@ } struct vsb * -macro_expand(char *name) +macro_expand(const char *name) { struct vsb *vsb; char *p, *q; @@ -525,7 +525,7 @@ static struct vtclog *vl; double tmax, t0, t00; const char *nmax; - char *tmpdir, *cmd; + char cmd[BUFSIZ]; setbuf(stdout, NULL); setbuf(stderr, NULL); @@ -556,10 +556,10 @@ init_macro(); init_sema(); - tmpdir = tempnam(NULL, "vtc"); - AN(tmpdir); - mkdir(tmpdir, 0700); - macro_def(vl, NULL, "tmpdir", tmpdir); + vtc_tmpdir = tempnam(NULL, "vtc"); + AN(vtc_tmpdir); + AZ(mkdir(vtc_tmpdir, 0700)); + macro_def(vl, NULL, "tmpdir", vtc_tmpdir); vtc_thread = pthread_self(); tmax = 0; nmax = NULL; @@ -579,12 +579,12 @@ break; } - /* XXX this will always remove the tmpdir even on failures. - * Maybe we should keep it in that case? */ - assert(asprintf(&cmd, "rm -rf %s", tmpdir) > 0); - AZ(system(cmd)); - free(tmpdir); - free(cmd); + /* Remove tmpdir on success or non-verbosity */ + if (vtc_error == 0 || vtc_verbosity == 0) { + bprintf(cmd, "rm -rf %s", vtc_tmpdir); + AZ(system(cmd)); + free(vtc_tmpdir); + } if (vtc_error) return (2); Modified: trunk/varnish-cache/bin/varnishtest/vtc.h =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-25 20:04:24 UTC (rev 4486) +++ trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-25 20:46:49 UTC (rev 4487) @@ -60,6 +60,7 @@ extern int vtc_error; /* Error, bail out */ extern int vtc_stop; /* Abandon current test, no error */ extern pthread_t vtc_thread; +extern char *vtc_tmpdir; void init_sema(void); @@ -78,4 +79,4 @@ void macro_def(struct vtclog *vl, const char *instance, const char *name, const char *fmt, ...); -struct vsb *macro_expand(char *name); +struct vsb *macro_expand(const char *name); Modified: trunk/varnish-cache/bin/varnishtest/vtc_varnish.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-25 20:04:24 UTC (rev 4486) +++ trunk/varnish-cache/bin/varnishtest/vtc_varnish.c 2010-01-25 20:46:49 UTC (rev 4487) @@ -46,7 +46,6 @@ #include #include -#include "compat/asprintf.h" #include "vqueue.h" #include "miniobj.h" #include "libvarnish.h" @@ -144,22 +143,19 @@ varnish_new(const char *name) { struct varnish *v; - char *c; + char buf[1024]; AN(name); ALLOC_OBJ(v, VARNISH_MAGIC); AN(v); REPLACE(v->name, name); - if (getuid() == 0) - assert(asprintf(&v->workdir, "/tmp/__%s", name) >= 0); - else - assert(asprintf(&v->workdir, "/tmp/__%s.%d", name, getuid()) >= 0); + bprintf(buf, "%s/%s", vtc_tmpdir, name); + v->workdir = strdup(buf); AN(v->workdir); - assert(asprintf(&c, "rm -rf %s ; mkdir -p %s", v->workdir, v->workdir) >= 0); - AN(c); - AZ(system(c)); + bprintf(buf, "rm -rf %s ; mkdir -p %s", v->workdir, v->workdir); + AZ(system(buf)); v->vl = vtc_logopen(name); AN(v->vl); From phk at projects.linpro.no Mon Jan 25 21:07:23 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 22:07:23 +0100 (CET) Subject: r4488 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100125210723.329951F7E37@projects.linpro.no> Author: phk Date: 2010-01-25 22:07:22 +0100 (Mon, 25 Jan 2010) New Revision: 4488 Modified: trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc Log: Reformat so that all magic port numbers show their nature in a grep Modified: trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc 2010-01-25 20:46:49 UTC (rev 4487) +++ trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc 2010-01-25 21:07:22 UTC (rev 4488) @@ -6,8 +6,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; - .port = "9080"; + .host = "127.0.0.2"; .port = "9080"; } } -start Modified: trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc 2010-01-25 20:46:49 UTC (rev 4487) +++ trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc 2010-01-25 21:07:22 UTC (rev 4488) @@ -4,8 +4,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; - .port = "9080"; + .host = "127.0.0.2"; .port = "9080"; } sub vcl_recv { @@ -27,8 +26,7 @@ director bar random { { .backend = { - .host = "127.0.0.2"; - .port = "9080"; + .host = "127.0.0.2"; .port = "9080"; } .weight = 1; } @@ -52,8 +50,7 @@ varnish v1 -vcl { director baz round-robin { { .backend = { - .host = "127.0.0.2"; - .port = "9080"; + .host = "127.0.0.2"; .port = "9080"; } } } Modified: trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc 2010-01-25 20:46:49 UTC (rev 4487) +++ trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc 2010-01-25 21:07:22 UTC (rev 4488) @@ -4,8 +4,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; - .port = "9080"; + .host = "127.0.0.2"; .port = "9080"; } sub vcl_recv { error 888; Modified: trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc 2010-01-25 20:46:49 UTC (rev 4487) +++ trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc 2010-01-25 21:07:22 UTC (rev 4488) @@ -11,9 +11,9 @@ # 198.18.1.1 is IANA-reserved for Network Interconnect Device # Benchmark Testing -varnish v1 -cliok "param.set listen_address 198.18.1.1:9082" +varnish v1 -cliok "param.set listen_address 127.0.0.2:0" varnish v1 -vcl+backend {} -clierr 300 start -varnish v1 -cliok "param.set listen_address 127.0.0.1:9081,127.0.0.2:9082" +varnish v1 -cliok "param.set listen_address 127.0.0.1:0,127.0.0.2:9082" varnish v1 -start client c1 { Modified: trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc 2010-01-25 20:46:49 UTC (rev 4487) +++ trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc 2010-01-25 21:07:22 UTC (rev 4488) @@ -4,8 +4,7 @@ varnish v1 -vcl { backend bad { - .host = "127.0.0.1"; - .port = "9090"; + .host = "127.0.0.2"; .port = "9090"; } sub vcl_recv { From phk at projects.linpro.no Mon Jan 25 21:15:49 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 22:15:49 +0100 (CET) Subject: r4489 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100125211549.1DBC81F7E37@projects.linpro.no> Author: phk Date: 2010-01-25 22:15:48 +0100 (Mon, 25 Jan 2010) New Revision: 4489 Modified: trunk/varnish-cache/bin/varnishtest/tests/v00010.vtc Log: Use the nice macros we have, to make even this test use randomly selected port numbers. I belive varnishtest is now entirely without magic TCP port numbers and multiple instances should be runable without collision. Modified: trunk/varnish-cache/bin/varnishtest/tests/v00010.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00010.vtc 2010-01-25 21:07:22 UTC (rev 4488) +++ trunk/varnish-cache/bin/varnishtest/tests/v00010.vtc 2010-01-25 21:15:48 UTC (rev 4489) @@ -24,15 +24,12 @@ varnish v1 -cliok "param.set diag_bitmap 0x00001000" -# We have to force a known listen address otherwise the client cannot find -# the varnish after the restart. +# Force the (random) port selected to be used again after restart. +varnish v1 -cliok "param.set listen_address ${v1_sock}" -varnish v1 -cliok "param.set listen_address 127.0.0.1:9080" -stop -start - # varnishtest defaults to auto_restart off, to avoid masking bugs. varnish v1 -cliok "param.set auto_restart on" - client c1 { txreq -url "/" rxresp From phk at projects.linpro.no Mon Jan 25 22:05:37 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 23:05:37 +0100 (CET) Subject: r4490 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125220537.76C5B1F7E3D@projects.linpro.no> Author: phk Date: 2010-01-25 23:05:37 +0100 (Mon, 25 Jan 2010) New Revision: 4490 Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c Log: Add a couple of asserts to be even more paranoid Modified: trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 21:15:48 UTC (rev 4489) +++ trunk/varnish-cache/bin/varnishd/cache_waiter_poll.c 2010-01-25 22:05:37 UTC (rev 4490) @@ -88,6 +88,7 @@ assert(fd < npoll); if (hpoll < fd) hpoll = fd; + assert(pollfd[fd].fd == -1); pollfd[fd].fd = fd; pollfd[fd].events = POLLIN; } @@ -99,6 +100,7 @@ assert(fd < npoll); assert(fd >= 0); vca_pollspace((unsigned)fd); + assert(pollfd[fd].fd == fd); pollfd[fd].fd = -1; pollfd[fd].events = 0; } From phk at projects.linpro.no Mon Jan 25 22:23:31 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 23:23:31 +0100 (CET) Subject: r4491 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125222331.61F061F7E3A@projects.linpro.no> Author: phk Date: 2010-01-25 23:23:31 +0100 (Mon, 25 Jan 2010) New Revision: 4491 Modified: trunk/varnish-cache/bin/varnishd/cache_hash.c Log: New objcore's didn't get a pointer set to their objhead. Add a couple of asserts to check that they all do now. Modified: trunk/varnish-cache/bin/varnishd/cache_hash.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-25 22:05:37 UTC (rev 4490) +++ trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-25 22:23:31 UTC (rev 4491) @@ -348,6 +348,7 @@ VTAILQ_FOREACH(oc, &oh->objcs, list) { /* Must be at least our own ref + the objcore we examine */ assert(oh->refcnt > 1); + assert(oc->objhead == oh); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); if (oc->flags & OC_F_PERSISTENT) @@ -405,6 +406,7 @@ if (oc != NULL) { o = oc->obj; CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); + assert(oc->objhead == oh); /* We found an object we like */ o->refcnt++; @@ -438,6 +440,7 @@ /* XXX: Should this not be ..._HEAD now ? */ VTAILQ_INSERT_TAIL(&oh->objcs, oc, list); + oc->objhead = oh; /* NB: do not deref objhead the new object inherits our reference */ Lck_Unlock(&oh->mtx); *poh = oh; From phk at projects.linpro.no Mon Jan 25 22:48:43 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Mon, 25 Jan 2010 23:48:43 +0100 (CET) Subject: r4492 - trunk/varnish-cache/bin/varnishd Message-ID: <20100125224843.6DE031F7E36@projects.linpro.no> Author: phk Date: 2010-01-25 23:48:42 +0100 (Mon, 25 Jan 2010) New Revision: 4492 Modified: trunk/varnish-cache/bin/varnishd/cache_expire.c trunk/varnish-cache/bin/varnishd/cache_hash.c trunk/varnish-cache/bin/varnishd/storage_persistent.c Log: Be even more paranoid about oc->objhead pointers. Modified: trunk/varnish-cache/bin/varnishd/cache_expire.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_expire.c 2010-01-25 22:23:31 UTC (rev 4491) +++ trunk/varnish-cache/bin/varnishd/cache_expire.c 2010-01-25 22:48:42 UTC (rev 4492) @@ -306,6 +306,8 @@ sp->objhead = oc->objhead; sp->objcore = oc; HSH_DerefObjCore(sp); + AZ(sp->objcore); + AZ(sp->objhead); sp->wrk->stats.n_vampireobject--; } } Modified: trunk/varnish-cache/bin/varnishd/cache_hash.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-25 22:23:31 UTC (rev 4491) +++ trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-25 22:48:42 UTC (rev 4492) @@ -547,14 +547,15 @@ sp->objcore = NULL; Lck_Lock(&oh->mtx); + assert(oc->objhead == oh); VTAILQ_REMOVE(&oh->objcs, oc, list); Lck_Unlock(&oh->mtx); + oc->objhead = NULL; assert(oh->refcnt > 0); FREE_OBJ(oc); sp->wrk->stats.n_objectcore--; - if (hash->deref(oh)) - return; - HSH_DeleteObjHead(sp->wrk, oh); + if (!hash->deref(oh)) + HSH_DeleteObjHead(sp->wrk, oh); } /******************************************************************* Modified: trunk/varnish-cache/bin/varnishd/storage_persistent.c =================================================================== --- trunk/varnish-cache/bin/varnishd/storage_persistent.c 2010-01-25 22:23:31 UTC (rev 4491) +++ trunk/varnish-cache/bin/varnishd/storage_persistent.c 2010-01-25 22:48:42 UTC (rev 4492) @@ -674,7 +674,6 @@ /* refcnt is one because the object is in the hash */ oc->obj->refcnt = 1; oc->obj->objcore = oc; - oc->objhead = oh; oc->obj->ban = oc->ban; sg->nfixed++; From phk at projects.linpro.no Tue Jan 26 21:58:31 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 26 Jan 2010 22:58:31 +0100 (CET) Subject: r4493 - trunk/varnish-cache/bin/varnishd Message-ID: <20100126215831.6B3131F7C4E@projects.linpro.no> Author: phk Date: 2010-01-26 22:58:30 +0100 (Tue, 26 Jan 2010) New Revision: 4493 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/bin/varnishd/cache_pool.c trunk/varnish-cache/bin/varnishd/cache_session.c Log: Go over the session structure memory management code. Always zap struct sess to zero before we start using it, but do so in the recycling workerthread if we can get away with it, to reduce the amount of work done in the acceptor. Let worker threads pick up some of the pool-expansion work by pre-creating sessmem structures, to offload this from the acceptor thread if possible. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-25 22:48:42 UTC (rev 4492) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-26 21:58:30 UTC (rev 4493) @@ -427,7 +427,6 @@ #endif }; - /* -------------------------------------------------------------------*/ /* Backend connection */ @@ -609,8 +608,8 @@ /* cache_session.c [SES] */ void SES_Init(void); -struct sess *SES_New(const struct sockaddr *addr, unsigned len); -struct sess *SES_Alloc(const struct sockaddr *addr, unsigned len); +struct sess *SES_New(void); +struct sess *SES_Alloc(void); void SES_Delete(struct sess *sp); void SES_Charge(struct sess *sp); void SES_ResetBackendTimeouts(struct sess *sp); Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-25 22:48:42 UTC (rev 4492) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-26 21:58:30 UTC (rev 4493) @@ -267,7 +267,7 @@ } continue; } - sp = SES_New(addr, l); + sp = SES_New(); if (sp == NULL) { AZ(close(i)); VSL_stats->client_drop++; @@ -278,6 +278,9 @@ sp->t_open = now; sp->t_end = now; sp->mylsock = ls; + assert(l < sp->sockaddrlen); + memcpy(sp->sockaddr, addr, l); + sp->sockaddrlen = l; sp->step = STP_FIRST; WRK_QueueSession(sp); Modified: trunk/varnish-cache/bin/varnishd/cache_pool.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-25 22:48:42 UTC (rev 4492) +++ trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-26 21:58:30 UTC (rev 4493) @@ -549,7 +549,7 @@ CAST_OBJ_NOTNULL(bt, arg, BGTHREAD_MAGIC); THR_SetName(bt->name); - sp = SES_Alloc(NULL, 0); + sp = SES_Alloc(); XXXAN(sp); memset(&ww, 0, sizeof ww); sp->wrk = &ww; Modified: trunk/varnish-cache/bin/varnishd/cache_session.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-25 22:48:42 UTC (rev 4492) +++ trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-26 21:58:30 UTC (rev 4493) @@ -94,50 +94,75 @@ #undef ACCT } -/*--------------------------------------------------------------------*/ +/*-------------------------------------------------------------------- + * This function allocates a session + assorted peripheral data + * structures in one single malloc operation. + */ -static struct sess * -ses_setup(struct sessmem *sm, const struct sockaddr *addr, unsigned len) +static struct sessmem * +ses_sm_alloc(void) { - struct sess *sp; - unsigned char *p; + struct sessmem *sm; + unsigned char *p, *q; volatile unsigned nws; volatile unsigned nhttp; unsigned l, hl; - if (sm == NULL) { - if (VSL_stats->n_sess_mem >= params->max_sess) - return (NULL); - /* - * It is not necessary to lock these, but we need to - * cache them locally, to make sure we get a consistent - * view of the value. - */ - nws = params->sess_workspace; - nhttp = params->http_headers; - hl = HTTP_estimate(nhttp); - l = sizeof *sm + nws + 2 * hl; - p = malloc(l); - if (p == NULL) - return (NULL); + if (VSL_stats->n_sess_mem >= params->max_sess) + return (NULL); + /* + * It is not necessary to lock these, but we need to + * cache them locally, to make sure we get a consistent + * view of the value. + */ + nws = params->sess_workspace; + nhttp = params->http_headers; + hl = HTTP_estimate(nhttp); + l = sizeof *sm + nws + 2 * hl; + p = malloc(l); + if (p == NULL) + return (NULL); + q = p + l; - /* Don't waste time zeroing the workspace */ - memset(p, 0, l - nws); + Lck_Lock(&stat_mtx); + VSL_stats->n_sess_mem++; + Lck_Unlock(&stat_mtx); - sm = (void*)p; - p += sizeof *sm; - sm->magic = SESSMEM_MAGIC; - sm->workspace = nws; - VSL_stats->n_sess_mem++; - sm->http[0] = HTTP_create(p, nhttp); - p += hl; - sm->http[1] = HTTP_create(p, nhttp); - p += hl; - sm->wsp = p; - } + /* Don't waste time zeroing the workspace */ + memset(p, 0, l - nws); + + sm = (void*)p; + p += sizeof *sm; + sm->magic = SESSMEM_MAGIC; + sm->workspace = nws; + sm->http[0] = HTTP_create(p, nhttp); + p += hl; + sm->http[1] = HTTP_create(p, nhttp); + p += hl; + sm->wsp = p; + p += nws; + assert(p == q); + + return (sm); +} + +/*-------------------------------------------------------------------- + * This prepares a session for use, based on its sessmem structure. + */ + +static void +ses_setup(struct sessmem *sm) +{ + struct sess *sp; + + CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC); - VSL_stats->n_sess++; sp = &sm->sess; + memset(sp, 0, sizeof *sp); + + /* We assume that the sess has been zeroed by the time we get here */ + AZ(sp->magic); + sp->magic = SESS_MAGIC; sp->mem = sm; sp->sockaddr = (void*)(&sm->sockaddr[0]); @@ -150,31 +175,23 @@ sp->t_resp = NAN; sp->t_end = NAN; sp->grace = NAN; - sp->disable_esi = 0; - assert(len <= sp->sockaddrlen); - if (addr != NULL) { - memcpy(sp->sockaddr, addr, len); - sp->sockaddrlen = len; - } - WS_Init(sp->ws, "sess", sm->wsp, sm->workspace); sp->http = sm->http[0]; sp->http0 = sm->http[1]; SES_ResetBackendTimeouts(sp); - - return (sp); } /*-------------------------------------------------------------------- - * Try to recycle an existing session. + * Get a new session, preferably by recycling an already ready one */ struct sess * -SES_New(const struct sockaddr *addr, unsigned len) +SES_New(void) { struct sessmem *sm; + struct sess *sp; assert(pthread_self() == VCA_thread); assert(ses_qp <= 1); @@ -189,27 +206,52 @@ Lck_Unlock(&ses_mem_mtx); sm = VTAILQ_FIRST(&ses_free_mem[ses_qp]); } - if (sm != NULL) + if (sm != NULL) { VTAILQ_REMOVE(&ses_free_mem[ses_qp], sm, list); - return (ses_setup(sm, addr, len)); + sp = &sm->sess; + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + } else { + sm = ses_sm_alloc(); + if (sm == NULL) + return (NULL); + ses_setup(sm); + sp = &sm->sess; + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + } + + VSL_stats->n_sess++; /* XXX: locking ? */ + + return (sp); } -/*--------------------------------------------------------------------*/ +/*-------------------------------------------------------------------- + * Allocate a session for use by background threads. + */ struct sess * -SES_Alloc(const struct sockaddr *addr, unsigned len) +SES_Alloc(void) { - return (ses_setup(NULL, addr, len)); + struct sess *sp; + struct sessmem *sm; + + sm = ses_sm_alloc(); + AN(sm); + ses_setup(sm); + sp = &sm->sess; + sp->sockaddrlen = 0; + return (sp); } -/*--------------------------------------------------------------------*/ +/*-------------------------------------------------------------------- + * Recycle a session. If the workspace has changed, deleted it, + * otherwise wash it, and put it up for adoption. + */ void SES_Delete(struct sess *sp) { struct acct *b = &sp->acct; struct sessmem *sm; - // unsigned workspace; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); sm = sp->mem; @@ -217,7 +259,7 @@ AZ(sp->obj); AZ(sp->vcl); - VSL_stats->n_sess--; + VSL_stats->n_sess--; /* XXX: locking ? */ assert(!isnan(b->first)); assert(!isnan(sp->t_end)); VSL(SLT_StatSess, sp->id, "%s %s %.0f %ju %ju %ju %ju %ju %ju %ju", @@ -225,28 +267,28 @@ b->sess, b->req, b->pipe, b->pass, b->fetch, b->hdrbytes, b->bodybytes); if (sm->workspace != params->sess_workspace) { + Lck_Lock(&stat_mtx); VSL_stats->n_sess_mem--; + Lck_Unlock(&stat_mtx); free(sm); } else { /* Clean and prepare for reuse */ - // workspace = sm->workspace; - // memset(sm, 0, sizeof *sm); - // sm->magic = SESSMEM_MAGIC; - // sm->workspace = workspace; - - /* XXX: cache_waiter_epoll.c evaluates data.ptr. If it's - * XXX: not nulled, things go wrong during load. - * XXX: Should probably find a better method to deal with - * XXX: this scenario... - */ -#if defined(HAVE_EPOLL_CTL) - sm->sess.ev.data.ptr = NULL; -#endif - + ses_setup(sm); Lck_Lock(&ses_mem_mtx); VTAILQ_INSERT_HEAD(&ses_free_mem[1 - ses_qp], sm, list); Lck_Unlock(&ses_mem_mtx); } + + /* Try to precreate some ses-mem so the acceptor will not have to */ + if (VSL_stats->n_sess_mem < VSL_stats->n_sess + 10) { + sm = ses_sm_alloc(); + if (sm != NULL) { + ses_setup(sm); + Lck_Lock(&ses_mem_mtx); + VTAILQ_INSERT_HEAD(&ses_free_mem[1 - ses_qp], sm, list); + Lck_Unlock(&ses_mem_mtx); + } + } } /*--------------------------------------------------------------------*/ @@ -259,6 +301,8 @@ Lck_New(&ses_mem_mtx); } +/* XXX: We should use NAN as default marker */ + void SES_ResetBackendTimeouts(struct sess *sp) { From phk at projects.linpro.no Tue Jan 26 22:56:39 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 26 Jan 2010 23:56:39 +0100 (CET) Subject: r4494 - trunk/varnish-cache/bin/varnishd Message-ID: <20100126225639.94DA61F7C92@projects.linpro.no> Author: phk Date: 2010-01-26 23:56:39 +0100 (Tue, 26 Jan 2010) New Revision: 4494 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_backend.c trunk/varnish-cache/bin/varnishd/cache_center.c trunk/varnish-cache/bin/varnishd/cache_fetch.c trunk/varnish-cache/bin/varnishd/cache_session.c trunk/varnish-cache/bin/varnishd/cache_vrt.c Log: Move the backend timeouts out of struct sess, they don't belong there. Put the fields VCL tweaks in the worker, and figure out the real numbers when we connect/have connected to a backend, and but the numbers in the vbe_conn. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-26 22:56:39 UTC (rev 4494) @@ -233,6 +233,12 @@ double ttl; double grace; unsigned do_esi; + + /* Timeouts */ + double connect_timeout; + double first_byte_timeout; + double between_bytes_timeout; + }; /* Work Request for worker thread ------------------------------------*/ @@ -390,11 +396,6 @@ double t_resp; double t_end; - /* Timeouts */ - double connect_timeout; - double first_byte_timeout; - double between_bytes_timeout; - /* Acceptable grace period */ double grace; @@ -436,6 +437,10 @@ VTAILQ_ENTRY(vbe_conn) list; struct backend *backend; int fd; + + /* Timeouts */ + double first_byte_timeout; + double between_bytes_timeout; }; /* Prototypes etc ----------------------------------------------------*/ @@ -612,8 +617,6 @@ struct sess *SES_Alloc(void); void SES_Delete(struct sess *sp); void SES_Charge(struct sess *sp); -void SES_ResetBackendTimeouts(struct sess *sp); -void SES_InheritBackendTimeouts(struct sess *sp); /* cache_shmlog.c */ void VSL_Init(void); Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-26 22:56:39 UTC (rev 4494) @@ -91,6 +91,17 @@ } } +#define FIND_TMO(tmx, dst, sp, be) \ + do { \ + dst = sp->wrk->tmx; \ + if (dst == 0.0) \ + dst = be->tmx; \ + if (dst == 0.0) \ + dst = params->tmx; \ + assert(dst > 0.0); \ + } while (0) + + /*-------------------------------------------------------------------- * Attempt to connect to a given addrinfo entry. * @@ -105,6 +116,7 @@ socklen_t salen, const struct backend *bp) { int s, i, tmo; + double tmod; char abuf1[TCP_ADDRBUFSIZE], abuf2[TCP_ADDRBUFSIZE]; char pbuf1[TCP_PORTBUFSIZE], pbuf2[TCP_PORTBUFSIZE]; @@ -114,10 +126,10 @@ if (s < 0) return (s); - tmo = (int)(sp->connect_timeout * 1000); - if (bp->connect_timeout > 10e-3) - tmo = (int)(bp->connect_timeout * 1000); + FIND_TMO(connect_timeout, tmod, sp, bp); + tmo = (int)(tmod * 1000.0); + if (tmo > 0) i = TCP_connect(s, sa, salen, tmo); else @@ -412,7 +424,7 @@ return (d->getfd(d, sp)); } -/* Cheack health -----------------------------------------------------*/ +/* Check health ------------------------------------------------------*/ int VBE_Healthy(const struct director *d, const struct sess *sp) @@ -442,11 +454,19 @@ vdi_simple_getfd(const struct director *d, struct sess *sp) { struct vdi_simple *vs; + struct vbe_conn *vc; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC); - return (vbe_GetVbe(sp, vs->backend)); + vc = vbe_GetVbe(sp, vs->backend); + if (vc != NULL) { + FIND_TMO(first_byte_timeout, + vc->first_byte_timeout, sp, vc->backend); + FIND_TMO(between_bytes_timeout, + vc->between_bytes_timeout, sp, vc->backend); + } + return (vc); } static unsigned Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2010-01-26 22:56:39 UTC (rev 4494) @@ -864,6 +864,9 @@ sp->wrk->bereq = sp->wrk->http[0]; http_Setup(sp->wrk->bereq, sp->wrk->ws); http_FilterHeader(sp, HTTPH_R_FETCH); + sp->wrk->connect_timeout = 0; + sp->wrk->first_byte_timeout = 0; + sp->wrk->between_bytes_timeout = 0; VCL_miss_method(sp); switch(sp->handling) { case VCL_RET_ERROR: @@ -930,6 +933,9 @@ http_Setup(sp->wrk->bereq, sp->wrk->ws); http_FilterHeader(sp, HTTPH_R_PASS); + sp->wrk->connect_timeout = 0; + sp->wrk->first_byte_timeout = 0; + sp->wrk->between_bytes_timeout = 0; VCL_pass_method(sp); if (sp->handling == VCL_RET_ERROR) { sp->step = STP_ERROR; @@ -1020,8 +1026,6 @@ CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC); AZ(sp->obj); - SES_ResetBackendTimeouts(sp); - /* By default we use the first backend */ AZ(sp->director); sp->director = sp->vcl->director[0]; Modified: trunk/varnish-cache/bin/varnishd/cache_fetch.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_fetch.c 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache_fetch.c 2010-01-26 22:56:39 UTC (rev 4494) @@ -335,6 +335,7 @@ char *b; struct http *hp; int i; + double tmo; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC); @@ -357,8 +358,6 @@ return (__LINE__); } vc = sp->vbe; - /* Inherit the backend timeouts from the selected backend */ - SES_InheritBackendTimeouts(sp); /* * Now that we know our backend, we can set a default Host: @@ -390,7 +389,10 @@ /* Receive response */ HTC_Init(sp->wrk->htc, sp->wrk->ws, vc->fd); - TCP_set_read_timeout(vc->fd, sp->first_byte_timeout); + + TCP_set_read_timeout(vc->fd, vc->first_byte_timeout); + tmo = vc->first_byte_timeout; + do { i = HTC_Rx(sp->wrk->htc); if (i < 0) { @@ -400,10 +402,13 @@ /* XXX: other cleanup ? */ return (__LINE__); } - TCP_set_read_timeout(vc->fd, sp->between_bytes_timeout); - } - while (i == 0); + if (vc->between_bytes_timeout != tmo) { + TCP_set_read_timeout(vc->fd, vc->between_bytes_timeout); + tmo = vc->between_bytes_timeout; + } + } while (i == 0); + hp = sp->wrk->beresp; if (http_DissectResponse(sp->wrk, sp->wrk->htc, hp)) { Modified: trunk/varnish-cache/bin/varnishd/cache_session.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache_session.c 2010-01-26 22:56:39 UTC (rev 4494) @@ -179,8 +179,6 @@ WS_Init(sp->ws, "sess", sm->wsp, sm->workspace); sp->http = sm->http[0]; sp->http0 = sm->http[1]; - - SES_ResetBackendTimeouts(sp); } /*-------------------------------------------------------------------- @@ -300,39 +298,3 @@ Lck_New(&stat_mtx); Lck_New(&ses_mem_mtx); } - -/* XXX: We should use NAN as default marker */ - -void -SES_ResetBackendTimeouts(struct sess *sp) -{ - sp->connect_timeout = params->connect_timeout; - sp->first_byte_timeout = params->first_byte_timeout; - sp->between_bytes_timeout = params->between_bytes_timeout; -} - -void -SES_InheritBackendTimeouts(struct sess *sp) -{ - struct backend *be; - - AN(sp); - AN(sp->vbe); - AN(sp->vbe->backend); - - be = sp->vbe->backend; - /* - * We only inherit the backend's timeout if the session timeout - * has not already been set in the VCL, as the order of precedence - * is parameter < backend definition < VCL. - */ - if (be->connect_timeout > 1e-3 && - sp->connect_timeout == params->connect_timeout) - sp->connect_timeout = be->connect_timeout; - if (be->first_byte_timeout > 1e-3 && - sp->first_byte_timeout == params->first_byte_timeout) - sp->first_byte_timeout = be->first_byte_timeout; - if (be->between_bytes_timeout > 1e-3 && - sp->between_bytes_timeout == params->between_bytes_timeout) - sp->between_bytes_timeout = be->between_bytes_timeout; -} Modified: trunk/varnish-cache/bin/varnishd/cache_vrt.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt.c 2010-01-26 21:58:30 UTC (rev 4493) +++ trunk/varnish-cache/bin/varnishd/cache_vrt.c 2010-01-26 22:56:39 UTC (rev 4494) @@ -453,42 +453,42 @@ VRT_l_bereq_connect_timeout(struct sess *sp, double num) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - sp->connect_timeout = (num > 0 ? num : 0); + sp->wrk->connect_timeout = (num > 0 ? num : 0); } double VRT_r_bereq_connect_timeout(struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - return sp->connect_timeout; + return (sp->wrk->connect_timeout); } void VRT_l_bereq_first_byte_timeout(struct sess *sp, double num) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - sp->first_byte_timeout = (num > 0 ? num : 0); + sp->wrk->first_byte_timeout = (num > 0 ? num : 0); } double VRT_r_bereq_first_byte_timeout(struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - return sp->first_byte_timeout; + return (sp->wrk->first_byte_timeout); } void VRT_l_bereq_between_bytes_timeout(struct sess *sp, double num) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - sp->between_bytes_timeout = (num > 0 ? num : 0); + sp->wrk->between_bytes_timeout = (num > 0 ? num : 0); } double VRT_r_bereq_between_bytes_timeout(struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - return sp->between_bytes_timeout; + return (sp->wrk->between_bytes_timeout); } /*--------------------------------------------------------------------*/ From phk at projects.linpro.no Wed Jan 27 15:14:51 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 27 Jan 2010 16:14:51 +0100 (CET) Subject: r4495 - trunk/varnish-cache/bin/varnishd Message-ID: <20100127151451.876E41F7C89@projects.linpro.no> Author: phk Date: 2010-01-27 16:14:50 +0100 (Wed, 27 Jan 2010) New Revision: 4495 Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c trunk/varnish-cache/bin/varnishd/mgt_pool.c trunk/varnish-cache/bin/varnishd/vparam.h Log: Make the min/mix param fields double Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-26 22:56:39 UTC (rev 4494) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-27 15:14:50 UTC (rev 4495) @@ -92,46 +92,79 @@ cli_out(cli, "%u", *dst); } +/*--------------------------------------------------------------------*/ + +void +tweak_timeout(struct cli *cli, const struct parspec *par, const char *arg) +{ + volatile unsigned *dest; + + dest = par->priv; + tweak_generic_timeout(cli, dest, arg); +} + static void -tweak_generic_timeout_double(struct cli *cli, volatile double *dst, +tweak_timeout_double(struct cli *cli, const struct parspec *par, const char *arg) { + volatile double *dest; double u; + dest = par->priv; if (arg != NULL) { u = strtod(arg, NULL); - if (u < 0) { + if (u < par->min) { cli_out(cli, - "Timeout must be greater or equal to zero\n"); + "Timeout must be greater or equal to %.g\n", + par->min); cli_result(cli, CLIS_PARAM); return; } - *dst = u; + if (u > par->max) { + cli_out(cli, + "Timeout must be less than or equal to %.g\n", + par->max); + cli_result(cli, CLIS_PARAM); + return; + } + *dest = u; } else - cli_out(cli, "%f", *dst); + cli_out(cli, "%.6f", *dest); } - +#if 0 /*--------------------------------------------------------------------*/ -void -tweak_timeout(struct cli *cli, const struct parspec *par, const char *arg) -{ - volatile unsigned *dest; - - dest = par->priv; - tweak_generic_timeout(cli, dest, arg); -} - static void -tweak_timeout_double(struct cli *cli, const struct parspec *par, +tweak_generic_double(struct cli *cli, const struct parspec *par, const char *arg) { volatile double *dest; + double u; dest = par->priv; - tweak_generic_timeout_double(cli, dest, arg); + if (arg != NULL) { + u = strtod(arg, NULL); + if (u < par->min) { + cli_out(cli, + "Must be greater or equal to %.g\n", + par->min); + cli_result(cli, CLIS_PARAM); + return; + } + if (u > par->max) { + cli_out(cli, + "Must be less than or equal to %.g\n", + par->max); + cli_result(cli, CLIS_PARAM); + return; + } + *dest = u; + } else + cli_out(cli, "%f", *dest); } +#endif + /*--------------------------------------------------------------------*/ static void @@ -213,7 +246,7 @@ volatile unsigned *dest; dest = par->priv; - tweak_generic_uint(cli, dest, arg, par->umin, par->umax); + tweak_generic_uint(cli, dest, arg, (uint)par->min, (uint)par->max); } /*-------------------------------------------------------------------- @@ -551,7 +584,7 @@ 0, "on", "bool" }, { "fetch_chunksize", - tweak_uint, &master.fetch_chunksize, 4, UINT_MAX / 1024, + tweak_uint, &master.fetch_chunksize, 4, UINT_MAX / 1024., "The default chunksize used by fetcher. " "This should be bigger than the majority of objects with " "short TTLs.\n" Modified: trunk/varnish-cache/bin/varnishd/mgt_pool.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_pool.c 2010-01-26 22:56:39 UTC (rev 4494) +++ trunk/varnish-cache/bin/varnishd/mgt_pool.c 2010-01-27 15:14:50 UTC (rev 4495) @@ -61,7 +61,7 @@ { tweak_generic_uint(cli, &master.wthread_min, arg, - par->umin, master.wthread_max); + (unsigned)par->min, master.wthread_max); } /*--------------------------------------------------------------------*/ Modified: trunk/varnish-cache/bin/varnishd/vparam.h =================================================================== --- trunk/varnish-cache/bin/varnishd/vparam.h 2010-01-26 22:56:39 UTC (rev 4494) +++ trunk/varnish-cache/bin/varnishd/vparam.h 2010-01-27 15:14:50 UTC (rev 4495) @@ -37,8 +37,8 @@ const char *name; tweak_t *func; volatile void *priv; - unsigned umin; - unsigned umax; + double min; + double max; const char *descr; int flags; #define DELAYED_EFFECT 1 From phk at projects.linpro.no Wed Jan 27 15:59:30 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 27 Jan 2010 16:59:30 +0100 (CET) Subject: r4496 - trunk/varnish-cache/bin/varnishd Message-ID: <20100127155930.E03951F7E42@projects.linpro.no> Author: phk Date: 2010-01-27 16:59:30 +0100 (Wed, 27 Jan 2010) New Revision: 4496 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_pool.c Log: Give WRK_QueueSession() a return value that tells if the session as indeed queued or if it was dropped. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2010-01-27 15:14:50 UTC (rev 4495) +++ trunk/varnish-cache/bin/varnishd/cache.h 2010-01-27 15:59:30 UTC (rev 4496) @@ -595,7 +595,7 @@ /* cache_pool.c */ void WRK_Init(void); int WRK_Queue(struct workreq *wrq); -void WRK_QueueSession(struct sess *sp); +int WRK_QueueSession(struct sess *sp); void WRK_SumStat(struct worker *w); void WRW_Reserve(struct worker *w, int *fd); Modified: trunk/varnish-cache/bin/varnishd/cache_pool.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-27 15:14:50 UTC (rev 4495) +++ trunk/varnish-cache/bin/varnishd/cache_pool.c 2010-01-27 15:59:30 UTC (rev 4496) @@ -298,7 +298,7 @@ /*--------------------------------------------------------------------*/ -void +int WRK_QueueSession(struct sess *sp) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); @@ -306,7 +306,7 @@ sp->workreq.func = wrk_do_cnt_sess; sp->workreq.priv = sp; if (WRK_Queue(&sp->workreq) == 0) - return; + return (0); /* * Couldn't queue it -- kill it. @@ -324,6 +324,7 @@ VCL_Rel(&sp->vcl); } SES_Delete(sp); + return (1); } /*-------------------------------------------------------------------- From phk at projects.linpro.no Wed Jan 27 16:06:46 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 27 Jan 2010 17:06:46 +0100 (CET) Subject: r4497 - in trunk/varnish-cache: bin/varnishd include Message-ID: <20100127160646.C76DC1F7E34@projects.linpro.no> Author: phk Date: 2010-01-27 17:06:46 +0100 (Wed, 27 Jan 2010) New Revision: 4497 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/bin/varnishd/cache_hash.c trunk/varnish-cache/bin/varnishd/heritage.h trunk/varnish-cache/bin/varnishd/mgt_param.c trunk/varnish-cache/include/stat_field.h Log: The acceptor already has a back-off feature for when we run out of filedescriptors, generalize that concept and use it for all cases where we cannot accept and/or serve the connection: (Lack of sessions, lack of workerthreads). This is controlled by three paramters: Everytime we run into trouble, we increase the sleep-time by: acceptor_sleep_incr (0.001 s) But we never let it get above acceptor_sleep_max (0.050 s) Once we manage to accept and schedule a connection, we multiply the sleep-time by: acceptor_sleep_decay (0.9) The default numbers are more or less picked out of thin air. Two new stats counters help us keep track of this: accept_fail where accept(2) returns error. This can be out of file-descriptors, but also clients which closed while they were stuck in the accept-queue. Under normal operation, a minor trickle is probably to be expected. client_drop New connection dropped, because we could not get a session for it, or because the workerthreads were too busy. client_drop_late An previously served connection was dropped for the same reasons. As always, feedback welcome. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-27 15:59:30 UTC (rev 4496) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2010-01-27 16:06:46 UTC (rev 4497) @@ -187,7 +187,7 @@ struct pollfd *pfd; struct listen_sock *ls; unsigned u; - double now; + double now, pace; THR_SetName("cache-acceptor"); (void)arg; @@ -207,6 +207,7 @@ } need_test = 1; + pace = 0; while (1) { #ifdef SO_SNDTIMEO_WORKS if (params->send_timeout != tv_sndtimeo.tv_sec) { @@ -234,6 +235,13 @@ } } #endif + /* Bound the pacing delay by parameter */ + if (pace > params->acceptor_sleep_max) + pace = params->acceptor_sleep_max; + if (pace < params->acceptor_sleep_incr) + pace = 0.0; + if (pace > 0.0) + TIM_sleep(pace); i = poll(pfd, heritage.nsocks, 1000); now = TIM_real(); u = 0; @@ -247,6 +255,7 @@ addr = (void*)&addr_s; i = accept(ls->sock, addr, &l); if (i < 0) { + VSL_stats->accept_fail++; switch (errno) { case EAGAIN: case ECONNABORTED: @@ -255,14 +264,13 @@ VSL(SLT_Debug, ls->sock, "Too many open files " "when accept(2)ing. Sleeping."); - TIM_sleep( - params->accept_fd_holdoff * 0.001); + pace += params->acceptor_sleep_incr; break; default: VSL(SLT_Debug, ls->sock, "Accept failed: %s", strerror(errno)); - /* XXX: stats ? */ + pace += params->acceptor_sleep_incr; break; } continue; @@ -271,6 +279,7 @@ if (sp == NULL) { AZ(close(i)); VSL_stats->client_drop++; + pace += params->acceptor_sleep_incr; continue; } sp->fd = i; @@ -283,7 +292,12 @@ sp->sockaddrlen = l; sp->step = STP_FIRST; - WRK_QueueSession(sp); + if (WRK_QueueSession(sp)) { + VSL_stats->client_drop++; + pace += params->acceptor_sleep_incr; + } else { + pace *= params->acceptor_sleep_decay; + } } } NEEDLESS_RETURN(NULL); @@ -306,7 +320,8 @@ break; case 1: sp->step = STP_START; - WRK_QueueSession(sp); + if (WRK_QueueSession(sp)) + VSL_stats->client_drop_late++; break; default: INCOMPL(); Modified: trunk/varnish-cache/bin/varnishd/cache_hash.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-27 15:59:30 UTC (rev 4496) +++ trunk/varnish-cache/bin/varnishd/cache_hash.c 2010-01-27 16:06:46 UTC (rev 4497) @@ -463,7 +463,14 @@ AZ(sp->wrk); VTAILQ_REMOVE(&oh->waitinglist, sp, list); DSL(0x20, SLT_Debug, sp->id, "off waiting list"); - WRK_QueueSession(sp); + if (WRK_QueueSession(sp)) { + /* + * We could not schedule the session, leave the + * rest on the busy list. + */ + VSL_stats->client_drop_late++; + break; + } } } Modified: trunk/varnish-cache/bin/varnishd/heritage.h =================================================================== --- trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-27 15:59:30 UTC (rev 4496) +++ trunk/varnish-cache/bin/varnishd/heritage.h 2010-01-27 16:06:46 UTC (rev 4497) @@ -186,9 +186,10 @@ /* Acceptable clockskew with backends */ unsigned clock_skew; - /* Amount of time to sleep when running out of file - descriptors. In msecs */ - unsigned accept_fd_holdoff; + /* Acceptor pacer parameters */ + double acceptor_sleep_max; + double acceptor_sleep_incr; + double acceptor_sleep_decay; /* Get rid of duplicate purges */ unsigned purge_dups; Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-27 15:59:30 UTC (rev 4496) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2010-01-27 16:06:46 UTC (rev 4497) @@ -132,7 +132,6 @@ cli_out(cli, "%.6f", *dest); } -#if 0 /*--------------------------------------------------------------------*/ static void @@ -163,7 +162,6 @@ } else cli_out(cli, "%f", *dest); } -#endif /*--------------------------------------------------------------------*/ @@ -703,12 +701,30 @@ "and backend request. This parameter does not apply to pipe.", 0, "60", "s" }, - { "accept_fd_holdoff", tweak_timeout, - &master.accept_fd_holdoff, 0, 3600*1000, - "If we run out of file descriptors, the accept thread will " - "sleep. This parameter control for how long it will sleep.", + { "acceptor_sleep_max", tweak_timeout_double, + &master.acceptor_sleep_max, 0, 10, + "If we run out of resources, such as file descriptors or " + "worker threads, the acceptor will sleep between accepts.\n" + "This parameter limits how long it can sleep between " + "attempts to accept new connections.", EXPERIMENTAL, - "50", "ms" }, + "0.050", "s" }, + { "acceptor_sleep_incr", tweak_timeout_double, + &master.acceptor_sleep_incr, 0, 1, + "If we run out of resources, such as file descriptors or " + "worker threads, the acceptor will sleep between accepts.\n" + "This parameter control how much longer we sleep, each time " + "we fail to accept a new connection.", + EXPERIMENTAL, + "0.001", "s" }, + { "acceptor_sleep_decay", tweak_generic_double, + &master.acceptor_sleep_decay, 0, 1, + "If we run out of resources, such as file descriptors or " + "worker threads, the acceptor will sleep between accepts.\n" + "This parameter (multiplicatively) reduce the sleep duration " + "for each succesfull accept. (ie: 0.9 = reduce by 10%)", + EXPERIMENTAL, + "0.900", "" }, { "clock_skew", tweak_uint, &master.clock_skew, 0, UINT_MAX, "How much clockskew we are willing to accept between the " "backend and our own clock.", Modified: trunk/varnish-cache/include/stat_field.h =================================================================== --- trunk/varnish-cache/include/stat_field.h 2010-01-27 15:59:30 UTC (rev 4496) +++ trunk/varnish-cache/include/stat_field.h 2010-01-27 16:06:46 UTC (rev 4497) @@ -33,7 +33,7 @@ */ MAC_STAT(client_conn, uint64_t, 0, 'a', "Client connections accepted") -MAC_STAT(client_drop, uint64_t, 0, 'a', "Connection dropped, no sess") +MAC_STAT(client_drop, uint64_t, 0, 'a', "Connection dropped, no sess/wrk") MAC_STAT(client_req, uint64_t, 1, 'a', "Client requests received") MAC_STAT(cache_hit, uint64_t, 1, 'a', "Cache hits") @@ -149,3 +149,5 @@ MAC_STAT(esi_parse, uint64_t, 0, 'a', "Objects ESI parsed (unlock)") MAC_STAT(esi_errors, uint64_t, 0, 'a', "ESI parse errors (unlock)") +MAC_STAT(accept_fail, uint64_t, 0, 'a', "Accept failures") +MAC_STAT(client_drop_late, uint64_t, 0, 'a', "Connection dropped late") From phk at projects.linpro.no Wed Jan 27 21:40:10 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 27 Jan 2010 22:40:10 +0100 (CET) Subject: r4498 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100127214010.37D891F7E52@projects.linpro.no> Author: phk Date: 2010-01-27 22:40:09 +0100 (Wed, 27 Jan 2010) New Revision: 4498 Modified: trunk/varnish-cache/bin/varnishtest/vtc_server.c Log: Close a race condition in server teardown. Modified: trunk/varnish-cache/bin/varnishtest/vtc_server.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_server.c 2010-01-27 16:06:46 UTC (rev 4497) +++ trunk/varnish-cache/bin/varnishtest/vtc_server.c 2010-01-27 21:40:09 UTC (rev 4498) @@ -55,6 +55,7 @@ char *name; struct vtclog *vl; VTAILQ_ENTRY(server) list; + char run; unsigned repeat; char *spec; @@ -184,6 +185,7 @@ macro_def(s->vl, s->name, "sock", "%s:%s", s->aaddr, s->aport); } vtc_log(s->vl, 1, "Listen on %s:%s", s->addr, s->port); + s->run = 1; AZ(pthread_create(&s->tp, NULL, server_thread, s)); } @@ -205,6 +207,7 @@ s->tp = 0; TCP_close(&s->sock); s->sock = -1; + s->run = 0; } /********************************************************************** @@ -241,7 +244,7 @@ /* Reset and free */ VTAILQ_FOREACH_SAFE(s, &servers, list, s2) { VTAILQ_REMOVE(&servers, s, list); - if (s->sock >= 0) { + if (s->run) { (void)pthread_cancel(s->tp); server_wait(s); } From phk at projects.linpro.no Wed Jan 27 21:45:53 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Wed, 27 Jan 2010 22:45:53 +0100 (CET) Subject: r4499 - in trunk/varnish-cache/bin/varnishtest: . tests Message-ID: <20100127214553.3DDBA1F7E52@projects.linpro.no> Author: phk Date: 2010-01-27 22:45:52 +0100 (Wed, 27 Jan 2010) New Revision: 4499 Modified: trunk/varnish-cache/bin/varnishtest/tests/b00008.vtc trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc trunk/varnish-cache/bin/varnishtest/tests/v00002.vtc trunk/varnish-cache/bin/varnishtest/vtc.c Log: Define a macro for "bad_ip", an IP number we cannot bind to. For now use 255.255.255.254 which is about as illegal as an IP# can be for TCP purposes. Use instead of 127.0.0.2, which appearantly Linux allows you to bind to. Modified: trunk/varnish-cache/bin/varnishtest/tests/b00008.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00008.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/b00008.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -2,7 +2,7 @@ test "Test CLI help and parameter functions" -varnish v1 -arg "-b 127.0.0.2:9080" +varnish v1 -arg "-b ${bad_ip}:9080" varnish v1 -cliok "help" Modified: trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/b00015.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -6,7 +6,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; .port = "9080"; + .host = "${bad_ip}"; .port = "9080"; } } -start Modified: trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/b00016.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -4,7 +4,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; .port = "9080"; + .host = "${bad_ip}"; .port = "9080"; } sub vcl_recv { @@ -26,7 +26,7 @@ director bar random { { .backend = { - .host = "127.0.0.2"; .port = "9080"; + .host = "${bad_ip}"; .port = "9080"; } .weight = 1; } @@ -50,7 +50,7 @@ varnish v1 -vcl { director baz round-robin { { .backend = { - .host = "127.0.0.2"; .port = "9080"; + .host = "${bad_ip}"; .port = "9080"; } } } Modified: trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/b00017.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -4,7 +4,7 @@ varnish v1 -vcl { backend foo { - .host = "127.0.0.2"; .port = "9080"; + .host = "${bad_ip}"; .port = "9080"; } sub vcl_recv { error 888; Modified: trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/c00003.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -9,11 +9,9 @@ txresp -hdr "Connection: close" -body "012345\n" } -start -# 198.18.1.1 is IANA-reserved for Network Interconnect Device -# Benchmark Testing -varnish v1 -cliok "param.set listen_address 127.0.0.2:0" +varnish v1 -cliok "param.set listen_address ${bad_ip}:0" varnish v1 -vcl+backend {} -clierr 300 start -varnish v1 -cliok "param.set listen_address 127.0.0.1:0,127.0.0.2:9082" +varnish v1 -cliok "param.set listen_address 127.0.0.1:0,${bad_ip}:9082" varnish v1 -start client c1 { Modified: trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/c00028.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -4,7 +4,7 @@ varnish v1 -vcl { backend bad { - .host = "127.0.0.2"; .port = "9090"; + .host = "${bad_ip}"; .port = "9090"; } sub vcl_recv { Modified: trunk/varnish-cache/bin/varnishtest/tests/v00002.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00002.vtc 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/tests/v00002.vtc 2010-01-27 21:45:52 UTC (rev 4499) @@ -59,7 +59,7 @@ # Check backend reference by name varnish v1 -badvcl { - backend b1 { .host = "127.0.0.2"; } + backend b1 { .host = "${bad_ip}"; } director r1 random { { .weight = 1; .backend = b1; } { .weight = 1; .backend = { .host = "127.0.0.3"; } } @@ -69,18 +69,18 @@ # Check backend reference by name varnish v1 -badvcl { - backend b1 { .host = "127.0.0.2"; } + backend b1 { .host = "${bad_ip}"; } backend b2 b1; } # Check backend reference by non-C-compat name varnish v1 -badvcl { - backend b-1 { .host = "127.0.0.2"; } + backend b-1 { .host = "${bad_ip}"; } } # Check backend reference by non-C-compat name varnish v1 -badvcl { - backend b1 { .host = "127.0.0.2"; } + backend b1 { .host = "${bad_ip}"; } sub vcl_recv { set req.backend = b-1; } Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-27 21:40:09 UTC (rev 4498) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-27 21:45:52 UTC (rev 4499) @@ -561,6 +561,8 @@ AZ(mkdir(vtc_tmpdir, 0700)); macro_def(vl, NULL, "tmpdir", vtc_tmpdir); vtc_thread = pthread_self(); + + macro_def(vl, NULL, "bad_ip", "255.255.255.254"); tmax = 0; nmax = NULL; t00 = TIM_mono(); From phk at projects.linpro.no Thu Jan 28 09:32:34 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 10:32:34 +0100 (CET) Subject: r4500 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128093234.F3E0B1F7E62@projects.linpro.no> Author: phk Date: 2010-01-28 10:32:34 +0100 (Thu, 28 Jan 2010) New Revision: 4500 Modified: trunk/varnish-cache/bin/varnishtest/vtc_log.c Log: Also force data-dumps under the new log-regime Modified: trunk/varnish-cache/bin/varnishtest/vtc_log.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_log.c 2010-01-27 21:45:52 UTC (rev 4499) +++ trunk/varnish-cache/bin/varnishtest/vtc_log.c 2010-01-28 09:32:34 UTC (rev 4500) @@ -120,6 +120,17 @@ #define NLEAD (sizeof(lead)/sizeof(lead[0])) +static void +vtc_log_emit(struct vtclog *vl, unsigned lvl) +{ + AZ(pthread_mutex_lock(&vtclog_mtx)); + vsb_cat(vtclog_full, vsb_data(vl->vsb)); + AZ(pthread_mutex_unlock(&vtclog_mtx)); + + if (lvl > 0 && lvl <= vtc_verbosity) + (void)fputs(vsb_data(vl->vsb), stdout); +} + //lint -e{818} void vtc_log(struct vtclog *vl, unsigned lvl, const char *fmt, ...) @@ -138,12 +149,8 @@ vsb_finish(vl->vsb); AZ(vsb_overflowed(vl->vsb)); - AZ(pthread_mutex_lock(&vtclog_mtx)); - vsb_cat(vtclog_full, vsb_data(vl->vsb)); - AZ(pthread_mutex_unlock(&vtclog_mtx)); + vtc_log_emit(vl, lvl); - if (lvl > 0 && lvl <= vtc_verbosity) - (void)fputs(vsb_data(vl->vsb), stdout); vsb_clear(vl->vsb); AZ(pthread_mutex_unlock(&vl->mtx)); if (lvl == 0) { @@ -165,8 +172,7 @@ CHECK_OBJ_NOTNULL(vl, VTCLOG_MAGIC); assert(lvl < NLEAD); - if (lvl > vtc_verbosity) - return; + AZ(pthread_mutex_lock(&vl->mtx)); vsb_clear(vl->vsb); if (pfx == NULL) pfx = ""; @@ -196,8 +202,11 @@ vsb_printf(vl->vsb, "\n"); vsb_finish(vl->vsb); AZ(vsb_overflowed(vl->vsb)); - (void)fputs(vsb_data(vl->vsb), stdout); + + vtc_log_emit(vl, lvl); + vsb_clear(vl->vsb); + AZ(pthread_mutex_unlock(&vl->mtx)); if (lvl == 0) { vtc_error = 1; if (pthread_self() != vtc_thread) From phk at projects.linpro.no Thu Jan 28 09:33:16 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 10:33:16 +0100 (CET) Subject: r4501 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128093316.3F9371F7C89@projects.linpro.no> Author: phk Date: 2010-01-28 10:33:16 +0100 (Thu, 28 Jan 2010) New Revision: 4501 Modified: trunk/varnish-cache/bin/varnishtest/vtc_sema.c Log: Seamlessly hand over the per-sema mutex, to avoid silly context-switches Modified: trunk/varnish-cache/bin/varnishtest/vtc_sema.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_sema.c 2010-01-28 09:32:34 UTC (rev 4500) +++ trunk/varnish-cache/bin/varnishtest/vtc_sema.c 2010-01-28 09:33:16 UTC (rev 4501) @@ -93,7 +93,6 @@ CHECK_OBJ_NOTNULL(r, SEMA_MAGIC); u = strtoul(av, NULL, 0); - AZ(pthread_mutex_lock(&r->mtx)); if (r->expected == 0) r->expected = u; if (r->expected != u) @@ -111,7 +110,6 @@ r->name, r->waiters, r->expected); AZ(pthread_cond_wait(&r->cond, &r->mtx)); } - AZ(pthread_mutex_unlock(&r->mtx)); } /********************************************************************** @@ -148,8 +146,9 @@ break; if (r == NULL) r = sema_new(av[0], vl); + av++; + AZ(pthread_mutex_lock(&r->mtx)); AZ(pthread_mutex_unlock(&sema_mtx)); - av++; for (; *av != NULL; av++) { if (!strcmp(*av, "sync")) { @@ -160,6 +159,7 @@ } vtc_log(vl, 0, "Unknown sema argument: %s", *av); } + AZ(pthread_mutex_unlock(&r->mtx)); } void From phk at projects.linpro.no Thu Jan 28 09:34:05 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 10:34:05 +0100 (CET) Subject: r4502 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128093405.84F751F7C89@projects.linpro.no> Author: phk Date: 2010-01-28 10:34:05 +0100 (Thu, 28 Jan 2010) New Revision: 4502 Modified: trunk/varnish-cache/bin/varnishtest/vtc_client.c Log: Don't pointlessly announce that the default #iterations s 1 Modified: trunk/varnish-cache/bin/varnishtest/vtc_client.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_client.c 2010-01-28 09:33:16 UTC (rev 4501) +++ trunk/varnish-cache/bin/varnishtest/vtc_client.c 2010-01-28 09:34:05 UTC (rev 4502) @@ -93,7 +93,8 @@ if (c->repeat == 0) c->repeat = 1; - vtc_log(vl, 2, "Started (%u iterations)", c->repeat); + if (c->repeat != 1) + vtc_log(vl, 2, "Started (%u iterations)", c->repeat); for (u = 0; u < c->repeat; u++) { vtc_log(vl, 3, "Connect to %s", vsb_data(vsb)); fd = VSS_open(vsb_data(vsb)); From phk at projects.linpro.no Thu Jan 28 09:36:15 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 10:36:15 +0100 (CET) Subject: r4503 - in trunk/varnish-cache/bin/varnishtest: . tests Message-ID: <20100128093615.793E61F7C89@projects.linpro.no> Author: phk Date: 2010-01-28 10:36:15 +0100 (Thu, 28 Jan 2010) New Revision: 4503 Modified: trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc trunk/varnish-cache/bin/varnishtest/vtc_server.c Log: Fix a potential race when retargeting a server by doing an implicit -wait before mucking with a running server. If running on a random port, record it for subsequent -starts, so we use the same (random) port throughout. Fix the v00014 testcase which made some really bad assumptions relative to the above, eliminating the delay it used in the process. Modified: trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc 2010-01-28 09:34:05 UTC (rev 4502) +++ trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc 2010-01-28 09:36:15 UTC (rev 4503) @@ -6,7 +6,7 @@ rxreq expect req.url == "/" txresp -body "slash" -} -start -repeat 4 +} -start varnish v1 -vcl { @@ -16,7 +16,7 @@ .max_connections = 1; .probe = { .url = "/"; - .timeout = 100 ms; + .timeout = 1s; .interval = 1s; .window = 3; .threshold = 2; @@ -39,11 +39,22 @@ expect resp.status == 500 } -run -delay 1 +server s1 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start +server s1 { + rxreq + expect req.url == "/foo" + txresp -body "foobar" +} -start + client c2 { - txreq + txreq -url "/foo" rxresp + expect resp.bodylen == 6 expect resp.status == 200 } -start Modified: trunk/varnish-cache/bin/varnishtest/vtc_server.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc_server.c 2010-01-28 09:34:05 UTC (rev 4502) +++ trunk/varnish-cache/bin/varnishtest/vtc_server.c 2010-01-28 09:36:15 UTC (rev 4503) @@ -183,6 +183,9 @@ macro_def(s->vl, s->name, "addr", "%s", s->aaddr); macro_def(s->vl, s->name, "port", "%s", s->aport); macro_def(s->vl, s->name, "sock", "%s:%s", s->aaddr, s->aport); + /* Record the actual port, and reuse it on subsequent starts */ + if (!strcmp(s->port, "0")) + REPLACE(s->port, s->aport); } vtc_log(s->vl, 1, "Listen on %s:%s", s->addr, s->port); s->run = 1; @@ -243,6 +246,7 @@ if (av == NULL) { /* Reset and free */ VTAILQ_FOREACH_SAFE(s, &servers, list, s2) { + CHECK_OBJ_NOTNULL(s, SERVER_MAGIC); VTAILQ_REMOVE(&servers, s, list); if (s->run) { (void)pthread_cancel(s->tp); @@ -261,11 +265,26 @@ break; if (s == NULL) s = server_new(av[0]); + CHECK_OBJ_NOTNULL(s, SERVER_MAGIC); av++; for (; *av != NULL; av++) { if (vtc_error) break; + if (!strcmp(*av, "-wait")) { + if (!s->run) + vtc_log(s->vl, 0, "Server not -started"); + server_wait(s); + continue; + } + /* + * We do an implict -wait if people muck about with a + * running server. + */ + if (s->run) + server_wait(s); + + assert(s->run == 0); if (!strcmp(*av, "-repeat")) { s->repeat = atoi(av[1]); av++; @@ -281,10 +300,6 @@ server_start(s); continue; } - if (!strcmp(*av, "-wait")) { - server_wait(s); - continue; - } if (**av == '-') vtc_log(s->vl, 0, "Unknown server argument: %s", *av); s->spec = *av; From phk at projects.linpro.no Thu Jan 28 09:58:09 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 10:58:09 +0100 (CET) Subject: r4504 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128095809.64A701F7E6B@projects.linpro.no> Author: phk Date: 2010-01-28 10:58:09 +0100 (Thu, 28 Jan 2010) New Revision: 4504 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c Log: Try to use 10.255.255.255 as un-bindable IP number, in the hope that there is some limit to linux kernels promiscuousness. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 09:36:15 UTC (rev 4503) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 09:58:09 UTC (rev 4504) @@ -562,7 +562,7 @@ macro_def(vl, NULL, "tmpdir", vtc_tmpdir); vtc_thread = pthread_self(); - macro_def(vl, NULL, "bad_ip", "255.255.255.254"); + macro_def(vl, NULL, "bad_ip", "10.255.255.255"); tmax = 0; nmax = NULL; t00 = TIM_mono(); From phk at projects.linpro.no Thu Jan 28 10:20:19 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 11:20:19 +0100 (CET) Subject: r4505 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100128102019.41F601F7E6B@projects.linpro.no> Author: phk Date: 2010-01-28 11:20:18 +0100 (Thu, 28 Jan 2010) New Revision: 4505 Modified: trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc Log: Fix this test-case to actually wait for the last and critical client transaction Modified: trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc 2010-01-28 09:58:09 UTC (rev 4504) +++ trunk/varnish-cache/bin/varnishtest/tests/v00014.vtc 2010-01-28 10:20:18 UTC (rev 4505) @@ -51,10 +51,9 @@ txresp -body "foobar" } -start -client c2 { +client c1 { txreq -url "/foo" rxresp - expect resp.bodylen == 6 expect resp.status == 200 -} -start +} -run From phk at projects.linpro.no Thu Jan 28 10:22:07 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 11:22:07 +0100 (CET) Subject: r4506 - trunk/varnish-cache/bin/varnishtest/tests Message-ID: <20100128102207.159C11F7E6B@projects.linpro.no> Author: phk Date: 2010-01-28 11:22:06 +0100 (Thu, 28 Jan 2010) New Revision: 4506 Modified: trunk/varnish-cache/bin/varnishtest/tests/c00016.vtc trunk/varnish-cache/bin/varnishtest/tests/r00386.vtc Log: Fix two other test-cases that did not wait for clients to finish Modified: trunk/varnish-cache/bin/varnishtest/tests/c00016.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/c00016.vtc 2010-01-28 10:20:18 UTC (rev 4505) +++ trunk/varnish-cache/bin/varnishtest/tests/c00016.vtc 2010-01-28 10:22:06 UTC (rev 4506) @@ -24,4 +24,4 @@ txreq -url "/bar" -hdr "Foo: bar2" -hdr "Connection: foo, close" rxresp expect req.http.Bar == "req.http.Bar" -} -start +} -run Modified: trunk/varnish-cache/bin/varnishtest/tests/r00386.vtc =================================================================== --- trunk/varnish-cache/bin/varnishtest/tests/r00386.vtc 2010-01-28 10:20:18 UTC (rev 4505) +++ trunk/varnish-cache/bin/varnishtest/tests/r00386.vtc 2010-01-28 10:22:06 UTC (rev 4506) @@ -26,4 +26,4 @@ txreq -url / -hdr "If-Modified-Since: Tue, 25 Nov 2008 00:00:00 GMT" rxresp expect resp.bodylen == 11 -} -start +} -run From phk at projects.linpro.no Thu Jan 28 11:09:18 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 12:09:18 +0100 (CET) Subject: r4507 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128110918.8ACCE1F7E73@projects.linpro.no> Author: phk Date: 2010-01-28 12:09:18 +0100 (Thu, 28 Jan 2010) New Revision: 4507 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c Log: Add a default 30 second timeout around all test-cases. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 10:22:06 UTC (rev 4506) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 11:09:18 UTC (rev 4507) @@ -53,12 +53,15 @@ #define MAX_FILESIZE (1024 * 1024) #define MAX_TOKENS 200 -const char *vtc_file; -char *vtc_desc; -int vtc_error; /* Error encountered */ -int vtc_stop; /* Stops current test without error */ -pthread_t vtc_thread; -char *vtc_tmpdir; +const char *vtc_file; +char *vtc_desc; +int vtc_error; /* Error encountered */ +int vtc_stop; /* Stops current test without error */ +pthread_t vtc_thread; +char *vtc_tmpdir; +static struct vtclog *vltop; +static pthread_mutex_t vtc_mtx; +static pthread_cond_t vtc_cond; /********************************************************************** * Macro facility @@ -460,33 +463,67 @@ { NULL, NULL } }; +struct priv_exec { + const char *fn; + char *buf; +}; + +static void * +exec_file_thread(void *priv) +{ + unsigned old_err; + struct priv_exec *pe; + + pe = priv; + + parse_string(pe->buf, cmds, NULL, vltop); + old_err = vtc_error; + vtc_stop = 1; + vtc_log(vltop, 1, "RESETTING after %s", pe->fn); + reset_cmds(cmds); + vtc_error = old_err; + AZ(pthread_cond_signal(&vtc_cond)); + return (NULL); +} + static double -exec_file(const char *fn, struct vtclog *vl) +exec_file(const char *fn, unsigned dur) { - char *buf; double t0; - unsigned old_err; + struct priv_exec pe; + pthread_t pt; + struct timespec ts; + void *v; + int i; t0 = TIM_mono(); vtc_stop = 0; vtc_file = fn; vtc_desc = NULL; - vtc_log(vl, 1, "TEST %s starting", fn); - buf = read_file(fn); - if (buf == NULL) - vtc_log(vl, 0, "Cannot read file '%s': %s", + vtc_log(vltop, 1, "TEST %s starting", fn); + pe.buf = read_file(fn); + if (pe.buf == NULL) + vtc_log(vltop, 0, "Cannot read file '%s': %s", fn, strerror(errno)); - parse_string(buf, cmds, NULL, vl); - old_err = vtc_error; - vtc_stop = 1; - vtc_log(vl, 1, "RESETTING after %s", fn); - reset_cmds(cmds); - vtc_error = old_err; + pe.fn = fn; + AZ(pthread_create(&pt, NULL, exec_file_thread, &pe)); + AZ(pthread_mutex_lock(&vtc_mtx)); + AZ(clock_gettime(CLOCK_REALTIME, &ts)); + ts.tv_sec += dur; + i = pthread_cond_timedwait(&vtc_cond, &vtc_mtx, &ts); + AZ(pthread_mutex_unlock(&vtc_mtx)); + if (i == ETIMEDOUT) { + vtc_log(vltop, 1, "Test timed out"); + vtc_error = 1; + } else { + AZ(pthread_join(pt, &v)); + } + if (vtc_error) - vtc_log(vl, 1, "TEST %s FAILED", fn); + vtc_log(vltop, 1, "TEST %s FAILED", fn); else { - vtc_log(vl, 1, "TEST %s completed", fn); + vtc_log(vltop, 1, "TEST %s completed", fn); vtc_logreset(); } @@ -517,22 +554,26 @@ * Main */ +/********************************************************************** + * Main + */ + int main(int argc, char * const *argv) { int ch, i, ntest = 1, ncheck = 0; FILE *fok; - static struct vtclog *vl; double tmax, t0, t00; + unsigned dur = 30; const char *nmax; char cmd[BUFSIZ]; setbuf(stdout, NULL); setbuf(stderr, NULL); vtc_loginit(); - vl = vtc_logopen("top"); - AN(vl); - while ((ch = getopt(argc, argv, "n:qv")) != -1) { + vltop = vtc_logopen("top"); + AN(vltop); + while ((ch = getopt(argc, argv, "n:qt:v")) != -1) { switch (ch) { case 'n': ntest = strtoul(optarg, NULL, 0); @@ -540,6 +581,9 @@ case 'q': vtc_verbosity--; break; + case 't': + dur = strtoul(optarg, NULL, 0); + break; case 'v': vtc_verbosity++; break; @@ -559,16 +603,19 @@ vtc_tmpdir = tempnam(NULL, "vtc"); AN(vtc_tmpdir); AZ(mkdir(vtc_tmpdir, 0700)); - macro_def(vl, NULL, "tmpdir", vtc_tmpdir); + macro_def(vltop, NULL, "tmpdir", vtc_tmpdir); vtc_thread = pthread_self(); - macro_def(vl, NULL, "bad_ip", "10.255.255.255"); + AZ(pthread_mutex_init(&vtc_mtx, NULL)); + AZ(pthread_cond_init(&vtc_cond, NULL)); + + macro_def(vltop, NULL, "bad_ip", "10.255.255.255"); tmax = 0; nmax = NULL; t00 = TIM_mono(); for (i = 0; i < ntest; i++) { for (ch = 0; ch < argc; ch++) { - t0 = exec_file(argv[ch], vl); + t0 = exec_file(argv[ch], dur); ncheck++; if (t0 > tmax) { tmax = t0; From phk at projects.linpro.no Thu Jan 28 11:21:26 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 12:21:26 +0100 (CET) Subject: r4508 - trunk/varnish-cache/bin/varnishd Message-ID: <20100128112126.0C8E11F7E73@projects.linpro.no> Author: phk Date: 2010-01-28 12:21:25 +0100 (Thu, 28 Jan 2010) New Revision: 4508 Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c Log: Allow backend timeouts to be zero. Fixes #635 Modified: trunk/varnish-cache/bin/varnishd/cache_backend.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-28 11:09:18 UTC (rev 4507) +++ trunk/varnish-cache/bin/varnishd/cache_backend.c 2010-01-28 11:21:25 UTC (rev 4508) @@ -98,7 +98,6 @@ dst = be->tmx; \ if (dst == 0.0) \ dst = params->tmx; \ - assert(dst > 0.0); \ } while (0) From phk at projects.linpro.no Thu Jan 28 11:51:09 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Thu, 28 Jan 2010 12:51:09 +0100 (CET) Subject: r4509 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100128115109.B560D1F7E7B@projects.linpro.no> Author: phk Date: 2010-01-28 12:51:09 +0100 (Thu, 28 Jan 2010) New Revision: 4509 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c Log: Be even more paranoid about the condvar used for the timeout. Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 11:21:25 UTC (rev 4508) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 11:51:09 UTC (rev 4509) @@ -482,7 +482,9 @@ vtc_log(vltop, 1, "RESETTING after %s", pe->fn); reset_cmds(cmds); vtc_error = old_err; + AZ(pthread_mutex_lock(&vtc_mtx)); AZ(pthread_cond_signal(&vtc_cond)); + AZ(pthread_mutex_unlock(&vtc_mtx)); return (NULL); } @@ -507,12 +509,14 @@ fn, strerror(errno)); pe.fn = fn; - AZ(pthread_create(&pt, NULL, exec_file_thread, &pe)); - AZ(pthread_mutex_lock(&vtc_mtx)); AZ(clock_gettime(CLOCK_REALTIME, &ts)); ts.tv_sec += dur; + + AZ(pthread_mutex_lock(&vtc_mtx)); + AZ(pthread_create(&pt, NULL, exec_file_thread, &pe)); i = pthread_cond_timedwait(&vtc_cond, &vtc_mtx, &ts); AZ(pthread_mutex_unlock(&vtc_mtx)); + if (i == ETIMEDOUT) { vtc_log(vltop, 1, "Test timed out"); vtc_error = 1; From phk at projects.linpro.no Fri Jan 29 09:56:02 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 29 Jan 2010 10:56:02 +0100 (CET) Subject: r4510 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100129095602.13D721F7E8E@projects.linpro.no> Author: phk Date: 2010-01-29 10:56:01 +0100 (Fri, 29 Jan 2010) New Revision: 4510 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c trunk/varnish-cache/bin/varnishtest/vtc.h Log: Try to unbreak, or at least diagnose the tinderbox Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-28 11:51:09 UTC (rev 4509) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-29 09:56:01 UTC (rev 4510) @@ -53,8 +53,8 @@ #define MAX_FILESIZE (1024 * 1024) #define MAX_TOKENS 200 -const char *vtc_file; -char *vtc_desc; +static const char *vtc_file; +static char *vtc_desc; int vtc_error; /* Error encountered */ int vtc_stop; /* Stops current test without error */ pthread_t vtc_thread; @@ -515,13 +515,18 @@ AZ(pthread_mutex_lock(&vtc_mtx)); AZ(pthread_create(&pt, NULL, exec_file_thread, &pe)); i = pthread_cond_timedwait(&vtc_cond, &vtc_mtx, &ts); - AZ(pthread_mutex_unlock(&vtc_mtx)); if (i == ETIMEDOUT) { + // AZ(pthread_cancel(pt)); + // AZ(pthread_join(pt, &v)); vtc_log(vltop, 1, "Test timed out"); vtc_error = 1; + } else if (i == 0) { + AZ(pthread_mutex_unlock(&vtc_mtx)); + AZ(pthread_join(pt, &v)); } else { - AZ(pthread_join(pt, &v)); + vtc_log(vltop, 1, "Weird return: %d %s", i, strerror(i)); + vtc_error = 1; } if (vtc_error) Modified: trunk/varnish-cache/bin/varnishtest/vtc.h =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-28 11:51:09 UTC (rev 4509) +++ trunk/varnish-cache/bin/varnishtest/vtc.h 2010-01-29 09:56:01 UTC (rev 4510) @@ -54,8 +54,6 @@ cmd_f cmd_varnish; cmd_f cmd_sema; -extern const char *vtc_file; -extern char *vtc_desc; extern int vtc_verbosity; extern int vtc_error; /* Error, bail out */ extern int vtc_stop; /* Abandon current test, no error */ From phk at projects.linpro.no Fri Jan 29 10:39:41 2010 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Fri, 29 Jan 2010 11:39:41 +0100 (CET) Subject: r4511 - trunk/varnish-cache/bin/varnishtest Message-ID: <20100129103941.A8B8A1F7E96@projects.linpro.no> Author: phk Date: 2010-01-29 11:39:41 +0100 (Fri, 29 Jan 2010) New Revision: 4511 Modified: trunk/varnish-cache/bin/varnishtest/vtc.c Log: Document why we don't clean up in case of a timeout Modified: trunk/varnish-cache/bin/varnishtest/vtc.c =================================================================== --- trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-29 09:56:01 UTC (rev 4510) +++ trunk/varnish-cache/bin/varnishtest/vtc.c 2010-01-29 10:39:41 UTC (rev 4511) @@ -516,16 +516,22 @@ AZ(pthread_create(&pt, NULL, exec_file_thread, &pe)); i = pthread_cond_timedwait(&vtc_cond, &vtc_mtx, &ts); - if (i == ETIMEDOUT) { - // AZ(pthread_cancel(pt)); - // AZ(pthread_join(pt, &v)); - vtc_log(vltop, 1, "Test timed out"); - vtc_error = 1; - } else if (i == 0) { + if (i == 0) { AZ(pthread_mutex_unlock(&vtc_mtx)); AZ(pthread_join(pt, &v)); } else { - vtc_log(vltop, 1, "Weird return: %d %s", i, strerror(i)); + if (i != ETIMEDOUT) + vtc_log(vltop, 1, "Weird condwait return: %d %s", + i, strerror(i)); + /* + * We are all going to die anyway, so don't waste time + * trying to clean things up, it seems to trigger a + * problem in the tinderbox + * AZ(pthread_mutex_unlock(&vtc_mtx)); + * AZ(pthread_cancel(pt)); + * AZ(pthread_join(pt, &v)); + */ + vtc_log(vltop, 1, "Test timed out"); vtc_error = 1; } From tfheen at projects.linpro.no Fri Jan 29 12:16:31 2010 From: tfheen at projects.linpro.no (tfheen at projects.linpro.no) Date: Fri, 29 Jan 2010 13:16:31 +0100 (CET) Subject: r4512 - in trunk/varnish-tools: . cache-channels Message-ID: <20100129121631.E7D711F7E9B@projects.linpro.no> Author: tfheen Date: 2010-01-29 13:16:31 +0100 (Fri, 29 Jan 2010) New Revision: 4512 Added: trunk/varnish-tools/cache-channels/ trunk/varnish-tools/cache-channels/README trunk/varnish-tools/cache-channels/varnish-cc trunk/varnish-tools/cache-channels/varnish-cc.init trunk/varnish-tools/cache-channels/varnish-cc.yaml Log: Add cache channel implementation Cache channels are useful for active cache management where a backend publishes a feed of objects that are to be expired. See http://ietfreport.isoc.org/idref/draft-nottingham-http-cache-channels/ for an expired internet draft about this. Added: trunk/varnish-tools/cache-channels/README =================================================================== --- trunk/varnish-tools/cache-channels/README (rev 0) +++ trunk/varnish-tools/cache-channels/README 2010-01-29 12:16:31 UTC (rev 4512) @@ -0,0 +1,27 @@ +Description + +Perl script to read a cache channel feed and send PURGE requests to varnish. +See http://www.mnot.net/blog/2008/01/04/cache_channels for a short +introduction on what cache channels are. + +Requirements + +perl +XML::Atom::Feed +DateTime::Format::ISO8601 +HTTP::Request +Sys::Syslog +YAML + +Varnish Configuration + +See http://varnish.projects.linpro.no/wiki/VCLExamplePurging + +Installation + +varnish-cc -> /usr/local/sbin/varnish-cc +varnish-cc.init -> /etc/init.d/varnish-cc +varnish-cc.yaml -> /etc/varnish-cc.yaml + +Edit /etc/varnish-cc.yaml. + Added: trunk/varnish-tools/cache-channels/varnish-cc =================================================================== --- trunk/varnish-tools/cache-channels/varnish-cc (rev 0) +++ trunk/varnish-tools/cache-channels/varnish-cc 2010-01-29 12:16:31 UTC (rev 4512) @@ -0,0 +1,84 @@ +#!/usr/bin/env perl +# +# cache control +# + + +use lib qw[ /root/inc ]; + +use XML::Atom::Feed; +use DateTime::Format::ISO8601; +use HTTP::Request; +use Sys::Syslog qw( :DEFAULT setlogsock); +use YAML; +use Data::Dumper; + + +open my $fh, '<', '/etc/varnish-cc.yaml' or die "can't open config file: $!"; +my $config = YAML::LoadFile($fh); + +my $channel = $config->{'channel'}; +my $interval = $config->{'default_interval'}; +my $host = $config->{'host'}; + +# +# + +my $latest = DateTime->now(time_zone => "Europe/Oslo"); #DateTime::Format::ISO8601->parse_datetime("2006-04-13T11:23:42Z"); +my $last_entry = $latest; +my $cc_ns = XML::Atom::Namespace->new(dc => 'http://purl.org/syndication/cache-channel'); +my $ua = LWP::UserAgent->new; +setlogsock('unix'); + +sub invalidate { + my $e = $_[0]; + + purge("http://localhost" . $e); + +} + +sub purge{ + my $uri = $_[0]; + my $req = HTTP::Request->new(PURGE => $uri); + my $response = $ua->request($req); + if ($response->is_error) { + openlog($0,'','user'); + syslog('err', $uri . ": " . $response->status_line . "\n"); + closelog; + #print STDERR $uri . ": " . $response->status_line . "\n"; + } +} + +sub get{ + my $uri = $_[0]; + my $req = HTTP::Request->new(GET => $uri); + my $response = $ua->request($req); + if ($response->is_error) { + openlog($0,'','user'); + syslog('err', $uri . ": " . $response->status_line . "\n"); + closelog; + #print STDERR $uri . ": " . $response->status_line . "\n"; + } +} + +while(1) { + my $feed = XML::Atom::Feed->new(URI->new($channel)); + $interval = $feed->get($cc_ns, 'precision'); # FIXME + for my $entry ($feed->entries()) { + $t = $entry->updated(); + $t =~ s/(?<=[+-]\d\d)(?=\d\d$)/:/; + my $entry_updated = DateTime::Format::ISO8601->parse_datetime($t); + if ($latest < $entry_updated) { + my $link = $entry->link->elem; + invalidate($link->getAttribute("href")); + if ($last_entry < $entry_updated) { + $last_entry = $entry_updated; + } + } + } + $latest = $last_entry; + + sleep $interval; +} + + Added: trunk/varnish-tools/cache-channels/varnish-cc.init =================================================================== --- trunk/varnish-tools/cache-channels/varnish-cc.init (rev 0) +++ trunk/varnish-tools/cache-channels/varnish-cc.init 2010-01-29 12:16:31 UTC (rev 4512) @@ -0,0 +1,75 @@ +#!/bin/bash +# +# init script for varnish cache control +# + +if [ -e /var/run/varnish-cc ] ; then + PID=$( /var/run/varnish-cc + exit 0 +} + +stop() { + kill $PID + sleep 1 + if running ; then + echo "$0 just won't quit" + exit 1 + else + rm /var/run/varnish-cc + exit 0 + fi +} + +status() { + if running ; then + echo "varnish-cc is running as pid $PID" + exit 0 + elif [ $PID = "bl?h" ] ; then + # exited cleanly + echo "trouble" + exit 3 + else + echo "trouble" + exit 1 + fi +} + +case "$1" in + "start") + start + ;; + "stop") + stop + ;; + "restart") + stop + start + ;; + "force-reload") + # you don't want to + exit 3 + ;; + "status") + status + ;; + *) + echo "usage: $0 (start|stop|restart|status)" +esac +exit 0 + Added: trunk/varnish-tools/cache-channels/varnish-cc.yaml =================================================================== --- trunk/varnish-tools/cache-channels/varnish-cc.yaml (rev 0) +++ trunk/varnish-tools/cache-channels/varnish-cc.yaml 2010-01-29 12:16:31 UTC (rev 4512) @@ -0,0 +1,3 @@ +channel: http://www.example.com:8888/atomizer/event/current +default_interval: 30 +host: localhost:9001