From stockrt at gmail.com Thu Oct 8 14:58:03 2009 From: stockrt at gmail.com (=?ISO-8859-1?Q?Rog=E9rio_Schneider?=) Date: Thu, 8 Oct 2009 11:58:03 -0300 Subject: about vcl In-Reply-To: <4aa68089.141bf30a.4f58.0849@mx.google.com> References: <4aa68089.141bf30a.4f58.0849@mx.google.com> Message-ID: <100657c90910080758w5b555440ge7fa028b03c86e0e@mail.gmail.com> Yuri, I think it is: > ?* Assignments are introduced with the set keyword. You must use the "set" keyword to assign a value to a variable, in vcl. > There are no > user-defined variables; An user can not create a variable, the only variables available to use are the ones that Varnish provide, like: req.http.host, obj.ttl... > values can only be assigned to variables attached to > backend, request or document objects. You can only alter the variables under the structures of: resp beresp req bereq obj > Most of these are typed, and the > values assigned to them must have a compatible unit suffix.? The majority of those variables are typed, they must receive values which are compatible with their type, as in, for a variable which type is "int" (an integer) you must use "set" to assign a value of an "int", for example: 10; Trying to set it to a string will result in failure. These are my understandings of the above, if anyone have something to add, or to fix, please do it. Regards, -- Rog?rio Schneider MSN: stockrt at hotmail.com GTalk: stockrt at gmail.com Skype: stockrt http://stockrt.github.com From izak at upfrontsystems.co.za Wed Oct 14 17:42:05 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Wed, 14 Oct 2009 19:42:05 +0200 Subject: Backend polling not working Message-ID: <4AD60D6D.1070909@upfrontsystems.co.za> Hi all, I tried varnish today for the first time. This happened after we upgraded to a newer version of squid and it hung itself in an infinite loop. But that is a different story. We want to use varnish to front a number of zope backends, all of which speak to the same database backend (so they serve the same content). The problem with this zope application is that it takes a long time to reload. Those who know plone knows it is very heavy on the catalog, and in our case it means that after restarting a backend, it can take up to 15 minutes for the caches to warm up again. During this time, we would like the zope backends to be marked as sick and not be used. Unfortunately this does not seem to work in version 2.0.4 (which seems the most recent if I check your svn tags). The odd thing is this: it seems that the probe succeeds, but zope's access log shows that zero bytes is transferred, eg: 10.0.0.12 - Anonymous [14/Oct/2009:18:38:38 +0200] "GET /plonesite HTTP/1.1" 200 0 "" Note the zero after the 200 status. I suspect this causes a problem on line 220 of bin/varnishd/cache_backend_poll.c, where rlen is the length of data read from zope: if (rlen == 0) return; The result of this is a sick backend, eg: 0 Backend_health - live1 Still sick 4--X-S--- 0 2 3 0.000000 0.000000 Using wireshark to log at the network traffic confirms that the HTTP request is sent, but no response, corresponding with the access log. This seems very similar to this report: http://projects.linpro.no/pipermail/varnish-dev/2009-February/002086.html Has anyone seen this problem before? Perhaps there is a fix in svn already? I could not see anything that looked promising while looking through the subversion logs. regards, Izak From phk at phk.freebsd.dk Wed Oct 14 18:34:52 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 14 Oct 2009 18:34:52 +0000 Subject: Backend polling not working In-Reply-To: Your message of "Wed, 14 Oct 2009 19:42:05 +0200." <4AD60D6D.1070909@upfrontsystems.co.za> Message-ID: <32970.1255545292@critter.freebsd.dk> In message <4AD60D6D.1070909 at upfrontsystems.co.za>, Izak Burger writes: >Hi all, > >10.0.0.12 - Anonymous [14/Oct/2009:18:38:38 +0200] "GET /plonesite >HTTP/1.1" 200 0 "" > >Note the zero after the 200 status. I suspect this causes a problem on >line 220 of bin/varnishd/cache_backend_poll.c, where rlen is the length >of data read from zope: > > if (rlen == 0) > return; > >The result of this is a sick backend, eg: Yes, if we get no bytes, we do not accept it as a good probe. >Using wireshark to log at the network traffic confirms that the HTTP >request is sent, but no response, corresponding with the access log. That would sort of indicate that the problem is on the backend. You may need to customize the HTTP request used for probing with the ".request =" line in your backend probe definition, in order to explain things to your backend. A Host: header maybe ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From izak at upfrontsystems.co.za Wed Oct 14 19:42:58 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Wed, 14 Oct 2009 21:42:58 +0200 Subject: Backend polling not working In-Reply-To: <4AD61BC5.1040608@upfrontsystems.co.za> References: <32970.1255545292@critter.freebsd.dk> <4AD61BC5.1040608@upfrontsystems.co.za> Message-ID: <4AD629C2.5050905@upfrontsystems.co.za> Poul-Henning Kamp wrote: > Yes, if we get no bytes, we do not accept it as a good probe. Yes, saw that in the svn logs, and I agree with it. > You may need to customize the HTTP request used for probing with > the ".request =" line in your backend probe definition, in order > to explain things to your backend. A Host: header maybe ? I had the same thought, so I took the request as captured with the packet grabber and tried it against zope using telnet, and it produces the required result. That in itself is perhaps a hint, I think the telnet client inserts the correct CRLF sequences which might not be immediately apparent. Also, if read(2) returns zero it means EOF, so the problem must somehow be on the backend. I'll reply again on this thread if I manage to figure it out. regards, Izak From izak at upfrontsystems.co.za Thu Oct 15 06:25:52 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Thu, 15 Oct 2009 08:25:52 +0200 Subject: Backend polling not working In-Reply-To: <4AD629C2.5050905@upfrontsystems.co.za> References: <32970.1255545292@critter.freebsd.dk> <4AD61BC5.1040608@upfrontsystems.co.za> <4AD629C2.5050905@upfrontsystems.co.za> Message-ID: <4AD6C070.20100@upfrontsystems.co.za> Izak Burger wrote: > I had the same thought, so I took the request as captured with the > packet grabber and tried it against zope using telnet, and it produces > the required result. That in itself is perhaps a hint, I think the > telnet client inserts the correct CRLF sequences which might not be > immediately apparent. > > Also, if read(2) returns zero it means EOF, so the problem must somehow > be on the backend. I'll reply again on this thread if I manage to figure > it out. When inspecting the network traffic I noticed that the client side (varnish) sends the first FIN packet, and not the server/backend (zope). This means that varnish closes the connection without reading the response. As far as I know, when you specify "Connection: close", the server will close the connection after the response rather than the other way round. I then added some extra log statements to the C code and confirmed that the do/while loop that is supposed to read the response terminates on the very first read() call with an EOF. In other words, there is definitely something wrong with varnish here. I suspect the problem might be with poll(), although I have yet to confirm that. Specifically, it seems that the case where poll() returns -1 is not handled properly. At the least I think it should handle EINTR? I'll check that later and let you know. regards, Izak From izak at upfrontsystems.co.za Thu Oct 15 07:46:54 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Thu, 15 Oct 2009 09:46:54 +0200 Subject: Backend polling not working In-Reply-To: <4AD6C070.20100@upfrontsystems.co.za> References: <32970.1255545292@critter.freebsd.dk> <4AD61BC5.1040608@upfrontsystems.co.za> <4AD629C2.5050905@upfrontsystems.co.za> <4AD6C070.20100@upfrontsystems.co.za> Message-ID: <4AD6D36E.9060704@upfrontsystems.co.za> Izak Burger wrote: > I then added some extra log statements to the C code and confirmed that > the do/while loop that is supposed to read the response terminates on > the very first read() call with an EOF. > > In other words, there is definitely something wrong with varnish here. > > I suspect the problem might be with poll(), although I have yet to > confirm that. Specifically, it seems that the case where poll() returns > -1 is not handled properly. At the least I think it should handle EINTR? > I'll check that later and let you know. Alright, I have to take that back. I changed the port so varnish talks to apache instead and that works. That means zope somehow manages to cause a POLLIN event on the file descriptor even though there is nothing to read. Will have to do more research on this. From various sources found by google, it seems backend probing fails in a similar manner with nginx. regards, Izak From izak at upfrontsystems.co.za Thu Oct 15 10:29:16 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Thu, 15 Oct 2009 12:29:16 +0200 Subject: Backend polling not working In-Reply-To: <4AD6D36E.9060704@upfrontsystems.co.za> References: <32970.1255545292@critter.freebsd.dk> <4AD61BC5.1040608@upfrontsystems.co.za> <4AD629C2.5050905@upfrontsystems.co.za> <4AD6C070.20100@upfrontsystems.co.za> <4AD6D36E.9060704@upfrontsystems.co.za> Message-ID: <4AD6F97C.0@upfrontsystems.co.za> Apologies for the (mostly) monologue this far :-) Izak Burger wrote: > Alright, I have to take that back. I changed the port so varnish talks > to apache instead and that works. That means zope somehow manages to > cause a POLLIN event on the file descriptor even though there is nothing > to read. Will have to do more research on this. It appears the fault is somewhere in zope. I ended up writing a test program to illustrate the problem. The reason why poll() returns and yet read() has nothing to read is that poll() always implicitly checks for the additional events POLLERR, POLLHUP and POLLLNVAL, so simply checking that poll() returns greater than zero is not a guarantee that there is data to be read. In my case, poll() returns immediately because POLLHUP has occured. It seems that zope reacts (incorrectly it would seem) on the shutdown() call and closes down the entire connection. I will have to take this up with zope-dev. regards, Izak From tfheen at redpill-linpro.com Fri Oct 16 14:20:28 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Fri, 16 Oct 2009 16:20:28 +0200 Subject: [PATCH] Add PCRE support Message-ID: <87skdj5sk3.fsf@qurzaw.linpro.no> We no longer use POSIX style regexes, but rather PCRE regexes. This introduces a hard dependency on libpcre. Test suite passes, but only lightly tested. --- varnish-cache/bin/varnishd/Makefile.am | 3 +- varnish-cache/bin/varnishd/cache_ban.c | 29 ++++++++------- varnish-cache/bin/varnishd/cache_ban.h | 2 +- varnish-cache/bin/varnishd/cache_vrt_re.c | 48 +++++++++++++----------- varnish-cache/bin/varnishlog/Makefile.am | 3 +- varnish-cache/bin/varnishlog/varnishlog.c | 18 +++++---- varnish-cache/configure.ac | 31 ++++++++++++++++ varnish-cache/include/vrt.h | 2 +- varnish-cache/lib/libvarnishapi/Makefile.am | 4 ++- varnish-cache/lib/libvarnishapi/shmlog.c | 52 +++++++++++++-------------- varnish-cache/lib/libvcl/Makefile.am | 4 ++- varnish-cache/lib/libvcl/vcc_compile.h | 2 +- varnish-cache/lib/libvcl/vcc_parse.c | 2 +- varnish-cache/lib/libvcl/vcc_string.c | 23 ++++++------ 14 files changed, 132 insertions(+), 91 deletions(-) diff --git a/varnish-cache/bin/varnishd/Makefile.am b/varnish-cache/bin/varnishd/Makefile.am index dfdef42..daa389b 100644 --- a/varnish-cache/bin/varnishd/Makefile.am +++ b/varnish-cache/bin/varnishd/Makefile.am @@ -1,6 +1,6 @@ # $Id$ -INCLUDES = -I$(top_srcdir)/include +INCLUDES = -I$(top_srcdir)/include @PCRE_CFLAGS@ sbin_PROGRAMS = varnishd @@ -87,6 +87,7 @@ varnishd_LDADD = \ $(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \ $(top_builddir)/lib/libvcl/libvcl.la \ @JEMALLOC_LDADD@ \ + @PCRE_LIBS@ \ ${DL_LIBS} ${PTHREAD_LIBS} ${NET_LIBS} ${LIBM} ${LIBUMEM} EXTRA_DIST = default.vcl diff --git a/varnish-cache/bin/varnishd/cache_ban.c b/varnish-cache/bin/varnishd/cache_ban.c index 57bdb05..6f6cc0e 100644 --- a/varnish-cache/bin/varnishd/cache_ban.c +++ b/varnish-cache/bin/varnishd/cache_ban.c @@ -50,7 +50,7 @@ SVNID("$Id$") #include #include #include -#include +#include #include "shmlog.h" #include "cli.h" @@ -116,7 +116,7 @@ BAN_Free(struct ban *b) bt = VTAILQ_FIRST(&b->tests); VTAILQ_REMOVE(&b->tests, bt, list); if (bt->flags & BAN_T_REGEXP) - regfree(&bt->re); + pcre_free(&bt->re); if (bt->dst != NULL) free(bt->dst); if (bt->src != NULL) @@ -137,10 +137,13 @@ ban_cond_str(const struct ban_test *bt, const char *p) if (p == NULL) return(!(bt->flags & BAN_T_NOT)); - if (bt->flags & BAN_T_REGEXP) - i = regexec(&bt->re, p, 0, NULL, 0); - else + if (bt->flags & BAN_T_REGEXP) { + i = pcre_exec(bt->re, NULL, p, strlen(p), 0, 0, NULL, 0); + if (i >= 0) + i = 0; + } else { i = strcmp(bt->dst, p); + } if (bt->flags & BAN_T_NOT) return (!i); return (i); @@ -199,15 +202,13 @@ ban_cond_obj_http(const struct ban_test *bt, const struct object *o, static int ban_parse_regexp(struct cli *cli, struct ban_test *bt, const char *a3) { - int i; - char buf[512]; - - i = regcomp(&bt->re, a3, REG_EXTENDED | REG_ICASE | REG_NOSUB); - if (i) { - (void)regerror(i, &bt->re, buf, sizeof buf); - regfree(&bt->re); - VSL(SLT_Debug, 0, "REGEX: <%s>", buf); - cli_out(cli, "%s", buf); + const char *error; + int erroroffset; + + bt->re = pcre_compile(a3, 0, &error, &erroroffset, NULL); + if (bt->re == NULL) { + VSL(SLT_Debug, 0, "REGEX: <%s>", error); + cli_out(cli, "%s", error); cli_result(cli, CLIS_PARAM); return (-1); } diff --git a/varnish-cache/bin/varnishd/cache_ban.h b/varnish-cache/bin/varnishd/cache_ban.h index 58a1ff8..673e5a9 100644 --- a/varnish-cache/bin/varnishd/cache_ban.h +++ b/varnish-cache/bin/varnishd/cache_ban.h @@ -44,7 +44,7 @@ struct ban_test { int flags; #define BAN_T_REGEXP (1 << 0) #define BAN_T_NOT (1 << 1) - regex_t re; + pcre *re; char *dst; char *src; }; diff --git a/varnish-cache/bin/varnishd/cache_vrt_re.c b/varnish-cache/bin/varnishd/cache_vrt_re.c index e91f3c1..0e9052a 100644 --- a/varnish-cache/bin/varnishd/cache_vrt_re.c +++ b/varnish-cache/bin/varnishd/cache_vrt_re.c @@ -35,12 +35,14 @@ SVNID("$Id$") #include +#include + #include #include #include #include -#include +#include #include "shmlog.h" #include "vrt.h" @@ -48,14 +50,15 @@ SVNID("$Id$") #include "cache.h" void -VRT_re_init(void **rep, const char *re, int sub) +VRT_re_init(void **rep, const char *re) { - regex_t *t; + pcre *t; + const char *error; + int erroroffset; - t = calloc(sizeof *t, 1); - XXXAN(t); /* This was already check-compiled by the VCL compiler */ - AZ(regcomp(t, re, REG_EXTENDED | REG_ICASE | (sub ? 0 : REG_NOSUB))); + t = pcre_compile(re, 0, &error, &erroroffset, NULL); + AN(t); *rep = t; } @@ -64,23 +67,23 @@ VRT_re_fini(void *rep) { if (rep != NULL) - regfree(rep); + pcre_free(rep); } int VRT_re_match(const char *s, void *re) { - regex_t *t; + pcre *t; int i; if (s == NULL) return (0); AN(re); t = re; - i = regexec(t, s, 0, NULL, 0); - if (i == 0) + i = pcre_exec(t, NULL, s, strlen(s), 0, 0, NULL, 0); + if (i >= 0) return (1); - assert(i == REG_NOMATCH); + assert(i == PCRE_ERROR_NOMATCH); return (0); } @@ -88,8 +91,8 @@ const char * VRT_regsub(const struct sess *sp, int all, const char *str, void *re, const char *sub) { - regmatch_t pm[10]; - regex_t *t; + int ovector[30]; + pcre *t; int i, l; txt res; char *b0; @@ -100,10 +103,11 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re, if (str == NULL) return (""); t = re; - i = regexec(t, str, 10, pm, 0); + memset(&ovector, 0, sizeof(ovector)); + i = pcre_exec(t, NULL, str, strlen(str), 0, 0, ovector, 30); /* If it didn't match, we can return the original string */ - if (i == REG_NOMATCH) + if (i == PCRE_ERROR_NOMATCH) return(str); u = WS_Reserve(sp->http->ws, 0); @@ -112,8 +116,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re, do { /* Copy prefix to match */ - Tadd(&res, str, pm[0].rm_so); - + Tadd(&res, str, ovector[0]); for (s = sub ; *s != '\0'; s++ ) { if (*s != '\\' || s[1] == '\0') { if (res.b < res.e) @@ -123,19 +126,20 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re, s++; if (isdigit(*s)) { x = *s - '0'; - l = pm[x].rm_eo - pm[x].rm_so; - Tadd(&res, str + pm[x].rm_so, l); + l = ovector[2*x+1] - ovector[2*x]; + Tadd(&res, str + ovector[2*x], l); continue; } else { if (res.b < res.e) *res.b++ = *s; } } - str += pm[0].rm_eo; + str += ovector[1]; if (!all) break; - i = regexec(t, str, 10, pm, 0); - } while (i != REG_NOMATCH); + memset(&ovector, 0, sizeof(ovector)); + i = pcre_exec(t, NULL, str, strlen(str), 0, 0, ovector, 30); + } while (i != PCRE_ERROR_NOMATCH); /* Copy suffix to match */ l = strlen(str) + 1; diff --git a/varnish-cache/bin/varnishlog/Makefile.am b/varnish-cache/bin/varnishlog/Makefile.am index 85c8b92..2dbd279 100644 --- a/varnish-cache/bin/varnishlog/Makefile.am +++ b/varnish-cache/bin/varnishlog/Makefile.am @@ -1,6 +1,6 @@ # $Id$ -INCLUDES = -I$(top_srcdir)/include +INCLUDES = -I$(top_srcdir)/include @PCRE_CFLAGS@ bin_PROGRAMS = varnishlog @@ -13,3 +13,4 @@ varnishlog_LDADD = \ $(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \ $(top_builddir)/lib/libvarnishapi/libvarnishapi.la \ ${PTHREAD_LIBS} + @PCRE_LIBS@ diff --git a/varnish-cache/bin/varnishlog/varnishlog.c b/varnish-cache/bin/varnishlog/varnishlog.c index 4e6245a..4c11981 100644 --- a/varnish-cache/bin/varnishlog/varnishlog.c +++ b/varnish-cache/bin/varnishlog/varnishlog.c @@ -36,7 +36,7 @@ SVNID("$Id$") #include #include -#include +#include #include #include #include @@ -80,7 +80,7 @@ static enum shmlogtag last[65536]; #define F_MATCH (1 << 1) static int match_tag = -1; -static regex_t match_re; +static pcre *match_re; static void h_order_finish(int fd) @@ -116,6 +116,7 @@ h_order(void *priv, enum shmlogtag tag, unsigned fd, unsigned len, unsigned spec, const char *ptr) { char type; + int ovector[30]; (void)priv; @@ -132,7 +133,8 @@ h_order(void *priv, enum shmlogtag tag, unsigned fd, unsigned len, assert(ob[fd] != NULL); } if (tag == match_tag && - !regexec(&match_re, ptr, 0, NULL, 0)) + pcre_exec(match_re, NULL, ptr, len, 0, 0, ovector, + sizeof(ovector)/sizeof(ovector[0])) > 0) flg[fd] |= F_MATCH; if ((tag == SLT_BackendOpen || tag == SLT_SessionOpen || @@ -198,6 +200,8 @@ static void do_order(struct VSL_data *vd, int argc, char **argv) { int i; + const char *error; + int erroroffset; if (argc == 2) { match_tag = name2tag(argv[0]); @@ -205,11 +209,9 @@ do_order(struct VSL_data *vd, int argc, char **argv) fprintf(stderr, "Tag \"%s\" unknown\n", argv[0]); exit(2); } - i = regcomp(&match_re, argv[1], REG_EXTENDED | REG_NOSUB); - if (i) { - char buf[BUFSIZ]; - regerror(i, &match_re, buf, sizeof buf); - fprintf(stderr, "%s\n", buf); + match_re = pcre_compile(argv[1], 0, &error, &erroroffset, NULL); + if (match_re == NULL) { + fprintf(stderr, "Invalid regex: %s\n", error); exit(2); } } diff --git a/varnish-cache/configure.ac b/varnish-cache/configure.ac index 5c2d67f..4fc88a5 100644 --- a/varnish-cache/configure.ac +++ b/varnish-cache/configure.ac @@ -75,6 +75,37 @@ AC_SUBST(NET_LIBS) AC_CHECK_LIBM AC_SUBST(LIBM) +PKG_PROG_PKG_CONFIG +if test -n $PKG_CONFIG; then + PKG_CHECK_MODULES([PCRE], [libpcre]) +else + AC_CHECK_PROG(PCRE_CONFIG, pcre-config, pcre-config) + AC_ARG_WITH(pcre-config, + AS_HELP_STRING([--with-pcre-config=PATH], + [Location of PCRE pcre-config (auto)]), + [pcre_config="$withval"], + [pcre_config=""]) + + if test "x$pcre_config" != "x" ; then + AC_MSG_CHECKING(for $pcre_config) + + if test -f $pcre_config ; then + PCRE_CONFIG=$pcre_config + AC_MSG_RESULT(yes) + else + AC_MSG_RESULT(no - searching PATH) + fi + fi + if test "x$PCRE_CONFIG" = "x"; then + AC_CHECK_PROGS(PCRE_CONFIG, pcre-config) + fi + PCRE_CFLAGS=`$PCRE_CONFIG --cflags` + PCRE_LIBS=`$PCRE_CONFIG --libs` +fi +AC_SUBST(PCRE_CFLAGS) +AC_SUBST(PCRE_LIBS) + + # Checks for header files. AC_HEADER_STDC AC_HEADER_SYS_WAIT diff --git a/varnish-cache/include/vrt.h b/varnish-cache/include/vrt.h index 150db3c..17dbf25 100644 --- a/varnish-cache/include/vrt.h +++ b/varnish-cache/include/vrt.h @@ -136,7 +136,7 @@ struct vrt_ref { void VRT_acl_log(const struct sess *, const char *msg); /* Regexp related */ -void VRT_re_init(void **, const char *, int sub); +void VRT_re_init(void **, const char *); void VRT_re_fini(void *); int VRT_re_match(const char *, void *re); const char *VRT_regsub(const struct sess *sp, int all, const char *, diff --git a/varnish-cache/lib/libvarnishapi/Makefile.am b/varnish-cache/lib/libvarnishapi/Makefile.am index b109811..4210d46 100644 --- a/varnish-cache/lib/libvarnishapi/Makefile.am +++ b/varnish-cache/lib/libvarnishapi/Makefile.am @@ -1,6 +1,8 @@ # $Id$ -INCLUDES = -I$(top_srcdir)/include +INCLUDES = -I$(top_srcdir)/include @PCRE_CFLAGS@ + +LIBS = @PCRE_LIBS@ lib_LTLIBRARIES = libvarnishapi.la diff --git a/varnish-cache/lib/libvarnishapi/shmlog.c b/varnish-cache/lib/libvarnishapi/shmlog.c index 37be28a..78be38a 100644 --- a/varnish-cache/lib/libvarnishapi/shmlog.c +++ b/varnish-cache/lib/libvarnishapi/shmlog.c @@ -40,7 +40,7 @@ SVNID("$Id$") #include #include #include -#include +#include #include #include #include @@ -84,8 +84,8 @@ struct VSL_data { #define M_SELECT (1 << 3) int regflags; - regex_t *regincl; - regex_t *regexcl; + pcre *regincl; + pcre *regexcl; unsigned long skip; unsigned long keep; @@ -170,7 +170,7 @@ VSL_New(void) assert(VSL_S_BACKEND == M_BACKEND); vd = calloc(sizeof *vd, 1); assert(vd != NULL); - vd->regflags = REG_EXTENDED | REG_NOSUB; + vd->regflags = 0; vd->magic = VSL_MAGIC; vd->fd = -1; return (vd); @@ -272,9 +272,9 @@ int VSL_NextLog(struct VSL_data *vd, unsigned char **pp) { unsigned char *p; - regmatch_t rm; unsigned u, l; int i; + int ovector[30]; CHECK_OBJ_NOTNULL(vd, VSL_MAGIC); while (1) { @@ -315,19 +315,23 @@ VSL_NextLog(struct VSL_data *vd, unsigned char **pp) if (vd->c_opt && !(vd->map[u] & M_CLIENT)) continue; if (vd->regincl != NULL) { - rm.rm_so = 0; - rm.rm_eo = l; - i = regexec(vd->regincl, - (char *)p + SHMLOG_DATA, 1, &rm, 0); - if (i == REG_NOMATCH) + i = pcre_exec(vd->regincl, + NULL, /* We didn't study the pattern*/ + (char *)p + SHMLOG_DATA, + SHMLOG_LEN(p) - SHMLOG_DATA, /* Length */ + 0, 0, ovector, + sizeof(ovector)/sizeof(ovector[0])); + if (i == PCRE_ERROR_NOMATCH) continue; } if (vd->regexcl != NULL) { - rm.rm_so = 0; - rm.rm_eo = l; - i = regexec(vd->regexcl, - (char *)p + SHMLOG_DATA, 1, &rm, 0); - if (i != REG_NOMATCH) + i = pcre_exec(vd->regincl, + NULL, /* We didn't study the pattern*/ + (char *)p + SHMLOG_DATA, + SHMLOG_LEN(p) - SHMLOG_DATA, /* Length */ + 0, 0, ovector, + sizeof(ovector)/sizeof(ovector[0])); + if (i != PCRE_ERROR_NOMATCH) continue; } *pp = p; @@ -412,9 +416,9 @@ vsl_r_arg(struct VSL_data *vd, const char *opt) static int vsl_IX_arg(struct VSL_data *vd, const char *opt, int arg) { - int i; - regex_t **rp; - char buf[BUFSIZ]; + pcre **rp; + const char *error; + int erroroffset; CHECK_OBJ_NOTNULL(vd, VSL_MAGIC); if (arg == 'I') @@ -425,15 +429,9 @@ vsl_IX_arg(struct VSL_data *vd, const char *opt, int arg) fprintf(stderr, "Option %c can only be given once", arg); return (-1); } - *rp = calloc(sizeof(regex_t), 1); + *rp = pcre_compile(opt, vd->regflags, &error, &erroroffset, NULL); if (*rp == NULL) { - perror("malloc"); - return (-1); - } - i = regcomp(*rp, opt, vd->regflags); - if (i) { - regerror(i, *rp, buf, sizeof buf); - fprintf(stderr, "%s", buf); + fprintf(stderr, "Illegal regex: %s\n", error); return (-1); } return (1); @@ -547,7 +545,7 @@ VSL_Arg(struct VSL_data *vd, int arg, const char *opt) case 'i': case 'x': return (vsl_ix_arg(vd, opt, arg)); case 'r': return (vsl_r_arg(vd, opt)); case 'I': case 'X': return (vsl_IX_arg(vd, opt, arg)); - case 'C': vd->regflags = REG_ICASE; return (1); + case 'C': vd->regflags = PCRE_CASELESS; return (1); case 's': return (vsl_s_arg(vd, opt)); case 'k': return (vsl_k_arg(vd, opt)); default: diff --git a/varnish-cache/lib/libvcl/Makefile.am b/varnish-cache/lib/libvcl/Makefile.am index 4bb2e4b..6d98e43 100644 --- a/varnish-cache/lib/libvcl/Makefile.am +++ b/varnish-cache/lib/libvcl/Makefile.am @@ -1,6 +1,8 @@ # $Id$ -INCLUDES = -I$(top_srcdir)/include +INCLUDES = -I$(top_srcdir)/include @PCRE_CFLAGS@ + +LDADD = @PCRE_LIBS@ lib_LTLIBRARIES = libvcl.la diff --git a/varnish-cache/lib/libvcl/vcc_compile.h b/varnish-cache/lib/libvcl/vcc_compile.h index 256da13..be44bfb 100644 --- a/varnish-cache/lib/libvcl/vcc_compile.h +++ b/varnish-cache/lib/libvcl/vcc_compile.h @@ -198,7 +198,7 @@ unsigned vcc_UintVal(struct tokenlist *tl); double vcc_DoubleVal(struct tokenlist *tl); /* vcc_string.c */ -char *vcc_regexp(struct tokenlist *tl, int sub); +char *vcc_regexp(struct tokenlist *tl); int vcc_StringVal(struct tokenlist *tl); void vcc_ExpectedStringval(struct tokenlist *tl); diff --git a/varnish-cache/lib/libvcl/vcc_parse.c b/varnish-cache/lib/libvcl/vcc_parse.c index d6079f2..ef2b09a 100644 --- a/varnish-cache/lib/libvcl/vcc_parse.c +++ b/varnish-cache/lib/libvcl/vcc_parse.c @@ -225,7 +225,7 @@ Cond_String(const struct var *vp, struct tokenlist *tl) tl->t->tok == '~' ? "" : "!"); vcc_NextToken(tl); ExpectErr(tl, CSTR); - p = vcc_regexp(tl, 0); + p = vcc_regexp(tl); ERRCHK(tl); vcc_NextToken(tl); Fb(tl, 1, "%s, %s)\n", vp->rname, p); diff --git a/varnish-cache/lib/libvcl/vcc_string.c b/varnish-cache/lib/libvcl/vcc_string.c index 564e3da..7fe34b7 100644 --- a/varnish-cache/lib/libvcl/vcc_string.c +++ b/varnish-cache/lib/libvcl/vcc_string.c @@ -34,7 +34,7 @@ SVNID("$Id$") #include #include -#include +#include #include "vsb.h" @@ -47,26 +47,25 @@ SVNID("$Id$") /*--------------------------------------------------------------------*/ char * -vcc_regexp(struct tokenlist *tl, int sub) +vcc_regexp(struct tokenlist *tl) { char buf[BUFSIZ], *p; - regex_t t; - int i; + pcre *t; + const char *error; + int erroroffset; Expect(tl, CSTR); if (tl->err) return (NULL); memset(&t, 0, sizeof t); - i = regcomp(&t, tl->t->dec, REG_EXTENDED | (sub ? 0 : REG_NOSUB)); - if (i != 0) { - (void)regerror(i, &t, buf, sizeof buf); + t = pcre_compile(tl->t->dec, 0, &error, &erroroffset, NULL); + if (t == NULL) { vsb_printf(tl->sb, - "Regexp compilation error:\n\n%s\n\n", buf); + "Regexp compilation error:\n\n%s\n\n", error); vcc_ErrWhere(tl, tl->t); - regfree(&t); return (NULL); } - regfree(&t); + pcre_free(t); sprintf(buf, "VGC_re_%u", tl->recnt++); p = TlAlloc(tl, strlen(buf) + 1); strcpy(p, buf); @@ -74,7 +73,7 @@ vcc_regexp(struct tokenlist *tl, int sub) Fh(tl, 0, "static void *%s;\n", buf); Fi(tl, 0, "\tVRT_re_init(&%s, ",buf); EncToken(tl->fi, tl->t); - Fi(tl, 0, ", %d);\n", sub); + Fi(tl, 0, ");\n"); Ff(tl, 0, "\tVRT_re_fini(%s);\n", buf); return (p); } @@ -108,7 +107,7 @@ vcc_regsub(struct tokenlist *tl, int all) Expect(tl, CSTR); if (tl->err) return (0); - p = vcc_regexp(tl, 1); + p = vcc_regexp(tl); vcc_NextToken(tl); Fb(tl, 0, ", %s, ", p); -- 1.6.3.3 -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From des at des.no Fri Oct 16 21:54:32 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 16 Oct 2009 23:54:32 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <87skdj5sk3.fsf@qurzaw.linpro.no> (Tollef Fog Heen's message of "Fri, 16 Oct 2009 16:20:28 +0200") References: <87skdj5sk3.fsf@qurzaw.linpro.no> Message-ID: <863a5j57jb.fsf@ds4.des.no> Tollef Fog Heen writes: > We no longer use POSIX style regexes, but rather PCRE regexes. This > introduces a hard dependency on libpcre. This is a radical departure from one of the basic design objectives for Varnish, which was that it should be possible to compile it with nothing more than a C compiler and headers for a more or less POSIX-compatible C library. DES -- Dag-Erling Sm?rgrav - des at des.no From aotto at mosso.com Fri Oct 16 22:08:25 2009 From: aotto at mosso.com (Adrian Otto) Date: Fri, 16 Oct 2009 15:08:25 -0700 Subject: [PATCH] Add PCRE support In-Reply-To: <863a5j57jb.fsf@ds4.des.no> References: <87skdj5sk3.fsf@qurzaw.linpro.no> <863a5j57jb.fsf@ds4.des.no> Message-ID: <8467848E-92E1-4F10-BEA2-14C856F10920@mosso.com> DES, I suggest making it an optional ./configure option. That way you could still build it the old way if you wanted. From my perspective the value of the feature would justify having two regex implementations in varnish. Make the default the current one. You can tell by looking at the patch that regex is not used extensively anyway so it's not likely to matter much. It's not a lot of different code. Adrian On Oct 16, 2009, at 2:54 PM, Dag-Erling Sm?rgrav wrote: > Tollef Fog Heen writes: >> We no longer use POSIX style regexes, but rather PCRE regexes. This >> introduces a hard dependency on libpcre. > > This is a radical departure from one of the basic design objectives > for > Varnish, which was that it should be possible to compile it with > nothing > more than a C compiler and headers for a more or less POSIX- > compatible C > library. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev From tfheen at redpill-linpro.com Tue Oct 20 07:53:36 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 20 Oct 2009 09:53:36 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <863a5j57jb.fsf@ds4.des.no> ("Dag-Erling =?utf-8?Q?Sm=C3=B8rg?= =?utf-8?Q?rav=22's?= message of "Fri, 16 Oct 2009 23:54:32 +0200") References: <87skdj5sk3.fsf@qurzaw.linpro.no> <863a5j57jb.fsf@ds4.des.no> Message-ID: <87fx9ev6v3.fsf@qurzaw.linpro.no> ]] Dag-Erling Sm?rgrav | Tollef Fog Heen writes: | > We no longer use POSIX style regexes, but rather PCRE regexes. This | > introduces a hard dependency on libpcre. | | This is a radical departure from one of the basic design objectives for | Varnish, which was that it should be possible to compile it with nothing | more than a C compiler and headers for a more or less POSIX-compatible C | library. I don't agree it's a radical departure, but I'd rather not end up in a straw-splitting discussion about that. :-) It's something that we agreed on doing during the Varnish user group meeting (as documented on http://varnish.projects.linpro.no/wiki/VUG1Report), as well as having been discussed extensively on IRC already. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Tue Oct 20 07:55:12 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 20 Oct 2009 09:55:12 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <8467848E-92E1-4F10-BEA2-14C856F10920@mosso.com> (Adrian Otto's message of "Fri, 16 Oct 2009 15:08:25 -0700") References: <87skdj5sk3.fsf@qurzaw.linpro.no> <863a5j57jb.fsf@ds4.des.no> <8467848E-92E1-4F10-BEA2-14C856F10920@mosso.com> Message-ID: <87bpk2v6sf.fsf@qurzaw.linpro.no> ]] Adrian Otto | I suggest making it an optional ./configure option. That way you could | still build it the old way if you wanted. No, having to support two different styles of regular expression is going to be a problem documentation- and support-wise. I'm strongly against that and would rather just stick with POSIX than to have both. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From ask at develooper.com Tue Oct 20 08:06:36 2009 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Tue, 20 Oct 2009 01:06:36 -0700 Subject: [PATCH] Add PCRE support In-Reply-To: <87skdj5sk3.fsf@qurzaw.linpro.no> References: <87skdj5sk3.fsf@qurzaw.linpro.no> Message-ID: c On Oct 16, 2009, at 7:20, Tollef Fog Heen wrote: > We no longer use POSIX style regexes, but rather PCRE regexes. This > introduces a hard dependency on libpcre. For a small project[1] I was working on today I've been cursing the POSIX regular expressions endlessly, so I'd welcome this patch in a future version of Varnish. - ask [1] http://ipv64.v6test.develooper.com/c/ip?callback=some_callback_123 - I'm testing if users can reach sites with only AAAA or AAAA+A records. I have varnish generate a bit of Javascript with the user IP address so the backend application only needs to get hit when the Javascript has collected all the data. -- http://develooper.com/ - http://askask.com/ From jos at dwim.org Tue Oct 20 08:41:40 2009 From: jos at dwim.org (Jos I. Boumans) Date: Tue, 20 Oct 2009 10:41:40 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <87fx9ev6v3.fsf@qurzaw.linpro.no> References: <87skdj5sk3.fsf@qurzaw.linpro.no> <863a5j57jb.fsf@ds4.des.no> <87fx9ev6v3.fsf@qurzaw.linpro.no> Message-ID: <9CAA6245-D239-4D32-83D0-E4DB30B147A9@dwim.org> Greetings, On Oct 20, 2009, at 9:53 AM, Tollef Fog Heen wrote: > ]] Dag-Erling Sm?rgrav > > | Tollef Fog Heen writes: > | > We no longer use POSIX style regexes, but rather PCRE regexes. > This > | > introduces a hard dependency on libpcre. > | > | This is a radical departure from one of the basic design > objectives for > | Varnish, which was that it should be possible to compile it with > nothing > | more than a C compiler and headers for a more or less POSIX- > compatible C > | library. > > I don't agree it's a radical departure, but I'd rather not end up in a > straw-splitting discussion about that. :-) > > It's something that we agreed on doing during the Varnish user group > meeting (as documented on > http://varnish.projects.linpro.no/wiki/VUG1Report), as well as having > been discussed extensively on IRC already. It indeed adds an external dependency, but I'm not quite understanding why you may think that's a bad thing. Are you worried about systems that aren't able to provide/build libpcre and stop varnish from working? Or are you worried about third party code that may introduce bugs or security issues? Or is there perhaps another reason I hadn't thought of? For all those at the VUG, the use and benefit of PCRE over POSIX regexes was quite apparent at least, and surely we don't want to implement our own version of PCRE. But perhaps there's a risk we didn't see. Would you care to elaborate? -- Jos Boumans "Whenever you find you are on the side of the majority, it is time to pause and reflect." - Mark Twain From des at des.no Tue Oct 20 09:39:42 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 20 Oct 2009 11:39:42 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <9CAA6245-D239-4D32-83D0-E4DB30B147A9@dwim.org> (Jos I. Boumans's message of "Tue, 20 Oct 2009 10:41:40 +0200") References: <87skdj5sk3.fsf@qurzaw.linpro.no> <863a5j57jb.fsf@ds4.des.no> <87fx9ev6v3.fsf@qurzaw.linpro.no> <9CAA6245-D239-4D32-83D0-E4DB30B147A9@dwim.org> Message-ID: <86zl7mtndt.fsf@ds4.des.no> "Jos I. Boumans" writes: > It indeed adds an external dependency, but I'm not quite understanding > why you may think that's a bad thing. Like I said: "it should be possible to compile [Varnish] with nothing more than a C compiler and headers for a more or less POSIX-compatible C library". Don't assume everyone will use a precompiled Varnish package provided by their distro. In fact, assume the opposite: assume the average user is running an LTS release for which there is no up-to-date Varnish package, and it's two in the morning, and they need it up and running NOW. Download, configure, make install. An experienced Debian / Ubuntu admin knows that he needs to apt-get install build-essential, but will not enjoy seeing the build fail and having to read and understand compiler messages and (hopefully) figure out that he also needs to apt-get install libpcre-dev. It's even worse on the BSDs, where prior to this change you can build and install Varnish on a fresh stock install with no other ports / packages installed. This is why some of the tools won't be built if you don't have ncurses (or to be precise the ncurses headers) installed. DES -- Dag-Erling Sm?rgrav - des at des.no From phk at phk.freebsd.dk Tue Oct 20 10:04:29 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Oct 2009 10:04:29 +0000 Subject: [PATCH] Add PCRE support In-Reply-To: Your message of "Tue, 20 Oct 2009 11:39:42 +0200." <86zl7mtndt.fsf@ds4.des.no> Message-ID: <50615.1256033069@critter.freebsd.dk> I think it is time to wrap this discussion up. Being as it may, that I happen to be the supreme master and arbiter of all bikeshed coloring in the Open Source domain. (Please see www.bikeshed.org if you doubt my credencials.) First of all, I my spectroscope clearly indicates that the discussion is in bikeshed territory now, reading lambda = 536 +/- 107 nm I will therefore don my Official Psycedelic Overalls, and wielding the Brush Of Authority, pronounce my judgment in the matter. Which shall be as follows: Having in mind, the many examples that Varnish users have proffered, why PCRE would be A Good Thing over POSIX regexps... Beeing keenly aware that having binary packages of Varnish differ in this point would be a catastrophe for documentation and user support. Having studied PCRE at some length, and reached the firm belief, that if Dennis and Ken had been in charge, PCRE would have been adopted as the standard UNIX regexp library on the spot... Recognizing, that PCRE is BSD licensed... Having in mind, that if the external dependency on PCRE causes us grief, we can import PCRE into Varnish, like we did with and ... I hereby decide: We will add a "vre.c" file to libvarnish, and all the varnish code will call vre_comp(), vre_exec() etc. functions from there. By default, this file will simply call PCRE functions, and as shipped Varnish will depend in PCRE. If somebody for reasons of policy does not want to use PCRE, they will only have to change the vre.c file to do so. Dixit. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tfheen at redpill-linpro.com Tue Oct 20 12:19:39 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 20 Oct 2009 14:19:39 +0200 Subject: [PATCH] Add PCRE support In-Reply-To: <50615.1256033069@critter.freebsd.dk> (Poul-Henning Kamp's message of "Tue, 20 Oct 2009 10:04:29 +0000") References: <50615.1256033069@critter.freebsd.dk> Message-ID: <873a5euujo.fsf@qurzaw.linpro.no> ]] "Poul-Henning Kamp" | We will add a "vre.c" file to libvarnish, and all the varnish | code will call vre_comp(), vre_exec() etc. functions from | there. | | By default, this file will simply call PCRE functions, and | as shipped Varnish will depend in PCRE. | | If somebody for reasons of policy does not want to use PCRE, | they will only have to change the vre.c file to do so. I've done that in the attached patch. If somebody could please read through it an ack it, that would be most appreciated. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 -------------- next part -------------- A non-text attachment was scrubbed... Name: pcre2.patch Type: text/x-diff Size: 22071 bytes Desc: not available URL: From kristian at redpill-linpro.com Wed Oct 21 08:37:35 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 21 Oct 2009 10:37:35 +0200 Subject: Saintmode in 2.0 Message-ID: <20091021083735.GB4527@kjeks.linpro.no> Revisions: 4208 4328 4335 Should be included in 2.0. The latter two adds a paramter which controls how long the saintmode list has to be until the backend is considered sick, and it also makes it possible to disable saintmode handling entirely. The first is beresp.saintmode, which I suppose you have to rename, but otherwise it should apply with very little hassle. In addition, I have a patch awaiting PHK-approval to add saintmode_threshold to the backend declaration too, which should probably be included too when it's accepted. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From izak at upfrontsystems.co.za Wed Oct 21 09:33:48 2009 From: izak at upfrontsystems.co.za (Izak Burger) Date: Wed, 21 Oct 2009 11:33:48 +0200 Subject: Backend polling not working In-Reply-To: <4AD6F97C.0@upfrontsystems.co.za> References: <32970.1255545292@critter.freebsd.dk> <4AD61BC5.1040608@upfrontsystems.co.za> <4AD629C2.5050905@upfrontsystems.co.za> <4AD6C070.20100@upfrontsystems.co.za> <4AD6D36E.9060704@upfrontsystems.co.za> <4AD6F97C.0@upfrontsystems.co.za> Message-ID: <4ADED57C.3020004@upfrontsystems.co.za> Hello again, I've discussed this issue on zope-dev. The agreement seems to be that there is some ambiguity in the HTTP1.1 specification, but that the general accepted interpretation is that if the other end closes the connection you should close your end as well (it signals a timeout). There is also no browser implementation out there that does half-closing (except where SSL is involved). None that I know of anyway. https://mail.zope.org/pipermail/zope-dev/2009-October/038133.html This also affects nginx. I would therefore like to submit a wishlist item to make this configurable. I've already hacked the offending shutdown() call out of the code to test, and it works perfectly with zope once you do that. If there are no objections, can I go ahead and log this as a wish/feature request on the bugtracker? regards, Izak From phk at phk.freebsd.dk Wed Oct 21 09:45:10 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 21 Oct 2009 09:45:10 +0000 Subject: Backend polling not working In-Reply-To: Your message of "Wed, 21 Oct 2009 11:33:48 +0200." <4ADED57C.3020004@upfrontsystems.co.za> Message-ID: <90463.1256118310@critter.freebsd.dk> In message <4ADED57C.3020004 at upfrontsystems.co.za>, Izak Burger writes: >Hello again, > >I've discussed this issue on zope-dev. The agreement seems to be that >there is some ambiguity in the HTTP1.1 specification, but that the >general accepted interpretation is that if the other end closes the >connection you should close your end as well (it signals a timeout). > >There is also no browser implementation out there that does half-closing >(except where SSL is involved). None that I know of anyway. I wasn't quite sure which way to interpret the text originally so I did it the way that was most convenient for the code. I'll change it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jfrias at gmail.com Tue Oct 27 17:42:14 2009 From: jfrias at gmail.com (Javier Frias) Date: Tue, 27 Oct 2009 13:42:14 -0400 Subject: Patch adding more Apache LogFormat support to varnishncsa ( based on previous patch ) Message-ID: <22964b960910271042u6419ad11l5fde89f84651e40c@mail.gmail.com> Figured some people will find this useful, until varnishncsa gets full LogFormat support. I've expanded Emil Isberg's initial path on the following items: List of what I added/fixed: - Fixed segfault in %s,%>s, when these where not found in shmlog - Fixed segfault in %{VarnishR}i, when df_H, Protocol version, was not found, http defaults to HTTP/1.0 - Added %D, ( proc_time + xmit_time ) in microseconds - Added $T, same as above, but in seconds - Added %V, faked as %{Host}i ( different meaning in apache, but usually meant as vhost ) - Added %{Age}i, age of object being served. Quite useful, as you can graph things like average age of served pages, cache hit ratios, etc etc. I attach a diff against stock 2.0.4 varnish, I've tested this against several multi-gigabyte varnish native logs, as well as have it run for a few days now, with no issues. -Javier -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishncsa-logformat-newfullpatch.diff Type: text/x-patch Size: 9306 bytes Desc: not available URL: