From des at projects.linpro.no Thu Nov 1 21:48:01 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 1 Nov 2007 22:48:01 +0100 (CET) Subject: r2222 - trunk/varnish-cache Message-ID: <20071101214801.90F071EC030@projects.linpro.no> Author: des Date: 2007-11-01 22:47:58 +0100 (Thu, 01 Nov 2007) New Revision: 2222 Modified: trunk/varnish-cache/autogen.sh Log: Add Solaris support. Avoid kshisms. Modified: trunk/varnish-cache/autogen.sh =================================================================== --- trunk/varnish-cache/autogen.sh 2007-10-30 14:41:11 UTC (rev 2221) +++ trunk/varnish-cache/autogen.sh 2007-11-01 21:47:58 UTC (rev 2222) @@ -17,12 +17,15 @@ Linux) LIBTOOLIZE=libtoolize ;; +SunOS) + LIBTOOLIZE=libtoolize + ;; *) warn "unrecognized platform:" `uname -s` LIBTOOLIZE=libtoolize esac -automake_version=$(automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+') +automake_version=`automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+'` if [ -z "$automake_version" ] ; then warn "unable to determine automake version" else From des at projects.linpro.no Thu Nov 1 21:48:55 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 1 Nov 2007 22:48:55 +0100 (CET) Subject: r2223 - in trunk/varnish-cache/lib: libvarnish libvarnishcompat Message-ID: <20071101214855.4E5F61EC2BA@projects.linpro.no> Author: des Date: 2007-11-01 22:48:54 +0100 (Thu, 01 Nov 2007) New Revision: 2223 Modified: trunk/varnish-cache/lib/libvarnish/vsb.c trunk/varnish-cache/lib/libvarnishcompat/strndup.c Log: Add missing Modified: trunk/varnish-cache/lib/libvarnish/vsb.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/vsb.c 2007-11-01 21:47:58 UTC (rev 2222) +++ trunk/varnish-cache/lib/libvarnish/vsb.c 2007-11-01 21:48:54 UTC (rev 2223) @@ -34,6 +34,7 @@ #include #include #include +#include #include "vsb.h" Modified: trunk/varnish-cache/lib/libvarnishcompat/strndup.c =================================================================== --- trunk/varnish-cache/lib/libvarnishcompat/strndup.c 2007-11-01 21:47:58 UTC (rev 2222) +++ trunk/varnish-cache/lib/libvarnishcompat/strndup.c 2007-11-01 21:48:54 UTC (rev 2223) @@ -34,6 +34,7 @@ #include #include +#include #ifndef HAVE_STRLCPY #include "compat/strlcpy.h" From des at projects.linpro.no Thu Nov 1 21:52:34 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 1 Nov 2007 22:52:34 +0100 (CET) Subject: r2224 - trunk/varnish-cache/lib/libvarnishcompat Message-ID: <20071101215234.D582B1ECF28@projects.linpro.no> Author: des Date: 2007-11-01 22:52:34 +0100 (Thu, 01 Nov 2007) New Revision: 2224 Modified: trunk/varnish-cache/lib/libvarnishcompat/daemon.c Log: Don't rely on non-portable Modified: trunk/varnish-cache/lib/libvarnishcompat/daemon.c =================================================================== --- trunk/varnish-cache/lib/libvarnishcompat/daemon.c 2007-11-01 21:48:54 UTC (rev 2223) +++ trunk/varnish-cache/lib/libvarnishcompat/daemon.c 2007-11-01 21:52:34 UTC (rev 2224) @@ -35,7 +35,6 @@ #include #include -#include #include #include #include @@ -79,7 +78,7 @@ if (!nochdir) (void)chdir("/"); - if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) { + if (!noclose && (fd = open("/dev/null", O_RDWR, 0)) != -1) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); (void)dup2(fd, STDERR_FILENO); From des at projects.linpro.no Thu Nov 1 21:53:59 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 1 Nov 2007 22:53:59 +0100 (CET) Subject: r2225 - trunk/varnish-cache/bin/varnishd Message-ID: <20071101215359.719611EC528@projects.linpro.no> Author: des Date: 2007-11-01 22:53:59 +0100 (Thu, 01 Nov 2007) New Revision: 2225 Modified: trunk/varnish-cache/bin/varnishd/cache_http.c Log: Header cleanup. Modified: trunk/varnish-cache/bin/varnishd/cache_http.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_http.c 2007-11-01 21:52:34 UTC (rev 2224) +++ trunk/varnish-cache/bin/varnishd/cache_http.c 2007-11-01 21:53:59 UTC (rev 2225) @@ -31,13 +31,14 @@ * HTTP request storage and manipulation */ -#include +#include #include #include #include +#include #include #include -#include +#include #include "heritage.h" #include "shmlog.h" From des at projects.linpro.no Thu Nov 1 21:57:44 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 1 Nov 2007 22:57:44 +0100 (CET) Subject: r2226 - trunk/varnish-cache/bin/varnishd Message-ID: <20071101215744.AFB981EC51C@projects.linpro.no> Author: des Date: 2007-11-01 22:57:44 +0100 (Thu, 01 Nov 2007) New Revision: 2226 Modified: trunk/varnish-cache/bin/varnishd/storage_file.c Log: Fix the struct statvfs issue. Modified: trunk/varnish-cache/bin/varnishd/storage_file.c =================================================================== --- trunk/varnish-cache/bin/varnishd/storage_file.c 2007-11-01 21:53:59 UTC (rev 2225) +++ trunk/varnish-cache/bin/varnishd/storage_file.c 2007-11-01 21:57:44 UTC (rev 2226) @@ -33,7 +33,6 @@ #include #include -#include #include #include @@ -122,7 +121,13 @@ static void smf_calcsize(struct smf_sc *sc, const char *size, int newfile) { +#if defined(HAVE_SYS_STATVFS_H) + struct statvfs fsst; +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) struct statfs fsst; +#else +#error no struct statfs / struct statvfs +#endif uintmax_t l, fssize; unsigned bs; char suff[2]; @@ -134,7 +139,13 @@ AZ(fstat(sc->fd, &st)); xxxassert(S_ISREG(st.st_mode)); +#if defined(HAVE_SYS_STATVFS_H) + AZ(fstatvfs(sc->fd, &fsst)); +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) AZ(fstatfs(sc->fd, &fsst)); +#else +#error no struct statfs / struct statvfs +#endif /* We use units of the larger of filesystem blocksize and pagesize */ bs = sc->pagesize; From des at projects.linpro.no Fri Nov 2 14:09:20 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 2 Nov 2007 15:09:20 +0100 (CET) Subject: r2227 - in trunk/varnish-cache: . bin/varnishd Message-ID: <20071102140920.6B24C1EC51E@projects.linpro.no> Author: des Date: 2007-11-02 15:09:17 +0100 (Fri, 02 Nov 2007) New Revision: 2227 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c trunk/varnish-cache/configure.ac Log: Test whether SO_{RCV,SND}TIMEO work (they are defined but unimplemented on Solaris, so simply checking for their presence is not enough) Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2007-11-01 21:57:44 UTC (rev 2226) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor.c 2007-11-02 14:09:17 UTC (rev 2227) @@ -90,17 +90,22 @@ if (memcmp(&lin, &linger, l)) need_linger = 1; +#ifdef SO_SNDTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_sndtimeo, l)) need_sndtimeo = 1; +#endif +#ifdef SO_RCVTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_rcvtimeo, l)) need_rcvtimeo = 1; +#endif + need_test = 0; } @@ -122,12 +127,16 @@ if (need_linger) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER, &linger, sizeof linger)); +#ifdef SO_SNDTIMEO_WORKS if (need_sndtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); +#endif +#ifdef SO_RCVTIMEO_WORKS if (need_rcvtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); +#endif } /*--------------------------------------------------------------------*/ @@ -160,6 +169,7 @@ need_test = 1; while (1) { +#ifdef SO_SNDTIMEO_WORKS if (params->send_timeout != tv_sndtimeo.tv_sec) { need_test = 1; tv_sndtimeo.tv_sec = params->send_timeout; @@ -168,6 +178,8 @@ SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); } +#endif +#ifdef SO_RCVTIMEO_WORKS if (params->sess_timeout != tv_rcvtimeo.tv_sec) { need_test = 1; tv_rcvtimeo.tv_sec = params->sess_timeout; @@ -176,6 +188,7 @@ SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); } +#endif i = poll(pfd, heritage.nsocks, 1000); now = TIM_real(); for (u = 0; u < heritage.nsocks; u++) { Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2007-11-01 21:57:44 UTC (rev 2226) +++ trunk/varnish-cache/configure.ac 2007-11-02 14:09:17 UTC (rev 2227) @@ -115,6 +115,45 @@ AC_CHECK_FUNCS([epoll_ctl]) AC_CHECK_FUNCS([poll]) +# Solaris defines SO_{RCV,SND}TIMEO, but does not implement them +AC_CACHE_CHECK([whether SO_RCVTIMEO works], + [ac_cv_so_rcvtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_rcvtimeo_works=yes], + [ac_cv_so_rcvtime_works=no]) + ]) +if test "$ac_cv_so_rcvtimeo_works" = yes; then + AC_DEFINE([SO_RCVTIMEO_WORKS], [1], [Define if SO_RCVTIMEO works]) +fi + +AC_CACHE_CHECK([whether SO_SNDTIMEO works], + [ac_cv_so_sndtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_sndtimeo_works=yes], + [ac_cv_so_sndtime_works=no]) + ]) +if test "$ac_cv_so_sndtimeo_works" = yes; then + AC_DEFINE([SO_SNDTIMEO_WORKS], [1], [Define if SO_SNDTIMEO works]) +fi + # Run-time directory VARNISH_STATE_DIR='${localstatedir}/varnish' AC_SUBST(VARNISH_STATE_DIR) From phk at phk.freebsd.dk Fri Nov 2 17:03:37 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 02 Nov 2007 17:03:37 +0000 Subject: r2227 - in trunk/varnish-cache: . bin/varnishd In-Reply-To: Your message of "Fri, 02 Nov 2007 15:09:20 +0100." <20071102140920.6B24C1EC51E@projects.linpro.no> Message-ID: <3917.1194023017@critter.freebsd.dk> In message <20071102140920.6B24C1EC51E at projects.linpro.no>, des at projects.linpro .no writes: >Author: des >Date: 2007-11-02 15:09:17 +0100 (Fri, 02 Nov 2007) >New Revision: 2227 > >Modified: > trunk/varnish-cache/bin/varnishd/cache_acceptor.c > trunk/varnish-cache/configure.ac >Log: >Test whether SO_{RCV,SND}TIMEO work (they are defined but unimplemented >on Solaris, so simply checking for their presence is not enough) That is not enough, if we don't have them, we need to have timeouts on writes and reads to emulate them :-( -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at projects.linpro.no Sat Nov 3 12:30:07 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Sat, 3 Nov 2007 13:30:07 +0100 (CET) Subject: r2228 - trunk/varnish-cache Message-ID: <20071103123007.6CCBF1EC030@projects.linpro.no> Author: des Date: 2007-11-03 13:30:06 +0100 (Sat, 03 Nov 2007) New Revision: 2228 Modified: trunk/varnish-cache/configure.ac Log: Typo in previous commit. Modified: trunk/varnish-cache/configure.ac =================================================================== --- trunk/varnish-cache/configure.ac 2007-11-02 14:09:17 UTC (rev 2227) +++ trunk/varnish-cache/configure.ac 2007-11-03 12:30:06 UTC (rev 2228) @@ -129,7 +129,7 @@ return !!setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); ]])], [ac_cv_so_rcvtimeo_works=yes], - [ac_cv_so_rcvtime_works=no]) + [ac_cv_so_rcvtimeo_works=no]) ]) if test "$ac_cv_so_rcvtimeo_works" = yes; then AC_DEFINE([SO_RCVTIMEO_WORKS], [1], [Define if SO_RCVTIMEO works]) @@ -148,7 +148,7 @@ return !!setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv); ]])], [ac_cv_so_sndtimeo_works=yes], - [ac_cv_so_sndtime_works=no]) + [ac_cv_so_sndtimeo_works=no]) ]) if test "$ac_cv_so_sndtimeo_works" = yes; then AC_DEFINE([SO_SNDTIMEO_WORKS], [1], [Define if SO_SNDTIMEO works]) From phk at phk.freebsd.dk Sun Nov 4 10:49:20 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 04 Nov 2007 10:49:20 +0000 Subject: r2180 - trunk/varnish-cache/lib/libvarnishapi In-Reply-To: Your message of "Mon, 29 Oct 2007 16:44:47 +0100." Message-ID: <13405.1194173360@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?= w rites: >"Poul-Henning Kamp" writes: >> "Dag-Erling Sm=F8rgrav" writes: >> > A cast would be more appropriate... and we should probably use >> > isprint() instead of hardcoding (*ptr >=3D ' ' && *ptr <=3D '~'). >> A cast we can discuss, but isprint() would be wrong because it would >> take the locale into consideration and HTTP is specifically specified >> in ASCII with no extension. > >We already force the time zone to UTC, there is no reason not to force >the locale to "C". > >Alternatively, we can provide our own ASCII-only implementation - I >really don't like (*ptr >=3D ' ' && *ptr <=3D '~'). I guess, strictly speaking that should be 0x20 and 0x7e... :-) I actually have the beginning of a HTTP-oriented ctype over in cache_http.c and I have been thinking about adding a couple of bits more and use that throughout. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at projects.linpro.no Thu Nov 8 16:27:53 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 17:27:53 +0100 (CET) Subject: r2229 - trunk/varnish-cache/bin/varnishlog Message-ID: <20071108162753.BB0511EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 17:27:53 +0100 (Thu, 08 Nov 2007) New Revision: 2229 Modified: trunk/varnish-cache/bin/varnishlog/varnishlog.1 trunk/varnish-cache/bin/varnishlog/varnishlog.c Log: Pick some nits. Modified: trunk/varnish-cache/bin/varnishlog/varnishlog.1 =================================================================== --- trunk/varnish-cache/bin/varnishlog/varnishlog.1 2007-11-03 12:30:06 UTC (rev 2228) +++ trunk/varnish-cache/bin/varnishlog/varnishlog.1 2007-11-08 16:27:53 UTC (rev 2229) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHLOG 1 .Os .Sh NAME @@ -44,9 +44,9 @@ .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag +.Op Fl n Ar varnish_name .Op Fl o .Op Fl P Ar file -.Op Fl n Ar varnish_name .Op Fl r Ar file .Op Fl V .Op Fl w Ar file Modified: trunk/varnish-cache/bin/varnishlog/varnishlog.c =================================================================== --- trunk/varnish-cache/bin/varnishlog/varnishlog.c 2007-11-03 12:30:06 UTC (rev 2228) +++ trunk/varnish-cache/bin/varnishlog/varnishlog.c 2007-11-08 16:27:53 UTC (rev 2229) @@ -283,9 +283,9 @@ { int i, c; int a_flag = 0, D_flag = 0, o_flag = 0; + const char *n_arg = NULL; const char *P_arg = NULL; const char *w_arg = NULL; - const char *n_arg = NULL; struct pidfh *pfh = NULL; struct VSL_data *vd; From des at projects.linpro.no Thu Nov 8 16:28:26 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 17:28:26 +0100 (CET) Subject: r2230 - trunk/varnish-cache/bin/varnishncsa Message-ID: <20071108162826.BC65D1EC46F@projects.linpro.no> Author: des Date: 2007-11-08 17:28:26 +0100 (Thu, 08 Nov 2007) New Revision: 2230 Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 trunk/varnish-cache/bin/varnishncsa/varnishncsa.c Log: Copy&paste -D (daemonize) from varnishlog. Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 =================================================================== --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 2007-11-08 16:27:53 UTC (rev 2229) +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 2007-11-08 16:28:26 UTC (rev 2230) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHNCSA 1 .Os .Sh NAME @@ -40,10 +40,12 @@ .Op Fl b .Op Fl C .Op Fl c +.Op Fl D .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag .Op Fl n Ar varnish_name +.Op Fl P Ar file .Op Fl r Ar file .Op Fl V .Op Fl w Ar file @@ -82,6 +84,8 @@ is specified, .Nm acts as if they both were. +.It Fl D +Daemonize. .It Fl d Process old log entries on startup. Normally, @@ -109,6 +113,9 @@ If .Fl n is not specified, the host name is used. +.It Fl P Ar file +Write the process's PID to the specified +.Ar file . .It Fl r Ar file Read log entries from .Ar file Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c =================================================================== --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-08 16:27:53 UTC (rev 2229) +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-08 16:28:26 UTC (rev 2230) @@ -69,10 +69,16 @@ #include #include +#ifndef HAVE_DAEMON +#include "compat/daemon.h" +#endif + +#include "vsb.h" +#include "vpf.h" + #include "libvarnish.h" #include "shmlog.h" #include "varnishapi.h" -#include "vsb.h" static struct logline { char *df_H; /* %H, Protocol version */ @@ -434,7 +440,7 @@ usage(void) { - fprintf(stderr, "usage: varnishncsa %s [-aV] [-n varnish_name] [-w file]\n", VSL_ARGS); + fprintf(stderr, "usage: varnishncsa %s [-aDV] [-n varnish_name] [-P file] [-w file]\n", VSL_ARGS); exit(1); } @@ -442,15 +448,17 @@ main(int argc, char *argv[]) { int i, c; + int a_flag = 0, D_flag = 0; + const char *n_arg = NULL; + const char *P_arg = NULL; + const char *w_arg = NULL; + struct pidfh *pfh = NULL; struct VSL_data *vd; - const char *ofn = NULL; - const char *n_arg = NULL; - int append = 0; FILE *of; vd = VSL_New(); - while ((c = getopt(argc, argv, VSL_ARGS "an:Vw:")) != -1) { + while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { i = VSL_Arg(vd, c, optarg); if (i < 0) exit (1); @@ -458,16 +466,22 @@ continue; switch (c) { case 'a': - append = 1; + a_flag = 1; break; + case 'D': + D_flag = 1; + break; case 'n': n_arg = optarg; break; + case 'P': + P_arg = optarg; + break; case 'V': varnish_version("varnishncsa"); exit(0); case 'w': - ofn = optarg; + w_arg = optarg; break; default: if (VSL_Arg(vd, c, optarg) > 0) @@ -479,22 +493,37 @@ if (VSL_OpenLog(vd, n_arg)) exit(1); - if (ofn) { - of = open_log(ofn, append); + if (P_arg && (pfh = vpf_open(P_arg, 0600, NULL)) == NULL) { + perror(P_arg); + exit(1); + } + + if (D_flag && daemon(0, 0) == -1) { + perror("daemon()"); + if (pfh != NULL) + vpf_remove(pfh); + exit(1); + } + + if (pfh != NULL) + vpf_write(pfh); + + if (w_arg) { + of = open_log(w_arg, a_flag); signal(SIGHUP, sighup); } else { - ofn = "stdout"; + w_arg = "stdout"; of = stdout; } while (VSL_Dispatch(vd, h_ncsa, of) == 0) { if (fflush(of) != 0) { - perror(ofn); + perror(w_arg); exit(1); } if (reopen && of != stdout) { fclose(of); - of = open_log(ofn, append); + of = open_log(w_arg, a_flag); reopen = 0; } } From des at projects.linpro.no Thu Nov 8 16:52:40 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 17:52:40 +0100 (CET) Subject: r2231 - in trunk/varnish-cache/bin: varnishhist varnishstat Message-ID: <20071108165240.82CDD1EC396@projects.linpro.no> Author: des Date: 2007-11-08 17:52:40 +0100 (Thu, 08 Nov 2007) New Revision: 2231 Modified: trunk/varnish-cache/bin/varnishhist/varnishhist.c trunk/varnish-cache/bin/varnishstat/varnishstat.c Log: Pressing 0-9 will change the delay to the corresponding power of two (e.g. press 3 for 2^3 = 8 seconds) Modified: trunk/varnish-cache/bin/varnishhist/varnishhist.c =================================================================== --- trunk/varnish-cache/bin/varnishhist/varnishhist.c 2007-11-08 16:28:26 UTC (rev 2230) +++ trunk/varnish-cache/bin/varnishhist/varnishhist.c 2007-11-08 16:52:40 UTC (rev 2231) @@ -279,6 +279,18 @@ case 'q': endwin(); return; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; Modified: trunk/varnish-cache/bin/varnishstat/varnishstat.c =================================================================== --- trunk/varnish-cache/bin/varnishstat/varnishstat.c 2007-11-08 16:28:26 UTC (rev 2230) +++ trunk/varnish-cache/bin/varnishstat/varnishstat.c 2007-11-08 16:52:40 UTC (rev 2231) @@ -148,6 +148,18 @@ case 'q': endwin(); exit(0); + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; From des at projects.linpro.no Thu Nov 8 20:30:07 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 21:30:07 +0100 (CET) Subject: r2232 - trunk/varnish-cache/bin/varnishd Message-ID: <20071108203007.3091F1EC872@projects.linpro.no> Author: des Date: 2007-11-08 21:30:06 +0100 (Thu, 08 Nov 2007) New Revision: 2232 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Log: Minimal intervention fix for #162: we can delay arming a kevent, but we can't delay disarming it, or we will have a race between the acceptor and a worker thread. This will need revisiting at some later date. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 16:52:40 UTC (rev 2231) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 20:30:06 UTC (rev 2232) @@ -66,8 +66,8 @@ if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); - if (++nki == NKEV) { - assert(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); + if (++nki == NKEV || arm == EV_DELETE) { + AZ(kevent(kq, ki, nki, NULL, 0, NULL)); nki = 0; } } From phk at phk.freebsd.dk Thu Nov 8 20:48:15 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 08 Nov 2007 20:48:15 +0000 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Thu, 08 Nov 2007 21:30:07 +0100." <20071108203007.3091F1EC872@projects.linpro.no> Message-ID: <52830.1194554895@critter.freebsd.dk> In message <20071108203007.3091F1EC872 at projects.linpro.no>, des at projects.linpro .no writes: >Author: des >Date: 2007-11-08 21:30:06 +0100 (Thu, 08 Nov 2007) >New Revision: 2232 > >Modified: > trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c >Log: >Minimal intervention fix for #162: we can delay arming a kevent, but we >can't delay disarming it, or we will have a race between the acceptor and >a worker thread. This will need revisiting at some later date. More thinking: If we get two events for the fd in the same batch, this fix will not save us. I'm not sure if FreeBSD's current kqueue implementation prevents that from happening. Long term and full fix: we should mark the session with a flag instead and ignore events when we don't own the session. Trouble with that is, we need to hold a ref on the session to check the flag. There are other things in the acceptor-end of varnish that needs revisited, for now this fix is plenty. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at projects.linpro.no Thu Nov 8 21:05:43 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:05:43 +0100 (CET) Subject: r2233 - in branches/1.2: . bin/varnishd lib/libvarnish lib/libvarnishcompat Message-ID: <20071108210543.E216A1EC46F@projects.linpro.no> Author: des Date: 2007-11-08 22:05:43 +0100 (Thu, 08 Nov 2007) New Revision: 2233 Modified: branches/1.2/ branches/1.2/autogen.sh branches/1.2/bin/varnishd/cache_acceptor.c branches/1.2/bin/varnishd/cache_http.c branches/1.2/bin/varnishd/storage_file.c branches/1.2/configure.ac branches/1.2/lib/libvarnish/vsb.c branches/1.2/lib/libvarnishcompat/daemon.c branches/1.2/lib/libvarnishcompat/strndup.c Log: Merged revisions 2222-2228 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2222 | des | 2007-11-01 22:47:58 +0100 (Thu, 01 Nov 2007) | 2 lines Add Solaris support. Avoid kshisms. ........ r2223 | des | 2007-11-01 22:48:54 +0100 (Thu, 01 Nov 2007) | 2 lines Add missing ........ r2224 | des | 2007-11-01 22:52:34 +0100 (Thu, 01 Nov 2007) | 2 lines Don't rely on non-portable ........ r2225 | des | 2007-11-01 22:53:59 +0100 (Thu, 01 Nov 2007) | 2 lines Header cleanup. ........ r2226 | des | 2007-11-01 22:57:44 +0100 (Thu, 01 Nov 2007) | 2 lines Fix the struct statvfs issue. ........ r2227 | des | 2007-11-02 15:09:17 +0100 (Fri, 02 Nov 2007) | 3 lines Test whether SO_{RCV,SND}TIMEO work (they are defined but unimplemented on Solaris, so simply checking for their presence is not enough) ........ r2228 | des | 2007-11-03 13:30:06 +0100 (Sat, 03 Nov 2007) | 2 lines Typo in previous commit. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2228 Modified: branches/1.2/autogen.sh =================================================================== --- branches/1.2/autogen.sh 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/autogen.sh 2007-11-08 21:05:43 UTC (rev 2233) @@ -17,12 +17,15 @@ Linux) LIBTOOLIZE=libtoolize ;; +SunOS) + LIBTOOLIZE=libtoolize + ;; *) warn "unrecognized platform:" `uname -s` LIBTOOLIZE=libtoolize esac -automake_version=$(automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+') +automake_version=`automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+'` if [ -z "$automake_version" ] ; then warn "unable to determine automake version" else Modified: branches/1.2/bin/varnishd/cache_acceptor.c =================================================================== --- branches/1.2/bin/varnishd/cache_acceptor.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/bin/varnishd/cache_acceptor.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -90,17 +90,22 @@ if (memcmp(&lin, &linger, l)) need_linger = 1; +#ifdef SO_SNDTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_sndtimeo, l)) need_sndtimeo = 1; +#endif +#ifdef SO_RCVTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_rcvtimeo, l)) need_rcvtimeo = 1; +#endif + need_test = 0; } @@ -122,12 +127,16 @@ if (need_linger) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER, &linger, sizeof linger)); +#ifdef SO_SNDTIMEO_WORKS if (need_sndtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); +#endif +#ifdef SO_RCVTIMEO_WORKS if (need_rcvtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); +#endif } /*--------------------------------------------------------------------*/ @@ -160,6 +169,7 @@ need_test = 1; while (1) { +#ifdef SO_SNDTIMEO_WORKS if (params->send_timeout != tv_sndtimeo.tv_sec) { need_test = 1; tv_sndtimeo.tv_sec = params->send_timeout; @@ -168,6 +178,8 @@ SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); } +#endif +#ifdef SO_RCVTIMEO_WORKS if (params->sess_timeout != tv_rcvtimeo.tv_sec) { need_test = 1; tv_rcvtimeo.tv_sec = params->sess_timeout; @@ -176,6 +188,7 @@ SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); } +#endif i = poll(pfd, heritage.nsocks, 1000); now = TIM_real(); for (u = 0; u < heritage.nsocks; u++) { Modified: branches/1.2/bin/varnishd/cache_http.c =================================================================== --- branches/1.2/bin/varnishd/cache_http.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/bin/varnishd/cache_http.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -31,13 +31,14 @@ * HTTP request storage and manipulation */ -#include +#include #include #include #include +#include #include #include -#include +#include #include "heritage.h" #include "shmlog.h" Modified: branches/1.2/bin/varnishd/storage_file.c =================================================================== --- branches/1.2/bin/varnishd/storage_file.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/bin/varnishd/storage_file.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -33,7 +33,6 @@ #include #include -#include #include #include @@ -122,7 +121,13 @@ static void smf_calcsize(struct smf_sc *sc, const char *size, int newfile) { +#if defined(HAVE_SYS_STATVFS_H) + struct statvfs fsst; +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) struct statfs fsst; +#else +#error no struct statfs / struct statvfs +#endif uintmax_t l, fssize; unsigned bs; char suff[2]; @@ -134,7 +139,13 @@ AZ(fstat(sc->fd, &st)); xxxassert(S_ISREG(st.st_mode)); +#if defined(HAVE_SYS_STATVFS_H) + AZ(fstatvfs(sc->fd, &fsst)); +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) AZ(fstatfs(sc->fd, &fsst)); +#else +#error no struct statfs / struct statvfs +#endif /* We use units of the larger of filesystem blocksize and pagesize */ bs = sc->pagesize; Modified: branches/1.2/configure.ac =================================================================== --- branches/1.2/configure.ac 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/configure.ac 2007-11-08 21:05:43 UTC (rev 2233) @@ -115,6 +115,45 @@ AC_CHECK_FUNCS([epoll_ctl]) AC_CHECK_FUNCS([poll]) +# Solaris defines SO_{RCV,SND}TIMEO, but does not implement them +AC_CACHE_CHECK([whether SO_RCVTIMEO works], + [ac_cv_so_rcvtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_rcvtimeo_works=yes], + [ac_cv_so_rcvtimeo_works=no]) + ]) +if test "$ac_cv_so_rcvtimeo_works" = yes; then + AC_DEFINE([SO_RCVTIMEO_WORKS], [1], [Define if SO_RCVTIMEO works]) +fi + +AC_CACHE_CHECK([whether SO_SNDTIMEO works], + [ac_cv_so_sndtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_sndtimeo_works=yes], + [ac_cv_so_sndtimeo_works=no]) + ]) +if test "$ac_cv_so_sndtimeo_works" = yes; then + AC_DEFINE([SO_SNDTIMEO_WORKS], [1], [Define if SO_SNDTIMEO works]) +fi + # Run-time directory VARNISH_STATE_DIR='${localstatedir}/varnish' AC_SUBST(VARNISH_STATE_DIR) Modified: branches/1.2/lib/libvarnish/vsb.c =================================================================== --- branches/1.2/lib/libvarnish/vsb.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/lib/libvarnish/vsb.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -34,6 +34,7 @@ #include #include #include +#include #include "vsb.h" Modified: branches/1.2/lib/libvarnishcompat/daemon.c =================================================================== --- branches/1.2/lib/libvarnishcompat/daemon.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/lib/libvarnishcompat/daemon.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -35,7 +35,6 @@ #include #include -#include #include #include #include @@ -79,7 +78,7 @@ if (!nochdir) (void)chdir("/"); - if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) { + if (!noclose && (fd = open("/dev/null", O_RDWR, 0)) != -1) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); (void)dup2(fd, STDERR_FILENO); Modified: branches/1.2/lib/libvarnishcompat/strndup.c =================================================================== --- branches/1.2/lib/libvarnishcompat/strndup.c 2007-11-08 20:30:06 UTC (rev 2232) +++ branches/1.2/lib/libvarnishcompat/strndup.c 2007-11-08 21:05:43 UTC (rev 2233) @@ -34,6 +34,7 @@ #include #include +#include #ifndef HAVE_STRLCPY #include "compat/strlcpy.h" From des at projects.linpro.no Thu Nov 8 21:06:50 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:06:50 +0100 (CET) Subject: r2234 - in branches/1.2: . bin/varnishhist bin/varnishlog bin/varnishncsa bin/varnishstat Message-ID: <20071108210650.1DCEE1EC51E@projects.linpro.no> Author: des Date: 2007-11-08 22:06:49 +0100 (Thu, 08 Nov 2007) New Revision: 2234 Modified: branches/1.2/ branches/1.2/bin/varnishhist/varnishhist.c branches/1.2/bin/varnishlog/varnishlog.1 branches/1.2/bin/varnishlog/varnishlog.c branches/1.2/bin/varnishncsa/varnishncsa.1 branches/1.2/bin/varnishncsa/varnishncsa.c branches/1.2/bin/varnishstat/varnishstat.c Log: Merged revisions 2229-2231 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2229 | des | 2007-11-08 17:27:53 +0100 (Thu, 08 Nov 2007) | 2 lines Pick some nits. ........ r2230 | des | 2007-11-08 17:28:26 +0100 (Thu, 08 Nov 2007) | 2 lines Copy&paste -D (daemonize) from varnishlog. ........ r2231 | des | 2007-11-08 17:52:40 +0100 (Thu, 08 Nov 2007) | 3 lines Pressing 0-9 will change the delay to the corresponding power of two (e.g. press 3 for 2^3 = 8 seconds) ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2228 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2231 Modified: branches/1.2/bin/varnishhist/varnishhist.c =================================================================== --- branches/1.2/bin/varnishhist/varnishhist.c 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishhist/varnishhist.c 2007-11-08 21:06:49 UTC (rev 2234) @@ -279,6 +279,18 @@ case 'q': endwin(); return; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; Modified: branches/1.2/bin/varnishlog/varnishlog.1 =================================================================== --- branches/1.2/bin/varnishlog/varnishlog.1 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishlog/varnishlog.1 2007-11-08 21:06:49 UTC (rev 2234) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHLOG 1 .Os .Sh NAME @@ -44,9 +44,9 @@ .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag +.Op Fl n Ar varnish_name .Op Fl o .Op Fl P Ar file -.Op Fl n Ar varnish_name .Op Fl r Ar file .Op Fl V .Op Fl w Ar file Modified: branches/1.2/bin/varnishlog/varnishlog.c =================================================================== --- branches/1.2/bin/varnishlog/varnishlog.c 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishlog/varnishlog.c 2007-11-08 21:06:49 UTC (rev 2234) @@ -283,9 +283,9 @@ { int i, c; int a_flag = 0, D_flag = 0, o_flag = 0; + const char *n_arg = NULL; const char *P_arg = NULL; const char *w_arg = NULL; - const char *n_arg = NULL; struct pidfh *pfh = NULL; struct VSL_data *vd; Modified: branches/1.2/bin/varnishncsa/varnishncsa.1 =================================================================== --- branches/1.2/bin/varnishncsa/varnishncsa.1 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishncsa/varnishncsa.1 2007-11-08 21:06:49 UTC (rev 2234) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHNCSA 1 .Os .Sh NAME @@ -40,10 +40,12 @@ .Op Fl b .Op Fl C .Op Fl c +.Op Fl D .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag .Op Fl n Ar varnish_name +.Op Fl P Ar file .Op Fl r Ar file .Op Fl V .Op Fl w Ar file @@ -82,6 +84,8 @@ is specified, .Nm acts as if they both were. +.It Fl D +Daemonize. .It Fl d Process old log entries on startup. Normally, @@ -109,6 +113,9 @@ If .Fl n is not specified, the host name is used. +.It Fl P Ar file +Write the process's PID to the specified +.Ar file . .It Fl r Ar file Read log entries from .Ar file Modified: branches/1.2/bin/varnishncsa/varnishncsa.c =================================================================== --- branches/1.2/bin/varnishncsa/varnishncsa.c 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishncsa/varnishncsa.c 2007-11-08 21:06:49 UTC (rev 2234) @@ -69,10 +69,16 @@ #include #include +#ifndef HAVE_DAEMON +#include "compat/daemon.h" +#endif + +#include "vsb.h" +#include "vpf.h" + #include "libvarnish.h" #include "shmlog.h" #include "varnishapi.h" -#include "vsb.h" static struct logline { char *df_H; /* %H, Protocol version */ @@ -434,7 +440,7 @@ usage(void) { - fprintf(stderr, "usage: varnishncsa %s [-aV] [-n varnish_name] [-w file]\n", VSL_ARGS); + fprintf(stderr, "usage: varnishncsa %s [-aDV] [-n varnish_name] [-P file] [-w file]\n", VSL_ARGS); exit(1); } @@ -442,15 +448,17 @@ main(int argc, char *argv[]) { int i, c; + int a_flag = 0, D_flag = 0; + const char *n_arg = NULL; + const char *P_arg = NULL; + const char *w_arg = NULL; + struct pidfh *pfh = NULL; struct VSL_data *vd; - const char *ofn = NULL; - const char *n_arg = NULL; - int append = 0; FILE *of; vd = VSL_New(); - while ((c = getopt(argc, argv, VSL_ARGS "an:Vw:")) != -1) { + while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { i = VSL_Arg(vd, c, optarg); if (i < 0) exit (1); @@ -458,16 +466,22 @@ continue; switch (c) { case 'a': - append = 1; + a_flag = 1; break; + case 'D': + D_flag = 1; + break; case 'n': n_arg = optarg; break; + case 'P': + P_arg = optarg; + break; case 'V': varnish_version("varnishncsa"); exit(0); case 'w': - ofn = optarg; + w_arg = optarg; break; default: if (VSL_Arg(vd, c, optarg) > 0) @@ -479,22 +493,37 @@ if (VSL_OpenLog(vd, n_arg)) exit(1); - if (ofn) { - of = open_log(ofn, append); + if (P_arg && (pfh = vpf_open(P_arg, 0600, NULL)) == NULL) { + perror(P_arg); + exit(1); + } + + if (D_flag && daemon(0, 0) == -1) { + perror("daemon()"); + if (pfh != NULL) + vpf_remove(pfh); + exit(1); + } + + if (pfh != NULL) + vpf_write(pfh); + + if (w_arg) { + of = open_log(w_arg, a_flag); signal(SIGHUP, sighup); } else { - ofn = "stdout"; + w_arg = "stdout"; of = stdout; } while (VSL_Dispatch(vd, h_ncsa, of) == 0) { if (fflush(of) != 0) { - perror(ofn); + perror(w_arg); exit(1); } if (reopen && of != stdout) { fclose(of); - of = open_log(ofn, append); + of = open_log(w_arg, a_flag); reopen = 0; } } Modified: branches/1.2/bin/varnishstat/varnishstat.c =================================================================== --- branches/1.2/bin/varnishstat/varnishstat.c 2007-11-08 21:05:43 UTC (rev 2233) +++ branches/1.2/bin/varnishstat/varnishstat.c 2007-11-08 21:06:49 UTC (rev 2234) @@ -148,6 +148,18 @@ case 'q': endwin(); exit(0); + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; From des at projects.linpro.no Thu Nov 8 21:07:37 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:07:37 +0100 (CET) Subject: r2235 - in branches/1.2: . bin/varnishd Message-ID: <20071108210737.564C61EC46F@projects.linpro.no> Author: des Date: 2007-11-08 22:07:37 +0100 (Thu, 08 Nov 2007) New Revision: 2235 Modified: branches/1.2/ branches/1.2/bin/varnishd/cache_acceptor_kqueue.c Log: Merged revisions 2232 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2232 | des | 2007-11-08 21:30:06 +0100 (Thu, 08 Nov 2007) | 4 lines Minimal intervention fix for #162: we can delay arming a kevent, but we can't delay disarming it, or we will have a race between the acceptor and a worker thread. This will need revisiting at some later date. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2231 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232 Modified: branches/1.2/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- branches/1.2/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 21:06:49 UTC (rev 2234) +++ branches/1.2/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 21:07:37 UTC (rev 2235) @@ -66,8 +66,8 @@ if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); - if (++nki == NKEV) { - assert(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); + if (++nki == NKEV || arm == EV_DELETE) { + AZ(kevent(kq, ki, nki, NULL, 0, NULL)); nki = 0; } } From des at linpro.no Thu Nov 8 21:16:46 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 22:16:46 +0100 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: <52830.1194554895@critter.freebsd.dk> (Poul-Henning Kamp's message of "Thu, 08 Nov 2007 20:48:15 +0000") References: <52830.1194554895@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > If we get two events for the fd in the same batch, this fix will > not save us. I'm not sure if FreeBSD's current kqueue implementation > prevents that from happening. I think that in such a case, we would get a single kevent with (kp->data > 0) and (kp->flags & EV_EOF). I'll have to check the code to be sure. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at projects.linpro.no Thu Nov 8 21:23:28 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:23:28 +0100 (CET) Subject: r2236 - branches/1.1 Message-ID: <20071108212328.27A501EC46F@projects.linpro.no> Author: des Date: 2007-11-08 22:23:27 +0100 (Thu, 08 Nov 2007) New Revision: 2236 Modified: branches/1.1/ branches/1.1/autogen.des branches/1.1/configure.ac Log: Merged revisions 2211-2212 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2211 | des | 2007-10-30 14:39:21 +0100 (Tue, 30 Oct 2007) | 2 lines Add an --enable-diagnostics option which causes DIAGNOSTICS to be defined. ........ r2212 | des | 2007-10-30 14:41:56 +0100 (Tue, 30 Oct 2007) | 2 lines Enable diagnostics ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212 Modified: branches/1.1/autogen.des =================================================================== --- branches/1.1/autogen.des 2007-11-08 21:07:37 UTC (rev 2235) +++ branches/1.1/autogen.des 2007-11-08 21:23:27 UTC (rev 2236) @@ -15,6 +15,7 @@ --enable-developer-warnings \ --enable-debugging-symbols \ --enable-dependency-tracking \ + --enable-diagnostics \ --enable-extra-developer-warnings \ --enable-stack-protector \ --enable-werror \ Modified: branches/1.1/configure.ac =================================================================== --- branches/1.1/configure.ac 2007-11-08 21:07:37 UTC (rev 2235) +++ branches/1.1/configure.ac 2007-11-08 21:23:27 UTC (rev 2236) @@ -133,6 +133,9 @@ AC_ARG_ENABLE(debugging-symbols, AS_HELP_STRING([--enable-debugging-symbols],[enable debugging symbols (default is NO)]), CFLAGS="${CFLAGS} -O0 -g -fno-inline") +AC_ARG_ENABLE(diagnostics, + AS_HELP_STRING([--enable-diagnostics],[enable run-time diagnostics (default is NO)]), + CFLAGS="${CFLAGS} -DDIAGNOSTICS") AC_ARG_ENABLE(extra-developer-warnings, AS_HELP_STRING([--enable-extra-developer-warnings],[enable even stricter warnings (default is NO)]), CFLAGS="${CFLAGS} ${EXTRA_DEVELOPER_CFLAGS}") From des at projects.linpro.no Thu Nov 8 21:23:48 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:23:48 +0100 (CET) Subject: r2237 - in branches/1.1: . bin/varnishd Message-ID: <20071108212348.85ECA1EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 22:23:48 +0100 (Thu, 08 Nov 2007) New Revision: 2237 Modified: branches/1.1/ branches/1.1/bin/varnishd/cache_center.c Log: Merged revisions 2215-2220 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2215 | des | 2007-10-30 15:06:37 +0100 (Tue, 30 Oct 2007) | 3 lines When DIAGNOSTICS is defined, log every step that each session goes through, along with some vital statistics, and flush the log regularly. ........ r2220 | des | 2007-10-30 15:40:55 +0100 (Tue, 30 Oct 2007) | 5 lines Refine the debugging string: we don't really need the XID, but we do need the thread ID in order to locate the correct thread in gdb. Furthermore, we must use VSL rather than WSL since the session might actually not have a worker at that point; but if it does, make sure to flush the log first. ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220 Modified: branches/1.1/bin/varnishd/cache_center.c =================================================================== --- branches/1.1/bin/varnishd/cache_center.c 2007-11-08 21:23:27 UTC (rev 2236) +++ branches/1.1/bin/varnishd/cache_center.c 2007-11-08 21:23:48 UTC (rev 2237) @@ -783,7 +783,22 @@ CHECK_OBJ(w->nobjhead, OBJHEAD_MAGIC); switch (sp->step) { -#define STEP(l,u) case STP_##u: done = cnt_##l(sp); break; +#ifdef DIAGNOSTICS +#define STEP(l,u) \ + case STP_##u: \ + if (sp->wrk) \ + WSL_Flush(sp->wrk); \ + VSL(SLT_Debug, sp->id, \ + "thr %p STP_%s sp %p obj %p vcl %p", \ + pthread_self(), #u, sp, sp->obj, sp->vcl); \ + done = cnt_##l(sp); \ + break; +#else +#define STEP(l,u) \ + case STP_##u: \ + done = cnt_##l(sp); \ + break; +#endif #include "steps.h" #undef STEP default: INCOMPL(); From des at projects.linpro.no Thu Nov 8 21:29:23 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:29:23 +0100 (CET) Subject: r2238 - branches/1.1/bin/varnishd Message-ID: <20071108212923.E553D1EC46F@projects.linpro.no> Author: des Date: 2007-11-08 22:29:23 +0100 (Thu, 08 Nov 2007) New Revision: 2238 Modified: branches/1.1/bin/varnishd/cache_backend.c Log: Fix r2191. Modified: branches/1.1/bin/varnishd/cache_backend.c =================================================================== --- branches/1.1/bin/varnishd/cache_backend.c 2007-11-08 21:23:48 UTC (rev 2237) +++ branches/1.1/bin/varnishd/cache_backend.c 2007-11-08 21:29:23 UTC (rev 2238) @@ -95,7 +95,7 @@ { CHECK_OBJ_NOTNULL(bereq, BEREQ_MAGIC); - WS_Reset(bereq->ws, NULL); + WS_Reset(bereq->http->ws); LOCK(&vbemtx); VTAILQ_INSERT_HEAD(&bereq_head, bereq, list); UNLOCK(&vbemtx); From des at projects.linpro.no Thu Nov 8 21:30:05 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:30:05 +0100 (CET) Subject: r2239 - in branches/1.1: . bin/varnishd lib/libvarnish lib/libvarnishcompat Message-ID: <20071108213005.61E851EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 22:30:05 +0100 (Thu, 08 Nov 2007) New Revision: 2239 Modified: branches/1.1/ branches/1.1/autogen.sh branches/1.1/bin/varnishd/cache_acceptor.c branches/1.1/bin/varnishd/cache_http.c branches/1.1/bin/varnishd/storage_file.c branches/1.1/configure.ac branches/1.1/lib/libvarnish/vsb.c branches/1.1/lib/libvarnishcompat/daemon.c branches/1.1/lib/libvarnishcompat/strndup.c Log: Merged revisions 2222-2228 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2222 | des | 2007-11-01 22:47:58 +0100 (Thu, 01 Nov 2007) | 2 lines Add Solaris support. Avoid kshisms. ........ r2223 | des | 2007-11-01 22:48:54 +0100 (Thu, 01 Nov 2007) | 2 lines Add missing ........ r2224 | des | 2007-11-01 22:52:34 +0100 (Thu, 01 Nov 2007) | 2 lines Don't rely on non-portable ........ r2225 | des | 2007-11-01 22:53:59 +0100 (Thu, 01 Nov 2007) | 2 lines Header cleanup. ........ r2226 | des | 2007-11-01 22:57:44 +0100 (Thu, 01 Nov 2007) | 2 lines Fix the struct statvfs issue. ........ r2227 | des | 2007-11-02 15:09:17 +0100 (Fri, 02 Nov 2007) | 3 lines Test whether SO_{RCV,SND}TIMEO work (they are defined but unimplemented on Solaris, so simply checking for their presence is not enough) ........ r2228 | des | 2007-11-03 13:30:06 +0100 (Sat, 03 Nov 2007) | 2 lines Typo in previous commit. ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2228 Modified: branches/1.1/autogen.sh =================================================================== --- branches/1.1/autogen.sh 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/autogen.sh 2007-11-08 21:30:05 UTC (rev 2239) @@ -17,12 +17,15 @@ Linux) LIBTOOLIZE=libtoolize ;; +SunOS) + LIBTOOLIZE=libtoolize + ;; *) warn "unrecognized platform:" `uname -s` LIBTOOLIZE=libtoolize esac -automake_version=$(automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+') +automake_version=`automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+'` if [ -z "$automake_version" ] ; then warn "unable to determine automake version" else Modified: branches/1.1/bin/varnishd/cache_acceptor.c =================================================================== --- branches/1.1/bin/varnishd/cache_acceptor.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/bin/varnishd/cache_acceptor.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -91,17 +91,22 @@ if (memcmp(&lin, &linger, l)) need_linger = 1; +#ifdef SO_SNDTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_sndtimeo, l)) need_sndtimeo = 1; +#endif +#ifdef SO_RCVTIMEO_WORKS l = sizeof tv; AZ(getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l)); assert(l == sizeof tv); if (memcmp(&tv, &tv_rcvtimeo, l)) need_rcvtimeo = 1; +#endif + need_test = 0; } @@ -118,12 +123,16 @@ if (need_linger) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER, &linger, sizeof linger)); +#ifdef SO_SNDTIMEO_WORKS if (need_sndtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); +#endif +#ifdef SO_RCVTIMEO_WORKS if (need_rcvtimeo) AZ(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); +#endif } /*--------------------------------------------------------------------*/ @@ -154,6 +163,7 @@ need_test = 1; while (1) { +#ifdef SO_SNDTIMEO_WORKS if (params->send_timeout != tv_sndtimeo.tv_sec) { need_test = 1; tv_sndtimeo.tv_sec = params->send_timeout; @@ -162,6 +172,8 @@ SO_SNDTIMEO, &tv_sndtimeo, sizeof tv_sndtimeo)); } +#endif +#ifdef SO_RCVTIMEO_WORKS if (params->sess_timeout != tv_rcvtimeo.tv_sec) { need_test = 1; tv_rcvtimeo.tv_sec = params->sess_timeout; @@ -170,6 +182,7 @@ SO_RCVTIMEO, &tv_rcvtimeo, sizeof tv_rcvtimeo)); } +#endif i = poll(pfd, heritage.nsocks, 1000); for (j = 0; j < heritage.nsocks; j++) { if (pfd[j].revents == 0) Modified: branches/1.1/bin/varnishd/cache_http.c =================================================================== --- branches/1.1/bin/varnishd/cache_http.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/bin/varnishd/cache_http.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -31,13 +31,14 @@ * HTTP request storage and manipulation */ -#include +#include #include #include #include +#include #include #include -#include +#include #include "heritage.h" #include "shmlog.h" Modified: branches/1.1/bin/varnishd/storage_file.c =================================================================== --- branches/1.1/bin/varnishd/storage_file.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/bin/varnishd/storage_file.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -33,7 +33,6 @@ #include #include -#include #include #include @@ -120,7 +119,13 @@ static void smf_calcsize(struct smf_sc *sc, const char *size, int newfile) { +#if defined(HAVE_SYS_STATVFS_H) + struct statvfs fsst; +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) struct statfs fsst; +#else +#error no struct statfs / struct statvfs +#endif uintmax_t l, fssize; unsigned bs; char suff[2]; @@ -132,7 +137,13 @@ AZ(fstat(sc->fd, &st)); xxxassert(S_ISREG(st.st_mode)); +#if defined(HAVE_SYS_STATVFS_H) + AZ(fstatvfs(sc->fd, &fsst)); +#elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) AZ(fstatfs(sc->fd, &fsst)); +#else +#error no struct statfs / struct statvfs +#endif /* We use units of the larger of filesystem blocksize and pagesize */ bs = sc->pagesize; Modified: branches/1.1/configure.ac =================================================================== --- branches/1.1/configure.ac 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/configure.ac 2007-11-08 21:30:05 UTC (rev 2239) @@ -114,6 +114,45 @@ AC_CHECK_FUNCS([epoll_ctl]) AC_CHECK_FUNCS([poll]) +# Solaris defines SO_{RCV,SND}TIMEO, but does not implement them +AC_CACHE_CHECK([whether SO_RCVTIMEO works], + [ac_cv_so_rcvtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_rcvtimeo_works=yes], + [ac_cv_so_rcvtimeo_works=no]) + ]) +if test "$ac_cv_so_rcvtimeo_works" = yes; then + AC_DEFINE([SO_RCVTIMEO_WORKS], [1], [Define if SO_RCVTIMEO works]) +fi + +AC_CACHE_CHECK([whether SO_SNDTIMEO works], + [ac_cv_so_sndtimeo_works], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM([[ +#include +#include +#include + ]],[[ +int sd = socket(AF_INET, SOCK_STREAM, 0); +struct timeval tv = { 1, 0 }; +return !!setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv); + ]])], + [ac_cv_so_sndtimeo_works=yes], + [ac_cv_so_sndtimeo_works=no]) + ]) +if test "$ac_cv_so_sndtimeo_works" = yes; then + AC_DEFINE([SO_SNDTIMEO_WORKS], [1], [Define if SO_SNDTIMEO works]) +fi + # Run-time directory VARNISH_STATE_DIR='${localstatedir}/varnish' AC_SUBST(VARNISH_STATE_DIR) Modified: branches/1.1/lib/libvarnish/vsb.c =================================================================== --- branches/1.1/lib/libvarnish/vsb.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/lib/libvarnish/vsb.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -34,6 +34,7 @@ #include #include #include +#include #include "vsb.h" Modified: branches/1.1/lib/libvarnishcompat/daemon.c =================================================================== --- branches/1.1/lib/libvarnishcompat/daemon.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/lib/libvarnishcompat/daemon.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -35,7 +35,6 @@ #include #include -#include #include #include #include @@ -79,7 +78,7 @@ if (!nochdir) (void)chdir("/"); - if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) { + if (!noclose && (fd = open("/dev/null", O_RDWR, 0)) != -1) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); (void)dup2(fd, STDERR_FILENO); Modified: branches/1.1/lib/libvarnishcompat/strndup.c =================================================================== --- branches/1.1/lib/libvarnishcompat/strndup.c 2007-11-08 21:29:23 UTC (rev 2238) +++ branches/1.1/lib/libvarnishcompat/strndup.c 2007-11-08 21:30:05 UTC (rev 2239) @@ -34,6 +34,7 @@ #include #include +#include #ifndef HAVE_STRLCPY #include "compat/strlcpy.h" From des at projects.linpro.no Thu Nov 8 21:30:58 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:30:58 +0100 (CET) Subject: r2240 - in branches/1.1: . bin/varnishhist bin/varnishlog bin/varnishncsa bin/varnishstat Message-ID: <20071108213058.5024F1EC51E@projects.linpro.no> Author: des Date: 2007-11-08 22:30:58 +0100 (Thu, 08 Nov 2007) New Revision: 2240 Modified: branches/1.1/ branches/1.1/bin/varnishhist/varnishhist.c branches/1.1/bin/varnishlog/varnishlog.1 branches/1.1/bin/varnishlog/varnishlog.c branches/1.1/bin/varnishncsa/varnishncsa.1 branches/1.1/bin/varnishncsa/varnishncsa.c branches/1.1/bin/varnishstat/varnishstat.c Log: Merged revisions 2229-2231 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2229 | des | 2007-11-08 17:27:53 +0100 (Thu, 08 Nov 2007) | 2 lines Pick some nits. ........ r2230 | des | 2007-11-08 17:28:26 +0100 (Thu, 08 Nov 2007) | 2 lines Copy&paste -D (daemonize) from varnishlog. ........ r2231 | des | 2007-11-08 17:52:40 +0100 (Thu, 08 Nov 2007) | 3 lines Pressing 0-9 will change the delay to the corresponding power of two (e.g. press 3 for 2^3 = 8 seconds) ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2228 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2231 Modified: branches/1.1/bin/varnishhist/varnishhist.c =================================================================== --- branches/1.1/bin/varnishhist/varnishhist.c 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishhist/varnishhist.c 2007-11-08 21:30:58 UTC (rev 2240) @@ -279,6 +279,18 @@ case 'q': endwin(); return; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; Modified: branches/1.1/bin/varnishlog/varnishlog.1 =================================================================== --- branches/1.1/bin/varnishlog/varnishlog.1 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishlog/varnishlog.1 2007-11-08 21:30:58 UTC (rev 2240) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHLOG 1 .Os .Sh NAME @@ -44,9 +44,9 @@ .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag +.Op Fl n Ar varnish_name .Op Fl o .Op Fl P Ar file -.Op Fl n Ar varnish_name .Op Fl r Ar file .Op Fl V .Op Fl w Ar file Modified: branches/1.1/bin/varnishlog/varnishlog.c =================================================================== --- branches/1.1/bin/varnishlog/varnishlog.c 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishlog/varnishlog.c 2007-11-08 21:30:58 UTC (rev 2240) @@ -283,9 +283,9 @@ { int i, c; int a_flag = 0, D_flag = 0, o_flag = 0; + const char *n_arg = NULL; const char *P_arg = NULL; const char *w_arg = NULL; - const char *n_arg = NULL; struct pidfh *pfh = NULL; struct VSL_data *vd; Modified: branches/1.1/bin/varnishncsa/varnishncsa.1 =================================================================== --- branches/1.1/bin/varnishncsa/varnishncsa.1 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishncsa/varnishncsa.1 2007-11-08 21:30:58 UTC (rev 2240) @@ -28,7 +28,7 @@ .\" .\" $Id$ .\" -.Dd June 15, 2007 +.Dd November 8, 2007 .Dt VARNISHNCSA 1 .Os .Sh NAME @@ -40,10 +40,12 @@ .Op Fl b .Op Fl C .Op Fl c +.Op Fl D .Op Fl d .Op Fl I Ar regex .Op Fl i Ar tag .Op Fl n Ar varnish_name +.Op Fl P Ar file .Op Fl r Ar file .Op Fl V .Op Fl w Ar file @@ -82,6 +84,8 @@ is specified, .Nm acts as if they both were. +.It Fl D +Daemonize. .It Fl d Process old log entries on startup. Normally, @@ -109,6 +113,9 @@ If .Fl n is not specified, the host name is used. +.It Fl P Ar file +Write the process's PID to the specified +.Ar file . .It Fl r Ar file Read log entries from .Ar file Modified: branches/1.1/bin/varnishncsa/varnishncsa.c =================================================================== --- branches/1.1/bin/varnishncsa/varnishncsa.c 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishncsa/varnishncsa.c 2007-11-08 21:30:58 UTC (rev 2240) @@ -69,10 +69,16 @@ #include #include +#ifndef HAVE_DAEMON +#include "compat/daemon.h" +#endif + +#include "vsb.h" +#include "vpf.h" + #include "libvarnish.h" #include "shmlog.h" #include "varnishapi.h" -#include "vsb.h" static struct logline { char *df_H; /* %H, Protocol version */ @@ -434,7 +440,7 @@ usage(void) { - fprintf(stderr, "usage: varnishncsa %s [-aV] [-n varnish_name] [-w file]\n", VSL_ARGS); + fprintf(stderr, "usage: varnishncsa %s [-aDV] [-n varnish_name] [-P file] [-w file]\n", VSL_ARGS); exit(1); } @@ -442,15 +448,17 @@ main(int argc, char *argv[]) { int i, c; + int a_flag = 0, D_flag = 0; + const char *n_arg = NULL; + const char *P_arg = NULL; + const char *w_arg = NULL; + struct pidfh *pfh = NULL; struct VSL_data *vd; - const char *ofn = NULL; - const char *n_arg = NULL; - int append = 0; FILE *of; vd = VSL_New(); - while ((c = getopt(argc, argv, VSL_ARGS "an:Vw:")) != -1) { + while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { i = VSL_Arg(vd, c, optarg); if (i < 0) exit (1); @@ -458,16 +466,22 @@ continue; switch (c) { case 'a': - append = 1; + a_flag = 1; break; + case 'D': + D_flag = 1; + break; case 'n': n_arg = optarg; break; + case 'P': + P_arg = optarg; + break; case 'V': varnish_version("varnishncsa"); exit(0); case 'w': - ofn = optarg; + w_arg = optarg; break; default: if (VSL_Arg(vd, c, optarg) > 0) @@ -479,22 +493,37 @@ if (VSL_OpenLog(vd, n_arg)) exit(1); - if (ofn) { - of = open_log(ofn, append); + if (P_arg && (pfh = vpf_open(P_arg, 0600, NULL)) == NULL) { + perror(P_arg); + exit(1); + } + + if (D_flag && daemon(0, 0) == -1) { + perror("daemon()"); + if (pfh != NULL) + vpf_remove(pfh); + exit(1); + } + + if (pfh != NULL) + vpf_write(pfh); + + if (w_arg) { + of = open_log(w_arg, a_flag); signal(SIGHUP, sighup); } else { - ofn = "stdout"; + w_arg = "stdout"; of = stdout; } while (VSL_Dispatch(vd, h_ncsa, of) == 0) { if (fflush(of) != 0) { - perror(ofn); + perror(w_arg); exit(1); } if (reopen && of != stdout) { fclose(of); - of = open_log(ofn, append); + of = open_log(w_arg, a_flag); reopen = 0; } } Modified: branches/1.1/bin/varnishstat/varnishstat.c =================================================================== --- branches/1.1/bin/varnishstat/varnishstat.c 2007-11-08 21:30:05 UTC (rev 2239) +++ branches/1.1/bin/varnishstat/varnishstat.c 2007-11-08 21:30:58 UTC (rev 2240) @@ -148,6 +148,18 @@ case 'q': endwin(); exit(0); + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + delay = 1 << (ch - '0'); + break; default: beep(); break; From des at projects.linpro.no Thu Nov 8 21:31:16 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:31:16 +0100 (CET) Subject: r2241 - in branches/1.1: . bin/varnishd Message-ID: <20071108213116.9689E1EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 22:31:16 +0100 (Thu, 08 Nov 2007) New Revision: 2241 Modified: branches/1.1/ branches/1.1/bin/varnishd/cache_acceptor_kqueue.c Log: Merged revisions 2232 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2232 | des | 2007-11-08 21:30:06 +0100 (Thu, 08 Nov 2007) | 4 lines Minimal intervention fix for #162: we can delay arming a kevent, but we can't delay disarming it, or we will have a race between the acceptor and a worker thread. This will need revisiting at some later date. ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2231 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2232 Modified: branches/1.1/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- branches/1.1/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 21:30:58 UTC (rev 2240) +++ branches/1.1/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 21:31:16 UTC (rev 2241) @@ -66,8 +66,8 @@ if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); - if (++nki == NKEV) { - assert(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); + if (++nki == NKEV || arm == EV_DELETE) { + AZ(kevent(kq, ki, nki, NULL, 0, NULL)); nki = 0; } } From des at projects.linpro.no Thu Nov 8 21:32:42 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:32:42 +0100 (CET) Subject: r2242 - trunk/varnish-tools/regress/lib/Varnish/Test Message-ID: <20071108213242.F28851EC51E@projects.linpro.no> Author: des Date: 2007-11-08 22:32:42 +0100 (Thu, 08 Nov 2007) New Revision: 2242 Modified: trunk/varnish-tools/regress/lib/Varnish/Test/Case.pm Log: Nit Modified: trunk/varnish-tools/regress/lib/Varnish/Test/Case.pm =================================================================== --- trunk/varnish-tools/regress/lib/Varnish/Test/Case.pm 2007-11-08 21:31:16 UTC (rev 2241) +++ trunk/varnish-tools/regress/lib/Varnish/Test/Case.pm 2007-11-08 21:32:42 UTC (rev 2242) @@ -168,9 +168,8 @@ no strict 'refs'; my @tests = @{ref($self)."::TESTS"}; - if (!@tests) { - @tests = sort grep {/^test(\w+)/} (keys %{ref($self) . '::'}); - } + @tests = sort grep {/^test(\w+)/} (keys %{ref($self) . '::'}) + unless @tests; $self->{'start'} = [gettimeofday()]; foreach my $method (@tests) { eval { From des at projects.linpro.no Thu Nov 8 21:33:53 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:33:53 +0100 (CET) Subject: r2243 - trunk/varnish-cache Message-ID: <20071108213353.48C571EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 22:33:53 +0100 (Thu, 08 Nov 2007) New Revision: 2243 Modified: trunk/varnish-cache/ Log: Ignore config.cache. Property changes on: trunk/varnish-cache ___________________________________________________________________ Name: svn:ignore - .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc + .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.cache config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc From des at projects.linpro.no Thu Nov 8 21:34:12 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:34:12 +0100 (CET) Subject: r2244 - branches/1.2 Message-ID: <20071108213412.AB8421EC46F@projects.linpro.no> Author: des Date: 2007-11-08 22:34:12 +0100 (Thu, 08 Nov 2007) New Revision: 2244 Modified: branches/1.2/ Log: Merged revisions 2243 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2243 | des | 2007-11-08 22:33:53 +0100 (Thu, 08 Nov 2007) | 2 lines Ignore config.cache. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svn:ignore - .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc + .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.cache config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243 From des at projects.linpro.no Thu Nov 8 21:34:15 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 8 Nov 2007 22:34:15 +0100 (CET) Subject: r2245 - branches/1.1 Message-ID: <20071108213415.77FD61EC2B2@projects.linpro.no> Author: des Date: 2007-11-08 22:34:15 +0100 (Thu, 08 Nov 2007) New Revision: 2245 Modified: branches/1.1/ Log: Merged revisions 2243 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2243 | des | 2007-11-08 22:33:53 +0100 (Thu, 08 Nov 2007) | 2 lines Ignore config.cache. ........ Property changes on: branches/1.1 ___________________________________________________________________ Name: svn:ignore - .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc + .deps Makefile Makefile.in aclocal.m4 autom4te.cache compile config.cache config.guess config.h config.h.in config.log config.status config.sub configure configure.lineno depcomp install-sh libtool ltmain.sh missing stamp-h1 varnishapi.pc Name: svnmerge-integrated - /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2232 + /trunk/varnish-cache:1-1722,1727-1729,1733,1738,1743-1777,1779-1798,1800-1815,1817,1819,1823,1830-1838,1846,1853-1855,1857-1859,1862,1865-1868,1871-1880,1883-1884,1886,1888-1889,1896,1898,1902-1905,1907,1909,1912-1916,1920-1928,1935-1939,1941-1949,1955,1957-1958,1967-1968,1970-1974,1976-1977,1984,1986-1989,1991-1998,2026,2031-2033,2057,2077-2080,2086,2088,2097,2106-2107,2116,2133,2154,2173,2181,2206,2211-2212,2215-2220,2222-2232,2243 From des at linpro.no Fri Nov 9 10:45:18 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 11:45:18 +0100 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav's?= message of "Thu, 08 Nov 2007 22:16:46 +0100") References: <52830.1194554895@critter.freebsd.dk> Message-ID: Dag-Erling Sm?rgrav writes: > "Poul-Henning Kamp" writes: > > If we get two events for the fd in the same batch, this fix will > > not save us. I'm not sure if FreeBSD's current kqueue implementation > > prevents that from happening. > I think that in such a case, we would get a single kevent with > (kp->data > 0) and (kp->flags & EV_EOF). I'll have to check the code > to be sure. I checked the code, this is indeed the case. The bad news is that we just caught an assert in SES_Delete() again, so there must be another bug in here somewhere. More thinking: - VTAILQ_REMOVE() is not idempotent, removing an element from the list twice (as will happen when this race occurs) will corrupt the list. - The tail queue should probably be a binary heap instead. - kevent changes can be buffered safely if we flush the buffer at the end of the loop in vca_kqueue_main(). Since we process (at most) NKEV kevents at a time, we will never have more than NKEV kevent changes to apply at that point, so it's safe to use a static buffer. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Nov 9 10:48:41 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 09 Nov 2007 10:48:41 +0000 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Fri, 09 Nov 2007 11:45:18 +0100." Message-ID: <55730.1194605321@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: > - kevent changes can be buffered safely if we flush the buffer at the > end of the loop in vca_kqueue_main(). Since we process (at most) > NKEV kevents at a time, we will never have more than NKEV kevent > changes to apply at that point, so it's safe to use a static > buffer. That's not true. We can receive multiple sessions in one read from the pipe. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Nov 9 10:51:57 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 11:51:57 +0100 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: <55730.1194605321@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 09 Nov 2007 10:48:41 +0000") References: <55730.1194605321@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg > rav?= writes: > > - kevent changes can be buffered safely if we flush the buffer at the > > end of the loop in vca_kqueue_main(). Since we process (at most) > > NKEV kevents at a time, we will never have more than NKEV kevent > > changes to apply at that point, so it's safe to use a static > > buffer. > That's not true. We can receive multiple sessions in one read from > the pipe. Ah, right. So we flush the buffer when it's full, plus at the end of the loop. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Nov 9 10:53:44 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 09 Nov 2007 10:53:44 +0000 Subject: r2232 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Fri, 09 Nov 2007 11:51:57 +0100." Message-ID: <55761.1194605624@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: >"Poul-Henning Kamp" writes: >> In message , =3D?iso-8859-1?Q?Dag-Erling= >_Sm=3DF8rg >> rav?=3D writes: >> > - kevent changes can be buffered safely if we flush the buffer at the >> > end of the loop in vca_kqueue_main(). Since we process (at most) >> > NKEV kevents at a time, we will never have more than NKEV kevent >> > changes to apply at that point, so it's safe to use a static >> > buffer. >> That's not true. We can receive multiple sessions in one read from >> the pipe. > >Ah, right. So we flush the buffer when it's full, plus at the end of >the loop. Yes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at projects.linpro.no Fri Nov 9 11:19:46 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 12:19:46 +0100 (CET) Subject: r2246 - trunk/varnish-cache/bin/varnishd Message-ID: <20071109111946.032171EC50E@projects.linpro.no> Author: des Date: 2007-11-09 12:19:45 +0100 (Fri, 09 Nov 2007) New Revision: 2246 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Log: When diagnostics are enabled, log every kevent that has a session associated with it. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-08 21:34:15 UTC (rev 2245) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-09 11:19:45 UTC (rev 2246) @@ -98,6 +98,11 @@ return; } CAST_OBJ_NOTNULL(sp, kp->udata, SESS_MAGIC); +#ifdef DIAGNOSTICS + VSL(SLT_Debug, sp->id, "sp %p kev data %lu flags 0x%x%s", + sp, (unsigned long)kp->data, kp->flags, + (kp->flags & EV_EOF) ? " EOF" : ""); +#endif if (kp->data > 0) { i = HTC_Rx(sp->htc); if (i == 0) From des at projects.linpro.no Fri Nov 9 12:43:10 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 13:43:10 +0100 (CET) Subject: r2247 - trunk/varnish-cache/lib/libvcl Message-ID: <20071109124310.22C131EC407@projects.linpro.no> Author: des Date: 2007-11-09 13:43:09 +0100 (Fri, 09 Nov 2007) New Revision: 2247 Modified: trunk/varnish-cache/lib/libvcl/vcc_compile.c Log: Whitespace cleanup. Modified: trunk/varnish-cache/lib/libvcl/vcc_compile.c =================================================================== --- trunk/varnish-cache/lib/libvcl/vcc_compile.c 2007-11-09 11:19:45 UTC (rev 2246) +++ trunk/varnish-cache/lib/libvcl/vcc_compile.c 2007-11-09 12:43:09 UTC (rev 2247) @@ -216,7 +216,7 @@ break; case '\n': vsb_printf(sb, "\\n"); - if (mode) + if (mode) vsb_printf(sb, "\"\n\t\""); break; case '\t': vsb_printf(sb, "\\t"); break; @@ -327,7 +327,7 @@ Fc(tl, 0, ",\n"); } Fc(tl, 0, "};\n"); - + Fc(tl, 0, "\nconst char *srcbody[%u] = {\n", tl->nsources); VTAILQ_FOREACH(sp, &tl->sources, list) { Fc(tl, 0, " /* "); @@ -355,7 +355,7 @@ Fc(tl, 0, "\t.nhashcount = %u,\n", tl->nhashcount); #define VCL_RET_MAC(l,u,b,n) #define VCL_MET_MAC(l,u,b) \ - Fc(tl, 0, "\t." #l "_func = VGC_function_vcl_" #l ",\n"); + Fc(tl, 0, "\t." #l "_func = VGC_function_vcl_" #l ",\n"); #include "vcl_returns.h" #undef VCL_MET_MAC #undef VCL_RET_MAC @@ -385,7 +385,7 @@ if (sp->freeit != NULL) free(sp->freeit); - free(sp->name); + free(sp->name); free(sp); } @@ -530,7 +530,7 @@ VTAILQ_REMOVE(&tl->sources, sp, list); vcc_destroy_source(sp); } - + vsb_delete(tl->fh); vsb_delete(tl->fc); vsb_delete(tl->fi); From des at projects.linpro.no Fri Nov 9 14:03:13 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 15:03:13 +0100 (CET) Subject: r2248 - in trunk/varnish-cache: include lib/libvarnish Message-ID: <20071109140313.D59631EC50E@projects.linpro.no> Author: des Date: 2007-11-09 15:03:13 +0100 (Fri, 09 Nov 2007) New Revision: 2248 Added: trunk/varnish-cache/lib/libvarnish/vtmpfile.c Modified: trunk/varnish-cache/include/libvarnish.h Log: Add our own mkstemp(3) clone which allows the template to contain non- placeholder characters after the placeholders. Modified: trunk/varnish-cache/include/libvarnish.h =================================================================== --- trunk/varnish-cache/include/libvarnish.h 2007-11-09 12:43:09 UTC (rev 2247) +++ trunk/varnish-cache/include/libvarnish.h 2007-11-09 14:03:13 UTC (rev 2248) @@ -54,7 +54,8 @@ /* from libvarnish/version.c */ void varnish_version(const char *); -/* from libvarnish/assert.c */ +/* from libvarnish/vtmpfile.c */ +int vtmpfile(char *); /* * assert(), AN() and AZ() are static checks that should not happen. Added: trunk/varnish-cache/lib/libvarnish/vtmpfile.c =================================================================== --- trunk/varnish-cache/lib/libvarnish/vtmpfile.c (rev 0) +++ trunk/varnish-cache/lib/libvarnish/vtmpfile.c 2007-11-09 14:03:13 UTC (rev 2248) @@ -0,0 +1,73 @@ +/*- + * Copyright (c) 2006 Verdens Gang AS + * Copyright (c) 2006-2007 Linpro AS + * All rights reserved. + * + * Author: Dag-Erling Sm?rgrav + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $Id$ + */ + +#include +#include +#include +#include + +#include "libvarnish.h" + +int +vtmpfile(char *template) +{ + char *b, *e, *p; + int fd, ran; + + for (b = template; *b != '#'; ++b) + /* nothing */ ; + if (*b == '\0') { + errno = EINVAL; + return (-1); + } + for (e = b; *e == '#'; ++e) + /* nothing */ ; + + for (;;) { + for (p = b; p < e; ++p) { + ran = random() % 63; + if (ran < 10) + *p = '0' + ran; + else if (ran < 36) + *p = 'A' + ran - 10; + else if (ran < 62) + *p = 'a' + ran - 36; + else + *p = '_'; + } + fd = open(template, O_RDWR|O_CREAT|O_EXCL, 0600); + if (fd >= 0) + return (fd); + if (errno != EEXIST) + return (-1); + } + /* not reached */ +} Property changes on: trunk/varnish-cache/lib/libvarnish/vtmpfile.c ___________________________________________________________________ Name: svn:keywords + Id From des at projects.linpro.no Fri Nov 9 14:10:15 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 15:10:15 +0100 (CET) Subject: r2249 - trunk/varnish-cache/bin/varnishd Message-ID: <20071109141015.BE5701EC407@projects.linpro.no> Author: des Date: 2007-11-09 15:10:15 +0100 (Fri, 09 Nov 2007) New Revision: 2249 Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c trunk/varnish-cache/bin/varnishd/mgt_vcc.c Log: Significant rewrite of the code used to invoke the C compiler. The most important functional change is that the source is no longer piped to the compiler; instead, the source file name (which now ends in .c) is passed on the command line. This makes it *much* easier to support non-GNU compilers. The cc_command parameter processed by a custom function instead of sprintf; "%s" is replaced with the source file name, "%o" is replaced with the output file name, and "%%" is replaced with a single "%". Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2007-11-09 14:03:13 UTC (rev 2248) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2007-11-09 14:10:15 UTC (rev 2249) @@ -541,14 +541,6 @@ if (arg == NULL) { cli_out(cli, "%s", mgt_cc_cmd); } else { -#if defined(HAVE_FMTCHECK) - if (arg != fmtcheck(arg, "%s %s")) { - cli_out(cli, - "Parameter has dangerous %%-string expansions."); - cli_result(cli, CLIS_PARAM); - return; - } -#endif free(mgt_cc_cmd); mgt_cc_cmd = strdup(arg); XXXAN(mgt_cc_cmd); @@ -764,16 +756,14 @@ "2", "seconds" }, { "cc_command", tweak_cc_command, "Command used for compiling the C source code to a " - "dlopen(3) loadable object.\n" - "NB: The string must contain two %s sequences which " - "will be replaced by the binary and source file names " - "respectively.\n" + "dlopen(3) loadable object. Any occurrence of %s in " + "the string will be replaced with the source file name, " + "and %o will be replaced with the output file name.\n" MUST_RELOAD, #ifdef __APPLE__ - "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %s -x c" - " - < %s" + "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" #else - "exec cc -nostdinc -fpic -shared -Wl,-x -o %s -x c - < %s" + "exec cc -fpic -shared -Wl,-x -o %o %s" #endif , NULL }, { "max_restarts", tweak_max_restarts, Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2007-11-09 14:03:13 UTC (rev 2248) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2007-11-09 14:10:15 UTC (rev 2249) @@ -32,13 +32,14 @@ */ #include +#include #include +#include #include #include #include #include -#include #ifndef HAVE_ASPRINTF #include "compat/asprintf.h" @@ -119,7 +120,7 @@ "\n" "sub vcl_fetch {\n" " if (!obj.valid) {\n" - " error;\n" + " error obj.status;\n" " }\n" " if (!obj.cacheable) {\n" " pass;\n" @@ -139,147 +140,154 @@ " discard;\n" "}\n"; +/* + * Prepare the compiler command line + */ +static void +mgt_make_cc_cmd(struct vsb *sb, const char *sf, const char *of) +{ + int pct; + char *p; + + for (p = mgt_cc_cmd, pct = 0; *p; ++p) { + if (pct) { + switch (*p) { + case 's': + vsb_cat(sb, sf); + break; + case 'o': + vsb_cat(sb, of); + break; + case '%': + vsb_putc(sb, '%'); + break; + default: + vsb_putc(sb, '%'); + vsb_putc(sb, *p); + break; + } + pct = 0; + } else if (*p == '%') { + pct = 1; + } else { + vsb_putc(sb, *p); + } + } + if (pct) + vsb_putc(sb, '%'); +} + /*-------------------------------------------------------------------- * Invoke system C compiler on source and return resulting dlfile. * Errors goes in sb; */ static char * -mgt_CallCc(const char *source, struct vsb *sb) +mgt_run_cc(const char *source, struct vsb *sb) { - FILE *fo, *fs; - char sf[] = "./vcl.XXXXXXXX"; + char cmdline[1024]; + struct vsb cmdsb; + char sf[] = "./vcl.########.c"; char *of; - struct vsb *cccmd; char buf[128]; - int i, j, sfd; - void *p; + int p[2], sfd, srclen, status; + pid_t pid; + void *dlh; /* Create temporary C source file */ - sfd = mkstemp(sf); + sfd = vtmpfile(sf); if (sfd < 0) { vsb_printf(sb, - "Cannot open temporary source file \"%s\": %s\n", - sf, strerror(errno)); + "%s(): failed to create %s: %s", + __func__, sf, strerror(errno)); return (NULL); } - fs = fdopen(sfd, "r+"); - XXXAN(fs); - - if (fputs(source, fs) < 0 || fflush(fs)) { + srclen = strlen(source); + if (write(sfd, source, srclen) != srclen) { vsb_printf(sb, - "Write error to C source file: %s\n", + "Failed to write C source to file: %s", strerror(errno)); AZ(unlink(sf)); - AZ(fclose(fs)); + AZ(close(sfd)); return (NULL); } - rewind(fs); + AZ(close(sfd)); - /* Name the output shared library by brutally overwriting "./vcl." */ + /* Name the output shared library by overwriting the final 'c' */ of = strdup(sf); XXXAN(of); - memcpy(of, "./bin", 5); + of[sizeof sf - 2] = 'o'; + vsb_new(&cmdsb, cmdline, sizeof cmdline, 0); + mgt_make_cc_cmd(&cmdsb, sf, of); + vsb_finish(&cmdsb); + /* XXX check vsb state */ - cccmd = vsb_new(NULL, NULL, 0, VSB_AUTOEXTEND); - XXXAN(cccmd); - - /* - * Attempt to open a pipe to the system C-compiler. - * ------------------------------------------------ - * - * The arguments to the C-compiler must be whatever it takes to - * create a dlopen(3) compatible object file named $of from a - * source file named $sf. - * - * The source code is entirely selfcontained, so options should be - * specified to prevent the C-compiler from doing any DWITYW - * processing. For GCC this amounts to at least "-nostdinc". - * - * We wrap the entire command in a 'sh -c "..." 2>&1' to get any - * errors from popen(3)'s shell redirected to our stderr as well. - * - */ - vsb_printf(cccmd, "exec /bin/sh -c \"" ); - vsb_printf(cccmd, mgt_cc_cmd, of, sf); - vsb_printf(cccmd, "\" 2>&1"); - vsb_finish(cccmd); - /* XXX: check that vsb is happy about cccmd */ - - fo = popen(vsb_data(cccmd), "r"); - if (fo == NULL) { - vsb_printf(sb, - "System error: Cannot execute C-compiler: %s\n" - "\tcommand attempted: %s\n", - strerror(errno), vsb_data(cccmd)); + if (pipe(p) < 0) { + vsb_printf(sb, "%s(): pipe() failed: %s", + __func__, strerror(errno)); + unlink(sf); free(of); - AZ(unlink(sf)); - AZ(fclose(fs)); - vsb_delete(cccmd); return (NULL); } - - /* Any output is considered fatal */ - j = 0; - while (1) { - i = fread(buf, 1, sizeof buf, fo); - if (i == 0) - break; - if (!j) { - vsb_printf(sb, - "System error:\n" - "C-compiler command complained.\n" - "Command attempted: %s\nMessage:\n", - vsb_data(cccmd)); - j++; - } - vsb_bcat(sb, buf, i); + if ((pid = fork()) < 0) { + vsb_printf(sb, "%s(): fork() failed: %s", + __func__, strerror(errno)); + close(p[0]); + close(p[1]); + unlink(sf); + free(of); + return (NULL); } - - i = pclose(fo); - - AZ(unlink(sf)); - AZ(fclose(fs)); - - if (j == 0 && i != 0) { - vsb_printf(sb, - "System error:\n" - "C-compiler command failed"); - if (WIFEXITED(i)) - vsb_printf(sb, ", exit %d", WEXITSTATUS(i)); - if (WIFSIGNALED(i)) - vsb_printf(sb, ", signal %d", WTERMSIG(i)); - if (WCOREDUMP(i)) - vsb_printf(sb, ", core dumped"); - vsb_printf(sb, - "\nCommand attempted: %s\n", vsb_data(cccmd)); + if (pid == 0) { + close(p[0]); + close(STDIN_FILENO); + assert(open("/dev/null", O_RDONLY) == STDIN_FILENO); + assert(dup2(p[1], STDOUT_FILENO) == STDOUT_FILENO); + assert(dup2(p[1], STDERR_FILENO) == STDERR_FILENO); + execl("/bin/sh", "/bin/sh", "-c", cmdline, NULL); + _exit(1); } - - vsb_delete(cccmd); - - /* If the compiler complained, or exited non-zero, fail */ - if (i || j) { - (void)unlink(of); /* May or may not have created file */ + close(p[1]); + while (read(p[0], buf, sizeof buf) > 0) + /* XXX nothing */ ; + close(p[0]); + unlink(sf); + if (waitpid(pid, &status, 0) < 0) { + vsb_printf(sb, "%s(): waitpid() failed: %s", + __func__, strerror(errno)); + unlink(of); free(of); return (NULL); } + if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) { + vsb_printf(sb, "%s(): Compiler failed", __func__); + if (WIFEXITED(status)) + vsb_printf(sb, ", exit %d", WEXITSTATUS(status)); + if (WIFSIGNALED(status)) + vsb_printf(sb, ", signal %d", WTERMSIG(status)); + if (WCOREDUMP(status)) + vsb_printf(sb, ", core dumped"); + unlink(of); + free(of); + return (NULL); + } /* Next, try to load the object into the management process */ - p = dlopen(of, RTLD_NOW | RTLD_LOCAL); - if (p == NULL) { - vsb_printf(sb, "Problem loading compiled VCL program:\n\t%s\n", - dlerror()); - AZ(unlink(of)); + if ((dlh = dlopen(of, RTLD_NOW | RTLD_LOCAL)) == NULL) { + vsb_printf(sb, + "%s(): failed to load compiled VCL program: %s", + __func__, dlerror()); + unlink(of); free(of); return (NULL); - } + } /* * XXX: we should look up and check the handle in the loaded * object */ - AZ(dlclose(p)); + AZ(dlclose(dlh)); return (of); } @@ -294,7 +302,7 @@ if (csrc != NULL) { if (C_flag) (void)fputs(csrc, stdout); - vf = mgt_CallCc(csrc, sb); + vf = mgt_run_cc(csrc, sb); if (C_flag && vf != NULL) AZ(unlink(vf)); free(csrc); @@ -311,7 +319,7 @@ if (csrc != NULL) { if (C_flag) fputs(csrc, stdout); - vf = mgt_CallCc(csrc, sb); + vf = mgt_run_cc(csrc, sb); if (C_flag && vf != NULL) AZ(unlink(vf)); free(csrc); From des at projects.linpro.no Fri Nov 9 14:44:27 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 15:44:27 +0100 (CET) Subject: r2250 - trunk/varnish-cache/bin/varnishd Message-ID: <20071109144427.B76931EC407@projects.linpro.no> Author: des Date: 2007-11-09 15:44:27 +0100 (Fri, 09 Nov 2007) New Revision: 2250 Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c Log: Clean up the code a bit, and warn the user when changing a parameter that requires the child to be restarted or the VCL script to be reloaded. Modified: trunk/varnish-cache/bin/varnishd/mgt_param.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_param.c 2007-11-09 14:10:15 UTC (rev 2249) +++ trunk/varnish-cache/bin/varnishd/mgt_param.c 2007-11-09 14:44:27 UTC (rev 2250) @@ -61,6 +61,11 @@ const char *name; tweak_t *func; const char *descr; + int flags; +#define DELAYED_EFFECT 1 +#define EXPERIMENTAL 2 +#define MUST_RESTART 4 +#define MUST_RELOAD 8 const char *def; const char *units; }; @@ -537,6 +542,7 @@ tweak_cc_command(struct cli *cli, struct parspec *par, const char *arg) { + /* XXX should have tweak_generic_string */ (void)par; if (arg == NULL) { cli_out(cli, "%s", mgt_cc_cmd); @@ -570,20 +576,20 @@ * formatting will go haywire. */ -#define DELAYED_EFFECT \ - "\nNB: This parameter will take some time to take effect.\n" +#define DELAYED_EFFECT_TEXT \ + "NB: This parameter will take some time to take effect." -#define MUST_RESTART \ - "\nNB: This parameter will not take any effect until the " \ - "child process has been restarted.\n" +#define MUST_RESTART_TEXT \ + "NB: This parameter will not take any effect until the " \ + "child process has been restarted." -#define MUST_RELOAD \ - "\nNB: This parameter will not take any effect until the " \ - "VCL programs have been reloaded.\n" +#define MUST_RELOAD_TEXT \ + "NB: This parameter will not take any effect until the " \ + "VCL programs have been reloaded." -#define EXPERIMENTAL \ - "\nNB: We don't know yet if it is a good idea to change " \ - "this parameter. Caution advised.\n" +#define EXPERIMENTAL_TEXT \ + "NB: We don't know yet if it is a good idea to change " \ + "this parameter. Caution advised." /* * Remember to update varnishd.1 whenever you add / remove a parameter or @@ -592,11 +598,11 @@ static struct parspec parspec[] = { { "user", tweak_user, "The unprivileged user to run as. Setting this will " - "also set \"group\" to the specified user's primary group.\n" + "also set \"group\" to the specified user's primary group.", MUST_RESTART, MAGIC_INIT_STRING }, { "group", tweak_group, - "The unprivileged group to run as.\n" + "The unprivileged group to run as.", MUST_RESTART, MAGIC_INIT_STRING }, { "default_ttl", tweak_default_ttl, @@ -606,89 +612,77 @@ "made until they are fetched from the backend again.\n" "To force an immediate effect at the expense of a total " "flush of the cache use \"url.purge .\"", + 0, "120", "seconds" }, { "thread_pools", tweak_thread_pools, "Number of worker pools. " "Increasing number of worker pools decreases lock " "contention but increases the number of threads as well. " "Can be increased on the fly, but decreases require a " - "restart to take effect.\n" + "restart to take effect.", EXPERIMENTAL, "1", "pools" }, { "thread_pool_max", tweak_thread_pool_max, "The maximum number of threads in the total worker pool.\n" - "-1 is unlimited.\n" - EXPERIMENTAL - DELAYED_EFFECT, + "-1 is unlimited.", + EXPERIMENTAL | DELAYED_EFFECT, "1000", "threads" }, { "thread_pool_min", tweak_thread_pool_min, "The minimum number of threads in the worker pool.\n" - "Minimum is 1 thread. " - EXPERIMENTAL - DELAYED_EFFECT, + "Minimum is 1 thread.", + EXPERIMENTAL | DELAYED_EFFECT, "1", "threads" }, { "thread_pool_timeout", tweak_thread_pool_timeout, "Thread dies after this many seconds of inactivity.\n" - "Minimum is 1 second. " - EXPERIMENTAL - DELAYED_EFFECT, + "Minimum is 1 second.", + EXPERIMENTAL | DELAYED_EFFECT, "120", "seconds" }, { "overflow_max", tweak_overflow_max, "Limit on overflow queue length in percent of " - "thread_pool_max parameter.\n" + "thread_pool_max parameter.", EXPERIMENTAL, "100", "%" }, { "http_workspace", tweak_http_workspace, "Bytes of HTTP protocol workspace allocated. " "This space must be big enough for the entire HTTP protocol " "header and any edits done to it in the VCL code.\n" - "Minimum is 1024 bytes. " + "Minimum is 1024 bytes.", DELAYED_EFFECT, "8192", "bytes" }, { "sess_timeout", tweak_sess_timeout, "Idle timeout for persistent sessions. " "If a HTTP request has not been received in this many " - "seconds, the session is closed.\n", + "seconds, the session is closed.", + 0, "5", "seconds" }, { "pipe_timeout", tweak_pipe_timeout, "Idle timeout for PIPE sessions. " "If nothing have been received in either direction for " "this many seconds, the session is closed.\n", + 0, "60", "seconds" }, { "send_timeout", tweak_send_timeout, "Send timeout for client connections. " "If no data has been sent to the client in this many seconds, " "the session is closed.\n" - "See setsockopt(2) under SO_SNDTIMEO for more information.\n" + "See setsockopt(2) under SO_SNDTIMEO for more information.", DELAYED_EFFECT, "600", "seconds" }, { "auto_restart", tweak_auto_restart, "Restart child process automatically if it dies.\n", + 0, "on", "bool" }, { "fetch_chunksize", tweak_fetch_chunksize, "The default chunksize used by fetcher. " "This should be bigger than the majority of objects with " "short TTLs.\n" "Internal limits in the storage_file module makes increases " - "above 128kb a dubious idea.\n" + "above 128kb a dubious idea.", EXPERIMENTAL, "128", "kilobytes" }, #ifdef HAVE_SENDFILE { "sendfile_threshold", tweak_sendfile_threshold, - "The minimum size of objects transmitted with sendfile.\n" -#if defined(__FreeBSD__) - "In \"plenty-of-RAM\" scenarios this is unlikely to " - "have any effect. Once disk-I/O becomes frequent " - "we guess smaller values are likely to be better.\n" -#elif defined(__Linux__) - "Linux sendfile(2) does not allow for inclusion of " - "header data and therefore using sendfile(2) means " - "an extra system call, compared to using writev(2) for " - "both the header and body.\n" - "We suspect that sendfile(2) on Linux will only start " - "to be beneficial in low-ram scenarios. Therefore it " - "may make sense to set this to \"unlimited\".\n" -#endif + "The minimum size of objects transmitted with sendfile.", EXPERIMENTAL, "-1", "bytes" }, #endif /* HAVE_SENDFILE */ @@ -697,53 +691,50 @@ "Enabling this will allow you to see the path each " "request has taken through the VCL program.\n" "This generates a lot of logrecords so it is off by " - "default. ", + "default.", + 0, "off", "bool" }, { "listen_address", tweak_listen_address, "Whitespace separated list of network endpoints where " "Varnish will accept requests.\n" - "Possible formats: host, host:port, :port\n" + "Possible formats: host, host:port, :port", MUST_RESTART, ":80" }, { "listen_depth", tweak_listen_depth, - "Listen(2) queue depth.\n" -#if defined(__FreeBSD__) - "Please see FreeBSDs tuning(7) manual page for more " - "information.\n" -#endif + "Listen queue depth.", MUST_RESTART, "1024", "connections" }, { "srcaddr_hash", tweak_srcaddr_hash, "Number of source address hash buckets.\n" - "Powers of two are bad, prime numbers are good.\n" - EXPERIMENTAL - MUST_RESTART, + "Powers of two are bad, prime numbers are good.", + EXPERIMENTAL | MUST_RESTART, "1049", "buckets" }, { "srcaddr_ttl", tweak_srcaddr_ttl, "Lifetime of srcaddr entries.\n" - "Zero will disable srcaddr accounting entirely.\n" + "Zero will disable srcaddr accounting entirely.", EXPERIMENTAL, "30", "seconds" }, { "backend_http11", tweak_backend_http11, "Force all backend requests to be HTTP/1.1.\n" "By default we copy the protocol version from the " - "incoming client request." + "incoming client request.", EXPERIMENTAL, "off", "bool" }, { "client_http11", tweak_client_http11, "Force all client responses to be HTTP/1.1.\n" "By default we copy the protocol version from the " - "backend response." + "backend response.", EXPERIMENTAL, "off", "bool" }, { "cli_timeout", tweak_cli_timeout, "Timeout for the childs replies to CLI requests from " - "the master.\n", + "the master.", + 0, "5", "seconds" }, { "ping_interval", tweak_ping_interval, "Interval between pings from parent to child.\n" "Zero will disable pinging entirely, which makes " - "it possible to attach a debugger to the child.\n" + "it possible to attach a debugger to the child.", MUST_RESTART, "3", "seconds" }, { "lru_interval", tweak_lru_timeout, @@ -751,14 +742,14 @@ "Objects are only moved to the front of the LRU " "list if they have not been moved there already inside " "this timeout period. This reduces the amount of lock " - "operations necessary for LRU list access.\n" + "operations necessary for LRU list access.", EXPERIMENTAL, "2", "seconds" }, { "cc_command", tweak_cc_command, "Command used for compiling the C source code to a " "dlopen(3) loadable object. Any occurrence of %s in " "the string will be replaced with the source file name, " - "and %o will be replaced with the output file name.\n" + "and %o will be replaced with the output file name.", MUST_RELOAD, #ifdef __APPLE__ "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" @@ -770,22 +761,49 @@ "Upper limit on how many times a request can restart." "\nBe aware that restarts are likely to cause a hit against " "the backend, so don't increase thoughtlessly.\n", + 0, "4", "restarts" }, { "max_esi_includes", tweak_max_esi_includes, "Maximum depth of esi:include processing." "\nBe aware that restarts are likely to cause a hit against " "the backend, so don't increase thoughtlessly.\n", + 0, "5", "restarts" }, { NULL, NULL, NULL } }; /*--------------------------------------------------------------------*/ +#define WIDTH 76 +#define MARGIN 20 + +static void +mcf_wrap(struct cli *cli, const char *text) +{ + const char *p, *q; + + /* Format text to COLUMNS width */ + for (p = text; *p != '\0'; ) { + q = strchr(p, '\n'); + if (q == NULL) + q = strchr(p, '\0'); + if (q > p + WIDTH - MARGIN) { + q = p + WIDTH - MARGIN; + while (q > p && *q != ' ') + q--; + AN(q); + } + cli_out(cli, "%*s %.*s\n", MARGIN, "", (int)(q - p), p); + p = q; + if (*p == ' ' || *p == '\n') + p++; + } +} + void mcf_param_show(struct cli *cli, const char * const *av, void *priv) { struct parspec *pp; - const char *p, *q; int lfmt; (void)priv; @@ -796,7 +814,7 @@ for (pp = parspec; pp->name != NULL; pp++) { if (av[2] != NULL && !lfmt && strcmp(pp->name, av[2])) continue; - cli_out(cli, "%-20s ", pp->name); + cli_out(cli, "%-*s ", MARGIN, pp->name); if (pp->func == NULL) { cli_out(cli, "Not implemented.\n"); if (av[2] != NULL && !lfmt) @@ -810,23 +828,17 @@ else cli_out(cli, "\n"); if (av[2] != NULL) { - cli_out(cli, "%-20s Default is %s\n", "", pp->def); - /* Format text to 72 col width */ - for (p = pp->descr; *p != '\0'; ) { - q = strchr(p, '\n'); - if (q == NULL) - q = strchr(p, '\0'); - if (q > p + 52) { - q = p + 52; - while (q > p && *q != ' ') - q--; - AN(q); - } - cli_out(cli, "%20s %.*s\n", "", q - p, p); - p = q; - if (*p == ' ' || *p == '\n') - p++; - } + cli_out(cli, "%-*s Default is %s\n", + MARGIN, "", pp->def); + mcf_wrap(cli, pp->descr); + if (pp->flags & DELAYED_EFFECT) + mcf_wrap(cli, DELAYED_EFFECT_TEXT); + if (pp->flags & EXPERIMENTAL) + mcf_wrap(cli, EXPERIMENTAL_TEXT); + if (pp->flags & MUST_RELOAD) + mcf_wrap(cli, MUST_RELOAD_TEXT); + if (pp->flags & MUST_RESTART) + mcf_wrap(cli, MUST_RESTART_TEXT); if (!lfmt) return; else @@ -835,7 +847,7 @@ } if (av[2] != NULL && !lfmt) { cli_result(cli, CLIS_PARAM); - cli_out(cli, "Unknown paramter \"%s\".", av[2]); + cli_out(cli, "Unknown parameter \"%s\".", av[2]); } } @@ -858,12 +870,18 @@ for (pp = parspec; pp->name != NULL; pp++) { if (!strcmp(pp->name, param)) { pp->func(cli, pp, val); + if (pp->flags & MUST_RESTART) + cli_out(cli, "change will take effect" + " when child is restarted"); + if (pp->flags & MUST_RELOAD) + cli_out(cli, "change will take effect" + " when VCL script is reloaded"); MCF_ParamSync(); return; } } cli_result(cli, CLIS_PARAM); - cli_out(cli, "Unknown paramter \"%s\".", param); + cli_out(cli, "Unknown parameter \"%s\".", param); } From des at projects.linpro.no Fri Nov 9 22:59:45 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Fri, 9 Nov 2007 23:59:45 +0100 (CET) Subject: r2251 - trunk/varnish-cache/lib/libvarnish Message-ID: <20071109225945.1E9871EC396@projects.linpro.no> Author: des Date: 2007-11-09 23:59:44 +0100 (Fri, 09 Nov 2007) New Revision: 2251 Modified: trunk/varnish-cache/lib/libvarnish/Makefile.am Log: Forgot to commit this when vtmpfile.c was added. Modified: trunk/varnish-cache/lib/libvarnish/Makefile.am =================================================================== --- trunk/varnish-cache/lib/libvarnish/Makefile.am 2007-11-09 14:44:27 UTC (rev 2250) +++ trunk/varnish-cache/lib/libvarnish/Makefile.am 2007-11-09 22:59:44 UTC (rev 2251) @@ -16,7 +16,8 @@ version.c \ vpf.c \ vsb.c \ - vss.c + vss.c \ + vtmpfile.c libvarnish_la_CFLAGS = -include config.h From des at projects.linpro.no Thu Nov 15 14:18:41 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 15:18:41 +0100 (CET) Subject: r2252 - trunk/varnish-cache/bin/varnishd Message-ID: <20071115141841.4AB901EC407@projects.linpro.no> Author: des Date: 2007-11-15 15:18:39 +0100 (Thu, 15 Nov 2007) New Revision: 2252 Modified: trunk/varnish-cache/bin/varnishd/cache_response.c Log: Whitespace cleanup Modified: trunk/varnish-cache/bin/varnishd/cache_response.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-09 22:59:44 UTC (rev 2251) +++ trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-15 14:18:39 UTC (rev 2252) @@ -126,10 +126,10 @@ http_CopyResp(sp->http, sp->obj->http); http_FilterFields(sp->wrk, sp->fd, sp->http, sp->obj->http, HTTPH_A_DELIVER); - + TIM_format(TIM_real(), time_str); http_PrintfHeader(sp->wrk, sp->fd, sp->http, "Date: %s", time_str); - + if (sp->xid != sp->obj->xid) http_PrintfHeader(sp->wrk, sp->fd, sp->http, "X-Varnish: %u %u", sp->xid, sp->obj->xid); @@ -167,7 +167,7 @@ sp->wrk->acct.hdrbytes += WRK_Write(sp->wrk, lenbuf, -1); } - + VTAILQ_FOREACH(st, &sp->obj->store, list) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); @@ -192,7 +192,7 @@ WRK_Write(sp->wrk, st->ptr, st->len); } assert(u == sp->obj->len); - if (sp->esis > 0) + if (sp->esis > 0) WRK_Write(sp->wrk, "\r\n", -1); } if (WRK_Flush(sp->wrk)) From des at projects.linpro.no Thu Nov 15 15:07:16 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 16:07:16 +0100 (CET) Subject: r2253 - trunk/varnish-cache/include Message-ID: <20071115150716.24B441EC290@projects.linpro.no> Author: des Date: 2007-11-15 16:07:15 +0100 (Thu, 15 Nov 2007) New Revision: 2253 Modified: trunk/varnish-cache/include/miniobj.h Log: Add a VALID_OBJ() predicate macro. Modified: trunk/varnish-cache/include/miniobj.h =================================================================== --- trunk/varnish-cache/include/miniobj.h 2007-11-15 14:18:39 UTC (rev 2252) +++ trunk/varnish-cache/include/miniobj.h 2007-11-15 15:07:15 UTC (rev 2253) @@ -12,6 +12,9 @@ free(to); \ } while (0) +#define VALID_OBJ(ptr, type_magic) \ + ((ptr) != NULL && (ptr)->magic == (type_magic)) + #define CHECK_OBJ(ptr, type_magic) \ do { \ assert((ptr)->magic == type_magic); \ From des at projects.linpro.no Thu Nov 15 16:01:22 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:01:22 +0100 (CET) Subject: r2254 - trunk/varnish-cache/bin/varnishd Message-ID: <20071115160122.0405F1EC528@projects.linpro.no> Author: des Date: 2007-11-15 17:01:21 +0100 (Thu, 15 Nov 2007) New Revision: 2254 Added: trunk/varnish-cache/bin/varnishd/cache_panic.c Modified: trunk/varnish-cache/bin/varnishd/Makefile.am trunk/varnish-cache/bin/varnishd/cache.h Log: Bigger hammer: panic() is similar to lbv_assert() but takes a struct sess * from which it tries to extract as much useful information as possible before it croaks. The corresponding macro is spassert(), which assumes you have a struct sess *sp in scope. This should help a lot with bugs like #167 where important information is not available post-mortem (because sp->obj points into the cache, which is not included in the core dump) Modified: trunk/varnish-cache/bin/varnishd/Makefile.am =================================================================== --- trunk/varnish-cache/bin/varnishd/Makefile.am 2007-11-15 15:07:15 UTC (rev 2253) +++ trunk/varnish-cache/bin/varnishd/Makefile.am 2007-11-15 16:01:21 UTC (rev 2254) @@ -24,6 +24,7 @@ cache_http.c \ cache_httpconn.c \ cache_main.c \ + cache_panic.c \ cache_pool.c \ cache_pipe.c \ cache_response.c \ Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2007-11-15 15:07:15 UTC (rev 2253) +++ trunk/varnish-cache/bin/varnishd/cache.h 2007-11-15 16:01:21 UTC (rev 2254) @@ -668,3 +668,17 @@ return ((unsigned)(t.e - t.b)); } + +#ifdef WITHOUT_ASSERTS +#define spassert(cond) ((void)0) +#else +void panic(const char *, int, const char *, + const struct sess *, const char *, ...); +#define spassert(cond) \ + do { \ + int ok = !!(cond); \ + if (!ok) \ + panic(__FILE__, __LINE__, __func__, sp, \ + "assertion failed: %s\n", #cond); \ + } while (0) +#endif Added: trunk/varnish-cache/bin/varnishd/cache_panic.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_panic.c (rev 0) +++ trunk/varnish-cache/bin/varnishd/cache_panic.c 2007-11-15 16:01:21 UTC (rev 2254) @@ -0,0 +1,188 @@ +/*- + * Copyright (c) 2006 Verdens Gang AS + * Copyright (c) 2006-2007 Linpro AS + * All rights reserved. + * + * Author: Dag-Erling Sm?rgrav + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $Id$ + */ + +#include +#include +#include + +#include "cache.h" + +#ifndef WITHOUT_ASSERTS + +/* panic string */ +char panicstr[65536]; +static char *pstr = panicstr; + +#define fp(...) \ + pstr += snprintf(pstr, \ + (panicstr + sizeof panicstr) - pstr, \ + __VA_ARGS__) +#define vfp(fmt, ap) \ + pstr += vsnprintf(pstr, \ + (panicstr + sizeof panicstr) - pstr, \ + (fmt), (ap)) + +/* step names */ +static const char *steps[] = { +#define STEP(l, u) "STP_" #u, +#include "steps.h" +#undef STEP +}; +static int nsteps = sizeof steps / sizeof *steps; + +/* dump a struct storage */ +static void +dump_storage(const struct storage *st) +{ + int i, j; + +#define MAX_BYTES (4*16) +#define show(ch) (((ch) > 31 && (ch) < 127) ? (ch) : '.') + + for (i = 0; i < MAX_BYTES && i < st->len; i += 16) { + fp(" "); + for (j = 0; j < 16; ++j) { + if (i + j < st->len) + fp("%02x ", st->ptr[i + j]); + else + fp(" "); + } + fp("|"); + for (j = 0; j < 16; ++j) + if (i + j < st->len) + fp("%c", show(st->ptr[i + j])); + fp("|\n"); + } + if (st->len > MAX_BYTES) + fp(" [%u more]\n", st->len - MAX_BYTES); + +#undef show +#undef MAX_BYTES +} + +/* dump a struct http */ +static void +dump_http(const struct http *h) +{ + int i; + + fp(" http = {\n"); + if (h->nhd > HTTP_HDR_FIRST) { + fp(" hd = {\n"); + for (i = HTTP_HDR_FIRST; i < h->nhd; ++i) + fp(" \"%.*s\",\n", + (int)(h->hd[i].e - h->hd[i].b), + h->hd[i].b); + fp(" },\n"); + } + fp(" },\n"); +} + +/* dump a struct object */ +static void +dump_object(const struct object *o) +{ + const struct storage *st; + + fp(" obj = %p {\n", o); + fp(" refcnt = %u, xid = %u,\n", o->refcnt, o->xid); + dump_http(o->http); + fp(" store = {\n"); + VTAILQ_FOREACH(st, &o->store, list) { + dump_storage(st); + } + fp(" },\n"); + fp(" },\n"); +} + +/* dump a struct backend */ +static void +dump_backend(const struct backend *be) +{ + + fp(" backend = %p {\n", be); + fp(" vcl_name = \"%s\",\n", + be->vcl_name ? be->vcl_name : "(null)"); + fp(" },\n"); +} + +/* dump a struct sess */ +static void +dump_sess(const struct sess *sp) +{ + const struct backend *be = sp->backend; + const struct object *obj = sp->obj; + + fp("sp = %p {\n", sp); + fp(" fd = %d, id = %d, xid = %u,\n", sp->fd, sp->id, sp->xid); + fp(" client = %s:%s,\n", + sp->addr ? sp->addr : "?.?.?.?", + sp->port ? sp->port : "?"); + if (sp->step < nsteps) + fp(" step = %s,\n", steps[sp->step]); + else + fp(" step = %d,\n", sp->step); + if (sp->err_code) + fp(" err_code = %d, err_reason = %s,\n", sp->err_code, + sp->err_reason ? sp->err_reason : "(null)"); + + if (VALID_OBJ(be, BACKEND_MAGIC)) + dump_backend(be); + + if (VALID_OBJ(obj, OBJECT_MAGIC)) + dump_object(obj); + + fp("},\n"); +} + +/* report as much information as we can before we croak */ +void +panic(const char *file, int line, const char *func, + const struct sess *sp, const char *fmt, ...) +{ + va_list ap; + + fp("panic in %s() at %s:%d\n", func, file, line); + va_start(ap, fmt); + vfp(fmt, ap); + va_end(ap); + + if (VALID_OBJ(sp, SESS_MAGIC)) + dump_sess(sp); + + fputs(panicstr, stderr); + + /* I wish there was a way to flush the log buffers... */ + signal(SIGABRT, SIG_DFL); + raise(SIGABRT); +} + +#endif Property changes on: trunk/varnish-cache/bin/varnishd/cache_panic.c ___________________________________________________________________ Name: svn:keywords + Id From des at projects.linpro.no Thu Nov 15 16:02:05 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:02:05 +0100 (CET) Subject: r2255 - trunk/varnish-cache/bin/varnishd Message-ID: <20071115160205.83C8C1EC407@projects.linpro.no> Author: des Date: 2007-11-15 17:02:05 +0100 (Thu, 15 Nov 2007) New Revision: 2255 Modified: trunk/varnish-cache/bin/varnishd/cache_response.c Log: Change the assert() that triggers in #167 to spassert(). Modified: trunk/varnish-cache/bin/varnishd/cache_response.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-15 16:01:21 UTC (rev 2254) +++ trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-15 16:02:05 UTC (rev 2255) @@ -191,7 +191,7 @@ VSL_stats->n_objwrite++; WRK_Write(sp->wrk, st->ptr, st->len); } - assert(u == sp->obj->len); + spassert(u == sp->obj->len); if (sp->esis > 0) WRK_Write(sp->wrk, "\r\n", -1); } From des at projects.linpro.no Thu Nov 15 16:03:22 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:03:22 +0100 (CET) Subject: r2256 - in branches/1.2: . bin/varnishd Message-ID: <20071115160322.18FC21EC407@projects.linpro.no> Author: des Date: 2007-11-15 17:03:21 +0100 (Thu, 15 Nov 2007) New Revision: 2256 Modified: branches/1.2/ branches/1.2/bin/varnishd/cache_acceptor_kqueue.c Log: Merged revisions 2246 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2246 | des | 2007-11-09 12:19:45 +0100 (Fri, 09 Nov 2007) | 3 lines When diagnostics are enabled, log every kevent that has a session associated with it. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246 Modified: branches/1.2/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- branches/1.2/bin/varnishd/cache_acceptor_kqueue.c 2007-11-15 16:02:05 UTC (rev 2255) +++ branches/1.2/bin/varnishd/cache_acceptor_kqueue.c 2007-11-15 16:03:21 UTC (rev 2256) @@ -98,6 +98,11 @@ return; } CAST_OBJ_NOTNULL(sp, kp->udata, SESS_MAGIC); +#ifdef DIAGNOSTICS + VSL(SLT_Debug, sp->id, "sp %p kev data %lu flags 0x%x%s", + sp, (unsigned long)kp->data, kp->flags, + (kp->flags & EV_EOF) ? " EOF" : ""); +#endif if (kp->data > 0) { i = HTC_Rx(sp->htc); if (i == 0) From des at projects.linpro.no Thu Nov 15 16:03:57 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:03:57 +0100 (CET) Subject: r2257 - in branches/1.2: . include lib/libvarnish Message-ID: <20071115160357.DBCE21EC290@projects.linpro.no> Author: des Date: 2007-11-15 17:03:57 +0100 (Thu, 15 Nov 2007) New Revision: 2257 Added: branches/1.2/lib/libvarnish/vtmpfile.c Modified: branches/1.2/ branches/1.2/include/libvarnish.h branches/1.2/lib/libvarnish/Makefile.am Log: Merged revisions 2248,2251 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2248 | des | 2007-11-09 15:03:13 +0100 (Fri, 09 Nov 2007) | 3 lines Add our own mkstemp(3) clone which allows the template to contain non- placeholder characters after the placeholders. ........ r2251 | des | 2007-11-09 23:59:44 +0100 (Fri, 09 Nov 2007) | 2 lines Forgot to commit this when vtmpfile.c was added. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248,2251 Modified: branches/1.2/include/libvarnish.h =================================================================== --- branches/1.2/include/libvarnish.h 2007-11-15 16:03:21 UTC (rev 2256) +++ branches/1.2/include/libvarnish.h 2007-11-15 16:03:57 UTC (rev 2257) @@ -54,7 +54,8 @@ /* from libvarnish/version.c */ void varnish_version(const char *); -/* from libvarnish/assert.c */ +/* from libvarnish/vtmpfile.c */ +int vtmpfile(char *); /* * assert(), AN() and AZ() are static checks that should not happen. Modified: branches/1.2/lib/libvarnish/Makefile.am =================================================================== --- branches/1.2/lib/libvarnish/Makefile.am 2007-11-15 16:03:21 UTC (rev 2256) +++ branches/1.2/lib/libvarnish/Makefile.am 2007-11-15 16:03:57 UTC (rev 2257) @@ -16,7 +16,8 @@ version.c \ vpf.c \ vsb.c \ - vss.c + vss.c \ + vtmpfile.c libvarnish_la_CFLAGS = -include config.h Copied: branches/1.2/lib/libvarnish/vtmpfile.c (from rev 2248, trunk/varnish-cache/lib/libvarnish/vtmpfile.c) =================================================================== --- branches/1.2/lib/libvarnish/vtmpfile.c (rev 0) +++ branches/1.2/lib/libvarnish/vtmpfile.c 2007-11-15 16:03:57 UTC (rev 2257) @@ -0,0 +1,73 @@ +/*- + * Copyright (c) 2006 Verdens Gang AS + * Copyright (c) 2006-2007 Linpro AS + * All rights reserved. + * + * Author: Dag-Erling Sm?rgrav + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $Id$ + */ + +#include +#include +#include +#include + +#include "libvarnish.h" + +int +vtmpfile(char *template) +{ + char *b, *e, *p; + int fd, ran; + + for (b = template; *b != '#'; ++b) + /* nothing */ ; + if (*b == '\0') { + errno = EINVAL; + return (-1); + } + for (e = b; *e == '#'; ++e) + /* nothing */ ; + + for (;;) { + for (p = b; p < e; ++p) { + ran = random() % 63; + if (ran < 10) + *p = '0' + ran; + else if (ran < 36) + *p = 'A' + ran - 10; + else if (ran < 62) + *p = 'a' + ran - 36; + else + *p = '_'; + } + fd = open(template, O_RDWR|O_CREAT|O_EXCL, 0600); + if (fd >= 0) + return (fd); + if (errno != EEXIST) + return (-1); + } + /* not reached */ +} From des at projects.linpro.no Thu Nov 15 16:04:25 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:04:25 +0100 (CET) Subject: r2258 - in branches/1.2: . bin/varnishd Message-ID: <20071115160425.167661EC407@projects.linpro.no> Author: des Date: 2007-11-15 17:04:24 +0100 (Thu, 15 Nov 2007) New Revision: 2258 Modified: branches/1.2/ branches/1.2/bin/varnishd/mgt_param.c branches/1.2/bin/varnishd/mgt_vcc.c Log: Merged revisions 2249 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2249 | des | 2007-11-09 15:10:15 +0100 (Fri, 09 Nov 2007) | 10 lines Significant rewrite of the code used to invoke the C compiler. The most important functional change is that the source is no longer piped to the compiler; instead, the source file name (which now ends in .c) is passed on the command line. This makes it *much* easier to support non-GNU compilers. The cc_command parameter processed by a custom function instead of sprintf; "%s" is replaced with the source file name, "%o" is replaced with the output file name, and "%%" is replaced with a single "%". ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248,2251 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2249,2251 Modified: branches/1.2/bin/varnishd/mgt_param.c =================================================================== --- branches/1.2/bin/varnishd/mgt_param.c 2007-11-15 16:03:57 UTC (rev 2257) +++ branches/1.2/bin/varnishd/mgt_param.c 2007-11-15 16:04:24 UTC (rev 2258) @@ -541,14 +541,6 @@ if (arg == NULL) { cli_out(cli, "%s", mgt_cc_cmd); } else { -#if defined(HAVE_FMTCHECK) - if (arg != fmtcheck(arg, "%s %s")) { - cli_out(cli, - "Parameter has dangerous %%-string expansions."); - cli_result(cli, CLIS_PARAM); - return; - } -#endif free(mgt_cc_cmd); mgt_cc_cmd = strdup(arg); XXXAN(mgt_cc_cmd); @@ -756,16 +748,14 @@ "2", "seconds" }, { "cc_command", tweak_cc_command, "Command used for compiling the C source code to a " - "dlopen(3) loadable object.\n" - "NB: The string must contain two %s sequences which " - "will be replaced by the binary and source file names " - "respectively.\n" + "dlopen(3) loadable object. Any occurrence of %s in " + "the string will be replaced with the source file name, " + "and %o will be replaced with the output file name.\n" MUST_RELOAD, #ifdef __APPLE__ - "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %s -x c" - " - < %s" + "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" #else - "exec cc -nostdinc -fpic -shared -Wl,-x -o %s -x c - < %s" + "exec cc -fpic -shared -Wl,-x -o %o %s" #endif , NULL }, { "max_restarts", tweak_max_restarts, Modified: branches/1.2/bin/varnishd/mgt_vcc.c =================================================================== --- branches/1.2/bin/varnishd/mgt_vcc.c 2007-11-15 16:03:57 UTC (rev 2257) +++ branches/1.2/bin/varnishd/mgt_vcc.c 2007-11-15 16:04:24 UTC (rev 2258) @@ -32,13 +32,14 @@ */ #include +#include #include +#include #include #include #include #include -#include #ifndef HAVE_ASPRINTF #include "compat/asprintf.h" @@ -119,7 +120,7 @@ "\n" "sub vcl_fetch {\n" " if (!obj.valid) {\n" - " error;\n" + " error obj.status;\n" " }\n" " if (!obj.cacheable) {\n" " pass;\n" @@ -139,147 +140,154 @@ " discard;\n" "}\n"; +/* + * Prepare the compiler command line + */ +static void +mgt_make_cc_cmd(struct vsb *sb, const char *sf, const char *of) +{ + int pct; + char *p; + + for (p = mgt_cc_cmd, pct = 0; *p; ++p) { + if (pct) { + switch (*p) { + case 's': + vsb_cat(sb, sf); + break; + case 'o': + vsb_cat(sb, of); + break; + case '%': + vsb_putc(sb, '%'); + break; + default: + vsb_putc(sb, '%'); + vsb_putc(sb, *p); + break; + } + pct = 0; + } else if (*p == '%') { + pct = 1; + } else { + vsb_putc(sb, *p); + } + } + if (pct) + vsb_putc(sb, '%'); +} + /*-------------------------------------------------------------------- * Invoke system C compiler on source and return resulting dlfile. * Errors goes in sb; */ static char * -mgt_CallCc(const char *source, struct vsb *sb) +mgt_run_cc(const char *source, struct vsb *sb) { - FILE *fo, *fs; - char sf[] = "./vcl.XXXXXXXX"; + char cmdline[1024]; + struct vsb cmdsb; + char sf[] = "./vcl.########.c"; char *of; - struct vsb *cccmd; char buf[128]; - int i, j, sfd; - void *p; + int p[2], sfd, srclen, status; + pid_t pid; + void *dlh; /* Create temporary C source file */ - sfd = mkstemp(sf); + sfd = vtmpfile(sf); if (sfd < 0) { vsb_printf(sb, - "Cannot open temporary source file \"%s\": %s\n", - sf, strerror(errno)); + "%s(): failed to create %s: %s", + __func__, sf, strerror(errno)); return (NULL); } - fs = fdopen(sfd, "r+"); - XXXAN(fs); - - if (fputs(source, fs) < 0 || fflush(fs)) { + srclen = strlen(source); + if (write(sfd, source, srclen) != srclen) { vsb_printf(sb, - "Write error to C source file: %s\n", + "Failed to write C source to file: %s", strerror(errno)); AZ(unlink(sf)); - AZ(fclose(fs)); + AZ(close(sfd)); return (NULL); } - rewind(fs); + AZ(close(sfd)); - /* Name the output shared library by brutally overwriting "./vcl." */ + /* Name the output shared library by overwriting the final 'c' */ of = strdup(sf); XXXAN(of); - memcpy(of, "./bin", 5); + of[sizeof sf - 2] = 'o'; + vsb_new(&cmdsb, cmdline, sizeof cmdline, 0); + mgt_make_cc_cmd(&cmdsb, sf, of); + vsb_finish(&cmdsb); + /* XXX check vsb state */ - cccmd = vsb_new(NULL, NULL, 0, VSB_AUTOEXTEND); - XXXAN(cccmd); - - /* - * Attempt to open a pipe to the system C-compiler. - * ------------------------------------------------ - * - * The arguments to the C-compiler must be whatever it takes to - * create a dlopen(3) compatible object file named $of from a - * source file named $sf. - * - * The source code is entirely selfcontained, so options should be - * specified to prevent the C-compiler from doing any DWITYW - * processing. For GCC this amounts to at least "-nostdinc". - * - * We wrap the entire command in a 'sh -c "..." 2>&1' to get any - * errors from popen(3)'s shell redirected to our stderr as well. - * - */ - vsb_printf(cccmd, "exec /bin/sh -c \"" ); - vsb_printf(cccmd, mgt_cc_cmd, of, sf); - vsb_printf(cccmd, "\" 2>&1"); - vsb_finish(cccmd); - /* XXX: check that vsb is happy about cccmd */ - - fo = popen(vsb_data(cccmd), "r"); - if (fo == NULL) { - vsb_printf(sb, - "System error: Cannot execute C-compiler: %s\n" - "\tcommand attempted: %s\n", - strerror(errno), vsb_data(cccmd)); + if (pipe(p) < 0) { + vsb_printf(sb, "%s(): pipe() failed: %s", + __func__, strerror(errno)); + unlink(sf); free(of); - AZ(unlink(sf)); - AZ(fclose(fs)); - vsb_delete(cccmd); return (NULL); } - - /* Any output is considered fatal */ - j = 0; - while (1) { - i = fread(buf, 1, sizeof buf, fo); - if (i == 0) - break; - if (!j) { - vsb_printf(sb, - "System error:\n" - "C-compiler command complained.\n" - "Command attempted: %s\nMessage:\n", - vsb_data(cccmd)); - j++; - } - vsb_bcat(sb, buf, i); + if ((pid = fork()) < 0) { + vsb_printf(sb, "%s(): fork() failed: %s", + __func__, strerror(errno)); + close(p[0]); + close(p[1]); + unlink(sf); + free(of); + return (NULL); } - - i = pclose(fo); - - AZ(unlink(sf)); - AZ(fclose(fs)); - - if (j == 0 && i != 0) { - vsb_printf(sb, - "System error:\n" - "C-compiler command failed"); - if (WIFEXITED(i)) - vsb_printf(sb, ", exit %d", WEXITSTATUS(i)); - if (WIFSIGNALED(i)) - vsb_printf(sb, ", signal %d", WTERMSIG(i)); - if (WCOREDUMP(i)) - vsb_printf(sb, ", core dumped"); - vsb_printf(sb, - "\nCommand attempted: %s\n", vsb_data(cccmd)); + if (pid == 0) { + close(p[0]); + close(STDIN_FILENO); + assert(open("/dev/null", O_RDONLY) == STDIN_FILENO); + assert(dup2(p[1], STDOUT_FILENO) == STDOUT_FILENO); + assert(dup2(p[1], STDERR_FILENO) == STDERR_FILENO); + execl("/bin/sh", "/bin/sh", "-c", cmdline, NULL); + _exit(1); } - - vsb_delete(cccmd); - - /* If the compiler complained, or exited non-zero, fail */ - if (i || j) { - (void)unlink(of); /* May or may not have created file */ + close(p[1]); + while (read(p[0], buf, sizeof buf) > 0) + /* XXX nothing */ ; + close(p[0]); + unlink(sf); + if (waitpid(pid, &status, 0) < 0) { + vsb_printf(sb, "%s(): waitpid() failed: %s", + __func__, strerror(errno)); + unlink(of); free(of); return (NULL); } + if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) { + vsb_printf(sb, "%s(): Compiler failed", __func__); + if (WIFEXITED(status)) + vsb_printf(sb, ", exit %d", WEXITSTATUS(status)); + if (WIFSIGNALED(status)) + vsb_printf(sb, ", signal %d", WTERMSIG(status)); + if (WCOREDUMP(status)) + vsb_printf(sb, ", core dumped"); + unlink(of); + free(of); + return (NULL); + } /* Next, try to load the object into the management process */ - p = dlopen(of, RTLD_NOW | RTLD_LOCAL); - if (p == NULL) { - vsb_printf(sb, "Problem loading compiled VCL program:\n\t%s\n", - dlerror()); - AZ(unlink(of)); + if ((dlh = dlopen(of, RTLD_NOW | RTLD_LOCAL)) == NULL) { + vsb_printf(sb, + "%s(): failed to load compiled VCL program: %s", + __func__, dlerror()); + unlink(of); free(of); return (NULL); - } + } /* * XXX: we should look up and check the handle in the loaded * object */ - AZ(dlclose(p)); + AZ(dlclose(dlh)); return (of); } @@ -294,7 +302,7 @@ if (csrc != NULL) { if (C_flag) (void)fputs(csrc, stdout); - vf = mgt_CallCc(csrc, sb); + vf = mgt_run_cc(csrc, sb); if (C_flag && vf != NULL) AZ(unlink(vf)); free(csrc); @@ -311,7 +319,7 @@ if (csrc != NULL) { if (C_flag) fputs(csrc, stdout); - vf = mgt_CallCc(csrc, sb); + vf = mgt_run_cc(csrc, sb); if (C_flag && vf != NULL) AZ(unlink(vf)); free(csrc); From des at projects.linpro.no Thu Nov 15 16:05:43 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:05:43 +0100 (CET) Subject: r2259 - in branches/1.2: . bin/varnishd Message-ID: <20071115160543.234061EC290@projects.linpro.no> Author: des Date: 2007-11-15 17:05:42 +0100 (Thu, 15 Nov 2007) New Revision: 2259 Modified: branches/1.2/ branches/1.2/bin/varnishd/mgt_param.c Log: Merged revisions 2250 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2250 | des | 2007-11-09 15:44:27 +0100 (Fri, 09 Nov 2007) | 3 lines Clean up the code a bit, and warn the user when changing a parameter that requires the child to be restarted or the VCL script to be reloaded. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2249,2251 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2251 Modified: branches/1.2/bin/varnishd/mgt_param.c =================================================================== --- branches/1.2/bin/varnishd/mgt_param.c 2007-11-15 16:04:24 UTC (rev 2258) +++ branches/1.2/bin/varnishd/mgt_param.c 2007-11-15 16:05:42 UTC (rev 2259) @@ -61,6 +61,11 @@ const char *name; tweak_t *func; const char *descr; + int flags; +#define DELAYED_EFFECT 1 +#define EXPERIMENTAL 2 +#define MUST_RESTART 4 +#define MUST_RELOAD 8 const char *def; const char *units; }; @@ -537,6 +542,7 @@ tweak_cc_command(struct cli *cli, struct parspec *par, const char *arg) { + /* XXX should have tweak_generic_string */ (void)par; if (arg == NULL) { cli_out(cli, "%s", mgt_cc_cmd); @@ -562,20 +568,20 @@ * formatting will go haywire. */ -#define DELAYED_EFFECT \ - "\nNB: This parameter will take some time to take effect.\n" +#define DELAYED_EFFECT_TEXT \ + "NB: This parameter will take some time to take effect." -#define MUST_RESTART \ - "\nNB: This parameter will not take any effect until the " \ - "child process has been restarted.\n" +#define MUST_RESTART_TEXT \ + "NB: This parameter will not take any effect until the " \ + "child process has been restarted." -#define MUST_RELOAD \ - "\nNB: This parameter will not take any effect until the " \ - "VCL programs have been reloaded.\n" +#define MUST_RELOAD_TEXT \ + "NB: This parameter will not take any effect until the " \ + "VCL programs have been reloaded." -#define EXPERIMENTAL \ - "\nNB: We don't know yet if it is a good idea to change " \ - "this parameter. Caution advised.\n" +#define EXPERIMENTAL_TEXT \ + "NB: We don't know yet if it is a good idea to change " \ + "this parameter. Caution advised." /* * Remember to update varnishd.1 whenever you add / remove a parameter or @@ -584,11 +590,11 @@ static struct parspec parspec[] = { { "user", tweak_user, "The unprivileged user to run as. Setting this will " - "also set \"group\" to the specified user's primary group.\n" + "also set \"group\" to the specified user's primary group.", MUST_RESTART, MAGIC_INIT_STRING }, { "group", tweak_group, - "The unprivileged group to run as.\n" + "The unprivileged group to run as.", MUST_RESTART, MAGIC_INIT_STRING }, { "default_ttl", tweak_default_ttl, @@ -598,89 +604,77 @@ "made until they are fetched from the backend again.\n" "To force an immediate effect at the expense of a total " "flush of the cache use \"url.purge .\"", + 0, "120", "seconds" }, { "thread_pools", tweak_thread_pools, "Number of worker pools. " "Increasing number of worker pools decreases lock " "contention but increases the number of threads as well. " "Can be increased on the fly, but decreases require a " - "restart to take effect.\n" + "restart to take effect.", EXPERIMENTAL, "1", "pools" }, { "thread_pool_max", tweak_thread_pool_max, "The maximum number of threads in the total worker pool.\n" - "-1 is unlimited.\n" - EXPERIMENTAL - DELAYED_EFFECT, + "-1 is unlimited.", + EXPERIMENTAL | DELAYED_EFFECT, "1000", "threads" }, { "thread_pool_min", tweak_thread_pool_min, "The minimum number of threads in the worker pool.\n" - "Minimum is 1 thread. " - EXPERIMENTAL - DELAYED_EFFECT, + "Minimum is 1 thread.", + EXPERIMENTAL | DELAYED_EFFECT, "1", "threads" }, { "thread_pool_timeout", tweak_thread_pool_timeout, "Thread dies after this many seconds of inactivity.\n" - "Minimum is 1 second. " - EXPERIMENTAL - DELAYED_EFFECT, + "Minimum is 1 second.", + EXPERIMENTAL | DELAYED_EFFECT, "120", "seconds" }, { "overflow_max", tweak_overflow_max, "Limit on overflow queue length in percent of " - "thread_pool_max parameter.\n" + "thread_pool_max parameter.", EXPERIMENTAL, "100", "%" }, { "http_workspace", tweak_http_workspace, "Bytes of HTTP protocol workspace allocated. " "This space must be big enough for the entire HTTP protocol " "header and any edits done to it in the VCL code.\n" - "Minimum is 1024 bytes. " + "Minimum is 1024 bytes.", DELAYED_EFFECT, "8192", "bytes" }, { "sess_timeout", tweak_sess_timeout, "Idle timeout for persistent sessions. " "If a HTTP request has not been received in this many " - "seconds, the session is closed.\n", + "seconds, the session is closed.", + 0, "5", "seconds" }, { "pipe_timeout", tweak_pipe_timeout, "Idle timeout for PIPE sessions. " "If nothing have been received in either direction for " "this many seconds, the session is closed.\n", + 0, "60", "seconds" }, { "send_timeout", tweak_send_timeout, "Send timeout for client connections. " "If no data has been sent to the client in this many seconds, " "the session is closed.\n" - "See setsockopt(2) under SO_SNDTIMEO for more information.\n" + "See setsockopt(2) under SO_SNDTIMEO for more information.", DELAYED_EFFECT, "600", "seconds" }, { "auto_restart", tweak_auto_restart, "Restart child process automatically if it dies.\n", + 0, "on", "bool" }, { "fetch_chunksize", tweak_fetch_chunksize, "The default chunksize used by fetcher. " "This should be bigger than the majority of objects with " "short TTLs.\n" "Internal limits in the storage_file module makes increases " - "above 128kb a dubious idea.\n" + "above 128kb a dubious idea.", EXPERIMENTAL, "128", "kilobytes" }, #ifdef HAVE_SENDFILE { "sendfile_threshold", tweak_sendfile_threshold, - "The minimum size of objects transmitted with sendfile.\n" -#if defined(__FreeBSD__) - "In \"plenty-of-RAM\" scenarios this is unlikely to " - "have any effect. Once disk-I/O becomes frequent " - "we guess smaller values are likely to be better.\n" -#elif defined(__Linux__) - "Linux sendfile(2) does not allow for inclusion of " - "header data and therefore using sendfile(2) means " - "an extra system call, compared to using writev(2) for " - "both the header and body.\n" - "We suspect that sendfile(2) on Linux will only start " - "to be beneficial in low-ram scenarios. Therefore it " - "may make sense to set this to \"unlimited\".\n" -#endif + "The minimum size of objects transmitted with sendfile.", EXPERIMENTAL, "-1", "bytes" }, #endif /* HAVE_SENDFILE */ @@ -689,53 +683,50 @@ "Enabling this will allow you to see the path each " "request has taken through the VCL program.\n" "This generates a lot of logrecords so it is off by " - "default. ", + "default.", + 0, "off", "bool" }, { "listen_address", tweak_listen_address, "Whitespace separated list of network endpoints where " "Varnish will accept requests.\n" - "Possible formats: host, host:port, :port\n" + "Possible formats: host, host:port, :port", MUST_RESTART, ":80" }, { "listen_depth", tweak_listen_depth, - "Listen(2) queue depth.\n" -#if defined(__FreeBSD__) - "Please see FreeBSDs tuning(7) manual page for more " - "information.\n" -#endif + "Listen queue depth.", MUST_RESTART, "1024", "connections" }, { "srcaddr_hash", tweak_srcaddr_hash, "Number of source address hash buckets.\n" - "Powers of two are bad, prime numbers are good.\n" - EXPERIMENTAL - MUST_RESTART, + "Powers of two are bad, prime numbers are good.", + EXPERIMENTAL | MUST_RESTART, "1049", "buckets" }, { "srcaddr_ttl", tweak_srcaddr_ttl, "Lifetime of srcaddr entries.\n" - "Zero will disable srcaddr accounting entirely.\n" + "Zero will disable srcaddr accounting entirely.", EXPERIMENTAL, "30", "seconds" }, { "backend_http11", tweak_backend_http11, "Force all backend requests to be HTTP/1.1.\n" "By default we copy the protocol version from the " - "incoming client request." + "incoming client request.", EXPERIMENTAL, "off", "bool" }, { "client_http11", tweak_client_http11, "Force all client responses to be HTTP/1.1.\n" "By default we copy the protocol version from the " - "backend response." + "backend response.", EXPERIMENTAL, "off", "bool" }, { "cli_timeout", tweak_cli_timeout, "Timeout for the childs replies to CLI requests from " - "the master.\n", + "the master.", + 0, "5", "seconds" }, { "ping_interval", tweak_ping_interval, "Interval between pings from parent to child.\n" "Zero will disable pinging entirely, which makes " - "it possible to attach a debugger to the child.\n" + "it possible to attach a debugger to the child.", MUST_RESTART, "3", "seconds" }, { "lru_interval", tweak_lru_timeout, @@ -743,14 +734,14 @@ "Objects are only moved to the front of the LRU " "list if they have not been moved there already inside " "this timeout period. This reduces the amount of lock " - "operations necessary for LRU list access.\n" + "operations necessary for LRU list access.", EXPERIMENTAL, "2", "seconds" }, { "cc_command", tweak_cc_command, "Command used for compiling the C source code to a " "dlopen(3) loadable object. Any occurrence of %s in " "the string will be replaced with the source file name, " - "and %o will be replaced with the output file name.\n" + "and %o will be replaced with the output file name.", MUST_RELOAD, #ifdef __APPLE__ "exec cc -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" @@ -765,17 +756,43 @@ #endif "\nBe aware that restarts are likely to cause a hit against " "the backend, so don't increase thoughtlessly.\n", + 0, "4", "restarts" }, { NULL, NULL, NULL } }; /*--------------------------------------------------------------------*/ +#define WIDTH 76 +#define MARGIN 20 + +static void +mcf_wrap(struct cli *cli, const char *text) +{ + const char *p, *q; + + /* Format text to COLUMNS width */ + for (p = text; *p != '\0'; ) { + q = strchr(p, '\n'); + if (q == NULL) + q = strchr(p, '\0'); + if (q > p + WIDTH - MARGIN) { + q = p + WIDTH - MARGIN; + while (q > p && *q != ' ') + q--; + AN(q); + } + cli_out(cli, "%*s %.*s\n", MARGIN, "", (int)(q - p), p); + p = q; + if (*p == ' ' || *p == '\n') + p++; + } +} + void mcf_param_show(struct cli *cli, const char * const *av, void *priv) { struct parspec *pp; - const char *p, *q; int lfmt; (void)priv; @@ -786,7 +803,7 @@ for (pp = parspec; pp->name != NULL; pp++) { if (av[2] != NULL && !lfmt && strcmp(pp->name, av[2])) continue; - cli_out(cli, "%-20s ", pp->name); + cli_out(cli, "%-*s ", MARGIN, pp->name); if (pp->func == NULL) { cli_out(cli, "Not implemented.\n"); if (av[2] != NULL && !lfmt) @@ -800,23 +817,17 @@ else cli_out(cli, "\n"); if (av[2] != NULL) { - cli_out(cli, "%-20s Default is %s\n", "", pp->def); - /* Format text to 72 col width */ - for (p = pp->descr; *p != '\0'; ) { - q = strchr(p, '\n'); - if (q == NULL) - q = strchr(p, '\0'); - if (q > p + 52) { - q = p + 52; - while (q > p && *q != ' ') - q--; - AN(q); - } - cli_out(cli, "%20s %.*s\n", "", q - p, p); - p = q; - if (*p == ' ' || *p == '\n') - p++; - } + cli_out(cli, "%-*s Default is %s\n", + MARGIN, "", pp->def); + mcf_wrap(cli, pp->descr); + if (pp->flags & DELAYED_EFFECT) + mcf_wrap(cli, DELAYED_EFFECT_TEXT); + if (pp->flags & EXPERIMENTAL) + mcf_wrap(cli, EXPERIMENTAL_TEXT); + if (pp->flags & MUST_RELOAD) + mcf_wrap(cli, MUST_RELOAD_TEXT); + if (pp->flags & MUST_RESTART) + mcf_wrap(cli, MUST_RESTART_TEXT); if (!lfmt) return; else @@ -825,7 +836,7 @@ } if (av[2] != NULL && !lfmt) { cli_result(cli, CLIS_PARAM); - cli_out(cli, "Unknown paramter \"%s\".", av[2]); + cli_out(cli, "Unknown parameter \"%s\".", av[2]); } } @@ -848,12 +859,18 @@ for (pp = parspec; pp->name != NULL; pp++) { if (!strcmp(pp->name, param)) { pp->func(cli, pp, val); + if (pp->flags & MUST_RESTART) + cli_out(cli, "change will take effect" + " when child is restarted"); + if (pp->flags & MUST_RELOAD) + cli_out(cli, "change will take effect" + " when VCL script is reloaded"); MCF_ParamSync(); return; } } cli_result(cli, CLIS_PARAM); - cli_out(cli, "Unknown paramter \"%s\".", param); + cli_out(cli, "Unknown parameter \"%s\".", param); } From des at projects.linpro.no Thu Nov 15 16:06:08 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:06:08 +0100 (CET) Subject: r2260 - in branches/1.2: . bin/varnishd include Message-ID: <20071115160608.A8D041EC528@projects.linpro.no> Author: des Date: 2007-11-15 17:06:08 +0100 (Thu, 15 Nov 2007) New Revision: 2260 Added: branches/1.2/bin/varnishd/cache_panic.c Modified: branches/1.2/ branches/1.2/bin/varnishd/Makefile.am branches/1.2/bin/varnishd/cache.h branches/1.2/include/miniobj.h Log: Merged revisions 2253-2254 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2253 | des | 2007-11-15 16:07:15 +0100 (Thu, 15 Nov 2007) | 2 lines Add a VALID_OBJ() predicate macro. ........ r2254 | des | 2007-11-15 17:01:21 +0100 (Thu, 15 Nov 2007) | 7 lines Bigger hammer: panic() is similar to lbv_assert() but takes a struct sess * from which it tries to extract as much useful information as possible before it croaks. The corresponding macro is spassert(), which assumes you have a struct sess *sp in scope. This should help a lot with bugs like #167 where important information is not available post-mortem (because sp->obj points into the cache, which is not included in the core dump) ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2251 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2251,2253-2254 Modified: branches/1.2/bin/varnishd/Makefile.am =================================================================== --- branches/1.2/bin/varnishd/Makefile.am 2007-11-15 16:05:42 UTC (rev 2259) +++ branches/1.2/bin/varnishd/Makefile.am 2007-11-15 16:06:08 UTC (rev 2260) @@ -24,6 +24,7 @@ cache_http.c \ cache_httpconn.c \ cache_main.c \ + cache_panic.c \ cache_pool.c \ cache_pipe.c \ cache_response.c \ Modified: branches/1.2/bin/varnishd/cache.h =================================================================== --- branches/1.2/bin/varnishd/cache.h 2007-11-15 16:05:42 UTC (rev 2259) +++ branches/1.2/bin/varnishd/cache.h 2007-11-15 16:06:08 UTC (rev 2260) @@ -659,3 +659,17 @@ return ((unsigned)(t.e - t.b)); } + +#ifdef WITHOUT_ASSERTS +#define spassert(cond) ((void)0) +#else +void panic(const char *, int, const char *, + const struct sess *, const char *, ...); +#define spassert(cond) \ + do { \ + int ok = !!(cond); \ + if (!ok) \ + panic(__FILE__, __LINE__, __func__, sp, \ + "assertion failed: %s\n", #cond); \ + } while (0) +#endif Copied: branches/1.2/bin/varnishd/cache_panic.c (from rev 2254, trunk/varnish-cache/bin/varnishd/cache_panic.c) =================================================================== --- branches/1.2/bin/varnishd/cache_panic.c (rev 0) +++ branches/1.2/bin/varnishd/cache_panic.c 2007-11-15 16:06:08 UTC (rev 2260) @@ -0,0 +1,188 @@ +/*- + * Copyright (c) 2006 Verdens Gang AS + * Copyright (c) 2006-2007 Linpro AS + * All rights reserved. + * + * Author: Dag-Erling Sm?rgrav + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $Id$ + */ + +#include +#include +#include + +#include "cache.h" + +#ifndef WITHOUT_ASSERTS + +/* panic string */ +char panicstr[65536]; +static char *pstr = panicstr; + +#define fp(...) \ + pstr += snprintf(pstr, \ + (panicstr + sizeof panicstr) - pstr, \ + __VA_ARGS__) +#define vfp(fmt, ap) \ + pstr += vsnprintf(pstr, \ + (panicstr + sizeof panicstr) - pstr, \ + (fmt), (ap)) + +/* step names */ +static const char *steps[] = { +#define STEP(l, u) "STP_" #u, +#include "steps.h" +#undef STEP +}; +static int nsteps = sizeof steps / sizeof *steps; + +/* dump a struct storage */ +static void +dump_storage(const struct storage *st) +{ + int i, j; + +#define MAX_BYTES (4*16) +#define show(ch) (((ch) > 31 && (ch) < 127) ? (ch) : '.') + + for (i = 0; i < MAX_BYTES && i < st->len; i += 16) { + fp(" "); + for (j = 0; j < 16; ++j) { + if (i + j < st->len) + fp("%02x ", st->ptr[i + j]); + else + fp(" "); + } + fp("|"); + for (j = 0; j < 16; ++j) + if (i + j < st->len) + fp("%c", show(st->ptr[i + j])); + fp("|\n"); + } + if (st->len > MAX_BYTES) + fp(" [%u more]\n", st->len - MAX_BYTES); + +#undef show +#undef MAX_BYTES +} + +/* dump a struct http */ +static void +dump_http(const struct http *h) +{ + int i; + + fp(" http = {\n"); + if (h->nhd > HTTP_HDR_FIRST) { + fp(" hd = {\n"); + for (i = HTTP_HDR_FIRST; i < h->nhd; ++i) + fp(" \"%.*s\",\n", + (int)(h->hd[i].e - h->hd[i].b), + h->hd[i].b); + fp(" },\n"); + } + fp(" },\n"); +} + +/* dump a struct object */ +static void +dump_object(const struct object *o) +{ + const struct storage *st; + + fp(" obj = %p {\n", o); + fp(" refcnt = %u, xid = %u,\n", o->refcnt, o->xid); + dump_http(o->http); + fp(" store = {\n"); + VTAILQ_FOREACH(st, &o->store, list) { + dump_storage(st); + } + fp(" },\n"); + fp(" },\n"); +} + +/* dump a struct backend */ +static void +dump_backend(const struct backend *be) +{ + + fp(" backend = %p {\n", be); + fp(" vcl_name = \"%s\",\n", + be->vcl_name ? be->vcl_name : "(null)"); + fp(" },\n"); +} + +/* dump a struct sess */ +static void +dump_sess(const struct sess *sp) +{ + const struct backend *be = sp->backend; + const struct object *obj = sp->obj; + + fp("sp = %p {\n", sp); + fp(" fd = %d, id = %d, xid = %u,\n", sp->fd, sp->id, sp->xid); + fp(" client = %s:%s,\n", + sp->addr ? sp->addr : "?.?.?.?", + sp->port ? sp->port : "?"); + if (sp->step < nsteps) + fp(" step = %s,\n", steps[sp->step]); + else + fp(" step = %d,\n", sp->step); + if (sp->err_code) + fp(" err_code = %d, err_reason = %s,\n", sp->err_code, + sp->err_reason ? sp->err_reason : "(null)"); + + if (VALID_OBJ(be, BACKEND_MAGIC)) + dump_backend(be); + + if (VALID_OBJ(obj, OBJECT_MAGIC)) + dump_object(obj); + + fp("},\n"); +} + +/* report as much information as we can before we croak */ +void +panic(const char *file, int line, const char *func, + const struct sess *sp, const char *fmt, ...) +{ + va_list ap; + + fp("panic in %s() at %s:%d\n", func, file, line); + va_start(ap, fmt); + vfp(fmt, ap); + va_end(ap); + + if (VALID_OBJ(sp, SESS_MAGIC)) + dump_sess(sp); + + fputs(panicstr, stderr); + + /* I wish there was a way to flush the log buffers... */ + signal(SIGABRT, SIG_DFL); + raise(SIGABRT); +} + +#endif Modified: branches/1.2/include/miniobj.h =================================================================== --- branches/1.2/include/miniobj.h 2007-11-15 16:05:42 UTC (rev 2259) +++ branches/1.2/include/miniobj.h 2007-11-15 16:06:08 UTC (rev 2260) @@ -12,6 +12,9 @@ free(to); \ } while (0) +#define VALID_OBJ(ptr, type_magic) \ + ((ptr) != NULL && (ptr)->magic == (type_magic)) + #define CHECK_OBJ(ptr, type_magic) \ do { \ assert((ptr)->magic == type_magic); \ From des at projects.linpro.no Thu Nov 15 16:07:47 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:07:47 +0100 (CET) Subject: r2261 - in branches/1.2: . bin/varnishd Message-ID: <20071115160747.D46CA1EC290@projects.linpro.no> Author: des Date: 2007-11-15 17:07:47 +0100 (Thu, 15 Nov 2007) New Revision: 2261 Modified: branches/1.2/ branches/1.2/bin/varnishd/cache_response.c Log: Merged revisions 2252,2255 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2252 | des | 2007-11-15 15:18:39 +0100 (Thu, 15 Nov 2007) | 2 lines Whitespace cleanup ........ r2255 | des | 2007-11-15 17:02:05 +0100 (Thu, 15 Nov 2007) | 2 lines Change the assert() that triggers in #167 to spassert(). ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2251,2253-2254 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2255 Modified: branches/1.2/bin/varnishd/cache_response.c =================================================================== --- branches/1.2/bin/varnishd/cache_response.c 2007-11-15 16:06:08 UTC (rev 2260) +++ branches/1.2/bin/varnishd/cache_response.c 2007-11-15 16:07:47 UTC (rev 2261) @@ -126,10 +126,10 @@ http_CopyResp(sp->http, sp->obj->http); http_FilterFields(sp->wrk, sp->fd, sp->http, sp->obj->http, HTTPH_A_DELIVER); - + TIM_format(TIM_real(), time_str); http_PrintfHeader(sp->wrk, sp->fd, sp->http, "Date: %s", time_str); - + if (sp->xid != sp->obj->xid) http_PrintfHeader(sp->wrk, sp->fd, sp->http, "X-Varnish: %u %u", sp->xid, sp->obj->xid); @@ -181,7 +181,7 @@ VSL_stats->n_objwrite++; WRK_Write(sp->wrk, st->ptr, st->len); } - assert(u == sp->obj->len); + spassert(u == sp->obj->len); } if (WRK_Flush(sp->wrk)) vca_close_session(sp, "remote closed"); From des at projects.linpro.no Thu Nov 15 16:08:43 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Thu, 15 Nov 2007 17:08:43 +0100 (CET) Subject: r2262 - in branches/1.2: . lib/libvcl Message-ID: <20071115160843.70B961EC407@projects.linpro.no> Author: des Date: 2007-11-15 17:08:43 +0100 (Thu, 15 Nov 2007) New Revision: 2262 Modified: branches/1.2/ branches/1.2/lib/libvcl/vcc_compile.c Log: Merged revisions 2247 via svnmerge from svn+ssh://projects.linpro.no/svn/varnish/trunk/varnish-cache ........ r2247 | des | 2007-11-09 13:43:09 +0100 (Fri, 09 Nov 2007) | 2 lines Whitespace cleanup. ........ Property changes on: branches/1.2 ___________________________________________________________________ Name: svnmerge-integrated - /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246,2248-2255 + /trunk/varnish-cache:1-2101,2104-2107,2115-2120,2122-2130,2133,2151,2153-2154,2157,2161-2162,2166-2168,2173,2175-2176,2180-2184,2186-2193,2206,2208,2210-2215,2220,2222-2232,2243,2246-2255 Modified: branches/1.2/lib/libvcl/vcc_compile.c =================================================================== --- branches/1.2/lib/libvcl/vcc_compile.c 2007-11-15 16:07:47 UTC (rev 2261) +++ branches/1.2/lib/libvcl/vcc_compile.c 2007-11-15 16:08:43 UTC (rev 2262) @@ -216,7 +216,7 @@ break; case '\n': vsb_printf(sb, "\\n"); - if (mode) + if (mode) vsb_printf(sb, "\"\n\t\""); break; case '\t': vsb_printf(sb, "\\t"); break; @@ -327,7 +327,7 @@ Fc(tl, 0, ",\n"); } Fc(tl, 0, "};\n"); - + Fc(tl, 0, "\nconst char *srcbody[%u] = {\n", tl->nsources); VTAILQ_FOREACH(sp, &tl->sources, list) { Fc(tl, 0, " /* "); @@ -355,7 +355,7 @@ Fc(tl, 0, "\t.nhashcount = %u,\n", tl->nhashcount); #define VCL_RET_MAC(l,u,b,n) #define VCL_MET_MAC(l,u,b) \ - Fc(tl, 0, "\t." #l "_func = VGC_function_vcl_" #l ",\n"); + Fc(tl, 0, "\t." #l "_func = VGC_function_vcl_" #l ",\n"); #include "vcl_returns.h" #undef VCL_MET_MAC #undef VCL_RET_MAC @@ -385,7 +385,7 @@ if (sp->freeit != NULL) free(sp->freeit); - free(sp->name); + free(sp->name); free(sp); } @@ -530,7 +530,7 @@ VTAILQ_REMOVE(&tl->sources, sp, list); vcc_destroy_source(sp); } - + vsb_delete(tl->fh); vsb_delete(tl->fc); vsb_delete(tl->fi); From phk at projects.linpro.no Tue Nov 20 11:07:47 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 12:07:47 +0100 (CET) Subject: r2263 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120110747.A382B1EC290@projects.linpro.no> Author: phk Date: 2007-11-20 12:07:47 +0100 (Tue, 20 Nov 2007) New Revision: 2263 Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c Log: Add some asserts to the plumbing around the C-compiler invocation. Collect any messages from the C-compiler and pass them on to the user, but do not take them as evidence of a failed compilation, check the returnvalue for that. To avoid confusion, explicitly say when the VCL program was compiled. Modified: trunk/varnish-cache/bin/varnishd/mgt_vcc.c =================================================================== --- trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2007-11-15 16:08:43 UTC (rev 2262) +++ trunk/varnish-cache/bin/varnishd/mgt_vcc.c 2007-11-20 11:07:47 UTC (rev 2263) @@ -232,25 +232,28 @@ if ((pid = fork()) < 0) { vsb_printf(sb, "%s(): fork() failed: %s", __func__, strerror(errno)); - close(p[0]); - close(p[1]); - unlink(sf); + AZ(close(p[0])); + AZ(close(p[1])); + (void)unlink(sf); free(of); return (NULL); } if (pid == 0) { - close(p[0]); - close(STDIN_FILENO); + AZ(close(p[0])); + AZ(close(STDIN_FILENO)); assert(open("/dev/null", O_RDONLY) == STDIN_FILENO); assert(dup2(p[1], STDOUT_FILENO) == STDOUT_FILENO); assert(dup2(p[1], STDERR_FILENO) == STDERR_FILENO); execl("/bin/sh", "/bin/sh", "-c", cmdline, NULL); _exit(1); } - close(p[1]); - while (read(p[0], buf, sizeof buf) > 0) - /* XXX nothing */ ; - close(p[0]); + AZ(close(p[1])); + do { + status = read(p[0], buf, sizeof buf); + if (status > 0) + vsb_printf(sb, "C-Compiler said: %.*s", status, buf); + } while (status > 0); + AZ(close(p[0])); unlink(sf); if (waitpid(pid, &status, 0) < 0) { vsb_printf(sb, "%s(): waitpid() failed: %s", @@ -410,14 +413,15 @@ vf = mgt_VccCompileFile(sb, f_arg, C_flag, f_fd); } vsb_finish(sb); - if (vsb_len(sb) > 0) { + if (vsb_len(sb) > 0) fprintf(stderr, "%s", vsb_data(sb)); - vsb_delete(sb); - return (1); - } vsb_delete(sb); if (C_flag) return (0); + if (vf == NULL) { + fprintf(stderr, "VCL compilation failed"); + return (1); + } vp = mgt_vcc_add("boot", vf); vp->active = 1; return (0); @@ -490,13 +494,15 @@ XXXAN(sb); vf = mgt_VccCompile(sb, av[3], NULL, 0); vsb_finish(sb); - if (vsb_len(sb) > 0) { + if (vsb_len(sb) > 0) cli_out(cli, "%s", vsb_data(sb)); - vsb_delete(sb); + vsb_delete(sb); + if (vf == NULL) { + cli_out(cli, "VCL compilation failed"); cli_result(cli, CLIS_PARAM); return; } - vsb_delete(sb); + cli_out(cli, "VCL compiled."); if (child_pid >= 0 && mgt_cli_askchild(&status, &p, "vcl.load %s %s\n", av[2], vf)) { cli_result(cli, status); @@ -521,13 +527,15 @@ XXXAN(sb); vf = mgt_VccCompileFile(sb, av[3], 0, -1); vsb_finish(sb); - if (vsb_len(sb) > 0) { + if (vsb_len(sb) > 0) cli_out(cli, "%s", vsb_data(sb)); - vsb_delete(sb); + vsb_delete(sb); + if (vf == NULL) { + cli_out(cli, "VCL compilation failed"); cli_result(cli, CLIS_PARAM); return; } - vsb_delete(sb); + cli_out(cli, "VCL compiled."); if (child_pid >= 0 && mgt_cli_askchild(&status, &p, "vcl.load %s %s\n", av[2], vf)) { cli_result(cli, status); From phk at projects.linpro.no Tue Nov 20 11:11:17 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 12:11:17 +0100 (CET) Subject: r2264 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120111117.A728A1EC396@projects.linpro.no> Author: phk Date: 2007-11-20 12:11:17 +0100 (Tue, 20 Nov 2007) New Revision: 2264 Modified: trunk/varnish-cache/bin/varnishd/cache_backend_simple.c Log: When closing a backend connection, we can run into a ECONNRESET which given circumstances is fair game. Modified: trunk/varnish-cache/bin/varnishd/cache_backend_simple.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_backend_simple.c 2007-11-20 11:07:47 UTC (rev 2263) +++ trunk/varnish-cache/bin/varnishd/cache_backend_simple.c 2007-11-20 11:11:17 UTC (rev 2264) @@ -274,12 +274,14 @@ static void bes_ClosedFd(struct worker *w, struct vbe_conn *vc) { + int i; CHECK_OBJ_NOTNULL(vc, VBE_CONN_MAGIC); CHECK_OBJ_NOTNULL(vc->backend, BACKEND_MAGIC); assert(vc->fd >= 0); WSL(w, SLT_BackendClose, vc->fd, "%s", vc->backend->vcl_name); - AZ(close(vc->fd)); + i = close(vc->fd); + assert(i == 0 || errno == ECONNRESET); vc->fd = -1; VBE_DropRef(vc->backend); vc->backend = NULL; From phk at projects.linpro.no Tue Nov 20 11:23:37 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 12:23:37 +0100 (CET) Subject: r2265 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120112337.A89291EC2B4@projects.linpro.no> Author: phk Date: 2007-11-20 12:23:37 +0100 (Tue, 20 Nov 2007) New Revision: 2265 Modified: trunk/varnish-cache/bin/varnishd/cache_center.c Log: We should never have an object when we hit the error state. Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2007-11-20 11:11:17 UTC (rev 2264) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2007-11-20 11:23:37 UTC (rev 2265) @@ -275,6 +275,7 @@ cnt_error(struct sess *sp) { + AZ(sp->obj); RES_Error(sp, sp->err_code, sp->err_reason); sp->err_code = 0; sp->err_reason = NULL; From phk at projects.linpro.no Tue Nov 20 12:01:40 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 13:01:40 +0100 (CET) Subject: r2266 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120120140.5EB5B1EC290@projects.linpro.no> Author: phk Date: 2007-11-20 13:01:40 +0100 (Tue, 20 Nov 2007) New Revision: 2266 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/bin/varnishd/cache_center.c trunk/varnish-cache/bin/varnishd/cache_response.c trunk/varnish-cache/bin/varnishd/cache_synthetic.c Log: Simplify the generation of error responses. Don't take a detour around cache_response.c, go directly to cache_synthetic.c. Don't wast time and energy mucking about with pseudo-objects, just synthesize the error body directly in the workspace and send it from there. Close the session after sending error response, freeing up resources sounds like a good strategy. (XXX: for futher study). This might be related to #167 Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2007-11-20 11:23:37 UTC (rev 2265) +++ trunk/varnish-cache/bin/varnishd/cache.h 2007-11-20 12:01:40 UTC (rev 2266) @@ -550,11 +550,10 @@ /* cache_response.c */ void RES_BuildHttp(struct sess *sp); -void RES_Error(struct sess *sp, int code, const char *reason); void RES_WriteObj(struct sess *sp); /* cache_synthetic.c */ -void SYN_ErrorPage(struct sess *sp, int status, const char *reason, int ttl); +void SYN_ErrorPage(struct sess *sp, int status, const char *reason); /* cache_vary.c */ void VRY_Create(const struct sess *sp); Modified: trunk/varnish-cache/bin/varnishd/cache_center.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_center.c 2007-11-20 11:23:37 UTC (rev 2265) +++ trunk/varnish-cache/bin/varnishd/cache_center.c 2007-11-20 12:01:40 UTC (rev 2266) @@ -276,7 +276,7 @@ { AZ(sp->obj); - RES_Error(sp, sp->err_code, sp->err_reason); + SYN_ErrorPage(sp, sp->err_code, sp->err_reason); sp->err_code = 0; sp->err_reason = NULL; sp->step = STP_DONE; @@ -829,7 +829,7 @@ *sp->http0 = *sp->http; if (done != 0) { - RES_Error(sp, done, NULL); /* XXX: STP_ERROR ? */ + SYN_ErrorPage(sp, done, NULL); /* XXX: STP_ERROR ? */ sp->step = STP_DONE; return (0); } Modified: trunk/varnish-cache/bin/varnishd/cache_response.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-20 11:23:37 UTC (rev 2265) +++ trunk/varnish-cache/bin/varnishd/cache_response.c 2007-11-20 12:01:40 UTC (rev 2266) @@ -41,29 +41,6 @@ /*--------------------------------------------------------------------*/ -void -RES_Error(struct sess *sp, int code, const char *reason) -{ - - /* get a pristine object */ - HSH_Prealloc(sp); - sp->obj = sp->wrk->nobj; - sp->wrk->nobj = NULL; - sp->obj->busy = 1; - - /* synthesize error page and send it */ - SYN_ErrorPage(sp, code, reason, 0); - RES_BuildHttp(sp); - RES_WriteObj(sp); - - /* GC the error page */ - HSH_Unbusy(sp->obj); - HSH_Deref(sp->obj); - sp->obj = NULL; -} - -/*--------------------------------------------------------------------*/ - static void res_do_304(struct sess *sp) { Modified: trunk/varnish-cache/bin/varnishd/cache_synthetic.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_synthetic.c 2007-11-20 11:23:37 UTC (rev 2265) +++ trunk/varnish-cache/bin/varnishd/cache_synthetic.c 2007-11-20 12:01:40 UTC (rev 2266) @@ -39,67 +39,64 @@ #include "stevedore.h" /* - * Synthesize an error page. This assumes the session already has an - * object - if it doesn't, you need to either call HSH_Lookup(), or call - * HSH_Prealloc() and grab sp->obj->nobj, before calling this. + * Synthesize an error page including headers. + * XXX: For now close the connection. Long term that should probably + * XXX: be either a paramter or VCL decision. + * XXX: VCL should get a shot at generating the page. */ + void -SYN_ErrorPage(struct sess *sp, int status, const char *reason, int ttl) +SYN_ErrorPage(struct sess *sp, int status, const char *reason) { - struct storage *st; - struct object *o; - struct worker *w; struct http *h; - struct vsb vsb; + struct worker *w; const char *msg; char date[40]; double now; + unsigned u; + struct vsb vsb; int fd; + int ttl = 0; /* XXX: ?? */ WSL_Flush(sp->wrk); assert(status >= 100 && status <= 999); CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC); - CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC); - CHECK_OBJ_NOTNULL(sp->obj->http, HTTP_MAGIC); - assert(sp->obj->busy > 0); /* shortcuts */ w = sp->wrk; + h = sp->http; fd = sp->fd; - o = sp->obj; - h = o->http; - now = TIM_real(); + now = TIM_real(); /* XXX: use cached val ? */ - /* Set up obj's workspace */ - st = o->objstore; - WS_Init(o->ws_o, "obj", st->ptr + st->len, st->space - st->len); - st->len = st->space; - WS_Assert(o->ws_o); - http_Setup(o->http, o->ws_o); + WRK_Reset(w, &sp->fd); /* look up HTTP response */ msg = http_StatusMessage(status); + AN(msg); if (reason == NULL) reason = msg; - AN(reason); - AN(msg); - /* populate metadata */ - o->response = status; - o->valid = 1; - o->entered = now; - o->ttl = now + ttl; - o->last_modified = now; - o->xid = sp->xid; + /* generate header */ + http_ClrHeader(h); + h->logtag = HTTP_Tx; + http_PutProtocol(w, fd, h, "HTTP/1.0"); /* XXX */ + http_PutStatus(w, fd, h, status); + http_PutResponse(w, fd, h, msg); + TIM_format(now, date); + http_PrintfHeader(w, fd, h, "Date: %s", date); + http_PrintfHeader(w, fd, h, "Server: Varnish"); + http_PrintfHeader(w, fd, h, "Retry-After: %d", ttl); + http_PrintfHeader(w, fd, h, "Content-Type: text/html; charset=utf-8"); + http_PrintfHeader(w, sp->fd, sp->http, "X-Varnish: %u", sp->xid); + http_PrintfHeader(w, fd, h, "Connection: close"); - /* allocate space for body */ - /* XXX what if the object already has a body? */ - st = STV_alloc(sp, 1024); - VTAILQ_INSERT_TAIL(&sp->obj->store, st, list); + w->acct.hdrbytes += http_Write(w, h, 1); /* generate body */ - AN(vsb_new(&vsb, (char *)st->ptr, st->space, VSB_FIXEDLEN)); + /* XXX: VCL should do this */ + u = WS_Reserve(h->ws, 0); + AN(vsb_new(&vsb, h->ws->f, u, VSB_FIXEDLEN)); vsb_printf(&vsb, "\n" "\n" "\n"); vsb_finish(&vsb); - o->len = st->len = vsb_len(&vsb); + w->acct.hdrbytes = WRK_Write(w, vsb_data(&vsb), vsb_len(&vsb)); + (void)WRK_Flush(w); vsb_delete(&vsb); - - /* allocate space for header */ - - WS_Init(h->ws, "error", malloc(1024), 1024); - - /* generate header */ - http_ClrHeader(h); - http_PutProtocol(w, fd, h, "HTTP/1.0"); /* XXX */ - http_PutStatus(w, fd, h, status); - http_PutResponse(w, fd, h, msg); - TIM_format(now, date); - http_PrintfHeader(w, fd, h, "Date: %s", date); - http_PrintfHeader(w, fd, h, "Server: Varnish"); - http_PrintfHeader(w, fd, h, "Retry-After: %ju", (uintmax_t)ttl); - http_PrintfHeader(w, fd, h, "Content-Type: text/html; charset=utf-8"); - http_PrintfHeader(w, fd, h, "Content-Length: %u", o->len); - /* DO NOT generate X-Varnish header, RES_BuildHttp will */ + vca_close_session(sp, "error returned"); } From phk at projects.linpro.no Tue Nov 20 12:11:04 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 13:11:04 +0100 (CET) Subject: r2267 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120121104.661AC1EC396@projects.linpro.no> Author: phk Date: 2007-11-20 13:11:04 +0100 (Tue, 20 Nov 2007) New Revision: 2267 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Log: Use a bitmap to check if we own a given file handle, before we do anything about the associated session. May be relevant relative to: #162 (Bitmap functions slightly general, for possible later reuse) Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 12:01:40 UTC (rev 2266) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 12:11:04 UTC (rev 2267) @@ -36,6 +36,7 @@ #if defined(HAVE_KQUEUE) #include +#include #include #include #include @@ -48,9 +49,84 @@ #include "cache.h" #include "cache_acceptor.h" +/********************************************************************** + * Generic bitmap functions, may be generalized at some point. + */ + +#define VBITMAP_TYPE unsigned /* Our preferred wordsize */ +#define VBITMAP_LUMP (32*1024) /* How many bits we alloc at a time */ +#define VBITMAP_WORD (sizeof(VBITMAP_TYPE) * 8) +#define VBITMAP_IDX(n) (n / VBITMAP_WORD) +#define VBITMAP_BIT(n) (1U << (n % VBITMAP_WORD)) + +struct vbitmap { + VBITMAP_TYPE *bits; + unsigned nbits; +}; + +static void +vbit_expand(struct vbitmap *vb, unsigned bit) +{ + unsigned char *p; + + bit += VBITMAP_LUMP - 1; + bit -= (bit % VBITMAP_LUMP); + VSL(SLT_Debug, 0, "Expanding KQ VBIT to %u", bit); + p = realloc(vb->bits, bit / 8); + AN(p); + memset(p + vb->nbits / 8, 0, (bit - vb->nbits) / 8); + vb->bits = (void*)p; + vb->nbits = bit; +} + +static struct vbitmap * +vbit_init(unsigned initial) +{ + struct vbitmap *vb; + + vb = calloc(sizeof *vb, 1); + AN(vb); + if (initial == 0) + initial = VBITMAP_LUMP; + vbit_expand(vb, initial); + return (vb); +} + +static void +vbit_set(struct vbitmap *vb, unsigned bit) +{ + + if (bit >= vb->nbits) + vbit_expand(vb, bit); + vb->bits[VBITMAP_IDX(bit)] |= VBITMAP_BIT(bit); +} + +static void +vbit_clr(struct vbitmap *vb, unsigned bit) +{ + + if (bit >= vb->nbits) + vbit_expand(vb, bit); + vb->bits[VBITMAP_IDX(bit)] &= ~VBITMAP_BIT(bit); +} + +static int +vbit_test(struct vbitmap *vb, unsigned bit) +{ + + if (bit >= vb->nbits) + vbit_expand(vb, bit); + return (vb->bits[VBITMAP_IDX(bit)] & VBITMAP_BIT(bit)); +} + +/**********************************************************************/ + + static pthread_t vca_kqueue_thread; static int kq = -1; +struct vbitmap *vca_kqueue_bits; + static VTAILQ_HEAD(,sess) sesshead = VTAILQ_HEAD_INITIALIZER(sesshead); #define NKEV 100 @@ -66,7 +142,7 @@ if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); - if (++nki == NKEV || arm == EV_DELETE) { + if (++nki == NKEV) { AZ(kevent(kq, ki, nki, NULL, 0, NULL)); nki = 0; } @@ -89,6 +165,8 @@ CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC); assert(ss[j]->fd >= 0); AZ(ss[j]->obj); + AZ(vbit_test(vca_kqueue_bits, ss[j]->fd)); + vbit_set(vca_kqueue_bits, ss[j]->fd); VTAILQ_INSERT_TAIL(&sesshead, ss[j], list); vca_kq_sess(ss[j], EV_ADD); j++; @@ -97,6 +175,13 @@ assert(i == 0); return; } + if (!vbit_test(vca_kqueue_bits, kp->ident)) { + VSL(SLT_Debug, kp->ident, + "KQ: not my fd %d, sp %p kev data %lu flags 0x%x%s", + kp->ident, kp->udata, (unsigned long)kp->data, kp->flags, + (kp->flags & EV_EOF) ? " EOF" : ""); + return; + } CAST_OBJ_NOTNULL(sp, kp->udata, SESS_MAGIC); #ifdef DIAGNOSTICS VSL(SLT_Debug, sp->id, "sp %p kev data %lu flags 0x%x%s", @@ -107,11 +192,13 @@ i = HTC_Rx(sp->htc); if (i == 0) return; /* more needed */ + vbit_clr(vca_kqueue_bits, sp->fd); VTAILQ_REMOVE(&sesshead, sp, list); vca_kq_sess(sp, EV_DELETE); vca_handover(sp, i); return; } else if (kp->flags == EV_EOF) { + vbit_clr(vca_kqueue_bits, sp->fd); VTAILQ_REMOVE(&sesshead, sp, list); vca_close_session(sp, "EOF"); SES_Delete(sp); @@ -164,6 +251,7 @@ break; if (sp->t_open > deadline) break; + vbit_clr(vca_kqueue_bits, sp->fd); VTAILQ_REMOVE(&sesshead, sp, list); vca_close_session(sp, "timeout"); SES_Delete(sp); @@ -182,6 +270,7 @@ i |= O_NONBLOCK; i = fcntl(vca_pipes[0], F_SETFL, i); + vca_kqueue_bits = vbit_init(0); AZ(pthread_create(&vca_kqueue_thread, NULL, vca_kqueue_main, NULL)); } From des at linpro.no Tue Nov 20 12:17:53 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 13:17:53 +0100 Subject: r2266 - trunk/varnish-cache/bin/varnishd In-Reply-To: <20071120120140.5EB5B1EC290@projects.linpro.no> (phk@projects.linpro.no's message of "Tue, 20 Nov 2007 13:01:40 +0100 (CET)") References: <20071120120140.5EB5B1EC290@projects.linpro.no> Message-ID: phk at projects.linpro.no writes: > Log: > Simplify the generation of error responses. This commit is a regression. Please revert it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Nov 20 12:20:55 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 13:20:55 +0100 Subject: r2267 - trunk/varnish-cache/bin/varnishd In-Reply-To: <20071120121104.661AC1EC396@projects.linpro.no> (phk@projects.linpro.no's message of "Tue, 20 Nov 2007 13:11:04 +0100 (CET)") References: <20071120121104.661AC1EC396@projects.linpro.no> Message-ID: phk at projects.linpro.no writes: > Author: phk > Date: 2007-11-20 13:11:04 +0100 (Tue, 20 Nov 2007) > New Revision: 2267 > > Modified: > trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c > Log: > Use a bitmap to check if we own a given file handle, before we > do anything about the associated session. > > May be relevant relative to: #162 You know very well that I objected to this patch. There is absolutely zero evidence that it helps #162, and non-zero evidence that it has a significant negative impact on performance. Please revert it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Nov 20 12:30:40 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Nov 2007 12:30:40 +0000 Subject: r2267 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Tue, 20 Nov 2007 13:20:55 +0100." Message-ID: <15094.1195561840@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: >> Modified: >> trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c >> Log: >> Use a bitmap to check if we own a given file handle, before we >> do anything about the associated session. >> >> May be relevant relative to: #162 > >You know very well that I objected to this patch. There is absolutely >zero evidence that it helps #162, and non-zero evidence that it has a >significant negative impact on performance. Please revert it. There is zero evidence that this has any performance impact. The CPU usage problem is also in 1.2 without my patch. Checking for the suspected condition behind #162s problem can only be correctly implemented by only looking at the file handle, we are not, despite what you may think or desire, allowed to access the session structure after we have handed the session back. A bitmap is the simplest, lightest and safest way to do implement that check. (And the binheap vs TAILQ discussion doesn't even remotely come close to this, as both of them would require dereferencing of the session structure before the check could be carried out). I already wrote to you, when you commited the temporary workaround, that it did not look 100% water-tight to me, but that it was a good bandgap, even at a increased keevent syscall load. This commit have no implemented the check in the correct way and reduced the syscall loading to its former level. Case closed, can we focus on more important things now ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Nov 20 12:57:42 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Nov 2007 12:57:42 +0000 Subject: r2266 - trunk/varnish-cache/bin/varnishd In-Reply-To: Your message of "Tue, 20 Nov 2007 13:17:53 +0100." Message-ID: <15269.1195563462@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: >phk at projects.linpro.no writes: >> Log: >> Simplify the generation of error responses. > >This commit is a regression. Please revert it. No, it is not. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anders at fupp.net Tue Nov 20 14:16:39 2007 From: anders at fupp.net (Anders Nordby) Date: Tue, 20 Nov 2007 15:16:39 +0100 Subject: r2230 - trunk/varnish-cache/bin/varnishncsa In-Reply-To: <20071108162826.BC65D1EC46F@projects.linpro.no> References: <20071108162826.BC65D1EC46F@projects.linpro.no> Message-ID: <20071120141639.GA51806@fupp.net> Hi, This does not seem to work well. Upon sending SIGHUP to varnishncsa, it does not start writing to the new log. On Thu, Nov 08, 2007 at 05:28:26PM +0100, des at projects.linpro.no wrote: > Author: des > Date: 2007-11-08 17:28:26 +0100 (Thu, 08 Nov 2007) > New Revision: 2230 > > Modified: > trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 > trunk/varnish-cache/bin/varnishncsa/varnishncsa.c > Log: > Copy&paste -D (daemonize) from varnishlog. > > > Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 > =================================================================== > --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 2007-11-08 16:27:53 UTC (rev 2229) > +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.1 2007-11-08 16:28:26 UTC (rev 2230) > @@ -28,7 +28,7 @@ > .\" > .\" $Id$ > .\" > -.Dd June 15, 2007 > +.Dd November 8, 2007 > .Dt VARNISHNCSA 1 > .Os > .Sh NAME > @@ -40,10 +40,12 @@ > .Op Fl b > .Op Fl C > .Op Fl c > +.Op Fl D > .Op Fl d > .Op Fl I Ar regex > .Op Fl i Ar tag > .Op Fl n Ar varnish_name > +.Op Fl P Ar file > .Op Fl r Ar file > .Op Fl V > .Op Fl w Ar file > @@ -82,6 +84,8 @@ > is specified, > .Nm > acts as if they both were. > +.It Fl D > +Daemonize. > .It Fl d > Process old log entries on startup. > Normally, > @@ -109,6 +113,9 @@ > If > .Fl n > is not specified, the host name is used. > +.It Fl P Ar file > +Write the process's PID to the specified > +.Ar file . > .It Fl r Ar file > Read log entries from > .Ar file > > Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c > =================================================================== > --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-08 16:27:53 UTC (rev 2229) > +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-08 16:28:26 UTC (rev 2230) > @@ -69,10 +69,16 @@ > #include > #include > > +#ifndef HAVE_DAEMON > +#include "compat/daemon.h" > +#endif > + > +#include "vsb.h" > +#include "vpf.h" > + > #include "libvarnish.h" > #include "shmlog.h" > #include "varnishapi.h" > -#include "vsb.h" > > static struct logline { > char *df_H; /* %H, Protocol version */ > @@ -434,7 +440,7 @@ > usage(void) > { > > - fprintf(stderr, "usage: varnishncsa %s [-aV] [-n varnish_name] [-w file]\n", VSL_ARGS); > + fprintf(stderr, "usage: varnishncsa %s [-aDV] [-n varnish_name] [-P file] [-w file]\n", VSL_ARGS); > exit(1); > } > > @@ -442,15 +448,17 @@ > main(int argc, char *argv[]) > { > int i, c; > + int a_flag = 0, D_flag = 0; > + const char *n_arg = NULL; > + const char *P_arg = NULL; > + const char *w_arg = NULL; > + struct pidfh *pfh = NULL; > struct VSL_data *vd; > - const char *ofn = NULL; > - const char *n_arg = NULL; > - int append = 0; > FILE *of; > > vd = VSL_New(); > > - while ((c = getopt(argc, argv, VSL_ARGS "an:Vw:")) != -1) { > + while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { > i = VSL_Arg(vd, c, optarg); > if (i < 0) > exit (1); > @@ -458,16 +466,22 @@ > continue; > switch (c) { > case 'a': > - append = 1; > + a_flag = 1; > break; > + case 'D': > + D_flag = 1; > + break; > case 'n': > n_arg = optarg; > break; > + case 'P': > + P_arg = optarg; > + break; > case 'V': > varnish_version("varnishncsa"); > exit(0); > case 'w': > - ofn = optarg; > + w_arg = optarg; > break; > default: > if (VSL_Arg(vd, c, optarg) > 0) > @@ -479,22 +493,37 @@ > if (VSL_OpenLog(vd, n_arg)) > exit(1); > > - if (ofn) { > - of = open_log(ofn, append); > + if (P_arg && (pfh = vpf_open(P_arg, 0600, NULL)) == NULL) { > + perror(P_arg); > + exit(1); > + } > + > + if (D_flag && daemon(0, 0) == -1) { > + perror("daemon()"); > + if (pfh != NULL) > + vpf_remove(pfh); > + exit(1); > + } > + > + if (pfh != NULL) > + vpf_write(pfh); > + > + if (w_arg) { > + of = open_log(w_arg, a_flag); > signal(SIGHUP, sighup); > } else { > - ofn = "stdout"; > + w_arg = "stdout"; > of = stdout; > } > > while (VSL_Dispatch(vd, h_ncsa, of) == 0) { > if (fflush(of) != 0) { > - perror(ofn); > + perror(w_arg); > exit(1); > } > if (reopen && of != stdout) { > fclose(of); > - of = open_log(ofn, append); > + of = open_log(w_arg, a_flag); > reopen = 0; > } > } > > _______________________________________________ > varnish-commit mailing list > varnish-commit at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-commit -- Anders. From phk at projects.linpro.no Tue Nov 20 14:20:36 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 15:20:36 +0100 (CET) Subject: r2268 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120142036.6B8A83BC1D2@projects.linpro.no> Author: phk Date: 2007-11-20 15:20:36 +0100 (Tue, 20 Nov 2007) New Revision: 2268 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Log: Handle that the sessions may be killed the worker thread before we get around to delete the kevent. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 12:11:04 UTC (rev 2267) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 14:20:36 UTC (rev 2268) @@ -125,7 +125,7 @@ static pthread_t vca_kqueue_thread; static int kq = -1; -struct vbitmap *vca_kqueue_bits; +static struct vbitmap *vca_kqueue_bits; static VTAILQ_HEAD(,sess) sesshead = VTAILQ_HEAD_INITIALIZER(sesshead); @@ -137,13 +137,25 @@ static void vca_kq_sess(struct sess *sp, short arm) { + int i; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); if (++nki == NKEV) { - AZ(kevent(kq, ki, nki, NULL, 0, NULL)); + i = kevent(kq, ki, nki, NULL, 0, NULL); + assert(i <= 0); + if (i < 0) { + /* + * We do not push kevents into the kernel before passing the session off + * to a worker thread, so by the time we get around to delete the event + * the fd may be closed and we get an ENOENT back once we do flush. + * + * XXX: Can we get EBADF if the client closes during this window ? + */ + assert(errno == ENOENT); + } nki = 0; } } @@ -188,6 +200,7 @@ sp, (unsigned long)kp->data, kp->flags, (kp->flags & EV_EOF) ? " EOF" : ""); #endif + assert(sp->fd == kp->ident); if (kp->data > 0) { i = HTC_Rx(sp->htc); if (i == 0) @@ -233,8 +246,10 @@ assert(n >= 1 && n <= NKEV); nki = 0; for (kp = ke, j = 0; j < n; j++, kp++) { - if (kp->flags & EV_ERROR) + if (kp->flags & EV_ERROR) { + /* See comment in vca_kq_sess() */ continue; + } if (kp->filter == EVFILT_TIMER) { dotimer = 1; continue; From phk at projects.linpro.no Tue Nov 20 14:32:01 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 15:32:01 +0100 (CET) Subject: r2269 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120143201.D8D0A3BC1D2@projects.linpro.no> Author: phk Date: 2007-11-20 15:32:01 +0100 (Tue, 20 Nov 2007) New Revision: 2269 Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Log: Also accept deferred EBADF errors. Modified: trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 14:20:36 UTC (rev 2268) +++ trunk/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c 2007-11-20 14:32:01 UTC (rev 2269) @@ -148,13 +148,17 @@ assert(i <= 0); if (i < 0) { /* - * We do not push kevents into the kernel before passing the session off - * to a worker thread, so by the time we get around to delete the event - * the fd may be closed and we get an ENOENT back once we do flush. - * - * XXX: Can we get EBADF if the client closes during this window ? + * We do not push kevents into the kernel before + * passing the session off to a worker thread, so + * by the time we get around to delete the event + * the fd may be closed and we get an ENOENT back + * once we do flush. + * We can get EBADF the same way if the client closes + * on us. In that case, we get no kevent on that + * socket, but the TAILQ still has it, and it will + * be GC'ed there after the timeout. */ - assert(errno == ENOENT); + assert(errno == ENOENT || errno == EBADF); } nki = 0; } From des at linpro.no Tue Nov 20 14:36:25 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 15:36:25 +0100 Subject: r2230 - trunk/varnish-cache/bin/varnishncsa In-Reply-To: <20071120141639.GA51806@fupp.net> (Anders Nordby's message of "Tue, 20 Nov 2007 15:16:39 +0100") References: <20071108162826.BC65D1EC46F@projects.linpro.no> <20071120141639.GA51806@fupp.net> Message-ID: Anders Nordby writes: > This does not seem to work well. Upon sending SIGHUP to varnishncsa, it > does not start writing to the new log. I'll look into it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at projects.linpro.no Tue Nov 20 14:55:49 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Tue, 20 Nov 2007 15:55:49 +0100 (CET) Subject: r2270 - trunk/varnish-cache/bin/varnishncsa Message-ID: <20071120145549.B52933BC1D2@projects.linpro.no> Author: des Date: 2007-11-20 15:55:49 +0100 (Tue, 20 Nov 2007) New Revision: 2270 Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c Log: Make SIGHUP work: VSL_Dispatch() will not return as long as it has data to feed to h_ncsa(), so it is up to h_ncsa() to interrupt it if reopen is non-zero. Also remove a superfluous call to VSL_Arg() as was previously done in varnishlog(1). Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c =================================================================== --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-20 14:32:01 UTC (rev 2269) +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-20 14:55:49 UTC (rev 2270) @@ -80,6 +80,8 @@ #include "shmlog.h" #include "varnishapi.h" +static volatile sig_atomic_t reopen; + static struct logline { char *df_H; /* %H, Protocol version */ char *df_Host; /* %{Host}i */ @@ -187,7 +189,7 @@ end = ptr + len; if (!(spec & VSL_S_CLIENT || spec & VSL_S_BACKEND)) - return (0); + return (reopen); if (fd >= nll) { struct logline **newll = ll; @@ -315,11 +317,11 @@ } if ((spec & VSL_S_CLIENT) && tag != SLT_ReqEnd) - return (0); + return (reopen); if ((spec & VSL_S_BACKEND) && tag != SLT_BackendReuse && (tag != SLT_BackendClose || lp->df_Uq)) - return (0); + return (reopen); if (tag == SLT_ReqEnd) { if (sscanf(ptr, "%*u %*u.%*u %ld.", &l) != 1) @@ -407,13 +409,11 @@ #undef freez lp->bogus = 0; - return (0); + return (reopen); } /*--------------------------------------------------------------------*/ -static volatile sig_atomic_t reopen; - static void sighup(int sig) { @@ -447,7 +447,7 @@ int main(int argc, char *argv[]) { - int i, c; + int c; int a_flag = 0, D_flag = 0; const char *n_arg = NULL; const char *P_arg = NULL; @@ -459,11 +459,6 @@ vd = VSL_New(); while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { - i = VSL_Arg(vd, c, optarg); - if (i < 0) - exit (1); - if (i > 0) - continue; switch (c) { case 'a': a_flag = 1; @@ -516,7 +511,7 @@ of = stdout; } - while (VSL_Dispatch(vd, h_ncsa, of) == 0) { + while (VSL_Dispatch(vd, h_ncsa, of) >= 0) { if (fflush(of) != 0) { perror(w_arg); exit(1); From des at projects.linpro.no Tue Nov 20 14:56:21 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Tue, 20 Nov 2007 15:56:21 +0100 (CET) Subject: r2271 - trunk/varnish-cache/bin/varnishlog Message-ID: <20071120145621.A7AB13BC1D2@projects.linpro.no> Author: des Date: 2007-11-20 15:56:21 +0100 (Tue, 20 Nov 2007) New Revision: 2271 Modified: trunk/varnish-cache/bin/varnishlog/varnishlog.c Log: Fix some style issues, and harmonize with varnishncsa(1). Modified: trunk/varnish-cache/bin/varnishlog/varnishlog.c =================================================================== --- trunk/varnish-cache/bin/varnishlog/varnishlog.c 2007-11-20 14:55:49 UTC (rev 2270) +++ trunk/varnish-cache/bin/varnishlog/varnishlog.c 2007-11-20 14:56:21 UTC (rev 2271) @@ -179,14 +179,14 @@ match_tag = name2tag(argv[0]); if (match_tag < 0) { fprintf(stderr, "Tag \"%s\" unknown\n", argv[0]); - exit (2); + exit(2); } i = regcomp(&match_re, argv[1], REG_EXTENDED | REG_NOSUB); if (i) { char buf[BUFSIZ]; regerror(i, &match_re, buf, sizeof buf); fprintf(stderr, "%s\n", buf); - exit (2); + exit(2); } } if (!b_flag) { @@ -235,7 +235,7 @@ fd = open(w_arg, flags, 0644); if (fd < 0) { perror(w_arg); - exit (1); + exit(1); } return (fd); } @@ -265,7 +265,7 @@ reopen = 0; } } - exit (0); + exit(0); } /*--------------------------------------------------------------------*/ @@ -281,7 +281,7 @@ int main(int argc, char **argv) { - int i, c; + int c; int a_flag = 0, D_flag = 0, o_flag = 0; const char *n_arg = NULL; const char *P_arg = NULL; @@ -358,12 +358,11 @@ if (o_flag) do_order(vd, argc - optind, argv + optind); - while (1) { - i = VSL_Dispatch(vd, VSL_H_Print, stdout); - if (i == 0) - fflush(stdout); - else if (i < 0) + while (VSL_Dispatch(vd, VSL_H_Print, stdout) >= 0) { + if (fflush(stdout) != 0) { + perror("stdout"); break; + } } if (pfh != NULL) From des at projects.linpro.no Tue Nov 20 14:58:51 2007 From: des at projects.linpro.no (des at projects.linpro.no) Date: Tue, 20 Nov 2007 15:58:51 +0100 (CET) Subject: r2272 - in trunk/varnish-cache: bin/varnishd include Message-ID: <20071120145851.9EB551EC2B4@projects.linpro.no> Author: des Date: 2007-11-20 15:58:51 +0100 (Tue, 20 Nov 2007) New Revision: 2272 Modified: trunk/varnish-cache/bin/varnishd/cache.h trunk/varnish-cache/include/libvarnish.h Log: Many of our assertions have side effects, so the condition needs to be evaluated even if we don't bother to check the result. We should trust the compiler to eliminate code that has no effect. Modified: trunk/varnish-cache/bin/varnishd/cache.h =================================================================== --- trunk/varnish-cache/bin/varnishd/cache.h 2007-11-20 14:56:21 UTC (rev 2271) +++ trunk/varnish-cache/bin/varnishd/cache.h 2007-11-20 14:58:51 UTC (rev 2272) @@ -669,7 +669,7 @@ } #ifdef WITHOUT_ASSERTS -#define spassert(cond) ((void)0) +#define spassert(cond) ((void)(cond)) #else void panic(const char *, int, const char *, const struct sess *, const char *, ...); Modified: trunk/varnish-cache/include/libvarnish.h =================================================================== --- trunk/varnish-cache/include/libvarnish.h 2007-11-20 14:56:21 UTC (rev 2271) +++ trunk/varnish-cache/include/libvarnish.h 2007-11-20 14:58:51 UTC (rev 2272) @@ -63,7 +63,7 @@ */ #ifdef WITHOUT_ASSERTS -#define assert(e) ((void)0) +#define assert(e) ((void)(e)) #else /* WITH_ASSERTS */ #define assert(e) \ do { \ From phk at phk.freebsd.dk Tue Nov 20 15:02:30 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Nov 2007 15:02:30 +0000 Subject: r2272 - in trunk/varnish-cache: bin/varnishd include In-Reply-To: Your message of "Tue, 20 Nov 2007 15:58:51 +0100." <20071120145851.9EB551EC2B4@projects.linpro.no> Message-ID: <19910.1195570950@critter.freebsd.dk> In message <20071120145851.9EB551EC2B4 at projects.linpro.no>, des at projects.linpro .no writes: >Modified: > trunk/varnish-cache/bin/varnishd/cache.h > trunk/varnish-cache/include/libvarnish.h >Log: >Many of our assertions have side effects, so the condition needs to be >evaluated even if we don't bother to check the result. We should trust >the compiler to eliminate code that has no effect. Are we going to allow the asserts to be compiled out ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at projects.linpro.no Tue Nov 20 16:30:24 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 17:30:24 +0100 (CET) Subject: r2273 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120163024.2A3AC1EC290@projects.linpro.no> Author: phk Date: 2007-11-20 17:30:23 +0100 (Tue, 20 Nov 2007) New Revision: 2273 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Make the incmt and remflg persistent across the parse chunks Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 14:58:51 UTC (rev 2272) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 16:30:23 UTC (rev 2273) @@ -77,6 +77,8 @@ struct esi_bit *ebl; /* list of */ int neb; int is_esi; + int remflg; /* inside */ + int incmt; /* inside comment */ }; /*-------------------------------------------------------------------- @@ -343,29 +345,25 @@ char *p, *q, *r; txt t, o; int celem; /* closing element */ - int remflg; /* inside */ - int incmt; /* inside comment */ int i; t.b = (char *)ew->st->ptr; t.e = t.b + ew->st->len; ew->dst.b = t.b; ew->dst.e = t.b; - remflg = 0; - incmt = 0; o.b = t.b; for (p = t.b; p < t.e; ) { - if (incmt && *p == '-') { + if (ew->incmt && *p == '-') { /* * We are inside an when we see it. */ if (p + 2 >= t.e) { - /* XXX: need to return pending incmt */ + /* XXX: need to return pending ew->incmt */ return (p - t.b); } if (!memcmp(p, "-->", 3)) { - incmt = 0; + ew->incmt = 0; o.e = p; esi_addverbatim(ew, o); p += 3; @@ -386,12 +384,12 @@ if (i < 2) return (p - t.b); - if (remflg == 0 && !memcmp(p, " * at least 10 char, but we only test on the * first seven because the tail is handled - * by the incmt flag. + * by the ew->incmt flag. */ ew->is_esi++; if (i < 7) @@ -402,7 +400,7 @@ p += 7; o.b = p; - incmt = 1; + ew->incmt = 1; continue; } @@ -461,15 +459,15 @@ ew->is_esi++; - if (celem != remflg) { + if (celem != ew->remflg) { /* * ESI 1.0 violation, ignore element */ - esi_error(ew, p, 1 + q - p, - remflg ? "ESI 1.0 forbids nested esi:remove" + esi_error(ew, p, 1 + q - p, ew->remflg ? + "ESI 1.0 forbids nested esi:remove" : "ESI 1.0 esi:remove not opened"); - if (!remflg) { + if (!ew->remflg) { o.e = p; esi_addverbatim(ew, o); } @@ -481,17 +479,17 @@ /* open element */ o.e = p; esi_addverbatim(ew, o); - remflg = !celem; + ew->remflg = !celem; } else { /* close element */ - remflg = !celem; + ew->remflg = !celem; } p = q + 1; o.b = p; continue; } - if (remflg && r + 3 < q && !memcmp(r, "esi:", 4)) { + if (ew->remflg && r + 3 < q && !memcmp(r, "esi:", 4)) { /* * ESI 1.0 violation, no esi: elements in esi:remove */ From anders at fupp.net Tue Nov 20 16:41:57 2007 From: anders at fupp.net (Anders Nordby) Date: Tue, 20 Nov 2007 17:41:57 +0100 Subject: r2270 - trunk/varnish-cache/bin/varnishncsa In-Reply-To: <20071120145549.B52933BC1D2@projects.linpro.no> References: <20071120145549.B52933BC1D2@projects.linpro.no> Message-ID: <20071120164157.GA56892@fupp.net> Hi, Finally I can use varnishncsa with a newsyslog to rotate the logs. Great, thanks! Other wishes: - It would be nice if varnishncsa could also log different virtual hosts in different files, and also make it possible to choose which virtual hosts to log for/not. - It would be nice to be able to specify what to log, as in Apache CustomLog/LogFormat. :-) On Tue, Nov 20, 2007 at 03:55:49PM +0100, des at projects.linpro.no wrote: > Author: des > Date: 2007-11-20 15:55:49 +0100 (Tue, 20 Nov 2007) > New Revision: 2270 > > Modified: > trunk/varnish-cache/bin/varnishncsa/varnishncsa.c > Log: > Make SIGHUP work: VSL_Dispatch() will not return as long as it has data > to feed to h_ncsa(), so it is up to h_ncsa() to interrupt it if reopen > is non-zero. > > Also remove a superfluous call to VSL_Arg() as was previously done in > varnishlog(1). > > > Modified: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c > =================================================================== > --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-20 14:32:01 UTC (rev 2269) > +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c 2007-11-20 14:55:49 UTC (rev 2270) > @@ -80,6 +80,8 @@ > #include "shmlog.h" > #include "varnishapi.h" > > +static volatile sig_atomic_t reopen; > + > static struct logline { > char *df_H; /* %H, Protocol version */ > char *df_Host; /* %{Host}i */ > @@ -187,7 +189,7 @@ > end = ptr + len; > > if (!(spec & VSL_S_CLIENT || spec & VSL_S_BACKEND)) > - return (0); > + return (reopen); > > if (fd >= nll) { > struct logline **newll = ll; > @@ -315,11 +317,11 @@ > } > > if ((spec & VSL_S_CLIENT) && tag != SLT_ReqEnd) > - return (0); > + return (reopen); > > if ((spec & VSL_S_BACKEND) && tag != SLT_BackendReuse && > (tag != SLT_BackendClose || lp->df_Uq)) > - return (0); > + return (reopen); > > if (tag == SLT_ReqEnd) { > if (sscanf(ptr, "%*u %*u.%*u %ld.", &l) != 1) > @@ -407,13 +409,11 @@ > #undef freez > lp->bogus = 0; > > - return (0); > + return (reopen); > } > > /*--------------------------------------------------------------------*/ > > -static volatile sig_atomic_t reopen; > - > static void > sighup(int sig) > { > @@ -447,7 +447,7 @@ > int > main(int argc, char *argv[]) > { > - int i, c; > + int c; > int a_flag = 0, D_flag = 0; > const char *n_arg = NULL; > const char *P_arg = NULL; > @@ -459,11 +459,6 @@ > vd = VSL_New(); > > while ((c = getopt(argc, argv, VSL_ARGS "aDn:P:Vw:")) != -1) { > - i = VSL_Arg(vd, c, optarg); > - if (i < 0) > - exit (1); > - if (i > 0) > - continue; > switch (c) { > case 'a': > a_flag = 1; > @@ -516,7 +511,7 @@ > of = stdout; > } > > - while (VSL_Dispatch(vd, h_ncsa, of) == 0) { > + while (VSL_Dispatch(vd, h_ncsa, of) >= 0) { > if (fflush(of) != 0) { > perror(w_arg); > exit(1); > > _______________________________________________ > varnish-commit mailing list > varnish-commit at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-commit -- Anders. From phk at projects.linpro.no Tue Nov 20 16:43:09 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 17:43:09 +0100 (CET) Subject: r2274 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120164309.E32061EC40A@projects.linpro.no> Author: phk Date: 2007-11-20 17:43:09 +0100 (Tue, 20 Nov 2007) New Revision: 2274 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Pass a txt instead of the raw storage into the esi_parser Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 16:30:23 UTC (rev 2273) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 16:43:09 UTC (rev 2274) @@ -71,7 +71,7 @@ struct esi_work { struct sess *sp; size_t off; - struct storage *st; + txt t; txt dst; struct esi_bit *eb; struct esi_bit *ebl; /* list of */ @@ -93,14 +93,13 @@ txt t; if (i == 0) - i = p - ((char *)ew->st->ptr + ew->st->len); + i = p - ew->t.b; if (i > 20) { i = 20; ellipsis = 1; } q = buf; - q += sprintf(buf, "at %zd: %s \"", - ew->off + (p - (char*)ew->st->ptr), err); + q += sprintf(buf, "at %zd: %s \"", ew->off + (p - ew->t.b), err); while (i > 0) { if (*p >= ' ' && *p <= '~') { *q++ = *p; @@ -347,8 +346,7 @@ int celem; /* closing element */ int i; - t.b = (char *)ew->st->ptr; - t.e = t.b + ew->st->len; + t = ew->t; ew->dst.b = t.b; ew->dst.e = t.b; o.b = t.b; @@ -574,7 +572,8 @@ ew->off = 1; VTAILQ_FOREACH(st, &sp->obj->store, list) { - ew->st = st; + ew->t.b = (void *)st->ptr; + ew->t.e = ew->t.b + st->len; i = esi_parse(ew); ew->off += st->len; if (i < st->len) { From des at linpro.no Tue Nov 20 16:49:50 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 17:49:50 +0100 Subject: r2272 - in trunk/varnish-cache: bin/varnishd include In-Reply-To: <19910.1195570950@critter.freebsd.dk> (Poul-Henning Kamp's message of "Tue, 20 Nov 2007 15:02:30 +0000") References: <19910.1195570950@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > Many of our assertions have side effects, so the condition needs to be > > evaluated even if we don't bother to check the result. We should trust > > the compiler to eliminate code that has no effect. > Are we going to allow the asserts to be compiled out ? We don't, currently, but we might. In fact, I think we *should*, in the long run. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Nov 20 16:53:06 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 17:53:06 +0100 Subject: r2270 - trunk/varnish-cache/bin/varnishncsa In-Reply-To: <20071120164157.GA56892@fupp.net> (Anders Nordby's message of "Tue, 20 Nov 2007 17:41:57 +0100") References: <20071120145549.B52933BC1D2@projects.linpro.no> <20071120164157.GA56892@fupp.net> Message-ID: Anders Nordby writes: > - It would be nice if varnishncsa could also log different virtual hosts > in different files, and also make it possible to choose which virtual > hosts to log for/not. I'm not sure how easy that would be. I guess at the most basic level you could run a separate varnishncsa process for each virtual host. > - It would be nice to be able to specify what to log, as in Apache > CustomLog/LogFormat. This is on my todo list, but there are more pressing issues... BTW, I should warn you that while it is very useful, varnishncsa is also currently very CPU intensive. It needs a lot of work for which I don't currently have the time. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Nov 20 17:01:22 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Nov 2007 17:01:22 +0000 Subject: r2272 - in trunk/varnish-cache: bin/varnishd include In-Reply-To: Your message of "Tue, 20 Nov 2007 17:49:50 +0100." Message-ID: <26134.1195578082@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: >"Poul-Henning Kamp" writes: >> Dag-Erling Sm=F8rgrav writes: >> > Many of our assertions have side effects, so the condition needs to be >> > evaluated even if we don't bother to check the result. We should trust >> > the compiler to eliminate code that has no effect. >> Are we going to allow the asserts to be compiled out ? > >We don't, currently, but we might. In fact, I think we *should*, in >the long run. Hmm, I generally tend to make sure asserts are cheap and then leave them in. It's not like the conditions become more acceptable because you run production :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at projects.linpro.no Tue Nov 20 17:20:48 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 18:20:48 +0100 (CET) Subject: r2275 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120172048.0BA561EC2B4@projects.linpro.no> Author: phk Date: 2007-11-20 18:20:47 +0100 (Tue, 20 Nov 2007) New Revision: 2275 Modified: trunk/varnish-cache/bin/varnishd/cache_fetch.c Log: Reset the objects length to zero after we zapped all its component parts, otherwise we will get trouble (#167 ?) later. Modified: trunk/varnish-cache/bin/varnishd/cache_fetch.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_fetch.c 2007-11-20 16:43:09 UTC (rev 2274) +++ trunk/varnish-cache/bin/varnishd/cache_fetch.c 2007-11-20 17:20:47 UTC (rev 2275) @@ -395,6 +395,7 @@ } VBE_UpdateHealth(sp, vc, -4); VBE_ClosedFd(sp->wrk, vc); + sp->obj->len = 0; return (__LINE__); } From des at linpro.no Tue Nov 20 17:30:53 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 18:30:53 +0100 Subject: r2272 - in trunk/varnish-cache: bin/varnishd include In-Reply-To: <26134.1195578082@critter.freebsd.dk> (Poul-Henning Kamp's message of "Tue, 20 Nov 2007 17:01:22 +0000") References: <26134.1195578082@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Hmm, I generally tend to make sure asserts are cheap and then leave > them in. It's not like the conditions become more acceptable because > you run production :-) That's a good principle, but where do you draw the line between keeping assertions cheap enough, and adding enough of them to matter? I prefer to have the option of compiling them out so I don't have to worry about how expensive they are. There is no obviously expensive stuff in our assertions today (the obviously expensive stuff is in #ifdef DIAGNOSTICS), but even the cheap stuff adds up (and increases code size, which may hurt cache locality) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Nov 20 18:07:28 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 20 Nov 2007 18:07:28 +0000 Subject: r2272 - in trunk/varnish-cache: bin/varnishd include In-Reply-To: Your message of "Tue, 20 Nov 2007 18:30:53 +0100." Message-ID: <27344.1195582048@critter.freebsd.dk> In message , =?iso-8859-1?Q?Dag-Erling_Sm=F8rg rav?= writes: >"Poul-Henning Kamp" writes: >> Hmm, I generally tend to make sure asserts are cheap and then leave >> them in. It's not like the conditions become more acceptable because >> you run production :-) > >That's a good principle, but where do you draw the line between >keeping assertions cheap enough, and adding enough of them to matter? >I prefer to have the option of compiling them out so I don't have to >worry about how expensive they are. There is no obviously expensive >stuff in our assertions today (the obviously expensive stuff is in >#ifdef DIAGNOSTICS), but even the cheap stuff adds up (and increases >code size, which may hurt cache locality) It is a tradeoff, but I do consider it on the same order as being able to drive faster without the wasted weight of seat-belts and airbags in the car :-) Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at projects.linpro.no Tue Nov 20 18:33:09 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 19:33:09 +0100 (CET) Subject: r2276 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120183309.D99C61ECF14@projects.linpro.no> Author: phk Date: 2007-11-20 19:33:09 +0100 (Tue, 20 Nov 2007) New Revision: 2276 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Make sure to add verbatim anything leading up to whatever we fail to parse due to lack of bytes. Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 17:20:47 UTC (rev 2275) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 18:33:09 UTC (rev 2276) @@ -72,6 +72,7 @@ struct sess *sp; size_t off; txt t; + txt o; txt dst; struct esi_bit *eb; struct esi_bit *ebl; /* list of */ @@ -165,12 +166,12 @@ */ static void -esi_addverbatim(struct esi_work *ew, txt t) +esi_addverbatim(struct esi_work *ew) { - if (t.b != ew->dst.e) - memmove(ew->dst.e, t.b, Tlen(t)); - ew->dst.e += Tlen(t); + if (ew->o.b != ew->dst.e) + memmove(ew->dst.e, ew->o.b, Tlen(ew->o)); + ew->dst.e += Tlen(ew->o); } /*-------------------------------------------------------------------- @@ -338,18 +339,18 @@ * Return value: number of bytes processed. */ -static int -esi_parse(struct esi_work *ew) +static char * +esi_parse2(struct esi_work *ew) { char *p, *q, *r; - txt t, o; + txt t; int celem; /* closing element */ int i; t = ew->t; ew->dst.b = t.b; ew->dst.e = t.b; - o.b = t.b; + ew->o.b = t.b; for (p = t.b; p < t.e; ) { if (ew->incmt && *p == '-') { /* @@ -358,14 +359,14 @@ */ if (p + 2 >= t.e) { /* XXX: need to return pending ew->incmt */ - return (p - t.b); + return (p); } if (!memcmp(p, "-->", 3)) { ew->incmt = 0; - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); p += 3; - o.b = p; + ew->o.b = p; } else p++; continue; @@ -380,7 +381,7 @@ i = t.e - p; if (i < 2) - return (p - t.b); + return (p); if (ew->remflg == 0 && !memcmp(p, " at least 7 char */ if (i < 7) - return (p - t.b); + return (p); for (q = p + 4; ; q++) { if (q + 2 >= t.e) - return (p - t.b); + return (p); if (!memcmp(q, "-->", 3)) break; } @@ -421,12 +422,13 @@ if (!memcmp(p, " 9 ? 9 : i)) { /* * cdata at least 12 char + * XXX: can span multiple chunks */ if (i < 12) - return (p - t.b); + return (p); for (q = p + 9; ; q++) { if (q + 2 >= t.e) - return (p - t.b); + return (p); if (!memcmp(q, "]]>", 3)) break; } @@ -438,7 +440,7 @@ for (q = p + 1; q < t.e && *q != '>'; q++) continue; if (*q != '>') - return (p - t.b); + return (p); /* Opening/empty or closing element ? */ if (p[1] == '/') { @@ -466,24 +468,24 @@ : "ESI 1.0 esi:remove not opened"); if (!ew->remflg) { - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); } } else if (!celem && q[-1] == '/') { /* empty element */ - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); } else if (!celem) { /* open element */ - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); ew->remflg = !celem; } else { /* close element */ ew->remflg = !celem; } p = q + 1; - o.b = p; + ew->o.b = p; continue; } @@ -501,19 +503,19 @@ ew->is_esi++; - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); if (celem == 0) { - o.b = r + 11; + ew->o.b = r + 11; if (q[-1] != '/') { esi_error(ew, p, 1 + q - p, "ESI 1.0 wants emtpy esi:include"); - o.e = q; + ew->o.e = q; } else { - o.e = q - 1; + ew->o.e = q - 1; } - esi_addinclude(ew, o); + esi_addinclude(ew, ew->o); ew->dst.b = q + 1; ew->dst.e = q + 1; } else { @@ -521,7 +523,7 @@ "ESI 1.0 closing esi:include illegal"); } p = q + 1; - o.b = p; + ew->o.b = p; continue; } @@ -531,21 +533,31 @@ */ esi_error(ew, p, 1 + q - p, "ESI 1.0 unimplemented element"); - o.e = p; - esi_addverbatim(ew, o); + ew->o.e = p; + esi_addverbatim(ew); p = q + 1; - o.b = p; + ew->o.b = p; continue; } /* Not an element we care about */ p = q + 1; } - o.e = p; - esi_addverbatim(ew, o); - return (p - t.b); + return (p); } +static int +esi_parse(struct esi_work *ew) +{ + char *p; + + p = esi_parse2(ew); + ew->o.e = p; + if (Tlen(ew->o)) + esi_addverbatim(ew); + return (p - ew->t.b); +} + /*--------------------------------------------------------------------*/ void From phk at projects.linpro.no Tue Nov 20 20:08:14 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 21:08:14 +0100 (CET) Subject: r2277 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120200814.0E1311EC2B4@projects.linpro.no> Author: phk Date: 2007-11-20 21:08:13 +0100 (Tue, 20 Nov 2007) New Revision: 2277 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Implement almost complete ESI parsing for objects spanning multiple storage chunks. Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 18:33:09 UTC (rev 2276) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 20:08:13 UTC (rev 2277) @@ -156,7 +156,8 @@ VTAILQ_INSERT_TAIL(&ew->sp->obj->esibits, ew->eb, list); ew->eb->verbatim = ew->dst; sprintf(ew->eb->chunk_length, "%x\r\n", Tlen(ew->dst)); - VSL(SLT_Debug, ew->sp->fd, "AddBit: %.*s", Tlen(ew->dst), ew->dst.b); + VSL(SLT_Debug, ew->sp->fd, "AddBit: %d <%.*s>", + Tlen(ew->dst), Tlen(ew->dst), ew->dst.b); return(ew->eb); } @@ -169,6 +170,8 @@ esi_addverbatim(struct esi_work *ew) { + VSL(SLT_Debug, ew->sp->fd, "AddVer: %d <%.*s>", + Tlen(ew->o), Tlen(ew->o), ew->o.b); if (ew->o.b != ew->dst.e) memmove(ew->dst.e, ew->o.b, Tlen(ew->o)); ew->dst.e += Tlen(ew->o); @@ -543,19 +546,27 @@ /* Not an element we care about */ p = q + 1; } + assert(p == t.e); return (p); } -static int +static char * esi_parse(struct esi_work *ew) { char *p; + VSL(SLT_Debug, ew->sp->fd, "Parse: %d <%.*s>", + Tlen(ew->t), Tlen(ew->t), ew->t.b); p = esi_parse2(ew); + assert(ew->o.b >= ew->t.b); + assert(ew->o.e <= ew->t.e); ew->o.e = p; if (Tlen(ew->o)) esi_addverbatim(ew); - return (p - ew->t.b); + if (Tlen(ew->dst)) + esi_addbit(ew); + ew->off += (p - ew->t.b); + return (p); } /*--------------------------------------------------------------------*/ @@ -563,11 +574,14 @@ void VRT_ESI(struct sess *sp) { - struct storage *st; + struct storage *st, *st2; struct esi_work *ew, eww[1]; - int i; + txt t; + unsigned u; + char *p, *q; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + assert(sp->obj->busy); if (sp->cur_method != VCL_MET_FETCH) { /* XXX: we should catch this at compile time */ WSP(sp, SLT_VCL_error, @@ -583,19 +597,64 @@ ew->sp = sp; ew->off = 1; + p = NULL; VTAILQ_FOREACH(st, &sp->obj->store, list) { - ew->t.b = (void *)st->ptr; - ew->t.e = ew->t.b + st->len; - i = esi_parse(ew); - ew->off += st->len; - if (i < st->len) { - /* XXX: Handle complications */ - if (VTAILQ_NEXT(st, list)) + if (p != NULL) { + assert ((void*)p > (void *)st->ptr); + assert ((void*)p <= (void *)(st->ptr + st->len)); + if (p == (void*)(st->ptr + st->len)) + break; + ew->t.b = p; + p = NULL; + } else + ew->t.b = (void *)st->ptr; + ew->t.e = (void *)(st->ptr + st->len); + p = esi_parse(ew); + if (p == ew->t.e) { + p = NULL; + continue; + } + + if (VTAILQ_NEXT(st, list) == NULL) { + /* + * XXX: illegal XML input, but did we handle + * XXX: all of it, or do we leave the final + * XXX: element dangling ? + */ + INCOMPL(); + } + + /* Move remainder to workspace */ + u = ew->t.e - p; + t.b = sp->obj->ws_o->f; + t.e = t.b + WS_Reserve(sp->obj->ws_o, 0); + assert(t.e > t.b + u); /* XXX incredibly long element ? */ + memcpy(t.b, p, u); + + /* Peel start off next chunk, until and including '<' */ + st2 = VTAILQ_NEXT(st, list); + AN(st2); + q = t.b + u; + p = (void*)st2->ptr; + while (1) { + if (p >= (char *)st2->ptr + st2->len) INCOMPL(); + if (q >= t.e) + INCOMPL(); + *q = *p++; + if (*q++ == '>') + break; } + WS_ReleaseP(sp->obj->ws_o, q); + t.e = q; + + /* Parse this bit */ + ew->t = t; + q = esi_parse(ew); + assert(q == ew->t.e); /* XXX */ + + /* 'p' is cached starting point for next storage part */ } - if (Tlen(ew->dst)) - esi_addbit(ew); if (!ew->is_esi) { ESI_Destroy(sp->obj); From phk at projects.linpro.no Tue Nov 20 20:41:19 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 21:41:19 +0100 (CET) Subject: r2278 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120204119.905911EC290@projects.linpro.no> Author: phk Date: 2007-11-20 21:41:19 +0100 (Tue, 20 Nov 2007) New Revision: 2278 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Storage border bug fixes Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 20:08:13 UTC (rev 2277) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 20:41:19 UTC (rev 2278) @@ -354,6 +354,7 @@ ew->dst.b = t.b; ew->dst.e = t.b; ew->o.b = t.b; + ew->o.e = t.b; for (p = t.b; p < t.e; ) { if (ew->incmt && *p == '-') { /* @@ -442,9 +443,11 @@ /* Find end of this element */ for (q = p + 1; q < t.e && *q != '>'; q++) continue; - if (*q != '>') + if (q >= t.e || *q != '>') return (p); + +VSL(SLT_Debug, ew->sp->fd, "Element: [%.*s]", q - p, p); /* Opening/empty or closing element ? */ if (p[1] == '/') { celem = 1; @@ -544,6 +547,7 @@ } /* Not an element we care about */ + assert(q < t.e); p = q + 1; } assert(p == t.e); @@ -561,7 +565,7 @@ assert(ew->o.b >= ew->t.b); assert(ew->o.e <= ew->t.e); ew->o.e = p; - if (Tlen(ew->o)) + if (Tlen(ew->o) && !ew->remflg) esi_addverbatim(ew); if (Tlen(ew->dst)) esi_addbit(ew); From phk at projects.linpro.no Tue Nov 20 21:01:00 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 22:01:00 +0100 (CET) Subject: r2279 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120210100.713071EC462@projects.linpro.no> Author: phk Date: 2007-11-20 22:01:00 +0100 (Tue, 20 Nov 2007) New Revision: 2279 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Warn about XML/ESI language elements that are not properly closed. Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 20:41:19 UTC (rev 2278) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 21:01:00 UTC (rev 2279) @@ -625,7 +625,12 @@ * XXX: all of it, or do we leave the final * XXX: element dangling ? */ - INCOMPL(); + esi_error(ew, p, ew->t.e -p, + "XML 1.0 incomplete language element"); + ew->dst.b = p; + ew->dst.e = ew->t.e; + esi_addbit(ew); + break; } /* Move remainder to workspace */ @@ -660,6 +665,13 @@ /* 'p' is cached starting point for next storage part */ } + if (ew->remflg) + esi_error(ew, NULL, 0, + "ESI 1.0 unterminated element"); + if (ew->incmt) + esi_error(ew, NULL, 0, + "ESI 1.0 unterminated comment */ + int incdata; /* inside */ }; /*-------------------------------------------------------------------- @@ -356,15 +357,33 @@ ew->o.b = t.b; ew->o.e = t.b; for (p = t.b; p < t.e; ) { + assert(p >= t.b); + assert(p < t.e); + if (ew->incdata) { + /* + * We are inside an . + */ + if (*p != ']') { + p++; + } else { + if (p + 2 >= t.e) + return (p); + if (!memcmp(p, "]]>", 3)) { + ew->incdata = 0; + p += 3; + } else + p++; + } + continue; + } if (ew->incmt && *p == '-') { /* * We are inside an when we see it. */ - if (p + 2 >= t.e) { - /* XXX: need to return pending ew->incmt */ + if (p + 2 >= t.e) return (p); - } if (!memcmp(p, "-->", 3)) { ew->incmt = 0; ew->o.e = p; @@ -425,18 +444,12 @@ if (!memcmp(p, " 9 ? 9 : i)) { /* - * cdata at least 12 char - * XXX: can span multiple chunks + * cdata = t.e) - return (p); - if (!memcmp(q, "]]>", 3)) - break; - } - p = q + 3; + ew->incdata = 1; + p += 9; continue; } @@ -683,6 +696,9 @@ /* 'p' is cached starting point for next storage part */ } + if (ew->incdata) + esi_error(ew, ew->t.e, -1, + "ESI 1.0 unterminated remflg) esi_error(ew, ew->t.e, -1, "ESI 1.0 unterminated element"); From phk at projects.linpro.no Tue Nov 20 22:21:06 2007 From: phk at projects.linpro.no (phk at projects.linpro.no) Date: Tue, 20 Nov 2007 23:21:06 +0100 (CET) Subject: r2282 - trunk/varnish-cache/bin/varnishd Message-ID: <20071120222106.2A4261EC2B4@projects.linpro.no> Author: phk Date: 2007-11-20 23:21:05 +0100 (Tue, 20 Nov 2007) New Revision: 2282 Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c Log: Add esi:comment support. "Support" is defined as: Silently ignore, as opposed to unhandled esi: elments which we complain about in varnishlog. Thus, if you want to get a short comment into the shmlog, the easiest way is to do something like which will result in a shmlog record: 11 ESI_xmlerror c at 25: ESI 1.0 unimplemented element "" But the length of the message is truncated to avoid dumping the entire source document into the shmlog. Snip out any unknown esi: element. While the ESI 1.0 specification doesn't address this directly, my impression from the document is that they should never leak through to the client. Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c =================================================================== --- trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 21:47:37 UTC (rev 2281) +++ trunk/varnish-cache/bin/varnishd/cache_vrt_esi.c 2007-11-20 22:21:05 UTC (rev 2282) @@ -459,8 +459,6 @@ if (q >= t.e || *q != '>') return (p); - -VSL(SLT_Debug, ew->sp->fd, "Element: [%.*s]", q - p, p); /* Opening/empty or closing element ? */ if (p[1] == '/') { celem = 1; @@ -474,6 +472,9 @@ r = p + 1; } + VSL(SLT_Debug, ew->sp->fd, "Element: clos=%d [%.*s]", + celem, q - r, r); + if (r + 9 < q && !memcmp(r, "esi:remove", 10)) { ew->is_esi++; @@ -517,11 +518,26 @@ p = q + 1; continue; } + ew->is_esi++; + if (r + 10 < q && !memcmp(r, "esi:comment", 11)) { + + ew->o.e = p; + esi_addverbatim(ew); + + if (celem == 1) { + esi_error(ew, p, 1 + q - p, + "ESI 1.0 closing esi:comment illegal"); + } else if (q[-1] != '/') { + esi_error(ew, p, 1 + q - p, + "ESI 1.0 wants emtpy esi:comment"); + } + p = q + 1; + ew->o.b = p; + continue; + } if (r + 10 < q && !memcmp(r, "esi:include", 11)) { - ew->is_esi++; - ew->o.e = p; esi_addverbatim(ew); @@ -696,6 +712,13 @@ /* 'p' is cached starting point for next storage part */ } + /* + * XXX: we could record the starting point of these elements + * XXX: so that the char-index were more useful, but we are + * XXX: not trivially able to print their contents, so leave + * XXX: it like this for now, pending more thought about the + * XXX: proper way to report these errors. + */ if (ew->incdata) esi_error(ew, ew->t.e, -1, "ESI 1.0 unterminated Author: phk Date: 2007-11-20 23:28:04 +0100 (Tue, 20 Nov 2007) New Revision: 2283 Added: trunk/varnish-cache/bin/varnishd/c.sh trunk/varnish-cache/bin/varnishd/s.sh Log: Plug these two test-scripts in for the night, until some day where I find out where to put ad-hoc stuff like this in the tree. Added: trunk/varnish-cache/bin/varnishd/c.sh =================================================================== --- trunk/varnish-cache/bin/varnishd/c.sh (rev 0) +++ trunk/varnish-cache/bin/varnishd/c.sh 2007-11-20 22:28:04 UTC (rev 2283) @@ -0,0 +1,15 @@ +#!/bin/sh + +# A client side script to test the ESI parsing, see s.sh for serverside + +set -e + +echo "" +while true +do + sleep 1 + echo "" + fetch -o - -q http://localhost:8080/ | hexdump -C | + sed 's/$//' + echo "" +done Added: trunk/varnish-cache/bin/varnishd/s.sh =================================================================== --- trunk/varnish-cache/bin/varnishd/s.sh (rev 0) +++ trunk/varnish-cache/bin/varnishd/s.sh 2007-11-20 22:28:04 UTC (rev 2283) @@ -0,0 +1,82 @@ +#!/bin/sh + +# A server side test-script for pushing the ESI XML parser over a +# storage boundary. +# The crucial trick here is that we send these objects HTTP/0.9 style +# so that cache_fetch puts the first 128k in one storage object and +# the rest in another, thus by putting around 128K space in our test +# data we can put it right before, over and after the storage boundary. +# +# Use c.sh as the client side, and run varnish with this vcl: +# +# backend b1 { +# set backend.host = "Localhost"; +# set backend.port = "8081"; +# } +# +# sub vcl_recv { +# pass; +# } +# +# sub vcl_fetch { +# esi; +# } + +serve () ( + ( + echo 'HTTP/1.0 200 OK' + echo '' + echo "$1" + dd if=/dev/zero bs=$2 count=1 2>/dev/null | tr '\0' ' ' + cat + sleep .1 + ) | nc -l 8081 +) + + + +if false ; then + echo -n " foo bar" | serve Test01 1 + echo -n " foo bar" | serve Test02 2 + # Unterminated CDATA + echo -n " foo { foo bar" | serve Test04::$i $i + done + + for i in `jot 40 131036` + do + echo -n " bar" | serve Test05::$i $i + done + + for i in `jot 22 131040` + do + echo -n "" | serve Test06::$i $i + done + + echo -n " " | serve Test07 10 + + echo -n "