From martin at varnish-software.com Fri Oct 3 14:43:10 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 3 Oct 2014 16:43:10 +0200 Subject: [PATCH 2/2] Remove the automatic file size reduction In-Reply-To: <1412347390-27757-1-git-send-email-martin@varnish-software.com> References: <1412347390-27757-1-git-send-email-martin@varnish-software.com> Message-ID: <1412347390-27757-2-git-send-email-martin@varnish-software.com> This never worked as intented, as it didn't take the actual size into account (sparse files). Also the calucation didn't take overflow into account leading to strange behavior. Replace it with an ARGV_ERR if the attempted file size exceeds the size of the filesystem. This is still not perfect as file system overhead can't be anticipated, making it possible to end up with IO errors on writes. posix_fallocate() could be run after the ftruncate call to make sure we can allocate all the blocks, but posix_fallocate() isn't available in FreeBSD (I believe). Fixes: #1343 --- bin/varnishd/storage/stevedore_utils.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index c865f6e..7ff5202 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -196,8 +196,9 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) * use its existing size. */ l = st.st_size; + } else if (size == NULL || *size == '\0') { + ARGV_ERR("(%s) missing size argument\n", ctx); } else { - AN(size); q = VNUM_2bytes(size, &l, fssize); if (q != NULL) @@ -206,6 +207,10 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) if (l < 1024*1024) ARGV_ERR("(%s) size \"%s\": too small, " "did you forget to specify M or G?\n", ctx, size); + + if (l > fssize) + ARGV_ERR("(%s) size \"%s\": larger than file system", + ctx, size); } /* @@ -223,11 +228,6 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) if (i) fprintf(stderr, "WARNING: (%s) file size reduced" " to %ju due to system \"off_t\" limitations\n", ctx, l); - else if (l - st.st_size > fssize) { - l = fssize * 80 / 100; - fprintf(stderr, "WARNING: (%s) file size reduced" - " to %ju (80%% of available disk space)\n", ctx, l); - } if (sizeof(void *) == 4 && l > INT32_MAX) { /*lint !e506 !e774 !e845 */ fprintf(stderr, -- 2.1.0 From martin at varnish-software.com Fri Oct 3 14:43:09 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 3 Oct 2014 16:43:09 +0200 Subject: [PATCH 1/2] Change stv_fsspace to be stv_fssize Message-ID: <1412347390-27757-1-git-send-email-martin@varnish-software.com> Doing e.g. -sfile,varnish.bin,90% sets the size of varnish.bin to 90% of the available file system space where the file resides. This doesn't take into account the size of varnish.bin and creates problems on subsequent invocations. The first time it becomes 90% of the free space. Next startup it becomes 90% of the 10% free space, which would be 9% of the total space causing the free space to be 91%. Next startup it becomes 90% of 91% etc. With this change we look at the total size of the filesystem, which can be a useful and well defined metric when using a dedicated data partion for the storage. --- bin/varnishd/storage/stevedore_utils.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index a86d57e..c865f6e 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -133,25 +133,25 @@ STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx) } /*-------------------------------------------------------------------- - * Figure out how much space is in a filesystem + * Figure out the size of the filesystem */ static uintmax_t -stv_fsspace(int fd, unsigned *bs) +stv_fssize(int fd, unsigned *bs) { - uintmax_t bsize, bavail; + uintmax_t bsize, blocks; #if defined(HAVE_SYS_STATVFS_H) struct statvfs fsst; AZ(fstatvfs(fd, &fsst)); bsize = fsst.f_frsize; - bavail = fsst.f_bavail; + blocks = fsst.f_blocks; #elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) struct statfs fsst; AZ(fstatfs(sc->fd, &fsst)); bsize = fsst.f_bsize; - bavail = fsst.f_bavail; + blocks = fsst.f_blocks; #else #error no struct statfs / struct statvfs #endif @@ -160,7 +160,7 @@ stv_fsspace(int fd, unsigned *bs) if (*bs < bsize) *bs = bsize; XXXAZ(*bs % bsize); - return (bsize * bavail); + return (bsize * blocks); } @@ -187,7 +187,7 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) xxxassert(S_ISREG(st.st_mode)); bs = *granularity; - fssize = stv_fsspace(fd, &bs); + fssize = stv_fssize(fd, &bs); XXXAZ(bs % *granularity); if ((size == NULL || *size == '\0') && st.st_size != 0) { -- 2.1.0 From martin at varnish-software.com Mon Oct 6 15:50:05 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:50:05 +0200 Subject: [PATCH 2/8] Change stv_fsspace to be stv_fssize In-Reply-To: <1412610611-15827-1-git-send-email-martin@varnish-software.com> References: <1412610611-15827-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610611-15827-2-git-send-email-martin@varnish-software.com> Calculate the total size of the file system instead of how much space is available --- bin/varnishd/storage/stevedore_utils.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index e30cbcc..ccb0614 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -133,25 +133,25 @@ STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx) } /*-------------------------------------------------------------------- - * Figure out how much space is in a filesystem + * Figure out the block size and total size of the filesystem */ static uintmax_t -stv_fsspace(int fd, unsigned *bs) +stv_fssize(int fd, unsigned *bs) { - uintmax_t bsize, bavail; + uintmax_t bsize, blocks; #if defined(HAVE_SYS_STATVFS_H) struct statvfs fsst; AZ(fstatvfs(fd, &fsst)); bsize = fsst.f_frsize; - bavail = fsst.f_bavail; + blocks = fsst.f_blocks; #elif defined(HAVE_SYS_MOUNT_H) || defined(HAVE_SYS_VFS_H) struct statfs fsst; AZ(fstatfs(sc->fd, &fsst)); bsize = fsst.f_bsize; - bavail = fsst.f_bavail; + blocks = fsst.f_blocks; #else #error no struct statfs / struct statvfs #endif @@ -160,7 +160,7 @@ stv_fsspace(int fd, unsigned *bs) if (*bs < bsize) *bs = bsize; XXXAZ(*bs % bsize); - return (bsize * bavail); + return (bsize * blocks); } @@ -187,7 +187,7 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) xxxassert(S_ISREG(st.st_mode)); bs = *granularity; - fssize = stv_fsspace(fd, &bs); + fssize = stv_fssize(fd, &bs); XXXAZ(bs % *granularity); if ((size == NULL || *size == '\0') && st.st_size != 0) { -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:50:04 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:50:04 +0200 Subject: [PATCH 1/8] Remove the 80% file system free space check Message-ID: <1412610611-15827-1-git-send-email-martin@varnish-software.com> This check never worked as intended for several reasons: - Didn't take the existing file size into account, causing it to succeed on first startup but fail on next - Sparse files were not handled, and can't be handled as we don't know where in the file a hole could be and whether truncation actually would change the number of blocks used at all Fixes: #1343 --- bin/varnishd/storage/stevedore_utils.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index a86d57e..e30cbcc 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -223,11 +223,6 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) if (i) fprintf(stderr, "WARNING: (%s) file size reduced" " to %ju due to system \"off_t\" limitations\n", ctx, l); - else if (l - st.st_size > fssize) { - l = fssize * 80 / 100; - fprintf(stderr, "WARNING: (%s) file size reduced" - " to %ju (80%% of available disk space)\n", ctx, l); - } if (sizeof(void *) == 4 && l > INT32_MAX) { /*lint !e506 !e774 !e845 */ fprintf(stderr, -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:50:06 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:50:06 +0200 Subject: [PATCH 3/8] Bail out with arg error if specifying a size larger than the file system size In-Reply-To: <1412610611-15827-1-git-send-email-martin@varnish-software.com> References: <1412610611-15827-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610611-15827-3-git-send-email-martin@varnish-software.com> This is to spot obvious typos --- bin/varnishd/storage/stevedore_utils.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index ccb0614..d5ed033 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -206,6 +206,10 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) if (l < 1024*1024) ARGV_ERR("(%s) size \"%s\": too small, " "did you forget to specify M or G?\n", ctx, size); + + if (l > fssize) + ARGV_ERR("(%s) size \"%s\": larger than file system\n", + ctx, size); } /* -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:50:07 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:50:07 +0200 Subject: [PATCH 4/8] Don't allow relative sizes for stevedore file size In-Reply-To: <1412610611-15827-1-git-send-email-martin@varnish-software.com> References: <1412610611-15827-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610611-15827-4-git-send-email-martin@varnish-software.com> --- bin/varnishd/storage/stevedore_utils.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index d5ed033..540f911 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -198,7 +198,7 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) l = st.st_size; } else { AN(size); - q = VNUM_2bytes(size, &l, fssize); + q = VNUM_2bytes(size, &l, 0); if (q != NULL) ARGV_ERR("(%s) size \"%s\": %s\n", ctx, size, q); -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:53:12 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:53:12 +0200 Subject: [PATCH 1/4] Add a default size argument to STV_FileSize Message-ID: <1412610795-15955-1-git-send-email-martin@varnish-software.com> --- bin/varnishd/storage/stevedore_utils.c | 29 ++++++++++++++++++--------- bin/varnishd/storage/storage.h | 4 ++-- bin/varnishd/storage/storage_file.c | 11 +++++----- bin/varnishd/storage/storage_persistent_mgt.c | 2 +- 4 files changed, 29 insertions(+), 17 deletions(-) diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index 540f911..6f55b69 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -168,13 +168,21 @@ stv_fssize(int fd, unsigned *bs) * Decide file size. * * If the sizespecification is empty and the file exists with non-zero - * size, use that, otherwise, interpret the specification. + * size + * use that + * else if the sizespecification is non-empty + * interpret the specification + * else if the default specification is non-empty + * interpret the default specification + * else + * bail with ARGV_ERR * * Handle off_t sizes and pointer width limitations. */ uintmax_t -STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) +STV_FileSize(int fd, const char *size, const char *defsize, + unsigned *granularity, const char *ctx) { uintmax_t l, fssize; unsigned bs; @@ -190,27 +198,30 @@ STV_FileSize(int fd, const char *size, unsigned *granularity, const char *ctx) fssize = stv_fssize(fd, &bs); XXXAZ(bs % *granularity); - if ((size == NULL || *size == '\0') && st.st_size != 0) { + l = 0; + if (size != NULL && *size == '\0') + size = NULL; + if (size == NULL && st.st_size != 0) { /* * We have no size specification, but an existing file, * use its existing size. */ l = st.st_size; - } else { - AN(size); - q = VNUM_2bytes(size, &l, 0); + } else if (size == NULL) + size = defsize; + if (size != NULL) { + q = VNUM_2bytes(size, &l, 0); if (q != NULL) ARGV_ERR("(%s) size \"%s\": %s\n", ctx, size, q); - if (l < 1024*1024) ARGV_ERR("(%s) size \"%s\": too small, " "did you forget to specify M or G?\n", ctx, size); - if (l > fssize) ARGV_ERR("(%s) size \"%s\": larger than file system\n", ctx, size); - } + } else if (l == 0) + ARGV_ERR("(%s) no size specified\n", ctx); /* * This trickery wouldn't be necessary if X/Open would diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h index 0d162ca..06264ae 100644 --- a/bin/varnishd/storage/storage.h +++ b/bin/varnishd/storage/storage.h @@ -194,8 +194,8 @@ extern struct stevedore *stv_transient; /*--------------------------------------------------------------------*/ int STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx); -uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity, - const char *ctx); +uintmax_t STV_FileSize(int fd, const char *size, const char *defsize, + unsigned *granularity, const char *ctx); struct object *STV_MkObject(struct stevedore *, struct objcore *, void *ptr); struct lru *LRU_Alloc(void); diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c index 863ad25..db634f2 100644 --- a/bin/varnishd/storage/storage_file.c +++ b/bin/varnishd/storage/storage_file.c @@ -98,19 +98,20 @@ struct smf_sc { /*--------------------------------------------------------------------*/ +static const char default_size[] = "100M"; +static const char default_filename[] = "."; + static void smf_initfile(struct smf_sc *sc, const char *size) { - sc->filesize = STV_FileSize(sc->fd, size, &sc->pagesize, "-sfile"); + sc->filesize = STV_FileSize(sc->fd, size, default_size, &sc->pagesize, + "-sfile"); AZ(ftruncate(sc->fd, (off_t)sc->filesize)); /* XXX: force block allocation here or in open ? */ } -static const char default_size[] = "100M"; -static const char default_filename[] = "."; - static void smf_init(struct stevedore *parent, int ac, char * const *av) { @@ -122,7 +123,7 @@ smf_init(struct stevedore *parent, int ac, char * const *av) AZ(av[ac]); fn = default_filename; - size = default_size; + size = NULL; page_size = getpagesize(); if (ac > 3) diff --git a/bin/varnishd/storage/storage_persistent_mgt.c b/bin/varnishd/storage/storage_persistent_mgt.c index 33f41e0..e3a41b0 100644 --- a/bin/varnishd/storage/storage_persistent_mgt.c +++ b/bin/varnishd/storage/storage_persistent_mgt.c @@ -168,7 +168,7 @@ smp_mgt_init(struct stevedore *parent, int ac, char * const *av) sc->align = sizeof(void*) * 2; sc->granularity = getpagesize(); - sc->mediasize = STV_FileSize(sc->fd, av[1], &sc->granularity, + sc->mediasize = STV_FileSize(sc->fd, av[1], NULL, &sc->granularity, "-spersistent"); AZ(ftruncate(sc->fd, sc->mediasize)); -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:53:13 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:53:13 +0200 Subject: [PATCH 2/4] Add a VFIL_fallocate function In-Reply-To: <1412610795-15955-1-git-send-email-martin@varnish-software.com> References: <1412610795-15955-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610795-15955-2-git-send-email-martin@varnish-software.com> This function uses fallocate to preallocate file system blocks, returning error if the available space is exhausted. Does nothing on platforms without fallocate --- configure.ac | 1 + include/vfil.h | 1 + lib/libvarnish/vfil.c | 16 ++++++++++++++++ 3 files changed, 18 insertions(+) diff --git a/configure.ac b/configure.ac index d4f5c5b..2facc53 100644 --- a/configure.ac +++ b/configure.ac @@ -205,6 +205,7 @@ AC_CHECK_FUNCS([getdtablesize]) AC_CHECK_FUNCS([timegm]) AC_CHECK_FUNCS([nanosleep]) AC_CHECK_FUNCS([setppriv]) +AC_CHECK_FUNCS([fallocate]) save_LIBS="${LIBS}" LIBS="${PTHREAD_LIBS}" diff --git a/include/vfil.h b/include/vfil.h index 533dfd8..8c4dcd6 100644 --- a/include/vfil.h +++ b/include/vfil.h @@ -34,3 +34,4 @@ int VFIL_tmpfile(char *); char *VFIL_readfile(const char *pfx, const char *fn, ssize_t *sz); char *VFIL_readfd(int fd, ssize_t *sz); int VFIL_nonblocking(int fd); +int VFIL_fallocate(int fd, off_t size); diff --git a/lib/libvarnish/vfil.c b/lib/libvarnish/vfil.c index 920a06f..cf734ea 100644 --- a/lib/libvarnish/vfil.c +++ b/lib/libvarnish/vfil.c @@ -136,3 +136,19 @@ VFIL_nonblocking(int fd) assert(i != -1); return (i); } + +int +VFIL_fallocate(int fd, off_t size) +{ +#ifdef HAVE_FALLOCATE + if (fallocate(fd, 0, 0, size) && errno == ENOSPC) + return (-1); + /* Ignore other errors */ + return (0); +#else + /* Do nothing */ + (void)fd; + (void)size; + return (0); +#endif +} -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:53:14 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:53:14 +0200 Subject: [PATCH 3/4] Use VFIL_fallocate in -sfile In-Reply-To: <1412610795-15955-1-git-send-email-martin@varnish-software.com> References: <1412610795-15955-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610795-15955-3-git-send-email-martin@varnish-software.com> --- bin/varnishd/storage/storage_file.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c index db634f2..0102302 100644 --- a/bin/varnishd/storage/storage_file.c +++ b/bin/varnishd/storage/storage_file.c @@ -40,6 +40,7 @@ #include "storage/storage.h" #include "vnum.h" +#include "vfil.h" #ifndef MAP_NOCORE #define MAP_NOCORE 0 /* XXX Linux */ @@ -108,6 +109,8 @@ smf_initfile(struct smf_sc *sc, const char *size) "-sfile"); AZ(ftruncate(sc->fd, (off_t)sc->filesize)); + if (VFIL_fallocate(sc->fd, (off_t)sc->filesize)) + ARGV_ERR("(-sfile) %s\n", strerror(errno));; /* XXX: force block allocation here or in open ? */ } -- 2.1.1 From martin at varnish-software.com Mon Oct 6 15:53:15 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 6 Oct 2014 17:53:15 +0200 Subject: [PATCH 4/4] Use VFIL_fallocate for the shmlog file In-Reply-To: <1412610795-15955-1-git-send-email-martin@varnish-software.com> References: <1412610795-15955-1-git-send-email-martin@varnish-software.com> Message-ID: <1412610795-15955-4-git-send-email-martin@varnish-software.com> --- bin/varnishd/mgt/mgt_shmem.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index dc1fa5c..5db385a 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -47,6 +47,7 @@ #include "flopen.h" #include "vapi/vsm_int.h" #include "vmb.h" +#include "vfil.h" #ifndef MAP_HASSEMAPHORE #define MAP_HASSEMAPHORE 0 /* XXX Linux */ @@ -173,6 +174,11 @@ vsm_zerofile(const char *fn, ssize_t size) u += i; } AZ(ftruncate(fd, (off_t)size)); + if (VFIL_fallocate(fd, (off_t)size)) { + fprintf(stderr, "Could not preallocate %s: %s\n", + fn, strerror(errno)); + return (-1); + } return (fd); } -- 2.1.1 From martin at varnish-software.com Tue Oct 7 14:31:52 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 7 Oct 2014 16:31:52 +0200 Subject: New patch set for the fallocate and related changes Message-ID: Sending as a tar file this time, to not spam. Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: patches.tar.gz Type: application/x-gzip Size: 6893 bytes Desc: not available URL: From slink at schokola.de Thu Oct 9 15:23:40 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 09 Oct 2014 17:23:40 +0200 Subject: [PATCH] make bereq.uncacheable read-only outside backend_fetch Message-ID: <5436A87C.6020908@schokola.de> Writing to bereq.uncacheable in backend_response and backend_error would only make sense in the context of a retry. In most real world cases, writing to be*resp*.uncacheable would be intended in backend_response and backend_error. --- lib/libvcc/generate.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index e82d610..0791671 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -360,7 +360,7 @@ sp_variables = [ ('bereq.uncacheable', 'BOOL', ( 'backend', ), - ( 'backend', ), """ + ( 'backend_fetch', ), """ Indicates whether the object requested from the backend is going to be uncacheable - either because the request was an explicit pass from the client side or a hit on an uncacheable -- 2.1.1 From fgsch at lodoss.net Tue Oct 21 22:26:32 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 21 Oct 2014 23:26:32 +0100 Subject: [PATCH] merge swr into ttl handling Message-ID: I think this makes more sense and allow us to log the grace as part of the TTL record, my goal here really. f.- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: merge-swr.patch Type: text/x-patch Size: 3027 bytes Desc: not available URL: From daghf at varnish-software.com Wed Oct 22 09:26:11 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Wed, 22 Oct 2014 11:26:11 +0200 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: <59017.1401095089@critter.freebsd.dk> References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> <59017.1401095089@critter.freebsd.dk> Message-ID: Hi I'm revisiting this old thread. To be a bit more concrete and less hand-wavy this time around, here's a VMOD: https://github.com/daghf/libvmod-saintmode Recent changes in Varnish have made things quite a bit simpler, in particular having bo->digest available (6b388a21), as well as having bo->director_resp (9ba9a8bb). That said, the issue that was brought up in my original email still stands, I still do need the digest to be available in vdir->healthy(). The github repo above includes a patch[1] that adds a busyobj parameter to vdir->healthy(). This permits the saintmode director access to the digest and thus lets us consider if that object is present on the saintmode trouble list. In the case that busyobj is not available in the calling context (i.e. on the client side), a NULL will be passed. In that case, the saintmode VMOD will merely fall back to returning the health status of the decorated backend. This is much like how it worked in 3.0, where the digest was zeroed and thus never found on the saintmode trouble list prior to executing vcl_hash. [1]: https://raw.githubusercontent.com/daghf/libvmod-saintmode/master/0001-Add-a-struct-busyobj-param-to-dir-healthy.patch (never mind that this patch says PATCH 1/3, the other two needed were varnish commits 9ec96da and 03a5a8d) Regards, Dag On Mon, May 26, 2014 at 11:04 AM, Poul-Henning Kamp wrote: > In message > , Dag Haavi Finstad writes: > >>The way I envisioned it is purely on the fetch side. So 'saintmode' in >>this case will be a tool that helps us avoid picking a backend for >>certain URLs. > > So how exactly does that work ? > > Once you havn't picked the backend, what backend do you then use ? > > Does the saint director pick another one ? (by what algorithm ?) > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! From daghf at varnish-software.com Wed Oct 22 16:28:15 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Wed, 22 Oct 2014 18:28:15 +0200 Subject: Move obj.hits to the objcore? Message-ID: Hi At the Varnish Summit in Paris last week, we got some feedback from a few early adopters of Varnish 4.0, who were very surprised with the new behavior of obj.hits. To understand obj.hits in its current form, we need to understand the concept of an objhead, which is fairly esoteric knowledge, and something most users aren't familiar with. Also, I find that obj.hits currently simply is not very useful. I have attached a patch that moves the obj.hits counter to struct objcore. This should give the old 3.0 functionality, where obj.hits is separate for each object. The cost of this turns out to be (on my computer) and extra 8 bytes per OC (and 8 less per OH), which IMO is manageable. Regards, Dag -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Move-obj.hits-to-the-objcore.patch Type: text/x-patch Size: 4213 bytes Desc: not available URL: From fgsch at lodoss.net Wed Oct 22 16:34:49 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 22 Oct 2014 17:34:49 +0100 Subject: Move obj.hits to the objcore? In-Reply-To: References: Message-ID: +1 Current behaviour is rather broken. On 22 Oct 2014 17:28, "Dag Haavi Finstad" wrote: > Hi > > At the Varnish Summit in Paris last week, we got some feedback from a > few early adopters of Varnish 4.0, who were very surprised with the > new behavior of obj.hits. > > To understand obj.hits in its current form, we need to understand the > concept of an objhead, which is fairly esoteric knowledge, and > something most users aren't familiar with. Also, I find that obj.hits > currently simply is not very useful. > > I have attached a patch that moves the obj.hits counter to struct > objcore. This should give the old 3.0 functionality, where obj.hits is > separate for each object. The cost of this turns out to be (on my > computer) and extra 8 bytes per OC (and 8 less per OH), which IMO is > manageable. > > Regards, > Dag > > -- > Dag Haavi Finstad > Software Developer | Varnish Software > Mobile: +47 476 64 134 > We Make Websites Fly! > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Thu Oct 23 14:21:45 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Thu, 23 Oct 2014 16:21:45 +0200 Subject: [PATCH] Expand SLT_Storage with allocation size. Message-ID: <20141023142141.GA9211@immer.varnish-software.com> While understanding how fetch_chunksize on chunked backend responses affect cache size, it was apparent that we don't log anywhere how much is actually allocated from a stevedore. This commit introduces two extra fields, header bytes allocated and body bytes allocated, on the Storage VSL line. --- bin/varnishd/cache/cache_obj.c | 1 + bin/varnishd/storage/stevedore.c | 4 ++-- include/tbl/vsl_tags.h | 6 ++++-- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c index 73eff9c..53361b7 100644 --- a/bin/varnishd/cache/cache_obj.c +++ b/bin/varnishd/cache/cache_obj.c @@ -239,6 +239,7 @@ objallocwithnuke(struct stevedore *stv, struct worker *wrk, size_t size) break; } CHECK_OBJ_ORNULL(st, STORAGE_MAGIC); + VSLb(wrk->vsl, SLT_Storage, "%s %s 0 %zu", stv->name, stv->ident, size); return (st); } diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c index 30d8422..53725a4 100644 --- a/bin/varnishd/storage/stevedore.c +++ b/bin/varnishd/storage/stevedore.c @@ -282,8 +282,8 @@ STV_NewObject(struct objcore *oc, struct worker *wrk, return (0); wrk->stats->n_object++; - VSLb(wrk->vsl, SLT_Storage, "%s %s", - oc->stobj->stevedore->name, oc->stobj->stevedore->ident); + VSLb(wrk->vsl, SLT_Storage, "%s %s %i 0", + oc->stobj->stevedore->name, oc->stobj->stevedore->ident, ltot); return (1); } diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h index b483f2c..8ae7193 100644 --- a/include/tbl/vsl_tags.h +++ b/include/tbl/vsl_tags.h @@ -419,8 +419,10 @@ SLTM(VSL, 0, "VSL API warnings and error message", SLTM(Storage, 0, "Where object is stored", "Type and name of the storage backend the object is stored in.\n\n" "The format is::\n\n" - "\t%s %s\n" - "\t| |\n" + "\t%s %s %d %d\n" + "\t| | | |\n" + "\t| | | +- Body bytes allocated\n" + "\t| | +- Header bytes allocated\n" "\t| +- Name of storage backend\n" "\t+---- Type (\"malloc\", \"file\", \"persistent\" etc.)\n" "\n" -- 2.1.1 -- Lasse Karstensen Varnish Software AS From daghf at varnish-software.com Fri Oct 24 11:42:35 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Fri, 24 Oct 2014 13:42:35 +0200 Subject: [PATCH] Prefer exact matches in varnishadm backend.set_health. Message-ID: Hi This patch is a fix for bug #1349 backend_find now does an extra traversal of the backend list to see if there is an exact match for the provided name argument. If ip and/or port is present, it will also check that these don't disagree with the configured ip/port of the backend. If no exact match can be found, we fall back to partial matching. Example: Consider a VCL with backends named foobar00, foobar0, foobar: backend.list foobar -> foobar (old behavior -> foobar00, foobar0, foobar) backend.list foobar0 -> foobar0 (old behavior -> foobar00, foobar0) backend.list foo -> foobar00, foobar0, foobar (old behavior -> same) Related to this, we should consider a different syntax for the ip/port specification, as the current one uses ':' as the separator, which breaks for IPv6. We could go with the square bracket syntax ([::1]:8080) or just use a different separator (::1,8080). Ideas? -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Prefer-exact-matches-in-varnishadm-backend.set_healt.patch Type: text/x-patch Size: 2924 bytes Desc: not available URL: From fgsch at lodoss.net Mon Oct 27 21:22:38 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 27 Oct 2014 21:22:38 +0000 Subject: [PATCH] Split gunzip counters updated Message-ID: Hi, This is a revision and update of the original patch as found in (1). As discussed on irc a while back instead of creating a new function to handle the gunzip testing, this iteration moves the counters into the calling code. I've also expanded the test slightly. And since I wasn't quite happy with this approach I'm also attaching an alternative patch (v2) based on the original one. Comments? OKs? Preferred bike colour? f.- 1. https://www.varnish-cache.org/patchwork/patch/186/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Split-gunzip-counters.patch Type: text/x-patch Size: 3766 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Split-gunzip-counters-v2.patch Type: text/x-patch Size: 4204 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue Oct 28 07:18:50 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 28 Oct 2014 07:18:50 +0000 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> <59017.1401095089@critter.freebsd.dk> Message-ID: <57435.1414480730@critter.freebsd.dk> -------- In message , Dag Haavi Finstad writes: >Recent changes in Varnish have made things quite a bit simpler, in >particular having bo->digest available (6b388a21), as well as having >bo->director_resp (9ba9a8bb). That said, the issue that was brought up >in my original email still stands, I still do need the digest to be >available in vdir->healthy(). I was offline, so I couldn't look at your patch, but I've added the const busyobj as param to vdi_healthy_f. You'll see this in -trunk next time I have a chance to push. (Approximately when you receive this email...) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From daghf at varnish-software.com Tue Oct 28 12:13:23 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Tue, 28 Oct 2014 13:13:23 +0100 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: <57435.1414480730@critter.freebsd.dk> References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> <59017.1401095089@critter.freebsd.dk> <57435.1414480730@critter.freebsd.dk> Message-ID: On Tue, Oct 28, 2014 at 8:18 AM, Poul-Henning Kamp wrote: > > I was offline, so I couldn't look at your patch, but I've added the > const busyobj as param to vdi_healthy_f. You'll see this in -trunk > next time I have a chance to push. (Approximately when you receive > this email...) Thanks a lot, that looks to be exactly what I needed. :-) -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly!