From dridi.boukelmoune at gmail.com Fri Apr 1 18:08:04 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 01 Apr 2016 20:08:04 +0200 Subject: [master] 6b66574 Missing bracket Message-ID: commit 6b66574dcc369eefba200bc53a9e7e8b8c7cfede Author: Dridi Boukelmoune Date: Fri Apr 1 20:07:05 2016 +0200 Missing bracket diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index cd63f49..5f60d73 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -39,7 +39,7 @@ WS_Assert(const struct ws *ws) { CHECK_OBJ_NOTNULL(ws, WS_MAGIC); - DSL(DBG_WORKSPACE, 0, "WS(%p = (%s, %p %u %u %u)", + DSL(DBG_WORKSPACE, 0, "WS(%p) = (%s, %p %u %u %u)", ws, ws->id, ws->s, pdiff(ws->s, ws->f), ws->r == NULL ? 0 : pdiff(ws->f, ws->r), pdiff(ws->s, ws->e)); From varnish-commit at varnish-cache.org Sat Apr 2 04:38:54 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 2 Apr 2016 10:38:54 +0600 Subject: Hello! Message-ID: <56FF4E82.7018.D95622@varnish-commit.varnish-cache.org> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-commit at varnish-cache.org Sat Apr 2 08:33:12 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 2 Apr 2016 11:33:12 +0300 Subject: beneficial offer Message-ID: <003801d18cbe$047f9c83$a52b51a9$@varnish-cache.org> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-commit at varnish-cache.org Sat Apr 2 19:56:14 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 3 Apr 2016 01:56:14 +0600 Subject: Wanted regional manageres Message-ID: <003101d18d1d$05c577fe$95fafaba$@varnish-cache.org> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-commit at varnish-cache.org Sun Apr 3 01:36:00 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 3 Apr 2016 02:36:00 +0100 Subject: The beautiful work in crisis! Message-ID: <28E9042792C5CAB1709DBE0B5C5328E9@6ST6797RF37> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-commit at varnish-cache.org Sun Apr 3 06:29:04 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 3 Apr 2016 12:29:04 +0600 Subject: Wanted regional manageres Message-ID: Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Sun Apr 3 11:24:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sun, 03 Apr 2016 13:24:04 +0200 Subject: [4.1] d01f76a Revive the backend_conn counter Message-ID: commit d01f76a45e9674480955f93b711c409b89b91af5 Author: Dag Haavi Finstad Date: Tue Mar 8 19:27:36 2016 +0100 Revive the backend_conn counter Discussed with Federico. Fixes: #1725 diff --git a/bin/varnishd/cache/cache_backend_tcp.c b/bin/varnishd/cache/cache_backend_tcp.c index 13a026c..9f28d22 100644 --- a/bin/varnishd/cache/cache_backend_tcp.c +++ b/bin/varnishd/cache/cache_backend_tcp.c @@ -382,7 +382,9 @@ VBT_Get(struct tcp_pool *tp, double tmo, const struct backend *be, Lck_Lock(&tp->mtx); tp->n_used--; // Nope, didn't work after all. Lck_Unlock(&tp->mtx); - } + } else + VSC_C_main->backend_conn++; + return (vbc); } From fgsch at lodoss.net Sun Apr 3 11:27:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sun, 03 Apr 2016 13:27:04 +0200 Subject: [4.1] b21bd69 Sync changelog Message-ID: commit b21bd69c467b53399787398579bc2a740d8235d9 Author: Federico G. Schwindt Date: Sun Apr 3 12:25:39 2016 +0100 Sync changelog diff --git a/doc/changes.rst b/doc/changes.rst index 4258e26..1a2bf7b 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -10,10 +10,12 @@ Bugs fixed * 1879_ - Correct handling of duplicate headers on IMS header merge * 1878_ - Fix a ESI+gzip corner case which had escaped notice until now * 1871_ - Missing error handling code in V1F_Setup_Fetch +* 1725_ - Revive the backend_conn counter .. _1879: https://github.com/varnishcache/varnish-cache/issues/1879 .. _1878: https://github.com/varnishcache/varnish-cache/issues/1878 .. _1871: https://github.com/varnishcache/varnish-cache/issues/1871 +.. _1725: https://github.com/varnishcache/varnish-cache/issues/1725 ================================ Varnish Cache 4.1.2 (2016-03-04) From varnish-commit at varnish-cache.org Mon Apr 4 01:17:55 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 3 Apr 2016 17:17:55 -0800 Subject: Working with partial occupancy Message-ID: <4DF9462FDBF2977089EC856E15E007FE@WMHLL2XIR> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at FreeBSD.org Mon Apr 4 08:52:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 04 Apr 2016 10:52:04 +0200 Subject: [master] 0ff9ab8 Add a short and to the point developer guide. Message-ID: commit 0ff9ab8dee9baae86d4bd55607928b6fc8b25315 Author: Poul-Henning Kamp Date: Mon Apr 4 08:51:24 2016 +0000 Add a short and to the point developer guide. Most of it is rescued from the old Trac wiki. diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index e47a60d..13e2bc6 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -109,6 +109,7 @@ EXTRA_DIST = \ phk \ tutorial \ users-guide \ + dev-guide \ whats-new dist-hook: diff --git a/doc/sphinx/dev-guide/index.rst b/doc/sphinx/dev-guide/index.rst new file mode 100644 index 0000000..453f201 --- /dev/null +++ b/doc/sphinx/dev-guide/index.rst @@ -0,0 +1,84 @@ +.. _dev-guide-index: + +The Varnish Developers Guide +============================ + +This is the deliberately short and to the point list of things +Varnish Developers should know. + +Behaviour +--------- + +* Be sensible. + +* If in doubt, think. + +* If still in doubt, ask. + +* Admit your mistakes, it's faster that way. + +* Thou SHALL not paint `bikesheds. `_ + +* We will toss you out of the project rather than add another rule. + +Technical stuff +---------------- + +* Our coding style guideline is FreeBSD's + `style(9) `_ + +* See autoconf.des script for developer options to the toolchain. + +* We always -Werror, there are no harmless warnings, only source code + that does not express intent well enough. + +* We prefer the source code, rather than the comments explain what is + going on, that way tools like FlexeLint and Coverity also gets a chance. + +* Our reference platforms are Ubuntu and FreeBSD. + +* Asserts have negative cost, they save developer time next time around. + +* Our license is BSD 2-clause or looser, no GPL or LGPL. + +* We havn't had a major security issue in 10 years, and we're not going + to start having them now. + +Bugs, issues, feature requests & VIPs +------------------------------------- + +Bugs, issues and feature requests start out as github issues. + +We do a "bug-wash" every monday at 13:00 EU time, where new and otherwise +worthy of discussion issues are pow-wowed. + +Tickets we cannot do anything about are closed. + +If feature-requests make sense, they get moved to a wiki/VIP page until +somebody implements them, and the issue is closed. + +Varnishtest cases for bugs is the norm, not the exception. + +Architectural stuff +------------------- + +These rules are imported from the X11 project: + +* It is as important to decide what a system is not as to decide what it is. + +* Do not serve all the world's needs; rather, make the system extensible so + that additional needs can be met in an upwardly compatible fashion. + +* The only thing worse than generalizing from one example is generalizing + from no examples at all. + +* If a problem is not completely understood, it is probably best to provide + no solution at all. + +* If you can get 90 percent of the desired effect for 10 percent of the work, + use the simpler solution. + +* Isolate complexity as much as possible. + +* Provide mechanism, rather than policy. + diff --git a/doc/sphinx/index.rst b/doc/sphinx/index.rst index 5e22e66..cc94f1c 100644 --- a/doc/sphinx/index.rst +++ b/doc/sphinx/index.rst @@ -70,6 +70,7 @@ Longer listings like example command output and VCL look like this:: users-guide/index.rst reference/index.rst whats-new/index.rst + dev-guide/index.rst phk/index.rst glossary/index.rst From varnish-commit at varnish-cache.org Mon Apr 4 15:33:46 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 4 Apr 2016 19:33:46 +0400 Subject: The part-time employment Message-ID: <57028EBE.8807.A1031F@varnish-commit.varnish-cache.org> Hello! We are looking for employees working remotely. My name is Lucas, am the personnel manager of a large International company. Most of the work you can do from home, that is, at a distance. Salary is $2000-$5000. If you are interested in this offer, please visit Our Web Page Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at FreeBSD.org Mon Apr 4 10:47:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 04 Apr 2016 12:47:04 +0200 Subject: [master] d411a89 Slightly clearer text. Message-ID: commit d411a89bb807d1d16b55024a2a7ef2513799d380 Author: Poul-Henning Kamp Date: Mon Apr 4 10:46:07 2016 +0000 Slightly clearer text. diff --git a/doc/sphinx/dev-guide/index.rst b/doc/sphinx/dev-guide/index.rst index 453f201..f4915d2 100644 --- a/doc/sphinx/dev-guide/index.rst +++ b/doc/sphinx/dev-guide/index.rst @@ -55,7 +55,7 @@ worthy of discussion issues are pow-wowed. Tickets we cannot do anything about are closed. If feature-requests make sense, they get moved to a wiki/VIP page until -somebody implements them, and the issue is closed. +somebody implements them. Varnishtest cases for bugs is the norm, not the exception. From phk at FreeBSD.org Mon Apr 4 10:56:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 04 Apr 2016 12:56:04 +0200 Subject: [master] 6d70fb9 Typo Message-ID: commit 6d70fb927b32e86cb9531f6eeeca4b2b5b1a6503 Author: Poul-Henning Kamp Date: Mon Apr 4 10:55:34 2016 +0000 Typo diff --git a/doc/sphinx/dev-guide/index.rst b/doc/sphinx/dev-guide/index.rst index f4915d2..5db5217 100644 --- a/doc/sphinx/dev-guide/index.rst +++ b/doc/sphinx/dev-guide/index.rst @@ -27,7 +27,7 @@ Technical stuff * Our coding style guideline is FreeBSD's `style(9) `_ -* See autoconf.des script for developer options to the toolchain. +* See autogen.des script for developer options to the toolchain. * We always -Werror, there are no harmless warnings, only source code that does not express intent well enough. From dridi.boukelmoune at gmail.com Mon Apr 4 11:40:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 04 Apr 2016 13:40:05 +0200 Subject: [master] a65aa02 Enable optional lcov support Message-ID: commit a65aa027fe3875374f95ec45cf32d166b3163e2b Author: Dridi Boukelmoune Date: Fri Apr 1 09:11:17 2016 +0200 Enable optional lcov support With a minor tweak so that running `make lcov` is enough, rather than the original `make check-code-coverage`: ./autogen.sh ./configure --enable-code-coverage make lcov Closes #1891 diff --git a/.gitignore b/.gitignore index fb9cde6..e11a81b 100644 --- a/.gitignore +++ b/.gitignore @@ -112,3 +112,9 @@ cscope.*out /bin/varnishtest/tests/*.log /bin/varnishtest/tests/*.log-t /bin/varnishtest/test-suite.log + +# Code coverage with lcov/gcov +*.gcno +*.gcda +*.info +/lcov/ diff --git a/Makefile.am b/Makefile.am index b947ceb..98b3091 100644 --- a/Makefile.am +++ b/Makefile.am @@ -25,6 +25,17 @@ distcleancheck_listfiles = \ find . -type f -exec sh -c 'test -f $(srcdir)/$$1 || echo $$1' \ sh '{}' ';' +# lcov support +CODE_COVERAGE_OUTPUT_DIRECTORY = lcov +CODE_COVERAGE_IGNORE_PATTERN = "/usr/*" +CODE_COVERAGE_GENHTML_OPTIONS = --prefix $(abs_top_srcdir) + + at CODE_COVERAGE_RULES@ + +if CODE_COVERAGE_ENABLED +lcov: check-code-coverage +endif + # XXX: This is a hack to ensure we have a built source tree when # running make dist If we had used non-recursive make we could have # solved it better, but we don't, so use this at least for now. diff --git a/autogen.des b/autogen.des index 3cfc1fa..9c81a91 100755 --- a/autogen.des +++ b/autogen.des @@ -1,6 +1,10 @@ #!/bin/sh # # Use this when doing code development +# +# Arguments to this script are passed to the resulting ./configure execution. +# You may pass `--enable-code-coverage` to measure code coverage with lcov for +# example. set -ex diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 9d8c0b5..66e114d 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -4,6 +4,10 @@ VTC_LOG_COMPILER = ./varnishtest -v -i TEST_EXTENSIONS = .vtc TESTS = @VTC_TESTS@ +if CODE_COVERAGE_ENABLED +VTC_LOG_COMPILER += -p cli_timeout=120 +endif + # Make sure we run check-local first check: check-am check-local check-am: check-local diff --git a/configure.ac b/configure.ac index edba56a..edd47c5 100644 --- a/configure.ac +++ b/configure.ac @@ -672,6 +672,12 @@ AC_SUBST(VTC_TESTS) # Make sure this include dir exists AC_CONFIG_COMMANDS([mkdir], [$MKDIR_P doc/sphinx/include]) +# lcov support +AX_CODE_COVERAGE +CPPFLAGS="$CPPFLAGS \$(CODE_COVERAGE_CPPFLAGS)" +CFLAGS="$CFLAGS \$(CODE_COVERAGE_CFLAGS)" +LDFLAGS="$LDFLAGS \$(CODE_COVERAGE_LDFLAGS)" + # Generate output AC_CONFIG_FILES([ Makefile diff --git a/include/vbm_test.c b/include/vbm_test.c index f680597..ba60ffb 100644 --- a/include/vbm_test.c +++ b/include/vbm_test.c @@ -88,5 +88,7 @@ main(void) vbit_destroy(vb); + (void)obits; /* needed when NDEBUG is defined */ + return (0); } diff --git a/m4/ax_code_coverage.m4 b/m4/ax_code_coverage.m4 new file mode 100644 index 0000000..6c985eb --- /dev/null +++ b/m4/ax_code_coverage.m4 @@ -0,0 +1,273 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_code_coverage.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_CODE_COVERAGE() +# +# DESCRIPTION +# +# Defines CODE_COVERAGE_CPPFLAGS, CODE_COVERAGE_CFLAGS, +# CODE_COVERAGE_CXXFLAGS and CODE_COVERAGE_LDFLAGS which should be +# included in the CPPFLAGS, CFLAGS CXXFLAGS and LIBS/LDFLAGS variables of +# every build target (program or library) which should be built with code +# coverage support. Also defines CODE_COVERAGE_RULES which should be +# substituted in your Makefile; and $enable_code_coverage which can be +# used in subsequent configure output. CODE_COVERAGE_ENABLED is defined +# and substituted, and corresponds to the value of the +# --enable-code-coverage option, which defaults to being disabled. +# +# Test also for gcov program and create GCOV variable that could be +# substituted. +# +# Note that all optimisation flags in CFLAGS must be disabled when code +# coverage is enabled. +# +# Usage example: +# +# configure.ac: +# +# AX_CODE_COVERAGE +# +# Makefile.am: +# +# @CODE_COVERAGE_RULES@ +# my_program_LIBS = ... $(CODE_COVERAGE_LDFLAGS) ... +# my_program_CPPFLAGS = ... $(CODE_COVERAGE_CPPFLAGS) ... +# my_program_CFLAGS = ... $(CODE_COVERAGE_CFLAGS) ... +# my_program_CXXFLAGS = ... $(CODE_COVERAGE_CXXFLAGS) ... +# +# This results in a "check-code-coverage" rule being added to any +# Makefile.am which includes "@CODE_COVERAGE_RULES@" (assuming the module +# has been configured with --enable-code-coverage). Running `make +# check-code-coverage` in that directory will run the module's test suite +# (`make check`) and build a code coverage report detailing the code which +# was touched, then print the URI for the report. +# +# This code was derived from Makefile.decl in GLib, originally licenced +# under LGPLv2.1+. +# +# LICENSE +# +# Copyright (c) 2012, 2016 Philip Withnall +# Copyright (c) 2012 Xan Lopez +# Copyright (c) 2012 Christian Persch +# Copyright (c) 2012 Paolo Borelli +# Copyright (c) 2012 Dan Winship +# Copyright (c) 2015 Bastien ROUCARIES +# +# This library is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or (at +# your option) any later version. +# +# This library is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser +# General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program. If not, see . + +#serial 15 + +AC_DEFUN([AX_CODE_COVERAGE],[ + dnl Check for --enable-code-coverage + AC_REQUIRE([AC_PROG_SED]) + + # allow to override gcov location + AC_ARG_WITH([gcov], + [AS_HELP_STRING([--with-gcov[=GCOV]], [use given GCOV for coverage (GCOV=gcov).])], + [_AX_CODE_COVERAGE_GCOV_PROG_WITH=$with_gcov], + [_AX_CODE_COVERAGE_GCOV_PROG_WITH=gcov]) + + AC_MSG_CHECKING([whether to build with code coverage support]) + AC_ARG_ENABLE([code-coverage], + AS_HELP_STRING([--enable-code-coverage], + [Whether to enable code coverage support]),, + enable_code_coverage=no) + + AM_CONDITIONAL([CODE_COVERAGE_ENABLED], [test x$enable_code_coverage = xyes]) + AC_SUBST([CODE_COVERAGE_ENABLED], [$enable_code_coverage]) + AC_MSG_RESULT($enable_code_coverage) + + AS_IF([ test "$enable_code_coverage" = "yes" ], [ + # check for gcov + AC_CHECK_TOOL([GCOV], + [$_AX_CODE_COVERAGE_GCOV_PROG_WITH], + [:]) + AS_IF([test "X$GCOV" = "X:"], + [AC_MSG_ERROR([gcov is needed to do coverage])]) + AC_SUBST([GCOV]) + + dnl Check if gcc is being used + AS_IF([ test "$GCC" = "no" ], [ + AC_MSG_ERROR([not compiling with gcc, which is required for gcov code coverage]) + ]) + + # List of supported lcov versions. + lcov_version_list="1.6 1.7 1.8 1.9 1.10 1.11 1.12" + + AC_CHECK_PROG([LCOV], [lcov], [lcov]) + AC_CHECK_PROG([GENHTML], [genhtml], [genhtml]) + + AS_IF([ test "$LCOV" ], [ + AC_CACHE_CHECK([for lcov version], ax_cv_lcov_version, [ + ax_cv_lcov_version=invalid + lcov_version=`$LCOV -v 2>/dev/null | $SED -e 's/^.* //'` + for lcov_check_version in $lcov_version_list; do + if test "$lcov_version" = "$lcov_check_version"; then + ax_cv_lcov_version="$lcov_check_version (ok)" + fi + done + ]) + ], [ + lcov_msg="To enable code coverage reporting you must have one of the following lcov versions installed: $lcov_version_list" + AC_MSG_ERROR([$lcov_msg]) + ]) + + case $ax_cv_lcov_version in + ""|invalid[)] + lcov_msg="You must have one of the following versions of lcov: $lcov_version_list (found: $lcov_version)." + AC_MSG_ERROR([$lcov_msg]) + LCOV="exit 0;" + ;; + esac + + AS_IF([ test -z "$GENHTML" ], [ + AC_MSG_ERROR([Could not find genhtml from the lcov package]) + ]) + + dnl Build the code coverage flags + CODE_COVERAGE_CPPFLAGS="-DNDEBUG" + CODE_COVERAGE_CFLAGS="-O0 -g -fprofile-arcs -ftest-coverage" + CODE_COVERAGE_CXXFLAGS="-O0 -g -fprofile-arcs -ftest-coverage" + CODE_COVERAGE_LDFLAGS="-lgcov" + + AC_SUBST([CODE_COVERAGE_CPPFLAGS]) + AC_SUBST([CODE_COVERAGE_CFLAGS]) + AC_SUBST([CODE_COVERAGE_CXXFLAGS]) + AC_SUBST([CODE_COVERAGE_LDFLAGS]) + ]) + +[CODE_COVERAGE_RULES=' +# Code coverage +# +# Optional: +# - CODE_COVERAGE_DIRECTORY: Top-level directory for code coverage reporting. +# Multiple directories may be specified, separated by whitespace. +# (Default: $(top_builddir)) +# - CODE_COVERAGE_OUTPUT_FILE: Filename and path for the .info file generated +# by lcov for code coverage. (Default: +# $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage.info) +# - CODE_COVERAGE_OUTPUT_DIRECTORY: Directory for generated code coverage +# reports to be created. (Default: +# $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage) +# - CODE_COVERAGE_BRANCH_COVERAGE: Set to 1 to enforce branch coverage, +# set to 0 to disable it and leave empty to stay with the default. +# (Default: empty) +# - CODE_COVERAGE_LCOV_SHOPTS_DEFAULT: Extra options shared between both lcov +# instances. (Default: based on $CODE_COVERAGE_BRANCH_COVERAGE) +# - CODE_COVERAGE_LCOV_SHOPTS: Extra options to shared between both lcov +# instances. (Default: $CODE_COVERAGE_LCOV_SHOPTS_DEFAULT) +# - CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH: --gcov-tool pathtogcov +# - CODE_COVERAGE_LCOV_OPTIONS_DEFAULT: Extra options to pass to the +# collecting lcov instance. (Default: $CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH) +# - CODE_COVERAGE_LCOV_OPTIONS: Extra options to pass to the collecting lcov +# instance. (Default: $CODE_COVERAGE_LCOV_OPTIONS_DEFAULT) +# - CODE_COVERAGE_LCOV_RMOPTS_DEFAULT: Extra options to pass to the filtering +# lcov instance. (Default: empty) +# - CODE_COVERAGE_LCOV_RMOPTS: Extra options to pass to the filtering lcov +# instance. (Default: $CODE_COVERAGE_LCOV_RMOPTS_DEFAULT) +# - CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT: Extra options to pass to the +# genhtml instance. (Default: based on $CODE_COVERAGE_BRANCH_COVERAGE) +# - CODE_COVERAGE_GENHTML_OPTIONS: Extra options to pass to the genhtml +# instance. (Default: $CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT) +# - CODE_COVERAGE_IGNORE_PATTERN: Extra glob pattern of files to ignore +# +# The generated report will be titled using the $(PACKAGE_NAME) and +# $(PACKAGE_VERSION). In order to add the current git hash to the title, +# use the git-version-gen script, available online. + +# Optional variables +CODE_COVERAGE_DIRECTORY ?= $(top_builddir) +CODE_COVERAGE_OUTPUT_FILE ?= $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage.info +CODE_COVERAGE_OUTPUT_DIRECTORY ?= $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage +CODE_COVERAGE_BRANCH_COVERAGE ?= +CODE_COVERAGE_LCOV_SHOPTS_DEFAULT ?= $(if $(CODE_COVERAGE_BRANCH_COVERAGE),\ +--rc lcov_branch_coverage=$(CODE_COVERAGE_BRANCH_COVERAGE)) +CODE_COVERAGE_LCOV_SHOPTS ?= $(CODE_COVERAGE_LCOV_SHOPTS_DEFAULT) +CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH ?= --gcov-tool "$(GCOV)" +CODE_COVERAGE_LCOV_OPTIONS_DEFAULT ?= $(CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH) +CODE_COVERAGE_LCOV_OPTIONS ?= $(CODE_COVERAGE_LCOV_OPTIONS_DEFAULT) +CODE_COVERAGE_LCOV_RMOPTS_DEFAULT ?= +CODE_COVERAGE_LCOV_RMOPTS ?= $(CODE_COVERAGE_LCOV_RMOPTS_DEFAULT) +CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT ?=\ +$(if $(CODE_COVERAGE_BRANCH_COVERAGE),\ +--rc genhtml_branch_coverage=$(CODE_COVERAGE_BRANCH_COVERAGE)) +CODE_COVERAGE_GENHTML_OPTIONS ?= $(CODE_COVERAGE_GENHTML_OPTIONS_DEFAULTS) +CODE_COVERAGE_IGNORE_PATTERN ?= + +code_coverage_v_lcov_cap = $(code_coverage_v_lcov_cap_$(V)) +code_coverage_v_lcov_cap_ = $(code_coverage_v_lcov_cap_$(AM_DEFAULT_VERBOSITY)) +code_coverage_v_lcov_cap_0 = @echo " LCOV --capture"\ + $(CODE_COVERAGE_OUTPUT_FILE); +code_coverage_v_lcov_ign = $(code_coverage_v_lcov_ign_$(V)) +code_coverage_v_lcov_ign_ = $(code_coverage_v_lcov_ign_$(AM_DEFAULT_VERBOSITY)) +code_coverage_v_lcov_ign_0 = @echo " LCOV --remove /tmp/*"\ + $(CODE_COVERAGE_IGNORE_PATTERN); +code_coverage_v_genhtml = $(code_coverage_v_genhtml_$(V)) +code_coverage_v_genhtml_ = $(code_coverage_v_genhtml_$(AM_DEFAULT_VERBOSITY)) +code_coverage_v_genhtml_0 = @echo " GEN " $(CODE_COVERAGE_OUTPUT_DIRECTORY); +code_coverage_quiet = $(code_coverage_quiet_$(V)) +code_coverage_quiet_ = $(code_coverage_quiet_$(AM_DEFAULT_VERBOSITY)) +code_coverage_quiet_0 = --quiet + +# sanitizes the test-name: replaces with underscores: dashes and dots +code_coverage_sanitize = $(subst -,_,$(subst .,_,$(1))) + +# Use recursive makes in order to ignore errors during check +check-code-coverage: +ifeq ($(CODE_COVERAGE_ENABLED),yes) + -$(A''M_V_at)$(MAKE) $(AM_MAKEFLAGS) -k check + $(A''M_V_at)$(MAKE) $(AM_MAKEFLAGS) code-coverage-capture +else + @echo "Need to reconfigure with --enable-code-coverage" +endif + +# Capture code coverage data +code-coverage-capture: code-coverage-capture-hook +ifeq ($(CODE_COVERAGE_ENABLED),yes) + $(code_coverage_v_lcov_cap)$(LCOV) $(code_coverage_quiet) $(addprefix --directory ,$(CODE_COVERAGE_DIRECTORY)) --capture --output-file "$(CODE_COVERAGE_OUTPUT_FILE).tmp" --test-name "$(call code_coverage_sanitize,$(PACKAGE_NAME)-$(PACKAGE_VERSION))" --no-checksum --compat-libtool $(CODE_COVERAGE_LCOV_SHOPTS) $(CODE_COVERAGE_LCOV_OPTIONS) + $(code_coverage_v_lcov_ign)$(LCOV) $(code_coverage_quiet) $(addprefix --directory ,$(CODE_COVERAGE_DIRECTORY)) --remove "$(CODE_COVERAGE_OUTPUT_FILE).tmp" "/tmp/*" $(CODE_COVERAGE_IGNORE_PATTERN) --output-file "$(CODE_COVERAGE_OUTPUT_FILE)" $(CODE_COVERAGE_LCOV_SHOPTS) $(CODE_COVERAGE_LCOV_RMOPTS) + - at rm -f $(CODE_COVERAGE_OUTPUT_FILE).tmp + $(code_coverage_v_genhtml)LANG=C $(GENHTML) $(code_coverage_quiet) $(addprefix --prefix ,$(CODE_COVERAGE_DIRECTORY)) --output-directory "$(CODE_COVERAGE_OUTPUT_DIRECTORY)" --title "$(PACKAGE_NAME)-$(PACKAGE_VERSION) Code Coverage" --legend --show-details "$(CODE_COVERAGE_OUTPUT_FILE)" $(CODE_COVERAGE_GENHTML_OPTIONS) + @echo "file://$(abs_builddir)/$(CODE_COVERAGE_OUTPUT_DIRECTORY)/index.html" +else + @echo "Need to reconfigure with --enable-code-coverage" +endif + +# Hook rule executed before code-coverage-capture, overridable by the user +code-coverage-capture-hook: + +ifeq ($(CODE_COVERAGE_ENABLED),yes) +clean: code-coverage-clean +code-coverage-clean: + -$(LCOV) --directory $(top_builddir) -z + -rm -rf $(CODE_COVERAGE_OUTPUT_FILE) $(CODE_COVERAGE_OUTPUT_FILE).tmp $(CODE_COVERAGE_OUTPUT_DIRECTORY) + -find . -name "*.gcda" -o -name "*.gcov" -delete +endif + +GITIGNOREFILES ?= +GITIGNOREFILES += $(CODE_COVERAGE_OUTPUT_FILE) $(CODE_COVERAGE_OUTPUT_DIRECTORY) + +A''M_DISTCHECK_CONFIGURE_FLAGS ?= +A''M_DISTCHECK_CONFIGURE_FLAGS += --disable-code-coverage + +.PHONY: check-code-coverage code-coverage-capture code-coverage-capture-hook code-coverage-clean +'] + + AC_SUBST([CODE_COVERAGE_RULES]) + m4_ifdef([_AM_SUBST_NOTMAKE], [_AM_SUBST_NOTMAKE([CODE_COVERAGE_RULES])]) +]) From dridi.boukelmoune at gmail.com Mon Apr 4 11:40:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 04 Apr 2016 13:40:05 +0200 Subject: [master] 25647b4 Retire VCL_EVENT_USE Message-ID: commit 25647b4833e8a3dfa163133ebfe01d92e7299e6f Author: Dridi Boukelmoune Date: Mon Apr 4 11:57:35 2016 +0200 Retire VCL_EVENT_USE It was decided during VDD15Q3 but this change didn't make it before the 4.1.0 release. It is kept in the 4.1 series but documented as deprecated and can now be retired as discussed during VDD15Q4. diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index bfcba8b..26454a2 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -475,11 +475,10 @@ vcl_failsafe_event(VRT_CTX, enum vcl_event_e ev) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); AN(ctx->handling); AN(ctx->vcl); - assert(ev == VCL_EVENT_COLD || ev == VCL_EVENT_DISCARD || - ev == VCL_EVENT_USE); + assert(ev == VCL_EVENT_COLD || ev == VCL_EVENT_DISCARD); if (ctx->vcl->conf->event_vcl(ctx, ev) != 0) - WRONG("A VMOD cannot fail USE, COLD or DISCARD events"); + WRONG("A VMOD cannot fail COLD or DISCARD events"); } static void @@ -784,8 +783,6 @@ static void __match_proto__(cli_func_t) ccf_config_use(struct cli *cli, const char * const *av, void *priv) { struct vcl *vcl; - struct vrt_ctx ctx; - unsigned hand = 0; ASSERT_CLI(); AN(cli); @@ -793,10 +790,6 @@ ccf_config_use(struct cli *cli, const char * const *av, void *priv) vcl = vcl_find(av[2]); AN(vcl); // MGT ensures this assert(vcl->temp == VCL_TEMP_WARM); // MGT ensures this - INIT_OBJ(&ctx, VRT_CTX_MAGIC); - ctx.handling = &hand; - ctx.vcl = vcl; - vcl_failsafe_event(&ctx, VCL_EVENT_USE); Lck_Lock(&vcl_mtx); vcl_active = vcl; Lck_Unlock(&vcl_mtx); diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index d3271b5..208070a 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -426,10 +426,6 @@ In the case where properly releasing resources may take some time, you can opt for an asynchronous worker, either by spawning a thread and tracking it, or by using Varnish's worker pools. -There is also a ``VCL_EVENT_USE`` event. Please note that this event is now -deprecated and may be removed in a future release. A warm VCL should be ready -to use so no additional task should be postponed at use time. - When to lock, and when not to lock ================================== diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 44b8b2b..5efcff1 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -1027,7 +1027,6 @@ struct worker; enum vcl_event_e { VCL_EVENT_LOAD, VCL_EVENT_WARM, - VCL_EVENT_USE, VCL_EVENT_COLD, VCL_EVENT_DISCARD, }; diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index 2161cb7..0ee166d 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -326,8 +326,7 @@ EmitCoordinates(const struct vcc *tl, struct vsb *vsb) * Init/Fini/Event * * We call DISCARD and COLD events in the opposite order of LOAD and - * WARM. The child will panic if a USE event fails, since a WARM event - * leads to a usable state. + * WARM. */ static void @@ -404,24 +403,6 @@ EmitInitFini(const struct vcc *tl) Fc(tl, 0, "}\n"); /* - * USE (deprecated) - */ - Fc(tl, 0, "\nstatic int\n"); - Fc(tl, 0, "VGC_Use(VRT_CTX, enum vcl_event_e ev)\n{\n\n"); - - VTAILQ_FOREACH(p, &tl->inifin, list) { - assert(p->n > 0); - if (VSB_len(p->event)) { - Fc(tl, 0, "\t/* %u */\n", p->n); - Fc(tl, 0, "\tif (%s)\n", VSB_data(p->event)); - Fc(tl, 0, "\t\treturn (1);\n\n"); - } - } - - Fc(tl, 0, "\treturn (0);\n"); - Fc(tl, 0, "}\n"); - - /* * COLD */ Fc(tl, 0, "\nstatic int\n"); @@ -453,8 +434,6 @@ EmitInitFini(const struct vcc *tl) if (has_event) { Fc(tl, 0, "\tif (ev == VCL_EVENT_WARM)\n"); Fc(tl, 0, "\t\treturn(VGC_Warmup(ctx, ev));\n"); - Fc(tl, 0, "\tif (ev == VCL_EVENT_USE)\n"); - Fc(tl, 0, "\t\treturn(VGC_Use(ctx, ev));\n"); Fc(tl, 0, "\tif (ev == VCL_EVENT_COLD)\n"); Fc(tl, 0, "\t\treturn(VGC_Cooldown(ctx, ev));\n"); } From phk at phk.freebsd.dk Mon Apr 4 19:24:09 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 Apr 2016 19:24:09 +0000 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: References: Message-ID: <59015.1459797849@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: Uhm, this seems to only work with GNU-make ? >commit a65aa027fe3875374f95ec45cf32d166b3163e2b >Author: Dridi Boukelmoune >Date: Fri Apr 1 09:11:17 2016 +0200 > > Enable optional lcov support > > With a minor tweak so that running `make lcov` is enough, rather than > the original `make check-code-coverage`: -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Apr 4 20:45:03 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 4 Apr 2016 22:45:03 +0200 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: <59015.1459797849@critter.freebsd.dk> References: <59015.1459797849@critter.freebsd.dk> Message-ID: > Uhm, this seems to only work with GNU-make ? My bad, I assumed OK meant it had been tested. I just installed bmake on my machine, I'll have a look and see what's tying it to gmake. I'm getting tired of GNU extensions creeping everywhere :( From phk at FreeBSD.org Mon Apr 4 20:59:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 04 Apr 2016 22:59:04 +0200 Subject: [master] 9a85dfd Clean up the code to configure parameter min/max/default values. Message-ID: commit 9a85dfdf327f0528dd67e9c4d2c2397a46eff92b Author: Poul-Henning Kamp Date: Mon Apr 4 20:58:08 2016 +0000 Clean up the code to configure parameter min/max/default values. diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index 55c78ae..8401707 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -130,7 +130,7 @@ ban_error(struct ban_proto *bp, const char *fmt, ...) /* Record the error message in the vsb */ VSB_clear(bp->vsb); va_start(ap, fmt); - (void)VSB_vprintf(bp->vsb, fmt, ap); + VSB_vprintf(bp->vsb, fmt, ap); va_end(ap); AZ(VSB_finish(bp->vsb)); bp->err = VSB_data(bp->vsb); diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index 3cd8ddd..d174206 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -133,9 +133,14 @@ void MGT_complain(const char *loud, const char *, ...) __v_printflike(2, 3); /* mgt_param.c */ void MCF_InitParams(struct cli *); void MCF_CollectParams(void); -void MCF_SetDefault(const char *param, const char *def); -void MCF_SetMinimum(const char *param, const char *def); -void MCF_SetMaximum(const char *param, const char *def); +enum mcf_which_e { + MCF_DEFAULT = 32, + MCF_MINIMUM = 33, + MCF_MAXIMUM = 34, +}; +void MCF_ParamConf(enum mcf_which_e, const char *param, const char *, ...) + __v_printflike(3, 4); + void MCF_ParamSet(struct cli *, const char *param, const char *val); void MCF_ParamProtect(struct cli *, const char *arg); void MCF_DumpRstParam(void); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 2dc452a..267fee7 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -425,9 +425,6 @@ make_secret(const char *dirname) /*--------------------------------------------------------------------*/ -static char stackmin[20]; -static char stackdef[20]; - static void init_params(struct cli *cli) { @@ -442,26 +439,24 @@ init_params(struct cli *cli) * Adjust default parameters for 32 bit systems to conserve * VM space. */ - MCF_SetDefault("workspace_client", "24k"); - MCF_SetDefault("workspace_backend", "16k"); - MCF_SetDefault("http_resp_size", "8k"); - MCF_SetDefault("http_req_size", "12k"); - MCF_SetDefault("gzip_buffer", "4k"); + MCF_ParamConf(MCF_DEFAULT, "workspace_client", "24k"); + MCF_ParamConf(MCF_DEFAULT, "workspace_backend", "16k"); + MCF_ParamConf(MCF_DEFAULT, "http_resp_size", "8k"); + MCF_ParamConf(MCF_DEFAULT, "http_req_size", "12k"); + MCF_ParamConf(MCF_DEFAULT, "gzip_buffer", "4k"); } #if !defined(HAVE_ACCEPT_FILTERS) || defined(__linux) - MCF_SetDefault("accept_filter", "off"); + MCF_ParamConf(MCF_DEFAULT, "accept_filter", "off"); #endif low = sysconf(_SC_THREAD_STACK_MIN); - bprintf(stackmin, "%jd", (intmax_t)low); - MCF_SetMinimum("thread_pool_stack", stackmin); + MCF_ParamConf(MCF_MINIMUM, "thread_pool_stack", "%jd", (intmax_t)low); def = 48 * 1024; if (def < low) def = low; - bprintf(stackdef, "%jd", (intmax_t)def); - MCF_SetDefault("thread_pool_stack", stackdef); + MCF_ParamConf(MCF_DEFAULT, "thread_pool_stack", "%jd", (intmax_t)def); MCF_InitParams(cli); } diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 71f9751..44c64d7 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -30,6 +30,7 @@ #include "config.h" #include +#include #include #include #include @@ -483,38 +484,35 @@ MCF_CollectParams(void) /*--------------------------------------------------------------------*/ void -MCF_SetDefault(const char *param, const char *new_def) -{ - struct parspec *pp; - - pp = mcf_findpar(param); - AN(pp); - pp->def = new_def; - AN(pp->def); -} - -void -MCF_SetMinimum(const char *param, const char *new_min) -{ - struct parspec *pp; - - AN(new_min); - pp = mcf_findpar(param); - AN(pp); - pp->min = new_min; - AN(pp->min); -} - -void -MCF_SetMaximum(const char *param, const char *new_max) +MCF_ParamConf(enum mcf_which_e which, const char *param, const char *fmt, ...) { struct parspec *pp; + struct vsb *vsb; + va_list ap; - AN(new_max); pp = mcf_findpar(param); AN(pp); - pp->max = new_max; - AN(pp->max); + vsb = VSB_new_auto(); + AN(vsb); + va_start(ap, fmt); + VSB_vprintf(vsb, fmt, ap); + va_end(ap); + AZ(VSB_finish(vsb)); + switch (which) { + case MCF_DEFAULT: + pp->def = strdup(VSB_data(vsb)); + AN(pp->def); + break; + case MCF_MINIMUM: + pp->min = strdup(VSB_data(vsb)); + AN(pp->min); + break; + case MCF_MAXIMUM: + pp->max = strdup(VSB_data(vsb)); + AN(pp->max); + break; + } + VSB_delete(vsb); } /*--------------------------------------------------------------------*/ diff --git a/bin/varnishd/mgt/mgt_param_tcp.c b/bin/varnishd/mgt/mgt_param_tcp.c index ab8f6d4..6d40097 100644 --- a/bin/varnishd/mgt/mgt_param_tcp.c +++ b/bin/varnishd/mgt/mgt_param_tcp.c @@ -79,17 +79,12 @@ tcp_probe(int sock, int nam, const char *param, unsigned def) int i; socklen_t l; unsigned u; - char buf[10]; - const char *p; l = sizeof u; i = getsockopt(sock, IPPROTO_TCP, nam, &u, &l); if (i < 0 || u == 0) u = def; - bprintf(buf, "%u", u); - p = strdup(buf); - AN(p); - MCF_SetDefault(param, p); + MCF_ParamConf(MCF_DEFAULT, param, "%u", u); } static void diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 00ad35c..d58e032 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -329,16 +329,14 @@ tweak_vsl_buffer(struct vsb *vsb, const struct parspec *par, const char *arg) { volatile unsigned *d1; volatile ssize_t dest; - char buf[20]; d1 = par->priv; dest = *d1; if (tweak_generic_bytes(vsb, &dest, arg, par->min, par->max)) return (-1); *d1 = dest; - bprintf(buf, "%u", *d1 - 12); - MCF_SetMaximum("vsl_reclen", strdup(buf)); - MCF_SetMaximum("shm_reclen", strdup(buf)); + MCF_ParamConf(MCF_MAXIMUM, "vsl_reclen", "%u", *d1 - 12); + MCF_ParamConf(MCF_MAXIMUM, "shm_reclen", "%u", *d1 - 12); return (0); } @@ -347,15 +345,13 @@ tweak_vsl_reclen(struct vsb *vsb, const struct parspec *par, const char *arg) { volatile unsigned *d1; volatile ssize_t dest; - char buf[20]; d1 = par->priv; dest = *d1; if (tweak_generic_bytes(vsb, &dest, arg, par->min, par->max)) return (-1); *d1 = dest; - bprintf(buf, "%u", *d1 + 12); - MCF_SetMinimum("vsl_buffer", strdup(buf)); + MCF_ParamConf(MCF_MINIMUM, "vsl_buffer", "%u", *d1 + 12); return (0); } diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c index d14a32f..398e468 100644 --- a/bin/varnishd/mgt/mgt_pool.c +++ b/bin/varnishd/mgt/mgt_pool.c @@ -53,21 +53,15 @@ * limit, so they don't end up crossing. */ -static char min_val[20]; -static char max_val[20]; - static int tweak_thread_pool_min(struct vsb *vsb, const struct parspec *par, const char *arg) { - struct vsb v2; if (tweak_generic_uint(vsb, par->priv, arg, par->min, par->max)) return (-1); - AN(VSB_new(&v2, min_val, sizeof min_val, 0)); - AZ(tweak_generic_uint(&v2, &mgt_param.wthread_min, NULL, NULL, NULL)); - AZ(VSB_finish(&v2)); - MCF_SetMinimum("thread_pool_max", min_val); + MCF_ParamConf(MCF_MINIMUM, "thread_pool_max", + "%u", mgt_param.wthread_min); return (0); } @@ -75,14 +69,11 @@ static int tweak_thread_pool_max(struct vsb *vsb, const struct parspec *par, const char *arg) { - struct vsb v2; if (tweak_generic_uint(vsb, par->priv, arg, par->min, par->max)) return (-1); - AN(VSB_new(&v2, max_val, sizeof max_val, 0)); - AZ(tweak_generic_uint(&v2, &mgt_param.wthread_max, NULL, NULL, NULL)); - AZ(VSB_finish(&v2)); - MCF_SetMaximum("thread_pool_min", max_val); + MCF_ParamConf(MCF_MAXIMUM, "thread_pool_min", + "%u", mgt_param.wthread_max); return (0); } From varnish-commit at varnish-cache.org Tue Apr 5 11:22:57 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: 5 Apr 2016 19:22:57 +0800 Subject: Are you Making Money Yet? Message-ID: Hey, Can I ask you a question? Are you interested in making money from home? Would you like to have a second income? If so, what if I could show you a fool-proof way to turn $100 into $300 ... over and over again, as many times as you wanted. What's more, you could start making money in 15 minutes from now. Would you be interested in learning how? Are you sick and tired of the endless 9 to 5 work day? Fed up with not making the kind of money you want? Sick of driving in rush hour traffic ... getting late to a job where you have to listen to a boss yell orders at you ... all to make a paycheck that barely covers your bills? If you can answer «Yes» to any or all of those ... you just have to check this out. Open the page now, and get the full story of how you can follow 3 simple steps and see 100% profits in an hour or less. It's a free video presentation and it will reveal the «binary code method» that is capable of making above $500 per day ... Just check it out now. Warmest regards, Matilda Raynolds -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Apr 5 09:45:15 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 5 Apr 2016 11:45:15 +0200 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: References: <59015.1459797849@critter.freebsd.dk> Message-ID: On Mon, Apr 4, 2016 at 10:45 PM, Dridi Boukelmoune wrote: >> Uhm, this seems to only work with GNU-make ? > > My bad, I assumed OK meant it had been tested. I just installed bmake > on my machine, I'll have a look and see what's tying it to gmake. I'm > getting tired of GNU extensions creeping everywhere :( I had a look at this, and it doesn't look like something that can be fixed in a reasonable amount of time. The conditional logic can be moved from the generated Makefile to the m4/autoconf parts. That whould be make-friendly while still being gmake-only once enabled. Would that be OK? I'll comment out the lcov support and try that later. Also once enabled it will require gmake because it uses sanitizing $(functions ...) and unlike conditionals I'm not sure I can make it portable. Dridi From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] 64ae2ad Make the maximum number of thread pools a compile time parameter. Message-ID: commit 64ae2adce3f748392732e2f9f7c928928fc7df41 Author: Poul-Henning Kamp Date: Mon Apr 4 21:01:26 2016 +0000 Make the maximum number of thread pools a compile time parameter. diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h index eb20956..7b6a6b8 100644 --- a/bin/varnishd/common/common.h +++ b/bin/varnishd/common/common.h @@ -42,6 +42,10 @@ /**********************************************************************/ +#if !defined(MAX_THREAD_POOLS) +# define MAX_THREAD_POOLS 32 +#endif + /* Name of transient storage */ #define TRANSIENT_STORAGE "Transient" diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 267fee7..36005a3 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -458,6 +458,8 @@ init_params(struct cli *cli) def = low; MCF_ParamConf(MCF_DEFAULT, "thread_pool_stack", "%jd", (intmax_t)def); + MCF_ParamConf(MCF_MAXIMUM, "thread_pools", "%d", MAX_THREAD_POOLS); + MCF_InitParams(cli); } From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] 5a8ec68 SES_RxInit() shouldnt know about the max length of HTTP headers. Message-ID: commit 5a8ec687e7f907cd7766d441fc41458cedd922d2 Author: Poul-Henning Kamp Date: Tue Apr 5 07:33:50 2016 +0000 SES_RxInit() shouldnt know about the max length of HTTP headers. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 44f1c81..3444656 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -963,8 +963,7 @@ enum htc_status_e { HTC_S_IDLE = 3, }; -void SES_RxInit(struct http_conn *htc, struct ws *ws, - unsigned maxbytes, unsigned maxhdr); +void SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes); void SES_RxReInit(struct http_conn *htc); enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, double *t1, double *t2, double ti, double tn); diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index 42fd3f0..982e1e8 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -376,8 +376,7 @@ vca_make_session(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws, - cache_param->http_req_size, cache_param->http_req_hdr_len); + SES_RxInit(req->htc, req->ws, cache_param->http_req_size); SES_SetTransport(wrk, sp, req, wa->acceptlsock->transport); } diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 4208fc5..b96aa9a 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -183,14 +183,12 @@ SES_Get_String_Attr(const struct sess *sp, enum sess_attr a) /*--------------------------------------------------------------------*/ void -SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes, - unsigned maxhdr) +SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes) { htc->magic = HTTP_CONN_MAGIC; htc->ws = ws; htc->maxbytes = maxbytes; - htc->maxhdr = maxhdr; (void)WS_Reserve(htc->ws, htc->maxbytes); htc->rxbuf_b = ws->f; diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index ef24a71..2c92ab6 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -149,8 +149,7 @@ V1F_FetchRespHdr(struct busyobj *bo) /* Receive response */ - SES_RxInit(htc, bo->ws, cache_param->http_resp_size, - cache_param->http_resp_hdr_len); + SES_RxInit(htc, bo->ws, cache_param->http_resp_size); CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 073684e..9dd8b3b 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -130,8 +130,7 @@ http1_unwait(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws, - cache_param->http_req_size, cache_param->http_req_hdr_len); + SES_RxInit(req->htc, req->ws, cache_param->http_req_size); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index 412370a..56d9cc1 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -349,6 +349,7 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp) CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC); + htc->maxhdr = cache_param->http_req_hdr_len; retval = http1_splitline(hp, htc, HTTP1_Req); if (retval != 0) return (retval); @@ -414,6 +415,7 @@ HTTP1_DissectResponse(struct http_conn *htc, struct http *hp, CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC); CHECK_OBJ_NOTNULL(req, HTTP_MAGIC); + htc->maxhdr = cache_param->http_resp_hdr_len; if (http1_splitline(hp, htc, HTTP1_Resp)) retval = 503; From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] 57adab3 Eliminate the maxhdr field from struct http_conn Message-ID: commit 57adab3dbb1ff8959d268d5092d56d70ca937b74 Author: Poul-Henning Kamp Date: Tue Apr 5 07:41:21 2016 +0000 Eliminate the maxhdr field from struct http_conn diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 3444656..5e4fb89 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -225,7 +225,6 @@ struct http_conn { int fd; enum sess_close doclose; unsigned maxbytes; - unsigned maxhdr; enum body_status body_status; struct ws *ws; char *rxbuf_b; diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index 56d9cc1..f8ac66f 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -106,7 +106,8 @@ HTTP1_Complete(struct http_conn *htc) */ static uint16_t -http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc) +http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc, + unsigned maxhdr) { char *q, *r; @@ -147,7 +148,7 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc) *q++ = ' '; } - if (q - p > htc->maxhdr) { + if (q - p > maxhdr) { VSLb(hp->vsl, SLT_BogoHeader, "Header too long: %.*s", (int)(q - p > 20 ? 20 : q - p), p); return (400); @@ -205,7 +206,8 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc) */ static uint16_t -http1_splitline(struct http *hp, struct http_conn *htc, const int *hf) +http1_splitline(struct http *hp, struct http_conn *htc, const int *hf, + unsigned maxhdr) { char *p; int i; @@ -271,7 +273,7 @@ http1_splitline(struct http *hp, struct http_conn *htc, const int *hf) *p = '\0'; p += i; - return (http1_dissect_hdrs(hp, p, htc)); + return (http1_dissect_hdrs(hp, p, htc, maxhdr)); } /*--------------------------------------------------------------------*/ @@ -349,8 +351,8 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp) CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC); - htc->maxhdr = cache_param->http_req_hdr_len; - retval = http1_splitline(hp, htc, HTTP1_Req); + retval = http1_splitline(hp, htc, + HTTP1_Req, cache_param->http_req_hdr_len); if (retval != 0) return (retval); hp->protover = http1_proto_ver(hp); @@ -415,8 +417,8 @@ HTTP1_DissectResponse(struct http_conn *htc, struct http *hp, CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC); CHECK_OBJ_NOTNULL(req, HTTP_MAGIC); - htc->maxhdr = cache_param->http_resp_hdr_len; - if (http1_splitline(hp, htc, HTTP1_Resp)) + if (http1_splitline(hp, htc, + HTTP1_Resp, cache_param->http_resp_hdr_len)) retval = 503; if (retval == 0) { From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] f9f6b2a Use SES_RxStuff() also for receiving beresp headers Message-ID: commit f9f6b2a4457dd96bcdb7983fc6377d540cc6281a Author: Poul-Henning Kamp Date: Tue Apr 5 09:04:14 2016 +0000 Use SES_RxStuff() also for receiving beresp headers diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 2c92ab6..4c3de64 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -136,8 +136,10 @@ V1F_FetchRespHdr(struct busyobj *bo) { struct http *hp; - int first, i; + int i; + double t; struct http_conn *htc; + enum htc_status_e hs; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC); @@ -153,41 +155,37 @@ V1F_FetchRespHdr(struct busyobj *bo) CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC); - VTCP_set_read_timeout(htc->fd, htc->first_byte_timeout); - - first = 1; - do { - i = (htc->ws->r - htc->rxbuf_e) - 1; /* space for NUL */ - if (i <= 0) { - bo->acct.beresp_hdrbytes += - htc->rxbuf_e - htc->rxbuf_b; - WS_ReleaseP(htc->ws, htc->rxbuf_b); - VSLb(bo->vsl, SLT_FetchError, - "http %sread error: overflow", - first ? "first " : ""); - htc->doclose = SC_RX_OVERFLOW; - return (-1); - } - i = read(htc->fd, htc->rxbuf_e, i); - if (i <= 0) { - bo->acct.beresp_hdrbytes += - htc->rxbuf_e - htc->rxbuf_b; - WS_ReleaseP(htc->ws, htc->rxbuf_b); - VSLb(bo->vsl, SLT_FetchError, "http %sread error: EOF", - first ? "first " : ""); + t = VTIM_real() + htc->first_byte_timeout; + hs = SES_RxStuff(htc, HTTP1_Complete, NULL, NULL, + t, t + htc->between_bytes_timeout); + if (hs != HTC_S_COMPLETE) { + bo->acct.beresp_hdrbytes += + htc->rxbuf_e - htc->rxbuf_b; + switch(hs) { + case HTC_S_JUNK: + VSLb(bo->vsl, SLT_FetchError, "Received junk"); + htc->doclose = SC_RX_JUNK; + break; + case HTC_S_CLOSE: + VSLb(bo->vsl, SLT_FetchError, "backend closed"); + htc->doclose = SC_RESP_CLOSE; + break; + case HTC_S_TIMEOUT: + VSLb(bo->vsl, SLT_FetchError, "timeout"); htc->doclose = SC_RX_TIMEOUT; - return (first ? 1 : -1); - } - if (first) { - first = 0; - VTCP_set_read_timeout(htc->fd, - htc->between_bytes_timeout); + break; + case HTC_S_OVERFLOW: + VSLb(bo->vsl, SLT_FetchError, "overflow"); + htc->doclose = SC_RX_OVERFLOW; + break; + default: + VSLb(bo->vsl, SLT_FetchError, "HTC status %d", hs); + htc->doclose = SC_RX_BAD; + break; } - htc->rxbuf_e += i; - *htc->rxbuf_e = '\0'; - } while (HTTP1_Complete(htc) != HTC_S_COMPLETE); - - WS_ReleaseP(htc->ws, htc->rxbuf_e); + return (htc->rxbuf_e == htc->rxbuf_b ? 1 : -1); + } + VTCP_set_read_timeout(htc->fd, htc->between_bytes_timeout); hp = bo->beresp; From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] a97bff7 Pass the max number of bytes to be received into SES_RxStuff() as parameter. Message-ID: commit a97bff7e4e6f4d3c41b4cd37e9e9b3460d967e80 Author: Poul-Henning Kamp Date: Tue Apr 5 09:16:43 2016 +0000 Pass the max number of bytes to be received into SES_RxStuff() as parameter. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 5e4fb89..9789b4d 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -965,7 +965,7 @@ enum htc_status_e { void SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes); void SES_RxReInit(struct http_conn *htc); enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, - double *t1, double *t2, double ti, double tn); + double *t1, double *t2, double ti, double tn, int maxbytes); #define SESS_ATTR(UP, low, typ, len) \ int SES_Set_##low(const struct sess *sp, const typ *src); \ diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index b96aa9a..6b5cccb 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -236,7 +236,7 @@ SES_RxReInit(struct http_conn *htc) enum htc_status_e SES_RxStuff(struct http_conn *htc, htc_complete_f *func, - double *t1, double *t2, double ti, double tn) + double *t1, double *t2, double ti, double tn, int maxbytes) { double tmo; double now; @@ -275,7 +275,8 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func, tmo = tn - now; if (!isnan(ti) && ti < tn) tmo = ti - now; - i = (htc->ws->r - htc->rxbuf_e) - 1; /* space for NUL */ + i = (htc->rxbuf_e - htc->rxbuf_b) + 1L; /* space for NUL */ + i = maxbytes - i; if (i <= 0) { WS_ReleaseP(htc->ws, htc->rxbuf_b); return (HTC_S_OVERFLOW); diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 4c3de64..902076a 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -157,7 +157,7 @@ V1F_FetchRespHdr(struct busyobj *bo) t = VTIM_real() + htc->first_byte_timeout; hs = SES_RxStuff(htc, HTTP1_Complete, NULL, NULL, - t, t + htc->between_bytes_timeout); + t, t + htc->between_bytes_timeout, cache_param->http_resp_size); if (hs != HTC_S_COMPLETE) { bo->acct.beresp_hdrbytes += htc->rxbuf_e - htc->rxbuf_b; diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 9dd8b3b..3dd63b0 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -323,7 +323,8 @@ HTTP1_Session(struct worker *wrk, struct req *req) hs = SES_RxStuff(req->htc, HTTP1_Complete, &req->t_first, &req->t_req, sp->t_idle + cache_param->timeout_linger, - sp->t_idle + cache_param->timeout_idle); + sp->t_idle + cache_param->timeout_idle, + cache_param->http_req_size); XXXAZ(req->htc->ws->r); if (hs < HTC_S_EMPTY) { req->acct.req_hdrbytes += diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index ea0d676..3c9cffe 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -351,8 +351,8 @@ vpx_new_session(struct worker *wrk, void *arg) assert(sizeof vpx2_sig == 12); hs = SES_RxStuff(req->htc, vpx_complete, - NULL, NULL, NAN, sp->t_idle + cache_param->timeout_idle); - XXXAZ(req->htc->ws->r); + NULL, NULL, NAN, sp->t_idle + cache_param->timeout_idle, + 1024); // XXX ? if (hs != HTC_S_COMPLETE) { Req_Release(req); SES_Delete(sp, SC_RX_JUNK, NAN); From phk at FreeBSD.org Tue Apr 5 10:13:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 12:13:05 +0200 Subject: [master] 8287ce7 Eliminate maxbytes entirely from struct http_conn Message-ID: commit 8287ce7d065e99f0621a06a671be15f2dc02bc75 Author: Poul-Henning Kamp Date: Tue Apr 5 10:11:48 2016 +0000 Eliminate maxbytes entirely from struct http_conn diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 9789b4d..ee024c5 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -224,7 +224,6 @@ struct http_conn { int fd; enum sess_close doclose; - unsigned maxbytes; enum body_status body_status; struct ws *ws; char *rxbuf_b; @@ -962,7 +961,7 @@ enum htc_status_e { HTC_S_IDLE = 3, }; -void SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes); +void SES_RxInit(struct http_conn *htc, struct ws *ws); void SES_RxReInit(struct http_conn *htc); enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, double *t1, double *t2, double ti, double tn, int maxbytes); diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index 982e1e8..faeaeb8 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -376,7 +376,7 @@ vca_make_session(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws, cache_param->http_req_size); + SES_RxInit(req->htc, req->ws); SES_SetTransport(wrk, sp, req, wa->acceptlsock->transport); } diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 6b5cccb..c9c7fa7 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -183,14 +183,13 @@ SES_Get_String_Attr(const struct sess *sp, enum sess_attr a) /*--------------------------------------------------------------------*/ void -SES_RxInit(struct http_conn *htc, struct ws *ws, unsigned maxbytes) +SES_RxInit(struct http_conn *htc, struct ws *ws) { htc->magic = HTTP_CONN_MAGIC; htc->ws = ws; - htc->maxbytes = maxbytes; - (void)WS_Reserve(htc->ws, htc->maxbytes); + (void)WS_Reserve(htc->ws, 0); htc->rxbuf_b = ws->f; htc->rxbuf_e = ws->f; *htc->rxbuf_e = '\0'; @@ -210,7 +209,7 @@ SES_RxReInit(struct http_conn *htc) ssize_t l; CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); - (void)WS_Reserve(htc->ws, htc->maxbytes); + (void)WS_Reserve(htc->ws, 0); htc->rxbuf_b = htc->ws->f; htc->rxbuf_e = htc->ws->f; if (htc->pipeline_b != NULL) { @@ -245,6 +244,9 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func, CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); + if (htc->ws->r - htc->rxbuf_b < maxbytes) + maxbytes = (htc->ws->r - htc->rxbuf_b); + AZ(isnan(tn)); if (t1 != NULL) assert(isnan(*t1)); diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 902076a..0c6a225 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -151,7 +151,7 @@ V1F_FetchRespHdr(struct busyobj *bo) /* Receive response */ - SES_RxInit(htc, bo->ws, cache_param->http_resp_size); + SES_RxInit(htc, bo->ws); CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 3dd63b0..ba937d2 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -130,7 +130,7 @@ http1_unwait(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws, cache_param->http_req_size); + SES_RxInit(req->htc, req->ws); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; From martin at varnish-software.com Tue Apr 5 13:58:05 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 05 Apr 2016 15:58:05 +0200 Subject: [4.0] 77bdac6 Update RFC reference Message-ID: commit 77bdac69e7f12ff98242692e6ad53a6efd4c689b Author: P?l Hermunn Johansen Date: Fri Apr 1 15:53:17 2016 +0200 Update RFC reference Fixes #1809, backported from master. Original patch by Federico G. Schwindt (fgsch) is 54471f2a86ae7234a31c97c0adb5e70a1b28ee38 diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index d05c823..9309f8d 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -224,7 +224,7 @@ RFC2616_Body(struct busyobj *bo, struct dstat *stats) if (hp->status == 204) { /* * 204 is "No Content", obviously don't expect a body. - * [RFC2616 10.2.5 p60] + * [RFC7230 3.3.1 p28 and 3.3.2 p30] */ stats->fetch_204++; return (BS_NONE); From martin at varnish-software.com Tue Apr 5 13:58:05 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 05 Apr 2016 15:58:05 +0200 Subject: [4.0] 32af5c5 Properly encode HTTP headers with weird characters to C identifiers. Message-ID: commit 32af5c5908a2c31acb8760edbb9641bbe3d4288b Author: Poul-Henning Kamp Date: Mon Aug 3 06:59:20 2015 +0000 Properly encode HTTP headers with weird characters to C identifiers. Please note that using underscore in HTTP headers is considered a really bad idea because many application frameworks map minus to underscore in environment variables. Fixes: #1768 Conflicts: lib/libvcc/vcc_var.c diff --git a/bin/varnishtest/tests/r01768.vtc b/bin/varnishtest/tests/r01768.vtc new file mode 100644 index 0000000..716701a --- /dev/null +++ b/bin/varnishtest/tests/r01768.vtc @@ -0,0 +1,19 @@ +varnishtest "http header collision -/_" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + if (req.http.foo_bar == req.http.foo-bar) { + set req.http.foo_bar = "xxx"; + } + } +} -start + +client c1 { + txreq + rxresp +} -run diff --git a/lib/libvcc/vcc_var.c b/lib/libvcc/vcc_var.c index 74713b8..a018f36 100644 --- a/lib/libvcc/vcc_var.c +++ b/lib/libvcc/vcc_var.c @@ -43,43 +43,43 @@ vcc_Var_Wildcard(struct vcc *tl, const struct token *t, const struct symbol *wc) struct symbol *sym; struct var *v; const struct var *vh; - int l, i; - char c; - char buf[258]; - char cnam[256]; + unsigned u; + const char *p, *leaf; + struct vsb *vsb; vh = wc->var; + assert(vh->fmt == HEADER); v = TlAlloc(tl, sizeof *v); AN(v); - - assert(vh->fmt == HEADER); v->name = TlDupTok(tl, t); v->r_methods = vh->r_methods; v->w_methods = vh->w_methods; v->fmt = vh->fmt; + leaf = v->name + vh->len; /* Create a C-name version of the header name */ - l = strlen(v->name + vh->len) + 1; - for (i = 0; i < l - 1; i++) { - c = *(v->name + vh->len + i); - if (vct_isalpha(c) || vct_isdigit(c)) - cnam[i] = c; + vsb = VSB_new_auto(); + AN(vsb); + VSB_printf(vsb, "&VGC_%s_", vh->rname); + for (p = leaf, u = 1; *p != '\0'; p++, u++) + if (vct_isalpha(*p) || vct_isdigit(*p)) + VSB_putc(vsb, *p); else - cnam[i] = '_'; - } - cnam[i] = '\0'; + VSB_printf(vsb, "_%02x_", *p); + AZ(VSB_finish(vsb)); /* Create the static identifier */ - Fh(tl, 0, "static const struct gethdr_s VGC_%s_%s =\n", - vh->rname, cnam); - Fh(tl, 0, " { %s, \"\\%03o%s:\"};\n", - vh->rname, (unsigned)l, v->name + vh->len); - - bprintf(buf, "&VGC_%s_%s", vh->rname, cnam); - v->rname = TlDup(tl, buf); - bprintf(buf, "VRT_SetHdr(ctx, %s, ", v->rname); - v->lname = TlDup(tl, buf); + Fh(tl, 0, "static const struct gethdr_s %s =\n", VSB_data(vsb) + 1); + Fh(tl, 0, " { %s, \"\\%03o%s:\"};\n", vh->rname, u, leaf); + + /* Create the symbol r/l values */ + v->rname = TlDup(tl, VSB_data(vsb)); + VSB_clear(vsb); + VSB_printf(vsb, "VRT_SetHdr(ctx, %s,", v->rname); + AZ(VSB_finish(vsb)); + v->lname = TlDup(tl, VSB_data(vsb)); + VSB_delete(vsb); sym = VCC_AddSymbolTok(tl, t, SYM_VAR); AN(sym); From martin at varnish-software.com Tue Apr 5 13:58:05 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 05 Apr 2016 15:58:05 +0200 Subject: [4.0] 9827989 Add 1815 and 1768 to the changelog Message-ID: commit 9827989f6fef28045eaa7076cfcb11f03f53e0c5 Author: Martin Blix Grydeland Date: Tue Apr 5 13:43:13 2016 +0200 Add 1815 and 1768 to the changelog diff --git a/doc/changes.rst b/doc/changes.rst index 9a84b74..8d8932c 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -12,6 +12,8 @@ Changes from 4.0.3 to (unreleased) Bugs fixed ---------- +.. _1768: https://www.varnish-cache.org/trac/ticket/1768 +.. _1815: https://www.varnish-cache.org/trac/ticket/1815 .. _1862: https://www.varnish-cache.org/trac/ticket/1862 .. _1744: https://www.varnish-cache.org/trac/ticket/1744 .. _1742: https://www.varnish-cache.org/trac/ticket/1742 @@ -20,6 +22,8 @@ Bugs fixed .. _1602: https://www.varnish-cache.org/trac/ticket/1602 .. _1598: https://www.varnish-cache.org/trac/ticket/1598 +- 1768_ - Properly encode HTTP headers with weird characters to C identifiers. +- 1815_ - Close piped sessions in the absence of backends - 1862_ - Request URL with whitespace is allowed - 1744_ - Update the users guide to for new -sfile syntax - 1742_ - Document varnishlog -w/-r with more details From fgsch at lodoss.net Tue Apr 5 15:03:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 05 Apr 2016 17:03:05 +0200 Subject: [master] bdc63fc Update device detection VCL Message-ID: commit bdc63fc38748ff6abe5f4652b62dbbd2f5a33145 Author: Federico G. Schwindt Date: Thu Mar 31 14:57:51 2016 +0100 Update device detection VCL diff --git a/etc/devicedetect.vcl b/etc/devicedetect.vcl index d1a85d9..6c7e659 100644 --- a/etc/devicedetect.vcl +++ b/etc/devicedetect.vcl @@ -42,7 +42,7 @@ sub devicedetect { if (req.http.User-Agent ~ "\(compatible; Googlebot-Mobile/2.1; \+http://www.google.com/bot.html\)" || (req.http.User-Agent ~ "iPhone" && req.http.User-Agent ~ "\(compatible; Googlebot/2.1; \+http://www.google.com/bot.html")) { set req.http.X-UA-Device = "mobile-bot"; } - elsif (req.http.User-Agent ~ "(?i)(ads|google|bing|msn|yandex|baidu|ro|career|)bot" || + elsif (req.http.User-Agent ~ "(?i)(ads|google|bing|msn|yandex|baidu|ro|career|seznam|)bot" || req.http.User-Agent ~ "(?i)(baidu|jike|symantec)spider" || req.http.User-Agent ~ "(?i)scanner" || req.http.User-Agent ~ "(?i)(web)crawler") { @@ -61,7 +61,10 @@ sub devicedetect { elsif (req.http.User-Agent ~ "PlayBook; U; RIM Tablet") { set req.http.X-UA-Device = "tablet-rim"; } elsif (req.http.User-Agent ~ "hp-tablet.*TouchPad") { set req.http.X-UA-Device = "tablet-hp"; } elsif (req.http.User-Agent ~ "Kindle/3") { set req.http.X-UA-Device = "tablet-kindle"; } - elsif (req.http.User-Agent ~ "Touch.+Tablet PC") { set req.http.X-UA-Device = "tablet-microsoft"; } + elsif (req.http.User-Agent ~ "Touch.+Tablet PC" || + req.http.User-Agent ~ "Windows NT [0-9.]+; ARM;" ) { + set req.http.X-UA-Device = "tablet-microsoft"; + } elsif (req.http.User-Agent ~ "Mobile.+Firefox") { set req.http.X-UA-Device = "mobile-firefoxos"; } elsif (req.http.User-Agent ~ "^HTC" || req.http.User-Agent ~ "Fennec" || From phk at FreeBSD.org Tue Apr 5 20:12:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 22:12:04 +0200 Subject: [master] d533839 Collapse SES_RxReInit() into SES_RxInit(). Message-ID: commit d533839d31835a7dfa43bdccd1ca1cfbbf72c726 Author: Poul-Henning Kamp Date: Tue Apr 5 19:48:05 2016 +0000 Collapse SES_RxReInit() into SES_RxInit(). diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index ee024c5..7df18fd 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -962,7 +962,6 @@ enum htc_status_e { }; void SES_RxInit(struct http_conn *htc, struct ws *ws); -void SES_RxReInit(struct http_conn *htc); enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, double *t1, double *t2, double ti, double tn, int maxbytes); diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index faeaeb8..9f3a341 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -376,7 +376,6 @@ vca_make_session(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws); SES_SetTransport(wrk, sp, req, wa->acceptlsock->transport); } diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index f4e4f61..63be280 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -66,6 +66,8 @@ Req_New(const struct worker *wrk, struct sess *sp) req->sp = sp; req->top = req; // esi overrides + INIT_OBJ(req->htc, HTTP_CONN_MAGIC); + e = (char*)req + sz; p = (char*)(req + 1); p = (void*)PRNDUP(p); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index c9c7fa7..52682aa 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -185,33 +185,13 @@ SES_Get_String_Attr(const struct sess *sp, enum sess_attr a) void SES_RxInit(struct http_conn *htc, struct ws *ws) { + ssize_t l; - htc->magic = HTTP_CONN_MAGIC; + CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); htc->ws = ws; - (void)WS_Reserve(htc->ws, 0); htc->rxbuf_b = ws->f; htc->rxbuf_e = ws->f; - *htc->rxbuf_e = '\0'; - htc->pipeline_b = NULL; - htc->pipeline_e = NULL; -} - -/*-------------------------------------------------------------------- - * Start over, and recycle any pipelined input. - * The WS_Reset is safe, even though the pipelined input is stored in - * the ws somewhere, because WS_Reset only fiddles pointers. - */ - -void -SES_RxReInit(struct http_conn *htc) -{ - ssize_t l; - - CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); - (void)WS_Reserve(htc->ws, 0); - htc->rxbuf_b = htc->ws->f; - htc->rxbuf_e = htc->ws->f; if (htc->pipeline_b != NULL) { l = htc->pipeline_e - htc->pipeline_b; assert(l > 0); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index ba937d2..220dfd0 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -112,6 +112,7 @@ http1_new_session(struct worker *wrk, void *arg) sp = req->sp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + SES_RxInit(req->htc, req->ws); SES_Reserve_xport_priv(sp, &u); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; @@ -394,7 +395,7 @@ HTTP1_Session(struct worker *wrk, struct req *req) } else if (st == H1CLEANUP) { if (Req_Cleanup(sp, wrk, req)) return; - SES_RxReInit(req->htc); + SES_RxInit(req->htc, req->ws); if (req->htc->rxbuf_e != req->htc->rxbuf_b) wrk->stats->sess_readahead++; http1_setstate(sp, H1NEWREQ); diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 3c9cffe..9d584ee 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -350,6 +350,7 @@ vpx_new_session(struct worker *wrk, void *arg) assert(sizeof vpx1_sig == 5); assert(sizeof vpx2_sig == 12); + SES_RxInit(req->htc, req->ws); hs = SES_RxStuff(req->htc, vpx_complete, NULL, NULL, NAN, sp->t_idle + cache_param->timeout_idle, 1024); // XXX ? @@ -376,7 +377,6 @@ vpx_new_session(struct worker *wrk, void *arg) req->htc->pipeline_b = NULL; else req->htc->pipeline_e = req->htc->rxbuf_e; - SES_RxReInit(req->htc); SES_SetTransport(wrk, sp, req, &HTTP1_transport); } From phk at FreeBSD.org Tue Apr 5 20:12:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 22:12:05 +0200 Subject: [master] a7c8544 Introduce a function to mark pipelined data in http_conn Message-ID: commit a7c85443ab08c2da7020e7681c1371da06ff8279 Author: Poul-Henning Kamp Date: Tue Apr 5 20:02:30 2016 +0000 Introduce a function to mark pipelined data in http_conn diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 7df18fd..a7abad5 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -962,6 +962,7 @@ enum htc_status_e { }; void SES_RxInit(struct http_conn *htc, struct ws *ws); +void SES_RxPipeline(struct http_conn *htc, void *); enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, double *t1, double *t2, double ti, double tn, int maxbytes); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 52682aa..db04b0a 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -203,6 +203,22 @@ SES_RxInit(struct http_conn *htc, struct ws *ws) *htc->rxbuf_e = '\0'; } +void +SES_RxPipeline(struct http_conn *htc, void *p) +{ + + CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); + if (p == NULL || (char*)p == htc->rxbuf_e) { + htc->pipeline_b = NULL; + htc->pipeline_e = NULL; + return; + } + assert((char*)p >= htc->rxbuf_b); + assert((char*)p < htc->rxbuf_e); + htc->pipeline_b = p; + htc->pipeline_e = htc->rxbuf_e; +} + /*---------------------------------------------------------------------- * Receive a request/packet/whatever, with timeouts * @@ -233,6 +249,8 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func, while (1) { now = VTIM_real(); + AZ(htc->pipeline_b); + AZ(htc->pipeline_e); hs = func(htc); if (hs == HTC_S_OVERFLOW || hs == HTC_S_JUNK) { WS_ReleaseP(htc->ws, htc->rxbuf_b); diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index f8ac66f..07b2fef 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -68,8 +68,6 @@ HTTP1_Complete(struct http_conn *htc) char *p; CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); - AZ(htc->pipeline_b); - AZ(htc->pipeline_e); assert(htc->rxbuf_e >= htc->rxbuf_b); assert(*htc->rxbuf_e == '\0'); @@ -193,11 +191,8 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc, } if (p < htc->rxbuf_e) p += vct_skipcrlf(p); - if (p < htc->rxbuf_e) { - htc->pipeline_b = p; - htc->pipeline_e = htc->rxbuf_e; - htc->rxbuf_e = p; - } + SES_RxPipeline(htc, p); + htc->rxbuf_e = p; return (0); } diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 9d584ee..6fb8cc4 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -150,7 +150,7 @@ vpx_proto1(const struct worker *wrk, struct req *req) VSL(SLT_Proxy, req->sp->vxid, "1 %s %s %s %s", fld[1], fld[3], fld[2], fld[4]); - req->htc->pipeline_b = q; + SES_RxPipeline(req->htc, q); return (0); } @@ -184,7 +184,7 @@ vpx_proto2(const struct worker *wrk, struct req *req) assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L); l = vbe16dec(req->htc->rxbuf_b + 14); assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L + l); - req->htc->pipeline_b = req->htc->rxbuf_b + 16L + l; + SES_RxPipeline(req->htc, req->htc->rxbuf_b + 16L + l); p = (const void *)req->htc->rxbuf_b; /* Version @12 top half */ @@ -299,8 +299,6 @@ vpx_complete(struct http_conn *htc) char *p, *q; CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); - AZ(htc->pipeline_b); - AZ(htc->pipeline_e); l = htc->rxbuf_e - htc->rxbuf_b; p = htc->rxbuf_b; @@ -373,10 +371,6 @@ vpx_new_session(struct worker *wrk, void *arg) return; } - if (req->htc->rxbuf_e == req->htc->pipeline_b) - req->htc->pipeline_b = NULL; - else - req->htc->pipeline_e = req->htc->rxbuf_e; SES_SetTransport(wrk, sp, req, &HTTP1_transport); } From phk at FreeBSD.org Tue Apr 5 20:12:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 22:12:05 +0200 Subject: [master] 5f25e69 When the completion function returns empty, reset the rxbuffer in the central code, rather than the completion function. Message-ID: commit 5f25e696c2c27f78864aaf13bdf7c60a758b5f83 Author: Poul-Henning Kamp Date: Tue Apr 5 20:10:10 2016 +0000 When the completion function returns empty, reset the rxbuffer in the central code, rather than the completion function. diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index db04b0a..c6823d0 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -269,7 +269,10 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func, /* Working on it */ if (t1 != NULL && isnan(*t1)) *t1 = now; - } else if (hs != HTC_S_EMPTY) + } else if (hs == HTC_S_EMPTY) { + htc->rxbuf_e = htc->rxbuf_b; + *htc->rxbuf_e = '\0'; + } else WRONG("htc_status_e"); tmo = tn - now; diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index 07b2fef..8bac16b 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -75,12 +75,9 @@ HTTP1_Complete(struct http_conn *htc) /* Skip any leading white space */ for (p = htc->rxbuf_b ; vct_islws(*p); p++) continue; - if (p == htc->rxbuf_e) { - /* All white space */ - htc->rxbuf_e = htc->rxbuf_b; - *htc->rxbuf_e = '\0'; + if (p == htc->rxbuf_e) return (HTC_S_EMPTY); - } + /* * Here we just look for NL[CR]NL to see that reception * is completed. More stringent validation happens later. From phk at FreeBSD.org Tue Apr 5 20:24:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 05 Apr 2016 22:24:05 +0200 Subject: [master] 82c21f9 Rename SES_Rx*() to HTC_Rx*(), but leave them in cache_session.c for the lack of any better place to put them right now. Message-ID: commit 82c21f91d3d93bc3a68d7708549ba6109d4313eb Author: Poul-Henning Kamp Date: Tue Apr 5 20:17:34 2016 +0000 Rename SES_Rx*() to HTC_Rx*(), but leave them in cache_session.c for the lack of any better place to put them right now. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index a7abad5..bb2a2b5 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -961,9 +961,9 @@ enum htc_status_e { HTC_S_IDLE = 3, }; -void SES_RxInit(struct http_conn *htc, struct ws *ws); -void SES_RxPipeline(struct http_conn *htc, void *); -enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *, +void HTC_RxInit(struct http_conn *htc, struct ws *ws); +void HTC_RxPipeline(struct http_conn *htc, void *); +enum htc_status_e HTC_RxStuff(struct http_conn *, htc_complete_f *, double *t1, double *t2, double ti, double tn, int maxbytes); #define SESS_ATTR(UP, low, typ, len) \ diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index c6823d0..4eaee4b 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -183,7 +183,7 @@ SES_Get_String_Attr(const struct sess *sp, enum sess_attr a) /*--------------------------------------------------------------------*/ void -SES_RxInit(struct http_conn *htc, struct ws *ws) +HTC_RxInit(struct http_conn *htc, struct ws *ws) { ssize_t l; @@ -204,7 +204,7 @@ SES_RxInit(struct http_conn *htc, struct ws *ws) } void -SES_RxPipeline(struct http_conn *htc, void *p) +HTC_RxPipeline(struct http_conn *htc, void *p) { CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); @@ -230,7 +230,7 @@ SES_RxPipeline(struct http_conn *htc, void *p) */ enum htc_status_e -SES_RxStuff(struct http_conn *htc, htc_complete_f *func, +HTC_RxStuff(struct http_conn *htc, htc_complete_f *func, double *t1, double *t2, double ti, double tn, int maxbytes) { double tmo; diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 0c6a225..6fd5f64 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -151,12 +151,12 @@ V1F_FetchRespHdr(struct busyobj *bo) /* Receive response */ - SES_RxInit(htc, bo->ws); + HTC_RxInit(htc, bo->ws); CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC); t = VTIM_real() + htc->first_byte_timeout; - hs = SES_RxStuff(htc, HTTP1_Complete, NULL, NULL, + hs = HTC_RxStuff(htc, HTTP1_Complete, NULL, NULL, t, t + htc->between_bytes_timeout, cache_param->http_resp_size); if (hs != HTC_S_COMPLETE) { bo->acct.beresp_hdrbytes += diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 220dfd0..5670421 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -112,7 +112,7 @@ http1_new_session(struct worker *wrk, void *arg) sp = req->sp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - SES_RxInit(req->htc, req->ws); + HTC_RxInit(req->htc, req->ws); SES_Reserve_xport_priv(sp, &u); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; @@ -131,7 +131,7 @@ http1_unwait(struct worker *wrk, void *arg) req = Req_New(wrk, sp); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->fd = sp->fd; - SES_RxInit(req->htc, req->ws); + HTC_RxInit(req->htc, req->ws); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; @@ -321,7 +321,7 @@ HTTP1_Session(struct worker *wrk, struct req *req) AZ(req->vcl); AZ(req->esi_level); - hs = SES_RxStuff(req->htc, HTTP1_Complete, + hs = HTC_RxStuff(req->htc, HTTP1_Complete, &req->t_first, &req->t_req, sp->t_idle + cache_param->timeout_linger, sp->t_idle + cache_param->timeout_idle, @@ -395,7 +395,7 @@ HTTP1_Session(struct worker *wrk, struct req *req) } else if (st == H1CLEANUP) { if (Req_Cleanup(sp, wrk, req)) return; - SES_RxInit(req->htc, req->ws); + HTC_RxInit(req->htc, req->ws); if (req->htc->rxbuf_e != req->htc->rxbuf_b) wrk->stats->sess_readahead++; http1_setstate(sp, H1NEWREQ); diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index 8bac16b..30b318c 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -188,7 +188,7 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc, } if (p < htc->rxbuf_e) p += vct_skipcrlf(p); - SES_RxPipeline(htc, p); + HTC_RxPipeline(htc, p); htc->rxbuf_e = p; return (0); } diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 6fb8cc4..ee6fef5 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -150,7 +150,7 @@ vpx_proto1(const struct worker *wrk, struct req *req) VSL(SLT_Proxy, req->sp->vxid, "1 %s %s %s %s", fld[1], fld[3], fld[2], fld[4]); - SES_RxPipeline(req->htc, q); + HTC_RxPipeline(req->htc, q); return (0); } @@ -184,7 +184,7 @@ vpx_proto2(const struct worker *wrk, struct req *req) assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L); l = vbe16dec(req->htc->rxbuf_b + 14); assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L + l); - SES_RxPipeline(req->htc, req->htc->rxbuf_b + 16L + l); + HTC_RxPipeline(req->htc, req->htc->rxbuf_b + 16L + l); p = (const void *)req->htc->rxbuf_b; /* Version @12 top half */ @@ -348,8 +348,8 @@ vpx_new_session(struct worker *wrk, void *arg) assert(sizeof vpx1_sig == 5); assert(sizeof vpx2_sig == 12); - SES_RxInit(req->htc, req->ws); - hs = SES_RxStuff(req->htc, vpx_complete, + HTC_RxInit(req->htc, req->ws); + hs = HTC_RxStuff(req->htc, vpx_complete, NULL, NULL, NAN, sp->t_idle + cache_param->timeout_idle, 1024); // XXX ? if (hs != HTC_S_COMPLETE) { From phk at FreeBSD.org Tue Apr 5 22:09:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 06 Apr 2016 00:09:05 +0200 Subject: [master] b477c37 Introduce SES_Ref() and SES_Rel(), so we can start to decorate the code with them. Message-ID: commit b477c3778bd503e229d45fdac9a5bafec317256e Author: Poul-Henning Kamp Date: Tue Apr 5 21:04:34 2016 +0000 Introduce SES_Ref() and SES_Rel(), so we can start to decorate the code with them. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index bb2a2b5..eef6d8b 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -634,6 +634,7 @@ struct sess { #define SESS_MAGIC 0x2c2f9c5a uint16_t sattr[SA_LAST]; + int refcnt; int fd; uint32_t vxid; @@ -944,7 +945,8 @@ struct sess *SES_New(struct pool *); void SES_Close(struct sess *, enum sess_close reason); void SES_Wait(struct sess *, const struct transport *); void SES_Delete(struct sess *, enum sess_close reason, double now); -void SES_NewPool(struct pool *, unsigned pool_no); +void SES_Ref(struct sess *sp); +void SES_Rel(struct sess *sp); int SES_Reschedule_Req(struct req *); void SES_SetTransport(struct worker *, struct sess *, struct req *, const struct transport *); diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index ecb01bc..876a0b6 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -89,6 +89,7 @@ ved_include(struct req *preq, const char *src, const char *host, return; req = Req_New(wrk, preq->sp); + SES_Ref(preq->sp); req->req_body_status = REQ_BODY_NONE; AZ(req->vsl->wid); req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER); @@ -182,6 +183,7 @@ ved_include(struct req *preq, const char *src, const char *host, req->wrk = NULL; THR_SetRequest(preq); + SES_Rel(req->sp); Req_Release(req); } diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h index 2676680..ff354a6 100644 --- a/bin/varnishd/cache/cache_priv.h +++ b/bin/varnishd/cache/cache_priv.h @@ -95,6 +95,9 @@ void Pool_Init(void); /* cache_proxy.c [VPX] */ task_func_t VPX_Proto_Sess; +/* cache_session.c */ +void SES_NewPool(struct pool *, unsigned pool_no); + /* cache_shmlog.c */ void VSM_Init(void); void VSL_Setup(struct vsl_log *vsl, void *ptr, size_t len); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 4eaee4b..6088fae 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -325,6 +325,7 @@ SES_New(struct pool *pp) sp = MPL_Get(pp->mpl_sess, &sz); sp->magic = SESS_MAGIC; sp->pool = pp; + sp->refcnt = 1; memset(sp->sattr, 0xff, sizeof sp->sattr); e = (char*)sp + sz; @@ -534,6 +535,31 @@ SES_Delete(struct sess *sp, enum sess_close reason, double now) } /*-------------------------------------------------------------------- + */ + +void +SES_Ref(struct sess *sp) +{ + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + Lck_Lock(&sp->mtx); + assert(sp->refcnt > 0); + sp->refcnt++; + Lck_Unlock(&sp->mtx); +} + +void +SES_Rel(struct sess *sp) +{ + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + Lck_Lock(&sp->mtx); + assert(sp->refcnt > 0); + sp->refcnt--; + Lck_Unlock(&sp->mtx); +} + +/*-------------------------------------------------------------------- * Create and delete pools */ From phk at FreeBSD.org Tue Apr 5 22:09:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 06 Apr 2016 00:09:05 +0200 Subject: [master] 4743b17 Deref the sess in SES_Delete() and release the storage on ref==0 Message-ID: commit 4743b17b0fb3b11525d6a4a0aeac959f68d0b396 Author: Poul-Henning Kamp Date: Tue Apr 5 21:57:19 2016 +0000 Deref the sess in SES_Delete() and release the storage on ref==0 diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 6088fae..6c0e45f 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -507,11 +507,8 @@ SES_Close(struct sess *sp, enum sess_close reason) void SES_Delete(struct sess *sp, enum sess_close reason, double now) { - struct pool *pp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - pp = sp->pool; - CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); if (reason != SC_NULL) SES_Close(sp, reason); @@ -529,9 +526,7 @@ SES_Delete(struct sess *sp, enum sess_close reason, double now) VSL(SLT_SessClose, sp->vxid, "%s %.3f", sess_close_2str(reason, 0), now - sp->t_open); VSL(SLT_End, sp->vxid, "%s", ""); - - Lck_Delete(&sp->mtx); - MPL_Free(pp->mpl_sess, sp); + SES_Rel(sp); } /*-------------------------------------------------------------------- @@ -551,12 +546,21 @@ SES_Ref(struct sess *sp) void SES_Rel(struct sess *sp) { + int i; + struct pool *pp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + pp = sp->pool; + CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + Lck_Lock(&sp->mtx); assert(sp->refcnt > 0); - sp->refcnt--; + i = --sp->refcnt; Lck_Unlock(&sp->mtx); + if (i) + return; + Lck_Delete(&sp->mtx); + MPL_Free(sp->pool->mpl_sess, sp); } /*-------------------------------------------------------------------- From phk at FreeBSD.org Tue Apr 5 22:09:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 06 Apr 2016 00:09:05 +0200 Subject: [master] abeded8 Vector request failuers through the transport protocol Message-ID: commit abeded8d1bf8601e0df5ea1cbd42190e24482618 Author: Poul-Henning Kamp Date: Tue Apr 5 22:08:20 2016 +0000 Vector request failuers through the transport protocol diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 63be280..72c3d95 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -39,6 +39,7 @@ #include "cache.h" #include "cache_pool.h" +#include "cache_transport.h" #include "vtim.h" @@ -209,11 +210,11 @@ Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req) /*---------------------------------------------------------------------- */ -void __match_proto__() +void __match_proto__(vtr_req_fail_f) Req_Fail(struct req *req, enum sess_close reason) { CHECK_OBJ_NOTNULL(req, REQ_MAGIC); - if (req->sp->fd >= 0) - SES_Close(req->sp, reason); + AN(req->transport->req_fail); + req->transport->req_fail(req, reason); } diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h index 0af1a11..40847a5 100644 --- a/bin/varnishd/cache/cache_transport.h +++ b/bin/varnishd/cache/cache_transport.h @@ -40,6 +40,7 @@ typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody); typedef void vtr_req_body_f (struct req *); typedef void vtr_sess_panic_f (struct vsb *, const struct sess *); typedef void vtr_req_panic_f (struct vsb *, const struct req *); +typedef void vtr_req_fail_f (struct req *, enum sess_close); struct transport { unsigned magic; @@ -52,6 +53,7 @@ struct transport { task_func_t *new_session; task_func_t *unwait; + vtr_req_fail_f *req_fail; vtr_req_body_f *req_body; vtr_deliver_f *deliver; vtr_sess_panic_f *sess_panic; diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 5670421..0174604 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -168,12 +168,20 @@ http1_req_panic(struct vsb *vsb, const struct req *req) VSB_printf(vsb, "state = %s\n", http1_getstate(req->sp)); } +static void __match_proto__(vtr_req_fail_f) +http1_req_fail(struct req *req, enum sess_close reason) +{ + if (req->sp->fd >= 0) + SES_Close(req->sp, reason); +} + struct transport HTTP1_transport = { .name = "HTTP/1", .magic = TRANSPORT_MAGIC, .deliver = V1D_Deliver, .unwait = http1_unwait, .req_body = http1_req_body, + .req_fail = http1_req_fail, .new_session = http1_new_session, .sess_panic = http1_sess_panic, .req_panic = http1_req_panic, From phk at FreeBSD.org Tue Apr 5 22:14:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 06 Apr 2016 00:14:04 +0200 Subject: [master] 6313596 Use correct printf spec for ssize_t Message-ID: commit 6313596ada4909b0202d264f5a07b92343a869c5 Author: Poul-Henning Kamp Date: Tue Apr 5 22:13:00 2016 +0000 Use correct printf spec for ssize_t diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index e00332b..9928991 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -509,6 +509,6 @@ vmod_barrier_sync(VRT_CTX, VCL_STRING addr) VSLb(ctx->vsl, SLT_Error, "Barrier read failed: %s (errno=%d)", strerror(i), i); if (sz > 0) - VSLb(ctx->vsl, SLT_Error, "Barrier unexpected data (%ldB)", sz); + VSLb(ctx->vsl, SLT_Error, "Barrier unexpected data (%zdB)", sz); return (0); } From phk at FreeBSD.org Tue Apr 5 22:21:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 06 Apr 2016 00:21:04 +0200 Subject: [master] 05065fe Fix another size_t printf format Message-ID: commit 05065fea0f0df8530ed56ccdd44ea0ea4223bce5 Author: Poul-Henning Kamp Date: Tue Apr 5 22:20:31 2016 +0000 Fix another size_t printf format diff --git a/bin/varnishtest/vtc_barrier.c b/bin/varnishtest/vtc_barrier.c index f0becfa..a7bffc4 100644 --- a/bin/varnishtest/vtc_barrier.c +++ b/bin/varnishtest/vtc_barrier.c @@ -328,7 +328,7 @@ barrier_sock_sync(struct barrier *b, struct vtclog *vl) vtc_log(vl, 0, "Barrier(%s) read failed: %s (errno=%d)", b->name, strerror(i), i); if (sz > 0) - vtc_log(vl, 0, "Barrier(%s) unexpected data (%ldB)", + vtc_log(vl, 0, "Barrier(%s) unexpected data (%zdB)", b->name, sz); } From varnish-commit at varnish-cache.org Wed Apr 6 13:10:23 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: Wed, 06 Apr 2016 20:10:23 +0700 Subject: Document(1) Message-ID: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Document(1).doc Type: application/msword Size: 76896 bytes Desc: Document(1).doc URL: From dridi.boukelmoune at gmail.com Thu Apr 7 08:27:04 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 07 Apr 2016 10:27:04 +0200 Subject: [master] 2624c7e Sync with reality Message-ID: commit 2624c7eb75110fe7898b006c1c0c5a8aa7214d63 Author: Dridi Boukelmoune Date: Thu Apr 7 10:25:32 2016 +0200 Sync with reality diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index 535b403..3999e20 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -315,7 +315,6 @@ Be aware that on 32 bit systems, certain default values are reduced relative to the values listed below, in order to conserve VM space: * workspace_client: 16k -* thread_pool_workspace: 16k * http_resp_size: 8k * http_req_size: 12k * gzip_stack_buffer: 4k From phk at FreeBSD.org Thu Apr 7 08:51:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 10:51:04 +0200 Subject: [master] 5550d12 Make VSA_Get_Sockaddr() return a const void*, since we don't know what sockaddr-hack it is destined for. Message-ID: commit 5550d1236547e13f03d1984058ad2427c76c50e9 Author: Poul-Henning Kamp Date: Thu Apr 7 08:44:57 2016 +0000 Make VSA_Get_Sockaddr() return a const void*, since we don't know what sockaddr-hack it is destined for. diff --git a/include/vsa.h b/include/vsa.h index 185be03..e692003 100644 --- a/include/vsa.h +++ b/include/vsa.h @@ -38,7 +38,7 @@ unsigned VSA_Port(const struct suckaddr *); int VSA_Compare(const struct suckaddr *, const struct suckaddr *); struct suckaddr *VSA_Clone(const struct suckaddr *sua); -const struct sockaddr *VSA_Get_Sockaddr(const struct suckaddr *, socklen_t *sl); +const void *VSA_Get_Sockaddr(const struct suckaddr *, socklen_t *sl); int VSA_Get_Proto(const struct suckaddr *); /* diff --git a/lib/libvarnish/vsa.c b/lib/libvarnish/vsa.c index 43d9a86..4809dfe 100644 --- a/lib/libvarnish/vsa.c +++ b/lib/libvarnish/vsa.c @@ -265,7 +265,7 @@ VSA_Build(void *d, const void *s, unsigned sal) return (NULL); } -const struct sockaddr * +const void * VSA_Get_Sockaddr(const struct suckaddr *sua, socklen_t *sl) { From phk at FreeBSD.org Thu Apr 7 08:51:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 10:51:04 +0200 Subject: [master] 67d1592 Have the busyobj hold a ref on the clients session, so we can get at client/server IPs. Message-ID: commit 67d15920302f4bf7486193ada259f1966853e3b3 Author: Poul-Henning Kamp Date: Thu Apr 7 08:46:35 2016 +0000 Have the busyobj hold a ref on the clients session, so we can get at client/server IPs. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index eef6d8b..77f62e1 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -472,6 +472,7 @@ struct busyobj { */ int retries; struct req *req; + struct sess *sp; struct worker *wrk; struct vfp_ctx vfc[1]; diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index fb7c638..8a513f2 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -989,6 +989,7 @@ vbf_fetch_thread(struct worker *wrk, void *priv) wrk->vsl = NULL; HSH_DerefBoc(wrk, bo->fetch_objcore); + SES_Rel(bo->sp); VBO_ReleaseBusyObj(wrk, &bo); THR_SetBusyobj(NULL); } @@ -1029,6 +1030,8 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc, THR_SetBusyobj(bo); + bo->sp = req->sp; + SES_Ref(bo->sp); AN(bo->vcl); @@ -1061,6 +1064,8 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc, if (bo->stale_oc != NULL) (void)HSH_DerefObjCore(wrk, &bo->stale_oc); HSH_DerefBoc(wrk, oc); + SES_Rel(bo->sp); + VBO_ReleaseBusyObj(wrk, &bo); } else { bo = NULL; /* ref transferred to fetch thread */ if (mode == VBF_BACKGROUND) { From phk at FreeBSD.org Thu Apr 7 08:51:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 10:51:04 +0200 Subject: [master] 48a844d Add a ".proxy_header" to the backend definitions. Message-ID: commit 48a844d07e19002fe8e1e04f0990c14040c94c38 Author: Poul-Henning Kamp Date: Thu Apr 7 08:49:03 2016 +0000 Add a ".proxy_header" to the backend definitions. diff --git a/include/vrt.h b/include/vrt.h index 2b243c3..427e45c 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -40,6 +40,7 @@ * Whenever something is deleted or changed in a way which is not * binary/load-time compatible, increment MAJOR version * + * 3.2 = vrt_backend grew .proxy_header field * Bump VRT_MINOR_VERSION due to: * - VCL_ACL type added * Bump VRT_MAJOR_VERSION due to: @@ -49,7 +50,7 @@ #define VRT_MAJOR_VERSION 3U -#define VRT_MINOR_VERSION 1U +#define VRT_MINOR_VERSION 2U /***********************************************************************/ @@ -176,7 +177,8 @@ extern const void * const vrt_magic_string_unset; double connect_timeout; \ double first_byte_timeout; \ double between_bytes_timeout; \ - unsigned max_connections; + unsigned max_connections; \ + unsigned proxy_header; #define VRT_BACKEND_HANDLE() \ do { \ @@ -189,6 +191,7 @@ extern const void * const vrt_magic_string_unset; DN(first_byte_timeout); \ DN(between_bytes_timeout); \ DN(max_connections); \ + DN(proxy_header); \ } while(0) struct vrt_backend { From phk at FreeBSD.org Thu Apr 7 08:51:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 10:51:04 +0200 Subject: [master] e1d1b66 Add support for sending PROXY headers to backends. Message-ID: commit e1d1b66e2333e0fa2cab8093dc1fb8614d7e01bf Author: Poul-Henning Kamp Date: Thu Apr 7 08:49:26 2016 +0000 Add support for sending PROXY headers to backends. diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index d544ffd..a76e95a 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -288,6 +288,7 @@ static void vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname) { struct token *t_field; + struct token *t_val; struct token *t_host = NULL; struct token *t_port = NULL; struct token *t_hosthdr = NULL; @@ -307,6 +308,7 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname) "?between_bytes_timeout", "?probe", "?max_connections", + "?proxy_header", NULL); SkipToken(tl, '{'); @@ -379,6 +381,18 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname) ERRCHK(tl); SkipToken(tl, ';'); Fb(tl, 0, "\t.max_connections = %u,\n", u); + } else if (vcc_IdIs(t_field, "proxy_header")) { + t_val = tl->t; + u = vcc_UintVal(tl); + ERRCHK(tl); + if (u != 1 && u != 2) { + VSB_printf(tl->sb, + ".proxy_header must be 1 or 2\n"); + vcc_ErrWhere(tl, t_val); + return; + } + SkipToken(tl, ';'); + Fb(tl, 0, "\t.proxy_header = %u,\n", u); } else if (vcc_IdIs(t_field, "probe") && tl->t->tok == '{') { vcc_ParseProbeSpec(tl, NULL, &p); Fb(tl, 0, "\t.probe = &%s,\n", p); From phk at FreeBSD.org Thu Apr 7 08:51:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 10:51:04 +0200 Subject: [master] 7b9a3a3 Add support for sending PROXY headers to backends. Message-ID: commit 7b9a3a30086ee5d3638d65597c316a278ef42991 Author: Poul-Henning Kamp Date: Thu Apr 7 08:50:11 2016 +0000 Add support for sending PROXY headers to backends. Connections to backends so configured are never reused. diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c index 2fff7c2..14e5c04 100644 --- a/bin/varnishd/cache/cache_backend.c +++ b/bin/varnishd/cache/cache_backend.c @@ -42,6 +42,7 @@ #include "cache_director.h" #include "cache_backend.h" +#include "cache_transport.h" #include "http1/cache_http1.h" #define FIND_TMO(tmx, dst, bo, be) \ @@ -108,6 +109,9 @@ vbe_dir_getfd(struct worker *wrk, struct backend *bp, struct busyobj *bo) bp->vsc->req++; Lck_Unlock(&bp->mtx); + if (bp->proxy_header != 0) + VPX_Send_Proxy(vc->fd, bp->proxy_header, bo->sp); + VTCP_myname(vc->fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1); VTCP_hisname(vc->fd, abuf2, sizeof abuf2, pbuf2, sizeof pbuf2); VSLb(bo->vsl, SLT_BackendOpen, "%d %s %s %s %s %s", @@ -152,7 +156,7 @@ vbe_dir_finish(const struct director *d, struct worker *wrk, bo->htc->priv = NULL; if (vbc->state != VBC_STATE_USED) VBT_Wait(wrk, vbc); - if (bo->htc->doclose != SC_NULL) { + if (bo->htc->doclose != SC_NULL || bp->proxy_header != 0) { VSLb(bo->vsl, SLT_BackendClose, "%d %s", vbc->fd, bp->display_name); VBT_Close(bp->tcp_pool, &vbc); diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h index 40847a5..2b22519 100644 --- a/bin/varnishd/cache/cache_transport.h +++ b/bin/varnishd/cache/cache_transport.h @@ -66,4 +66,4 @@ extern struct transport PROXY_transport; extern struct transport HTTP1_transport; const struct transport *XPORT_ByNumber(uint16_t no); - +void VPX_Send_Proxy(int fd, int version, const struct sess *); diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index ee6fef5..fafbd0a 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -43,6 +43,7 @@ #include "vend.h" #include "vsa.h" +#include "vsb.h" #include "vtcp.h" /********************************************************************** @@ -379,3 +380,98 @@ struct transport PROXY_transport = { .magic = TRANSPORT_MAGIC, .new_session = vpx_new_session, }; + +static void +vpx_enc_addr(struct vsb *vsb, int proto, const struct suckaddr *s) +{ + const struct sockaddr_in *sin4; + const struct sockaddr_in6 *sin6; + socklen_t sl; + + if (proto == PF_INET6) { + sin6 = VSA_Get_Sockaddr(s, &sl); //lint !e826 + AN(sin6); + assert(sl >= sizeof *sin6); + VSB_bcat(vsb, &sin6->sin6_addr, sizeof(sin6->sin6_addr)); + } else { + sin4 = VSA_Get_Sockaddr(s, &sl); //lint !e826 + AN(sin4); + assert(sl >= sizeof *sin4); + VSB_bcat(vsb, &sin4->sin_addr, sizeof(sin4->sin_addr)); + } +} + +static void +vpx_enc_port(struct vsb *vsb, const struct suckaddr *s) +{ + uint8_t b[2]; + + vbe16enc(b, (uint16_t)VSA_Port(s)); + VSB_bcat(vsb, b, sizeof(b)); +} + +void +VPX_Send_Proxy(int fd, int version, const struct sess *sp) +{ + struct vsb *vsb, *vsb2; + const char *p1, *p2; + struct suckaddr *sac, *sas; + char ha[VTCP_ADDRBUFSIZE]; + char pa[VTCP_PORTBUFSIZE]; + int proto; + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + assert(version == 1 || version == 2); + vsb = VSB_new_auto(); + AN(vsb); + + AZ(SES_Get_server_addr(sp, &sas)); + AN(sas); + proto = VSA_Get_Proto(sas); + assert(proto == PF_INET6 || proto == PF_INET); + + if (version == 1) { + VSB_bcat(vsb, vpx1_sig, sizeof(vpx1_sig)); + p1 = SES_Get_String_Attr(sp, SA_CLIENT_IP); + AN(p1); + p2 = SES_Get_String_Attr(sp, SA_CLIENT_PORT); + AN(p2); + VTCP_name(sas, ha, sizeof ha, pa, sizeof pa); + if (proto == PF_INET6) + VSB_printf(vsb, " TCP6 "); + else if (proto == PF_INET) + VSB_printf(vsb, " TCP4 "); + VSB_printf(vsb, "%s %s %s %s\r\n", p1, ha, p2, pa); + } else if (version == 2) { + AZ(SES_Get_client_addr(sp, &sac)); + AN(sac); + + VSB_bcat(vsb, vpx2_sig, sizeof(vpx2_sig)); + VSB_putc(vsb, 0x21); + if (proto == PF_INET6) { + VSB_putc(vsb, 0x21); + VSB_putc(vsb, 0x00); + VSB_putc(vsb, 0x24); + } else if (proto == PF_INET) { + VSB_putc(vsb, 0x11); + VSB_putc(vsb, 0x00); + VSB_putc(vsb, 0x0c); + } + vpx_enc_addr(vsb, proto, sac); + vpx_enc_addr(vsb, proto, sas); + vpx_enc_port(vsb, sac); + vpx_enc_port(vsb, sas); + } else + WRONG("Wrong proxy version"); + + AZ(VSB_finish(vsb)); + (void)write(fd, VSB_data(vsb), VSB_len(vsb)); + vsb2 = VSB_new_auto(); + AN(vsb2); + VSB_quote(vsb2, VSB_data(vsb), VSB_len(vsb), + version == 2 ? VSB_QUOTE_HEX : 0); + AZ(VSB_finish(vsb2)); + VSL(SLT_Debug, 999, "PROXY_HDR %s", VSB_data(vsb2)); + VSB_delete(vsb); + VSB_delete(vsb2); +} diff --git a/bin/varnishtest/tests/o00002.vtc b/bin/varnishtest/tests/o00002.vtc new file mode 100644 index 0000000..5c6fee0 --- /dev/null +++ b/bin/varnishtest/tests/o00002.vtc @@ -0,0 +1,103 @@ +varnishtest "Sending proxy headers to backend" + +# This test is kind of hairy. +# We don't have code in server to validate PROXY headers +# so use a pipe of: c1 [proxy] v2 [proxy] v1 [http] s1 +# Using proxy also between c1 and v2 allows us to test +# IPv6 processing over a IPv4 connection. + +server s1 { + rxreq + expect req.url == "/1" + expect req.http.xyzzy1 == req.http.xyzzy2 + expect req.http.xyzzy1 == 1111 + expect req.http.x-forwarded-for == "1.2.3.4, 1.2.3.4" + txresp -body "proxy1" + + rxreq + expect req.url == "/2" + expect req.http.xyzzy1 == req.http.xyzzy2 + expect req.http.xyzzy1 == 2222 + expect req.http.x-forwarded-for == "1.2.3.4, 1.2.3.4" + txresp -body "proxy2" + + rxreq + expect req.url == "/3" + expect req.http.xyzzy1 == req.http.xyzzy2 + expect req.http.xyzzy1 == 3333 + expect req.http.x-forwarded-for == "1:f::2, 1:f::2" + txresp -body "proxy3" + + rxreq + expect req.url == "/4" + expect req.http.xyzzy1 == req.http.xyzzy2 + expect req.http.xyzzy1 == 4444 + expect req.http.x-forwarded-for == "1:f::2, 1:f::2" + txresp -body "proxy4" +} -start + +varnish v1 -proto PROXY -vcl+backend { + import std; + + sub vcl_recv { + set req.http.xyzzy1 = std.port(client.ip); + } +} -start + +varnish v2 -proto PROXY -vcl { + import std; + + backend bp1 { + .host = "${v1_addr}"; + .port = "${v1_port}"; + .proxy_header = 1; + } + backend bp2 { + .host = "${v1_addr}"; + .port = "${v1_port}"; + .proxy_header = 2; + } + sub vcl_recv { + set req.http.xyzzy2 = std.port(client.ip); + if (req.url == "/1" || req.url == "/3") { + set req.backend_hint = bp1; + } else { + set req.backend_hint = bp2; + } + } + sub vcl_deliver { + set resp.http.connection = "close"; + } +} -start + +client c1 -connect ${v2_sock} { + send "PROXY TCP4 1.2.3.4 5.6.7.8 1111 5678\r\n" + txreq -url /1 + rxresp + expect resp.body == "proxy1" +} -run +delay .2 + +client c1 -connect ${v2_sock} { + send "PROXY TCP4 1.2.3.4 5.6.7.8 2222 5678\r\n" + txreq -url /2 + rxresp + expect resp.body == "proxy2" +} -run +delay .2 + +client c1 -connect ${v2_sock} { + send "PROXY TCP6 1:f::2 5:a::8 3333 5678\r\n" + txreq -url /3 + rxresp + expect resp.body == "proxy3" +} -run +delay .2 + +client c1 -connect ${v2_sock} { + send "PROXY TCP6 1:f::2 5:a::8 4444 5678\r\n" + txreq -url /4 + rxresp + expect resp.body == "proxy4" +} -run +delay .2 From dridi.boukelmoune at gmail.com Thu Apr 7 09:36:04 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 07 Apr 2016 11:36:04 +0200 Subject: [4.0] 2d511c9 Don't consider backends as resources Message-ID: commit 2d511c90c4c5f3bc0b5b514ceacd95160066acdf Author: Dridi Boukelmoune Date: Thu Apr 7 11:33:41 2016 +0200 Don't consider backends as resources Closes #1815 diff --git a/bin/varnishd/cache/cache_pipe.c b/bin/varnishd/cache/cache_pipe.c index 6507121..7f75a3a 100644 --- a/bin/varnishd/cache/cache_pipe.c +++ b/bin/varnishd/cache/cache_pipe.c @@ -116,7 +116,7 @@ PipeRequest(struct req *req, struct busyobj *bo) if (bo->director == NULL) { VSLb(bo->vsl, SLT_FetchError, "No backend"); pipecharge(req, &acct_pipe, NULL); - SES_Close(req->sp, SC_OVERLOAD); + SES_Close(req->sp, SC_TX_ERROR); return; } From phk at phk.freebsd.dk Thu Apr 7 09:22:21 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 07 Apr 2016 09:22:21 +0000 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: References: <59015.1459797849@critter.freebsd.dk> Message-ID: <18434.1460020941@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >On Mon, Apr 4, 2016 at 10:45 PM, Dridi Boukelmoune wrote: >>> Uhm, this seems to only work with GNU-make ? >> >> My bad, I assumed OK meant it had been tested. I just installed bmake >> on my machine, I'll have a look and see what's tying it to gmake. I'm >> getting tired of GNU extensions creeping everywhere :( > >I had a look at this, and it doesn't look like something that can be >fixed in a reasonable amount of time. The conditional logic can be >moved from the generated Makefile to the m4/autoconf parts. That >whould be make-friendly while still being gmake-only once enabled. > >Would that be OK? Have I mentioned how much I hate the entire autocrap thing for reducing portability instead of increasing it ? Does this LCOV stuff even work with LLVM or does it tie you to GCC ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Thu Apr 7 12:16:08 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Thu, 7 Apr 2016 13:16:08 +0100 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: <18434.1460020941@critter.freebsd.dk> References: <59015.1459797849@critter.freebsd.dk> <18434.1460020941@critter.freebsd.dk> Message-ID: If we inline CODE_COVERAGE_RULES it builds fine in FreeBSD. Haven't really tested trying to get coverage working there. Answering your question, this is tied to GCC, yes. On Thu, Apr 7, 2016 at 10:22 AM, Poul-Henning Kamp wrote: > -------- > In message EcgTyD0EraqmfLD+bBp77FQ at mail.gmail.com> > , Dridi Boukelmoune writes: > >On Mon, Apr 4, 2016 at 10:45 PM, Dridi Boukelmoune > wrote: > >>> Uhm, this seems to only work with GNU-make ? > >> > >> My bad, I assumed OK meant it had been tested. I just installed bmake > >> on my machine, I'll have a look and see what's tying it to gmake. I'm > >> getting tired of GNU extensions creeping everywhere :( > > > >I had a look at this, and it doesn't look like something that can be > >fixed in a reasonable amount of time. The conditional logic can be > >moved from the generated Makefile to the m4/autoconf parts. That > >whould be make-friendly while still being gmake-only once enabled. > > > >Would that be OK? > > Have I mentioned how much I hate the entire autocrap thing for > reducing portability instead of increasing it ? > > Does this LCOV stuff even work with LLVM or does it tie you to GCC ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at FreeBSD.org Thu Apr 7 22:13:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 08 Apr 2016 00:13:04 +0200 Subject: [master] dd0959d Give the vrt_ctx a distinct session pointer which will be available in both client and backend context. Message-ID: commit dd0959d62964a014911d1f1866b3c5b45ec5ccdc Author: Poul-Henning Kamp Date: Thu Apr 7 22:11:14 2016 +0000 Give the vrt_ctx a distinct session pointer which will be available in both client and backend context. diff --git a/include/vrt.h b/include/vrt.h index 427e45c..9963460 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -40,7 +40,11 @@ * Whenever something is deleted or changed in a way which is not * binary/load-time compatible, increment MAJOR version * - * 3.2 = vrt_backend grew .proxy_header field + * 3.2: + * vrt_backend grew .proxy_header field + * vrt_ctx grew .sp field. + * + * older version: * Bump VRT_MINOR_VERSION due to: * - VCL_ACL type added * Bump VRT_MAJOR_VERSION due to: @@ -108,6 +112,8 @@ struct vrt_ctx { struct vcl *vcl; struct ws *ws; + struct sess *sp; + struct req *req; struct http *http_req; struct http *http_req_top; From phk at FreeBSD.org Thu Apr 7 22:13:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 08 Apr 2016 00:13:04 +0200 Subject: [master] bc5db18 Fill vrt_ctx->sp Message-ID: commit bc5db18c80543536bb91a4cfc6ca7231ea294672 Author: Poul-Henning Kamp Date: Thu Apr 7 22:11:56 2016 +0000 Fill vrt_ctx->sp diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 26454a2..1afabb9 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -862,6 +862,7 @@ vcl_call_method(struct worker *wrk, struct req *req, struct busyobj *bo, ctx.http_req_top = req->top->http; ctx.http_resp = req->resp; ctx.req = req; + ctx.sp = req->sp; ctx.now = req->t_prev; ctx.ws = req->ws; } @@ -873,6 +874,7 @@ vcl_call_method(struct worker *wrk, struct req *req, struct busyobj *bo, ctx.http_bereq = bo->bereq; ctx.http_beresp = bo->beresp; ctx.bo = bo; + ctx.sp = bo->sp; ctx.now = bo->t_prev; ctx.ws = bo->ws; } From phk at FreeBSD.org Thu Apr 7 22:13:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 08 Apr 2016 00:13:04 +0200 Subject: [master] 44fc7c4 Make the four IP#s available also in backend vcl methods. Message-ID: commit 44fc7c413debeb19bb34ec3e6a452da15b23f956 Author: Poul-Henning Kamp Date: Thu Apr 7 22:12:16 2016 +0000 Make the four IP#s available also in backend vcl methods. diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 442bd47..10879c0 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -604,9 +604,8 @@ VRT_r_req_##field(VRT_CTX) \ struct suckaddr *sa; \ \ CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ - CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); \ - CHECK_OBJ_NOTNULL(ctx->req->sp, SESS_MAGIC); \ - AZ(SES_Get_##fld##_addr(ctx->req->sp, &sa)); \ + CHECK_OBJ_NOTNULL(ctx->sp, SESS_MAGIC); \ + AZ(SES_Get_##fld##_addr(ctx->sp, &sa)); \ return (sa); \ } diff --git a/bin/varnishtest/tests/o00003.vtc b/bin/varnishtest/tests/o00003.vtc new file mode 100644 index 0000000..d821ce7 --- /dev/null +++ b/bin/varnishtest/tests/o00003.vtc @@ -0,0 +1,24 @@ +varnishtest "VCL backend side access to IP#s" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -proto PROXY -vcl+backend { + sub vcl_backend_response { + set beresp.http.li = local.ip; + set beresp.http.ri = remote.ip; + set beresp.http.ci = client.ip; + set beresp.http.si = server.ip; + } +} -start + +client c1 { + send "PROXY TCP4 1.2.3.4 5.6.7.8 1111 5678\r\n" + txreq + rxresp + expect resp.http.li == ${v1_addr} + expect resp.http.ci == 1.2.3.4 + expect resp.http.si == 5.6.7.8 +} -run diff --git a/bin/varnishtest/tests/v00017.vtc b/bin/varnishtest/tests/v00017.vtc index e00ad4f..d3a0ccf 100644 --- a/bin/varnishtest/tests/v00017.vtc +++ b/bin/varnishtest/tests/v00017.vtc @@ -64,12 +64,6 @@ varnish v1 -vcl { sub vcl_recv { if (client.ip ~ a) { return(pass); } } } -varnish v1 -errvcl {'client.ip': Not available in method 'vcl_backend_response'} { - backend b { .host = "127.0.0.1"; } - sub vcl_recv { if (client.ip == "127.0.0.1") { return(pass); } } - sub vcl_backend_response { if (client.ip != "127.0.0.1") { return(retry); } } -} - varnish v1 -errvcl {Operator > not possible on IP} { backend b { .host = "127.0.0.1"; } sub vcl_recv { if (client.ip > "127.0.0.1") { return(pass); } } diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 5efcff1..61cfc4d 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -160,7 +160,7 @@ returns =( sp_variables = [ ('remote.ip', 'IP', - ( 'client',), + ( 'client', 'backend'), ( ), """ The IP address of the other end of the TCP connection. This can either be the clients IP, or the outgoing IP @@ -169,7 +169,7 @@ sp_variables = [ ), ('client.ip', 'IP', - ( 'client',), + ( 'client', 'backend'), ( ), """ The client's IP address. """ @@ -184,14 +184,14 @@ sp_variables = [ ), ('local.ip', 'IP', - ( 'client',), + ( 'client', 'backend'), ( ), """ The IP address of the local end of the TCP connection. """ ), ('server.ip', 'IP', - ( 'client',), + ( 'client', 'backend'), ( ), """ The IP address of the socket on which the client connection was received. From phk at FreeBSD.org Fri Apr 8 22:19:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Sat, 09 Apr 2016 00:19:05 +0200 Subject: [master] 2d51524 These programs should respect SIGHUP and die, rather pointlessly eat all CPU time. Message-ID: commit 2d51524ac36dd72efd9002381b711936dd5e305c Author: Poul-Henning Kamp Date: Fri Apr 8 22:16:34 2016 +0000 These programs should respect SIGHUP and die, rather pointlessly eat all CPU time. diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 9eab9bf..f279dc3 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -294,7 +294,7 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], static int __match_proto__(VUT_cb_f) sighup(void) { - return (1); + exit(1); } static void * diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index f56b9a7..9c555cc 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -183,7 +183,7 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], static int __match_proto__(VUT_cb_f) sighup(void) { - return (1); + exit (1); } static void From hermunn at varnish-software.com Mon Apr 11 11:38:03 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 11 Apr 2016 13:38:03 +0200 Subject: [master] fce2ff2 Handle return(synth()) in vcl_pipe Message-ID: commit fce2ff230ebb20f04c8a42e14792f9274b8fd53b Author: P?l Hermunn Johansen Date: Tue Apr 5 11:39:30 2016 +0200 Handle return(synth()) in vcl_pipe Instead of panicking on a return(synth()) in vcl_pipe, we just proceed to a normal synth. Test by Dridi Fixes #1890 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 321a47a..889cf40 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -569,6 +569,7 @@ static enum req_fsm_nxt cnt_pipe(struct worker *wrk, struct req *req) { struct busyobj *bo; + enum req_fsm_nxt nxt; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); @@ -588,15 +589,22 @@ cnt_pipe(struct worker *wrk, struct req *req) VCL_pipe_method(req->vcl, wrk, req, bo, NULL); - if (wrk->handling == VCL_RET_SYNTH) - INCOMPL(); - assert(wrk->handling == VCL_RET_PIPE); - - SES_Close(req->sp, VDI_Http1Pipe(req, bo)); + switch (wrk->handling) { + case VCL_RET_SYNTH: + req->req_step = R_STP_SYNTH; + nxt = REQ_FSM_MORE; + break; + case VCL_RET_PIPE: + SES_Close(req->sp, VDI_Http1Pipe(req, bo)); + nxt = REQ_FSM_DONE; + break; + default: + WRONG("Illegal return from vcl_pipe{}"); + } http_Teardown(bo->bereq); VBO_ReleaseBusyObj(wrk, &bo); THR_SetBusyobj(NULL); - return (REQ_FSM_DONE); + return (nxt); } /*-------------------------------------------------------------------- diff --git a/bin/varnishtest/tests/r01890.vtc b/bin/varnishtest/tests/r01890.vtc new file mode 100644 index 0000000..6403c13 --- /dev/null +++ b/bin/varnishtest/tests/r01890.vtc @@ -0,0 +1,16 @@ +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + sub vcl_pipe { + return (synth(401)); + } +} -start + +client c1 { + txreq -req PROPFIND + rxresp + expect resp.status == 401 +} -run From dridi at varni.sh Tue Apr 12 07:58:58 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 12 Apr 2016 09:58:58 +0200 Subject: [master] a65aa02 Enable optional lcov support In-Reply-To: References: <59015.1459797849@critter.freebsd.dk> <18434.1460020941@critter.freebsd.dk> Message-ID: > Answering your question, this is tied to GCC, yes. That's not accurate, it is tied to gcov's file format. I have successfully used lcov with clang in the past. But you might run into issues when gcov changes its output file format [1] because lcov and clang will catch up at different paces or need additional flags. > Have I mentioned how much I hate the entire autocrap thing for > reducing portability instead of increasing it ? The lcov plumbing I imported [2] are not even trying to be portable, despite a fancy generic name of "ax_code_coverage" that only supports lcov, gcc and gmake. I have looked closer at autotools after this lcov fiasco and came to the conclusion that we are doing it wrong, probably like most people (I'm looking at ax_code_coverage.) For instance, we build libraries using libtool, and invoking a one-liner LT_INIT would be enough to check a lot of Varnish's toolchain because libtool requires at least C support. I'm not saying that after reading some docs and macros implementations that I suddenly find autotools great, but used right and if the planets align (read not-recent-gnu-linux platforms) it could get less in the way and possibly even become useful. That's a topic for another thread. My last question was about reverting the commit. Since it wasn't answered and since ax_code_coverage is still in the source tree, should I assume it doesn't harm when disabled and can be left alone? Dridi [1] http://clang-developers.42468.n3.nabble.com/Code-coverage-on-clang-td4033066.html [2] http://www.gnu.org/software/autoconf-archive/ax_code_coverage.html From fgsch at lodoss.net Tue Apr 12 22:22:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 13 Apr 2016 00:22:05 +0200 Subject: [master] 90afdda Start the return values at 1 Message-ID: commit 90afddae68d0811a59ba1507ff51908cc094feff Author: Federico G. Schwindt Date: Tue Apr 12 23:20:45 2016 +0100 Start the return values at 1 Use 0 to indicate it has not been set instead of displaying abandon. diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 61cfc4d..e70068c 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -1055,7 +1055,7 @@ fo.write("\n" + tbl40("#define VCL_MET_MASK", "0x%x\n" % ((1 << n) - 1))) fo.write("\n/* VCL Returns */\n") -n = 0 +n = 1 l = list(rets.keys()) l.sort() for i in l: From lkarsten at varnish-software.com Thu Apr 14 07:06:04 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Thu, 14 Apr 2016 09:06:04 +0200 Subject: [master] b318692 Extend parts on heredoc in vcl.inline. Message-ID: commit b3186921c80c6ed9423f7bdd394328e450dd8d3b Author: Lasse Karstensen Date: Thu Apr 14 08:52:03 2016 +0200 Extend parts on heredoc in vcl.inline. diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst index 17e86de..fa2fc76 100644 --- a/doc/sphinx/reference/varnish-cli.rst +++ b/doc/sphinx/reference/varnish-cli.rst @@ -51,15 +51,18 @@ Syntax ------ Commands are usually terminated with a newline. Long command can be -entered using sh style *here documents*. The format of here-documents -is:: +entered using sh style *here documents* (colloquially heredocs). +The format of here-documents is:: << word here document word +.. :ref:`heredoc` + *word* can be any continuous string chosen to make sure it doesn't -appear naturally in the following *here document*. +appear naturally in the following *here document*. Commonly "EOF" +is used. When using the here document style of input there are no restrictions on length. When using newline-terminated commands maximum length is @@ -105,7 +108,7 @@ vcl.load [auto|cold|warm] Compile and load the VCL file under the name provided. vcl.inline [auto|cold|warm] - Compile and load the VCL data under the name provided. + Compile and load the VCL data under the name provided. Multi-line VCL can be input using :ref:`heredoc`. vcl.use Switch to the named configuration immediately. From lkarsten at varnish-software.com Thu Apr 14 07:06:04 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Thu, 14 Apr 2016 09:06:04 +0200 Subject: [master] beac5f6 Remove unexplained acronym. Message-ID: commit beac5f65084cd8accfc71a7302f7c92d246e7026 Author: Lasse Karstensen Date: Thu Apr 14 08:52:45 2016 +0200 Remove unexplained acronym. I don't think we can assume that everyone knows that PSK is, and with the extensive acronym soup in use in Varnish it is hard for the user to know if we have overloaded it. diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst index fa2fc76..203ac60 100644 --- a/doc/sphinx/reference/varnish-cli.rst +++ b/doc/sphinx/reference/varnish-cli.rst @@ -242,8 +242,8 @@ the functions Varnish code uses to read and write CLI response. .. _ref_psk_auth: -How -S/PSK Authentication Works -------------------------------- +Authentication with -S +---------------------- If the -S secret-file is given as argument to varnishd, all network CLI connections must authenticate, by proving they know the contents From fgsch at lodoss.net Thu Apr 14 10:22:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 14 Apr 2016 12:22:04 +0200 Subject: [master] 84dc0ca Improve examples Message-ID: commit 84dc0ca9d378203f406de8464f55acb30f2b4d94 Author: Federico G. Schwindt Date: Wed Apr 13 23:45:07 2016 +0100 Improve examples diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst index 203ac60..210d3c8 100644 --- a/doc/sphinx/reference/varnish-cli.rst +++ b/doc/sphinx/reference/varnish-cli.rst @@ -321,16 +321,15 @@ secret file, and the challenge string. EXAMPLES ======== -Simple example: All requests where req.url exactly matches the string -/news are banned from the cache:: +Ban all requests where req.url exactly matches the string /news:: - req.url == "/news" + ban req.url == "/news" -Example: Ban all documents where the serving host is "example.com" -or "www.example.com", and where the Set-Cookie header received from -the backend contains "USERID=1663":: +Ban all documents where the serving host is "example.com" or +"www.example.com", and where the Set-Cookie header received from the +backend contains "USERID=1663":: - req.http.host ~ "^(?i)(www\.)example.com$" && obj.http.set-cookie ~ "USERID=1663" + ban req.http.host ~ "^(?i)(www\\.)example.com$" && obj.http.set-cookie ~ "USERID=1663" AUTHORS ======= From fgsch at lodoss.net Thu Apr 14 10:24:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 14 Apr 2016 12:24:05 +0200 Subject: [master] dc360a4 Correct prompt when using heredoc Message-ID: commit dc360a451d54696e5b52ac254a5b5ff1b24c909a Author: Federico G. Schwindt Date: Wed Apr 13 00:46:59 2016 +0100 Correct prompt when using heredoc Tested by scn. Fixes #1899. diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index 621d696..9bd42f4 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -191,7 +191,7 @@ static void send_line(char *l) cli_write(_line_sock, l); cli_write(_line_sock, "\n"); add_history(l); - rl_callback_handler_install("varnish> ", send_line); + rl_callback_handler_install("", send_line); } else { RL_EXIT(0); } @@ -246,7 +246,7 @@ interactive(int sock) unsigned u, status; _line_sock = sock; rl_already_prompted = 1; - rl_callback_handler_install("varnish> ", send_line); + rl_callback_handler_install("", send_line); rl_attempted_completion_function = varnishadm_completion; fds[0].fd = sock; @@ -309,6 +309,7 @@ interactive(int sock) free(answer); answer = NULL; } + rl_set_prompt("varnish> "); rl_forced_update_display(); } if (fds[1].revents & POLLIN) { From fgsch at lodoss.net Thu Apr 14 23:06:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 15 Apr 2016 01:06:05 +0200 Subject: [master] a5b5376 Correct backend expression and other improvements Message-ID: commit a5b5376ad4c38e2b85d2e622bf78740e3af87cb6 Author: Federico G. Schwindt Date: Fri Apr 15 00:00:59 2016 +0100 Correct backend expression and other improvements diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst index 210d3c8..2ae20e5 100644 --- a/doc/sphinx/reference/varnish-cli.rst +++ b/doc/sphinx/reference/varnish-cli.rst @@ -46,23 +46,21 @@ available. In debug mode (-d) the CLI will be in the foreground, with varnishd will connect back to a listening service *pushing* the CLI to that service. Please see :ref:`varnishd(1)` for details. +.. _ref_syntax: Syntax ------ Commands are usually terminated with a newline. Long command can be -entered using sh style *here documents* (colloquially heredocs). -The format of here-documents is:: +entered using shell style *here document* (here-document or heredoc). +The format of here document is:: << word here document word -.. :ref:`heredoc` - -*word* can be any continuous string chosen to make sure it doesn't -appear naturally in the following *here document*. Commonly "EOF" -is used. +*word* can be any continuous string chosen to make sure it doesn't appear +naturally in the following *here document*. Often EOF or END is used. When using the here document style of input there are no restrictions on length. When using newline-terminated commands maximum length is @@ -108,7 +106,8 @@ vcl.load [auto|cold|warm] Compile and load the VCL file under the name provided. vcl.inline [auto|cold|warm] - Compile and load the VCL data under the name provided. Multi-line VCL can be input using :ref:`heredoc`. + Compile and load the VCL data under the name provided. + Multi-line VCL can be input using the here document :ref:`ref_syntax`. vcl.use Switch to the named configuration immediately. @@ -173,14 +172,16 @@ ban.list Backend Expression ------------------ -A backend expression can be a backend name or a combination of backend -name, IP address and port in "name(IP address:port)" format. All fields -are optional. If no exact matching backend is found, partial matching -will be attempted based on the provided name, IP address and port fields. +A backend expression can be a backend name or a combination of a VCL name +and backend name in "VCL.backend" format. If the VCL name is omitted, +the active VCL is assumed. Partial matching on the backend name, VCL +name or both is possible using shell-style wilcards, i.e. asterisk (*), +question mark (?) and brackets ([]). Examples:: backend.list def* + backend.list b*.def* backend.set_health default sick backend.set_health def* healthy backend.set_health * auto @@ -268,10 +269,13 @@ An authenticated session looks like this:: Authentication required. auth 455ce847f0073c7ab3b1465f74507b75d3dc064c1e7de3b71e00de9092fdc89a - 200 193 + 200 279 ----------------------------- - Varnish HTTP accelerator CLI. + Varnish Cache CLI 1.0 ----------------------------- + Linux,4.4.0-1-amd64,x86_64,-jnone,-smalloc,-smalloc,-hcritbit + varnish-trunk revision dc360a4 + Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. @@ -321,6 +325,17 @@ secret file, and the challenge string. EXAMPLES ======== +Load a multi-line VCL using shell style *here document*:: + + vcl.inline example << EOF + vcl 4.0; + + backend www { + .host = "127.0.0.1"; + .port = "8080"; + } + EOF + Ban all requests where req.url exactly matches the string /news:: ban req.url == "/news" From nils.goroll at uplex.de Fri Apr 15 08:23:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 15 Apr 2016 10:23:05 +0200 Subject: [master] b045f92 nit-picking: we just checked for t1 == t0 Message-ID: commit b045f92903f0df0af6d64a5cbcaacf96cfa8a86a Author: Nils Goroll Date: Fri Apr 15 10:19:56 2016 +0200 nit-picking: we just checked for t1 == t0 diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c index 7494ee0..2565a68 100644 --- a/bin/varnishd/cache/cache_ban.c +++ b/bin/varnishd/cache/cache_ban.c @@ -264,7 +264,7 @@ BAN_FindBan(double t0) t1 = ban_time(b->spec); if (t1 == t0) return (b); - if (t1 <= t0) + if (t1 < t0) break; } return (NULL); From nils.goroll at uplex.de Fri Apr 15 16:05:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 15 Apr 2016 18:05:05 +0200 Subject: [master] 5f0dc30 improve ban lurker parameter documentation Message-ID: commit 5f0dc307550e09be39308ffbe7778bbb10c6faf4 Author: Nils Goroll Date: Fri Apr 15 18:03:34 2016 +0200 improve ban lurker parameter documentation diff --git a/include/tbl/params.h b/include/tbl/params.h index 402497f..2d6446a 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -137,8 +137,10 @@ PARAM( /* s-text */ "The ban lurker will ignore bans until they are this old. " "When a ban is added, the active traffic will be tested against it " - "as part of object lookup. This parameter " - "holds the ban-lurker off, until the rush is over.", + "as part of object lookup. Because many applications issue bans in " + "bursts, this parameter holds the ban-lurker off until the rush is " + "over.\n" + "This should be set to the approximate time which a ban-burst takes.", /* l-text */ "", /* func */ NULL ) @@ -169,7 +171,7 @@ PARAM( /* flags */ 0, /* s-text */ "How long the ban lurker sleeps after examining ${ban_lurker_batch} " - "objects." + "objects and when it detects lock contention on object heads." " Use this to pace the ban-lurker if it eats too many resources.\n" "A value of zero will disable the ban lurker entirely.", /* l-text */ "", From phk at FreeBSD.org Mon Apr 18 07:56:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 18 Apr 2016 09:56:04 +0200 Subject: [master] 649d9f4 Add some oc->refcnt paranoid asserts. Message-ID: commit 649d9f4abe0fd26f9636369b9c3f987407f9d938 Author: Poul-Henning Kamp Date: Mon Apr 18 07:54:42 2016 +0000 Add some oc->refcnt paranoid asserts. diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c index 2565a68..6ade8d4 100644 --- a/bin/varnishd/cache/cache_ban.c +++ b/bin/varnishd/cache/cache_ban.c @@ -530,6 +530,7 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req) CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); Lck_AssertHeld(&oc->objhead->mtx); + assert(oc->refcnt > 0); vsl = req->vsl; @@ -597,8 +598,6 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req) } else { VSLb(vsl, SLT_ExpBan, "%u banned lookup", ObjGetXID(wrk, oc)); VSC_C_main->bans_obj_killed++; - oc->flags |= OC_F_DYING; - EXP_Poke(oc); return (1); } } diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index a6c2d3f..9d474ba 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -83,6 +83,7 @@ static void exp_mail_it(struct objcore *oc, uint8_t cmds) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + assert(oc->refcnt > 0); Lck_Lock(&exphdl->mtx); if (!(oc->exp_flags & OC_EF_POSTED)) { @@ -174,20 +175,21 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags) CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + assert(oc->refcnt > 0); - VSLb(&ep->vsl, SLT_ExpKill, "EXP_Inbox p=%p e=%.9f f=0x%x", oc, - oc->timer_when, oc->flags); + VSLb(&ep->vsl, SLT_ExpKill, "EXP_Inbox flg=%x p=%p e=%.9f f=0x%x", + flags, oc, oc->timer_when, oc->flags); if (oc->flags & OC_F_DYING) { - oc->exp_flags &= ~OC_EF_EXP; - VSLb(&ep->vsl, SLT_ExpKill, "EXP_Kill p=%p e=%.9f f=0x%x", oc, - oc->timer_when, oc->flags); if (!(flags & OC_EF_INSERT)) { assert(oc->timer_idx != BINHEAP_NOIDX); binheap_delete(ep->heap, oc->timer_idx); } assert(oc->timer_idx == BINHEAP_NOIDX); ObjSendEvent(ep->wrk, oc, OEV_REMOVE); + oc->exp_flags &= ~OC_EF_EXP; + assert(oc->refcnt > 0); + AZ(oc->exp_flags); (void)HSH_DerefObjCore(ep->wrk, &oc); return; } @@ -304,7 +306,9 @@ exp_thread(struct worker *wrk, void *priv) Lck_Lock(&ep->mtx); oc = VSTAILQ_FIRST(&ep->inbox); + CHECK_OBJ_ORNULL(oc, OBJCORE_MAGIC); if (oc != NULL) { + assert(oc->refcnt >= 1); VSTAILQ_REMOVE(&ep->inbox, oc, objcore, exp_list); VSC_C_main->exp_received++; tnext = 0; diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index fc5c368..1fb627c 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -385,6 +385,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp, assert(oh->refcnt > 1); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); assert(oc->objhead == oh); + assert(oc->refcnt > 0); if (oc->flags & OC_F_DYING) continue; @@ -408,8 +409,11 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp, if (oc->ttl <= 0.) continue; - if (BAN_CheckObject(wrk, oc, req)) + if (BAN_CheckObject(wrk, oc, req)) { + oc->flags |= OC_F_DYING; + EXP_Poke(oc); continue; + } if (ObjHasAttr(wrk, oc, OA_VARY)) { vary = ObjGetAttr(wrk, oc, OA_VARY, NULL); From varnish-commit at varnish-cache.org Mon Apr 18 17:10:00 2016 From: varnish-commit at varnish-cache.org (snlgqkifk) Date: Tue, 19 Apr 2016 01:10:00 +0800 Subject: varnish-commit:1ctwg Message-ID: <20160419011008241885@zlbgbvady.org> ???????? ???????? 2016?4?26-27? ?? ???????? ???????????????????????????????????????? ???????? ???? + ???? + ???? +???? + ???? ???????? 5200?/????????????????? ????????021-31006787 18917870808 ????????jsqyxuxin at vip.163.com QQ/??:320588808 ????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ????? ???????????????????????????????????????????????????? 1?????????????????????,???????,????? A????? B?????????????? C???????? ????: ??? ????????? ????? ????? ????? ??????? ???? ??????? ????? ???? ?? ???? ????????????? ?????????????? ??????? ?????????? ??????????? ?????????????? ?? ?? ???????? ??????? ??????????????? ??????????SMART???? ????????? ???????????? ?? ?????????? ????? ??????????????? ????????????????????????? ???? ?????????WBS ?RAM ????????? ??????????WBS? WBS????? WBS?????? ?????WBS?????????? ????????(RAM) ?????????????WBS?RAM ? ?? ?? ??????????3??? ????? ??????????????? ????????WBS?RAM ???????????????????? ????????????????PDM?ADM?GERT?Network Board?????? ??????? ???????????? ????????????????????? ???????????CPM????????? ????? ??????? ???????????? ???? ????---??????? ????---??????????????? ?????????9????CPM????10???? ??CPM???????????? ?????? ?????? ????????????? ?????? ???????? ???? ?????????4????WBS?RAM?CPM?????? ??????? ??????? ??????????? ?????? ?????? ????????????? ?????[???] ????? * ????????????? ???? * ??????????????? * ????????????? * ????????????? * ??????????????????????????? * ?????????????????????????? * ???????????????????? * ??????????????????????? ????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????? ??????BASF???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????3 M (??)??????????????????????BP?????MBA???????? ????????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????(??)???????????????????????????????????????????????????????????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-commit at varnish-cache.org Mon Apr 18 17:41:11 2016 From: varnish-commit at varnish-cache.org (ugfyavlp) Date: Tue, 19 Apr 2016 01:41:11 +0800 Subject: =?utf-8?B?dmFybmlzaC1jb21taXQ65oub55So5b6F5bKX44CB?= =?utf-8?B?5YaF6YCA44CB5YGc6Jaq55WZ6IGM55qE5Lq65ZGY77yM5bqU5rOo5oSP5ZOq5Lqb57uG6IqC6Zeu6aKY?= =?utf-8?B?77yf?= Message-ID: <20160419014118812103@iph.com> varnish-commit: ??! 1.???????????????????????????? 2.?????????????????? 3.???????????????????????? 4.?????????????????????????? 5.?????????????? 6.?????????????????????????? 7.?????????????????????????? 8.??????????????????? 9.??????????????????????? ??????????????????? ???????????? 2016/4/19 ???1:41:17 ??? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ??????????????????????.docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 27448 bytes Desc: not available URL: From fgsch at lodoss.net Tue Apr 19 12:20:07 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 19 Apr 2016 14:20:07 +0200 Subject: [master] 8602c8a Revert recent heredoc handling change for now Message-ID: commit 8602c8a1a34931c1f5105fe360185cc61db7c415 Author: Federico G. Schwindt Date: Tue Apr 19 13:14:01 2016 +0100 Revert recent heredoc handling change for now Older libedit versions lack rl_set_prompt() and in some cases we ended up with an empty prompt. diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index 9bd42f4..621d696 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -191,7 +191,7 @@ static void send_line(char *l) cli_write(_line_sock, l); cli_write(_line_sock, "\n"); add_history(l); - rl_callback_handler_install("", send_line); + rl_callback_handler_install("varnish> ", send_line); } else { RL_EXIT(0); } @@ -246,7 +246,7 @@ interactive(int sock) unsigned u, status; _line_sock = sock; rl_already_prompted = 1; - rl_callback_handler_install("", send_line); + rl_callback_handler_install("varnish> ", send_line); rl_attempted_completion_function = varnishadm_completion; fds[0].fd = sock; @@ -309,7 +309,6 @@ interactive(int sock) free(answer); answer = NULL; } - rl_set_prompt("varnish> "); rl_forced_update_display(); } if (fds[1].revents & POLLIN) { From fgsch at lodoss.net Tue Apr 19 12:20:07 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 19 Apr 2016 14:20:07 +0200 Subject: [master] 0cc4adf Do not add empty strings to the cli history Message-ID: commit 0cc4adf09ba64928a7fbb94f6c246533a2996427 Author: Federico G. Schwindt Date: Tue Apr 19 13:16:23 2016 +0100 Do not add empty strings to the cli history diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index 621d696..e426f0b 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -190,7 +190,8 @@ static void send_line(char *l) if (l) { cli_write(_line_sock, l); cli_write(_line_sock, "\n"); - add_history(l); + if (*l) + add_history(l); rl_callback_handler_install("varnish> ", send_line); } else { RL_EXIT(0); From fgsch at lodoss.net Tue Apr 19 15:52:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 19 Apr 2016 17:52:05 +0200 Subject: [master] ddcaf41 Revert "Enable optional lcov support" Message-ID: commit ddcaf41f5e96a7fcf73da72f4a21df277527deed Author: Federico G. Schwindt Date: Tue Apr 19 16:44:15 2016 +0100 Revert "Enable optional lcov support" This made gmake mandatory which is not ideal. The code will be fixed outside the tree and added back once it's free from any GNUism. Discussed with dridi at . diff --git a/.gitignore b/.gitignore index e11a81b..fb9cde6 100644 --- a/.gitignore +++ b/.gitignore @@ -112,9 +112,3 @@ cscope.*out /bin/varnishtest/tests/*.log /bin/varnishtest/tests/*.log-t /bin/varnishtest/test-suite.log - -# Code coverage with lcov/gcov -*.gcno -*.gcda -*.info -/lcov/ diff --git a/Makefile.am b/Makefile.am index 98b3091..b947ceb 100644 --- a/Makefile.am +++ b/Makefile.am @@ -25,17 +25,6 @@ distcleancheck_listfiles = \ find . -type f -exec sh -c 'test -f $(srcdir)/$$1 || echo $$1' \ sh '{}' ';' -# lcov support -CODE_COVERAGE_OUTPUT_DIRECTORY = lcov -CODE_COVERAGE_IGNORE_PATTERN = "/usr/*" -CODE_COVERAGE_GENHTML_OPTIONS = --prefix $(abs_top_srcdir) - - at CODE_COVERAGE_RULES@ - -if CODE_COVERAGE_ENABLED -lcov: check-code-coverage -endif - # XXX: This is a hack to ensure we have a built source tree when # running make dist If we had used non-recursive make we could have # solved it better, but we don't, so use this at least for now. diff --git a/autogen.des b/autogen.des index 9c81a91..3cfc1fa 100755 --- a/autogen.des +++ b/autogen.des @@ -1,10 +1,6 @@ #!/bin/sh # # Use this when doing code development -# -# Arguments to this script are passed to the resulting ./configure execution. -# You may pass `--enable-code-coverage` to measure code coverage with lcov for -# example. set -ex diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 66e114d..9d8c0b5 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -4,10 +4,6 @@ VTC_LOG_COMPILER = ./varnishtest -v -i TEST_EXTENSIONS = .vtc TESTS = @VTC_TESTS@ -if CODE_COVERAGE_ENABLED -VTC_LOG_COMPILER += -p cli_timeout=120 -endif - # Make sure we run check-local first check: check-am check-local check-am: check-local diff --git a/configure.ac b/configure.ac index edd47c5..edba56a 100644 --- a/configure.ac +++ b/configure.ac @@ -672,12 +672,6 @@ AC_SUBST(VTC_TESTS) # Make sure this include dir exists AC_CONFIG_COMMANDS([mkdir], [$MKDIR_P doc/sphinx/include]) -# lcov support -AX_CODE_COVERAGE -CPPFLAGS="$CPPFLAGS \$(CODE_COVERAGE_CPPFLAGS)" -CFLAGS="$CFLAGS \$(CODE_COVERAGE_CFLAGS)" -LDFLAGS="$LDFLAGS \$(CODE_COVERAGE_LDFLAGS)" - # Generate output AC_CONFIG_FILES([ Makefile diff --git a/include/vbm_test.c b/include/vbm_test.c index ba60ffb..f680597 100644 --- a/include/vbm_test.c +++ b/include/vbm_test.c @@ -88,7 +88,5 @@ main(void) vbit_destroy(vb); - (void)obits; /* needed when NDEBUG is defined */ - return (0); } diff --git a/m4/ax_code_coverage.m4 b/m4/ax_code_coverage.m4 deleted file mode 100644 index 6c985eb..0000000 --- a/m4/ax_code_coverage.m4 +++ /dev/null @@ -1,273 +0,0 @@ -# =========================================================================== -# http://www.gnu.org/software/autoconf-archive/ax_code_coverage.html -# =========================================================================== -# -# SYNOPSIS -# -# AX_CODE_COVERAGE() -# -# DESCRIPTION -# -# Defines CODE_COVERAGE_CPPFLAGS, CODE_COVERAGE_CFLAGS, -# CODE_COVERAGE_CXXFLAGS and CODE_COVERAGE_LDFLAGS which should be -# included in the CPPFLAGS, CFLAGS CXXFLAGS and LIBS/LDFLAGS variables of -# every build target (program or library) which should be built with code -# coverage support. Also defines CODE_COVERAGE_RULES which should be -# substituted in your Makefile; and $enable_code_coverage which can be -# used in subsequent configure output. CODE_COVERAGE_ENABLED is defined -# and substituted, and corresponds to the value of the -# --enable-code-coverage option, which defaults to being disabled. -# -# Test also for gcov program and create GCOV variable that could be -# substituted. -# -# Note that all optimisation flags in CFLAGS must be disabled when code -# coverage is enabled. -# -# Usage example: -# -# configure.ac: -# -# AX_CODE_COVERAGE -# -# Makefile.am: -# -# @CODE_COVERAGE_RULES@ -# my_program_LIBS = ... $(CODE_COVERAGE_LDFLAGS) ... -# my_program_CPPFLAGS = ... $(CODE_COVERAGE_CPPFLAGS) ... -# my_program_CFLAGS = ... $(CODE_COVERAGE_CFLAGS) ... -# my_program_CXXFLAGS = ... $(CODE_COVERAGE_CXXFLAGS) ... -# -# This results in a "check-code-coverage" rule being added to any -# Makefile.am which includes "@CODE_COVERAGE_RULES@" (assuming the module -# has been configured with --enable-code-coverage). Running `make -# check-code-coverage` in that directory will run the module's test suite -# (`make check`) and build a code coverage report detailing the code which -# was touched, then print the URI for the report. -# -# This code was derived from Makefile.decl in GLib, originally licenced -# under LGPLv2.1+. -# -# LICENSE -# -# Copyright (c) 2012, 2016 Philip Withnall -# Copyright (c) 2012 Xan Lopez -# Copyright (c) 2012 Christian Persch -# Copyright (c) 2012 Paolo Borelli -# Copyright (c) 2012 Dan Winship -# Copyright (c) 2015 Bastien ROUCARIES -# -# This library is free software; you can redistribute it and/or modify it -# under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation; either version 2.1 of the License, or (at -# your option) any later version. -# -# This library is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser -# General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see . - -#serial 15 - -AC_DEFUN([AX_CODE_COVERAGE],[ - dnl Check for --enable-code-coverage - AC_REQUIRE([AC_PROG_SED]) - - # allow to override gcov location - AC_ARG_WITH([gcov], - [AS_HELP_STRING([--with-gcov[=GCOV]], [use given GCOV for coverage (GCOV=gcov).])], - [_AX_CODE_COVERAGE_GCOV_PROG_WITH=$with_gcov], - [_AX_CODE_COVERAGE_GCOV_PROG_WITH=gcov]) - - AC_MSG_CHECKING([whether to build with code coverage support]) - AC_ARG_ENABLE([code-coverage], - AS_HELP_STRING([--enable-code-coverage], - [Whether to enable code coverage support]),, - enable_code_coverage=no) - - AM_CONDITIONAL([CODE_COVERAGE_ENABLED], [test x$enable_code_coverage = xyes]) - AC_SUBST([CODE_COVERAGE_ENABLED], [$enable_code_coverage]) - AC_MSG_RESULT($enable_code_coverage) - - AS_IF([ test "$enable_code_coverage" = "yes" ], [ - # check for gcov - AC_CHECK_TOOL([GCOV], - [$_AX_CODE_COVERAGE_GCOV_PROG_WITH], - [:]) - AS_IF([test "X$GCOV" = "X:"], - [AC_MSG_ERROR([gcov is needed to do coverage])]) - AC_SUBST([GCOV]) - - dnl Check if gcc is being used - AS_IF([ test "$GCC" = "no" ], [ - AC_MSG_ERROR([not compiling with gcc, which is required for gcov code coverage]) - ]) - - # List of supported lcov versions. - lcov_version_list="1.6 1.7 1.8 1.9 1.10 1.11 1.12" - - AC_CHECK_PROG([LCOV], [lcov], [lcov]) - AC_CHECK_PROG([GENHTML], [genhtml], [genhtml]) - - AS_IF([ test "$LCOV" ], [ - AC_CACHE_CHECK([for lcov version], ax_cv_lcov_version, [ - ax_cv_lcov_version=invalid - lcov_version=`$LCOV -v 2>/dev/null | $SED -e 's/^.* //'` - for lcov_check_version in $lcov_version_list; do - if test "$lcov_version" = "$lcov_check_version"; then - ax_cv_lcov_version="$lcov_check_version (ok)" - fi - done - ]) - ], [ - lcov_msg="To enable code coverage reporting you must have one of the following lcov versions installed: $lcov_version_list" - AC_MSG_ERROR([$lcov_msg]) - ]) - - case $ax_cv_lcov_version in - ""|invalid[)] - lcov_msg="You must have one of the following versions of lcov: $lcov_version_list (found: $lcov_version)." - AC_MSG_ERROR([$lcov_msg]) - LCOV="exit 0;" - ;; - esac - - AS_IF([ test -z "$GENHTML" ], [ - AC_MSG_ERROR([Could not find genhtml from the lcov package]) - ]) - - dnl Build the code coverage flags - CODE_COVERAGE_CPPFLAGS="-DNDEBUG" - CODE_COVERAGE_CFLAGS="-O0 -g -fprofile-arcs -ftest-coverage" - CODE_COVERAGE_CXXFLAGS="-O0 -g -fprofile-arcs -ftest-coverage" - CODE_COVERAGE_LDFLAGS="-lgcov" - - AC_SUBST([CODE_COVERAGE_CPPFLAGS]) - AC_SUBST([CODE_COVERAGE_CFLAGS]) - AC_SUBST([CODE_COVERAGE_CXXFLAGS]) - AC_SUBST([CODE_COVERAGE_LDFLAGS]) - ]) - -[CODE_COVERAGE_RULES=' -# Code coverage -# -# Optional: -# - CODE_COVERAGE_DIRECTORY: Top-level directory for code coverage reporting. -# Multiple directories may be specified, separated by whitespace. -# (Default: $(top_builddir)) -# - CODE_COVERAGE_OUTPUT_FILE: Filename and path for the .info file generated -# by lcov for code coverage. (Default: -# $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage.info) -# - CODE_COVERAGE_OUTPUT_DIRECTORY: Directory for generated code coverage -# reports to be created. (Default: -# $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage) -# - CODE_COVERAGE_BRANCH_COVERAGE: Set to 1 to enforce branch coverage, -# set to 0 to disable it and leave empty to stay with the default. -# (Default: empty) -# - CODE_COVERAGE_LCOV_SHOPTS_DEFAULT: Extra options shared between both lcov -# instances. (Default: based on $CODE_COVERAGE_BRANCH_COVERAGE) -# - CODE_COVERAGE_LCOV_SHOPTS: Extra options to shared between both lcov -# instances. (Default: $CODE_COVERAGE_LCOV_SHOPTS_DEFAULT) -# - CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH: --gcov-tool pathtogcov -# - CODE_COVERAGE_LCOV_OPTIONS_DEFAULT: Extra options to pass to the -# collecting lcov instance. (Default: $CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH) -# - CODE_COVERAGE_LCOV_OPTIONS: Extra options to pass to the collecting lcov -# instance. (Default: $CODE_COVERAGE_LCOV_OPTIONS_DEFAULT) -# - CODE_COVERAGE_LCOV_RMOPTS_DEFAULT: Extra options to pass to the filtering -# lcov instance. (Default: empty) -# - CODE_COVERAGE_LCOV_RMOPTS: Extra options to pass to the filtering lcov -# instance. (Default: $CODE_COVERAGE_LCOV_RMOPTS_DEFAULT) -# - CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT: Extra options to pass to the -# genhtml instance. (Default: based on $CODE_COVERAGE_BRANCH_COVERAGE) -# - CODE_COVERAGE_GENHTML_OPTIONS: Extra options to pass to the genhtml -# instance. (Default: $CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT) -# - CODE_COVERAGE_IGNORE_PATTERN: Extra glob pattern of files to ignore -# -# The generated report will be titled using the $(PACKAGE_NAME) and -# $(PACKAGE_VERSION). In order to add the current git hash to the title, -# use the git-version-gen script, available online. - -# Optional variables -CODE_COVERAGE_DIRECTORY ?= $(top_builddir) -CODE_COVERAGE_OUTPUT_FILE ?= $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage.info -CODE_COVERAGE_OUTPUT_DIRECTORY ?= $(PACKAGE_NAME)-$(PACKAGE_VERSION)-coverage -CODE_COVERAGE_BRANCH_COVERAGE ?= -CODE_COVERAGE_LCOV_SHOPTS_DEFAULT ?= $(if $(CODE_COVERAGE_BRANCH_COVERAGE),\ ---rc lcov_branch_coverage=$(CODE_COVERAGE_BRANCH_COVERAGE)) -CODE_COVERAGE_LCOV_SHOPTS ?= $(CODE_COVERAGE_LCOV_SHOPTS_DEFAULT) -CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH ?= --gcov-tool "$(GCOV)" -CODE_COVERAGE_LCOV_OPTIONS_DEFAULT ?= $(CODE_COVERAGE_LCOV_OPTIONS_GCOVPATH) -CODE_COVERAGE_LCOV_OPTIONS ?= $(CODE_COVERAGE_LCOV_OPTIONS_DEFAULT) -CODE_COVERAGE_LCOV_RMOPTS_DEFAULT ?= -CODE_COVERAGE_LCOV_RMOPTS ?= $(CODE_COVERAGE_LCOV_RMOPTS_DEFAULT) -CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT ?=\ -$(if $(CODE_COVERAGE_BRANCH_COVERAGE),\ ---rc genhtml_branch_coverage=$(CODE_COVERAGE_BRANCH_COVERAGE)) -CODE_COVERAGE_GENHTML_OPTIONS ?= $(CODE_COVERAGE_GENHTML_OPTIONS_DEFAULTS) -CODE_COVERAGE_IGNORE_PATTERN ?= - -code_coverage_v_lcov_cap = $(code_coverage_v_lcov_cap_$(V)) -code_coverage_v_lcov_cap_ = $(code_coverage_v_lcov_cap_$(AM_DEFAULT_VERBOSITY)) -code_coverage_v_lcov_cap_0 = @echo " LCOV --capture"\ - $(CODE_COVERAGE_OUTPUT_FILE); -code_coverage_v_lcov_ign = $(code_coverage_v_lcov_ign_$(V)) -code_coverage_v_lcov_ign_ = $(code_coverage_v_lcov_ign_$(AM_DEFAULT_VERBOSITY)) -code_coverage_v_lcov_ign_0 = @echo " LCOV --remove /tmp/*"\ - $(CODE_COVERAGE_IGNORE_PATTERN); -code_coverage_v_genhtml = $(code_coverage_v_genhtml_$(V)) -code_coverage_v_genhtml_ = $(code_coverage_v_genhtml_$(AM_DEFAULT_VERBOSITY)) -code_coverage_v_genhtml_0 = @echo " GEN " $(CODE_COVERAGE_OUTPUT_DIRECTORY); -code_coverage_quiet = $(code_coverage_quiet_$(V)) -code_coverage_quiet_ = $(code_coverage_quiet_$(AM_DEFAULT_VERBOSITY)) -code_coverage_quiet_0 = --quiet - -# sanitizes the test-name: replaces with underscores: dashes and dots -code_coverage_sanitize = $(subst -,_,$(subst .,_,$(1))) - -# Use recursive makes in order to ignore errors during check -check-code-coverage: -ifeq ($(CODE_COVERAGE_ENABLED),yes) - -$(A''M_V_at)$(MAKE) $(AM_MAKEFLAGS) -k check - $(A''M_V_at)$(MAKE) $(AM_MAKEFLAGS) code-coverage-capture -else - @echo "Need to reconfigure with --enable-code-coverage" -endif - -# Capture code coverage data -code-coverage-capture: code-coverage-capture-hook -ifeq ($(CODE_COVERAGE_ENABLED),yes) - $(code_coverage_v_lcov_cap)$(LCOV) $(code_coverage_quiet) $(addprefix --directory ,$(CODE_COVERAGE_DIRECTORY)) --capture --output-file "$(CODE_COVERAGE_OUTPUT_FILE).tmp" --test-name "$(call code_coverage_sanitize,$(PACKAGE_NAME)-$(PACKAGE_VERSION))" --no-checksum --compat-libtool $(CODE_COVERAGE_LCOV_SHOPTS) $(CODE_COVERAGE_LCOV_OPTIONS) - $(code_coverage_v_lcov_ign)$(LCOV) $(code_coverage_quiet) $(addprefix --directory ,$(CODE_COVERAGE_DIRECTORY)) --remove "$(CODE_COVERAGE_OUTPUT_FILE).tmp" "/tmp/*" $(CODE_COVERAGE_IGNORE_PATTERN) --output-file "$(CODE_COVERAGE_OUTPUT_FILE)" $(CODE_COVERAGE_LCOV_SHOPTS) $(CODE_COVERAGE_LCOV_RMOPTS) - - at rm -f $(CODE_COVERAGE_OUTPUT_FILE).tmp - $(code_coverage_v_genhtml)LANG=C $(GENHTML) $(code_coverage_quiet) $(addprefix --prefix ,$(CODE_COVERAGE_DIRECTORY)) --output-directory "$(CODE_COVERAGE_OUTPUT_DIRECTORY)" --title "$(PACKAGE_NAME)-$(PACKAGE_VERSION) Code Coverage" --legend --show-details "$(CODE_COVERAGE_OUTPUT_FILE)" $(CODE_COVERAGE_GENHTML_OPTIONS) - @echo "file://$(abs_builddir)/$(CODE_COVERAGE_OUTPUT_DIRECTORY)/index.html" -else - @echo "Need to reconfigure with --enable-code-coverage" -endif - -# Hook rule executed before code-coverage-capture, overridable by the user -code-coverage-capture-hook: - -ifeq ($(CODE_COVERAGE_ENABLED),yes) -clean: code-coverage-clean -code-coverage-clean: - -$(LCOV) --directory $(top_builddir) -z - -rm -rf $(CODE_COVERAGE_OUTPUT_FILE) $(CODE_COVERAGE_OUTPUT_FILE).tmp $(CODE_COVERAGE_OUTPUT_DIRECTORY) - -find . -name "*.gcda" -o -name "*.gcov" -delete -endif - -GITIGNOREFILES ?= -GITIGNOREFILES += $(CODE_COVERAGE_OUTPUT_FILE) $(CODE_COVERAGE_OUTPUT_DIRECTORY) - -A''M_DISTCHECK_CONFIGURE_FLAGS ?= -A''M_DISTCHECK_CONFIGURE_FLAGS += --disable-code-coverage - -.PHONY: check-code-coverage code-coverage-capture code-coverage-capture-hook code-coverage-clean -'] - - AC_SUBST([CODE_COVERAGE_RULES]) - m4_ifdef([_AM_SUBST_NOTMAKE], [_AM_SUBST_NOTMAKE([CODE_COVERAGE_RULES])]) -]) From phk at FreeBSD.org Tue Apr 19 16:15:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:15:06 +0200 Subject: [master] d30d554 Isolate the EXP_ interactions in cache_hash.c and mark the functions private. Message-ID: commit d30d554ca2779026305a16918bcea1242c87b414 Author: Poul-Henning Kamp Date: Tue Apr 19 09:00:20 2016 +0000 Isolate the EXP_ interactions in cache_hash.c and mark the functions private. Rename EXP_Poke() to EXP_Remove() diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 77f62e1..52fcef9 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -712,12 +712,6 @@ extern pthread_t cli_thread; #define EXP_WHEN(to) \ ((to)->t_origin + (to)->ttl + (to)->grace + (to)->keep) -double EXP_Ttl(const struct req *, const struct objcore *); -void EXP_Insert(struct worker *wrk, struct objcore *oc); -void EXP_Rearm(struct objcore *, double now, double ttl, double grace, - double keep); -void EXP_Poke(struct objcore *); - /* cache_fetch.c */ enum vbf_fetch_mode_e { VBF_NORMAL = 0, diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index 9d474ba..b0524cc 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -104,7 +104,7 @@ exp_mail_it(struct objcore *oc, uint8_t cmds) */ void -EXP_Poke(struct objcore *oc) +EXP_Remove(struct objcore *oc) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index 1fb627c..6df5160 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -411,7 +411,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp, if (BAN_CheckObject(wrk, oc, req)) { oc->flags |= OC_F_DYING; - EXP_Poke(oc); + EXP_Remove(oc); continue; } @@ -706,7 +706,7 @@ HSH_Kill(struct objcore *oc) Lck_Lock(&oc->objhead->mtx); oc->flags |= OC_F_DYING; Lck_Unlock(&oc->objhead->mtx); - EXP_Poke(oc); + EXP_Remove(oc); } /*==================================================================== @@ -734,6 +734,8 @@ HSH_Snipe(const struct worker *wrk, struct objcore *oc) } Lck_Unlock(&oc->objhead->mtx); } + if (retval) + EXP_Remove(oc); return (retval); } diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h index ff354a6..315cf96 100644 --- a/bin/varnishd/cache/cache_priv.h +++ b/bin/varnishd/cache/cache_priv.h @@ -42,7 +42,12 @@ void VBE_Poll(void); /* cache_backend_poll.c */ void VBP_Init(void); -/* == cache_ban.c == */ +/* cache_exp.c */ +double EXP_Ttl(const struct req *, const struct objcore *); +void EXP_Insert(struct worker *wrk, struct objcore *oc); +void EXP_Rearm(struct objcore *, double now, double ttl, double grace, + double keep); +void EXP_Remove(struct objcore *); /* From cache_main.c */ void BAN_Init(void); diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c index 9ca5380..fddd4a0 100644 --- a/bin/varnishd/storage/storage_lru.c +++ b/bin/varnishd/storage/storage_lru.c @@ -189,8 +189,6 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru) /* XXX: We could grab and return one storage segment to our caller */ ObjSlim(wrk, oc); - EXP_Poke(oc); - VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc)); (void)HSH_DerefObjCore(wrk, &oc); // Ref from ObjSnipe return (1); From phk at FreeBSD.org Tue Apr 19 16:15:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:15:06 +0200 Subject: [master] 6e6cc0b Rename OEV_REMOVE to OEV_EXPIRE to precisely capture its meaning. Message-ID: commit 6e6cc0bdb8bcc4c5a381f40dc7ca1bd35324d72a Author: Poul-Henning Kamp Date: Tue Apr 19 09:06:08 2016 +0000 Rename OEV_REMOVE to OEV_EXPIRE to precisely capture its meaning. The fact that the oc expired does not mean that the OC is going away any time soon: It may be delivering War&Peace to a mobile phone in outher Elbonia for quite some time still. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 52fcef9..eecb05b 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -889,12 +889,12 @@ int ObjGetU64(struct worker *, struct objcore *, enum obj_attr, uint64_t *); int ObjCheckFlag(struct worker *, struct objcore *, enum obj_flags of); void ObjSetFlag(struct worker *, struct objcore *, enum obj_flags of, int val); -#define OEV_BANCHG (1U<<1) -#define OEV_TTLCHG (1U<<2) -#define OEV_INSERT (1U<<3) -#define OEV_REMOVE (1U<<4) +#define OEV_INSERT (1U<<1) +#define OEV_BANCHG (1U<<2) +#define OEV_TTLCHG (1U<<3) +#define OEV_EXPIRE (1U<<4) -#define OEV_MASK (OEV_BANCHG|OEV_TTLCHG|OEV_INSERT|OEV_REMOVE) +#define OEV_MASK (OEV_INSERT|OEV_BANCHG|OEV_TTLCHG|OEV_EXPIRE) typedef void obj_event_f(struct worker *, void *priv, struct objcore *, unsigned); diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index b0524cc..39c4626 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -115,8 +115,7 @@ EXP_Remove(struct objcore *oc) /*-------------------------------------------------------------------- * Insert new object. * - * We grab a reference to the object, which will keep it around until - * we decide its time to let it go. + * Caller got a oc->refcnt for us. */ void @@ -186,10 +185,10 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags) binheap_delete(ep->heap, oc->timer_idx); } assert(oc->timer_idx == BINHEAP_NOIDX); - ObjSendEvent(ep->wrk, oc, OEV_REMOVE); oc->exp_flags &= ~OC_EF_EXP; assert(oc->refcnt > 0); AZ(oc->exp_flags); + ObjSendEvent(ep->wrk, oc, OEV_EXPIRE); (void)HSH_DerefObjCore(ep->wrk, &oc); return; } @@ -258,7 +257,7 @@ exp_expire(struct exp_priv *ep, double now) CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC); VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f", ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, oc) - now); - ObjSendEvent(ep->wrk, oc, OEV_REMOVE); + ObjSendEvent(ep->wrk, oc, OEV_EXPIRE); (void)HSH_DerefObjCore(ep->wrk, &oc); return (0); } diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 9928991..ee3a005 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -212,7 +212,7 @@ obj_cb(struct worker *wrk, void *priv, struct objcore *oc, unsigned event) CAST_OBJ_NOTNULL(priv_vcl, priv, PRIV_VCL_MAGIC); switch (event) { case OEV_INSERT: what = "insert"; break; - case OEV_REMOVE: what = "remove"; break; + case OEV_EXPIRE: what = "expire"; break; default: WRONG("Wrong object event"); } @@ -230,7 +230,7 @@ vmod_register_obj_events(VRT_CTX, struct vmod_priv *priv) CAST_OBJ_NOTNULL(priv_vcl, priv->priv, PRIV_VCL_MAGIC); AZ(priv_vcl->obj_cb); priv_vcl->obj_cb = ObjSubscribeEvents(obj_cb, priv_vcl, - OEV_INSERT|OEV_REMOVE); + OEV_INSERT|OEV_EXPIRE); VSL(SLT_Debug, 0, "Subscribed to Object Events"); } From phk at FreeBSD.org Tue Apr 19 16:15:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:15:06 +0200 Subject: [master] 1a67d3e Fix test-case to match OEV_REMOVE now being called OEV_EXPIRE Message-ID: commit 1a67d3e2faaabc473ab4262d025d1de2c15bdba6 Author: Poul-Henning Kamp Date: Tue Apr 19 09:09:25 2016 +0000 Fix test-case to match OEV_REMOVE now being called OEV_EXPIRE diff --git a/bin/varnishtest/tests/m00021.vtc b/bin/varnishtest/tests/m00021.vtc index 007728a..02a6cbe 100644 --- a/bin/varnishtest/tests/m00021.vtc +++ b/bin/varnishtest/tests/m00021.vtc @@ -12,7 +12,7 @@ varnish v1 -cliok "param.set debug +vclrel" logexpect l1 -v v1 -g raw { expect * 0 Debug "Subscribed to Object Events" expect * 0 Debug "Object Event: insert 0x[0-9a-f]+" - expect * 0 Debug "Object Event: remove 0x[0-9a-f]+" + expect * 0 Debug "Object Event: expire 0x[0-9a-f]+" expect * 0 Debug "Unsubscribed from Object Events" } -start From phk at FreeBSD.org Tue Apr 19 16:15:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:15:06 +0200 Subject: [master] d089c57 Use correct test for busyness Message-ID: commit d089c5732bc59a53a3abed43679110d424d00e6a Author: Poul-Henning Kamp Date: Tue Apr 19 15:56:03 2016 +0000 Use correct test for busyness diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 8a513f2..c076bcc 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -908,13 +908,10 @@ vbf_stp_fail(struct worker *wrk, const struct busyobj *bo) assert(bo->fetch_objcore->boc->state < BOS_FINISHED); HSH_Fail(bo->fetch_objcore); - if (bo->fetch_objcore->exp_flags & OC_EF_EXP) { - /* Already unbusied - expire it */ - AN(bo->fetch_objcore); + if (!(bo->fetch_objcore->flags & OC_F_BUSY)) HSH_Kill(bo->fetch_objcore); - } - wrk->stats->fetch_failed++; ObjSetState(wrk, bo->fetch_objcore, BOS_FAILED); + wrk->stats->fetch_failed++; return (F_STP_DONE); } From phk at FreeBSD.org Tue Apr 19 16:15:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:15:06 +0200 Subject: [master] 33a49c4 Attempt to close the race in #1909. Message-ID: commit 33a49c426938d2bc1d9e1c27c5fce9d309665572 Author: Poul-Henning Kamp Date: Tue Apr 19 16:08:33 2016 +0000 Attempt to close the race in #1909. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index eecb05b..ef583f1 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -430,9 +430,10 @@ struct objcore { uint8_t exp_flags; #define OC_EF_POSTED (1<<1) -#define OC_EF_MOVE (1<<2) -#define OC_EF_INSERT (1<<3) -#define OC_EF_EXP (1<<4) +#define OC_EF_REFD (1<<2) +#define OC_EF_MOVE (1<<3) +#define OC_EF_INSERT (1<<4) +#define OC_EF_REMOVE (1<<5) uint16_t oa_present; diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index 39c4626..f0b7002 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -86,16 +86,20 @@ exp_mail_it(struct objcore *oc, uint8_t cmds) assert(oc->refcnt > 0); Lck_Lock(&exphdl->mtx); - if (!(oc->exp_flags & OC_EF_POSTED)) { - if (oc->flags & OC_F_DYING) - VSTAILQ_INSERT_HEAD(&exphdl->inbox, oc, exp_list); - else - VSTAILQ_INSERT_TAIL(&exphdl->inbox, oc, exp_list); + if ((cmds | oc->exp_flags) & OC_EF_REFD) { + if (!(oc->exp_flags & OC_EF_POSTED)) { + if (cmds & OC_EF_REMOVE) + VSTAILQ_INSERT_HEAD(&exphdl->inbox, + oc, exp_list); + else + VSTAILQ_INSERT_TAIL(&exphdl->inbox, + oc, exp_list); + } + oc->exp_flags |= cmds | OC_EF_POSTED; + AN(oc->exp_flags & OC_EF_REFD); + VSC_C_main->exp_mailed++; + AZ(pthread_cond_signal(&exphdl->condvar)); } - oc->exp_flags |= cmds | OC_EF_POSTED; - AN(oc->exp_flags & OC_EF_EXP); - VSC_C_main->exp_mailed++; - AZ(pthread_cond_signal(&exphdl->condvar)); Lck_Unlock(&exphdl->mtx); } @@ -108,8 +112,8 @@ EXP_Remove(struct objcore *oc) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); - if (oc->exp_flags & OC_EF_EXP) - exp_mail_it(oc, 0); + if (oc->exp_flags & OC_EF_REFD) + exp_mail_it(oc, OC_EF_REMOVE); } /*-------------------------------------------------------------------- @@ -130,7 +134,7 @@ EXP_Insert(struct worker *wrk, struct objcore *oc) AZ(oc->flags & OC_F_DYING); ObjSendEvent(wrk, oc, OEV_INSERT); - exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE); + exp_mail_it(oc, OC_EF_INSERT | OC_EF_REFD | OC_EF_MOVE); } /*-------------------------------------------------------------------- @@ -146,7 +150,7 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep) CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); assert(oc->refcnt > 0); - AN(oc->exp_flags & OC_EF_EXP); + AN(oc->exp_flags & OC_EF_REFD); if (!isnan(ttl)) oc->ttl = now + ttl - oc->t_origin; @@ -179,13 +183,13 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags) VSLb(&ep->vsl, SLT_ExpKill, "EXP_Inbox flg=%x p=%p e=%.9f f=0x%x", flags, oc, oc->timer_when, oc->flags); - if (oc->flags & OC_F_DYING) { + if (flags & OC_EF_REMOVE) { if (!(flags & OC_EF_INSERT)) { assert(oc->timer_idx != BINHEAP_NOIDX); binheap_delete(ep->heap, oc->timer_idx); } assert(oc->timer_idx == BINHEAP_NOIDX); - oc->exp_flags &= ~OC_EF_EXP; + oc->exp_flags &= ~OC_EF_REFD; assert(oc->refcnt > 0); AZ(oc->exp_flags); ObjSendEvent(ep->wrk, oc, OEV_EXPIRE); @@ -244,7 +248,7 @@ exp_expire(struct exp_priv *ep, double now) return (oc->timer_when); VSC_C_main->n_expired++; - oc->exp_flags &= ~OC_EF_EXP; + oc->exp_flags &= ~OC_EF_REFD; if (!(oc->flags & OC_F_DYING)) HSH_Kill(oc); @@ -312,8 +316,7 @@ exp_thread(struct worker *wrk, void *priv) VSC_C_main->exp_received++; tnext = 0; flags = oc->exp_flags; - oc->exp_flags &= - ~(OC_EF_INSERT | OC_EF_MOVE | OC_EF_POSTED); + oc->exp_flags &= OC_EF_REFD; } else if (tnext > t) { VSL_Flush(&ep->vsl, 0); Pool_Sumstat(wrk); From phk at FreeBSD.org Tue Apr 19 16:36:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 19 Apr 2016 18:36:05 +0200 Subject: [master] 9796bd9 Fix (insignificant) memory leak with mal-formed ESI directives. Message-ID: commit 9796bd939173e5ddc9bb3b26e5731c19fe776ccd Author: Poul-Henning Kamp Date: Tue Apr 19 16:34:23 2016 +0000 Fix (insignificant) memory leak with mal-formed ESI directives. Fixes #1912 diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index 6ba5eb1..815bac4 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -820,6 +820,8 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l) vep_mark_skip(vep, p); vep->in_esi_tag = 0; vep->state = VEP_NEXTTAG; + if (vep->attr_vsb) + VSB_destroy(&vep->attr_vsb); } /****************************************************** @@ -854,6 +856,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l) vep->state = VEP_TAGERROR; } } else if (vep->state == VEP_ATTRGETVAL) { + AZ(vep->attr_vsb); vep->attr_vsb = VSB_new_auto(); vep->state = VEP_ATTRDELIM; } else if (vep->state == VEP_ATTRDELIM) { From phk at FreeBSD.org Wed Apr 20 06:29:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 20 Apr 2016 08:29:05 +0200 Subject: [master] 1694e15 If an expiring oc is in the mail, stick OC_EF_REMOVE on it and wait for it to arrive. Message-ID: commit 1694e156b8e5597d917d26b909f5587f3c482c80 Author: Poul-Henning Kamp Date: Wed Apr 20 06:27:41 2016 +0000 If an expiring oc is in the mail, stick OC_EF_REMOVE on it and wait for it to arrive. diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index f0b7002..e226fee 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -248,21 +248,30 @@ exp_expire(struct exp_priv *ep, double now) return (oc->timer_when); VSC_C_main->n_expired++; - oc->exp_flags &= ~OC_EF_REFD; - if (!(oc->flags & OC_F_DYING)) - HSH_Kill(oc); + Lck_Lock(&ep->mtx); + if (oc->exp_flags & OC_EF_POSTED) { + oc->exp_flags |= OC_EF_REMOVE; + oc = NULL; + } else { + oc->exp_flags &= ~OC_EF_REFD; + } + Lck_Unlock(&ep->mtx); + if (oc != NULL) { + if (!(oc->flags & OC_F_DYING)) + HSH_Kill(oc); - /* Remove from binheap */ - assert(oc->timer_idx != BINHEAP_NOIDX); - binheap_delete(ep->heap, oc->timer_idx); - assert(oc->timer_idx == BINHEAP_NOIDX); + /* Remove from binheap */ + assert(oc->timer_idx != BINHEAP_NOIDX); + binheap_delete(ep->heap, oc->timer_idx); + assert(oc->timer_idx == BINHEAP_NOIDX); - CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC); - VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f", - ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, oc) - now); - ObjSendEvent(ep->wrk, oc, OEV_EXPIRE); - (void)HSH_DerefObjCore(ep->wrk, &oc); + CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC); + VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f", + ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, oc) - now); + ObjSendEvent(ep->wrk, oc, OEV_EXPIRE); + (void)HSH_DerefObjCore(ep->wrk, &oc); + } return (0); } From fgsch at lodoss.net Wed Apr 20 07:46:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 20 Apr 2016 09:46:05 +0200 Subject: [master] 3a06210 Also remove directory if compilation failed Message-ID: commit 3a06210b095d09e3070bfaf1aa41bb15cc0f80e0 Author: Federico G. Schwindt Date: Tue Apr 19 23:21:45 2016 +0100 Also remove directory if compilation failed diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c index 996985e..c0f8ed2 100644 --- a/bin/varnishd/mgt/mgt_vcc.c +++ b/bin/varnishd/mgt/mgt_vcc.c @@ -299,8 +299,7 @@ mgt_VccCompile(struct cli *cli, const char *vclname, const char *vclsrc, if (status || C_flag) { (void)unlink(vp.libfile); free(vp.libfile); - if (C_flag) - AZ(rmdir(vp.dir)); + AZ(rmdir(vp.dir)); free(vp.dir); if (status) { VCLI_Out(cli, "VCL compilation failed"); From fgsch at lodoss.net Wed Apr 20 09:51:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 20 Apr 2016 11:51:05 +0200 Subject: [master] d270a00 Maintain `-d` behaviour when log is overrun and reacquired Message-ID: commit d270a009be6f1966330643bc3de846334a30513e Author: Jason Stangroome Date: Wed Apr 20 16:13:44 2016 +1000 Maintain `-d` behaviour when log is overrun and reacquired diff --git a/lib/libvarnishtools/vut.c b/lib/libvarnishtools/vut.c index 40d001b..f534266 100644 --- a/lib/libvarnishtools/vut.c +++ b/lib/libvarnishtools/vut.c @@ -372,7 +372,8 @@ VUT_Main(void) continue; } c = VSL_CursorVSM(VUT.vsl, VUT.vsm, - VSL_COPT_TAIL | VSL_COPT_BATCH); + (VUT.d_opt ? VSL_COPT_TAILSTOP : VSL_COPT_TAIL) + | VSL_COPT_BATCH); if (c == NULL) { VSL_ResetError(VUT.vsl); VSM_Close(VUT.vsm); From fgsch at lodoss.net Wed Apr 20 09:54:06 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 20 Apr 2016 11:54:06 +0200 Subject: [4.1] 9f933fd Maintain `-d` behaviour when log is overrun and reacquired Message-ID: commit 9f933fd22ce99a7d3a1065e78115468f5c053017 Author: Jason Stangroome Date: Wed Apr 20 16:13:44 2016 +1000 Maintain `-d` behaviour when log is overrun and reacquired diff --git a/lib/libvarnishtools/vut.c b/lib/libvarnishtools/vut.c index 40d001b..f534266 100644 --- a/lib/libvarnishtools/vut.c +++ b/lib/libvarnishtools/vut.c @@ -372,7 +372,8 @@ VUT_Main(void) continue; } c = VSL_CursorVSM(VUT.vsl, VUT.vsm, - VSL_COPT_TAIL | VSL_COPT_BATCH); + (VUT.d_opt ? VSL_COPT_TAILSTOP : VSL_COPT_TAIL) + | VSL_COPT_BATCH); if (c == NULL) { VSL_ResetError(VUT.vsl); VSM_Close(VUT.vsm); From nils.goroll at uplex.de Wed Apr 20 13:08:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 20 Apr 2016 15:08:05 +0200 Subject: [master] ae69070 dont assert on a failing rmdir Message-ID: commit ae69070c3aa5073808c5ba8c94009cfb50da789a Author: Nils Goroll Date: Wed Apr 20 15:06:44 2016 +0200 dont assert on a failing rmdir We may not have permission to unlink vp.dir $ /etc/init.d/varnish_master restart Stopping Varnish Cache: [FAILED] Starting Varnish Cache: Assert error in mgt_VccCompile(), mgt/mgt_vcc.c line 302: Condition((rmdir(vp.dir)) == 0) not true. errno = 13 (Permission denied) diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c index c0f8ed2..9a094fb 100644 --- a/bin/varnishd/mgt/mgt_vcc.c +++ b/bin/varnishd/mgt/mgt_vcc.c @@ -299,7 +299,7 @@ mgt_VccCompile(struct cli *cli, const char *vclname, const char *vclsrc, if (status || C_flag) { (void)unlink(vp.libfile); free(vp.libfile); - AZ(rmdir(vp.dir)); + (void)rmdir(vp.dir); free(vp.dir); if (status) { VCLI_Out(cli, "VCL compilation failed"); From nils.goroll at uplex.de Wed Apr 20 15:35:06 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 20 Apr 2016 17:35:06 +0200 Subject: [master] 76315ee make EXP_Rearm() non-private again - vmods use it Message-ID: commit 76315ee568370b6cd3219fc336d3449c8e4c92fa Author: Nils Goroll Date: Wed Apr 20 17:32:40 2016 +0200 make EXP_Rearm() non-private again - vmods use it Lack of this interface breaks existing vmods, so should we really need to remove it, we problable should provide an alternative. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index ef583f1..2b14c9b 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -713,6 +713,10 @@ extern pthread_t cli_thread; #define EXP_WHEN(to) \ ((to)->t_origin + (to)->ttl + (to)->grace + (to)->keep) +/* cache_exp.c */ +void EXP_Rearm(struct objcore *, double now, double ttl, double grace, + double keep); + /* cache_fetch.c */ enum vbf_fetch_mode_e { VBF_NORMAL = 0, diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h index 315cf96..47330ee 100644 --- a/bin/varnishd/cache/cache_priv.h +++ b/bin/varnishd/cache/cache_priv.h @@ -45,8 +45,6 @@ void VBP_Init(void); /* cache_exp.c */ double EXP_Ttl(const struct req *, const struct objcore *); void EXP_Insert(struct worker *wrk, struct objcore *oc); -void EXP_Rearm(struct objcore *, double now, double ttl, double grace, - double keep); void EXP_Remove(struct objcore *); /* From cache_main.c */ From nils.goroll at uplex.de Thu Apr 21 05:46:04 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 21 Apr 2016 07:46:04 +0200 Subject: [master] 911d4d5 Initialize wrk->lastused if still NaN Message-ID: commit 911d4d5afd49f3f3165755ef0d6deccd5f35f927 Author: Nils Goroll Date: Thu Apr 21 07:36:53 2016 +0200 Initialize wrk->lastused if still NaN Fixes #1917 diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c index dea59de..075d866 100644 --- a/bin/varnishd/storage/storage_simple.c +++ b/bin/varnishd/storage/storage_simple.c @@ -41,6 +41,8 @@ #include "vsb.h" +#include "vtim.h" + /*-------------------------------------------------------------------*/ static struct storage * @@ -498,8 +500,11 @@ sml_bocdone(struct worker *wrk, struct objcore *oc, struct boc *boc) sml_stv_free(stv, st); } - if (stv->lru != NULL) + if (stv->lru != NULL) { + if (isnan(wrk->lastused)) + wrk->lastused = VTIM_real(); LRU_Add(oc, wrk->lastused); // approx timestamp is OK + } } static const void * __match_proto__(objgetattr_f) From nils.goroll at uplex.de Thu Apr 21 05:46:04 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 21 Apr 2016 07:46:04 +0200 Subject: [master] d3e9232 No LRU for private objects Message-ID: commit d3e9232618befed3ac8fbf55a71cb4b52aaa274d Author: Nils Goroll Date: Thu Apr 21 07:42:09 2016 +0200 No LRU for private objects diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c index fddd4a0..fc1987a 100644 --- a/bin/varnishd/storage/storage_lru.c +++ b/bin/varnishd/storage/storage_lru.c @@ -85,6 +85,10 @@ LRU_Add(struct objcore *oc, double now) struct lru *lru; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + + if (oc->flags & OC_F_PRIVATE) + return; + AZ(oc->boc); AN(isnan(oc->last_lru)); AZ(isnan(now)); @@ -103,6 +107,10 @@ LRU_Remove(struct objcore *oc) struct lru *lru; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + + if (oc->flags & OC_F_PRIVATE) + return; + AZ(oc->boc); lru = lru_get(oc); CHECK_OBJ_NOTNULL(lru, LRU_MAGIC); @@ -121,7 +129,7 @@ LRU_Touch(struct worker *wrk, struct objcore *oc, double now) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); - if (isnan(oc->last_lru)) + if (oc->flags & OC_F_PRIVATE || isnan(oc->last_lru)) return; /* From phk at FreeBSD.org Fri Apr 22 23:23:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Sat, 23 Apr 2016 01:23:04 +0200 Subject: [master] c8d062a Recover the Architect notes from the old wiki Message-ID: commit c8d062ab89b67a53f25e3506b3469a19fdd6401e Author: Poul-Henning Kamp Date: Fri Apr 22 23:21:41 2016 +0000 Recover the Architect notes from the old wiki diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst index 60e2846..7dc5978 100644 --- a/doc/sphinx/phk/index.rst +++ b/doc/sphinx/phk/index.rst @@ -32,4 +32,5 @@ You may or may not want to know what Poul-Henning thinks. thoughts.rst autocrap.rst sphinx.rst + notes.rst diff --git a/doc/sphinx/phk/notes.rst b/doc/sphinx/phk/notes.rst new file mode 100644 index 0000000..16065ec --- /dev/null +++ b/doc/sphinx/phk/notes.rst @@ -0,0 +1,254 @@ +.. _phk_notes: + +======================== +Notes from the Architect +======================== + +Once you start working with the Varnish source code, you will notice +that Varnish is not your average run of the mill application. + +That is not a coincidence. + +I have spent many years working on the FreeBSD kernel, and only +rarely did I venture into userland programming, but when I had +occation to do so, I invariably found that people programmed like +it was still 1975. + +So when I was approached about the Varnish project I wasn't really +interested until I realized that this would be a good opportunity +to try to put some of all my knowledge of how hardware and kernels +work to good use, and now that we have reached alpha stage, I can +say I have really enjoyed it. + +So what's wrong with 1975 programming ? +--------------------------------------- + +The really short answer is that computers do not have two kinds of +storage any more. + +It used to be that you had the primary store, and it was anything +from acoustic delaylines filled with mercury via small magnetic +dougnuts via transistor flip-flops to dynamic RAM. + +And then there were the secondary store, paper tape, magnetic tape, +disk drives the size of houses, then the size of washing machines +and these days so small that girls get disappointed if think they +got hold of something else than the MP3 player you had in your +pocket. + +And people program this way. + +They have variables in "memory" and move data to and from "disk". + +Take Squid for instance, a 1975 program if I ever saw one: You tell +it how much RAM it can use and how much disk it can use. It will +then spend inordinate amounts of time keeping track of what HTTP +objects are in RAM and which are on disk and it will move them forth +and back depending on traffic patterns. + +Well, today computers really only have one kind of storage, and it +is usually some sort of disk, the operating system and the virtual +memory management hardware has converted the RAM to a cache for the +disk storage. + +So what happens with squids elaborate memory management is that it +gets into fights with the kernels elaborate memory management, and +like any civil war, that never gets anything done. + +What happens is this: Squid creates a HTTP object in "RAM" and it +gets used some times rapidly after creation. Then after some time +it get no more hits and the kernel notices this. Then somebody tries +to get memory from the kernel for something and the kernel decides +to push those unused pages of memory out to swap space and use the +(cache-RAM) more sensibly for some data which is actually used by +a program. This however, is done without squid knowing about it. +Squid still thinks that these http objects are in RAM, and they +will be, the very second it tries to access them, but until then, +the RAM is used for something productive. + +This is what Virtual Memory is all about. + +If squid did nothing else, things would be fine, but this is where +the 1975 programming kicks in. + +After some time, squid will also notice that these objects are +unused, and it decides to move them to disk so the RAM can be used +for more busy data. So squid goes out, creates a file and then it +writes the http objects to the file. + +Here we switch to the high-speed camera: Squid calls write(2), the +address i gives is a "virtual address" and the kernel has it marked +as "not at home". + +So the CPU hardwares paging unit will raise a trap, a sort of +interrupt to the operating system telling it "fix the memory please". + +The kernel tries to find a free page, if there are none, it will +take a little used page from somewhere, likely another little used +squid object, write it to the paging poll space on the disk (the +"swap area") when that write completes, it will read from another +place in the paging pool the data it "paged out" into the now unused +RAM page, fix up the paging tables, and retry the instruction which +failed. + +Squid knows nothing about this, for squid it was just a single +normal memory acces. + +So now squid has the object in a page in RAM and written to the +disk two places: one copy in the operating systems paging space and +one copy in the filesystem. + +Squid now uses this RAM for something else but after some time, the +HTTP object gets a hit, so squid needs it back. + +First squid needs some RAM, so it may decide to push another HTTP +object out to disk (repeat above), then it reads the filesystem +file back into RAM, and then it sends the data on the network +connections socket. + +Did any of that sound like wasted work to you ? + +Here is how Varnish does it: + +Varnish allocate some virtual memory, it tells the operating system +to back this memory with space from a disk file. When it needs to +send the object to a client, it simply refers to that piece of +virtual memory and leaves the rest to the kernel. + +If/when the kernel decides it needs to use RAM for something else, +the page will get written to the backing file and the RAM page +reused elsewhere. + +When Varnish next time refers to the virtual memory, the operating +system will find a RAM page, possibly freeing one, and read the +contents in from the backing file. + +And that's it. Varnish doesn't really try to control what is cached +in RAM and what is not, the kernel has code and hardware support +to do a good job at that, and it does a good job. + +Varnish also only has a single file on the disk whereas squid puts +one object in its own separate file. The HTTP objects are not needed +as filesystem objects, so there is no point in wasting time in the +filesystem name space (directories, filenames and all that) for +each object, all we need to have in Varnish is a pointer into virtual +memory and a length, the kernel does the rest. + +Virtual memory was meant to make it easier to program when data was +larger than the physical memory, but people have still not caught +on. + +More caches +----------- + +But there are more caches around, the silicon mafia has more or +less stalled at 4GHz CPU clock and to get even that far they have +had to put level 1, 2 and sometimes 3 caches between the CPU and +the RAM (which is the level 4 cache), there are also things like +write buffers, pipeline and page-mode fetches involved, all to make +it a tad less slow to pick up something from memory. + +And since they have hit the 4GHz limit, but decreasing silicon +feature sizes give them more and more transistors to work with, +multi-cpu designs have become the fancy of the world, despite the +fact that they suck as a programming model. + +Multi-CPU systems is nothing new, but writing programs that use +more than one CPU at a time has always been tricky and it still is. + +Writing programs that perform well on multi-CPU systems is even trickier. + +Imagine I have two statistics counters: + + unsigned n_foo; + unsigned n_bar; + +So one CPU is chugging along and has to execute n_foo++ + +To do that, it read n_foo and then write n_foo back. It may or may +not involve a load into a CPU register, but that is not important. + +To read a memory location means to check if we have it in the CPUs +level 1 cache. It is unlikely to be unless it is very frequently +used. Next check the level two cache, and let us assume that is a +miss as well. + +If this is a single CPU system, the game ends here, we pick it out +of RAM and move on. + +On a Multi-CPU system, and it doesn't matter if the CPUs share a +socket or have their own, we first have to check if any of the other +CPUs have a modified copy of n_foo stored in their caches, so a +special bus-transaction goes out to find this out, if if some cpu +comes back and says "yeah, I have it" that cpu gets to write it to +RAM. On good hardware designs, our CPU will listen in on the bus +during that write operation, on bad designs it will have to do a +memory read afterwards. + +Now the CPU can increment the value of n_foo, and write it back. +But it is unlikely to go directly back to memory, we might need it +again quickly, so the modified value gets stored in our own L1 cache +and then at some point, it will end up in RAM. + +Now imagine that another CPU wants to n_bar+++ at the same time, +can it do that ? No. Caches operate not on bytes but on some +"linesize" of bytes, typically from 8 to 128 bytes in each line. +So since the first cpu was busy dealing with n_foo, the second CPU +will be trying to grab the same cache-line, so it will have to wait, +even through it is a different variable. + +Starting to get the idea ? + +Yes, it's ugly. + +How do we cope ? +---------------- + +Avoid memory operations if at all possible. + +Here are some ways Varnish tries to do that: + +When we need to handle a HTTP request or response, we have an array +of pointers and a workspace. We do not call malloc(3) for each +header. We call it once for the entire workspace and then we pick +space for the headers from there. The nice thing about this is that +we usually free the entire header in one go and we can do that +simply by resetting a pointer to the start of the workspace. + +When we need to copy a HTTP header from one request to another (or +from a response to another) we don't copy the string, we just copy +the pointer to it. Provided we do not change or free the source +headers, this is perfectly safe, a good example is copying from the +client request to the request we will send to the backend. + +When the new header has a longer lifetime than the source, then we +have to copy it. For instance when we store headers in a cached +object. But in that case we build the new header in a workspace, +and once we know how big it will be, we do a single malloc(3) to +get the space and then we put the entire header in that space. + +We also try to reuse memory which is likely to be in the caches. + +The worker threads are used in "most recently busy" fashion, when +a workerthread becomes free it goes to the front of the queue where +it is most likely to get the next request, so that all the memory +it already has cached, stack space, variables etc, can be reused +while in the cache, instead of having the expensive fetches from +RAM. + +We also give each worker thread a private set of variables it is +likely to need, all allocated on the stack of the thread. That way +we are certain that they occupy a page in RAM which none of the +other CPUs will ever think about touching as long as this thread +runs on its own CPU. That way they will not fight about the cachelines. + +If all this sounds foreign to you, let me just assure you that it +works: we spend less than 18 system calls on serving a cache hit, +and even many of those are calls tog get timestamps for statistics. + +These techniques are also nothing new, we have used them in the +kernel for more than a decade, now it's your turn to learn them :-) + +So Welcome to Varnish, a 2006 architecture program. + +*phk* From hermunn at varnish-software.com Mon Apr 25 13:44:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 15:44:05 +0200 Subject: [master] f542cab Release memory instead of crashing on malformed ESI Message-ID: commit f542cabbf977a09df89cf71c39ceda9c11d774ec Author: P?l Hermunn Johansen Date: Tue Apr 19 16:59:03 2016 +0200 Release memory instead of crashing on malformed ESI When an ESI tag contains an unterminated quote or misses its ending '>', we release the relevant buffers instead of crashing. Fixes: #1904 diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index 815bac4..eac151e 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -1047,8 +1047,11 @@ VEP_Finish(struct vep_state *vep) CHECK_OBJ_NOTNULL(vep, VEP_MAGIC); - AZ(vep->include_src); - AZ(vep->attr_vsb); + if (vep->include_src) + VSB_destroy(&vep->include_src); + if (vep->attr_vsb) + VSB_destroy(&vep->attr_vsb); + if (vep->o_pending) vep_mark_common(vep, vep->ver_p, vep->last_mark); if (vep->o_wait > 0) { From hermunn at varnish-software.com Mon Apr 25 13:44:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 15:44:06 +0200 Subject: [master] d47a144 Add simple error checking in ESI processing Message-ID: commit d47a144a05cad189dff673ac869ff86f6e80352f Author: P?l Hermunn Johansen Date: Tue Apr 19 17:00:24 2016 +0200 Add simple error checking in ESI processing When the VEP state machine is in a bad state during finalization (VEP_Finish), we log this as an ESI error. diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index eac151e..744089a 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -1052,6 +1052,14 @@ VEP_Finish(struct vep_state *vep) if (vep->attr_vsb) VSB_destroy(&vep->attr_vsb); + if (vep->state != VEP_START && + vep->state != VEP_BOM && + vep->state != VEP_TESTXML && + vep->state != VEP_NOTXML && + vep->state != VEP_NEXTTAG) { + vep_error(vep, "VEP ended inside a tag"); + } + if (vep->o_pending) vep_mark_common(vep, vep->ver_p, vep->last_mark); if (vep->o_wait > 0) { From hermunn at varnish-software.com Mon Apr 25 13:57:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 15:57:06 +0200 Subject: [master] e171090 Add a SLT_BackendStart log tag, and log address and port on backend req Message-ID: commit e171090e7fc84ced76a5aeae822b9d11b59ee803 Author: Martin Blix Grydeland Date: Mon Apr 25 11:26:49 2016 +0200 Add a SLT_BackendStart log tag, and log address and port on backend req This log record is a counterpart to the SLT_ReqStart for backends, and shows the socket endpoint we are talking to at the start of backend requests. Today this information is only logged on opening a backend connection (SLT_BackendOpen), and is thus not available on the backend log transaction when on a reused backend connection. We need this information to complete the backend view varnishncsa extension. diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 6fd5f64..1cf127c 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -80,6 +80,8 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr, ssize_t i; struct http_conn *htc; int do_chunked = 0; + char abuf[VTCP_ADDRBUFSIZE]; + char pbuf[VTCP_PORTBUFSIZE]; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); @@ -95,6 +97,9 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr, do_chunked = 1; } + VTCP_hisname(htc->fd, abuf, sizeof abuf, pbuf, sizeof pbuf); + VSLb(bo->vsl, SLT_BackendStart, "%s %s", abuf, pbuf); + (void)VTCP_blocking(htc->fd); /* XXX: we should timeout instead */ V1L_Reserve(wrk, wrk->aws, &htc->fd, bo->vsl, bo->t_prev); *ctr += HTTP1_Write(wrk, hp, HTTP1_Req); diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h index e1c946d..26d8216 100644 --- a/include/tbl/vsl_tags.h +++ b/include/tbl/vsl_tags.h @@ -522,5 +522,16 @@ SLTM(Witness, 0, "Lock order witness records", "Diagnostic recording of locking order.\n" ) +SLTM(BackendStart, 0, "Backend request start", + "Start of backend processing. Logs the backend IP address and port" + " number.\n\n" + "The format is::\n\n" + "\t%s %s\n" + "\t| |\n" + "\t| +- Backend Port number\n" + "\t+---- Backend IP4/6 address\n" + "\n" +) + #undef NODEF_NOTICE /*lint -restore */ From hermunn at varnish-software.com Mon Apr 25 13:57:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 15:57:06 +0200 Subject: [master] 9c13026 Backend mode in varnishncsa Message-ID: commit 9c13026752259bb4d317fdbd9bd420b7142f9019 Author: P?l Hermunn Johansen Date: Mon Apr 25 12:43:54 2016 +0200 Backend mode in varnishncsa A new mode, backend mode, in varnishncsa is introduced. It lets you log trafic from varnish to the backends. The normal (default) mode is now named "client mode" and can be explicitly selected using -c. You can run varnishncsa in backend mode in addition to varnishncsa in client mode, but it is also possible to run in mixed mode by specifying both -b and -c. New formatter added: %{Varnish:side}x will be either "b" or "c", depending on where the request was made. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 885743d..326b6ae 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -50,6 +50,7 @@ #include #include #include +#include #include #include #include @@ -125,6 +126,8 @@ VTAILQ_HEAD(watch_head, watch); struct ctx { /* Options */ int a_opt; + int b_opt; + int c_opt; char *w_arg; FILE *fo; @@ -134,11 +137,12 @@ struct ctx { /* State */ struct watch_head watch_vcl_log; - struct watch_head watch_reqhdr; - struct watch_head watch_resphdr; + struct watch_head watch_reqhdr; /* also bereqhdr */ + struct watch_head watch_resphdr; /* also beresphdr */ struct fragment frag[F__MAX]; const char *hitmiss; const char *handling; + const char *side; } CTX; static void @@ -658,6 +662,10 @@ parse_format(const char *format) addf_strptr(&CTX.handling); break; } + if (!strcmp(buf, "Varnish:side")) { + addf_strptr(&CTX.side); + break; + } if (!strncmp(buf, "VCL_Log:", 8)) { addf_vcl_log(buf + 8, ""); break; @@ -717,7 +725,8 @@ frag_fields(int force, const char *b, const char *e, ...) field = va_arg(ap, int); frag = va_arg(ap, struct fragment *); for (n = 1, q = b; q < e; n++) { - assert(field > 0); + /* caller must sort the fields, or this loop will not work: */ + assert(field >= n); AN(frag); p = q; @@ -786,14 +795,25 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], unsigned tag; const char *b, *e, *p; struct watch *w; - int i, skip; + int i, skip, be_mark; (void)vsl; (void)priv; +#define BACKEND_MARKER (INT_MAX / 2 + 1) + assert(BACKEND_MARKER >= VSL_t__MAX); + for (t = pt[0]; t != NULL; t = *++pt) { CTX.gen++; - if (t->type != VSL_t_req) - /* Only look at client requests */ + + /* Consider client requests only if in client mode. + Consider backend requests only if in backend mode. */ + if (t->type == VSL_t_req && CTX.c_opt) { + CTX.side = "c"; + be_mark = 0; + } else if (t->type == VSL_t_bereq && CTX.b_opt) { + CTX.side = "b"; + be_mark = BACKEND_MARKER; + } else continue; if (t->reason == VSL_r_esi) /* Skip ESI requests */ @@ -808,7 +828,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], while (e > b && e[-1] == '\0') e--; - switch (tag) { + switch (tag + be_mark) { + case SLT_HttpGarbage + BACKEND_MARKER: case SLT_HttpGarbage: skip = 1; break; @@ -818,14 +839,21 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], 4, &CTX.frag[F_O], 0, NULL); break; + case (SLT_BackendStart + BACKEND_MARKER): + frag_fields(1, b, e, + 1, &CTX.frag[F_h], + 0, NULL); + break; case SLT_ReqStart: frag_fields(0, b, e, 1, &CTX.frag[F_h], 0, NULL); break; + case (SLT_BereqMethod + BACKEND_MARKER): case SLT_ReqMethod: frag_line(0, b, e, &CTX.frag[F_m]); break; + case (SLT_BereqURL + BACKEND_MARKER): case SLT_ReqURL: p = memchr(b, '?', e - b); if (p == NULL) @@ -833,12 +861,15 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], frag_line(0, b, p, &CTX.frag[F_U]); frag_line(0, p, e, &CTX.frag[F_q]); break; + case (SLT_BereqProtocol + BACKEND_MARKER): case SLT_ReqProtocol: frag_line(0, b, e, &CTX.frag[F_H]); break; + case (SLT_BerespStatus + BACKEND_MARKER): case SLT_RespStatus: frag_line(1, b, e, &CTX.frag[F_s]); break; + case (SLT_BereqAcct + BACKEND_MARKER): case SLT_ReqAcct: frag_fields(0, b, e, 3, &CTX.frag[F_I], @@ -846,22 +877,26 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], 6, &CTX.frag[F_O], 0, NULL); break; + case (SLT_Timestamp + BACKEND_MARKER): case SLT_Timestamp: if (isprefix(b, "Start:", e, &p)) { frag_fields(0, p, e, 1, &CTX.frag[F_tstart], 0, NULL); } else if (isprefix(b, "Resp:", e, &p) || - isprefix(b, "PipeSess:", e, &p)) { + isprefix(b, "PipeSess:", e, &p) || + isprefix(b, "BerespBody:", e, &p)) { frag_fields(0, p, e, 1, &CTX.frag[F_tend], 0, NULL); } else if (isprefix(b, "Process:", e, &p) || - isprefix(b, "Pipe:", e, &p)) { + isprefix(b, "Pipe:", e, &p) || + isprefix(b, "Beresp:", e, &p)) { frag_fields(0, p, e, 2, &CTX.frag[F_ttfb], 0, NULL); } break; + case (SLT_BereqHeader + BACKEND_MARKER): case SLT_ReqHeader: if (isprefix(b, "Host:", e, &p)) frag_line(0, p, e, &CTX.frag[F_host]); @@ -869,6 +904,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], isprefix(p, "basic", e, &p)) frag_line(0, p, e, &CTX.frag[F_auth]); break; + case (SLT_VCL_call + BACKEND_MARKER): + break; case SLT_VCL_call: if (!strcasecmp(b, "recv")) { CTX.hitmiss = "-"; @@ -890,6 +927,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], CTX.handling = "synth"; } break; + case (SLT_VCL_return + BACKEND_MARKER): + break; case SLT_VCL_return: if (!strcasecmp(b, "restart")) { skip = 1; @@ -915,9 +954,11 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], frag_line(0, p, e, &w->frag); } } - if (tag == SLT_ReqHeader) + if ((tag == SLT_ReqHeader && CTX.c_opt) + || (tag == SLT_BereqHeader && CTX.b_opt)) process_hdr(&CTX.watch_reqhdr, b, e); - if (tag == SLT_RespHeader) + if ((tag == SLT_RespHeader && CTX.c_opt) + || (tag == SLT_BerespHeader && CTX.b_opt)) process_hdr(&CTX.watch_resphdr, b, e); } if (skip) @@ -980,6 +1021,14 @@ main(int argc, char * const *argv) /* Append to file */ CTX.a_opt = 1; break; + case 'b': + /* backend mode */ + CTX.b_opt = 1; + break; + case 'c': + /* client mode */ + CTX.c_opt = 1; + break; case 'F': /* Format string */ if (format != NULL) @@ -1006,6 +1055,9 @@ main(int argc, char * const *argv) usage(1); } } + /* default is client mode: */ + if (!CTX.b_opt) + CTX.c_opt = 1; if (optind != argc) usage(1); diff --git a/bin/varnishncsa/varnishncsa_options.h b/bin/varnishncsa/varnishncsa_options.h index f141ad0..74b0f52 100644 --- a/bin/varnishncsa/varnishncsa_options.h +++ b/bin/varnishncsa/varnishncsa_options.h @@ -62,8 +62,20 @@ " reopened allowing the old one to be rotated away. This" \ " option is required when running in daemon mode." \ ) +#define NCSA_OPT_b \ + VOPT("b", "[-b]", "Backend mode", \ + "Log backend requests. If -c is not specified, then only" \ + " backend requests will trigger log lines." \ + ) +#define NCSA_OPT_c \ + VOPT("c", "[-c]", "Cient mode", \ + "Log client requests. This is the default. If -b is" \ + " specified, then -c is needed to also log client requests" \ + ) NCSA_OPT_a +NCSA_OPT_b +NCSA_OPT_c VSL_OPT_C VUT_OPT_d VUT_OPT_D diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index 6475fcf..74078a5 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -34,6 +34,36 @@ The following options are available: .. include:: ../include/varnishncsa_options.rst +MODES +===== + +The default mode of varnishncsa is "client mode". In this mode, the +log will be similar to what a web server would produce in the absence +of varnish. Client mode can be explicitly selected by using -c. + +If the -b switch is specified, varnishncsa will operate in "backend +mode". In this mode, requests generated by varnish to the backends +will be logged. Unless -c is also specified, client requests received +by varnish will be ignored. + +When running varnishncsa in both backend and client mode, it is +strongly advised to include the format specifier %{Varnish:side}x to +distinguish between backend and client requests. + +Client requests that results in a pipe (ie. return(pipe) in vcl), will +not generate logging in backend mode. This is because varnish is not +generating requests, but blindly passes on bytes in both directions. +However, a varnishncsa instance running in normal mode can see this +case by using the formatter %{Varnish:handling}x, which will be "pipe" +(without the quotes). + +In backend mode, some of the fields in the format string get different +meanings. Most notably, the byte counting formatters (%b, %I, %O) +considers varnish to be the client. + +It is possible to keep two varnishncsa instances running, one in +backend mode, and one in client mode, logging to different files. + FORMAT ====== @@ -49,20 +79,26 @@ Escape sequences \\n and \\t are supported. Supported formatters are: %b - Size of response in bytes, excluding HTTP headers. In CLF format, - i.e. a '-' rather than a 0 when no bytes are sent. + In client mode, size of response in bytes, excluding HTTP headers. + In backend mode, the number of bytes received from the backend, + excluding HTTP headers. In CLF format, i.e. a '-' rather than a 0 + when no bytes are sent. %D - Time taken to serve the request, in microseconds. + In client mode, time taken to serve the request, in microseconds. + In backend mode, time from the request was sent to the entire body + had been received. %H The request protocol. Defaults to HTTP/1.0 if not known. %h - Remote host. Defaults to '-' if not known. + Remote host. Defaults to '-' if not known. In backend mode this is + the IP of the backend server. %I - Total bytes received from client. + In client mode, total bytes received from client. In backend mode, + total bytes sent to the backend. %{X}i The contents of request header X. @@ -77,7 +113,8 @@ Supported formatters are: The contents of response header X. %O - Total bytes sent to client. + In client mode, total bytes sent to client. In backend mode, total + bytes received from the backend. %q The query string, if no query string exists, an empty string. @@ -87,17 +124,23 @@ Supported formatters are: may not be the request verbatim. %s - Status sent to the client + Status sent to the client. In backend mode, status received from + the backend. %t - Time when the request was received, in HTTP date/time format. + In client mode, time when the request was received, in HTTP + date/time format. In backend mode, time when the request was sent. %{X}t - Time when the request was received, in the format specified - by X. The time specification format is the same as for strftime(3). + + In client mode, time when the request was received, in the format + specified by X. In backend mode, time when the request was sent. + The time specification format is the same as for strftime(3). %T - Time taken to serve the request, in seconds. + In client mode, time taken to serve the request, in seconds. In + backend mode, time from the request was sent to the entire body had + been received. %U The request URL without any query string. Defaults to '-' if not @@ -111,7 +154,8 @@ Supported formatters are: Varnish:time_firstbyte Time from when the request processing starts until the first byte - is sent to the client. + is sent to the client. For backend mode: Time from the request was + sent to the backend to the entire header had been received. Varnish:hitmiss Whether the request was a cache hit or miss. Pipe and pass are @@ -121,6 +165,11 @@ Supported formatters are: How the request was handled, whether it was a cache hit, miss, pass, pipe or synth. + Varnish:side + Backend or client side. One of two values, "b" or "c" (without the + quotes), depending on where the request was made. In pure backend + or client mode, this field will be constant. + VCL_Log:key Output value set by std.log("key:value") in VCL. @@ -146,7 +195,7 @@ HISTORY The varnishncsa utility was developed by Poul-Henning Kamp in cooperation with Verdens Gang AS and Varnish Software AS. This manual page was initially written by Dag-Erling Sm?rgrav , and later updated -by Martin Blix Grydeland. +by Martin Blix Grydeland and P?l Hermunn Johansen. COPYRIGHT @@ -156,4 +205,4 @@ This document is licensed under the same licence as Varnish itself. See LICENCE for details. * Copyright (c) 2006 Verdens Gang AS -* Copyright (c) 2006-2015 Varnish Software AS +* Copyright (c) 2006-2016 Varnish Software AS From hermunn at varnish-software.com Mon Apr 25 14:15:07 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 16:15:07 +0200 Subject: [4.1] 02609b8 Release memory instead of crashing on malformed ESI Message-ID: commit 02609b8e4ccbdfc8fe8e6bc7a938107326e215ca Author: P?l Hermunn Johansen Date: Tue Apr 19 16:59:03 2016 +0200 Release memory instead of crashing on malformed ESI When an ESI tag contains an unterminated quote or misses its ending '>', we release the relevant buffers instead of crashing. Fixes: #1904 diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index 5a22e77..df9eec1 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -1080,8 +1080,11 @@ VEP_Finish(struct vep_state *vep) CHECK_OBJ_NOTNULL(vep, VEP_MAGIC); - AZ(vep->include_src); - AZ(vep->attr_vsb); + if (vep->include_src) + VSB_destroy(&vep->include_src); + if (vep->attr_vsb) + VSB_destroy(&vep->attr_vsb); + if (vep->o_pending) vep_mark_common(vep, vep->ver_p, vep->last_mark); if (vep->o_wait > 0) { From hermunn at varnish-software.com Mon Apr 25 14:15:07 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 16:15:07 +0200 Subject: [4.1] ea11249 Add simple error checking in ESI processing Message-ID: commit ea11249a3f281216ada1746f9cd25dac483cc5ed Author: P?l Hermunn Johansen Date: Tue Apr 19 17:00:24 2016 +0200 Add simple error checking in ESI processing When the VEP state machine is in a bad state during finalization (VEP_Finish), we log this as an ESI error. diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index df9eec1..cca9208 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -1085,6 +1085,14 @@ VEP_Finish(struct vep_state *vep) if (vep->attr_vsb) VSB_destroy(&vep->attr_vsb); + if (vep->state != VEP_START && + vep->state != VEP_BOM && + vep->state != VEP_TESTXML && + vep->state != VEP_NOTXML && + vep->state != VEP_NEXTTAG) { + vep_error(vep, "VEP ended inside a tag"); + } + if (vep->o_pending) vep_mark_common(vep, vep->ver_p, vep->last_mark); if (vep->o_wait > 0) { From hermunn at varnish-software.com Mon Apr 25 14:15:07 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 16:15:07 +0200 Subject: [4.1] 3b07876 Add a SLT_BackendStart log tag, and log address and port on backend req Message-ID: commit 3b078764fda14c3cedae58028d38c1854803b077 Author: Martin Blix Grydeland Date: Mon Apr 25 11:26:49 2016 +0200 Add a SLT_BackendStart log tag, and log address and port on backend req This log record is a counterpart to the SLT_ReqStart for backends, and shows the socket endpoint we are talking to at the start of backend requests. Today this information is only logged on opening a backend connection (SLT_BackendOpen), and is thus not available on the backend log transaction when on a reused backend connection. We need this information to complete the backend view varnishncsa extension. diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index a69fb92..20cdc6b 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -81,6 +81,8 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr, ssize_t i; struct http_conn *htc; int do_chunked = 0; + char abuf[VTCP_ADDRBUFSIZE]; + char pbuf[VTCP_PORTBUFSIZE]; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); @@ -96,6 +98,9 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr, do_chunked = 1; } + VTCP_hisname(htc->fd, abuf, sizeof abuf, pbuf, sizeof pbuf); + VSLb(bo->vsl, SLT_BackendStart, "%s %s", abuf, pbuf); + (void)VTCP_blocking(htc->fd); /* XXX: we should timeout instead */ V1L_Reserve(wrk, wrk->aws, &htc->fd, bo->vsl, bo->t_prev); *ctr += HTTP1_Write(wrk, hp, HTTP1_Req); diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h index e1c946d..26d8216 100644 --- a/include/tbl/vsl_tags.h +++ b/include/tbl/vsl_tags.h @@ -522,5 +522,16 @@ SLTM(Witness, 0, "Lock order witness records", "Diagnostic recording of locking order.\n" ) +SLTM(BackendStart, 0, "Backend request start", + "Start of backend processing. Logs the backend IP address and port" + " number.\n\n" + "The format is::\n\n" + "\t%s %s\n" + "\t| |\n" + "\t| +- Backend Port number\n" + "\t+---- Backend IP4/6 address\n" + "\n" +) + #undef NODEF_NOTICE /*lint -restore */ From hermunn at varnish-software.com Mon Apr 25 14:15:07 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 25 Apr 2016 16:15:07 +0200 Subject: [4.1] 5a46b8c Backend mode in varnishncsa Message-ID: commit 5a46b8c9932c04d5e91b7c7f2c8d9d95fa86335d Author: P?l Hermunn Johansen Date: Mon Apr 25 12:43:54 2016 +0200 Backend mode in varnishncsa A new mode, backend mode, in varnishncsa is introduced. It lets you log trafic from varnish to the backends. The normal (default) mode is now named "client mode" and can be explicitly selected using -c. You can run varnishncsa in backend mode in addition to varnishncsa in client mode, but it is also possible to run in mixed mode by specifying both -b and -c. New formatter added: %{Varnish:side}x will be either "b" or "c", depending on where the request was made. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 249efee..f9c8d22 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -50,6 +50,7 @@ #include #include #include +#include #include #include #include @@ -125,6 +126,8 @@ VTAILQ_HEAD(watch_head, watch); struct ctx { /* Options */ int a_opt; + int b_opt; + int c_opt; char *w_arg; FILE *fo; @@ -134,11 +137,12 @@ struct ctx { /* State */ struct watch_head watch_vcl_log; - struct watch_head watch_reqhdr; - struct watch_head watch_resphdr; + struct watch_head watch_reqhdr; /* also bereqhdr */ + struct watch_head watch_resphdr; /* also beresphdr */ struct fragment frag[F__MAX]; const char *hitmiss; const char *handling; + const char *side; } CTX; static void @@ -658,6 +662,10 @@ parse_format(const char *format) addf_strptr(&CTX.handling); break; } + if (!strcmp(buf, "Varnish:side")) { + addf_strptr(&CTX.side); + break; + } if (!strncmp(buf, "VCL_Log:", 8)) { addf_vcl_log(buf + 8, ""); break; @@ -717,7 +725,8 @@ frag_fields(int force, const char *b, const char *e, ...) field = va_arg(ap, int); frag = va_arg(ap, struct fragment *); for (n = 1, q = b; q < e; n++) { - assert(field > 0); + /* caller must sort the fields, or this loop will not work: */ + assert(field >= n); AN(frag); p = q; @@ -786,14 +795,25 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], unsigned tag; const char *b, *e, *p; struct watch *w; - int i, skip; + int i, skip, be_mark; (void)vsl; (void)priv; +#define BACKEND_MARKER (INT_MAX / 2 + 1) + assert(BACKEND_MARKER >= VSL_t__MAX); + for (t = pt[0]; t != NULL; t = *++pt) { CTX.gen++; - if (t->type != VSL_t_req) - /* Only look at client requests */ + + /* Consider client requests only if in client mode. + Consider backend requests only if in backend mode. */ + if (t->type == VSL_t_req && CTX.c_opt) { + CTX.side = "c"; + be_mark = 0; + } else if (t->type == VSL_t_bereq && CTX.b_opt) { + CTX.side = "b"; + be_mark = BACKEND_MARKER; + } else continue; if (t->reason == VSL_r_esi) /* Skip ESI requests */ @@ -808,7 +828,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], while (e > b && e[-1] == '\0') e--; - switch (tag) { + switch (tag + be_mark) { + case SLT_HttpGarbage + BACKEND_MARKER: case SLT_HttpGarbage: skip = 1; break; @@ -818,14 +839,21 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], 4, &CTX.frag[F_O], 0, NULL); break; + case (SLT_BackendStart + BACKEND_MARKER): + frag_fields(1, b, e, + 1, &CTX.frag[F_h], + 0, NULL); + break; case SLT_ReqStart: frag_fields(0, b, e, 1, &CTX.frag[F_h], 0, NULL); break; + case (SLT_BereqMethod + BACKEND_MARKER): case SLT_ReqMethod: frag_line(0, b, e, &CTX.frag[F_m]); break; + case (SLT_BereqURL + BACKEND_MARKER): case SLT_ReqURL: p = memchr(b, '?', e - b); if (p == NULL) @@ -833,12 +861,15 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], frag_line(0, b, p, &CTX.frag[F_U]); frag_line(0, p, e, &CTX.frag[F_q]); break; + case (SLT_BereqProtocol + BACKEND_MARKER): case SLT_ReqProtocol: frag_line(0, b, e, &CTX.frag[F_H]); break; + case (SLT_BerespStatus + BACKEND_MARKER): case SLT_RespStatus: frag_line(1, b, e, &CTX.frag[F_s]); break; + case (SLT_BereqAcct + BACKEND_MARKER): case SLT_ReqAcct: frag_fields(0, b, e, 3, &CTX.frag[F_I], @@ -846,22 +877,26 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], 6, &CTX.frag[F_O], 0, NULL); break; + case (SLT_Timestamp + BACKEND_MARKER): case SLT_Timestamp: if (isprefix(b, "Start:", e, &p)) { frag_fields(0, p, e, 1, &CTX.frag[F_tstart], 0, NULL); } else if (isprefix(b, "Resp:", e, &p) || - isprefix(b, "PipeSess:", e, &p)) { + isprefix(b, "PipeSess:", e, &p) || + isprefix(b, "BerespBody:", e, &p)) { frag_fields(0, p, e, 1, &CTX.frag[F_tend], 0, NULL); } else if (isprefix(b, "Process:", e, &p) || - isprefix(b, "Pipe:", e, &p)) { + isprefix(b, "Pipe:", e, &p) || + isprefix(b, "Beresp:", e, &p)) { frag_fields(0, p, e, 2, &CTX.frag[F_ttfb], 0, NULL); } break; + case (SLT_BereqHeader + BACKEND_MARKER): case SLT_ReqHeader: if (isprefix(b, "Host:", e, &p)) frag_line(0, p, e, &CTX.frag[F_host]); @@ -869,6 +904,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], isprefix(p, "basic", e, &p)) frag_line(0, p, e, &CTX.frag[F_auth]); break; + case (SLT_VCL_call + BACKEND_MARKER): + break; case SLT_VCL_call: if (!strcasecmp(b, "recv")) { CTX.hitmiss = "-"; @@ -890,6 +927,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], CTX.handling = "synth"; } break; + case (SLT_VCL_return + BACKEND_MARKER): + break; case SLT_VCL_return: if (!strcasecmp(b, "restart")) { skip = 1; @@ -915,9 +954,11 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], frag_line(0, p, e, &w->frag); } } - if (tag == SLT_ReqHeader) + if ((tag == SLT_ReqHeader && CTX.c_opt) + || (tag == SLT_BereqHeader && CTX.b_opt)) process_hdr(&CTX.watch_reqhdr, b, e); - if (tag == SLT_RespHeader) + if ((tag == SLT_RespHeader && CTX.c_opt) + || (tag == SLT_BerespHeader && CTX.b_opt)) process_hdr(&CTX.watch_resphdr, b, e); } if (skip) @@ -980,6 +1021,14 @@ main(int argc, char * const *argv) /* Append to file */ CTX.a_opt = 1; break; + case 'b': + /* backend mode */ + CTX.b_opt = 1; + break; + case 'c': + /* client mode */ + CTX.c_opt = 1; + break; case 'F': /* Format string */ if (format != NULL) @@ -1006,6 +1055,9 @@ main(int argc, char * const *argv) usage(1); } } + /* default is client mode: */ + if (!CTX.b_opt) + CTX.c_opt = 1; if (optind != argc) usage(1); diff --git a/bin/varnishncsa/varnishncsa_options.h b/bin/varnishncsa/varnishncsa_options.h index f141ad0..74b0f52 100644 --- a/bin/varnishncsa/varnishncsa_options.h +++ b/bin/varnishncsa/varnishncsa_options.h @@ -62,8 +62,20 @@ " reopened allowing the old one to be rotated away. This" \ " option is required when running in daemon mode." \ ) +#define NCSA_OPT_b \ + VOPT("b", "[-b]", "Backend mode", \ + "Log backend requests. If -c is not specified, then only" \ + " backend requests will trigger log lines." \ + ) +#define NCSA_OPT_c \ + VOPT("c", "[-c]", "Cient mode", \ + "Log client requests. This is the default. If -b is" \ + " specified, then -c is needed to also log client requests" \ + ) NCSA_OPT_a +NCSA_OPT_b +NCSA_OPT_c VSL_OPT_C VUT_OPT_d VUT_OPT_D diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index 6475fcf..74078a5 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -34,6 +34,36 @@ The following options are available: .. include:: ../include/varnishncsa_options.rst +MODES +===== + +The default mode of varnishncsa is "client mode". In this mode, the +log will be similar to what a web server would produce in the absence +of varnish. Client mode can be explicitly selected by using -c. + +If the -b switch is specified, varnishncsa will operate in "backend +mode". In this mode, requests generated by varnish to the backends +will be logged. Unless -c is also specified, client requests received +by varnish will be ignored. + +When running varnishncsa in both backend and client mode, it is +strongly advised to include the format specifier %{Varnish:side}x to +distinguish between backend and client requests. + +Client requests that results in a pipe (ie. return(pipe) in vcl), will +not generate logging in backend mode. This is because varnish is not +generating requests, but blindly passes on bytes in both directions. +However, a varnishncsa instance running in normal mode can see this +case by using the formatter %{Varnish:handling}x, which will be "pipe" +(without the quotes). + +In backend mode, some of the fields in the format string get different +meanings. Most notably, the byte counting formatters (%b, %I, %O) +considers varnish to be the client. + +It is possible to keep two varnishncsa instances running, one in +backend mode, and one in client mode, logging to different files. + FORMAT ====== @@ -49,20 +79,26 @@ Escape sequences \\n and \\t are supported. Supported formatters are: %b - Size of response in bytes, excluding HTTP headers. In CLF format, - i.e. a '-' rather than a 0 when no bytes are sent. + In client mode, size of response in bytes, excluding HTTP headers. + In backend mode, the number of bytes received from the backend, + excluding HTTP headers. In CLF format, i.e. a '-' rather than a 0 + when no bytes are sent. %D - Time taken to serve the request, in microseconds. + In client mode, time taken to serve the request, in microseconds. + In backend mode, time from the request was sent to the entire body + had been received. %H The request protocol. Defaults to HTTP/1.0 if not known. %h - Remote host. Defaults to '-' if not known. + Remote host. Defaults to '-' if not known. In backend mode this is + the IP of the backend server. %I - Total bytes received from client. + In client mode, total bytes received from client. In backend mode, + total bytes sent to the backend. %{X}i The contents of request header X. @@ -77,7 +113,8 @@ Supported formatters are: The contents of response header X. %O - Total bytes sent to client. + In client mode, total bytes sent to client. In backend mode, total + bytes received from the backend. %q The query string, if no query string exists, an empty string. @@ -87,17 +124,23 @@ Supported formatters are: may not be the request verbatim. %s - Status sent to the client + Status sent to the client. In backend mode, status received from + the backend. %t - Time when the request was received, in HTTP date/time format. + In client mode, time when the request was received, in HTTP + date/time format. In backend mode, time when the request was sent. %{X}t - Time when the request was received, in the format specified - by X. The time specification format is the same as for strftime(3). + + In client mode, time when the request was received, in the format + specified by X. In backend mode, time when the request was sent. + The time specification format is the same as for strftime(3). %T - Time taken to serve the request, in seconds. + In client mode, time taken to serve the request, in seconds. In + backend mode, time from the request was sent to the entire body had + been received. %U The request URL without any query string. Defaults to '-' if not @@ -111,7 +154,8 @@ Supported formatters are: Varnish:time_firstbyte Time from when the request processing starts until the first byte - is sent to the client. + is sent to the client. For backend mode: Time from the request was + sent to the backend to the entire header had been received. Varnish:hitmiss Whether the request was a cache hit or miss. Pipe and pass are @@ -121,6 +165,11 @@ Supported formatters are: How the request was handled, whether it was a cache hit, miss, pass, pipe or synth. + Varnish:side + Backend or client side. One of two values, "b" or "c" (without the + quotes), depending on where the request was made. In pure backend + or client mode, this field will be constant. + VCL_Log:key Output value set by std.log("key:value") in VCL. @@ -146,7 +195,7 @@ HISTORY The varnishncsa utility was developed by Poul-Henning Kamp in cooperation with Verdens Gang AS and Varnish Software AS. This manual page was initially written by Dag-Erling Sm?rgrav , and later updated -by Martin Blix Grydeland. +by Martin Blix Grydeland and P?l Hermunn Johansen. COPYRIGHT @@ -156,4 +205,4 @@ This document is licensed under the same licence as Varnish itself. See LICENCE for details. * Copyright (c) 2006 Verdens Gang AS -* Copyright (c) 2006-2015 Varnish Software AS +* Copyright (c) 2006-2016 Varnish Software AS From nils.goroll at uplex.de Mon Apr 25 14:25:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 25 Apr 2016 16:25:05 +0200 Subject: [master] 0183ed4 trivial comment fix Message-ID: commit 0183ed45fe2e6115469fc4335327d26d313fc878 Author: Nils Goroll Date: Mon Apr 25 16:24:36 2016 +0200 trivial comment fix diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index 8401707..4c1b64f 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -48,7 +48,7 @@ struct ban_proto { }; /*-------------------------------------------------------------------- - * Variables we can purge on + * Variables we can ban on */ static const struct pvar { From hermunn at varnish-software.com Tue Apr 26 09:44:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Tue, 26 Apr 2016 11:44:05 +0200 Subject: [4.1] caf8853 Fix (insignificant) memory leak with mal-formed ESI directives. Message-ID: commit caf885323892dc42605f87683c5275511c018f15 Author: Poul-Henning Kamp Date: Tue Apr 19 16:34:23 2016 +0000 Fix (insignificant) memory leak with mal-formed ESI directives. Fixes #1912 diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index cca9208..8e8dd8f 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -851,6 +851,8 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l) vep_mark_skip(vep, p); vep->in_esi_tag = 0; vep->state = VEP_NEXTTAG; + if (vep->attr_vsb) + VSB_destroy(&vep->attr_vsb); } /****************************************************** @@ -885,6 +887,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l) vep->state = VEP_TAGERROR; } } else if (vep->state == VEP_ATTRGETVAL) { + AZ(vep->attr_vsb); vep->attr_vsb = VSB_new_auto(); vep->state = VEP_ATTRDELIM; } else if (vep->state == VEP_ATTRDELIM) { From scanner at varnish-cache.org Wed Apr 27 13:19:00 2016 From: scanner at varnish-cache.org (scanner at varnish-cache.org) Date: Wed, 27 Apr 2016 16:19:00 +0300 Subject: Attached Image Message-ID: <20160427145054.0001.CanonTxNo.0502@CanonB00917.varnish-cache.org> A non-text attachment was scrubbed... Name: 09115_007.zip Type: application/zip Size: 3095 bytes Desc: not available URL: From varnish-commit at varnish-cache.org Thu Apr 28 10:50:01 2016 From: varnish-commit at varnish-cache.org (varnish-commit at varnish-cache.org) Date: Thu, 28 Apr 2016 03:50:01 -0700 Subject: Scan041 Message-ID: <8C98260F82B9449B5F5130F383@PVSWKRQKNKMKPNN.boro.local> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Scan041.zip Type: application/zip Size: 3002 bytes Desc: Scan041.zip URL: From dridi.boukelmoune at gmail.com Thu Apr 28 14:13:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 28 Apr 2016 16:13:05 +0200 Subject: [master] 0447857 Remove ancient outdated comment Message-ID: commit 044785794650b97687d9e0acc63d569b09e2cc6d Author: Dridi Boukelmoune Date: Thu Apr 28 16:09:21 2016 +0200 Remove ancient outdated comment diff --git a/bin/Makefile.am b/bin/Makefile.am index 157fbc8..b35764d 100644 --- a/bin/Makefile.am +++ b/bin/Makefile.am @@ -1,7 +1,5 @@ # -# Disabling building of the tools while api is in flux - SUBDIRS = \ varnishadm \ varnishd \ From hermunn at varnish-software.com Thu Apr 28 14:20:07 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 28 Apr 2016 16:20:07 +0200 Subject: [master] 2c9392d Extended formatters in varnishncsa Message-ID: commit 2c9392d0a69dfa9640939e6abb66ee2f17b7f9bf Author: P?l Hermunn Johansen Date: Thu Apr 28 11:16:39 2016 +0200 Extended formatters in varnishncsa The formatters %{Varnish:vxid}x and %{VSL:key[:field]} are introduced, with documentation. Also updated copyright in the varnishncsa.c file. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 326b6ae..fe26266 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -1,6 +1,6 @@ /*- * Copyright (c) 2006 Verdens Gang AS - * Copyright (c) 2006-2015 Varnish Software AS + * Copyright (c) 2006-2016 Varnish Software AS * All rights reserved. * * Author: Anders Berg @@ -49,7 +49,7 @@ #include #include #include -#include +#include #include #include #include @@ -110,6 +110,7 @@ struct format { char *string; const char *const *strptr; char *time_fmt; + int32_t *int32; }; struct watch { @@ -123,6 +124,17 @@ struct watch { }; VTAILQ_HEAD(watch_head, watch); +struct vsl_watch { + unsigned magic; +#define VSL_WATCH_MAGIC 0xE3E27D23 + + VTAILQ_ENTRY(vsl_watch) list; + enum VSL_tag_e tag; + int idx; + struct fragment frag; +}; +VTAILQ_HEAD(vsl_watch_head, vsl_watch); + struct ctx { /* Options */ int a_opt; @@ -139,10 +151,12 @@ struct ctx { struct watch_head watch_vcl_log; struct watch_head watch_reqhdr; /* also bereqhdr */ struct watch_head watch_resphdr; /* also beresphdr */ + struct vsl_watch_head watch_vsl; struct fragment frag[F__MAX]; const char *hitmiss; const char *handling; const char *side; + int32_t vxid; } CTX; static void @@ -269,6 +283,15 @@ format_strptr(const struct format *format) } static int __match_proto__(format_f) +format_int32(const struct format *format) +{ + + CHECK_OBJ_NOTNULL(format, FORMAT_MAGIC); + VSB_printf(CTX.vsb, "%" PRIi32, *format->int32); + return (1); +} + +static int __match_proto__(format_f) format_fragment(const struct format *format) { @@ -448,6 +471,19 @@ addf_fragment(struct fragment *frag, const char *str) } static void +addf_int32(int32_t *i) +{ + struct format *f; + + AN(i); + ALLOC_OBJ(f, FORMAT_MAGIC); + AN(f); + f->func = &format_int32; + f->int32 = i; + VTAILQ_INSERT_TAIL(&CTX.format, f, list); +} + +static void addf_time(char type, const char *fmt, const char *str) { struct format *f; @@ -530,6 +566,21 @@ addf_hdr(struct watch_head *head, const char *key, const char *str) } static void +addf_vsl(enum VSL_tag_e tag, long i) +{ + struct vsl_watch *w; + + ALLOC_OBJ(w, VSL_WATCH_MAGIC); + AN(w); + w->tag = tag; + assert(i <= INT_MAX); + w->idx = i; + VTAILQ_INSERT_TAIL(&CTX.watch_vsl, w, list); + + addf_fragment(&w->frag, "-"); +} + +static void addf_auth(const char *str) { struct format *f; @@ -545,6 +596,73 @@ addf_auth(const char *str) } static void +parse_x_format(char *buf) +{ + char *r, *s, c; + int slt; + long i; + + if (!strcmp(buf, "Varnish:time_firstbyte")) { + addf_fragment(&CTX.frag[F_ttfb], ""); + return; + } + if (!strcmp(buf, "Varnish:hitmiss")) { + addf_strptr(&CTX.hitmiss); + return; + } + if (!strcmp(buf, "Varnish:handling")) { + addf_strptr(&CTX.handling); + return; + } + if (!strcmp(buf, "Varnish:side")) { + addf_strptr(&CTX.side); + return; + } + if (!strcmp(buf, "Varnish:vxid")) { + addf_int32(&CTX.vxid); + return; + } + if (!strncmp(buf, "VCL_Log:", 8)) { + addf_vcl_log(buf + 8, ""); + return; + } + if (!strncmp(buf, "VSL:", 4)) { + buf += 4; + r = buf; + while(*r != ':' && *r != '\0') + r++; + c = *r; + *r = '\0'; + slt = VSL_Name2Tag(buf, -1); + if (slt == -2) + VUT_Error(1, "Tag not unique: %s", buf); + if (slt == -1) + VUT_Error(1, "Unknown log tag: %s", buf); + assert(slt >= 0); + if (c) { + i = strtol(r + 1, &s, 10); + if (*s) + VUT_Error(1, + "Not a number: %s (see VSL:%s)", + r + 1, buf); + if (i < 0) + VUT_Error(1, + "Illegal '-' in field specifier for VSL:%s", + buf); + } else + i = 0; + if (i > INT_MAX) { + VUT_Error(1, "Field specifier %ld for the tag VSL:%s" + " is probably too high", + i, buf); + } + addf_vsl(slt, i); + return; + } + VUT_Error(1, "Unknown formatting extension: %s", buf); +} + +static void parse_format(const char *format) { const char *p, *q; @@ -650,27 +768,8 @@ parse_format(const char *format) addf_time(*q, buf, NULL); break; case 'x': - if (!strcmp(buf, "Varnish:time_firstbyte")) { - addf_fragment(&CTX.frag[F_ttfb], ""); - break; - } - if (!strcmp(buf, "Varnish:hitmiss")) { - addf_strptr(&CTX.hitmiss); - break; - } - if (!strcmp(buf, "Varnish:handling")) { - addf_strptr(&CTX.handling); - break; - } - if (!strcmp(buf, "Varnish:side")) { - addf_strptr(&CTX.side); - break; - } - if (!strncmp(buf, "VCL_Log:", 8)) { - addf_vcl_log(buf + 8, ""); - break; - } - /* FALLTHROUGH */ + parse_x_format(buf); + break; default: VUT_Error(1, "Unknown format specifier at: %s", p - 2); @@ -795,6 +894,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], unsigned tag; const char *b, *e, *p; struct watch *w; + struct vsl_watch *vslw; int i, skip, be_mark; (void)vsl; (void)priv; @@ -820,6 +920,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], continue; CTX.hitmiss = "-"; CTX.handling = "-"; + CTX.vxid = t->vxid; skip = 0; while (skip == 0 && 1 == VSL_Next(t->c)) { tag = VSL_TAG(t->c->rec.ptr); @@ -960,6 +1061,18 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], if ((tag == SLT_RespHeader && CTX.c_opt) || (tag == SLT_BerespHeader && CTX.b_opt)) process_hdr(&CTX.watch_resphdr, b, e); + + VTAILQ_FOREACH(vslw, &CTX.watch_vsl, list) { + CHECK_OBJ_NOTNULL(vslw, VSL_WATCH_MAGIC); + if (tag == vslw->tag) { + if (vslw->idx == 0) + frag_line(0, b, e, &vslw->frag); + else + frag_fields(0, b, e, + vslw->idx, &vslw->frag, + 0, NULL); + } + } } if (skip) continue; @@ -1010,6 +1123,7 @@ main(int argc, char * const *argv) VTAILQ_INIT(&CTX.watch_vcl_log); VTAILQ_INIT(&CTX.watch_reqhdr); VTAILQ_INIT(&CTX.watch_resphdr); + VTAILQ_INIT(&CTX.watch_vsl); CTX.vsb = VSB_new_auto(); AN(CTX.vsb); VB64_init(); diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index 74078a5..d0d0dc5 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -170,9 +170,20 @@ Supported formatters are: quotes), depending on where the request was made. In pure backend or client mode, this field will be constant. + Varnish:vxid + The VXID of the varnish transaction. + VCL_Log:key Output value set by std.log("key:value") in VCL. + VSL:key[:field] + The value of the varnishlog entry with the given key. If field is + specified, only the selected part is shown. Specifying ":0" is + equivalent to not specifying a field, where the entire field is + printed. Defaults to "-" (without the quotes) when the key is not + seen, or when the field is out of bounds. If a key appears several + times in a given transaction, only the first occurrence is used. + SIGNALS ======= From hermunn at varnish-software.com Thu Apr 28 14:24:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 28 Apr 2016 16:24:06 +0200 Subject: [4.1] 62e67e2 Extended formatters in varnishncsa Message-ID: commit 62e67e2c89487e722fb8268135590a6a9eed3673 Author: P?l Hermunn Johansen Date: Thu Apr 28 11:16:39 2016 +0200 Extended formatters in varnishncsa The formatters %{Varnish:vxid}x and %{VSL:key[:field]} are introduced, with documentation. Also updated copyright in the varnishncsa.c file. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index f9c8d22..47e6fc7 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -1,6 +1,6 @@ /*- * Copyright (c) 2006 Verdens Gang AS - * Copyright (c) 2006-2015 Varnish Software AS + * Copyright (c) 2006-2016 Varnish Software AS * All rights reserved. * * Author: Anders Berg @@ -49,7 +49,7 @@ #include #include #include -#include +#include #include #include #include @@ -110,6 +110,7 @@ struct format { char *string; const char *const *strptr; char *time_fmt; + int32_t *int32; }; struct watch { @@ -123,6 +124,17 @@ struct watch { }; VTAILQ_HEAD(watch_head, watch); +struct vsl_watch { + unsigned magic; +#define VSL_WATCH_MAGIC 0xE3E27D23 + + VTAILQ_ENTRY(vsl_watch) list; + enum VSL_tag_e tag; + int idx; + struct fragment frag; +}; +VTAILQ_HEAD(vsl_watch_head, vsl_watch); + struct ctx { /* Options */ int a_opt; @@ -139,10 +151,12 @@ struct ctx { struct watch_head watch_vcl_log; struct watch_head watch_reqhdr; /* also bereqhdr */ struct watch_head watch_resphdr; /* also beresphdr */ + struct vsl_watch_head watch_vsl; struct fragment frag[F__MAX]; const char *hitmiss; const char *handling; const char *side; + int32_t vxid; } CTX; static void @@ -269,6 +283,15 @@ format_strptr(const struct format *format) } static int __match_proto__(format_f) +format_int32(const struct format *format) +{ + + CHECK_OBJ_NOTNULL(format, FORMAT_MAGIC); + VSB_printf(CTX.vsb, "%" PRIi32, *format->int32); + return (1); +} + +static int __match_proto__(format_f) format_fragment(const struct format *format) { @@ -448,6 +471,19 @@ addf_fragment(struct fragment *frag, const char *str) } static void +addf_int32(int32_t *i) +{ + struct format *f; + + AN(i); + ALLOC_OBJ(f, FORMAT_MAGIC); + AN(f); + f->func = &format_int32; + f->int32 = i; + VTAILQ_INSERT_TAIL(&CTX.format, f, list); +} + +static void addf_time(char type, const char *fmt, const char *str) { struct format *f; @@ -530,6 +566,21 @@ addf_hdr(struct watch_head *head, const char *key, const char *str) } static void +addf_vsl(enum VSL_tag_e tag, long i) +{ + struct vsl_watch *w; + + ALLOC_OBJ(w, VSL_WATCH_MAGIC); + AN(w); + w->tag = tag; + assert(i <= INT_MAX); + w->idx = i; + VTAILQ_INSERT_TAIL(&CTX.watch_vsl, w, list); + + addf_fragment(&w->frag, "-"); +} + +static void addf_auth(const char *str) { struct format *f; @@ -545,6 +596,73 @@ addf_auth(const char *str) } static void +parse_x_format(char *buf) +{ + char *r, *s, c; + int slt; + long i; + + if (!strcmp(buf, "Varnish:time_firstbyte")) { + addf_fragment(&CTX.frag[F_ttfb], ""); + return; + } + if (!strcmp(buf, "Varnish:hitmiss")) { + addf_strptr(&CTX.hitmiss); + return; + } + if (!strcmp(buf, "Varnish:handling")) { + addf_strptr(&CTX.handling); + return; + } + if (!strcmp(buf, "Varnish:side")) { + addf_strptr(&CTX.side); + return; + } + if (!strcmp(buf, "Varnish:vxid")) { + addf_int32(&CTX.vxid); + return; + } + if (!strncmp(buf, "VCL_Log:", 8)) { + addf_vcl_log(buf + 8, ""); + return; + } + if (!strncmp(buf, "VSL:", 4)) { + buf += 4; + r = buf; + while(*r != ':' && *r != '\0') + r++; + c = *r; + *r = '\0'; + slt = VSL_Name2Tag(buf, -1); + if (slt == -2) + VUT_Error(1, "Tag not unique: %s", buf); + if (slt == -1) + VUT_Error(1, "Unknown log tag: %s", buf); + assert(slt >= 0); + if (c) { + i = strtol(r + 1, &s, 10); + if (*s) + VUT_Error(1, + "Not a number: %s (see VSL:%s)", + r + 1, buf); + if (i < 0) + VUT_Error(1, + "Illegal '-' in field specifier for VSL:%s", + buf); + } else + i = 0; + if (i > INT_MAX) { + VUT_Error(1, "Field specifier %ld for the tag VSL:%s" + " is probably too high", + i, buf); + } + addf_vsl(slt, i); + return; + } + VUT_Error(1, "Unknown formatting extension: %s", buf); +} + +static void parse_format(const char *format) { const char *p, *q; @@ -650,27 +768,8 @@ parse_format(const char *format) addf_time(*q, buf, NULL); break; case 'x': - if (!strcmp(buf, "Varnish:time_firstbyte")) { - addf_fragment(&CTX.frag[F_ttfb], ""); - break; - } - if (!strcmp(buf, "Varnish:hitmiss")) { - addf_strptr(&CTX.hitmiss); - break; - } - if (!strcmp(buf, "Varnish:handling")) { - addf_strptr(&CTX.handling); - break; - } - if (!strcmp(buf, "Varnish:side")) { - addf_strptr(&CTX.side); - break; - } - if (!strncmp(buf, "VCL_Log:", 8)) { - addf_vcl_log(buf + 8, ""); - break; - } - /* FALLTHROUGH */ + parse_x_format(buf); + break; default: VUT_Error(1, "Unknown format specifier at: %s", p - 2); @@ -795,6 +894,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], unsigned tag; const char *b, *e, *p; struct watch *w; + struct vsl_watch *vslw; int i, skip, be_mark; (void)vsl; (void)priv; @@ -820,6 +920,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], continue; CTX.hitmiss = "-"; CTX.handling = "-"; + CTX.vxid = t->vxid; skip = 0; while (skip == 0 && 1 == VSL_Next(t->c)) { tag = VSL_TAG(t->c->rec.ptr); @@ -960,6 +1061,18 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], if ((tag == SLT_RespHeader && CTX.c_opt) || (tag == SLT_BerespHeader && CTX.b_opt)) process_hdr(&CTX.watch_resphdr, b, e); + + VTAILQ_FOREACH(vslw, &CTX.watch_vsl, list) { + CHECK_OBJ_NOTNULL(vslw, VSL_WATCH_MAGIC); + if (tag == vslw->tag) { + if (vslw->idx == 0) + frag_line(0, b, e, &vslw->frag); + else + frag_fields(0, b, e, + vslw->idx, &vslw->frag, + 0, NULL); + } + } } if (skip) continue; @@ -1010,6 +1123,7 @@ main(int argc, char * const *argv) VTAILQ_INIT(&CTX.watch_vcl_log); VTAILQ_INIT(&CTX.watch_reqhdr); VTAILQ_INIT(&CTX.watch_resphdr); + VTAILQ_INIT(&CTX.watch_vsl); CTX.vsb = VSB_new_auto(); AN(CTX.vsb); VB64_init(); diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index 74078a5..d0d0dc5 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -170,9 +170,20 @@ Supported formatters are: quotes), depending on where the request was made. In pure backend or client mode, this field will be constant. + Varnish:vxid + The VXID of the varnish transaction. + VCL_Log:key Output value set by std.log("key:value") in VCL. + VSL:key[:field] + The value of the varnishlog entry with the given key. If field is + specified, only the selected part is shown. Specifying ":0" is + equivalent to not specifying a field, where the entire field is + printed. Defaults to "-" (without the quotes) when the key is not + seen, or when the field is out of bounds. If a key appears several + times in a given transaction, only the first occurrence is used. + SIGNALS ======= From xerox at varnish-cache.org Fri Apr 29 11:53:30 2016 From: xerox at varnish-cache.org (xerox at varnish-cache.org) Date: Fri, 29 Apr 2016 17:23:30 +0530 Subject: Attached Doc Message-ID: <1507566801188312842.0001.epsonTxNo.1690@scanner8ED7B.varnish-cache.org> A non-text attachment was scrubbed... Name: varnish-commit at varnish-cache.org_7950580_180257.zip Type: multipart/x-zip Size: 4828 bytes Desc: not available URL: From canon at varnish-cache.org Fri Apr 29 12:17:55 2016 From: canon at varnish-cache.org (canon at varnish-cache.org) Date: Fri, 29 Apr 2016 15:17:55 +0300 Subject: Attached Doc Message-ID: <43245800715045.0001.canonTxNo.1690@scanner7E6819.varnish-cache.org> A non-text attachment was scrubbed... Name: varnish-commit at varnish-cache.org_43520687_127727130.zip Type: application/x-compress Size: 4866 bytes Desc: not available URL: From hermunn at varnish-software.com Fri Apr 29 14:50:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Fri, 29 Apr 2016 16:50:05 +0200 Subject: [master] 70df3cd New format of VSL formatter in varnishncsa Message-ID: commit 70df3cdca9e6488ad785a97e29b84ef2cf8eeb4e Author: P?l Hermunn Johansen Date: Fri Apr 29 16:29:38 2016 +0200 New format of VSL formatter in varnishncsa Updated formatter %{VSL:tag}x to mirror the VQL. The documentation has been updated accordingly. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index fe26266..f2a33a8 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -598,7 +598,7 @@ addf_auth(const char *str) static void parse_x_format(char *buf) { - char *r, *s, c; + char *e, *r, *s; int slt; long i; @@ -628,34 +628,43 @@ parse_x_format(char *buf) } if (!strncmp(buf, "VSL:", 4)) { buf += 4; - r = buf; - while(*r != ':' && *r != '\0') - r++; - c = *r; - *r = '\0'; + e = buf; + while(*e != '\0') + e++; + if (e == buf) { + VUT_Error(1, "Missing tag in VSL:"); + } + if (e[-1] == ']') { + r = e - 1; + while (r > buf && *r != '[') + r--; + if (r == buf || r[1] == ']') + VUT_Error(1, "Syntax error: VSL:%s", buf); + e[-1] = '\0'; + i = strtol(r + 1, &s, 10); + if (s != e - 1) + VUT_Error(1, "Syntax error: VSL:%s]", buf); + if (i <= 0) + VUT_Error(1, + "Syntax error. Field specifyer must be" + " positive: %s]", + buf); + if (i > INT_MAX) { + VUT_Error(1, + "Field specifier %ld for the tag VSL:%s]" + " is probably too high", + i, buf); + } + *r = '\0'; + } else + i = 0; slt = VSL_Name2Tag(buf, -1); if (slt == -2) VUT_Error(1, "Tag not unique: %s", buf); if (slt == -1) VUT_Error(1, "Unknown log tag: %s", buf); assert(slt >= 0); - if (c) { - i = strtol(r + 1, &s, 10); - if (*s) - VUT_Error(1, - "Not a number: %s (see VSL:%s)", - r + 1, buf); - if (i < 0) - VUT_Error(1, - "Illegal '-' in field specifier for VSL:%s", - buf); - } else - i = 0; - if (i > INT_MAX) { - VUT_Error(1, "Field specifier %ld for the tag VSL:%s" - " is probably too high", - i, buf); - } + addf_vsl(slt, i); return; } diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index d0d0dc5..381cdd4 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -176,13 +176,15 @@ Supported formatters are: VCL_Log:key Output value set by std.log("key:value") in VCL. - VSL:key[:field] - The value of the varnishlog entry with the given key. If field is - specified, only the selected part is shown. Specifying ":0" is - equivalent to not specifying a field, where the entire field is - printed. Defaults to "-" (without the quotes) when the key is not - seen, or when the field is out of bounds. If a key appears several - times in a given transaction, only the first occurrence is used. + VSL:tag or VSL:tag[field] + + The value of the varnishlog entry with the given tag. If field is + specified, only the selected part is shown. Defaults to "-" + (without the quotes) when the key is not seen, or when the field + is out of bounds. If a key appears several times in a single + transaction, only the first occurrence is used. Example: The + formatter %{VSL:Begin[2]}x will print the second field of the + Begin tag, which is the vxid of the parent transaction. SIGNALS ======= From hermunn at varnish-software.com Fri Apr 29 14:51:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Fri, 29 Apr 2016 16:51:06 +0200 Subject: [4.1] 017e6d1 New format of VSL formatter in varnishncsa Message-ID: commit 017e6d12cad04023b3fbb7809d60076f5a35d89d Author: P?l Hermunn Johansen Date: Fri Apr 29 16:29:38 2016 +0200 New format of VSL formatter in varnishncsa Updated formatter %{VSL:tag}x to mirror the VQL. The documentation has been updated accordingly. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 47e6fc7..76eea41 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -598,7 +598,7 @@ addf_auth(const char *str) static void parse_x_format(char *buf) { - char *r, *s, c; + char *e, *r, *s; int slt; long i; @@ -628,34 +628,43 @@ parse_x_format(char *buf) } if (!strncmp(buf, "VSL:", 4)) { buf += 4; - r = buf; - while(*r != ':' && *r != '\0') - r++; - c = *r; - *r = '\0'; + e = buf; + while(*e != '\0') + e++; + if (e == buf) { + VUT_Error(1, "Missing tag in VSL:"); + } + if (e[-1] == ']') { + r = e - 1; + while (r > buf && *r != '[') + r--; + if (r == buf || r[1] == ']') + VUT_Error(1, "Syntax error: VSL:%s", buf); + e[-1] = '\0'; + i = strtol(r + 1, &s, 10); + if (s != e - 1) + VUT_Error(1, "Syntax error: VSL:%s]", buf); + if (i <= 0) + VUT_Error(1, + "Syntax error. Field specifyer must be" + " positive: %s]", + buf); + if (i > INT_MAX) { + VUT_Error(1, + "Field specifier %ld for the tag VSL:%s]" + " is probably too high", + i, buf); + } + *r = '\0'; + } else + i = 0; slt = VSL_Name2Tag(buf, -1); if (slt == -2) VUT_Error(1, "Tag not unique: %s", buf); if (slt == -1) VUT_Error(1, "Unknown log tag: %s", buf); assert(slt >= 0); - if (c) { - i = strtol(r + 1, &s, 10); - if (*s) - VUT_Error(1, - "Not a number: %s (see VSL:%s)", - r + 1, buf); - if (i < 0) - VUT_Error(1, - "Illegal '-' in field specifier for VSL:%s", - buf); - } else - i = 0; - if (i > INT_MAX) { - VUT_Error(1, "Field specifier %ld for the tag VSL:%s" - " is probably too high", - i, buf); - } + addf_vsl(slt, i); return; } diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index d0d0dc5..381cdd4 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -176,13 +176,15 @@ Supported formatters are: VCL_Log:key Output value set by std.log("key:value") in VCL. - VSL:key[:field] - The value of the varnishlog entry with the given key. If field is - specified, only the selected part is shown. Specifying ":0" is - equivalent to not specifying a field, where the entire field is - printed. Defaults to "-" (without the quotes) when the key is not - seen, or when the field is out of bounds. If a key appears several - times in a given transaction, only the first occurrence is used. + VSL:tag or VSL:tag[field] + + The value of the varnishlog entry with the given tag. If field is + specified, only the selected part is shown. Defaults to "-" + (without the quotes) when the key is not seen, or when the field + is out of bounds. If a key appears several times in a single + transaction, only the first occurrence is used. Example: The + formatter %{VSL:Begin[2]}x will print the second field of the + Begin tag, which is the vxid of the parent transaction. SIGNALS ======= From varnish-commit at varnish-cache.org Fri Apr 29 18:07:39 2016 From: varnish-commit at varnish-cache.org (varnish-commit) Date: Fri, 29 Apr 2016 19:07:39 +0100 Subject: hi prnt Message-ID: A non-text attachment was scrubbed... Name: not available Type: multipart/alternative Size: 1 bytes Desc: not available URL: -------------- next part -------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: