From bjorn at varnish-cache.org  Tue Mar  1 18:03:13 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:03:13 +0100
Subject: [2.1] varnish-2.1.5-3-g5cdfbfc
Message-ID: <E1PuTuj-00031G-IL@project.varnish-software.com>

commit 5cdfbfc561f3503f7734c4199124b2d44ec248d9
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 18:58:24 2011 +0100

    fix typos and link

diff --git a/doc/sphinx/tutorial/cookies.rst b/doc/sphinx/tutorial/cookies.rst
index 5cb8f2c..e8fab9f 100644
--- a/doc/sphinx/tutorial/cookies.rst
+++ b/doc/sphinx/tutorial/cookies.rst
@@ -15,7 +15,7 @@ interest to the server.
 For a lot of web application it makes sense to completely disregard the
 cookies unless you are accessing a special part of the web site. This
 VCL snippet in vcl_recv will disregard cookies unless you are
-accessing /admin/.::
+accessing /admin/::
 
   if ( !( req.url ~ ^/admin/) ) {
     unset req.http.Cookie;
@@ -26,8 +26,8 @@ like removing one out of several cookies, things get
 difficult. Unfortunately Varnish doesn't have good tools for
 manipulating the Cookies. We have to use regular expressions to do the
 work. If you are familiar with regular expressions you'll understand
-whats going on. If you don't I suggest you either pick up a book on
-the subject, read through the *pcrepattern* man page or read through
+what's going on. If you don't, I suggest you either pick up a book on
+the subject, read through the *pcrepattern* man page, or read through
 one of many online guides.
 
 Let me show you what Varnish Software uses. We use some cookies for
@@ -37,15 +37,15 @@ cookies and since Varnish will cease caching of pages when the client
 sends cookies we will discard these unnecessary cookies in VCL. 
 
 In the following VCL we discard all cookies that start with a
-underscore.::
+underscore::
 
   // Remove has_js and Google Analytics __* cookies.
   set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
   // Remove a ";" prefix, if present.
   set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
 
-Let me show you an example where we remove everything the the cookies
-named COOKIE1 and COOKIE2 and you can marvel at it.::
+Let me show you an example where we remove everything but the cookies
+named COOKIE1 and COOKIE2 and you can marvel at it::
 
   sub vcl_recv {
     if (req.http.Cookie) {
diff --git a/doc/sphinx/tutorial/sizing_your_cache.rst b/doc/sphinx/tutorial/sizing_your_cache.rst
index 1b2449c..4df3f9d 100644
--- a/doc/sphinx/tutorial/sizing_your_cache.rst
+++ b/doc/sphinx/tutorial/sizing_your_cache.rst
@@ -12,7 +12,7 @@ task. A few things to consider:
    to only cache images a little while or not to cache them at all if
    they are cheap to serve from the backend and you have a limited
    amount of memory.
- * Watch the n_lru_nuked counter with varnishstat or some other
+ * Watch the n_lru_nuked counter with :ref:`varnishstat`_ or some other
    tool. If you have a lot of LRU activity then your cache is evicting
    objects due to space constraints and you should consider increasing
    the size of the cache.



From bjorn at varnish-cache.org  Tue Mar  1 18:03:13 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:03:13 +0100
Subject: [2.1] varnish-2.1.5-4-g4184ed1
Message-ID: <E1PuTuj-00031M-MW@project.varnish-software.com>

commit 4184ed18a0a20433641c98ef173277d3d691bf39
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 18:59:46 2011 +0100

    fix typos and link

diff --git a/doc/sphinx/tutorial/esi.rst b/doc/sphinx/tutorial/esi.rst
index e2002bd..13a4478 100644
--- a/doc/sphinx/tutorial/esi.rst
+++ b/doc/sphinx/tutorial/esi.rst
@@ -20,13 +20,13 @@ have three ESI statements:
  * <!--esi ...-->
 
 Content substitution based on variables and cookies is not implemented
-but is on the roadmap. 
+but is on the roadmap.
 
 Example: esi include
 ~~~~~~~~~~~~~~~~~~~~
 
 Lets see an example how this could be used. This simple cgi script
-outputs the date:::
+outputs the date::
 
      #!/bin/sh
      
@@ -34,7 +34,7 @@ outputs the date:::
      echo ''
      date "+%Y-%m-%d %H:%M"
 
-Now, lets have an HTML file that has an ESI include statement:::
+Now, lets have an HTML file that has an ESI include statement::
 
      <HTML>
      <BODY>
@@ -43,7 +43,7 @@ Now, lets have an HTML file that has an ESI include statement:::
      </BODY>
      </HTML>
 
-For ESI to work you need to activate ESI processing in VCL, like this:::
+For ESI to work you need to activate ESI processing in VCL, like this::
 
     sub vcl_fetch {
     	if (req.url == "/test.html") {
@@ -59,7 +59,7 @@ Example: esi remove
 ~~~~~~~~~~~~~~~~~~~
 
 The *remove* keyword allows you to remove output. You can use this to make
-a fall back of sorts, when ESI is not available, like this:::
+a fall back of sorts, when ESI is not available, like this::
 
   <esi:include src="http://www.example.com/ad.html"/> 
   <esi:remove> 
diff --git a/doc/sphinx/tutorial/vary.rst b/doc/sphinx/tutorial/vary.rst
index 4da6744..a0bfc71 100644
--- a/doc/sphinx/tutorial/vary.rst
+++ b/doc/sphinx/tutorial/vary.rst
@@ -21,9 +21,9 @@ And another one sends::
   Accept-Encoding:: deflate,gzip
 
 Varnish will keep two variants of the page requested due to the
-different Accept-Encoding headers. Normalizing the accept-encoding
+different Accept-Encoding headers. Normalizing the Accept-Encoding
 header will sure that you have as few variants as possible. The
-following VCL code will normalize the Accept-Encoding headers.::
+following VCL code will normalize the Accept-Encoding headers::
 
     if (req.http.Accept-Encoding) {
         if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {



From bjorn at varnish-cache.org  Tue Mar  1 18:09:54 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:09:54 +0100
Subject: [master] ccb59d9 add link
Message-ID: <E1PuU1C-0003Vj-MT@project.varnish-software.com>

commit ccb59d9261d03644aca94bb9e172b7e7062994bd
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 19:09:36 2011 +0100

    add link

diff --git a/doc/sphinx/tutorial/sizing_your_cache.rst b/doc/sphinx/tutorial/sizing_your_cache.rst
index 5e23796..34d46fa 100644
--- a/doc/sphinx/tutorial/sizing_your_cache.rst
+++ b/doc/sphinx/tutorial/sizing_your_cache.rst
@@ -12,7 +12,7 @@ task. A few things to consider:
    to only cache images a little while or not to cache them at all if
    they are cheap to serve from the backend and you have a limited
    amount of memory.
- * Watch the n_lru_nuked counter with varnishstat or some other
+ * Watch the n_lru_nuked counter with :ref:`varnishstat`_ or some other
    tool. If you have a lot of LRU activity then your cache is evicting
    objects due to space constraints and you should consider increasing
    the size of the cache.



From bjorn at varnish-cache.org  Tue Mar  1 18:12:58 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:12:58 +0100
Subject: [master] 79a8de9 Take 2
Message-ID: <E1PuU4A-0004u8-1i@project.varnish-software.com>

commit 79a8de98466e03ae4932b969f593caa32134683c
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 19:12:46 2011 +0100

    Take 2

diff --git a/doc/sphinx/tutorial/sizing_your_cache.rst b/doc/sphinx/tutorial/sizing_your_cache.rst
index 34d46fa..5f81a31 100644
--- a/doc/sphinx/tutorial/sizing_your_cache.rst
+++ b/doc/sphinx/tutorial/sizing_your_cache.rst
@@ -12,7 +12,7 @@ task. A few things to consider:
    to only cache images a little while or not to cache them at all if
    they are cheap to serve from the backend and you have a limited
    amount of memory.
- * Watch the n_lru_nuked counter with :ref:`varnishstat`_ or some other
+ * Watch the n_lru_nuked counter with `varnishstat`_ or some other
    tool. If you have a lot of LRU activity then your cache is evicting
    objects due to space constraints and you should consider increasing
    the size of the cache.



From bjorn at varnish-cache.org  Tue Mar  1 18:17:50 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:17:50 +0100
Subject: [master] 2e50392 Take 3
Message-ID: <E1PuU8s-0006PK-Rg@project.varnish-software.com>

commit 2e503928d08f9a1813a67f38738770433bb72d54
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 19:17:33 2011 +0100

    Take 3

diff --git a/doc/sphinx/tutorial/sizing_your_cache.rst b/doc/sphinx/tutorial/sizing_your_cache.rst
index 5f81a31..c19647c 100644
--- a/doc/sphinx/tutorial/sizing_your_cache.rst
+++ b/doc/sphinx/tutorial/sizing_your_cache.rst
@@ -12,7 +12,7 @@ task. A few things to consider:
    to only cache images a little while or not to cache them at all if
    they are cheap to serve from the backend and you have a limited
    amount of memory.
- * Watch the n_lru_nuked counter with `varnishstat`_ or some other
+ * Watch the n_lru_nuked counter with :ref:`reference-varnishstat` or some other
    tool. If you have a lot of LRU activity then your cache is evicting
    objects due to space constraints and you should consider increasing
    the size of the cache.



From bjorn at varnish-cache.org  Tue Mar  1 18:34:55 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 19:34:55 +0100
Subject: [master] 97e918d Added reference
Message-ID: <E1PuUPP-0000Oc-NS@project.varnish-software.com>

commit 97e918d5eb2c5baea583ce361efc2fd96fbec3bf
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 19:34:33 2011 +0100

    Added reference

diff --git a/doc/sphinx/reference/varnishstat.rst b/doc/sphinx/reference/varnishstat.rst
index 5447785..23d4266 100644
--- a/doc/sphinx/reference/varnishstat.rst
+++ b/doc/sphinx/reference/varnishstat.rst
@@ -1,3 +1,5 @@
+.. _reference-varnishstat:
+
 ===========
 varnishstat
 ===========



From bjorn at varnish-cache.org  Tue Mar  1 21:07:21 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:07:21 +0100
Subject: [2.1] varnish-2.1.5-5-g24fe3cd
Message-ID: <E1PuWmv-00047B-FP@project.varnish-software.com>

commit 24fe3cd67370ef11cfd53d7df783fe74b5d5136b
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:06:56 2011 +0100

    typos

diff --git a/doc/sphinx/tutorial/cookies.rst b/doc/sphinx/tutorial/cookies.rst
index e8fab9f..496579c 100644
--- a/doc/sphinx/tutorial/cookies.rst
+++ b/doc/sphinx/tutorial/cookies.rst
@@ -44,7 +44,7 @@ underscore::
   // Remove a ";" prefix, if present.
   set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
 
-Let me show you an example where we remove everything but the cookies
+Let me show you an example where we remove everything except the cookies
 named COOKIE1 and COOKIE2 and you can marvel at it::
 
   sub vcl_recv {



From bjorn at varnish-cache.org  Tue Mar  1 21:19:37 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:19:37 +0100
Subject: [master] 193a8c7 Fix typos
Message-ID: <E1PuWyn-0004mv-1L@project.varnish-software.com>

commit 193a8c716da3298df5ae6ba85af8a38241f53e1f
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:18:30 2011 +0100

    Fix typos

diff --git a/doc/sphinx/tutorial/advanced_topics.rst b/doc/sphinx/tutorial/advanced_topics.rst
index 5bdb7b4..4491c28 100644
--- a/doc/sphinx/tutorial/advanced_topics.rst
+++ b/doc/sphinx/tutorial/advanced_topics.rst
@@ -43,7 +43,7 @@ Edge Side Includes
 Varnish can cache create web pages by putting different pages
 together. These *fragments* can have individual cache policies. If you
 have a web site with a list showing the 5 most popular articles on
-your site this list can probably be cached as a fragment and included
+your site, this list can probably be cached as a fragment and included
 in all the other pages. Used properly it can dramatically increase
 your hit rate and reduce the load on your servers. ESI looks like this::
 
diff --git a/doc/sphinx/tutorial/logging.rst b/doc/sphinx/tutorial/logging.rst
index 9dce29a..c79fbd3 100644
--- a/doc/sphinx/tutorial/logging.rst
+++ b/doc/sphinx/tutorial/logging.rst
@@ -45,11 +45,11 @@ with a tag indicating what sort of activity is beeing logged. Tags
 starting with Rx indicate Varnish is recieving data and Tx indicates
 sending data.
 
-The third column tell us whether this is is data comming or going to
+The third column tell us whether this is is data coming or going to
 the client (c) or to/from the backend (b). The forth column is the
 data being logged.
 
-Now, you can filter quite a bit with varnishlog. The basics option you
+Now, you can filter quite a bit with varnishlog. The basic option you
 want to know are:
 
 -b
@@ -73,5 +73,5 @@ want to know are:
  Group log entries by request ID.
 
 
-Now that Varnish seem to work OK its time to put Varnish on port 80
+Now that Varnish seem to work OK it's time to put Varnish on port 80
 while we tune it.
diff --git a/doc/sphinx/tutorial/putting_varnish_on_port_80.rst b/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
index 5742806..73a80ff 100644
--- a/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
+++ b/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
@@ -7,7 +7,7 @@ purposes. You should test your application and if it works OK we can
 switch, so Varnish will be running on port 80 and your web server on a
 high port.
 
-First we kill off varnishd.::
+First we kill off varnishd::
 
      # pkill varnishd
 
@@ -15,11 +15,11 @@ and stop your web server. Edit the configuration for your web server
 and make it bind to port 8080 instead of 80. Now open the Varnish
 default.vcl and change the port of the *default* backend to 8080.
 
-Start up your web server and then start varnish.::
+Start up your web server and then start varnish::
 
       # varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000
 
 Note that we've removed the -a option. Now Varnish, as its default
-setting dictates, will bind to the http port (80). Now everyone thats
+setting dictates, will bind to the http port (80). Now everyone
 accessing your site will be accessing through Varnish.
 
diff --git a/doc/sphinx/tutorial/starting_varnish.rst b/doc/sphinx/tutorial/starting_varnish.rst
index cbf94f8..6c89f54 100644
--- a/doc/sphinx/tutorial/starting_varnish.rst
+++ b/doc/sphinx/tutorial/starting_varnish.rst
@@ -21,14 +21,14 @@ I added a few options, lets go through them:
  - one gigabyte. 
 
 ``-T 127.0.0.1:2000``
- Varnish has a buildt in text-based administration
+ Varnish has a built-in text-based administration
  interface. Activating the interface makes Varnish manageble without
  stopping it. You can specify what interface the management interface
  should listen to. Make sure you don't expose the management interface
  to the world as you can easily gain root access to a system via the
- Varnish management interace. I recommend tieing it to localhost. If
- you have users on your system that you don't fully trust use firewall
- rules to restrict access to the interace to root only.
+ Varnish management interface. I recommend tieing it to localhost. If
+ you have users on your system that you don't fully trust, use firewall
+ rules to restrict access to the interface to root only.
 
 ``-a 0.0.0.0:8080``
  I specify that I want Varnish to listen on port 8080 for incomming
@@ -36,8 +36,9 @@ I added a few options, lets go through them:
  Varnish listen on port 80, which is the default.
 
 Now you have Varnish running. Let us make sure that it works
-properly. Use your browser to go to http://192.168.2.2:8080/ - you
-should now see your web application running there.
+properly. Use your browser to go to http://192.168.2.2:8080/
+(obviously, you should replace the IP address with one on your own
+system) - you should now see your web application running there.
 
 Whether or not the application actually goes faster when run through
 Varnish depends on a few factors. If you application uses cookies for



From bjorn at varnish-cache.org  Tue Mar  1 21:24:59 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:24:59 +0100
Subject: [master] 1dd0560 typos
Message-ID: <E1PuX3z-0006G8-2v@project.varnish-software.com>

commit 1dd056021cb1d8941de760cb7b535b2fce985e6a
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:24:35 2011 +0100

    typos

diff --git a/doc/sphinx/tutorial/cookies.rst b/doc/sphinx/tutorial/cookies.rst
index 5cb8f2c..1d6f291 100644
--- a/doc/sphinx/tutorial/cookies.rst
+++ b/doc/sphinx/tutorial/cookies.rst
@@ -15,7 +15,7 @@ interest to the server.
 For a lot of web application it makes sense to completely disregard the
 cookies unless you are accessing a special part of the web site. This
 VCL snippet in vcl_recv will disregard cookies unless you are
-accessing /admin/.::
+accessing /admin/::
 
   if ( !( req.url ~ ^/admin/) ) {
     unset req.http.Cookie;
@@ -37,15 +37,15 @@ cookies and since Varnish will cease caching of pages when the client
 sends cookies we will discard these unnecessary cookies in VCL. 
 
 In the following VCL we discard all cookies that start with a
-underscore.::
+underscore::
 
   // Remove has_js and Google Analytics __* cookies.
   set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
   // Remove a ";" prefix, if present.
   set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
 
-Let me show you an example where we remove everything the the cookies
-named COOKIE1 and COOKIE2 and you can marvel at it.::
+Let me show you an example where we remove everything except the
+cookies named COOKIE1 and COOKIE2 and you can marvel at it::
 
   sub vcl_recv {
     if (req.http.Cookie) {



From bjorn at varnish-cache.org  Tue Mar  1 21:31:38 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:31:38 +0100
Subject: [master] cc6482f typo patrol
Message-ID: <E1PuXAQ-0007zh-GE@project.varnish-software.com>

commit cc6482f32d0a14a7c3ed2f76e83b5a699e4a7065
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:31:18 2011 +0100

    typo patrol

diff --git a/doc/sphinx/tutorial/statistics.rst b/doc/sphinx/tutorial/statistics.rst
index 98af37b..65e3bc3 100644
--- a/doc/sphinx/tutorial/statistics.rst
+++ b/doc/sphinx/tutorial/statistics.rst
@@ -4,7 +4,7 @@
 Statistics
 ----------
 
-Now that your varnish is up and running lets have a look at how it is
+Now that your varnish is up and running let's have a look at how it is
 doing. There are several tools that can help.
 
 varnishtop
@@ -21,7 +21,7 @@ agents, or any other information which is recorded in the log.
 by the client. ``varnishtop -i txurl`` will show you what your backend
 is beeing asked the most. ``varnishtop -i RxHeader -I
 Accept-Encoding`` will show the most popular Accept-Encoding header
-the client are sendning you.
+the client are sending you.
 
 varnishhist
 ~~~~~~~~~~~



From bjorn at varnish-cache.org  Tue Mar  1 21:33:41 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:33:41 +0100
Subject: [master] 20454d7 typo patrol
Message-ID: <E1PuXCP-0000oX-QR@project.varnish-software.com>

commit 20454d7a297e5730bc1d6ec342aeabaea5c45265
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:33:38 2011 +0100

    typo patrol

diff --git a/doc/sphinx/tutorial/statistics.rst b/doc/sphinx/tutorial/statistics.rst
index 65e3bc3..520a25a 100644
--- a/doc/sphinx/tutorial/statistics.rst
+++ b/doc/sphinx/tutorial/statistics.rst
@@ -17,7 +17,7 @@ With suitable filtering using the -I, -i, -X and -x options, it can be
 used to display a ranking of requested documents, clients, user
 agents, or any other information which is recorded in the log.
 
-``varnishtop -i rxurl`` will show you what URLs are beeing asked for
+``varnishtop -i rxurl`` will show you what URLs are being asked for
 by the client. ``varnishtop -i txurl`` will show you what your backend
 is beeing asked the most. ``varnishtop -i RxHeader -I
 Accept-Encoding`` will show the most popular Accept-Encoding header



From bjorn at varnish-cache.org  Tue Mar  1 21:35:53 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:35:53 +0100
Subject: [master] 0aeeb84 typo patrol^3
Message-ID: <E1PuXEX-0002Cf-JX@project.varnish-software.com>

commit 0aeeb84de4874fa83881ee889518294b6a83b2cd
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:35:31 2011 +0100

    typo patrol^3

diff --git a/doc/sphinx/tutorial/statistics.rst b/doc/sphinx/tutorial/statistics.rst
index 520a25a..4386111 100644
--- a/doc/sphinx/tutorial/statistics.rst
+++ b/doc/sphinx/tutorial/statistics.rst
@@ -19,7 +19,7 @@ agents, or any other information which is recorded in the log.
 
 ``varnishtop -i rxurl`` will show you what URLs are being asked for
 by the client. ``varnishtop -i txurl`` will show you what your backend
-is beeing asked the most. ``varnishtop -i RxHeader -I
+is being asked the most. ``varnishtop -i RxHeader -I
 Accept-Encoding`` will show the most popular Accept-Encoding header
 the client are sending you.
 



From bjorn at varnish-cache.org  Tue Mar  1 21:38:07 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:38:07 +0100
Subject: [master] 6b11aa2 More typo patrol
Message-ID: <E1PuXGh-0003WF-Vd@project.varnish-software.com>

commit 6b11aa28db811d19ce80994339de042aba2216a1
Author: Bjoern Ruberg <bjorn at doffen.hjemme.ruberg.no>
Date:   Tue Mar 1 22:38:04 2011 +0100

    More typo patrol

diff --git a/doc/sphinx/tutorial/logging.rst b/doc/sphinx/tutorial/logging.rst
index c79fbd3..070dc3d 100644
--- a/doc/sphinx/tutorial/logging.rst
+++ b/doc/sphinx/tutorial/logging.rst
@@ -41,7 +41,7 @@ app. You'll see lines like these.::
 The first column is an arbitrary number, it defines the request. Lines
 with the same number are part of the same HTTP transaction. The second
 column is the *tag* of the log message. All log entries are tagged
-with a tag indicating what sort of activity is beeing logged. Tags
+with a tag indicating what sort of activity is being logged. Tags
 starting with Rx indicate Varnish is recieving data and Tx indicates
 sending data.
 



From bjorn at varnish-cache.org  Tue Mar  1 21:42:27 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Tue, 01 Mar 2011 22:42:27 +0100
Subject: [master] 6e95c71 Typo patrol - again
Message-ID: <E1PuXKt-000584-EN@project.varnish-software.com>

commit 6e95c71cb7b0a237f25b170cf2a39881b5d32ff6
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Tue Mar 1 22:42:04 2011 +0100

    Typo patrol - again

diff --git a/doc/sphinx/tutorial/purging.rst b/doc/sphinx/tutorial/purging.rst
index 2c1a287..61e1e69 100644
--- a/doc/sphinx/tutorial/purging.rst
+++ b/doc/sphinx/tutorial/purging.rst
@@ -16,11 +16,11 @@ bans. First, let me explain the HTTP purges.
 HTTP Purges
 ~~~~~~~~~~~
 
-An HTTP purge is similar to a HTTP GET request, except that the
+An HTTP purge is similar to an HTTP GET request, except that the
 *method* is PURGE. Actually you can call the method whatever you'd
 like, but most people refer to this as purging. Squid supports the
 same mechanism. In order to support purging in Varnish you need the
-following VCL in place:::
+following VCL in place::
 
   acl purge {
 	  "localhost";
@@ -61,7 +61,7 @@ corresponding subroutine is called. In vcl_hit the object that is
 stored in cache is available and we can set the TTL.
 
 So for vg.no to invalidate their front page they would call out to
-Varnish like this:::
+Varnish like this::
 
   PURGE / HTTP/1.0
   Host: vg.no
@@ -69,7 +69,7 @@ Varnish like this:::
 And Varnish would then discard the front page. If there are several
 variants of the same URL in the cache however, only the matching
 variant will be purged. To purge a gzip variant of the same page the
-request would have to look like this:::
+request would have to look like this::
 
   PURGE / HTTP/1.0
   Host: vg.no
@@ -85,7 +85,7 @@ have.
 
 Support for bans is built into Varnish and available in the CLI
 interface. For VG to ban every png object belonging on vg.no they could
-issue:::
+issue::
 
   purge req.http.host == "vg.no" && req.http.url ~ "\.png$"
 
@@ -96,7 +96,7 @@ deliver it. An object is only checked against newer bans. If you have
 a lot of objects with long TTL in your cache you should be aware of a
 potential performance impact of having many bans.
 
-You can also add bans to Varnish via HTTP. Doing so requires a bit of VCL.::
+You can also add bans to Varnish via HTTP. Doing so requires a bit of VCL::
 
   sub vcl_recv {
 	  if (req.request == "BAN") {
@@ -108,12 +108,11 @@ You can also add bans to Varnish via HTTP. Doing so requires a bit of VCL.::
 		        "&& req.url == " req.url);
 
 		  # Throw a synthetic page so the
-                  # request wont go to the backend.
+                  # request won't go to the backend.
 		  error 200 "Ban added"
 	  }
   }
 
-This VCL sniplet enables Varnish to handle a HTTP BAN method. Adding a
+This VCL sniplet enables Varnish to handle an HTTP BAN method, adding a
 ban on the URL, including the host part.
 
-



From phk at varnish-cache.org  Wed Mar  2 12:19:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 13:19:19 +0100
Subject: [master] 6bf50b2 Silence warnings about the VMOD interface functions
	not being used etc.
Message-ID: <E1Pul1T-00064K-NP@project.varnish-software.com>

commit 6bf50b299d36990996cecd2619a16507d5d93806
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 12:18:59 2011 +0000

    Silence warnings about the VMOD interface functions not being used etc.

diff --git a/bin/varnishd/flint.lnt b/bin/varnishd/flint.lnt
index 6d1686d..9d9a231 100644
--- a/bin/varnishd/flint.lnt
+++ b/bin/varnishd/flint.lnt
@@ -49,6 +49,19 @@
 -efunc(838, VRT_purge)
 -efunc(838, VRT_purge_string)
 
+// Stuff in VMODs which is used through dl*(3) functions
+-esym(754, Vmod_Func*)
+-esym(765, Vmod_Func)
+-esym(552, Vmod_Func)
+-esym(765, Vmod_Len)
+-esym(714, Vmod_Len)
+-esym(765, Vmod_Name)
+-esym(714, Vmod_Name)
+-esym(765, Vmod_Proto)
+-esym(714, Vmod_Proto)
+-esym(765, Vmod_Spec)
+-esym(714, Vmod_Spec)
+
 //-sem (pthread_mutex_lock, thread_lock)
 -sem (pthread_mutex_trylock, thread_lock)
 -sem (VBE_DropRefLocked, thread_unlock)



From phk at varnish-cache.org  Wed Mar  2 13:06:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 14:06:17 +0100
Subject: [master] e10a410 Make default_oc_methods static to stevedore.c
Message-ID: <E1Pulkv-0003rZ-FR@project.varnish-software.com>

commit e10a410d14c15d249385983ec89a413493e61fa7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 12:36:10 2011 +0000

    Make default_oc_methods static to stevedore.c

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 9ca8034..05cdcf9 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -363,8 +363,6 @@ struct objcore_methods {
 	getlru_f	*getlru;
 };
 
-extern struct objcore_methods default_oc_methods;
-
 struct objcore {
 	unsigned		magic;
 #define OBJCORE_MAGIC		0x4d301302
diff --git a/bin/varnishd/stevedore.c b/bin/varnishd/stevedore.c
index 912fd68..9d16c6c 100644
--- a/bin/varnishd/stevedore.c
+++ b/bin/varnishd/stevedore.c
@@ -53,6 +53,7 @@ static VTAILQ_HEAD(, stevedore)	stevedores =
 static const struct stevedore * volatile stv_next;
 
 static struct stevedore *stv_transient;
+static struct objcore_methods default_oc_methods;
 
 /*--------------------------------------------------------------------
  */
@@ -339,7 +340,7 @@ default_oc_getlru(const struct objcore *oc)
 	return (o->objstore->stevedore->lru);
 }
 
-struct objcore_methods default_oc_methods = {
+static struct objcore_methods default_oc_methods = {
 	.getobj = default_oc_getobj,
 	.freeobj = default_oc_freeobj,
 	.getlru = default_oc_getlru,



From phk at varnish-cache.org  Wed Mar  2 13:06:18 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 14:06:18 +0100
Subject: [master] b8f33e0 Remove unused variable
Message-ID: <E1Pulkw-0003rg-EP@project.varnish-software.com>

commit b8f33e0f4b27f9056d12f2e419dde5d107ef2f60
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 13:04:04 2011 +0000

    Remove unused variable

diff --git a/bin/varnishd/cache_vrt_vmod.c b/bin/varnishd/cache_vrt_vmod.c
index 8b1ed3e..0c74e02 100644
--- a/bin/varnishd/cache_vrt_vmod.c
+++ b/bin/varnishd/cache_vrt_vmod.c
@@ -69,7 +69,6 @@ VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm, const char *path)
 	struct vmod *v;
 	void *x;
 	const int *i;
-	const char *p;
 
 	ASSERT_CLI();
 
@@ -91,7 +90,6 @@ VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm, const char *path)
 
 		x = dlsym(v->hdl, "Vmod_Name");
 		AN(x);
-		p = x;
 
 		x = dlsym(v->hdl, "Vmod_Len");
 		AN(x);



From phk at varnish-cache.org  Wed Mar  2 13:06:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 14:06:19 +0100
Subject: [master] ddc3902 Separate sandboxing of child processes to a separate
	source file in order to encapsulate the lack of of API
	coordination between different OS's in this arena.
Message-ID: <E1Pulkx-0003ru-5q@project.varnish-software.com>

commit ddc3902f9418a4b6c0037872ac41dcf8e8abea1a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 13:05:24 2011 +0000

    Separate sandboxing of child processes to a separate source file
    in order to encapsulate the lack of of API coordination between
    different OS's in this arena.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index b8db6e0..1ca60b9 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -57,6 +57,7 @@ varnishd_SOURCES = \
 	mgt_cli.c \
 	mgt_param.c \
 	mgt_pool.c \
+	mgt_sandbox.c \
 	mgt_shmem.c \
 	mgt_vcc.c \
 	rfc2616.c \
diff --git a/bin/varnishd/mgt.h b/bin/varnishd/mgt.h
index 122d384..a2dc912 100644
--- a/bin/varnishd/mgt.h
+++ b/bin/varnishd/mgt.h
@@ -71,6 +71,9 @@ void MCF_ParamSet(struct cli *, const char *param, const char *val);
 void MCF_DumpRst(void);
 #endif
 
+/* mgt_sandbox.c */
+void mgt_sandbox(void);
+
 /* mgt_shmem.c */
 void mgt_SHM_Init(const char *arg);
 void mgt_SHM_Pid(void);
diff --git a/bin/varnishd/mgt_child.c b/bin/varnishd/mgt_child.c
index 8ae55c8..d8cbc59 100644
--- a/bin/varnishd/mgt_child.c
+++ b/bin/varnishd/mgt_child.c
@@ -46,18 +46,10 @@ SVNID("$Id$")
 #include <sys/types.h>
 #include <sys/wait.h>
 
-#ifdef HAVE_PRIV_H
-#include <priv.h>
-#endif
-
 #ifndef HAVE_SETPROCTITLE
 #include "compat/setproctitle.h"
 #endif
 
-#ifdef __linux__
-#include <sys/prctl.h>
-#endif
-
 #include "mgt.h"
 #include "vsm.h"
 #include "heritage.h"
@@ -280,44 +272,6 @@ close_sockets(void)
 
 /*--------------------------------------------------------------------*/
 
-/* Waive all privileges in the child, it does not need any */
-
-static inline void
-waive_privileges(void)
-{
-
-#ifdef HAVE_SETPPRIV
-	priv_set_t *empty, *minimal;
-
-	if (!(empty = priv_allocset()) ||
-	    !(minimal = priv_allocset())) {
-		perror("priv_allocset_failed");
-		return;
-	}
-	priv_emptyset(empty);
-	priv_emptyset(minimal);
-
-	/* new privilege, silently ignore any errors if it doesn't exist */
-	priv_addset(minimal, "net_access");
-
-#define SETPPRIV(which, set)				       \
-	if (setppriv(PRIV_SET, which, set))		       \
-		perror("Waiving privileges failed on " #which)
-
-	/* need to set I after P to avoid SNOCD being set */
-	SETPPRIV(PRIV_LIMIT, minimal);
-	SETPPRIV(PRIV_PERMITTED, minimal); /* implies PRIV_EFFECTIVE */
-	SETPPRIV(PRIV_INHERITABLE, empty);
-
-	priv_freeset(empty);
-	priv_freeset(minimal);
-#else
-	return;
-#endif
-}
-
-/*--------------------------------------------------------------------*/
-
 static void
 start_child(struct cli *cli)
 {
@@ -370,19 +324,6 @@ start_child(struct cli *cli)
 		exit(1);
 	}
 	if (pid == 0) {
-		if (geteuid() == 0) {
-			XXXAZ(setgid(params->gid));
-			XXXAZ(setuid(params->uid));
-		}
-
-		/* On Linux >= 2.4, you need to set the dumpable flag
-		   to get core dumps after you have done a setuid. */
-#ifdef __linux__
-		if (prctl(PR_SET_DUMPABLE, 1) != 0) {
-		  printf("Could not set dumpable bit.  Core dumps turned "
-			 "off\n");
-		}
-#endif
 
 		/* Redirect stdin/out/err */
 		AZ(close(STDIN_FILENO));
@@ -397,14 +338,13 @@ start_child(struct cli *cli)
 				continue;
 			(void)(close(i) == 0);
 		}
-		printf("\n");
-
-		waive_privileges();
-
 		setproctitle("Varnish-Chld %s", heritage.name);
 
 		(void)signal(SIGINT, SIG_DFL);
 		(void)signal(SIGTERM, SIG_DFL);
+
+		mgt_sandbox();
+
 		child_main();
 
 		exit(1);
diff --git a/bin/varnishd/mgt_sandbox.c b/bin/varnishd/mgt_sandbox.c
new file mode 100644
index 0000000..c6ec1e8
--- /dev/null
+++ b/bin/varnishd/mgt_sandbox.c
@@ -0,0 +1,124 @@
+/*-
+ * Copyright (c) 2006-2011 Linpro AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Sandboxing child processes
+ *
+ * The worker/manager process border is one of the major security barriers
+ * in Varnish, and therefore subject to whatever restrictions we have access
+ * to under the given operating system.
+ *
+ * Unfortunately there is no consensus on APIs for this purpose, so each
+ * operating system will require its own methods.
+ *
+ * This sourcefile tries to encapsulate the resulting mess on place.
+ *
+ * TODO:
+ *	Unix:	chroot
+ *	FreeBSD: jail
+ *	FreeBSD: capsicum
+ */
+
+#include "config.h"
+
+#include "svnid.h"
+SVNID("$Id$")
+
+#include <stdio.h>
+#include <syslog.h>
+#include <unistd.h>
+
+#ifdef HAVE_PRIV_H
+#include <priv.h>
+#endif
+
+#ifndef HAVE_SETPROCTITLE
+#include "compat/setproctitle.h"
+#endif
+
+#ifdef __linux__
+#include <sys/prctl.h>
+#endif
+
+#include "mgt.h"
+#include "heritage.h"
+
+/*--------------------------------------------------------------------*/
+
+/* Waive all privileges in the child, it does not need any */
+
+void
+mgt_sandbox(void)
+{
+
+	if (geteuid() == 0) {
+		XXXAZ(setgid(params->gid));
+		XXXAZ(setuid(params->uid));
+	} else {
+		REPORT0(LOG_INFO, "Not running as root, no priv-sep");
+	}
+
+	/* On Linux >= 2.4, you need to set the dumpable flag
+	   to get core dumps after you have done a setuid. */
+
+#ifdef __linux__
+	if (prctl(PR_SET_DUMPABLE, 1) != 0)
+		REPORT0(LOG_INFO,
+		    "Could not set dumpable bit.  Core dumps turned off\n");
+#endif
+
+#ifdef HAVE_SETPPRIV
+	priv_set_t *empty, *minimal;
+
+	if (!(empty = priv_allocset()) ||
+	    !(minimal = priv_allocset())) {
+		REPORT0(LOG_ERR, "priv_allocset_failed");
+	} else {
+		priv_emptyset(empty);
+		priv_emptyset(minimal);
+
+		/*
+		 * new privilege,
+		 * silently ignore any errors if it doesn't exist
+		 */
+		priv_addset(minimal, "net_access");
+
+#define SETPPRIV(which, set)				       		\
+		if (setppriv(PRIV_SET, which, set))			\
+			REPORT0(LOG_ERR,				\
+			    "Waiving privileges failed on " #which)
+
+		/* need to set I after P to avoid SNOCD being set */
+		SETPPRIV(PRIV_LIMIT, minimal);
+		SETPPRIV(PRIV_PERMITTED, minimal); /* implies PRIV_EFFECTIVE */
+		SETPPRIV(PRIV_INHERITABLE, empty);
+
+		priv_freeset(empty);
+		priv_freeset(minimal);
+	}
+#endif
+
+}



From phk at varnish-cache.org  Wed Mar  2 14:29:54 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 15:29:54 +0100
Subject: [master] 72d94d7 Add missing checks for Z_OK
Message-ID: <E1Pun3q-0003pJ-VG@project.varnish-software.com>

commit 72d94d7c87713cbdbf3f4f3ef5818cc146ae7056
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 14:29:30 2011 +0000

    Add missing checks for Z_OK
    
    Found & Patch by:	Dmitry Panov

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 7af71c8..9b5180c 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -428,7 +428,7 @@ vfp_gunzip_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		assert(i == Z_OK || i == Z_STREAM_END);
 		sp->obj->len += dl;
 	}
-	if (i == Z_STREAM_END)
+	if (i == Z_OK || i == Z_STREAM_END)
 		return (1);
 	return (-1);
 }
@@ -580,7 +580,7 @@ vfp_testgzip_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 			assert(i == Z_OK || i == Z_STREAM_END);
 		}
 	}
-	if (i == Z_STREAM_END)
+	if (i == Z_OK || i == Z_STREAM_END)
 		return (1);
 	return (-1);
 }



From phk at varnish-cache.org  Wed Mar  2 16:30:59 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 17:30:59 +0100
Subject: [master] 7264c91 Move grace and ttl variables into a common structure
	and rename HSH_Grace() to EXP_Grace()
Message-ID: <E1Puox1-0006xn-Kf@project.varnish-software.com>

commit 7264c919ecda6787489232fd15b9cc45c4adaf7f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 16:30:05 2011 +0000

    Move grace and ttl variables into a common structure and rename
    HSH_Grace() to EXP_Grace()

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 05cdcf9..ecad892 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -232,6 +232,13 @@ extern struct vfp vfp_esi;
 
 /*--------------------------------------------------------------------*/
 
+struct exp {
+	double			ttl;
+	double			grace;
+};
+
+/*--------------------------------------------------------------------*/
+
 struct worker {
 	unsigned		magic;
 #define WORKER_MAGIC		0x6391adcf
@@ -272,8 +279,7 @@ struct worker {
 
 	double			age;
 	double			entered;
-	double			ttl;
-	double			grace;
+	struct exp		exp;
 
 	/* This is only here so VRT can find it */
 	const char		*storage_hint;
@@ -449,10 +455,9 @@ struct object {
 
 	ssize_t			len;
 
-	double			ttl;
 	double			age;
 	double			entered;
-	double			grace;
+	struct exp		exp;
 
 	double			last_modified;
 	double			last_lru;
@@ -517,7 +522,7 @@ struct sess {
 	double			t_end;
 
 	/* Acceptable grace period */
-	double			grace;
+	struct exp		exp;
 
 	enum step		step;
 	unsigned		cur_method;
@@ -628,6 +633,7 @@ extern pthread_t cli_thread;
 #define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
 
 /* cache_expiry.c */
+double EXP_Grace(double g);
 void EXP_Insert(struct object *o);
 void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
 void EXP_Init(void);
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 2563f0c..a14c54e 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -475,8 +475,8 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 		oc_updatemeta(oc);
 		return (0);
 	} else {
-		o->ttl = 0;
-		o->grace = 0;
+		o->exp.ttl = 0;
+		o->exp.grace = 0;
 		oc->ban = NULL;
 		oc_updatemeta(oc);
 		/* BAN also changed, but that is not important any more */
@@ -543,7 +543,7 @@ ban_lurker(struct sess *sp, void *priv)
 		// AZ(oc->flags & OC_F_PERSISTENT);
 		o = oc_getobj(sp->wrk, oc);
 		i = ban_check_object(o, sp, 0);
-		WSP(sp, SLT_Debug, "lurker: %p %g %d", oc, o->ttl, i);
+		WSP(sp, SLT_Debug, "lurker: %p %g %d", oc, o->exp.ttl, i);
 		(void)HSH_Deref(sp->wrk, NULL, &o);
 		TIM_sleep(params->ban_lurker_sleep);
 	}
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index f6065f0..f5e7718 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -527,7 +527,7 @@ cnt_fetch(struct sess *sp)
 	 */
 	sp->wrk->entered = TIM_real();
 	sp->wrk->age = 0;
-	sp->wrk->ttl = RFC2616_Ttl(sp);
+	sp->wrk->exp.ttl = RFC2616_Ttl(sp);
 
 	/*
 	 * Initial cacheability determination per [RFC2616, 13.4]
@@ -543,16 +543,16 @@ cnt_fetch(struct sess *sp)
 	case 404: /* Not Found */
 		break;
 	default:
-		sp->wrk->ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = sp->t_req - 1.;
 		break;
 	}
 
 	/* pass from vclrecv{} has negative TTL */
 	if (sp->objcore == NULL)
-		sp->wrk->ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = sp->t_req - 1.;
 
 	sp->wrk->do_esi = 0;
-	sp->wrk->grace = NAN;
+	sp->wrk->exp.grace = NAN;
 
 	sp->wrk->body_status = RFC2616_Body(sp);
 
@@ -564,7 +564,7 @@ cnt_fetch(struct sess *sp)
 		/* This is a pass from vcl_recv */
 		pass = 1;
 		/* VCL may have fiddled this, but that doesn't help */
-		sp->wrk->ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = sp->t_req - 1.;
 	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
 		/* pass from vcl_fetch{} -> hit-for-pass */
 		/* XXX: the bereq was not filtered pass... */
@@ -653,12 +653,12 @@ cnt_fetch(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (sp->wrk->ttl < sp->t_req + params->shortlived ||
+	if (sp->wrk->exp.ttl < sp->t_req + params->shortlived ||
 	    sp->objcore == NULL)
 		sp->wrk->storage_hint = TRANSIENT_STORAGE;
 
 	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
-	    sp->wrk->ttl, nhttp);
+	    sp->wrk->exp.ttl, nhttp);
 	/* XXX: -> 513 */
 	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
 
@@ -678,7 +678,7 @@ cnt_fetch(struct sess *sp)
 
 	sp->obj->xid = sp->xid;
 	sp->obj->response = sp->err_code;
-	sp->obj->grace = sp->wrk->grace;
+	sp->obj->exp.grace = sp->wrk->exp.grace;
 	sp->obj->age = sp->wrk->age;
 	sp->obj->entered = sp->wrk->entered;
 	WS_Assert(sp->obj->ws_o);
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 5d2e588..2393ca5 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -57,6 +57,14 @@ static pthread_t exp_thread;
 static struct binheap *exp_heap;
 static struct lock exp_mtx;
 
+double
+EXP_Grace(double g)
+{
+	if (isnan(g))
+		return (double)(params->default_grace);
+	return (g);
+}
+
 /*--------------------------------------------------------------------
  * When & why does the timer fire for this object ?
  */
@@ -72,7 +80,7 @@ update_object_when(const struct object *o)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	Lck_AssertHeld(&exp_mtx);
 
-	when = o->ttl + HSH_Grace(o->grace);
+	when = o->exp.ttl + EXP_Grace(o->exp.grace);
 	assert(!isnan(when));
 	if (when == oc->timer_when)
 		return (0);
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index b3363cc..107c149 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -72,14 +72,6 @@ SVNID("$Id$")
 static const struct hash_slinger *hash;
 
 /*---------------------------------------------------------------------*/
-double
-HSH_Grace(double g)
-{
-	if (isnan(g))
-		return (double)(params->default_grace);
-	return (g);
-}
-
 /* Precreate an objhead and object for later use */
 void
 HSH_Prealloc(const struct sess *sp)
@@ -362,7 +354,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 		o = oc_getobj(sp->wrk, oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 
-		if (o->ttl == 0)
+		if (o->exp.ttl == 0)
 			continue;
 		if (BAN_CheckObject(o, sp))
 			continue;
@@ -370,17 +362,17 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 			continue;
 
 		/* If still valid, use it */
-		if (o->ttl >= sp->t_req)
+		if (o->exp.ttl >= sp->t_req)
 			break;
 
 		/*
 		 * Remember any matching objects inside their grace period
 		 * and if there are several, use the least expired one.
 		 */
-		if (o->ttl + HSH_Grace(o->grace) >= sp->t_req) {
-			if (grace_oc == NULL || grace_ttl < o->ttl) {
+		if (o->exp.ttl + EXP_Grace(o->exp.grace) >= sp->t_req) {
+			if (grace_oc == NULL || grace_ttl < o->exp.ttl) {
 				grace_oc = oc;
-				grace_ttl = o->ttl;
+				grace_ttl = o->exp.ttl;
 			}
 		}
 	}
@@ -404,7 +396,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 					/* Or it is impossible to fetch */
 		o = oc_getobj(sp->wrk, grace_oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		if (o->ttl + HSH_Grace(sp->grace) >= sp->t_req)
+		if (o->exp.ttl + EXP_Grace(sp->exp.grace) >= sp->t_req)
 			oc = grace_oc;
 	}
 	sp->objcore = NULL;
@@ -552,9 +544,9 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
 		if (o == NULL)
 			continue;
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		o->ttl = sp->t_req + ttl;
+		o->exp.ttl = sp->t_req + ttl;
 		if (!isnan(grace))
-			o->grace = grace;
+			o->exp.grace = grace;
 		EXP_Rearm(o);
 		(void)HSH_Deref(sp->wrk, NULL, &o);
 	}
@@ -577,7 +569,7 @@ HSH_Drop(struct sess *sp)
 	o = sp->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	AssertObjPassOrBusy(o);
-	o->ttl = 0;
+	o->exp.ttl = 0;
 	if (o->objcore != NULL)		/* Pass has no objcore */
 		HSH_Unbusy(sp);
 	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
diff --git a/bin/varnishd/cache_session.c b/bin/varnishd/cache_session.c
index 8f26890..0fa662f 100644
--- a/bin/varnishd/cache_session.c
+++ b/bin/varnishd/cache_session.c
@@ -174,7 +174,7 @@ ses_setup(struct sessmem *sm)
 	sp->t_req = NAN;
 	sp->t_resp = NAN;
 	sp->t_end = NAN;
-	sp->grace = NAN;
+	sp->exp.grace = NAN;
 
 	WS_Init(sp->ws, "sess", sm->wsp, sm->workspace);
 	sp->http = sm->http[0];
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index 9fa56ba..b562d6d 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -247,17 +247,17 @@ VRT_l_beresp_ttl(const struct sess *sp, double a)
 	 * We special case and make sure that rounding does not surprise.
 	 */
 	if (a <= 0) {
-		sp->wrk->ttl = sp->t_req - 1;
-		sp->wrk->grace = 0.;
+		sp->wrk->exp.ttl = sp->t_req - 1;
+		sp->wrk->exp.grace = 0.;
 	} else
-		sp->wrk->ttl = sp->t_req + a;
+		sp->wrk->exp.ttl = sp->t_req + a;
 }
 
 double
 VRT_r_beresp_ttl(const struct sess *sp)
 {
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->wrk->ttl - sp->t_req);
+	return (sp->wrk->exp.ttl - sp->t_req);
 }
 
 /*--------------------------------------------------------------------*/
@@ -351,10 +351,10 @@ VRT_l_obj_ttl(const struct sess *sp, double a)
 	 * We special case and make sure that rounding does not surprise.
 	 */
 	if (a <= 0) {
-		sp->obj->ttl = sp->t_req - 1;
-		sp->obj->grace = 0;
+		sp->obj->exp.ttl = sp->t_req - 1;
+		sp->obj->exp.grace = 0;
 	} else
-		sp->obj->ttl = sp->t_req + a;
+		sp->obj->exp.ttl = sp->t_req + a;
 	EXP_Rearm(sp->obj);
 }
 
@@ -365,7 +365,7 @@ VRT_r_obj_ttl(const struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
 	if (sp->obj->objcore == NULL)
 		return (0.0);
-	return (sp->obj->ttl - sp->t_req);
+	return (sp->obj->exp.ttl - sp->t_req);
 }
 
 /*--------------------------------------------------------------------*/
@@ -451,12 +451,12 @@ VRT_r_##which##_grace(struct sess *sp)				\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(HSH_Grace(fld));					\
+	return(EXP_Grace(fld));					\
 }
 
-VRT_DO_GRACE(req, sp->grace, )
-VRT_DO_GRACE(obj, sp->obj->grace, EXP_Rearm(sp->obj))
-VRT_DO_GRACE(beresp, sp->wrk->grace, )
+VRT_DO_GRACE(req, sp->exp.grace, )
+VRT_DO_GRACE(obj, sp->obj->exp.grace, EXP_Rearm(sp->obj))
+VRT_DO_GRACE(beresp, sp->wrk->exp.grace, )
 
 /*--------------------------------------------------------------------
  * req.xid
diff --git a/bin/varnishd/hash_slinger.h b/bin/varnishd/hash_slinger.h
index d4cd87d..d5e81e3 100644
--- a/bin/varnishd/hash_slinger.h
+++ b/bin/varnishd/hash_slinger.h
@@ -58,7 +58,6 @@ struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh);
 void HSH_Unbusy(const struct sess *sp);
 void HSH_Ref(struct objcore *o);
 void HSH_Drop(struct sess *sp);
-double HSH_Grace(double g);
 void HSH_Init(void);
 void HSH_AddString(const struct sess *sp, const char *str);
 void HSH_FindBan(const struct sess *sp, struct objcore **oc);
diff --git a/bin/varnishd/stevedore.c b/bin/varnishd/stevedore.c
index 9d16c6c..915eadc 100644
--- a/bin/varnishd/stevedore.c
+++ b/bin/varnishd/stevedore.c
@@ -198,9 +198,9 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
 
 	http_Setup(o->http, o->ws_o);
 	o->http->magic = HTTP_MAGIC;
-	o->grace = NAN;
+	o->exp.grace = NAN;
 	o->entered = NAN;
-	o->ttl = soc->ttl;
+	o->exp.ttl = soc->ttl;
 	VTAILQ_INIT(&o->store);
 	sp->wrk->stats.n_object++;
 
diff --git a/bin/varnishd/storage_persistent.c b/bin/varnishd/storage_persistent.c
index 0fd2ee4..789f86e 100644
--- a/bin/varnishd/storage_persistent.c
+++ b/bin/varnishd/storage_persistent.c
@@ -493,7 +493,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 
 	CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC);
 	AN(sp->objcore);
-	AN(sp->wrk->ttl >= 0);
+	AN(sp->wrk->exp.ttl >= 0);
 
 	ltot = IRNUP(sc, ltot);
 
@@ -519,7 +519,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	/* We have to do this somewhere, might as well be here... */
 	assert(sizeof so->hash == DIGEST_LEN);
 	memcpy(so->hash, oc->objhead->digest, DIGEST_LEN);
-	so->ttl = o->ttl;	/* XXX: grace? */
+	so->ttl = o->exp.ttl;	/* XXX: grace? */
 	so->ptr = (uint8_t*)o - sc->base;
 	so->ban = o->ban_t;
 
diff --git a/bin/varnishd/storage_persistent_silo.c b/bin/varnishd/storage_persistent_silo.c
index 7a8bac2..86f487b 100644
--- a/bin/varnishd/storage_persistent_silo.c
+++ b/bin/varnishd/storage_persistent_silo.c
@@ -433,8 +433,8 @@ smp_oc_getobj(struct worker *wrk, struct objcore *oc)
 			bad |= 0x100;
 
 		if(bad) {
-			o->ttl = 0;
-			o->grace = 0;
+			o->exp.ttl = 0;
+			o->exp.grace = 0;
 			so->ttl = 0;
 		}
 
@@ -463,10 +463,10 @@ smp_oc_updatemeta(struct objcore *oc)
 	CHECK_OBJ_NOTNULL(sg->sc, SMP_SC_MAGIC);
 	so = smp_find_so(sg, oc);
 
-	if (isnan(o->grace))
-		mttl = o->ttl;
+	if (isnan(o->exp.grace))
+		mttl = o->exp.ttl;
 	else
-		mttl = - (o->ttl + o->grace);
+		mttl = - (o->exp.ttl + o->exp.grace);
 
 	if (sg == sg->sc->cur_seg) {
 		/* Lock necessary, we might race close_seg */



From phk at varnish-cache.org  Wed Mar  2 19:24:58 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 20:24:58 +0100
Subject: [master] d6b3388 Change the ttl variables to be relative to
	o->entered, since most manipulation is actually the relative value,
	not the absolute time.
Message-ID: <E1PurfO-0002xm-WB@project.varnish-software.com>

commit d6b338852eb23dd24cf475eb7f2884daf8703ea8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 19:24:21 2011 +0000

    Change the ttl variables to be relative to o->entered, since most
    manipulation is actually the relative value, not the absolute time.

diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index a14c54e..241001b 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -475,8 +475,8 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 		oc_updatemeta(oc);
 		return (0);
 	} else {
-		o->exp.ttl = 0;
-		o->exp.grace = 0;
+		o->exp.ttl = -1.;
+		o->exp.grace = -1.;
 		oc->ban = NULL;
 		oc_updatemeta(oc);
 		/* BAN also changed, but that is not important any more */
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index f5e7718..34be2c6 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -543,13 +543,13 @@ cnt_fetch(struct sess *sp)
 	case 404: /* Not Found */
 		break;
 	default:
-		sp->wrk->exp.ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = -1.;
 		break;
 	}
 
 	/* pass from vclrecv{} has negative TTL */
 	if (sp->objcore == NULL)
-		sp->wrk->exp.ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = -1.;
 
 	sp->wrk->do_esi = 0;
 	sp->wrk->exp.grace = NAN;
@@ -564,7 +564,7 @@ cnt_fetch(struct sess *sp)
 		/* This is a pass from vcl_recv */
 		pass = 1;
 		/* VCL may have fiddled this, but that doesn't help */
-		sp->wrk->exp.ttl = sp->t_req - 1.;
+		sp->wrk->exp.ttl = -1.;
 	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
 		/* pass from vcl_fetch{} -> hit-for-pass */
 		/* XXX: the bereq was not filtered pass... */
@@ -653,8 +653,7 @@ cnt_fetch(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (sp->wrk->exp.ttl < sp->t_req + params->shortlived ||
-	    sp->objcore == NULL)
+	if (sp->wrk->exp.ttl < params->shortlived || sp->objcore == NULL)
 		sp->wrk->storage_hint = TRANSIENT_STORAGE;
 
 	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 2393ca5..60392c5 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -80,7 +80,7 @@ update_object_when(const struct object *o)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	Lck_AssertHeld(&exp_mtx);
 
-	when = o->exp.ttl + EXP_Grace(o->exp.grace);
+	when = o->entered + o->exp.ttl + EXP_Grace(o->exp.grace);
 	assert(!isnan(when));
 	if (when == oc->timer_when)
 		return (0);
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index 107c149..56528ab 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -354,7 +354,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 		o = oc_getobj(sp->wrk, oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 
-		if (o->exp.ttl == 0)
+		if (o->exp.ttl <= 0.)
 			continue;
 		if (BAN_CheckObject(o, sp))
 			continue;
@@ -362,17 +362,17 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 			continue;
 
 		/* If still valid, use it */
-		if (o->exp.ttl >= sp->t_req)
+		if (o->entered + o->exp.ttl >= sp->t_req)
 			break;
 
 		/*
 		 * Remember any matching objects inside their grace period
 		 * and if there are several, use the least expired one.
 		 */
-		if (o->exp.ttl + EXP_Grace(o->exp.grace) >= sp->t_req) {
-			if (grace_oc == NULL || grace_ttl < o->exp.ttl) {
+		if (o->entered + o->exp.ttl + EXP_Grace(o->exp.grace) >= sp->t_req) {
+			if (grace_oc == NULL || grace_ttl < o->entered + o->exp.ttl) {
 				grace_oc = oc;
-				grace_ttl = o->exp.ttl;
+				grace_ttl = o->entered + o->exp.ttl;
 			}
 		}
 	}
@@ -396,7 +396,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 					/* Or it is impossible to fetch */
 		o = oc_getobj(sp->wrk, grace_oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		if (o->exp.ttl + EXP_Grace(sp->exp.grace) >= sp->t_req)
+		if (o->entered + o->exp.ttl + EXP_Grace(sp->exp.grace) >= sp->t_req)
 			oc = grace_oc;
 	}
 	sp->objcore = NULL;
@@ -536,7 +536,7 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
 	Lck_Unlock(&oh->mtx);
 
 	if (ttl <= 0)
-		ttl = -1;
+		ttl = -1.;
 	for (n = 0; n < nobj; n++) {
 		oc = ocp[n];
 		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -544,7 +544,7 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
 		if (o == NULL)
 			continue;
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		o->exp.ttl = sp->t_req + ttl;
+		o->exp.ttl = ttl;
 		if (!isnan(grace))
 			o->exp.grace = grace;
 		EXP_Rearm(o);
@@ -569,7 +569,7 @@ HSH_Drop(struct sess *sp)
 	o = sp->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	AssertObjPassOrBusy(o);
-	o->exp.ttl = 0;
+	o->exp.ttl = -1.;
 	if (o->objcore != NULL)		/* Pass has no objcore */
 		HSH_Unbusy(sp);
 	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index b562d6d..ef89cdd 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -247,17 +247,17 @@ VRT_l_beresp_ttl(const struct sess *sp, double a)
 	 * We special case and make sure that rounding does not surprise.
 	 */
 	if (a <= 0) {
-		sp->wrk->exp.ttl = sp->t_req - 1;
+		sp->wrk->exp.ttl =  -1.;
 		sp->wrk->exp.grace = 0.;
 	} else
-		sp->wrk->exp.ttl = sp->t_req + a;
+		sp->wrk->exp.ttl = a;
 }
 
 double
 VRT_r_beresp_ttl(const struct sess *sp)
 {
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->wrk->exp.ttl - sp->t_req);
+	return (sp->wrk->exp.ttl);
 }
 
 /*--------------------------------------------------------------------*/
@@ -351,10 +351,10 @@ VRT_l_obj_ttl(const struct sess *sp, double a)
 	 * We special case and make sure that rounding does not surprise.
 	 */
 	if (a <= 0) {
-		sp->obj->exp.ttl = sp->t_req - 1;
-		sp->obj->exp.grace = 0;
+		sp->obj->exp.ttl = -1.;
+		sp->obj->exp.grace = 0.;
 	} else
-		sp->obj->exp.ttl = sp->t_req + a;
+		sp->obj->exp.ttl = a;
 	EXP_Rearm(sp->obj);
 }
 
@@ -365,7 +365,7 @@ VRT_r_obj_ttl(const struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
 	if (sp->obj->objcore == NULL)
 		return (0.0);
-	return (sp->obj->exp.ttl - sp->t_req);
+	return (sp->obj->exp.ttl);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index b571381..42d9856 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -152,16 +152,16 @@ RFC2616_Ttl(const struct sess *sp)
 
 	} while (0);
 
-	if (ttl > 0 && ttd == 0)
-		ttd = sp->wrk->entered + ttl;
+	if (ttd > 0)
+		ttl = ttd - sp->wrk->entered;
 
 	/* calculated TTL, Our time, Date, Expires, max-age, age */
 	WSP(sp, SLT_TTL, "%u RFC %d %d %d %d %u %u", sp->xid,
-	    ttd ? (int)(ttd - sp->wrk->entered) : 0,
+	    ttd ? (int)(ttl) : 0,
 	    (int)sp->wrk->entered, (int)h_date,
 	    (int)h_expires, max_age, age);
 
-	return (ttd);
+	return (ttl);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/storage_persistent.c b/bin/varnishd/storage_persistent.c
index 789f86e..c5cf339 100644
--- a/bin/varnishd/storage_persistent.c
+++ b/bin/varnishd/storage_persistent.c
@@ -493,7 +493,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 
 	CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC);
 	AN(sp->objcore);
-	AN(sp->wrk->exp.ttl >= 0);
+	AN(sp->wrk->exp.ttl > 0.);
 
 	ltot = IRNUP(sc, ltot);
 
@@ -519,7 +519,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	/* We have to do this somewhere, might as well be here... */
 	assert(sizeof so->hash == DIGEST_LEN);
 	memcpy(so->hash, oc->objhead->digest, DIGEST_LEN);
-	so->ttl = o->exp.ttl;	/* XXX: grace? */
+	so->ttl = o->entered + o->exp.ttl;	/* XXX: grace? */
 	so->ptr = (uint8_t*)o - sc->base;
 	so->ban = o->ban_t;
 
diff --git a/bin/varnishd/storage_persistent_silo.c b/bin/varnishd/storage_persistent_silo.c
index 86f487b..dde8347 100644
--- a/bin/varnishd/storage_persistent_silo.c
+++ b/bin/varnishd/storage_persistent_silo.c
@@ -433,8 +433,8 @@ smp_oc_getobj(struct worker *wrk, struct objcore *oc)
 			bad |= 0x100;
 
 		if(bad) {
-			o->exp.ttl = 0;
-			o->exp.grace = 0;
+			o->exp.ttl = -1.;
+			o->exp.grace = 0.;
 			so->ttl = 0;
 		}
 
@@ -464,9 +464,9 @@ smp_oc_updatemeta(struct objcore *oc)
 	so = smp_find_so(sg, oc);
 
 	if (isnan(o->exp.grace))
-		mttl = o->exp.ttl;
+		mttl = o->entered + o->exp.ttl;
 	else
-		mttl = - (o->exp.ttl + o->exp.grace);
+		mttl = - (o->entered + o->exp.ttl + o->exp.grace);
 
 	if (sg == sg->sc->cur_seg) {
 		/* Lock necessary, we might race close_seg */



From phk at varnish-cache.org  Wed Mar  2 19:52:18 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 20:52:18 +0100
Subject: [master] 61cc67d Try to handle ttl and grace next to each other.
Message-ID: <E1Pus5q-00059G-Rm@project.varnish-software.com>

commit 61cc67d4a51fbc10516c94fc3f4eef6c519eb25f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 19:40:10 2011 +0000

    Try to handle ttl and grace next to each other.

diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 34be2c6..32dc4eb 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -528,6 +528,7 @@ cnt_fetch(struct sess *sp)
 	sp->wrk->entered = TIM_real();
 	sp->wrk->age = 0;
 	sp->wrk->exp.ttl = RFC2616_Ttl(sp);
+	sp->wrk->exp.grace = NAN;
 
 	/*
 	 * Initial cacheability determination per [RFC2616, 13.4]
@@ -552,7 +553,6 @@ cnt_fetch(struct sess *sp)
 		sp->wrk->exp.ttl = -1.;
 
 	sp->wrk->do_esi = 0;
-	sp->wrk->exp.grace = NAN;
 
 	sp->wrk->body_status = RFC2616_Body(sp);
 



From phk at varnish-cache.org  Wed Mar  2 19:52:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 20:52:19 +0100
Subject: [master] 3101d97 Move ttl and grace handling together
Message-ID: <E1Pus5r-00059O-2B@project.varnish-software.com>

commit 3101d97cc0fe04ea8c2f66299f2b2d6bd04e6f64
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 19:52:03 2011 +0000

    Move ttl and grace handling together

diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 60392c5..bed8e94 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -57,6 +57,10 @@ static pthread_t exp_thread;
 static struct binheap *exp_heap;
 static struct lock exp_mtx;
 
+/*--------------------------------------------------------------------
+ * struct exp manipulations
+ */
+
 double
 EXP_Grace(double g)
 {
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index ef89cdd..104718d 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -223,43 +223,6 @@ VRT_l_client_identity(struct sess *sp, const char *str, ...)
 	sp->client_identity = b;
 }
 
-/*--------------------------------------------------------------------
- * XXX: Working relative to t_req is maybe not the right thing, we could
- * XXX: have spent a long time talking to the backend since then.
- * XXX: It might make sense to cache a timestamp as "current time"
- * XXX: before vcl_recv (== t_req) and vcl_fetch.
- * XXX: On the other hand, that might lead to inconsistent behaviour
- * XXX: where an object expires while we are running VCL code, and
- * XXX: and that may not be a good idea either.
- * XXX: See also related t_req use in cache_hash.c
- */
-
-void
-VRT_l_beresp_ttl(const struct sess *sp, double a)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f", sp->xid, a, sp->t_req);
-	/*
-	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
-	 * any longer, but it will still be for up to 1s - epsilon because
-	 * of the rounding to seconds.
-	 * We special case and make sure that rounding does not surprise.
-	 */
-	if (a <= 0) {
-		sp->wrk->exp.ttl =  -1.;
-		sp->wrk->exp.grace = 0.;
-	} else
-		sp->wrk->exp.ttl = a;
-}
-
-double
-VRT_r_beresp_ttl(const struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->wrk->exp.ttl);
-}
-
 /*--------------------------------------------------------------------*/
 
 #define BEREQ_TIMEOUT(which)					\
@@ -334,40 +297,6 @@ VRT_l_beresp_storage(struct sess *sp, const char *str, ...)
 	sp->wrk->storage_hint = b;
 }
 
-void
-VRT_l_obj_ttl(const struct sess *sp, double a)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	if (sp->obj->objcore == NULL)
-		return;
-	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f",
-	    sp->obj->xid, a, sp->t_req);
-	/*
-	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
-	 * any longer, but it will still be for up to 1s - epsilon because
-	 * of the rounding to seconds.
-	 * We special case and make sure that rounding does not surprise.
-	 */
-	if (a <= 0) {
-		sp->obj->exp.ttl = -1.;
-		sp->obj->exp.grace = 0.;
-	} else
-		sp->obj->exp.ttl = a;
-	EXP_Rearm(sp->obj);
-}
-
-double
-VRT_r_obj_ttl(const struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	if (sp->obj->objcore == NULL)
-		return (0.0);
-	return (sp->obj->exp.ttl);
-}
-
 /*--------------------------------------------------------------------*/
 
 void
@@ -434,15 +363,91 @@ VRT_r_req_restarts(const struct sess *sp)
 	return (sp->restarts);
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * ttl and grace have special magical relationships, encapsulated in
+ * cache_expire.c
+ */
+
+void
+VRT_l_obj_ttl(const struct sess *sp, double a)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
+	if (sp->obj->objcore == NULL)
+		return;
+	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f",
+	    sp->obj->xid, a, sp->t_req);
+	/*
+	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
+	 * any longer, but it will still be for up to 1s - epsilon because
+	 * of the rounding to seconds.
+	 * We special case and make sure that rounding does not surprise.
+	 */
+	if (a <= 0) {
+		sp->obj->exp.ttl = -1.;
+		sp->obj->exp.grace = 0.;
+	} else
+		sp->obj->exp.ttl = a;
+	EXP_Rearm(sp->obj);
+}
+
+double
+VRT_r_obj_ttl(const struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
+	if (sp->obj->objcore == NULL)
+		return (0.0);
+	return (sp->obj->exp.ttl);
+}
+
+
+/*--------------------------------------------------------------------
+ * XXX: Working relative to t_req is maybe not the right thing, we could
+ * XXX: have spent a long time talking to the backend since then.
+ * XXX: It might make sense to cache a timestamp as "current time"
+ * XXX: before vcl_recv (== t_req) and vcl_fetch.
+ * XXX: On the other hand, that might lead to inconsistent behaviour
+ * XXX: where an object expires while we are running VCL code, and
+ * XXX: and that may not be a good idea either.
+ * XXX: See also related t_req use in cache_hash.c
+ */
+
+void
+VRT_l_beresp_ttl(const struct sess *sp, double a)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f", sp->xid, a, sp->t_req);
+	/*
+	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
+	 * any longer, but it will still be for up to 1s - epsilon because
+	 * of the rounding to seconds.
+	 * We special case and make sure that rounding does not surprise.
+	 */
+	if (a <= 0) {
+		sp->wrk->exp.ttl =  -1.;
+		sp->wrk->exp.grace = 0.;
+	} else
+		sp->wrk->exp.ttl = a;
+}
+
+double
+VRT_r_beresp_ttl(const struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (sp->wrk->exp.ttl);
+}
+
 
-#define VRT_DO_GRACE(which, fld, extra)				\
+#define VRT_DO_EXP(which, exp, fld, extra)			\
 void __match_proto__()						\
 VRT_l_##which##_grace(struct sess *sp, double a)		\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	fld = a >= 0.0 ? a : NAN;				\
+	exp.fld = a >= 0.0 ? a : NAN;				\
 	extra;							\
 }								\
 								\
@@ -451,12 +456,12 @@ VRT_r_##which##_grace(struct sess *sp)				\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(EXP_Grace(fld));					\
+	return(EXP_Grace(exp.fld));				\
 }
 
-VRT_DO_GRACE(req, sp->exp.grace, )
-VRT_DO_GRACE(obj, sp->obj->exp.grace, EXP_Rearm(sp->obj))
-VRT_DO_GRACE(beresp, sp->wrk->exp.grace, )
+VRT_DO_EXP(req, sp->exp, grace, )
+VRT_DO_EXP(obj, sp->obj->exp, grace, EXP_Rearm(sp->obj))
+VRT_DO_EXP(beresp, sp->wrk->exp, grace, )
 
 /*--------------------------------------------------------------------
  * req.xid



From phk at varnish-cache.org  Wed Mar  2 20:35:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 21:35:40 +0100
Subject: [master] a21746d Unify .grace and .ttl VCL variable handling
Message-ID: <E1Puslo-0007ux-5K@project.varnish-software.com>

commit a21746d23d4047ce209a0c283e12ff684f478b72
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 20:35:07 2011 +0000

    Unify .grace and .ttl VCL variable handling

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index ecad892..dfeb6be 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -633,6 +633,11 @@ extern pthread_t cli_thread;
 #define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
 
 /* cache_expiry.c */
+double EXP_Get_grace(const struct exp *e);
+double EXP_Get_ttl(const struct exp *e);
+void EXP_Set_grace(struct exp *e, double v);
+void EXP_Set_ttl(struct exp *e, double v);
+
 double EXP_Grace(double g);
 void EXP_Insert(struct object *o);
 void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index bed8e94..3c5f661 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -59,8 +59,47 @@ static struct lock exp_mtx;
 
 /*--------------------------------------------------------------------
  * struct exp manipulations
+ *
+ * The Get/Set functions encapsulate the mutual magic between the
+ * fields in one single place.
  */
 
+void
+EXP_Set_grace(struct exp *e, double v)
+{
+
+	if (v > 0.)
+		e->grace = v;
+	else
+		e->grace = NAN;
+}
+
+double
+EXP_Get_grace(const struct exp *e)
+{
+
+	return (e->grace > 0. ? e->grace : -1.);
+}
+
+void
+EXP_Set_ttl(struct exp *e, double v)
+{
+
+	if (v > 0.)
+		e->ttl = v;
+	else {
+		e->ttl = -1.;
+		e->grace = NAN;
+	}
+}
+
+double
+EXP_Get_ttl(const struct exp *e)
+{
+
+	return (e->ttl > 0. ? e->ttl : -1.);
+}
+
 double
 EXP_Grace(double g)
 {
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index 104718d..e943526 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -363,105 +363,32 @@ VRT_r_req_restarts(const struct sess *sp)
 	return (sp->restarts);
 }
 
-/*--------------------------------------------------------------------
- * ttl and grace have special magical relationships, encapsulated in
- * cache_expire.c
- */
-
-void
-VRT_l_obj_ttl(const struct sess *sp, double a)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	if (sp->obj->objcore == NULL)
-		return;
-	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f",
-	    sp->obj->xid, a, sp->t_req);
-	/*
-	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
-	 * any longer, but it will still be for up to 1s - epsilon because
-	 * of the rounding to seconds.
-	 * We special case and make sure that rounding does not surprise.
-	 */
-	if (a <= 0) {
-		sp->obj->exp.ttl = -1.;
-		sp->obj->exp.grace = 0.;
-	} else
-		sp->obj->exp.ttl = a;
-	EXP_Rearm(sp->obj);
-}
-
-double
-VRT_r_obj_ttl(const struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	if (sp->obj->objcore == NULL)
-		return (0.0);
-	return (sp->obj->exp.ttl);
-}
-
-
-/*--------------------------------------------------------------------
- * XXX: Working relative to t_req is maybe not the right thing, we could
- * XXX: have spent a long time talking to the backend since then.
- * XXX: It might make sense to cache a timestamp as "current time"
- * XXX: before vcl_recv (== t_req) and vcl_fetch.
- * XXX: On the other hand, that might lead to inconsistent behaviour
- * XXX: where an object expires while we are running VCL code, and
- * XXX: and that may not be a good idea either.
- * XXX: See also related t_req use in cache_hash.c
- */
-
-void
-VRT_l_beresp_ttl(const struct sess *sp, double a)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f", sp->xid, a, sp->t_req);
-	/*
-	 * If people set obj.ttl = 0s, they don't expect it to be cacheable
-	 * any longer, but it will still be for up to 1s - epsilon because
-	 * of the rounding to seconds.
-	 * We special case and make sure that rounding does not surprise.
-	 */
-	if (a <= 0) {
-		sp->wrk->exp.ttl =  -1.;
-		sp->wrk->exp.grace = 0.;
-	} else
-		sp->wrk->exp.ttl = a;
-}
-
-double
-VRT_r_beresp_ttl(const struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->wrk->exp.ttl);
-}
-
+/*--------------------------------------------------------------------*/
 
 #define VRT_DO_EXP(which, exp, fld, extra)			\
+								\
 void __match_proto__()						\
-VRT_l_##which##_grace(struct sess *sp, double a)		\
+VRT_l_##which##_##fld(struct sess *sp, double a)		\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	exp.fld = a >= 0.0 ? a : NAN;				\
+	EXP_Set_##fld(&exp, a);					\
 	extra;							\
 }								\
 								\
 double __match_proto__()					\
-VRT_r_##which##_grace(struct sess *sp)				\
+VRT_r_##which##_##fld(struct sess *sp)				\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(EXP_Grace(exp.fld));				\
+	return(EXP_Get_##fld(&exp));				\
 }
 
 VRT_DO_EXP(req, sp->exp, grace, )
 VRT_DO_EXP(obj, sp->obj->exp, grace, EXP_Rearm(sp->obj))
+VRT_DO_EXP(obj, sp->obj->exp, ttl, EXP_Rearm(sp->obj))
 VRT_DO_EXP(beresp, sp->wrk->exp, grace, )
+VRT_DO_EXP(beresp, sp->wrk->exp, ttl, )
 
 /*--------------------------------------------------------------------
  * req.xid
diff --git a/bin/varnishtest/tests/b00030.vtc b/bin/varnishtest/tests/b00030.vtc
index d578822..787aa0e 100644
--- a/bin/varnishtest/tests/b00030.vtc
+++ b/bin/varnishtest/tests/b00030.vtc
@@ -21,5 +21,5 @@ varnish v1 -vcl+backend {
 client c1 {
 	txreq
 	rxresp
-	expect resp.msg == "10.000"
+	expect resp.msg == "-1.000"
 } -run



From phk at varnish-cache.org  Wed Mar  2 20:38:37 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 21:38:37 +0100
Subject: [master] f04a08e Change prototypes for .ttl and .grace VRT functions
	to be consistent
Message-ID: <E1Pusof-0008AA-G3@project.varnish-software.com>

commit f04a08e82ec694e455acd459dde63089dfb5e5ba
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 20:37:56 2011 +0000

    Change prototypes for .ttl and .grace VRT functions to be consistent

diff --git a/lib/libvcl/generate.py b/lib/libvcl/generate.py
index 6e27946..72edc9b 100755
--- a/lib/libvcl/generate.py
+++ b/lib/libvcl/generate.py
@@ -314,7 +314,7 @@ sp_variables = (
 		'DURATION',
 		( 'fetch',),
 		( 'fetch',),
-		'const struct sess *'
+		'struct sess *'
 	),
 	('beresp.grace',
 		'DURATION',
@@ -380,7 +380,7 @@ sp_variables = (
 		'DURATION',
 		( 'hit', 'error',),
 		( 'hit', 'error',),
-		'const struct sess *'
+		'struct sess *'
 	),
 	('obj.grace',
 		'DURATION',



From phk at varnish-cache.org  Wed Mar  2 21:40:51 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 22:40:51 +0100
Subject: [master] fc836dd Encapsulate the calculation of ttl and grace times
	in functions in cache_expire.c
Message-ID: <E1Putmt-0003fP-1H@project.varnish-software.com>

commit fc836dd23d4aa035ee50067e54cfe3ba6268ae16
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 21:40:00 2011 +0000

    Encapsulate the calculation of ttl and grace times in functions
    in cache_expire.c
    
    All this interior decoration should make it less painful to add
    yet a timer for IMS processing.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index dfeb6be..f317e57 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -638,7 +638,8 @@ double EXP_Get_ttl(const struct exp *e);
 void EXP_Set_grace(struct exp *e, double v);
 void EXP_Set_ttl(struct exp *e, double v);
 
-double EXP_Grace(double g);
+double EXP_Grace(const struct sess *, const struct object*);
+double EXP_Ttl(const struct sess *, const struct object*);
 void EXP_Insert(struct object *o);
 void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
 void EXP_Init(void);
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 3c5f661..b586922 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -71,7 +71,7 @@ EXP_Set_grace(struct exp *e, double v)
 	if (v > 0.)
 		e->grace = v;
 	else
-		e->grace = NAN;
+		e->grace = -1.;
 }
 
 double
@@ -89,7 +89,7 @@ EXP_Set_ttl(struct exp *e, double v)
 		e->ttl = v;
 	else {
 		e->ttl = -1.;
-		e->grace = NAN;
+		e->grace = -1.;
 	}
 }
 
@@ -100,12 +100,33 @@ EXP_Get_ttl(const struct exp *e)
 	return (e->ttl > 0. ? e->ttl : -1.);
 }
 
+/*--------------------------------------------------------------------
+ * Calculate when an object is out of ttl or grace, possibly constrained
+ * by per-session limits.
+ */
+
+double
+EXP_Grace(const struct sess *sp, const struct object *o)
+{
+	double r;
+
+	r = (double)params->default_grace;
+	if (o->exp.grace > 0.)
+		r = o->exp.grace;
+	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace > r)
+		r = sp->exp.grace;
+	return (EXP_Ttl(sp, o) + r);
+}
+
 double
-EXP_Grace(double g)
+EXP_Ttl(const struct sess *sp, const struct object *o)
 {
-	if (isnan(g))
-		return (double)(params->default_grace);
-	return (g);
+	double r;
+
+	r = o->exp.ttl;
+	if (sp != NULL && sp->exp.ttl > 0. && sp->exp.ttl > r)
+		r = sp->exp.ttl;
+	return (o->entered + r);
 }
 
 /*--------------------------------------------------------------------
@@ -123,7 +144,7 @@ update_object_when(const struct object *o)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	Lck_AssertHeld(&exp_mtx);
 
-	when = o->entered + o->exp.ttl + EXP_Grace(o->exp.grace);
+	when = EXP_Grace(NULL, o);
 	assert(!isnan(when));
 	if (when == oc->timer_when)
 		return (0);
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index 56528ab..ea07f2b 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -362,14 +362,14 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 			continue;
 
 		/* If still valid, use it */
-		if (o->entered + o->exp.ttl >= sp->t_req)
+		if (EXP_Ttl(sp, o) >= sp->t_req)
 			break;
 
 		/*
 		 * Remember any matching objects inside their grace period
 		 * and if there are several, use the least expired one.
 		 */
-		if (o->entered + o->exp.ttl + EXP_Grace(o->exp.grace) >= sp->t_req) {
+		if (EXP_Grace(sp, o) >= sp->t_req) {
 			if (grace_oc == NULL || grace_ttl < o->entered + o->exp.ttl) {
 				grace_oc = oc;
 				grace_ttl = o->entered + o->exp.ttl;
@@ -396,8 +396,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 					/* Or it is impossible to fetch */
 		o = oc_getobj(sp->wrk, grace_oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		if (o->entered + o->exp.ttl + EXP_Grace(sp->exp.grace) >= sp->t_req)
-			oc = grace_oc;
+		oc = grace_oc;
 	}
 	sp->objcore = NULL;
 
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index e943526..779af7b 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -34,7 +34,6 @@
 SVNID("$Id$")
 
 #include <stdio.h>
-#include <math.h>
 #include <stdarg.h>
 #include <stdlib.h>
 



From phk at varnish-cache.org  Wed Mar  2 22:28:38 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 02 Mar 2011 23:28:38 +0100
Subject: [master] c2b0f14 Final (?) Part of ttl/grace clean up: Add an
	EXP_Clr() function to initialize struct exp, and use it.
Message-ID: <E1PuuX8-0006dF-9e@project.varnish-software.com>

commit c2b0f14963a2915521e864e32cc2b30de1fcd8d9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 2 22:26:42 2011 +0000

    Final (?) Part of ttl/grace clean up:  Add an EXP_Clr() function to
    initialize struct exp, and use it.
    
    Pass an 'exp' instead of just the ttl to STV_NewObject()
    
    Have -spersistent always use the graced timeout for object index,
    rearm when we resurrect the zombie object.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index f317e57..7c97012 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -633,6 +633,7 @@ extern pthread_t cli_thread;
 #define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
 
 /* cache_expiry.c */
+void EXP_Clr(struct exp *e);
 double EXP_Get_grace(const struct exp *e);
 double EXP_Get_ttl(const struct exp *e);
 void EXP_Set_grace(struct exp *e, double v);
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 241001b..4480c98 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -475,8 +475,7 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 		oc_updatemeta(oc);
 		return (0);
 	} else {
-		o->exp.ttl = -1.;
-		o->exp.grace = -1.;
+		EXP_Clr(&o->exp);
 		oc->ban = NULL;
 		oc_updatemeta(oc);
 		/* BAN also changed, but that is not important any more */
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 32dc4eb..c991400 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -381,7 +381,8 @@ cnt_error(struct sess *sp)
 	if (sp->obj == NULL) {
 		HSH_Prealloc(sp);
 		/* XXX: 1024 is a pure guess */
-		sp->obj = STV_NewObject(sp, NULL, 1024, 0,
+		EXP_Clr(&w->exp);
+		sp->obj = STV_NewObject(sp, NULL, 1024, &w->exp,
 		     params->http_max_hdr);
 		sp->obj->xid = sp->xid;
 		sp->obj->entered = sp->t_req;
@@ -527,8 +528,8 @@ cnt_fetch(struct sess *sp)
 	 */
 	sp->wrk->entered = TIM_real();
 	sp->wrk->age = 0;
+	EXP_Clr(&sp->wrk->exp);
 	sp->wrk->exp.ttl = RFC2616_Ttl(sp);
-	sp->wrk->exp.grace = NAN;
 
 	/*
 	 * Initial cacheability determination per [RFC2616, 13.4]
@@ -657,7 +658,7 @@ cnt_fetch(struct sess *sp)
 		sp->wrk->storage_hint = TRANSIENT_STORAGE;
 
 	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
-	    sp->wrk->exp.ttl, nhttp);
+	    &sp->wrk->exp, nhttp);
 	/* XXX: -> 513 */
 	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
 
@@ -677,7 +678,6 @@ cnt_fetch(struct sess *sp)
 
 	sp->obj->xid = sp->xid;
 	sp->obj->response = sp->err_code;
-	sp->obj->exp.grace = sp->wrk->exp.grace;
 	sp->obj->age = sp->wrk->age;
 	sp->obj->entered = sp->wrk->entered;
 	WS_Assert(sp->obj->ws_o);
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index b586922..6c2ccb5 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -65,6 +65,14 @@ static struct lock exp_mtx;
  */
 
 void
+EXP_Clr(struct exp *e)
+{
+
+	e->grace = -1;
+	e->ttl = -1;
+}
+
+void
 EXP_Set_grace(struct exp *e, double v)
 {
 
@@ -113,7 +121,7 @@ EXP_Grace(const struct sess *sp, const struct object *o)
 	r = (double)params->default_grace;
 	if (o->exp.grace > 0.)
 		r = o->exp.grace;
-	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace > r)
+	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace < r)
 		r = sp->exp.grace;
 	return (EXP_Ttl(sp, o) + r);
 }
@@ -124,7 +132,7 @@ EXP_Ttl(const struct sess *sp, const struct object *o)
 	double r;
 
 	r = o->exp.ttl;
-	if (sp != NULL && sp->exp.ttl > 0. && sp->exp.ttl > r)
+	if (sp != NULL && sp->exp.ttl > 0. && sp->exp.ttl < r)
 		r = sp->exp.ttl;
 	return (o->entered + r);
 }
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index ea07f2b..21007b5 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -370,7 +370,8 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 		 * and if there are several, use the least expired one.
 		 */
 		if (EXP_Grace(sp, o) >= sp->t_req) {
-			if (grace_oc == NULL || grace_ttl < o->entered + o->exp.ttl) {
+			if (grace_oc == NULL ||
+			    grace_ttl < o->entered + o->exp.ttl) {
 				grace_oc = oc;
 				grace_ttl = o->entered + o->exp.ttl;
 			}
@@ -534,8 +535,11 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
 	}
 	Lck_Unlock(&oh->mtx);
 
-	if (ttl <= 0)
+	/* NB: inverse test to catch NAN also */
+	if (!(ttl > 0.))
 		ttl = -1.;
+	if (!(grace > 0.))
+		grace = -1.;
 	for (n = 0; n < nobj; n++) {
 		oc = ocp[n];
 		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -544,8 +548,7 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
 			continue;
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 		o->exp.ttl = ttl;
-		if (!isnan(grace))
-			o->exp.grace = grace;
+		o->exp.grace = grace;
 		EXP_Rearm(o);
 		(void)HSH_Deref(sp->wrk, NULL, &o);
 	}
diff --git a/bin/varnishd/cache_session.c b/bin/varnishd/cache_session.c
index 0fa662f..17d228d 100644
--- a/bin/varnishd/cache_session.c
+++ b/bin/varnishd/cache_session.c
@@ -174,7 +174,7 @@ ses_setup(struct sessmem *sm)
 	sp->t_req = NAN;
 	sp->t_resp = NAN;
 	sp->t_end = NAN;
-	sp->exp.grace = NAN;
+	EXP_Clr(&sp->exp);
 
 	WS_Init(sp->ws, "sess", sm->wsp, sm->workspace);
 	sp->http = sm->http[0];
diff --git a/bin/varnishd/stevedore.c b/bin/varnishd/stevedore.c
index 915eadc..1fcf75c 100644
--- a/bin/varnishd/stevedore.c
+++ b/bin/varnishd/stevedore.c
@@ -158,7 +158,7 @@ struct stv_objsecrets {
 	unsigned	nhttp;
 	unsigned	lhttp;
 	unsigned	wsl;
-	double		ttl;
+	struct exp	*exp;
 };
 
 /*--------------------------------------------------------------------
@@ -198,9 +198,8 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
 
 	http_Setup(o->http, o->ws_o);
 	o->http->magic = HTTP_MAGIC;
-	o->exp.grace = NAN;
 	o->entered = NAN;
-	o->exp.ttl = soc->ttl;
+	o->exp = *soc->exp;
 	VTAILQ_INIT(&o->store);
 	sp->wrk->stats.n_object++;
 
@@ -251,7 +250,7 @@ stv_default_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
  */
 
 struct object *
-STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, double ttl,
+STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, struct exp *ep,
     unsigned nhttp)
 {
 	struct object *o;
@@ -270,7 +269,7 @@ STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, double ttl,
 	soc.nhttp = nhttp;
 	soc.lhttp = lhttp;
 	soc.wsl = wsl;
-	soc.ttl = ttl;
+	soc.exp = ep;
 
 	ltot = sizeof *o + wsl + lhttp;
 
diff --git a/bin/varnishd/stevedore.h b/bin/varnishd/stevedore.h
index e97d70f..6036b32 100644
--- a/bin/varnishd/stevedore.h
+++ b/bin/varnishd/stevedore.h
@@ -30,6 +30,7 @@
  */
 
 struct stevedore;
+struct exp;
 struct sess;
 struct iovec;
 struct object;
@@ -91,7 +92,7 @@ struct object *STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
     const struct stv_objsecrets *soc);
 
 struct object *STV_NewObject(struct sess *sp, const char *hint, unsigned len,
-    double ttl, unsigned nhttp);
+    struct exp *, unsigned nhttp);
 struct storage *STV_alloc(const struct sess *sp, size_t size);
 void STV_trim(struct storage *st, size_t size);
 void STV_free(struct storage *st);
diff --git a/bin/varnishd/storage_persistent.c b/bin/varnishd/storage_persistent.c
index c5cf339..de28e7a 100644
--- a/bin/varnishd/storage_persistent.c
+++ b/bin/varnishd/storage_persistent.c
@@ -519,7 +519,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	/* We have to do this somewhere, might as well be here... */
 	assert(sizeof so->hash == DIGEST_LEN);
 	memcpy(so->hash, oc->objhead->digest, DIGEST_LEN);
-	so->ttl = o->entered + o->exp.ttl;	/* XXX: grace? */
+	so->ttl = EXP_Grace(NULL, o);
 	so->ptr = (uint8_t*)o - sc->base;
 	so->ban = o->ban_t;
 
diff --git a/bin/varnishd/storage_persistent_silo.c b/bin/varnishd/storage_persistent_silo.c
index dde8347..12de04a 100644
--- a/bin/varnishd/storage_persistent_silo.c
+++ b/bin/varnishd/storage_persistent_silo.c
@@ -152,9 +152,7 @@ smp_load_seg(const struct sess *sp, const struct smp_sc *sc,
 	/* Clear the bogus "hold" count */
 	sg->nobj = 0;
 	for (;no > 0; so++,no--) {
-		if (so->ttl > 0 && so->ttl < t_now)
-			continue;
-		if (so->ttl < 0 && -so->ttl < t_now)
+		if (so->ttl == 0 || so->ttl < t_now)
 			continue;
 		HSH_Prealloc(sp);
 		oc = sp->wrk->nobjcore;
@@ -165,7 +163,7 @@ smp_load_seg(const struct sess *sp, const struct smp_sc *sc,
 		memcpy(sp->wrk->nobjhead->digest, so->hash, SHA256_LEN);
 		(void)HSH_Insert(sp);
 		AZ(sp->wrk->nobjcore);
-		EXP_Inject(oc, sg->lru, fabs(so->ttl));
+		EXP_Inject(oc, sg->lru, so->ttl);
 		sg->nobj++;
 	}
 	WRK_SumStat(sp->wrk);
@@ -433,10 +431,9 @@ smp_oc_getobj(struct worker *wrk, struct objcore *oc)
 			bad |= 0x100;
 
 		if(bad) {
-			o->exp.ttl = -1.;
-			o->exp.grace = 0.;
+			EXP_Set_ttl(&o->exp, -1);
 			so->ttl = 0;
-		}
+		} 
 
 		sg->nfixed++;
 		wrk->stats.n_object++;
@@ -444,6 +441,7 @@ smp_oc_getobj(struct worker *wrk, struct objcore *oc)
 		oc->flags &= ~OC_F_NEEDFIXUP;
 	}
 	Lck_Unlock(&sg->sc->mtx);
+	EXP_Rearm(o);
 	return (o);
 }
 
@@ -463,10 +461,7 @@ smp_oc_updatemeta(struct objcore *oc)
 	CHECK_OBJ_NOTNULL(sg->sc, SMP_SC_MAGIC);
 	so = smp_find_so(sg, oc);
 
-	if (isnan(o->exp.grace))
-		mttl = o->entered + o->exp.ttl;
-	else
-		mttl = - (o->entered + o->exp.ttl + o->exp.grace);
+	mttl = EXP_Grace(NULL, o);
 
 	if (sg == sg->sc->cur_seg) {
 		/* Lock necessary, we might race close_seg */



From bjorn at varnish-cache.org  Thu Mar  3 07:49:57 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 08:49:57 +0100
Subject: [master] f615d8a glossary typo
Message-ID: <E1Pv3IL-0007Wz-PJ@project.varnish-software.com>

commit f615d8ab100953acab3ef383bc6a35c430097d27
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 08:48:24 2011 +0100

    glossary typo

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index f1aa0e6..70a40ed 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -82,8 +82,8 @@ Varnish Glossary
 	to varnishd they are just bytes.
 
    object
-	The cached version of a response.  Varnishd receives a reponse
-	from the backend and creates an object, from which it can
+	The (possibly cached version of a) response.  Varnishd receives a reponse
+	from the backend and creates an object, from which it may
 	produce cached responses to clients.
 
    .. comment: "configuration of varnishd -----------------------------"
@@ -99,7 +99,7 @@ Varnish Glossary
 
    miss
 	An object Varnish fetches from the backend.  It may or may not
-	be putin the cache, that depends.
+	be put in the cache, that depends.
 
    pass
 	An object Varnish does not try to cache, but simply fetches



From bjorn at varnish-cache.org  Thu Mar  3 08:03:56 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 09:03:56 +0100
Subject: [2.1] varnish-2.1.5-6-g6382fbc
Message-ID: <E1Pv3Vs-0002EX-1K@project.varnish-software.com>

commit 6382fbc3da92e80ea6c1c335a3d52f6a7988351d
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 08:48:24 2011 +0100

    glossary typo

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index f1aa0e6..70a40ed 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -82,8 +82,8 @@ Varnish Glossary
 	to varnishd they are just bytes.
 
    object
-	The cached version of a response.  Varnishd receives a reponse
-	from the backend and creates an object, from which it can
+	The (possibly cached version of a) response.  Varnishd receives a reponse
+	from the backend and creates an object, from which it may
 	produce cached responses to clients.
 
    .. comment: "configuration of varnishd -----------------------------"
@@ -99,7 +99,7 @@ Varnish Glossary
 
    miss
 	An object Varnish fetches from the backend.  It may or may not
-	be putin the cache, that depends.
+	be put in the cache, that depends.
 
    pass
 	An object Varnish does not try to cache, but simply fetches



From bjorn at varnish-cache.org  Thu Mar  3 08:20:53 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 09:20:53 +0100
Subject: [master] 7427be8 nicer definitions
Message-ID: <E1Pv3mH-0005uZ-WD@project.varnish-software.com>

commit 7427be86c924c82ab28c444e6706ed8db0e71e4e
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 09:19:39 2011 +0100

    nicer definitions

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index 70a40ed..94a104d 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -30,12 +30,12 @@ Varnish Glossary
 	The worker process is started and configured by the master
 	process.  This is the process that does all the work you actually
 	want varnish to do.  If the worker dies, the master will try start
-	it again, to keep your website alive..
+	it again, to keep your website alive.
 
    backend
 	The HTTP server varnishd is caching for.  This can be
 	any sort of device that handles HTTP requests, including, but
-	not limited to:  a webserver, a CMS, a load-balancer 
+	not limited to: a webserver, a CMS, a load-balancer 
 	another varnishd, etc.
 
    client
@@ -82,9 +82,11 @@ Varnish Glossary
 	to varnishd they are just bytes.
 
    object
-	The (possibly cached version of a) response.  Varnishd receives a reponse
-	from the backend and creates an object, from which it may
-	produce cached responses to clients.
+	The (possibly) cached version of a backend response. Varnishd
+	receives a reponse from the backend and creates an object,
+	from which it may produce cached responses to clients. If the
+	backend response is created from a request which is passed, it
+	will not be stored for caching.
 
    .. comment: "configuration of varnishd -----------------------------"
 
@@ -98,8 +100,9 @@ Varnish Glossary
 	An object Varnish delivers from cache.
 
    miss
-	An object Varnish fetches from the backend.  It may or may not
-	be put in the cache, that depends.
+	An object Varnish fetches from the backend before it is served
+	to the client.  The object may or may not be put in the cache,
+	that depends.
 
    pass
 	An object Varnish does not try to cache, but simply fetches



From bjorn at varnish-cache.org  Thu Mar  3 08:24:00 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 09:24:00 +0100
Subject: [2.1] varnish-2.1.5-7-g0605c36
Message-ID: <E1Pv3pI-0008Hw-9P@project.varnish-software.com>

commit 0605c368edbb7c6fe979050bf74abde62a703866
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 09:19:39 2011 +0100

    nicer definitions

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index 70a40ed..94a104d 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -30,12 +30,12 @@ Varnish Glossary
 	The worker process is started and configured by the master
 	process.  This is the process that does all the work you actually
 	want varnish to do.  If the worker dies, the master will try start
-	it again, to keep your website alive..
+	it again, to keep your website alive.
 
    backend
 	The HTTP server varnishd is caching for.  This can be
 	any sort of device that handles HTTP requests, including, but
-	not limited to:  a webserver, a CMS, a load-balancer 
+	not limited to: a webserver, a CMS, a load-balancer 
 	another varnishd, etc.
 
    client
@@ -82,9 +82,11 @@ Varnish Glossary
 	to varnishd they are just bytes.
 
    object
-	The (possibly cached version of a) response.  Varnishd receives a reponse
-	from the backend and creates an object, from which it may
-	produce cached responses to clients.
+	The (possibly) cached version of a backend response. Varnishd
+	receives a reponse from the backend and creates an object,
+	from which it may produce cached responses to clients. If the
+	backend response is created from a request which is passed, it
+	will not be stored for caching.
 
    .. comment: "configuration of varnishd -----------------------------"
 
@@ -98,8 +100,9 @@ Varnish Glossary
 	An object Varnish delivers from cache.
 
    miss
-	An object Varnish fetches from the backend.  It may or may not
-	be put in the cache, that depends.
+	An object Varnish fetches from the backend before it is served
+	to the client.  The object may or may not be put in the cache,
+	that depends.
 
    pass
 	An object Varnish does not try to cache, but simply fetches



From bjorn at varnish-cache.org  Thu Mar  3 08:57:16 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 09:57:16 +0100
Subject: [master] 802cfca clarifications (again)
Message-ID: <E1Pv4LU-0004Qf-No@project.varnish-software.com>

commit 802cfca7527de99e6cbaa07dbe44b8a7c0c6cc70
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 09:56:59 2011 +0100

    clarifications (again)

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index 94a104d..7a2787d 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -76,6 +76,11 @@ Varnish Glossary
 	the client.  When the response is stored in varnishd's cache,
 	we call it an object.
 
+   backend response
+        The response specifically served from a backend to
+        varnishd. The backend response may be manipulated in
+        vcl_fetch.
+
    body
 	The bytes that make up the contents of the object, varnishd
 	does not care if they are in HTML, XML, JPEG or even EBCDIC,
@@ -84,8 +89,8 @@ Varnish Glossary
    object
 	The (possibly) cached version of a backend response. Varnishd
 	receives a reponse from the backend and creates an object,
-	from which it may produce cached responses to clients. If the
-	backend response is created from a request which is passed, it
+	from which it may deliver cached responses to clients. If the
+	object is created as a result of a request which is passed, it
 	will not be stored for caching.
 
    .. comment: "configuration of varnishd -----------------------------"



From bjorn at varnish-cache.org  Thu Mar  3 09:00:00 2011
From: bjorn at varnish-cache.org (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=)
Date: Thu, 03 Mar 2011 10:00:00 +0100
Subject: [2.1] varnish-2.1.5-8-gf3710ac
Message-ID: <E1Pv4O8-0005pi-OR@project.varnish-software.com>

commit f3710ac081854803005791e49ad65712fdf28ba2
Author: Bjoern Ruberg <bjorn at ruberg.no>
Date:   Thu Mar 3 09:56:59 2011 +0100

    clarifications (again)

diff --git a/doc/sphinx/glossary/index.rst b/doc/sphinx/glossary/index.rst
index 94a104d..7a2787d 100644
--- a/doc/sphinx/glossary/index.rst
+++ b/doc/sphinx/glossary/index.rst
@@ -76,6 +76,11 @@ Varnish Glossary
 	the client.  When the response is stored in varnishd's cache,
 	we call it an object.
 
+   backend response
+        The response specifically served from a backend to
+        varnishd. The backend response may be manipulated in
+        vcl_fetch.
+
    body
 	The bytes that make up the contents of the object, varnishd
 	does not care if they are in HTML, XML, JPEG or even EBCDIC,
@@ -84,8 +89,8 @@ Varnish Glossary
    object
 	The (possibly) cached version of a backend response. Varnishd
 	receives a reponse from the backend and creates an object,
-	from which it may produce cached responses to clients. If the
-	backend response is created from a request which is passed, it
+	from which it may deliver cached responses to clients. If the
+	object is created as a result of a request which is passed, it
 	will not be stored for caching.
 
    .. comment: "configuration of varnishd -----------------------------"



From phk at varnish-cache.org  Thu Mar  3 15:14:33 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 03 Mar 2011 16:14:33 +0100
Subject: [master] 2b4e631 Make default_ttl/grace doubles,
	polish rfc2616 ttl calculation accordingly.
Message-ID: <E1PvAEb-00081U-6s@project.varnish-software.com>

commit 2b4e631769eb1fba1219b0d38fe83be56ae05a60
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 3 09:39:43 2011 +0000

    Make default_ttl/grace doubles, polish rfc2616 ttl calculation
    accordingly.

diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index c027838..06b6f5d 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -71,7 +71,7 @@ struct params {
 	gid_t			gid;
 
 	/* TTL used for lack of anything better */
-	unsigned		default_ttl;
+	double			default_ttl;
 
 	/* Maximum concurrent sessions */
 	unsigned		max_sess;
@@ -163,7 +163,7 @@ struct params {
 	unsigned		diag_bitmap;
 
 	/* Default grace period */
-	unsigned		default_grace;
+	double			default_grace;
 
 	/* Log hash string to shm */
 	unsigned		log_hash;
diff --git a/bin/varnishd/mgt_param.c b/bin/varnishd/mgt_param.c
index 201aaab..1f7eb50 100644
--- a/bin/varnishd/mgt_param.c
+++ b/bin/varnishd/mgt_param.c
@@ -502,7 +502,8 @@ static const struct parspec input_parspec[] = {
 		"The unprivileged group to run as.",
 		MUST_RESTART,
 		MAGIC_INIT_STRING },
-	{ "default_ttl", tweak_uint, &master.default_ttl, 0, UINT_MAX,
+	{ "default_ttl", tweak_timeout_double, &master.default_ttl,
+		0, UINT_MAX,
 		"The TTL assigned to objects if neither the backend nor "
 		"the VCL code assigns one.\n"
 		"Objects already cached will not be affected by changes "
@@ -573,7 +574,8 @@ static const struct parspec input_parspec[] = {
 		"Maximum is 65535 bytes.",
 		0,
 		"255", "bytes" },
-	{ "default_grace", tweak_uint, &master.default_grace, 0, UINT_MAX,
+	{ "default_grace", tweak_timeout_double, &master.default_grace,
+		0, UINT_MAX,
 		"Default grace period.  We will deliver an object "
 		"this long after it has expired, provided another thread "
 		"is attempting to get a new copy.\n"
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index 42d9856..1c07751 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -71,9 +71,9 @@ SVNID("$Id$")
 double
 RFC2616_Ttl(const struct sess *sp)
 {
-	int ttl;
+	double ttl;
 	unsigned max_age, age;
-	double h_date, h_expires, ttd;
+	double h_date, h_expires;
 	char *p;
 	const struct http *hp;
 
@@ -84,7 +84,6 @@ RFC2616_Ttl(const struct sess *sp)
 	ttl = params->default_ttl;
 
 	max_age = age = 0;
-	ttd = 0;
 	h_expires = 0;
 	h_date = 0;
 
@@ -116,6 +115,8 @@ RFC2616_Ttl(const struct sess *sp)
 
 		if (http_GetHdr(hp, H_Expires, &p))
 			h_expires = TIM_parse(p);
+
+		/* No expire header, fall back to default */
 		if (h_expires == 0)
 			break;
 
@@ -129,8 +130,7 @@ RFC2616_Ttl(const struct sess *sp)
 		}
 
 		if (h_date == 0 ||
-		    (h_date < sp->wrk->entered + params->clock_skew &&
-		    h_date + params->clock_skew > sp->wrk->entered)) {
+		    fabs(h_date - sp->wrk->entered) < params->clock_skew) {
 			/*
 			 * If we have no Date: header or if it is
 			 * sufficiently close to our clock we will
@@ -139,27 +139,22 @@ RFC2616_Ttl(const struct sess *sp)
 			if (h_expires < sp->wrk->entered)
 				ttl = 0;
 			else
-				ttd = h_expires;
+				ttl = h_expires - sp->wrk->entered;
 			break;
+		} else {
+			/*
+			 * But even if the clocks are out of whack we can still
+			 * derive a relative time from the two headers.
+			 * (the negative ttl case is caught above)
+			 */
+			ttl = (int)(h_expires - h_date);
 		}
 
-		/*
-		 * But even if the clocks are out of whack we can still
-		 * derive a relative time from the two headers.
-		 * (the negative ttl case is caught above)
-		 */
-		ttl = (int)(h_expires - h_date);
-
 	} while (0);
 
-	if (ttd > 0)
-		ttl = ttd - sp->wrk->entered;
-
 	/* calculated TTL, Our time, Date, Expires, max-age, age */
-	WSP(sp, SLT_TTL, "%u RFC %d %d %d %d %u %u", sp->xid,
-	    ttd ? (int)(ttl) : 0,
-	    (int)sp->wrk->entered, (int)h_date,
-	    (int)h_expires, max_age, age);
+	WSP(sp, SLT_TTL, "%u RFC %g %g %g %g %u %u", sp->xid,
+	    ttl, sp->wrk->entered, h_date, h_expires, max_age, age);
 
 	return (ttl);
 }



From tfheen at varnish-cache.org  Thu Mar  3 19:30:19 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 03 Mar 2011 20:30:19 +0100
Subject: [master] 6587c5d Fix typo in example
Message-ID: <E1PvEE7-00062o-TB@project.varnish-software.com>

commit 6587c5d5c7033a92cecdea4f8ade3eae74346ad0
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 3 20:30:01 2011 +0100

    Fix typo in example

diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst
index 9a65466..7e5d4aa 100644
--- a/doc/sphinx/tutorial/vcl.rst
+++ b/doc/sphinx/tutorial/vcl.rst
@@ -152,7 +152,7 @@ Here we override the TTL of a object comming from the backend if it
 matches certain criteria:::
 
   sub vcl_fetch {
-     if (beresp.url ~ "\.(png|gif|jpg)$") {
+     if (req.url ~ "\.(png|gif|jpg)$") {
        unset beresp.http.set-cookie;
        set beresp.ttl = 3600;
     }



From tfheen at varnish-cache.org  Thu Mar  3 19:30:20 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 03 Mar 2011 20:30:20 +0100
Subject: [master] e4a0af7 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1PvEE8-000633-Js@project.varnish-software.com>

commit e4a0af767763ccd3381a720b634599462f9201d5
Merge: 6587c5d 2b4e631
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 3 20:30:07 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From tfheen at varnish-cache.org  Thu Mar  3 19:31:58 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 03 Mar 2011 20:31:58 +0100
Subject: [2.1] varnish-2.1.5-9-gaed56fa
Message-ID: <E1PvEFi-0007Nq-7w@project.varnish-software.com>

commit aed56fa6b5e42a4b0606c4b0112a3b82fc1b32d0
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 3 20:30:01 2011 +0100

    Fix typo in example

diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst
index 9a65466..7e5d4aa 100644
--- a/doc/sphinx/tutorial/vcl.rst
+++ b/doc/sphinx/tutorial/vcl.rst
@@ -152,7 +152,7 @@ Here we override the TTL of a object comming from the backend if it
 matches certain criteria:::
 
   sub vcl_fetch {
-     if (beresp.url ~ "\.(png|gif|jpg)$") {
+     if (req.url ~ "\.(png|gif|jpg)$") {
        unset beresp.http.set-cookie;
        set beresp.ttl = 3600;
     }



From phk at varnish-cache.org  Mon Mar  7 10:12:15 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Mar 2011 11:12:15 +0100
Subject: [master] a9dc4a2 Extract the two bits of information we need from the
	unadultered beresp, rather than saving the entire thing.
Message-ID: <E1PwXQF-0008WO-70@project.varnish-software.com>

commit a9dc4a2abb92dfca3273ca20616eed5ad3e78b1a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 7 10:11:08 2011 +0000

    Extract the two bits of information we need from the unadultered beresp,
    rather than saving the entire thing.
    
    This also moves a bit more of the fetch-policy/standards-compliance into
    rfc2616.c

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 7c97012..ca1cdff 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -273,7 +273,6 @@ struct worker {
 	struct ws		ws[1];
 	struct http		*http[3];
 	struct http		*bereq;
-	struct http		*beresp1;
 	struct http		*beresp;
 	struct http		*resp;
 
@@ -294,6 +293,8 @@ struct worker {
 	unsigned		is_gzip;
 	unsigned		do_gunzip;
 	unsigned		is_gunzip;
+	unsigned		do_close;
+	char			*h_content_length;
 
 	/* ESI stuff */
 	struct vep_state	*vep;
@@ -651,7 +652,7 @@ int EXP_NukeOne(const struct sess *sp, struct lru *lru);
 /* cache_fetch.c */
 struct storage *FetchStorage(const struct sess *sp, ssize_t sz);
 int FetchHdr(struct sess *sp);
-int FetchBody(struct sess *sp, const struct http *hp);
+int FetchBody(struct sess *sp);
 int FetchReqBody(struct sess *sp);
 void Fetch_Init(void);
 
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index c991400..30ad63a 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -229,7 +229,6 @@ cnt_deliver(struct sess *sp)
 		sp->director = NULL;
 		sp->wrk->bereq = NULL;
 		sp->wrk->beresp = NULL;
-		sp->wrk->beresp1 = NULL;
 		sp->wrk->resp = NULL;
 		sp->step = STP_RECV;
 		return (0);
@@ -475,6 +474,8 @@ cnt_fetch(struct sess *sp)
 
 	AN(sp->director);
 	AZ(sp->vbc);
+	AZ(sp->wrk->h_content_length);
+	AZ(sp->wrk->do_close);
 
 	/* sp->wrk->http[0] is (still) bereq */
 	sp->wrk->beresp = sp->wrk->http[1];
@@ -500,11 +501,11 @@ cnt_fetch(struct sess *sp)
 	http_CollectHdr(sp->wrk->beresp, H_Vary);
 
 	/*
-	 * Save a copy before it might get mangled in VCL.  When it comes to
-	 * dealing with the body, we want to see the unadultered headers.
-	 */
-	sp->wrk->beresp1 = sp->wrk->http[2];
-	*sp->wrk->beresp1 = *sp->wrk->beresp;
+	 * Figure out how the fetch is supposed to happen, before the
+	 * headers are adultered by VCL
+	 * Also sets other sp->wrk variables 
+	 */ 
+	sp->wrk->body_status = RFC2616_Body(sp);
 
 	if (i) {
 		if (sp->objcore != NULL) {
@@ -513,9 +514,10 @@ cnt_fetch(struct sess *sp)
 			sp->objcore = NULL;
 		}
 		AZ(sp->obj);
+		sp->wrk->do_close = 0;
+		sp->wrk->h_content_length = NULL;
 		sp->wrk->bereq = NULL;
 		sp->wrk->beresp = NULL;
-		sp->wrk->beresp1 = NULL;
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
 		return (0);
@@ -555,7 +557,6 @@ cnt_fetch(struct sess *sp)
 
 	sp->wrk->do_esi = 0;
 
-	sp->wrk->body_status = RFC2616_Body(sp);
 
 	AZ(sp->wrk->storage_hint);
 
@@ -698,11 +699,13 @@ cnt_fetch(struct sess *sp)
 		sp->obj->last_modified = sp->wrk->entered;
 
 	/* Use unmodified headers*/
-	i = FetchBody(sp, sp->wrk->beresp1);
+	i = FetchBody(sp);
+
+	sp->wrk->do_close = 0;
+	sp->wrk->h_content_length = NULL;
 
 	sp->wrk->bereq = NULL;
 	sp->wrk->beresp = NULL;
-	sp->wrk->beresp1 = NULL;
 	sp->wrk->vfp = NULL;
 	AZ(sp->wrk->wfd);
 	AZ(sp->vbc);
diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
index 911d576..cc17733 100644
--- a/bin/varnishd/cache_fetch.c
+++ b/bin/varnishd/cache_fetch.c
@@ -482,9 +482,8 @@ FetchHdr(struct sess *sp)
 /*--------------------------------------------------------------------*/
 
 int
-FetchBody(struct sess *sp, const struct http *hp)
+FetchBody(struct sess *sp)
 {
-	char *b;
 	int cls;
 	struct storage *st;
 	int mklen;
@@ -517,8 +516,8 @@ FetchBody(struct sess *sp, const struct http *hp)
 		mklen = 1;
 		break;
 	case BS_LENGTH:
-		AN(http_GetHdr(hp, H_Content_Length, &b));
-		cls = fetch_straight(sp, sp->wrk->htc, b);
+		cls = fetch_straight(sp, sp->wrk->htc,
+		    sp->wrk->h_content_length);
 		mklen = 1;
 		break;
 	case BS_CHUNKED:
@@ -555,11 +554,7 @@ FetchBody(struct sess *sp, const struct http *hp)
 		return (__LINE__);
 	}
 
-	if (cls == 0 && http_HdrIs(hp, H_Connection, "close"))
-		cls = 1;
-
-	if (cls == 0 && hp->protover < 1.1 &&
-	    !http_HdrIs(hp, H_Connection, "keep-alive"))
+	if (cls == 0 && sp->wrk->do_close)
 		cls = 1;
 
 	if (cls < 0) {
@@ -593,9 +588,6 @@ FetchBody(struct sess *sp, const struct http *hp)
 		    "Content-Length: %u", sp->obj->len);
 	}
 
-	if (http_HdrIs(hp, H_Connection, "close"))
-		cls = 1;
-
 	if (cls)
 		VDI_CloseFd(sp);
 	else
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 49cfc9f..361743f 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -179,13 +179,11 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 		w->lastused = NAN;
 		WS_Reset(w->ws, NULL);
 		w->bereq = NULL;
-		w->beresp1 = NULL;
 		w->beresp = NULL;
 		w->resp = NULL;
 		w->storage_hint = NULL;
 		w->wrq->func(w, w->wrq->priv);
 		AZ(w->bereq);
-		AZ(w->beresp1);
 		AZ(w->beresp);
 		AZ(w->resp);
 		WS_Assert(w->ws);
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index 1c07751..0295be5 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -160,7 +160,7 @@ RFC2616_Ttl(const struct sess *sp)
 }
 
 /*--------------------------------------------------------------------
- * Body existence and fetch method
+ * Body existence, fetch method and close policy.
  */
 
 enum body_status
@@ -169,7 +169,14 @@ RFC2616_Body(const struct sess *sp)
 	struct http *hp;
 	char *b;
 
-	hp = sp->wrk->beresp1;
+	hp = sp->wrk->beresp;
+
+	if (hp->protover < 1.1 && !http_HdrIs(hp, H_Connection, "keep-alive"))
+		sp->wrk->do_close = 1;
+	else if (http_HdrIs(hp, H_Connection, "close"))
+		sp->wrk->do_close = 1;
+	else
+		sp->wrk->do_close = 0;
 
 	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
 		/*
@@ -218,7 +225,7 @@ RFC2616_Body(const struct sess *sp)
 		return (BS_ERROR);
 	}
 
-	if (http_GetHdr(hp, H_Content_Length, &b)) {
+	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
 		sp->wrk->stats.fetch_length++;
 		return (BS_LENGTH);
 	}



From phk at varnish-cache.org  Mon Mar  7 10:34:15 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Mar 2011 11:34:15 +0100
Subject: [master] c799158 Avoid the indirection of the bereq,
	beresp and resp http structures, it no longer saves us memory.
Message-ID: <E1PwXlX-0002x1-1d@project.varnish-software.com>

commit c7991588e3eab660f295fd30a54a5bf17166c2df
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 7 10:33:49 2011 +0000

    Avoid the indirection of the bereq,beresp and resp http structures,
    it no longer saves us memory.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index ca1cdff..66fa848 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -271,7 +271,6 @@ struct worker {
 
 	struct http_conn	htc[1];
 	struct ws		ws[1];
-	struct http		*http[3];
 	struct http		*bereq;
 	struct http		*beresp;
 	struct http		*resp;
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 30ad63a..981e4f2 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -213,7 +213,6 @@ cnt_deliver(struct sess *sp)
 			sp->obj->last_lru = sp->t_resp;
 		sp->obj->last_use = sp->t_resp;	/* XXX: locking ? */
 	}
-	sp->wrk->resp = sp->wrk->http[2];
 	http_Setup(sp->wrk->resp, sp->wrk->ws);
 	RES_BuildHttp(sp);
 	VCL_deliver_method(sp);
@@ -227,9 +226,9 @@ cnt_deliver(struct sess *sp)
 		AZ(sp->obj);
 		sp->restarts++;
 		sp->director = NULL;
-		sp->wrk->bereq = NULL;
-		sp->wrk->beresp = NULL;
-		sp->wrk->resp = NULL;
+		http_Setup(sp->wrk->bereq, NULL);
+		http_Setup(sp->wrk->beresp, NULL);
+		http_Setup(sp->wrk->resp, NULL);
 		sp->step = STP_RECV;
 		return (0);
 	default:
@@ -243,7 +242,7 @@ cnt_deliver(struct sess *sp)
 
 	AZ(sp->wrk->wfd);
 	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-	sp->wrk->resp = NULL;
+	http_Setup(sp->wrk->resp, NULL);
 	sp->step = STP_DONE;
 	return (0);
 }
@@ -425,7 +424,7 @@ cnt_error(struct sess *sp)
 	assert(sp->handling == VCL_RET_DELIVER);
 	sp->err_code = 0;
 	sp->err_reason = NULL;
-	sp->wrk->bereq = NULL;
+	http_Setup(sp->wrk->bereq, NULL);
 	sp->step = STP_DELIVER;
 	return (0);
 }
@@ -477,8 +476,6 @@ cnt_fetch(struct sess *sp)
 	AZ(sp->wrk->h_content_length);
 	AZ(sp->wrk->do_close);
 
-	/* sp->wrk->http[0] is (still) bereq */
-	sp->wrk->beresp = sp->wrk->http[1];
 	http_Setup(sp->wrk->beresp, sp->wrk->ws);
 
 	i = FetchHdr(sp);
@@ -516,8 +513,8 @@ cnt_fetch(struct sess *sp)
 		AZ(sp->obj);
 		sp->wrk->do_close = 0;
 		sp->wrk->h_content_length = NULL;
-		sp->wrk->bereq = NULL;
-		sp->wrk->beresp = NULL;
+		http_Setup(sp->wrk->bereq, NULL);
+		http_Setup(sp->wrk->beresp, NULL);
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
 		return (0);
@@ -704,8 +701,8 @@ cnt_fetch(struct sess *sp)
 	sp->wrk->do_close = 0;
 	sp->wrk->h_content_length = NULL;
 
-	sp->wrk->bereq = NULL;
-	sp->wrk->beresp = NULL;
+	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(sp->wrk->beresp, NULL);
 	sp->wrk->vfp = NULL;
 	AZ(sp->wrk->wfd);
 	AZ(sp->vbc);
@@ -812,7 +809,8 @@ cnt_hit(struct sess *sp)
 	if (sp->handling == VCL_RET_DELIVER) {
 		/* Dispose of any body part of the request */
 		(void)FetchReqBody(sp);
-		sp->wrk->bereq = NULL;
+		AZ(sp->wrk->bereq->ws);
+		AZ(sp->wrk->beresp->ws);
 		sp->step = STP_DELIVER;
 		return (0);
 	}
@@ -953,7 +951,6 @@ cnt_miss(struct sess *sp)
 	AZ(sp->obj);
 	AN(sp->objcore);
 	WS_Reset(sp->wrk->ws, NULL);
-	sp->wrk->bereq = sp->wrk->http[0];
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_FETCH);
 	http_ForceGet(sp->wrk->bereq);
@@ -1035,7 +1032,6 @@ cnt_pass(struct sess *sp)
 	AZ(sp->obj);
 
 	WS_Reset(sp->wrk->ws, NULL);
-	sp->wrk->bereq = sp->wrk->http[0];
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_PASS);
 
@@ -1088,7 +1084,6 @@ cnt_pipe(struct sess *sp)
 
 	sp->acct_tmp.pipe++;
 	WS_Reset(sp->wrk->ws, NULL);
-	sp->wrk->bereq = sp->wrk->http[0];
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_PIPE);
 
@@ -1100,7 +1095,7 @@ cnt_pipe(struct sess *sp)
 
 	PipeSession(sp);
 	AZ(sp->wrk->wfd);
-	sp->wrk->bereq = NULL;
+	http_Setup(sp->wrk->bereq, NULL);
 	sp->step = STP_DONE;
 	return (0);
 }
diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c
index 55c04d5..3405d92 100644
--- a/bin/varnishd/cache_panic.c
+++ b/bin/varnishd/cache_panic.c
@@ -198,10 +198,12 @@ pan_wrk(const struct worker *wrk)
 
 	vsb_printf(vsp, "  worker = %p {\n", wrk);
 	pan_ws(wrk->ws, 4);
-	if (wrk->bereq != NULL)
+	if (wrk->bereq->ws != NULL)
 		pan_http("bereq", wrk->bereq, 4);
-	if (wrk->beresp != NULL)
+	if (wrk->beresp->ws != NULL)
 		pan_http("beresp", wrk->beresp, 4);
+	if (wrk->resp->ws != NULL)
+		pan_http("resp", wrk->resp, 4);
 	vsb_printf(vsp, "    },\n");
 }
 
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 361743f..0c80772 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -140,9 +140,9 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 	w->wlb = w->wlp = wlog;
 	w->wle = wlog + (sizeof wlog) / 4;
 	w->sha256ctx = &sha256;
-	w->http[0] = HTTP_create(http0, nhttp);
-	w->http[1] = HTTP_create(http1, nhttp);
-	w->http[2] = HTTP_create(http2, nhttp);
+	w->bereq = HTTP_create(http0, nhttp);
+	w->beresp = HTTP_create(http1, nhttp);
+	w->resp = HTTP_create(http2, nhttp);
 	w->iov = iov;
 	w->siov = siov;
 	AZ(pthread_cond_init(&w->cond, NULL));
@@ -155,6 +155,9 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 	qp->nthr++;
 	stats_clean = 1;
 	while (1) {
+		CHECK_OBJ_NOTNULL(w->bereq, HTTP_MAGIC);
+		CHECK_OBJ_NOTNULL(w->beresp, HTTP_MAGIC);
+		CHECK_OBJ_NOTNULL(w->resp, HTTP_MAGIC);
 		CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 
 		/* Process queued requests, if any */
@@ -178,15 +181,14 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 		AN(w->wrq->func);
 		w->lastused = NAN;
 		WS_Reset(w->ws, NULL);
-		w->bereq = NULL;
-		w->beresp = NULL;
-		w->resp = NULL;
 		w->storage_hint = NULL;
+
 		w->wrq->func(w, w->wrq->priv);
-		AZ(w->bereq);
-		AZ(w->beresp);
-		AZ(w->resp);
+
 		WS_Assert(w->ws);
+		AZ(w->bereq->ws);
+		AZ(w->beresp->ws);
+		AZ(w->resp->ws);
 		AZ(w->wfd);
 		AZ(w->storage_hint);
 		assert(w->wlp == w->wlb);



From phk at varnish-cache.org  Mon Mar  7 10:47:18 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Mar 2011 11:47:18 +0100
Subject: [master] 8559412 Move the body_status enum to cache.h
Message-ID: <E1PwXyA-0004ua-Vz@project.varnish-software.com>

commit 85594125d8e6e13a38e39377bf77b0cc1133020f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 7 10:47:05 2011 +0000

    Move the body_status enum to cache.h

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 66fa848..e195151 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -65,7 +65,15 @@
 
 #include "vsc.h"
 #include "vsl.h"
-#include "vtypes.h"
+
+enum body_status {
+	BS_NONE,
+	BS_ZERO,
+	BS_ERROR,
+	BS_CHUNKED,
+	BS_LENGTH,
+	BS_EOF
+};
 
 /*
  * NB: HDR_STATUS is only used in cache_http.c, everybody else uses the
diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
index cc17733..594aa4e 100644
--- a/bin/varnishd/cache_fetch.c
+++ b/bin/varnishd/cache_fetch.c
@@ -499,11 +499,6 @@ FetchBody(struct sess *sp)
 	AN(sp->director);
 	AssertObjPassOrBusy(sp->obj);
 
-	/*
-	 * Determine if we have a body or not
-	 * XXX: Missing:  RFC2616 sec. 4.4 in re 1xx, 204 & 304 responses
-	 */
-
 	AZ(sp->wrk->vgz_rx);
 	AZ(VTAILQ_FIRST(&sp->obj->store));
 	switch (sp->wrk->body_status) {
@@ -542,9 +537,9 @@ FetchBody(struct sess *sp)
 	/*
 	 * It is OK for ->end to just leave the last storage segment
 	 * sitting on sp->wrk->storage, we will always call vfp_nop_end()
-	 * to get it trimmed and added to the object.
+	 * to get it trimmed or thrown out if empty.
 	 */
-	XXXAZ(vfp_nop_end(sp));
+	AZ(vfp_nop_end(sp));
 
 	WSL(sp->wrk, SLT_Fetch_Body, sp->vbc->fd, "%u %d %u",
 	    sp->wrk->body_status, cls, mklen);
@@ -554,9 +549,6 @@ FetchBody(struct sess *sp)
 		return (__LINE__);
 	}
 
-	if (cls == 0 && sp->wrk->do_close)
-		cls = 1;
-
 	if (cls < 0) {
 		sp->wrk->stats.fetch_failed++;
 		/* XXX: Wouldn't this store automatically be released ? */
@@ -570,6 +562,9 @@ FetchBody(struct sess *sp)
 		return (__LINE__);
 	}
 
+	if (cls == 0 && sp->wrk->do_close)
+		cls = 1;
+
 	WSL(sp->wrk, SLT_Length, sp->vbc->fd, "%u", sp->obj->len);
 
 	{
diff --git a/include/Makefile.am b/include/Makefile.am
index 94eabc4..a46ae72 100644
--- a/include/Makefile.am
+++ b/include/Makefile.am
@@ -50,8 +50,7 @@ nobase_noinst_HEADERS = \
 	vrt.h \
 	vrt_obj.h \
 	vrt_stv_var.h \
-	vss.h \
-	vtypes.h
+	vss.h
 
 vrt_stv_var.h vcl_returns.h vcl.h vrt_obj.h: $(top_srcdir)/lib/libvcl/generate.py $(top_srcdir)/include/vrt.h
 	@PYTHON@ $(top_srcdir)/lib/libvcl/generate.py $(top_srcdir) $(top_builddir)
diff --git a/include/vtypes.h b/include/vtypes.h
deleted file mode 100644
index a8e1599..0000000
--- a/include/vtypes.h
+++ /dev/null
@@ -1,41 +0,0 @@
-/*-
- * Copyright (c) 2010 Linpro AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * $Id$
- *
- * Very special types, widely used, in their own #include to keep
- * #include-infection minimal.
- */
-
-enum body_status {
-	BS_NONE,
-	BS_ZERO,
-	BS_ERROR,
-	BS_CHUNKED,
-	BS_LENGTH,
-	BS_EOF
-};



From phk at varnish-cache.org  Mon Mar  7 11:09:14 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Mar 2011 12:09:14 +0100
Subject: [master] 25c5f2e Move more policy stuff to rfc2616.c
Message-ID: <E1PwYJO-0007yF-FV@project.varnish-software.com>

commit 25c5f2ed3229e41e99eadff57374c3a93b41a356
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 7 11:09:03 2011 +0000

    Move more policy stuff to rfc2616.c

diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 981e4f2..9146392 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -530,24 +530,6 @@ cnt_fetch(struct sess *sp)
 	EXP_Clr(&sp->wrk->exp);
 	sp->wrk->exp.ttl = RFC2616_Ttl(sp);
 
-	/*
-	 * Initial cacheability determination per [RFC2616, 13.4]
-	 * We do not support ranges yet, so 206 is out.
-	 */
-	switch (sp->err_code) {
-	case 200: /* OK */
-	case 203: /* Non-Authoritative Information */
-	case 300: /* Multiple Choices */
-	case 301: /* Moved Permanently */
-	case 302: /* Moved Temporarily */
-	case 410: /* Gone */
-	case 404: /* Not Found */
-		break;
-	default:
-		sp->wrk->exp.ttl = -1.;
-		break;
-	}
-
 	/* pass from vclrecv{} has negative TTL */
 	if (sp->objcore == NULL)
 		sp->wrk->exp.ttl = -1.;
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index 0295be5..2c6c851 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -87,8 +87,22 @@ RFC2616_Ttl(const struct sess *sp)
 	h_expires = 0;
 	h_date = 0;
 
-	do {	/* Allows us to break when we want out */
+	/*
+	 * Initial cacheability determination per [RFC2616, 13.4]
+	 * We do not support ranges yet, so 206 is out.
+	 */
 
+	switch (sp->err_code) {
+	default:
+		sp->wrk->exp.ttl = -1.;
+		break;
+	case 200: /* OK */
+	case 203: /* Non-Authoritative Information */
+	case 300: /* Multiple Choices */
+	case 301: /* Moved Permanently */
+	case 302: /* Moved Temporarily */
+	case 410: /* Gone */
+	case 404: /* Not Found */
 		/*
 		 * First find any relative specification from the backend
 		 * These take precedence according to RFC2616, 13.2.4
@@ -150,7 +164,7 @@ RFC2616_Ttl(const struct sess *sp)
 			ttl = (int)(h_expires - h_date);
 		}
 
-	} while (0);
+	} 
 
 	/* calculated TTL, Our time, Date, Expires, max-age, age */
 	WSP(sp, SLT_TTL, "%u RFC %g %g %g %g %u %u", sp->xid,



From tfheen at varnish-cache.org  Tue Mar  8 09:35:56 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 08 Mar 2011 10:35:56 +0100
Subject: [master] ab05eda Test getsockopt, not just setsockopt in configure.ac
Message-ID: <E1PwtKe-0003Vk-FN@project.varnish-software.com>

commit ab05edab53ce7d40185db8c7d9153cd92e9f50ef
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 8 10:34:12 2011 +0100

    Test getsockopt, not just setsockopt in configure.ac
    
    Fixes: #853

diff --git a/configure.ac b/configure.ac
index d7f5b94..19b194c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -351,7 +351,13 @@ AC_CACHE_CHECK([whether SO_RCVTIMEO works],
     ]],[[
 int sd = socket(AF_INET, SOCK_STREAM, 0);
 struct timeval tv = { 1, 0 };
-return !!setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv);
+if (setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv) == 0) {
+    socklen_t l = sizeof tv;
+    if (getsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv_get, &l) == 0) {
+        return (l == sizeof tv);
+    }
+}
+return 1;
     ]])],
     [ac_cv_so_rcvtimeo_works=yes],
     [ac_cv_so_rcvtimeo_works=no])
@@ -370,7 +376,13 @@ AC_CACHE_CHECK([whether SO_SNDTIMEO works],
     ]],[[
 int sd = socket(AF_INET, SOCK_STREAM, 0);
 struct timeval tv = { 1, 0 };
-return !!setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv);
+if (setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv) == 0) {
+    socklen_t l = sizeof tv;
+    if (getsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv_get, &l) == 0) {
+        return (l == sizeof tv);
+    }
+}
+return 1;
     ]])],
     [ac_cv_so_sndtimeo_works=yes],
     [ac_cv_so_sndtimeo_works=no])



From tfheen at varnish-cache.org  Tue Mar  8 09:53:04 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 08 Mar 2011 10:53:04 +0100
Subject: [master] ab70df5 Fix typo
Message-ID: <E1PwtbE-0005bN-SZ@project.varnish-software.com>

commit ab70df53d69d3f89653321f66218e6c8283e7212
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 8 10:51:29 2011 +0100

    Fix typo
    
    I forgot to rename the tv_get variable to tv in configure.ac
    
    Fixes #853
    
    (again)

diff --git a/configure.ac b/configure.ac
index 19b194c..2ac0c94 100644
--- a/configure.ac
+++ b/configure.ac
@@ -353,8 +353,8 @@ int sd = socket(AF_INET, SOCK_STREAM, 0);
 struct timeval tv = { 1, 0 };
 if (setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv) == 0) {
     socklen_t l = sizeof tv;
-    if (getsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv_get, &l) == 0) {
-        return (l == sizeof tv);
+    if (getsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l) == 0) {
+        return 0;
     }
 }
 return 1;
@@ -378,8 +378,8 @@ int sd = socket(AF_INET, SOCK_STREAM, 0);
 struct timeval tv = { 1, 0 };
 if (setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv) == 0) {
     socklen_t l = sizeof tv;
-    if (getsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv_get, &l) == 0) {
-        return (l == sizeof tv);
+    if (getsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l) == 0) {
+        return 0;
     }
 }
 return 1;



From tfheen at varnish-cache.org  Tue Mar  8 10:04:30 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 08 Mar 2011 11:04:30 +0100
Subject: [master] cd458db Tighten SO{RCV,SND}TIMEO check
Message-ID: <E1PwtmI-00073R-2n@project.varnish-software.com>

commit cd458db6d792022164b5221f6313ae4079982b58
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 8 11:03:50 2011 +0100

    Tighten SO{RCV,SND}TIMEO check
    
    We can check the size of the return value too.

diff --git a/configure.ac b/configure.ac
index 2ac0c94..ab2bd65 100644
--- a/configure.ac
+++ b/configure.ac
@@ -354,7 +354,7 @@ struct timeval tv = { 1, 0 };
 if (setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv) == 0) {
     socklen_t l = sizeof tv;
     if (getsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l) == 0) {
-        return 0;
+        return (l != sizeof tv);
     }
 }
 return 1;
@@ -379,7 +379,7 @@ struct timeval tv = { 1, 0 };
 if (setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv) == 0) {
     socklen_t l = sizeof tv;
     if (getsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l) == 0) {
-        return 0;
+        return (l != sizeof tv);
     }
 }
 return 1;



From phk at varnish-cache.org  Tue Mar  8 10:14:41 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Mar 2011 11:14:41 +0100
Subject: [master] eda2f19 Add {req,obj,beresp}.keep and req.ttl variables.
Message-ID: <E1Pwtw9-0007rY-1m@project.varnish-software.com>

commit eda2f19150b62688b8d664798f7e35a8eb7a7968
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 8 10:13:35 2011 +0000

    Add {req,obj,beresp}.keep and req.ttl variables.
    
    Keep is for future use by conditional backend requests.
    
    req.ttl provides a way to force a fetch of a not yet stale object,
    see how in testcase c00042.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index e195151..ea82edd 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -243,6 +243,7 @@ extern struct vfp vfp_esi;
 struct exp {
 	double			ttl;
 	double			grace;
+	double			keep;
 };
 
 /*--------------------------------------------------------------------*/
@@ -642,13 +643,16 @@ extern pthread_t cli_thread;
 
 /* cache_expiry.c */
 void EXP_Clr(struct exp *e);
-double EXP_Get_grace(const struct exp *e);
 double EXP_Get_ttl(const struct exp *e);
-void EXP_Set_grace(struct exp *e, double v);
+double EXP_Get_grace(const struct exp *e);
+double EXP_Get_keep(const struct exp *e);
 void EXP_Set_ttl(struct exp *e, double v);
+void EXP_Set_grace(struct exp *e, double v);
+void EXP_Set_keep(struct exp *e, double v);
 
-double EXP_Grace(const struct sess *, const struct object*);
 double EXP_Ttl(const struct sess *, const struct object*);
+double EXP_Grace(const struct sess *, const struct object*);
+double EXP_Keep(const struct sess *, const struct object*);
 void EXP_Insert(struct object *o);
 void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
 void EXP_Init(void);
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 6c2ccb5..c95c43f 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -68,51 +68,51 @@ void
 EXP_Clr(struct exp *e)
 {
 
-	e->grace = -1;
 	e->ttl = -1;
+	e->grace = -1;
+	e->keep = -1;
 }
 
-void
-EXP_Set_grace(struct exp *e, double v)
-{
-
-	if (v > 0.)
-		e->grace = v;
-	else
-		e->grace = -1.;
-}
-
-double
-EXP_Get_grace(const struct exp *e)
-{
-
-	return (e->grace > 0. ? e->grace : -1.);
-}
-
-void
-EXP_Set_ttl(struct exp *e, double v)
-{
+#define EXP_ACCESS(fld, extra)					\
+	double							\
+	EXP_Get_##fld(const struct exp *e)			\
+	{							\
+		return (e->fld > 0. ? e->fld : -1.);		\
+	}							\
+								\
+	void							\
+	EXP_Set_##fld(struct exp *e, double v)			\
+	{							\
+		if (v > 0.)					\
+			e->fld = v;				\
+		else {						\
+			e->fld = -1.;				\
+			extra;					\
+		}						\
+	}							\
+
+EXP_ACCESS(ttl, (e->grace = e->keep = -1.))
+EXP_ACCESS(grace,)
+EXP_ACCESS(keep,)
 
-	if (v > 0.)
-		e->ttl = v;
-	else {
-		e->ttl = -1.;
-		e->grace = -1.;
-	}
-}
+/*--------------------------------------------------------------------
+ * Calculate when an object is out of ttl or grace, possibly constrained
+ * by per-session limits.
+ */
 
 double
-EXP_Get_ttl(const struct exp *e)
+EXP_Keep(const struct sess *sp, const struct object *o)
 {
+	double r;
 
-	return (e->ttl > 0. ? e->ttl : -1.);
+	r = (double)params->default_keep;
+	if (o->exp.keep > 0.)
+		r = o->exp.keep;
+	if (sp != NULL && sp->exp.keep > 0. && sp->exp.keep < r)
+		r = sp->exp.keep;
+	return (EXP_Grace(sp, o) + r);
 }
 
-/*--------------------------------------------------------------------
- * Calculate when an object is out of ttl or grace, possibly constrained
- * by per-session limits.
- */
-
 double
 EXP_Grace(const struct sess *sp, const struct object *o)
 {
@@ -152,7 +152,7 @@ update_object_when(const struct object *o)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	Lck_AssertHeld(&exp_mtx);
 
-	when = EXP_Grace(NULL, o);
+	when = EXP_Keep(NULL, o);
 	assert(!isnan(when));
 	if (when == oc->timer_when)
 		return (0);
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index 779af7b..8faf2c4 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -383,11 +383,15 @@ VRT_r_##which##_##fld(struct sess *sp)				\
 	return(EXP_Get_##fld(&exp));				\
 }
 
+VRT_DO_EXP(req, sp->exp, ttl, )
 VRT_DO_EXP(req, sp->exp, grace, )
+VRT_DO_EXP(req, sp->exp, keep, )
 VRT_DO_EXP(obj, sp->obj->exp, grace, EXP_Rearm(sp->obj))
 VRT_DO_EXP(obj, sp->obj->exp, ttl, EXP_Rearm(sp->obj))
+VRT_DO_EXP(obj, sp->obj->exp, keep, EXP_Rearm(sp->obj))
 VRT_DO_EXP(beresp, sp->wrk->exp, grace, )
 VRT_DO_EXP(beresp, sp->wrk->exp, ttl, )
+VRT_DO_EXP(beresp, sp->wrk->exp, keep, )
 
 /*--------------------------------------------------------------------
  * req.xid
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index 06b6f5d..c1a251f 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -73,6 +73,12 @@ struct params {
 	/* TTL used for lack of anything better */
 	double			default_ttl;
 
+	/* Default grace period */
+	double			default_grace;
+
+	/* Default keep period */
+	double			default_keep;
+
 	/* Maximum concurrent sessions */
 	unsigned		max_sess;
 
@@ -162,9 +168,6 @@ struct params {
 	/* Control diagnostic code */
 	unsigned		diag_bitmap;
 
-	/* Default grace period */
-	double			default_grace;
-
 	/* Log hash string to shm */
 	unsigned		log_hash;
 
diff --git a/bin/varnishd/mgt_param.c b/bin/varnishd/mgt_param.c
index 1f7eb50..ed70fc7 100644
--- a/bin/varnishd/mgt_param.c
+++ b/bin/varnishd/mgt_param.c
@@ -583,6 +583,15 @@ static const struct parspec input_parspec[] = {
 		"made until they are fetched from the backend again.\n",
 		DELAYED_EFFECT,
 		"10", "seconds" },
+	{ "default_keep", tweak_timeout_double, &master.default_keep,
+		0, UINT_MAX,
+		"Default keep period.  We will keep a useless object "
+		"around this long, making it available for conditional "
+		"backend fetches.  "
+		"That means that the object will be removed from the "
+		"cache at the end of ttl+grace+keep.",
+		DELAYED_EFFECT,
+		"0", "seconds" },
 	{ "sess_timeout", tweak_timeout, &master.sess_timeout, 0, 0,
 		"Idle timeout for persistent sessions. "
 		"If a HTTP request has not been received in this many "
diff --git a/bin/varnishtest/tests/c00042.vtc b/bin/varnishtest/tests/c00042.vtc
new file mode 100644
index 0000000..de94e38
--- /dev/null
+++ b/bin/varnishtest/tests/c00042.vtc
@@ -0,0 +1,47 @@
+# $Id$
+
+test "using req.ttl to force fetch"
+
+server s1 {
+	rxreq
+	expect req.url == "/1"
+	txresp -bodylen 1
+	rxreq
+	expect req.url == "/1"
+	txresp -bodylen 2
+
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_recv {
+		if (req.http.short) {
+			set req.ttl = 1s;
+			set req.grace = 0s;
+		}
+	}
+} -start
+
+client c1 {
+	txreq -url "/1" 
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 1
+
+	txreq -url "/1" 
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 1
+
+	delay 2
+
+	txreq -url "/1" -hdr "short: yes"
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 2
+
+	txreq -url "/1" 
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 2
+
+} -run
diff --git a/lib/libvcl/generate.py b/lib/libvcl/generate.py
index 72edc9b..969fbc8 100755
--- a/lib/libvcl/generate.py
+++ b/lib/libvcl/generate.py
@@ -178,12 +178,24 @@ sp_variables = (
 		( ),
 		'const struct sess *'
 	),
+	('req.ttl',
+		'DURATION',
+		( 'all',),
+		( 'all',),
+		'struct sess *'
+	),
 	('req.grace',
 		'DURATION',
 		( 'all',),
 		( 'all',),
 		'struct sess *'
 	),
+	('req.keep',
+		'DURATION',
+		( 'all',),
+		( 'all',),
+		'struct sess *'
+	),
 	('req.xid',
 		'STRING',
 		( 'all',),
@@ -322,6 +334,12 @@ sp_variables = (
 		( 'fetch',),
 		'struct sess *'
 	),
+	('beresp.keep',
+		'DURATION',
+		( 'fetch',),
+		( 'fetch',),
+		'struct sess *'
+	),
 	('beresp.backend.name',
 		'STRING',
 		( 'fetch',),
@@ -388,6 +406,12 @@ sp_variables = (
 		( 'hit', 'error',),
 		'struct sess *'
 	),
+	('obj.keep',
+		'DURATION',
+		( 'hit', 'error',),
+		( 'hit', 'error',),
+		'struct sess *'
+	),
 	('obj.lastuse',
 		'DURATION',
 		( 'hit', 'deliver', 'error',),



From kristian at varnish-cache.org  Tue Mar  8 11:56:50 2011
From: kristian at varnish-cache.org (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=)
Date: Tue, 08 Mar 2011 12:56:50 +0100
Subject: [master] cb86f7e doc: Change 3600 to 1h
Message-ID: <E1PwvX0-0001ga-JZ@project.varnish-software.com>

commit cb86f7eff6a85d61b7efb8d5767345004d2fc582
Author: Kristian Lyngstol <kristian at bohemians.org>
Date:   Tue Mar 8 12:56:20 2011 +0100

    doc: Change 3600 to 1h
    
    Thanks to askalski.
    
    closes #867

diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst
index 7e5d4aa..ccf43d2 100644
--- a/doc/sphinx/tutorial/vcl.rst
+++ b/doc/sphinx/tutorial/vcl.rst
@@ -154,7 +154,7 @@ matches certain criteria:::
   sub vcl_fetch {
      if (req.url ~ "\.(png|gif|jpg)$") {
        unset beresp.http.set-cookie;
-       set beresp.ttl = 3600;
+       set beresp.ttl = 1h;
     }
   }
 



From tfheen at varnish-cache.org  Tue Mar  8 12:10:21 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 08 Mar 2011 13:10:21 +0100
Subject: [master] 71a99b6 Add TAGS (etags) file to ignore list
Message-ID: <E1Pwvk5-0004Vs-0M@project.varnish-software.com>

commit 71a99b67f57c296b8b0f6c434f78b33cb4e2b77e
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 8 13:04:44 2011 +0100

    Add TAGS (etags) file to ignore list

diff --git a/.gitignore b/.gitignore
index 9136788..b6c617e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,6 +33,7 @@ Makefile.in
 /missing
 /stamp-h1
 /varnishapi.pc
+TAGS
 
 # Default vcl made from bin/varnishd/default.vcl
 /bin/varnishd/default_vcl.h



From tfheen at varnish-cache.org  Tue Mar  8 12:10:21 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 08 Mar 2011 13:10:21 +0100
Subject: [master] 5f8041a Reduce http_resp_size to 8k on 32 bit
Message-ID: <E1Pwvk5-0004W1-Sh@project.varnish-software.com>

commit 5f8041ac0745ba30e9190a69ec7b5d8ac2d4c1ec
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 8 13:06:01 2011 +0100

    Reduce http_resp_size to 8k on 32 bit
    
    http_resp_size is allocated on the thread_pool_workspace.  This means
    we will hit assert errors if http_resp_size is larger than the
    thread_pool_workspace.  8k should be enough in most cases, from a
    light sample of web sites in the wild.

diff --git a/bin/varnishd/varnishd.c b/bin/varnishd/varnishd.c
index 9583cc4..f635322 100644
--- a/bin/varnishd/varnishd.c
+++ b/bin/varnishd/varnishd.c
@@ -421,6 +421,9 @@ main(int argc, char * const *argv)
 		MCF_ParamSet(cli, "thread_pool_workspace", "16384");
 		cli_check(cli);
 
+		MCF_ParamSet(cli, "http_resp_size", "8192");
+		cli_check(cli);
+
 		MCF_ParamSet(cli, "thread_pool_stack", "32bit");
 		cli_check(cli);
 



From perbu at varnish-cache.org  Tue Mar  8 13:23:02 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Tue, 08 Mar 2011 14:23:02 +0100
Subject: [master] 33135df Elaborate on the semantics of grace mode.
Message-ID: <E1PwwsQ-0004bd-OK@project.varnish-software.com>

commit 33135dfd8237202d5c0d8c39e69d5c1c0c6a8192
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Mar 8 13:15:44 2011 +0100

    Elaborate on the semantics of grace mode.
    
    Also added Kristians workaround for dealing with errors

diff --git a/doc/sphinx/tutorial/handling_misbehaving_servers.rst b/doc/sphinx/tutorial/handling_misbehaving_servers.rst
index e0890de..cc8a636 100644
--- a/doc/sphinx/tutorial/handling_misbehaving_servers.rst
+++ b/doc/sphinx/tutorial/handling_misbehaving_servers.rst
@@ -13,13 +13,20 @@ Grace mode
 
 When several clients are requesting the same page Varnish will send
 one request to the backend and place the others on hold while fetching
-one copy from the back end. 
+one copy from the back end. In some products this is called request
+coalescing and Varnish does this automatically.
 
-If you are serving thousands of hits per second this queue can get
-huge. Nobody likes to wait so there is a possibility to serve stale
-content to waiting users. In order to do this we must instruct Varnish
-to keep the objects in cache beyond their TTL. So, to keep all objects
-for 30 minutes beyond their TTL use the following VCL:::
+If you are serving thousands of hits per second the queue of waiting
+requests can get huge. There are to potential problems - one is a
+thundering heard problem - suddenly releasing a thousand threads to
+serve content might send to load sky high. Secondly - nobody likes to
+wait. To deal with this Varnish can we can instruct Varnish to keep
+the objects in cache beyond their TTL and to serve the waiting
+requests somewhat stale content.
+
+So, in order to serve stale content we must first have some content to
+serve. So to make Varnish keep all objects for 30 minutes beyond their
+TTL use the following VCL:::
 
   sub vcl_fetch {
     set beresp.grace = 30m;
@@ -36,14 +43,19 @@ request. Lets us say that we accept serving 15s old object.::
 You might wonder why we should keep the objects in the cache for 30
 minutes if we are unable to serve them? Well, if you have enabled
 :ref:`tutorial-advanced_backend_servers-health` you can check if the
-backend is healthy and serve the content for as longer.::
+backend is sick and if it is we can serve the stale content for a bit
+longer.::
 
    if (! req.backend.healthy) {
       set req.grace = 5m;
    } else {
       set req.grace = 15s;
    }
-  
+
+So, to sum up, grace mode solves two problems:
+ * it serves stale content to avoid request pile-up.
+ * it serves stale content if the backend is not healthy.
+
 Saint mode
 ~~~~~~~~~~
 
@@ -70,6 +82,21 @@ Varnish will serve the content from its stale cache.
 
 This can really be a life saver.
 
+Known limitations on grace- and saint mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If your request fails while it is being fetched you're thrown into
+vcl_error. vcl_error has access to a rather limited set of data so you
+can't enable saint mode or grace mode here. This will be addressed in a
+future release but a work-around is available.
+
+* Declare a backend that is always sick.
+* Set a magic marker in vcl_error
+* Restart the transaction
+* Note the magic marker in vcl_recv and set the backend to the one mentioned
+* Varnish will now serve stale data is any is available
+
+
 God mode
 ~~~~~~~~
 Not implemented yet. :-)



From perbu at varnish-cache.org  Tue Mar  8 13:23:02 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Tue, 08 Mar 2011 14:23:02 +0100
Subject: [master] 9b3a999 Elaborating on troubleshooting.
Message-ID: <E1PwwsQ-0004bm-Uz@project.varnish-software.com>

commit 9b3a999b8cd6e40bc322e3dac087f0e7a6bef67b
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Mar 8 13:48:20 2011 +0100

    Elaborating on troubleshooting.
    
    How to deal with Varnish crashing (syslog, etc) and how to find
    503 errors in Varnishlog.

diff --git a/doc/sphinx/tutorial/troubleshooting.rst b/doc/sphinx/tutorial/troubleshooting.rst
index 5965ac0..aeb2786 100644
--- a/doc/sphinx/tutorial/troubleshooting.rst
+++ b/doc/sphinx/tutorial/troubleshooting.rst
@@ -10,9 +10,9 @@ leave clues of whats going on.
 When Varnish won't start
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-Sometimes Varnish wont start. There is a pletphora of reasons why
+Sometimes Varnish wont start. There is a plethora of reasons why
 Varnish wont start on your machine. We've seen everything from wrong
-permissions on /dev/null to other processses blocking the ports.
+permissions on /dev/null to other processes blocking the ports.
 
 Starting Varnish in debug mode to see what is going on.
 
@@ -53,14 +53,44 @@ on IRC.
 Varnish is crashing
 ~~~~~~~~~~~~~~~~~~~
 
-When varnish goes bust.
+When varnish goes bust the child processes crashes. Usually the mother
+process will manage this by restarting the child process again. Any
+errors will be logged in syslog. It might look like this:::
 
+       Mar  8 13:23:38 smoke varnishd[15670]: Child (15671) not responding to CLI, killing it.
+       Mar  8 13:23:43 smoke varnishd[15670]: last message repeated 2 times
+       Mar  8 13:23:43 smoke varnishd[15670]: Child (15671) died signal=3
+       Mar  8 13:23:43 smoke varnishd[15670]: Child cleanup complete
+       Mar  8 13:23:43 smoke varnishd[15670]: child (15697) Started
+
+Specifically if you see the "Error in munmap" error on Linux you might
+want to increase the amount of maps available. Linux is limited to a
+maximum of 64k maps. Setting vm.max_max_count i sysctl.conf will
+enable you to increase this limit. You can inspect the number of maps
+your program is consuming by counting the lines in /proc/$PID/maps
+
+This is a rather odd thing to document here - but hopefully Google
+will serve you this page if you ever encounter this error. 
 
 Varnish gives me Guru meditation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 First find the relevant log entries in varnishlog. That will probably
-give you a clue.
+give you a clue. Since varnishlog logs so much data it might be hard
+to track the entries down. You can set varnishlog to log all your 503
+errors by issuing the following command:::
+
+   $ varnishlog -c -o TxStatus 503
+
+If the error happened just a short time ago the transaction might still
+be in the shared memory log segment. To get varnishlog to process the
+whole shared memory log just add the -d option:::
+
+   $ varnishlog -d -c -o TxStatus 503
+
+Please see the varnishlog man page for elaborations on further
+filtering capabilities and explanation of the various options.
+
 
 Varnish doesn't cache
 ~~~~~~~~~~~~~~~~~~~~~



From perbu at varnish-cache.org  Tue Mar  8 13:23:03 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Tue, 08 Mar 2011 14:23:03 +0100
Subject: [master] 095f6b2 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1PwwsR-0004by-8K@project.varnish-software.com>

commit 095f6b23ef0ace15cf19234da7d7607307e7ed1b
Merge: 9b3a999 5f8041a
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Mar 8 14:22:59 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Tue Mar  8 13:48:21 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Tue, 08 Mar 2011 14:48:21 +0100
Subject: [master] 13c60b5 Spelling errors (thanks to Camiel Dobbelaar)
Message-ID: <E1PwxGv-0000cn-8T@project.varnish-software.com>

commit 13c60b522a6c7abe9cc1052c2a244f25f2ba5227
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Mar 8 14:47:57 2011 +0100

    Spelling errors (thanks to Camiel Dobbelaar)

diff --git a/doc/sphinx/tutorial/handling_misbehaving_servers.rst b/doc/sphinx/tutorial/handling_misbehaving_servers.rst
index cc8a636..c258f46 100644
--- a/doc/sphinx/tutorial/handling_misbehaving_servers.rst
+++ b/doc/sphinx/tutorial/handling_misbehaving_servers.rst
@@ -17,10 +17,10 @@ one copy from the back end. In some products this is called request
 coalescing and Varnish does this automatically.
 
 If you are serving thousands of hits per second the queue of waiting
-requests can get huge. There are to potential problems - one is a
-thundering heard problem - suddenly releasing a thousand threads to
-serve content might send to load sky high. Secondly - nobody likes to
-wait. To deal with this Varnish can we can instruct Varnish to keep
+requests can get huge. There are two potential problems - one is a
+thundering herd problem - suddenly releasing a thousand threads to
+serve content might send the load sky high. Secondly - nobody likes to
+wait. To deal with this we can instruct Varnish to keep
 the objects in cache beyond their TTL and to serve the waiting
 requests somewhat stale content.
 
@@ -88,7 +88,7 @@ Known limitations on grace- and saint mode
 If your request fails while it is being fetched you're thrown into
 vcl_error. vcl_error has access to a rather limited set of data so you
 can't enable saint mode or grace mode here. This will be addressed in a
-future release but a work-around is available.
+future release but a work-around available.
 
 * Declare a backend that is always sick.
 * Set a magic marker in vcl_error



From tfheen at varnish-cache.org  Wed Mar  9 10:10:12 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 09 Mar 2011 11:10:12 +0100
Subject: [master] 589f8f6 Switch from mkpasswd to uuidgen in rpm spec file
Message-ID: <E1PxGLM-0005i0-S6@project.varnish-software.com>

commit 589f8f6586609d681acc0a3cdf8130083d707002
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 9 09:39:56 2011 +0100

    Switch from mkpasswd to uuidgen in rpm spec file
    
    uuidgen lives in e2fsprogs which is almost always installed, unlike
    mkpasswd which lives in expect, so change to using uuidgen.  No change
    for installed systems.

diff --git a/redhat/varnish.spec b/redhat/varnish.spec
index bb6dda9..3ea0b11 100644
--- a/redhat/varnish.spec
+++ b/redhat/varnish.spec
@@ -17,7 +17,7 @@ Requires: logrotate
 Requires: ncurses
 Requires: pcre
 Requires(pre): shadow-utils
-Requires(post): /sbin/chkconfig, /usr/bin/mkpasswd
+Requires(post): /sbin/chkconfig, /usr/bin/uuidgen
 Requires(preun): /sbin/chkconfig
 Requires(preun): /sbin/service
 Requires(preun): initscripts
@@ -218,7 +218,7 @@ exit 0
 /sbin/chkconfig --add varnish
 /sbin/chkconfig --add varnishlog
 /sbin/chkconfig --add varnishncsa 
-test -f /etc/varnish/secret || (mkpasswd > /etc/varnish/secret && chmod 0600 /etc/varnish/secret)
+test -f /etc/varnish/secret || (uuidgen > /etc/varnish/secret && chmod 0600 /etc/varnish/secret)
 
 %preun
 if [ $1 -lt 1 ]; then



From tfheen at varnish-cache.org  Wed Mar  9 10:10:13 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 09 Mar 2011 11:10:13 +0100
Subject: [master] f2b1e0a Add support for system libjemalloc
Message-ID: <E1PxGLN-0005i8-47@project.varnish-software.com>

commit f2b1e0a2e31e9906200e9739bc311746066d3b6e
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 9 11:09:35 2011 +0100

    Add support for system libjemalloc
    
    Use system libjemalloc if found, if not we use the bundled one.

diff --git a/configure.ac b/configure.ac
index ab2bd65..22f0651 100644
--- a/configure.ac
+++ b/configure.ac
@@ -236,6 +236,29 @@ if test "$ac_cv_have_viz" = no; then
 fi 
 CFLAGS="${save_CFLAGS}" 
 
+# Use jemalloc on Linux
+JEMALLOC_SUBDIR=
+JEMALLOC_LDADD=
+AC_ARG_WITH([jemalloc],
+            [AS_HELP_STRING([--with-jemalloc],
+              [use jemalloc memory allocator.  Default is yes on Linux,  no elsewhere])],
+            [],
+            [with_jemalloc=check])
+
+case $target in
+    *-*-linux*)
+        if test "x$with_jemalloc" != xno; then
+            AC_CHECK_LIB([jemalloc], [malloc_conf],
+                  [JEMALLOC_LDADD="-ljemalloc"],
+                  [AC_MSG_NOTICE([No system jemalloc found, using bundled version])
+                   JEMALLOC_SUBDIR=libjemalloc
+                   JEMALLOC_LDADD='$(top_builddir)/lib/libjemalloc/libjemalloc_mt.la'])
+	fi
+	;;
+esac
+AC_SUBST(JEMALLOC_SUBDIR)
+AC_SUBST(JEMALLOC_LDADD)
+
 # Userland slab allocator, available only on Solaris
 case $target in
 *-*-solaris*)
@@ -477,28 +500,6 @@ fi
 
 AC_DEFINE_UNQUOTED([VCC_CC],"$VCC_CC",[C compiler command line for VCL code])
 
-# Use jemalloc on Linux
-JEMALLOC_SUBDIR=
-JEMALLOC_LDADD=
-AC_ARG_ENABLE(jemalloc,
-AS_HELP_STRING([--disable-jemalloc],[do not use jemalloc (default is yes on Linux, no everywhere else)]),
-[if test "x$enableval" = "xyes"; then
-	JEMALLOC_SUBDIR=libjemalloc
-	JEMALLOC_LDADD='$(top_builddir)/lib/libjemalloc/libjemalloc_mt.la'
-fi],
-[case $target in #(
-*-*-linux*)
-	JEMALLOC_SUBDIR=libjemalloc
-	JEMALLOC_LDADD='$(top_builddir)/lib/libjemalloc/libjemalloc_mt.la'
-	;; #(
-*)
-	true
-	;;
-esac])
-
-AC_SUBST(JEMALLOC_SUBDIR)
-AC_SUBST(JEMALLOC_LDADD)
-
 # Generate output
 AC_CONFIG_FILES([
     Makefile



From phk at varnish-cache.org  Wed Mar  9 12:21:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:21:40 +0100
Subject: [master] c95acca Use brain in low-power mode while sneezing my way
	through the first cold of spring and add the "duration"
	conversion function to vmod_std.
Message-ID: <E1PxIOa-0008Kj-LK@project.varnish-software.com>

commit c95acca7351e708cabb3f6b3126aaed878155c2b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:19:46 2011 +0000

    Use brain in low-power mode while sneezing my way through the first
    cold of spring and add the "duration" conversion function to vmod_std.
    
    The way these conversions work is that you give them the string you
    want converted, and the value to use if they cannot.
    
    like so:
    
    	set beresp.ttl = std.duration(beresp.http.x-ttl, 10 s);

diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am
index 5437c02..a284b09 100644
--- a/lib/libvmod_std/Makefile.am
+++ b/lib/libvmod_std/Makefile.am
@@ -11,7 +11,8 @@ libvmod_std_la_SOURCES = \
 	vcc_if.c \
 	vcc_if.h \
 	vmod_std.c \
-	vmod_std_fileread.c
+	vmod_std_fileread.c \
+	vmod_std_conversions.c
 
 vcc_if.c vcc_if.h: $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc
 	@PYTHON@ $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc
diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index b2923dc..d77d49a 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -34,9 +34,5 @@ Function REAL random(REAL, REAL)
 Function VOID log(STRING_LIST)
 Function VOID syslog(INT, STRING_LIST)
 Function STRING fileread(PRIV_CALL, STRING)
-Function STRING author(ENUM {
-	phk,
-	des,
-	kristian,
-	mithrandir
-})
+Function STRING author(ENUM { phk, des, kristian, mithrandir })
+Function DURATION duration(STRING, DURATION)
diff --git a/lib/libvmod_std/vmod_std_conversions.c b/lib/libvmod_std/vmod_std_conversions.c
new file mode 100644
index 0000000..f17b752
--- /dev/null
+++ b/lib/libvmod_std/vmod_std_conversions.c
@@ -0,0 +1,86 @@
+/*-
+ * Copyright (c) 2010 Linpro AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include <ctype.h>
+#include <math.h>
+#include <stdlib.h>
+
+#include "../../bin/varnishd/cache.h"
+
+#include "vcc_if.h"
+
+double __match_proto__()
+vmod_duration(struct sess *sp, const char *p, double d)
+{
+	char *e;
+	double r;
+
+	(void)sp;
+
+	if (p == NULL)
+		return (d);
+
+	while(isspace(*p))
+		p++;
+
+	if (*p != '+' && *p != '-' && !isdigit(*p))
+		return (d);
+
+	e = NULL;
+
+	r = strtod(p, &e);
+
+	if (!isfinite(r))
+		return (d);
+
+	if (e == NULL)
+		return (d);
+
+	while(isspace(*e))
+		e++;
+
+	/* NB: Keep this list synchronized with VCC */
+	switch (*e++) {
+	case 's': break;
+	case 'm': r *= 60.; break;
+	case 'h': r *= 60.*60.; break;
+	case 'd': r *= 60.*60.*24.; break;
+	case 'w': r *= 60.*60.*24.*7.; break;
+	default:
+		return (d);
+	}
+
+	while(isspace(*e))
+		e++;
+
+	if (*e != '\0')
+		return (d);
+
+	return (r);
+}



From phk at varnish-cache.org  Wed Mar  9 12:21:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:21:40 +0100
Subject: [master] 6038894 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1PxIOa-0008Ks-Sd@project.varnish-software.com>

commit 6038894bb8e1863b81e3fe2798e79923a79eb0cf
Merge: c95acca f2b1e0a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:21:37 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Wed Mar  9 12:48:34 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:48:34 +0100
Subject: [master] b8f4cd4 Flexelint silencing.
Message-ID: <E1PxIoc-0001H0-6u@project.varnish-software.com>

commit b8f4cd4d45448c46a8de1ddfa38cde45a2760ad2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:36:26 2011 +0000

    Flexelint silencing.
    
    (Did you remember that toupper() can be called with EOF ?)

diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 2807af1..4be07c7 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -59,9 +59,9 @@ vmod_updown(struct sess *sp, int up, const char *s, va_list ap)
 		if (p != NULL) {
 			for (; b < e && *p != '\0'; p++)
 				if (up)
-					*b++ = toupper(*p);
+					*b++ = (char)toupper(*p);
 				else
-					*b++ = tolower(*p);
+					*b++ = (char)tolower(*p);
 		}
 		p = va_arg(ap, const char *);
 	}



From phk at varnish-cache.org  Wed Mar  9 12:48:34 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:48:34 +0100
Subject: [master] ba51600 Add missing assert
Message-ID: <E1PxIoc-0001H9-El@project.varnish-software.com>

commit ba516000f1dfd3c267f66de58319cc02467f3a08
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:47:43 2011 +0000

    Add missing assert

diff --git a/lib/libvmod_std/vmod_std_fileread.c b/lib/libvmod_std/vmod_std_fileread.c
index a1d9798..8ab7b70 100644
--- a/lib/libvmod_std/vmod_std_fileread.c
+++ b/lib/libvmod_std/vmod_std_fileread.c
@@ -125,7 +125,7 @@ vmod_fileread(struct sess *sp, struct vmod_priv *priv, const char *file_name)
 	if ((fd = open(file_name, O_RDONLY)) == -1)
 		return "";
 
-	fstat(fd, &buf);
+	AZ(fstat(fd, &buf)i);
 
 	AZ(pthread_rwlock_wrlock(&filelist_lock));
 



From phk at varnish-cache.org  Wed Mar  9 12:48:34 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:48:34 +0100
Subject: [master] 638e22a Clarify code a tiny bit,
	making it more understandable for FlexeLint
Message-ID: <E1PxIoc-0001HM-Pf@project.varnish-software.com>

commit 638e22acdb045ee22320edf1eeb4522f8c0746b1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:48:16 2011 +0000

    Clarify code a tiny bit, making it more understandable for FlexeLint

diff --git a/bin/varnishd/cache_vrt.c b/bin/varnishd/cache_vrt.c
index 944f583..7c7e107 100644
--- a/bin/varnishd/cache_vrt.c
+++ b/bin/varnishd/cache_vrt.c
@@ -154,13 +154,10 @@ VRT_StringList(char *d, unsigned dl, const char *p, va_list ap)
 		}
 		p = va_arg(ap, const char *);
 	}
-	if (b < e)
-		*b = '\0';
-	b++;
-	if (b > e)
+	if (b >= e)
 		return (NULL);
-	else
-		return (b);
+	*b++ = '\0';
+	return (b);
 }
 
 /*--------------------------------------------------------------------



From phk at varnish-cache.org  Wed Mar  9 12:51:10 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:51:10 +0100
Subject: [master] 1dfaf8a Editor-malfunction
Message-ID: <E1PxIr8-0001cI-Qe@project.varnish-software.com>

commit 1dfaf8aa53f556ab99359dbf1d948f9f87c865da
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:51:02 2011 +0000

    Editor-malfunction

diff --git a/lib/libvmod_std/vmod_std_fileread.c b/lib/libvmod_std/vmod_std_fileread.c
index 8ab7b70..9171a4b 100644
--- a/lib/libvmod_std/vmod_std_fileread.c
+++ b/lib/libvmod_std/vmod_std_fileread.c
@@ -125,7 +125,7 @@ vmod_fileread(struct sess *sp, struct vmod_priv *priv, const char *file_name)
 	if ((fd = open(file_name, O_RDONLY)) == -1)
 		return "";
 
-	AZ(fstat(fd, &buf)i);
+	AZ(fstat(fd, &buf));
 
 	AZ(pthread_rwlock_wrlock(&filelist_lock));
 



From phk at varnish-cache.org  Wed Mar  9 12:52:58 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Mar 2011 13:52:58 +0100
Subject: [master] 281c818 Clarify a pointers value for FlexeLint
Message-ID: <E1PxIss-0001lx-Ut@project.varnish-software.com>

commit 281c81870d9c6ba1a75ff685e425bccc12d771e7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 9 12:52:46 2011 +0000

    Clarify a pointers value for FlexeLint

diff --git a/bin/varnishd/cache_dir_random.c b/bin/varnishd/cache_dir_random.c
index 45a74d0..9727232 100644
--- a/bin/varnishd/cache_dir_random.c
+++ b/bin/varnishd/cache_dir_random.c
@@ -90,7 +90,7 @@ vdi_random_getfd(const struct director *d, struct sess *sp)
 	struct vbc *vbe;
 	struct director *d2;
 	struct SHA256Context ctx;
-	uint8_t sign[SHA256_LEN], *hp;
+	uint8_t sign[SHA256_LEN], *hp = NULL;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
@@ -128,6 +128,7 @@ vdi_random_getfd(const struct director *d, struct sess *sp)
 	 * amongst the healthy set.
 	 */
 	if (vs->criteria != c_random) {
+		AN(hp);
 		u = vle32dec(hp);
 		r = u / 4294967296.0;
 		assert(r >= 0.0 && r < 1.0);



From martin at varnish-cache.org  Fri Mar 11 12:25:40 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 11 Mar 2011 13:25:40 +0100
Subject: [master] 0a23fd6 Add terminal reset call to libedit when exiting
Message-ID: <E1Py1PY-0004IA-3x@project.varnish-software.com>

commit 0a23fd621900329ef00a428c06eabc0d734e757f
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Mar 11 13:14:26 2011 +0100

    Add terminal reset call to libedit when exiting

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 147438e..e0a6119 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -50,6 +50,16 @@ SVNID("$Id$")
 #include "varnishapi.h"
 #include "vss.h"
 
+#ifdef HAVE_LIBEDIT
+#define RL_EXIT(status) \
+	do { \
+		rl_callback_handler_remove(); \
+		exit(status); \
+	} while (0);
+#else
+#define RL_EXIT(status) exit(status);
+#endif
+
 static double timeout = 5;
 
 static void
@@ -62,7 +72,7 @@ cli_write(int sock, const char *s)
 	if (i == l)
 		return;
 	perror("Write error CLI socket");
-	exit (1);
+	RL_EXIT(1);
 }
 
 /*
@@ -164,9 +174,13 @@ do_args(int sock, int argc, char * const *argv)
 static int _line_sock;
 void send_line(char *l)
 {
-	cli_write(_line_sock, l);
-	cli_write(_line_sock, "\n");
-	add_history(l);
+	if (l) {
+		cli_write(_line_sock, l);
+		cli_write(_line_sock, "\n");
+		add_history(l);
+	} else {
+		RL_EXIT(0);
+	}
 }
 #endif
 
@@ -202,10 +216,10 @@ pass(int sock)
 		if (fds[0].revents & POLLIN) {
 			n = read(fds[0].fd, buf, sizeof buf);
 			if (n == 0)
-				exit (0);
+				RL_EXIT(0);
 			if (n < 0) {
 				perror("Read error reading CLI socket");
-				exit (0);
+				RL_EXIT(1);
 			}
 			assert(n > 0);
 			/* Get rid of the prompt, kinda hackish */
@@ -213,7 +227,7 @@ pass(int sock)
 			m = write(1, buf, n);
 			if (n != m) {
 				perror("Write error writing stdout");
-				exit (1);
+				RL_EXIT(1);
 			}
 #ifdef HAVE_LIBEDIT
 			rl_forced_update_display();
@@ -228,7 +242,7 @@ pass(int sock)
 				AZ(shutdown(sock, SHUT_WR));
 				fds[1].fd = -1;
 			} else if (n < 0) {
-				exit(0);
+				RL_EXIT(0);
 			} else {
 				buf[n] = '\0';
 				cli_write(sock, buf);



From martin at varnish-cache.org  Fri Mar 11 14:09:41 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 11 Mar 2011 15:09:41 +0100
Subject: [master] ef59a3c Remove extra semicolon in macros
Message-ID: <E1Py32D-00022C-0z@project.varnish-software.com>

commit ef59a3c7a2a46a14b2406c817fa84118e6ddbeee
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Mar 11 14:28:23 2011 +0100

    Remove extra semicolon in macros

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index e0a6119..628a995 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -55,9 +55,9 @@ SVNID("$Id$")
 	do { \
 		rl_callback_handler_remove(); \
 		exit(status); \
-	} while (0);
+	} while (0)
 #else
-#define RL_EXIT(status) exit(status);
+#define RL_EXIT(status) exit(status)
 #endif
 
 static double timeout = 5;



From martin at varnish-cache.org  Fri Mar 11 14:09:41 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 11 Mar 2011 15:09:41 +0100
Subject: [master] 7058d47 Use cli_readres for varnish responses in interactive
	mode
Message-ID: <E1Py32D-00022L-7q@project.varnish-software.com>

commit 7058d471b586151e01a8167b04d5845edffa7fd5
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Mar 11 15:07:26 2011 +0100

    Use cli_readres for varnish responses in interactive mode

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 628a995..4f3d013 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -193,7 +193,9 @@ pass(int sock)
 {
 	struct pollfd fds[2];
 	char buf[1024];
-	int i, n, m;
+	int i, n;
+	char *answer = NULL;
+	unsigned u, status;
 
 #ifdef HAVE_LIBEDIT
 	_line_sock = sock;
@@ -214,21 +216,25 @@ pass(int sock)
 		i = poll(fds, 2, -1);
 		assert(i > 0);
 		if (fds[0].revents & POLLIN) {
-			n = read(fds[0].fd, buf, sizeof buf);
-			if (n == 0)
-				RL_EXIT(0);
-			if (n < 0) {
-				perror("Read error reading CLI socket");
-				RL_EXIT(1);
-			}
-			assert(n > 0);
 			/* Get rid of the prompt, kinda hackish */
-			write(1, "\r           \r", 13);
-			m = write(1, buf, n);
-			if (n != m) {
-				perror("Write error writing stdout");
+			u = write(1, "\r           \r", 13);
+			u = cli_readres(fds[0].fd, &status, &answer, timeout);
+			if (u) {
+				if (status == CLIS_COMMS)
+					RL_EXIT(0);
+				if (answer)
+					fprintf(stderr, "%s\n", answer);
 				RL_EXIT(1);
 			}
+
+			sprintf(buf, "%u\n", status);
+			u = write(1, buf, strlen(buf));
+			if (answer) {
+				u = write(1, answer, strlen(answer));
+				u = write(1, "\n", 1);
+				free(answer);
+				answer = NULL;
+			}
 #ifdef HAVE_LIBEDIT
 			rl_forced_update_display();
 #endif



From phk at varnish-cache.org  Fri Mar 11 20:06:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 11 Mar 2011 21:06:24 +0100
Subject: [master] 1ebd861 grace=0 should not mean "use default"
Message-ID: <E1Py8bQ-0001zD-CM@project.varnish-software.com>

commit 1ebd86156a21a62bda61059958a3a0146e78bb1f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Mar 11 14:15:10 2011 +0000

    grace=0 should not mean "use default"
    
    Spotted by:	Geoff

diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index c95c43f..3a691a6 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -119,7 +119,7 @@ EXP_Grace(const struct sess *sp, const struct object *o)
 	double r;
 
 	r = (double)params->default_grace;
-	if (o->exp.grace > 0.)
+	if (o->exp.grace >= 0.)
 		r = o->exp.grace;
 	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace < r)
 		r = sp->exp.grace;



From phk at varnish-cache.org  Fri Mar 11 20:06:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 11 Mar 2011 21:06:25 +0100
Subject: [master] b530ed4 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Py8bR-0001zM-8a@project.varnish-software.com>

commit b530ed433bb70e2aa1c212ac4fd11ec243d2635e
Merge: 1ebd861 7058d47
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Mar 11 14:15:35 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Fri Mar 11 20:06:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 11 Mar 2011 21:06:25 +0100
Subject: [master] 9c9ce2a Make the power-of-two rounding functions generally
	available and use them for VSM allocation
Message-ID: <E1Py8bR-0001zT-SK@project.varnish-software.com>

commit 9c9ce2a61d3efc9084663b8b801988bf6398c393
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Mar 11 19:55:05 2011 +0000

    Make the power-of-two rounding functions generally available and
    use them for VSM allocation

diff --git a/bin/varnishd/common.h b/bin/varnishd/common.h
index 6cbbfd7..bccac52 100644
--- a/bin/varnishd/common.h
+++ b/bin/varnishd/common.h
@@ -93,3 +93,11 @@ void vsm_iter_n(struct vsm_chunk **pp);
 
 /* cache_lck.c */
 struct lock { void *priv; };		// Opaque
+
+/*---------------------------------------------------------------------
+ * Generic power-2 rounding macros
+ */
+
+#define PWR2(x)     ((((x)-1)&(x))==0)		/* Is a power of two */
+#define RDN2(x, y)  ((x)&(~((y)-1)))		/* if y is powers of two */
+#define RUP2(x, y)  (((x)+((y)-1))&(~((y)-1)))	/* if y is powers of two */
diff --git a/bin/varnishd/storage_persistent.h b/bin/varnishd/storage_persistent.h
index 4841a2d..e7c7344 100644
--- a/bin/varnishd/storage_persistent.h
+++ b/bin/varnishd/storage_persistent.h
@@ -148,11 +148,6 @@ struct smp_sc {
 
 /*--------------------------------------------------------------------*/
 
-/* Generic power-2 rounding */
-#define PWR2(x)     ((((x)-1)&(x))==0)		/* Is a power of two */
-#define RDN2(x, y)  ((x)&(~((y)-1)))		/* if y is powers of two */
-#define RUP2(x, y)  (((x)+((y)-1))&(~((y)-1)))	/* if y is powers of two */
-
 /* Pointer round up/down & assert */
 #define PRNDN(sc, x)	((void*)RDN2((uintptr_t)(x), sc->align))
 #define PRNUP(sc, x)	((void*)RUP2((uintptr_t)(x), sc->align))
diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
index 7585bee..ccdfa38 100644
--- a/bin/varnishd/vsm.c
+++ b/bin/varnishd/vsm.c
@@ -165,8 +165,7 @@ VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident)
 	vsm_cleanup();
 
 	/* Round up to pointersize */
-	size += sizeof(void *) - 1;
-	size &= ~(sizeof(void *) - 1);
+	size = RUP2(size, sizeof(void*));
 
 	size += sizeof *sha;		/* Make space for the header */
 



From phk at varnish-cache.org  Fri Mar 11 20:06:26 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 11 Mar 2011 21:06:26 +0100
Subject: [master] 72e08e2 Fix a VSM segment list corruption bug,
	spotted by inspection.
Message-ID: <E1Py8bS-0001zi-PD@project.varnish-software.com>

commit 72e08e2a9de1b15b27c07a4f08764f1238308e13
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Mar 11 20:05:57 2011 +0000

    Fix a VSM segment list corruption bug, spotted by inspection.

diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
index ccdfa38..4d689ab 100644
--- a/bin/varnishd/vsm.c
+++ b/bin/varnishd/vsm.c
@@ -181,7 +181,7 @@ VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident)
 			/* Mark as inconsistent while we write string fields */
 		seq = vsm_mark();
 
-		if (size < sha->len) {
+		if (size + sizeof (*sha) < sha->len) {
 			sha2 = (void*)((uintptr_t)sha + size);
 
 			memset(sha2, 0, sizeof *sha2);



From phk at varnish-cache.org  Sat Mar 12 14:47:09 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sat, 12 Mar 2011 15:47:09 +0100
Subject: [master] ad65850 Actually free the memory libvgz allocated to work in.
Message-ID: <E1PyQ61-0007Ga-DO@project.varnish-software.com>

commit ad6585009cd59831cf85d2e264e811d408d34425
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Mar 12 14:45:27 2011 +0000

    Actually free the memory libvgz allocated to work in.
    
    No wonder gzip support leaked memory...
    
    Orginally this memory came from the the thread workspace so deallocation
    was automatic.  Not so when we switched to malloc...

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 9b5180c..50f8c83 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -80,6 +80,7 @@ SVNID("$Id$")
 struct vgz {
 	unsigned		magic;
 #define VGZ_MAGIC		0x162df0cb
+	enum {VGZ_GZ,VGZ_UN}	dir;
 	struct sess		*sess;
 	const char		*id;
 	struct ws		*tmp;
@@ -162,6 +163,7 @@ VGZ_NewUngzip(struct sess *sp, const char *id)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	vg = vgz_alloc_vgz(sp, id);
+	vg->dir = VGZ_UN;
 	VSC_main->n_gunzip++;
 
 	/*
@@ -182,6 +184,7 @@ VGZ_NewGzip(struct sess *sp, const char *id)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	vg = vgz_alloc_vgz(sp, id);
+	vg->dir = VGZ_GZ;
 	VSC_main->n_gzip++;
 
 	/*
@@ -380,6 +383,10 @@ VGZ_Destroy(struct vgz **vgp)
 	    (intmax_t)vg->vz.stop_bit);
 	if (vg->tmp != NULL)
 		WS_Reset(vg->tmp, vg->tmp_snapshot);
+	if (vg->dir == VGZ_GZ)
+		AZ(deflateEnd(&vg->vz));
+	else
+		AZ(inflateEnd(&vg->vz));
 	FREE_OBJ(vg);
 }
 



From phk at varnish-cache.org  Sat Mar 12 14:47:09 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sat, 12 Mar 2011 15:47:09 +0100
Subject: [master] 8838884 Make the VSM_ITER macro available to all vsm users.
Message-ID: <E1PyQ61-0007Gj-Qx@project.varnish-software.com>

commit 8838884f34e50453318873960d975a1ade908ea1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Mar 12 14:46:54 2011 +0000

    Make the VSM_ITER macro available to all vsm users.

diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c
index 2dc6256..94e9a6c 100644
--- a/bin/varnishd/cache_shmlog.c
+++ b/bin/varnishd/cache_shmlog.c
@@ -36,9 +36,9 @@ SVNID("$Id$")
 #include <unistd.h>
 #include <stdarg.h>
 
-#include "vsm.h"
 #include "cache.h"
 #include "vmb.h"
+#include "vsm.h"
 
 static pthread_mutex_t vsl_mtx;
 
diff --git a/bin/varnishd/common.h b/bin/varnishd/common.h
index bccac52..78d59c4 100644
--- a/bin/varnishd/common.h
+++ b/bin/varnishd/common.h
@@ -78,12 +78,6 @@ void *VSM_Alloc(unsigned size, const char *class, const char *type,
 void VSM_Free(const void *ptr);
 void VSM_Clean(void);
 
-
-struct vsm_chunk *vsm_iter_0(void);
-void vsm_iter_n(struct vsm_chunk **pp);
-
-#define VSM_ITER(vd) for ((vd) = vsm_iter_0(); (vd) != NULL; vsm_iter_n(&vd))
-
 /* These classes are opaque to other programs, so we define the here */
 #define VSM_CLASS_FREE	"Free"
 #define VSM_CLASS_COOL	"Cool"
diff --git a/bin/varnishd/mgt_shmem.c b/bin/varnishd/mgt_shmem.c
index 3af3e7b..e55bcc2 100644
--- a/bin/varnishd/mgt_shmem.c
+++ b/bin/varnishd/mgt_shmem.c
@@ -98,12 +98,12 @@ SVNID("$Id$")
 #include <sys/mman.h>
 #include <sys/stat.h>
 
-#include "vsm.h"
 #include "vsc.h"
 #include "vsl.h"
 #include "mgt.h"
 #include "heritage.h"
 #include "vmb.h"
+#include "vsm.h"
 
 #ifndef MAP_HASSEMAPHORE
 #define MAP_HASSEMAPHORE 0 /* XXX Linux */
diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
index 4d689ab..c5f6748 100644
--- a/bin/varnishd/vsm.c
+++ b/bin/varnishd/vsm.c
@@ -72,31 +72,6 @@ vsm_release(unsigned seq)
 
 /*--------------------------------------------------------------------*/
 
-struct vsm_chunk *
-vsm_iter_0(void)
-{
-
-	CHECK_OBJ_NOTNULL(vsm_head, VSM_HEAD_MAGIC);
-	CHECK_OBJ_NOTNULL(&vsm_head->head, VSM_CHUNK_MAGIC);
-	return (&vsm_head->head);
-}
-
-void
-vsm_iter_n(struct vsm_chunk **pp)
-{
-
-	CHECK_OBJ_NOTNULL(vsm_head, VSM_HEAD_MAGIC);
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-	*pp = VSM_NEXT(*pp);
-	if (*pp >= vsm_end) {
-		*pp = NULL;
-		return;
-	}
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-}
-
-/*--------------------------------------------------------------------*/
-
 static void
 vsm_cleanup(void)
 {
@@ -178,7 +153,7 @@ VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident)
 		if (size > sha->len)
 			continue;
 
-			/* Mark as inconsistent while we write string fields */
+		/* Mark as inconsistent while we write string fields */
 		seq = vsm_mark();
 
 		if (size + sizeof (*sha) < sha->len) {
diff --git a/include/vsm.h b/include/vsm.h
index 09d42c4..caaae06 100644
--- a/include/vsm.h
+++ b/include/vsm.h
@@ -78,4 +78,37 @@ struct vsm_head {
 	struct vsm_chunk	head;
 };
 
+/*
+ * You must include "miniobj.h" and have an assert function to be
+ * able to use the VSM_ITER() macro.
+ */
+#ifdef CHECK_OBJ_NOTNULL
+
+static inline struct vsm_chunk *
+vsm_iter_0(void)
+{
+
+	CHECK_OBJ_NOTNULL(vsm_head, VSM_HEAD_MAGIC);
+	CHECK_OBJ_NOTNULL(&vsm_head->head, VSM_CHUNK_MAGIC);
+	return (&vsm_head->head);
+}
+ 
+static inline void
+vsm_iter_n(struct vsm_chunk **pp)
+{
+
+	CHECK_OBJ_NOTNULL(vsm_head, VSM_HEAD_MAGIC);
+	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
+	*pp = VSM_NEXT(*pp);
+	if (*pp >= vsm_end) {
+		*pp = NULL;
+		return;
+	}
+	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
+}
+
+#define VSM_ITER(vd) for ((vd) = vsm_iter_0(); (vd) != NULL; vsm_iter_n(&vd))
+
+#endif /* CHECK_OBJ_NOTNULL */
+
 #endif



From phk at varnish-cache.org  Sat Mar 12 20:37:21 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sat, 12 Mar 2011 21:37:21 +0100
Subject: [master] 958114d Commenting
Message-ID: <E1PyVYv-0004PK-E0@project.varnish-software.com>

commit 958114da8bc760dfc318b7312ac73dfd6bfdf8ea
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Mar 12 20:37:13 2011 +0000

    Commenting

diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
index c5f6748..624d5a5 100644
--- a/bin/varnishd/vsm.c
+++ b/bin/varnishd/vsm.c
@@ -27,6 +27,22 @@
  *
  * VSM stuff common to manager and child.
  *
+ * We have three potential conflicts we need to lock against here:
+ *
+ * VSM-studying programs (varnishstat...) vs. everybody else
+ *	The VSM studying programs only have read-only access to the VSM
+ *	so everybody else must use memory barriers, stable storage and
+ *	similar tricks to keep the VSM image in sync (long enough) for
+ * 	the studying programs.
+ *
+ * Manager process vs child process.
+ *	Will only muck about in VSM when child process is not running
+ *	Responsible for cleaning up any mess left behind by dying child.
+ *
+ * Child process threads
+ *	Pthread locking necessary.
+ *
+ * XXX: not all of this is in place yet.
  */
 
 #include "config.h"
@@ -89,7 +105,7 @@ vsm_cleanup(void)
 	if (sha == NULL)
 		return;
 	seq = vsm_mark();
-	/* First pass, free, and collaps with next if applicable */
+	/* First pass, free, and collapse with next if applicable */
 	VSM_ITER(sha) {
 		if (strcmp(sha->class, VSM_CLASS_COOL))
 			continue;



From phk at varnish-cache.org  Sun Mar 13 16:48:41 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Mar 2011 17:48:41 +0100
Subject: [master] 7259fab Lock the VSM alloc/free cals in the child process
Message-ID: <E1PyoTB-0007py-8I@project.varnish-software.com>

commit 7259fab64227cbfb35eaa1b56942d2ee2eda7fa9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Mar 12 21:11:19 2011 +0000

    Lock the VSM alloc/free cals in the child process

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index ea82edd..4c31a7e 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -803,6 +803,9 @@ void SES_Charge(struct sess *sp);
 
 /* cache_shmlog.c */
 void VSL_Init(void);
+void *VSM_Alloc(unsigned size, const char *class, const char *type,
+    const char *ident);
+void VSM_Free(const void *ptr);
 #ifdef VSL_ENDMARKER
 void VSL(enum vsl_tag tag, int id, const char *fmt, ...);
 void WSLR(struct worker *w, enum vsl_tag tag, int id, txt t);
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index d874a98..3387e9e 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -104,7 +104,7 @@ child_main(void)
 
 	VSL_Init();	/* First, LCK needs it. */
 
-	LCK_Init();	/* Locking, must be first */
+	LCK_Init();	/* Second, locking */
 
 	PAN_Init();
 	CLI_Init();
diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c
index 94e9a6c..62d6f92 100644
--- a/bin/varnishd/cache_shmlog.c
+++ b/bin/varnishd/cache_shmlog.c
@@ -40,7 +40,9 @@ SVNID("$Id$")
 #include "vmb.h"
 #include "vsm.h"
 
+/* These cannot be struct lock, which depends on vsm/vsl working */
 static pthread_mutex_t vsl_mtx;
+static pthread_mutex_t vsm_mtx;
 
 static uint32_t			*vsl_start;
 static const uint32_t		*vsl_end;
@@ -275,8 +277,9 @@ VSL_Init(void)
 	struct vsm_chunk *vsc;
 
 	AZ(pthread_mutex_init(&vsl_mtx, NULL));
+	AZ(pthread_mutex_init(&vsm_mtx, NULL));
 
-	VSM_Clean();
+	VSM__Clean();
 
 	VSM_ITER(vsc)
 		if (!strcmp(vsc->class, VSL_CLASS))
@@ -292,3 +295,26 @@ VSL_Init(void)
 	memset(VSC_main, 0, sizeof *VSC_main);
 	vsm_head->child_pid = getpid();
 }
+
+/*--------------------------------------------------------------------*/
+
+void *
+VSM_Alloc(unsigned size, const char *class, const char *type,
+    const char *ident)
+{
+	void *p;
+
+	AZ(pthread_mutex_lock(&vsm_mtx));
+	p = VSM__Alloc(size, class, type, ident);
+	AZ(pthread_mutex_unlock(&vsm_mtx));
+	return (p);
+}
+
+void
+VSM_Free(const void *ptr)
+{
+
+	AZ(pthread_mutex_lock(&vsm_mtx));
+	VSM__Free(ptr);
+	AZ(pthread_mutex_unlock(&vsm_mtx));
+}
diff --git a/bin/varnishd/common.h b/bin/varnishd/common.h
index 78d59c4..d17eca8 100644
--- a/bin/varnishd/common.h
+++ b/bin/varnishd/common.h
@@ -73,10 +73,14 @@ const void *pick(const struct choice *cp, const char *which, const char *kind);
 extern struct vsm_head		*vsm_head;
 extern const struct vsm_chunk	*vsm_end;
 
-void *VSM_Alloc(unsigned size, const char *class, const char *type,
+/*
+ * These three should not be called directly, but only through
+ * proper vectors in mgt.h/cache.h, hence the __
+ */
+void *VSM__Alloc(unsigned size, const char *class, const char *type,
     const char *ident);
-void VSM_Free(const void *ptr);
-void VSM_Clean(void);
+void VSM__Free(const void *ptr);
+void VSM__Clean(void);
 
 /* These classes are opaque to other programs, so we define the here */
 #define VSM_CLASS_FREE	"Free"
diff --git a/bin/varnishd/mgt.h b/bin/varnishd/mgt.h
index a2dc912..d4ecc47 100644
--- a/bin/varnishd/mgt.h
+++ b/bin/varnishd/mgt.h
@@ -99,3 +99,8 @@ extern unsigned mgt_vcc_err_unref;
 		fprintf(stderr, fmt "\n", __VA_ARGS__);	\
 		syslog(pri, fmt, __VA_ARGS__);		\
 	} while (0)
+
+#define VSM_Alloc(a, b, c, d)	VSM__Alloc(a,b,c,d)
+#define VSM_Free(a)		VSM__Free(a)
+#define VSM_Clean()		VSM__Clean()
+
diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
index 624d5a5..92c3830 100644
--- a/bin/varnishd/vsm.c
+++ b/bin/varnishd/vsm.c
@@ -60,6 +60,7 @@ SVNID("$Id$")
 #include "vsm.h"
 #include "vmb.h"
 
+/* These two come from beyond (mgt_shmem.c actually) */
 struct vsm_head		*vsm_head;
 const struct vsm_chunk	*vsm_end;
 
@@ -146,7 +147,7 @@ vsm_cleanup(void)
 /*--------------------------------------------------------------------*/
 
 void *
-VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident)
+VSM__Alloc(unsigned size, const char *class, const char *type, const char *ident)
 {
 	struct vsm_chunk *sha, *sha2;
 	unsigned seq;
@@ -195,7 +196,7 @@ VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_Free(const void *ptr)
+VSM__Free(const void *ptr)
 {
 	struct vsm_chunk *sha;
 	unsigned seq;
@@ -216,7 +217,7 @@ VSM_Free(const void *ptr)
  */
 
 void
-VSM_Clean(void)
+VSM__Clean(void)
 {
 	struct vsm_chunk *sha;
 	unsigned f, seq;
@@ -233,7 +234,7 @@ VSM_Clean(void)
 			continue;
 		if (strcmp(sha->class, VSM_CLASS_FREE) &&
 		    strcmp(sha->class, VSM_CLASS_COOL))
-			VSM_Free(VSM_PTR(sha));
+			VSM__Free(VSM_PTR(sha));
 	}
 	vsm_release(seq);
 }



From phk at varnish-cache.org  Sun Mar 13 16:48:41 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Mar 2011 17:48:41 +0100
Subject: [master] c3c2307 Add a couple of asserts to detect the panic Per sees
	somewhat earlier
Message-ID: <E1PyoTB-0007q9-Ml@project.varnish-software.com>

commit c3c23079c171ee0049005fe335c041fdd2addcf1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Mar 13 16:48:18 2011 +0000

    Add a couple of asserts to detect the panic Per sees somewhat
    earlier

diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
index 385ee1c..d3562fa 100644
--- a/bin/varnishd/cache_esi_fetch.c
+++ b/bin/varnishd/cache_esi_fetch.c
@@ -335,6 +335,7 @@ vfp_esi_begin(struct sess *sp, size_t estimate)
 	}
 
 	(void)estimate;
+	AN(sp->wrk->vep);
 }
 
 static int __match_proto__()
@@ -343,6 +344,7 @@ vfp_esi_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 	int i;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AN(sp->wrk->vep);
 	if (sp->wrk->is_gzip && sp->wrk->do_gunzip)
 		i = vfp_esi_bytes_gu(sp, htc, bytes);
 	else if (sp->wrk->is_gunzip && sp->wrk->do_gzip)
@@ -351,6 +353,7 @@ vfp_esi_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		i = vfp_esi_bytes_gg(sp, htc, bytes);
 	else
 		i = vfp_esi_bytes_uu(sp, htc, bytes);
+	AN(sp->wrk->vep);
 	return (i);
 }
 
@@ -362,6 +365,7 @@ vfp_esi_end(struct sess *sp)
 	ssize_t l;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AN(sp->wrk->vep);
 
 	vsb = VEP_Finish(sp);
 



From phk at varnish-cache.org  Mon Mar 14 08:38:56 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 09:38:56 +0100
Subject: [master] 6cd23c2 Fix HEAD requests with ESI processing: Only call
	vfp->end if we called vfp->begin.
Message-ID: <E1Pz3Im-0006kx-1e@project.varnish-software.com>

commit 6cd23c2a5a9028e42161389fac750f95d5bb77ec
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 08:38:00 2011 +0000

    Fix HEAD requests with ESI processing:  Only call vfp->end if we
    called vfp->begin.
    
    Fixes: #871, #870

diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
index 594aa4e..78c7af8 100644
--- a/bin/varnishd/cache_fetch.c
+++ b/bin/varnishd/cache_fetch.c
@@ -514,14 +514,17 @@ FetchBody(struct sess *sp)
 		cls = fetch_straight(sp, sp->wrk->htc,
 		    sp->wrk->h_content_length);
 		mklen = 1;
+		XXXAZ(sp->wrk->vfp->end(sp));
 		break;
 	case BS_CHUNKED:
 		cls = fetch_chunked(sp, sp->wrk->htc);
 		mklen = 1;
+		XXXAZ(sp->wrk->vfp->end(sp));
 		break;
 	case BS_EOF:
 		cls = fetch_eof(sp, sp->wrk->htc);
 		mklen = 1;
+		XXXAZ(sp->wrk->vfp->end(sp));
 		break;
 	case BS_ERROR:
 		cls = 1;
@@ -532,8 +535,8 @@ FetchBody(struct sess *sp)
 		mklen = 0;
 		INCOMPL();
 	}
-	XXXAZ(sp->wrk->vfp->end(sp));
 	AZ(sp->wrk->vgz_rx);
+	
 	/*
 	 * It is OK for ->end to just leave the last storage segment
 	 * sitting on sp->wrk->storage, we will always call vfp_nop_end()
diff --git a/bin/varnishtest/tests/e00027.vtc b/bin/varnishtest/tests/e00027.vtc
new file mode 100644
index 0000000..041c0ee
--- /dev/null
+++ b/bin/varnishtest/tests/e00027.vtc
@@ -0,0 +1,27 @@
+# $Id$
+
+test "test HEAD against ESI processing"
+
+server s1 {
+	rxreq
+	expect req.url == "/1"
+	txresp -bodylen 1
+
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_fetch {
+		set beresp.do_esi = true;
+	}
+} -start
+
+client c1 {
+	txreq  -url "/1" 
+	rxresp 
+	expect resp.status == 200
+	expect resp.bodylen == 1
+
+	txreq -req "HEAD" -url "/1" 
+	rxresp -no_obj
+	expect resp.status == 200
+} -run



From phk at varnish-cache.org  Mon Mar 14 09:21:26 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 10:21:26 +0100
Subject: [master] 4dac230 Don't explode if ErrWhere2() comes with a
	NULL-pointer, fall back to ErrWhere()
Message-ID: <E1Pz3xu-0001jI-Np@project.varnish-software.com>

commit 4dac2303f87627ae5c37cb85723acef3998c5543
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 08:56:59 2011 +0000

    Don't explode if ErrWhere2() comes with a NULL-pointer, fall back to
    ErrWhere()

diff --git a/lib/libvcl/vcc_token.c b/lib/libvcl/vcc_token.c
index 28f5cdc..6818777 100644
--- a/lib/libvcl/vcc_token.c
+++ b/lib/libvcl/vcc_token.c
@@ -197,6 +197,10 @@ vcc_ErrWhere2(struct vcc *tl, const struct token *t, const struct token *t2)
 {
 	const char  *l1, *l2, *l3;
 
+	if (t == NULL) {
+		vcc_ErrWhere(tl, t2);
+		return;
+	}
 	vcc_iline(t, &l1, 0);
 	t2 = VTAILQ_PREV(t2, tokenhead, list);
 	vcc_iline(t2, &l2, 1);



From phk at varnish-cache.org  Mon Mar 14 09:21:27 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 10:21:27 +0100
Subject: [master] 384d673 Record the http-header-spec for wildcard'ed
	variables.
Message-ID: <E1Pz3xv-0001jU-Ta@project.varnish-software.com>

commit 384d67320a8c748330cd134367e3152cb52f025f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 09:20:52 2011 +0000

    Record the http-header-spec for wildcard'ed variables.

diff --git a/lib/libvcl/vcc_action.c b/lib/libvcl/vcc_action.c
index 8850909..dd36d23 100644
--- a/lib/libvcl/vcc_action.c
+++ b/lib/libvcl/vcc_action.c
@@ -166,7 +166,7 @@ parse_unset(struct vcc *tl)
 	vp = vcc_FindVar(tl, tl->t, 1, "cannot be unset");
 	ERRCHK(tl);
 	assert(vp != NULL);
-	if (vp->fmt != STRING || vp->hdr == NULL) {
+	if (vp->fmt != STRING || vp->http == NULL) {
 		vsb_printf(tl->sb,
 		    "Only http header variables can be unset.\n");
 		vcc_ErrWhere(tl, tl->t);
diff --git a/lib/libvcl/vcc_compile.h b/lib/libvcl/vcc_compile.h
index 3d570b3..8453f1a 100644
--- a/lib/libvcl/vcc_compile.h
+++ b/lib/libvcl/vcc_compile.h
@@ -178,6 +178,7 @@ struct var {
 	unsigned		r_methods;
 	const char		*lname;
 	unsigned		w_methods;
+	const char		*http;
 	const char		*hdr;
 };
 
diff --git a/lib/libvcl/vcc_var.c b/lib/libvcl/vcc_var.c
index ca91d1c..8bec564 100644
--- a/lib/libvcl/vcc_var.c
+++ b/lib/libvcl/vcc_var.c
@@ -61,15 +61,15 @@ vcc_Var_Wildcard(struct vcc *tl, const struct token *t, const struct symbol *wc)
 	v->r_methods = vh->r_methods;
 	v->w_methods = vh->w_methods;
 	v->fmt = STRING;
-	v->hdr = vh->hdr;
+	v->http = vh->http;
 	l = strlen(v->name + vh->len) + 1;
 
-	bprintf(buf, "VRT_GetHdr(sp, %s, \"\\%03o%s:\")",
-	    v->hdr, (unsigned)l, v->name + vh->len);
+	bprintf(buf, "\\%03o%s:", (unsigned)l, v->name + vh->len);
+	v->hdr = TlDup(tl, buf);
+	bprintf(buf, "VRT_GetHdr(sp, %s, \"%s\")", v->http, v->hdr);
 	v->rname = TlDup(tl, buf);
 
-	bprintf(buf, "VRT_SetHdr(sp, %s, \"\\%03o%s:\", ",
-	    v->hdr, (unsigned)l, v->name + vh->len);
+	bprintf(buf, "VRT_SetHdr(sp, %s, \"%s\", ", v->http, v->hdr);
 	v->lname = TlDup(tl, buf);
 
 	sym = VCC_AddSymbolTok(tl, t, SYM_VAR);



From phk at varnish-cache.org  Mon Mar 14 09:48:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 10:48:48 +0100
Subject: [master] ee9e803 Add support for passing a HTTP header-identity to a
	VMOD function.
Message-ID: <E1Pz4OO-0003yh-Da@project.varnish-software.com>

commit ee9e8034a5dc799f5f467f51cf712d5e3f537a98
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 09:45:47 2011 +0000

    Add support for passing a HTTP header-identity to a VMOD function.

diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c
index 1a5b074..de2df31 100644
--- a/lib/libvcl/vcc_expr.c
+++ b/lib/libvcl/vcc_expr.c
@@ -576,6 +576,32 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 			SkipToken(tl, ID);
 			if (*p != '\0')
 				SkipToken(tl, ',');
+		} else if (fmt == HEADER) {
+			const struct var *v;
+			sym = VCC_FindSymbol(tl, tl->t, SYM_NONE);
+			vcc_AddUses(tl, tl->t, sym->r_methods, "Not available");
+			ERRCHK(tl);
+			SkipToken(tl, ID);
+			if (sym == NULL) {
+				vsb_printf(tl->sb, "Symbol not found.\n");
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
+			if (sym->kind != SYM_VAR) {
+				vsb_printf(tl->sb, "Wrong kind of symbol.\n");
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
+			AN(sym->var);
+			v = sym->var;
+			if (v->http == NULL) {
+				vsb_printf(tl->sb,
+				    "Variable not an HTTP header.\n");
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
+			e1 = vcc_mk_expr(VOID, "%s, \"%s\"", v->http, v->hdr);
+			p += strlen(p) + 1;
 		} else {
 			vcc_expr0(tl, &e1, fmt);
 			ERRCHK(tl);
diff --git a/lib/libvmod_std/vmod.py b/lib/libvmod_std/vmod.py
index a799dd8..4ad37d0 100755
--- a/lib/libvmod_std/vmod.py
+++ b/lib/libvmod_std/vmod.py
@@ -58,7 +58,7 @@ ctypes = {
 	'REAL':		"double",
 	'DURATION':	"double",
 	'INT':		"int",
-	'HEADER':	"const char *",
+	'HEADER':	"enum gethdr_e, const char *",
 	'PRIV_VCL':	"struct vmod_priv *",
 	'PRIV_CALL':	"struct vmod_priv *",
 	'VOID':		"void",
@@ -281,8 +281,8 @@ fh.write("\n");
 fh.write(plist)
 
 
-fc.write('#include "vcc_if.h"\n')
 fc.write('#include "vrt.h"\n')
+fc.write('#include "vcc_if.h"\n')
 fc.write("\n");
 
 fc.write("\n");
diff --git a/lib/libvmod_std/vmod_std_conversions.c b/lib/libvmod_std/vmod_std_conversions.c
index f17b752..1167f55 100644
--- a/lib/libvmod_std/vmod_std_conversions.c
+++ b/lib/libvmod_std/vmod_std_conversions.c
@@ -33,6 +33,7 @@
 
 #include "../../bin/varnishd/cache.h"
 
+#include "vrt.h"
 #include "vcc_if.h"
 
 double __match_proto__()



From phk at varnish-cache.org  Mon Mar 14 09:48:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 10:48:48 +0100
Subject: [master] 395a5c8 Add std.collect() which will collect multiple HTTP
	headers into one single header.
Message-ID: <E1Pz4OO-0003yr-KD@project.varnish-software.com>

commit 395a5c8deee8cd9410964ccbfd9cce9843cbb3a2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 09:47:16 2011 +0000

    Add std.collect() which will collect multiple HTTP headers into
    one single header.
    
    Example usage:
    
    	import std;
    
    	sub vcl_recv {
    		std.collect(req.http.foo);
    	}
    	sub vcl_fetch {
    		std.collect(beresp.http.bar);
    	}
    
    Fixes: #866

diff --git a/bin/varnishtest/tests/m00006.vtc b/bin/varnishtest/tests/m00006.vtc
new file mode 100644
index 0000000..cdec1f9
--- /dev/null
+++ b/bin/varnishtest/tests/m00006.vtc
@@ -0,0 +1,40 @@
+# $Id$
+
+test "test vmod_std.collect()"
+
+server s1 {
+	rxreq
+	expect req.url == "/1"
+	expect req.http.foo == "1, 2"
+	txresp -hdr "bar: a" -hdr "bar: b" -bodylen 1
+
+} -start
+
+varnish v1 -vcl+backend {
+	import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+
+	sub vcl_recv {
+		std.collect(req.http.foo);
+	}
+	sub vcl_fetch {
+		std.collect(beresp.http.bar);
+	}
+} -start
+
+client c1 {
+	txreq  -url "/1" -hdr "Foo: 1" -hdr "Foo: 2"
+	rxresp 
+	expect resp.http.bar == "a, b"
+	expect resp.status == 200
+	expect resp.bodylen == 1
+} -run
+
+varnish v1 -badvcl {
+	import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+
+	backend b { .host = "127.0.0.1"; }
+
+	sub vcl_recv {
+		std.collect(beresp.http.bar);
+	}
+}
diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index d77d49a..121b375 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -36,3 +36,4 @@ Function VOID syslog(INT, STRING_LIST)
 Function STRING fileread(PRIV_CALL, STRING)
 Function STRING author(ENUM { phk, des, kristian, mithrandir })
 Function DURATION duration(STRING, DURATION)
+Function VOID collect(HEADER)
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 4be07c7..5a132ac 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -176,3 +176,16 @@ vmod_author(struct sess *sp, const char *id)
 		return ("Tollef");
 	WRONG("Illegal VMOD enum");
 }
+
+void __match_proto__()
+vmod_collect(struct sess *sp, enum gethdr_e e, const char *h)
+{
+	(void)e;
+	(void)sp;
+	(void)h;
+	if (e == HDR_REQ)
+		http_CollectHdr(sp->http, h);
+	else if (e == HDR_BERESP)
+		http_CollectHdr(sp->wrk->beresp, h);
+}
+



From phk at varnish-cache.org  Mon Mar 14 09:53:00 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 10:53:00 +0100
Subject: [master] 6fe073a Don't acces symbol until we are sure we have one.
Message-ID: <E1Pz4SS-0004OR-5e@project.varnish-software.com>

commit 6fe073aee49a1fe48b182e0cfffb0bfe27c6c684
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 09:52:45 2011 +0000

    Don't acces symbol until we are sure we have one.

diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c
index de2df31..952c2a7 100644
--- a/lib/libvcl/vcc_expr.c
+++ b/lib/libvcl/vcc_expr.c
@@ -579,7 +579,6 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 		} else if (fmt == HEADER) {
 			const struct var *v;
 			sym = VCC_FindSymbol(tl, tl->t, SYM_NONE);
-			vcc_AddUses(tl, tl->t, sym->r_methods, "Not available");
 			ERRCHK(tl);
 			SkipToken(tl, ID);
 			if (sym == NULL) {
@@ -587,6 +586,7 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 				vcc_ErrWhere(tl, tl->t);
 				return;
 			}
+			vcc_AddUses(tl, tl->t, sym->r_methods, "Not available");
 			if (sym->kind != SYM_VAR) {
 				vsb_printf(tl->sb, "Wrong kind of symbol.\n");
 				vcc_ErrWhere(tl, tl->t);



From phk at varnish-cache.org  Mon Mar 14 10:21:26 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 11:21:26 +0100
Subject: [master] 13d6057 Make foo.grace and foo.keep return 0. as lowest
	value. foo.ttl still returns -1. to close gap due to integer
	second timestamps.
Message-ID: <E1Pz4ty-0006cL-BN@project.varnish-software.com>

commit 13d6057a61d010b015e58ae97b30b1ac9edad254
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 10:16:12 2011 +0000

    Make foo.grace and foo.keep return 0. as lowest value.
    foo.ttl still returns -1. to close gap due to integer second timestamps.

diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index 3a691a6..f71a630 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -73,11 +73,11 @@ EXP_Clr(struct exp *e)
 	e->keep = -1;
 }
 
-#define EXP_ACCESS(fld, extra)					\
+#define EXP_ACCESS(fld, low_val, extra)				\
 	double							\
 	EXP_Get_##fld(const struct exp *e)			\
 	{							\
-		return (e->fld > 0. ? e->fld : -1.);		\
+		return (e->fld > 0. ? e->fld : low_val);	\
 	}							\
 								\
 	void							\
@@ -91,9 +91,9 @@ EXP_Clr(struct exp *e)
 		}						\
 	}							\
 
-EXP_ACCESS(ttl, (e->grace = e->keep = -1.))
-EXP_ACCESS(grace,)
-EXP_ACCESS(keep,)
+EXP_ACCESS(ttl, -1., (e->grace = e->keep = -1.))
+EXP_ACCESS(grace, 0., )
+EXP_ACCESS(keep, 0.,)
 
 /*--------------------------------------------------------------------
  * Calculate when an object is out of ttl or grace, possibly constrained
diff --git a/bin/varnishtest/tests/b00030.vtc b/bin/varnishtest/tests/b00030.vtc
index 787aa0e..8cc5c7f 100644
--- a/bin/varnishtest/tests/b00030.vtc
+++ b/bin/varnishtest/tests/b00030.vtc
@@ -21,5 +21,5 @@ varnish v1 -vcl+backend {
 client c1 {
 	txreq
 	rxresp
-	expect resp.msg == "-1.000"
+	expect resp.msg == "0.000"
 } -run



From phk at varnish-cache.org  Mon Mar 14 10:21:26 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 11:21:26 +0100
Subject: [master] 3e087be Kill objects on "ttl + max(grace,
	keep)" rather than "ttl + grace + keep"
Message-ID: <E1Pz4ty-0006cU-Sw@project.varnish-software.com>

commit 3e087be3c7fd588a88235bee1998df7c132a4bea
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 10:21:02 2011 +0000

    Kill objects on "ttl + max(grace,keep)" rather than "ttl + grace + keep"

diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index f71a630..ec6ff66 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -110,7 +110,7 @@ EXP_Keep(const struct sess *sp, const struct object *o)
 		r = o->exp.keep;
 	if (sp != NULL && sp->exp.keep > 0. && sp->exp.keep < r)
 		r = sp->exp.keep;
-	return (EXP_Grace(sp, o) + r);
+	return (EXP_Ttl(sp, o) + r);
 }
 
 double
@@ -145,7 +145,7 @@ static int
 update_object_when(const struct object *o)
 {
 	struct objcore *oc;
-	double when;
+	double when, w2;
 
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	oc = o->objcore;
@@ -153,6 +153,9 @@ update_object_when(const struct object *o)
 	Lck_AssertHeld(&exp_mtx);
 
 	when = EXP_Keep(NULL, o);
+	w2 = EXP_Grace(NULL, o);
+	if (w2 > when)
+		when = w2;
 	assert(!isnan(when));
 	if (when == oc->timer_when)
 		return (0);



From phk at varnish-cache.org  Mon Mar 14 10:47:58 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Mar 2011 11:47:58 +0100
Subject: [master] 8ef6c17 Don't skip in the argument string,
	HEADER is a simple token.
Message-ID: <E1Pz5Je-0008EP-5a@project.varnish-software.com>

commit 8ef6c17500e916b52adda44f785ebef4bfe0e68a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 14 10:47:34 2011 +0000

    Don't skip in the argument string, HEADER is a simple token.

diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c
index 952c2a7..9261963 100644
--- a/lib/libvcl/vcc_expr.c
+++ b/lib/libvcl/vcc_expr.c
@@ -601,7 +601,8 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 				return;
 			}
 			e1 = vcc_mk_expr(VOID, "%s, \"%s\"", v->http, v->hdr);
-			p += strlen(p) + 1;
+			if (*p != '\0')
+				SkipToken(tl, ',');
 		} else {
 			vcc_expr0(tl, &e1, fmt);
 			ERRCHK(tl);



From tfheen at varnish-cache.org  Mon Mar 14 12:31:03 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Mar 2011 13:31:03 +0100
Subject: [master] 28dafef Add panic.show and panic.clear CLI commands
Message-ID: <E1Pz6vP-0007sd-Mf@project.varnish-software.com>

commit 28dafefee8dd72f52fad515e24add2f44905d1a9
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 10 13:34:58 2011 +0100

    Add panic.show and panic.clear CLI commands
    
    panic.show shows the last panic from the child
    panic.clear removes it.

diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
index dae6af5..056a63c 100644
--- a/bin/varnishd/cache_cli.c
+++ b/bin/varnishd/cache_cli.c
@@ -1,6 +1,7 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
  * Copyright (c) 2006-2010 Redpill Linpro AS
+ * Copyright (c) 2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/mgt_child.c b/bin/varnishd/mgt_child.c
index d8cbc59..a670a36 100644
--- a/bin/varnishd/mgt_child.c
+++ b/bin/varnishd/mgt_child.c
@@ -91,6 +91,8 @@ static struct vev	*ev_poker;
 static struct vev	*ev_listen;
 static struct vlu	*vlu;
 
+static struct vsb *child_panic = NULL;
+
 /*--------------------------------------------------------------------
  * Track the highest file descriptor the parent knows is being used.
  *
@@ -433,6 +435,31 @@ mgt_report_panic(pid_t r)
 	    (intmax_t)r, vsm_head->panicstr);
 }
 
+static void
+mgt_save_panic(void)
+{
+	char time_str[30];
+	if (vsm_head->panicstr[0] == '\0')
+		return;
+
+	if (child_panic)
+		vsb_delete(child_panic);
+	child_panic = vsb_newauto();
+	XXXAN(child_panic);
+	TIM_format(TIM_real(), time_str);
+	vsb_printf(child_panic, "Last panic at: %s\n", time_str);
+	vsb_cat(child_panic, vsm_head->panicstr);
+	vsb_finish(child_panic);
+	AZ(vsb_overflowed(child_panic));
+}
+
+static void
+mgt_clear_panic(void)
+{
+	vsb_delete(child_panic);
+	child_panic = NULL;
+}
+
 /*--------------------------------------------------------------------*/
 
 static int
@@ -478,6 +505,7 @@ mgt_sigchld(const struct vev *e, int what)
 	vsb_delete(vsb);
 
 	mgt_report_panic(r);
+	mgt_save_panic();
 
 	child_pid = -1;
 
@@ -614,3 +642,33 @@ mcf_server_status(struct cli *cli, const char * const *av, void *priv)
 	(void)priv;
 	cli_out(cli, "Child in state %s", ch_state[child_state]);
 }
+
+void
+mcf_panic_show(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)av;
+	(void)priv;
+
+	if (!child_panic) {
+	  cli_result(cli, CLIS_CANT);
+	  cli_out(cli, "Child has not panicked or panic has been cleared");
+	  return;
+	}
+
+	cli_out(cli, "%s\n", vsb_data(child_panic));
+}
+
+void
+mcf_panic_clear(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)av;
+	(void)priv;
+
+	if (!child_panic) {
+	  cli_result(cli, CLIS_CANT);
+	  cli_out(cli, "No panic to clear");
+	  return;
+	}
+
+	mgt_clear_panic();
+}
diff --git a/bin/varnishd/mgt_cli.c b/bin/varnishd/mgt_cli.c
index 7f719a0..0b364fd 100644
--- a/bin/varnishd/mgt_cli.c
+++ b/bin/varnishd/mgt_cli.c
@@ -127,6 +127,8 @@ static struct cli_proto cli_proto[] = {
 	{ CLI_VCL_SHOW,		"", mcf_config_show, NULL },
 	{ CLI_PARAM_SHOW,	"", mcf_param_show, NULL },
 	{ CLI_PARAM_SET,	"", mcf_param_set, NULL },
+	{ CLI_PANIC_SHOW,	"", mcf_panic_show, NULL },
+	{ CLI_PANIC_CLEAR,	"", mcf_panic_clear, NULL },
 	{ NULL }
 };
 
diff --git a/bin/varnishd/mgt_cli.h b/bin/varnishd/mgt_cli.h
index c4bfd5a..f7a1797 100644
--- a/bin/varnishd/mgt_cli.h
+++ b/bin/varnishd/mgt_cli.h
@@ -32,6 +32,8 @@
 /* mgt_child.c */
 cli_func_t mcf_server_startstop;
 cli_func_t mcf_server_status;
+cli_func_t mcf_panic_show;
+cli_func_t mcf_panic_clear;
 
 /* mgt_param.c */
 cli_func_t mcf_param_show;
diff --git a/bin/varnishtest/tests/v00010.vtc b/bin/varnishtest/tests/v00010.vtc
index 4ef81f2..259894f 100644
--- a/bin/varnishtest/tests/v00010.vtc
+++ b/bin/varnishtest/tests/v00010.vtc
@@ -45,3 +45,7 @@ client c1 {
 	rxresp
 	expect resp.http.foo == "foo"
 } -run
+
+varnish v1 -cliok "panic.show"
+varnish v1 -cliok "panic.clear"
+varnish v1 -clierr 300 "panic.clear"
diff --git a/include/cli.h b/include/cli.h
index e16ff60..3734b0b 100644
--- a/include/cli.h
+++ b/include/cli.h
@@ -230,6 +230,18 @@
 	"\tAuthenticate.",						\
 	1, 1
 
+#define CLI_PANIC_SHOW							\
+	"panic.show",							\
+	"panic.show",							\
+	"\tReturn the last panic, if any.",				\
+	0, 0
+
+#define CLI_PANIC_CLEAR							\
+	"panic.clear",							\
+	"panic.clear",							\
+	"\tClear the last panic, if any.",				\
+	0, 0
+
 #define CLI_HIDDEN(foo, min_arg, max_arg)				\
 	foo, NULL, NULL, min_arg, max_arg,
 



From tfheen at varnish-cache.org  Mon Mar 14 12:31:05 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Mar 2011 13:31:05 +0100
Subject: [master] 5101756 Look for C99 compilers somewhat harder
Message-ID: <E1Pz6vR-0007sp-1O@project.varnish-software.com>

commit 5101756880bbe0369c679ea13558241b4fd1de48
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Mar 14 12:56:59 2011 +0100

    Look for C99 compilers somewhat harder
    
    RHEL5/CentOS 5 has an ancient version of autoconf which doesn't pick
    up the necessary runes to enable C99 mode, leading to compilation
    failures.  Hack around this in configure.ac

diff --git a/configure.ac b/configure.ac
index 22f0651..b954106 100644
--- a/configure.ac
+++ b/configure.ac
@@ -20,8 +20,21 @@ AM_INIT_AUTOMAKE([foreign])
 AC_GNU_SOURCE
 AC_PROG_CC
 AC_PROG_CC_STDC
-if test "x$ac_cv_prog_cc_c99" = xno; then
-  AC_MSG_ERROR([Could not find a C99 compatible compiler])
+if test "x$ac_cv_prog_cc_c99" = "xno" || test "x$ac_cv_prog_cc_c99" = "x"; then
+# We might be on RHEL5 with a git checkout and so broken
+# autoconf. Check if CC is gcc and if it bails when given -std=gnu99.
+# If not, use that.  Yuck.
+  if test "x$ac_cv_c_compiler_gnu" = "yes"; then
+     CC="$CC -std=gnu99"
+     AC_RUN_IFELSE(
+	[AC_LANG_PROGRAM([],[[
+return 0;
+    ]])],
+    [],
+    [AC_MSG_ERROR([Could not find a C99 compatible compiler])])
+  else
+    AC_MSG_ERROR([Could not find a C99 compatible compiler])
+  fi
 fi
 AC_PROG_CPP
 



From tfheen at varnish-cache.org  Mon Mar 14 12:40:49 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Mar 2011 13:40:49 +0100
Subject: [master] 30dc7dd Fix typo
Message-ID: <E1Pz74r-0000iH-Sw@project.varnish-software.com>

commit 30dc7dd090df8ca8f14c534e6f042fe1d987f4a2
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Mar 14 13:39:57 2011 +0100

    Fix typo

diff --git a/configure.ac b/configure.ac
index b954106..9849783 100644
--- a/configure.ac
+++ b/configure.ac
@@ -24,7 +24,7 @@ if test "x$ac_cv_prog_cc_c99" = "xno" || test "x$ac_cv_prog_cc_c99" = "x"; then
 # We might be on RHEL5 with a git checkout and so broken
 # autoconf. Check if CC is gcc and if it bails when given -std=gnu99.
 # If not, use that.  Yuck.
-  if test "x$ac_cv_c_compiler_gnu" = "yes"; then
+  if test "x$ac_cv_c_compiler_gnu" = "xyes"; then
      CC="$CC -std=gnu99"
      AC_RUN_IFELSE(
 	[AC_LANG_PROGRAM([],[[



From tfheen at varnish-cache.org  Mon Mar 14 13:33:06 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Mar 2011 14:33:06 +0100
Subject: [master] 17d7143 Validate EXTRA_DEVELOPER_CFLAGS and -fstack-protector
Message-ID: <E1Pz7tS-000639-7t@project.varnish-software.com>

commit 17d7143cd1b00f092edc8787eb26261d1c27af02
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Mar 14 14:21:20 2011 +0100

    Validate EXTRA_DEVELOPER_CFLAGS and -fstack-protector
    
    This fixes buildbot problems on Solaris, since the solaris buildbot
    runs an ancient gcc.

diff --git a/configure.ac b/configure.ac
index 9849783..1f19044 100644
--- a/configure.ac
+++ b/configure.ac
@@ -466,12 +466,34 @@ AC_ARG_ENABLE(diagnostics,
 # --enable-extra-developer-warnings
 AC_ARG_ENABLE(extra-developer-warnings,
 	AS_HELP_STRING([--enable-extra-developer-warnings],[enable even stricter warnings (default is NO)]),
-	CFLAGS="${CFLAGS} ${EXTRA_DEVELOPER_CFLAGS}")
+	[],
+	[enable_extra_developer_warnings=no])
+
+if test "x$enable_stack_protector" != "xno"; then
+	save_CFLAGS="$CFLAGS"
+	CFLAGS="${CFLAGS} ${EXTRA_DEVELOPER_CFLAGS}"
+	AC_COMPILE_IFELSE(
+		[AC_LANG_PROGRAM([],[],[])],
+		[],
+		[AC_MSG_WARN([All of ${EXTRA_DEVELOPER_CFLAGS} not supported, disabling])
+		    CFLAGS="$save_CFLAGS"])
+fi
 
 # --enable-stack-protector
 AC_ARG_ENABLE(stack-protector,
 	AS_HELP_STRING([--enable-stack-protector],[enable stack protector (default is NO)]),
-	CFLAGS="${CFLAGS} -fstack-protector-all")
+	[],
+	[enable_stack_protector=no])
+
+if test "x$enable_stack_protector" != "xno"; then
+	save_CFLAGS="$CFLAGS"
+	CFLAGS="${CFLAGS} -fstack-protector-all"
+	AC_COMPILE_IFELSE(
+		[AC_LANG_PROGRAM([],[],[])],
+		[],
+		[AC_MSG_WARN([-fstack-protector not supported, disabling])
+		    CFLAGS="$save_CFLAGS"])
+fi
 
 # --enable-tests
 AC_ARG_ENABLE(tests,



From phk at varnish-cache.org  Tue Mar 15 08:27:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Mar 2011 09:27:17 +0100
Subject: [master] f04cff3 still has a chance to compile.
Message-ID: <E1PzPb3-0000EJ-Iq@project.varnish-software.com>

commit f04cff326121c57593d16f19066a0525c4143a0a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 15 08:26:25 2011 +0000

    still has a chance to compile.

diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index e43f2c4..4bdb9ff 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -203,7 +203,12 @@ vca_acct(void *arg)
 	socklen_t l;
 	struct sockaddr_storage addr_s;
 	struct sockaddr *addr;
-	double send_timeout = 0, sess_timeout = 0;
+#ifdef SO_RCVTIMEO_WORKS
+	double sess_timeout = 0;
+#endif
+#ifdef SO_SNDTIMEO_WORKS
+	double send_timeout = 0;
+#endif
 	int i;
 	struct pollfd *pfd;
 	struct listen_sock *ls;



From phk at varnish-cache.org  Tue Mar 15 08:27:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Mar 2011 09:27:17 +0100
Subject: [master] 9c47426 Cast bit numbers before printf.
Message-ID: <E1PzPb3-0000ER-U1@project.varnish-software.com>

commit 9c474268879b411e0a9da1ee72b109fef0900163
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 15 08:27:05 2011 +0000

    Cast bit numbers before printf.

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 5f8cd1b..7d0dba2 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -561,11 +561,14 @@ cmd_http_gunzip_body(CMD_ARGS)
 	vtc_dump(hp->vl, 4, "body", hp->body, hp->bodyl);
 	bprintf(hp->bodylen, "%u", hp->bodyl);
 	vtc_log(hp->vl, 4, "startbit = %ju %ju/%ju",
-	    vz.start_bit, vz.start_bit >> 3, vz.start_bit & 7);
+	    (uintmax_t)vz.start_bit,
+	    (uintmax_t)vz.start_bit >> 3, (uintmax_t)vz.start_bit & 7);
 	vtc_log(hp->vl, 4, "lastbit = %ju %ju/%ju",
-	    vz.last_bit, vz.last_bit >> 3, vz.last_bit & 7);
+	    (uintmax_t)vz.last_bit,
+	    (uintmax_t)vz.last_bit >> 3, (uintmax_t)vz.last_bit & 7);
 	vtc_log(hp->vl, 4, "stopbit = %ju %ju/%ju",
-	    vz.stop_bit, vz.stop_bit >> 3, vz.stop_bit & 7);
+	    (uintmax_t)vz.stop_bit,
+	    (uintmax_t)vz.stop_bit >> 3, (uintmax_t)vz.stop_bit & 7);
 	if (i != Z_STREAM_END)
 		vtc_log(hp->vl, 0, "Gunzip error = %d (%s) in:%jd out:%jd",
 		    i, vz.msg, (intmax_t)vz.total_in, (intmax_t)vz.total_out);



From phk at varnish-cache.org  Tue Mar 15 09:21:30 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Mar 2011 10:21:30 +0100
Subject: [master] 516e69e Fix an indentation bug,
	which caused too much input data to be made "pending" if an input
	snippet ended with an XML comment or CDATA block.
Message-ID: <E1PzQRW-0003v8-Tp@project.varnish-software.com>

commit 516e69e035c431c782ea1cbfeab36cdf6e0042b7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 15 09:20:03 2011 +0000

    Fix an indentation bug, which caused too much input data to be made
    "pending" if an input snippet ended with an XML comment or CDATA block.
    
    Not spotted by:	FlexeLint
    
    Fixes: #873

diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
index d3562fa..b80de68 100644
--- a/bin/varnishd/cache_esi_fetch.c
+++ b/bin/varnishd/cache_esi_fetch.c
@@ -226,6 +226,7 @@ vfp_esi_bytes_ug(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		bytes -= w;
 		vef->bufp = ibuf;
 		VEP_parse(sp, ibuf, w);
+		assert(vef->bufp >= ibuf && vef->bufp <= ibuf + w);
 		if (vef->error) {
 			errno = vef->error;
 			return (-1);
diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
index e979b44..5e1915e 100644
--- a/bin/varnishd/cache_esi_parse.c
+++ b/bin/varnishd/cache_esi_parse.c
@@ -698,6 +698,7 @@ VEP_parse(const struct sess *sp, const char *p, size_t l)
 					vep->until_p = vep->until = "-->";
 					vep->until_s = VEP_NEXTTAG;
 					vep->state = VEP_UNTIL;
+					vep_mark_verbatim(vep, p);
 					break;
 				}
 				p++;
@@ -943,9 +944,9 @@ VEP_parse(const struct sess *sp, const char *p, size_t l)
 					vep->state = vep->until_s;
 					break;
 				}
+			}
 			if (p == e && !vep->remove)
 				vep_mark_verbatim(vep, p);
-			}
 		} else {
 			Debug("*** Unknown state %s\n", vep->state);
 			INCOMPL();
diff --git a/bin/varnishtest/tests/r00873.vtc b/bin/varnishtest/tests/r00873.vtc
new file mode 100644
index 0000000..6797559
--- /dev/null
+++ b/bin/varnishtest/tests/r00873.vtc
@@ -0,0 +1,29 @@
+# $Id$
+
+test "Ticket #873"
+
+server s1 {
+	rxreq 
+	txresp -nolen -hdr "Transfer-encoding: chunked"
+	chunked {<!-- jfsljfslkjfslkfjsljflskjflskjdj -->}
+	chunked {<![CDATA[fooooooooooooooooooobar]]>}
+	chunkedlen 0
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_fetch {
+		set beresp.do_esi = true;
+		set beresp.do_gzip = true;
+	}
+} -start 
+
+client c1 {
+	txreq  -hdr "Accept-Encoding: gzip"
+	rxresp
+	expect resp.http.content-encoding == gzip
+	gunzip
+	expect resp.status == 200
+	expect resp.bodylen == 75
+} -run
+
+varnish v1 -expect esi_errors == 0



From perbu at varnish-cache.org  Tue Mar 15 09:57:00 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Tue, 15 Mar 2011 10:57:00 +0100
Subject: [master] 18db545 change send_timeout doc to reflect reality rather
	then intention
Message-ID: <E1PzQzs-0006JD-MD@project.varnish-software.com>

commit 18db5450ffb31b04bbd2db2300737823f26932d1
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Mar 15 10:56:52 2011 +0100

    change send_timeout doc to reflect reality rather then intention

diff --git a/bin/varnishd/mgt_param.c b/bin/varnishd/mgt_param.c
index ed70fc7..a1f33d0 100644
--- a/bin/varnishd/mgt_param.c
+++ b/bin/varnishd/mgt_param.c
@@ -611,8 +611,8 @@ static const struct parspec input_parspec[] = {
 		"60", "seconds" },
 	{ "send_timeout", tweak_timeout, &master.send_timeout, 0, 0,
 		"Send timeout for client connections. "
-		"If no data has been sent to the client in this many seconds, "
-		"the session is closed.\n"
+		"If the HTTP response hasn't been transmitted in this many\n"
+                "seconds the session is closed. \n"
 		"See setsockopt(2) under SO_SNDTIMEO for more information.",
 		DELAYED_EFFECT,
 		"60", "seconds" },



From phk at varnish-cache.org  Tue Mar 15 10:01:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Mar 2011 11:01:40 +0100
Subject: [master] f6a5e7e Allways call with VGZ_FINISH for proper gzip
	termination.
Message-ID: <E1PzR4O-0006oL-Lv@project.varnish-software.com>

commit f6a5e7e0f8e23b3b19a296561d25818844226c6d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 15 10:00:30 2011 +0000

    Allways call with VGZ_FINISH for proper gzip termination.
    
    Fixes	#872

diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
index 5e1915e..937a857 100644
--- a/bin/varnishd/cache_esi_parse.c
+++ b/bin/varnishd/cache_esi_parse.c
@@ -1043,8 +1043,8 @@ VEP_Finish(const struct sess *sp)
 	if (vep->o_wait > 0) {
 		lcb = vep->cb(vep->sp, 0, VGZ_ALIGN);
 		vep_emit_common(vep, lcb - vep->o_last, vep->last_mark);
-		(void)vep->cb(vep->sp, 0, VGZ_FINISH);
 	}
+	(void)vep->cb(vep->sp, 0, VGZ_FINISH);
 
 	sp->wrk->vep = NULL;
 



From phk at varnish-cache.org  Tue Mar 15 10:01:42 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Mar 2011 11:01:42 +0100
Subject: [master] 03b5604 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1PzR4Q-0006oc-5n@project.varnish-software.com>

commit 03b56048dc7af6945715505e858d3b3c7caaffb9
Merge: f6a5e7e 18db545
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 15 10:01:35 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From kristian at varnish-cache.org  Tue Mar 15 11:45:34 2011
From: kristian at varnish-cache.org (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=)
Date: Tue, 15 Mar 2011 12:45:34 +0100
Subject: [master] 4340f0a varnishd.rst: Indent the literal block for a-e
Message-ID: <E1PzSgw-0004oB-Fp@project.varnish-software.com>

commit 4340f0a800977e3abf0720cabfaf04c6c6eb138d
Author: Kristian Lyngstol <kristian at bohemians.org>
Date:   Tue Mar 15 12:45:07 2011 +0100

    varnishd.rst: Indent the literal block for a-e
    
    Fixes rst-warning in the man-page.

diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index c929baf..42e14be 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -517,7 +517,7 @@ http_gzip_support
 
 	Enable gzip support. When enabled Varnish will compress uncompressed objects before they are stored in the cache. If a client does not support gzip encoding Varnish will uncompress compressed objects on demand. Varnish will also rewrite the Accept-Encoding header of clients indicating support for gzip to::
 
-	Accept-Encoding: gzip
+        	Accept-Encoding: gzip
 
 	Clients that do not support gzip will have their Accept-Encoding header removed. For more information on how gzip is implemented please see the chapter on gzip in the Varnish reference.
 



From tfheen at varnish-cache.org  Tue Mar 15 12:57:38 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 15 Mar 2011 13:57:38 +0100
Subject: [master] e0842f5 Fix build failures
Message-ID: <E1PzTog-00048b-2H@project.varnish-software.com>

commit e0842f52bbdc5c2af197399f661c5c1b4f117454
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 15 13:57:04 2011 +0100

    Fix build failures
    
    Get rid of a unused variable, add a missing prototype

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 4f3d013..3c5c7d3 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -56,6 +56,9 @@ SVNID("$Id$")
 		rl_callback_handler_remove(); \
 		exit(status); \
 	} while (0)
+
+void send_line(char *l);
+
 #else
 #define RL_EXIT(status) exit(status)
 #endif
@@ -193,7 +196,7 @@ pass(int sock)
 {
 	struct pollfd fds[2];
 	char buf[1024];
-	int i, n;
+	int i;
 	char *answer = NULL;
 	unsigned u, status;
 



From tfheen at varnish-cache.org  Tue Mar 15 13:00:27 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 15 Mar 2011 14:00:27 +0100
Subject: [master] f014b45 Fix up compilation error
Message-ID: <E1PzTrP-0004NE-Le@project.varnish-software.com>

commit f014b451d16511472914f944f604fe51bc23aa13
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 15 14:00:25 2011 +0100

    Fix up compilation error

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 3c5c7d3..be400d0 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -199,6 +199,9 @@ pass(int sock)
 	int i;
 	char *answer = NULL;
 	unsigned u, status;
+#ifndef HAVE_LIBEDIT
+	int n;
+#endif
 
 #ifdef HAVE_LIBEDIT
 	_line_sock = sock;



From tfheen at varnish-cache.org  Wed Mar 16 09:15:36 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 16 Mar 2011 10:15:36 +0100
Subject: [master] 8742421 svn_version.c is renamed to vcs_version.c,
	ignore the new name
Message-ID: <E1PzmpM-0002pI-Ve@project.varnish-software.com>

commit 87424219f1bec91b355121fc90f144f0d27e8985
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 16 10:13:36 2011 +0100

    svn_version.c is renamed to vcs_version.c, ignore the new name

diff --git a/.gitignore b/.gitignore
index b6c617e..2a94b5c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,7 +44,7 @@ TAGS
 /include/vcl_returns.h
 /include/vrt_obj.h
 /include/vrt_stv_var.h
-/lib/libvarnish/svn_version.c
+/lib/libvarnish/vcs_version.c
 /lib/libvcl/vcc_fixed_token.c
 /lib/libvcl/vcc_obj.c
 /lib/libvcl/vcc_token_defs.h



From tfheen at varnish-cache.org  Wed Mar 16 09:15:38 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 16 Mar 2011 10:15:38 +0100
Subject: [master] 1c0e14e Bump the value of http_resp_hdr_len and
	http_req_hdr_len
Message-ID: <E1PzmpO-0002pQ-6m@project.varnish-software.com>

commit 1c0e14e9fd32c6c5c7914026acd20b046f275c37
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 16 10:14:36 2011 +0100

    Bump the value of http_resp_hdr_len and http_req_hdr_len
    
    We have seen larger than 512 values in the wild, so increase the
    default values a bit to give us breathing room.

diff --git a/bin/varnishd/mgt_param.c b/bin/varnishd/mgt_param.c
index a1f33d0..d0e9c20 100644
--- a/bin/varnishd/mgt_param.c
+++ b/bin/varnishd/mgt_param.c
@@ -525,7 +525,7 @@ static const struct parspec input_parspec[] = {
 		"Maximum length of any HTTP client request header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
-		"512", "bytes" },
+		"2048", "bytes" },
 	{ "http_req_size", tweak_uint, &master.http_req_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP client request we will deal "
@@ -541,7 +541,7 @@ static const struct parspec input_parspec[] = {
 		"Maximum length of any HTTP backend response header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
-		"512", "bytes" },
+		"2048", "bytes" },
 	{ "http_resp_size", tweak_uint, &master.http_resp_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP backend resonse we will deal "



From tfheen at varnish-cache.org  Wed Mar 16 09:56:14 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 16 Mar 2011 10:56:14 +0100
Subject: [master] 6c957b4 Make varnishadm able to find args automatically
Message-ID: <E1PznSg-00060r-1g@project.varnish-software.com>

commit 6c957b4d8acf6e1bea0814e2ef6b500fef7407b9
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 16 10:54:31 2011 +0100

    Make varnishadm able to find args automatically
    
    varnishadm already supported -n for getting the connection arguments.
    However, unlike the other tools, you needed to pass an -n argument, it
    did not default to anything.  It now uses the compiled-in default.
    
    Fixes: #875

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index be400d0..8d98181 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -355,7 +355,9 @@ main(int argc, char * const *argv)
 		if (sock < 0)
 			exit(2);
 	} else if (T_arg == NULL) {
-		usage();
+		sock = n_arg_sock("");
+		if (sock < 0)
+			exit(2);
 	} else {
 		assert(T_arg != NULL);
 		sock = cli_sock(T_arg, S_arg);



From tfheen at varnish-cache.org  Wed Mar 16 10:39:41 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 16 Mar 2011 11:39:41 +0100
Subject: [master] 7109360 Use printf rather than echo when writing default.vcl
Message-ID: <E1Pzo8j-0001mp-0Y@project.varnish-software.com>

commit 7109360d88af36c0b621fda81c4e93e8bdf609a4
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 16 11:37:16 2011 +0100

    Use printf rather than echo when writing default.vcl
    
    echo -e isn't in POSIX so it's not supported by all shells.  Use
    printf instead of echo to work around this.

diff --git a/etc/Makefile.am b/etc/Makefile.am
index c68f266..a250571 100644
--- a/etc/Makefile.am
+++ b/etc/Makefile.am
@@ -6,7 +6,7 @@ DISTCLEANFILES = default.vcl
 dist_varnishconf_DATA = default.vcl
 
 default.vcl:	$(top_srcdir)/bin/varnishd/default.vcl
-	@echo -e "This is a basic VCL configuration file for varnish.  See the vcl(7)\n\
+	@printf "This is a basic VCL configuration file for varnish.  See the vcl(7)\n\
 man page for details on VCL syntax and semantics.\n\
 \n\
 Default backend definition.  Set this to point to your content\n\



From tfheen at varnish-cache.org  Wed Mar 16 10:39:41 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 16 Mar 2011 11:39:41 +0100
Subject: [master] 5ec2bb0 Get rid of temporary intermediate file
Message-ID: <E1Pzo8j-0001n3-UY@project.varnish-software.com>

commit 5ec2bb09c50629eafaabc9f6d6f43ae145e4f254
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 16 11:39:14 2011 +0100

    Get rid of temporary intermediate file
    
    Simplify generation of default.vcl somewhat.

diff --git a/etc/Makefile.am b/etc/Makefile.am
index a250571..24faf56 100644
--- a/etc/Makefile.am
+++ b/etc/Makefile.am
@@ -6,7 +6,7 @@ DISTCLEANFILES = default.vcl
 dist_varnishconf_DATA = default.vcl
 
 default.vcl:	$(top_srcdir)/bin/varnishd/default.vcl
-	@printf "This is a basic VCL configuration file for varnish.  See the vcl(7)\n\
+	( printf "This is a basic VCL configuration file for varnish.  See the vcl(7)\n\
 man page for details on VCL syntax and semantics.\n\
 \n\
 Default backend definition.  Set this to point to your content\n\
@@ -19,7 +19,6 @@ backend default {\n\
 \n\
 Below is a commented-out copy of the default VCL logic.  If you\n\
 redefine any of these subroutines, the built-in logic will be\n\
-appended to your code.\n" > tmp.vcl
-	@sed -n '/vcl_recv/,$$p' $(top_srcdir)/bin/varnishd/default.vcl >> tmp.vcl
-	@sed 's/^\(.*\)$$/# \1/' tmp.vcl > default.vcl
-	@rm tmp.vcl
+appended to your code.\n" ; \
+	sed -n '/vcl_recv/,$$p' $(top_srcdir)/bin/varnishd/default.vcl ) | \
+	sed 's/^\(.*\)$$/# \1/' > default.vcl



From martin at varnish-cache.org  Tue Mar 22 08:07:49 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Tue, 22 Mar 2011 09:07:49 +0100
Subject: [master] cab6b85 Add -D macro option to varnishtest
Message-ID: <E1Q1wd3-0001B6-6P@project.varnish-software.com>

commit cab6b85befea47fdff8cdd61ce456d97a3ba0485
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Mar 18 14:30:09 2011 +0100

    Add -D macro option to varnishtest

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index f601dbc..909e588 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -76,6 +76,8 @@ struct macro {
 
 static VTAILQ_HEAD(,macro) macro_list = VTAILQ_HEAD_INITIALIZER(macro_list);
 
+struct _extmacro_list extmacro_list = VTAILQ_HEAD_INITIALIZER(extmacro_list);
+
 static pthread_mutex_t		macro_mtx;
 
 static void
@@ -479,6 +481,7 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	char *cwd, *p;
 	char topbuild[BUFSIZ];
 	FILE *f;
+	struct extmacro *m;
 
 	vtc_loginit(logbuf, loglen);
 	vltop = vtc_logopen("top");
@@ -501,6 +504,10 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	AZ(chdir(tmpdir));
 	macro_def(vltop, NULL, "tmpdir", tmpdir);
 
+	/* Apply extmacro definitions */
+	VTAILQ_FOREACH(m, &extmacro_list, list)
+		macro_def(vltop, NULL, m->name, m->val);
+
 	/* Drop file to tell what was going on here */
 	f = fopen("INFO", "w");
 	AN(f);
diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h
index 843b8ec..28a5ca3 100644
--- a/bin/varnishtest/vtc.h
+++ b/bin/varnishtest/vtc.h
@@ -34,6 +34,7 @@
 #ifdef HAVE_PTHREAD_NP_H
 #include <pthread_np.h>
 #endif
+#include "vqueue.h"
 
 struct vsb;
 struct vtclog;
@@ -49,6 +50,15 @@ struct cmds {
 	cmd_f		*cmd;
 };
 
+struct extmacro {
+	VTAILQ_ENTRY(extmacro)	list;
+	char			*name;
+	char			*val;
+};
+
+VTAILQ_HEAD(_extmacro_list, extmacro);
+extern struct _extmacro_list extmacro_list;
+
 void parse_string(char *buf, const struct cmds *cmd, void *priv,
     struct vtclog *vl);
 
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index cd67225..07a876e 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -92,6 +92,32 @@ static int vtc_fail;
 static int leave_temp;
 
 /**********************************************************************
+ * Parse a -D option argument into a name/val pair, and insert
+ * into extmacro list
+ */
+
+int
+parse_D_opt(char *arg)
+{
+	int i;
+	char *p, *q;
+	struct extmacro *m;
+
+	p = arg;
+	q = strchr(p, '=');
+	if (!q)
+		return (0);
+	*q++ = '\0';
+	m = calloc(sizeof *m, 1);
+	AN(m);
+	REPLACE(m->name, p);
+	REPLACE(m->val, q);
+	VTAILQ_INSERT_TAIL(&extmacro_list, m, list);
+
+	return (1);
+}
+
+/**********************************************************************
  * Read a file into memory
  */
 
@@ -130,6 +156,7 @@ usage(void)
 {
 	fprintf(stderr, "usage: varnishtest [options] file ...\n");
 #define FMT "    %-28s # %s\n"
+	fprintf(stderr, FMT, "-D name=val", "Define macro for use in scripts");
 	fprintf(stderr, FMT, "-j jobs", "Run this many tests in parallel");
 	fprintf(stderr, FMT, "-k", "Continue on test failure");
 	fprintf(stderr, FMT, "-l", "Leave /tmp/vtc.* if test fails");
@@ -310,8 +337,15 @@ main(int argc, char * const *argv)
 
 	setbuf(stdout, NULL);
 	setbuf(stderr, NULL);
-	while ((ch = getopt(argc, argv, "j:klLn:qt:v")) != -1) {
+	while ((ch = getopt(argc, argv, "D:j:klLn:qt:v")) != -1) {
 		switch (ch) {
+		case 'D':
+			if (!parse_D_opt(optarg)) {
+				fprintf(stderr, "Cannot parse D opt '%s'\n", 
+					optarg);
+				exit (2);
+			}
+			break;
 		case 'j':
 			npar = strtoul(optarg, NULL, 0);
 			break;



From martin at varnish-cache.org  Tue Mar 22 08:07:49 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Tue, 22 Mar 2011 09:07:49 +0100
Subject: [master] 9b620f8 Apply extmacros first
Message-ID: <E1Q1wd3-0001BE-Es@project.varnish-software.com>

commit 9b620f86618318e48423b1e40afc5015d28746de
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 21 12:01:27 2011 +0100

    Apply extmacros first

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 909e588..1cc16cf 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -490,6 +490,10 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	init_macro();
 	init_sema();
 
+	/* Apply extmacro definitions */
+	VTAILQ_FOREACH(m, &extmacro_list, list)
+		macro_def(vltop, NULL, m->name, m->val);
+
 	/* We are still in bin/varnishtest at this point */
 	cwd = getcwd(NULL, PATH_MAX);
 	bprintf(topbuild, "%s/%s", cwd, TOP_BUILDDIR);
@@ -504,10 +508,6 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	AZ(chdir(tmpdir));
 	macro_def(vltop, NULL, "tmpdir", tmpdir);
 
-	/* Apply extmacro definitions */
-	VTAILQ_FOREACH(m, &extmacro_list, list)
-		macro_def(vltop, NULL, m->name, m->val);
-
 	/* Drop file to tell what was going on here */
 	f = fopen("INFO", "w");
 	AN(f);



From martin at varnish-cache.org  Tue Mar 22 08:07:49 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Tue, 22 Mar 2011 09:07:49 +0100
Subject: [master] 55e9e49 Change 'exit (' to 'exit('
Message-ID: <E1Q1wd3-0001BR-S8@project.varnish-software.com>

commit 55e9e499ffe8a2d085f33dc9f0d9cf2ddb73c936
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 21 15:48:42 2011 +0100

    Change 'exit (' to 'exit('

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 07a876e..3ae243e 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -233,7 +233,7 @@ tst_cb(const struct vev *ve, int what)
 			    jp->tst->filename, t);
 			if (!vtc_continue) {
 				/* XXX kill -9 other jobs ? */
-				exit (2);
+				exit(2);
 			}
 		} else if (vtc_verbosity) {
 			printf("#     top  TEST %s passed (%.3f)\n",
@@ -343,7 +343,7 @@ main(int argc, char * const *argv)
 			if (!parse_D_opt(optarg)) {
 				fprintf(stderr, "Cannot parse D opt '%s'\n", 
 					optarg);
-				exit (2);
+				exit(2);
 			}
 			break;
 		case 'j':
@@ -386,7 +386,7 @@ main(int argc, char * const *argv)
 			    *argv, strerror(errno));
 			if (vtc_continue)
 				continue;
-			exit (2);
+			exit(2);
 		}
 		ALLOC_OBJ(tp, TST_MAGIC);
 		AN(tp);



From martin at varnish-cache.org  Tue Mar 22 08:07:50 2011
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Tue, 22 Mar 2011 09:07:50 +0100
Subject: [master] 87587f1 Merge branch 'varnishtest'
Message-ID: <E1Q1wd4-0001Be-3d@project.varnish-software.com>

commit 87587f1fa2c360fa57833ab0b143a2a240105d2e
Merge: 5ec2bb0 55e9e49
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 21 15:58:31 2011 +0100

    Merge branch 'varnishtest'



From phk at varnish-cache.org  Tue Mar 22 09:02:38 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 22 Mar 2011 10:02:38 +0100
Subject: [master] 293f332 Deal with a couple of warnings/errors.
Message-ID: <E1Q1xU6-0000tt-RJ@project.varnish-software.com>

commit 293f332181cfe48c981a61fd8d93ede3c246708b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 22 09:01:58 2011 +0000

    Deal with a couple of warnings/errors.
    
    Please use autogen.des to test patches before committing.

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 3ae243e..20f7fac 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -96,10 +96,9 @@ static int leave_temp;
  * into extmacro list
  */
 
-int
+static int
 parse_D_opt(char *arg)
 {
-	int i;
 	char *p, *q;
 	struct extmacro *m;
 



From phk at varnish-cache.org  Tue Mar 22 11:59:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 22 Mar 2011 12:59:23 +0100
Subject: [master] a1a62dc Cut cnt_fetch() into two,
	one that fetches the header and one that fetches the body.
Message-ID: <E1Q20F9-0004YR-Q8@project.varnish-software.com>

commit a1a62dc724fe77ca3b6a5c6bc0e78742eec1522d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 22 11:58:08 2011 +0000

    Cut cnt_fetch() into two, one that fetches the header and one
    that fetches the body.
    
    vcl_fetch{} is called between them.
    
    If vcl_fetch{} results in restart or error, don't bother fetching
    the body, just summarily close the backend connection.

diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 9146392..e3cc4e2 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -430,12 +430,12 @@ cnt_error(struct sess *sp)
 }
 
 /*--------------------------------------------------------------------
- * Fetch an object from the backend, either for pass or for caching.
+ * Fetch response headers from the backend
  *
 DOT subgraph xcluster_fetch {
 DOT	fetch [
 DOT		shape=ellipse
-DOT		label="fetch from backend\n(find obj.ttl)"
+DOT		label="fetch hdr\nfrom backend\n(find obj.ttl)"
 DOT	]
 DOT	vcl_fetch [
 DOT		shape=record
@@ -446,11 +446,10 @@ DOT	fetch_pass [
 DOT		shape=ellipse
 DOT		label="obj.pass=true"
 DOT	]
-DOT	vcl_fetch -> fetch_pass [label="pass",style=bold,color=red]
+DOT	vcl_fetch -> fetch_pass [label="hit_for_pass",style=bold,color=red]
 DOT }
-DOT fetch_pass -> deliver [style=bold,color=red]
-DOT vcl_fetch -> deliver [label="deliver",style=bold,color=blue,weight=2]
-DOT vcl_fetch -> recv [label="restart"]
+DOT fetch_pass -> fetchbody [style=bold,color=red]
+DOT vcl_fetch -> fetchbody [label="deliver",style=bold,color=blue,weight=2]
 DOT vcl_fetch -> rstfetch [label="restart",color=purple]
 DOT rstfetch [label="RESTART",shape=plaintext]
 DOT fetch -> errfetch
@@ -462,11 +461,6 @@ static int
 cnt_fetch(struct sess *sp)
 {
 	int i;
-	struct http *hp, *hp2;
-	char *b;
-	unsigned l, nhttp;
-	int varyl = 0, pass;
-	struct vsb *vary = NULL;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
@@ -489,21 +483,6 @@ cnt_fetch(struct sess *sp)
 		i = FetchHdr(sp);
 	}
 
-	/*
-	 * These two headers can be spread over multiple actual headers
-	 * and we rely on their content outside of VCL, so collect them
-	 * into one line here.
-	 */
-	http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
-	http_CollectHdr(sp->wrk->beresp, H_Vary);
-
-	/*
-	 * Figure out how the fetch is supposed to happen, before the
-	 * headers are adultered by VCL
-	 * Also sets other sp->wrk variables 
-	 */ 
-	sp->wrk->body_status = RFC2616_Body(sp);
-
 	if (i) {
 		if (sp->objcore != NULL) {
 			CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
@@ -520,6 +499,22 @@ cnt_fetch(struct sess *sp)
 		return (0);
 	}
 
+	/*
+	 * These two headers can be spread over multiple actual headers
+	 * and we rely on their content outside of VCL, so collect them
+	 * into one line here.
+	 */
+	http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
+	http_CollectHdr(sp->wrk->beresp, H_Vary);
+
+	/*
+	 * Figure out how the fetch is supposed to happen, before the
+	 * headers are adultered by VCL
+	 * Also sets other sp->wrk variables 
+	 */ 
+	sp->wrk->body_status = RFC2616_Body(sp);
+
+
 	sp->err_code = http_GetStatus(sp->wrk->beresp);
 
 	/*
@@ -541,6 +536,75 @@ cnt_fetch(struct sess *sp)
 
 	VCL_fetch_method(sp);
 
+	switch (sp->handling) {
+	case VCL_RET_HIT_FOR_PASS:
+		if (sp->objcore != NULL)
+			sp->objcore->flags |= OC_F_PASS;
+		sp->step = STP_FETCHBODY;
+		return (0);
+	case VCL_RET_DELIVER:
+		sp->step = STP_FETCHBODY;
+		return (0);
+	default:
+		break;
+	}
+
+	/*
+	 * We are not going to fetch the body
+	 * Close the connection and clean up...
+	 */
+
+	VDI_CloseFd(sp);
+	if (sp->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
+		sp->objcore = NULL;
+	}
+	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(sp->wrk->beresp, NULL);
+	sp->wrk->h_content_length = NULL;
+	sp->director = NULL;
+	sp->wrk->storage_hint = NULL;
+
+	switch (sp->handling) {
+	case VCL_RET_RESTART:
+		sp->restarts++;
+		sp->step = STP_RECV;
+		return (0);
+	case VCL_RET_ERROR:
+		sp->step = STP_ERROR;
+		return (0);
+	default:
+		WRONG("Illegal action in vcl_fetch{}");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Fetch response body from the backend
+ *
+DOT subgraph xcluster_body {
+DOT	fetchbody [
+DOT		shape=ellipse
+DOT		label="fetch body\nfrom backend\n"
+DOT	]
+DOT }
+DOT fetchbody -> deliver [style=bold,color=red]
+ */
+
+
+static int
+cnt_fetchbody(struct sess *sp)
+{
+	int i;
+	struct http *hp, *hp2;
+	char *b;
+	unsigned l, nhttp;
+	struct vsb *vary = NULL;
+	int varyl = 0, pass;
+
+	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
+	    sp->handling == VCL_RET_DELIVER);
+
 	if (sp->objcore == NULL) {
 		/* This is a pass from vcl_recv */
 		pass = 1;
@@ -550,7 +614,6 @@ cnt_fetch(struct sess *sp)
 		/* pass from vcl_fetch{} -> hit-for-pass */
 		/* XXX: the bereq was not filtered pass... */
 		pass = 1;
-		sp->objcore->flags |= OC_F_PASS;
 	} else {
 		/* regular object */
 		pass = 0;
@@ -698,24 +761,6 @@ cnt_fetch(struct sess *sp)
 		return (0);
 	}
 
-	switch (sp->handling) {
-	case VCL_RET_RESTART:
-		HSH_Drop(sp);
-		sp->director = NULL;
-		sp->restarts++;
-		sp->step = STP_RECV;
-		return (0);
-	case VCL_RET_HIT_FOR_PASS:
-	case VCL_RET_DELIVER:
-		break;
-	case VCL_RET_ERROR:
-		HSH_Drop(sp);
-		sp->step = STP_ERROR;
-		return (0);
-	default:
-		WRONG("Illegal action in vcl_fetch{}");
-	}
-
 	if (sp->obj->objcore != NULL) {
 		EXP_Insert(sp->obj);
 		AN(sp->obj->objcore);
@@ -1334,7 +1379,6 @@ CNT_Session(struct sess *sp)
 		CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
 		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
 		WS_Assert(w->ws);
-		AZ(sp->wrk->storage_hint);
 
 		switch (sp->step) {
 #define STEP(l,u) \
diff --git a/bin/varnishd/steps.h b/bin/varnishd/steps.h
index 3234c2d..59d4779 100644
--- a/bin/varnishd/steps.h
+++ b/bin/varnishd/steps.h
@@ -40,6 +40,7 @@ STEP(lookup,	LOOKUP)
 STEP(miss,	MISS)
 STEP(hit,	HIT)
 STEP(fetch,	FETCH)
+STEP(fetchbody,	FETCHBODY)
 STEP(deliver,	DELIVER)
 STEP(error,	ERROR)
 STEP(done,	DONE)
diff --git a/bin/varnishtest/tests/b00019.vtc b/bin/varnishtest/tests/b00019.vtc
index 729ca84..84a1f32 100644
--- a/bin/varnishtest/tests/b00019.vtc
+++ b/bin/varnishtest/tests/b00019.vtc
@@ -5,14 +5,19 @@ test "Check that max_restarts works and that we don't fall over"
 server s1 {
 	rxreq
 	txresp -body "012345\n"
+	accept
 	rxreq
 	txresp -body "012345\n"
+	accept
 	rxreq
 	txresp -body "012345\n"
+	accept
 	rxreq
 	txresp -body "012345\n"
+	accept
 	rxreq
 	txresp -body "012345\n"
+	accept
 	rxreq
 	txresp -body "012345\n"
 } -start
diff --git a/bin/varnishtest/tests/c00029.vtc b/bin/varnishtest/tests/c00029.vtc
index 7b87883..8d59f88 100644
--- a/bin/varnishtest/tests/c00029.vtc
+++ b/bin/varnishtest/tests/c00029.vtc
@@ -18,12 +18,18 @@ server s1 {
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 } -start
diff --git a/bin/varnishtest/tests/c00030.vtc b/bin/varnishtest/tests/c00030.vtc
index 656dee9..d5a6a7a 100644
--- a/bin/varnishtest/tests/c00030.vtc
+++ b/bin/varnishtest/tests/c00030.vtc
@@ -18,12 +18,18 @@ server s1 {
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 
+	accept
+
 	rxreq
 	txresp -hdr "X-Saint: yes"
 } -start
diff --git a/bin/varnishtest/tests/c00032.vtc b/bin/varnishtest/tests/c00032.vtc
index 671088a..e207ac6 100644
--- a/bin/varnishtest/tests/c00032.vtc
+++ b/bin/varnishtest/tests/c00032.vtc
@@ -8,6 +8,7 @@ server s1 {
 	expect req.url == "/foo"
 	expect req.http.foobar == "harck-coff"
 	txresp -status 400
+	accept
 	rxreq
 	expect req.url == "/bar"
 	expect req.http.foobar == "snark"
diff --git a/bin/varnishtest/tests/r00412.vtc b/bin/varnishtest/tests/r00412.vtc
index 5fcd43a..4e04a0b 100644
--- a/bin/varnishtest/tests/r00412.vtc
+++ b/bin/varnishtest/tests/r00412.vtc
@@ -6,6 +6,7 @@ server s1 {
 	rxreq
 	expect req.url == "/"
 	txresp -status 303 -hdr "Location: /foo" 
+	accept
 	rxreq
 	expect req.url == "/foo"
 	txresp -body "12345"
diff --git a/bin/varnishtest/tests/s00003.vtc b/bin/varnishtest/tests/s00003.vtc
index 162f1b6..4277a97 100644
--- a/bin/varnishtest/tests/s00003.vtc
+++ b/bin/varnishtest/tests/s00003.vtc
@@ -11,6 +11,7 @@ server s1 {
 	rxreq
 	expect req.url == "/"
 	txresp -status 200 -hdr "foo: 2"
+	accept
 	rxreq
 	expect req.url == "/"
 	txresp -status 200 -hdr "foo: 3"



From perbu at varnish-cache.org  Wed Mar 23 17:39:41 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:41 +0100
Subject: [master] bd9cb5d Pulled out CLI stuff and put it into separate man
	page
Message-ID: <E1Q2S21-0008Th-1Z@project.varnish-software.com>

commit bd9cb5d51c6025badf94be68f99f24b833236913
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 15:22:32 2011 +0100

    Pulled out CLI stuff and put it into separate man page

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
new file mode 100644
index 0000000..32a7394
--- /dev/null
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -0,0 +1,311 @@
+=======
+varnish
+=======
+
+------------------------------
+Varnish Command Line Interface
+------------------------------
+
+:Author: Per Buer
+:Date:   2011-03-23
+:Version: 0.1
+:Manual section: 7
+
+DESCRIPTION
+===========
+
+Varnish as a command line interface (CLI) which can control and change
+most of the operational parameters and the configuration of Varnish,
+without interrupting the running service.
+
+The CLI can be used for the following tasks:
+
+configuration
+     You can upload, change and delete VCL files from the CLI. 
+
+parameters 
+     You can inspect and change the various parameters Varnish has
+     available through the CLI. The individual parameters are
+     documented in the varnishd(1) man page.
+
+statistics
+     Statistic counters are available from the CLI.
+
+bans 
+     Bans are filters that are applied to hinder Varnish from serving
+     stale content. When you issue a ban Varnish will not serve any
+     *banned* object from cache, but rather re-fetch it from it's back
+     end servers.
+
+process management
+     You can stop and start the cache (child) process though the
+     CLI. You can also retrieve the lastst stack trace if the child
+     process has crashed.
+
+If you invoke varnishd(1) with -T, -M or -d the CLI will be
+available. In debug mode (-d) the CLI will be in the foreground, with
+-T you can connect to it with varnishadm and with -M varnishd will
+connect back to a listening service *pushing* the CLI to that
+service. Please see varnishd(1) for details.
+
+Sessions can be authenticated
+
+Commands
+--------
+
+help [command]
+      Display a list of available commands.
+
+      If the command is specified, display help for this command.
+
+param.set param value
+      Set the parameter specified by param to the specified value.  See Run-Time Parameters for a list of parame?
+      ters.
+
+param.show [-l] [param]
+      Display a list if run-time parameters and their values.
+
+      If the -l option is specified, the list includes a brief explanation of each parameter.
+
+      If a param is specified, display only the value and explanation for this parameter.
+
+ping  [timestamp]
+      Ping the Varnish cache process, keeping the connection alive.
+
+ban   *field operator argument* [&& field operator argument [...]]
+      Immediately invalidate all documents matching the ban
+      expression.  See *Ban Expressions* for more documentation and
+      examples.
+
+ban.list
+      Display the ban list.
+
+      All requests for objects from the cache are matched against
+      items on the ban list.  If an object in the cache is older than
+      a matching ban list item, it is considered "banned", and will be
+      fetched from the backend instead.
+
+      When a ban expression is older than all the objects in the
+      cache, it is removed from the list.
+
+ban.url regexp
+      Immediately invalidate all documents whose URL matches the
+      specified regular expression. Please note that the Host part of
+      the URL is ignored, so if you have several virtual hosts all of
+      them will be purged. Use *ban* to specify a complete ban if you
+      need to narrow it down.
+
+quit
+      Close the connection to the varnish admin port.
+
+start
+      Start the Varnish cache process if it is not already running.
+
+stats
+      Show summary statistics.
+
+      All the numbers presented are totals since server startup; for a
+      better idea of the current situation, use the varnishstat(1)
+      utility.
+
+status
+      Check the status of the Varnish cache process.
+
+stop
+      Stop the Varnish cache process.
+
+vcl.discard configname
+      Discard the configuration specified by configname.  This will
+      have no effect if the specified configuration has a non-zero
+      reference count.
+
+vcl.inline configname vcl
+      Create a new configuration named configname with the VCL code
+      specified by vcl, which must be a quoted string.
+
+vcl.list
+      List available configurations and their respective reference
+      counts.  The active configuration is indicated with an asterisk
+      ("*").
+
+vcl.load configname filename
+      Create a new configuration named configname with the contents of
+      the specified file.
+
+vcl.show configname
+      Display the source code for the specified configuration.
+
+vcl.use configname
+      Start using the configuration specified by configname for all
+      new requests.  Existing requests will con? tinue using whichever
+      configuration was in use when they arrived.
+
+Ban Expressions
+---------------
+
+A ban expression consists of one or more conditions.  A condition
+consists of a field, an operator, and an argument.  Conditions can be
+ANDed together with "&&".
+
+A field can be any of the variables from VCL, for instance req.url,
+req.http.host or obj.set-cookie.
+
+Operators are "==" for direct comparision, "~" for a regular
+expression match, and ">" or "<" for size comparisons.  Prepending
+an operator with "!" negates the expression.
+
+The argument could be a quoted string, a regexp, or an integer.
+Integers can have "KB", "MB", "GB" or "TB" appended for size related
+fields.
+
+Escapes
+
+
+
+The commands are tokenized. So if you need spaces they need to be
+wrapped in "". \" can be used to escape ".
+
+\n, \r, \t works as thought.
+
+\123 are used for bytes given in octals.
+
+\x12 are used for hexadecial bytes.
+
+
+Here documents
+--------------
+When giving long commands 
+
+"<< MAGIC" can be used.
+
+The here documents only work for authenticated sessions.
+
+Scripting
+---------
+
+If you are going to write a script that talks CLI to varnishd, the
+include/cli.h contains the relevant magic numbers.
+
+One particular magic number to know, is that the line with the status
+code and length field always is exactly 13 characters long, including
+the NL character.
+
+For your reference the sourcefile lib/libvarnish/cli_common.h contains
+the functions varnish code uses to read and write CLI response.
+
+Details on authentication
+-------------------------
+
+If the -S secret-file is given as argument to varnishd, all network
+CLI connections must authenticate, by proving they know the contents
+of that file.
+
+The file is read at the time the auth command is issued and the
+contents is not cached in varnishd, so it is possible to update the
+file on the fly.
+
+Use the unix file permissions to control access to the file.
+
+An authenticated session looks like this:::
+
+   critter phk> telnet localhost 1234
+   Trying ::1...
+   Trying 127.0.0.1...
+   Connected to localhost.
+   Escape character is '^]'.
+   107 59      
+   ixslvvxrgkjptxmcgnnsdxsvdmvfympg
+   
+   Authentication required.
+   
+   auth 455ce847f0073c7ab3b1465f74507b75d3dc064c1e7de3b71e00de9092fdc89a
+   200 193     
+   -----------------------------
+   Varnish HTTP accelerator CLI.
+   -----------------------------
+   Type 'help' for command list.
+   Type 'quit' to close CLI session.
+   Type 'start' to launch worker process.
+
+The CLI status of 107 indicates that authentication is necessary. The
+first 32 characters of the reponse text is the challenge
+"ixsl...mpg". The challenge is randomly generated for each CLI
+connection, and changes each time a 107 is emitted.
+
+The most recently emitted challenge must be used for calculating the
+authenticator "455c...c89a".
+
+The authenticator is calculated by applying the SHA256 function to the
+following byte sequence:
+
+* Challenge string
+* Newline (0x0a) character.
+* Contents of the secret file
+* Challenge string
+* Newline (0x0a) character.
+
+and dumping the resulting digest in lower-case hex.
+
+In the above example, the secret file contained foo\n and thus:::
+
+   critter phk> cat > _
+   ixslvvxrgkjptxmcgnnsdxsvdmvfympg
+   foo
+   ixslvvxrgkjptxmcgnnsdxsvdmvfympg
+   ^D
+   critter phk> hexdump -C _
+   00000000  69 78 73 6c 76 76 78 72  67 6b 6a 70 74 78 6d 63  |ixslvvxrgkjptxmc|
+   00000010  67 6e 6e 73 64 78 73 76  64 6d 76 66 79 6d 70 67  |gnnsdxsvdmvfympg|
+   00000020  0a 66 6f 6f 0a 69 78 73  6c 76 76 78 72 67 6b 6a  |.foo.ixslvvxrgkj|
+   00000030  70 74 78 6d 63 67 6e 6e  73 64 78 73 76 64 6d 76  |ptxmcgnnsdxsvdmv|
+   00000040  66 79 6d 70 67 0a                                 |fympg.|
+   00000046
+   critter phk> sha256 _ 
+   SHA256 (_) = 455ce847f0073c7ab3b1465f74507b75d3dc064c1e7de3b71e00de9092fdc89a
+   critter phk> openssl dgst -sha256 < _
+   455ce847f0073c7ab3b1465f74507b75d3dc064c1e7de3b71e00de9092fdc89a
+
+The sourcefile lib/libvarnish/cli_auth.c contains a useful function
+which calculates the response, given an open filedescriptor to the
+secret file, and the challenge string.
+
+EXAMPLES
+========
+
+Simple example: All requests where req.url exactly matches the string
+/news are purged from the cache:::
+
+    req.url == "/news"
+
+Example: Purge all documents where the name does not end with ".ogg",
+and where the size of the object is greater than 10 megabytes:::
+
+    req.url !~ "\.ogg$" && obj.size > 10MB
+
+Example: Purge all documents where the serving host is "example.com"
+or "www.example.com", and where the Set-Cookie header received from
+the backend contains "USERID=1663":::
+
+    req.http.host ~ "^(?i)(www\.)example.com$" && obj.set-cookie ~ "USERID=1663"
+
+SEE ALSO
+========
+
+* varnishd(1)
+* vanrishadm(1)
+* vcl(7)
+
+HISTORY
+=======
+
+The varnish manual page was written by Per Buer in 2011. Some of the
+text was taken from the Varnish Cache wiki, the varnishd(7) man page
+or the varnish source code.
+
+COPYRIGHT
+=========
+
+This document is licensed under the same licence as Varnish
+itself. See LICENCE for details.
+
+* Copyright (c) 2011 Varnish Software AS
diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 42e14be..1d3add5 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -93,6 +93,10 @@ OPTIONS
             Offer a management interface on the specified address and port.  See Management
             Interface for a list of management commands.
 
+-M address:port
+            Connect to this port and offer the command line
+            interface. Think of it as a reverse shell.
+
 -t ttl      
    	    Specifies a hard minimum time to live for cached documents.  This is a shortcut for
             specifying the default_ttl run-time parameter.
@@ -204,85 +208,9 @@ Management Interface
 --------------------
 
 If the -T option was specified, varnishd will offer a command-line management interface on the specified address
-and port.  The following commands are available:
-
-help [command]
-      Display a list of available commands.
-
-      If the command is specified, display help for this command.
-
-param.set param value
-      Set the parameter specified by param to the specified value.  See Run-Time Parameters for a list of parame?
-      ters.
-
-param.show [-l] [param]
-      Display a list if run-time parameters and their values.
-
-      If the -l option is specified, the list includes a brief explanation of each parameter.
-
-      If a param is specified, display only the value and explanation for this parameter.
-
-ping [timestamp]
-      Ping the Varnish cache process, keeping the connection alive.
-
-purge field operator argument [&& field operator argument [...]]
-      Immediately invalidate all documents matching the purge expression.  See Purge expressions for more docu?
-      mentation and examples.
-
-purge.list
-      Display the purge list.
-
-      All requests for objects from the cache are matched against items on the purge list.  If an object in the
-      cache is older than a matching purge list item, it is considered "purged", and will be fetched from the
-      backend instead.
-
-      When a purge expression is older than all the objects in the cache, it is removed from the list.
-
-purge.url regexp
-      Immediately invalidate all documents whose URL matches the specified regular expression.
-
-quit
-      Close the connection to the varnish admin port.
-
-start
-      Start the Varnish cache process if it is not already running.
-
-stats
-      Show summary statistics.
-
-      All the numbers presented are totals since server startup; for a better idea of the current situation, use
-      the varnishstat(1) utility.
-
-status
-      Check the status of the Varnish cache process.
-
-stop
-      Stop the Varnish cache process.
+and port.  The recommended way of connecting to the command-line management interface is through varnishadm(1).
 
-url.purge regexp
-      Deprecated, see purge.url instead.
-
-vcl.discard configname
-      Discard the configuration specified by configname.  This will have no effect if the specified configuration
-      has a non-zero reference count.
-
-vcl.inline configname vcl
-      Create a new configuration named configname with the VCL code specified by vcl, which must be a quoted
-      string.
-
-vcl.list
-      List available configurations and their respective reference counts.  The active configuration is indicated
-      with an asterisk ("*").
-
-vcl.load configname filename
-      Create a new configuration named configname with the contents of the specified file.
-
-vcl.show configname
-      Display the source code for the specified configuration.
-
-vcl.use configname
-      Start using the configuration specified by configname for all new requests.  Existing requests will con?
-      tinue using whichever configuration was in use when they arrived.
+The commands available are documented in varnish(7).
 
 Run-Time Parameters
 -------------------
@@ -835,41 +763,6 @@ waiter
 
 	Select the waiter kernel interface.
 
-
-Purge expressions
------------------
-
-A purge expression consists of one or more conditions.  A condition
-consists of a field, an operator, and an argument.  Conditions can be
-ANDed together with "&&".
-
-A field can be any of the variables from VCL, for instance req.url,
-req.http.host or obj.set-cookie.
-
-Operators are "==" for direct comparision, "~" for a regular
-expression match, and ">" or "<" for size comparisons.  Prepending
-an operator with "!" negates the expression.
-
-The argument could be a quoted string, a regexp, or an integer.
-Integers can have "KB", "MB", "GB" or "TB" appended for size related
-fields.
-
-Simple example: All requests where req.url exactly matches the string
-/news are purged from the cache:::
-
-    req.url == "/news"
-
-Example: Purge all documents where the name does not end with ".ogg",
-and where the size of the object is greater than 10 megabytes:::
-
-    req.url !~ "\.ogg$" && obj.size > 10MB
-
-Example: Purge all documents where the serving host is "example.com"
-or "www.example.com", and where the Set- Cookie header received from
-the backend contains "USERID=1663":::
-
-    req.http.host ~ "^(?i)(www\.)example.com$" && obj.set-cookie ~ "USERID=1663"
-
 SEE ALSO
 ========
 



From perbu at varnish-cache.org  Wed Mar 23 17:39:41 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:41 +0100
Subject: [master] 9105474 Document -n and default behaviour
Message-ID: <E1Q2S21-0008Tq-9N@project.varnish-software.com>

commit 9105474f474234f7fc02ee0f1853a388942d2698
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 15:23:01 2011 +0100

    Document -n and default behaviour

diff --git a/doc/sphinx/reference/varnishadm.rst b/doc/sphinx/reference/varnishadm.rst
index 9386cdf..af68355 100644
--- a/doc/sphinx/reference/varnishadm.rst
+++ b/doc/sphinx/reference/varnishadm.rst
@@ -15,12 +15,13 @@ Control a running varnish instance
 SYNOPSIS
 ========
 
-       varnishadm [-t timeout] [-S secret_file] -T address:port [command [...]]
+       varnishadm [-t timeout] [-S secret_file] [-T address:port] [-n name] [command [...]]
 
 DESCRIPTION
 ===========
 
-The varnishadm utility establishes a CLI connection using the -T and -S arguments.
+The varnishadm utility establishes a CLI connection to varnishd either
+using -n *name* or using the -T and -S arguments. If -n *name* is the location of the secret file and the address:port is looked up in shared memory. If neither is given varnishadm will look for an instance without a given name. 
 
 If a command is given, the command and arguments are sent over the CLI
 connection and the result returned on stdout.
@@ -42,6 +43,8 @@ OPTIONS
 -T address:port   
         Connect to the management interface at the specified address and port.
 
+-n name
+	Connect to the instance of varnishd with this name.
 
 Available commands and parameters are documented in the varnishd(1)
 manual page.  Additionally, a summary of commands can be obtained by



From perbu at varnish-cache.org  Wed Mar 23 17:39:41 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:41 +0100
Subject: [master] feaa02b finished the here docs + cleanup
Message-ID: <E1Q2S21-0008U3-M0@project.varnish-software.com>

commit feaa02b32942f8140a0beea2d428bb527bf15369
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 18:36:30 2011 +0100

    finished the here docs + cleanup

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 32a7394..145f960 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -32,7 +32,7 @@ statistics
      Statistic counters are available from the CLI.
 
 bans 
-     Bans are filters that are applied to hinder Varnish from serving
+     Bans are filters that are applied to keep Varnish from serving
      stale content. When you issue a ban Varnish will not serve any
      *banned* object from cache, but rather re-fetch it from it's back
      end servers.
@@ -44,11 +44,37 @@ process management
 
 If you invoke varnishd(1) with -T, -M or -d the CLI will be
 available. In debug mode (-d) the CLI will be in the foreground, with
--T you can connect to it with varnishadm and with -M varnishd will
-connect back to a listening service *pushing* the CLI to that
-service. Please see varnishd(1) for details.
+-T you can connect to it with varnishadm or telnet and with -M
+varnishd will connect back to a listening service *pushing* the CLI to
+that service. Please see varnishd(1) for details.
 
-Sessions can be authenticated
+
+Syntax
+------
+
+Commands are usually terminated with a newline. Long command can be
+entered using sh style *here documents*. The format of here-documents
+is:::
+
+   << word
+	here document
+   word
+
+*word* can be any continuous string choosen to make sure it doesn't
+appear naturally in the following *here document*.
+
+When using the here document style of input there are no restrictions
+on lenght. When using newline-terminated commands maximum lenght is
+limited by the varnishd parameter *cli_buffer*.
+
+When commands are newline terminated they get *tokenized* before
+parsing so if you have significant spaces enclose your strings in
+double quotes. Within the quotes you can escape characters with
+\\. The \n, \r and \t get translated to newlines, carrage returns and
+tabs. Double quotes themselves can be escaped with a backslash.
+
+To enter characters in octals use the \\nnn syntax. Hexadecimals can
+be entered with the \\xnn syntax.
 
 Commands
 --------
@@ -59,15 +85,17 @@ help [command]
       If the command is specified, display help for this command.
 
 param.set param value
-      Set the parameter specified by param to the specified value.  See Run-Time Parameters for a list of parame?
-      ters.
+      Set the parameter specified by param to the specified value.
+      See Run-Time Parameters for a list of parame? ters.
 
 param.show [-l] [param]
       Display a list if run-time parameters and their values.
 
-      If the -l option is specified, the list includes a brief explanation of each parameter.
+      If the -l option is specified, the list includes a brief
+      explanation of each parameter.
 
-      If a param is specified, display only the value and explanation for this parameter.
+      If a param is specified, display only the value and explanation
+      for this parameter.
 
 ping  [timestamp]
       Ping the Varnish cache process, keeping the connection alive.
@@ -158,27 +186,6 @@ The argument could be a quoted string, a regexp, or an integer.
 Integers can have "KB", "MB", "GB" or "TB" appended for size related
 fields.
 
-Escapes
-
-
-
-The commands are tokenized. So if you need spaces they need to be
-wrapped in "". \" can be used to escape ".
-
-\n, \r, \t works as thought.
-
-\123 are used for bytes given in octals.
-
-\x12 are used for hexadecial bytes.
-
-
-Here documents
---------------
-When giving long commands 
-
-"<< MAGIC" can be used.
-
-The here documents only work for authenticated sessions.
 
 Scripting
 ---------



From perbu at varnish-cache.org  Wed Mar 23 17:39:42 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:42 +0100
Subject: [master] 3fba51b advice the user to rely on the default caching rules
Message-ID: <E1Q2S22-0008UD-6r@project.varnish-software.com>

commit 3fba51b01b44059250f35f6d6349a43682bd14a5
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 18:37:43 2011 +0100

    advice the user to rely on the default caching rules

diff --git a/doc/sphinx/tutorial/increasing_your_hitrate.rst b/doc/sphinx/tutorial/increasing_your_hitrate.rst
index 3ecd6de..437461a 100644
--- a/doc/sphinx/tutorial/increasing_your_hitrate.rst
+++ b/doc/sphinx/tutorial/increasing_your_hitrate.rst
@@ -168,6 +168,18 @@ beresp.ttl to whatever you want::
       }
   }
 
+The example will set the TTL to 5 days for the old legacy stuff on
+your site.
+
+Forcing caching for certain requests and certain responses
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since you still have this cumbersome backend that isn't very friendly
+to work with you might want to override more stuff in Varnish. We
+recommend that you rely as much as you can on the default caching
+rules. It is perfectly easy to force Varnish to lookup an object in
+the cache but it isn't really recommended.
+
 
 Normalizing your namespace
 ~~~~~~~~~~~~~~~~~~~~~~~~~~



From perbu at varnish-cache.org  Wed Mar 23 17:39:42 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:42 +0100
Subject: [master] 3587d1b Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Q2S22-0008UW-OD@project.varnish-software.com>

commit 3587d1b28245983a82ddd0c6b20fb5c450218668
Merge: 3fba51b a1a62dc
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 18:37:54 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Wed Mar 23 17:39:43 2011
From: perbu at varnish-cache.org (Per Andreas Buer)
Date: Wed, 23 Mar 2011 18:39:43 +0100
Subject: [master] 9652705 3.0 string concats + line breaks
Message-ID: <E1Q2S23-0008Un-8J@project.varnish-software.com>

commit 96527057e97421b20485d7f0fcdad2763c5c093a
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Mar 23 18:38:37 2011 +0100

    3.0 string concats + line breaks

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 6355913..85553f4 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -35,16 +35,15 @@ Blocks are delimited by curly braces, statements end with semicolons,
 and comments may be written as in C, C++ or Perl according to your own
 preferences.
 
-In addition to the C-like assignment (=), comparison (==) and boolean
-(!, && and \|\|) operators, VCL supports regular expression and ACL
-matching using the ~ operator.
+In addition to the C-like assignment (=), comparison (==, !=) and
+boolean (!, && and \|\|) operators, VCL supports both regular
+expression and ACL matching using the ~ and the !~ operators.
 
 Unlike C and Perl, the backslash (\) character has no special meaning
 in strings in VCL, so it can be freely used in regular expressions
 without doubling.
 
-Strings are concatenated by putting them one after each other
-with a '+' operator between.
+Strings are concatenated using the '+' operator. 
 
 Assignments are introduced with the *set* keyword.  There are no
 user-defined variables; values can only be assigned to variables



From phk at varnish-cache.org  Thu Mar 24 09:45:00 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 10:45:00 +0100
Subject: [master] 3441f41 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Q2h6C-0003xG-Qg@project.varnish-software.com>

commit 3441f4148d60185ebb659c32a26ae223361e28f2
Merge: bd007e8 9652705
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 08:54:34 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Thu Mar 24 09:45:00 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 10:45:00 +0100
Subject: [master] bd007e8 Forgot to git add this one some days ago
Message-ID: <E1Q2h6C-0003x7-Fv@project.varnish-software.com>

commit bd007e8310e557eb828dbc4ab6e579dfc958d344
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 22 12:50:48 2011 +0000

    Forgot to git add this one some days ago

diff --git a/bin/varnishtest/tests/m00005.vtc b/bin/varnishtest/tests/m00005.vtc
new file mode 100644
index 0000000..80dfb74
--- /dev/null
+++ b/bin/varnishtest/tests/m00005.vtc
@@ -0,0 +1,90 @@
+# $Id$
+
+test "test vmod_std.duration conversion"
+
+server s1 {
+	rxreq
+	expect req.url == "/1"
+	txresp -bodylen 1
+
+} -start
+
+varnish v1 -vcl+backend {
+	import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+
+	sub vcl_recv {
+		set req.ttl = 1000000s + std.duration(req.http.ttl, 1s);
+	}
+	sub vcl_deliver {
+		set resp.http.ttl = (req.ttl + 1s);
+	}
+} -start
+
+client c1 {
+	txreq -url "/1"  -hdr "ttl: 10s "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl == 1000011.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 10m "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl == 1000601.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 10h "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl == 1036001.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 10d "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl == 1864001.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 10w "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl == 7048001.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: -100s "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  999901.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: s "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  1000002.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 3wx "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  1000002.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: -inf "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  1000002.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 2x "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  1000002.000
+	expect resp.bodylen == 1
+
+	txreq -url "/1"  -hdr "ttl: 2h x "
+	rxresp
+	expect resp.status == 200
+	expect resp.http.ttl ==  1000002.000
+	expect resp.bodylen == 1
+
+} -run



From phk at varnish-cache.org  Thu Mar 24 09:45:01 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 10:45:01 +0100
Subject: [master] d9037dd Move some of the magic bits related to gzip-editing
	onto the worker thread workspace,
	to make sure they are not overwritten before they make it out the wire.
Message-ID: <E1Q2h6D-0003xs-Ed@project.varnish-software.com>

commit d9037ddff65bed9cafb0de2b8bb944fda0596b60
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 09:44:31 2011 +0000

    Move some of the magic bits related to gzip-editing onto the worker
    thread workspace, to make sure they are not overwritten before they
    make it out the wire.

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 48a9948..179c31f 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -467,7 +467,7 @@ ESI_DeliverChild(const struct sess *sp)
 	u_char *p, cc;
 	uint32_t icrc;
 	uint32_t ilen;
-	uint8_t pad[6] = { 0 };
+	uint8_t *dbits;
 
 	if (!sp->obj->gziped) {
 		VTAILQ_FOREACH(st, &sp->obj->store, list)
@@ -479,6 +479,9 @@ ESI_DeliverChild(const struct sess *sp)
 	 * blocks, stripping the "LAST" bit of the last one and
 	 * padding it, as necessary, to a byte boundary.
 	 */
+
+	dbits = (void*)WS_Alloc(sp->wrk->ws, 8);
+	AN(dbits);
 	obj = sp->obj;
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	start = obj->gzip_start;
@@ -489,7 +492,6 @@ ESI_DeliverChild(const struct sess *sp)
 	assert(stop > 0 && stop < obj->len * 8);
 	assert(last >= start);
 	assert(last < stop);
-//printf("BITS %jd %jd %jd\n", start, last, stop);
 
 	/* The start bit must be byte aligned. */
 	AZ(start & 7);
@@ -498,12 +500,10 @@ ESI_DeliverChild(const struct sess *sp)
 	 * XXX: optimize for the case where the 'last'
 	 * XXX: bit is in a empty copy block
 	 */
-	cc = ved_deliver_byterange(sp, start/8, last/8);
-//printf("CC_LAST %x\n", cc);
-	cc &= ~(1U << (last & 7));
-	ved_sendchunk(sp, NULL, 0, &cc, 1);
+	*dbits = ved_deliver_byterange(sp, start/8, last/8);
+	*dbits &= ~(1U << (last & 7));
+	ved_sendchunk(sp, NULL, 0, dbits, 1);
 	cc = ved_deliver_byterange(sp, 1 + last/8, stop/8);
-//printf("CC_STOP %x (%d)\n", cc, (int)(stop & 7));
 	switch((int)(stop & 7)) {
 	case 0: /* xxxxxxxx */
 		/* I think we have an off by one here, but that's OK */
@@ -512,46 +512,47 @@ ESI_DeliverChild(const struct sess *sp)
 	case 1: /* x000.... 00000000 00000000 11111111 11111111 */
 	case 3: /* xxx000.. 00000000 00000000 11111111 11111111 */
 	case 5: /* xxxxx000 00000000 00000000 11111111 11111111 */
-		pad[0] = cc | 0x00;
-		pad[1] = 0x00; pad[2] = 0x00; pad[3] = 0xff; pad[4] = 0xff;
+		dbits[1] = cc | 0x00;
+		dbits[2] = 0x00; dbits[3] = 0x00;
+	        dbits[4] = 0xff; dbits[5] = 0xff;
 		lpad = 5;
 		break;
 	case 2: /* xx010000 00000100 00000001 00000000 */
-		pad[0] = cc | 0x08;
-		pad[1] = 0x20;
-		pad[2] = 0x80;
-		pad[3] = 0x00;
+		dbits[1] = cc | 0x08;
+		dbits[2] = 0x20;
+		dbits[3] = 0x80;
+		dbits[4] = 0x00;
 		lpad = 4;
 		break;
 	case 4: /* xxxx0100 00000001 00000000 */
-		pad[0] = cc | 0x20;
-		pad[1] = 0x80;
-		pad[2] = 0x00;
+		dbits[1] = cc | 0x20;
+		dbits[2] = 0x80;
+		dbits[3] = 0x00;
 		lpad = 3;
 		break;
 	case 6: /* xxxxxx01 00000000 */
-		pad[0] = cc | 0x80;
-		pad[1] = 0x00;
+		dbits[1] = cc | 0x80;
+		dbits[2] = 0x00;
 		lpad = 2;
 		break;
 	case 7:	/* xxxxxxx0 00...... 00000000 00000000 11111111 11111111 */
-		pad[0] = cc | 0x00;
-		pad[1] = 0x00;
-		pad[2] = 0x00; pad[3] = 0x00; pad[4] = 0xff; pad[5] = 0xff;
+		dbits[1] = cc | 0x00;
+		dbits[2] = 0x00;
+		dbits[3] = 0x00; dbits[4] = 0x00;
+	        dbits[5] = 0xff; dbits[6] = 0xff;
 		lpad = 6;
 		break;
 	default:
 		INCOMPL();
 	}
 	if (lpad > 0)
-		ved_sendchunk(sp, NULL, 0, pad, lpad);
+		ved_sendchunk(sp, NULL, 0, dbits + 1, lpad);
 	st = VTAILQ_LAST(&sp->obj->store, storagehead);
 	assert(st->len > 8);
 
 	p = st->ptr + st->len - 8;
 	icrc = vle32dec(p);
 	ilen = vle32dec(p + 4);
-//printf("CRC %08x LEN %d\n", icrc, ilen);
 	sp->wrk->crc = crc32_combine(sp->wrk->crc, icrc, ilen);
 	sp->wrk->l_crc += ilen;
 }



From phk at varnish-cache.org  Thu Mar 24 11:54:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 12:54:31 +0100
Subject: [master] 83ea35d Segregate the WRW related fields in their own
	structure for code clarity.
Message-ID: <E1Q2j7X-00029B-GT@project.varnish-software.com>

commit 83ea35df7297afd8ff8bdce494737164d8d3191b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 11:52:57 2011 +0000

    Segregate the WRW related fields in their own structure for code
    clarity.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 4c31a7e..840c774 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -248,6 +248,16 @@ struct exp {
 
 /*--------------------------------------------------------------------*/
 
+/* WRW related fields */
+struct wrw {
+	int			*wfd;
+	unsigned		werr;	/* valid after WRK_Flush() */
+	struct iovec		*iov;
+	unsigned		siov;
+	unsigned		niov;
+	ssize_t			liov;
+};
+
 struct worker {
 	unsigned		magic;
 #define WORKER_MAGIC		0x6391adcf
@@ -259,18 +269,13 @@ struct worker {
 
 	double			lastused;
 
+	struct wrw		wrw;
+
 	pthread_cond_t		cond;
 
 	VTAILQ_ENTRY(worker)	list;
 	struct workreq		*wrq;
 
-	int			*wfd;
-	unsigned		werr;	/* valid after WRK_Flush() */
-	struct iovec		*iov;
-	unsigned		siov;
-	unsigned		niov;
-	ssize_t			liov;
-
 	struct VCL_conf		*vcl;
 
 	uint32_t		*wlb, *wlp, *wle;
@@ -781,6 +786,7 @@ int WRK_Queue(struct workreq *wrq);
 int WRK_QueueSession(struct sess *sp);
 void WRK_SumStat(struct worker *w);
 
+#define WRW_IsReleased(w)	((w)->wrw.wfd == NULL)
 void WRW_Reserve(struct worker *w, int *fd);
 unsigned WRW_Flush(struct worker *w);
 unsigned WRW_FlushRelease(struct worker *w);
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index e3cc4e2..d3a9a43 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -240,7 +240,7 @@ cnt_deliver(struct sess *sp)
 
 	RES_WriteObj(sp);
 
-	AZ(sp->wrk->wfd);
+	assert(WRW_IsReleased(sp->wrk));
 	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
 	http_Setup(sp->wrk->resp, NULL);
 	sp->step = STP_DONE;
@@ -749,7 +749,7 @@ cnt_fetchbody(struct sess *sp)
 	http_Setup(sp->wrk->bereq, NULL);
 	http_Setup(sp->wrk->beresp, NULL);
 	sp->wrk->vfp = NULL;
-	AZ(sp->wrk->wfd);
+	assert(WRW_IsReleased(sp->wrk));
 	AZ(sp->vbc);
 	AN(sp->director);
 
@@ -1121,7 +1121,7 @@ cnt_pipe(struct sess *sp)
 	assert(sp->handling == VCL_RET_PIPE);
 
 	PipeSession(sp);
-	AZ(sp->wrk->wfd);
+	assert(WRW_IsReleased(sp->wrk));
 	http_Setup(sp->wrk->bereq, NULL);
 	sp->step = STP_DONE;
 	return (0);
@@ -1396,7 +1396,7 @@ CNT_Session(struct sess *sp)
 		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
 	}
 	WSL_Flush(w, 0);
-	AZ(w->wfd);
+	assert(WRW_IsReleased(w));
 }
 
 /*
diff --git a/bin/varnishd/cache_http.c b/bin/varnishd/cache_http.c
index 4ebf878..b8cdb3a 100644
--- a/bin/varnishd/cache_http.c
+++ b/bin/varnishd/cache_http.c
@@ -1054,10 +1054,11 @@ unsigned
 http_Write(struct worker *w, const struct http *hp, int resp)
 {
 	unsigned u, l;
+	int fd = *(w->wrw.wfd);
 
 	if (resp) {
 		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], " ");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_PROTO);
+		WSLH(w, fd, hp, HTTP_HDR_PROTO);
 
 		hp->hd[HTTP_HDR_STATUS].b = WS_Alloc(w->ws, 4);
 		AN(hp->hd[HTTP_HDR_STATUS].b);
@@ -1066,18 +1067,18 @@ http_Write(struct worker *w, const struct http *hp, int resp)
 		hp->hd[HTTP_HDR_STATUS].e = hp->hd[HTTP_HDR_STATUS].b + 3;
 
 		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_STATUS], " ");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_STATUS);
+		WSLH(w, fd, hp, HTTP_HDR_STATUS);
 
 		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_RESPONSE], "\r\n");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_RESPONSE);
+		WSLH(w, fd, hp, HTTP_HDR_RESPONSE);
 	} else {
 		AN(hp->hd[HTTP_HDR_URL].b);
 		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_REQ], " ");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_REQ);
+		WSLH(w, fd, hp, HTTP_HDR_REQ);
 		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_URL], " ");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_URL);
+		WSLH(w, fd, hp, HTTP_HDR_URL);
 		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], "\r\n");
-		WSLH(w, *w->wfd, hp, HTTP_HDR_PROTO);
+		WSLH(w, fd, hp, HTTP_HDR_PROTO);
 	}
 	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
 		if (hp->hd[u].b == NULL)
@@ -1085,7 +1086,7 @@ http_Write(struct worker *w, const struct http *hp, int resp)
 		AN(hp->hd[u].b);
 		AN(hp->hd[u].e);
 		l += WRW_WriteH(w, &hp->hd[u], "\r\n");
-		WSLH(w, *w->wfd, hp, u);
+		WSLH(w, fd, hp, u);
 	}
 	l += WRW_Write(w, "\r\n", -1);
 	return (l);
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 0c80772..99ca6c3 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -143,8 +143,8 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 	w->bereq = HTTP_create(http0, nhttp);
 	w->beresp = HTTP_create(http1, nhttp);
 	w->resp = HTTP_create(http2, nhttp);
-	w->iov = iov;
-	w->siov = siov;
+	w->wrw.iov = iov;
+	w->wrw.siov = siov;
 	AZ(pthread_cond_init(&w->cond, NULL));
 
 	WS_Init(w->ws, "wrk", ws, sess_workspace);
@@ -189,7 +189,7 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 		AZ(w->bereq->ws);
 		AZ(w->beresp->ws);
 		AZ(w->resp->ws);
-		AZ(w->wfd);
+		AZ(w->wrw.wfd);
 		AZ(w->storage_hint);
 		assert(w->wlp == w->wlb);
 		w->wrq = NULL;
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index 37d09ae..f6ddec8 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -72,45 +72,51 @@ SVNID("$Id$")
 void
 WRW_Reserve(struct worker *w, int *fd)
 {
+	struct wrw *wrw;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->wfd);
-	w->werr = 0;
-	w->liov = 0;
-	w->niov = 0;
-	w->wfd = fd;
+	wrw = &w->wrw;
+	AZ(wrw->wfd);
+	wrw->werr = 0;
+	wrw->liov = 0;
+	wrw->niov = 0;
+	wrw->wfd = fd;
 }
 
 static void
 WRW_Release(struct worker *w)
 {
+	struct wrw *wrw;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	w->werr = 0;
-	w->liov = 0;
-	w->niov = 0;
-	w->wfd = NULL;
+	wrw = &w->wrw;
+	wrw->werr = 0;
+	wrw->liov = 0;
+	wrw->niov = 0;
+	wrw->wfd = NULL;
 }
 
 unsigned
 WRW_Flush(struct worker *w)
 {
 	ssize_t i;
+	struct wrw *wrw;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wfd);
-	if (*w->wfd >= 0 && w->niov > 0 && w->werr == 0) {
-		i = writev(*w->wfd, w->iov, w->niov);
-		if (i != w->liov) {
-			w->werr++;
-			WSL(w, SLT_Debug, *w->wfd,
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+	if (*wrw->wfd >= 0 && wrw->niov > 0 && wrw->werr == 0) {
+		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
+		if (i != wrw->liov) {
+			wrw->werr++;
+			WSL(w, SLT_Debug, *wrw->wfd,
 			    "Write error, retval = %d, len = %d, errno = %s",
-			    i, w->liov, strerror(errno));
+			    i, wrw->liov, strerror(errno));
 		}
 	}
-	w->liov = 0;
-	w->niov = 0;
-	return (w->werr);
+	wrw->liov = 0;
+	wrw->niov = 0;
+	return (wrw->werr);
 }
 
 unsigned
@@ -119,7 +125,7 @@ WRW_FlushRelease(struct worker *w)
 	unsigned u;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wfd);
+	AN(w->wrw.wfd);
 	u = WRW_Flush(w);
 	WRW_Release(w);
 	return (u);
@@ -131,7 +137,7 @@ WRW_WriteH(struct worker *w, const txt *hh, const char *suf)
 	unsigned u;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wfd);
+	AN(w->wrw.wfd);
 	AN(w);
 	AN(hh);
 	AN(hh->b);
@@ -145,19 +151,21 @@ WRW_WriteH(struct worker *w, const txt *hh, const char *suf)
 unsigned
 WRW_Write(struct worker *w, const void *ptr, int len)
 {
+	struct wrw *wrw;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wfd);
-	if (len == 0 || *w->wfd < 0)
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+	if (len == 0 || *wrw->wfd < 0)
 		return (0);
 	if (len == -1)
 		len = strlen(ptr);
-	if (w->niov == w->siov)
+	if (wrw->niov == wrw->siov)
 		(void)WRW_Flush(w);
-	w->iov[w->niov].iov_base = TRUST_ME(ptr);
-	w->iov[w->niov].iov_len = len;
-	w->liov += len;
-	w->niov++;
+	wrw->iov[wrw->niov].iov_base = TRUST_ME(ptr);
+	wrw->iov[wrw->niov].iov_len = len;
+	wrw->liov += len;
+	wrw->niov++;
 	return (len);
 }
 
@@ -165,9 +173,11 @@ WRW_Write(struct worker *w, const void *ptr, int len)
 void
 WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
 {
+	struct wrw *wrw;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wfd);
+	wrw = &w->wrw;
+	AN(wrw->wfd);
 	assert(fd >= 0);
 	assert(len > 0);
 
@@ -175,31 +185,31 @@ WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
 	do {
 		struct sf_hdtr sfh;
 		memset(&sfh, 0, sizeof sfh);
-		if (w->niov > 0) {
-			sfh.headers = w->iov;
-			sfh.hdr_cnt = w->niov;
+		if (wrw->niov > 0) {
+			sfh.headers = wrw->iov;
+			sfh.hdr_cnt = wrw->niov;
 		}
-		if (sendfile(fd, *w->wfd, off, len, &sfh, NULL, 0) != 0)
-			w->werr++;
-		w->liov = 0;
-		w->niov = 0;
+		if (sendfile(fd, *wrw->wfd, off, len, &sfh, NULL, 0) != 0)
+			wrw->werr++;
+		wrw->liov = 0;
+		wrw->niov = 0;
 	} while (0);
 #elif defined(__linux__)
 	do {
 		if (WRK_Flush(w) == 0 &&
-		    sendfile(*w->wfd, fd, &off, len) != len)
-			w->werr++;
+		    sendfile(*wrw->wfd, fd, &off, len) != len)
+			wrw->werr++;
 	} while (0);
 #elif defined(__sun) && defined(HAVE_SENDFILEV)
 	do {
 		sendfilevec_t svvec[params->http_headers * 2 + 1];
 		size_t xferred = 0, expected = 0;
 		int i;
-		for (i = 0; i < w->niov; i++) {
+		for (i = 0; i < wrw->niov; i++) {
 			svvec[i].sfv_fd = SFV_FD_SELF;
 			svvec[i].sfv_flag = 0;
-			svvec[i].sfv_off = (off_t) w->iov[i].iov_base;
-			svvec[i].sfv_len = w->iov[i].iov_len;
+			svvec[i].sfv_off = (off_t) wrw->iov[i].iov_base;
+			svvec[i].sfv_len = wrw->iov[i].iov_len;
 			expected += svvec[i].sfv_len;
 		}
 		svvec[i].sfv_fd = fd;
@@ -207,17 +217,17 @@ WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
 		svvec[i].sfv_off = off;
 		svvec[i].sfv_len = len;
 		expected += svvec[i].sfv_len;
-		if (sendfilev(*w->wfd, svvec, i, &xferred) == -1 ||
+		if (sendfilev(*wrw->wfd, svvec, i, &xferred) == -1 ||
 		    xferred != expected)
-			w->werr++;
-		w->liov = 0;
-		w->niov = 0;
+			wrw->werr++;
+		wrw->liov = 0;
+		wrw->niov = 0;
 	} while (0);
 #elif defined(__sun) && defined(HAVE_SENDFILE)
 	do {
 		if (WRK_Flush(w) == 0 &&
-		    sendfile(*w->wfd, fd, &off, len) != len)
-			w->werr++;
+		    sendfile(*wrw->wfd, fd, &off, len) != len)
+			wrw->werr++;
 	} while (0);
 #else
 #error Unknown sendfile() implementation



From phk at varnish-cache.org  Thu Mar 24 14:01:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 15:01:31 +0100
Subject: [master] d4ac001 Add WRW support for chunked encoding.
Message-ID: <E1Q2l6R-0006tr-Ge@project.varnish-software.com>

commit d4ac001ad8c959bd9b755454a3975c0d20b5ba0a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 12:24:22 2011 +0000

    Add WRW support for chunked encoding.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 840c774..423d452 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -256,6 +256,8 @@ struct wrw {
 	unsigned		siov;
 	unsigned		niov;
 	ssize_t			liov;
+	ssize_t			cliov;
+	unsigned		ciov;	/* Chunked header marker */
 };
 
 struct worker {
@@ -787,6 +789,7 @@ int WRK_QueueSession(struct sess *sp);
 void WRK_SumStat(struct worker *w);
 
 #define WRW_IsReleased(w)	((w)->wrw.wfd == NULL)
+void WRW_Chunked(struct worker *w);
 void WRW_Reserve(struct worker *w, int *fd);
 unsigned WRW_Flush(struct worker *w);
 unsigned WRW_FlushRelease(struct worker *w);
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index f6ddec8..d43ea78 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -45,6 +45,7 @@
 #include "svnid.h"
 SVNID("$Id$")
 
+#include <stdio.h>
 #include <sys/types.h>
 #include <sys/uio.h>
 
@@ -80,6 +81,7 @@ WRW_Reserve(struct worker *w, int *fd)
 	wrw->werr = 0;
 	wrw->liov = 0;
 	wrw->niov = 0;
+	wrw->ciov = wrw->siov;
 	wrw->wfd = fd;
 }
 
@@ -90,9 +92,11 @@ WRW_Release(struct worker *w)
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	wrw = &w->wrw;
+	AN(wrw->wfd);
 	wrw->werr = 0;
 	wrw->liov = 0;
 	wrw->niov = 0;
+	wrw->ciov = wrw->siov;
 	wrw->wfd = NULL;
 }
 
@@ -101,11 +105,28 @@ WRW_Flush(struct worker *w)
 {
 	ssize_t i;
 	struct wrw *wrw;
+	char cbuf[32];
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	wrw = &w->wrw;
 	AN(wrw->wfd);
+
+	/* For chunked, there must be one slot reserved for the chunked tail */
+	if (wrw->ciov < wrw->siov)
+		assert(wrw->niov < wrw->siov);
+
 	if (*wrw->wfd >= 0 && wrw->niov > 0 && wrw->werr == 0) {
+		if (wrw->ciov < wrw->siov && wrw->liov > 0) {
+			bprintf(cbuf, "%jx\r\n", (intmax_t)wrw->cliov);
+			i = strlen(cbuf);
+			wrw->iov[wrw->ciov].iov_base = cbuf;
+			wrw->iov[wrw->ciov].iov_len = i;
+			wrw->liov += i;
+
+			wrw->iov[wrw->niov].iov_base = cbuf + i - 2;
+			wrw->iov[wrw->niov++].iov_len = 2;
+			wrw->liov += 2;
+		}
 		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
 		if (i != wrw->liov) {
 			wrw->werr++;
@@ -115,7 +136,10 @@ WRW_Flush(struct worker *w)
 		}
 	}
 	wrw->liov = 0;
+	wrw->cliov = 0;
 	wrw->niov = 0;
+	if (wrw->ciov < wrw->siov)
+		wrw->ciov = wrw->niov++;
 	return (wrw->werr);
 }
 
@@ -160,15 +184,38 @@ WRW_Write(struct worker *w, const void *ptr, int len)
 		return (0);
 	if (len == -1)
 		len = strlen(ptr);
-	if (wrw->niov == wrw->siov)
+	if (wrw->niov == wrw->siov + (wrw->ciov < wrw->siov ? 1 : 0))
 		(void)WRW_Flush(w);
 	wrw->iov[wrw->niov].iov_base = TRUST_ME(ptr);
 	wrw->iov[wrw->niov].iov_len = len;
 	wrw->liov += len;
+	if (wrw->ciov < wrw->siov)
+		wrw->cliov += len;
 	wrw->niov++;
 	return (len);
 }
 
+void
+WRW_Chunked(struct worker *w)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+
+	assert(wrw->ciov == wrw->siov);
+	/*
+	 * If there are not space for chunked header, a chunk of data and
+	 * a chunk tail, we might as well flush right away.
+	 */
+	if (wrw->niov + 3 >= wrw->siov)
+		(void)WRW_Flush(w);
+	wrw->ciov = wrw->niov++;
+	wrw->cliov = wrw->liov;
+	assert(wrw->ciov < wrw->siov);
+}
+
+
 #ifdef SENDFILE_WORKS
 void
 WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)



From phk at varnish-cache.org  Thu Mar 24 14:01:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 15:01:31 +0100
Subject: [master] 306f74b Remove unwarranted assert
Message-ID: <E1Q2l6R-0006ty-Ph@project.varnish-software.com>

commit 306f74bbf50d1f9491dd48df186a142797e75c71
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 12:30:00 2011 +0000

    Remove unwarranted assert

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 179c31f..9e983b7 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -64,7 +64,7 @@ ved_include(struct sess *sp, const char *src, const char *host)
 		return;
 	}
 
-	AZ(WRW_FlushRelease(w));
+	(void)WRW_FlushRelease(w);
 
 	obj = sp->obj;
 	sp->obj = NULL;



From phk at varnish-cache.org  Thu Mar 24 14:01:32 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 15:01:32 +0100
Subject: [master] 6e68c08 Further flesh out the WRW chunked support
Message-ID: <E1Q2l6S-0006uH-6s@project.varnish-software.com>

commit 6e68c08149cc213fed4f13ecc1a96bd68d96fc4e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 13:43:30 2011 +0000

    Further flesh out the WRW chunked support

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 423d452..66a4de7 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -790,6 +790,7 @@ void WRK_SumStat(struct worker *w);
 
 #define WRW_IsReleased(w)	((w)->wrw.wfd == NULL)
 void WRW_Chunked(struct worker *w);
+void WRW_EndChunk(struct worker *w);
 void WRW_Reserve(struct worker *w, int *fd);
 unsigned WRW_Flush(struct worker *w);
 unsigned WRW_FlushRelease(struct worker *w);
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index d3a9a43..84e6004 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -241,6 +241,7 @@ cnt_deliver(struct sess *sp)
 	RES_WriteObj(sp);
 
 	assert(WRW_IsReleased(sp->wrk));
+	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
 	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
 	http_Setup(sp->wrk->resp, NULL);
 	sp->step = STP_DONE;
@@ -1153,6 +1154,7 @@ cnt_recv(struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 	AZ(sp->obj);
+	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
 
 	/* By default we use the first backend */
 	AZ(sp->director);
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index d43ea78..2340aa3 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -115,9 +115,9 @@ WRW_Flush(struct worker *w)
 	if (wrw->ciov < wrw->siov)
 		assert(wrw->niov < wrw->siov);
 
-	if (*wrw->wfd >= 0 && wrw->niov > 0 && wrw->werr == 0) {
-		if (wrw->ciov < wrw->siov && wrw->liov > 0) {
-			bprintf(cbuf, "%jx\r\n", (intmax_t)wrw->cliov);
+	if (*wrw->wfd >= 0 && wrw->liov > 0 && wrw->werr == 0) {
+		if (wrw->ciov < wrw->siov && wrw->cliov > 0) {
+			bprintf(cbuf, "00%jx\r\n", (intmax_t)wrw->cliov);
 			i = strlen(cbuf);
 			wrw->iov[wrw->ciov].iov_base = cbuf;
 			wrw->iov[wrw->ciov].iov_len = i;
@@ -126,6 +126,9 @@ WRW_Flush(struct worker *w)
 			wrw->iov[wrw->niov].iov_base = cbuf + i - 2;
 			wrw->iov[wrw->niov++].iov_len = 2;
 			wrw->liov += 2;
+		} else if (wrw->ciov < wrw->siov) {
+			wrw->iov[wrw->ciov].iov_base = cbuf;
+			wrw->iov[wrw->ciov].iov_len = 0;
 		}
 		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
 		if (i != wrw->liov) {
@@ -211,8 +214,24 @@ WRW_Chunked(struct worker *w)
 	if (wrw->niov + 3 >= wrw->siov)
 		(void)WRW_Flush(w);
 	wrw->ciov = wrw->niov++;
-	wrw->cliov = wrw->liov;
+	wrw->cliov = 0;
+	assert(wrw->ciov < wrw->siov);
+}
+
+void
+WRW_EndChunk(struct worker *w)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+
 	assert(wrw->ciov < wrw->siov);
+	(void)WRW_Flush(w);
+	wrw->ciov = wrw->siov;
+	(void)WRW_Flush(w);
+	wrw->cliov = 0;
+	WRW_Write(w, "0\r\n\r\n", -1);
 }
 
 



From phk at varnish-cache.org  Thu Mar 24 14:01:32 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 15:01:32 +0100
Subject: [master] 702f640 Leave generation of chunked headers to WRW code
Message-ID: <E1Q2l6S-0006uT-I2@project.varnish-software.com>

commit 702f6401ce51279114cfeda01e05e10a64c113e0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 13:49:30 2011 +0000

    Leave generation of chunked headers to WRW code

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 9e983b7..2295358 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -120,6 +120,8 @@ ved_include(struct sess *sp, const char *src, const char *host)
 	WS_Reset(sp->ws, ws_wm);
 
 	WRW_Reserve(sp->wrk, &sp->fd);
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		WRW_Chunked(sp->wrk);
 }
 
 /*--------------------------------------------------------------------*/
@@ -128,28 +130,6 @@ ved_include(struct sess *sp, const char *src, const char *host)
 //#define Debug(fmt, ...) printf(fmt, __VA_ARGS__)
 #define Debug(fmt, ...) /**/
 
-static void
-ved_sendchunk(const struct sess *sp, const void *cb, ssize_t cl,
-    const void *ptr, ssize_t l)
-{
-	char chunk[20];
-
-	assert(l > 0);
-	if (sp->wrk->res_mode & RES_CHUNKED) {
-		if (cb == NULL) {
-			bprintf(chunk, "%jx\r\n", (intmax_t)l);
-			(void)WRW_Write(sp->wrk, chunk, -1);
-		} else
-			(void)WRW_Write(sp->wrk, cb, cl);
-	}
-	(void)WRW_Write(sp->wrk, ptr, l);
-	if (sp->wrk->res_mode & RES_CHUNKED) {
-		(void)WRW_Write(sp->wrk, "\r\n", -1);
-		if (cb == NULL)
-			(void)WRW_Flush(sp->wrk);
-	}
-}
-
 static ssize_t
 ved_decode_len(uint8_t **pp)
 {
@@ -200,32 +180,32 @@ ved_decode_len(uint8_t **pp)
 static void
 ved_pretend_gzip(const struct sess *sp, const uint8_t *p, ssize_t l)
 {
-	uint8_t buf[5];
-	char chunk[20];
+	uint8_t buf1[5], buf2[5];
 	uint16_t lx;
 
+	lx = 65535;
+	buf1[0] = 0;
+	vle16enc(buf1 + 1, lx);
+	vle16enc(buf1 + 3, ~lx);
+
 	while (l > 0) {
-		if (l > 65535)
+		if (l >= 65535) {
 			lx = 65535;
-		else
+			(void)WRW_Write(sp->wrk, buf1, sizeof buf1);
+		} else {
 			lx = (uint16_t)l;
-		buf[0] = 0;
-		vle16enc(buf + 1, lx);
-		vle16enc(buf + 3, ~lx);
-		if (sp->wrk->res_mode & RES_CHUNKED) {
-			bprintf(chunk, "%x\r\n", lx + 5);
-			(void)WRW_Write(sp->wrk, chunk, -1);
+			buf2[0] = 0;
+			vle16enc(buf2 + 1, lx);
+			vle16enc(buf2 + 3, ~lx);
+			(void)WRW_Write(sp->wrk, buf2, sizeof buf2);
 		}
-		(void)WRW_Write(sp->wrk, buf, sizeof buf);
 		(void)WRW_Write(sp->wrk, p, lx);
-		if (sp->wrk->res_mode & RES_CHUNKED)
-			(void)WRW_Write(sp->wrk, "\r\n", -1);
-		(void)WRW_Flush(sp->wrk);
 		sp->wrk->crc = crc32(sp->wrk->crc, p, lx);
 		sp->wrk->l_crc += lx;
 		l -= lx;
 		p += lx;
 	}
+	(void)WRW_Flush(sp->wrk);
 }
 
 /*---------------------------------------------------------------------
@@ -280,7 +260,7 @@ ESI_Deliver(struct sess *sp)
 		if (isgzip && !(sp->wrk->res_mode & RES_GUNZIP)) {
 			assert(sizeof gzip_hdr == 10);
 			/* Send out the gzip header */
-			ved_sendchunk(sp, "a\r\n", 3, gzip_hdr, 10);
+			WRW_Write(sp->wrk, gzip_hdr, 10);
 			sp->wrk->l_crc = 0;
 			sp->wrk->gzip_resp = 1;
 			sp->wrk->crc = crc32(0L, Z_NULL, 0);
@@ -328,7 +308,7 @@ ESI_Deliver(struct sess *sp)
 				sp->wrk->crc = crc32_combine(
 				    sp->wrk->crc, icrc, l_icrc);
 				sp->wrk->l_crc += l_icrc;
-				ved_sendchunk(sp, r, q - r, st->ptr + off, l);
+				WRW_Write(sp->wrk, st->ptr + off, l);
 			} else if (sp->wrk->gzip_resp) {
 				/*
 				 * A gzip'ed ESI response, but the VEC was
@@ -344,8 +324,7 @@ ESI_Deliver(struct sess *sp)
 				VGZ_Ibuf(vgz, st->ptr + off, l);
 				do {
 					if (obufl == sizeof obuf) {
-						ved_sendchunk(sp, NULL, 0,
-						    obuf, obufl);
+						WRW_Write(sp->wrk, obuf, obufl);
 						obufl = 0;
 					}
 					VGZ_Obuf(vgz, obuf + obufl,
@@ -358,7 +337,7 @@ ESI_Deliver(struct sess *sp)
 				/*
 				 * Ungzip'ed VEC, ungzip'ed ESI response
 				 */
-				ved_sendchunk(sp, r, q - r, st->ptr + off, l);
+				WRW_Write(sp->wrk, st->ptr + off, l);
 			}
 			off += l;
 			break;
@@ -377,8 +356,7 @@ ESI_Deliver(struct sess *sp)
 			r = (void*)strchr((const char*)q, '\0');
 			AN(r);
 			if (obufl > 0) {
-				ved_sendchunk(sp, NULL, 0,
-				    obuf, obufl);
+				WRW_Write(sp->wrk, obuf, obufl);
 				obufl = 0;
 			}
 			Debug("INCL [%s][%s] BEGIN\n", q, p);
@@ -393,7 +371,7 @@ ESI_Deliver(struct sess *sp)
 	}
 	if (vgz != NULL) {
 		if (obufl > 0)
-			ved_sendchunk(sp, NULL, 0, obuf, obufl);
+			WRW_Write(sp->wrk, obuf, obufl);
 		VGZ_Destroy(&vgz);
 	}
 	if (sp->wrk->gzip_resp && sp->esi_level == 0) {
@@ -410,7 +388,7 @@ ESI_Deliver(struct sess *sp)
 		/* MOD(2^32) length */
 		vle32enc(tailbuf + 9, sp->wrk->l_crc);
 
-		ved_sendchunk(sp, "d\r\n", 3, tailbuf, 13);
+		WRW_Write(sp->wrk, tailbuf, 13);
 	}
 	(void)WRW_Flush(sp->wrk);
 }
@@ -450,7 +428,7 @@ ved_deliver_byterange(const struct sess *sp, ssize_t low, ssize_t high)
 //printf("[2-] %jd %jd\n", lx, lx + l);
 		assert(lx >= low && lx + l <= high);
 		if (l != 0)
-			ved_sendchunk(sp, NULL, 0, p, l);
+			WRW_Write(sp->wrk, p, l);
 		if (lx + st->len > high)
 			return(p[l]);
 		lx += st->len;
@@ -502,7 +480,7 @@ ESI_DeliverChild(const struct sess *sp)
 	 */
 	*dbits = ved_deliver_byterange(sp, start/8, last/8);
 	*dbits &= ~(1U << (last & 7));
-	ved_sendchunk(sp, NULL, 0, dbits, 1);
+	WRW_Write(sp->wrk, dbits, 1);
 	cc = ved_deliver_byterange(sp, 1 + last/8, stop/8);
 	switch((int)(stop & 7)) {
 	case 0: /* xxxxxxxx */
@@ -546,7 +524,7 @@ ESI_DeliverChild(const struct sess *sp)
 		INCOMPL();
 	}
 	if (lpad > 0)
-		ved_sendchunk(sp, NULL, 0, dbits + 1, lpad);
+		WRW_Write(sp->wrk, dbits + 1, lpad);
 	st = VTAILQ_LAST(&sp->obj->store, storagehead);
 	assert(st->len > 8);
 
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 99ca6c3..5b52934 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -145,6 +145,7 @@ wrk_thread_real(struct wq *qp, unsigned shm_workspace, unsigned sess_workspace,
 	w->resp = HTTP_create(http2, nhttp);
 	w->wrw.iov = iov;
 	w->wrw.siov = siov;
+	w->wrw.ciov = siov;
 	AZ(pthread_cond_init(&w->cond, NULL));
 
 	WS_Init(w->ws, "wrk", ws, sess_workspace);
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
index 96ff344..8098d64 100644
--- a/bin/varnishd/cache_response.c
+++ b/bin/varnishd/cache_response.c
@@ -246,7 +246,6 @@ res_WriteGunzipObj(struct sess *sp)
 	unsigned u = 0;
 	struct vgz *vg;
 	const void *dp;
-	char lenbuf[20];
 	char obuf[params->gzip_stack_buffer];
 	size_t dl;
 	int i;
@@ -268,13 +267,7 @@ res_WriteGunzipObj(struct sess *sp)
 			VGZ_Obuf(vg, obuf, sizeof obuf);
 			i = VGZ_Gunzip(vg, &dp, &dl);
 			if (dl != 0) {
-				if (sp->wrk->res_mode & RES_CHUNKED) {
-					bprintf(lenbuf, "%x\r\n", (unsigned)dl);
-					(void)WRW_Write(sp->wrk, lenbuf, -1);
-				}
 				(void)WRW_Write(sp->wrk, dp, dl);
-				if (sp->wrk->res_mode & RES_CHUNKED)
-					(void)WRW_Write(sp->wrk, "\r\n", -1);
 				if (WRW_Flush(sp->wrk))
 					break;
 			}
@@ -288,7 +281,7 @@ res_WriteGunzipObj(struct sess *sp)
 /*--------------------------------------------------------------------*/
 
 static void
-res_WriteDirObj(struct sess *sp, char *lenbuf, size_t low, size_t high)
+res_WriteDirObj(struct sess *sp, size_t low, size_t high)
 {
 	unsigned u = 0;
 	size_t ptr, off, len;
@@ -296,11 +289,6 @@ res_WriteDirObj(struct sess *sp, char *lenbuf, size_t low, size_t high)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
-	if (sp->wrk->res_mode & RES_CHUNKED) {
-		sprintf(lenbuf, "%jx\r\n", (intmax_t)sp->obj->len);
-		(void)WRW_Write(sp->wrk, lenbuf, -1);
-	}
-
 	ptr = 0;
 	VTAILQ_FOREACH(st, &sp->obj->store, list) {
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
@@ -344,8 +332,6 @@ res_WriteDirObj(struct sess *sp, char *lenbuf, size_t low, size_t high)
 		(void)WRW_Write(sp->wrk, st->ptr + off, len);
 	}
 	assert(u == sp->obj->len);
-	if (sp->wrk->res_mode & RES_CHUNKED)
-		(void)WRW_Write(sp->wrk, "\r\n", -1);
 }
 
 /*--------------------------------------------------------------------*/
@@ -355,7 +341,6 @@ RES_WriteObj(struct sess *sp)
 {
 	char *r;
 	unsigned low, high;
-	char lenbuf[20];
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
@@ -384,6 +369,9 @@ RES_WriteObj(struct sess *sp)
 		sp->acct_tmp.hdrbytes +=
 		    http_Write(sp->wrk, sp->wrk->resp, 1);
 
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		WRW_Chunked(sp->wrk);
+
 	if (!sp->wantbody) {
 		/* This was a HEAD request */
 	} else if (sp->obj->len == 0) {
@@ -395,12 +383,12 @@ RES_WriteObj(struct sess *sp)
 	} else if (sp->wrk->res_mode & RES_GUNZIP) {
 		res_WriteGunzipObj(sp);
 	} else {
-		res_WriteDirObj(sp, lenbuf, low, high);
+		res_WriteDirObj(sp, low, high);
 	}
 
 	if (sp->wrk->res_mode & RES_CHUNKED &&
 	    !(sp->wrk->res_mode & RES_ESI_CHILD))
-		(void)WRW_Write(sp->wrk, "0\r\n\r\n", -1);
+		WRW_EndChunk(sp->wrk);
 
 	if (WRW_FlushRelease(sp->wrk))
 		vca_close_session(sp, "remote closed");



From phk at varnish-cache.org  Thu Mar 24 14:01:34 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Mar 2011 15:01:34 +0100
Subject: [master] 285deaa Don't put chunked headers in the esi-specification,
	we won't need them.
Message-ID: <E1Q2l6U-0006up-8P@project.varnish-software.com>

commit 285deaaf1dc357923231f1506dd59672fb78f001
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 24 14:00:57 2011 +0000

    Don't put chunked headers in the esi-specification, we won't need them.

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 2295358..22cb765 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -211,7 +211,7 @@ ved_pretend_gzip(const struct sess *sp, const uint8_t *p, ssize_t l)
 /*---------------------------------------------------------------------
  */
 
-static const char gzip_hdr[] = {
+static const uint8_t gzip_hdr[] = {
 	0x1f, 0x8b, 0x08,
 	0x00, 0x00, 0x00, 0x00,
 	0x00,
@@ -291,8 +291,6 @@ ESI_Deliver(struct sess *sp)
 		case VEC_V8:
 			l = ved_decode_len(&p);
 			r = p;
-			q = (void*)strchr((const char*)p, '\0');
-			p = q + 1;
 			if (isgzip) {
 				assert(*p == VEC_C1 || *p == VEC_C2 ||
 				    *p == VEC_C8);
diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
index 937a857..48ca71e 100644
--- a/bin/varnishd/cache_esi_parse.c
+++ b/bin/varnishd/cache_esi_parse.c
@@ -287,8 +287,6 @@ vep_emit_verbatim(const struct vep_state *vep, ssize_t l, ssize_t l_crc)
 		Debug("---> VERBATIM(%jd)\n", (intmax_t)l);
 	}
 	vep_emit_len(vep, l, VEC_V1, VEC_V2, VEC_V8);
-	/* Emit Chunked header */
-	vsb_printf(vep->vsb, "%lx\r\n%c", l, 0);
 	if (vep->dogzip) {
 		vep_emit_len(vep, l_crc, VEC_C1, VEC_C2, VEC_C8);
 		vbe32enc(buf, vep->crc);



From phk at varnish-cache.org  Fri Mar 25 09:54:28 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 25 Mar 2011 10:54:28 +0100
Subject: [master] 27422d8 Unify code path for abandoning a fetch.
Message-ID: <E1Q33iu-0005Vv-OZ@project.varnish-software.com>

commit 27422d81987612e113cf4cbb234d8945d77d9806
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Mar 25 09:54:01 2011 +0000

    Unify code path for abandoning a fetch.

diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 84e6004..45470f6 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -470,6 +470,7 @@ cnt_fetch(struct sess *sp)
 	AZ(sp->vbc);
 	AZ(sp->wrk->h_content_length);
 	AZ(sp->wrk->do_close);
+	AZ(sp->wrk->storage_hint);
 
 	http_Setup(sp->wrk->beresp, sp->wrk->ws);
 
@@ -485,77 +486,62 @@ cnt_fetch(struct sess *sp)
 	}
 
 	if (i) {
-		if (sp->objcore != NULL) {
-			CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-			AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-			sp->objcore = NULL;
-		}
-		AZ(sp->obj);
-		sp->wrk->do_close = 0;
-		sp->wrk->h_content_length = NULL;
-		http_Setup(sp->wrk->bereq, NULL);
-		http_Setup(sp->wrk->beresp, NULL);
+		sp->handling = VCL_RET_ERROR;
 		sp->err_code = 503;
-		sp->step = STP_ERROR;
-		return (0);
-	}
-
-	/*
-	 * These two headers can be spread over multiple actual headers
-	 * and we rely on their content outside of VCL, so collect them
-	 * into one line here.
-	 */
-	http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
-	http_CollectHdr(sp->wrk->beresp, H_Vary);
-
-	/*
-	 * Figure out how the fetch is supposed to happen, before the
-	 * headers are adultered by VCL
-	 * Also sets other sp->wrk variables 
-	 */ 
-	sp->wrk->body_status = RFC2616_Body(sp);
+		sp->wrk->do_close = 0;
+	} else {
+		/*
+		 * These two headers can be spread over multiple actual headers
+		 * and we rely on their content outside of VCL, so collect them
+		 * into one line here.
+		 */
+		http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
+		http_CollectHdr(sp->wrk->beresp, H_Vary);
 
+		/*
+		 * Figure out how the fetch is supposed to happen, before the
+		 * headers are adultered by VCL
+		 * NB: Also sets other sp->wrk variables 
+		 */ 
+		sp->wrk->body_status = RFC2616_Body(sp);
 
-	sp->err_code = http_GetStatus(sp->wrk->beresp);
+		sp->err_code = http_GetStatus(sp->wrk->beresp);
 
-	/*
-	 * What does RFC2616 think about TTL ?
-	 */
-	sp->wrk->entered = TIM_real();
-	sp->wrk->age = 0;
-	EXP_Clr(&sp->wrk->exp);
-	sp->wrk->exp.ttl = RFC2616_Ttl(sp);
-
-	/* pass from vclrecv{} has negative TTL */
-	if (sp->objcore == NULL)
-		sp->wrk->exp.ttl = -1.;
+		/*
+		 * What does RFC2616 think about TTL ?
+		 */
+		sp->wrk->entered = TIM_real();
+		sp->wrk->age = 0;
+		EXP_Clr(&sp->wrk->exp);
+		sp->wrk->exp.ttl = RFC2616_Ttl(sp);
 
-	sp->wrk->do_esi = 0;
+		/* pass from vclrecv{} has negative TTL */
+		if (sp->objcore == NULL)
+			sp->wrk->exp.ttl = -1.;
 
+		sp->wrk->do_esi = 0;
 
-	AZ(sp->wrk->storage_hint);
+		VCL_fetch_method(sp);
 
-	VCL_fetch_method(sp);
+		switch (sp->handling) {
+		case VCL_RET_HIT_FOR_PASS:
+			if (sp->objcore != NULL)
+				sp->objcore->flags |= OC_F_PASS;
+			sp->step = STP_FETCHBODY;
+			return (0);
+		case VCL_RET_DELIVER:
+			sp->step = STP_FETCHBODY;
+			return (0);
+		default:
+			break;
+		}
 
-	switch (sp->handling) {
-	case VCL_RET_HIT_FOR_PASS:
-		if (sp->objcore != NULL)
-			sp->objcore->flags |= OC_F_PASS;
-		sp->step = STP_FETCHBODY;
-		return (0);
-	case VCL_RET_DELIVER:
-		sp->step = STP_FETCHBODY;
-		return (0);
-	default:
-		break;
+		/* We are not going to fetch the body, Close the connection */
+		VDI_CloseFd(sp);
 	}
 
-	/*
-	 * We are not going to fetch the body
-	 * Close the connection and clean up...
-	 */
+	/* Clean up partial fetch */
 
-	VDI_CloseFd(sp);
 	if (sp->objcore != NULL) {
 		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
 		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
@@ -717,7 +703,6 @@ cnt_fetchbody(struct sess *sp)
 		AN(sp->obj->vary);
 		memcpy(sp->obj->vary, vsb_data(vary), varyl);
 		vsb_delete(vary);
-		vary = NULL;
 	}
 
 	sp->obj->xid = sp->xid;



From phk at varnish-cache.org  Mon Mar 28 08:10:59 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 10:10:59 +0200
Subject: [master] 3a56dbd Also correctly initialize subsequent references to
	vmods.
Message-ID: <E1Q47XP-0002Sz-QM@project.varnish-software.com>

commit 3a56dbdd70cfddfe4e497b0ff94e4c094c8f64f1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 08:10:12 2011 +0000

    Also correctly initialize subsequent references to vmods.
    
    Fixes:	878
    
    Solved by:	tmagnien

diff --git a/bin/varnishd/cache_vrt_vmod.c b/bin/varnishd/cache_vrt_vmod.c
index 0c74e02..77e37e7 100644
--- a/bin/varnishd/cache_vrt_vmod.c
+++ b/bin/varnishd/cache_vrt_vmod.c
@@ -90,19 +90,20 @@ VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm, const char *path)
 
 		x = dlsym(v->hdl, "Vmod_Name");
 		AN(x);
+		/* XXX: check that name is correct */
 
 		x = dlsym(v->hdl, "Vmod_Len");
 		AN(x);
 		i = x;
-		assert(len == *i);
+		v->funclen = *i;
 
 		x = dlsym(v->hdl, "Vmod_Func");
 		AN(x);
-		memcpy(ptr, x, len);
-
 		v->funcs = x;
-		v->funclen = *i;
 	}
+
+	assert(len == v->funclen);
+	memcpy(ptr, v->funcs, v->funclen);
 	v->ref++;
 
 	*hdl = v;
diff --git a/bin/varnishtest/tests/r00878.vtc b/bin/varnishtest/tests/r00878.vtc
new file mode 100644
index 0000000..c57e511
--- /dev/null
+++ b/bin/varnishtest/tests/r00878.vtc
@@ -0,0 +1,44 @@
+#!/bin/sh
+
+test "Loading vmods in subsequent VCLs"
+
+server s1 {
+	rxreq 
+	txresp -bodylen 4
+} -start
+
+varnish v1 -vcl+backend {
+        import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+	sub vcl_deliver {
+		set resp.http.who = std.author(phk);
+	}
+} -start
+
+
+client c1 {
+	txreq
+	rxresp
+} -run
+varnish v1 -vcl+backend {
+        import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+	sub vcl_deliver {
+		set resp.http.who = std.author(des);
+	}
+}
+
+client c1 {
+	txreq
+	rxresp
+} -run
+
+varnish v1 -vcl+backend {
+        import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so" ;
+	sub vcl_deliver {
+		set resp.http.who = std.author(kristian);
+	}
+}
+
+client c1 {
+	txreq
+	rxresp
+} -run



From phk at varnish-cache.org  Mon Mar 28 08:27:14 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 10:27:14 +0200
Subject: [master] f378c80 If we fail to allocate storage,
	it can be because the request is too large to serve in one bite,
	so chop it down until we reach fetch_chunksize before we start LRU
	flushing to make space for it.
Message-ID: <E1Q47n8-0003dg-Gc@project.varnish-software.com>

commit f378c80aec1e2b958d8bc9d67cb9fa203b5702fb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 08:25:53 2011 +0000

    If we fail to allocate storage, it can be because the request is
    too large to serve in one bite, so chop it down until we reach
    fetch_chunksize before we start LRU flushing to make space for it.
    
    Fixes #880

diff --git a/bin/varnishd/stevedore.c b/bin/varnishd/stevedore.c
index 1fcf75c..fbc9557 100644
--- a/bin/varnishd/stevedore.c
+++ b/bin/varnishd/stevedore.c
@@ -134,6 +134,11 @@ stv_alloc(const struct sess *sp, size_t size)
 		if (st != NULL)
 			break;
 
+		if (size > params->fetch_chunksize) {
+			size >>= 1;
+			continue;
+		}
+
 		/* no luck; try to free some space and keep trying */
 		if (EXP_NukeOne(sp, stv->lru) == -1)
 			break;



From phk at varnish-cache.org  Mon Mar 28 09:17:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 11:17:17 +0200
Subject: [master] 3cf1648 Make gzip data corruption on fetch a non-assert
	fetch error.
Message-ID: <E1Q48ZZ-0007TG-UI@project.varnish-software.com>

commit 3cf16486c61fca2e4502c0fa34297e29fcb4b4dc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 09:16:57 2011 +0000

    Make gzip data corruption on fetch a non-assert fetch error.

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 50f8c83..61b083c 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -584,11 +584,16 @@ vfp_testgzip_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		while (!VGZ_IbufEmpty(vg)) {
 			VGZ_Obuf(vg, ibuf, sizeof ibuf);
 			i = VGZ_Gunzip(vg, &dp, &dl);
-			assert(i == Z_OK || i == Z_STREAM_END);
+			if (i != Z_OK && i != Z_STREAM_END) {
+				WSP(sp, SLT_FetchError,
+				    "Invalid Gzip data: %s", vg->vz.msg);
+				return (-1);
+			}
 		}
 	}
-	if (i == Z_OK || i == Z_STREAM_END)
+	if (i == Z_STREAM_END)
 		return (1);
+	WSP(sp, SLT_FetchError, "Incomplete Gzip data (not STREAM_END)");
 	return (-1);
 }
 



From phk at varnish-cache.org  Mon Mar 28 10:28:14 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 12:28:14 +0200
Subject: [master] ec66fc0 Proper casting for debugging printout
Message-ID: <E1Q49gE-0005pZ-Om@project.varnish-software.com>

commit ec66fc07e63e4afd1a39f41cadc4d91c46704ed0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 10:27:43 2011 +0000

    Proper casting for debugging printout

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 7d0dba2..66c0308 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -606,11 +606,14 @@ gzip_body(struct http *hp, const char *txt, char **body, int *bodylen)
 		    i, hp->gzipresidual);
 	*bodylen = vz.total_out;
 	vtc_log(hp->vl, 4, "startbit = %ju %ju/%ju",
-	    vz.start_bit, vz.start_bit >> 3, vz.start_bit & 7);
+	    (uintmax_t)vz.start_bit,
+	    (uintmax_t)vz.start_bit >> 3, (uintmax_t)vz.start_bit & 7);
 	vtc_log(hp->vl, 4, "lastbit = %ju %ju/%ju",
-	    vz.last_bit, vz.last_bit >> 3, vz.last_bit & 7);
+	    (uintmax_t)vz.last_bit,
+	    (uintmax_t)vz.last_bit >> 3, (uintmax_t)vz.last_bit & 7);
 	vtc_log(hp->vl, 4, "stopbit = %ju %ju/%ju",
-	    vz.stop_bit, vz.stop_bit >> 3, vz.stop_bit & 7);
+	    (uintmax_t)vz.stop_bit,
+	    (uintmax_t)vz.stop_bit >> 3, (uintmax_t)vz.stop_bit & 7);
 	assert(Z_OK == deflateEnd(&vz));
 }
 



From phk at varnish-cache.org  Mon Mar 28 10:36:12 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 12:36:12 +0200
Subject: [master] 149fde9 Try to solve the 32/64 bit issue with a less
	draconian #ifdef, this looks like it helps PPC64 platform.
Message-ID: <E1Q49nw-0006NP-Kx@project.varnish-software.com>

commit 149fde9211490ccfa2c5984abe4c2f7e9eb7cf53
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 10:35:41 2011 +0000

    Try to solve the 32/64 bit issue with a less draconian #ifdef,
    this looks like it helps PPC64 platform.

diff --git a/lib/libvgz/vgz.h b/lib/libvgz/vgz.h
index 1b8da5e..f082e41 100644
--- a/lib/libvgz/vgz.h
+++ b/lib/libvgz/vgz.h
@@ -1575,14 +1575,14 @@ ZEXTERN int ZEXPORT inflateBackInit_ OF((z_streamp strm, int windowBits,
    ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, z_off64_t));
 #endif
 
-#if defined(NOTDEF) && !defined(ZLIB_INTERNAL) && _FILE_OFFSET_BITS-0 == 64 && _LFS64_LARGEFILE-0
+#if !defined(ZLIB_INTERNAL) && _FILE_OFFSET_BITS-0 == 64 && _LFS64_LARGEFILE-0
 #  define gzopen gzopen64
 #  define gzseek gzseek64
 #  define gztell gztell64
 #  define gzoffset gzoffset64
 #  define adler32_combine adler32_combine64
 #  define crc32_combine crc32_combine64
-#  ifdef _LARGEFILE64_SOURCE
+#  ifdef _LARGEFILE64_SOURCExxx
      ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *));
      ZEXTERN z_off_t ZEXPORT gzseek64 OF((gzFile, z_off_t, int));
      ZEXTERN z_off_t ZEXPORT gztell64 OF((gzFile));



From phk at varnish-cache.org  Mon Mar 28 21:42:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Mar 2011 23:42:23 +0200
Subject: [master] fd0ff2b Be more systematic about the returns from
	VGZ_Unzip() and pay attention to the VGZ_STUCK indication.
Message-ID: <E1Q4KCd-0001lo-P4@project.varnish-software.com>

commit fd0ff2b08aeb5dfe0f25cc2503f39fc8f287f230
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 28 21:41:35 2011 +0000

    Be more systematic about the returns from VGZ_Unzip() and pay attention
    to the VGZ_STUCK indication.
    
    Hopefully helps with #869

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 66a4de7..9bb84e6 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -690,6 +690,12 @@ int VGZ_Gunzip(struct vgz *, const void **, size_t *len);
 void VGZ_Destroy(struct vgz **);
 void VGZ_UpdateObj(const struct vgz*, struct object *);
 
+/* Return values */
+#define VGZ_ERROR	-1
+#define VGZ_OK		0
+#define VGZ_END		1
+#define VGZ_STUCK	2
+
 /* cache_http.c */
 unsigned HTTP_estimate(unsigned nhttp);
 void HTTP_Copy(struct http *to, const struct http * const fm);
diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 22cb765..509649f 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -274,7 +274,7 @@ ESI_Deliver(struct sess *sp)
 		VGZ_Ibuf(vgz, gzip_hdr, sizeof gzip_hdr);
 		VGZ_Obuf(vgz, obuf, sizeof obuf);
 		i = VGZ_Gunzip(vgz, &dp, &dl);
-		assert(i == Z_OK || i == Z_STREAM_END);
+		assert(i == VGZ_OK);
 		assert(VGZ_IbufEmpty(vgz));
 		assert(dl == 0);
 
@@ -321,16 +321,20 @@ ESI_Deliver(struct sess *sp)
 				AN(vgz);
 				VGZ_Ibuf(vgz, st->ptr + off, l);
 				do {
-					if (obufl == sizeof obuf) {
-						WRW_Write(sp->wrk, obuf, obufl);
-						obufl = 0;
-					}
 					VGZ_Obuf(vgz, obuf + obufl,
 					    sizeof obuf - obufl);
 					i = VGZ_Gunzip(vgz, &dp, &dl);
-					assert(i == Z_OK || i == Z_STREAM_END);
+					assert(i >= VGZ_OK);
 					obufl += dl;
+					assert(obufl <= sizeof obuf);
+					if (obufl == sizeof obuf ||
+					    i == VGZ_STUCK) {
+						WRW_Write(sp->wrk, obuf, obufl);
+						WRW_Flush(sp->wrk);
+						obufl = 0;
+					}
 				} while (!VGZ_IbufEmpty(vgz));
+				assert (i == VGZ_OK || i == VGZ_END);
 			} else {
 				/*
 				 * Ungzip'ed VEC, ungzip'ed ESI response
diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
index b80de68..8c94460 100644
--- a/bin/varnishd/cache_esi_fetch.c
+++ b/bin/varnishd/cache_esi_fetch.c
@@ -39,7 +39,6 @@ SVNID("$Id")
 #include "cache.h"
 #include "cache_esi.h"
 #include "vct.h"
-#include "vgz.h"
 #include "stevedore.h"
 
 /*---------------------------------------------------------------------
@@ -120,7 +119,7 @@ vfp_esi_bytes_gu(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		if (VGZ_ObufStorage(sp, vg))
 			return (-1);
 		i = VGZ_Gunzip(vg, &dp, &dl);
-		xxxassert(i == Z_OK || i == Z_STREAM_END);
+		xxxassert(i == VGZ_OK || i == VGZ_END);
 		VEP_parse(sp, dp, dl);
 		sp->obj->len += dl;
 	}
@@ -274,7 +273,7 @@ vfp_esi_bytes_gg(struct sess *sp, struct http_conn *htc, size_t bytes)
 			VGZ_Obuf(sp->wrk->vgz_rx, ibuf2, sizeof ibuf2);
 			i = VGZ_Gunzip(sp->wrk->vgz_rx, &dp, &dl);
 			/* XXX: check i */
-			assert(i >= 0);
+			assert(i >= VGZ_OK);
 			vef->bufp = ibuf2;
 			if (dl > 0)
 				VEP_parse(sp, ibuf2, dl);
diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 61b083c..69fdb44 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -300,13 +300,13 @@ VGZ_Gunzip(struct vgz *vg, const void **pptr, size_t *plen)
 			vg->obuf->len += l;
 	}
 	if (i == Z_OK)
-		return (0);
+		return (VGZ_OK);
 	if (i == Z_STREAM_END)
-		return (1);
+		return (VGZ_END);
 	if (i == Z_BUF_ERROR)
-		return (2);
+		return (VGZ_STUCK);
 printf("INFLATE=%d (%s)\n", i, vg->vz.msg);
-	return (-1);
+	return (VGZ_ERROR);
 }
 
 /*--------------------------------------------------------------------*/
@@ -432,7 +432,7 @@ vfp_gunzip_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		if (VGZ_ObufStorage(sp, vg))
 			return (-1);
 		i = VGZ_Gunzip(vg, &dp, &dl);
-		assert(i == Z_OK || i == Z_STREAM_END);
+		assert(i == VGZ_OK || i == VGZ_END);
 		sp->obj->len += dl;
 	}
 	if (i == Z_OK || i == Z_STREAM_END)
@@ -584,7 +584,7 @@ vfp_testgzip_bytes(struct sess *sp, struct http_conn *htc, ssize_t bytes)
 		while (!VGZ_IbufEmpty(vg)) {
 			VGZ_Obuf(vg, ibuf, sizeof ibuf);
 			i = VGZ_Gunzip(vg, &dp, &dl);
-			if (i != Z_OK && i != Z_STREAM_END) {
+			if (i != VGZ_OK && i != VGZ_END) {
 				WSP(sp, SLT_FetchError,
 				    "Invalid Gzip data: %s", vg->vz.msg);
 				return (-1);
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
index 8098d64..c97ffea 100644
--- a/bin/varnishd/cache_response.c
+++ b/bin/varnishd/cache_response.c
@@ -266,13 +266,13 @@ res_WriteGunzipObj(struct sess *sp)
 		do {
 			VGZ_Obuf(vg, obuf, sizeof obuf);
 			i = VGZ_Gunzip(vg, &dp, &dl);
+			assert(i >= VGZ_OK);
 			if (dl != 0) {
 				(void)WRW_Write(sp->wrk, dp, dl);
 				if (WRW_Flush(sp->wrk))
 					break;
 			}
-			assert(i >= 0);
-		} while (i == 0);
+		} while (!VGZ_IbufEmpty(vg));
 	}
 	VGZ_Destroy(&vg);
 	assert(u == sp->obj->len);



From tfheen at varnish-cache.org  Tue Mar 29 07:23:13 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 29 Mar 2011 09:23:13 +0200
Subject: [master] 714bde1 Build varnish-cli.7 from rst
Message-ID: <E1Q4TGj-0000yK-Kh@project.varnish-software.com>

commit 714bde15e8f786f0fc1009229af76501d4ec6a8a
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 29 09:22:26 2011 +0200

    Build varnish-cli.7 from rst
    
    Generate varnish-cli.7 and clean up the generated man pages on
    maintainer-clean

diff --git a/man/Makefile.am b/man/Makefile.am
index f0d23f0..2a1e36c 100644
--- a/man/Makefile.am
+++ b/man/Makefile.am
@@ -1,6 +1,7 @@
 # $Id$
 
-dist_man_MANS = vcl.7
+dist_man_MANS = vcl.7 varnish-cli.7
+MAINTAINERCLEANFILES = $(dist_man_MANS)
 
 vcl.7: $(top_srcdir)/doc/sphinx/reference/vcl.rst \
 	$(top_srcdir)/bin/varnishd/default.vcl
@@ -12,3 +13,13 @@ else
 	@echo "========================================"
 	@false
 endif
+
+varnish-cli.7: $(top_srcdir)/doc/sphinx/reference/varnish-cli.rst
+if HAVE_RST2MAN
+	${RST2MAN} $(top_srcdir)/doc/sphinx/reference/varnish-cli.rst $@
+else
+	@echo "========================================"
+	@echo "You need rst2man installed to make dist"
+	@echo "========================================"
+	@false
+endif



From tfheen at varnish-cache.org  Tue Mar 29 07:52:07 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 29 Mar 2011 09:52:07 +0200
Subject: [master] 8850b65 Add varnish-cli.rst to list of distributed files
Message-ID: <E1Q4Tih-00060C-6a@project.varnish-software.com>

commit 8850b659c74a9a5b7738bd5e3b597e96283e4382
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Mar 29 09:52:00 2011 +0200

    Add varnish-cli.rst to list of distributed files

diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am
index 8fac4af..73371db 100644
--- a/doc/sphinx/Makefile.am
+++ b/doc/sphinx/Makefile.am
@@ -123,6 +123,7 @@ EXTRA_DIST = \
 	reference/varnishstat.rst \
 	reference/varnishtest.rst \
 	reference/varnishtop.rst \
+	reference/varnish-cli.rst \
 	reference/vcl.rst \
 	reference/vmod.rst \
 	tutorial/advanced_backend_servers.rst \



From phk at varnish-cache.org  Tue Mar 29 09:18:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Mar 2011 11:18:40 +0200
Subject: [master] bfde25a Minor cleanups.
Message-ID: <E1Q4V4S-0006iC-BI@project.varnish-software.com>

commit bfde25ad811ccb19bbc9303b31922260dd0b74be
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 09:18:22 2011 +0000

    Minor cleanups.

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 509649f..aec0961 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -205,6 +205,7 @@ ved_pretend_gzip(const struct sess *sp, const uint8_t *p, ssize_t l)
 		l -= lx;
 		p += lx;
 	}
+	/* buf2 is local, have to flush */
 	(void)WRW_Flush(sp->wrk);
 }
 
diff --git a/bin/varnishd/stevedore.c b/bin/varnishd/stevedore.c
index fbc9557..5344bac 100644
--- a/bin/varnishd/stevedore.c
+++ b/bin/varnishd/stevedore.c
@@ -53,7 +53,51 @@ static VTAILQ_HEAD(, stevedore)	stevedores =
 static const struct stevedore * volatile stv_next;
 
 static struct stevedore *stv_transient;
-static struct objcore_methods default_oc_methods;
+
+/*---------------------------------------------------------------------
+ * Default objcore methods
+ */
+
+static struct object * __match_proto__(getobj_f)
+default_oc_getobj(struct worker *wrk, struct objcore *oc)
+{
+	struct object *o;
+
+	(void)wrk;
+	if (oc->priv == NULL)
+		return (NULL);
+	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
+	return (o);
+}
+
+static void
+default_oc_freeobj(struct objcore *oc)
+{
+	struct object *o;
+
+	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
+	oc->priv = NULL;
+	oc->methods = NULL;
+
+	STV_Freestore(o);
+	STV_free(o->objstore);
+}
+
+static struct lru *
+default_oc_getlru(const struct objcore *oc)
+{
+	struct object *o;
+
+	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
+	return (o->objstore->stevedore->lru);
+}
+
+static struct objcore_methods default_oc_methods = {
+	.getobj = default_oc_getobj,
+	.freeobj = default_oc_freeobj,
+	.getlru = default_oc_getlru,
+};
+
 
 /*--------------------------------------------------------------------
  */
@@ -306,50 +350,6 @@ STV_Freestore(struct object *o)
 	}
 }
 
-/*---------------------------------------------------------------------
- * Default objcore methods
- */
-
-static struct object * __match_proto__(getobj_f)
-default_oc_getobj(struct worker *wrk, struct objcore *oc)
-{
-	struct object *o;
-
-	(void)wrk;
-	if (oc->priv == NULL)
-		return (NULL);
-	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
-	return (o);
-}
-
-static void
-default_oc_freeobj(struct objcore *oc)
-{
-	struct object *o;
-
-	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
-	oc->priv = NULL;
-	oc->methods = NULL;
-
-	STV_Freestore(o);
-	STV_free(o->objstore);
-}
-
-static struct lru *
-default_oc_getlru(const struct objcore *oc)
-{
-	struct object *o;
-
-	CAST_OBJ_NOTNULL(o, oc->priv, OBJECT_MAGIC);
-	return (o->objstore->stevedore->lru);
-}
-
-static struct objcore_methods default_oc_methods = {
-	.getobj = default_oc_getobj,
-	.freeobj = default_oc_freeobj,
-	.getlru = default_oc_getlru,
-};
-
 /*-------------------------------------------------------------------*/
 
 struct storage *



From phk at varnish-cache.org  Tue Mar 29 09:18:40 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Mar 2011 11:18:40 +0200
Subject: [master] 0393086 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Q4V4S-0006iK-PO@project.varnish-software.com>

commit 0393086a6a2b82a0abf025136cca32c884c59584
Merge: bfde25a 8850b65
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 09:18:38 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Tue Mar 29 09:36:06 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Mar 2011 11:36:06 +0200
Subject: [master] 79c2d96 Minor cleanups from FlexeLint
Message-ID: <E1Q4VLK-0007uw-M6@project.varnish-software.com>

commit 79c2d962221bb7ce3582caa1a0be5df4841e0832
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 09:35:56 2011 +0000

    Minor cleanups from FlexeLint

diff --git a/bin/varnishncsa/flint.lnt b/bin/varnishncsa/flint.lnt
index 7614a7e..ce8b3f4 100644
--- a/bin/varnishncsa/flint.lnt
+++ b/bin/varnishncsa/flint.lnt
@@ -33,4 +33,5 @@
 -e557	// unrecog format
 -e732   // Loss of sign (arg. no. 2) (int to unsigned
 -e737	// [45]  Loss of sign in promotion from int to unsigned
+-e788	// tag not used in defaulted switch
 
diff --git a/bin/varnishncsa/flint.sh b/bin/varnishncsa/flint.sh
index 8b65b9f..f7d7988 100755
--- a/bin/varnishncsa/flint.sh
+++ b/bin/varnishncsa/flint.sh
@@ -4,6 +4,7 @@ T=/tmp/_$$
 flexelint \
 	-I/usr/include \
 	-I. \
+	-I../.. \
 	-I../../include \
 	../flint.lnt \
 	flint.lnt \
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 396111c..8367ba4 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -65,14 +65,12 @@
 SVNID("$Id$")
 
 #include <ctype.h>
-#include <errno.h>
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
-#include <time.h>
+#include <strings.h>
 #include <unistd.h>
-#include <limits.h>
 
 #include "compat/daemon.h"
 
@@ -112,7 +110,7 @@ static size_t nll;
 
 static int o_flag = 0;
 static int match_tag;
-static vre_t *match_tag_re;
+static const vre_t *match_tag_re;
 
 static const char *format;
 
@@ -677,6 +675,7 @@ h_ncsa(void *priv, enum vsl_tag tag, unsigned fd,
 				}
 			}
 			/* Fall through if we haven't handled something */
+			/* FALLTHROUGH*/
 		default:
 			fprintf(stderr, "Unknown format character: %c\n", *p);
 			exit(1);



From phk at varnish-cache.org  Tue Mar 29 09:43:26 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Mar 2011 11:43:26 +0200
Subject: [master] 12a85b9 Another nit, trying to make opensolaris compile
Message-ID: <E1Q4VSQ-0008R9-Fo@project.varnish-software.com>

commit 12a85b93450ac455eb08ecbd34c8f78835daeaed
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 09:43:11 2011 +0000

    Another nit, trying to make opensolaris compile

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 20f7fac..c2066b0 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -271,7 +271,7 @@ start_test(void)
 	memset(jp->buf, 0, jp->bufsiz);
 
 	srandomdev();
-	bprintf(tmpdir, "/tmp/vtc.%d.%08x", getpid(), (unsigned)random());
+	bprintf(tmpdir, "/tmp/vtc.%d.%08x", (int)getpid(), (unsigned)random());
 	AZ(mkdir(tmpdir, 0711));
 
 	tp = VTAILQ_FIRST(&tst_head);



From phk at varnish-cache.org  Tue Mar 29 10:40:44 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Mar 2011 12:40:44 +0200
Subject: [master] 9434db5 More nitpickery to make OpenSolaris compile
Message-ID: <E1Q4WLs-0003wQ-Vh@project.varnish-software.com>

commit 9434db579ee07c9b97a945d89d7578d8663ee4ad
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 10:40:08 2011 +0000

    More nitpickery to make OpenSolaris compile
    
    Having printw() take a "char*" rather than "const char*" as fmt argument
    feels sooo 1980 to me...

diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index d00e0a6..8ce1690 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -224,10 +224,11 @@ do_curses(struct VSM_data *vd, const struct vsc_main *VSC_main,
 					    (ju >> 24) & 0xffffffffffLL));
 					for (ch = 0x800000; ch; ch >>= 1)
 						if (ju & ch)
-							AC(printw("V"));
+							AC(addstr("V"));
 						else
-							AC(printw("_"));
-					AC(printw(" %s", pt->name));
+							AC(addstr("_"));
+					AC(addstr(" "));
+					AC(addstr(pt->name));
 				} else {
 					AC(mvprintw(line, 0,
 					    "%12ju %12s %12s %s\n",



From tfheen at varnish-cache.org  Wed Mar 30 07:09:50 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 30 Mar 2011 09:09:50 +0200
Subject: [master] e01527c Stop stripping rpath when building RPMs
Message-ID: <E1Q4pXK-0001Oo-El@project.varnish-software.com>

commit e01527c2a72649436c7655ae9f1cc648bbf04154
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Mar 30 09:08:48 2011 +0200

    Stop stripping rpath when building RPMs
    
    The varnish tools need some private libraries and find then via rpath,
    so avoid stripping the rpath when building RPMs

diff --git a/redhat/varnish.spec b/redhat/varnish.spec
index 3ea0b11..13e628e 100644
--- a/redhat/varnish.spec
+++ b/redhat/varnish.spec
@@ -92,8 +92,8 @@ cp bin/varnishd/default.vcl etc/zope-plone.vcl examples
 
 # We have to remove rpath - not allowed in Fedora
 # (This problem only visible on 64 bit arches)
-sed -i 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g;
-	s|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' libtool
+#sed -i 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g;
+#	s|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' libtool
 
 %{__make} %{?_smp_mflags}
 



From phk at varnish-cache.org  Wed Mar 30 10:01:05 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Mar 2011 12:01:05 +0200
Subject: [master] 4fac23d Unify the task of gunzip'ing a chunk of data and
	putting it in a buf to be flushed to WRW.
Message-ID: <E1Q4sD3-0003Z1-F8@project.varnish-software.com>

commit 4fac23da161b8885f4f4c35876ac30b971015d2a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 29 12:00:15 2011 +0000

    Unify the task of gunzip'ing a chunk of data and putting it in
    a buf to be flushed to WRW.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 9bb84e6..337f3ee 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -689,6 +689,8 @@ int VGZ_Gzip(struct vgz *, const void **, size_t *len, enum vgz_flag);
 int VGZ_Gunzip(struct vgz *, const void **, size_t *len);
 void VGZ_Destroy(struct vgz **);
 void VGZ_UpdateObj(const struct vgz*, struct object *);
+int VGZ_WrwGunzip(struct sess *, struct vgz *, void *ibuf, ssize_t ibufl,
+    char *obuf, ssize_t obufl, ssize_t *obufp);
 
 /* Return values */
 #define VGZ_ERROR	-1
diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index aec0961..b074cd5 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -320,21 +320,9 @@ ESI_Deliver(struct sess *sp)
 				 * A gzip'ed VEC, but ungzip'ed ESI response
 				 */
 				AN(vgz);
-				VGZ_Ibuf(vgz, st->ptr + off, l);
-				do {
-					VGZ_Obuf(vgz, obuf + obufl,
-					    sizeof obuf - obufl);
-					i = VGZ_Gunzip(vgz, &dp, &dl);
-					assert(i >= VGZ_OK);
-					obufl += dl;
-					assert(obufl <= sizeof obuf);
-					if (obufl == sizeof obuf ||
-					    i == VGZ_STUCK) {
-						WRW_Write(sp->wrk, obuf, obufl);
-						WRW_Flush(sp->wrk);
-						obufl = 0;
-					}
-				} while (!VGZ_IbufEmpty(vgz));
+				i = VGZ_WrwGunzip(sp, vgz,
+					st->ptr + off, l,
+					obuf, sizeof obuf, &obufl);
 				assert (i == VGZ_OK || i == VGZ_END);
 			} else {
 				/*
diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 69fdb44..0222cfd 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -350,6 +350,40 @@ VGZ_Gzip(struct vgz *vg, const void **pptr, size_t *plen, enum vgz_flag flags)
 	return (-1);
 }
 
+/*--------------------------------------------------------------------
+ * Gunzip ibuf into outb, if it runs full, emit it with WRW.
+ * Leave flushing to caller, more data may be coming.
+ */
+
+int
+VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
+    char *obuf, ssize_t obufl, ssize_t *obufp)
+{
+	int i;
+	size_t dl;
+	const void *dp;
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	VGZ_Ibuf(vg, ibuf, ibufl);
+	VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
+	do {
+		i = VGZ_Gunzip(vg, &dp, &dl);
+		if (i < VGZ_OK) {
+			/* XXX: VSL ? */
+			return (-1);
+		}
+		*obufp += dl;
+		if (obufl == *obufp || i == VGZ_STUCK) {
+			WRW_Write(sp->wrk, obuf, *obufp);
+			if (WRW_Flush(sp->wrk))
+				return (-1);
+			*obufp = 0;
+			VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
+		}
+	} while (!VGZ_IbufEmpty(vg));
+	return (i);
+}
+
 /*--------------------------------------------------------------------*/
 
 void
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
index c97ffea..87f6f6f 100644
--- a/bin/varnishd/cache_response.c
+++ b/bin/varnishd/cache_response.c
@@ -245,15 +245,15 @@ res_WriteGunzipObj(struct sess *sp)
 	struct storage *st;
 	unsigned u = 0;
 	struct vgz *vg;
-	const void *dp;
 	char obuf[params->gzip_stack_buffer];
-	size_t dl;
+	ssize_t obufl = 0;
 	int i;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
 	vg = VGZ_NewUngzip(sp, "U D -");
 
+	VGZ_Obuf(vg, obuf, sizeof obuf);
 	VTAILQ_FOREACH(st, &sp->obj->store, list) {
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
@@ -262,17 +262,14 @@ res_WriteGunzipObj(struct sess *sp)
 		sp->acct_tmp.bodybytes += st->len;	/* XXX ? */
 		VSC_main->n_objwrite++;
 
-		VGZ_Ibuf(vg, st->ptr, st->len);
-		do {
-			VGZ_Obuf(vg, obuf, sizeof obuf);
-			i = VGZ_Gunzip(vg, &dp, &dl);
-			assert(i >= VGZ_OK);
-			if (dl != 0) {
-				(void)WRW_Write(sp->wrk, dp, dl);
-				if (WRW_Flush(sp->wrk))
-					break;
-			}
-		} while (!VGZ_IbufEmpty(vg));
+		i = VGZ_WrwGunzip(sp, vg,
+		    st->ptr, st->len,
+		    obuf, sizeof obuf, &obufl);
+		/* XXX: error check */
+	}
+	if (obufl) {
+		(void)WRW_Write(sp->wrk, obuf, obufl);
+		WRW_Flush(sp->wrk);
 	}
 	VGZ_Destroy(&vg);
 	assert(u == sp->obj->len);



From phk at varnish-cache.org  Wed Mar 30 10:01:06 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Mar 2011 12:01:06 +0200
Subject: [master] 2e3b5a5 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Q4sD4-0003ZA-A4@project.varnish-software.com>

commit 2e3b5a5d3256f2f917019bd3b8c4253c0e0a0d1f
Merge: 4fac23d e01527c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 30 09:35:38 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Wed Mar 30 11:01:44 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Mar 2011 13:01:44 +0200
Subject: [master] feafbe2 Handle the case where we already have a full buffer
	on entry.
Message-ID: <E1Q4t9k-0007My-CG@project.varnish-software.com>

commit feafbe2548f10b72be151bc4d4abf0e927d31938
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 30 11:01:29 2011 +0000

    Handle the case where we already have a full buffer on entry.

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 0222cfd..4809a3e 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -367,12 +367,16 @@ VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
 	VGZ_Ibuf(vg, ibuf, ibufl);
 	VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
 	do {
-		i = VGZ_Gunzip(vg, &dp, &dl);
+		if (obufl == *obufp)
+			i = VGZ_STUCK;
+		else {
+			i = VGZ_Gunzip(vg, &dp, &dl);
+			*obufp += dl;
+		}
 		if (i < VGZ_OK) {
 			/* XXX: VSL ? */
 			return (-1);
 		}
-		*obufp += dl;
 		if (obufl == *obufp || i == VGZ_STUCK) {
 			WRW_Write(sp->wrk, obuf, *obufp);
 			if (WRW_Flush(sp->wrk))



From phk at varnish-cache.org  Wed Mar 30 12:59:22 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Mar 2011 14:59:22 +0200
Subject: [master] c29c358 Fix a really stupid typo: s/ibufl/obufl/
Message-ID: <E1Q4uza-0007N8-VT@project.varnish-software.com>

commit c29c3582f4cd1ccf45155cef0d8814b734c3607c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 30 12:59:06 2011 +0000

    Fix a really stupid typo: s/ibufl/obufl/

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 4809a3e..fd360cf 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -364,7 +364,10 @@ VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
 	const void *dp;
 
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	assert(obufl > 16);
 	VGZ_Ibuf(vg, ibuf, ibufl);
+	if (ibufl == 0)
+		return (VGZ_OK);
 	VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
 	do {
 		if (obufl == *obufp)
@@ -382,7 +385,7 @@ VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
 			if (WRW_Flush(sp->wrk))
 				return (-1);
 			*obufp = 0;
-			VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
+			VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
 		}
 	} while (!VGZ_IbufEmpty(vg));
 	return (i);



From tfheen at varnish-cache.org  Thu Mar 31 10:26:15 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 31 Mar 2011 12:26:15 +0200
Subject: [master] 3b23f37 Fix up -o documentation for varnishlog
Message-ID: <E1Q5F4x-00084R-TE@project.varnish-software.com>

commit 3b23f3725afe80f324c089941026b99ecf937d99
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 31 12:25:16 2011 +0200

    Fix up -o documentation for varnishlog
    
    -o is needed for matching on tag/regex, the documentation did not make
     this clear.

diff --git a/doc/sphinx/reference/varnishlog.rst b/doc/sphinx/reference/varnishlog.rst
index b7fd67f..8af8030 100644
--- a/doc/sphinx/reference/varnishlog.rst
+++ b/doc/sphinx/reference/varnishlog.rst
@@ -17,8 +17,8 @@ SYNOPSIS
 ========
 
 varnishlog [-a] [-b] [-C] [-c] [-D] [-d] [-I regex] [-i tag] [-k keep] 
-[-n varnish_name] [-o] [-P file] [-r file] [-s num] [-u] [-V] 
-[-w file] [-X regex] [-x tag] [tag regex]
+[-n varnish_name] [-o [tag regex]] [-P file] [-r file] [-s num] [-u] [-V]
+[-w file] [-X regex] [-x tag]
 
 DESCRIPTION
 ===========



From tfheen at varnish-cache.org  Thu Mar 31 11:19:27 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 31 Mar 2011 13:19:27 +0200
Subject: [master] cd43465 Make varnishlog complain if regex and tag is passed
	but no -o
Message-ID: <E1Q5FuR-0000Tv-BF@project.varnish-software.com>

commit cd4346594e48acc97e9f4731f4d469bde6d16d82
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 31 13:18:57 2011 +0200

    Make varnishlog complain if regex and tag is passed but no -o

diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c
index f7075ac..0d78bf2 100644
--- a/bin/varnishlog/varnishlog.c
+++ b/bin/varnishlog/varnishlog.c
@@ -303,7 +303,7 @@ static void
 usage(void)
 {
 	fprintf(stderr, "usage: varnishlog "
-	    "%s [-aDoV] [-n varnish_name] [-P file] [-w file]\n", VSL_USAGE);
+	    "%s [-aDV] [-o [tag regex]] [-n varnish_name] [-P file] [-w file]\n", VSL_USAGE);
 	exit(1);
 }
 
@@ -361,6 +361,9 @@ main(int argc, char * const *argv)
 	if (o_flag && w_arg != NULL)
 		usage();
 
+	if ((argc - optind) > 0 && !o_flag)
+		usage();
+
 	if (VSL_Open(vd, 1))
 		exit(1);
 



From phk at varnish-cache.org  Thu Mar 31 11:24:28 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 13:24:28 +0200
Subject: [master] 66c8c9c Treat RFC-illegal negative (s-)max-age values as
	zero.
Message-ID: <E1Q5FzI-0000qs-Cl@project.varnish-software.com>

commit 66c8c9c358bc85f0601f03eb175e01da79251733
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 11:24:05 2011 +0000

    Treat RFC-illegal negative (s-)max-age values as zero.
    
    Fixes	#887

diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index 2c6c851..d581444 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -112,7 +112,10 @@ RFC2616_Ttl(const struct sess *sp)
 		    http_GetHdrField(hp, H_Cache_Control, "max-age", &p)) &&
 		    p != NULL) {
 
-			max_age = strtoul(p, NULL, 0);
+			if (*p == '-')
+				max_age = 0;
+			else
+				max_age = strtoul(p, NULL, 0);
 			if (http_GetHdr(hp, H_Age, &p)) {
 				age = strtoul(p, NULL, 0);
 				sp->wrk->age = age;
diff --git a/bin/varnishtest/tests/r00887.vtc b/bin/varnishtest/tests/r00887.vtc
new file mode 100644
index 0000000..697be6e
--- /dev/null
+++ b/bin/varnishtest/tests/r00887.vtc
@@ -0,0 +1,22 @@
+# $Id$
+
+test "Ticket #887"
+
+server s1 {
+	rxreq 
+	txresp -hdr "Cache-control: max-age=-1000" -body "FOO"
+	rxreq 
+	txresp -body "FOOBAR"
+} -start
+
+varnish v1 -vcl+backend {
+} -start 
+
+client c1 {
+	txreq  
+	rxresp
+	expect resp.bodylen == 3
+	txreq  
+	rxresp
+	expect resp.bodylen == 6
+} -run



From phk at varnish-cache.org  Thu Mar 31 12:14:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 14:14:24 +0200
Subject: [master] 4c3a1f2 Duh!  Same s/i/o/ bug was copy & pasted.
Message-ID: <E1Q5Glc-0001RB-FE@project.varnish-software.com>

commit 4c3a1f2506c41a228a61b20c8043a34580c793df
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 12:14:10 2011 +0000

    Duh!  Same s/i/o/ bug was copy & pasted.

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index fd360cf..e398583 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -368,7 +368,7 @@ VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
 	VGZ_Ibuf(vg, ibuf, ibufl);
 	if (ibufl == 0)
 		return (VGZ_OK);
-	VGZ_Obuf(vg, obuf + *obufp, ibufl - *obufp);
+	VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
 	do {
 		if (obufl == *obufp)
 			i = VGZ_STUCK;



From tfheen at varnish-cache.org  Thu Mar 31 12:31:40 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 31 Mar 2011 14:31:40 +0200
Subject: [master] e89cffc Add -p $period to varnishtop
Message-ID: <E1Q5H2K-0003tj-VY@project.varnish-software.com>

commit e89cffc006159d2dd9a42cb891225b32b03d153e
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Mar 31 14:30:15 2011 +0200

    Add -p $period to varnishtop
    
    Make it possible to specify the period varnishtop should integrate
    numbers over.  This makes it easier to figure out per time period
    stats.

diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index d01c290..3b834a0 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -147,12 +147,13 @@ accumulate(uint32_t * const p)
 }
 
 static void
-update(const struct VSM_data *vd)
+update(const struct VSM_data *vd, int period)
 {
 	struct top *tp, *tp2;
 	int l, len;
 	double t = 0;
 	static time_t last = 0;
+	static unsigned n;
 	time_t now;
 
 	now = time(NULL);
@@ -161,6 +162,8 @@ update(const struct VSM_data *vd)
 	last = now;
 
 	l = 1;
+	if (n < period)
+		n++;
 	AC(erase());
 	AC(mvprintw(0, 0, "%*s", COLS - 1, VSM_Name(vd)));
 	AC(mvprintw(0, 0, "list length %u", ntop));
@@ -175,7 +178,7 @@ update(const struct VSM_data *vd)
 			    len, len, tp->rec_data));
 			t = tp->count;
 		}
-		tp->count *= .999;
+		tp->count += (1.0/3.0 - tp->count) / (double)n;
 		if (tp->count * 10 < t || l > LINES * 10) {
 			VTAILQ_REMOVE(&top_head, tp, list);
 			free(tp->rec_data);
@@ -211,7 +214,7 @@ accumulate_thread(void *arg)
 }
 
 static void
-do_curses(struct VSM_data *vd)
+do_curses(struct VSM_data *vd, int period)
 {
 	pthread_t thr;
 	int i;
@@ -237,7 +240,7 @@ do_curses(struct VSM_data *vd)
 	AC(erase());
 	for (;;) {
 		AZ(pthread_mutex_lock(&mtx));
-		update(vd);
+		update(vd, period);
 		AZ(pthread_mutex_unlock(&mtx));
 
 		timeout(1000);
@@ -310,11 +313,12 @@ main(int argc, char **argv)
 {
 	struct VSM_data *vd;
 	int o, once = 0;
+	float period = 60; /* seconds */
 
 	vd = VSM_New();
 	VSL_Setup(vd);
 
-	while ((o = getopt(argc, argv, VSL_ARGS "1fV")) != -1) {
+	while ((o = getopt(argc, argv, VSL_ARGS "1fVp:")) != -1) {
 		switch (o) {
 		case '1':
 			AN(VSL_Arg(vd, 'd', NULL));
@@ -323,6 +327,14 @@ main(int argc, char **argv)
 		case 'f':
 			f_flag = 1;
 			break;
+		case 'p':
+			errno = 0;
+			period = strtol(optarg, NULL, 0);
+			if (errno != 0)  {
+				fprintf(stderr, "Syntax error, %s is not a number", optarg);
+				exit(1);
+			}
+			break;
 		case 'V':
 			varnish_version("varnishtop");
 			exit(0);
@@ -340,7 +352,7 @@ main(int argc, char **argv)
 		VSL_NonBlocking(vd, 1);
 		do_once(vd);
 	} else {
-		do_curses(vd);
+		do_curses(vd, period);
 	}
 	exit(0);
 }
diff --git a/doc/sphinx/reference/varnishtop.rst b/doc/sphinx/reference/varnishtop.rst
index 4c4899a..0fe65fc 100644
--- a/doc/sphinx/reference/varnishtop.rst
+++ b/doc/sphinx/reference/varnishtop.rst
@@ -60,6 +60,10 @@ The following options are available:
 -i tag      Include log entries with the specified tag.  If neither ``-I``
    	    nor ``-i`` is specified, all log entries are included.
 
+-p period   Specifies the number of seconds to measure over, the default
+            is 60 seconds. The first number in the list is the average
+            number of requests seen over this time period.
+
 -n          Specifies the name of the varnishd instance to get logs from.  
 	    If ``-n`` is not specified, the host name is used.
 



From phk at varnish-cache.org  Thu Mar 31 16:58:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 18:58:31 +0200
Subject: [master] 40e09c2 Don't return VGZ_STUCK if we have emptied the input.
Message-ID: <E1Q5LCZ-00050D-0R@project.varnish-software.com>

commit 40e09c27b8c58f2ca879a6f6743ae04e4caca8eb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 14:42:28 2011 +0000

    Don't return VGZ_STUCK if we have emptied the input.

diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index e398583..b4d78dd 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -388,6 +388,8 @@ VGZ_WrwGunzip(struct sess *sp, struct vgz *vg, void *ibuf, ssize_t ibufl,
 			VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
 		}
 	} while (!VGZ_IbufEmpty(vg));
+	if (i == VGZ_STUCK)
+		i = VGZ_OK;
 	return (i);
 }
 



From phk at varnish-cache.org  Thu Mar 31 16:58:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 18:58:31 +0200
Subject: [master] 8aa9494 Don't assume esi-spec lengths match up with storage
	segments.
Message-ID: <E1Q5LCZ-00050L-Bl@project.varnish-software.com>

commit 8aa949400268d83069173bac1a155f175e66aacf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 16:58:05 2011 +0000

    Don't assume esi-spec lengths match up with storage segments.

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index b074cd5..71bbb4b 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -225,7 +225,7 @@ ESI_Deliver(struct sess *sp)
 	struct storage *st;
 	uint8_t *p, *e, *q, *r;
 	unsigned off;
-	ssize_t l, l_icrc = 0;
+	ssize_t l, l2, l_icrc = 0;
 	uint32_t icrc = 0;
 	uint8_t tailbuf[8 + 5];
 	int isgzip;
@@ -291,7 +291,6 @@ ESI_Deliver(struct sess *sp)
 		case VEC_V2:
 		case VEC_V8:
 			l = ved_decode_len(&p);
-			r = p;
 			if (isgzip) {
 				assert(*p == VEC_C1 || *p == VEC_C2 ||
 				    *p == VEC_C8);
@@ -299,45 +298,76 @@ ESI_Deliver(struct sess *sp)
 				icrc = vbe32dec(p);
 				p += 4;
 			}
-			if (sp->wrk->gzip_resp && isgzip) {
-				/*
-				 * We have a gzip'ed VEC and delivers a
-				 * gzip'ed ESI response.
-				 */
-				sp->wrk->crc = crc32_combine(
-				    sp->wrk->crc, icrc, l_icrc);
-				sp->wrk->l_crc += l_icrc;
-				WRW_Write(sp->wrk, st->ptr + off, l);
-			} else if (sp->wrk->gzip_resp) {
-				/*
-				 * A gzip'ed ESI response, but the VEC was
-				 * not gzip'ed.
-				 */
-				ved_pretend_gzip(sp, st->ptr + off, l);
-				off += l;
-			} else if (isgzip) {
-				/*
-				 * A gzip'ed VEC, but ungzip'ed ESI response
-				 */
-				AN(vgz);
-				i = VGZ_WrwGunzip(sp, vgz,
-					st->ptr + off, l,
-					obuf, sizeof obuf, &obufl);
-				assert (i == VGZ_OK || i == VGZ_END);
-			} else {
-				/*
-				 * Ungzip'ed VEC, ungzip'ed ESI response
-				 */
-				WRW_Write(sp->wrk, st->ptr + off, l);
+			/*
+			 * There is no guarantee that the 'l' bytes are all
+			 * in the same storage segment, so loop over storage
+			 * until we have processed them all.
+			 */
+			while (l > 0) {
+				l2 = l;
+				if (l2 > st->len - off)
+					l2 = st->len - off;
+				l -= l2;
+
+				if (sp->wrk->gzip_resp && isgzip) {
+					/*
+					 * We have a gzip'ed VEC and delivers
+					 * a gzip'ed ESI response.
+					 */
+					sp->wrk->crc = crc32_combine(
+					    sp->wrk->crc, icrc, l_icrc);
+					sp->wrk->l_crc += l_icrc;
+					WRW_Write(sp->wrk, st->ptr + off, l2);
+				} else if (sp->wrk->gzip_resp) {
+					/*
+					 * A gzip'ed ESI response, but the VEC
+					 * was not gzip'ed.
+					 */
+					ved_pretend_gzip(sp, st->ptr + off, l2);
+				} else if (isgzip) {
+					/*
+					 * A gzip'ed VEC, but ungzip'ed ESI
+					 * response
+					 */
+					AN(vgz);
+					i = VGZ_WrwGunzip(sp, vgz,
+						st->ptr + off, l2,
+						obuf, sizeof obuf, &obufl);
+					assert (i == VGZ_OK || i == VGZ_END);
+				} else {
+					/*
+					 * Ungzip'ed VEC, ungzip'ed ESI response
+					 */
+					WRW_Write(sp->wrk, st->ptr + off, l2);
+				}
+				off += l2;
+				if (off == st->len) {
+					st = VTAILQ_NEXT(st, list);
+					off = 0;
+				}
 			}
-			off += l;
 			break;
 		case VEC_S1:
 		case VEC_S2:
 		case VEC_S8:
 			l = ved_decode_len(&p);
 			Debug("SKIP1(%d)\n", (int)l);
-			off += l;
+			/*
+			 * There is no guarantee that the 'l' bytes are all
+			 * in the same storage segment, so loop over storage
+			 * until we have processed them all.
+			 */
+			while (l > 0) {
+				l2 = l;
+				if (l2 > st->len - off)
+					l2 = st->len - off;
+				l -= l2;
+				off += l2;
+				if (off == st->len) {
+					st = VTAILQ_NEXT(st, list);
+					off = 0;
+				}
+			}
 			break;
 		case VEC_INCL:
 			p++;



From phk at varnish-cache.org  Thu Mar 31 16:58:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 18:58:31 +0200
Subject: [master] 3c8e520 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1Q5LCZ-00050V-NL@project.varnish-software.com>

commit 3c8e5203ebd7ad232c80406511d4288a41a6fc87
Merge: 8aa9494 e89cffc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 16:58:28 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Thu Mar 31 17:44:33 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 19:44:33 +0200
Subject: [master] c7e3738 Only do CRC calculation once.
Message-ID: <E1Q5Lv7-0008Aw-Ay@project.varnish-software.com>

commit c7e373808971c3814d159df89b8a42b06f6a235a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 17:44:22 2011 +0000

    Only do CRC calculation once.

diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 71bbb4b..b666c79 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -297,6 +297,11 @@ ESI_Deliver(struct sess *sp)
 				l_icrc = ved_decode_len(&p);
 				icrc = vbe32dec(p);
 				p += 4;
+				if (sp->wrk->gzip_resp) {
+					sp->wrk->crc = crc32_combine(
+					    sp->wrk->crc, icrc, l_icrc);
+					sp->wrk->l_crc += l_icrc;
+				}
 			}
 			/*
 			 * There is no guarantee that the 'l' bytes are all
@@ -314,9 +319,6 @@ ESI_Deliver(struct sess *sp)
 					 * We have a gzip'ed VEC and delivers
 					 * a gzip'ed ESI response.
 					 */
-					sp->wrk->crc = crc32_combine(
-					    sp->wrk->crc, icrc, l_icrc);
-					sp->wrk->l_crc += l_icrc;
 					WRW_Write(sp->wrk, st->ptr + off, l2);
 				} else if (sp->wrk->gzip_resp) {
 					/*



From phk at varnish-cache.org  Thu Mar 31 19:14:16 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 21:14:16 +0200
Subject: [master] 19ff8f1 Fix the VBE/VCL reference count anomaly
Message-ID: <E1Q5NJw-0000qc-Kj@project.varnish-software.com>

commit 19ff8f182b582fbea85cfef40142b5b45865276e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 19:13:59 2011 +0000

    Fix the VBE/VCL reference count anomaly

diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index a1886e1..93f4a77 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -479,7 +479,7 @@ vdi_simple_fini(const struct director *d)
 	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
 
 	VBP_Stop(vs->backend, vs->vrt->probe);
-	VBE_DropRef(vs->backend);
+	VBE_DropRefVcl(vs->backend);
 	free(vs->dir.vcl_name);
 	vs->dir.magic = 0;
 	FREE_OBJ(vs);
diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
index e620068..a437256 100644
--- a/bin/varnishd/cache_backend.h
+++ b/bin/varnishd/cache_backend.h
@@ -143,7 +143,7 @@ struct backend *vdi_get_backend_if_simple(const struct director *d);
 /* cache_backend_cfg.c */
 extern struct lock VBE_mtx;
 void VBE_DropRefConn(struct backend *);
-void VBE_DropRef(struct backend *);
+void VBE_DropRefVcl(struct backend *);
 void VBE_DropRefLocked(struct backend *b);
 
 /* cache_backend_poll.c */
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
index 17aed49..15dcc80 100644
--- a/bin/varnishd/cache_backend_cfg.c
+++ b/bin/varnishd/cache_backend_cfg.c
@@ -106,7 +106,6 @@ VBE_DropRefLocked(struct backend *b)
 	assert(b->refcount > 0);
 
 	i = --b->refcount;
-	b->vsc->vcls--;
 	Lck_Unlock(&b->mtx);
 	if (i > 0)
 		return;
@@ -125,12 +124,13 @@ VBE_DropRefLocked(struct backend *b)
 }
 
 void
-VBE_DropRef(struct backend *b)
+VBE_DropRefVcl(struct backend *b)
 {
 
 	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
 
 	Lck_Lock(&b->mtx);
+	b->vsc->vcls--;
 	VBE_DropRefLocked(b);
 }
 



From phk at varnish-cache.org  Thu Mar 31 20:02:37 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 31 Mar 2011 22:02:37 +0200
Subject: [master] ad3f134 Add some asserts to help catch #886
Message-ID: <E1Q5O4j-000469-H3@project.varnish-software.com>

commit ad3f1341c4960ecf12499c0ca0debb5bfd962e26
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 31 20:02:24 2011 +0000

    Add some asserts to help catch #886

diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 4480c98..f156018 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -434,6 +434,7 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
 
 	b0 = ban_start;
+	CHECK_OBJ_NOTNULL(b0, BAN_MAGIC);
 
 	if (b0 == oc->ban)
 		return (0);
@@ -445,6 +446,7 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 	 */
 	tests = 0;
 	for (b = b0; b != oc->ban; b = VTAILQ_NEXT(b, list)) {
+		CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
 		if (b->flags & BAN_F_GONE)
 			continue;
 		if (!has_req && (b->flags & BAN_F_REQ))