From kenshaw at gmail.com  Mon Apr  1 11:21:14 2013
From: kenshaw at gmail.com (Kenneth Shaw)
Date: Mon, 1 Apr 2013 18:21:14 +0700
Subject: libvmod-dns (super alpha)
Message-ID: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>

Hi,

I spent a bit of time today developing a DNS module for Varnish.

It is available here:

https://github.com/kenshaw/libvmod-dns/

The reason for this development is to cut off bots that abuse the
User-Agent string (ie, claiming to be Googlebot/bingbot/etc.) by doing a
reverse and then forward DNS against the client.ip/X-Forwarded-For header
and comparing with a regex against the resultant domain.

The logic is meant to work something like this:

sub vcl_recv {
    # do a dns check on "good" crawlers
    if (req.http.user-agent ~ "(?i)(googlebot|bingbot|slurp|teoma)") {
        # do a reverse lookup on the client.ip (X-Forwarded-For) and check
that its in the allowed domains
        set req.http.X-Crawler-DNS-Reverse =
dns.rresolve(req.http.X-Forwarded-For);

        # check that the RDNS points to an allowed domain -- 403 error if
it doesn't
        if (req.http.X-Crawler-DNS-Reverse !~
"(?i)\.(googlebot\.com|search\.msn\.com|crawl\.yahoo\.net|ask\.com)$") {
            error 403 "Forbidden";
        }

        # do a forward lookup on the DNS
        set req.http.X-Crawler-DNS-Forward =
dns.resolve(req.http.X-Crawler-DNS-Reverse);

        # if the client.ip/X-Forwarded-For doesn't match, then the
user-agent is fake
        if (req.http.X-Crawler-DNS-Forward != req.http.X-Forwarded-For) {
            error 403 "Forbidden";
        }
    }
}

While this is not being used in production (yet), I plan to do so later
this week against a production system receiving ~10,000+ requests/sec. I
will report back afterwards.

I realize the code currently has issues (memory, documentation, etc.),
which will be fixed in the near future.

I also realize there are better ways to head malicious bots off at the pass
through DNS, etc (which we are doing as well). The largest issue here for
my purposes is that it is difficult / impossible to identify all traffic.
Additionally, it is nice to be able to monitor the actual traffic coming
through and not completely dropping it at the edge.

Any input/comments against what I've written so far would be gladly
appreciated! Thanks!

-Ken
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130401/18279237/attachment.html>

From phk at phk.freebsd.dk  Tue Apr  2 09:36:01 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 02 Apr 2013 09:36:01 +0000
Subject: libvmod-dns (super alpha)
In-Reply-To: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
References: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
Message-ID: <41051.1364895361@critter.freebsd.dk>

In message <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw at mail.gmail.com>
, Kenneth Shaw writes:

>I spent a bit of time today developing a DNS module for Varnish.

I think that is a really good idea, and once we have it figured out
and are happy with it, it should probably become a standard VMOD
in the varnish releases.

I don't have much concrete feedback though, I hope others can weigh
in with that...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From gonzalo.paniagua at acquia.com  Tue Apr  2 13:49:21 2013
From: gonzalo.paniagua at acquia.com (Gonzalo Paniagua)
Date: Tue, 2 Apr 2013 09:49:21 -0400
Subject: libvmod-dns (super alpha)
In-Reply-To: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
References: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
Message-ID: <CAOvFWRXEazu96D-UOCX_4t9nn5WEKaL=2Q3VbDsRLUBYEKa4Dg@mail.gmail.com>

On Mon, Apr 1, 2013 at 7:21 AM, Kenneth Shaw <kenshaw at gmail.com> wrote:
[...]
>
> Any input/comments against what I've written so far would be gladly
> appreciated! Thanks!

Great work.

My only comment is that AFAIR, using getaddrinfo() from multiple
threads makes all of them go through a single lock to access a socket.
You might want to consider an alternative resolver library.

-Gonzalo



From ruben at varnish-software.com  Sat Apr  6 00:27:37 2013
From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=)
Date: Sat, 6 Apr 2013 02:27:37 +0200
Subject: Welcome to VUG7 in New York City, USA - May 30th & 31st 2013
Message-ID: <CAEfnkDYvbGRQ_w83tfhT3GUpgPRHk7SF300uz3U48vkUJ0aypA@mail.gmail.com>

Howdy,

Once again it is my pleasure to announce, in behalf of the Varnish Team,
our next Varnish User Group meeting. Under the Seventh VUG (Second time we
hold it in the USA we will be joining the vibrant Varnish community in the
Big Apple at the very end of May. The meeting is kindly sponsored by Vimeo
and 10gen.


*** Registration is open, so get your ticket before it is too late: <
http://vug7.eventbrite.co.uk> ***


* The Varnish User Day will be on Friday 31st of May at Vimeo. For agenda
and general information [1]

* The Varnish Developer meeting will on Thursday 30th of May at 10gen. The
exact location of the event will only be communicated to the hardcore C
developers working with Varnish Core and VMOD hacking that are allowed to
attend ;-) See and use the wiki for coordinating details [2]

* For those interested in training, there is a session on June 3rd & 4th in
Manhattan. More information, pricing and registration [3]

A cool thing we will have during the User Day at this VUG is the *VCL
speed-dating session* where everyone can share their best tricks, propose
puzzles for other to solve and/or even changes for VCL in 4.0. Hopefully,
everyone can take something to try at home under this session. Please start
thinking about what you are going to show ;-)

As usual, presentation are welcome, and they can be added to the agenda by
replying to this email or directly in the wiki planning page [4].
Otherwise, I will eventually reach to some you and ask you if can hold a
presentation :-)

Feel free to tell your colleagues and friends about the event!

Links:
[1] <http://www.varnish.org/vug7>
[2] <https://www.varnish-cache.org/trac/wiki/VDD13Q2>
[3] <https://www.varnish-software.com/training/varnish-administration-course
>
[4] <https://www.varnish-cache.org/trac/wiki/VUG7>


All the best,
-- 
*Rub?n Romero*
Global Sales Executive & Community Hygiene Keeper  | Varnish Software AS
Cell: +47 95964088 / Office: +47 21989260
Skype & Twitter: ruben_varnish
We Make Websites Fly!



PD: For everyone in the Bay Area and the West Coast: There is hope! Varnish
Software will visit your neighborhood as we will be at Velocity 2013 (yes,
we have a booth). We expect to have an event there the day after Velocity
(soon to be announce). Stay tuned!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130406/f681b42c/attachment.html>

From phk at phk.freebsd.dk  Sun Apr  7 20:20:45 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 07 Apr 2013 20:20:45 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <5149B5F4.2050300@uplex.de>
References: <8648.1363781143@critter.freebsd.dk>
 <CABUevEw2P64dcayQ2_rH_ZR-_sX7YcG95qb7q2wQJ4o_hNHyCg@mail.gmail.com>
 <61572.1363784190@critter.freebsd.dk>
 <CABtDKm4wyChQ5BMptPiJmNpC9RrgB0i5uFVcQv9nM1txhE=MFA@mail.gmail.com>
 <5149B5F4.2050300@uplex.de>
Message-ID: <5867.1365366045@critter.freebsd.dk>


I have tried to distill the new vcl_lookup{} into one easy to understand
ASCII-graph (if you can't document it etc...)

   vcl_hash{}
       |
       |
       v
   ALWYAS_MISS ? --- N ----+
       |                   |
       |                   v
       |                lookup()
       |                   |
       v                   v
    "MISS" <-----------+---+------------+---------------+--------------+
       |               |                |               |              |
       |               v                v               v              v
       |           "BUSY ONLY"    "EXPIRED+BUSY"    "EXPIRED"        "HIT"
       |               |                |               |              |
       v               v                v               v              |
       +<--- Y ----IGNORE_BUSY ?   IGNORE_BUSY? -- Y -->+              |
       |               |                |               |              |
       v               N                N               v              | 
    insert             |                |            insert            | 
    our busyobj        v                |           our busyobj        |
       |          [WAITING LIST]        |               |              |
       |                                |               |              |
       |                                v               v              v
       +--------->  ret(bo)        ret(exp_obj)   ret(exp_obj+bo)  ret(obj)
		      |                 |               |              |
		      |                 |               |              |
		      |                 |               |              |
   vcl_lookup{}       |                 |               |              |
   returns            |                 |               |              |
     |                |                 |               |              |
     v                v                 v               v              v
   pass
		   unbusy                            unbusy        
		   fetch(pass)      fetch(pass)      fetch(pass)   fetch(pass)
		   deliver          deliver          deliver       deliver

   synth
		   unbusy                            unbusy        
		   synth            synth            synth         synth

   deliver         fetch            stream           fetch         deliver obj
		   deliver          busyobj          deliver


   deliver_stale   fetch            deliver          bg-fetch      deliver obj
		   deliver             exp_obj       deliver
							exp_obj

There is a footnote that I simply could not get in there: hit_for_pass
objects.

If vcl_lookup{} returns deliver or deliver_stale on one such, it will
be converted to "pass" and a SLT_VCL_Error will be logged.

That leaves one interesting corner case:  A hit-for-pass object has
expired but is still in cache and we come down the "EXPIRED" path
where we insert our own busyobj, expecting to refresh the cache
with our own fetch.

Ideally, we should keep that busyobject in there, do the fetch(pass)
and if the object still qualifies for hit-for-pass, mark the
busyobj as such, so that there is never a gap where we take a MISS
on a hit-for-pass.

If that's trivial, I'll do it, of not, I wont.

Comments etc...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From sky at crucially.net  Sun Apr  7 20:28:58 2013
From: sky at crucially.net (sky at crucially.net)
Date: Sun, 7 Apr 2013 20:28:58 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <5867.1365366045@critter.freebsd.dk>
References: <8648.1363781143@critter.freebsd.dk>
 <CABUevEw2P64dcayQ2_rH_ZR-_sX7YcG95qb7q2wQJ4o_hNHyCg@mail.gmail.com>
 <61572.1363784190@critter.freebsd.dk>
 <CABtDKm4wyChQ5BMptPiJmNpC9RrgB0i5uFVcQv9nM1txhE=MFA@mail.gmail.com>
 <5149B5F4.2050300@uplex.de> <5867.1365366045@critter.freebsd.dk>
Message-ID: <377519748-1365366540-cardhu_decombobulator_blackberry.rim.net-1478674699-@b15.c3.bise6.blackberry>

If you get a 304 from origin on a IMS to origin? Where would you end up. 

And why would you want to return stale from lookup? (I feel like I am missing part of the puzzle)

And is exp-obj the stale object?

Artur 

Sent via BlackBerry by AT&T

-----Original Message-----
From: "Poul-Henning Kamp" <phk at phk.freebsd.dk>
Sender: varnish-dev-bounces at varnish-cache.org
Date: Sun, 07 Apr 2013 20:20:45 
To: <varnish-dev at varnish-cache.org>
Subject: Re: V4 VCL roadmappery...


I have tried to distill the new vcl_lookup{} into one easy to understand
ASCII-graph (if you can't document it etc...)

   vcl_hash{}
       |
       |
       v
   ALWYAS_MISS ? --- N ----+
       |                   |
       |                   v
       |                lookup()
       |                   |
       v                   v
    "MISS" <-----------+---+------------+---------------+--------------+
       |               |                |               |              |
       |               v                v               v              v
       |           "BUSY ONLY"    "EXPIRED+BUSY"    "EXPIRED"        "HIT"
       |               |                |               |              |
       v               v                v               v              |
       +<--- Y ----IGNORE_BUSY ?   IGNORE_BUSY? -- Y -->+              |
       |               |                |               |              |
       v               N                N               v              | 
    insert             |                |            insert            | 
    our busyobj        v                |           our busyobj        |
       |          [WAITING LIST]        |               |              |
       |                                |               |              |
       |                                v               v              v
       +--------->  ret(bo)        ret(exp_obj)   ret(exp_obj+bo)  ret(obj)
		      |                 |               |              |
		      |                 |               |              |
		      |                 |               |              |
   vcl_lookup{}       |                 |               |              |
   returns            |                 |               |              |
     |                |                 |               |              |
     v                v                 v               v              v
   pass
		   unbusy                            unbusy        
		   fetch(pass)      fetch(pass)      fetch(pass)   fetch(pass)
		   deliver          deliver          deliver       deliver

   synth
		   unbusy                            unbusy        
		   synth            synth            synth         synth

   deliver         fetch            stream           fetch         deliver obj
		   deliver          busyobj          deliver


   deliver_stale   fetch            deliver          bg-fetch      deliver obj
		   deliver             exp_obj       deliver
							exp_obj

There is a footnote that I simply could not get in there: hit_for_pass
objects.

If vcl_lookup{} returns deliver or deliver_stale on one such, it will
be converted to "pass" and a SLT_VCL_Error will be logged.

That leaves one interesting corner case:  A hit-for-pass object has
expired but is still in cache and we come down the "EXPIRED" path
where we insert our own busyobj, expecting to refresh the cache
with our own fetch.

Ideally, we should keep that busyobject in there, do the fetch(pass)
and if the object still qualifies for hit-for-pass, mark the
busyobj as such, so that there is never a gap where we take a MISS
on a hit-for-pass.

If that's trivial, I'll do it, of not, I wont.

Comments etc...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

From phk at phk.freebsd.dk  Sun Apr  7 20:43:12 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 07 Apr 2013 20:43:12 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <377519748-1365366540-cardhu_decombobulator_blackberry.rim.net-1478674699-@b15.c3.bise6.blackberry>
References: <8648.1363781143@critter.freebsd.dk>
 <CABUevEw2P64dcayQ2_rH_ZR-_sX7YcG95qb7q2wQJ4o_hNHyCg@mail.gmail.com>
 <61572.1363784190@critter.freebsd.dk>
 <CABtDKm4wyChQ5BMptPiJmNpC9RrgB0i5uFVcQv9nM1txhE=MFA@mail.gmail.com>
 <5149B5F4.2050300@uplex.de> <5867.1365366045@critter.freebsd.dk>
 <377519748-1365366540-cardhu_decombobulator_blackberry.rim.net-1478674699-@b15.c3.bise6.blackberry>
Message-ID: <5968.1365367392@critter.freebsd.dk>


> If you get a 304 from origin on a IMS to origin? Where would you end up. 

The bits on this diagram is all on the "client-side" of things.

The places where you see "fetch" we will (mostly) fork another thread
to do the actual fetching. (exception: pass ?)

If we have an expired obj ("exp_obj"), we'll pass that to the thread,
that will enable it to do conditional fetch (will do by default) and
if we get 304 back well stitch the exp_obj's body into the new object.

> And why would you want to return stale from lookup? (I feel like I am
> missing part of the puzzle)

That's basically what "grace" does today, only now we expose that
decision in VCL.

The default.vcl will look something like:

	sub vcl_lookup {
	    if (obj.uncacheable) {
		return (pass);
	    }
	    if (obj.ttl >= 0s) {
		return (deliver);
	    }
	    if (obj.ttl + obj.grace > 0s) {
		return (stale);
	    }
	    return (fetch);
	}

(input very welcome!)

> And is exp-obj the stale object?

Yes.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From sky at crucially.net  Sun Apr  7 20:46:29 2013
From: sky at crucially.net (sky at crucially.net)
Date: Sun, 7 Apr 2013 20:46:29 +0000
Subject: V4 VCL roadmappery...
Message-ID: <864851000-1365367590-cardhu_decombobulator_blackberry.rim.net-1614404448-@b15.c3.bise6.blackberry>

But wouldn't you want to know what the backend thinks before you return stale?

IE, would return stale make more sense in fetch or error?
------Original Message------
From: Poul-Henning Kamp
To: Artur Bergman
Cc: varnish-dev at varnish-cache.org
Subject: Re: V4 VCL roadmappery...
Sent: Apr 7, 2013 13:43


> If you get a 304 from origin on a IMS to origin? Where would you end up. 

The bits on this diagram is all on the "client-side" of things.

The places where you see "fetch" we will (mostly) fork another thread
to do the actual fetching. (exception: pass ?)

If we have an expired obj ("exp_obj"), we'll pass that to the thread,
that will enable it to do conditional fetch (will do by default) and
if we get 304 back well stitch the exp_obj's body into the new object.

> And why would you want to return stale from lookup? (I feel like I am
> missing part of the puzzle)

That's basically what "grace" does today, only now we expose that
decision in VCL.

The default.vcl will look something like:

	sub vcl_lookup {
	    if (obj.uncacheable) {
		return (pass);
	    }
	    if (obj.ttl >= 0s) {
		return (deliver);
	    }
	    if (obj.ttl + obj.grace > 0s) {
		return (stale);
	    }
	    return (fetch);
	}

(input very welcome!)

> And is exp-obj the stale object?

Yes.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

Sent via BlackBerry by AT&T

From phk at phk.freebsd.dk  Sun Apr  7 21:05:28 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 07 Apr 2013 21:05:28 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <864851000-1365367590-cardhu_decombobulator_blackberry.rim.net-1614404448-@b15.c3.bise6.blackberry>
References: <864851000-1365367590-cardhu_decombobulator_blackberry.rim.net-1614404448-@b15.c3.bise6.blackberry>
Message-ID: <6137.1365368728@critter.freebsd.dk>


> But wouldn't you want to know what the backend thinks before you return stale?
> IE, would return stale make more sense in fetch or error?

Mostly returning stale will be for the same reason we have grace:
get a backend fetch/refresh going, without piling up clients on
the waiting list.

You have two places where you can do something of the sort you ask for.

In vcl_backend_response{} you decide if you want to use the
backends response or not.  (I'm pondring if "restart" should have
its own closed circuit on the backend side, so that restart fom
vcl_backend_response{} sends you back to vcl_backend_fetch{}. Input ?)

In vcl_deliver{} you decide if you want to deliver whatever
object you have at that time to the client.  Your alternatives is
to go ahead with it, restart or synthetic.


(One of the overall goals of decoupling client/backend more, is to
make things like parallel ESI possible)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From sky at crucially.net  Mon Apr  8 06:20:09 2013
From: sky at crucially.net (Artur Bergman)
Date: Sun, 7 Apr 2013 23:20:09 -0700
Subject: V4 VCL roadmappery...
In-Reply-To: <6137.1365368728@critter.freebsd.dk>
References: <864851000-1365367590-cardhu_decombobulator_blackberry.rim.net-1614404448-@b15.c3.bise6.blackberry>
 <6137.1365368728@critter.freebsd.dk>
Message-ID: <02E01EF8-1131-4357-B2CD-594434F0E9E5@crucially.net>

Ok,

If I end up in error and want to return the stale object, how would I do that?

Artur

On Apr 7, 2013, at 2:05 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

> 
>> But wouldn't you want to know what the backend thinks before you return stale?
>> IE, would return stale make more sense in fetch or error?
> 
> Mostly returning stale will be for the same reason we have grace:
> get a backend fetch/refresh going, without piling up clients on
> the waiting list.
> 
> You have two places where you can do something of the sort you ask for.
> 
> In vcl_backend_response{} you decide if you want to use the
> backends response or not.  (I'm pondring if "restart" should have
> its own closed circuit on the backend side, so that restart fom
> vcl_backend_response{} sends you back to vcl_backend_fetch{}. Input ?)
> 
> In vcl_deliver{} you decide if you want to deliver whatever
> object you have at that time to the client.  Your alternatives is
> to go ahead with it, restart or synthetic.
> 
> 
> (One of the overall goals of decoupling client/backend more, is to
> make things like parallel ESI possible)
> 
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe    
> Never attribute to malice what can adequately be explained by incompetence.




From phk at phk.freebsd.dk  Mon Apr  8 07:24:50 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 08 Apr 2013 07:24:50 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <02E01EF8-1131-4357-B2CD-594434F0E9E5@crucially.net>
References: <864851000-1365367590-cardhu_decombobulator_blackberry.rim.net-1614404448-@b15.c3.bise6.blackberry>
 <6137.1365368728@critter.freebsd.dk>
 <02E01EF8-1131-4357-B2CD-594434F0E9E5@crucially.net>
Message-ID: <1668.1365405890@critter.freebsd.dk>

In message <02E01EF8-1131-4357-B2CD-594434F0E9E5 at crucially.net>, Artur Bergman 
writes:

>If I end up in error and want to return the stale object, how would I do =
>that?

Do a restart.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From sky at crucially.net  Mon Apr  8 07:38:06 2013
From: sky at crucially.net (sky at crucially.net)
Date: Mon, 8 Apr 2013 07:38:06 +0000
Subject: V4 VCL roadmappery...
Message-ID: <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307430474-@b15.c3.bise6.blackberry>

Why not people do return stale from inside there?

Restart has problems because it forces people to guard against recv running multiple times?

------Original Message------
From: Poul-Henning Kamp
To: Artur Bergman
Cc: varnish-dev at varnish-cache.org
Subject: Re: V4 VCL roadmappery...
Sent: Apr 8, 2013 00:24

In message <02E01EF8-1131-4357-B2CD-594434F0E9E5 at crucially.net>, Artur Bergman 
writes:

>If I end up in error and want to return the stale object, how would I do =
>that?

Do a restart.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

Sent via BlackBerry by AT&T



From phk at phk.freebsd.dk  Mon Apr  8 07:41:45 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 08 Apr 2013 07:41:45 +0000
Subject: V4 VCL roadmappery...
In-Reply-To: <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307430474-@b15.c3.bise6.blackberry>
References: <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307430474-@b15.c3.bise6.blackberry>
Message-ID: <12984.1365406905@critter.freebsd.dk>

In message <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307
430474- at b15.c3.bise6.blackberry>, sky at crucially.net writes:

>Why not people do return stale from inside there?

You mean return stale from vcl_deliver{} ?

Then you'd have to rerun vcl_deliver, because resp.* is based on
the wrong object...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From sky at crucially.net  Mon Apr  8 07:43:42 2013
From: sky at crucially.net (Artur Bergman)
Date: Mon, 8 Apr 2013 00:43:42 -0700
Subject: V4 VCL roadmappery...
In-Reply-To: <12984.1365406905@critter.freebsd.dk>
References: <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307430474-@b15.c3.bise6.blackberry>
 <12984.1365406905@critter.freebsd.dk>
Message-ID: <5A89B275-1B90-4171-999C-BD1021A555A1@crucially.net>

No, in vcl_error, or in the replacement for vcl_fetch (assuming of course it is not a background fetch).

Artur

On Apr 8, 2013, at 12:41 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

> In message <1504564633-1365406685-cardhu_decombobulator_blackberry.rim.net-1307
> 430474- at b15.c3.bise6.blackberry>, sky at crucially.net writes:
> 
>> Why not people do return stale from inside there?
> 
> You mean return stale from vcl_deliver{} ?
> 
> Then you'd have to rerun vcl_deliver, because resp.* is based on
> the wrong object...
> 
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe    
> Never attribute to malice what can adequately be explained by incompetence.




From kenshaw at gmail.com  Mon Apr  8 14:40:52 2013
From: kenshaw at gmail.com (Kenneth Shaw)
Date: Mon, 8 Apr 2013 21:40:52 +0700
Subject: libvmod-dns (super alpha)
In-Reply-To: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
References: <CAAyX=LFoLivEXXBvdG33PJP_jprzW+wUVRwZWgcw2qXPK6XbGw@mail.gmail.com>
Message-ID: <CAAyX=LH4DXAuyhTTzkBMxvtBOrJsZao3-cu=BPY0zxLdLEA2AQ@mail.gmail.com>

Hi All,

This has been successfully deployed in production, and the code (as-is) is
handling "many thousands" of connections per second from fake and
legitimate bots advertising themselves as Googlebot/Bingbot/etc with no
apparent issues/problems. The configuration we've deployed is essentially
the same as provided here (and in the code base).

Anyway, if anyone else ends up finding libvmod-dns helpful, please consider
it "emailware" -- ie, drop me an email and let me know (off-the-record, of
course) how you're making use of it. I'm curious more than anything!



-Ken


On Mon, Apr 1, 2013 at 6:21 PM, Kenneth Shaw <kenshaw at gmail.com> wrote:

> Hi,
>
> I spent a bit of time today developing a DNS module for Varnish.
>
> It is available here:
>
> https://github.com/kenshaw/libvmod-dns/
>
> The reason for this development is to cut off bots that abuse the
> User-Agent string (ie, claiming to be Googlebot/bingbot/etc.) by doing a
> reverse and then forward DNS against the client.ip/X-Forwarded-For header
> and comparing with a regex against the resultant domain.
>
> The logic is meant to work something like this:
>
> sub vcl_recv {
>     # do a dns check on "good" crawlers
>     if (req.http.user-agent ~ "(?i)(googlebot|bingbot|slurp|teoma)") {
>         # do a reverse lookup on the client.ip (X-Forwarded-For) and check
> that its in the allowed domains
>         set req.http.X-Crawler-DNS-Reverse =
> dns.rresolve(req.http.X-Forwarded-For);
>
>         # check that the RDNS points to an allowed domain -- 403 error if
> it doesn't
>         if (req.http.X-Crawler-DNS-Reverse !~
> "(?i)\.(googlebot\.com|search\.msn\.com|crawl\.yahoo\.net|ask\.com)$") {
>             error 403 "Forbidden";
>         }
>
>         # do a forward lookup on the DNS
>         set req.http.X-Crawler-DNS-Forward =
> dns.resolve(req.http.X-Crawler-DNS-Reverse);
>
>         # if the client.ip/X-Forwarded-For doesn't match, then the
> user-agent is fake
>         if (req.http.X-Crawler-DNS-Forward != req.http.X-Forwarded-For) {
>             error 403 "Forbidden";
>         }
>     }
> }
>
> While this is not being used in production (yet), I plan to do so later
> this week against a production system receiving ~10,000+ requests/sec. I
> will report back afterwards.
>
> I realize the code currently has issues (memory, documentation, etc.),
> which will be fixed in the near future.
>
> I also realize there are better ways to head malicious bots off at the
> pass through DNS, etc (which we are doing as well). The largest issue here
> for my purposes is that it is difficult / impossible to identify all
> traffic. Additionally, it is nice to be able to monitor the actual traffic
> coming through and not completely dropping it at the edge.
>
> Any input/comments against what I've written so far would be gladly
> appreciated! Thanks!
>
> -Ken
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130408/34772f1d/attachment.html>

From tfheen at varnish-software.com  Tue Apr  9 06:25:26 2013
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue,  9 Apr 2013 08:25:26 +0200
Subject: [PATCH] Set the waiter pipe as non-blocking and record overflows
Message-ID: <1365488726-19331-1-git-send-email-tfheen@varnish-software.com>

Fixes #1285
---

I wonder if it would make sense to pull the "set this fd as non-blocking" into a libvarnish function.  Tested (make check) on Linux with epoll and poll.  Not tested on FreeBSD and Solaris.

 bin/varnishd/waiter/cache_waiter_epoll.c  |   19 ++++++++++++++++---
 bin/varnishd/waiter/cache_waiter_kqueue.c |   15 +++++++++++++--
 bin/varnishd/waiter/cache_waiter_poll.c   |   19 +++++++++++++++++--
 bin/varnishd/waiter/cache_waiter_ports.c  |   10 +++++++---
 bin/varnishd/waiter/waiter.h              |    2 +-
 include/tbl/vsc_f_main.h                  |    5 +++++
 6 files changed, 59 insertions(+), 11 deletions(-)

diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index a15bbe5..35b0e20 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -39,6 +39,7 @@
 
 #include <fcntl.h>
 #include <stdlib.h>
+#include <math.h>
 
 #include "cache/cache.h"
 
@@ -49,7 +50,7 @@
 #  define EPOLLRDHUP 0
 #endif
 
-#define NEEV	100
+#define NEEV	8192
 
 struct vwe {
 	unsigned		magic;
@@ -220,12 +221,18 @@ vwe_timeout_idle_ticker(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwe_pass(void *priv, const struct sess *sp)
+vwe_pass(void *priv, struct sess *sp)
 {
 	struct vwe *vwe;
+	int written;
 
 	CAST_OBJ_NOTNULL(vwe, priv, VWE_MAGIC);
-	assert(sizeof sp == write(vwe->pipes[1], &sp, sizeof sp));
+
+	written = write(vwe->pipes[1], &sp, sizeof sp);
+	if (written != sizeof sp) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_OVERLOAD, NAN);
+	}
 }
 
 /*--------------------------------------------------------------------*/
@@ -248,6 +255,12 @@ vwe_init(void)
 	i = fcntl(vwe->pipes[0], F_SETFL, i);
 	assert(i != -1);
 
+	i = fcntl(vwe->pipes[1], F_GETFL);
+	assert(i != -1);
+	i |= O_NONBLOCK;
+	i = fcntl(vwe->pipes[1], F_SETFL, i);
+	assert(i != -1);
+
 	i = fcntl(vwe->timer_pipes[0], F_GETFL);
 	assert(i != -1);
 	i |= O_NONBLOCK;
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index 8d37ad6..20e03a9 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -210,12 +210,17 @@ vwk_thread(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwk_pass(void *priv, const struct sess *sp)
+vwk_pass(void *priv, struct sess *sp)
 {
 	struct vwk *vwk;
 
 	CAST_OBJ_NOTNULL(vwk, priv, VWK_MAGIC);
-	assert(sizeof sp == write(vwk->pipes[1], &sp, sizeof sp));
+
+	written = write(vwk->pipes[1], &sp, sizeof sp);
+	if (written != sizeof sp) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_OVERLOAD, NAN);
+	}
 }
 
 /*--------------------------------------------------------------------*/
@@ -238,6 +243,12 @@ vwk_init(void)
 	i = fcntl(vwk->pipes[0], F_SETFL, i);
 	assert(i != -1);
 
+	i = fcntl(vwe->pipes[1], F_GETFL);
+	assert(i != -1);
+	i |= O_NONBLOCK;
+	i = fcntl(vwe->pipes[1], F_SETFL, i);
+	assert(i != -1);
+
 	AZ(pthread_create(&vwk->thread, NULL, vwk_thread, vwk));
 	return (vwk);
 }
diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c
index 4339617..34bafc2 100644
--- a/bin/varnishd/waiter/cache_waiter_poll.c
+++ b/bin/varnishd/waiter/cache_waiter_poll.c
@@ -30,8 +30,10 @@
 
 #include "config.h"
 
+#include <fcntl.h>
 #include <poll.h>
 #include <stdlib.h>
+#include <math.h>
 
 #include "cache/cache.h"
 
@@ -191,13 +193,18 @@ vwp_main(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwp_poll_pass(void *priv, const struct sess *sp)
+vwp_poll_pass(void *priv, struct sess *sp)
 {
 	struct vwp *vwp;
+	int written;
 
 	CAST_OBJ_NOTNULL(vwp, priv, VWP_MAGIC);
 
-	assert(sizeof sp == write(vwp->pipes[1], &sp, sizeof sp));
+	written = write(vwp->pipes[1], &sp, sizeof sp);
+	if (written != sizeof sp) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_OVERLOAD, NAN);
+	}
 }
 
 /*--------------------------------------------------------------------*/
@@ -205,12 +212,20 @@ vwp_poll_pass(void *priv, const struct sess *sp)
 static void *
 vwp_poll_init(void)
 {
+	int i;
 	struct vwp *vwp;
 
 	ALLOC_OBJ(vwp, VWP_MAGIC);
 	AN(vwp);
 	VTAILQ_INIT(&vwp->sesshead);
 	AZ(pipe(vwp->pipes));
+
+	i = fcntl(vwp->pipes[1], F_GETFL);
+	assert(i != -1);
+	i |= O_NONBLOCK;
+	i = fcntl(vwp->pipes[1], F_SETFL, i);
+	assert(i != -1);
+
 	vwp_pollspace(vwp, 256);
 	AZ(pthread_create(&vwp->poll_thread, NULL, vwp_main, vwp));
 	return (vwp);
diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index 3c07b95..e6825f3 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -240,14 +240,18 @@ vws_thread(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vws_pass(void *priv, const struct sess *sp)
+vws_pass(void *priv, struct sess *sp)
 {
 	int r;
 	struct vws *vws;
 
 	CAST_OBJ_NOTNULL(vws, priv, VWS_MAGIC);
-	while((r = port_send(vws->dport, 0, TRUST_ME(sp))) == -1 &&
-		errno == EAGAIN);
+	r = port_send(vws->dport, 0, TRUST_ME(sp));
+	if (r == -1 && errno == EAGAIN) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_OVERLOAD, NAN);
+		return;
+	}
 	AZ(r);
 }
 
diff --git a/bin/varnishd/waiter/waiter.h b/bin/varnishd/waiter/waiter.h
index 9f9f795..ce46e4d 100644
--- a/bin/varnishd/waiter/waiter.h
+++ b/bin/varnishd/waiter/waiter.h
@@ -31,7 +31,7 @@
 struct sess;
 
 typedef void* waiter_init_f(void);
-typedef void waiter_pass_f(void *priv, const struct sess *);
+typedef void waiter_pass_f(void *priv, struct sess *);
 
 #define WAITER_DEFAULT		"platform dependent"
 
diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h
index 3f6d3fe..6506668 100644
--- a/include/tbl/vsc_f_main.h
+++ b/include/tbl/vsc_f_main.h
@@ -88,6 +88,11 @@ VSC_F(sess_fail,		uint64_t, 1, 'c',
 	" some resource like filedescriptors."
 )
 
+VSC_F(sess_pipe_overflow,		uint64_t, 1, 'c',
+    "Session pipe overflow",
+	"Count of sessions dropped due to the session pipe overflowing."
+)
+
 /*---------------------------------------------------------------------*/
 
 VSC_F(client_req_400,		uint64_t, 1, 'a',
-- 
1.7.10.4




From mikael.carlsson at warpnine.se  Tue Apr  9 09:18:05 2013
From: mikael.carlsson at warpnine.se (Mikael Carlsson)
Date: Tue, 9 Apr 2013 11:18:05 +0200
Subject: check varnish cache hitrate ratio bashscript
Message-ID: <CAAmeg-p07ZSLP=u42r-kTyqa14JVoRAjfxDUy8iBz1=8VOVG6Q@mail.gmail.com>

Hello.

We use nagios and had a varnish bash script that returned the cache hitrate
ratio to our nagios server.
Now this script is gone and i cannot find it on the web.

Does anyone here know any solution for a varnish bash script that returns
the cache hitrate ratio to our nagios server?

Kind regards.

Med v?nliga h?lsningar

Mikael Carlsson
IT-Support and Developer
Mobil: +46 729 64 13 35
V?xel: +46 31 600 161
Email: mikael.carlsson at warpnine.se

info at warpnine.se
http://www.warpnine.se
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130409/ed2dfafc/attachment.html>

From tfheen at varnish-software.com  Tue Apr  9 12:40:54 2013
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue,  9 Apr 2013 14:40:54 +0200
Subject: [PATCH] Set the waiter pipe as non-blocking and record overflows
In-Reply-To: <1365488726-19331-1-git-send-email-tfheen@varnish-software.com>
References: <1365488726-19331-1-git-send-email-tfheen@varnish-software.com>
Message-ID: <1365511254-27510-1-git-send-email-tfheen@varnish-software.com>

Fixes #1285
---
 bin/varnishd/cache/cache.h                |    1 +
 bin/varnishd/waiter/cache_waiter.c        |   14 ++++++++++++++
 bin/varnishd/waiter/cache_waiter_epoll.c  |   24 ++++++++----------------
 bin/varnishd/waiter/cache_waiter_kqueue.c |   15 ++++++---------
 bin/varnishd/waiter/cache_waiter_poll.c   |    9 +++++++--
 bin/varnishd/waiter/cache_waiter_ports.c  |   10 +++++++---
 bin/varnishd/waiter/waiter.h              |    2 +-
 include/tbl/sess_close.h                  |    1 +
 include/tbl/vsc_f_main.h                  |    5 +++++
 include/vfil.h                            |    1 +
 lib/libvarnish/vfil.c                     |   13 +++++++++++++
 11 files changed, 64 insertions(+), 31 deletions(-)

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 31b8384..cd52d8b 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1035,6 +1035,7 @@ void VMOD_Init(void);
 void WAIT_Enter(struct sess *sp);
 void WAIT_Init(void);
 const char *WAIT_GetName(void);
+void WAIT_Write_Session(struct sess *sp, int fd);
 
 /* cache_wrk.c */
 
diff --git a/bin/varnishd/waiter/cache_waiter.c b/bin/varnishd/waiter/cache_waiter.c
index 53d7f0f..1b868fe 100644
--- a/bin/varnishd/waiter/cache_waiter.c
+++ b/bin/varnishd/waiter/cache_waiter.c
@@ -32,6 +32,7 @@
 #include <stdio.h>
 #include <unistd.h>
 #include <string.h>
+#include <math.h>
 
 #include "cache/cache.h"
 
@@ -77,3 +78,16 @@ WAIT_Enter(struct sess *sp)
 		SES_Close(sp, SC_REM_CLOSE);
 	waiter->pass(waiter_priv, sp);
 }
+
+void
+WAIT_Write_Session(struct sess *sp, int fd)
+{
+	ssize_t written;
+	written = write(fd, &sp, sizeof sp);
+	if (written != sizeof sp && (errno == EAGAIN || errno == EWOULDBLOCK)) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_SESS_PIPE_OVERFLOW, NAN);
+		return;
+	}
+	assert (written == sizeof sp);
+}
diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index a15bbe5..ba41210 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -37,19 +37,19 @@
 
 #include <sys/epoll.h>
 
-#include <fcntl.h>
 #include <stdlib.h>
 
 #include "cache/cache.h"
 
 #include "waiter/waiter.h"
 #include "vtim.h"
+#include "vfil.h"
 
 #ifndef EPOLLRDHUP
 #  define EPOLLRDHUP 0
 #endif
 
-#define NEEV	100
+#define NEEV	8192
 
 struct vwe {
 	unsigned		magic;
@@ -220,12 +220,13 @@ vwe_timeout_idle_ticker(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwe_pass(void *priv, const struct sess *sp)
+vwe_pass(void *priv, struct sess *sp)
 {
 	struct vwe *vwe;
 
 	CAST_OBJ_NOTNULL(vwe, priv, VWE_MAGIC);
-	assert(sizeof sp == write(vwe->pipes[1], &sp, sizeof sp));
+
+	WAIT_Write_Session(sp, vwe->pipes[1]);
 }
 
 /*--------------------------------------------------------------------*/
@@ -233,7 +234,6 @@ vwe_pass(void *priv, const struct sess *sp)
 static void *
 vwe_init(void)
 {
-	int i;
 	struct vwe *vwe;
 
 	ALLOC_OBJ(vwe, VWE_MAGIC);
@@ -242,17 +242,9 @@ vwe_init(void)
 	AZ(pipe(vwe->pipes));
 	AZ(pipe(vwe->timer_pipes));
 
-	i = fcntl(vwe->pipes[0], F_GETFL);
-	assert(i != -1);
-	i |= O_NONBLOCK;
-	i = fcntl(vwe->pipes[0], F_SETFL, i);
-	assert(i != -1);
-
-	i = fcntl(vwe->timer_pipes[0], F_GETFL);
-	assert(i != -1);
-	i |= O_NONBLOCK;
-	i = fcntl(vwe->timer_pipes[0], F_SETFL, i);
-	assert(i != -1);
+	AZ(VFIL_nonblocking(vwe->pipes[0]));
+	AZ(VFIL_nonblocking(vwe->pipes[1]));
+	AZ(VFIL_nonblocking(vwe->timer_pipes[0]));
 
 	AZ(pthread_create(&vwe->timer_thread,
 	    NULL, vwe_timeout_idle_ticker, vwe));
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index 8d37ad6..1c51399 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -38,7 +38,6 @@
 #include <sys/types.h>
 #include <sys/event.h>
 
-#include <fcntl.h>
 #include <stdlib.h>
 #include <unistd.h>
 
@@ -46,6 +45,7 @@
 
 #include "waiter/waiter.h"
 #include "vtim.h"
+#include "vfil.h"
 
 #define NKEV	100
 
@@ -210,12 +210,13 @@ vwk_thread(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwk_pass(void *priv, const struct sess *sp)
+vwk_pass(void *priv, struct sess *sp)
 {
 	struct vwk *vwk;
 
 	CAST_OBJ_NOTNULL(vwk, priv, VWK_MAGIC);
-	assert(sizeof sp == write(vwk->pipes[1], &sp, sizeof sp));
+
+	WAIT_Write_Session(sp, vwk->pipes[1]);
 }
 
 /*--------------------------------------------------------------------*/
@@ -223,7 +224,6 @@ vwk_pass(void *priv, const struct sess *sp)
 static void *
 vwk_init(void)
 {
-	int i;
 	struct vwk *vwk;
 
 	ALLOC_OBJ(vwk, VWK_MAGIC);
@@ -232,11 +232,8 @@ vwk_init(void)
 	VTAILQ_INIT(&vwk->sesshead);
 	AZ(pipe(vwk->pipes));
 
-	i = fcntl(vwk->pipes[0], F_GETFL);
-	assert(i != -1);
-	i |= O_NONBLOCK;
-	i = fcntl(vwk->pipes[0], F_SETFL, i);
-	assert(i != -1);
+	AZ(VFIL_nonblocking(vwk->pipes[0]));
+	AZ(VFIL_nonblocking(vwk->pipes[1]));
 
 	AZ(pthread_create(&vwk->thread, NULL, vwk_thread, vwk));
 	return (vwk);
diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c
index 4339617..d2c29d1 100644
--- a/bin/varnishd/waiter/cache_waiter_poll.c
+++ b/bin/varnishd/waiter/cache_waiter_poll.c
@@ -30,6 +30,7 @@
 
 #include "config.h"
 
+#include <fcntl.h>
 #include <poll.h>
 #include <stdlib.h>
 
@@ -37,6 +38,7 @@
 
 #include "waiter/waiter.h"
 #include "vtim.h"
+#include "vfil.h"
 
 #define NEEV	128
 
@@ -191,13 +193,13 @@ vwp_main(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vwp_poll_pass(void *priv, const struct sess *sp)
+vwp_poll_pass(void *priv, struct sess *sp)
 {
 	struct vwp *vwp;
 
 	CAST_OBJ_NOTNULL(vwp, priv, VWP_MAGIC);
 
-	assert(sizeof sp == write(vwp->pipes[1], &sp, sizeof sp));
+	WAIT_Write_Session(sp, vwp->pipes[1]);
 }
 
 /*--------------------------------------------------------------------*/
@@ -211,6 +213,9 @@ vwp_poll_init(void)
 	AN(vwp);
 	VTAILQ_INIT(&vwp->sesshead);
 	AZ(pipe(vwp->pipes));
+
+	AZ(VFIL_nonblocking(vwp->pipes[1]));
+
 	vwp_pollspace(vwp, 256);
 	AZ(pthread_create(&vwp->poll_thread, NULL, vwp_main, vwp));
 	return (vwp);
diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index 3c07b95..a3bd96d 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -240,14 +240,18 @@ vws_thread(void *priv)
 /*--------------------------------------------------------------------*/
 
 static void
-vws_pass(void *priv, const struct sess *sp)
+vws_pass(void *priv, struct sess *sp)
 {
 	int r;
 	struct vws *vws;
 
 	CAST_OBJ_NOTNULL(vws, priv, VWS_MAGIC);
-	while((r = port_send(vws->dport, 0, TRUST_ME(sp))) == -1 &&
-		errno == EAGAIN);
+	r = port_send(vws->dport, 0, TRUST_ME(sp));
+	if (r == -1 && errno == EAGAIN) {
+		VSC_C_main->sess_pipe_overflow++;
+		SES_Delete(sp, SC_SESS_PIPE_OVERFLOW, NAN);
+		return;
+	}
 	AZ(r);
 }
 
diff --git a/bin/varnishd/waiter/waiter.h b/bin/varnishd/waiter/waiter.h
index 9f9f795..ce46e4d 100644
--- a/bin/varnishd/waiter/waiter.h
+++ b/bin/varnishd/waiter/waiter.h
@@ -31,7 +31,7 @@
 struct sess;
 
 typedef void* waiter_init_f(void);
-typedef void waiter_pass_f(void *priv, const struct sess *);
+typedef void waiter_pass_f(void *priv, struct sess *);
 
 #define WAITER_DEFAULT		"platform dependent"
 
diff --git a/include/tbl/sess_close.h b/include/tbl/sess_close.h
index c9d28ba..6246a51 100644
--- a/include/tbl/sess_close.h
+++ b/include/tbl/sess_close.h
@@ -39,5 +39,6 @@ SESS_CLOSE(TX_PIPE,	"Piped transaction")
 SESS_CLOSE(TX_ERROR,	"Error transaction")
 SESS_CLOSE(TX_EOF,	"EOF transmission")
 SESS_CLOSE(OVERLOAD,	"Out of some resource")
+SESS_CLOSE(SESS_PIPE_OVERFLOW,	"Session pipe overflow")
 
 /*lint -restore */
diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h
index 3f6d3fe..6506668 100644
--- a/include/tbl/vsc_f_main.h
+++ b/include/tbl/vsc_f_main.h
@@ -88,6 +88,11 @@ VSC_F(sess_fail,		uint64_t, 1, 'c',
 	" some resource like filedescriptors."
 )
 
+VSC_F(sess_pipe_overflow,		uint64_t, 1, 'c',
+    "Session pipe overflow",
+	"Count of sessions dropped due to the session pipe overflowing."
+)
+
 /*---------------------------------------------------------------------*/
 
 VSC_F(client_req_400,		uint64_t, 1, 'a',
diff --git a/include/vfil.h b/include/vfil.h
index 74885ef..533dfd8 100644
--- a/include/vfil.h
+++ b/include/vfil.h
@@ -33,3 +33,4 @@ int seed_random(void);
 int VFIL_tmpfile(char *);
 char *VFIL_readfile(const char *pfx, const char *fn, ssize_t *sz);
 char *VFIL_readfd(int fd, ssize_t *sz);
+int VFIL_nonblocking(int fd);
diff --git a/lib/libvarnish/vfil.c b/lib/libvarnish/vfil.c
index 6bcbde5..df02a48 100644
--- a/lib/libvarnish/vfil.c
+++ b/lib/libvarnish/vfil.c
@@ -123,3 +123,16 @@ VFIL_readfile(const char *pfx, const char *fn, ssize_t *sz)
 	errno = err;
 	return (r);
 }
+
+int
+VFIL_nonblocking(int fd)
+{
+	int i;
+
+	i = fcntl(fd, F_GETFL);
+	assert(i != -1);
+	i |= O_NONBLOCK;
+	i = fcntl(fd, F_SETFL, i);
+       assert(i != -1);
+       return (i);
+}
-- 
1.7.10.4




From tfheen at varnish-software.com  Wed Apr 10 10:18:40 2013
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Wed, 10 Apr 2013 12:18:40 +0200
Subject: [PATCH 5/8] Return 503 when Vary-headers references header names
 more than 127 (out limit) characters long.
In-Reply-To: <40262.1363697446@critter.freebsd.dk>
References: <1363625849-2790-1-git-send-email-martin@varnish-software.com>
 <1363625849-2790-5-git-send-email-martin@varnish-software.com>
 <CANTTouXMDQRcO_xscL+H8YS-C=9Ze9PgM6z-WeY8=VvHB2kU8Q@mail.gmail.com>
 <CANTn4cq0uCpwzJf=198QJL78rS7eX0jkP+YhGNwxE8LYWyg9mg@mail.gmail.com>
 <40262.1363697446@critter.freebsd.dk>
Message-ID: <20130410101840.GA3316@err.no>

]] Poul-Henning Kamp

> In message <CANTn4cq0uCpwzJf=198QJL78rS7eX0jkP+YhGNwxE8LYWyg9mg at mail.gmail.com>, Martin Blix Grydeland wr
> ites:
> 
> >True, but that isn't the only place where that is used as an unsigned char
> >(e.g. http_findhdr). So Varnish header names are limited to signed char I
> >believe.
> 
> Are there real world examples of non-insane uses of headers longer
> than 127 characters ?

For the contents of headers?  Sure, cookies spring to mind.  The the
names of header fields?  Not that I know of-

-- 
Tollef Fog Heen
Technical lead | Varnish Software AS
t: +47 21 98 92 64
We Make Websites Fly!



From bilbo at hobbiton.org  Sun Apr 14 03:33:44 2013
From: bilbo at hobbiton.org (Leif Pedersen)
Date: Sat, 13 Apr 2013 22:33:44 -0500
Subject: Cache file on ZFS volume or raw block device
Message-ID: <CAK-wPOg3D=vwVE+GgMWp8b3sLxKF1cD2=90Q=aP+VS+19M=_rA@mail.gmail.com>

I just got serious about using ZFS on FreeBSD over the last few weeks. I
got to thinking, why incur the overhead of a filesystem to the Varnish
cache file? I figured I'd try an experiment on my desktop, and created a
ZFS volume for the cache file. Then I set varnishd_storage (/etc/rc.conf)
to "file,/dev/zvol/tank/varnish-cache,8G". But when I tried to start
Varnish, I got the error: "/dev/zvol/tank/varnish-cache" is neither file
nor directory. Searching, I found some easy code about it in
storage/stevedore_utils.c<https://www.varnish-cache.org/trac/browser/bin/varnishd/storage/stevedore_utils.c>,
and naturally the error check makes sense.

What do you guys think? Am I wasting my time, or is it worth the effort to
avoid some filesystem overhead for cache IO? I thought that perhaps
allowing the storage to be a raw block device might also be useful for some
people (maybe my future self) who want to put it on a dedicated SSD.

Thoughts?

- Leif

-- 

As implied by email protocols, the information in this message is
not confidential.  Any middle-man or recipient may inspect, modify,
copy, forward, reply to, delete, or filter email for any purpose unless
said parties are otherwise obligated.  As the sender, I acknowledge that
I have a lower expectation of the control and privacy of this message
than I would a post-card.  Further, nothing in this message is
legally binding without cryptographic evidence of its integrity.

http://bilbo.hobbiton.org/wiki/Eat_My_Sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130413/4e1dca2c/attachment.html>

From perbu at varnish-software.com  Mon Apr 15 07:44:59 2013
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 15 Apr 2013 09:44:59 +0200
Subject: Cache file on ZFS volume or raw block device
In-Reply-To: <CAK-wPOg3D=vwVE+GgMWp8b3sLxKF1cD2=90Q=aP+VS+19M=_rA@mail.gmail.com>
References: <CAK-wPOg3D=vwVE+GgMWp8b3sLxKF1cD2=90Q=aP+VS+19M=_rA@mail.gmail.com>
Message-ID: <CAOXZevDWsfhRkHa086kAGsT+YU_0Z5gPOaMYA0YPkq3fE-i-Jw@mail.gmail.com>

hi,

On Sun, Apr 14, 2013 at 5:33 AM, Leif Pedersen <bilbo at hobbiton.org> wrote:

> What do you guys think? Am I wasting my time, or is it worth the effort to
> avoid some filesystem overhead for cache IO? I thought that perhaps
> allowing the storage to be a raw block device might also be useful for some
> people (maybe my future self) who want to put it on a dedicated SSD.
>

We tried disabling those checks and running Varnish on raw Linux IO devices
a couple of years back. Not completely raw IO devices but those that still
keep the page cache active. The results where rather disappointing with a
significant reduction in performance. I guess the
more aggressive preemptive caching in the filesystem help out a bit more.

It is not a costly experiment to conduct. Go for it. The VM your playing
with is a totally different one.

-- 
 <http://www.varnish-software.com/> *Per Buer*
CEO | Varnish Software AS
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20130415/42cfc093/attachment.html>

From klaus.brunner at gmail.com  Tue Apr 16 13:24:36 2013
From: klaus.brunner at gmail.com (Klaus Brunner)
Date: Tue, 16 Apr 2013 15:24:36 +0200
Subject: [PATCH] log "total time" in varnishncsa
Message-ID: <CAOPOstJZu+1HEiE9_5nt=2LRq_e8S2mkB_5+MVDmXjB5T0fg3Q@mail.gmail.com>

This is a patch against master to provide the "total time" log field
in varnishncsa (I posted a previous version of this against 3.0.3 back
in January). As requested by tfheen, now it doesn't add another
sscanf() call and the doc is updated:

https://gist.github.com/KlausBrunner/4175582#file-varnishncsa-request-time-vs-master-patch

Caveat: I've had to do some guesswork thanks to the new format of
ReqEnd log entries. It seems to work correctly based on my extremely
limited understanding of what CNT_Request() does, but I wouldn't bet a
Mars rover on it.

Klaus



From mailman at haukebruno.de  Tue Apr 30 08:21:06 2013
From: mailman at haukebruno.de (Hauke)
Date: Tue, 30 Apr 2013 10:21:06 +0200
Subject: Error handling for missing secret-file
Message-ID: <3778574.KJNA5Idu5F@notenbuch.haukebruno.de>

Hi there,

I would like to suggest a simple file check if using the '-S mysecretfile' 
option on varnishd startup. If there is no file given, varnishd does not report 
that circumstance yet (selfbuild 3.0.3).

-- 
Best regards,
Hauke