From perbu at varnish-software.com  Wed Nov  2 08:42:23 2011
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 2 Nov 2011 09:42:23 +0100
Subject: Doing tcp splicing for pipe?
Message-ID: <CAOXZevBKDbHjq65M3XYWmU_Bnp9je51h92tgm-i41NL03v7HGQ@mail.gmail.com>

Hi.

We're messing around with websockets these days (for the varnish mgmt
console). So, in varnish we end up detecting an upgrade and doing
return(pipe) for the connections. It works well on the scale we are using
it. However, if we where to handle 100k concurrent users there would have
to be 100k threads to handle it which would take a fair bit of resources
doing silly bit-shuffling.

Would it make sense just to splice the connections in vcl_pipe? We'd still
need to keep an eye on the connection for book-keeping, handling errors and
such so we'd still probably need a thread looking after all of the
connections.

Does it makes sense? Is it premature to do such optimization now?

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111102/7cba524b/attachment.html>

From harm.verhagen at gmail.com  Wed Nov  2 12:27:59 2011
From: harm.verhagen at gmail.com (Harm Verhagen)
Date: Wed, 2 Nov 2011 13:27:59 +0100
Subject: varnishtop feature request: show _expensive_ urls
In-Reply-To: <20110629143027.E55234@ishtar.drwilco.net>
References: <BANLkTikCG8oxOa7StRXKKkuKE8mo4cd6vA@mail.gmail.com>
	<20110629143027.E55234@ishtar.drwilco.net>
Message-ID: <CAKWpb6_ZuUxpAYow4k18f+y16O9y2VSoxrGt+5pMEf+X-XTRHg@mail.gmail.com>

On Wed, Jun 29, 2011 at 3:16 PM, Rogier R. Mulhuijzen
<drwilco at drwilco.net>wrote:

>
>  Currently varnishtop shows the top urls purely based on frequency.
>> Depending
>> on your usecase that might be usefull, but might also NOT give you the
>> bottlenecks you are looking for.
>>
>
> Agreed.
>
>
>  I'd like to use varnish for the following (example) scenario:
>>
>
> ---- SNIP ----
>
>
>  expensive url  =   backend url with high  total time spend  (freq x
>> duration)    (So the slow ones, that are called often)
>>
>> What do you guys think ?
>> This would instantly give the problem areas of a website, no ?
>>
>
> I think this would work quite nicely. I would just change the "freq x
> duration" definition to "cumulative duration".
>
> It would mean adding the 5th (or 6th) field from ReqEnd to the counter
> instead of 1. The hard part is probably (guessing here, have yet to look at
> v-top source) getting the ReqEnd values to add to the individual counters.
>
> Throw it on the shopping list, I'd say.
>


Whats the best way to get this on the shopping list ?

Regards,
Harm Verhagen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111102/c683bb79/attachment.html>

From dtwork.7 at mailbolt.com  Thu Nov  3 18:44:35 2011
From: dtwork.7 at mailbolt.com (dtwork.7 at mailbolt.com)
Date: Thu, 03 Nov 2011 11:44:35 -0700
Subject: Mission:Impossible -- a known-to-(really!)-work Varnish3 + Drupal7
	VCL?
Message-ID: <1320345875.17570.140660994294229@webmail.messagingengine.com>

Hi,

I'm on a 'mission' -- to identify a solid, recommended, known-to-work
with/on Drupal 7 Varnish3 VCL, including module selections/configs,
settings.php.

Basically, a solid starting point.

I've posted about it here:

  "Looking for (just) one, good, complete example of Varnish3+Drupal7
  configuration"
  http://groups.drupal.org/node/187799

What troubles me is the statement, there:

	"There's no cookie-cutter varnish configuration because each
	instance is very site specific."

For peak-performance tweaking, I agree.

For basic functionality, my current perception is that that's flatly
wrong.

The same exists for Varnish2 + Pressflow6 -- it's referenced from the
Varnish wiki.  It works, really well.  Install Pressflow6, install
Varnish 2, set up the modules/settings as per the FourKitchens wiki --
done. Works for the "90%" of basic installs.

For V3 + D7, however, it's a solution that's not readily found.  At
least, I haven't -- and, apparently, neither have lots of others
similarly thrashing about.

I'm quite certain that there are vetted, working V3+D7 VCLs in
production.  Heck, people are building companies and giving
presentations on solutions using it. I've contacted a number of them
asking them to consider contributing their solutions.  So far, noone's
taken me up on the idea.  Iiuc (?), Varnish Co. itself is using Drupal
-- I'd guess that's with a known-to-work Varnish3 VCL?

To THIS list, I'd like to ask:

	Is there a known-to-work-in-real-production V3+D7 VCL that
	someone will share -- that's solid enough that it can/will be
	referenced as a good starting point on the Varnish wiki?

It would be hugely useful to short-circuit the random noise out there,
get a good working starting point, so that we can get sites up and
running -- and then pay money to folks as needed to optimize & tweak as
we grow.

And, yes, it's possible to scour countless posts, etc. and (randomly)
piece together snippets -- that's part of the problem.  For every
snippet that someone says 'works', someone else in the same thread
typically says, no it doesn't, and then the discussion peters out.  I
certainly admit I am unable to understand what the fundamental
difference between the V2+PF6 and V3+D7 scenarios is that explains why
we HAVE a 'good' community recommended/documented starter config for the
former, but not for the latter.

If, otoh, the answer really IS "every installation is different", and
that there is no longer a recommended (by community members who have
already done) starting point, that too would be really valuable to hear
& know -- from this list, where I'm assuming the expertise abounds.  At
the very least, that could usefully be communicated to the rest of us
mere-mortals and we could all stop looking for something that's simply
not there.

Thanks!



From mattias at nucleus.be  Fri Nov  4 07:53:42 2011
From: mattias at nucleus.be (Mattias Geniar)
Date: Fri, 4 Nov 2011 08:53:42 +0100
Subject: Mission:Impossible -- a known-to-(really!)-work Varnish3 +
	Drupal7VCL?
In-Reply-To: <1320345875.17570.140660994294229@webmail.messagingengine.com>
References: <1320345875.17570.140660994294229@webmail.messagingengine.com>
Message-ID: <18834F5BEC10824891FB8B22AC821A5A01C6CFA2@nucleus-srv01.Nucleus.local>

> 	Is there a known-to-work-in-real-production V3+D7 VCL that
> 	someone will share -- that's solid enough that it can/will be
> 	referenced as a good starting point on the Varnish wiki?

Hi there,

I've created a pretty basic template for Drupal 7 (which needs further
extending), but to my experience is pretty solid. As you say, it will be
dependant on modules that set specific cookies etc.
You can have a peak here:
https://github.com/mattiasgeniar/varnish-3.0-configuration-templates

The VCL you're looking for is in the conf.d directory, I've separated
the fetch/receive actions.

Regards,
Mattias Geniar



From ask at develooper.com  Thu Nov 10 08:47:31 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Thu, 10 Nov 2011 00:47:31 -0800
Subject: gzip crash
Message-ID: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>

I noticed today that our varnish servers a few times have crashed with an error like the ones below.  We're running 3.0.2 on CentOS/Linux x86_64.  Most of the startup variables are the defaults from the rpm.

Our .vcl distributes requests to various backends depending on host header and request URL and sets beresp.do_gzip for content-type ~ text/.* requests.

The load is really low (~30-100 requests a second per box).


  - ask


Nov  9 23:21:16 varnish3 varnishd[31763]: Child (31764) Panic message: Assert error in VGZ_Ibuf(), cache_gzip.c line 222:   Condition((vg->vz.avail_in) == 0) not true. errno = 12 (Cannot allocate memory) thread = (cache-worker) ident = Linux,2.6.18-274.7.1.el5,x86_64,-sfile,-smalloc,-hcritbit,epoll Backtrace:   0x42c7a6: /usr/sbin/varnishd [0x42c7a6]   0x4227da: /usr/sbin/varnishd(VGZ_Ibuf+0x7a) [0x4227da]   0x422e19: /usr/sbin/varnishd [0x422e19]   0x4215fd: /usr/sbin/varnishd(FetchBody+0x3fd) [0x4215fd]   0x4153e8: /usr/sbin/varnishd [0x4153e8]   0x417ab6: /usr/sbin/varnishd(CNT_Session+0x9f6) [0x417ab6]   0x42efb8: /usr/sbin/varnishd [0x42efb8]   0x42e19b: /usr/sbin/varnishd [0x42e19b]   0x3b8420673d: /lib64/libpthread.so.0 [0x3b8420673d]   0x3b83ad44bd: /lib64/libc.so.6(clone+0x6d) [0x3b83ad44bd] sp = 0x2aaaecbc0008 {   fd = 18, id = 18, xid = 556098001,   client = 10.50.0.51 54247,   step = STP_FETCHBODY,   handling = deliver,   err_code = 200, err_reason = (null),   restarts = 0, esi_level = 0   flags =  


Nov  9 10:12:43 varnish2 varnishd[9043]: Child (9044) Panic message: Assert error in VGZ_Ibuf(), cache_gzip.c line 222:   Condition((vg->vz.avail_in) == 0) not true. errno = 12 (Cannot allocate memory) thread = (cache-worker) ident = Linux,2.6.18-238.9.1.el5,x86_64,-sfile,-smalloc,-hcritbit,epoll Backtrace:   0x42c7a6: /usr/sbin/varnishd [0x42c7a6]   0x4227da: /usr/sbin/varnishd(VGZ_Ibuf+0x7a) [0x4227da]   0x422e19: /usr/sbin/varnishd [0x422e19]   0x4215fd: /usr/sbin/varnishd(FetchBody+0x3fd) [0x4215fd]   0x4153e8: /usr/sbin/varnishd [0x4153e8]   0x417ab6: /usr/sbin/varnishd(CNT_Session+0x9f6) [0x417ab6]   0x42efb8: /usr/sbin/varnishd [0x42efb8]   0x42e19b: /usr/sbin/varnishd [0x42e19b]   0x314880673d: /lib64/libpthread.so.0 [0x314880673d]   0x31480d44bd: /lib64/libc.so.6(clone+0x6d) [0x31480d44bd] sp = 0x2aaaece29008 {   fd = 18, id = 18, xid = 1952412410,   client = 10.50.0.77 58679,   step = STP_FETCHBODY,   handling = deliver,   err_code = 200, err_reason = (null),   restarts = 0, esi_level = 0   flags =  d

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111110/3c7e286a/attachment.bin>

From apj at mutt.dk  Thu Nov 10 08:52:25 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 10 Nov 2011 09:52:25 +0100
Subject: gzip crash
In-Reply-To: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
References: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
Message-ID: <20111110085225.GY13866@nerd.dk>

On Thu, Nov 10, 2011 at 12:47:31AM -0800, Ask Bj?rn Hansen wrote:

> I noticed today that our varnish servers a few times have crashed with an
> error like the ones below.  We're running 3.0.2 on CentOS/Linux x86_64.  Most
> of the startup variables are the defaults from the rpm.

https://www.varnish-cache.org/trac/ticket/1036

-- 
Andreas



From perbu at varnish-software.com  Thu Nov 10 08:58:41 2011
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 10 Nov 2011 09:58:41 +0100
Subject: gzip crash
In-Reply-To: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
References: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
Message-ID: <CAOXZevB9LME1qgh9NfxaCiwGngcrYqsuA+LrZtJgKBO7Mz+sbA@mail.gmail.com>

On Thu, Nov 10, 2011 at 9:47 AM, Ask Bj?rn Hansen <ask at develooper.com>wrote:

> I noticed today that our varnish servers a few times have crashed with an
> error like the ones below.


Tip of the day; panic.show in the CLI creates really nice assert messages.
Syslog also chops of the tail.

Bonus: panic.clear clears the panic so you can automate reporting of panics.


Per.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111110/981dee1d/attachment.html>

From ask at develooper.com  Thu Nov 10 15:51:35 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Thu, 10 Nov 2011 07:51:35 -0800
Subject: gzip crash
In-Reply-To: <CAOXZevB9LME1qgh9NfxaCiwGngcrYqsuA+LrZtJgKBO7Mz+sbA@mail.gmail.com>
References: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
	<CAOXZevB9LME1qgh9NfxaCiwGngcrYqsuA+LrZtJgKBO7Mz+sbA@mail.gmail.com>
Message-ID: <337049ED-ADA3-4F32-8E59-F8C113036EA8@develooper.com>


On Nov 10, 2011, at 0:58, Per Buer wrote:

> Tip of the day; panic.show in the CLI creates really nice assert messages. Syslog also chops of the tail.

Sorry about that; the poor formatting tripped me up and I didn't realize how incomplete that was.  Thank you for the hint!

Seeing more data I see that most crashes are from un-gzip'ing responses for our nagios check (not supporting compression).


 - ask

Last panic at: Wed, 09 Nov 2011 10:12:43 GMT
Assert error in VGZ_Ibuf(), cache_gzip.c line 222:
  Condition((vg->vz.avail_in) == 0) not true.
errno = 12 (Cannot allocate memory)
thread = (cache-worker)
ident = Linux,2.6.18-238.9.1.el5,x86_64,-sfile,-smalloc,-hcritbit,epoll
Backtrace:
  0x42c7a6: /usr/sbin/varnishd [0x42c7a6]
  0x4227da: /usr/sbin/varnishd(VGZ_Ibuf+0x7a) [0x4227da]
  0x422e19: /usr/sbin/varnishd [0x422e19]
  0x4215fd: /usr/sbin/varnishd(FetchBody+0x3fd) [0x4215fd]
  0x4153e8: /usr/sbin/varnishd [0x4153e8]
  0x417ab6: /usr/sbin/varnishd(CNT_Session+0x9f6) [0x417ab6]
  0x42efb8: /usr/sbin/varnishd [0x42efb8]
  0x42e19b: /usr/sbin/varnishd [0x42e19b]
  0x314880673d: /lib64/libpthread.so.0 [0x314880673d]
  0x31480d44bd: /lib64/libc.so.6(clone+0x6d) [0x31480d44bd]
sp = 0x2aaaece29008 {
  fd = 18, id = 18, xid = 1952412410,
  client = 10.50.0.77 58679,
  step = STP_FETCHBODY,
  handling = deliver,
  err_code = 200, err_reason = (null),
  restarts = 0, esi_level = 0
  flags =  do_gzip do_close is_gunzip
  bodystatus = 4
  ws = 0x2aaaece29080 { 
    id = "sess",
    {s,f,r,e} = {0x2aaaece29c90,+264,(nil),+65536},
  },
  http[req] = {
    ws = 0x2aaaece29080[sess]
      "GET",
      "/static/robots.txt",
      "HTTP/1.1",
      "User-Agent: check_http/v1.4.14 (nagios-plugins 1.4.14)",
      "Connection: close",
      "Host: fromnagiosrsc1",
      "X-Forwarded-For: 174.136.111.98",
  },
  worker = 0x7235dcf0 {
    ws = 0x7235df30 { 
      id = "wrk",
      {s,f,r,e} = {0x7234bca0,+520,(nil),+65536},
    },
    http[bereq] = {
      ws = 0x7235df30[wrk]
        "GET",
        "/static/robots.txt",
        "HTTP/1.1",
        "User-Agent: check_http/v1.4.14 (nagios-plugins 1.4.14)",
        "Host: fromnagiosrsc1",
        "X-Forwarded-For: 174.136.111.98",
        "X-Varnish: 1952412410",
        "Accept-Encoding: gzip",
    },
    http[beresp] = {
      ws = 0x7235df30[wrk]
        "HTTP/1.0",
        "200",
        "OK",
        "Date: Wed, 09 Nov 2011 10:12:43 GMT",
        "Server: Apache/1.3.37 (Unix) mod_perl/1.29",
        "Last-Modified: Tue, 08 Jun 2010 23:46:54 GMT",
        "Accept-Ranges: bytes",
        "Content-Length: 230",
        "Content-Type: text/plain",
        "Content-Encoding: gzip",
    },
    },
    vcl = {
      srcname = {
        "input",
        "Default",
      },
    },
  obj = 0x2aaab41dd000 {
    xid = 1952412410,
    ws = 0x2aaab41dd018 { 
      id = "obj",
      {s,f,r,e} = {0x2aaab41dd1e0,+216,(nil),+272},
    },
    http[obj] = {
      ws = 0x2aaab41dd018[obj]
        "HTTP/1.1",
        "OK",
        "Date: Wed, 09 Nov 2011 10:12:43 GMT",
        "Server: Apache/1.3.37 (Unix) mod_perl/1.29",
        "Last-Modified: Tue, 08 Jun 2010 23:46:54 GMT",
        "Content-Type: text/plain",
        "Content-Encoding: gzip",
    },
    len = 0,
    store = {
    },
  },
},


Last panic at: Wed, 09 Nov 2011 23:21:16 GMT
Assert error in VGZ_Ibuf(), cache_gzip.c line 222:
  Condition((vg->vz.avail_in) == 0) not true.
errno = 12 (Cannot allocate memory)
thread = (cache-worker)
ident = Linux,2.6.18-274.7.1.el5,x86_64,-sfile,-smalloc,-hcritbit,epoll
Backtrace:
  0x42c7a6: /usr/sbin/varnishd [0x42c7a6]
  0x4227da: /usr/sbin/varnishd(VGZ_Ibuf+0x7a) [0x4227da]
  0x422e19: /usr/sbin/varnishd [0x422e19]
  0x4215fd: /usr/sbin/varnishd(FetchBody+0x3fd) [0x4215fd]
  0x4153e8: /usr/sbin/varnishd [0x4153e8]
  0x417ab6: /usr/sbin/varnishd(CNT_Session+0x9f6) [0x417ab6]
  0x42efb8: /usr/sbin/varnishd [0x42efb8]
  0x42e19b: /usr/sbin/varnishd [0x42e19b]
  0x3b8420673d: /lib64/libpthread.so.0 [0x3b8420673d]
  0x3b83ad44bd: /lib64/libc.so.6(clone+0x6d) [0x3b83ad44bd]
sp = 0x2aaaecbc0008 {
  fd = 18, id = 18, xid = 556098001,
  client = 10.50.0.51 54247,
  step = STP_FETCHBODY,
  handling = deliver,
  err_code = 200, err_reason = (null),
  restarts = 0, esi_level = 0
  flags =  do_gzip do_close is_gunzip
  bodystatus = 4
  ws = 0x2aaaecbc0080 { 
    id = "sess",
    {s,f,r,e} = {0x2aaaecbc0c90,+232,(nil),+65536},
  },
  http[req] = {
    ws = 0x2aaaecbc0080[sess]
      "GET",
      "/static/robots.txt",
      "HTTP/1.1",
      "User-Agent: check_http/v1.4.15 (nagios-plugins 1.4.15)",
      "Connection: close",
      "Host: fromnagios:6083",
  },
  worker = 0x6f4a8cf0 {
    ws = 0x6f4a8f30 { 
      id = "wrk",
      {s,f,r,e} = {0x6f496ca0,+520,(nil),+65536},
    },
    http[bereq] = {
      ws = 0x6f4a8f30[wrk]
        "GET",
        "/static/robots.txt",
        "HTTP/1.1",
        "User-Agent: check_http/v1.4.15 (nagios-plugins 1.4.15)",
        "Host: fromnagios:6083",
        "X-Varnish: 556098001",
        "Accept-Encoding: gzip",
    },
    http[beresp] = {
      ws = 0x6f4a8f30[wrk]
        "HTTP/1.0",
        "200",
        "OK",
        "Date: Wed, 09 Nov 2011 23:21:16 GMT",
        "Server: Apache/1.3.37 (Unix) mod_perl/1.29",
        "Last-Modified: Tue, 08 Jun 2010 23:46:54 GMT",
        "Accept-Ranges: bytes",
        "Content-Length: 230",
        "Content-Type: text/plain",
        "Content-Encoding: gzip",
    },
    },
    vcl = {
      srcname = {
        "input",
        "Default",
      },
    },
  obj = 0x2aaaba292000 {
    xid = 556098001,
    ws = 0x2aaaba292018 { 
      id = "obj",
      {s,f,r,e} = {0x2aaaba2921e0,+216,(nil),+272},
    },
    http[obj] = {
      ws = 0x2aaaba292018[obj]
        "HTTP/1.1",
        "OK",
        "Date: Wed, 09 Nov 2011 23:21:16 GMT",
        "Server: Apache/1.3.37 (Unix) mod_perl/1.29",
        "Last-Modified: Tue, 08 Jun 2010 23:46:54 GMT",
        "Content-Type: text/plain",
        "Content-Encoding: gzip",
    },
    len = 0,
    store = {
    },
  },
},

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111110/d334c685/attachment.bin>

From ask at develooper.com  Thu Nov 10 19:03:46 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Thu, 10 Nov 2011 11:03:46 -0800
Subject: gzip crash
In-Reply-To: <20111110085225.GY13866@nerd.dk>
References: <BEB17889-F64A-4B86-90EA-50E9029C7453@develooper.com>
	<20111110085225.GY13866@nerd.dk>
Message-ID: <1B3575FD-8914-4F52-811E-2D80427378BE@develooper.com>


On Nov 10, 2011, at 0:52, Andreas Plesner Jacobsen wrote:

>> I noticed today that our varnish servers a few times have crashed with an
>> error like the ones below.  We're running 3.0.2 on CentOS/Linux x86_64.  Most
>> of the startup variables are the defaults from the rpm.
> 
> https://www.varnish-cache.org/trac/ticket/1036

Ah, indeed - that looks like the one.  It looks like it was fixed the same day 3.0.2 was built.  Thanks Andreas.


Ask

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111110/060976ed/attachment.bin>

From phk at phk.freebsd.dk  Sat Nov 12 09:12:09 2011
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 12 Nov 2011 09:12:09 +0000
Subject: gzip crash
In-Reply-To: Your message of "Thu, 10 Nov 2011 07:51:35 PST."
	<337049ED-ADA3-4F32-8E59-F8C113036EA8@develooper.com>
Message-ID: <33665.1321089129@critter.freebsd.dk>

In message <337049ED-ADA3-4F32-8E59-F8C113036EA8 at develooper.com>, =?iso-8859-1?
Q?Ask_Bj=F8rn_Hansen?= writes:

>Last panic at: Wed, 09 Nov 2011 10:12:43 GMT
>Assert error in VGZ_Ibuf(), cache_gzip.c line 222:
>  Condition((vg->vz.avail_in) == 0) not true.


I think I just fixed this one in -trunk.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From thierry.magnien at sfr.com  Tue Nov 15 10:53:21 2011
From: thierry.magnien at sfr.com (MAGNIEN, Thierry)
Date: Tue, 15 Nov 2011 11:53:21 +0100
Subject: TR: [PATCH proposal] RE: Backend DNS lookup refresh
In-Reply-To: <4A029B1A60B8E340A50D654D2F130DAA2FF45759CB@EXCV001.encara.local.ads>
References: <4A029B1A60B8E340A50D654D2F130DAA2FF45759CB@EXCV001.encara.local.ads>
Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FF45759D4@EXCV001.encara.local.ads>

And sending in the correct mailing-list is also not a bad idea ;-)

Sorry for duplicate,
Thierry

-----Message d'origine-----
De?: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] De la part de MAGNIEN, Thierry
Envoy??: mardi 15 novembre 2011 11:51
??: 'varnish-misc at varnish-cache.org' (varnish-misc at varnish-cache.org)
Objet?: [PATCH proposal] RE: Backend DNS lookup refresh

Hi,

Here is a first version of a patch to add the "vcl.reload" function to CLI, in order to refresh DNS lookups without having to send a new VCL or restart varnish.

It is far from perfect but may be used as a starting point. It applies on master and of course all comments are welcome.

Regards,
Thierry

-----Message d'origine-----
De?: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] De la part de MAGNIEN, Thierry
Envoy??: jeudi 3 novembre 2011 17:15
??: 'varnish-misc at varnish-cache.org' (varnish-misc at varnish-cache.org)
Objet?: Backend DNS lookup refresh

Hi all,

We are dealing with a great number of backends in our VCL file and we have problems with some backends changing IP address (for example when hosted on services such as Amazon S3, that switch IP address without telling anyone).

We would like to be able to refresh the DNS resolution of backends when needed and after a quick look at the source code, it seems there are several ways to handle this.

I would like to get your feeling about what would be the best solution before trying to write a patch, so here are my thoughts, just as it comes out of my mind ;-)

DNS resolution of backends is done when the VCL is loaded. IP address is stored in the compiled VCL as a sockaddr struct.

The first way to solve the problem would be to implement a "vcl reload" CLI function. In that case, this needs some tricks to be able to recompile a VCL which has the same name of the one used.

A 2nd way would be to make backend IP addresses resolved when needed (at first backend request and then cached for a "TTL" time), and not at VCL loading. However, this breaks most of the the optimization brought by the sockaddr structure.

A point discussed quickly with Per Buer on IRC was to have probes checking the IP addresses of the backends and update when an IP change is detected.

All this makes me think that it's not so easy as it could seem, and that's why I would accept any advice/idea/comment about this.

Regards,
Thierry


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vcl.reload.patch
Type: application/octet-stream
Size: 5198 bytes
Desc: vcl.reload.patch
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111115/c41f9049/attachment.obj>

From martin at varnish-software.com  Mon Nov 21 14:36:08 2011
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 21 Nov 2011 15:36:08 +0100
Subject: Proposed restructuring of http_conn and where the data is stored
Message-ID: <CANTn4cquE5c=kGokaObHiJ1AnBCFNAgmT1qScybSR+6bmhG+UA@mail.gmail.com>

For the streaming development, some changes will be needed to the http_conn
and where it stores it's data (buffers while reading headers and such, as
well as read-ahead and pipeline for the http protocol). Today the data is
stored on the session workspace (for the client communication) and on the
worker workspace for the backend communication.

For the streaming development this causes problems when we want to hand
over the body fetching to another worker, as there is read-ahead data in
the http_conn buffer that it needs access to, but this will then be
pointing into the workspace of the previous worker. I'd rather decouple
this, as it creates a strong relationship between the two threads and
troubles will come if they are not synchronized with regard to this address
space (e.g. if the client hangs up, the client thread needs to make sure
the body fetcher thread have finished with the data before it can reuse
it's workspace).

To come around this, I'm proposing to make the http_conn's a pooled
resource of their own, with their own internal buffer space. Something
along these lines:

   - Each thread pool have a list of unused http_conn's
   - Each worker thread have a pool of unused http_conn's. When the worker
   is idle (goes into pool or starts processing a new request), this pool is
   increased/reduced from the thread pool's list (or creating new ones) to a
   size of 2. Number of 2 as it will need 1 for the client request, and maybe
   one for the backend fetch.
   - Worker thread takes http_conns from it's pool when it needs a HTC (or
   creating a new one if it goes empty). It returns them to it's pool when
   they are not used anymore
   - http_conn's can then be transferred from one thread to another and
   take their data with them. (The receiving worker will then end up with one
   more, but this is returned to the thread pool's list when it's finished
   with the request. The thread giving one away gets one from the thread pool)
   - Whenever the house keeping is done on the worker's list, it will check
   the buffer sizes against the current parameter sizes, and free the
   http_conn's and creating new ones if they have changed.

I believe this creates a mostly lock free system, but still make these data
structures decoupled from the session/thread and can be transferred between
them when that is needed.

Any comments?

-- 
Martin Blix Grydeland
Varnish Software AS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111121/3b782174/attachment.html>

From phk at phk.freebsd.dk  Tue Nov 22 19:19:55 2011
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 22 Nov 2011 19:19:55 +0000
Subject: Proposed restructuring of http_conn and where the data is stored
In-Reply-To: Your message of "Tue, 22 Nov 2011 00:04:24 +0100."
	<20111121224046.C37583@ishtar.drwilco.net>
Message-ID: <2201.1321989595@critter.freebsd.dk>

In message <20111121224046.C37583 at ishtar.drwilco.net>, "Rogier R. Mulhuijzen" w
rites:

>What about decoupling the workspace instead (or maybe as well)? That way 
>the workspace can be released (back into a pool) at the end of a request, 

Yes, this is an aspect we should look at too.

If the session workspace were only alive during request processing, the
need for the worker workspace mostly disappear, at the cost of a larger
"request workspace"

There are some nasty corner cases with ESI & restarts to consider.

Not sure this isn't a 4.x thing.

Please make sure to put this in a Future_* wiki page.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From geoff at uplex.de  Wed Nov 23 16:35:08 2011
From: geoff at uplex.de (Geoff Simmons)
Date: Wed, 23 Nov 2011 17:35:08 +0100
Subject: [PATCH] Build fails on Solaris (include struct params)
Message-ID: <4ECD20BC.8080008@uplex.de>

Hello all,

A build for Solaris on the present trunk fails with the error shown
below, the problem being that an include is missing for the declaration
of struct params.

The attached one-liner makes it better.


Best,
Geoff


mgt/mgt_sandbox_solaris.c: In function `mgt_sandbox_solaris_privsep':
mgt/mgt_sandbox_solaris.c:161: error: invalid use of undefined type
`struct params'
mgt/mgt_sandbox_solaris.c:162: error: invalid use of undefined type
`struct params'
mgt/mgt_sandbox_solaris.c:163: error: invalid use of undefined type
`struct params'
mgt/mgt_sandbox_solaris.c:164: error: invalid use of undefined type
`struct params'

-- 
** * * UPLEX - Nils Goroll Systemoptimierung

Schwanenwik 24
22087 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: params.patch
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111123/807ab8f1/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 896 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111123/807ab8f1/attachment.bin>

From drwilco at drwilco.net  Mon Nov 21 23:04:24 2011
From: drwilco at drwilco.net (Rogier R. Mulhuijzen)
Date: Tue, 22 Nov 2011 00:04:24 +0100 (CET)
Subject: Proposed restructuring of http_conn and where the data is stored
In-Reply-To: <CANTn4cquE5c=kGokaObHiJ1AnBCFNAgmT1qScybSR+6bmhG+UA@mail.gmail.com>
References: <CANTn4cquE5c=kGokaObHiJ1AnBCFNAgmT1qScybSR+6bmhG+UA@mail.gmail.com>
Message-ID: <20111121224046.C37583@ishtar.drwilco.net>


What about decoupling the workspace instead (or maybe as well)? That way 
the workspace can be released (back into a pool) at the end of a request, 
and not eat up memory during idle-time for a session. And then the 
workspace can go with the http_conn to the next worker in your scenario.

This is from one of our servers:

       1 VCL_Recv
       3 Pipe
       3 Reading_Backend
       3 Waiting_List
      10 Connect_Backend
      28 Background
     223 Writing_Client
     225 Linger
     226 Reading_Backend_Hdr
   17274 Waiting_Client_Poll
  182004 Idle

With a 128K sess_workspace, that's 22 gigs in Idle sessions.

Now I know this is sess_workspace and the worker workspace is what you're 
after here, but if we decouple workspaces as a whole we can move them 
around for this and not waste extra memory on this problem.

My 2 cents, at least.

Cheers,

 	DocWilco

On Mon, 21 Nov 2011, Martin Blix Grydeland wrote:

> For the streaming development, some changes will be needed to the http_conn
> and where it stores it's data (buffers while reading headers and such, as
> well as read-ahead and pipeline for the http protocol). Today the data is
> stored on the session workspace (for the client communication) and on the
> worker workspace for the backend communication.
>
> For the streaming development this causes problems when we want to hand
> over the body fetching to another worker, as there is read-ahead data in
> the http_conn buffer that it needs access to, but this will then be
> pointing into the workspace of the previous worker. I'd rather decouple
> this, as it creates a strong relationship between the two threads and
> troubles will come if they are not synchronized with regard to this address
> space (e.g. if the client hangs up, the client thread needs to make sure
> the body fetcher thread have finished with the data before it can reuse
> it's workspace).
>
> To come around this, I'm proposing to make the http_conn's a pooled
> resource of their own, with their own internal buffer space. Something
> along these lines:
>
>   - Each thread pool have a list of unused http_conn's
>   - Each worker thread have a pool of unused http_conn's. When the worker
>   is idle (goes into pool or starts processing a new request), this pool is
>   increased/reduced from the thread pool's list (or creating new ones) to a
>   size of 2. Number of 2 as it will need 1 for the client request, and maybe
>   one for the backend fetch.
>   - Worker thread takes http_conns from it's pool when it needs a HTC (or
>   creating a new one if it goes empty). It returns them to it's pool when
>   they are not used anymore
>   - http_conn's can then be transferred from one thread to another and
>   take their data with them. (The receiving worker will then end up with one
>   more, but this is returned to the thread pool's list when it's finished
>   with the request. The thread giving one away gets one from the thread pool)
>   - Whenever the house keeping is done on the worker's list, it will check
>   the buffer sizes against the current parameter sizes, and free the
>   http_conn's and creating new ones if they have changed.
>
> I believe this creates a mostly lock free system, but still make these data
> structures decoupled from the session/thread and can be transferred between
> them when that is needed.
>
> Any comments?
>
> -- 
> Martin Blix Grydeland
> Varnish Software AS
>



From manju26m at googlemail.com  Wed Nov 30 10:27:40 2011
From: manju26m at googlemail.com (manju m)
Date: Wed, 30 Nov 2011 10:27:40 +0000
Subject: vcl_error function not getting executed
Message-ID: <CAMTD9-kaRZGsMCv_W5J1k4OQFsabhb8MTDU89Ub=TVRDm2GxKQ@mail.gmail.com>

Hi ,

I have a requirement for customizing 404 error page for varnish, and i
have done the changes.
But the changes are not getting picked up, i still get the error page
from the application , not the one which i have configured in varnish,
even after restarting varnish.

Request to advise what has gone wrong.

Many Thanks,
Manju
-------------- next part --------------
A non-text attachment was scrubbed...
Name: default.vcl
Type: application/octet-stream
Size: 4054 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20111130/ef865f21/attachment.obj>