From matthewm at boedicker.org  Sun Jun  2 06:28:09 2013
From: matthewm at boedicker.org (Matthew M. Boedicker)
Date: Sat, 1 Jun 2013 23:28:09 -0700
Subject: VMOD for sending messages over UDP
Message-ID: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>

I wrote a VMOD to send UDP messages from VCL:

https://github.com/mmb/vmod_dgram

I wanted something simple that could notify another process about urls
that Varnish has served.

Thanks for the excellent example VMOD.



From perbu at varnish-software.com  Mon Jun  3 07:36:19 2013
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 3 Jun 2013 09:36:19 +0200
Subject: VMOD for sending messages over UDP
In-Reply-To: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>
References: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>
Message-ID: <CAOXZevD4SnPvAqa6Dbkntk6EGQDf28zAwb9nxsQb7GTVi6A3ug@mail.gmail.com>

Great.

However, the VMODs lacks a license, which you stripped out from the example
vmod. It is sort of problematic to use one without a license. (insert rant
about how Github should force a distribution license here). u

Also, if you want it listed at varnish-cache.org/vmods please sign up and
register it there.

Per



On Sun, Jun 2, 2013 at 8:28 AM, Matthew M. Boedicker <matthewm at boedicker.org
> wrote:

> I wrote a VMOD to send UDP messages from VCL:
>
> https://github.com/mmb/vmod_dgram
>
> I wanted something simple that could notify another process about urls
> that Varnish has served.
>
> Thanks for the excellent example VMOD.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
 <http://www.varnish-software.com/> *Per Buer*
CEO | Varnish Software AS
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!

Winner of the Red Herring Top 100 Europe Award 2013
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130603/185a34d2/attachment.html>

From matthewm at boedicker.org  Tue Jun  4 04:25:24 2013
From: matthewm at boedicker.org (Matthew M. Boedicker)
Date: Mon, 3 Jun 2013 21:25:24 -0700
Subject: VMOD for sending messages over UDP
In-Reply-To: <CAOXZevD4SnPvAqa6Dbkntk6EGQDf28zAwb9nxsQb7GTVi6A3ug@mail.gmail.com>
References: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>
 <CAOXZevD4SnPvAqa6Dbkntk6EGQDf28zAwb9nxsQb7GTVi6A3ug@mail.gmail.com>
Message-ID: <CANfC_4JD-6yD2WUiX3r8m8aE2jfDrRuASSu5Bmq3hHrAxwBLmA@mail.gmail.com>

Thanks Per. I have re-added the BSD license and submitted it for inclusion
on the site.


On Mon, Jun 3, 2013 at 12:36 AM, Per Buer <perbu at varnish-software.com>wrote:

> Great.
>
> However, the VMODs lacks a license, which you stripped out from the
> example vmod. It is sort of problematic to use one without a license.
> (insert rant about how Github should force a distribution license here). u
>
> Also, if you want it listed at varnish-cache.org/vmods please sign up and
> register it there.
>
> Per
>
>
>
> On Sun, Jun 2, 2013 at 8:28 AM, Matthew M. Boedicker <
> matthewm at boedicker.org> wrote:
>
>> I wrote a VMOD to send UDP messages from VCL:
>>
>> https://github.com/mmb/vmod_dgram
>>
>> I wanted something simple that could notify another process about urls
>> that Varnish has served.
>>
>> Thanks for the excellent example VMOD.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
>
> --
>  <http://www.varnish-software.com/> *Per Buer*
> CEO | Varnish Software AS
> Phone: +47 958 39 117 | Skype: per.buer
> We Make Websites Fly!
>
> Winner of the Red Herring Top 100 Europe Award 2013
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130603/5cad3c79/attachment.html>

From straightflush at gmail.com  Wed Jun  5 13:33:03 2013
From: straightflush at gmail.com (AD)
Date: Wed, 5 Jun 2013 09:33:03 -0400
Subject: VMOD for sending messages over UDP
In-Reply-To: <CANfC_4JD-6yD2WUiX3r8m8aE2jfDrRuASSu5Bmq3hHrAxwBLmA@mail.gmail.com>
References: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>
 <CAOXZevD4SnPvAqa6Dbkntk6EGQDf28zAwb9nxsQb7GTVi6A3ug@mail.gmail.com>
 <CANfC_4JD-6yD2WUiX3r8m8aE2jfDrRuASSu5Bmq3hHrAxwBLmA@mail.gmail.com>
Message-ID: <CANZWjw4ZyBF9oKg1FDWZV09moVEF7UFp9nZA=ru+y5NHb0GDMQ@mail.gmail.com>

interesting, i wonder if you can somehow wire this up with a statsd server



On Tue, Jun 4, 2013 at 12:25 AM, Matthew M. Boedicker <
matthewm at boedicker.org> wrote:

> Thanks Per. I have re-added the BSD license and submitted it for inclusion
> on the site.
>
>
> On Mon, Jun 3, 2013 at 12:36 AM, Per Buer <perbu at varnish-software.com>wrote:
>
>> Great.
>>
>> However, the VMODs lacks a license, which you stripped out from the
>> example vmod. It is sort of problematic to use one without a license.
>> (insert rant about how Github should force a distribution license here). u
>>
>> Also, if you want it listed at varnish-cache.org/vmods please sign up
>> and register it there.
>>
>> Per
>>
>>
>>
>> On Sun, Jun 2, 2013 at 8:28 AM, Matthew M. Boedicker <
>> matthewm at boedicker.org> wrote:
>>
>>> I wrote a VMOD to send UDP messages from VCL:
>>>
>>> https://github.com/mmb/vmod_dgram
>>>
>>> I wanted something simple that could notify another process about urls
>>> that Varnish has served.
>>>
>>> Thanks for the excellent example VMOD.
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>>
>> --
>>  <http://www.varnish-software.com/> *Per Buer*
>> CEO | Varnish Software AS
>> Phone: +47 958 39 117 | Skype: per.buer
>> We Make Websites Fly!
>>
>> Winner of the Red Herring Top 100 Europe Award 2013
>>
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130605/8a27a34e/attachment.html>

From cosimo at streppone.it  Wed Jun  5 13:47:14 2013
From: cosimo at streppone.it (Cosimo Streppone)
Date: Wed, 05 Jun 2013 15:47:14 +0200
Subject: VMOD for sending messages over UDP
In-Reply-To: <CANZWjw4ZyBF9oKg1FDWZV09moVEF7UFp9nZA=ru+y5NHb0GDMQ@mail.gmail.com>
References: <CANfC_4J2sWFkY1g-fH5TPzBPvahMW4QkpN=4d-5PaSAjfEzhbA@mail.gmail.com>
 <CAOXZevD4SnPvAqa6Dbkntk6EGQDf28zAwb9nxsQb7GTVi6A3ug@mail.gmail.com>
 <CANfC_4JD-6yD2WUiX3r8m8aE2jfDrRuASSu5Bmq3hHrAxwBLmA@mail.gmail.com>
 <CANZWjw4ZyBF9oKg1FDWZV09moVEF7UFp9nZA=ru+y5NHb0GDMQ@mail.gmail.com>
Message-ID: <51AF4162.7060805@streppone.it>

On 06/05/2013 03:33 PM, AD wrote:

> [... vmod_dgram ...]
> interesting, i wonder if you can somehow wire this up with a statsd server

Maybe you should look at this one:

- https://www.varnish-cache.org/vmod/statsd-send-stats-varnish-statsd-0
- https://github.com/jib/libvmod-statsd

IIUC, it also caches the udp socket fd, which
may or may not be important, depending on the
volume of requests you're getting.

-- 
Cosimo




From jpotter-varnish at codepuppy.com  Thu Jun  6 01:16:39 2013
From: jpotter-varnish at codepuppy.com (Jeff Potter)
Date: Wed, 5 Jun 2013 21:16:39 -0400
Subject: varnish lock-up
Message-ID: <C66E32E7-BBC5-4DAE-A9B5-360D994D2D09@codepuppy.com>


Hi Varnish Folk,

We had an OOM condition today on a dev box that I thought was worth passing along, as it could've happened in production or may happen to others.

This is on Varnish 3.0.3 / Ubuntu 10.04.

It looks like the parent varnish process is not handling unexpected child exits correctly while inside a signal, leading to a situation where the parent process locks up because of a race condition where the signal handlers stomp on the global values of child_state and child_pid.

Here's a snippet of log -- the OOM killer kills the child processes each time that varnish attempts to run them (28004, 15403, 15640/-1).

[7270495.691960] Out of memory: kill process 27086 (varnishd) score 9908734 or a child
[7270495.691971] Killed process 28004 (varnishd)

1370466543.3  Child (28004) died signal=9
1370466543.3  Child cleanup complete
1370466543.4  child (15403) Started
1370466545.4  Child (15403) said Not running as root, no priv-sep
1370466545.4  Child (15403) said Child starts
1370466545.4  Child (15403) said SMF.s0 mmap'ed 268435456 bytes of 268435456

[7270511.916123] Out of memory: kill process 27086 (varnishd) score 9724109 or a child
[7270511.916134] Killed process 15403 (varnishd)

1370466558.8  Child (15403) not responding to CLI, killing it.
1370466559.6  Child (15403) died signal=9
1370466559.6  Child cleanup complete
1370466559.6  child (15640) Started

[7270520.798539] Out of memory: kill process 27086 (varnishd) score 6705406 or a child
[7270520.798551] Killed process 15640 (varnishd)

1370466568.1  Pushing vcls failed:
1370466568.1  CLI communication error (hdr)
1370466568.1  Stopping Child
1370466568.1  Child (15640) died signal=9

# (The -1 value is "interesting" given the check for >0; child_pid is a global and it's being set to -1 elsewhere -- so presumably start_child is being called from somewhere else; i.e. start_child is running, mgt_sigchild gets triggered, then flow returns to start_child right after and assigns -1 to child_pid)

1370466568.1  Child (-1) said Not running as root, no priv-sep
1370466568.1  Child (-1) said Child starts
1370466568.1  Child (-1) said SMF.s0 mmap'ed 268435456 bytes of 268435456
1370466568.1  Child cleanup complete


Reading through mgt_child.c, here's what I think is going on:

1) The signal handler is getting invoked when the child process is killed by the OOM killer
2) Then the parent process, inside the sig handler (mgt_sigchld), calls fork (inside mgt_sigchld's call to start_child), which causes a new child process to start
3) The OOM killer kills that second child process right away, before the sig handler has a chance to return

Does it maybe make sense to set the signal to be blocked for sigchild so that the events can be processed synchronously without losing them?

-Jeff




From dev at serviidbgroup.info  Tue Jun 11 22:18:00 2013
From: dev at serviidbgroup.info (Mark Strickland)
Date: Tue, 11 Jun 2013 18:18:00 -0400
Subject: help with cache misses Apache2 mod.fcgid with Varnish 3
Message-ID: <000901ce66f1$89e571b0$9db05510$@serviidbgroup.info>

I am running Apache2 mod.fcgid with Varnish 3 as a cache.  I am getting
around 50% cache misses.   I would appreciate any help or hints on what I am
doing wrong with my default.vcl.  

 

cid:image001.png at 01CE654D.EC949250

 

Here is my default.vcl 

 

 

# This is a basic VCL configuration file for varnish.  See the vcl(7)

# man page for details on VCL syntax and semantics.

#

 

# TODO: Update internal subnet ACL and security.

 

# Define the internal network subnet.

# These are used below to allow internal access to certain files while not

# allowing access from the public internet.

# acl internal {

#  "192.10.0.0"/24;

# }

 

# Default backend definition.  Set this to point to your content

# server.

#

backend default {

.host = "127.0.0.1";

.port = "8080";

.connect_timeout = 600s;

.first_byte_timeout = 600s;

.between_bytes_timeout = 600s;

.max_connections = 800;

}

 

acl purge {

  "localhost";

  "127.0.0.1";

}

 

 

# Respond to incoming requests.

sub vcl_recv {

                # Use anonymous, cached pages if all backends are down.

  if (!req.backend.healthy) {

    unset req.http.Cookie;

  }

  

  # Allow the backend to serve up stale content if it is responding slowly.

  set req.grace = 6h;

                

  if (req.restarts == 0) {

  if (req.http.X-Forwarded-For) {

    set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
client.ip;

  } else {

    set req.http.X-Forwarded-For = client.ip;

  }

}

   

# Check the incoming request type is "PURGE", not "GET" or "POST"

if (req.request == "PURGE") {

  # Check if the ip coresponds with the acl purge

  if (!client.ip ~ purge) {

  # Return error code 405 (Forbidden) when not

    error 405 "Not allowed.";

  }

  return (lookup);

}

   

  # Pipe these paths directly to Apache for streaming.

  #if (req.url ~ "^/admin/content/backup_migrate/export") {

  #  return (pipe);

  #}

  #req.url ~ "^.*/api/.*$" ||

  # Do not cache these paths.

  if (req.url ~ "^/status\.php$" ||

                 req.url ~ "^/update\.php$" ||

                req.url ~ "^/admin$" ||

                req.url ~ "^/admin/.*$" ||

req.url ~ "^.*/apidev/.*$" ||

                req.url ~ "^/flag/.*$" ||

                req.url ~ "^.*/munin/.*$" ||

                  req.url ~ "^.*/server-status/.*$" ||

                  req.url ~ "^.*/store_us/.*$" ||

                  req.url ~ "^.*/store_us_test/.*$" ||

                  req.url ~ "^.*/store_ca/.*$" ||

                  req.url ~ "^.*/store_uk/.*$" ||

                  req.url ~ "^.*/store_de/.*$" ||

                  req.url ~ "^.*/store_fr/.*$" ||

                  req.url ~ "^.*/store_it/.*$" ||

                  req.url ~ "^.*/store_cn/.*$" ||

                  req.url ~ "^.*/store_es/.*$" ||

                  req.url ~ "^.*/store_jp/.*$" ||

                 req.url ~ "^.*/ajax/.*$" ||

                  req.url ~ "^.*/dev.serviidb.com/.*$" ||

                  req.url ~ "^.*/user/.*$" ||

                  req.url ~ "^.*/cacti/.*$" ||

                  req.url ~ "^.*/admin/.*$" ||

                  req.url ~ "^.*/install/.*$" ||

                  req.url ~ "^.*/serviioweb/.*$" ||

                 req.url ~ "^.*/mediabrowser/.*$" ||

                 req.url ~ "^.*/phpmyadmin/.*$" ||

                  req.url ~ "^.*/xataface/.*$") {

       return (pass);

  }

 

  # Do not allow outside access to cron.php or install.php.

  #if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {

    # Have Varnish throw the error directly.

  #  error 404 "Page not found.";

    # Use a custom error page that you've defined in Drupal at the path
"404".

    # set req.url = "/404";

  #}

 

  # Always cache the following file types for all users. This list of
extensions

  # appears twice, once here and again in vcl_fetch so make sure you edit
both

  # and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset req.http.Cookie;

  }

 

 # Remove all cookies that Drupal doesn't need to know about. We explicitly 

  # list the ones that Drupal does need, the SESS and NO_CACHE. If, after 

  # running this code we find that either of these two cookies remains, we 

  # will pass as the page cannot be cached.

  if (req.http.Cookie) {

    # 1. Append a semi-colon to the front of the cookie string.

    # 2. Remove all spaces that appear after semi-colons.

    # 3. Match the cookies we want to keep, adding the space we removed 

    #    previously back. (\1) is first matching group in the regsuball.

    # 4. Remove all other cookies, identifying them by the fact that they
have

    #    no space after the preceding semi-colon.

    # 5. Remove all spaces and semi-colons from the beginning and end of the


    #    cookie string. 

    set req.http.Cookie = ";" + req.http.Cookie;

    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");    

    set req.http.Cookie = regsuball(req.http.Cookie,
";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");

    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");

    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

 

    if (req.http.Cookie == "") {

      # If there are no remaining cookies, remove the cookie header. If
there

      # aren't any cookie headers, Varnish's default behavior will be to
cache

      # the page.

      unset req.http.Cookie;

    }

    else {

      # If there is any cookies left (a session or NO_CACHE cookie), do not

      # cache the page. Pass it on to Apache directly.

      return (pass);

    }

  }

 

}

 

# Set a header to track a cache HIT/MISS.

sub vcl_deliver {

  if (obj.hits > 0) {

    set resp.http.X-Varnish-Cache = "HIT";

  }

  else {

    set resp.http.X-Varnish-Cache = "MISS";

  }

}

 

# Code determining what to do when serving items from the Apache servers.

# beresp == Back-end response from the web server.

sub vcl_fetch {

  # We need this to cache 404s, 301s, 500s. Otherwise, depending on backend
but 

  # definitely in Drupal's case these responses are not cacheable by
default.

  if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500)
{

    set beresp.ttl = 10m;

  }

  # this section of code sets the API to return a response code of 200 not
302.  DO NOT MODIFY OR DELETE COULD MESS UP CLIENTS.

    if(beresp.status == 302 && !beresp.http.Location){

    set beresp.status = 200; 

    set beresp.response = "OK";

  }

 

  # Don't allow static files to set cookies. 

  # (?i) denotes case insensitive in PCRE (perl compatible regular
expressions).

  # This list of extensions appears twice, once here and again in vcl_recv
so 

  # make sure you edit both and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset beresp.http.set-cookie;

  }

 

  # Allow items to be stale if needed.

  set beresp.grace = 6h;

}

 

sub vcl_hit {

        if (req.request == "PURGE") {

                purge;

                error 200 "Purged.";

        }

}

 

sub vcl_miss {

        if (req.request == "PURGE") {

                purge;

                error 200 "Purged.";

        }

}

 

# In the event of an error, show friendlier messages.

sub vcl_error {

  # Redirect to some other URL in the case of a homepage failure.

  #if (req.url ~ "^/?$") {

  #  set obj.status = 302;

  #  set obj.http.Location = "http://backup.example.com/";

  #}

 

  # Otherwise redirect to the homepage, which will likely be in the cache.

  set obj.http.Content-Type = "text/html; charset=utf-8";

  synthetic {"

<html>

<head>

  <title>Page Unavailable</title>

  <style>

    body { background: #303030; text-align: center; color: white; }

    #page { border: 1px solid #CCC; width: 500px; margin: 100px auto 0;
padding: 30px; background: #323232; }

    a, a:link, a:visited { color: #CCC; }

    .error { color: #222; }

  </style>

</head>

<body onload="setTimeout(function() { window.location = '/' }, 5000)">

  <div id="page">

    <h1 class="title">Page Unavailable</h1>

    <p>The page you requested is temporarily unavailable.</p>

    <p>We're redirecting you to the <a href="/">homepage</a> in 5
seconds.</p>

    <div class="error">(Error "} + obj.status + " " + obj.response +
{")</div>

  </div>

</body>

</html>

"};

  return (deliver);

}

 

 

Requests.log

 

/ hit

/ hit

/api/media.json?client=ServiiDroid&num=0 miss

/api/plugin.json?client=ServiiDroid&num=0 miss

/sites/default/files/hulu.png miss

/sites/default/files/comedycentral.jpg miss

/sites/default/files/CBS_0.jpg miss

/sites/default/files/abc_iview.jpg miss

/sites/default/files/4OD-logo.jpg miss

/ hit

/ hit

/sl/taxonomy/term/116/all miss

/ hit

/ru/node/317?language=de miss

/ hit

/ hit

/ hit

/sl/taxonomy/term/130/all miss

/plugin-api-name/youtubetop100 miss

/ hit

/ hit

/da/content/abc-iview-australia?language=en miss

/ hit

/sl/taxonomy/term/130/all miss

/ru/content/haha-sport-baseball?language=de miss

/ hit

/ hit

/ hit

/?q=node/add miss

/plugin-api-name/youtubetop100 miss

/node/add miss

/ hit

/user/register miss

/user/register miss

/ hit

/ja/comment/7?language=ja miss

/sites/default/files/itv.jpg miss

/sites/default/files/syfy.jpg miss

/sl/taxonomy/term/130/all miss

/sites/default/files/coco.jpg miss

/sites/default/files/css/css_fUXc0IbfJFVGhZ_CC63oYAw5rSt49epRhzcGmscaISU.css
hit

/sites/default/files/css/css_SKdZ0BB5XafXDbZJFEkdKM2wno6BIT2kaNm5ZvTeTgA.css
miss

/sites/default/files/css/css_KhS183x7gviqG0CuFRJJAOmVB1hHNwf9ofpIxQ9WMmE.css
hit

/sites/default/files/eclp_3.png hit

/sites/all/modules/addtoany/images/share_save_171_16.png hit

/sites/default/files/css/css_5_EBsvYf_U3gqgv1Idal1sImw7zcXB1kaIR6NOugp8o.css
hit

/sites/default/files/bbciplayer.png miss

/misc/feed.png hit

/sites/default/files/castalba.jpg miss

/sites/default/files/skysportsplus.png miss

/sites/default/files/hahasport.png miss

/sites/default/files/coolsporttv.png miss

/ hit

/ hit

/sites/default/files/sopcast.jpg miss

/sites/default/files/stopstream_0.png miss

/ar/ar/content/data/view/api_access?client_op=contains&client=&page=16 miss

/sites/default/files/dailymotion.jpg miss

/sites/default/files/twitch.png miss

/sites/default/files/Ustream.jpg miss

/sites/default/files/vimeo.jpg miss

/sites/default/files/YouTube.jpg miss

/sites/default/files/canalplus.jpg miss

/sites/default/files/m6groupe.png miss

/plugin-api-name/youtubetop100 miss

/ hit

/ hit

/zh-hans/node/199?language=en miss

/zh-hans/node/863?language=en miss

/api/install?client=ServiiDroid&nid=175.0 miss

/ hit

/sl/content/pilgrims-death miss

/content/how-cook-heston?language=ru miss

/ hit

/plugin-api-name/youtubetop100 miss

/sl/node/239 miss

/da/edit-plugin miss

/ hit

/ hit

/ hit

/plugin-api-name/youtubetop100 miss

/api/install?client=ServiiDroid&nid=771.0 miss

/ hit

/ hit

/ hit

/es/media-type/video?language=en miss

/es/media-type/video?language=en miss

/ hit

/plugin-api-name/youtubetop100 miss

/unpublished-forum miss

/ hit

/robots.txt hit

/ hit

/unpublished-forum miss

/content/pokemon miss

/content/conan-celebrity-interviews?language=es miss

/sl/region/uk?language=en miss

/unpublished-forum miss

/batch?op=start&id=1516 miss

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130611/96b69a72/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 22305 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130611/96b69a72/attachment.png>

From marco at nucleus.it  Wed Jun 12 12:49:24 2013
From: marco at nucleus.it (Marco Felettigh)
Date: Wed, 12 Jun 2013 14:49:24 +0200
Subject: Questiona about req.backend.healthy
Message-ID: <20130612144924.55b3e0cc@lobo.lobo.dom>

Hi ,
i'am new about Varnish and i need help understanding
req.backend.healthy .

req.backend.healthy is a state for all the backends or i can check the
healthy of a single backend ?
How can i grace only the content from a single backend ?

site1 -> be1
site2 -> be2

example:
backend be1 {
   .host = "x.x.x.x";
   .port = "80";
   .probe = {
   .url = "/index.php";
   .interval = 5s;
   .timeout = 1 s;
   .window = 5;
   .threshold = 3;
  }
}

backend be2 {
   .host = "y.y.y.y";
   .port = "80";
   .probe = {
   .url = "/index.php";
   .interval = 5s;
   .timeout = 1 s;
   .window = 5;
   .threshold = 3;
  }
}


thanks
Marco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130612/8a244eee/attachment.bin>

From jpotter-varnish at codepuppy.com  Wed Jun 12 13:02:03 2013
From: jpotter-varnish at codepuppy.com (Jeff Potter)
Date: Wed, 12 Jun 2013 09:02:03 -0400
Subject: varnish lock-up
In-Reply-To: <C66E32E7-BBC5-4DAE-A9B5-360D994D2D09@codepuppy.com>
References: <C66E32E7-BBC5-4DAE-A9B5-360D994D2D09@codepuppy.com>
Message-ID: <D9C9B337-1E41-4C3C-BB9D-F1B4D94194BC@codepuppy.com>


Hi --

Haven't heard anything on this, so thought I'd try to post it into a bug tracking system. However, I can't seem to figure out where that'd be -- https://www.varnish-cache.org/trac ?

Thanks,
Jeff

On Jun 5, 2013, at 9:16 PM, Jeff Potter <jpotter-varnish at codepuppy.com> wrote:

> 
> Hi Varnish Folk,
> 
> We had an OOM condition today on a dev box that I thought was worth passing along, as it could've happened in production or may happen to others.
> 
> This is on Varnish 3.0.3 / Ubuntu 10.04.
> 
> It looks like the parent varnish process is not handling unexpected child exits correctly while inside a signal, leading to a situation where the parent process locks up because of a race condition where the signal handlers stomp on the global values of child_state and child_pid.
> 
> Here's a snippet of log -- the OOM killer kills the child processes each time that varnish attempts to run them (28004, 15403, 15640/-1).
> 
> [7270495.691960] Out of memory: kill process 27086 (varnishd) score 9908734 or a child
> [7270495.691971] Killed process 28004 (varnishd)
> 
> 1370466543.3  Child (28004) died signal=9
> 1370466543.3  Child cleanup complete
> 1370466543.4  child (15403) Started
> 1370466545.4  Child (15403) said Not running as root, no priv-sep
> 1370466545.4  Child (15403) said Child starts
> 1370466545.4  Child (15403) said SMF.s0 mmap'ed 268435456 bytes of 268435456
> 
> [7270511.916123] Out of memory: kill process 27086 (varnishd) score 9724109 or a child
> [7270511.916134] Killed process 15403 (varnishd)
> 
> 1370466558.8  Child (15403) not responding to CLI, killing it.
> 1370466559.6  Child (15403) died signal=9
> 1370466559.6  Child cleanup complete
> 1370466559.6  child (15640) Started
> 
> [7270520.798539] Out of memory: kill process 27086 (varnishd) score 6705406 or a child
> [7270520.798551] Killed process 15640 (varnishd)
> 
> 1370466568.1  Pushing vcls failed:
> 1370466568.1  CLI communication error (hdr)
> 1370466568.1  Stopping Child
> 1370466568.1  Child (15640) died signal=9
> 
> # (The -1 value is "interesting" given the check for >0; child_pid is a global and it's being set to -1 elsewhere -- so presumably start_child is being called from somewhere else; i.e. start_child is running, mgt_sigchild gets triggered, then flow returns to start_child right after and assigns -1 to child_pid)
> 
> 1370466568.1  Child (-1) said Not running as root, no priv-sep
> 1370466568.1  Child (-1) said Child starts
> 1370466568.1  Child (-1) said SMF.s0 mmap'ed 268435456 bytes of 268435456
> 1370466568.1  Child cleanup complete
> 
> 
> Reading through mgt_child.c, here's what I think is going on:
> 
> 1) The signal handler is getting invoked when the child process is killed by the OOM killer
> 2) Then the parent process, inside the sig handler (mgt_sigchld), calls fork (inside mgt_sigchld's call to start_child), which causes a new child process to start
> 3) The OOM killer kills that second child process right away, before the sig handler has a chance to return
> 
> Does it maybe make sense to set the signal to be blocked for sigchild so that the events can be processed synchronously without losing them?
> 
> -Jeff
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From Raul.Rangel at disney.com  Wed Jun 12 14:15:11 2013
From: Raul.Rangel at disney.com (Rangel, Raul)
Date: Wed, 12 Jun 2013 07:15:11 -0700
Subject: Questiona about req.backend.healthy
In-Reply-To: <20130612144924.55b3e0cc@lobo.lobo.dom>
References: <20130612144924.55b3e0cc@lobo.lobo.dom>
Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F261BD87C69@SM-CALA-VXMB04A.swna.wdpr.disney.com>

req.backend is the default backend you have selected. If you want your request to go to a specific backend just
set req.backend = be1;

You can reference each individual backend by name. So be1.healthy, or be2.healthy.

Raul

-----Original Message-----
From: varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org [mailto:varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org] On Behalf Of Marco Felettigh
Sent: Wednesday, June 12, 2013 6:49 AM
To: varnish-misc at varnish-cache.org
Subject: Questiona about req.backend.healthy

Hi ,
i'am new about Varnish and i need help understanding req.backend.healthy .

req.backend.healthy is a state for all the backends or i can check the healthy of a single backend ?
How can i grace only the content from a single backend ?

site1 -> be1
site2 -> be2

example:
backend be1 {
   .host = "x.x.x.x";
   .port = "80";
   .probe = {
   .url = "/index.php";
   .interval = 5s;
   .timeout = 1 s;
   .window = 5;
   .threshold = 3;
  }
}

backend be2 {
   .host = "y.y.y.y";
   .port = "80";
   .probe = {
   .url = "/index.php";
   .interval = 5s;
   .timeout = 1 s;
   .window = 5;
   .threshold = 3;
  }
}


thanks
Marco



From marco at nucleus.it  Wed Jun 12 14:40:23 2013
From: marco at nucleus.it (Marco Felettigh)
Date: Wed, 12 Jun 2013 16:40:23 +0200
Subject: Questiona about req.backend.healthy
In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F261BD87C69@SM-CALA-VXMB04A.swna.wdpr.disney.com>
References: <20130612144924.55b3e0cc@lobo.lobo.dom>
 <2465AAEEC8B8A242B26ED5F44BCA805F261BD87C69@SM-CALA-VXMB04A.swna.wdpr.disney.com>
Message-ID: <20130612164023.09b3475f@lobo.lobo.dom>

Ok thanks but if i write 

sub vcl_recv {
    set req.backend = be1;
    if (be1.healthy) {
    set req.grace = 30s;
    std.log("###################### Grace 30s");
  } else {
    std.log("###################### Grace 1h");
    set req.grace = 1h;
  }	
...

and restart Varnish the compiler fails.

Message from VCC-compiler:
Symbol not found: 'be1.healthy' (expected type BOOL):
('input' Line 125 Pos 9)
    if (be1.healthy) {
--------###############---

Running VCC-compiler failed, exit 1

I'm missing something ?

Thanks


On Wed, 12 Jun 2013 07:15:11 -0700
"Rangel, Raul" <Raul.Rangel at disney.com> wrote:

> req.backend is the default backend you have selected. If you want
> your request to go to a specific backend just set req.backend = be1;
> 
> You can reference each individual backend by name. So be1.healthy, or
> be2.healthy.
> 
> Raul
> 
> -----Original Message-----
> From: varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org
> [mailto:varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org]
> On Behalf Of Marco Felettigh Sent: Wednesday, June 12, 2013 6:49 AM
> To: varnish-misc at varnish-cache.org Subject: Questiona about
> req.backend.healthy
> 
> Hi ,
> i'am new about Varnish and i need help understanding
> req.backend.healthy .
> 
> req.backend.healthy is a state for all the backends or i can check
> the healthy of a single backend ? How can i grace only the content
> from a single backend ?
> 
> site1 -> be1
> site2 -> be2
> 
> example:
> backend be1 {
>    .host = "x.x.x.x";
>    .port = "80";
>    .probe = {
>    .url = "/index.php";
>    .interval = 5s;
>    .timeout = 1 s;
>    .window = 5;
>    .threshold = 3;
>   }
> }
> 
> backend be2 {
>    .host = "y.y.y.y";
>    .port = "80";
>    .probe = {
>    .url = "/index.php";
>    .interval = 5s;
>    .timeout = 1 s;
>    .window = 5;
>    .threshold = 3;
>   }
> }
> 
> 
> thanks
> Marco
> 



-- 
----
  Cordiali saluti
    Marco Felettigh
    Nucleus S.r.l.
    Tel. 0432526336
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130612/1bf5b321/attachment.bin>

From Raul.Rangel at disney.com  Wed Jun 12 14:45:01 2013
From: Raul.Rangel at disney.com (Rangel, Raul)
Date: Wed, 12 Jun 2013 07:45:01 -0700
Subject: Questiona about req.backend.healthy
In-Reply-To: <20130612164023.09b3475f@lobo.lobo.dom>
References: <20130612144924.55b3e0cc@lobo.lobo.dom>
 <2465AAEEC8B8A242B26ED5F44BCA805F261BD87C69@SM-CALA-VXMB04A.swna.wdpr.disney.com>
 <20130612164023.09b3475f@lobo.lobo.dom>
Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F261BD87C85@SM-CALA-VXMB04A.swna.wdpr.disney.com>

Oops,
I guess you can't reference the backend like I said.

In that case just check req.backend.healthy because you already assigned the backend to the request.

Raul

-----Original Message-----
From: Marco Felettigh [mailto:marco at nucleus.it] 
Sent: Wednesday, June 12, 2013 8:40 AM
To: Rangel, Raul
Cc: 'varnish-misc at varnish-cache.org'
Subject: Re: Questiona about req.backend.healthy

Ok thanks but if i write 

sub vcl_recv {
    set req.backend = be1;
    if (be1.healthy) {
    set req.grace = 30s;
    std.log("###################### Grace 30s");
  } else {
    std.log("###################### Grace 1h");
    set req.grace = 1h;
  }	
...

and restart Varnish the compiler fails.

Message from VCC-compiler:
Symbol not found: 'be1.healthy' (expected type BOOL):
('input' Line 125 Pos 9)
    if (be1.healthy) {
--------###############---

Running VCC-compiler failed, exit 1

I'm missing something ?

Thanks


On Wed, 12 Jun 2013 07:15:11 -0700
"Rangel, Raul" <Raul.Rangel at disney.com> wrote:

> req.backend is the default backend you have selected. If you want your 
> request to go to a specific backend just set req.backend = be1;
> 
> You can reference each individual backend by name. So be1.healthy, or 
> be2.healthy.
> 
> Raul
> 
> -----Original Message-----
> From: varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org
> [mailto:varnish-misc-bounces+raul.rangel=disney.com at varnish-cache.org]
> On Behalf Of Marco Felettigh Sent: Wednesday, June 12, 2013 6:49 AM
> To: varnish-misc at varnish-cache.org Subject: Questiona about 
> req.backend.healthy
> 
> Hi ,
> i'am new about Varnish and i need help understanding 
> req.backend.healthy .
> 
> req.backend.healthy is a state for all the backends or i can check the 
> healthy of a single backend ? How can i grace only the content from a 
> single backend ?
> 
> site1 -> be1
> site2 -> be2
> 
> example:
> backend be1 {
>    .host = "x.x.x.x";
>    .port = "80";
>    .probe = {
>    .url = "/index.php";
>    .interval = 5s;
>    .timeout = 1 s;
>    .window = 5;
>    .threshold = 3;
>   }
> }
> 
> backend be2 {
>    .host = "y.y.y.y";
>    .port = "80";
>    .probe = {
>    .url = "/index.php";
>    .interval = 5s;
>    .timeout = 1 s;
>    .window = 5;
>    .threshold = 3;
>   }
> }
> 
> 
> thanks
> Marco
> 



--
----
  Cordiali saluti
    Marco Felettigh
    Nucleus S.r.l.
    Tel. 0432526336



From jdmls at yahoo.com  Wed Jun 12 16:08:35 2013
From: jdmls at yahoo.com (John Doe)
Date: Wed, 12 Jun 2013 09:08:35 -0700 (PDT)
Subject: Timeout with large files...
Message-ID: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>

Hi,

I have a little issue with large files time outing front end while varnish caches the file sent by the backend.
Is there a way to tell varnish to start sending the beginning of the file right away?
Or is there a way to directly "pipe" only for files with a size > n?

Thx,
JD



From james at ifixit.com  Wed Jun 12 21:48:23 2013
From: james at ifixit.com (James Pearson)
Date: Wed, 12 Jun 2013 14:48:23 -0700
Subject: Timeout with large files...
In-Reply-To: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
Message-ID: <1371073562-sup-1527@geror.local>

Excerpts from John Doe's message of 2013-06-12 09:08:35 -0700:
> Hi,
> 
> I have a little issue with large files time outing front end while varnish caches the file sent by the backend.
> Is there a way to tell varnish to start sending the beginning of the file right away?
> Or is there a way to directly "pipe" only for files with a size > n?

There is a varnish-streaming build that was released some time ago[0].  We
played around with it a bit on our development server because of issues we'd
seen with large files.  There was some major issue that prevented us from
pushing it to production, but I'm afraid I can't remember what it was.  It's
fairly likely it was something particular to our setup.

HTH,
 - P

[0]: https://www.varnish-cache.org/lists/pipermail/varnish-announce/2011-November/000677.html



From jdmls at yahoo.com  Thu Jun 13 08:24:33 2013
From: jdmls at yahoo.com (John Doe)
Date: Thu, 13 Jun 2013 01:24:33 -0700 (PDT)
Subject: Timeout with large files...
In-Reply-To: <4E4FA8CEA7D2CA46B10FAAE3F58E8422DB06EB47CD@HOLANDAMBX.elmercurio.org>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
 <4E4FA8CEA7D2CA46B10FAAE3F58E8422DB06EB47CD@HOLANDAMBX.elmercurio.org>
Message-ID: <1371111873.13560.YahooMailNeo@web121602.mail.ne1.yahoo.com>

From: "juan.salinas at mercurio.cl" <juan.salinas at mercurio.cl>

> I think this link might help you...if you are using a version > 3.0
> https://www.varnish-software.com/blog/streaming-varnish-30

Cool, thx for the link.
Any idea how it handles client disconnection at the first download/caching?
Does it continue to cache the whole file even though the client closed the connection?? or does it give up?

Thx,
JD



From james at ifixit.com  Thu Jun 13 17:47:22 2013
From: james at ifixit.com (James Pearson)
Date: Thu, 13 Jun 2013 10:47:22 -0700
Subject: Timeout with large files...
In-Reply-To: <1371111873.13560.YahooMailNeo@web121602.mail.ne1.yahoo.com>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
 <4E4FA8CEA7D2CA46B10FAAE3F58E8422DB06EB47CD@HOLANDAMBX.elmercurio.org>
 <1371111873.13560.YahooMailNeo@web121602.mail.ne1.yahoo.com>
Message-ID: <1371145598-sup-6981@geror.local>

Excerpts from John Doe's message of 2013-06-13 01:24:33 -0700:
> From: "juan.salinas at mercurio.cl" <juan.salinas at mercurio.cl>
> 
> > I think this link might help you...if you are using a version > 3.0
> > https://www.varnish-software.com/blog/streaming-varnish-30
> 
> Cool, thx for the link.
> Any idea how it handles client disconnection at the first download/caching?
> Does it continue to cache the whole file even though the client closed the connection?? or does it give up?

I don't know terribly much personally.  There is some more information,
including a whitepaper, on the website:
https://www.varnish-software.com/blog/http-streaming-varnish

 - P



From nwarren at nazarene.org  Thu Jun 13 17:56:27 2013
From: nwarren at nazarene.org (Nathan Warren)
Date: Thu, 13 Jun 2013 12:56:27 -0500
Subject: varnish using dropped backend connections?
Message-ID: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>

Good day,

I have a varnish cache serving requests to two play framework instances with a cisco asa firewall in-between. Everything works great at first - cache is working, acceptable cache hit ratio, fast, etc. But then after about an hour (without hits to the service), when the cache misses and pulls from the backend, there is no reply - the pound server times out while varnish hangs. Eventually varnish gives a 503 meditation, but pound doesn't wait that long to time out. Packet traces on the varnish box show packets being sent from varnish, but packet traces never show it reaching the backend server. The health checks keep polling successfully throughout.

My suspicion was that varnish was creating a keepalive session to the play server that was persistent. This seems to be the case with tcp sessions visible (via netstat) on both sides when no traffic is passing through. It seems as though the firewall was dropping the keepalive session, but varnish kept using it as if it were there. Then it kept re-using it over and over. After refreshing the page a few times (in error), the page will start working.

I found that cisco asa firewalls drop idle sessions after one hour, and that my version of linux has a 7200 time to wait before interrogating a possibly failed tcp session. I dropped this to 1800 via /proc/sys/net/ipv4/tcp_keepalive_time and restarted varnish. After waiting for an hour, the problem continues as before.

I would appreciate any insight you can provide, even if it's a problem with my config that doesn't seem to be related to this problem.

Thanks,
Nate

####  default.vcl  ####
backend backend11 {
        .host = "10.1.1.20";
        .port = "9010";
        .probe = {
                .url = "/sys-status";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

backend backend12 {
        .host = "10.1.1.21";
        .port = "9010";
        .probe = {
                .url = "/sys-status";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director backend1 random {
        .retries = 5;
        {
                .backend = backend11;
                .weight = 3;
        }
        {
                .backend = backend12;
                .weight = 3;
        }
}

backend backend21 {
        .host = "10.1.1.20";
        .port = "9003";
        .probe = {
                .url = "/ping";
                .interval = 10s;
                .timeout = 2s;
                .window = 5;
                .threshold = 3;
        }
}

backend backend22 {
        .host = "10.1.1.21";
        .port = "9003";
        .probe = {
                .url = "/ping";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director backend2 random {
        {
                .backend = backend11;
                .weight = 3;
        }
        {
                .backend = backend12;
                .weight = 3;
        }
}

sub vcl_recv {

        #serve stale data is ok
        set req.grace = 55m;

        #we don't care about any cookies
        unset req.http.Cookie;

        # secure redirects
        if (req.http.host ~ "^server1.nazarene.org$"
                        && req.http.X-Forwarded-Proto !~ "(?i)https" ) {
                set req.http.x-Redir-Url = "https://" + req.http.host + req.url;
                error 750 req.http.x-Redir-Url;
        }

        if (req.url ~ "(noaccess|notallowed)" ) {
                error 404 "Not Found";
        }

        # send requests to server1 to go to backend1
        if (req.http.host ~ "^server1.domain.local$") {
                set req.backend = backend1;

                #if the request is going to /foo/p/blah/blah then we can't cache, so don't try
                if (req.url ~ "^/foo/p/.*") {
                        return (pipe);
                }

                return (lookup);
        }

        # send requests to server2.domain.local
        if ( req.http.host ~ "^server2.domain.local$" ) {
                # but only for certain urls
                if ( req.url ~ "^/foo/(bar|foobar)" ) {
                        set req.backend = backend2;
                        return (lookup);
                }
        }

        # If you're not served at this point, I don't have anything for you
        error 404 "Not Found";
}

sub vcl_fetch {
        set beresp.grace = 60m;

        #strip cookies from backend so varnish will cache
        unset beresp.http.Set-Cookie;

        # Properly handle different encoding types
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

        if (beresp.http.content-type ~ "text") {
          set beresp.do_gzip = true;
        }
}

sub vcl_error {
        #redirect the client to the specified location
        if (obj.status == 750) {
                set obj.http.Location = obj.response;
                set obj.status = 302;
                return (deliver);
        }
}

#####  END default.vcl  ######

user at host:~$ sudo varnishstat -1 -n varnish
client_conn                 21         0.00 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req                 146         0.02 Client requests received
cache_hit                  111         0.01 Cache hits
cache_hitpass                0         0.00 Cache hits for pass
cache_miss                  35         0.00 Cache misses
backend_conn                10         0.00 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail                 0         0.00 Backend conn. failures
backend_reuse               27         0.00 Backend conn. reuses
backend_toolate              0         0.00 Backend conn. was closed
backend_recycle             30         0.00 Backend conn. recycles
backend_retry                5         0.00 Backend conn. retry
fetch_head                   0         0.00 Fetch head
fetch_length                30         0.00 Fetch with Length
fetch_chunked                0         0.00 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                  0         0.00 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed                 0         0.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                    0         0.00 Fetch no body (304)
n_sess_mem                  14          .   N struct sess_mem
n_sess                       0          .   N struct sess
n_object                    16          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore                18          .   N struct objectcore
n_objecthead                21          .   N struct objecthead
n_waitinglist                7          .   N struct waitinglist
n_vbc                        3          .   N struct vbc
n_wrk                      800          .   N worker threads
n_wrk_create               800         0.09 N worker threads created
n_wrk_failed                 0         0.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued                 0         0.00 N queued work requests
n_wrk_drop                   0         0.00 N dropped work requests
n_backend                    4          .   N backends
n_expired                   14          .   N expired objects
n_lru_nuked                  0          .   N LRU nuked objects
n_lru_moved                 39          .   N LRU moved objects
losthdr                      0         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite                  25         0.00 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess                      21         0.00 Total Sessions
s_req                      146         0.02 Total Requests
s_pipe                       0         0.00 Total pipe
s_pass                       0         0.00 Total pass
s_fetch                     30         0.00 Total fetch
s_hdrbytes               45932         5.26 Total header bytes
s_bodybytes              66053         7.57 Total body bytes
sess_closed                  6         0.00 Session Closed
sess_pipeline                0         0.00 Session Pipeline
sess_readahead               0         0.00 Session Read Ahead
sess_linger                141         0.02 Session Linger
sess_herd                   90         0.01 Session herd
shm_records              17801         2.04 SHM records
shm_writes               10492         1.20 SHM writes
shm_flushes                  0         0.00 SHM flushes due to overflow
shm_cont                     0         0.00 SHM MTX contention
shm_cycles                   0         0.00 SHM cycles through buffer
sms_nreq                     5         0.00 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc                2090          .   SMS bytes allocated
sms_bfree                 2090          .   SMS bytes freed
backend_req                 37         0.00 Backend requests made
n_vcl                        1         0.00 N vcl total
n_vcl_avail                  1         0.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                        1          .   N total active bans
n_ban_add                    1         0.00 N new bans added
n_ban_retire                 0         0.00 N old bans deleted
n_ban_obj_test               0         0.00 N objects tested
n_ban_re_test                0         0.00 N regexps tested against
n_ban_dups                   0         0.00 N duplicate bans removed
hcb_nolock                 146         0.02 HCB Lookups without lock
hcb_lock                    18         0.00 HCB Lookups with lock
hcb_insert                  18         0.00 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             0         0.00 Connection dropped late
uptime                    8729         1.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                       9         0.00 Gzip operations
n_gunzip                     0         0.00 Gunzip operations
LCK.sms.creat                1         0.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks               15         0.00 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                2         0.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks              103         0.01 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                0         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks                0         0.00 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                1         0.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks               71         0.01 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                1         0.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks                9         0.00 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               1         0.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks              14         0.00 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            1         0.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks             37         0.00 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                1         0.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks            26237         3.01 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               1         0.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks               1         0.00 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                   2         0.00 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks               18452         2.11 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat              25         0.00 Created locks
LCK.objhdr.destroy             4         0.00 Destroyed locks
LCK.objhdr.locks             624         0.07 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  1         0.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks               8773         1.01 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  2         0.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks                 30         0.00 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  1         0.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks               2922         0.33 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  1         0.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks               8774         1.01 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  1         0.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks               3488         0.40 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  1         0.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks                 17         0.00 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat              4         0.00 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks            185         0.02 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMA.s0.c_req                  62         0.01 Allocator requests
SMA.s0.c_fail                  0         0.00 Allocator failures
SMA.s0.c_bytes           1798968       206.09 Bytes allocated
SMA.s0.c_freed           1548284       177.37 Bytes freed
SMA.s0.g_alloc                30          .   Allocations outstanding
SMA.s0.g_bytes            250684          .   Bytes outstanding
SMA.s0.g_space         268184772          .   Bytes available
SMA.Transient.c_req            0         0.00 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes            0         0.00 Bytes allocated
SMA.Transient.c_freed            0         0.00 Bytes freed
SMA.Transient.g_alloc            0          .   Allocations outstanding
SMA.Transient.g_bytes            0          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.backend11(10.1.1.20,,9010).vcls            1          .   VCL references
VBE.backend11(10.1.1.20,,9010).happy4611687117937967103          .   Happy health probes
VBE.backend12(10.1.1.21,,9010).vcls            1          .   VCL references
VBE.backend12(10.1.1.21,,9010).happy4611687117937967103          .   Happy health probes
VBE.backend21(10.1.1.20,,9003).vcls             1          .   VCL references
VBE.backend21(10.1.1.20,,9003).happy 18446744073709551615          .   Happy health probes
VBE.backend22(10.1.1.21,,9003).vcls             1          .   VCL references
VBE.backend22(10.1.1.21,,9003).happy 18446744073709551615          .   Happy health probes



From jdh132 at psu.edu  Thu Jun 13 19:13:00 2013
From: jdh132 at psu.edu (Jason Heffner)
Date: Thu, 13 Jun 2013 15:13:00 -0400
Subject: Timeout with large files...
In-Reply-To: <1371145598-sup-6981@geror.local>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
 <4E4FA8CEA7D2CA46B10FAAE3F58E8422DB06EB47CD@HOLANDAMBX.elmercurio.org>
 <1371111873.13560.YahooMailNeo@web121602.mail.ne1.yahoo.com>
 <1371145598-sup-6981@geror.local>
Message-ID: <D76A41C2-18A2-4785-B3E2-616AD5053B64@psu.edu>

On Jun 13, 2013, at 1:47 PM, James Pearson <james at ifixit.com> wrote:

> Excerpts from John Doe's message of 2013-06-13 01:24:33 -0700:
>> From: "juan.salinas at mercurio.cl" <juan.salinas at mercurio.cl>
>> 
>>> I think this link might help you...if you are using a version > 3.0
>>> https://www.varnish-software.com/blog/streaming-varnish-30
>> 
>> Cool, thx for the link.
>> Any idea how it handles client disconnection at the first download/caching?
>> Does it continue to cache the whole file even though the client closed the connection?  or does it give up?
> 
> I don't know terribly much personally.  There is some more information,
> including a whitepaper, on the website:
> https://www.varnish-software.com/blog/http-streaming-varnish
> 
> - P
> 

I didn?t see this mentioned but wanted to offer it as a solution..

https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-September/022469.html


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130613/a4d8b2dd/attachment.html>

From jdmls at yahoo.com  Fri Jun 14 15:22:03 2013
From: jdmls at yahoo.com (John Doe)
Date: Fri, 14 Jun 2013 08:22:03 -0700 (PDT)
Subject: Timeout with large files...
In-Reply-To: <CAHzQQx804qZs_yXQ2pB40oV54V9SBmfsrtrFmmLnb0H7_WNX6Q@mail.gmail.com>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
 <CAHzQQx804qZs_yXQ2pB40oV54V9SBmfsrtrFmmLnb0H7_WNX6Q@mail.gmail.com>
Message-ID: <1371223323.32547.YahooMailNeo@web121606.mail.ne1.yahoo.com>

From: Jocelyn <dreewoo at gmail.com>

>On Jun 12, 2013 5:15 PM, "John Doe" <jdmls at yahoo.com> wrote:
>> I have a little issue with large files time outing front end while varnish caches the file sent by the backend.
>> Is there a way to tell varnish to start sending the beginning of the file right away?
>> Or is there a way to directly "pipe" only for files with a size > n?
>The plus branch of Varnish available at:
>https://github.com/mbgrydeland/varnish-cache/tree/3.0.3-plus
>Includes support to "stream" objects from the backend using beresp.do_stream = true; in vcl_fetch. 


Thx to all for the info!
JD




From lkarsten at varnish-software.com  Mon Jun 17 08:55:37 2013
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 17 Jun 2013 10:55:37 +0200
Subject: varnish using dropped backend connections?
In-Reply-To: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>
References: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>
Message-ID: <20130617085536.GB29771@immer.varnish-software.com>

On Thu, Jun 13, 2013 at 12:56:27PM -0500, Nathan Warren wrote:
[..]
> My suspicion was that varnish was creating a keepalive session to the play server that was persistent. This seems to be the case with tcp sessions visible (via netstat) on both sides when no traffic is passing through. It seems as though the firewall was dropping the keepalive session, but varnish kept using it as if it were there. Then it kept re-using it over and over. After refreshing the page a few times (in error), the page will start working.

This sounds like a plausible explanation for the symptoms you are describing.

The "proper" fix is to find out which NAT/connection tracking table that
purges this entry and fix that.

If you just want to fix the problem, you can ask Varnish to stop reusing
backend connections. Of course this will give you TCP slow start on all backend
fetches, but this is usually acceptable.

Here is a VCL snippet for disabling reuse:

"""
sub vcl_miss {
    set bereq.http.Connection = "close";
}

sub vcl_pass {
    set bereq.http.Connection = "close";
}
"""


-- 
With regards,
Lasse Karstensen
Varnish Software AS



From dev at serviidbgroup.info  Mon Jun 10 00:22:16 2013
From: dev at serviidbgroup.info (Mark Strickland)
Date: Sun, 9 Jun 2013 20:22:16 -0400
Subject: help with cache misses Apache2 mod.fcgid with Varnish 3
Message-ID: <002201ce6570$9122c2c0$b3684840$@serviidbgroup.info>

I am running Apache2 mod.fcgid with Varnish 3 as a cache.  I am getting
around 50% cache misses.   I would appreciate any help or hints on what I am
doing wrong with my default.vcl.  

 

cid:image001.png at 01CE654D.EC949250

 

Here is my default.vcl 

 

 

# This is a basic VCL configuration file for varnish.  See the vcl(7)

# man page for details on VCL syntax and semantics.

#

 

# TODO: Update internal subnet ACL and security.

 

# Define the internal network subnet.

# These are used below to allow internal access to certain files while not

# allowing access from the public internet.

# acl internal {

#  "192.10.0.0"/24;

# }

 

# Default backend definition.  Set this to point to your content

# server.

#

backend default {

.host = "127.0.0.1";

.port = "8080";

.connect_timeout = 600s;

.first_byte_timeout = 600s;

.between_bytes_timeout = 600s;

.max_connections = 800;

}

 

acl purge {

  "localhost";

  "127.0.0.1";

}

 

 

# Respond to incoming requests.

sub vcl_recv {

                # Use anonymous, cached pages if all backends are down.

  if (!req.backend.healthy) {

    unset req.http.Cookie;

  }

  

  # Allow the backend to serve up stale content if it is responding slowly.

  set req.grace = 6h;

                

  if (req.restarts == 0) {

  if (req.http.X-Forwarded-For) {

    set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
client.ip;

  } else {

    set req.http.X-Forwarded-For = client.ip;

  }

}

   

# Check the incoming request type is "PURGE", not "GET" or "POST"

if (req.request == "PURGE") {

  # Check if the ip coresponds with the acl purge

  if (!client.ip ~ purge) {

  # Return error code 405 (Forbidden) when not

    error 405 "Not allowed.";

  }

  return (lookup);

}

   

  # Pipe these paths directly to Apache for streaming.

  #if (req.url ~ "^/admin/content/backup_migrate/export") {

  #  return (pipe);

  #}

  #req.url ~ "^.*/api/.*$" ||

  # Do not cache these paths.

  if (req.url ~ "^/status\.php$" ||

                 req.url ~ "^/update\.php$" ||

                req.url ~ "^/admin$" ||

                req.url ~ "^/admin/.*$" ||

req.url ~ "^.*/apidev/.*$" ||

                req.url ~ "^/flag/.*$" ||

                req.url ~ "^.*/munin/.*$" ||

                  req.url ~ "^.*/server-status/.*$" ||

                  req.url ~ "^.*/store_us/.*$" ||

                  req.url ~ "^.*/store_us_test/.*$" ||

                  req.url ~ "^.*/store_ca/.*$" ||

                  req.url ~ "^.*/store_uk/.*$" ||

                  req.url ~ "^.*/store_de/.*$" ||

                  req.url ~ "^.*/store_fr/.*$" ||

                  req.url ~ "^.*/store_it/.*$" ||

                  req.url ~ "^.*/store_cn/.*$" ||

                  req.url ~ "^.*/store_es/.*$" ||

                  req.url ~ "^.*/store_jp/.*$" ||

                 req.url ~ "^.*/ajax/.*$" ||

                  req.url ~ "^.*/dev.serviidb.com/.*$" ||

                  req.url ~ "^.*/user/.*$" ||

                  req.url ~ "^.*/cacti/.*$" ||

                  req.url ~ "^.*/admin/.*$" ||

                  req.url ~ "^.*/install/.*$" ||

                  req.url ~ "^.*/serviioweb/.*$" ||

                 req.url ~ "^.*/mediabrowser/.*$" ||

                 req.url ~ "^.*/phpmyadmin/.*$" ||

                  req.url ~ "^.*/xataface/.*$") {

       return (pass);

  }

 

  # Do not allow outside access to cron.php or install.php.

  #if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {

    # Have Varnish throw the error directly.

  #  error 404 "Page not found.";

    # Use a custom error page that you've defined in Drupal at the path
"404".

    # set req.url = "/404";

  #}

 

  # Always cache the following file types for all users. This list of
extensions

  # appears twice, once here and again in vcl_fetch so make sure you edit
both

  # and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset req.http.Cookie;

  }

 

 # Remove all cookies that Drupal doesn't need to know about. We explicitly 

  # list the ones that Drupal does need, the SESS and NO_CACHE. If, after 

  # running this code we find that either of these two cookies remains, we 

  # will pass as the page cannot be cached.

  if (req.http.Cookie) {

    # 1. Append a semi-colon to the front of the cookie string.

    # 2. Remove all spaces that appear after semi-colons.

    # 3. Match the cookies we want to keep, adding the space we removed 

    #    previously back. (\1) is first matching group in the regsuball.

    # 4. Remove all other cookies, identifying them by the fact that they
have

    #    no space after the preceding semi-colon.

    # 5. Remove all spaces and semi-colons from the beginning and end of the


    #    cookie string. 

    set req.http.Cookie = ";" + req.http.Cookie;

    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");    

    set req.http.Cookie = regsuball(req.http.Cookie,
";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");

    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");

    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

 

    if (req.http.Cookie == "") {

      # If there are no remaining cookies, remove the cookie header. If
there

      # aren't any cookie headers, Varnish's default behavior will be to
cache

      # the page.

      unset req.http.Cookie;

    }

    else {

      # If there is any cookies left (a session or NO_CACHE cookie), do not

      # cache the page. Pass it on to Apache directly.

      return (pass);

    }

  }

 

}

 

# Set a header to track a cache HIT/MISS.

sub vcl_deliver {

  if (obj.hits > 0) {

    set resp.http.X-Varnish-Cache = "HIT";

  }

  else {

    set resp.http.X-Varnish-Cache = "MISS";

  }

}

 

# Code determining what to do when serving items from the Apache servers.

# beresp == Back-end response from the web server.

sub vcl_fetch {

  # We need this to cache 404s, 301s, 500s. Otherwise, depending on backend
but 

  # definitely in Drupal's case these responses are not cacheable by
default.

  if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500)
{

    set beresp.ttl = 10m;

  }

  # this section of code sets the API to return a response code of 200 not
302.  DO NOT MODIFY OR DELETE COULD MESS UP CLIENTS.

    if(beresp.status == 302 && !beresp.http.Location){

    set beresp.status = 200; 

    set beresp.response = "OK";

  }

 

  # Don't allow static files to set cookies. 

  # (?i) denotes case insensitive in PCRE (perl compatible regular
expressions).

  # This list of extensions appears twice, once here and again in vcl_recv
so 

  # make sure you edit both and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset beresp.http.set-cookie;

  }

 

  # Allow items to be stale if needed.

  set beresp.grace = 6h;

}

 

sub vcl_hit {

        if (req.request == "PURGE") {

                purge;

                error 200 "Purged.";

        }

}

 

sub vcl_miss {

        if (req.request == "PURGE") {

                purge;

                error 200 "Purged.";

        }

}

 

# In the event of an error, show friendlier messages.

sub vcl_error {

  # Redirect to some other URL in the case of a homepage failure.

  #if (req.url ~ "^/?$") {

  #  set obj.status = 302;

  #  set obj.http.Location = "http://backup.example.com/";

  #}

 

  # Otherwise redirect to the homepage, which will likely be in the cache.

  set obj.http.Content-Type = "text/html; charset=utf-8";

  synthetic {"

<html>

<head>

  <title>Page Unavailable</title>

  <style>

    body { background: #303030; text-align: center; color: white; }

    #page { border: 1px solid #CCC; width: 500px; margin: 100px auto 0;
padding: 30px; background: #323232; }

    a, a:link, a:visited { color: #CCC; }

    .error { color: #222; }

  </style>

</head>

<body onload="setTimeout(function() { window.location = '/' }, 5000)">

  <div id="page">

    <h1 class="title">Page Unavailable</h1>

    <p>The page you requested is temporarily unavailable.</p>

    <p>We're redirecting you to the <a href="/">homepage</a> in 5
seconds.</p>

    <div class="error">(Error "} + obj.status + " " + obj.response +
{")</div>

  </div>

</body>

</html>

"};

  return (deliver);

}

 

 

Requests.log

 

/ hit

/ hit

/api/media.json?client=ServiiDroid&num=0 miss

/api/plugin.json?client=ServiiDroid&num=0 miss

/sites/default/files/hulu.png miss

/sites/default/files/comedycentral.jpg miss

/sites/default/files/CBS_0.jpg miss

/sites/default/files/abc_iview.jpg miss

/sites/default/files/4OD-logo.jpg miss

/ hit

/ hit

/sl/taxonomy/term/116/all miss

/ hit

/ru/node/317?language=de miss

/ hit

/ hit

/ hit

/sl/taxonomy/term/130/all miss

/plugin-api-name/youtubetop100 miss

/ hit

/ hit

/da/content/abc-iview-australia?language=en miss

/ hit

/sl/taxonomy/term/130/all miss

/ru/content/haha-sport-baseball?language=de miss

/ hit

/ hit

/ hit

/?q=node/add miss

/plugin-api-name/youtubetop100 miss

/node/add miss

/ hit

/user/register miss

/user/register miss

/ hit

/ja/comment/7?language=ja miss

/sites/default/files/itv.jpg miss

/sites/default/files/syfy.jpg miss

/sl/taxonomy/term/130/all miss

/sites/default/files/coco.jpg miss

/sites/default/files/css/css_fUXc0IbfJFVGhZ_CC63oYAw5rSt49epRhzcGmscaISU.css
hit

/sites/default/files/css/css_SKdZ0BB5XafXDbZJFEkdKM2wno6BIT2kaNm5ZvTeTgA.css
miss

/sites/default/files/css/css_KhS183x7gviqG0CuFRJJAOmVB1hHNwf9ofpIxQ9WMmE.css
hit

/sites/default/files/eclp_3.png hit

/sites/all/modules/addtoany/images/share_save_171_16.png hit

/sites/default/files/css/css_5_EBsvYf_U3gqgv1Idal1sImw7zcXB1kaIR6NOugp8o.css
hit

/sites/default/files/bbciplayer.png miss

/misc/feed.png hit

/sites/default/files/castalba.jpg miss

/sites/default/files/skysportsplus.png miss

/sites/default/files/hahasport.png miss

/sites/default/files/coolsporttv.png miss

/ hit

/ hit

/sites/default/files/sopcast.jpg miss

/sites/default/files/stopstream_0.png miss

/ar/ar/content/data/view/api_access?client_op=contains&client=&page=16 miss

/sites/default/files/dailymotion.jpg miss

/sites/default/files/twitch.png miss

/sites/default/files/Ustream.jpg miss

/sites/default/files/vimeo.jpg miss

/sites/default/files/YouTube.jpg miss

/sites/default/files/canalplus.jpg miss

/sites/default/files/m6groupe.png miss

/plugin-api-name/youtubetop100 miss

/ hit

/ hit

/zh-hans/node/199?language=en miss

/zh-hans/node/863?language=en miss

/api/install?client=ServiiDroid&nid=175.0 miss

/ hit

/sl/content/pilgrims-death miss

/content/how-cook-heston?language=ru miss

/ hit

/plugin-api-name/youtubetop100 miss

/sl/node/239 miss

/da/edit-plugin miss

/ hit

/ hit

/ hit

/plugin-api-name/youtubetop100 miss

/api/install?client=ServiiDroid&nid=771.0 miss

/ hit

/ hit

/ hit

/es/media-type/video?language=en miss

/es/media-type/video?language=en miss

/ hit

/plugin-api-name/youtubetop100 miss

/unpublished-forum miss

/ hit

/robots.txt hit

/ hit

/unpublished-forum miss

/content/pokemon miss

/content/conan-celebrity-interviews?language=es miss

/sl/region/uk?language=en miss

/unpublished-forum miss

/batch?op=start&id=1516 miss

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130609/be6207b5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 22305 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130609/be6207b5/attachment.png>

From juan.salinas at mercurio.cl  Wed Jun 12 16:35:16 2013
From: juan.salinas at mercurio.cl (juan.salinas at mercurio.cl)
Date: Wed, 12 Jun 2013 12:35:16 -0400
Subject: Timeout with large files...
In-Reply-To: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
References: <1371053315.61217.YahooMailNeo@web121605.mail.ne1.yahoo.com>
Message-ID: <4E4FA8CEA7D2CA46B10FAAE3F58E8422DB06EB47CD@HOLANDAMBX.elmercurio.org>

I think this link might help you...if you are using a version > 3.0

https://www.varnish-software.com/blog/streaming-varnish-30


Juan Salinas Contreras
Correo: juan.salinas at mercurio.cl
Telefono: +56-2-3301897


-----Mensaje original-----
De: varnish-misc-bounces+jsalinas=mercurio.cl at varnish-cache.org [mailto:varnish-misc-bounces+jsalinas=mercurio.cl at varnish-cache.org] En nombre de John Doe
Enviado el: mi?rcoles, 12 de junio de 2013 12:09
Para: ' varnish-misc at varnish-cache.org' 
Asunto: Timeout with large files...

Hi,

I have a little issue with large files time outing front end while varnish caches the file sent by the backend.
Is there a way to tell varnish to start sending the beginning of the file right away?
Or is there a way to directly "pipe" only for files with a size > n?

Thx,
JD

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From mattias at nucleus.be  Mon Jun 17 11:17:46 2013
From: mattias at nucleus.be (Mattias Geniar)
Date: Mon, 17 Jun 2013 11:17:46 +0000
Subject: help with cache misses Apache2 mod.fcgid with Varnish 3
In-Reply-To: <002201ce6570$9122c2c0$b3684840$@serviidbgroup.info>
Message-ID: <CDE4BCBD.6333C%mattias@nucleus.be>

> I am running Apache2 mod.fcgid with Varnish 3 as a cache.  I am getting around 50% cache misses.   I would appreciate any help or hints on what I am doing wrong with my default.vcl.

Hi Marc,

You may want to expand your monitoring and check for what kind of misses you have: pass, pipe, cache miss, ?
Your VCL has some custom logic to exclude certain URLs that are explicitly passed, this may just be what you're seeing here.

Mattias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130617/c37c401f/attachment.html>

From nwarren at nazarene.org  Tue Jun 18 15:22:10 2013
From: nwarren at nazarene.org (Nathan Warren)
Date: Tue, 18 Jun 2013 10:22:10 -0500
Subject: varnish using dropped backend connections?
In-Reply-To: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>
References: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>
Message-ID: <3680FCDD1B637D41B890408E6A2D6ADD07130607BB@ex07mailbox1.nazarene.org>

Thank you to all who at least put thought to my inquiry. Here is what I was able to discover:

1 ) The ASA firewall kills idle sessions after 1 hour
2) Varnish doesn't set the keepalive flag for backend connections
3) Once the tcp session is dropped by the firewall, varnish doesn't detect this, and keeps trying to use this backend connection until it times out
4) With multiple established (and failed) tcp sessions, the fetch from backend will lag severely waiting for timeouts until it either hits a usable tcp session or creates a new one
4) Health probes aren't able to detect this, since they create and close a tcp session for each probe

I ended up having to put apache in-between varnish and my backend since Apache does have a keepalive option in ProxyPass.

So my question is, should Varnish have an option to set the keepalive flag on backend connections? Should I make a feature request? This seems relatively simple.

Nate

-----Original Message-----
From: varnish-misc-bounces+nwarren=nazarene.org at varnish-cache.org [mailto:varnish-misc-bounces+nwarren=nazarene.org at varnish-cache.org] On Behalf Of Nathan Warren
Sent: Thursday, June 13, 2013 12:56 PM
To: varnish-misc at varnish-cache.org
Subject: varnish using dropped backend connections?

Good day,

I have a varnish cache serving requests to two play framework instances with a cisco asa firewall in-between. Everything works great at first - cache is working, acceptable cache hit ratio, fast, etc. But then after about an hour (without hits to the service), when the cache misses and pulls from the backend, there is no reply - the pound server times out while varnish hangs. Eventually varnish gives a 503 meditation, but pound doesn't wait that long to time out. Packet traces on the varnish box show packets being sent from varnish, but packet traces never show it reaching the backend server. The health checks keep polling successfully throughout.

My suspicion was that varnish was creating a keepalive session to the play server that was persistent. This seems to be the case with tcp sessions visible (via netstat) on both sides when no traffic is passing through. It seems as though the firewall was dropping the keepalive session, but varnish kept using it as if it were there. Then it kept re-using it over and over. After refreshing the page a few times (in error), the page will start working.

I found that cisco asa firewalls drop idle sessions after one hour, and that my version of linux has a 7200 time to wait before interrogating a possibly failed tcp session. I dropped this to 1800 via /proc/sys/net/ipv4/tcp_keepalive_time and restarted varnish. After waiting for an hour, the problem continues as before.

I would appreciate any insight you can provide, even if it's a problem with my config that doesn't seem to be related to this problem.

Thanks,
Nate

####  default.vcl  ####
backend backend11 {
        .host = "10.1.1.20";
        .port = "9010";
        .probe = {
                .url = "/sys-status";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

backend backend12 {
        .host = "10.1.1.21";
        .port = "9010";
        .probe = {
                .url = "/sys-status";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director backend1 random {
        .retries = 5;
        {
                .backend = backend11;
                .weight = 3;
        }
        {
                .backend = backend12;
                .weight = 3;
        }
}

backend backend21 {
        .host = "10.1.1.20";
        .port = "9003";
        .probe = {
                .url = "/ping";
                .interval = 10s;
                .timeout = 2s;
                .window = 5;
                .threshold = 3;
        }
}

backend backend22 {
        .host = "10.1.1.21";
        .port = "9003";
        .probe = {
                .url = "/ping";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director backend2 random {
        {
                .backend = backend11;
                .weight = 3;
        }
        {
                .backend = backend12;
                .weight = 3;
        }
}

sub vcl_recv {

        #serve stale data is ok
        set req.grace = 55m;

        #we don't care about any cookies
        unset req.http.Cookie;

        # secure redirects
        if (req.http.host ~ "^server1.nazarene.org$"
                        && req.http.X-Forwarded-Proto !~ "(?i)https" ) {
                set req.http.x-Redir-Url = "https://" + req.http.host + req.url;
                error 750 req.http.x-Redir-Url;
        }

        if (req.url ~ "(noaccess|notallowed)" ) {
                error 404 "Not Found";
        }

        # send requests to server1 to go to backend1
        if (req.http.host ~ "^server1.domain.local$") {
                set req.backend = backend1;

                #if the request is going to /foo/p/blah/blah then we can't cache, so don't try
                if (req.url ~ "^/foo/p/.*") {
                        return (pipe);
                }

                return (lookup);
        }

        # send requests to server2.domain.local
        if ( req.http.host ~ "^server2.domain.local$" ) {
                # but only for certain urls
                if ( req.url ~ "^/foo/(bar|foobar)" ) {
                        set req.backend = backend2;
                        return (lookup);
                }
        }

        # If you're not served at this point, I don't have anything for you
        error 404 "Not Found";
}

sub vcl_fetch {
        set beresp.grace = 60m;

        #strip cookies from backend so varnish will cache
        unset beresp.http.Set-Cookie;

        # Properly handle different encoding types
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

        if (beresp.http.content-type ~ "text") {
          set beresp.do_gzip = true;
        }
}

sub vcl_error {
        #redirect the client to the specified location
        if (obj.status == 750) {
                set obj.http.Location = obj.response;
                set obj.status = 302;
                return (deliver);
        }
}

#####  END default.vcl  ######

user at host:~$ sudo varnishstat -1 -n varnish
client_conn                 21         0.00 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req                 146         0.02 Client requests received
cache_hit                  111         0.01 Cache hits
cache_hitpass                0         0.00 Cache hits for pass
cache_miss                  35         0.00 Cache misses
backend_conn                10         0.00 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail                 0         0.00 Backend conn. failures
backend_reuse               27         0.00 Backend conn. reuses
backend_toolate              0         0.00 Backend conn. was closed
backend_recycle             30         0.00 Backend conn. recycles
backend_retry                5         0.00 Backend conn. retry
fetch_head                   0         0.00 Fetch head
fetch_length                30         0.00 Fetch with Length
fetch_chunked                0         0.00 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                  0         0.00 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed                 0         0.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                    0         0.00 Fetch no body (304)
n_sess_mem                  14          .   N struct sess_mem
n_sess                       0          .   N struct sess
n_object                    16          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore                18          .   N struct objectcore
n_objecthead                21          .   N struct objecthead
n_waitinglist                7          .   N struct waitinglist
n_vbc                        3          .   N struct vbc
n_wrk                      800          .   N worker threads
n_wrk_create               800         0.09 N worker threads created
n_wrk_failed                 0         0.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued                 0         0.00 N queued work requests
n_wrk_drop                   0         0.00 N dropped work requests
n_backend                    4          .   N backends
n_expired                   14          .   N expired objects
n_lru_nuked                  0          .   N LRU nuked objects
n_lru_moved                 39          .   N LRU moved objects
losthdr                      0         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite                  25         0.00 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess                      21         0.00 Total Sessions
s_req                      146         0.02 Total Requests
s_pipe                       0         0.00 Total pipe
s_pass                       0         0.00 Total pass
s_fetch                     30         0.00 Total fetch
s_hdrbytes               45932         5.26 Total header bytes
s_bodybytes              66053         7.57 Total body bytes
sess_closed                  6         0.00 Session Closed
sess_pipeline                0         0.00 Session Pipeline
sess_readahead               0         0.00 Session Read Ahead
sess_linger                141         0.02 Session Linger
sess_herd                   90         0.01 Session herd
shm_records              17801         2.04 SHM records
shm_writes               10492         1.20 SHM writes
shm_flushes                  0         0.00 SHM flushes due to overflow
shm_cont                     0         0.00 SHM MTX contention
shm_cycles                   0         0.00 SHM cycles through buffer
sms_nreq                     5         0.00 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc                2090          .   SMS bytes allocated
sms_bfree                 2090          .   SMS bytes freed
backend_req                 37         0.00 Backend requests made
n_vcl                        1         0.00 N vcl total
n_vcl_avail                  1         0.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                        1          .   N total active bans
n_ban_add                    1         0.00 N new bans added
n_ban_retire                 0         0.00 N old bans deleted
n_ban_obj_test               0         0.00 N objects tested
n_ban_re_test                0         0.00 N regexps tested against
n_ban_dups                   0         0.00 N duplicate bans removed
hcb_nolock                 146         0.02 HCB Lookups without lock
hcb_lock                    18         0.00 HCB Lookups with lock
hcb_insert                  18         0.00 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             0         0.00 Connection dropped late
uptime                    8729         1.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                       9         0.00 Gzip operations
n_gunzip                     0         0.00 Gunzip operations
LCK.sms.creat                1         0.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks               15         0.00 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                2         0.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks              103         0.01 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                0         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks                0         0.00 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                1         0.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks               71         0.01 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                1         0.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks                9         0.00 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               1         0.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks              14         0.00 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            1         0.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks             37         0.00 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                1         0.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks            26237         3.01 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               1         0.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks               1         0.00 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                   2         0.00 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks               18452         2.11 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat              25         0.00 Created locks
LCK.objhdr.destroy             4         0.00 Destroyed locks
LCK.objhdr.locks             624         0.07 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  1         0.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks               8773         1.01 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  2         0.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks                 30         0.00 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  1         0.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks               2922         0.33 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  1         0.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks               8774         1.01 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  1         0.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks               3488         0.40 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  1         0.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks                 17         0.00 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat              4         0.00 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks            185         0.02 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMA.s0.c_req                  62         0.01 Allocator requests
SMA.s0.c_fail                  0         0.00 Allocator failures
SMA.s0.c_bytes           1798968       206.09 Bytes allocated
SMA.s0.c_freed           1548284       177.37 Bytes freed
SMA.s0.g_alloc                30          .   Allocations outstanding
SMA.s0.g_bytes            250684          .   Bytes outstanding
SMA.s0.g_space         268184772          .   Bytes available
SMA.Transient.c_req            0         0.00 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes            0         0.00 Bytes allocated
SMA.Transient.c_freed            0         0.00 Bytes freed
SMA.Transient.g_alloc            0          .   Allocations outstanding
SMA.Transient.g_bytes            0          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.backend11(10.1.1.20,,9010).vcls            1          .   VCL references
VBE.backend11(10.1.1.20,,9010).happy4611687117937967103          .   Happy health probes
VBE.backend12(10.1.1.21,,9010).vcls            1          .   VCL references
VBE.backend12(10.1.1.21,,9010).happy4611687117937967103          .   Happy health probes
VBE.backend21(10.1.1.20,,9003).vcls             1          .   VCL references
VBE.backend21(10.1.1.20,,9003).happy 18446744073709551615          .   Happy health probes
VBE.backend22(10.1.1.21,,9003).vcls             1          .   VCL references
VBE.backend22(10.1.1.21,,9003).happy 18446744073709551615          .   Happy health probes

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From nwarren at nazarene.org  Tue Jun 18 17:16:07 2013
From: nwarren at nazarene.org (Nathan Warren)
Date: Tue, 18 Jun 2013 12:16:07 -0500
Subject: varnish using dropped backend connections?
In-Reply-To: <3680FCDD1B637D41B890408E6A2D6ADD07130607BB@ex07mailbox1.nazarene.org>
References: <3680FCDD1B637D41B890408E6A2D6ADD0712FD5BA2@ex07mailbox1.nazarene.org>
 <3680FCDD1B637D41B890408E6A2D6ADD07130607BB@ex07mailbox1.nazarene.org>
Message-ID: <3680FCDD1B637D41B890408E6A2D6ADD0713060809@ex07mailbox1.nazarene.org>

On Mon Jun 17 10:55:37 CEST 2013, Lasse Karstensen wrote:

>> My suspicion was that varnish was creating a keepalive session to the play server that was persistent. This seems to be the case with tcp sessions visible via netstat) on both sides when no traffic is passing through. It seems as though the firewall was dropping the keepalive session, but varnish kept using it as if it were there. Then it kept re-using it over and over. After refreshing the page a few times (in error), the page will start working.

>The "proper" fix is to find out which NAT/connection tracking table that purges this entry and fix that.

>If you just want to fix the problem, you can ask Varnish to stop reusing backend connections. Of course this will give you TCP slow start on all backend fetches, but this is usually acceptable.

>Here is a VCL snippet for disabling reuse:

>"""
>sub vcl_miss {
>    set bereq.http.Connection = "close";
>}

>sub vcl_pass {
>    set bereq.http.Connection = "close";
>}
>"""

Thanks, Lasse. Your message didn't make it to my inbox strangely, but I caught it in the archive. This makes perfect sense, and I believe I was missing the vlc_miss definition to force each session closed. I don't think the additional network overhead in my situation would be much trouble.

As I stated earlier, I ended up using proxypass in apache as an intermediate backend, since it has a feature which passes the SO_KEEPALIVE flag to the OS. I had to drop the keepalive timer on the OS as well via /proc/sys/net/ipv4/tcp_keepalive_time. At some point I'll probably try to cut apache out of the loop.

I'm still curious if it would be worthwhile to make an optional parameter to the backend definition to set the SO_KEEPALIVE flag. It seems to be an option for frontend connections, but not backend.

Nate





From lastcosmonaut at me.com  Wed Jun 19 08:39:36 2013
From: lastcosmonaut at me.com (Thilo Ettelt)
Date: Wed, 19 Jun 2013 10:39:36 +0200
Subject: SessionClose EOF
Message-ID: <D46889D7-6DCE-4E93-A3CD-41A96129C50D@me.com>

Hey there,

I'm new on this list.
I am experiencing sporadic (4 per Minute, with 45000 Requests/min served) SessionClose EOFs, which result in Socket Read TimeOuts on the client side. What does this EOF mean?

The documentation seems incomplete ;)



> SessionClose
> [...]
> EOF
> TODO
> 


My server setup looks like this:

14 Blades <> LB <> 2x Varnish <> 2x Backend

(I've tried using only 1 Varnish in the LoadBalancer, but that had no effect)

Sincerely,
Thilo


From davidurbansky at gmail.com  Wed Jun 19 23:26:12 2013
From: davidurbansky at gmail.com (David Urbansky)
Date: Thu, 20 Jun 2013 01:26:12 +0200
Subject: varnish and mod_pagespeed
Message-ID: <CADOBX8i+-jozGaSMmkiUabQSNmtUSLs6iAsH7XtNPZdsY688aA@mail.gmail.com>

Hello,

I am using Apache with mod_pagespeed (with
"ModPagespeedModifyCachingHeaders off" to allow caching) but
mod_pagespeed does usually not optimize the first request to a page.
Optimizations are done in the background and are then present for
subsequent requests.

The problem is that varnish caches the first request - an unoptimized
page. I wonder whether somebody has found a way to tell varnish to
ignore the first request after the ttl is over and cache the second
one?

Does anybody have this setup running?

Thank you and kind regards,
         David



From me at ibotty.net  Thu Jun 20 07:17:51 2013
From: me at ibotty.net (me at ibotty.net)
Date: Thu, 20 Jun 2013 09:17:51 +0200
Subject: varnish and mod_pagespeed
In-Reply-To: <CADOBX8i+-jozGaSMmkiUabQSNmtUSLs6iAsH7XtNPZdsY688aA@mail.gmail.com>
References: <CADOBX8i+-jozGaSMmkiUabQSNmtUSLs6iAsH7XtNPZdsY688aA@mail.gmail.com>
Message-ID: <51C2AC9F.70900@ibotty.net>

hi,

> The problem is that varnish caches the first request - an unoptimized
> page. I wonder whether somebody has found a way to tell varnish to
> ignore the first request after the ttl is over and cache the second
> one?

maybe you can configure mod_pagespeed to set a header when it is only
passing through the page.

and i thought the recommended setup is to have mod_pagespeed before
varnish, but i've never set it up.

good luck,
 tobias florek




From hans at moc.net  Thu Jun 20 07:21:00 2013
From: hans at moc.net (Hans Schou)
Date: Thu, 20 Jun 2013 09:21:00 +0200
Subject: Swap exceed & Child not responding to CLI, killing it
Message-ID: <51C2AD5C.5010005@moc.net>

Hi

I have just recently upgraded to Varnish 3.0.2 64bit Debian Squeeze on a 
new server an encounter some problems Ive not seen before.

There is 32G physical RAM and I use a 50G backend storage file.

# ps -eo rss,vsz,comm,cmd | grep varnishd
32439516 96968568 varnishd   /usr/sbin/varnishd -P /var/run/varnishd.pid 
-a :80,:8080 -T localhost:6082 -f /etc/varnish/default.vcl -s 
file,/var/lib/varnish/v02/varnish_storage.bin,50G -t 0 -p sess_workspace 
262144 -p thread_pool_min 200 -p thread_pool_max 4000 -p 
thread_pool_add_delay 2 -p session_linger 100

After running 8-10 hours the server is starting to using swap. When 
swapping increases to about 40% the server becomes slow and it ends up 
with a kill in syslog:

varnishd[7632]: Child (31696) not responding to CLI, killing it.
varnishd[7632]: Child (31696) died signal=3
varnishd[7632]: Child cleanup complete
varnishd[7632]: child (9203) Started
varnishd[7632]: Child (9203) said Child starts
varnishd[7632]: Child (9203) said SMF.s0 mmap'ed 53687091200 bytes of 
53687091200

My expectation was that varnish would use ~ 50G virtual RAM (vsz) and 
that Linux would use 32G RAM mapped with the file but not using swap. 
How can I prevent varnish from using more than the 50G vRAM and not 
swapping.

Any hint or help much appreciated.

-- 
Venlig hilsen
Hans Schou
tel:46923438




From hans at moc.net  Thu Jun 20 07:23:47 2013
From: hans at moc.net (Hans Schou)
Date: Thu, 20 Jun 2013 09:23:47 +0200
Subject: Swap exceed & Child not responding to CLI, killing it
In-Reply-To: <51C2AD5C.5010005@moc.net>
References: <51C2AD5C.5010005@moc.net>
Message-ID: <51C2AE03.8060502@moc.net>


S? er det bare at holde med et svar her:
https://www.varnish-cache.org/lists/pipermail/varnish-misc/2013-June/023157.html





From lury at critical.pl  Thu Jun 20 08:07:09 2013
From: lury at critical.pl (=?UTF-8?Q?=C5=81ukasz_Rysiak?=)
Date: Thu, 20 Jun 2013 10:07:09 +0200
Subject: varnish and mod_pagespeed
In-Reply-To: <51C2AC9F.70900@ibotty.net>
References: <CADOBX8i+-jozGaSMmkiUabQSNmtUSLs6iAsH7XtNPZdsY688aA@mail.gmail.com>
 <51C2AC9F.70900@ibotty.net>
Message-ID: <CAGeq0-cdbBaNNr-Q4H5KnXjcXS8EbpQ9i5jgOfh0OQ7NQOqfRQ@mail.gmail.com>

David, please read this thread on google mps forum, where i described the
same problem and got the solution from mps developers.
https://groups.google.com/forum/#!topic/mod-pagespeed-discuss/j7X4uSckXlg

You can add header which will force mps to do full optimisation before
sending a response to varnish.




Regards,

?ukasz Rysiak
Critical - Conversion Experts
tel. 792 448 678
http://critical.pl



2013/6/20 <me at ibotty.net>

> hi,
>
> > The problem is that varnish caches the first request - an unoptimized
> > page. I wonder whether somebody has found a way to tell varnish to
> > ignore the first request after the ttl is over and cache the second
> > one?
>
> maybe you can configure mod_pagespeed to set a header when it is only
> passing through the page.
>
> and i thought the recommended setup is to have mod_pagespeed before
> varnish, but i've never set it up.
>
> good luck,
>  tobias florek
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130620/b604826c/attachment.html>

From r at roze.lv  Thu Jun 20 14:51:08 2013
From: r at roze.lv (Reinis Rozitis)
Date: Thu, 20 Jun 2013 17:51:08 +0300
Subject: Swap exceed & Child not responding to CLI, killing it
In-Reply-To: <51C2AD5C.5010005@moc.net>
References: <51C2AD5C.5010005@moc.net>
Message-ID: <0423F797BEB04005A87E7E5669927081@MezhRoze>

> Any hint or help much appreciated.

Try setting sysctl vm.swappiness=0 (for permanent add it /etc/sysctl.conf 
and then sysctl -p).
It should instruct linux to (try) not to use swap at all.


p.s. I'm running my varnish boxes without swap at all - the 
behaviour/performance is much more predictable that way (you can test how it 
works out for you with 'swapoff -a')


rr 




From ruben at varnish-software.com  Fri Jun 21 00:34:47 2013
From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=)
Date: Fri, 21 Jun 2013 02:34:47 +0200
Subject: Varnish Utilities Directory (and some more)
Message-ID: <CAEfnkDbgfrA+gqLuXvWwa-BPkCEhEmBjHrgZ3cL14uMf2YeePA@mail.gmail.com>

Hello,

The Utilities Directory is now online. Feel free to add your own Varnish
Cache related project there, specially those that are for generic use and
that have been used in production somewhere, although that is not a
requirement:
 http://varnish.org/utilities

And for those of you that have a VMOD somewhere (yes I am thinking, among
others, of the GeoIP modules) but have not yet added it to the directory,
this is your chance:
 http://varnish.org/vmods

If you for any reason would rather have me adding your project or VMOD,
then reply to this email and send me the link to the description or
download link and I will get to it eventually.

Enjoy!


All the best,
-- 
<http://www.varnish-software.com/>*Rub?n Romero*
Varnish Software AS
Cell: +47 95964088 / Office: +47 21989260
Skype & Twitter: ruben_varnish
We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Europe Awards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130621/493ab9ba/attachment.html>

From lkarsten at varnish-software.com  Fri Jun 21 09:07:47 2013
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 21 Jun 2013 11:07:47 +0200
Subject: Swap exceed & Child not responding to CLI, killing it
In-Reply-To: <51C2AD5C.5010005@moc.net>
References: <51C2AD5C.5010005@moc.net>
Message-ID: <20130621090746.GB14369@immer.varnish-software.com>

On Thu, Jun 20, 2013 at 09:21:00AM +0200, Hans Schou wrote:
> I have just recently upgraded to Varnish 3.0.2 64bit Debian Squeeze
> on a new server an encounter some problems Ive not seen before.
> There is 32G physical RAM and I use a 50G backend storage file.
> # ps -eo rss,vsz,comm,cmd | grep varnishd
> 32439516 96968568 varnishd   /usr/sbin/varnishd -P
> /var/run/varnishd.pid -a :80,:8080 -T localhost:6082 -f
> /etc/varnish/default.vcl -s
> file,/var/lib/varnish/v02/varnish_storage.bin,50G -t 0 -p
> sess_workspace 262144 -p thread_pool_min 200 -p thread_pool_max 4000
> -p thread_pool_add_delay 2 -p session_linger 100

Why is your Varnish using 96GB virtual memory?

Please post more information. varnishstat -1 output and some VCL.


> After running 8-10 hours the server is starting to using swap. When
> swapping increases to about 40% the server becomes slow and it ends
> up with a kill in syslog:
> varnishd[7632]: Child (31696) not responding to CLI, killing it.
> varnishd[7632]: Child (31696) died signal=3
[..]

You can increase cli_timeout to stop the management process from killing the
child. Of course this is to keep your cached content slightly longer, your website
will be slow/dead while linux arranges its fingers/toes.


-- 
With regards,
Lasse Karstensen
Varnish Software AS



From hans at moc.net  Fri Jun 21 12:14:58 2013
From: hans at moc.net (Hans Schou)
Date: Fri, 21 Jun 2013 14:14:58 +0200
Subject: Swap exceed & Child not responding to CLI, killing it
In-Reply-To: <0423F797BEB04005A87E7E5669927081@MezhRoze>
References: <51C2AD5C.5010005@moc.net>
 <0423F797BEB04005A87E7E5669927081@MezhRoze>
Message-ID: <51C443C2.40207@moc.net>

Den 06/20/2013 04:51 PM, Reinis Rozitis skrev:
> Try setting sysctl vm.swappiness=0 (for permanent add it 
> /etc/sysctl.conf and then sysctl -p).
> It should instruct linux to (try) not to use swap at all.
It works much better now. Load less than 1. Fast response. And obviously 
no swapping. Thanks!

> p.s. I'm running my varnish boxes without swap at all - the 
> behaviour/performance is much more predictable that way (you can test 
> how it works out for you with 'swapoff -a')
Will do.

It took rather long time to get rid of the used swap and response became 
rather slow, so I was forced to do a restart (and lost a few seconds 
uptime). I guess the best way to stop using swap, is by first do a 
restart of varnish and then "echo 0 > /proc/sys/vm/swappiness".


-- 
Venlig hilsen
Hans Schou
tel:46923438




From varnish at tengu.ch  Mon Jun 24 05:30:21 2013
From: varnish at tengu.ch (Cedric Jeanneret)
Date: Mon, 24 Jun 2013 07:30:21 +0200
Subject: Ban URL - req.backend unsupported?
Message-ID: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>

Hello!

I'm trying to ban some content matching the configured req.backend, but it seems it's not working as expected:

varnish> ban req.backend == "foo_bar" && req.url ~ "^/baz/foo"
106        
unknown or unsupported field "req.backend"

I thought the ban syntax would take any of VCL variable (req.url, obj.set-cookie, req.http.host? and so on)? ?

Any help would be appreciated.

Thanks in advance.

Cheers,

C.



From daghf at varnish-software.com  Mon Jun 24 11:16:43 2013
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Mon, 24 Jun 2013 13:16:43 +0200
Subject: Ban URL - req.backend unsupported?
In-Reply-To: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
References: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
Message-ID: <CAEh05Vb+MHo4Jhw=wAuYMtzt=-0QXME4mungMXz9UkXkr=kD0Q@mail.gmail.com>

On Mon, Jun 24, 2013 at 7:30 AM, Cedric Jeanneret <varnish at tengu.ch> wrote:
>
>
> I'm trying to ban some content matching the configured req.backend, but it seems it's not working as expected:
>
> varnish> ban req.backend == "foo_bar" && req.url ~ "^/baz/foo"
> 106
> unknown or unsupported field "req.backend"
>
> I thought the ban syntax would take any of VCL variable (req.url, obj.set-cookie, req.http.host? and so on)? ?


Currently, ban expressions are restricted to the following variables:
  req.url
  req.http.*
  obj.http.*
  obj.status


So yes, you won't be able to use req.backend in a ban expression. I'm
not sure if there are any plans to expand on this.

--
Dag Haavi Finstad
Software Developer | Varnish Software AS
Mobile: +47 476 64 134 | Office: +47 21 98 92 60
We Make Websites Fly!



From varnish at tengu.ch  Mon Jun 24 11:53:44 2013
From: varnish at tengu.ch (Cedric Jeanneret)
Date: Mon, 24 Jun 2013 13:53:44 +0200
Subject: Ban URL - req.backend unsupported?
In-Reply-To: <CAEh05Vb+MHo4Jhw=wAuYMtzt=-0QXME4mungMXz9UkXkr=kD0Q@mail.gmail.com>
References: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
 <CAEh05Vb+MHo4Jhw=wAuYMtzt=-0QXME4mungMXz9UkXkr=kD0Q@mail.gmail.com>
Message-ID: <20130624135344.247c9f6e@wrk042.wrk.lsn.camptocamp.com>

On Mon, 24 Jun 2013 13:16:43 +0200
Dag Haavi Finstad <daghf at varnish-software.com> wrote:

> On Mon, Jun 24, 2013 at 7:30 AM, Cedric Jeanneret <varnish at tengu.ch> wrote:
> >
> >
> > I'm trying to ban some content matching the configured req.backend, but it seems it's not working as expected:
> >
> > varnish> ban req.backend == "foo_bar" && req.url ~ "^/baz/foo"
> > 106
> > unknown or unsupported field "req.backend"
> >
> > I thought the ban syntax would take any of VCL variable (req.url, obj.set-cookie, req.http.host? and so on)? ?
> 
> 
> Currently, ban expressions are restricted to the following variables:
>   req.url
>   req.http.*
>   obj.http.*
>   obj.status
> 
> 
> So yes, you won't be able to use req.backend in a ban expression. I'm
> not sure if there are any plans to expand on this.
> 
> --
> Dag Haavi Finstad
> Software Developer | Varnish Software AS
> Mobile: +47 476 64 134 | Office: +47 21 98 92 60
> We Make Websites Fly!

Hello Dag,

Thank you for your fast answer. It would be really great to add some new expression, as it may help people to flush/ban content in a more convenient way. For example, having many hosts aliases hitting the same backend would make use of the req.backend filter?

Anyway, this kind of solve my problem - in the current application it doesn't matter whether we can or not use the req.backend.

Cheers,

C.



From perbu at varnish-software.com  Mon Jun 24 11:58:25 2013
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 24 Jun 2013 13:58:25 +0200
Subject: Ban URL - req.backend unsupported?
In-Reply-To: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
References: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
Message-ID: <CAOXZevCT-MC7AZKV+4jgJ2t6qw6JTm7x+Lhw7Vjs-p2e=29ABg@mail.gmail.com>

Hi,

On Mon, Jun 24, 2013 at 7:30 AM, Cedric Jeanneret <varnish at tengu.ch> wrote:

> Hello!
>
> I'm trying to ban some content matching the configured req.backend, but it
> seems it's not working as expected:
>
> varnish> ban req.backend == "foo_bar" && req.url ~ "^/baz/foo"
> 106
> unknown or unsupported field "req.backend"
>

As Dag pointed out, this is unsupported.

However, you could solve it by assigning a HTTP response header that
indicates the backend. So, in fetch you do a resp.http.backend = "foo_bar"
and then ban on it later.


-- 
 <http://www.varnish-software.com/> *Per Buer*
Varnish Software AS
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130624/d8ebc478/attachment.html>

From varnish at tengu.ch  Mon Jun 24 12:06:16 2013
From: varnish at tengu.ch (Cedric Jeanneret)
Date: Mon, 24 Jun 2013 14:06:16 +0200
Subject: Ban URL - req.backend unsupported?
In-Reply-To: <CAOXZevCT-MC7AZKV+4jgJ2t6qw6JTm7x+Lhw7Vjs-p2e=29ABg@mail.gmail.com>
References: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
 <CAOXZevCT-MC7AZKV+4jgJ2t6qw6JTm7x+Lhw7Vjs-p2e=29ABg@mail.gmail.com>
Message-ID: <20130624140616.6add7ea5@wrk042.wrk.lsn.camptocamp.com>

On Mon, 24 Jun 2013 13:58:25 +0200
Per Buer <perbu at varnish-software.com> wrote:

> Hi,
> 
> On Mon, Jun 24, 2013 at 7:30 AM, Cedric Jeanneret <varnish at tengu.ch> wrote:
> 
> > Hello!
> >
> > I'm trying to ban some content matching the configured req.backend, but it
> > seems it's not working as expected:
> >
> > varnish> ban req.backend == "foo_bar" && req.url ~ "^/baz/foo"
> > 106
> > unknown or unsupported field "req.backend"
> >
> 
> As Dag pointed out, this is unsupported.
> 
> However, you could solve it by assigning a HTTP response header that
> indicates the backend. So, in fetch you do a resp.http.backend = "foo_bar"
> and then ban on it later.
> 
> 

Hello Per,

Hmm, yep, good point. I'll discuss with my colleague and see if this may be applied.

Thanks for the hint!

Cheers,

C.



From enno at tty.nl  Tue Jun 25 12:01:41 2013
From: enno at tty.nl (Enno van Amerongen)
Date: Tue, 25 Jun 2013 14:01:41 +0200
Subject: varnish crash: Error in munmap() / Cannot allocate memory
Message-ID: <20130625120141.11ec1b23@groupware.tty.nl>

Dear Varnish List,


Yesterday I upgraded memory of one of our Varnish servers (from 32GB to 96GB).
Everything was running perfectly fine (hitrate increased from 75% to 85%), until today when I tried to load new VCL using varnishadm:


    Reloading varnish config: live_20130625_131055
    VCL compiled.dlopen(./vcl.ZpQp7cNR.so): ./vcl.ZpQp7cNR.so: failed to map segment from shared object: Cannot allocate memory


    Command failed with error code 106


Immediately after vcl.load failed, the following errors started popping up in syslog:


    Jun 25 13:11:19 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): P
    Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #001
    Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #020???X#177
    Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #020???X#177


Shortly after, Varnish crashed completely and lost the whole cache. Before Varnish crashed, there was 16GB memory free on the server.


I have no idea yet what caused the crash, so hopefully someone on the list can shed some light on it.


Varnish is started with the following settings:


#!/bin/sh
echo starting varnish daemon
ulimit -n 131072
ulimit -l unlimited
sysctl -p /path/to/sysctl.conf
/path/to/varnishd \
    -s malloc,70G \
    -a 0.0.0.0:80 \
    -T 127.0.0.1:7777 \
    -p thread_pools=2 \
    -p thread_pool_add_delay=2 \
    -p thread_pool_min=500 \
    -p thread_pool_max=3000 \
    -p session_linger=50 \
    -p sess_workspace=65536 \
    -p connect_timeout=1 \
    -p lru_interval=10 \
    -n /var/www/varnish \
    -f /path/to/server.vcl
echo done




varnishstat showed the following:


$ varnishstat -1 | grep -i trans
SMA.Transient.c_req       845311        10.32 Allocator requests
SMA.Transient.c_fail         104         0.00 Allocator failures
SMA.Transient.c_bytes  10835594005    132281.74 Bytes allocated
SMA.Transient.c_freed  10835063135    132275.26 Bytes freed
SMA.Transient.g_alloc           68          .   Allocations outstanding
SMA.Transient.g_bytes       530870          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available


104 Allocator failures on this instance, while we normally never see Allocator failures.


Any ideas what caused this, or how I can fix it?


Kind regards,


Enno van Amerongen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130625/f1783e9e/attachment.html>

From isharov at yandex-team.ru  Tue Jun 25 12:41:28 2013
From: isharov at yandex-team.ru (Iliya Sharov)
Date: Tue, 25 Jun 2013 16:41:28 +0400
Subject: varnish crash: Error in munmap() / Cannot allocate memory
In-Reply-To: <20130625120141.11ec1b23@groupware.tty.nl>
References: <20130625120141.11ec1b23@groupware.tty.nl>
Message-ID: <51C98FF8.9000803@yandex-team.ru>

Hi!
Increase vm.max_map_count limit in the sysctl.
https://www.varnish-cache.org/docs/3.0/tutorial/troubleshooting.html

25.06.2013 16:01, Enno van Amerongen ?????:
> Error in munmap


-- 
Iliya Sharov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130625/cf25fd69/attachment.html>

From enno at tty.nl  Tue Jun 25 12:52:25 2013
From: enno at tty.nl (Enno van Amerongen)
Date: Tue, 25 Jun 2013 14:52:25 +0200
Subject: varnish crash: Error in munmap() / Cannot allocate memory
In-Reply-To: <51C98FF8.9000803@yandex-team.ru>
Message-ID: <20130625125225.6554e79f@groupware.tty.nl>

_____  

From: Iliya Sharov [mailto:isharov at yandex-team.ru]

                        
Hi!
        Increase vm.max_map_count limit in the sysctl.
  https://www.varnish-cache.org/docs/3.0/tutorial/troubleshooting.html
        
        25.06.2013 16:01, Enno van Amerongen ?????:
                      Error in munmap
heh, excellent, I just found that piece of documentation too and was gonna send it to the list for confirmation, but you beat me to it! awesome ;)


thanks a bunch!


Enno


from the docs:


"Specifically if you see the "Error in munmap" error on Linux you might want to increase the amount of maps available. Linux is limited to a maximum of 64k maps. Setting vm.max_map_count in sysctl.conf will enable you to increase this limit. You can inspect the number of maps your program is consuming by counting the lines in /proc/$PID/maps

This is a rather odd thing to document here - but hopefully Google will serve you this page if you ever encounter this error."  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130625/b50c6205/attachment.html>

From phk at phk.freebsd.dk  Mon Jun 24 06:39:43 2013
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 24 Jun 2013 06:39:43 +0000
Subject: Ban URL - req.backend unsupported?
In-Reply-To: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
References: <20130624073021.3fd8c3bb@wrk042.wrk.lsn.camptocamp.com>
Message-ID: <18698.1372055983@critter.freebsd.dk>

In message <20130624073021.3fd8c3bb at wrk042.wrk.lsn.camptocamp.com>, Cedric Jean
neret writes:

> I'm trying to ban some content matching the configured req.backend,
> but it seems it's not working as expected:

It is not supported, and probably doesn't work as well as you
expect it to.

Set the backend name into a HTTP header on the object,
and ban on that instead.

If you don't want the clients to see that header, remove it in
vcl_deliver{}


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From smwood4 at gmail.com  Wed Jun 26 22:58:56 2013
From: smwood4 at gmail.com (Stephen Wood)
Date: Wed, 26 Jun 2013 15:58:56 -0700
Subject: Effective way of monitoring cache hit/miss ratio?
Message-ID: <CA+pzux9aBLefexk27pDmcXepqwHicCVtsbBPjFzeWUperN4a7Q@mail.gmail.com>

Can anyone provide some advice for monitoring the cache hit/miss ratio?

The current way I'm doing it seems overly complicated. I'm running a script
every minute that executes"varnishstat -f cache_hit,cache_miss -1" and
stores the hits and misses as a delta. It stores it as a delta because I
need the hit/miss ratio over a period of time and not since the beginning
of the application starting.

It's cumbersome and requires two data points. It's also inflexible in terms
of hit/miss ratio over arbitrary periods of time.

Has anyone else encountered this and can provide some advice?

-- 
Stephen Wood
Dev/Ops Engineer
Moz, Inc.
Website: www.heystephenwood.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130626/7f3a0bb8/attachment.html>

From james at ifixit.com  Wed Jun 26 23:39:44 2013
From: james at ifixit.com (James Pearson)
Date: Wed, 26 Jun 2013 16:39:44 -0700
Subject: Effective way of monitoring cache hit/miss ratio?
In-Reply-To: <CA+pzux9aBLefexk27pDmcXepqwHicCVtsbBPjFzeWUperN4a7Q@mail.gmail.com>
References: <CA+pzux9aBLefexk27pDmcXepqwHicCVtsbBPjFzeWUperN4a7Q@mail.gmail.com>
Message-ID: <1372289670-sup-2307@geror.local>

Excerpts from Stephen Wood's message of 2013-06-26 15:58:56 -0700:
> Can anyone provide some advice for monitoring the cache hit/miss ratio?
> 
> The current way I'm doing it seems overly complicated. I'm running a script
> every minute that executes"varnishstat -f cache_hit,cache_miss -1" and
> stores the hits and misses as a delta. It stores it as a delta because I
> need the hit/miss ratio over a period of time and not since the beginning
> of the application starting.
> 
> It's cumbersome and requires two data points. It's also inflexible in terms
> of hit/miss ratio over arbitrary periods of time.
> 
> Has anyone else encountered this and can provide some advice?

We wrote a small script called pipe-to-graphite[0] and run it with the included
varnish.sh to send all of the stats from varnishstat into Graphite[1], a really
cool project that replaces rrdtool for data storage and also has a nifty
graph-based frontend.  Graphite has a wide variety of functions[2] for
processing data (nonNegativeDerivative is useful for things like Varnish's hit
stats), and also support json and plaintext output via its API, so it's not
terribly difficult to use it as a data-source for things like Nagios.

HTH.
 - P

[0]: https://github.com/iFixit/pipe-to-graphite
[1]: http://graphite.readthedocs.org/
[2]: http://graphite.readthedocs.org/en/0.9.10/functions.html



From herberto.graca at gmail.com  Sat Jun 29 12:44:00 2013
From: herberto.graca at gmail.com (=?ISO-8859-1?Q?Herberto_Gra=E7a?=)
Date: Sat, 29 Jun 2013 14:44:00 +0200
Subject: Unused backend code_sample_director, defined
Message-ID: <CAKOjejU+niqJhhWVcYrBZfXmN153ztoVDAu5wi4TxVBU3t9EBw@mail.gmail.com>

Hi,

Im new to varnish, and I've been following the tutorial to set it up.

Currently Im trying to set up varnish also as a load balancer, however it
comes out an error:

[vagrant at cache ~]$ sudo varnishd -n foo -f /etc/varnish/default.vcl -s
malloc,1G -T 127.0.0.1:2000  -a 0.0.0.0:8080 -d
Message from VCC-compiler:
Unused backend code_sample_director, defined:
('input' Line 29 Pos 10)
director code_sample_director round-robin {
---------####################--------------

Running VCC-compiler failed, exit 1

VCL compilation failed


This is my default.vcl:

backend app1 {
    .host = "app1.hgraca.com";
    .port = "80";
    .probe = {
        .url = "/";
        .interval = 5s;
        .timeout = 1s;
        .window = 5;
        .threshold = 3;
    }
}
backend app2 {
    .host = "app2.hgraca.com";
    .port = "80";
    .probe = {
        .url = "/";
        .interval = 5s;
        .timeout = 1s;
        .window = 5;
        .threshold = 3;
    }
}
director workspace_director round-robin {
    # server1
    {
        .backend = app1;
    }
    # server2
    {
        .backend = app2;
    }
}

Any ideaas about what Im missing?

Tkx,

Herberto Gra?a
_________________________________
<http://pt.linkedin.com/pub/herberto-gra%C3%A7a/8/10/b32>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20130629/dd634c20/attachment.html>