From drupalnut at gmail.com  Tue Sep  2 02:37:36 2008
From: drupalnut at gmail.com (Jack Tuhman)
Date: Mon, 1 Sep 2008 22:37:36 -0400
Subject: Ip Address
Message-ID: <f401c82a0809011937g2165f38w1122ce37ad13d4a9@mail.gmail.com>

Hi all,

We have a very old webapp that has an embedded server.   We have put varnish
in front of it to help with load while we rewrite it.   However, it is
detecting the IP address of the varnish box rather then the client.  I know
that in apache, we can fix this by looking at HTTP_X_FORWARDED_FOR, however,
our old app does not let us do that.  Is there anything we can do in varnish
to pass the real ip in the REMOTE_ADDR server var?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080901/9d2c8fe5/attachment.html>

From perbu at linpro.no  Tue Sep  2 06:10:12 2008
From: perbu at linpro.no (Per Buer)
Date: Tue, 02 Sep 2008 08:10:12 +0200
Subject: Ip Address
In-Reply-To: <f401c82a0809011937g2165f38w1122ce37ad13d4a9@mail.gmail.com>
References: <f401c82a0809011937g2165f38w1122ce37ad13d4a9@mail.gmail.com>
Message-ID: <48BCD8C4.8040601@linpro.no>

Jack Tuhman skrev:
>
> We have a very old webapp that has an embedded server.   We have put
> varnish in front of it to help with load while we rewrite it.   However,
> it is detecting the IP address of the varnish box rather then the
> client.  I know that in apache, we can fix this by looking at
> HTTP_X_FORWARDED_FOR, however, our old app does not let us do that.  Is
> there anything we can do in varnish to pass the real ip in the
> REMOTE_ADDR server var?

The REMOTE_ADDR server variable is set locally on the web server. So,
unless you can write some custom code to override REMOTE_ADDR you are
out of luck. :-(


Per.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080902/32b8e2b5/attachment.bin>

From varnish-misc at projects.linpro.no  Wed Sep  3 16:07:02 2008
From: varnish-misc at projects.linpro.no (VIAGRA INC)
Date: Wed,  3 Sep 2008 18:07:02 +0200 (CEST)
Subject: SALE 89% OFF
Message-ID: <20080903050706.13083.qmail@rrcs-24-106-36-66.west.biz.rr.com>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080903/fc32733b/attachment.html>

From drupalnut at gmail.com  Wed Sep  3 22:22:05 2008
From: drupalnut at gmail.com (Jack Tuhman)
Date: Wed, 3 Sep 2008 18:22:05 -0400
Subject: accept-encoding header
Message-ID: <f401c82a0809031522s7a8b415fmd0272b00a83996fd@mail.gmail.com>

Hi all,


We have a site behind varnish, and people are getting a blank page if they
they are behind squid.   It seems that varnish won't load anything unless
they sees an accept-encoding header, is there a way to default one if there
is not one?

Thanks,
Jack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080903/13ed1dec/attachment.html>

From fredrik.nygren at gotamedia.se  Thu Sep  4 03:49:23 2008
From: fredrik.nygren at gotamedia.se (Fredrik Nygren)
Date: Thu, 4 Sep 2008 05:49:23 +0200
Subject: accept-encoding header
In-Reply-To: <f401c82a0809031522s7a8b415fmd0272b00a83996fd@mail.gmail.com>
References: <f401c82a0809031522s7a8b415fmd0272b00a83996fd@mail.gmail.com>
Message-ID: <04967E0C-F827-4EAA-926F-F502FAFB1EC3@gotamedia.se>

Which version are you running?

We had the same problem with vistors using Squid and it affected  
Varnish 1.2. Maybe the problem is related to the HTTP/1.0 error:

http://varnish.projects.linpro.no/ticket/197
http://projects.linpro.no/pipermail/varnish-misc/2008-February/001429.html
http://projects.linpro.no/pipermail/varnish-dev/2008-February/000740.html

Regards
<fredrik />

On 4 sep 2008, at 00.22, Jack Tuhman wrote:

Hi all,


We have a site behind varnish, and people are getting a blank page if  
they they are behind squid.   It seems that varnish won't load  
anything unless they sees an accept-encoding header, is there a way to  
default one if there is not one?

Thanks,
Jack
_______________________________________________
varnish-misc mailing list
varnish-misc at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080904/10ac5a5f/attachment.html>

From cjohnson at socialvibe.com  Fri Sep  5 01:02:15 2008
From: cjohnson at socialvibe.com (Chris Johnson)
Date: Thu, 4 Sep 2008 18:02:15 -0700
Subject: rotating the varnishncsa log file
Message-ID: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>

Does anyone successfully have logrotate working with varnishncsa?

Thanks,
Chrs


From kova70 at gmail.com  Fri Sep  5 02:37:16 2008
From: kova70 at gmail.com (Raymond Hall)
Date: Thu, 4 Sep 2008 21:37:16 -0500
Subject: rotating the varnishncsa log file
In-Reply-To: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
References: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
Message-ID: <53dee190809041937g3fcacb1fq9261eb399b738da4@mail.gmail.com>

On Thu, Sep 4, 2008 at 8:02 PM, Chris Johnson <cjohnson at socialvibe.com> wrote:
> Does anyone successfully have logrotate working with varnishncsa?

On a RHEL5 machine:
(the DAEMON var in the initscript is set to varnishncsa-hack because
its a havcked version that writes the host header, just replace it
with varnishncsa, presto! :)

I have this logrotate conf:

[kovalski at 175384-proxy1 ~]$ cat /etc/logrotate.d/varnish
/var/log/varnish/varnish.log {
    daily
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/varnish/varnishlog.pid
2>/dev/null` 2> /dev/null || true
    endscript
}

And this init script:

[kovalski at 175384-proxy1 ~]$ cat /etc/init.d/varnishlog
#! /bin/sh
#
# varnishlog    Control the varnish HTTP accelerator logging daemon
#
# chkconfig: - 90 10
# description: HTTP accelerator logging daemon
# processname: varnishlog
# config:
# pidfile: /var/run/varnish/varnishlog.pid

# Source function library.
. /etc/init.d/functions

RETVAL=0
PROCNAME=varnishlog

DAEMON="/usr/bin/varnishncsa-hack"
PIDFILE="/var/run/varnish/varnishlog.pid"
LOCKFILE="/var/lock/subsys/varnishlog"
LOGFILE="/var/log/varnish/varnish.log"

DAEMON_OPTS="-c -a -w ${LOGFILE} -D -P $PIDFILE"

mkdir -p /var/run/varnish 2>/dev/null

# See how we were called.
case "$1" in
  start)
        echo -n "Starting varnish logging daeon: "
        daemon --pidfile $PIDFILE $DAEMON "$DAEMON_OPTS"
        sleep 1
        pkill -0 $PROCNAME
        RETVAL=$?
        if [ $RETVAL -eq 0 ]
        then
                echo_success
                touch $LOCKFILE
        else
                echo_failure
        fi
        echo
        ;;
  stop)
        echo -n "Stopping varnish logging daemon: "
        killproc -p $PIDFILE $DAEMON
        RETVAL=$?
        if [ $RETVAL -eq 0 ]
        then
                echo_success
                rm -f $LOCKFILE $PIDFILE
        else
                echo_failure
        fi
        echo
        ;;
  status)
        status -p $PIDFILE $PROCNAME
        RETVAL=$?
        ;;
  restart|reload)
        $0 stop
        $0 start
        RETVAL=$?
        ;;
  condrestart)
        if [ -f $PIDFILE ]; then
            $0 stop
            $0 start
            RETVAL=$?
        fi
        ;;
  *)
        echo "Usage: $0 {start|stop|status|restart|condrestart}"
        exit 1
esac

exit $RETVAL


>
> Thanks,
> Chrs
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>



-- 
I'd rather have an inch of a dog than miles of pedigree.

Dana Burnet


From nkinkade at creativecommons.org  Fri Sep  5 17:39:23 2008
From: nkinkade at creativecommons.org (Nathan Kinkade)
Date: Fri, 5 Sep 2008 10:39:23 -0700
Subject: rotating the varnishncsa log file
In-Reply-To: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
References: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
Message-ID: <1c5831080809051039v7bd99e38s18ea534ceb4920f1@mail.gmail.com>

2008/9/4 Chris Johnson <cjohnson at socialvibe.com>:
> Does anyone successfully have logrotate working with varnishncsa?
>
> Thanks,
> Chrs

Below is the solution I came up with.  The lastaction part may not be
necessary for anyone else, but on occasion it seemed to be necessary
for us, and rather than spending much time looking into the underlying
cause I'm okay with this solution.

Nathan

/etc/logrotate.d/varnish
===========================
/var/log/varnish/varnish.log {
  daily
  rotate 52
  compress
  create
  delaycompress
  dateext
  postrotate
    /usr/sbin/invoke-rc.d varnishlog reload > /dev/null
    DATE=$(date +%Y%m%d)
    VDIR=/var/log/varnish
    # varnishlog or varnishncsa(??) seems to produce a lot of entries
that are null,
    # the grep -v in the middle just attempts to eliminate them
    /usr/bin/ionice -c3 /usr/bin/varnishncsa -r
$VDIR/varnish.log-$DATE | grep -v '^- - -' | gzip >
$VDIR/varnish.log-$DATE.gz && rm $VDIR/varnish.log-
$DATE
  endscript
  lastaction
    # Sometimes, for reasons that I can't explain, varnishlog stops
logging after
    # rotation.  The new log just sits there at 0 bytes and it takes a reload to
    # make it start logging again.  It could be some complex timing issue, but
    # hopefully just adding one last restart of varnishlog will be enough.
    /usr/sbin/invoke-rc.d varnishlog force-reload > /dev/null
  endscript
}
===========================


From ottolski at web.de  Mon Sep  8 09:11:27 2008
From: ottolski at web.de (Sascha Ottolski)
Date: Mon, 8 Sep 2008 11:11:27 +0200
Subject: on linux: only one cpu in use?
Message-ID: <200809081111.28168.ottolski@web.de>

on a 4 core machine, running varnish exclusively, all the I/O goes to 
one core only:

# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:  603869381          0          0          0    IO-APIC-edge  timer
  6:          3          0          0          0    IO-APIC-edge  floppy
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 14:         63          0          0          0    IO-APIC-edge  ide0
 50:          0          0          0          0   IO-APIC-level  libata
 58:  451845938          0          0          0   IO-APIC-level  
aacraid
 66: 1457978435          0          0          0       PCI-MSI-X  eth0
 74: 1082456599          0          0          0       PCI-MSI-X  eth0
 82:  242944858          0          0          0       PCI-MSI-X  eth0
217:          0          0          0          0   IO-APIC-level  
ohci_hcd:usb1, libata
225:        131          0          0          0   IO-APIC-level  
ehci_hcd:usb2
233:          0          0          0          0   IO-APIC-level  libata
NMI:      75579      30102       9582       7646
LOC:  603897625  603897604  603897576  603897556
ERR:          0
MIS:          0


it's debian etch with the stock 2.6.18 kernel. performance is ok, beside 
the fact that the load climbs up when the cache is about 40-50 % full 
(with a 517GB cache file); below 40 % the load stays below 3, but after 
that level it rises to over 7 and even 10 at peak times. the response 
times are still good, though.

we just installed irqbalance to see if this makes a difference. after 
some minutes (without restarting varnish), the picture according to top 
to "top" is, that 3 out of 4 core do most of the I/O:

Tasks:  89 total,   1 running,  88 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.8%us,  0.8%sy,  0.0%ni, 36.7%id, 60.8%wa,  0.8%hi,  0.0%si,  
0.0%st
Cpu1  :  0.8%us,  1.7%sy,  0.0%ni,  0.0%id, 96.6%wa,  0.0%hi,  0.8%si,  
0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 97.5%id,  0.8%wa,  0.0%hi,  1.7%si,  
0.0%st
Cpu3  :  0.0%us,  0.8%sy,  0.0%ni,  4.2%id, 95.0%wa,  0.0%hi,  0.0%si,  
0.0%st


I'm posting mostly out of curiosity if this is an expected behavior.


Thanks, Sascha


From varnish-list at itiva.com  Mon Sep  8 19:39:18 2008
From: varnish-list at itiva.com (DHF)
Date: Mon, 08 Sep 2008 12:39:18 -0700
Subject: Range Requests
Message-ID: <48C57F66.9080802@itiva.com>

Does varnish support range requests?  I want each range request of a
large file to be treated as a unique web object in varnish, and I'm
wondering if its possible.  Are the headers used in the hashing, or a
better question would be can the range-request header be included in the
hash?  Would that work or am I missing some pieces that would screw up
the plan?

Thanks,

--DHF



From smiley at npr.org  Mon Sep  8 20:13:19 2008
From: smiley at npr.org (Shain Miley)
Date: Mon, 08 Sep 2008 16:13:19 -0400
Subject: HttpError Received errno 11
Message-ID: <48C5875F.3000502@npr.org>

Hello,
I see 'HttpError Received errno 11' listed at the top of list when I am 
using varnishtop. 
I was wondering if anyone knew what that error means and how/if I need 
to correct this problem.  We seem to have a very very large number of 
these...it just makes me a little uneasy.

Thanks,

Shain


-- 
Shain Miley | Level IV Unix System Administrator | NPR Digital Media | 202-513-3649



From phk at phk.freebsd.dk  Tue Sep  9 08:09:06 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 09 Sep 2008 08:09:06 +0000
Subject: HttpError Received errno 11 
In-Reply-To: Your message of "Mon, 08 Sep 2008 16:13:19 -0400."
	<48C5875F.3000502@npr.org> 
Message-ID: <60488.1220947746@critter.freebsd.dk>

In message <48C5875F.3000502 at npr.org>, Shain Miley writes:
>Hello,

>I see 'HttpError Received errno 11' listed at the top of list when I am 
>using varnishtop. 

On which operating system ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Tue Sep  9 08:09:32 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 09 Sep 2008 08:09:32 +0000
Subject: Range Requests 
In-Reply-To: Your message of "Mon, 08 Sep 2008 12:39:18 MST."
	<48C57F66.9080802@itiva.com> 
Message-ID: <60497.1220947772@critter.freebsd.dk>

In message <48C57F66.9080802 at itiva.com>, DHF writes:

>Does varnish support range requests? 

No.

It's probably not a big deal, but it's not on the plan for 2.0

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From smiley at npr.org  Tue Sep  9 15:21:21 2008
From: smiley at npr.org (Shain Miley)
Date: Tue, 09 Sep 2008 11:21:21 -0400
Subject: HttpError Received errno 11
In-Reply-To: <60488.1220947746@critter.freebsd.dk>
References: <60488.1220947746@critter.freebsd.dk>
Message-ID: <48C69471.8060906@npr.org>

Poul,

I am using Debian (Linux 2.6.18-4-686-bigmem #1 SMP).

Thanks,

Shain

Poul-Henning Kamp wrote:
> In message <48C5875F.3000502 at npr.org>, Shain Miley writes:
>   
>> Hello,
>>     
>
>   
>> I see 'HttpError Received errno 11' listed at the top of list when I am 
>> using varnishtop. 
>>     
>
> On which operating system ?
>
>   


-- 
Shain Miley | Level IV Unix System Administrator | NPR Digital Media | 202-513-3649



From wangmd0127 at gmail.com  Tue Sep  9 15:42:49 2008
From: wangmd0127 at gmail.com (mingdawang)
Date: Tue, 9 Sep 2008 23:42:49 +0800
Subject: trouble about varnish
Message-ID: <e15793d10809090842k25698590p6f7cc9beefcb7a0e@mail.gmail.com>

Hello,
i install varnish(rev 3175) in as5.1 64bit, i have some trouble:
1. when i used varnishstat to view varnish's status, i found that "worker
threads" always uqual 0. why?
2. "worker threads limited", what is the meaning of this parameter?

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080909/7c56824b/attachment.html>

From brian.pan at light-mc.com  Wed Sep 10 14:44:57 2008
From: brian.pan at light-mc.com (Brian Pan)
Date: Wed, 10 Sep 2008 09:44:57 -0500
Subject: IP and mod_rpaf
Message-ID: <6616347254C43B4F98EB5C7C421BCAA1250983@mail-33ps.atlarge.net>

Here's an apache module that might be useful for some people:

 

http://stderr.net/apache/rpaf/

 

It's similar to "mod_extracted_forward'.  It rewrites the remote_addr
for apache on the backend, so that Apache logs will see the client IP
address and not varnish server's address. 

 

I also added this to the .vcl as was recommended in the Varnish FAQ:

 

sub vcl_recv {
  # Add a unique header containing the client address
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;
  # [...]
}

 

It's also useful if you're using mod_bwshare - and are trying to
throttle usage by IP address. 

 

 

------

Brian Y. Pan

Light Management Consulting LLC

Princeton, NJ 08540   USA

 

Tel: +1 (609) 651-4779

Fax: +1 (609) 651-4799

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080910/b41629d4/attachment.html>

From chakatz at yahoo.com  Wed Sep 10 17:31:45 2008
From: chakatz at yahoo.com (Chaim Keren Tzion)
Date: Wed, 10 Sep 2008 10:31:45 -0700 (PDT)
Subject: Varnish Memory Consumption
Message-ID: <905966.19788.qm@web57005.mail.re3.yahoo.com>

Hi,

I'm running Varnish -r2626 on CentOS 5.2

The problem I'm having is that Varnish eats up all 16GB of RAM that I have and then fills up the 2GB Swap space. Also, one of the three Varnish machines I have is a Virtual Machine (8GB RAM +2GB Swap). The VM panics some time after the Swap fills up and becomes unresponsive. The hardware machines keep on running.

My questions are:

Regarding the memory consumption:
Is this normal?
Do I need to enlarge my Swap?
Do I have to define a storage file that is less than the RAM+Swap that I can give to Varnish, in order to ensure that they don't get filled up?
Should I switch to malloc?
Will an upgrade to a newer revision help?

Regarding the Virtual Machine:
Is there a way to protect the Virtual Machine?
It it a bad idea to use a Virtual Machine?

Thanks,
Chaim

My settings are below:

NFILES=131072
MEMLOCK=82000
VARNISH_STORAGE_SIZE=75% ( This is 91GB on the Hardware machines)
-h classic,500009
-s file,/var/lib/varnish/varnish_storage.bin,75%

user                       varnish (101)
group                      varnish (102)
default_ttl                864000 [seconds]
thread_pools               4 [pools]
thread_pool_max            4000 [threads]
thread_pool_min            1 [threads]
thread_pool_timeout        120 [seconds]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_add_delay      20 [milliseconds]
thread_pool_fail_delay     200 [milliseconds]
overflow_max               100 [%]
rush_exponent              3 [requests per request]
sess_workspace             8192 [bytes]
obj_workspace              2048 [bytes]
default_grace              10
sess_timeout               5 [seconds]
pipe_timeout               864000 [seconds]
send_timeout               600 [seconds]
auto_restart               on [bool]
fetch_chunksize            128 [kilobytes]
vcl_trace                  off [bool]
listen_address             :80
listen_depth               4096 [connections]
srcaddr_hash               1049 [buckets]
srcaddr_ttl                30 [seconds]
backend_http11             off [bool]
client_http11              off [bool]
cli_timeout                5 [seconds]
ping_interval              864000 [seconds]
lru_interval               3600 [seconds]
cc_command                 "exec cc -fpic -shared -Wl,-x -o %o %s"
max_restarts               4 [restarts]
esi_syntax                 0 [restarts]
max_esi_includes           5 [restarts]
cache_vbe_conns            off [bool]
connect_timeout            400 [ms]
clock_skew                 10 [s]
prefer_ipv6                off [bool]
session_linger             0 [ms]
cli_buffer                 8192 [bytes]
log_hashstring             off [bool]
log_local_address          off [bool]
acceptor                   default (epoll, poll)
diag_bitmap                0x0 [bitmap]
err_ttl                    0 [seconds]


      


From chakatz at yahoo.com  Wed Sep 10 21:29:59 2008
From: chakatz at yahoo.com (Chaim Keren Tzion)
Date: Wed, 10 Sep 2008 14:29:59 -0700 (PDT)
Subject: Varnish Memory Consumption
Message-ID: <687242.72099.qm@web57001.mail.re3.yahoo.com>

Just wanted to mention in order to avoid confusion, that the settings below are actually from another installation that I have which is running release 3175 (2.0B1+). It is also having the same problems described below.



----- Original Message ----
From: Chaim Keren Tzion <chakatz at yahoo.com>
To: varnish-misc at projects.linpro.no
Sent: Wednesday, September 10, 2008 8:31:45 PM
Subject: Varnish Memory Consumption

Hi,

I'm running Varnish -r2626 on CentOS 5.2

The problem I'm having is that Varnish eats up all 16GB of RAM that I have and then fills up the 2GB Swap space. Also, one of the three Varnish machines I have is a Virtual Machine (8GB RAM +2GB Swap). The VM panics some time after the Swap fills up and becomes unresponsive. The hardware machines keep on running.

My questions are:

Regarding the memory consumption:
Is this normal?
Do I need to enlarge my Swap?
Do I have to define a storage file that is less than the RAM+Swap that I can give to Varnish, in order to ensure that they don't get filled up?
Should I switch to malloc?
Will an upgrade to a newer revision help?

Regarding the Virtual Machine:
Is there a way to protect the Virtual Machine?
It it a bad idea to use a Virtual Machine?

Thanks,
Chaim

My settings are below:

NFILES=131072
MEMLOCK=82000
VARNISH_STORAGE_SIZE=75% ( This is 91GB on the Hardware machines)
-h classic,500009
-s file,/var/lib/varnish/varnish_storage.bin,75%

user                       varnish (101)
group                      varnish (102)
default_ttl                864000 [seconds]
thread_pools               4 [pools]
thread_pool_max            4000 [threads]
thread_pool_min            1 [threads]
thread_pool_timeout        120 [seconds]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_add_delay      20 [milliseconds]
thread_pool_fail_delay     200 [milliseconds]
overflow_max               100 [%]
rush_exponent              3 [requests per request]
sess_workspace             8192 [bytes]
obj_workspace              2048 [bytes]
default_grace              10
sess_timeout               5 [seconds]
pipe_timeout               864000 [seconds]
send_timeout               600 [seconds]
auto_restart               on [bool]
fetch_chunksize            128 [kilobytes]
vcl_trace                  off [bool]
listen_address             :80
listen_depth               4096 [connections]
srcaddr_hash               1049 [buckets]
srcaddr_ttl                30 [seconds]
backend_http11             off [bool]
client_http11              off [bool]
cli_timeout                5 [seconds]
ping_interval              864000 [seconds]
lru_interval               3600 [seconds]
cc_command                 "exec cc -fpic -shared -Wl,-x -o %o %s"
max_restarts               4 [restarts]
esi_syntax                 0 [restarts]
max_esi_includes           5 [restarts]
cache_vbe_conns            off [bool]
connect_timeout            400 [ms]
clock_skew                 10 [s]
prefer_ipv6                off [bool]
session_linger             0 [ms]
cli_buffer                 8192 [bytes]
log_hashstring             off [bool]
log_local_address          off [bool]
acceptor                   default (epoll, poll)
diag_bitmap                0x0 [bitmap]
err_ttl                    0 [seconds]


      
_______________________________________________
varnish-misc mailing list
varnish-misc at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc



      


From franck.leprette at decisionnews.com  Thu Sep 11 08:18:47 2008
From: franck.leprette at decisionnews.com (Franck LEPRETTE)
Date: Thu, 11 Sep 2008 10:18:47 +0200
Subject: URL truncated in varnish-1.1.2 debian/testing
Message-ID: <766DBDCE4DDF0146B01C511F227398A80A65543B@MAIL.mpl.decisionnews.lan>

Hello,
 
We encounter a problem using varnishnsca because the urls field are
truncated (limit of 256 bytes).
I tried to patch our source files like explained in this url 
http://projects.linpro.no/pipermail/varnish-misc/2007-September/001057.h
tml
 
but this is not working because it's not the good revision.
We currently use the varnish-1.1.2  in debian and the revision number is
1693 for bin/varnishd/shmlog.c
 
Do you know how to fix it ? is there any patch about this issue ?
 
Thanks in advance
Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080911/d6814b71/attachment.html>

From tfheen at linpro.no  Thu Sep 11 09:29:16 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Thu, 11 Sep 2008 11:29:16 +0200
Subject: Varnish Memory Consumption
In-Reply-To: <905966.19788.qm@web57005.mail.re3.yahoo.com> (Chaim Keren
	Tzion's message of "Wed, 10 Sep 2008 10:31:45 -0700 (PDT)")
References: <905966.19788.qm@web57005.mail.re3.yahoo.com>
Message-ID: <87wshjav4j.fsf@qurzaw.linpro.no>

]] Chaim Keren Tzion 

| I'm running Varnish -r2626 on CentOS 5.2
| 
| The problem I'm having is that Varnish eats up all 16GB of RAM that I
| have and then fills up the 2GB Swap space. Also, one of the three
| Varnish machines I have is a Virtual Machine (8GB RAM +2GB Swap). The
| VM panics some time after the Swap fills up and becomes
| unresponsive. The hardware machines keep on running.
| 
| My questions are:
| 
| Regarding the memory consumption:
| Is this normal?

Yes, fairly normal, given that you've given Varnish ~91G to use.  Apart
from the crashing, that is.  It shouldn't crash the virtual machine (but
then, that's a bug in your kernel rather than Varnish).

| Do I need to enlarge my Swap?

swap shouldn't really matter with mmap.

| Do I have to define a storage file that is less than the RAM+Swap that
| I can give to Varnish, in order to ensure that they don't get filled
| up? Should I switch to malloc? Will an upgrade to a newer revision
| help?

I'd recommend trying a swap partition the same size as the backing file
you use today and going with malloc and see if the problem then goes
away.

| Regarding the Virtual Machine:
| Is there a way to protect the Virtual Machine?
| It it a bad idea to use a Virtual Machine?

Which virtualisation environment are you using?

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From chakatz at yahoo.com  Thu Sep 11 10:24:29 2008
From: chakatz at yahoo.com (Chaim Keren Tzion)
Date: Thu, 11 Sep 2008 03:24:29 -0700 (PDT)
Subject: Varnish Memory Consumption
Message-ID: <999300.48050.qm@web57005.mail.re3.yahoo.com>

Thanks Tolle!,

Comments inline below:



----- Original Message ----
> From: Tollef Fog Heen <tfheen at linpro.no>
> To: varnish-misc at projects.linpro.no
> Sent: Thursday, September 11, 2008 12:29:16 PM
> Subject: Re: Varnish Memory Consumption
> 
> ]] Chaim Keren Tzion 
> 
> | I'm running Varnish -r2626 on CentOS 5.2
> | 
> | The problem I'm having is that Varnish eats up all 16GB of RAM that I
> | have and then fills up the 2GB Swap space. Also, one of the three
> | Varnish machines I have is a Virtual Machine (8GB RAM +2GB Swap). The
> | VM panics some time after the Swap fills up and becomes
> | unresponsive. The hardware machines keep on running.
> | 
> | My questions are:
> | 
> | Regarding the memory consumption:
> | Is this normal?
> 
> Yes, fairly normal, given that you've given Varnish ~91G to use.  Apart
> from the crashing, that is.  It shouldn't crash the virtual machine (but
> then, that's a bug in your kernel rather than Varnish).

The VM doesn't actually have a kernel panic. It just starts throws complaints about swap space to the console and won't allow logins or ssh.
 
> | Do I need to enlarge my Swap?
> 
> swap shouldn't really matter with mmap.
> 
> | Do I have to define a storage file that is less than the RAM+Swap that
> | I can give to Varnish, in order to ensure that they don't get filled
> | up? Should I switch to malloc? Will an upgrade to a newer revision
> | help?
> 
> I'd recommend trying a swap partition the same size as the backing file
> you use today and going with malloc and see if the problem then goes
> away.

Actually on the VM the swap file is 2G and the backing file is 2.5G. Close, but I will try with a smaller backing file.

> 
> | Regarding the Virtual Machine:
> | Is there a way to protect the Virtual Machine?
> | It it a bad idea to use a Virtual Machine?
> 
> Which virtualisation environment are you using?

VMware ESX 3.5

> 
> -- 
> Tollef Fog Heen 
> Redpill Linpro -- Changing the game!
> t: +47 21 54 41 73
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc



      


From jeanmarc.pouchoulon at gmail.com  Fri Sep 12 18:01:06 2008
From: jeanmarc.pouchoulon at gmail.com (jean-marc pouchoulon)
Date: Fri, 12 Sep 2008 20:01:06 +0200
Subject: req.restart or director to provide high availability 
Message-ID: <48CAAE62.8060508@gmail.com>

Hello,

I "implement" req.restart to provide high availbility, changing the 
backend if req.restart > 0
But I saw that director skips backend if it is unaivalable.

Both of them provides the same fonctionnality regards to high 
availability or Did I misunderstand something ?

thanks


jean-marc



From phk at phk.freebsd.dk  Fri Sep 12 18:37:33 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 12 Sep 2008 18:37:33 +0000
Subject: req.restart or director to provide high availability 
In-Reply-To: Your message of "Fri, 12 Sep 2008 20:01:06 +0200."
	<48CAAE62.8060508@gmail.com> 
Message-ID: <40282.1221244653@critter.freebsd.dk>

In message <48CAAE62.8060508 at gmail.com>, jean-marc pouchoulon writes:
>Hello,
>
>I "implement" req.restart to provide high availbility, changing the 
>backend if req.restart > 0
>But I saw that director skips backend if it is unaivalable.
>
>Both of them provides the same fonctionnality regards to high 
>availability or Did I misunderstand something ?

well, they _can_ overlap.

restarts are more flexible, for instance, instead of changing
the backend you could change the url or some other header when
the first attempt didn't work.

The directors only disable backends based on when their probe
requests fail, not on when a "production" request fails.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From tfheen at linpro.no  Mon Sep 15 14:11:40 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 15 Sep 2008 16:11:40 +0200
Subject: trouble about varnish
In-Reply-To: <e15793d10809090842k25698590p6f7cc9beefcb7a0e@mail.gmail.com>
	(mingdawang's message of "Tue, 9 Sep 2008 23:42:49 +0800")
References: <e15793d10809090842k25698590p6f7cc9beefcb7a0e@mail.gmail.com>
Message-ID: <871vzlfqhv.fsf@qurzaw.linpro.no>

]] mingdawang 

| i install varnish(rev 3175) in as5.1 64bit, i have some trouble:
| 1. when i used varnishstat to view varnish's status, i found that "worker
| threads" always uqual 0. why?

This is bug #308; I have a fix for this I need to commit.

| 2. "worker threads limited", what is the meaning of this parameter?

?How many threads did we not create because we ran into the limit on max
number of threads??

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From tfheen at linpro.no  Mon Sep 15 14:14:55 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 15 Sep 2008 16:14:55 +0200
Subject: Varnish Memory Consumption
In-Reply-To: <999300.48050.qm@web57005.mail.re3.yahoo.com> (Chaim Keren
	Tzion's message of "Thu, 11 Sep 2008 03:24:29 -0700 (PDT)")
References: <999300.48050.qm@web57005.mail.re3.yahoo.com>
Message-ID: <87wshdebs0.fsf@qurzaw.linpro.no>

]] Chaim Keren Tzion 

| > Yes, fairly normal, given that you've given Varnish ~91G to use.  Apart
| > from the crashing, that is.  It shouldn't crash the virtual machine (but
| > then, that's a bug in your kernel rather than Varnish).
| 
| The VM doesn't actually have a kernel panic. It just starts throws
| complaints about swap space to the console and won't allow logins or
| ssh.

Weird; we have not seen memory leaks with mmap.  We have a known leak
with malloc, though.

| > I'd recommend trying a swap partition the same size as the backing file
| > you use today and going with malloc and see if the problem then goes
| > away.
| 
| Actually on the VM the swap file is 2G and the backing file is
| 2.5G. Close, but I will try with a smaller backing file.

Did it help?

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From tfheen at linpro.no  Mon Sep 15 14:16:33 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 15 Sep 2008 16:16:33 +0200
Subject: on linux: only one cpu in use?
In-Reply-To: <200809081111.28168.ottolski@web.de> (Sascha Ottolski's message
	of "Mon, 8 Sep 2008 11:11:27 +0200")
References: <200809081111.28168.ottolski@web.de>
Message-ID: <87sks1ebpa.fsf@qurzaw.linpro.no>

]] Sascha Ottolski 

| it's debian etch with the stock 2.6.18 kernel. performance is ok, beside 
| the fact that the load climbs up when the cache is about 40-50 % full 
| (with a 517GB cache file); below 40 % the load stays below 3, but after 
| that level it rises to over 7 and even 10 at peak times. the response 
| times are still good, though.

Is this with the mmap or the malloc backend?

| we just installed irqbalance to see if this makes a difference. after 
| some minutes (without restarting varnish), the picture according to top 
| to "top" is, that 3 out of 4 core do most of the I/O:

Unsure why this happens.

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From phk at phk.freebsd.dk  Mon Sep 15 14:22:30 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 15 Sep 2008 14:22:30 +0000
Subject: Varnish Memory Consumption 
In-Reply-To: Your message of "Mon, 15 Sep 2008 16:14:55 +0200."
	<87wshdebs0.fsf@qurzaw.linpro.no> 
Message-ID: <1912.1221488550@critter.freebsd.dk>

In message <87wshdebs0.fsf at qurzaw.linpro.no>, Tollef Fog Heen writes:

>Weird; we have not seen memory leaks with mmap.  We have a known leak
>with malloc, though.

Uhm, no, we don't, we have no credible evidence that there is a leak
on our side of the malloc interface.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From ottolski at web.de  Tue Sep 16 08:50:59 2008
From: ottolski at web.de (Sascha Ottolski)
Date: Tue, 16 Sep 2008 10:50:59 +0200
Subject: on linux: only one cpu in use?
In-Reply-To: <87sks1ebpa.fsf@qurzaw.linpro.no>
References: <200809081111.28168.ottolski@web.de>
	<87sks1ebpa.fsf@qurzaw.linpro.no>
Message-ID: <200809161050.59439.ottolski@web.de>

Am Montag 15 September 2008 16:16:33 schrieb Tollef Fog Heen:
> ]] Sascha Ottolski
>
> | it's debian etch with the stock 2.6.18 kernel. performance is ok,
> | beside the fact that the load climbs up when the cache is about
> | 40-50 % full (with a 517GB cache file); below 40 % the load stays
> | below 3, but after that level it rises to over 7 and even 10 at
> | peak times. the response times are still good, though.
>
> Is this with the mmap or the malloc backend?

well, as the man page says about malloc "Not recommended for production 
use.", I use the file backend :-)

In case it matters, it's with trunk r2640.


Thanks, Sascha


From lukas.loesche at bertelsmann.de  Tue Sep 16 12:11:39 2008
From: lukas.loesche at bertelsmann.de (Loesche, Lukas, ZD-Scoyo)
Date: Tue, 16 Sep 2008 14:11:39 +0200
Subject: Custom varnishncsa Log Format?
In-Reply-To: <1912.1221488550@critter.freebsd.dk>
Message-ID: <C4F56F1B.9929%lukas.loesche@bertelsmann.de>

Hi,

I got a question regarding custom ncsa log formats. Are they possible?

We got varnish behind a loadbalancer. The loadbalancer sets the
X-Forwarded-For Header. When processing logs using varnishncsa I'd like to
see the content of X-Forwarded-For as remote addr, instead of the
loadbalancer's source IP.

E.g. for a request with
   16 ReqStart     c 172.19.0.28 36123 1596686019
...
   16 RxHeader     c X-Forwarded-For: 217.115.75.232

Instead of ncsa lines like:

172.19.0.28 - - [16/Sep/2008:14:07:15 +0200] "GET / HTTP/1.0" 200 898 "-"
"-"

I'd like to see

217.115.75.232 - - [16/Sep/2008:14:07:15 +0200] "GET / HTTP/1.0" 200 898 "-"
"-"


Is there an easy way to achieve this, without having to write a custom
varnishlog parser that generates ncsa output?


Regards,
-- Lukas



From jeanmarc.pouchoulon at gmail.com  Fri Sep 19 21:34:43 2008
From: jeanmarc.pouchoulon at gmail.com (jean-marc pouchoulon)
Date: Fri, 19 Sep 2008 23:34:43 +0200
Subject: when a backend in a director is stopped varnish is no longer
	responding ( 2.0 beta1)
Message-ID: <48D41AF3.5060703@gmail.com>

Hi,

I configured varnish (64bits server)  with two  backends as a director:

backend www2 {
     .host = "XX.XX.XX.XX";
     .port = "80";
     .probe = {
                .url = "/";
                .timeout = 2s;
                .interval = 5s;
                .window = 2;
                .threshold = 1;
     }
}


backend www2 {
     .host = "XX.XX.XX.XX";
     .port = "80";
     .probe = {
                .request =
                "GET / HTTP/1.1"
                "Host: www...."
                "Connection: close";
                .timeout = 500ms;
                .interval = 5s;
                .window = 2;
                .threshold = 1;
     }
}

director www_director random{
    { .backend = www1; .weight = 2; }
    { .backend = www2; .weight=1; }
}
...
sub vcl_recv {

     set req.grace = 30s;

     if (req.url ~ "/pub") {
                if (req.restarts == 0) {
                       set req.backend = pub8;
                } else {
                       set req.backend = pub6;
                }
     } else {
                set req.backend = www_director;
                }
  
    #

     if (req.http.Cookie ~ "www1\...." && req.http.Cookie ~ "auth_id=" ) {
                                 set req.backend = www1;
     }
     if (req.http.Cookie ~ "www2\...." && req.http.Cookie ~ "auth_id=" ) {
                                 set req.backend = www2;
     }


When I stopped one of the backend, varnish is no longer responding.
I can see clearly in varnishlog  that the backend regards to ill state 
but the failover is not working.

A telnet on port 80 of varnish didn't work either.
ReLauching the backend and varnish restarts working.
Using a vcl.conf without director is working ( The polling detects the 
fail of the backend)

thanks for your help

jean-marc
 




 





From tfheen at linpro.no  Mon Sep 22 19:16:49 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 22 Sep 2008 21:16:49 +0200
Subject: when a backend in a director is stopped varnish is no longer
	responding ( 2.0 beta1)
In-Reply-To: <48D41AF3.5060703@gmail.com> (jean-marc pouchoulon's message of
	"Fri, 19 Sep 2008 23:34:43 +0200")
References: <48D41AF3.5060703@gmail.com>
Message-ID: <87k5d4at3y.fsf@qurzaw.linpro.no>

]] jean-marc pouchoulon 


[...]

| When I stopped one of the backend, varnish is no longer responding.
| I can see clearly in varnishlog  that the backend regards to ill state 
| but the failover is not working.

This is already fixed in SVN; A new beta is happening shortly, so please
use that when it arrives.

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From cjohnson at socialvibe.com  Mon Sep 22 23:44:38 2008
From: cjohnson at socialvibe.com (Chris Johnson)
Date: Mon, 22 Sep 2008 16:44:38 -0700
Subject: rotating the varnishncsa log file
In-Reply-To: <1c5831080809051039v7bd99e38s18ea534ceb4920f1@mail.gmail.com>
References: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
	<1c5831080809051039v7bd99e38s18ea534ceb4920f1@mail.gmail.com>
Message-ID: <C01186DA-51CD-453C-8A10-1E2B26AD28E8@socialvibe.com>

Thank you for your replies Raymond and Nathan.

Can you possibly shed some light on the " /usr/sbin/invoke-rc.d"  
part?  Read on for more context...

I want to use logrotate to rotate the varnish log files.  I found the  
file (/home/username/varnish-1.1.1/debian/varnish.logrotate)  
containing the following:
 >>>
/var/log/varnish/varnish.log {
  daily
  rotate 7
  compress
  delaycompress
  postrotate
    /usr/sbin/invoke-rc.d varnishlog reload > /dev/null
  endscript
}
 >>>

However, we are using varnishncsa and not varnishlog. Running "ps aux  
| grep varnish" yields:
 >>
/opt/varnish/sbin/varnishd
/opt/varnish/bin/varnishncsa -a -c -w /var/log/varnish.log
 >>

Is setting up logrotate for varnishncsa as simple as changing the  
invoke-rc.d line of the logrotate file to be:
 >>
    /usr/sbin/invoke-rc.d varnishncsa reload > /dev/null
 >>

If so, how to pass in the ncsa commandline parameters?

Thanks for any guidance.
-Chris

On Sep 5, 2008, at 10:39 AM, Nathan Kinkade wrote:

> 2008/9/4 Chris Johnson <cjohnson at socialvibe.com>:
>> Does anyone successfully have logrotate working with varnishncsa?
>>
>> Thanks,
>> Chrs
>
> Below is the solution I came up with.  The lastaction part may not be
> necessary for anyone else, but on occasion it seemed to be necessary
> for us, and rather than spending much time looking into the underlying
> cause I'm okay with this solution.
>
> Nathan
>
> /etc/logrotate.d/varnish
> ===========================
> /var/log/varnish/varnish.log {
>  daily
>  rotate 52
>  compress
>  create
>  delaycompress
>  dateext
>  postrotate
>    /usr/sbin/invoke-rc.d varnishlog reload > /dev/null
>    DATE=$(date +%Y%m%d)
>    VDIR=/var/log/varnish
>    # varnishlog or varnishncsa(??) seems to produce a lot of entries
> that are null,
>    # the grep -v in the middle just attempts to eliminate them
>    /usr/bin/ionice -c3 /usr/bin/varnishncsa -r
> $VDIR/varnish.log-$DATE | grep -v '^- - -' | gzip >
> $VDIR/varnish.log-$DATE.gz && rm $VDIR/varnish.log-
> $DATE
>  endscript
>  lastaction
>    # Sometimes, for reasons that I can't explain, varnishlog stops
> logging after
>    # rotation.  The new log just sits there at 0 bytes and it takes  
> a reload to
>    # make it start logging again.  It could be some complex timing  
> issue, but
>    # hopefully just adding one last restart of varnishlog will be  
> enough.
>    /usr/sbin/invoke-rc.d varnishlog force-reload > /dev/null
>  endscript
> }
> ===========================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080922/46300377/attachment.html>

From kova70 at gmail.com  Tue Sep 23 17:12:43 2008
From: kova70 at gmail.com (Raymond Hall)
Date: Tue, 23 Sep 2008 12:12:43 -0500
Subject: rotating the varnishncsa log file
In-Reply-To: <C01186DA-51CD-453C-8A10-1E2B26AD28E8@socialvibe.com>
References: <88A63F89-D13C-46D5-8570-C5A32B513DB1@socialvibe.com>
	<1c5831080809051039v7bd99e38s18ea534ceb4920f1@mail.gmail.com>
	<C01186DA-51CD-453C-8A10-1E2B26AD28E8@socialvibe.com>
Message-ID: <53dee190809231012r1efb99bdx83f67cac9c62de2e@mail.gmail.com>

Hi Chris,

It all depends on you init scripts setup.
On a red hat based system (centos, rhel, fedora, mandrake, suse,
etc.), I just hacked the standard varnishlog init
(/etc/init.d/varnishlog) from the rpm, the relevant changes are these:

DAEMON="/usr/bin/varnishncsa-hack"
PIDFILE="/var/run/varnish/varnishlog.pid"
LOCKFILE="/var/lock/subsys/varnishlog"
LOGFILE="/var/log/varnish/varnish.log"
DAEMON_OPTS="-c -a -w ${LOGFILE} -D -P $PIDFILE"

So when logrotate does: /bin/kill -HUP `cat
/var/run/varnish/varnishlog.pid 2>/dev/null` 2> /dev/null || true
It actually resets varnishncsa logging.  I guess it'd be nicer to
write a proper varnishncsa init.d and logrotate scripts. Its easy
based on the original varnishlog scripts.

It seems that you're using debian. Did you write an init/d script for
varnishncsa or are you starting it elsewhere? If you wrote an init.d
script, then that's the one you want in the invoke-rc.d line. If you
didn't, that you have to notice where are you saving the pid the
process so you can do a kill -HUP on it. (which is what the init.d
script with reload arg does)

regards,
Ray

On Mon, Sep 22, 2008 at 6:44 PM, Chris Johnson <cjohnson at socialvibe.com> wrote:
> Thank you for your replies Raymond and Nathan.
> Can you possibly shed some light on the " /usr/sbin/invoke-rc.d" part?  Read
> on for more context...
>
> I want to use logrotate to rotate the varnish log files.  I found the file
> (/home/username/varnish-1.1.1/debian/varnish.logrotate) containing the
> following:
>>>>
> /var/log/varnish/varnish.log {
>  daily
>  rotate 7
>  compress
>  delaycompress
>  postrotate
>    /usr/sbin/invoke-rc.d varnishlog reload > /dev/null
>  endscript
> }
>>>>
>
> However, we are using varnishncsa and not varnishlog. Running "ps aux | grep
> varnish" yields:
>>>
> /opt/varnish/sbin/varnishd
> /opt/varnish/bin/varnishncsa -a -c -w /var/log/varnish.log
>>>
>
> Is setting up logrotate for varnishncsa as simple as changing the
> invoke-rc.d line of the logrotate file to be:
>>>
>    /usr/sbin/invoke-rc.d varnishncsa reload > /dev/null
>>>
>
> If so, how to pass in the ncsa commandline parameters?
>
> Thanks for any guidance.
> -Chris
>
> On Sep 5, 2008, at 10:39 AM, Nathan Kinkade wrote:
>
> 2008/9/4 Chris Johnson <cjohnson at socialvibe.com>:
>
> Does anyone successfully have logrotate working with varnishncsa?
>
> Thanks,
>
> Chrs
>
> Below is the solution I came up with.  The lastaction part may not be
> necessary for anyone else, but on occasion it seemed to be necessary
> for us, and rather than spending much time looking into the underlying
> cause I'm okay with this solution.
>
> Nathan
>
> /etc/logrotate.d/varnish
> ===========================
> /var/log/varnish/varnish.log {
>  daily
>  rotate 52
>  compress
>  create
>  delaycompress
>  dateext
>  postrotate
>    /usr/sbin/invoke-rc.d varnishlog reload > /dev/null
>    DATE=$(date +%Y%m%d)
>    VDIR=/var/log/varnish
>    # varnishlog or varnishncsa(??) seems to produce a lot of entries
> that are null,
>    # the grep -v in the middle just attempts to eliminate them
>    /usr/bin/ionice -c3 /usr/bin/varnishncsa -r
> $VDIR/varnish.log-$DATE | grep -v '^- - -' | gzip >
> $VDIR/varnish.log-$DATE.gz && rm $VDIR/varnish.log-
> $DATE
>  endscript
>  lastaction
>    # Sometimes, for reasons that I can't explain, varnishlog stops
> logging after
>    # rotation.  The new log just sits there at 0 bytes and it takes a reload
> to
>    # make it start logging again.  It could be some complex timing issue,
> but
>    # hopefully just adding one last restart of varnishlog will be enough.
>    /usr/sbin/invoke-rc.d varnishlog force-reload > /dev/null
>  endscript
> }
> ===========================
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>



-- 
I'd rather have an inch of a dog than miles of pedigree.

Dana Burnet


From duja at torlen.net  Wed Sep 24 09:40:24 2008
From: duja at torlen.net (duja at torlen.net)
Date: Wed, 24 Sep 2008 11:40:24 +0200
Subject: Cannot compile vcl
Message-ID: <ldzi2hw5q47saim.240920081140@torlen.net>

Hi,

Im having problem compiling the following VCL on varnish-2.0-tp2:
[VCL]
backend child_1 {
        .host = "192.168.0.192";
        .port = "80";
}

backend child_2 {
        .host = "192.168.0.193";
        .port = "80";
}


director www_director random {
        {
          .backend = child_1;
          .weight = 1;
        }

        {
          .backend = child_2;
          .weight = 2;
        }

}

sub vcl_recv {

        req.backend = www_director;

        # Cacha endast filer av denna filtyp
        # *.jpg , *.jpeg , *.gif , *.png, *.js , *.css , *.swf
        if (req.url ~ "\.(jpg|gif|png|js|css|swf)$") {
                lookup;
        }

        pass;

} #RECV

sub vcl_fetch {

        # Cacha endast filer av denna filtyp
        # *.jpg , *.jpeg , *.gif , *.png, *.js , *.css , *.swf
        if (req.url ~ "\.(jpg|gif|png|js|css|swf)$") {
                set obj.ttl = 1h;

                insert;
        }

        pass;

} #FETCH
[/VCL]

Im getting the following error:

Starting Varnish: varnishmgt_child_inherit(3, storage_file)
Expected action, 'if' or '}'
(/usr/local/etc/main.vcl Line 27 Pos 9)
        req.backend = www_director;
--------###########----------------

VCL compilation failed

I tried using probing in one of the backends but that gave me an error to :/

/ E



From phk at phk.freebsd.dk  Wed Sep 24 09:44:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 24 Sep 2008 09:44:50 +0000
Subject: Cannot compile vcl 
In-Reply-To: Your message of "Wed, 24 Sep 2008 11:40:24 +0200."
	<ldzi2hw5q47saim.240920081140@torlen.net> 
Message-ID: <89910.1222249490@critter.freebsd.dk>

In message <ldzi2hw5q47saim.240920081140 at torlen.net>, duja at torlen.net writes:
>Hi,
>
>Im having problem compiling the following VCL on varnish-2.0-tp2:
>[VCL]

>sub vcl_recv {
>
>        req.backend = www_director;

You need:

	set req.backend = www_director;

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Wed Sep 24 10:29:14 2008
From: duja at torlen.net (duja at torlen.net)
Date: Wed, 24 Sep 2008 12:29:14 +0200
Subject: Cannot compile vcl
Message-ID: <un5gmktrn66vfjk.240920081229@torlen.net>

Hmm, I must been blind :/

But now I get this, even if it used in the director.

Starting Varnish: varnishmgt_child_inherit(3, storage_file)
Unused backend child_2, defined:
(/usr/local/etc/main.vcl Line 6 Pos 9)
backend child_2 {
--------#######--

VCL compilation failed
.


/ E



From phk at phk.freebsd.dk  Wed Sep 24 11:16:55 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 24 Sep 2008 11:16:55 +0000
Subject: Cannot compile vcl 
In-Reply-To: Your message of "Wed, 24 Sep 2008 12:29:14 +0200."
	<un5gmktrn66vfjk.240920081229@torlen.net> 
Message-ID: <78987.1222255015@critter.freebsd.dk>

In message <un5gmktrn66vfjk.240920081229 at torlen.net>, duja at torlen.net writes:
>Hmm, I must been blind :/
>
>But now I get this, even if it used in the director.
>
>Starting Varnish: varnishmgt_child_inherit(3, storage_file)
>Unused backend child_2, defined:
>(/usr/local/etc/main.vcl Line 6 Pos 9)
>backend child_2 {
>--------#######--
>
>VCL compilation failed

Yes, the compiler says you have references to that backend anywhere
in your VCL program, that's considered an error.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Wed Sep 24 13:02:41 2008
From: duja at torlen.net (duja at torlen.net)
Date: Wed, 24 Sep 2008 15:02:41 +0200
Subject: Cannot compile vcl
Message-ID: <tcd85t75bxubk84.240920081502@torlen.net>

The only places where I use the backend child_2 is these:

backend child_1 {
        .host = "192.168.0.192";
        .port = "80";
}

backend child_2 {
        .host = "192.168.0.193";
        .port = "80";
}


director www_director random {
        {
          .backend = child_1;
          .weight = 1;
        }

        {
          .backend = child_2;
          .weight = 2;
        }

}

I cant find any errors in that code and I dont think that im blind again :)

/ E



From phk at phk.freebsd.dk  Wed Sep 24 13:05:42 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 24 Sep 2008 13:05:42 +0000
Subject: Cannot compile vcl 
In-Reply-To: Your message of "Wed, 24 Sep 2008 15:02:41 +0200."
	<tcd85t75bxubk84.240920081502@torlen.net> 
Message-ID: <10047.1222261542@critter.freebsd.dk>

In message <tcd85t75bxubk84.240920081502 at torlen.net>, duja at torlen.net writes:

>The only places where I use the backend child_2 is these:
>
>backend child_1 {
>        .host = "192.168.0.192";
>        .port = "80";
>}
>
>backend child_2 {
>        .host = "192.168.0.193";
>        .port = "80";
>}
>
>
>director www_director random {
>        {
>          .backend = child_1;
>          .weight = 1;
>        }
>
>        {
>          .backend = child_2;
>          .weight = 2;
>        }
>
>}
>
>I cant find any errors in that code and I dont think that im blind again :)

I don't see any errors either, and it works for me if I add to the above


	sub vcl_recv {
		set req.backend = www_director;
	}


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Thu Sep 25 07:37:38 2008
From: duja at torlen.net (duja at torlen.net)
Date: Thu, 25 Sep 2008 09:37:38 +0200
Subject: Cannot compile vcl
Message-ID: <2zqfa1p8hqn3jhv.250920080937@torlen.net>

Found the problem. The thing was that I used varnish-2.0-tp2 :/ I upgraded to beta2 today and now it compiles fine. 

/ E



From duja at torlen.net  Thu Sep 25 09:33:45 2008
From: duja at torlen.net (duja at torlen.net)
Date: Thu, 25 Sep 2008 11:33:45 +0200
Subject: Strange behaviour using director random
Message-ID: <onzb4r321qfrrus.250920081133@torlen.net>

Im testing directors in varnish 2.0 beta 2 with two backends.

The thing is that only one machine is getting all the traffic when using random directors and .weight = 1 on both backends.

It seems like the last assigned backend in the director is the one that's getting all the traffic.

If I use probing on the backends they both work.

[VCL]
backend child_1 {
        .host = "192.168.0.192";
        .port = "80";
/*
        .probe = {
                .url = "/Portals/0/logo1.gif";
                .timeout = 1s;
                .interval = 1s;
                .window = 8;
                .threshold = 6;
        }
*/
}


backend child_2 {
        .host = "192.168.0.193";
        .port = "80";
/*
        .probe = {
                .url = "/Portals/0/logo1.gif";
                .timeout = 100 ms;
                .interval = 1s;
                .window = 10;
                .threshold = 8;
        }
*/
}

director www_director random {
        {
          .backend = child_2;
          .weight = 1;
        }

        {
          .backend = child_1;
          .weight = 1;
        }

}
sub vcl_recv {

        set req.backend = www_director;

        # Cacha endast filer av denna filtyp
        # *.jpg , *.jpeg , *.gif , *.png, *.js , *.css , *.swf
        if (req.url ~ "\.(jpg|gif|png|js|css|swf)$") {

                lookup;
        }

        pass;

} #RECV

sub vcl_fetch {

        # Cacha endast filer av denna filtyp
        # *.jpg , *.jpeg , *.gif , *.png, *.js , *.css , *.swf
        if (req.url ~ "\.(jpg|gif|png|js|css|swf)$") {
                set obj.ttl = 1h;
                deliver;
        }

        pass;

} #FETCH
[/VCL]

In this case child_1 is getting all the traffic.
If I switch child_1 with child_2 in the director then child_2 is the one that gets all the traffic.

/ E



From nick at loman.net  Thu Sep 25 11:58:18 2008
From: nick at loman.net (Nick Loman)
Date: Thu, 25 Sep 2008 12:58:18 +0100
Subject: Keep-Alive acceleration, is this possible?
Message-ID: <48DB7CDA.9000604@loman.net>

Hi there,

On our platform we have had to disable Keep-Alive support on our 
Apache/FastCGI/PHP setup because it holds open too many backend 
processes under load even with KeepAliveTimeout set low.

I am looking at Varnish as a web accelerator (it looks great!), and I 
wonder if it is possible to get the best of both worlds, i.e. the user 
still gets KeepAlive support, but the backends server one script per 
connection.

I was thinking it might be possible to strip out the "Connection: Close" 
header returned by Apache, hopefully resulting in the client keeping its 
connection to Varnish. My question is: would Varnish realise that the 
backend connection has been closed but still keep the client connection 
open so that that future requests could be made on that same connection?

Would this have any adverse side-effects?

Grateful for your help

Nick



From phk at phk.freebsd.dk  Thu Sep 25 14:38:47 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 25 Sep 2008 14:38:47 +0000
Subject: Keep-Alive acceleration, is this possible? 
In-Reply-To: Your message of "Thu, 25 Sep 2008 12:58:18 +0100."
	<48DB7CDA.9000604@loman.net> 
Message-ID: <1604.1222353527@critter.freebsd.dk>

In message <48DB7CDA.9000604 at loman.net>, Nick Loman writes:

>I am looking at Varnish as a web accelerator (it looks great!), and I 
>wonder if it is possible to get the best of both worlds, i.e. the user 
>still gets KeepAlive support, but the backends server one script per 
>connection.

That will work.

>I was thinking it might be possible to strip out the "Connection: Close" 
>header returned by Apache, [...]

You don't need to do anything.

"Connection:" is a hop-by-hop header, so Varnish already deletes it before
sending the reply to the client

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Thu Sep 25 14:40:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 25 Sep 2008 14:40:50 +0000
Subject: Strange behaviour using director random 
In-Reply-To: Your message of "Thu, 25 Sep 2008 11:33:45 +0200."
	<onzb4r321qfrrus.250920081133@torlen.net> 
Message-ID: <1624.1222353650@critter.freebsd.dk>

In message <onzb4r321qfrrus.250920081133 at torlen.net>, duja at torlen.net writes:
>Im testing directors in varnish 2.0 beta 2 with two backends.
>
>The thing is that only one machine is getting all the traffic when
>using random directors and .weight = 1 on both backends.

This could be because Varnish manages to reuse the connection to the
backend so it never needs to choose backend server.

To test this hypothesis:  disable keepalive on the backend.

If that is inconvenient or unpractical, set a "Connection: close"
header in vcl_miss/vcl_pass/vcl_pipe.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From nick at loman.net  Thu Sep 25 21:28:59 2008
From: nick at loman.net (Nick Loman)
Date: Thu, 25 Sep 2008 22:28:59 +0100
Subject: Keep-Alive acceleration, is this possible?
In-Reply-To: <1604.1222353527@critter.freebsd.dk>
References: <1604.1222353527@critter.freebsd.dk>
Message-ID: <48DC029B.9030108@loman.net>

Poul-Henning Kamp wrote:

>> I was thinking it might be possible to strip out the "Connection: Close" 
>> header returned by Apache, [...]
> 
> You don't need to do anything.
> 
> "Connection:" is a hop-by-hop header, so Varnish already deletes it before
> sending the reply to the client

Perfect!

Many thanks,

Nick


From duja at torlen.net  Fri Sep 26 07:59:14 2008
From: duja at torlen.net (duja at torlen.net)
Date: Fri, 26 Sep 2008 09:59:14 +0200
Subject: Strange behaviour using director random
Message-ID: <f10l645gwv8f7px.260920080959@torlen.net>

I added this to my vcl:

sub vcl_pass {
        set bereq.http.Connection = "close";
}

sub vcl_miss {
        set bereq.http.Connection = "close";
}

But I got the same behaviour, only the last child in the director section is getting the traffic.

Should I open a ticket about this?

/ E



From 191919 at gmail.com  Sun Sep 28 14:10:50 2008
From: 191919 at gmail.com (Leo Wang)
Date: Sun, 28 Sep 2008 22:10:50 +0800
Subject: Varnish cache lost after restarted
Message-ID: <de894c690809280710ye556655xbd6727f3203429ac@mail.gmail.com>

When I restart varnish, the cache file is still there but it seems that all
cache contents are lost. All objects will have to be re-fetched from
backends even they are not expired.

Is it a designed behavior?

Thank you.

--
Regards,
LW
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080928/dc67a701/attachment.html>

From perbu at linpro.no  Sun Sep 28 16:32:05 2008
From: perbu at linpro.no (Per Buer)
Date: Sun, 28 Sep 2008 18:32:05 +0200
Subject: Varnish cache lost after restarted
In-Reply-To: <de894c690809280710ye556655xbd6727f3203429ac@mail.gmail.com>
References: <de894c690809280710ye556655xbd6727f3203429ac@mail.gmail.com>
Message-ID: <48DFB185.6020305@linpro.no>

Leo Wang skrev:
> When I restart varnish, the cache file is still there but it seems that
> all cache contents are lost. All objects will have to be re-fetched from
> backends even they are not expired.
> 
> Is it a designed behavior?

Maybe not designed, but definitively expected. A persistent storage
engine is more or less planned for 2.1.

-- 
Per Buer - Leder Infrastruktur og Drift - Redpill Linpro
Telefon: 21 54 41 21 - Mobil: 958 39 117
http://linpro.no/ | http://redpill.se/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080928/38620a7c/attachment.bin>

From tfheen at linpro.no  Mon Sep 29 12:12:16 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 29 Sep 2008 14:12:16 +0200
Subject: on linux: only one cpu in use?
In-Reply-To: <200809161050.59439.ottolski@web.de> (Sascha Ottolski's message
	of "Tue, 16 Sep 2008 10:50:59 +0200")
References: <200809081111.28168.ottolski@web.de>
	<87sks1ebpa.fsf@qurzaw.linpro.no> <200809161050.59439.ottolski@web.de>
Message-ID: <87ej33m9r3.fsf@qurzaw.linpro.no>

]] Sascha Ottolski 

| Am Montag 15 September 2008 16:16:33 schrieb Tollef Fog Heen:
| > ]] Sascha Ottolski
| >
| > | it's debian etch with the stock 2.6.18 kernel. performance is ok,
| > | beside the fact that the load climbs up when the cache is about
| > | 40-50 % full (with a 517GB cache file); below 40 % the load stays
| > | below 3, but after that level it rises to over 7 and even 10 at
| > | peak times. the response times are still good, though.
| >
| > Is this with the mmap or the malloc backend?
| 
| well, as the man page says about malloc "Not recommended for production 
| use.", I use the file backend :-)

Indeed.  Fixed that just now.  It seems like newer kernels has some
performance problems we are tickling.  I haven't had the time to track
down which commit breaks us, but it's fairly noticeable on busy sites.

| In case it matters, it's with trunk r2640.

This is fairly old, though.

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73


From tfheen at linpro.no  Mon Sep 29 12:13:12 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 29 Sep 2008 14:13:12 +0200
Subject: Custom varnishncsa Log Format?
In-Reply-To: <C4F56F1B.9929%lukas.loesche@bertelsmann.de> (Lukas Loesche's
	message of "Tue, 16 Sep 2008 14:11:39 +0200")
References: <C4F56F1B.9929%lukas.loesche@bertelsmann.de>
Message-ID: <87abdrm9pj.fsf@qurzaw.linpro.no>

]] "Loesche, Lukas, ZD-Scoyo" 

| I got a question regarding custom ncsa log formats. Are they possible?

Not at the moment.

| We got varnish behind a loadbalancer. The loadbalancer sets the
| X-Forwarded-For Header. When processing logs using varnishncsa I'd like to
| see the content of X-Forwarded-For as remote addr, instead of the
| loadbalancer's source IP.

[...]

| Is there an easy way to achieve this, without having to write a custom
| varnishlog parser that generates ncsa output?

I added the -f switch to varnishncsa in commit 3237 which allows you to
do this.

-- 
Tollef Fog Heen 
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73