From anders at fupp.net  Wed Jan  2 10:42:24 2008
From: anders at fupp.net (Anders Nordby)
Date: Wed, 2 Jan 2008 11:42:24 +0100
Subject: Varnish 1.1.2, still core dumps
Message-ID: <20080102104224.GA58498@fupp.net>

This is ticket #142 biting me again, now with Varnish 1.1.2:

(gdb) bt
#0  0x0000000000408a90 in vbe_sock_conn (ai=0x0) at cache_backend.c:162
#1  0x0000000000408b98 in vbe_conn_try (bp=0xaf1b80, pai=0x7ffffd1e7838)
    at cache_backend.c:190
#2  0x0000000000408d14 in vbe_connect (sp=0xc15008, bp=0xaf1b80)
    at cache_backend.c:228
#3  0x000000000040912e in vbe_nextfd (sp=0xc15008) at
cache_backend.c:299
#4  0x0000000000409372 in VBE_GetFd (sp=0xc15008) at cache_backend.c:336
#5  0x000000000040cafa in Fetch (sp=0xc15008) at cache_fetch.c:278
#6  0x000000000040a124 in cnt_fetch (sp=0xc15008) at cache_center.c:300
#7  0x000000000040b18d in CNT_Session (sp=0xc15008) at steps.h:40
#8  0x0000000000412cf8 in wrk_do_one (w=0x7ffffd1e7ad0) at
cache_pool.c:194
#9  0x00000000004131f3 in wrk_thread (priv=0x8562e0) at cache_pool.c:248
#10 0x000000080098755b in pthread_create () from /usr/lib/libthr.so.2
#11 0x00007ffffcfea000 in ?? ()
Cannot access memory at address 0x7ffffd1ea000
(gdb) frame 5
#5  0x000000000040cafa in Fetch (sp=0xc15008) at cache_fetch.c:278
278     cache_fetch.c: No such file or directory.
        in cache_fetch.c
(gdb) print *sp
$1 = {magic = 741317722, fd = 65, id = 65, xid = 652152521, 
  wrk = 0x7ffffd1e7ad0, sockaddrlen = 16, mysockaddrlen = 128, 
  sockaddr = 0xc15410, mysockaddr = 0xc15490, 
  addr = "84.211.134.255", '\0' <repeats 49 times>, 
  port = "16051\000\000\000\000\000\000\000\000\000\000", srcaddr =
0x28cbf00, 
  doclose = 0x0, http = 0xc15180, t_open = 1199268349.6441126, 
  t_req = 1199268349.9035521, t_resp = nan(0x8000000000000), 
  t_end = 1199268349.6441126, step = STP_FETCH, handling = 32, 
  wantbody = 1 '\001', err_code = 0, err_reason = 0x0, list = {
    vtqe_next = 0xd07008, vtqe_prev = 0xbb50d8}, backend = 0xaf1b80, 
  bereq = 0xbbb000, obj = 0x2c33800, vcl = 0x2d816c49c0, mem = 0xc15000, 
  workreq = {list = {vtqe_next = 0x0, vtqe_prev = 0x0}, sess =
0xc15008}, 
  acct = {first = 1199268348.5672638, sess = 1, req = 3, pipe = 0, pass
= 0, 
    fetch = 3, hdrbytes = 1024, bodybytes = 26656}, nhashptr = 10, 
  ihashptr = 4, lhashptr = 69, hashptr = 0xc15798}
(gdb) print sp
$2 = (struct sess *) 0xc15008
(gdb) print *hp
$3 = {magic = 1680389577, ws = {{s = 0xbbb2a0 "X-Varnish: 652152521", 
      e = 0xbbd2a0 "", 
      f = 0xbbb2d5 "\nServer: Apache/2.2.3 (Debian) Resin/3.1.3", r =
0x0}}, 
  rx_s = 0x0, rx_e = 0x0, pl_s = 0x0, pl_e = 0x0, conds = 0 '\0', 
  logtag = HTTP_Tx, hd = {{b = 0x42a3e2 "GET", e = 0x42a3e5 ""}, {
      b = 0xc15514 "/dynamic/tms.jsp/1.0.0/finnvector/14/1660/1653.png", 
      e = 0xc15546 ""}, {b = 0x42a11a "HTTP/1.1", e = 0x42a122 ""}, {b =
0x0, 
      e = 0x0}, {b = 0x0, e = 0x0}, {b = 0xc15551 "Accept: */*", 
      e = 0xc1555c ""}, {
      b = 0xc1555e "Referer:
http://finn.no/finn/mappopup?finnkode=11172143&utmx=249915&utmy=6825202&mapTitle=Skotten+Seter,++F??vang&airPossible=1", 
      e = 0xc155dd ""}, {b = 0xc155df "Accept-Language: no", e =
0xc155f2 ""}, 
    {b = 0xc155f4 "UA-CPU: x86", e = 0xc155ff ""}, {
      b = 0xc15601 "Accept-Encoding: gzip, deflate", e = 0xc1561f ""}, {
      b = 0xc15621 "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.1; .NET CLR 1.1.4322)", e = 0xc15672 ""}, {
      b = 0xc15674 "Host: maptiles2.finn.no", e = 0xc1568b ""}, {
      b = 0xc156a5 "Cookie: finn_ld=aeb69f3baeb69f34baeebabd;
JSESSIONID=cbapacpyQ6ArR95jm95Cr; RMID=54d386ff477b6170;
WT_FPC=id=2a8d6d3b7025c46a8bc1199268218640:lv=1199268345093:ss=1199268218640;
IADVISITOR=406416167; f"..., 
      e = 0xc1578d ""}, {b = 0xbbb2a0 "X-Varnish: 652152521", 
      e = 0xbbb2b4 ""}, {b = 0xbbb2b5 "X-Forwarded-for: 84.211.134.255", 
      e = 0xbbb2d4 ""}, {b = 0x0, e = 0x0} <repeats 17 times>}, 
  hdf = '\0' <repeats 31 times>, nhd = 15}

Bye,

-- 
Anders.


From anders at fupp.net  Wed Jan  2 18:15:41 2008
From: anders at fupp.net (Anders Nordby)
Date: Wed, 2 Jan 2008 19:15:41 +0100
Subject: Varnish 1.1.2, still core dumps
In-Reply-To: <20080102104224.GA58498@fupp.net>
References: <20080102104224.GA58498@fupp.net>
Message-ID: <20080102181541.GA73120@fupp.net>

Hi,

On Wed, Jan 02, 2008 at 11:42:24AM +0100, Anders Nordby wrote:
> This is ticket #142 biting me again, now with Varnish 1.1.2:
> 
> (gdb) bt
> #0  0x0000000000408a90 in vbe_sock_conn (ai=0x0) at cache_backend.c:162
> #1  0x0000000000408b98 in vbe_conn_try (bp=0xaf1b80, pai=0x7ffffd1e7838)
>     at cache_backend.c:190
> #2  0x0000000000408d14 in vbe_connect (sp=0xc15008, bp=0xaf1b80)
>     at cache_backend.c:228
> #3  0x000000000040912e in vbe_nextfd (sp=0xc15008) at
> cache_backend.c:299
> #4  0x0000000000409372 in VBE_GetFd (sp=0xc15008) at cache_backend.c:336
> #5  0x000000000040cafa in Fetch (sp=0xc15008) at cache_fetch.c:278
> #6  0x000000000040a124 in cnt_fetch (sp=0xc15008) at cache_center.c:300
> #7  0x000000000040b18d in CNT_Session (sp=0xc15008) at steps.h:40
> #8  0x0000000000412cf8 in wrk_do_one (w=0x7ffffd1e7ad0) at
> cache_pool.c:194
> #9  0x00000000004131f3 in wrk_thread (priv=0x8562e0) at cache_pool.c:248
> #10 0x000000080098755b in pthread_create () from /usr/lib/libthr.so.2
> #11 0x00007ffffcfea000 in ?? ()
> Cannot access memory at address 0x7ffffd1ea000

Discussing this with Daniel on the IRC channel, I wonder why commits
like changeset 1892 and 1965 etc. in cache_backend.c have been merged to
the 1.2 branch, but not to 1.1 and the 1.1.2 release? Are there other
merges that should have been done but were forgotten?

DES?

Bye,

-- 
Anders.


From anders at fupp.net  Thu Jan  3 07:35:13 2008
From: anders at fupp.net (Anders Nordby)
Date: Thu, 3 Jan 2008 08:35:13 +0100
Subject: Another Varnish 1.1.2 crash, in cache_backend.c
Message-ID: <20080103073513.GA99459@fupp.net>

Hi,

(gdb) bt
#0  0x28129f37 in thr_kill () from /lib/libc.so.6
#1  0x280cf1a5 in pthread_mutex_unlock () from /usr/lib/libthr.so.2
#2  0x280c72ae in raise () from /usr/lib/libthr.so.2
#3  0x281a4b78 in abort () from /lib/libc.so.6
#4  0x280aa0ed in lbv_assert (func=0x806ed70 "VBE_ClosedFd", 
    file=0x806ebfb "cache_backend.c", line=357, 
    cond=0x806edba "(close(vc->fd)) == 0", err=22) at assert.c:58
#5  0x0804fbdf in VBE_ClosedFd (w=0xb3b3ecf0, vc=0xb1ce040)
    at cache_backend.c:357
#6  0x0805a713 in PipeSession (sp=0x9099004) at cache_pipe.c:137
#7  0x080511df in cnt_pipe (sp=0x9099004) at cache_center.c:668
#8  0x080517c8 in CNT_Session (sp=0x9099004) at steps.h:35
#9  0x080590e2 in wrk_do_one (w=0xb3b3ecf0) at cache_pool.c:194
#10 0x08059650 in wrk_thread (priv=0x807e1f0) at cache_pool.c:248
#11 0x280c95cf in pthread_create () from /usr/lib/libthr.so.2
#12 0xb8489fec in ?? ()

Got this one during the night.

Bye,

-- 
Anders.


From chulmin2 at hotmail.com  Thu Jan  3 08:49:27 2008
From: chulmin2 at hotmail.com (MontyRee)
Date: Thu, 3 Jan 2008 08:49:27 +0000
Subject: about logging the file size..
Message-ID: <BLU116-W12B69EE7FE1D2EB006E1C85530@phx.gbl>


Hello list.
 
When downloading the large file for example, 100Mbyte through varnishd (1.1.2), if someone cancel the downloading at 10Mbyte, 
varnishncsa logging 100Mbyte not 10Mbyte.
 
If I download from the apache directly, apache logging just 10Mbyte.
 
I think 10Mbyte would be reasonable.
 
Is there any method or patch to solve this problem?
 
 
Thanks for your help.
 
 
 
Regards.
 
 
 
 
 
_________________________________________________________________
???? ??? ??? ?? ??, Live Search!
http://search.live.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080103/361a9eff/attachment.html>

From mike at dubdubdub.co.uk  Thu Jan  3 11:32:51 2008
From: mike at dubdubdub.co.uk (Mike Wallis)
Date: Thu, 3 Jan 2008 11:32:51 +0000
Subject: about logging the file size..
In-Reply-To: <BLU116-W12B69EE7FE1D2EB006E1C85530@phx.gbl>
References: <BLU116-W12B69EE7FE1D2EB006E1C85530@phx.gbl>
Message-ID: <E4EF25AB-3EFD-43B5-8B64-FB722338A5D6@dubdubdub.co.uk>

This looks like this bug within sendfile()

http://varnish.projects.linpro.no/ticket/187

regards,
-- mike

On 3 Jan 2008, at 08:49, MontyRee wrote:

> Hello list.
>
> When downloading the large file for example, 100Mbyte through  
> varnishd (1.1.2),
> if someone cancel the downloading at 10Mbyte,
> varnishncsa logging 100Mbyte not 10Mbyte.
>
> If I download from the apache directly, apache logging just 10Mbyte.
>
> I think 10Mbyte would be reasonable.
>
> Is there any method or patch to solve this problem?
>
>
> Thanks for your help.
>
>
>
> Regards.
>
>
>
>
>
>
> ???? ???? ???? ?? ??, live.com? ? 
> ? ?? ?? ?? ??? ??? Live Search! ??? 
> ? ??? ??? ?? ??, Live Search! 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080103/07d8fe16/attachment.html>

From linuxbsdfreak at gmail.com  Thu Jan  3 12:49:39 2008
From: linuxbsdfreak at gmail.com (kevin fernandes)
Date: Thu, 3 Jan 2008 18:19:39 +0530
Subject: varnish as a reverse proxy
Message-ID: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>

Hello All,

I require the varnish configuration for adding multiple backend webservers.
Please provide me few examples on how to do it. In short i require the
configuration of distributing the http-requests to
different backend webservers.

Regards,
Kevin Fernandes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080103/e002f2ed/attachment.html>

From perbu at linpro.no  Thu Jan  3 14:36:36 2008
From: perbu at linpro.no (Per Andreas Buer)
Date: Thu, 03 Jan 2008 15:36:36 +0100
Subject: varnish as a reverse proxy
In-Reply-To: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
Message-ID: <477CF2F4.9060702@linpro.no>

Hi Kevin.

kevin fernandes:

> I require the varnish configuration for adding multiple backend 
> webservers. Please provide me few examples on how to do it. In short i 
> require the configuration of distributing the http-requests to 
> different backend webservers.

Is this what you are looking for? 
http://varnish.projects.linpro.no/wiki/Backends



-- 
Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117                   
http://linpro.no/ - Ledende p? Linux og ?pen kildekode.



From linuxbsdfreak at gmail.com  Fri Jan  4 07:19:39 2008
From: linuxbsdfreak at gmail.com (kevin fernandes)
Date: Fri, 4 Jan 2008 12:49:39 +0530
Subject: varnish as a reverse proxy
In-Reply-To: <477CF2F4.9060702@linpro.no>
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
	<477CF2F4.9060702@linpro.no>
Message-ID: <1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>

Hello Per,

Thanks for your reply, i have tried the configuration given on the link.
However i receive the following error when i start varnish

Expected 'acl', 'sub' or 'backend', found 'backend_round_robin' at
(/etc/varnish/test.vcl Line 1 Pos 1)
backend_round_robin rr {

Can you please clarify why the error comes. I tried different ways to set
it. But no luck. Also attaching the vcl FYR.

Regards,
Kevin Fernandes

On Jan 3, 2008 8:06 PM, Per Andreas Buer <perbu at linpro.no> wrote:

> Hi Kevin.
>
> kevin fernandes:
>
> > I require the varnish configuration for adding multiple backend
> > webservers. Please provide me few examples on how to do it. In short i
> > require the configuration of distributing the http-requests to
> > different backend webservers.
>
> Is this what you are looking for?
> http://varnish.projects.linpro.no/wiki/Backends
>
>
>
> --
> Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117
> http://linpro.no/ - Ledende p? Linux og ?pen kildekode.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080104/4c308fa9/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.vcl.txt
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080104/4c308fa9/attachment.txt>

From ssm at linpro.no  Fri Jan  4 08:10:34 2008
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Fri, 04 Jan 2008 09:10:34 +0100
Subject: varnish as a reverse proxy
In-Reply-To: <1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>
	(kevin fernandes's message of "Fri, 4 Jan 2008 12:49:39 +0530")
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
	<477CF2F4.9060702@linpro.no>
	<1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>
Message-ID: <7x3atem2o5.fsf@iostat.linpro.no>

"kevin fernandes" <linuxbsdfreak at gmail.com> writes:

> Hello Per,
>
> Thanks for your reply, i have tried the configuration given on the
> link. However i receive the following error when i start varnish
>
> Expected 'acl', 'sub' or 'backend', found 'backend_round_robin' at
> (/etc/varnish/test.vcl Line 1 Pos 1)
> backend_round_robin rr {
>
> Can you please clarify why the error comes. I tried different ways
> to set it. But no luck. Also attaching the vcl FYR.

Which version of Varnish are you using?

-- 
Stig Sandbeck Mathisen, Linpro


From linuxbsdfreak at gmail.com  Fri Jan  4 08:22:11 2008
From: linuxbsdfreak at gmail.com (kevin fernandes)
Date: Fri, 4 Jan 2008 13:52:11 +0530
Subject: varnish as a reverse proxy
In-Reply-To: <7x3atem2o5.fsf@iostat.linpro.no>
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
	<477CF2F4.9060702@linpro.no>
	<1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>
	<7x3atem2o5.fsf@iostat.linpro.no>
Message-ID: <1c752e950801040022o1073693ek2c05c972741f6aae@mail.gmail.com>

Version is varnish-1.1.2


On Jan 4, 2008 1:40 PM, Stig Sandbeck Mathisen <ssm at linpro.no> wrote:

> "kevin fernandes" <linuxbsdfreak at gmail.com> writes:
>
> > Hello Per,
> >
> > Thanks for your reply, i have tried the configuration given on the
> > link. However i receive the following error when i start varnish
> >
> > Expected 'acl', 'sub' or 'backend', found 'backend_round_robin' at
> > (/etc/varnish/test.vcl Line 1 Pos 1)
> > backend_round_robin rr {
> >
> > Can you please clarify why the error comes. I tried different ways
> > to set it. But no luck. Also attaching the vcl FYR.
>
> Which version of Varnish are you using?
>
> --
> Stig Sandbeck Mathisen, Linpro
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080104/21ae5152/attachment.html>

From perbu at linpro.no  Fri Jan  4 09:19:21 2008
From: perbu at linpro.no (Per Andreas Buer)
Date: Fri, 04 Jan 2008 10:19:21 +0100
Subject: varnish as a reverse proxy
In-Reply-To: <1c752e950801040022o1073693ek2c05c972741f6aae@mail.gmail.com>
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>	<477CF2F4.9060702@linpro.no>	<1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>	<7x3atem2o5.fsf@iostat.linpro.no>
	<1c752e950801040022o1073693ek2c05c972741f6aae@mail.gmail.com>
Message-ID: <477DFA19.4000801@linpro.no>

kevin fernandes skrev:
> Version is varnish-1.1.2

I think load balancing is only available in trunk. Sorry.

-- 
Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117                   
http://linpro.no/ - Ledende p? Linux og ?pen kildekode.



From duja at torlen.net  Fri Jan  4 13:57:47 2008
From: duja at torlen.net (Erik)
Date: Fri, 4 Jan 2008 14:57:47 +0100
Subject: hash.url missing in varnish 1.1.2
Message-ID: <mhaw28zotjxo12o.040120081457@torlen.net>

Hi,

Downloaded and installed 1.1.2 from SF and noticed that hash.url has been removed from varnishadm. Didn't find anything about that on http://varnish.projects.linpro.no/wiki/Releases/1.1.2 .

Why has it been removed?

Cheers
Erik



From gslin at gslin.org  Sat Jan  5 09:15:29 2008
From: gslin at gslin.org (Gea-Suan Lin)
Date: Sat, 5 Jan 2008 17:15:29 +0800
Subject: varnish as a reverse proxy
In-Reply-To: <1c752e950801040022o1073693ek2c05c972741f6aae@mail.gmail.com>
References: <1c752e950801030449o3b03fc55v4122a1eee654a6cd@mail.gmail.com>
	<477CF2F4.9060702@linpro.no>
	<1c752e950801032319u1f2846cfsfb6eca5553e85e6@mail.gmail.com>
	<7x3atem2o5.fsf@iostat.linpro.no>
	<1c752e950801040022o1073693ek2c05c972741f6aae@mail.gmail.com>
Message-ID: <20080105091529.GA88466@gslin.org>

You need to use trunk version, or use other solution like Pound or
HAProxy to emulate multiple backend.

On Fri, Jan 04, 2008 at 01:52:11PM +0530, kevin fernandes wrote:
> Version is varnish-1.1.2
> 
> 
> On Jan 4, 2008 1:40 PM, Stig Sandbeck Mathisen <ssm at linpro.no> wrote:
> 
> > "kevin fernandes" <linuxbsdfreak at gmail.com> writes:
> >
> > > Hello Per,
> > >
> > > Thanks for your reply, i have tried the configuration given on the
> > > link. However i receive the following error when i start varnish
> > >
> > > Expected 'acl', 'sub' or 'backend', found 'backend_round_robin' at
> > > (/etc/varnish/test.vcl Line 1 Pos 1)
> > > backend_round_robin rr {
> > >
> > > Can you please clarify why the error comes. I tried different ways
> > > to set it. But no luck. Also attaching the vcl FYR.
> >
> > Which version of Varnish are you using?

-- 
* Gea-Suan Lin                                    http://blog.gslin.org/
* If you cannot convince them, confuse them.           -- Harry S Truman


From duja at torlen.net  Mon Jan  7 09:34:45 2008
From: duja at torlen.net (Erik)
Date: Mon, 7 Jan 2008 10:34:45 +0100
Subject: varnishtop hangs
Message-ID: <vx8f8g4j4c483hq.070120081034@torlen.net>

I do the following in varnishtop: varnishtop -i TxURL

after some request it hangs and from strace I get:

rt_sigaction(SIGTSTP, {SIG_IGN}, {0xb7f86fe0, [], SA_RESTART}, 8) = 0
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
rt_sigaction(SIGTSTP, {0xb7f86fe0, [], SA_RESTART}, NULL, 8) = 0
gettimeofday({1199697921, 208862}, NULL) = 0
select(1, [0], NULL, NULL, {1, 0})      = 0 (Timeout)
gettimeofday({1199697922, 208654}, NULL) = 0
time(NULL)                              = 1199697922
rt_sigaction(SIGTSTP, {SIG_IGN}, {0xb7f86fe0, [], SA_RESTART}, 8) = 0
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
rt_sigaction(SIGTSTP, {0xb7f86fe0, [], SA_RESTART}, NULL, 8) = 0
gettimeofday({1199697922, 208821}, NULL) = 0
select(1, [0], NULL, NULL, {1, 0})      = 0 (Timeout)
gettimeofday({1199697923, 208618}, NULL) = 0
time(NULL)                              = 1199697923
rt_sigaction(SIGTSTP, {SIG_IGN}, {0xb7f86fe0, [], SA_RESTART}, 8) = 0
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
rt_sigaction(SIGTSTP, {0xb7f86fe0, [], SA_RESTART}, NULL, 8) = 0
gettimeofday({1199697923, 208781}, NULL) = 0
select(1, [0], NULL, NULL, {1, 0})      = 0 (Timeout)
gettimeofday({1199697924, 208580}, NULL) = 0
time(NULL)                              = 1199697924
rt_sigaction(SIGTSTP, {SIG_IGN}, {0xb7f86fe0, [], SA_RESTART}, 8) = 0
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
rt_sigaction(SIGTSTP, {0xb7f86fe0, [], SA_RESTART}, NULL, 8) = 0
gettimeofday({1199697924, 208748}, NULL) = 0
select(1, [0], NULL, NULL, {1, 0})      = 0 (Timeout)
gettimeofday({1199697925, 208553}, NULL) = 0
time(NULL)                              = 1199697925
rt_sigaction(SIGTSTP, {SIG_IGN}, {0xb7f86fe0, [], SA_RESTART}, 8) = 0
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
select(1, [0], NULL, NULL, {0, 0})      = 0 (Timeout)
write(1, "\01013\33[3;6H1.00 TxURL     /\33[4;6H1"..., 584) = 584
rt_sigaction(SIGTSTP, {0xb7f86fe0, [], SA_RESTART}, NULL, 8) = 0
gettimeofday({1199697925, 209013}, NULL) = 0
select(1, [0], NULL, NULL, {1, 0})      = 0 (Timeout)
gettimeofday({1199697926, 224518}, NULL) = 0
futex(0x804af08, FUTEX_WAIT, 2, NULL

# varnishd -V
varnishd (varnish-1.1.2)
Copyright (c) 2006-2007 Linpro AS / Verdens Gang AS

Running ubuntu
# uname -a 
Linux web1 2.6.20-16-generic #2 SMP Sun Sep 23 19:50:39 UTC 2007 i686 GNU/Linux

Im using varnish from the .tar.gz package on SF. 

Thank you
Erik



From phk at phk.freebsd.dk  Mon Jan  7 09:43:29 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 07 Jan 2008 09:43:29 +0000
Subject: varnishtop hangs 
In-Reply-To: Your message of "Mon, 07 Jan 2008 10:34:45 +0100."
	<vx8f8g4j4c483hq.070120081034@torlen.net> 
Message-ID: <8961.1199699009@critter.freebsd.dk>

In message <vx8f8g4j4c483hq.070120081034 at torlen.net>, Erik writes:

>I do the following in varnishtop: varnishtop -i TxURL

The trace doesn't look wrong as far as I can see.

Your problem may simply be that there isn't enough backend traffic
for varnishtop to have anything to show.

Or possibly, that varnishtop is looking at the wrong shared memory
area if you have multiple varnish instances (remember to specify
the same instance name).

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chulmin2 at hotmail.com  Mon Jan  7 11:45:02 2008
From: chulmin2 at hotmail.com (MontyRee)
Date: Mon, 7 Jan 2008 11:45:02 +0000
Subject: hiding varnish headers from requests
In-Reply-To: <4aa1ddab0712131623y421558f3r18013df6dea528f3@mail.gmail.com>
References: <18219.16591.185911.307821@localhost.localdomain>
	<4aa1ddab0712131623y421558f3r18013df6dea528f3@mail.gmail.com>
Message-ID: <BLU116-W44FEAA8D7F52DE366C6FB3854F0@phx.gbl>


Hello, list.
 
I have set like below at varnishd 1.1.2  and works well.
But like below just connect to port 80 and quit, we can see all headers.
 
Any idea?
 
 
 
# telnet 127.0.0.1 80Trying 127.0.0.1...Connected to 127.0.0.1.Escape character is '^]'.quit
 
HTTP/1.0 400 Bad RequestServer: VarnishRetry-After: 0Content-Type: text/html; charset=utf-8Content-Length: 428Date: Mon, 07 Jan 2008 11:42:33 GMTX-Varnish: 1556702737Age: nanVia: 1.1 varnishConnection: keep-alive
 
 
Thanks 


Date: Thu, 13 Dec 2007 16:23:02 -0800From: jbittman at chewcorp.comTo: dwetzel at nerim.netSubject: Re: hiding varnish headers from requestsCC: varnish-misc at projects.linpro.noEdit your vcl file and modify deliver like so.sub vcl_deliver {    remove resp.http.X-Varnish;    remove resp.http.Via;}
On 11/2/07, Damien Wetzel <dwetzel at nerim.net> wrote: 
Hello,I wondered if there is a way to prevent Varnish fromputing headers like :X-varnishVia: 1.1 varnishin its responses to clients.thanks,--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL (ATANAR TECHNOLOGIES)        ("`-/")_.-'"``-._http://www.atanar.com                      . . `; -._    )-;-,_`)                                          (v_,)'  _  )`-.\  ``-' Phone:+33 6 62 29 61 77                   _.- _..-_/ / ((.'- So much to do, so little time -       ((,.-'   ((,/~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_______________________________________________ varnish-misc mailing listvarnish-misc at projects.linpro.nohttp://projects.linpro.no/mailman/listinfo/varnish-misc -- Jobe BittmanChief Network ArchitectZenGarden, Inc.(858) 200-6099 
_________________________________________________________________
MSN ???? ??? ??, Windows Live Messenger!
http://windowslive.msn.co.kr/wlm/messenger/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080107/c8bf5183/attachment.html>

From duja at torlen.net  Mon Jan  7 12:17:52 2008
From: duja at torlen.net (Erik)
Date: Mon, 7 Jan 2008 13:17:52 +0100
Subject: varnishtop hangs
Message-ID: <i2szdwej8zo3pz5.070120081317@torlen.net>


>I do the following in varnishtop: varnishtop -i TxURL

The trace doesn't look wrong as far as I can see.

Your problem may simply be that there isn't enough backend traffic
for varnishtop to have anything to show.

Or possibly, that varnishtop is looking at the wrong shared memory
area if you have multiple varnish instances (remember to specify
the same instance name).

---------------------------------
I'm making alot of requests and the varnishtop shows the correct stats but after a while with some certain tags (e.g TxURL) it just hangs. I cannot make e Ctrl+C or anything, simply dead.

Im running only one varnishinstance. Can I provide some more info?

BTW. This subject: "hash.url missing in varnish 1.1.2", do you have an answer of that one?

Regards
Erik



From ssm at linpro.no  Tue Jan  8 06:10:12 2008
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Tue, 08 Jan 2008 07:10:12 +0100
Subject: hiding varnish headers from requests
In-Reply-To: <BLU116-W44FEAA8D7F52DE366C6FB3854F0@phx.gbl>
	(chulmin2@hotmail.com's message of "Mon, 7 Jan 2008 11:45:02 +0000")
References: <18219.16591.185911.307821@localhost.localdomain>
	<4aa1ddab0712131623y421558f3r18013df6dea528f3@mail.gmail.com>
	<BLU116-W44FEAA8D7F52DE366C6FB3854F0@phx.gbl>
Message-ID: <7x3at8j1a3.fsf@iostat.linpro.no>

MontyRee <chulmin2 at hotmail.com> writes:

> Hello, list.
>  
> I have set like below at varnishd 1.1.2 and works well.  But like
> below just connect to port 80 and quit, we can see all headers.
>  
> Any idea?

Your "quit" request, which is not a valid HTTP request, causes Varnish
to respond with an error message.

This error message contains the response "HTTP/1.0 400 Bad Request",
as well as a set of headers describing the format of the error
message, and information that can be used to find the cause of the
error.

Your example shows that you are using telnet to connect to Varnish.
The proper way to exit this client is not to type "quit", but rather
with the escape character '^]', followed by the command "close", which
may also be abbreviated to "c".  For more information, see your
system's telnet(1) man page.

-- 
Stig Sandbeck Mathisen, Linpro


From phk at phk.freebsd.dk  Tue Jan  8 09:16:58 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 08 Jan 2008 09:16:58 +0000
Subject: varnishtop hangs 
In-Reply-To: Your message of "Mon, 07 Jan 2008 13:17:52 +0100."
	<i2szdwej8zo3pz5.070120081317@torlen.net> 
Message-ID: <28209.1199783818@critter.freebsd.dk>

In message <i2szdwej8zo3pz5.070120081317 at torlen.net>, Erik writes:

>I'm making alot of requests and the varnishtop shows the correct=
> stats but after a while with some certain tags (e.g TxURL) it=
> just hangs. I cannot make e Ctrl+C or anything, simply dead.

Hmm, I wonder if your shmlog segment is overrun too fast.  In that
case all the shmlog tailing programs can get confused.

How much traffic does your server take ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Tue Jan  8 10:26:54 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 08 Jan 2008 10:26:54 +0000
Subject: Another Varnish 1.1.2 crash, in cache_backend.c 
In-Reply-To: Your message of "Thu, 03 Jan 2008 08:35:13 +0100."
	<20080103073513.GA99459@fupp.net> 
Message-ID: <54427.1199788014@critter.freebsd.dk>

In message <20080103073513.GA99459 at fupp.net>, Anders Nordby writes:
>Hi,
>
>(gdb) bt
>#0  0x28129f37 in thr_kill () from /lib/libc.so.6
>#1  0x280cf1a5 in pthread_mutex_unlock () from /usr/lib/libthr.so.2
>#2  0x280c72ae in raise () from /usr/lib/libthr.so.2
>#3  0x281a4b78 in abort () from /lib/libc.so.6
>#4  0x280aa0ed in lbv_assert (func=0x806ed70 "VBE_ClosedFd", 
>    file=0x806ebfb "cache_backend.c", line=357, 
>    cond=0x806edba "(close(vc->fd)) == 0", err=22) at assert.c:58
>#5  0x0804fbdf in VBE_ClosedFd (w=0xb3b3ecf0, vc=0xb1ce040)
>    at cache_backend.c:357
>#6  0x0805a713 in PipeSession (sp=0x9099004) at cache_pipe.c:137
>#7  0x080511df in cnt_pipe (sp=0x9099004) at cache_center.c:668
>#8  0x080517c8 in CNT_Session (sp=0x9099004) at steps.h:35
>#9  0x080590e2 in wrk_do_one (w=0xb3b3ecf0) at cache_pool.c:194
>#10 0x08059650 in wrk_thread (priv=0x807e1f0) at cache_pool.c:248
>#11 0x280c95cf in pthread_create () from /usr/lib/libthr.so.2
>#12 0xb8489fec in ?? ()
>
>Got this one during the night.

Was there any errno information available ?  I belive the assert
message would have printed it ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From anders at fupp.net  Tue Jan  8 13:01:58 2008
From: anders at fupp.net (Anders Nordby)
Date: Tue, 8 Jan 2008 14:01:58 +0100
Subject: Another Varnish 1.1.2 crash, in cache_backend.c
In-Reply-To: <54427.1199788014@critter.freebsd.dk>
References: <20080103073513.GA99459@fupp.net>
	<54427.1199788014@critter.freebsd.dk>
Message-ID: <20080108130158.GA39925@fupp.net>

Hi,

On Tue, Jan 08, 2008 at 10:26:54AM +0000, Poul-Henning Kamp wrote:
> Was there any errno information available ?  I belive the assert
> message would have printed it ?

Nope, no assert.

Got two more of them here:

Case 1:

(gdb) bt
#0  0x0000000000408a90 in vbe_sock_conn (ai=0x0) at cache_backend.c:162
#1  0x0000000000408b98 in vbe_conn_try (bp=0xaf2d00, pai=0x7ffffadd5838)
    at cache_backend.c:190
#2  0x0000000000408d14 in vbe_connect (sp=0xb4d008, bp=0xaf2d00)
    at cache_backend.c:228
#3  0x000000000040912e in vbe_nextfd (sp=0xb4d008) at
cache_backend.c:299
#4  0x0000000000409372 in VBE_GetFd (sp=0xb4d008) at cache_backend.c:336
#5  0x000000000040cafa in Fetch (sp=0xb4d008) at cache_fetch.c:278
#6  0x000000000040a124 in cnt_fetch (sp=0xb4d008) at cache_center.c:300
#7  0x000000000040b18d in CNT_Session (sp=0xb4d008) at steps.h:40
#8  0x0000000000412cf8 in wrk_do_one (w=0x7ffffadd5ad0) at
cache_pool.c:194
#9  0x00000000004131f3 in wrk_thread (priv=0x856300) at cache_pool.c:248
#10 0x000000080098755b in pthread_create () from /usr/lib/libthr.so.2
#11 0x00007ffffabd8000 in ?? ()
Cannot access memory at address 0x7ffffadd8000

Child said (2, 51256): <<Child starts
managed to mmap 161061273600 bytes of 161061273600
Ready
CLI ready
>>
Cache child died pid=51256 status=0x8b
Clean child
Child cleaned

Case 2:

(gdb) bt
#0  0x0000000000408a90 in vbe_sock_conn (ai=0x0) at cache_backend.c:162
#1  0x0000000000408b98 in vbe_conn_try (bp=0xaf2c80, pai=0x7ffffd1e7838)
    at cache_backend.c:190
#2  0x0000000000408d14 in vbe_connect (sp=0xb85008, bp=0xaf2c80)
    at cache_backend.c:228
#3  0x000000000040912e in vbe_nextfd (sp=0xb85008) at
cache_backend.c:299
#4  0x0000000000409372 in VBE_GetFd (sp=0xb85008) at cache_backend.c:336
#5  0x000000000040cafa in Fetch (sp=0xb85008) at cache_fetch.c:278
#6  0x000000000040a124 in cnt_fetch (sp=0xb85008) at cache_center.c:300
#7  0x000000000040b18d in CNT_Session (sp=0xb85008) at steps.h:40
#8  0x0000000000412cf8 in wrk_do_one (w=0x7ffffd1e7ad0) at
cache_pool.c:194
#9  0x00000000004131f3 in wrk_thread (priv=0x856300) at cache_pool.c:248
#10 0x000000080098755b in pthread_create () from /usr/lib/libthr.so.2
#11 0x00007ffffcfea000 in ?? ()
Cannot access memory at address 0x7ffffd1ea000

Child said (2, 52204): <<Child starts
managed to mmap 161061273600 bytes of 161061273600
Ready
CLI ready
>>
Cache child died pid=52204 status=0x8b
Clean child
Child cleaned
start child pid 53434

I'm running FreeBSD 6.3-RC2/amd64 where I get this.

Bye,

-- 
Anders.


From phk at phk.freebsd.dk  Tue Jan  8 14:42:08 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 08 Jan 2008 14:42:08 +0000
Subject: Varnish1.1.2 returns 200OK,but no content 
In-Reply-To: Your message of "Sun, 23 Dec 2007 10:49:30 +0800."
	<20071223024223.M58534@wst.net.cn> 
Message-ID: <78611.1199803328@critter.freebsd.dk>

In message <20071223024223.M58534 at wst.net.cn>, "game" writes:

Please don't send HTML mail, I can't read it in my antique mailprogram.

(Yeah, I can see the irony, but I'm not going to change mailprogram
just because of some modern fad like HTML :-)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From dwetzel at nerim.net  Tue Jan  8 15:21:46 2008
From: dwetzel at nerim.net (Damien Wetzel)
Date: Tue, 8 Jan 2008 16:21:46 +0100
Subject: Varnish1.1.2 returns 200OK,but no content 
In-Reply-To: <78611.1199803328@critter.freebsd.dk>
References: <20071223024223.M58534@wst.net.cn>
	<78611.1199803328@critter.freebsd.dk>
Message-ID: <18307.38154.28654.207803@localhost.localdomain>

Hello Poul-Henning,
I'm in the same situation than you, i'm
using VM under Xemacs and i wonder how long i will
be able to keep it as my mail program since i have difficulties
to read 1 mail out of 4 :)

Poul-Henning Kamp writes:
 > In message <20071223024223.M58534 at wst.net.cn>, "game" writes:
 > 
 > Please don't send HTML mail, I can't read it in my antique mailprogram.
 > 
 > (Yeah, I can see the irony, but I'm not going to change mailprogram
 > just because of some modern fad like HTML :-)
 > 
 > -- 
 > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 > phk at FreeBSD.ORG         | TCP/IP since RFC 956
 > FreeBSD committer       | BSD since 4.3-tahoe    
 > Never attribute to malice what can adequately be explained by incompetence.
 > _______________________________________________
 > varnish-misc mailing list
 > varnish-misc at projects.linpro.no
 > http://projects.linpro.no/mailman/listinfo/varnish-misc


From des at linpro.no  Tue Jan  8 15:35:45 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 16:35:45 +0100
Subject: disable sendfile in Varnish, please
In-Reply-To: <EEF3518D-EA59-4B21-9CB0-D3F71075E8DA@anduin.net> (Eirik
	=?utf-8?Q?=C3=98verby's?=
	message of "Mon, 10 Dec 2007 15:23:22 +0100")
References: <5362.1178896342@critter.freebsd.dk>
	<EEF3518D-EA59-4B21-9CB0-D3F71075E8DA@anduin.net>
Message-ID: <ujrsl18fhym.fsf@login.linpro.no>

Eirik ?verby <ltning at anduin.net> writes:
> is this still a problem?

[a little late, but] yes, even more so now that we discovered that
sendfile() semantics are different from what we expected (the devil is
in the undocumented details) and that even with all known bugs fixed,
sendfile() may result in corrupted data being sent to the client.

This may be possible to fix by changing the way Varnish uses
sendfile(), but that's not currently a priority.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 15:42:55 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 16:42:55 +0100
Subject: purge_url
In-Reply-To: <f2u4qbu8zy4kwmw.131220071202@torlen.net> (duja@torlen.net's
	message of "Thu, 13 Dec 2007 12:02:38 +0100")
References: <f2u4qbu8zy4kwmw.131220071202@torlen.net>
Message-ID: <ujrodbwfhmo.fsf@login.linpro.no>

Erik <duja at torlen.net> writes:
> [redacted]
>
> sub vcl_recv {
>         if (req.http.host == "10.1.1.54" || req.http.host == "test.mysite.com") {
>                 set req.http.host = "www.mysite.com";
>         }
>
>         if (req.request == "PURGE") {
>                 if (client.ip ~ purge) {
>                         purge_url(req.url);
>                 }
>         }
> }
>
> sub vcl_miss {
>         if (req.request == "PURGE") {
>                 error 404 "The url could not be found in the cache.";
>         }
> }
>
> sub vcl_hit {
>         if (req.request == "PURGE") {
>                 set obj.ttl = 0s;
>                 error 200 "Purged successfully!";
>         }
> }

You are mixing two different purge mechanisms.  I think what you want
is listed in the EXAMPLES section in the vcl(7) manual page.

Also, the PURGE request you showed did not include a Host: header at
all, so your host normalization code never kicked in, and you tried to
purge a different document from the one you later requested.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 15:46:39 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 16:46:39 +0100
Subject: varnishd 1.1.2 not useing epoll in kernel 2.6
In-Reply-To: <e15793d10712200800j184dec4ci8a8746b487a17877@mail.gmail.com>
	(wangmd0127@gmail.com's message of "Fri, 21 Dec 2007 00:00:29 +0800")
References: <e15793d10712200800j184dec4ci8a8746b487a17877@mail.gmail.com>
Message-ID: <ujrk5mkfhgg.fsf@login.linpro.no>

"[illegible]" <wangmd0127 at gmail.com> writes:
> I install varnishd 1.1.2 in Red Hat AS4.4 (kernel: 2.6.9-42.ELsmp).
> I'm very strange, it doesn't useing epoll.

Yes it does, just not for the management interface.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 15:51:17 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 16:51:17 +0100
Subject: problems with varnish 1.1.2
In-Reply-To: <33667D03-8703-446B-99DE-DE10C759A0FF@zeno.org> (Denis Ahrens's
	message of "Fri, 21 Dec 2007 17:39:57 +0100")
References: <33667D03-8703-446B-99DE-DE10C759A0FF@zeno.org>
Message-ID: <ujrfxx8fh8q.fsf@login.linpro.no>

Denis Ahrens <denis at zeno.org> writes:
> I tried today to switch to varnish 1.1.2 but have a problem.
>
> [continues to provide almost none of the information required to
> diagnose and solve the problem]

Please.  Send.  Logs.

> GET http://www.test.org/index.html HTTP/1.1
>

Not a valid HTTP/1.1 request.  Varnish should actually 400 that one.

> HTTP/1.1 200 OK
> Server: ZenoServer/2.1.10 * DBSERVER/r1456 * Middleware/1444
> Vary: Accept-Encoding
> Content-Type: text/html
> Cache-Control: s-maxage=300, max-age=120
> Last-Modified: Fri, 21 Dec 2007 15:20:42 GMT
> Date: Fri, 21 Dec 2007 16:33:38 GMT
> X-Varnish: 1220401610
> Age: 0
> Via: 1.1 varnish
> Connection: keep-alive
>
> Connection closed by foreign host.

You got your response.  No Content-Length and no Content-Encoding
means no body.  Varnish waited for a second request but grew tired and
hanged up on you.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 16:02:00 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 17:02:00 +0100
Subject: Two New HTTP Caching Extensions
In-Reply-To: <63a3d0be0712280625i7cdd347du7c5b215bd7b50b2b@mail.gmail.com>
	(paul.marsh@gmail.com's message of "Fri, 28 Dec 2007 09:25:45 -0500")
References: <63a3d0be0712280625i7cdd347du7c5b215bd7b50b2b@mail.gmail.com>
Message-ID: <ujrbq7wfgqv.fsf@login.linpro.no>

Paul <paul.marsh at gmail.com> writes:
> Found these ideas very interesting, made it into Squid 2.7. Was
> curious if Varnish would ever support similar functionality.
>
> http://www.mnot.net/blog/2007/12/12/stale

Isn't it fascinating to see the Squid developers discussing this
approximately 18 months after we put it on our roadmap?  It was
actually planned for Varnish 1.0, but was left out as it turned out
not to be necessary for vg.no (which was our target at the time)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 16:15:04 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 17:15:04 +0100
Subject: Another Varnish 1.1.2 crash, in cache_backend.c
In-Reply-To: <20080108130158.GA39925@fupp.net> (Anders Nordby's message of
	"Tue, 8 Jan 2008 14:01:58 +0100")
References: <20080103073513.GA99459@fupp.net>
	<54427.1199788014@critter.freebsd.dk>
	<20080108130158.GA39925@fupp.net>
Message-ID: <ujr7iikfg53.fsf@login.linpro.no>

Anders Nordby <anders at fupp.net> writes:
> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
> > Was there any errno information available ?  I belive the assert
> > message would have printed it ?
> Nope, no assert.

Yes there was, let me show the backtrace which phk missed and you
edited out of your reply:

#0  0x28129f37 in thr_kill () from /lib/libc.so.6
#1  0x280cf1a5 in pthread_mutex_unlock () from /usr/lib/libthr.so.2
#2  0x280c72ae in raise () from /usr/lib/libthr.so.2
#3  0x281a4b78 in abort () from /lib/libc.so.6
#4  0x280aa0ed in lbv_assert (func=0x806ed70 "VBE_ClosedFd",
    file=0x806ebfb "cache_backend.c", line=357,
    cond=0x806edba "(close(vc->fd)) == 0", err=22) at assert.c:58
#5  0x0804fbdf in VBE_ClosedFd (w=0xb3b3ecf0, vc=0xb1ce040)
    at cache_backend.c:357

errno 22 is EINVAL.

Similar issues in trunk were addressed by a number of commits
(including r2264 and r2285) which unfortunately can't be merged back
to 1.1, because the code has been completely reorganized in trunk.
They need to be reimplemented for 1.1.

> Got two more of them here:

Actually, those are two almost identical instances of a completely
different bug, a segfault in vbe_sock_conn() due to vbe_conn_try()
passing a NULL pointer:

> (gdb) bt
> #0  0x0000000000408a90 in vbe_sock_conn (ai=0x0) at cache_backend.c:162
> #1  0x0000000000408b98 in vbe_conn_try (bp=0xaf2d00, pai=0x7ffffadd5838)
>     at cache_backend.c:190
> #2  0x0000000000408d14 in vbe_connect (sp=0xb4d008, bp=0xaf2d00)
>     at cache_backend.c:228

The fault lies in the following loop on bin/varnishd/cache_backend.c
lines 188-196:

        /* Then try the list until the cached last good address */
        for (ai = bp->addr; ai != bp->last_addr; ai = ai->ai_next) {
                s = vbe_sock_conn(ai);
                if (s >= 0) {
                        bp->last_addr = ai;
                        *pai = ai;
                        return (s);
                }
        }

The loop condition should check that ai != NULL.

There may or may not be a race condition at the bottom of this
(between vbe_conn_try() and something else modifying the backend
struct that bp points to)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 16:39:02 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 17:39:02 +0100
Subject: hash.url missing in varnish 1.1.2
In-Reply-To: <mhaw28zotjxo12o.040120081457@torlen.net> (duja@torlen.net's
	message of "Fri, 4 Jan 2008 14:57:47 +0100")
References: <mhaw28zotjxo12o.040120081457@torlen.net>
Message-ID: <ujr3at8ff15.fsf@login.linpro.no>

Erik <duja at torlen.net> writes:
> Downloaded and installed 1.1.2 from SF and noticed that hash.url has
> been removed from varnishadm. Didn't find anything about that on
> http://varnish.projects.linpro.no/wiki/Releases/1.1.2 .
>
> Why has it been removed?

There is no such thing as "hash.url" in Varnish (that I am aware of).

Assuming you mean the hash.purge management command, it has never been
in 1.1 - only in trunk and the as-yet unreleased 1.2.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jan  8 16:46:25 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 08 Jan 2008 17:46:25 +0100
Subject: Varnish1.1.2 returns 200OK,but no content
In-Reply-To: <20071223024223.M58534@wst.net.cn> (game@wst.net.cn's message of
	"Sun, 23 Dec 2007 10:49:30 +0800")
References: <20071223024223.M58534@wst.net.cn>
Message-ID: <ujry7b0e04e.fsf@login.linpro.no>

"game" <game at wst.net.cn> writes:
> I'm running into some trouble upgrading from 1.1.1 to 1.1.2.
> It seems to be the same one with http://projects.linpro.no/pipermail/varnish-
> misc/2007-December/001259.html

No, it is completely unrelated.  What Denis is seeing is a combination
of a backend bug and pilot error.  What you are seeing is a backend
bug which 1.2 apparently manages to recover from but 1.1.2 doesn't.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From duja at torlen.net  Tue Jan  8 16:51:30 2008
From: duja at torlen.net (Erik Torlen)
Date: Tue, 08 Jan 2008 17:51:30 +0100
Subject: varnishtop hangs
In-Reply-To: <28209.1199783818@critter.freebsd.dk>
References: <28209.1199783818@critter.freebsd.dk>
Message-ID: <4783AA12.5060702@torlen.net>

Its not near an overload when it hangs. I do some requests against it 
and *poff*, varnishtop hangs and
I can't even close it with ctrl+c. If you have any requests on some more 
info am more then glad to help.
Cheers
Erik

Poul-Henning Kamp skrev:
> In message <i2szdwej8zo3pz5.070120081317 at torlen.net>, Erik writes:
>
>   
>> I'm making alot of requests and the varnishtop shows the correct=
>> stats but after a while with some certain tags (e.g TxURL) it=
>> just hangs. I cannot make e Ctrl+C or anything, simply dead.
>>     
>
> Hmm, I wonder if your shmlog segment is overrun too fast.  In that
> case all the shmlog tailing programs can get confused.
>
> How much traffic does your server take ?
>
>   



From linuxbsdfreak at gmail.com  Wed Jan  9 10:25:21 2008
From: linuxbsdfreak at gmail.com (kevin fernandes)
Date: Wed, 9 Jan 2008 15:55:21 +0530
Subject: varnishtop hangs
In-Reply-To: <4783AA12.5060702@torlen.net>
References: <28209.1199783818@critter.freebsd.dk> <4783AA12.5060702@torlen.net>
Message-ID: <1c752e950801090225q7fe10073h134ab302d59548b9@mail.gmail.com>

Hello All,

I am running varnish and lighttpd. lighttpd webserver runs on port 8080 and
on localhost. Varnish runs on port 80 and on the public ip. While checking
the access.log of lighttpd. The webrequests are shown to come from 127.0.0.1.
I would like to know wht configruation changes are required in varnish to
log the remote client ip in access.log.

Kevin

On Jan 8, 2008 10:21 PM, Erik Torlen <duja at torlen.net> wrote:

> Its not near an overload when it hangs. I do some requests against it
> and *poff*, varnishtop hangs and
> I can't even close it with ctrl+c. If you have any requests on some more
> info am more then glad to help.
> Cheers
> Erik
>
> Poul-Henning Kamp skrev:
> > In message <i2szdwej8zo3pz5.070120081317 at torlen.net>, Erik writes:
> >
> >
> >> I'm making alot of requests and the varnishtop shows the correct=
> >> stats but after a while with some certain tags (e.g TxURL) it=
> >> just hangs. I cannot make e Ctrl+C or anything, simply dead.
> >>
> >
> > Hmm, I wonder if your shmlog segment is overrun too fast.  In that
> > case all the shmlog tailing programs can get confused.
> >
> > How much traffic does your server take ?
> >
> >
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080109/f9147e92/attachment.html>

From phk at phk.freebsd.dk  Wed Jan  9 10:27:55 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 09 Jan 2008 10:27:55 +0000
Subject: varnishtop hangs 
In-Reply-To: Your message of "Wed, 09 Jan 2008 15:55:21 +0530."
	<1c752e950801090225q7fe10073h134ab302d59548b9@mail.gmail.com> 
Message-ID: <82951.1199874475@critter.freebsd.dk>

In message <1c752e950801090225q7fe10073h134ab302d59548b9 at mail.gmail.com>, "kevi
n fernandes" writes:
>------=_Part_44100_26483652.1199874321414
>Content-Type: text/plain; charset=ISO-8859-1
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline
>
>Hello All,
>
>I am running varnish and lighttpd. lighttpd webserver runs on port 8080 and
>on localhost. Varnish runs on port 80 and on the public ip. While checking
>the access.log of lighttpd. The webrequests are shown to come from 127.0.0.1.
>I would like to know wht configruation changes are required in varnish to
>log the remote client ip in access.log.

Varnish sends a header "X-Forwarded-for:" to lightttpd which it should
use in the logfile.  You need to check the lighttpd config how to enable that.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From gslin at gslin.org  Wed Jan  9 16:40:32 2008
From: gslin at gslin.org (Gea-Suan Lin)
Date: Thu, 10 Jan 2008 00:40:32 +0800
Subject: varnishtop hangs
In-Reply-To: <1c752e950801090225q7fe10073h134ab302d59548b9@mail.gmail.com>
References: <28209.1199783818@critter.freebsd.dk> <4783AA12.5060702@torlen.net>
	<1c752e950801090225q7fe10073h134ab302d59548b9@mail.gmail.com>
Message-ID: <20080109164032.GA79528@gslin.org>

You can use mod_extforward to get real IP address:

http://trac.lighttpd.net/trac/wiki/Docs/ModExtForward

On Wed, Jan 09, 2008 at 03:55:21PM +0530, kevin fernandes wrote:
> Hello All,
> 
> I am running varnish and lighttpd. lighttpd webserver runs on port 8080 and
> on localhost. Varnish runs on port 80 and on the public ip. While checking
> the access.log of lighttpd. The webrequests are shown to come from 127.0.0.1.
> I would like to know wht configruation changes are required in varnish to
> log the remote client ip in access.log.
> 
> Kevin

-- 
* Gea-Suan Lin                                    http://blog.gslin.org/
* If you cannot convince them, confuse them.           -- Harry S Truman


From Phil.Cryer at edwardjones.com  Wed Jan  9 16:35:58 2008
From: Phil.Cryer at edwardjones.com (Cryer,Phil)
Date: Wed, 9 Jan 2008 10:35:58 -0600
Subject: varnishtop hangs 
In-Reply-To: <82951.1199874475@critter.freebsd.dk>
References: Your message of "Wed, 09 Jan 2008 15:55:21
	+0530."<1c752e950801090225q7fe10073h134ab302d59548b9@mail.gmail.com>
	<82951.1199874475@critter.freebsd.dk>
Message-ID: <3ECD7F7DDE95BA4FA598E8DDE71F1A51063EAD63@nwpsrv08.edj.ad.edwardjones.com>

<snip>
>>and on localhost. Varnish runs on port 80 and on the public ip. While 
>>checking the access.log of lighttpd. The webrequests are shown to come
from 127.0.0.1.
>>I would like to know wht configruation changes are required in varnish

>>to log the remote client ip in access.log.

>Varnish sends a header "X-Forwarded-for:" to lightttpd which it should
use in the logfile.  You need 
>to check the lighttpd config how to enable that.

In your lighttpd.conf, add:

accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
\"%{Referer}i\" \"%{User-Agent}i\""

http://fak3r.com/2008/01/09/howto-log-the-users-ip-not-the-proxys-in-lig
httpd-access-log/

Thanks for bringing this up, I finally got around to fixing it!

P
 
 If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments.  If you no longer wish to receive e-mail from Edward Jones, please send this request to messages at edwardjones.com.  You must include the e-mail address that you wish not to receive e-mail communications.  For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure


From chulmin2 at hotmail.com  Thu Jan 10 04:32:09 2008
From: chulmin2 at hotmail.com (MontyRee)
Date: Thu, 10 Jan 2008 04:32:09 +0000
Subject: What's the meaning of req.http.Expect?
Message-ID: <BLU116-W270839BDEE99BC1BB0D98D854A0@phx.gbl>


Hello, list.
 
 
I can see req.http.Expect like below from the lots of vcl example.
  if (req.http.Expect) {                  pipe;              }
Then, what's the meaning of the req.http.Expect?
 
 
Thanks in advance.
 
_________________________________________________________________
MSN ???? ??? ??, Windows Live Messenger!
http://windowslive.msn.co.kr/wlm/messenger/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080110/16da4a79/attachment.html>

From phk at phk.freebsd.dk  Thu Jan 10 08:18:24 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 10 Jan 2008 08:18:24 +0000
Subject: What's the meaning of req.http.Expect? 
In-Reply-To: Your message of "Thu, 10 Jan 2008 04:32:09 GMT."
	<BLU116-W270839BDEE99BC1BB0D98D854A0@phx.gbl> 
Message-ID: <88782.1199953104@critter.freebsd.dk>

In message <BLU116-W270839BDEE99BC1BB0D98D854A0 at phx.gbl>, MontyRee writes:

>I can see req.http.Expect like below from the lots of vcl example.
>  if (req.http.Expect) {                  pipe;              }
>Then, what's the meaning of the req.http.Expect?

The test above is "If the request has a "Expect:" HTTP header"

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Thu Jan 10 08:50:54 2008
From: duja at torlen.net (Erik)
Date: Thu, 10 Jan 2008 09:50:54 +0100
Subject: What's the meaning of req.http.Expect?
Message-ID: <fan5p0wc8kipve6.100120080950@torlen.net>

The expect header:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20

Original Message -----------------------
In message <BLU116-W270839BDEE99BC1BB0D98D854A0 at phx.gbl>, MontyRee writes:

>I can see req.http.Expect like below from the lots of vcl example.
>  if (req.http.Expect) {                  pipe;              }
>Then, what's the meaning of the req.http.Expect?

The test above is "If the request has a "Expect:" HTTP header"

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-misc mailing list
varnish-misc at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc



From james at nyi.net  Thu Jan 10 23:26:14 2008
From: james at nyi.net (james)
Date: Thu, 10 Jan 2008 18:26:14 -0500
Subject: Question about Cache Size
Message-ID: <4786A996.1030101@nyi.net>

Hey everyone,

In trying to diagnose why the varnish child dies at seemingly random 
times, I started varnish with -d -d and waited for a crash. This is what 
I got in the output:

Child said (2, 16094): <<Assert error in STV_alloc(), stevedore.c line 70:
  Condition((st) != NULL) not true.
  errno = 107 (Transport endpoint is not connected)
 >>
Cache child died pid=16094 status=0x6
Clean child
Child cleaned
start child pid 18886
Child said (2, 18886): <<Child starts
managed to mmap 565284864 bytes of 565284864
Ready
CLI ready
 >>


Any idea what this might be an indication of? Also, when starting 
varnish from the command line (and not an init.d script), I noticed this 
output which I haven't noticed before:

[root at www ~]# /usr/local/sbin/varnishd -a <ipaddr>:8000 -T <ipaddr>:8080 
-s file,/var/cache/varnish/,10G -p group=nobody -p thread_pool_max=1500 
-p thread_pools=5 -p listen_depth=512 -p client_http11=on -p 
backend_http11=on -p default_ttl=36000 -P /var/run/varnishd.pid -f 
/etc/varnish.conf
WARNING: storage file size reduced to 1342177280 due to system limitations
WARNING: storage file size reduced to 599837900 (80% of available disk 
space)
file /var/cache/varnish//varnish.z8brik (unlinked) size 599834624 bytes 
(146444 fs-blocks, 146444 pages)
Using old SHMFILE


However, 80% of my available disk space is not 599834624 bytes ( 572 
Mbytes ).

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             131G   16G  109G  13% /
/dev/sda1              99M   15M   79M  16% /boot
tmpfs                 1.8G     0  1.8G   0% /dev/shm

Any idea if these things are related? Currently running varnishd-1.2

Thanks!

-- james


From quigley at emerose.com  Thu Jan 10 23:48:17 2008
From: quigley at emerose.com (Sam Quigley)
Date: Thu, 10 Jan 2008 15:48:17 -0800
Subject: Two New HTTP Caching Extensions
In-Reply-To: <ujrbq7wfgqv.fsf@login.linpro.no>
References: <63a3d0be0712280625i7cdd347du7c5b215bd7b50b2b@mail.gmail.com>
	<ujrbq7wfgqv.fsf@login.linpro.no>
Message-ID: <4232A186-6F90-4713-8723-0925E4BD286A@emerose.com>

...just thought I'd point out another seemingly-nifty thing the Squid  
folks are working on:

http://www.mnot.net/cache_channels/
and
http://www.mnot.net/blog/2008/01/04/cache_channels

-sq


From phk at phk.freebsd.dk  Fri Jan 11 08:50:04 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 11 Jan 2008 08:50:04 +0000
Subject: Two New HTTP Caching Extensions 
In-Reply-To: Your message of "Thu, 10 Jan 2008 15:48:17 PST."
	<4232A186-6F90-4713-8723-0925E4BD286A@emerose.com> 
Message-ID: <3503.1200041404@critter.freebsd.dk>

In message <4232A186-6F90-4713-8723-0925E4BD286A at emerose.com>, Sam Quigley writ
es:
>...just thought I'd point out another seemingly-nifty thing the Squid  
>folks are working on:
>
>http://www.mnot.net/cache_channels/
>and
>http://www.mnot.net/blog/2008/01/04/cache_channels

Interesting to see what hoops they try to jump through these days...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jan 11 10:30:45 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 11 Jan 2008 10:30:45 +0000
Subject: stale delivery and prefetch sanity check...
Message-ID: <4074.1200047445@critter.freebsd.dk>


If any of you have time, your comments to this outline for degraded
mode and prefetching would be appreciated.


Prefetch
--------

Prefetching is easy to dispatch with: at some VCL determined time
before the TTL expires, we try to refresh the object from the backend
so that it never grows stale.

Deciding that prefetching should be an option for an object, must
be decided at fetch time, in order to cache the requesting headers,
at the same time, a time is set for when to prefetch.

The timer can either be absolute, or if negative relative to obj.ttl.

The interesting question is which headers to send to the backend,
and which client to pretend to be.

The simplest solution is probably to replay the headers used to
fetch the object in the first place, but this may wrongly account
the fetch to a particular client/cookie/user/account.

The alternative is to filter headers to the bare minimum, respecting
Vary:, and hope that gives the expected result.

Either way, VCL control is desirable, and an filtering/additive
approach is preferable, since it gives a known set of headers to
work from.

	sub vcl_fetch {
		set obj.prefetch = -2m;
		unset obj.req.http.cookie;
		set obj.req.myhdr = req.myhdr;
	}

When the pretech timer expires, vcl_prefetch{} is invoked to call
the shot:

	sub vcl_prefetch {
		if (obj.uses > 10) {
			prefetch;
		}
		pass;
	}

When prefetching an object, the new incomplete object should not
be marked "busy" in the "hold up" sense of the concept, since we can
reasonably expect a serviceable copy of the object to exists (the one
which triggered the prefetch).

If however, this copy fails to satisfy (eg. Vary:) the prefetching
object should hold up the client request.

XXX: I wonder if we hold up more clients than we need with Vary: today ?

Prefetching should probably be separately quota'ed on the backends,
along with general rate-limiting on these.

Prefetching is not guaranteed to happen in the same VCL as it was
armed, and therefore requests to prefetch must enter in vcl_recv{}
like every other request.  For that reason, the backend used is not
a hard reference, but a soft reference only for affinity.

For the lack of any better idea, I think all prefetching will look like
it happend from a client with IP# 127.0.0.2



Degraded mode
-------------

Degraded mode is the intentional serving of technically stale objects
instead of returning errors.

The condition for returning stale content is:

* Client must be marked as accepting degraded objects (VCL:
  "client.degraded = true", default true) before lookup.

* Object must be within it's timelimit for degraded mode (VCL:
  "obj.stale_time = 1h", default 30 seconds).

* An attempt to fetch the object from the backend must be in progress
  or recently (VCL: "backend.backoff = 1m", default 15 seconds) have failed.

In vcl_fetch, it would be desirable to be able to check good from bad
responses:

	sub vcl_fetch {
		if (obj.status > 200 && oldobj) {
			oldobj.stale_time += 5m;
			set obj = oldobj;
		}
	}

XXX: This requires holding refs/locks on both objects, that may be
too tricky, and possibly a restart is a better way ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From christophe.husson at univ-reims.fr  Fri Jan 11 15:00:49 2008
From: christophe.husson at univ-reims.fr (Christophe Husson)
Date: Fri, 11 Jan 2008 16:00:49 +0100
Subject: Varnish1.1.2 returns 200OK,but no content
References: 20071223024223.M58534@wst.net.cn
Message-ID: <478784A1.5000404@univ-reims.fr>

I've a similar problem with http 1.0 protocol on dynamic objects like
php generated data.

Some log from varnishlog:

15 TxRequest    b GET
15 TxURL        b /bug.php
15 TxProtocol   b HTTP/1.0
...
15 RxProtocol   b HTTP/1.1
15 RxStatus     b 200
15 RxResponse   b OK
15 RxHeader     b Date: Fri, 11 Jan 2008 14:32:33 GMT
15 RxHeader     b Server: Microsoft-IIS/4.0
15 RxHeader     b Connection: close
15 RxHeader     b Content-Type: text/css; charset: UTF-8
13 ObjProtocol  c HTTP/1.1
13 ObjStatus    c 200
13 ObjResponse  c OK
13 ObjHeader    c Date: Fri, 11 Jan 2008 14:32:33 GMT
13 ObjHeader    c Server: -
13 ObjHeader    c Content-Type: text/css; charset: UTF-8
15 BackendClose b default
13 TTL          c 1588262781 RFC 120 1200061953 1200061953 0 0 0
13 VCL_call     c fetch
13 VCL_return   c pass
13 Length       c 0
13 VCL_call     c deliver

The web server (Apache 2) don't send content-length or transfer encoding
chuncked header.
Varnish calculate the objet length only when the get query use http 1.1
protocol (/bin/varnish/cache_fetch.c line 346) or if the server reply
whith a content-length or transfert encoding chunk header.

Can this be fixed ?
Thanks



From duja at torlen.net  Tue Jan 15 12:04:03 2008
From: duja at torlen.net (Erik)
Date: Tue, 15 Jan 2008 13:04:03 +0100
Subject: Find timestamp in log
Message-ID: <hnaf7m28jhrk74n.150120081304@torlen.net>

Im doing a POST request against my varnishserver and getting a very long wait time until the response is received by the client.

As I understand by the vcl man the POST requests are all piped, right?

Anyway, I was looking for some timestamp in the log to understand where the request hangs.

Is their any line in the log where I can see the timestamp/time?

/ Erik



From phk at phk.freebsd.dk  Tue Jan 15 12:06:36 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 15 Jan 2008 12:06:36 +0000
Subject: Find timestamp in log 
In-Reply-To: Your message of "Tue, 15 Jan 2008 13:04:03 +0100."
	<hnaf7m28jhrk74n.150120081304@torlen.net> 
Message-ID: <61652.1200398796@critter.freebsd.dk>

In message <hnaf7m28jhrk74n.150120081304 at torlen.net>, Erik writes:
>Im doing a POST request against my varnishserver and getting a very long wait time until the response is received by the client.
>
>As I understand by the vcl man the POST requests are all piped, right?
>
>Anyway, I was looking for some timestamp in the log to understand where the request hangs.
>
>Is their any line in the log where I can see the timestamp/time?

Which version of varnish are you using ?

We can't really do much logging for a piped request, since all we
do is pass bytes back and forth.

For normal requests timestamps are in the ReqEnd record.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Tue Jan 15 15:46:31 2008
From: duja at torlen.net (Erik)
Date: Tue, 15 Jan 2008 16:46:31 +0100
Subject: Find timestamp in log
Message-ID: <kqhmbyk1xla0cac.150120081646@torlen.net>

Im using 1.1.2 from SF. I noticed in my log that the POST
request isn't piped, it is passed. Probably thats why the
request takes such a long time to finish. 

I thought that all POST were piped auto like this?
if (req.request != "GET" && req.request != "HEAD") {
        pipe;
}

Could you explain the values in the ReqEnd:
   12 ReqEnd       c 100929047 1200397302.365636110 1200397902.687728643 0.016155005 600.287691593 0.034400940

Thanks
Erik

----------- Poul-Henning Kamp wrote --------

Which version of varnish are you using ?

We can't really do much logging for a piped request, since all we
do is pass bytes back and forth.

For normal requests timestamps are in the ReqEnd record.



From fredrik at nettby.no  Tue Jan 15 22:34:32 2008
From: fredrik at nettby.no (Fredrik Kristiansen)
Date: Tue, 15 Jan 2008 23:34:32 +0100
Subject: Prevent varnish from deleting its entire cache(obj.lastuse)
Message-ID: <e9379eda0801151434vd8476e7s8864f9a183a12c10@mail.gmail.com>

Hi,

I'm trying to prevent varnish from deleting its entire cache when full, so I
have added this to our VCL confguration:

sub vcl_discard {
    # Keep objects in use last 5 minutes
    if (obj.lastuse > 300) {
        discard;
    }

    keep;
}

But variable obj.lastuse don't seem to be available and varnish won't start
with this script.

Fredrik Kristiansen
Nettby.no <http://nettby.no/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080115/428a06d9/attachment.html>

From fredrik at nettby.no  Tue Jan 15 22:46:51 2008
From: fredrik at nettby.no (Fredrik Kristiansen)
Date: Tue, 15 Jan 2008 23:46:51 +0100
Subject: How to prevent varnish from deleting its entire cache when full?
Message-ID: <e9379eda0801151446y6386f313h317d01321369effc@mail.gmail.com>

Hi,

I'm trying to prevent varnish from deleting its entire cache when full. So I
have added this our VCL confguration:

sub vcl_discard {
    # Keep objects in use last 5 minutes
    if (obj.lastuse > 300) {
        discard;
    }

    keep;
}

But variable obj.lastuse don't seem to be available and varnish won't start
with this script.

Fredrik Kristiansen
Nettby.no
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080115/4096dc02/attachment.html>

From phk at phk.freebsd.dk  Wed Jan 16 09:58:33 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 16 Jan 2008 09:58:33 +0000
Subject: How to prevent varnish from deleting its entire cache when full? 
In-Reply-To: Your message of "Tue, 15 Jan 2008 23:46:51 +0100."
	<e9379eda0801151446y6386f313h317d01321369effc@mail.gmail.com> 
Message-ID: <67958.1200477513@critter.freebsd.dk>

In message <e9379eda0801151446y6386f313h317d01321369effc at mail.gmail.com>, "Fred
rik Kristiansen" writes:

>I'm trying to prevent varnish from deleting its entire cache when full. So I
>have added this our VCL confguration:

I'm currently working on the prefetch/degraded mode code, and that will
change most if not all of this.

Stay tuned.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Wed Jan 16 10:12:27 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 16 Jan 2008 10:12:27 +0000
Subject: Find timestamp in log 
In-Reply-To: Your message of "Tue, 15 Jan 2008 16:46:31 +0100."
	<kqhmbyk1xla0cac.150120081646@torlen.net> 
Message-ID: <68992.1200478347@critter.freebsd.dk>

In message <kqhmbyk1xla0cac.150120081646 at torlen.net>, Erik writes:

>Could you explain the values in the ReqEnd:
>   12 ReqEnd       c 100929047 1200397302.365636110
> 1200397902.687728643 0.016155005 600.287691593 0.034400940

See:	http://varnish.projects.linpro.no/ticket/44



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Thu Jan 17 11:28:52 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 17 Jan 2008 12:28:52 +0100
Subject: Network outage
Message-ID: <ujrzlv4n11n.fsf@login.linpro.no>

I imagine this won't reach you until the problem has been solved, but
we are currently experiencing technical problems with one of our core
routers.  This does not affect customer systems, which are in a
different location, but it does affect Varnish services, which are in
a DMZ off of our office network.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ivoras at fer.hr  Thu Jan 17 15:35:04 2008
From: ivoras at fer.hr (Ivan Voras)
Date: Thu, 17 Jan 2008 16:35:04 +0100
Subject: Load balancing?
Message-ID: <478F75A8.3080005@fer.hr>

Hi,

I remember there were discussions about possible HTTP load balancing
features in Varnish, but I don't know what was the result. Does Varnish
support load balancing (even simple round-robin would be enough, IP
address hashing would be better)?

If not, does anyone has a recommendation about a nice and stable load
balancer (ideally, it would also have an integrated HTTP accelerator to
reduce the number of components to maintain in the system)?


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 250 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080117/e294544a/attachment.bin>

From max.clark at gmail.com  Thu Jan 17 18:26:43 2008
From: max.clark at gmail.com (Max Clark)
Date: Thu, 17 Jan 2008 10:26:43 -0800
Subject: Migration from Squid
Message-ID: <2fa1e1780801171026s5072b89o71af038514dd11d2@mail.gmail.com>

Hello,

I have been looking at Varnish as a replacement to our Squid
infrastructure for a little while now. I am very impressed with the
application and interested in implemented Varnish as a test for some
of our infrastructure. Reading through the wiki there are three things
that I have not figured out how to do / or if it's even possible which
I've detailed below.

1. Cache Peer
  - There are two use cases for the cache peer for our sites. The
first is the proxy only cache peer where the system will check another
proxy for an object and retrieve that object from cache vs. requesting
it from the origin server. This has proven to be extremely effective
at reducing the overall disk footprint of our caches while maintaining
a low hit rate on the origin server. The second of course is querying
the cache on another proxy, fetching and then caching on the local
box.

2. Redirect / Rewrite
  - Obviously running a redirector / rewrite application via a perl
script isn't ideal for performance but has been proven to be an
amazing resource when migrating CMS platforms or to work around
"features" of a specific application platform.

3. Header Replace
  - By default Squid enforces cache policy based on headers served
from the origin system. In some cases we need to then change those
headers when returning data to the client browser. An example would be
to modify the cache-control and expires headers to instruct the
browser not to cache. For a given site we could be setting this as a
global value, for specific URL patterns / directories, or for file
extensions.

Does Varnish support this currently? If not is it on the roadmap?

Thanks,
Max


From fredrik at nettby.no  Thu Jan 17 18:28:27 2008
From: fredrik at nettby.no (Fredrik Kristiansen)
Date: Thu, 17 Jan 2008 19:28:27 +0100
Subject: Load balancing?
In-Reply-To: <478F75A8.3080005@fer.hr>
References: <478F75A8.3080005@fer.hr>
Message-ID: <e9379eda0801171028g32c7873x316db0a71d2ff61e@mail.gmail.com>

Hi,

We have just built us a function in PHP to load balance the images between 4
varnish servers. This function will always assign the same image to a server
every time. It doesn't select a random server. So the cache is distributed
across our 4 servers.

// Get varnish server path
function get_varnish_path($path) {
    return 'http://'.$GLOBALS['servers'][crc32($path) %
count($GLOBALS['servers'])].$path;
}

// Varnish servers
$servers[] = 'varnish1.site.com';
$servers[] = 'varnish2.site.com';
$servers[] = 'varnish3.site.com';
$servers[] = 'varnish4.site.com';

// Example
print get_varnish_path('/image1.jpg').' ';
print get_varnish_path('/image12.jpg').' ';
print get_varnish_path('/image54.jpg').' ';
print get_varnish_path('/image488.jpg').' ';
print get_varnish_path('/image1.jpg').' ';

// Result
http://varnish4.site.com/image1.jpg
http://varnish1.site.com/image12.jpg
http://varnish3.site.com/image54.jpg
http://varnish1.site.com/image488.jpg
http://varnish4.site.com/image1.jpg

Fredrik Kristiansen
Nettby.no

On 17/01/2008, Ivan Voras <ivoras at fer.hr> wrote:
>
> Hi,
>
> I remember there were discussions about possible HTTP load balancing
> features in Varnish, but I don't know what was the result. Does Varnish
> support load balancing (even simple round-robin would be enough, IP
> address hashing would be better)?
>
> If not, does anyone has a recommendation about a nice and stable load
> balancer (ideally, it would also have an integrated HTTP accelerator to
> reduce the number of components to maintain in the system)?
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080117/f375f059/attachment.html>

From caleb.anthony at gmail.com  Thu Jan 17 21:41:34 2008
From: caleb.anthony at gmail.com (Caleb Anthony)
Date: Thu, 17 Jan 2008 14:41:34 -0700
Subject: Load balancing?
In-Reply-To: <478F75A8.3080005@fer.hr>
References: <478F75A8.3080005@fer.hr>
Message-ID: <33afbbe70801171341w37171c26od4780a263bd17b8f@mail.gmail.com>

Check out this page:

http://varnish.projects.linpro.no/wiki/Backends

This feature is only available in /trunk currently. So you'll have to
download the source from SVN.

I also agree about hashing. I'm not currently using varnish in full
production, but i'm worried that when I do, the round robin mode will
cause my clients to loose session information if/when they get
directed to a different backend server.

It would be great to have a sourcehash mode, where a specific IP would
always get directed to the same backend server as long as that backend
is up and functioning.

On Jan 17, 2008 8:35 AM, Ivan Voras <ivoras at fer.hr> wrote:
> Hi,
>
> I remember there were discussions about possible HTTP load balancing
> features in Varnish, but I don't know what was the result. Does Varnish
> support load balancing (even simple round-robin would be enough, IP
> address hashing would be better)?
>
> If not, does anyone has a recommendation about a nice and stable load
> balancer (ideally, it would also have an integrated HTTP accelerator to
> reduce the number of components to maintain in the system)?
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>


From phk at phk.freebsd.dk  Fri Jan 18 08:12:27 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 18 Jan 2008 08:12:27 +0000
Subject: Load balancing? 
In-Reply-To: Your message of "Thu, 17 Jan 2008 14:41:34 MST."
	<33afbbe70801171341w37171c26od4780a263bd17b8f@mail.gmail.com> 
Message-ID: <3593.1200643947@critter.freebsd.dk>

In message <33afbbe70801171341w37171c26od4780a263bd17b8f at mail.gmail.com>, "Cale
b Anthony" writes:

>Check out this page:
>
>http://varnish.projects.linpro.no/wiki/Backends
>
>This feature is only available in /trunk currently. So you'll have to
>download the source from SVN.

I'm not sure that core is entirely ready to be used, I have an audit
of it on my list.

With respect to client-backend-affinity, I plan to add a soft version
of that, more about that later.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From ssm at linpro.no  Fri Jan 18 10:33:50 2008
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Fri, 18 Jan 2008 11:33:50 +0100
Subject: Migration from Squid
In-Reply-To: <2fa1e1780801171026s5072b89o71af038514dd11d2@mail.gmail.com> (Max
	Clark's message of "Thu, 17 Jan 2008 10:26:43 -0800")
References: <2fa1e1780801171026s5072b89o71af038514dd11d2@mail.gmail.com>
Message-ID: <7xabn3pgmp.fsf@iostat.linpro.no>

"Max Clark" <max.clark at gmail.com> writes:

> Reading through the wiki there are three things that I have not
> figured out how to do / or if it's even possible which I've detailed
> below.
>
> 1. Cache Peer

> - There are two use cases for the cache peer for our sites.  The
> first is the proxy only cache peer where the system will check
> another proxy for an object and retrieve that object from cache
> vs. requesting it from the origin server. This has proven to be
> extremely effective at reducing the overall disk footprint of our
> caches while maintaining a low hit rate on the origin server.
>
> The second of course is querying the cache on another proxy,
> fetching and then caching on the local box.

Sibling cache_peers is not a feature in Varnish.  These are features
used to increase squid's performance by clustering, and implementing
different methods of retrieving cached data from another cache.

The functionality of parent cache_peers overlap roughly with the
backend declarations of varnish.

You will most likely will see bottlenecs in other places with varnish
than with squid.  You can, of course, put varnish in front of varnish,
or load balance between varnish instances if the impact of losing your
cache is too big, and you need to safeguard against this.

The cache_peer functionality of squid is a very nice feature if you
are using squid, but you may design your caching in another way with
varnish.

> 2. Redirect / Rewrite

> - Obviously running a redirector / rewrite application via a perl
> script isn't ideal for performance but has been proven to be an
> amazing resource when migrating CMS platforms or to work around
> "features" of a specific application platform.

As long as you don't execute perl script for every request, you should
in theory have quite good performance with this method of rewriting.

With the released (up to 1.1.2) versions of varnish, it seems you have
to use an external url rewriter.  While I have used apache, lighttpd
and nginx for this, there may be other, and better, alternatives
available.

> 3. Header Replace
> - By default Squid enforces cache policy based on headers served
> from the origin system. In some cases we need to then change those
> headers when returning data to the client browser. An example would be
> to modify the cache-control and expires headers to instruct the
> browser not to cache. For a given site we could be setting this as a
> global value, for specific URL patterns / directories, or for file
> extensions.
>
> Does Varnish support this currently? If not is it on the roadmap?

Yes, these examples from the vcl man page shows how to manipulate
headers:

,----[ vcl(7) ]
| 
|          sub vcl_recv {
|              # Normalize the Host: header
|              if (req.http.host ~ "^(www.)?example.com$") {
|                  set req.http.host = "www.example.com";
|              }
|          }
| [...]
|          sub vcl_fetch {
|              # Don?t cache cookies
|              remove obj.http.Set-Cookie;
|          }
| 
`----

You can manipulate headers received from the client (req.http.*), as
well as the request headers sent to the backend (bereq.http.*), object
headers retrieved from cache or backend (obj.http.*) and response
headers (resp.http.*) sent to the client in the same way.

-- 
Stig Sandbeck Mathisen, Linpro


From armdan20 at gmail.com  Fri Jan 18 14:46:24 2008
From: armdan20 at gmail.com (andan andan)
Date: Fri, 18 Jan 2008 15:46:24 +0100
Subject: Return 304 status code
Message-ID: <c1dd32e40801180646g69f1f9b9v6610019955f4948c@mail.gmail.com>

Hi.

How can I configure Varnish to return 304 status code to the client when the
object was not changed in base of cache headers like if-modified-since (or
others). We want to avoid to send the object if the client has an updated
version.

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080118/30c6738e/attachment.html>

From phk at phk.freebsd.dk  Fri Jan 18 14:52:04 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 18 Jan 2008 14:52:04 +0000
Subject: Return 304 status code 
In-Reply-To: Your message of "Fri, 18 Jan 2008 15:46:24 +0100."
	<c1dd32e40801180646g69f1f9b9v6610019955f4948c@mail.gmail.com> 
Message-ID: <6169.1200667924@critter.freebsd.dk>

In message <c1dd32e40801180646g69f1f9b9v6610019955f4948c at mail.gmail.com>, "anda
n andan" writes:

>How can I configure Varnish to return 304 status code to the client when the
>object was not changed in base of cache headers like if-modified-since (or
>others). We want to avoid to send the object if the client has an updated
>version.

Varnish already handles if-modified-since correctly.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From caleb.anthony at gmail.com  Fri Jan 18 17:33:04 2008
From: caleb.anthony at gmail.com (Caleb Anthony)
Date: Fri, 18 Jan 2008 10:33:04 -0700
Subject: Load balancing?
In-Reply-To: <3593.1200643947@critter.freebsd.dk>
References: <33afbbe70801171341w37171c26od4780a263bd17b8f@mail.gmail.com>
	<3593.1200643947@critter.freebsd.dk>
Message-ID: <33afbbe70801180933u6ec417eeg310968e1d1f9222e@mail.gmail.com>

You're right. I should have mentioned that /trunk is of course
bleeding-edge and may not be entirely stable for production use.

I look forward to what you have to say about client affinity.

On Jan 18, 2008 1:12 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <33afbbe70801171341w37171c26od4780a263bd17b8f at mail.gmail.com>, "Cale
> b Anthony" writes:
>
> >Check out this page:
> >
> >http://varnish.projects.linpro.no/wiki/Backends
> >
> >This feature is only available in /trunk currently. So you'll have to
> >download the source from SVN.
>
> I'm not sure that core is entirely ready to be used, I have an audit
> of it on my list.
>
> With respect to client-backend-affinity, I plan to add a soft version
> of that, more about that later.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>


From rob.g.tucker at gmail.com  Sat Jan 19 21:14:22 2008
From: rob.g.tucker at gmail.com (Rob Tucker)
Date: Sat, 19 Jan 2008 13:14:22 -0800
Subject: varnish on flash drives
Message-ID: <f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com>

I'd like to run a reverse proxy on a machine with solid state drives. The
hope is to get more queries served by utilizing high random I/O performance
(the drive I have supports about 5K requests per second of average 4KB
objects).
I'll have 16GB of RAM, 128GB of flash, and tons of CPU on the machine. The
entire 128GB of cache will be active in general.

The question is whether I should run Varnish on this hardware. SSD random
write performance is very poor, and I don't understand how Varnish replaces
objects. LRU won't be friendly to SSD. FIFO would be good enough for my
purpose. Is it possible to configure it that way?

Also, when using the flash drive as a disk, would I be guaranteed to make
exactly one read on the flash per object I fetch from it (for objects that
don't reside in the memory)?

Thanks.
Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080119/b7a58a8c/attachment.html>

From fragfutter at gmail.com  Sat Jan 19 22:22:33 2008
From: fragfutter at gmail.com (C. Handel)
Date: Sat, 19 Jan 2008 23:22:33 +0100
Subject: varnish on flash drives
In-Reply-To: <f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com>
References: <f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com>
Message-ID: <3d62bd5f0801191422x163f4f5ei2a1b737c8959a753@mail.gmail.com>

On Jan 19, 2008 10:14 PM, Rob Tucker <rob.g.tucker at gmail.com> wrote:
> I'd like to run a reverse proxy on a machine with solid state drives. The
> hope is to get more queries served by utilizing high random I/O performance
> (the drive I have supports about 5K requests per second of average 4KB
> objects).
>  I'll have 16GB of RAM, 128GB of flash, and tons of CPU on the machine. The
> entire 128GB of cache will be active in general.

Flash Drives are bad for systems with a high io-load (Flash Drives
have poor write Performance and a limited count of write access).
Get more Ram. You should be able to fit 32GB of ram in any
Serversystem for a reasonable Price. To get a good harddisk IO
Performance, get hardware raidcontroller an stripe a lot of small fast
scsi-disks.

Greetings
   Christoph


From rob.g.tucker at gmail.com  Sun Jan 20 03:17:01 2008
From: rob.g.tucker at gmail.com (Rob Tucker)
Date: Sat, 19 Jan 2008 19:17:01 -0800
Subject: varnish on flash drives
In-Reply-To: <f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com>
References: <f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com>
Message-ID: <f4b65cf70801191917r62ba7a0cr2f8fdd1fddec0b22@mail.gmail.com>

The flash drive I have support 1M cycles so if I treat it as circular
buffer and keep writing to it, at my rate, it would last me more than
10 years.
So it is not a concern. So if I go ahead with this config, I wonder if
it would work out ok. i.e.
1) Can I choose a FIFO replacement policy that ensures writes happen
sequentially to the media.
2) Would randomly reading one object that is not in memory result in
more than 1 access on the underlying device. Objects I'm talking about
would rarely exceed 10KB.

Rob

>Flash Drives are bad for systems with a high io-load (Flash Drives
have poor write Performance and a limited count of write access).
> Get more Ram. You should be able to fit 32GB of ram in any Serversystem for a reasonable Price. To get a good harddisk IO Performance, get hardware raidcontroller an stripe a lot of small fast scsi-disks.

>Greetings
> Christoph


On Jan 19, 2008 1:14 PM, Rob Tucker <rob.g.tucker at gmail.com> wrote:
> I'd like to run a reverse proxy on a machine with solid state drives. The hope is to get more queries served by utilizing high random I/O performance (the drive I have supports about 5K requests per second of average 4KB objects).
> I'll have 16GB of RAM, 128GB of flash, and tons of CPU on the machine. The entire 128GB of cache will be active in general.
>
> The question is whether I should run Varnish on this hardware. SSD random write performance is very poor, and I don't understand how Varnish replaces objects. LRU won't be friendly to SSD. FIFO would be good enough for my purpose. Is it possible to configure it that way?
>
> Also, when using the flash drive as a disk, would I be guaranteed to make exactly one read on the flash per object I fetch from it (for objects that don't reside in the memory)?
>
> Thanks.
> Rob
>
>


From james at nyi.net  Mon Jan 21 05:11:40 2008
From: james at nyi.net (james)
Date: Mon, 21 Jan 2008 00:11:40 -0500
Subject: Question about Cache Size
In-Reply-To: <4786A996.1030101@nyi.net>
References: <4786A996.1030101@nyi.net>
Message-ID: <4794298C.6000206@nyi.net>

james wrote:
> Hey everyone,
>
> In trying to diagnose why the varnish child dies at seemingly random 
> times, I started varnish with -d -d and waited for a crash. This is what 
> I got in the output:
>
> Child said (2, 16094): <<Assert error in STV_alloc(), stevedore.c line 70:
>   Condition((st) != NULL) not true.
>   errno = 107 (Transport endpoint is not connected)
>  >>
> Cache child died pid=16094 status=0x6
> Clean child
> Child cleaned
> start child pid 18886
> Child said (2, 18886): <<Child starts
> managed to mmap 565284864 bytes of 565284864
> Ready
> CLI ready
>  >>
>   

This issue is not effecting me anymore after moving to a 64bit system.

Alas, I am experiencing another issue using varnish-1.2. After purging 
from the cache (using url.purge *.css), the child will die and the 
management process goes to 100% cpu. From running in debug mode:

storage_file: filename: /var/cache/varnishd//varnish.BIT37f (unlinked) 
size 10240 MegaBytes.
Using old SHMFILE
rolling(1)...
rolling(2)...
start
start child pid 20178
200 0      

Child said (2, 20178): <<Child starts
managed to mmap 10737418240 bytes of 10737418240
Ready
CLI ready
 >>
Cache child died pid=20178 status=0xb
Clean child
Child cleaned
start child pid 20199
Child said (2, 20199): <<Child starts
managed to mmap 10737418240 bytes of 10737418240
Ready
CLI ready
 >>

Is there anything else I should to to help track down the issue? Is it 
advisable to go to 1.1.2?

-- james


From max.clark at gmail.com  Mon Jan 21 16:55:05 2008
From: max.clark at gmail.com (Max Clark)
Date: Mon, 21 Jan 2008 08:55:05 -0800
Subject: Migration from Squid
In-Reply-To: <7xabn3pgmp.fsf@iostat.linpro.no>
References: <2fa1e1780801171026s5072b89o71af038514dd11d2@mail.gmail.com>
	<7xabn3pgmp.fsf@iostat.linpro.no>
Message-ID: <2fa1e1780801210855i3a605c8fvd3197559e1a96e5c@mail.gmail.com>

Thanks for the response... is "cache clustering" on the roadmap for
Varnish? The sibling cache_peer configuration has been an excellent
performance boost for us but my primary concern is more resource
utilization. If the cache server has less resources than the origin
server on a busy site then the object expiration on the cache will
increase and as a result hits against the origin. Placing another
cache inbetween the external cache and the origin is an interesting
idea but adds to much additional complexity for my taste.

-Max

2008/1/18 Stig Sandbeck Mathisen <ssm at linpro.no>:
> "Max Clark" <max.clark at gmail.com> writes:
>
> > Reading through the wiki there are three things that I have not
> > figured out how to do / or if it's even possible which I've detailed
> > below.
> >
> > 1. Cache Peer
>
> > - There are two use cases for the cache peer for our sites.  The
> > first is the proxy only cache peer where the system will check
> > another proxy for an object and retrieve that object from cache
> > vs. requesting it from the origin server. This has proven to be
> > extremely effective at reducing the overall disk footprint of our
> > caches while maintaining a low hit rate on the origin server.
> >
> > The second of course is querying the cache on another proxy,
> > fetching and then caching on the local box.
>
> Sibling cache_peers is not a feature in Varnish.  These are features
> used to increase squid's performance by clustering, and implementing
> different methods of retrieving cached data from another cache.
>
> The functionality of parent cache_peers overlap roughly with the
> backend declarations of varnish.
>
> You will most likely will see bottlenecs in other places with varnish
> than with squid.  You can, of course, put varnish in front of varnish,
> or load balance between varnish instances if the impact of losing your
> cache is too big, and you need to safeguard against this.
>
> The cache_peer functionality of squid is a very nice feature if you
> are using squid, but you may design your caching in another way with
> varnish.
>
> > 2. Redirect / Rewrite
>
> > - Obviously running a redirector / rewrite application via a perl
> > script isn't ideal for performance but has been proven to be an
> > amazing resource when migrating CMS platforms or to work around
> > "features" of a specific application platform.
>
> As long as you don't execute perl script for every request, you should
> in theory have quite good performance with this method of rewriting.
>
> With the released (up to 1.1.2) versions of varnish, it seems you have
> to use an external url rewriter.  While I have used apache, lighttpd
> and nginx for this, there may be other, and better, alternatives
> available.
>
> > 3. Header Replace
> > - By default Squid enforces cache policy based on headers served
> > from the origin system. In some cases we need to then change those
> > headers when returning data to the client browser. An example would be
> > to modify the cache-control and expires headers to instruct the
> > browser not to cache. For a given site we could be setting this as a
> > global value, for specific URL patterns / directories, or for file
> > extensions.
> >
> > Does Varnish support this currently? If not is it on the roadmap?
>
> Yes, these examples from the vcl man page shows how to manipulate
> headers:
>
> ,----[ vcl(7) ]
> |
> |          sub vcl_recv {
> |              # Normalize the Host: header
> |              if (req.http.host ~ "^(www.)?example.com$") {
> |                  set req.http.host = "www.example.com";
> |              }
> |          }
> | [...]
> |          sub vcl_fetch {
> |              # Don't cache cookies
> |              remove obj.http.Set-Cookie;
> |          }
> |
> `----
>
> You can manipulate headers received from the client (req.http.*), as
> well as the request headers sent to the backend (bereq.http.*), object
> headers retrieved from cache or backend (obj.http.*) and response
> headers (resp.http.*) sent to the client in the same way.
>
> --
> Stig Sandbeck Mathisen, Linpro
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>


From varnish at streikt.net  Mon Jan 21 21:48:33 2008
From: varnish at streikt.net (Philipp Winkler)
Date: Mon, 21 Jan 2008 22:48:33 +0100
Subject: Configure varnish to pipe a subdomain
Message-ID: <47951331.900@streikt.net>

Hello list,

I installed varnish on a staging system today and I am totally impressed 
by the performance.
Unfortunately I'm no regex expert, so I will need your help with this 
config issue.

I want to use varnish for a domain - lets call it "example.com"
The domain has got a subdomain "admin.example.com" - any GET / POST to 
this subdomain and any pages underneath should not be cached but should 
directly passed to the backend.

This is my config - which does not work for as the requests to 
admin.example.com as they are also being cached :-(

Thanks so much for your help!
Philipp



(comments are in german - please dont bother!):



/* Definition Default Backend */
backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "1234";
}

sub vcl_recv {
        /* Falls etwas an den Server geschickt wird, geben wir das durch */
        if (req.request == "POST") {
                pipe;
        }

        /* Anfragen an die "admin" Seite werden direkt durchgereicht /*
         if (req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Expect) {
                pipe;
        }

        /* Wir cachen Bilder, Dokumente, Audio und FLV Filme*/
        if (req.url ~ 
"\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|flv|ico|doc|ppt|xls|pdf|mp3|mp4|mp4a|ogg)$")
        {
                lookup;
        }
        /* Wir cachen auch CSS und javascript */
        if (req.url ~ "\.(css|js)$") {
                lookup;
        }

        /* Und wir cachen HTML Code */
        if (req.url ~ "\.(html|htm)$") {
                lookup;
        }
        /* Filme cachen wir auch - testweise */
        if (req.url ~ "\.(mov|avi|wmv)$") {
                lookup;
        }
        /* ZIP Dateien cachen wir experimentell auch */
        if (req.url ~ "\.(zip)$") {
                lookup;
        }

        /* Cookie Requests an die Hauptseite cachen wir weg */
        if (req.http.Cookie) {
                lookup;
        }

        /* Kein Caching fuer Content hinter http authentification */
        if (req.http.Authenticate || req.http.Authorization) {
                pipe;
        }

        /* Oh und wir cachen alles andere */
       lookup;
}

sub vcl_fetch {
        /* Wir setzen die TTL fuer jedes Objekt das eine TTL von unter 
60sec hat auf 60sec */
        if (obj.ttl < 60s) {
        }

        if (!obj.valid) {
                error;
        }
        if (!obj.cacheable) {
                pass;
        }
        if (obj.http.Set-Cookie) {
                pass;
        }
        /* Wir cachen HTML maximal 30 Minuten */
        if (req.url ~ "\.(htm|html)$") {
                set obj.ttl = 1800s;
        }

/* Wir cachen jedes Objekt maximal fuer drei Stunden */

        set obj.ttl = 10800s;
}

sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
}





From duja at torlen.net  Tue Jan 22 09:37:50 2008
From: duja at torlen.net (Erik)
Date: Tue, 22 Jan 2008 10:37:50 +0100
Subject: Configure varnish to pipe a subdomain
Message-ID: <4nbq3fzft8fc02n.220120081037@torlen.net>



Original Message -----------------------
Hello list,

I installed varnish on a staging system today and I am totally impressed 
by the performance.
Unfortunately I'm no regex expert, so I will need your help with this 
config issue.

I want to use varnish for a domain - lets call it "example.com"
The domain has got a subdomain "admin.example.com" - any GET / POST to 
this subdomain and any pages underneath should not be cached but should 
directly passed to the backend.

This is my config - which does not work for as the requests to 
admin.example.com as they are also being cached :-(

Thanks so much for your help!
Philipp
Neither am I a regex expert, but I dont think that the regex is the problem. You are using http.url instead of using http.host.
And if im not totally wrong, you dont need the ~ to check the host, you can use ==. 

Change this:
if (req.http.url ~ "admin.example.com$") {
    pipe;
}
To this:
if (req.http.host == "admin.example.com") {
    pipe;
}

And this:
if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
    pipe;
}

To this:
if (req.http.Cookie && req.http.host == "admin.example.com") {
    pipe;
}

Althought I do not know if you should pipe the cookies. I think its pass that would be used when cookies is set but Im not an expert, maybe im wrong.

Cheers
Erik


-------------------------------------------

(comments are in german - please dont bother!):



/* Definition Default Backend */
backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "1234";
}

sub vcl_recv {
        /* Falls etwas an den Server geschickt wird, geben wir das durch */
        if (req.request == "POST") {
                pipe;
        }

        /* Anfragen an die "admin" Seite werden direkt durchgereicht /*
         if (req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Expect) {
                pipe;
        }

        /* Wir cachen Bilder, Dokumente, Audio und FLV Filme*/
        if (req.url ~ 
"\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|flv|ico|doc|ppt|xls|pdf|mp3|mp4|mp4a|ogg)$")
        {
                lookup;
        }
        /* Wir cachen auch CSS und javascript */
        if (req.url ~ "\.(css|js)$") {
                lookup;
        }

        /* Und wir cachen HTML Code */
        if (req.url ~ "\.(html|htm)$") {
                lookup;
        }
        /* Filme cachen wir auch - testweise */
        if (req.url ~ "\.(mov|avi|wmv)$") {
                lookup;
        }
        /* ZIP Dateien cachen wir experimentell auch */
        if (req.url ~ "\.(zip)$") {
                lookup;
        }

        /* Cookie Requests an die Hauptseite cachen wir weg */
        if (req.http.Cookie) {
                lookup;
        }

        /* Kein Caching fuer Content hinter http authentification */
        if (req.http.Authenticate || req.http.Authorization) {
                pipe;
        }

        /* Oh und wir cachen alles andere */
       lookup;
}

sub vcl_fetch {
        /* Wir setzen die TTL fuer jedes Objekt das eine TTL von unter 
60sec hat auf 60sec */
        if (obj.ttl < 60s) {
        }

        if (!obj.valid) {
                error;
        }
        if (!obj.cacheable) {
                pass;
        }
        if (obj.http.Set-Cookie) {
                pass;
        }
        /* Wir cachen HTML maximal 30 Minuten */
        if (req.url ~ "\.(htm|html)$") {
                set obj.ttl = 1800s;
        }

/* Wir cachen jedes Objekt maximal fuer drei Stunden */

        set obj.ttl = 10800s;
}

sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
}



_______________________________________________
varnish-misc mailing list
varnish-misc at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc



From duja at torlen.net  Tue Jan 22 09:40:46 2008
From: duja at torlen.net (Erik)
Date: Tue, 22 Jan 2008 10:40:46 +0100
Subject: Configure varnish to pipe a subdomain
Message-ID: <84o62yo3haxzcu4.220120081040@torlen.net>

Sending another one, the previous got messed up (using a webmail client)

Neither am I a regex expert, but I dont think that the regex is the problem. You are using http.url instead of using http.host.
And if im not totally wrong, you dont need the ~ to check the host, you can use ==. 

Change this:
if (req.http.url ~ "admin.example.com$") {
    pipe;
}
To this:
if (req.http.host == "admin.example.com") {
    pipe;
}

And this:
if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
    pipe;
}

To this:
if (req.http.Cookie && req.http.host == "admin.example.com") {
    pipe;
}

Althought I do not know if you should pipe the cookies. I think its pass that would be used when cookies is set but Im not an expert, maybe im wrong.

Cheers
Erik


Original Message -----------------------


Hello list,

I installed varnish on a staging system today and I am totally impressed 
by the performance.
Unfortunately I'm no regex expert, so I will need your help with this 
config issue.

I want to use varnish for a domain - lets call it "example.com"
The domain has got a subdomain "admin.example.com" - any GET / POST to 
this subdomain and any pages underneath should not be cached but should 
directly passed to the backend.

This is my config - which does not work for as the requests to 
admin.example.com as they are also being cached :-(

Thanks so much for your help!
Philipp

(comments are in german - please dont bother!):



/* Definition Default Backend */
backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "1234";
}

sub vcl_recv {
        /* Falls etwas an den Server geschickt wird, geben wir das durch */
        if (req.request == "POST") {
                pipe;
        }

        /* Anfragen an die "admin" Seite werden direkt durchgereicht /*
         if (req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
                pipe;
        }

        if (req.http.Expect) {
                pipe;
        }

        /* Wir cachen Bilder, Dokumente, Audio und FLV Filme*/
        if (req.url ~ 
"\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|flv|ico|doc|ppt|xls|pdf|mp3|mp4|mp4a|ogg)$")
        {
                lookup;
        }
        /* Wir cachen auch CSS und javascript */
        if (req.url ~ "\.(css|js)$") {
                lookup;
        }

        /* Und wir cachen HTML Code */
        if (req.url ~ "\.(html|htm)$") {
                lookup;
        }
        /* Filme cachen wir auch - testweise */
        if (req.url ~ "\.(mov|avi|wmv)$") {
                lookup;
        }
        /* ZIP Dateien cachen wir experimentell auch */
        if (req.url ~ "\.(zip)$") {
                lookup;
        }

        /* Cookie Requests an die Hauptseite cachen wir weg */
        if (req.http.Cookie) {
                lookup;
        }

        /* Kein Caching fuer Content hinter http authentification */
        if (req.http.Authenticate || req.http.Authorization) {
                pipe;
        }

        /* Oh und wir cachen alles andere */
       lookup;
}

sub vcl_fetch {
        /* Wir setzen die TTL fuer jedes Objekt das eine TTL von unter 
60sec hat auf 60sec */
        if (obj.ttl < 60s) {
        }

        if (!obj.valid) {
                error;
        }
        if (!obj.cacheable) {
                pass;
        }
        if (obj.http.Set-Cookie) {
                pass;
        }
        /* Wir cachen HTML maximal 30 Minuten */
        if (req.url ~ "\.(htm|html)$") {
                set obj.ttl = 1800s;
        }

/* Wir cachen jedes Objekt maximal fuer drei Stunden */

        set obj.ttl = 10800s;
}

sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
}



From varnish at streikt.net  Tue Jan 22 12:33:30 2008
From: varnish at streikt.net (varnish at streikt.net)
Date: Tue, 22 Jan 2008 13:33:30 +0100 (CET)
Subject: Configure varnish to pipe a subdomain
In-Reply-To: <84o62yo3haxzcu4.220120081040@torlen.net>
References: <84o62yo3haxzcu4.220120081040@torlen.net>
Message-ID: <34108.82.113.116.97.1201005210.squirrel@it-winkler.de>

Thanks a lot Erik, this did it! Requests to the subdomain are no longer
cached. Great! :-) Going live with this configuration right now..

> Sending another one, the previous got messed up (using a webmail client)
>
> Neither am I a regex expert, but I dont think that the regex is the
> problem. You are using http.url instead of using http.host.
> And if im not totally wrong, you dont need the ~ to check the host, you
> can use ==.
>
> Change this:
> if (req.http.url ~ "admin.example.com$") {
>     pipe;
> }
> To this:
> if (req.http.host == "admin.example.com") {
>     pipe;
> }
>
> And this:
> if (req.http.Cookie && req.http.url ~ "admin.example.com$") {
>     pipe;
> }
>
> To this:
> if (req.http.Cookie && req.http.host == "admin.example.com") {
>     pipe;
> }
>
> Althought I do not know if you should pipe the cookies. I think its pass
> that would be used when cookies is set but Im not an expert, maybe im
> wrong.
>
> Cheers
> Erik


From max at funnyordie.com  Wed Jan 23 00:26:54 2008
From: max at funnyordie.com (Max Baumann)
Date: Tue, 22 Jan 2008 16:26:54 -0800
Subject: Getting the most out of varnish with large files / slow connections 
Message-ID: <1423ECB1-61AB-46AA-B575-1C3B3CC9F5B1@funnyordie.com>


Hi,

We are seeing a large drop off in our ability to handle a load with  
large file sizes(70k) and slower connections.  My guess is that this  
is an issue around the number of concurrent connections we're able to  
handle.  Does this sound right?  Any tips on varnish / os /  
infrastructure config for getting more concurrent connections?

Many thanks,
Max


From tb-lists at delc.dk  Wed Jan 23 07:51:36 2008
From: tb-lists at delc.dk (Thilo Bangert)
Date: Wed, 23 Jan 2008 08:51:36 +0100
Subject: Getting the most out of varnish with large files / slow
	connections
In-Reply-To: <1423ECB1-61AB-46AA-B575-1C3B3CC9F5B1@funnyordie.com>
References: <1423ECB1-61AB-46AA-B575-1C3B3CC9F5B1@funnyordie.com>
Message-ID: <200801230851.36142.tb-lists@delc.dk>

On Wednesday 23 January 2008 01:26:54 Max Baumann wrote:
> Hi,
>
> We are seeing a large drop off in our ability to handle a load with
> large file sizes(70k) and slower connections.  My guess is that this
> is an issue around the number of concurrent connections we're able to
> handle.  

why? what is the basis for this guess? how many concurrent connections do you 
see? on what hardware? around where and what is the 'drop off'?

to me, 70k doesnt sound like a large file. OTOH if that is in fact the number 
of objects in your working set, i would consider it large.

Thilo


From phk at phk.freebsd.dk  Mon Jan 28 12:29:49 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 28 Jan 2008 12:29:49 +0000
Subject: varnish on flash drives 
In-Reply-To: Your message of "Sat, 19 Jan 2008 13:14:22 PST."
	<f4b65cf70801191314q91a6b47of3a8d95d0b508838@mail.gmail.com> 
Message-ID: <49576.1201523389@critter.freebsd.dk>

In message <f4b65cf70801191314q91a6b47of3a8d95d0b508838 at mail.gmail.com>, "Rob T
ucker" writes:

>I'd like to run a reverse proxy on a machine with solid state drives. 

If you run it on the "real" kind of solid state drives, you can just do
it with no concerns.  I've been recommended "MTRON" as a really good
and some of the fastest drives for this kind of purpose.

If you run it on "lets stick a SATA converter in front of a camera-flash"
drives, you'll suffer terrible write service time (0.1+ second) and wear
them out in no time.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at startsiden.no  Thu Jan 31 13:17:22 2008
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Thu, 31 Jan 2008 14:17:22 +0100 (CET)
Subject: Help debugging cacheability / ttl information?
Message-ID: <17076163.9291201785442139.JavaMail.root@ms1.startsiden.no>

Hi,

We have a setup with varnish in front of a lighttpd+fastcgi combo. For some reason or other the application doesn't send any cache control headers. Instead of doing the sensible thing (prodding the developers to fix it in the backend) I have taken it upon myself to fix it via VCL. 

However my feeble attempts at being smart have so far failed elegantly, and I am left scratching my head a bit.

What I really am looking for is some help with how to best debug the setup with regards to TTLs and cacheability. I'd like to be able to inspect a specific cached objects TTL to see if my "set obj.ttl" overrides actually work or not. What is the best way to do that? 

On the client I use firebug, so all headers and responess are readily available on that end. The reason I don't think everything is as it should is that the Age header is very low most of the time.

For the record we are still at varnish 1.0.4 and here is the current VCL (I've taken out the stuff that was obviously not working anyway) :

"
backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "80";
}

sub vcl_recv {
    if (req.request == "GET" && req.http.cookie) {
        lookup;
    }
    if (req.request == "GET" && req.url ~ "\.(gif|jpg|swf|css|js)$") {
        lookup;
    }
}

sub vcl_fetch {
    if (obj.ttl < 120s) {
        set obj.ttl = 120s;
    }
    if (resp.http.Set-Cookie) {
        insert;
    }
}
"

Any help will be highly appreciated!

Regards
--
Denis


From duja at torlen.net  Thu Jan 31 21:01:02 2008
From: duja at torlen.net (Erik Torlen)
Date: Thu, 31 Jan 2008 22:01:02 +0100
Subject: IP & X-Forward-for
Message-ID: <47A2370E.7040108@torlen.net>

Hi,

Im experiencing a strange problem when using varnish in front of apache2.
I m running a php website where I log the client IP to the usertable in 
the database.

I first check the Client IP to see if it is a valid IP, if not, I check 
the "x-forward-for" for a valid ip (I check for multiple IPs in 
x-forward-for).
If neither client ip or x-forward-for contained a valid ip, it returns 
the client ip (in most cases 127.0.0.1).

The strange thing is that I get alot of IPs that comes from 127.0.0.1 
which means that x-forward-for is NOT set.

This is the log from varnish:
81.224.31.xxx - - [31/Jan/2008:20:52:09 +0100] "GET
127.0.0.1 - - [31/Jan/2008:20:52:09 +0100] "(null) (null) (null)" (null) 
(null) "-" "-"
backendserver - - [31/Jan/2008:19:52:09 +0100] "GET
81.227.81.xxx - - [31/Jan/2008:20:52:09 +0100] "GET
127.0.0.1 - - [31/Jan/2008:19:52:11 +0100] "GET
217.197.62.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
127.0.0.1 - - [31/Jan/2008:19:52:11 +0100] "GET
85.229.6.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
127.0.0.1 - - [31/Jan/2008:19:52:11 +0100] "GET
85.229.6.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
127.0.0.1 - - [31/Jan/2008:19:52:12 +0100] "GET
83.233.193.xxx - - [31/Jan/2008:20:52:12 +0100] "GET

Why does it have so many connections from 127.0.0.1 and why is the date 
an hour late?
It also got a "connection" from "backendserver" which is the name of my 
backendserver in the vcl config.

The logfile from apache2 shows me this whit the same connections:
81.224.31.xxx - - [31/Jan/2008:20:52:08 +0100] "GET
81.227.81.xxx - - [31/Jan/2008:20:52:09 +0100] "GET
217.197.62.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
85.229.6.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
85.229.6.xxx - - [31/Jan/2008:20:52:11 +0100] "GET
83.233.193.xxx - - [31/Jan/2008:20:52:12 +0100] "GET

No 127.0.0.1 connections in this logfile. Is the 127.0.0.1 connections 
from apache2 when varnish fetches the object from back their?

Im not sure if these logentries say enough about my problem. I will try 
to find some more entries that could show some more about the issue.

I appreciate all help I can get on this one.

/ Erik









From aaron.pfeifer at gmail.com  Thu Jan 31 21:48:56 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Thu, 31 Jan 2008 16:48:56 -0500
Subject: Not caching immediately?
Message-ID: <2c7df3ed0801311348t759eb02eyf7cb5cfaaf567d3f@mail.gmail.com>

Is there a possible reason Varnish would not immediately cache a file when
put in front of s3, requiring multiple subsequent hits before caching?  The
scenario is:

The file "foo.png" is uploaded to the web server and then stored onto S3.
When a request is made for http://localhost/files/foo.png, Varnish is
configured to look for the file in S3 and cache it.  Below is my
configuration:

backend www {
  set backend.host = "127.0.0.1";
  set backend.port = "81";
}

backend media {
  set backend.host = "s3.amazonaws.com";
  set backend.port = "80";
}

sub vcl_recv {
  if (req.url ~ "\.png$") {
    set req.url = regsub(req.url, "^", "/my_bucket_name");
    set req.backend = media;
    lookup;
  }
}

However, for some reason when accessing that url, I get 404 errors.  After
hitting refresh a bunch of times, I stop getting the 404's.  I've verified
during this time that the file is already available at the url on S3.

Any help would be appreciated.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080131/2a745ae3/attachment.html>

From aaron.pfeifer at gmail.com  Thu Jan 31 23:14:24 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Thu, 31 Jan 2008 18:14:24 -0500
Subject: Not caching immediately?
In-Reply-To: <2c7df3ed0801311348t759eb02eyf7cb5cfaaf567d3f@mail.gmail.com>
References: <2c7df3ed0801311348t759eb02eyf7cb5cfaaf567d3f@mail.gmail.com>
Message-ID: <2c7df3ed0801311514h1faa5f1dq573ce0ef5a193421@mail.gmail.com>

In case anyone runs into the same problem, with the help from the folks on
#varnish, the problem is that piping a POST will cause the cache to be
bypassed if the session remains open.  The solution is to either:

(1) Upgrade to trunk (I was using 1.1.2)
(2) Add "set req.http.connection = "close";" immediately before the pipe for
a POST

On Jan 31, 2008 4:48 PM, Aaron Pfeifer <aaron.pfeifer at gmail.com> wrote:

> Is there a possible reason Varnish would not immediately cache a file when
> put in front of s3, requiring multiple subsequent hits before caching?  The
> scenario is:
>
> The file "foo.png" is uploaded to the web server and then stored onto S3.
> When a request is made for http://localhost/files/foo.png, Varnish is
> configured to look for the file in S3 and cache it.  Below is my
> configuration:
>
> backend www {
>   set backend.host = "127.0.0.1";
>   set backend.port = "81";
> }
>
> backend media {
>   set backend.host = "s3.amazonaws.com";
>   set backend.port = "80";
> }
>
> sub vcl_recv {
>   if (req.url ~ "\.png$") {
>     set req.url = regsub(req.url, "^", "/my_bucket_name");
>     set req.backend = media;
>     lookup;
>   }
> }
>
> However, for some reason when accessing that url, I get 404 errors.  After
> hitting refresh a bunch of times, I stop getting the 404's.  I've verified
> during this time that the file is already available at the url on S3.
>
> Any help would be appreciated.
>
> Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080131/eb4109a7/attachment.html>