From rafael.umann at terra.com.br  Mon Jun  2 19:02:05 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Mon, 2 Jun 2008 16:02:05 -0300
Subject: Linux performance enhancement
Message-ID: <20080602160205.633e479c@sup-dig.wrk.terra.com.br>


Hey guys,

We are using varnish in large scale here to provide static files, but
we still have to use our own patch to improve the performance using
Linux.

We've a ticket and would like to hear anything from you:

http://varnish.projects.linpro.no/ticket/235

Thanks,

[]s,
-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: 55 (51) 3284-4344


From phk at phk.freebsd.dk  Mon Jun  2 19:32:46 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jun 2008 19:32:46 +0000
Subject: Linux performance enhancement 
In-Reply-To: Your message of "Mon, 02 Jun 2008 16:02:05 -0300."
	<20080602160205.633e479c@sup-dig.wrk.terra.com.br> 
Message-ID: <6958.1212435166@critter.freebsd.dk>

In message <20080602160205.633e479c at sup-dig.wrk.terra.com.br>, Rafael Umann wri
tes:

>We've a ticket and would like to hear anything from you:
>
>http://varnish.projects.linpro.no/ticket/235

Hi Rafael,

I don't have linux machines in my lab to test this on, so I was hoping
to get Dag-Erling to give it a spin.

He is on another project right now, so that is probably not happening.

So if you tell me the patch is guaranteed to work better than what's 
in trunk, I'll commit it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From rafael.umann at terra.com.br  Mon Jun  2 21:52:08 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Mon, 2 Jun 2008 18:52:08 -0300
Subject: Linux performance enhancement
In-Reply-To: <6958.1212435166@critter.freebsd.dk>
References: <20080602160205.633e479c@sup-dig.wrk.terra.com.br>
	<6958.1212435166@critter.freebsd.dk>
Message-ID: <20080602185208.7fa20e73@sup-dig.wrk.terra.com.br>


Hi Poul,

The problem with this patch is that it keeps too many sockets opened.
In a high load server (maybe with load balancers), it can be a potential
problem, and much worst in atacks!

So, we would like to know your (and anybody else) opinion about it.
What is better?

1. Performance with lots of sockets waiting 60 seconds to be cleaned.
2. Performance with the normal amount of sockets, with a thread
controling all this socket stuff.
3. No performance :(

Anyway, with this patch (option 1) the performance is ok, but it also
implement some problems. And we could not use varnish with option 3.

For us, the ideal solution would be implementing a new thread to
control all this stuff. What do you think about it?

Thanks,

Br,
-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: +55 (51) 3284-4344


On Mon, 02 Jun 2008 19:32:46 +0000
"Poul-Henning Kamp" <phk at phk.freebsd.dk> wrote:

> In message <20080602160205.633e479c at sup-dig.wrk.terra.com.br>, Rafael
> Umann wri tes:
> 
> >We've a ticket and would like to hear anything from you:
> >
> >http://varnish.projects.linpro.no/ticket/235
> 
> Hi Rafael,
> 
> I don't have linux machines in my lab to test this on, so I was hoping
> to get Dag-Erling to give it a spin.
> 
> He is on another project right now, so that is probably not happening.
> 
> So if you tell me the patch is guaranteed to work better than what's 
> in trunk, I'll commit it.
> 





From phk at phk.freebsd.dk  Tue Jun  3 06:56:31 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Jun 2008 06:56:31 +0000
Subject: Linux performance enhancement 
In-Reply-To: Your message of "Mon, 02 Jun 2008 18:52:08 -0300."
	<20080602185208.7fa20e73@sup-dig.wrk.terra.com.br> 
Message-ID: <63230.1212476191@critter.freebsd.dk>

In message <20080602185208.7fa20e73 at sup-dig.wrk.terra.com.br>, Rafael Umann wri
tes:

>For us, the ideal solution would be implementing a new thread to
>control all this stuff. What do you think about it?

I'm reconsidering the accept thread right now for other reasons, so
I'll keep this in mind as well.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chenxy.china at gmail.com  Mon Jun 16 10:07:15 2008
From: chenxy.china at gmail.com (chen xiaoyong)
Date: Mon, 16 Jun 2008 18:07:15 +0800
Subject: about big flv files
Message-ID: <7ab169e90806160307w778fb06ay1eb92b19d6e17ad1@mail.gmail.com>

Hey guys,
  I'm thinking about big flv files in varnishd. Now,many web sites support
video, Typically,  these files more than 5M.
  I don't konw what's better ..  split file in function fetch_straight ?
or do nothing , set up tcp parameters?
  I would like to hear anything from you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080616/d06695d3/attachment.html>

From phk at phk.freebsd.dk  Mon Jun 16 10:15:22 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Jun 2008 10:15:22 +0000
Subject: about big flv files 
In-Reply-To: Your message of "Mon, 16 Jun 2008 18:07:15 +0800."
	<7ab169e90806160307w778fb06ay1eb92b19d6e17ad1@mail.gmail.com> 
Message-ID: <18376.1213611322@critter.freebsd.dk>

In message <7ab169e90806160307w778fb06ay1eb92b19d6e17ad1 at mail.gmail.com>, "chen
 xiaoyong" writes:

>Hey guys,
>  I'm thinking about big flv files in varnishd. Now,many web sites support
>video, Typically,  these files more than 5M.

There is no intrinsic limit on object size in varnish, but it is not
particular geared towards such use.  The major stumbling block is that
Varnish will fetch the entire file from the backend before starting
transmission to the client.

>  I don't konw what's better ..  split file in function fetch_straight ?
>or do nothing , set up tcp parameters?

I'm not sure I understand what you're trying to say here ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From aotto at mosso.com  Mon Jun 16 15:06:27 2008
From: aotto at mosso.com (Adrian Otto)
Date: Mon, 16 Jun 2008 08:06:27 -0700
Subject: about big flv files 
In-Reply-To: <18376.1213611322@critter.freebsd.dk>
References: <18376.1213611322@critter.freebsd.dk>
Message-ID: <06F9AFC0-24C5-4315-9B22-F7F8EE1913C4@mosso.com>


On Jun 16, 2008, at 3:15 AM, Poul-Henning Kamp wrote:

> In message  
> <7ab169e90806160307w778fb06ay1eb92b19d6e17ad1 at mail.gmail.com>, "chen
>  xiaoyong" writes:
>
>> Hey guys,
>>  I'm thinking about big flv files in varnishd. Now,many web sites  
>> support
>> video, Typically,  these files more than 5M.
>
> There is no intrinsic limit on object size in varnish, but it is not
> particular geared towards such use.  The major stumbling block is that
> Varnish will fetch the entire file from the backend before starting
> transmission to the client.

Have you considered allowing a configuration for streaming a file to  
a client while it's being fetched from the backend server? Upon the  
first successful sending of the complete file to the client, it can  
then be stored in the cache for subsequent serving. Also, it would be  
helpful to be able to limit the number of backend connections that  
are streaming in this way so that it's possible to prevent swamping  
the origin server when an object is very popular. Perhaps if the  
limit is reached, it could reply with a 502 Server Busy reply. In  
addition, I think this would be a fantastic addition.

>>  I don't konw what's better ..  split file in function  
>> fetch_straight ?
>> or do nothing , set up tcp parameters?
>
> I'm not sure I understand what you're trying to say here ?

It sounds like he's trying to optimize download performance.





From phk at phk.freebsd.dk  Mon Jun 16 15:28:57 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Jun 2008 15:28:57 +0000
Subject: about big flv files 
In-Reply-To: Your message of "Mon, 16 Jun 2008 08:06:27 MST."
	<06F9AFC0-24C5-4315-9B22-F7F8EE1913C4@mosso.com> 
Message-ID: <19853.1213630137@critter.freebsd.dk>

In message <06F9AFC0-24C5-4315-9B22-F7F8EE1913C4 at mosso.com>, Adrian Otto writes
:

>> There is no intrinsic limit on object size in varnish, but it is not
>> particular geared towards such use.  The major stumbling block is that
>> Varnish will fetch the entire file from the backend before starting
>> transmission to the client.
>
>Have you considered allowing a configuration for streaming a file to  
>a client while it's being fetched from the backend server? 

Yes.

Do I have a sponsor for it ?  No.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From aotto at mosso.com  Mon Jun 16 15:28:27 2008
From: aotto at mosso.com (Adrian Otto)
Date: Mon, 16 Jun 2008 08:28:27 -0700
Subject: about big flv files 
In-Reply-To: <19853.1213630137@critter.freebsd.dk>
References: <19853.1213630137@critter.freebsd.dk>
Message-ID: <C25651DA-E934-4E79-8A6F-4C3896419547@mosso.com>

I'll proudly sponsor it, along with at least one other important  
feature I'm seeking. Simply contact me directly and we can make  
arrangements.

Thanks,

Adrian

On Jun 16, 2008, at 8:28 AM, Poul-Henning Kamp wrote:

> In message <06F9AFC0-24C5-4315-9B22-F7F8EE1913C4 at mosso.com>, Adrian  
> Otto writes
> :
>
>>> There is no intrinsic limit on object size in varnish, but it is not
>>> particular geared towards such use.  The major stumbling block is  
>>> that
>>> Varnish will fetch the entire file from the backend before starting
>>> transmission to the client.
>>
>> Have you considered allowing a configuration for streaming a file to
>> a client while it's being fetched from the backend server?
>
> Yes.
>
> Do I have a sponsor for it ?  No.
>
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.



From jfbustarret at wat.tv  Mon Jun 16 15:53:00 2008
From: jfbustarret at wat.tv (BUSTARRET, Jean-francois)
Date: Mon, 16 Jun 2008 17:53:00 +0200
Subject: about big flv files 
In-Reply-To: <C25651DA-E934-4E79-8A6F-4C3896419547@mosso.com>
References: <19853.1213630137@critter.freebsd.dk>
	<C25651DA-E934-4E79-8A6F-4C3896419547@mosso.com>
Message-ID: <53C652A09719C54DA24741D0157CB269011238C0@TFPRDEXS1.tf1.groupetf1.fr>


I might also be interested. 

> -----Message d'origine-----
> De : varnish-dev-bounces at projects.linpro.no 
> [mailto:varnish-dev-bounces at projects.linpro.no] De la part de 
> Adrian Otto
> Envoy? : lundi 16 juin 2008 17:28
> ? : varnish-dev at projects.linpro.no
> Objet : Re: about big flv files 
> 
> I'll proudly sponsor it, along with at least one other 
> important feature I'm seeking. Simply contact me directly and 
> we can make arrangements.
> 
> Thanks,
> 
> Adrian
> 
> On Jun 16, 2008, at 8:28 AM, Poul-Henning Kamp wrote:
> 
> > In message <06F9AFC0-24C5-4315-9B22-F7F8EE1913C4 at mosso.com>, Adrian 
> > Otto writes
> > :
> >
> >>> There is no intrinsic limit on object size in varnish, 
> but it is not 
> >>> particular geared towards such use.  The major stumbling block is 
> >>> that Varnish will fetch the entire file from the backend before 
> >>> starting transmission to the client.
> >>
> >> Have you considered allowing a configuration for streaming 
> a file to 
> >> a client while it's being fetched from the backend server?
> >
> > Yes.
> >
> > Do I have a sponsor for it ?  No.
> >
> >
> > -- 
> > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> > phk at FreeBSD.ORG         | TCP/IP since RFC 956
> > FreeBSD committer       | BSD since 4.3-tahoe
> > Never attribute to malice what can adequately be explained by 
> > incompetence.
> 
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev
> 


From chenxy.china at gmail.com  Thu Jun 19 08:34:42 2008
From: chenxy.china at gmail.com (chen xiaoyong)
Date: Thu, 19 Jun 2008 16:34:42 +0800
Subject: a problem about _.vsl log file
Message-ID: <7ab169e90806190134p68d978b0td58480e07ae70a60@mail.gmail.com>

Hey guys,

I do a high load test, the test keep one hour.  One hour later , varnish
works fine with restarting child process some times but don't write log
file.  It don't write anything in _.vsl file after a period of time , even
these keep alive words  'Rd ping ...' ,'Wr 0 200 PONG...' . Prior to this,
it wrote about 16 M .    What's wrong ?

I don't know how to get gdb information , so Not  'Include a backtrace of
the core-dump' ,sorry

The test case detail:
    servers:
       10.217.12.67      A siege (a opensource software), used to simulate
with the http requests
       10.217.12.19      varnish server
       10.210.128.199  apache server , the backend of varnish with flv files
    command( run in 10.217.12.67):
        siege -c 100 -i -l  -f test2.url
    test2.url content :

http://10.217.12.19/files/0d8549c4833a494f1f7d399fdbdcb81413845300.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434007.flv

http://10.217.12.19/files/0e164d6c9214fef3c62273d1dc48558013845658.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434005.flv

http://10.217.12.19/files/125c8370ce789702aa77724c9f5f7b8413819002.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434006.flv

http://10.217.12.19/files/53a01e8576a2482212879607aa8d9bc013845100.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434002.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434004.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434828.flv

http://10.217.12.19/files/bb4cfcd666d8c0fb3bdea04a6991d3458966027.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434003.flv

http://10.217.12.19/files/ee360dfee7e0937cd3771c5940cd120913845212.flv

http://10.217.12.19/files/716bccbfe6cdac7e51e117b2e0cea61d11434001.flv

The environment of varnishd
   os :
       FreeBSD 7.0-RELEASE , 5G Disk free , 512M

   varnishd version :
       trunk  2008-05-26

   command:
       /usr/local/sbin/varnishd -n /home/varnish/ -f /home/varnish/flv.vcl
-a 0.0.0.0:80 <http://0.0.0.0/> -s
file,/home/varnish/varnish_cache.data,500M -g varnish -u varnish -w
3000,5120,10 -T 127.0.0.1:3500 -p client_http11=on

   data and log files:
       %ls -l /home/varnish
       total 508672
       -rw-r--r--  1 root     varnish   83886744 Jun 17 03:03 _.vsl
       -rw-r--r--  1 varnish  varnish       1166 May 30 17:56 flv.vcl
       -rw-------  1 root     varnish  524288000 Jun 19 03:02
varnish_cache.data
       -rwxr-xr-x  1 root     varnish      14195 Jun 16 14:44 vcl.1P9zoqAU.o

   process info:
       %ps auwx
       root     1103  0.0  0.1 85324   312  ??  Is   Mon02PM   0:00.12
varnishd: Varnish-Mgr /home/varnish/ (varnishd)
       varnish  1472  0.0 54.2 620160 272956  ??  I    Mon04PM   1:02.91
varnishd: Varnish-Chld /home/varnish/ (varnishd)

  assert message:
      tail -n 100 /var/log/messages

Jun 16 06:44:47 bsd_cxy varnishd: Child (1104) said <<Child starts>>
Jun 16 06:44:47 bsd_cxy varnishd: Child (1104) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 06:44:47 bsd_cxy varnishd: Child (1104) said <<Ready>>
Jun 16 16:02:29 bsd_cxy kernel: pid 1104 (varnishd), uid 1002: exited on
signal 6
Jun 16 08:02:29 bsd_cxy varnishd: Child (1104) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 16 08:02:29 bsd_cxy varnishd: Child (1104) said <<  Condition((st) !=
NULL) not true.>>
Jun 16 08:02:29 bsd_cxy varnishd: Child (1104) said <<  errno = 32 (Broken
pipe)>>
Jun 16 08:02:29 bsd_cxy varnishd: Child (1403) said <<Child starts>>
Jun 16 08:02:29 bsd_cxy varnishd: Child (1403) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 08:02:29 bsd_cxy varnishd: Child (1403) said <<Ready>>
Jun 16 16:11:02 bsd_cxy kernel: pid 1403 (varnishd), uid 1002: exited on
signal 6
Jun 16 08:11:02 bsd_cxy varnishd: Child (1403) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 16 08:11:02 bsd_cxy varnishd: Child (1403) said <<  Condition((st) !=
NULL) not true.>>
Jun 16 08:11:02 bsd_cxy varnishd: Child (1423) said <<Child starts>>
Jun 16 08:11:02 bsd_cxy varnishd: Child (1423) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 08:11:02 bsd_cxy varnishd: Child (1423) said <<Ready>>
Jun 16 16:20:24 bsd_cxy kernel: pid 1423 (varnishd), uid 1002: exited on
signal 6
Jun 16 08:20:24 bsd_cxy varnishd: Child (1423) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 16 08:20:24 bsd_cxy varnishd: Child (1423) said <<  Condition((st) !=
NULL) not true.>>
Jun 16 08:20:24 bsd_cxy varnishd: Child (1432) said <<Child starts>>
Jun 16 08:20:24 bsd_cxy varnishd: Child (1432) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 08:20:24 bsd_cxy varnishd: Child (1432) said <<Ready>>
Jun 16 16:33:06 bsd_cxy kernel: pid 1432 (varnishd), uid 1002: exited on
signal 6
Jun 16 08:33:06 bsd_cxy varnishd: Child (1432) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 16 08:33:06 bsd_cxy varnishd: Child (1432) said <<  Condition((st) !=
NULL) not true.>>
Jun 16 08:33:06 bsd_cxy varnishd: Child (1432) said <<  errno = 32 (Broken
pipe)>>
Jun 16 08:33:06 bsd_cxy varnishd: Child (1465) said <<Child starts>>
Jun 16 08:33:06 bsd_cxy varnishd: Child (1465) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 08:33:06 bsd_cxy varnishd: Child (1465) said <<Ready>>
Jun 16 16:42:42 bsd_cxy kernel: pid 1465 (varnishd), uid 1002: exited on
signal 6
Jun 16 08:42:42 bsd_cxy varnishd: Child (1465) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 16 08:42:42 bsd_cxy varnishd: Child (1465) said <<  Condition((st) !=
NULL) not true.>>
Jun 16 08:42:42 bsd_cxy varnishd: Child (1472) said <<Child starts>>
Jun 16 08:42:42 bsd_cxy varnishd: Child (1472) said <<managed to mmap
524288000 bytes of 524288000>>
Jun 16 08:42:42 bsd_cxy varnishd: Child (1472) said <<Ready>>

     flv.vcl :

%cat flv.vcl
         backend host2{
             .host = "10.210.128.199";
             .port = "http";
         }
         acl local {
             "localhost";         /* myself */
             "10.217.12.0"/24;      /* and everyone on the local network */
         }

sub vcl_recv {

    if (req.request == "PURGE") {
       if (!client.ip ~ local) {
             error 405 "Not allowed.";
       }
       lookup;
    }


    if (req.request != "GET" && req.request != "HEAD") {
        pipe;
    }

    if (req.http.Expect) {
        pipe;
    }
    if (req.http.Authenticate) { // was (req.http.Authenticate ||
req.http.Cookie)
        pass;
    }
    lookup;
}

sub vcl_pipe {
    pipe;
}

sub vcl_pass {
    pass;
}

sub vcl_hit {
    if (!obj.cacheable) {
        pass;
    }
    deliver;
}

sub vcl_miss {
    fetch;
}

sub vcl_fetch {
    if (!obj.valid) {
        error;
    }
    if (!obj.cacheable) {
        pass;
    }
    if (obj.http.Set-Cookie) {
        pass;
    }
    insert;
}

sub vcl_deliver {
    deliver;
}

sub vcl_timeout {
    discard;
}

sub vcl_discard {
    discard;
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080619/eba21ecb/attachment.html>

From phk at phk.freebsd.dk  Fri Jun 20 07:15:09 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 20 Jun 2008 07:15:09 +0000
Subject: a problem about _.vsl log file 
In-Reply-To: Your message of "Thu, 19 Jun 2008 16:34:42 +0800."
	<7ab169e90806190134p68d978b0td58480e07ae70a60@mail.gmail.com> 
Message-ID: <10372.1213946109@critter.freebsd.dk>

In message <7ab169e90806190134p68d978b0td58480e07ae70a60 at mail.gmail.com>, "chen
 xiaoyong" writes:

>I do a high load test, the test keep one hour.  One hour later , varnish
>works fine with restarting child process some times but don't write log
>file.  It don't write anything in _.vsl file after a period of time , even
>these keep alive words  'Rd ping ...' ,'Wr 0 200 PONG...' . Prior to this,
>it wrote about 16 M .    What's wrong ?

>Jun 16 08:02:29 bsd_cxy varnishd: Child (1104) said <<Assert error in
>STV_alloc(), stevedore.c line 71:>>
>Jun 16 08:02:29 bsd_cxy varnishd: Child (1104) said <<  Condition((st) !=
>NULL) not true.>>

You seem to run out of storage.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Sat Jun 21 10:11:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 21 Jun 2008 10:11:50 +0000
Subject: pre-2.0 Trac Ticket Triage
Message-ID: <3462.1214043110@critter.freebsd.dk>


As some of you will have discovered, I am busy going through the
tickets in Trac in preparation for 2.0 and have closed all those
tickets that I deemed to be of no help to the project any longer.

Since the 2.0 release will likely increase interest in Varnish
considerably, I want to avoid having 1.x related bugs cluttering
the database, when the default answer to any problem with 1.x
pretty soon will be: Update to 2.x.

Quite a number of tickets I have closed were "heisenbugs" at the
time of filing, and the majority of them relates to the two major
race conditions we had: kqueue/sess and backend/addr.  Obviously
I have no iron-clad guarantee that there is not another bug
hiding there as well, but I trust my users to open new tickets
in that case.

Other more planning-oriented tickets have been moved to the wiki.

If you feel any of your tickets were closed in error, please don't
hessitate to say so.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From sky at crucially.net  Mon Jun 23 07:46:03 2008
From: sky at crucially.net (Artur Bergman)
Date: Mon, 23 Jun 2008 00:46:03 -0700
Subject: HTCP Purger
Message-ID: <E10DDF5D-3C2A-498D-8C93-E1630C51C352@crucially.net>

Hi,

I have written an multicast HTCP purger, essentially a perl script  
that listens to HTCP purges and then purges from the local varnish. I  
wrote it for use with Mediawiki, and I am wondering if I should  
commit it here or over at mediawiki svn?

I might look into adding the purge code in C, I don't think it would  
be hard but I was in a hurry and it was before I started looking more  
at the core. Any directions on how you would want that implemented so  
I don't go off in a wrong direction. Instinctively I would say a  
dedicated thread listening on the multicast port.

Also, I am talking about Varnish at Velocity on Tuesday :)

Cheers
Artur




From phk at phk.freebsd.dk  Mon Jun 23 08:26:54 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 23 Jun 2008 08:26:54 +0000
Subject: HTCP Purger 
In-Reply-To: Your message of "Mon, 23 Jun 2008 00:46:03 MST."
	<E10DDF5D-3C2A-498D-8C93-E1630C51C352@crucially.net> 
Message-ID: <12893.1214209614@critter.freebsd.dk>

In message <E10DDF5D-3C2A-498D-8C93-E1630C51C352 at crucially.net>, Artur Bergman 
writes:

>I have written an multicast HTCP purger, essentially a perl script  
>that listens to HTCP purges and then purges from the local varnish. 

So this is only for HTCP CLR requests ?

We should probably integrate support for that in Varnish (after 2.0
is out the door!)

I suggest we add your perl script to the wiki somewhere, as an interrim
workaround.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From sky at crucially.net  Mon Jun 23 08:31:31 2008
From: sky at crucially.net (Artur Bergman)
Date: Mon, 23 Jun 2008 01:31:31 -0700
Subject: HTCP Purger
In-Reply-To: <12893.1214209614@critter.freebsd.dk>
References: <12893.1214209614@critter.freebsd.dk>
Message-ID: <B8BA3625-BADA-41F7-8A07-3DF657D8EC7D@crucially.net>


On Jun 23, 2008, at 1:26 AM, Poul-Henning Kamp wrote:

> In message <E10DDF5D-3C2A-498D-8C93-E1630C51C352 at crucially.net>,  
> Artur Bergman
> writes:
>
>> I have written an multicast HTCP purger, essentially a perl script
>> that listens to HTCP purges and then purges from the local varnish.
>
> So this is only for HTCP CLR requests ?
>

Yes

> We should probably integrate support for that in Varnish (after 2.0
> is out the door!)
>

Awww, that is no fun :), would you take ifdefed patches?

> I suggest we add your perl script to the wiki somewhere, as an  
> interrim
> workaround.

Can I get a commit bit and commit it into the tools directory? I hate  
having code on a wiki.

Cheers
Artur



From phk at phk.freebsd.dk  Mon Jun 23 08:36:01 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 23 Jun 2008 08:36:01 +0000
Subject: HTCP Purger 
In-Reply-To: Your message of "Mon, 23 Jun 2008 01:31:31 MST."
	<B8BA3625-BADA-41F7-8A07-3DF657D8EC7D@crucially.net> 
Message-ID: <12946.1214210161@critter.freebsd.dk>

In message <B8BA3625-BADA-41F7-8A07-3DF657D8EC7D at crucially.net>, Artur Bergman 
writes:
>On Jun 23, 2008, at 1:26 AM, Poul-Henning Kamp wrote:
>
>> In message <E10DDF5D-3C2A-498D-8C93-E1630C51C352 at crucially.net>,  
>> Artur Bergman
>> writes:
>>
>>> I have written an multicast HTCP purger, essentially a perl script
>>> that listens to HTCP purges and then purges from the local varnish.
>>
>> So this is only for HTCP CLR requests ?
>
>Yes
>
>> We should probably integrate support for that in Varnish (after 2.0
>> is out the door!)
>
>Awww, that is no fun :), would you take ifdefed patches?

Sure.

>> I suggest we add your perl script to the wiki somewhere, as an  
>> interrim
>> workaround.
>
>Can I get a commit bit and commit it into the tools directory? I hate  
>having code on a wiki.

I'd really prefer to not put temporary tools in the source tree,
because that way it gets an official part of the release and all
that jazz.

How long is it anyway ?  Can you mail it to me ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Wed Jun 25 22:31:27 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 25 Jun 2008 22:31:27 +0000
Subject: GCOV and automated testing
Message-ID: <5985.1214433087@critter.freebsd.dk>


I have spent the evening running a gcov'ed varnishd and writing
a couple of test-cases to get more code covered.

Unfortunately GCC's gcov does not support shared libraries,
I'm not sure I understand why that would be hard, but they don't
and I'm not going to spend time on it right now.

The lack of shared library coverage means that we only have the
central bin/varnishd directory covered and not the VCL compiler and
libvarnish libraries.

I have not spent time on the other programs (varnishlog, varnishstat
etc).

For bin/varnishd, We have 62.12% coverage right now, which is pretty
decent for just 27 testcases.

I'll add more testcases to get better coverage as we go, but
I need to work a bit on the tester to make it less tedious.

Some bits of code are going to be annoying to test, in particular
expiry and grace: even with very short TTLs, the tests will run on
timescales of a minute, rather than a few seconds.

The plan is to get this stuff running in a crontjob on projects.linpro.no
so that the results and statistics can be accessed via the web, but
various weird things prevent that right now, so I'll just run it
ad-hoc on my laptop.

If interested, I have uploaded a copy of the colorcoded results here:

	http://phk.freebsd.dk/misc/varnish-gcov/bin/varnishd/

Red code has not been executed by any test-case.

Testcases are of course welcome :-)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.