From zabrane3 at gmail.com  Sun Aug  1 01:58:48 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Sun, 1 Aug 2010 03:58:48 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
Message-ID: <AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>

Hi guys,

Could someone from varnish dev team tell me where to start looking into to
the source code to fix this "blocking problem".
I'll quickly provide a patch!

Thanks in advance guys.
-- 
Regards
Zabrane

2010/7/12 zabrane Mikael <zabrane3 at gmail.com>

> Hi Caunter,
>
> Microsft IIS sends these headers, our stack is IIS7 to varnish.
>>
>
>
>> 316 TxHeader     c Content-Type: text/xml; charset=utf-8
>>
>  316 TxHeader     c Content-Encoding: gzip
>>  316 TxHeader     c Vary: Accept-Encoding
>> ...
>>
>
> Yep I can see that, but not in our case.
>
> Something is stripping headers, see varnishlog output below
>
>
> Accessing our caching server directly works for us for years. We noticed
> that problem only today
> (as we started using Varnish only last week)
>
> --
> Regards
> Zabrane
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100801/26a45687/attachment.html>

From hc at codecompany.dk  Sun Aug  1 18:44:57 2010
From: hc at codecompany.dk (HC Saustrup)
Date: Sun, 01 Aug 2010 20:44:57 +0200
Subject: varnish-2.1.3.tar.gz - typo in varnish.spec
Message-ID: <4C55C0A9.5020703@codecompany.dk>

Version says "2.1.2". Works find when correcting to "2.1.3".

Cheers,
HC




From hc at codecompany.dk  Sun Aug  1 19:38:15 2010
From: hc at codecompany.dk (HC Saustrup)
Date: Sun, 01 Aug 2010 21:38:15 +0200
Subject: Sporadic "backend write error: 11" errors
In-Reply-To: <4C52E4FE.3090001@codecompany.dk>
References: <4C52E4FE.3090001@codecompany.dk>
Message-ID: <4C55CD27.1040402@codecompany.dk>

On 7/30/10 4:43 PM, HC Saustrup wrote:
>
>    46 FetchError   c backend write error: 11 <-- oops!
Error 11 seems to be "Resource temporarily unavailable."

I'm grasping at straws here, but could this error show up if the client 
closed the connection before the request was answered by the backend?

I tried dumping a request that failed with tcpdump and wrote a script to 
run 20 concurrent requests like it to the backend, but all respond with 
correct data and code 200 (3000-16000ms/request - way over the 5000ms 
where most of the sporadic 503's happen). I simply can't get it to fail 
myself, neither by script or via the browser. I get plenty errors in 
varnishlog still, though :-(

Any clues?

Cheers,
HC




From schmidt at ze.tum.de  Mon Aug  2 07:50:19 2010
From: schmidt at ze.tum.de (Gerhard Schmidt)
Date: Mon, 02 Aug 2010 09:50:19 +0200
Subject: Logging request size
Message-ID: <4C5678BB.4060201@ze.tum.de>

Hi,

I'm trying to set up some statistiks for our webserver and using varnishlog to
gather the information.

I can't find information about bytes send to a client neither per request nor
per session.

The size of the object is available if the backend delivers the Length Header
but for dynamic content no size information is available.

Are there plans to include this information in future releases?

Regards
	Estartu

-- 
-------------------------------------------------
Gerhard Schmidt       | E-Mail: schmidt at ze.tum.de
TU-M?nchen	      |
WWW & Online Services |
Tel: 089/289-25270    |
Fax: 089/289-25257    | PGP-Publickey auf Anfrage


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 544 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100802/acac31ae/attachment.bin>

From zabrane3 at gmail.com  Mon Aug  2 13:53:41 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Mon, 2 Aug 2010 15:53:41 +0200
Subject: Disable caching for a specific backend
Message-ID: <AANLkTikK8VD6psQq+EdsEsk078V=XZeXOk+uSatEKzWi@mail.gmail.com>

Hi guys,

Is there a way to tell varnish to "disable caching" all pages coming from a
specific backend?

-- 
Regards
Zabrane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100802/6fcc39e6/attachment.html>

From martin.boer at netclever.nl  Mon Aug  2 14:02:52 2010
From: martin.boer at netclever.nl (Martin Boer)
Date: Mon, 02 Aug 2010 16:02:52 +0200
Subject: X-varnish and Via question
Message-ID: <4C56D00C.101@netclever.nl>

Hi all,

In our setup we have 2 varnishes next in parallel.
If one of them gets a question which it cannot serve it will 'ask' the 
other first. If the other also doesn't have the object the question goes 
through to the backend servers.
This works fine with as side effect is that we get a 'Via 1.1 varnish, 
1.1 varnish' response header and 2 entries in the X-varnish response as 
well.
So far so good. But when I tested the HA parts of our site and 
stop/started both varnishes a couple of times the headers have been 
increasing rapidly to
'Via 1.1 varnish, 1.1 varnish, 1.1 varnish, 1.1 varnish...'
This is because when the restarted varnish asks its brother the answer 
already contains 2 'varnishes'. When restarting the other varnish it 
will ask the other one for the object and the headers increase again.

I can probably 'unset' and 'set' the response headers but is there a 
better way ?
Regards,
Martin



From perbu at varnish-software.com  Mon Aug  2 14:43:21 2010
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 2 Aug 2010 16:43:21 +0200
Subject: Disable caching for a specific backend
In-Reply-To: <AANLkTikK8VD6psQq+EdsEsk078V=XZeXOk+uSatEKzWi@mail.gmail.com>
References: <AANLkTikK8VD6psQq+EdsEsk078V=XZeXOk+uSatEKzWi@mail.gmail.com>
Message-ID: <AANLkTikNVTzRZjM4e9xTf=cr=znOfVgmGgDnVu_bQMCv@mail.gmail.com>

Hi,

On Mon, Aug 2, 2010 at 3:53 PM, zabrane Mikael <zabrane3 at gmail.com> wrote:

> Is there a way to tell varnish to "disable caching" all pages coming from a
> specific backend?

Yes. Just "pass" the requests in vcl_recv when setting the backend.

sub vcl_recv {
   if (req.host ~ "www.foo.com") {
     pass;
   }
}


-- 
Per Buer,? Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer



From jdixon at omniti.com  Mon Aug  2 16:05:03 2010
From: jdixon at omniti.com (Jason Dixon)
Date: Mon, 2 Aug 2010 12:05:03 -0400
Subject: Register now for Surge 2010
Message-ID: <20100802160503.GL4578@omniti.com>

Registration for Surge Scalability Conference 2010 is open for all
attendees!  We have an awesome lineup of leaders from across the various
communities that support highly scalable architectures, as well as the
companies that implement them.  Here's a small sampling from our list of
speakers:

John Allspaw, Etsy
Theo Schlossnagle, OmniTI
Rasmus Lerdorf, creator of PHP
Tom Cook, Facebook
Benjamin Black, fast_ip
Artur Bergman, Wikia
Christopher Brown, Opscode
Bryan Cantrill, Joyent
Baron Schwartz, Percona
Paul Querna, Cloudkick

Surge 2010 focuses on real case studies from production environments;
the lessons learned from failure and how to re-engineer your way to a
successful, highly scalable Internet architecture.  The conference takes
place at the Tremont Grand Historic Venue on Sept 30 and Oct 1, 2010 in
Baltimore, MD.  Register now to enjoy the Early Bird discount and
guarantee your seat to this year's event!

http://omniti.com/surge/2010/register

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241



From ccastro at altavoz.net  Mon Aug  2 17:10:02 2010
From: ccastro at altavoz.net (Claudio Castro)
Date: Mon, 02 Aug 2010 13:10:02 -0400
Subject: Varnish keep crashin on pipe
Message-ID: <4C56FBEA.2020804@altavoz.net>

Hi all,

I was using varnish 2.0.1, and i was force to use pipe on a php script, 
since then varnish keep crashing, now im trying with 2.1.3, but the 
crashing behavior persist.

 Child (25414) Panic message: Assert error in STV_alloc(), stevedore.c 
line 192:   Condition((st) != NULL) not true. thread = (cache-worker) 
ident = FreeBSD,6.2-RELEASE-p9,amd64,-sfile,-hcritbit,kqueue sp = 
0x1b0d008 {   fd = 223, id = 223, xid = 1979199372,   client = 
186.104.42.132:13109,   step = STP_FETCH,   handling = deliver,   
err_code = 200, err_reason = (null),   restarts = 0, esis = 0   ws = 
0x1b0d078 {      id = "sess",     {s,f,r,e} = 
{0x1b0dcc0,+568,0x0,+65536},   },   http[req] = {     ws = 
0x1b0d078[sess]       "GET",       
"/prontus_multimedia/site/artic/20100730/mmedia/MULTIMEDIA_FLV120100730163451.flv",       
"HTTP/1.1",       "Host: portada.diariosregionales.cl",       
"User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.3) 
Gecko/20100401 Firefox/3.6.3",       "Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",       
"Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3",       
"Accept-Encoding: gzip,defl


Child (25278) Panic message: Assert error in TCP_close(), tcp.c line 
243:   Condition(TCP_Check(i)) not true. errno = 22 (Invalid argument) 
thread = (cache-worker) ident = 
FreeBSD,6.2-RELEASE-p9,amd64,-sfile,-hcritbit,kqueue sp = 0x1cfc008 {   
fd = -1, id = 351, xid = 2039895765,   client = 201.241.79.76:1539,   
step = STP_PIPE,   handling = pipe,   restarts = 0, esis = 0   ws = 
0x1cfc078 {      id = "sess",     {s,f,r,e} = 
{0x1cfccc0,+456,0x0,+65536},   },   http[req] = {     ws = 
0x1cfc078[sess]       "GET",       
"/ads2/ads2.clk?ts=20081001123342&cli=admin&url=http%3A//www.mercuriovalpo.cl/clasificados/",       
"HTTP/1.1",       "Accept: */*",       "Referer: 
http://www.mercuriovalpo.cl/matriz/index.html",       "Accept-Language: 
es",       "Accept-Encoding: gzip, deflate",       "User-Agent: 
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6; .NET CLR 
2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)",       "Host: 
portada.diariosregionales.

Any ideas?

Thanks in advance.

-- 
Claudio Castro N.
Ingeniero Jefe Area de Plataforma
AltaVoz S.A.
  http://www.altavoz.net
Vi?a del Mar:
  1 Norte 461 of 208
  +56 32 276 8060
Santiago:
  Pedro de Valdivia 555 of 315
  +56 2 585 4264




From phk at phk.freebsd.dk  Tue Aug  3 06:08:54 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Aug 2010 06:08:54 +0000
Subject: Varnish keep crashin on pipe 
In-Reply-To: Your message of "Mon, 02 Aug 2010 13:10:02 -0400."
	<4C56FBEA.2020804@altavoz.net> -------- In message
	<4C56FBEA.2020804@altavoz.net>, Claudio Castro writes:
Message-ID: <54314.1280815734@critter.freebsd.dk>

> Child (25414) Panic message: Assert error in STV_alloc(), stevedore.c 
>line 192:   Condition((st) != NULL) not true. thread = (cache-worker) 
>[...]
>Any ideas?

You ran out of storage.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Tue Aug  3 07:09:42 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Aug 2010 07:09:42 +0000
Subject: FYI: FlexeLint on dns-director
Message-ID: <69076.1280819382@critter.freebsd.dk>


Please try to eliminate as many as at all possible...


FlexeLint for C/C++ (Unix) Vers. 9.00d, Copyright Gimpel Software 1985-2009

[...]

--- Module:   cache_dir_dns.c (C)
                                                                 _
	struct sockaddr_in *bps = (struct sockaddr_in *) bp->ipv4;
cache_dir_dns.c  94  Info 740: Unusual pointer cast (incompatible indirect
    types)
                                          _
	if (bp->ipv4len != len || len <= 0)
cache_dir_dns.c  96  Info 775: non-negative quantity cannot be less than zero
_
}
cache_dir_dns.c  103  Info 818: Pointer parameter 'bp' (line 89) could be
    declared as pointing to const
cache_dir_dns.c  89  Info 830: Location cited in prior message
                                                                   _
	struct sockaddr_in6 *bps = (struct sockaddr_in6 *) bp->ipv6;
cache_dir_dns.c  113  Info 740: Unusual pointer cast (incompatible indirect
    types)
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)
                                          _
	if (bp->ipv6len != len || len <= 0)
cache_dir_dns.c  115  Info 775: non-negative quantity cannot be less than zero
_
}
cache_dir_dns.c  127  Info 818: Pointer parameter 'bp' (line 107) could be
    declared as pointing to const
cache_dir_dns.c  107  Info 830: Location cited in prior message
                            _
			addr, len));
cache_dir_dns.c  139  Info 740: Unusual pointer cast (incompatible indirect
    types)
                            _
			addr, len));
cache_dir_dns.c  142  Info 740: Unusual pointer cast (incompatible indirect
    types)
cache_dir_dns.c  142  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)
_
}
cache_dir_dns.c  145  Info 818: Pointer parameter 'dir' (line 131) could be
    declared as pointing to const
cache_dir_dns.c  131  Info 830: Location cited in prior message
                                                                _
	new->hostname = calloc(sizeof(char), strlen(hostname)+1);
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                    _
#... !(new->hostname != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "n
	assert(new->hostname != NULL);
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                       _
	strcpy(new->hostname, hostname);
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                        _
		return 0;
cache_dir_dns.c  291  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message
                                                               _
				new->hosts[host] = vs->hosts[i];
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... ew->hosts[host]) != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "
#... ssert((new->hosts[host]) != NULL); assert((new->hosts[host])->magic == 0x3
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... sts[host])->magic == 0x3336351d)) vas_fail(__func__, __FILE__, __LINE__, "
#... osts[host])->magic == 0x3336351d); } while (0)
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                          _
	new->nhosts = host;
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                _
	return 1;
cache_dir_dns.c  313  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message
                                          _
	pthread_rwlock_unlock(&vs->rwlock);
cache_dir_dns.c  330  Warning 534: Ignoring return value of function
    'pthread_rwlock_unlock(struct pthread_rwlock **)' (compare with line 234,
    file /usr/include/pthread.h, module cache_acceptor.c)
/usr/include/pthread.h  234  Info 830: Location cited in prior message
                                                  _
		pthread_rwlock_unlock(&vs->rwlock);
cache_dir_dns.c  334  Warning 534: Ignoring return value of function
    'pthread_rwlock_unlock(struct pthread_rwlock **)' (compare with line 234,
    file /usr/include/pthread.h, module cache_acceptor.c)
/usr/include/pthread.h  234  Info 830: Location cited in prior message
                      _
	return backend;
cache_dir_dns.c  341  Warning 438: Last value assigned to variable 'ret'
    (defined at line 327) not used
cache_dir_dns.c  327  Info 830: Location cited in prior message
                                              _
	strncpy(hostname, p, sizeof(hostname));
cache_dir_dns.c  369  Warning 534: Ignoring return value of function
    'strncpy(char *, const char *, unsigned long)' (compare with line 98, file
    /usr/include/string.h, module cache_acceptor.c)
/usr/include/string.h  98  Info 830: Location cited in prior message
                                                                                  _
		strncat(hostname, vs->suffix, sizeof(hostname) - strlen(hostname));
cache_dir_dns.c  380  Warning 534: Ignoring return value of function
    'strncat(char *, const char *, unsigned long)' (compare with line 96, file
    /usr/include/string.h, module cache_acceptor.c)
/usr/include/string.h  96  Info 830: Location cited in prior message
        _
	return 1;
cache_dir_dns.c  414  Warning 527: Unreachable code at token 'return'
                                           _
	pthread_rwlock_destroy(&vs->rwlock);
cache_dir_dns.c  447  Warning 534: Ignoring return value of function
    'pthread_rwlock_destroy(struct pthread_rwlock **)' (compare with line 224,
    file /usr/include/pthread.h, module cache_acceptor.c)
/usr/include/pthread.h  224  Info 830: Location cited in prior message
_
}
cache_dir_dns.c  449  Warning 438: Last value assigned to variable 'vh'
    (defined at line 437) not used
cache_dir_dns.c  437  Info 830: Location cited in prior message
_
}
cache_dir_dns.c  449  Warning 550: Symbol 'vh' (line 437) not accessed
cache_dir_dns.c  437  Info 830: Location cited in prior message
                                              _
	pthread_rwlock_init(&vs->rwlock, NULL);
cache_dir_dns.c  485  Warning 534: Ignoring return value of function
    'pthread_rwlock_init(struct pthread_rwlock **, struct pthread_rwlockattr
    *const *)' (compare with line 226, file /usr/include/pthread.h, module
    cache_acceptor.c)
/usr/include/pthread.h  226  Info 830: Location cited in prior message

    --- Wrap-up for Module: cache_dir_dns.c

Info 754: local structure member 'vdi_dns::max_cache_size' (line 82, file
    cache_dir_dns.c) not referenced
cache_dir_dns.c  82  Info 830: Location cited in prior message

[...]

--- Module:   ../../lib/libvcl/vcc_dir_dns.c (C)
_
}
../../lib/libvcl/vcc_dir_dns.c  129  Info 818: Pointer parameter 'ip' (line 80)
    could be declared as pointing to const
../../lib/libvcl/vcc_dir_dns.c  80  Info 830: Location cited in prior message
                         _
	ip4 |= a[0] << 24;
../../lib/libvcl/vcc_dir_dns.c  148  Info 701: Shift left of signed quantity
    (int)
_
}
../../lib/libvcl/vcc_dir_dns.c  167  Info 818: Pointer parameter 'a' (line 141)
    could be declared as pointing to const
../../lib/libvcl/vcc_dir_dns.c  141  Info 830: Location cited in prior message
                                                                    _
		ret = sscanf(tl->t->dec, "%hhu.%hhu.%hhu.%hhu",&a[0],&a[1],&a[2],&a[3]);
../../lib/libvcl/vcc_dir_dns.c  261  Warning 561: (arg. no. 3) indirect object
    inconsistent with format
../../lib/libvcl/vcc_dir_dns.c  261  Warning 561: (arg. no. 4) indirect object
    inconsistent with format
../../lib/libvcl/vcc_dir_dns.c  261  Warning 561: (arg. no. 5) indirect object
    inconsistent with format
../../lib/libvcl/vcc_dir_dns.c  261  Warning 561: (arg. no. 6) indirect object
    inconsistent with format
                                              _
			mask = vcc_UintVal(tl);
../../lib/libvcl/vcc_dir_dns.c  266  Info 734: Loss of precision (assignment)
    (32 bits to 8 bits)
_
}
../../lib/libvcl/vcc_dir_dns.c  358  Warning 438: Last value assigned to
    variable 'nbh' (defined at line 280) not used
../../lib/libvcl/vcc_dir_dns.c  280  Info 830: Location cited in prior message
_
}
../../lib/libvcl/vcc_dir_dns.c  358  Warning 550: Symbol 'nbh' (line 280) not
    accessed
../../lib/libvcl/vcc_dir_dns.c  280  Info 830: Location cited in prior message

    --- Wrap-up for Module: ../../lib/libvcl/vcc_dir_dns.c

Info 844: Pointer variable 'dns_first' of type 'struct token *' (line 75, file
    ../../lib/libvcl/vcc_dir_dns.c) could be declared as pointing to const
../../lib/libvcl/vcc_dir_dns.c  75  Info 830: Location cited in prior message
Info 766: Header file '/usr/include/netdb.h' not used in module
    '../../lib/libvcl/vcc_dir_dns.c'

[...]

--- Module:   cache_dir_dns.c (C)
                                                                   _
	struct sockaddr_in6 *bps = (struct sockaddr_in6 *) bp->ipv6;
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 142: vdi_dns_comp_addrinfo6(?, ?, ?) #1
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)
                                   _
			addr, len));
cache_dir_dns.c  142  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 301: vdi_dns_comp_addrinfo(?, ?, ?) #1
cache_dir_dns.c  142  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  154  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  156  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  156  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  157  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  160  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  161  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  167  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  168  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  169  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message
                                                                _
	new->hostname = calloc(sizeof(char), strlen(hostname)+1);
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                    _
#... !(new->hostname != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "n
	assert(new->hostname != NULL);
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                       _
	strcpy(new->hostname, hostname);
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                        _
		return 0;
cache_dir_dns.c  291  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message
                                                               _
				new->hosts[host] = vs->hosts[i];
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... ew->hosts[host]) != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "
#... ssert((new->hosts[host]) != NULL); assert((new->hosts[host])->magic == 0x3
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... sts[host])->magic == 0x3336351d)) vas_fail(__func__, __FILE__, __LINE__, "
#... osts[host])->magic == 0x3336351d); } while (0)
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                          _
	new->nhosts = host;
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                _
	return 1;
cache_dir_dns.c  313  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  284  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  310  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

[...]

--- Module:   cache_dir_dns.c (C)
                                                                   _
	struct sockaddr_in6 *bps = (struct sockaddr_in6 *) bp->ipv6;
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 142: vdi_dns_comp_addrinfo6(?, ?, ?) #1
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 301: vdi_dns_comp_addrinfo(?, ?, ?) #1
  File cache_dir_dns.c line 142: vdi_dns_comp_addrinfo6(?, ?, ?) #2
cache_dir_dns.c  113  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)
                                                  _
	if (addr->sa_family == PF_INET && bp->ipv4) {
cache_dir_dns.c  137  Warning 613: Possible use of null pointer 'bp' in left
    argument to operator '->' [Reference: file cache_backend.c: line 552; file
    cache_dir_dns.c: line 136]
cache_backend.c  552  Info 831: Reference cited in prior message
cache_dir_dns.c  136  Info 831: Reference cited in prior message
                                                          _
	} else if (addr->sa_family == PF_INET6 && bp->ipv6) {
cache_dir_dns.c  140  Warning 613: Possible use of null pointer 'bp' in left
    argument to operator '->' [Reference: file cache_backend.c: line 552; file
    cache_dir_dns.c: line 136]
cache_backend.c  552  Info 831: Reference cited in prior message
cache_dir_dns.c  136  Info 831: Reference cited in prior message
                                   _
			addr, len));
cache_dir_dns.c  142  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 301: vdi_dns_comp_addrinfo(?, ?, ?) #1
cache_dir_dns.c  137  Warning 613: Possible use of null pointer 'bp' in left
    argument to operator '->' [Reference: file cache_backend.c: line 552; file
    cache_dir_dns.c: line 136]
cache_backend.c  552  Info 831: Reference cited in prior message
cache_dir_dns.c  136  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 301: vdi_dns_comp_addrinfo(?, ?, ?) #1
cache_dir_dns.c  140  Warning 613: Possible use of null pointer 'bp' in left
    argument to operator '->' [Reference: file cache_backend.c: line 552; file
    cache_dir_dns.c: line 136]
cache_backend.c  552  Info 831: Reference cited in prior message
cache_dir_dns.c  136  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 301: vdi_dns_comp_addrinfo(?, ?, ?) #1
cache_dir_dns.c  142  Info 826: Suspicious pointer-to-pointer conversion (area
    too small)

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  154  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  156  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  156  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  157  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  160  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  161  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  167  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  168  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 312: vdi_dns_pick_host(?, [1]? | 0?) #2
cache_dir_dns.c  169  Warning 613: Possible use of null pointer 'group' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: lines 281, 312]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
cache_dir_dns.c  312  Info 831: Reference cited in prior message
                                                                _
	new->hostname = calloc(sizeof(char), strlen(hostname)+1);
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                    _
#... !(new->hostname != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "n
	assert(new->hostname != NULL);
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                       _
	strcpy(new->hostname, hostname);
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                        _
		return 0;
cache_dir_dns.c  291  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message
                                                               _
				new->hosts[host] = vs->hosts[i];
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... ew->hosts[host]) != ((void *)0))) vas_fail(__func__, __FILE__, __LINE__, "
#... ssert((new->hosts[host]) != NULL); assert((new->hosts[host])->magic == 0x3
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                                                                                  _
#... sts[host])->magic == 0x3336351d)) vas_fail(__func__, __FILE__, __LINE__, "
#... osts[host])->magic == 0x3336351d); } while (0)
				CHECK_OBJ_NOTNULL(new->hosts[host], DIRECTOR_MAGIC);
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                          _
	new->nhosts = host;
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message
                _
	return 1;
cache_dir_dns.c  313  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  284  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #1
cache_dir_dns.c  310  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  284  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, ?, [1]) #2
cache_dir_dns.c  310  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  282  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  283  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  284  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  284  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  302  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  303  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  310  Warning 613: Possible use of null pointer 'new' in left
    argument to operator '->' [Reference: file cache_dir_dns.c: line 281]
cache_dir_dns.c  281  Info 831: Reference cited in prior message

During Specific Walk:
  File cache_dir_dns.c line 382: vdi_dns_walk_cache(?, ?, [1025]) #1
  File cache_dir_dns.c line 333: vdi_dns_cache_add(?, ?, [1025], [1]) #3
cache_dir_dns.c  310  Warning 429: Custodial pointer 'new' (line 267) has not
    been freed or returned
cache_dir_dns.c  267  Info 830: Location cited in prior message

[...]

--- Global Wrap-up

[...]
Info 765: external 'b_defaults' (line 62, file ../../lib/libvcl/vcc_dir_dns.c)
    could be made static
../../lib/libvcl/vcc_dir_dns.c  62  Info 830: Location cited in prior message

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Tue Aug  3 07:16:11 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Aug 2010 07:16:11 +0000
Subject: FYI: FlexeLint on dns-director 
In-Reply-To: Your message of "Tue, 03 Aug 2010 07:09:42 GMT."
	<69076.1280819382@critter.freebsd.dk> 
Message-ID: <69113.1280819771@critter.freebsd.dk>

In message <69076.1280819382 at critter.freebsd.dk>, "Poul-Henning Kamp" writes:

Sorry, didn't mean to Cc' v-misc on this, I just pulled an old email
to reply to and forgot to wash the cc header.

>Please try to eliminate as many as at all possible...
>
-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From checker at d6.com  Wed Aug  4 04:30:54 2010
From: checker at d6.com (Chris Hecker)
Date: Tue, 03 Aug 2010 21:30:54 -0700
Subject: 302 redirect and hitpass objects
Message-ID: <4C58ECFE.90007@d6.com>


I'd like to directly cache 302 redirects.  Right now, they're generating 
HitPass objects, and still hitting the backend.  Can I do this?

Chris




From checker at d6.com  Wed Aug  4 04:59:06 2010
From: checker at d6.com (Chris Hecker)
Date: Tue, 03 Aug 2010 21:59:06 -0700
Subject: 302 redirect and hitpass objects
In-Reply-To: <4C58ECFE.90007@d6.com>
References: <4C58ECFE.90007@d6.com>
Message-ID: <4C58F39A.8030008@d6.com>


Okay, I figured this out in my vcl, it looks like.

Mediawiki is setting Cache-Control: private on 302's it sends back, and 
my vcl (part of which was ripped from wikia's) was passing all private 
pages.  For now, I just check obj.status == 302 and force deliver.  Is 
this a bad thing for any reason?

if (obj.http.Cache-Control ~ "private") {
         if(req.http.Cookie ~"(UserID|_session)") {
                 set obj.http.X-Cacheable = "NO:Got Session";
         } else if(obj.status == 302) {
                 #force cache this
                 set obj.ttl = 600s;
                 set obj.grace = 600s;
                 set obj.http.X-Cacheable = "YES - FORCED";
                 deliver;
         } else {
                 set obj.http.X-Cacheable = "NO:Cache-Control=private";
         }
         pass;
}

It seems to be working, although I'm still digging in my logs.

Also, is there any problem running this script:

http://kristian.blog.linpro.no/2009/02/18/easy-reloading-of-varnish-vcl/

instead of restarting varnish when I change the vcl while the site is live?

Thanks,
Chris


On 2010/08/03 21:30, Chris Hecker wrote:
>
> I'd like to directly cache 302 redirects. Right now, they're generating
> HitPass objects, and still hitting the backend. Can I do this?
>
> Chris
>



From yves at varnish-software.com  Wed Aug  4 14:34:00 2010
From: yves at varnish-software.com (Yves Hwang)
Date: Wed, 4 Aug 2010 16:34:00 +0200
Subject: hi guys!
Message-ID: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>

Hi guys!


As a newly employed developer @ Varnish Software, I thought I'd pop in an
email to introduce myself to the Varnish community.


Hello, my name is Yves and its a pleasure to be part of the Varnish Software
family.


I will be working on the Varnish management suite that is expected to come!
Keep your eyes peeled :)



Best regards,


Yves
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100804/0d1fecb7/attachment.html>

From jdixon at omniti.com  Wed Aug  4 14:38:38 2010
From: jdixon at omniti.com (Jason Dixon)
Date: Wed, 4 Aug 2010 10:38:38 -0400
Subject: hi guys!
In-Reply-To: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
References: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
Message-ID: <20100804143838.GB4578@omniti.com>

On Wed, Aug 04, 2010 at 04:34:00PM +0200, Yves Hwang wrote:
> 
> As a newly employed developer @ Varnish Software, I thought I'd pop in an
> email to introduce myself to the Varnish community.
 
Congratulations!
 
> I will be working on the Varnish management suite that is expected to come!
> Keep your eyes peeled :)

Please consider offering metrics/statistics that don't require "root
access" to Varnish.  Circonus already supports pulling in Varnish
metrics but it's not ideal to have to connect over the management port.
Discussed with the other Varnish guys at Velocity.

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241



From joao.lisanti at locaweb.com.br  Wed Aug  4 14:39:27 2010
From: joao.lisanti at locaweb.com.br (=?iso-8859-1?Q?Jo=E3o_Gabriel?=)
Date: Wed, 4 Aug 2010 11:39:27 -0300
Subject: hi guys!
In-Reply-To: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
References: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
Message-ID: <2AB8077E-34BD-4522-9AB9-0C73AFBE5DC3@locaweb.com.br>

Hi Yves,

welcome, and good luck =]


Jo?o Gabriel
CT-Linux

Locaweb 
L?der em Hosting no Brasil e na Am?rica Latina em 2009 
http://www.locaweb.com.br


On Aug 4, 2010, at 11:34 AM, Yves Hwang wrote:

> Hi guys!
> 
> 
> As a newly employed developer @ Varnish Software, I thought I'd pop in an email to introduce myself to the Varnish community.
> 
> 
> Hello, my name is Yves and its a pleasure to be part of the Varnish Software family. 
> 
> 
> I will be working on the Varnish management suite that is expected to come! Keep your eyes peeled :)
> 
> 
> 
> Best regards,
> 
> 
> Yves 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100804/6ce9cbcb/attachment.html>

From rodrigo at mercadolibre.com  Wed Aug  4 15:02:18 2010
From: rodrigo at mercadolibre.com (Rodrigo Benzaquen)
Date: Wed, 4 Aug 2010 12:02:18 -0300
Subject: hi guys!
In-Reply-To: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
References: <AANLkTi=GgV+CQK-O4Jt9mko-6QG9UptUQpbwMV3cbhJm@mail.gmail.com>
Message-ID: <00c101cb33e6$0af45330$20dcf990$@com>

Yves, welcome from Argentina.

 

 

De: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] En nombre de Yves Hwang
Enviado el: mi?rcoles, 04 de agosto de 2010 11:34 a.m.
Para: varnish-misc at varnish-cache.org
Asunto: hi guys!

 

Hi guys!

 

 

As a newly employed developer @ Varnish Software, I thought I'd pop in an
email to introduce myself to the Varnish community.

 

 

Hello, my name is Yves and its a pleasure to be part of the Varnish Software
family. 

 

 

I will be working on the Varnish management suite that is expected to come!
Keep your eyes peeled :)

 

 

 

Best regards,

 

 

Yves 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100804/cdda8fdd/attachment.html>

From scaunter at topscms.com  Wed Aug  4 20:41:52 2010
From: scaunter at topscms.com (Caunter, Stefan)
Date: Wed, 4 Aug 2010 16:41:52 -0400
Subject: 302 redirect and hitpass objects
In-Reply-To: <4C58F39A.8030008@d6.com>
References: <4C58ECFE.90007@d6.com> <4C58F39A.8030008@d6.com>
Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C22523D@TMG-EVS02.torstar.net>

Hi Chris,

302 is cacheable. Your vcl logic needs to reverse engineer the site
behaviour you want. 

You may find that using && compound conditions gets you the 302 cached
properly

if (obj.http.Cache-Control ~ "private" && obj.status == 302) {
	set obj.ttl = 600s;
	return(deliver);
}

Watch for a less specific condition grabbing the request and not leaving
it for the more specific condition.

The reload script is essential for an admin. It allows you to extract a
good version if you don't have svn set up. Note the limitations on
reload though. You cannot redefine directors on a reload, it requires a
restart.

Stefan Caunter :: Senior Systems Administrator :: TOPS
e: scaunter at topscms.com  ::  m: (416) 561-4871
www.thestar.com www.topscms.com


-----Original Message-----
From: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Chris
Hecker
Sent: August-04-10 12:59 AM
To: varnish-misc at varnish-cache.org
Subject: Re: 302 redirect and hitpass objects


Okay, I figured this out in my vcl, it looks like.

Mediawiki is setting Cache-Control: private on 302's it sends back, and 
my vcl (part of which was ripped from wikia's) was passing all private 
pages.  For now, I just check obj.status == 302 and force deliver.  Is 
this a bad thing for any reason?

if (obj.http.Cache-Control ~ "private") {
         if(req.http.Cookie ~"(UserID|_session)") {
                 set obj.http.X-Cacheable = "NO:Got Session";
         } else if(obj.status == 302) {
                 #force cache this
                 set obj.ttl = 600s;
                 set obj.grace = 600s;
                 set obj.http.X-Cacheable = "YES - FORCED";
                 deliver;
         } else {
                 set obj.http.X-Cacheable = "NO:Cache-Control=private";
         }
         pass;
}

It seems to be working, although I'm still digging in my logs.

Also, is there any problem running this script:

http://kristian.blog.linpro.no/2009/02/18/easy-reloading-of-varnish-vcl/

instead of restarting varnish when I change the vcl while the site is
live?

Thanks,
Chris


On 2010/08/03 21:30, Chris Hecker wrote:
>
> I'd like to directly cache 302 redirects. Right now, they're
generating
> HitPass objects, and still hitting the backend. Can I do this?
>
> Chris
>

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc



From mloftis at wgops.com  Wed Aug  4 21:42:36 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Wed, 04 Aug 2010 15:42:36 -0600
Subject: Possible (probable...) varnish 2.1.3 bug...
Message-ID: <3655546BF3224A408DF8FF35@[192.168.1.100]>

If varnish receives any data on a client socket while it's sending then it 
closes the socket...

I havne't tried any other test cases but happens 100% of the time on 
freebsd 8.0 w/ varnish 2.1.3 simple test case, cut and paste in a GET 
request, and hit enter twice, really quickly, it'll disconnect...

Anyone else seeing this?  I want to check others' before I go open a bug on 
it.





From rtshilston at gmail.com  Wed Aug  4 21:53:16 2010
From: rtshilston at gmail.com (Rob S)
Date: Wed, 04 Aug 2010 22:53:16 +0100
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <3655546BF3224A408DF8FF35@[192.168.1.100]>
References: <3655546BF3224A408DF8FF35@[192.168.1.100]>
Message-ID: <4C59E14C.5050602@gmail.com>

Michael,

I'm using quite a different version (varnish-2.0.4-1.el5 on centos 54), 
and I'm pretty certain I'm seeing the same behaviour.  If I make a 
simple GET request, then send some more line breaks whilst the data is 
being returned, then the response is truncated.

Rob


Michael Loftis wrote:
> If varnish receives any data on a client socket while it's sending 
> then it closes the socket...
>
> I havne't tried any other test cases but happens 100% of the time on 
> freebsd 8.0 w/ varnish 2.1.3 simple test case, cut and paste in a GET 
> request, and hit enter twice, really quickly, it'll disconnect...
>
> Anyone else seeing this?  I want to check others' before I go open a 
> bug on it.
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc




From mloftis at wgops.com  Wed Aug  4 22:25:31 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Wed, 04 Aug 2010 16:25:31 -0600
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <3655546BF3224A408DF8FF35@[192.168.1.100]>
References: <3655546BF3224A408DF8FF35@[192.168.1.100]>
Message-ID: <85C9DB5CB2EF476205B5123F@[192.168.1.100]>

Another one,

Varnishlog does not filter as advertised when using the -o option.  I don't 
know what it *does* do, but it most certainly isn't "to select only 
requests which generated a log entry with the given
     tag whose contents match the given regex." -- I just get a bunch of 
entries -- seems like all of them from files, and completely random ones 
from shared memory, none of which have my match even

(IP REPLACED BELOW)
varnishlog -r <LOGFILE> -o RxStatus 503


  674 TxHeader     - X-Varnish: 290112796
  666 SessionClose - EOF
  666 StatSess     - X.Y.Z 51110 7 1 1 0 1 1 507 11905
  660 RxProtocol   - HTTP/1.1
  660 RxStatus     - 200
  660 RxResponse   - OK
  660 RxHeader     - X-Powered-By: PHP/5.2.11
  660 RxHeader     - Expires: Thu, 19 Nov 1981 08:52:00 GMT
  660 RxHeader     - Cache-Control: no-store, no-cache, must-revalidate, 
post-check=0, pre-check=0
  660 RxHeader     - Pragma: no-cache
  660 RxHeader     - Content-Encoding: gzip


(I have around 500 hits/sec on this box)


--On Wednesday, August 04, 2010 3:42 PM -0600 Michael Loftis 
<mloftis at wgops.com> wrote:

> If varnish receives any data on a client socket while it's sending then
> it closes the socket...
>
> I havne't tried any other test cases but happens 100% of the time on
> freebsd 8.0 w/ varnish 2.1.3 simple test case, cut and paste in a GET
> request, and hit enter twice, really quickly, it'll disconnect...
>
> Anyone else seeing this?  I want to check others' before I go open a bug
> on it.
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc







From mloftis at wgops.com  Wed Aug  4 22:26:20 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Wed, 04 Aug 2010 16:26:20 -0600
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <4C59E14C.5050602@gmail.com>
References: <3655546BF3224A408DF8FF35@[192.168.1.100]>
	<4C59E14C.5050602@gmail.com>
Message-ID: <86CF8515B0B9DB48A589D088@[192.168.1.100]>

That's exactly what I'm seeing here...

I don't think it is likely to actually happen in the wild, except for from 
broken clients...And we all know there are LOTS of broken clients out there.

--On Wednesday, August 04, 2010 10:53 PM +0100 Rob S <rtshilston at gmail.com> 
wrote:

> Michael,
>
> I'm using quite a different version (varnish-2.0.4-1.el5 on centos 54),
> and I'm pretty certain I'm seeing the same behaviour.  If I make a simple
> GET request, then send some more line breaks whilst the data is being
> returned, then the response is truncated.
>
> Rob
>
>
> Michael Loftis wrote:
>> If varnish receives any data on a client socket while it's sending
>> then it closes the socket...
>>
>> I havne't tried any other test cases but happens 100% of the time on
>> freebsd 8.0 w/ varnish 2.1.3 simple test case, cut and paste in a GET
>> request, and hit enter twice, really quickly, it'll disconnect...
>>
>> Anyone else seeing this?  I want to check others' before I go open a
>> bug on it.
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>







From perbu at varnish-software.com  Thu Aug  5 07:13:57 2010
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 5 Aug 2010 09:13:57 +0200
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <85C9DB5CB2EF476205B5123F@192.168.1.100>
References: <3655546BF3224A408DF8FF35@192.168.1.100>
	<85C9DB5CB2EF476205B5123F@192.168.1.100>
Message-ID: <AANLkTikwiccbei6qpy2t2ty-Jmge-jn+t4othu9x-d-O@mail.gmail.com>

Hi,

On Thu, Aug 5, 2010 at 12:25 AM, Michael Loftis <mloftis at wgops.com> wrote:

> Varnishlog does not filter as advertised when using the -o option. ?I don't
> know what it *does* do, but it most certainly isn't "to select only requests
> which generated a log entry with the given
> ? ?tag whose contents match the given regex."

Where is the quote from?

>  I just get a bunch of
> entries -- seems like all of them from files, and completely random ones
> from shared memory, none of which have my match even
>
> (IP REPLACED BELOW)
> varnishlog -r <LOGFILE> -o RxStatus 503

The -o option doesn't take options. It just groups the transactions
together. You probably mean the -I or -i option. Check the man page
(online at http://www.varnish-cache.org/docs/reference/varnishlog/).


-- 
Per Buer,? Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer



From havardf at met.no  Thu Aug  5 07:55:48 2010
From: havardf at met.no (=?utf-8?Q?H=C3=A5vard_Futs=C3=A6ter?=)
Date: Thu, 5 Aug 2010 07:55:48 +0000 (UTC)
Subject: response time in varnishncsa
In-Reply-To: <1408779592.6152.1280994653503.JavaMail.root@imap1b>
Message-ID: <717687862.6179.1280994948003.JavaMail.root@imap1b>

Hi! Are there any plans for introducing response time in the varnishncsa log format? I mean response time as in '%D' from the apache2 log format.

I am using varnish 2.0.4, so I suppose this might have been introduced in later versions, although I haven't seen this from the changelogs.

-- 
------
Mvh,
H?vard Futs?ter
Telefon: +47 22 96 32 69



From martin at varnish-software.com  Thu Aug  5 10:55:10 2010
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 5 Aug 2010 12:55:10 +0200
Subject: Hello everyone
Message-ID: <AANLkTim-Py87zgdrNYUmiBGn=L5qg63a8Q4h5--6gvsT@mail.gmail.com>

Hello everyone.

I'll follow Yves' example, and do a quick check in with the community. My
name is Martin, and I'm a new employee at Varnish Software starting this
week. I'm a C programmer and will be working on developing the varnish
web-cache software.

Looking forward to working on this exciting product :)

Best regards,
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100805/ac3d27f1/attachment.html>

From tony.primerano at gmail.com  Thu Aug  5 14:33:17 2010
From: tony.primerano at gmail.com (Tony Primerano)
Date: Thu, 5 Aug 2010 10:33:17 -0400
Subject: Using Varnish to Proxy 1000s of different sites
Message-ID: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>

I suspect what I am trying to do here falls outside of Varnish's intended
purpose but here is what I am trying to do...

Instead of using Varnish to cache content for a single site with several
backends,  I want to use it to allow me to serve existing sites on different
domains.

For example I can run varnish on test.com and serve content from
example.comusing this configuration.

backend test {
  .host = "example.com";
  .port = "80";
}

sub vcl_recv {
  set req.http.host = "example.com";
  set req.backend = test;
  return(pass);
}

But what if I have 1000s of backends and I choose them based on the domain
that user's hit varnish with.   Is this something Varnish handles or is it
only intended to work with a handful of backends?

Also, it would be really cool if I could do something like this..
sub vcl_recv {
  set req.http.host = "example.com";
  set req.backend.host  = req.http.host;
  return(pass);
}

I'm guessing backends are defined ahead of time for connection pooling but
maybe not.

Anyway, this is long enough,  tell me if I would be nuts to try to do this
with Varnish and if so what would be better.   I'm tempted to write my own
proxy but that seems like such a waste.  :-)

Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100805/23ca7e05/attachment.html>

From perbu at varnish-software.com  Thu Aug  5 14:50:10 2010
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 5 Aug 2010 16:50:10 +0200
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
Message-ID: <AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>

On Thu, Aug 5, 2010 at 4:33 PM, Tony Primerano <tony.primerano at gmail.com> wrote:
> (..)
> But what if I have 1000s of backends and I choose them based on the domain
> that user's hit varnish with.?? Is this something Varnish handles or is it
> only intended to work with a handful of backends?

It's not built for that.  Kristians dns director might help you out a
bit, but it just entered trunk (will be in 2.1.4).

> Also, it would be really cool if I could do something like this..
> sub vcl_recv {
> ? set req.http.host = "example.com";
> ? set req.backend.host? = req.http.host;
> ? return(pass);
> }

That would be an open proxy. Spammers bonanza. Please don't do that. :-)

If you still want to do it you could have a look at
http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/

and use a non-caching proxy in stead of squid. That would work.



-- 
Per Buer,? Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer



From outbackdingo at gmail.com  Thu Aug  5 15:00:07 2010
From: outbackdingo at gmail.com (Outback Dingo)
Date: Thu, 5 Aug 2010 11:00:07 -0400
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
	<AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
Message-ID: <AANLkTinQGPZ+w-+C8ejiQaMCGew_Pfa3YWPFak=dK5XN@mail.gmail.com>

>
>
> If you still want to do it you could have a look at
>
> http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/
>
> and use a non-caching proxy in stead of squid. That would work.
>


would this make one think of delegate or nginx for the proxy part... along
with varnish ?


>
>
> --
> Per Buer,  Varnish Software
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100805/a1b7fda9/attachment.html>

From tony.primerano at gmail.com  Thu Aug  5 16:21:07 2010
From: tony.primerano at gmail.com (Tony Primerano)
Date: Thu, 5 Aug 2010 12:21:07 -0400
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
	<AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
Message-ID: <AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>

On Thu, Aug 5, 2010 at 10:50 AM, Per Buer <perbu at varnish-software.com>wrote:

> On Thu, Aug 5, 2010 at 4:33 PM, Tony Primerano <tony.primerano at gmail.com>
> wrote:
> > (..)
> > But what if I have 1000s of backends and I choose them based on the
> domain
> > that user's hit varnish with.   Is this something Varnish handles or is
> it
> > only intended to work with a handful of backends?
>
> It's not built for that.  Kristians dns director might help you out a
> bit, but it just entered trunk (will be in 2.1.4).
>

http://kristianlyng.wordpress.com/2010/08/02/varnish-backend-selection-through-dns/

Looks promising but in the end I think I end up with the same amount of
code. (a vcl_fetch with 1000s of ifs)

Some clarification on what I am trying to do..  without code samples ;-)

All traffic to foo.com should go to bar.com (update Host header and open
connection to bar.com)
All traffic to some.food.com should go to meal.com (update Host header and
open connection to meal.com)

repeat for 1000 domains and or subdomains.

I'm just allowing bar.com content to show on foo.com without modifying
foo.com's virtual host configurations.



> > Also, it would be really cool if I could do something like this..
> > sub vcl_recv {
> >   set req.http.host = "example.com";
> >   set req.backend.host  = req.http.host;
> >   return(pass);
> > }
>
> That would be an open proxy. Spammers bonanza. Please don't do that. :-)
>
> If you still want to do it you could have a look at
>
> http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/
>
> and use a non-caching proxy in stead of squid. That would work.
>
>
>
> --
> Per Buer,  Varnish Software
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100805/269d42da/attachment.html>

From mloftis at wgops.com  Thu Aug  5 17:49:34 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Thu, 05 Aug 2010 11:49:34 -0600
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <AANLkTikwiccbei6qpy2t2ty-Jmge-jn+t4othu9x-d-O@mail.gmail.com>
References: <3655546BF3224A408DF8FF35@192.168.1.100>
	<85C9DB5CB2EF476205B5123F@192.168.1.100>
	<AANLkTikwiccbei6qpy2t2ty-Jmge-jn+t4othu9x-d-O@mail.gmail.com>
Message-ID: <4C19DED8400F5E999BD31747@[192.168.1.100]>



--On Thursday, August 05, 2010 9:13 AM +0200 Per Buer 
<perbu at varnish-software.com> wrote:

> Hi,
>
> On Thu, Aug 5, 2010 at 12:25 AM, Michael Loftis <mloftis at wgops.com> wrote:
>
>> Varnishlog does not filter as advertised when using the -o option. ?I
>> don't know what it *does* do, but it most certainly isn't "to select
>> only requests which generated a log entry with the given
>> ? ?tag whose contents match the given regex."
>
> Where is the quote from?

man varnishlog



>
>>  I just get a bunch of
>> entries -- seems like all of them from files, and completely random ones
>> from shared memory, none of which have my match even
>>
>> (IP REPLACED BELOW)
>> varnishlog -r <LOGFILE> -o RxStatus 503
>
> The -o option doesn't take options. It just groups the transactions
> together. You probably mean the -I or -i option. Check the man page
> (online at http://www.varnish-cache.org/docs/reference/varnishlog/).

I didn't mean to imply -o took args.  The man page explains that a trailing 
<tag regex> is only valid with -o and filters for transactions that have 
the <tag regex> occuring in them.  So you can get all the lines relating to 
a particular <tag regex>, grouped together, which is NOT what it's doing.

Botht the web page you linked me to, and the man page, state exactly this:

"If the -o option was specified, an additional tag and regex may be 
specified to select only requests which generated a log entry with the 
given tag whose contents match the given regex."


Right after the options synopsis.  -o has a side effect enabling the 
trailing (at the end of the command line <tag regex>).

Thus, bug, or atleast, documentation error.  I lean towards bug since this 
seems like a very useful and sensible feature that should be part of the 
transaction matching code.





From adi at netstyle.ch  Thu Aug  5 20:19:59 2010
From: adi at netstyle.ch (Adrian Lienhard)
Date: Thu, 5 Aug 2010 22:19:59 +0200
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
	<AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
	<AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>
Message-ID: <B858EF97-56AC-44B0-A4B2-4CC3D1EA4480@netstyle.ch>

Hi Tony,

I've a setup with Varnish for a couple hundred domains as follows:

Client -> Varnish -> Apache -> Backend 1, ... Backend n

Varnish has Apache as a single backend. Apache serves as a reverse proxy to rely requests to the respective backend instance based on the domain name. That is, the selection of the backend does not happen in Varnish but in Apache using RewriteMap and RewriteRule. Works very well, although, we haven't rolled it out yet.

Cheers,
Adrian

___________________
http://www.adrian-lienhard.ch/

On Aug 5, 2010, at 18:21 , Tony Primerano wrote:

> On Thu, Aug 5, 2010 at 10:50 AM, Per Buer <perbu at varnish-software.com>wrote:
> 
>> On Thu, Aug 5, 2010 at 4:33 PM, Tony Primerano <tony.primerano at gmail.com>
>> wrote:
>>> (..)
>>> But what if I have 1000s of backends and I choose them based on the
>> domain
>>> that user's hit varnish with.   Is this something Varnish handles or is
>> it
>>> only intended to work with a handful of backends?
>> 
>> It's not built for that.  Kristians dns director might help you out a
>> bit, but it just entered trunk (will be in 2.1.4).
>> 
> 
> http://kristianlyng.wordpress.com/2010/08/02/varnish-backend-selection-through-dns/
> 
> Looks promising but in the end I think I end up with the same amount of
> code. (a vcl_fetch with 1000s of ifs)
> 
> Some clarification on what I am trying to do..  without code samples ;-)
> 
> All traffic to foo.com should go to bar.com (update Host header and open
> connection to bar.com)
> All traffic to some.food.com should go to meal.com (update Host header and
> open connection to meal.com)
> 
> repeat for 1000 domains and or subdomains.
> 
> I'm just allowing bar.com content to show on foo.com without modifying
> foo.com's virtual host configurations.
> 
> 
> 
>>> Also, it would be really cool if I could do something like this..
>>> sub vcl_recv {
>>>  set req.http.host = "example.com";
>>>  set req.backend.host  = req.http.host;
>>>  return(pass);
>>> }
>> 
>> That would be an open proxy. Spammers bonanza. Please don't do that. :-)
>> 
>> If you still want to do it you could have a look at
>> 
>> http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/
>> 
>> and use a non-caching proxy in stead of squid. That would work.
>> 
>> 
>> 
>> --
>> Per Buer,  Varnish Software
>> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
>> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc








From varnish at mm.quex.org  Fri Aug  6 02:14:54 2010
From: varnish at mm.quex.org (Michael Alger)
Date: Fri, 6 Aug 2010 10:14:54 +0800
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
	<AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
	<AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>
Message-ID: <20100806021454.GA22378@grum.quex.org>

On Thu, Aug 05, 2010 at 12:21:07PM -0400, Tony Primerano wrote:
> On Thu, Aug 5, 2010 at 10:50 AM, Per Buer <perbu at varnish-software.com>wrote:
> 
> > On Thu, Aug 5, 2010 at 4:33 PM, Tony Primerano <tony.primerano at gmail.com>
> > wrote:
> >
> > > But what if I have 1000s of backends and I choose them based on
> > > the domain that user's hit varnish with.   Is this something
> > > Varnish handles or is it only intended to work with a handful of
> > > backends?
> >
> > It's not built for that.  Kristians dns director might help you out
> > a bit, but it just entered trunk (will be in 2.1.4).
> 
> http://kristianlyng.wordpress.com/2010/08/02/varnish-backend-selection-through-dns/
> 
> Looks promising but in the end I think I end up with the same amount of
> code. (a vcl_fetch with 1000s of ifs)

I don't think there's any avoiding that unless there's some kind of
logical, programmable pattern for mapping the requested host to the
destination host. The examples you gave sound fairly arbitrary, which
just means you'll have to have a great big list.

How many actual servers (as opposed to domains) are serving as
destinations? If it's still in the thousands, then you're probably
better off looking at a forward-proxy for your solution, although as
mentioned you could put Varnish in front of it to provide caching.

If the number of servers you connect to is more reasonable, then it
might be plausible to use varnish directly. The Host: header can be
changed by VCL and is independent of the backend definition.

The main performance issue you'd have, I think, is from the sheer number
of rewrite rules you'd have to have. This is especially the case if
you're having to run a string comparison or regular expression match
over every rule until you find a match. If you control the DNS for the
proxied domain, you can mitigate this a bit by spreading them across
multiple IP addresses.

For a better solution, I think you'd want to have something that can
look up the requested domain name using a hash table or similar, so you
get essentially constant-time lookups. I don't know of anything that can
do that "out of the box", but you could do it with Varnish using inline
C. It might be worthwhile doing a small-scale test with regular VCL for
the rewrites on a few dozen domains as a proof-of-concept, then if
you're happy with the results look at creating a more scalable rewriter.



From zabrane3 at gmail.com  Fri Aug  6 07:30:25 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Fri, 6 Aug 2010 09:30:25 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
Message-ID: <AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>

After days of researchs, this is a "Varnish" bug (Tiket 733:
http://varnish-cache.org/ticket/733).
Varnish should leave the HTTP response as it is without computing any
"Content-Length" in this case.

Please, help guys? We're blocked for more than 4 weeks!
I'm ready to send patch is someone can tell me where to tool in Varnish's C
code.
Thx

-- 
Regards
Zabrane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100806/e75694c9/attachment.html>

From ask at develooper.com  Fri Aug  6 08:22:20 2010
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Fri, 6 Aug 2010 01:22:20 -0700
Subject: Varnish computes Content-Length and set it to 0 for one of my URL!
In-Reply-To: <AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
Message-ID: <C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>


On Aug 6, 2010, at 0:30, zabrane Mikael wrote:

> After days of researchs, this is a "Varnish" bug (Tiket 733: http://varnish-cache.org/ticket/733).
> Varnish should leave the HTTP response as it is without computing any "Content-Length" in this case.
> 
> Please, help guys? We're blocked for more than 4 weeks!

You need to at least post full Varnish logs for anyone to be able to help you I suspect.   Most reasonable HTTP/1.1 implementations use either a Content-Length header or Transfer-Encoding to manage the message body length.


 - ask

-- 
http://develooper.com/ - http://askask.com/





From zabrane3 at gmail.com  Fri Aug  6 13:17:10 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Fri, 6 Aug 2010 15:17:10 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
Message-ID: <AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>

Hi Ask,

Which part of the log file do you want exactly (as a company, we've
privacy stuffs to respect)?

But the problem can be easily reproduced.?Here's an example:
$ cat bug.vcl
backend squid {
??.host = "127.0.0.1";
??.port = "3128";
}
backend cache {
??.host = "127.0.0.1";
??.port = "7676";
}
sub vcl_recv {
??// set default backend
??set req.backend = squid;
??// set default proto
??set req.proto = "HTTP/1.0";
  [...]
}

1) suppose squid is up and running, and used as a Varnish backend
# /etc/init.d/squid start

2) start Varnish:
# varnishd -n foo -f bug.vcl -s malloc,512M -a :6081
storage_malloc: max size 512 MB.
Using old SHMFILE

3) Test with CURL:
3.1) Getting ressource directly from live web without Varnish :

$ curl -0 -v -I?--no-keepalive?http://www.groupama.fr/
HTTP/1.1 200 OK
Connection: close
Date: Fri, 06 Aug 2010 12:42:24 GMT
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: jsessionid=McDQbrxZf2xTWGbzVGZ3G8GcG7hvMWv8Kv8Dk1PVpm3kQZty23yz!204719863;
path=/
[...]

3.2) Using CURL throught Squid (without varnish) :

HTTP/1.0 200 OK
Date: Fri, 06 Aug 2010 13:00:57 GMT
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: jsessionid=McHJmT7mp3hk23V4LQl1GkzQgMkS5JD7JSVhZk0hdyKFhM1shD2Z!204719863;
path=/
X-Cache: MISS from soron
X-Cache-Lookup: MISS from soron:3128
Via: 1.1 soron:3128 (squid/2.7.STABLE3)
Connection: keep-alive
Proxy-Connection: keep-alive
[...]

3.3) Now, using Varnish :
$ curl -0 -v -I?--no-keepalive --proxy 127.0.0.1:6081?http://www.groupama.fr/

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: jsessionid=McGpGDynT5phSP7QFlQ6Q8Wxqzk1vQTNpLcG210s1Sdp6QwD5wHb!204719863;
path=/
X-Cache: MISS from lyes-desktop
X-Cache-Lookup: MISS from lyes-desktop:3128
Via: 1.1 soron:3128 (squid/2.7.STABLE3)
Content-Length: 48930
Date: Fri, 06 Aug 2010 12:58:46 GMT
X-Varnish: 1996041575
Age: 7
Via: 1.1 varnish
Connection: close
[...]

Notice that for scenario 3.2, Squid replies with HTTP/1.0 (as
requested by "curl -0").
For scenario 3.3, Varnish reply with HTTP/1.1.

This is half of the problem. In my first email pointing out this bug,
our caching backend (running on port 7676) replies correctly:

$ curl -0 -v -I?--no-keepalive --proxy 127.0.0.1:7676?http://www.groupama.fr/
Date	Tue, 15 Jun 2010 09:18:29 GMT
Content-Type	text/html; charset=UTF-8
X-Powered-By	ASP.NET
Set-Cookie	jsessionid=MXFJmqLZ2cwnR1gyFhVDvzG59zx2KcP2vKcg6lNP4Gnz5nVky4D1!1556069365;
path=/
X-INEODEV-CRAWL-UUID	d4807176-43f8-4547-89a9-5847bab5c326
X-INEODEV-PAGE-UUID	1a6c0abd-44dd-4c32-afc5-b8e8a7f72099
X-INEODEV-RESOURCE-LEVEL	primary

Finally, accessing our caching backend through Varnish failed (notice
the Content-Length: 0):
$ curl -0 -v -I?--no-keepalive --proxy 127.0.0.1:6081?http://www.groupama.fr/

Content-Type	text/html; charset=UTF-8
X-Powered-By	ASP.NET
Set-Cookie	jsessionid=MXFJmqLZ2cwnR1gyFhVDvzG59zx2KcP2vKcg6lNP4Gnz5nVky4D1!1556069365;
path=/
X-INEODEV-CRAWL-UUID	d4807176-43f8-4547-89a9-5847bab5c326
X-INEODEV-PAGE-UUID	1a6c0abd-44dd-4c32-afc5-b8e8a7f72099
X-INEODEV-RESOURCE-LEVEL	primary
Content-Length	0        <=====
Date	Fri, 06 Aug 2010 13:09:18 GMT
X-Varnish	544081384
Age	0
Via	1.1 varnish
Connection	close


Hope this help.

--
Regards
Zabrane

2010/8/6 Ask Bj?rn Hansen <ask at develooper.com>
>
> On Aug 6, 2010, at 0:30, zabrane Mikael wrote:
>
> > After days of researchs, this is a "Varnish" bug (Tiket 733: http://varnish-cache.org/ticket/733).
> > Varnish should leave the HTTP response as it is without computing any "Content-Length" in this case.
> >
> > Please, help guys? We're blocked for more than 4 weeks!
>
> You need to at least post full Varnish logs for anyone to be able to help you I suspect. ? Most reasonable HTTP/1.1 implementations use either a Content-Length header or Transfer-Encoding to manage the message body length.
>
>
> ?- ask
>
> --
> http://develooper.com/ - http://askask.com/



From varnish at mm.quex.org  Fri Aug  6 13:55:48 2010
From: varnish at mm.quex.org (Michael Alger)
Date: Fri, 6 Aug 2010 21:55:48 +0800
Subject: Varnish computes Content-Length and set it to 0 for one of my URL!
In-Reply-To: <AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
Message-ID: <20100806135548.GA19410@grum.quex.org>

On Fri, Aug 06, 2010 at 03:17:10PM +0200, zabrane Mikael wrote:
> But the problem can be easily reproduced. Here's an example:
> $ cat bug.vcl
> backend squid {
>   host = "127.0.0.1";
>   port = "3128";
> }
> backend cache {
>   .host = "127.0.0.1";
>   .port = "7676";
> }
> 
> 1) suppose squid is up and running, and used as a Varnish backend
> # /etc/init.d/squid start
> 
> 2) start Varnish:
> # varnishd -n foo -f bug.vcl -s malloc,512M -a :6081
> storage_malloc: max size 512 MB.
> Using old SHMFILE
> 
> 3) Test with CURL:
> 3.1) Getting resource directly from live web without Varnish :
> 
> HTTP/1.0 200 OK
> [no content-length header]
> 
> 3.2) Using CURL through Squid (without varnish) :
> 
> HTTP/1.0 200 OK
> [no content-length header]
> 
> 3.3) Now, using Varnish :
> $ curl -0 -v -I --no-keepalive --proxy 127.0.0.1:6081 http://www.groupama.fr/
> 
> HTTP/1.1 200 OK
> [no content-length header]
> 
> Notice that for scenario 3.2, Squid replies with HTTP/1.0 (as
> requested by "curl -0").
> For scenario 3.3, Varnish reply with HTTP/1.1.

I don't think is likely to be relevant, and merely indicates that
squid doesn't support HTTP/1.1.

RFC 2145:

   An HTTP server SHOULD send a response version equal to the
   highest version for which the server is at least conditionally
   compliant, and whose major version is less than or equal to the
   one received in the request.

and RFC 2616:

   Due to interoperability problems with HTTP/1.0 proxies discovered
   since the publication of RFC 2068, caching proxies MUST, [...]
   upgrade the request to the highest version they support.

Squid's HTTP/1.0 response suggests that it doesn't support HTTP 1.1.

> In my first email pointing out this bug,
> our caching backend (running on port 7676) replies correctly:
> 
> $ curl -0 -v -I --no-keepalive --proxy 127.0.0.1:7676 http://www.groupama.fr/
> [no content-length header]
> 
> Finally, accessing our caching backend through Varnish failed
> (notice the Content-Length: 0):
> $ curl -0 -v -I --no-keepalive --proxy 127.0.0.1:6081 http://www.groupama.fr/
> 
> Content-Length	0

Have you examined the response from your caching backend and
compared it to the direct response, or the response from squid? It
seems like Varnish is only adding the erroneous header when it is
using your "caching backend", yes? (i.e. the thing on port 7676)

Would capturing the traffic Varnish sees with tcpdump and comparing
the two (squid and the caching backend) be feasible?

Does Varnish also have this behaviour if it fetches the data
straight from the source server, or is it only present if it fetches
it via the caching backend?



From zabrane3 at gmail.com  Fri Aug  6 15:43:03 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Fri, 6 Aug 2010 17:43:03 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <20100806135548.GA19410@grum.quex.org>
References: <AANLkTilyL-KO-EbuGBveEMgzsvKyOySM3c_c7Oc3MrtK@mail.gmail.com>
	<5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
	<20100806135548.GA19410@grum.quex.org>
Message-ID: <AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>

Hi Michael,

>> Notice that for scenario 3.2, Squid replies with HTTP/1.0 (as
>> requested by "curl -0").
>> For scenario 3.3, Varnish reply with HTTP/1.1.
>
> I don't think is likely to be relevant

I agree on that.

> and merely indicates that
> squid doesn't support HTTP/1.1.
>
> RFC 2145:
>
> ? An HTTP server SHOULD send a response version equal to the
> ? highest version for which the server is at least conditionally
> ? compliant, and whose major version is less than or equal to the
> ? one received in the request.
> and RFC 2616:
>
> ? Due to interoperability problems with HTTP/1.0 proxies discovered
> ? since the publication of RFC 2068, caching proxies MUST, [...]
> ? upgrade the request to the highest version they support.

Thanks for this pointer.

> Squid's HTTP/1.0 response suggests that it doesn't support HTTP 1.1.

Again, you're also right on that:

$ curl -1 -D -  -v -I --no-keepalive --proxy 127.0.0.1:3128
http://www.groupama.fr/
Date: Fri, 06 Aug 2010 14:34:49 GMT
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: jsessionid=McdJcjTszTRJZtf4jQVLR7RH2Z19ny2D2JSsqj04G2vKZYmjTQbj!204719863;
path=/
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: 1.0 localhost (squid/3.0.STABLE8)
Proxy-Connection: keep-alive
[...]

> Have you examined the response from your caching backend and
> compared it to the direct response, or the response from squid?

As I wrote it before, accessing our backend directly with "curl" or
any web browser
worked for us for years now. Here's an example:

$  curl -0 -D -  -v -I --no-keepalive --proxy 127.0.0.1:7676
http://www.groupama.fr/
Date	Tue, 15 Jun 2010 09:18:29 GMT
Content-Type	text/html; charset=UTF-8
X-Powered-By	ASP.NET
Set-Cookie	jsessionid=MXFJmqLZ2cwnR1gyFhVDvzG59zx2KcP2vKcg6lNP4Gnz5nVky4D1!1556069365;
path=/
[...]

> It seems like Varnish is only adding the erroneous header when it is
> using your "caching backend", yes? (i.e. the thing on port 7676)

Yep, only with our backend.

> Would capturing the traffic Varnish sees with tcpdump and comparing
> the two (squid and the caching backend) be feasible?

with wireshark, it's my first time using it (please, see enclosed files).

> Does Varnish also have this behaviour if it fetches the data
> straight from the source server, or is it only present if it fetches
> it via the caching backend?

A bit complicated to explain, but there no "source server" in reality.
Varnish can only fetch data from our "caching server" or from "squid"
(if no special HTTP
header is included in the request).

Thanks again.

-- 
Regards
Zabrane
-------------- next part --------------
No.     Time        Source                Destination           Protocol Info
      1 0.000000    127.0.0.1             127.0.0.1             TCP      43882 > ndl-aas [SYN] Seq=0 Win=32792 Len=0 MSS=16396 TSV=6247556 TSER=0 WS=6

Frame 1 (76 bytes on wire, 76 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 0, Len: 0

No.     Time        Source                Destination           Protocol Info
      2 0.000048    127.0.0.1             127.0.0.1             TCP      ndl-aas > 43882 [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=16396 TSV=6247556 TSER=6247556 WS=6

Frame 2 (76 bytes on wire, 76 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: ndl-aas (3128), Dst Port: 43882 (43882), Seq: 0, Ack: 1, Len: 0

No.     Time        Source                Destination           Protocol Info
      3 0.000069    127.0.0.1             127.0.0.1             TCP      43882 > ndl-aas [ACK] Seq=1 Ack=1 Win=32832 Len=0 TSV=6247556 TSER=6247556

Frame 3 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 1, Ack: 1, Len: 0

No.     Time        Source                Destination           Protocol Info
      4 0.000368    127.0.0.1             127.0.0.1             HTTP     HEAD http://www.groupama.fr/ HTTP/1.1 

Frame 4 (293 bytes on wire, 293 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 1, Ack: 1, Len: 225
Hypertext Transfer Protocol

No.     Time        Source                Destination           Protocol Info
      5 0.000392    127.0.0.1             127.0.0.1             TCP      ndl-aas > 43882 [ACK] Seq=1 Ack=226 Win=33856 Len=0 TSV=6247556 TSER=6247556

Frame 5 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: ndl-aas (3128), Dst Port: 43882 (43882), Seq: 1, Ack: 226, Len: 0

No.     Time        Source                Destination           Protocol Info
      6 1.769455    127.0.0.1             127.0.0.1             HTTP     HTTP/1.0 200 OK 

Frame 6 (451 bytes on wire, 451 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: ndl-aas (3128), Dst Port: 43882 (43882), Seq: 1, Ack: 226, Len: 383
Hypertext Transfer Protocol

No.     Time        Source                Destination           Protocol Info
      7 1.769490    127.0.0.1             127.0.0.1             TCP      43882 > ndl-aas [ACK] Seq=226 Ack=384 Win=33920 Len=0 TSV=6247998 TSER=6247998

Frame 7 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 226, Ack: 384, Len: 0

No.     Time        Source                Destination           Protocol Info
      8 1.770082    127.0.0.1             127.0.0.1             TCP      43882 > ndl-aas [FIN, ACK] Seq=226 Ack=384 Win=33920 Len=0 TSV=6247999 TSER=6247998

Frame 8 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 226, Ack: 384, Len: 0

No.     Time        Source                Destination           Protocol Info
      9 1.770157    127.0.0.1             127.0.0.1             TCP      ndl-aas > 43882 [FIN, ACK] Seq=384 Ack=227 Win=33856 Len=0 TSV=6247999 TSER=6247999

Frame 9 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: ndl-aas (3128), Dst Port: 43882 (43882), Seq: 384, Ack: 227, Len: 0

No.     Time        Source                Destination           Protocol Info
     10 1.770180    127.0.0.1             127.0.0.1             TCP      43882 > ndl-aas [ACK] Seq=227 Ack=385 Win=33920 Len=0 TSV=6247999 TSER=6247999

Frame 10 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 43882 (43882), Dst Port: ndl-aas (3128), Seq: 227, Ack: 385, Len: 0
-------------- next part --------------
No.     Time        Source                Destination           Protocol Info
      1 0.000000    127.0.0.1             127.0.0.1             TCP      41781 > http-alt [SYN] Seq=0 Win=32792 Len=0 MSS=16396 TSV=6371041 TSER=0 WS=6

Frame 1 (76 bytes on wire, 76 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 41781 (41781), Dst Port: http-alt (8008), Seq: 0, Len: 0
    Source port: 41781 (41781)
    Destination port: http-alt (8008)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
    Window size: 32792
    Checksum: 0x3842 [correct]
    Options: (20 bytes)
        Maximum segment size: 16396 bytes
        SACK permitted
        Timestamps: TSval 6371041, TSecr 0
        NOP
        Window scale: 6 (multiply by 64)

No.     Time        Source                Destination           Protocol Info
      2 0.000042    127.0.0.1             127.0.0.1             TCP      http-alt > 41781 [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=16396 TSV=6371041 TSER=6371041 WS=6

Frame 2 (76 bytes on wire, 76 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: http-alt (8008), Dst Port: 41781 (41781), Seq: 0, Ack: 1, Len: 0
    Source port: http-alt (8008)
    Destination port: 41781 (41781)
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
    Window size: 32768
    Checksum: 0x05c7 [correct]
    Options: (20 bytes)
        Maximum segment size: 16396 bytes
        SACK permitted
        Timestamps: TSval 6371041, TSecr 6371041
        NOP
        Window scale: 6 (multiply by 64)
    [SEQ/ACK analysis]

No.     Time        Source                Destination           Protocol Info
      3 0.000063    127.0.0.1             127.0.0.1             TCP      41781 > http-alt [ACK] Seq=1 Ack=1 Win=32832 Len=0 TSV=6371041 TSER=6371041

Frame 3 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 41781 (41781), Dst Port: http-alt (8008), Seq: 1, Ack: 1, Len: 0
    Source port: 41781 (41781)
    Destination port: http-alt (8008)
    Sequence number: 1    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x10 (ACK)
    Window size: 32832 (scaled)
    Checksum: 0xece9 [correct]
    Options: (12 bytes)
        NOP
        NOP
        Timestamps: TSval 6371041, TSecr 6371041
    [SEQ/ACK analysis]

No.     Time        Source                Destination           Protocol Info
      4 0.000132    127.0.0.1             127.0.0.1             TCP      41781 > http-alt [PSH, ACK] Seq=1 Ack=1 Win=32832 [TCP CHECKSUM INCORRECT] Len=732 TSV=6371041 TSER=6371041

Frame 4 (800 bytes on wire, 800 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: 41781 (41781), Dst Port: http-alt (8008), Seq: 1, Ack: 1, Len: 732
    Source port: 41781 (41781)
    Destination port: http-alt (8008)
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 733    (relative sequence number)]
    Acknowledgement number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x18 (PSH, ACK)
    Window size: 32832 (scaled)
    Checksum: 0x0105 [incorrect, should be 0x4b6f (maybe caused by "TCP checksum offload"?)]
    Options: (12 bytes)
        NOP
        NOP
        Timestamps: TSval 6371041, TSecr 6371041
Data (732 bytes)

0000  47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 67   GET http://www.g
0010  72 6f 75 70 61 6d 61 2e 66 72 2f 20 48 54 54 50   roupama.fr/ HTTP
0020  2f 31 2e 30 0d 0a 48 6f 73 74 3a 20 77 77 77 2e   /1.0..Host: www.
0030  67 72 6f 75 70 61 6d 61 2e 66 72 0d 0a 55 73 65   groupama.fr..Use
0040  72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61   r-Agent: Mozilla
0050  2f 35 2e 30 20 28 58 31 31 3b 20 55 3b 20 4c 69   /5.0 (X11; U; Li
0060  6e 75 78 20 69 36 38 36 3b 20 66 72 3b 20 72 76   nux i686; fr; rv
0070  3a 31 2e 39 2e 31 2e 31 31 29 20 47 65 63 6b 6f   :1.9.1.11) Gecko
0080  2f 32 30 31 30 30 37 30 31 20 46 69 72 65 66 6f   /20100701 Firefo
0090  78 2f 33 2e 35 2e 31 31 0d 0a 41 63 63 65 70 74   x/3.5.11..Accept
00a0  3a 20 74 65 78 74 2f 68 74 6d 6c 2c 61 70 70 6c   : text/html,appl
00b0  69 63 61 74 69 6f 6e 2f 78 68 74 6d 6c 2b 78 6d   ication/xhtml+xm
00c0  6c 2c 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d   l,application/xm
00d0  6c 3b 71 3d 30 2e 39 2c 2a 2f 2a 3b 71 3d 30 2e   l;q=0.9,*/*;q=0.
00e0  38 0d 0a 41 63 63 65 70 74 2d 4c 61 6e 67 75 61   8..Accept-Langua
00f0  67 65 3a 20 66 72 2c 66 72 2d 66 72 3b 71 3d 30   ge: fr,fr-fr;q=0
0100  2e 38 2c 65 6e 2d 75 73 3b 71 3d 30 2e 35 2c 65   .8,en-us;q=0.5,e
0110  6e 3b 71 3d 30 2e 33 0d 0a 41 63 63 65 70 74 2d   n;q=0.3..Accept-
0120  45 6e 63 6f 64 69 6e 67 3a 20 67 7a 69 70 2c 64   Encoding: gzip,d
0130  65 66 6c 61 74 65 0d 0a 41 63 63 65 70 74 2d 43   eflate..Accept-C
0140  68 61 72 73 65 74 3a 20 49 53 4f 2d 38 38 35 39   harset: ISO-8859
0150  2d 31 2c 75 74 66 2d 38 3b 71 3d 30 2e 37 2c 2a   -1,utf-8;q=0.7,*
0160  3b 71 3d 30 2e 37 0d 0a 43 6f 6e 6e 65 63 74 69   ;q=0.7..Connecti
0170  6f 6e 3a 20 63 6c 6f 73 65 0d 0a 50 72 6f 78 79   on: close..Proxy
0180  2d 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f   -Connection: clo
[...]
02d0  6e 6f 2d 63 61 63 68 65 0d 0a 0d 0a               no-cache....
    Data: 47455420687474703A2F2F7777772E67726F7570616D612E...

No.     Time        Source                Destination           Protocol Info
      5 0.000150    127.0.0.1             127.0.0.1             TCP      http-alt > 41781 [ACK] Seq=1 Ack=733 Win=34240 Len=0 TSV=6371041 TSER=6371041

Frame 5 (68 bytes on wire, 68 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: http-alt (8008), Dst Port: 41781 (41781), Seq: 1, Ack: 733, Len: 0
    Source port: http-alt (8008)
    Destination port: 41781 (41781)
    Sequence number: 1    (relative sequence number)
    Acknowledgement number: 733    (relative ack number)
    Header length: 32 bytes
    Flags: 0x10 (ACK)
    Window size: 34240 (scaled)
    Checksum: 0xe9f7 [correct]
    Options: (12 bytes)
        NOP
        NOP
        Timestamps: TSval 6371041, TSecr 6371041
    [SEQ/ACK analysis]

No.     Time        Source                Destination           Protocol Info
      6 0.181025    127.0.0.1             127.0.0.1             TCP      http-alt > 41781 [ACK] Seq=1 Ack=733 Win=34240 [TCP CHECKSUM INCORRECT] Len=16384 TSV=6371086 TSER=6371041

Frame 6 (16452 bytes on wire, 16452 bytes captured)
Linux cooked capture
Internet Protocol, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: http-alt (8008), Dst Port: 41781 (41781), Seq: 1, Ack: 733, Len: 16384
    Source port: http-alt (8008)
    Destination port: 41781 (41781)
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 16385    (relative sequence number)]
    Acknowledgement number: 733    (relative ack number)
    Header length: 32 bytes
    Flags: 0x10 (ACK)
    Window size: 34240 (scaled)
    Checksum: 0x3e29 [incorrect, should be 0xe331 (maybe caused by "TCP checksum offload"?)]
    Options: (12 bytes)
        NOP
        NOP
        Timestamps: TSval 6371086, TSecr 6371041
Data (16384 bytes)

0000  48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.
0010  0a 44 61 74 65 3a 20 54 75 65 2c 20 31 35 20 4a   .Date: Tue, 15 J
0020  75 6e 20 32 30 31 30 20 30 39 3a 31 38 3a 32 39   un 2010 09:18:29
0030  20 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79    GMT..Content-Ty
0040  70 65 3a 20 74 65 78 74 2f 68 74 6d 6c 3b 20 63   pe: text/html; c
0050  68 61 72 73 65 74 3d 55 54 46 2d 38 0d 0a 58 2d   harset=UTF-8..X-
0060  50 6f 77 65 72 65 64 2d 42 79 3a 20 41 53 50 2e   Powered-By: ASP.
0070  4e 45 54 0d 0a 53 65 74 2d 43 6f 6f 6b 69 65 3a   NET..Set-Cookie:
0080  20 6a 73 65 73 73 69 6f 6e 69 64 3d 4d 58 46 4a    jsessionid=MXFJ
0090  6d 71 4c 5a 32 63 77 6e 52 31 67 79 46 68 56 44   mqLZ2cwnR1gyFhVD
00a0  76 7a 47 35 39 7a 78 32 4b 63 50 32 76 4b 63 67   vzG59zx2KcP2vKcg
00b0  36 6c 4e 50 34 47 6e 7a 35 6e 56 6b 79 34 44 31   6lNP4Gnz5nVky4D1
00c0  21 31 35 35 36 30 36 39 33 36 35 3b 20 70 61 74   !1556069365; pat
00d0  68 3d 2f 0d 0a 58 2d 49 4e 45 4f 44 45 56 2d 43   h=/..X-INEODEV-C
[...]
0170  0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d   ................
0180  0a 0d 0a 20 0d 0a 20 0d 0a 0d 0a 0d 0a 0d 0a 0d   ... .. .........
0190  0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d   ................
01a0  0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d   ................
01b0  0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d   ................
01c0  0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d   ................
01d0  0a 09 0d 0a 09 09 0d 0a 09 0d 0a 09 0d 0a 0d 0a   ................
01e0  0d 0a 0d 0a 09 0d 0a 09 09 0d 0a 09 0d 0a 0d 0a   ................
01f0  0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a   ................
0200  0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a   ................
0210  0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a   ................
0220  0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a 0d 0a   ................
0230  0d 0a 0d 0a 0d 0a 0d 0a 09 0d 0a 09 0d 0a 0d 0a   ................
0240  0d 0a 0d 0a 0d 0a 09 0d 0a 0d 0a 0d 0a 3c 21 44   .............<!D
0250  4f 43 54 59 50 45 20 68 74 6d 6c 20 50 55 42 4c   OCTYPE html PUBL
0260  49 43 20 22 2d 2f 2f 57 33 43 2f 2f 44 54 44 20   IC "-//W3C//DTD 
0270  58 48 54 4d 4c 20 31 2e 30 20 54 72 61 6e 73 69   XHTML 1.0 Transi
0280  74 69 6f 6e 61 6c 2f 2f 45 4e 22 20 22 68 74 74   tional//EN" "htt
0290  70 3a 2f 2f 77 77 77 2e 77 33 2e 6f 72 67 2f 54   p://www.w3.org/T
02a0  52 2f 78 68 74 6d 6c 31 2f 44 54 44 2f 78 68 74   R/xhtml1/DTD/xht
02b0  6d 6c 31 2d 74 72 61 6e 73 69 74 69 6f 6e 61 6c   ml1-transitional
02c0  2e 64 74 64 22 3e 0d 0a 3c 68 74 6d 6c 20 78 6d   .dtd">..<html xm
02d0  6c 6e 73 3d 22 68 74 74 70 3a 2f 2f 77 77 77 2e   lns="http://www.
02e0  77 33 2e 6f 72 67 2f 31 39 39 39 2f 78 68 74 6d   w3.org/1999/xhtm
02f0  6c 22 20 6c 61 6e 67 3d 22 66 72 22 3e 0d 0a 0d   l" lang="fr">...
0300  0a 3c 68 65 61 64 3e 09 0d 0a 09 3c 6d 65 74 61   .<head>....<meta
0310  20 68 74 74 70 2d 65 71 75 69 76 3d 22 63 6f 6e    http-equiv="con
0320  74 65 6e 74 2d 74 79 70 65 22 20 63 6f 6e 74 65   tent-type" conte
0330  6e 74 3d 22 74 65 78 74 2f 68 74 6d 6c 3b 20 63   nt="text/html; c
0340  68 61 72 73 65 74 3d 49 53 4f 2d 38 38 35 39 2d   harset=ISO-8859-
0350  31 22 20 2f 3e 0d 0a 0d 0a 09 0d 0a 09 09 0d 0a   1" />...........
0360  09 09 0d 0a 09 09 09 0d 0a 0d 0a 0d 0a 0d 0a 09   ................

From perbu at varnish-software.com  Sat Aug  7 08:51:30 2010
From: perbu at varnish-software.com (Per Buer)
Date: Sat, 7 Aug 2010 10:51:30 +0200
Subject: Possible (probable...) varnish 2.1.3 bug...
In-Reply-To: <4C19DED8400F5E999BD31747@192.168.1.100>
References: <3655546BF3224A408DF8FF35@192.168.1.100>
	<85C9DB5CB2EF476205B5123F@192.168.1.100>
	<AANLkTikwiccbei6qpy2t2ty-Jmge-jn+t4othu9x-d-O@mail.gmail.com>
	<4C19DED8400F5E999BD31747@192.168.1.100>
Message-ID: <AANLkTi==inkXygpo453aa3KhHNU-O2uZNpKspo_nvdVA@mail.gmail.com>

On Thu, Aug 5, 2010 at 7:49 PM, Michael Loftis <mloftis at wgops.com> wrote:
>
> I didn't mean to imply -o took args. ?The man page explains that a trailing
> <tag regex> is only valid with -o and filters for transactions that have the
> <tag regex> occuring in them. ?So you can get all the lines relating to a
> particular <tag regex>, grouped together, which is NOT what it's doing.

Of course, you're right.

> (..)
> Thus, bug, or atleast, documentation error. ?I lean towards bug since this
> seems like a very useful and sensible feature that should be part of the
> transaction matching code.

Right.


-- 
Per Buer,? Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer



From varnish at mm.quex.org  Sun Aug  8 07:10:24 2010
From: varnish at mm.quex.org (Michael Alger)
Date: Sun, 8 Aug 2010 15:10:24 +0800
Subject: Varnish computes Content-Length and set it to 0 for one of my URL!
In-Reply-To: <AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>
References: <5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
	<20100806135548.GA19410@grum.quex.org>
	<AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>
Message-ID: <20100808071024.GA21508@grum.quex.org>

On Fri, Aug 06, 2010 at 05:43:03PM +0200, zabrane Mikael wrote:
> 
> As I wrote it before, accessing our backend directly with "curl"
> or any web browser worked for us for years now. Here's an example:
> 
> $  curl -0 -D -  -v -I --no-keepalive --proxy 127.0.0.1:7676 http://www.groupama.fr/
> Date	Tue, 15 Jun 2010 09:18:29 GMT
> Content-Type	text/html; charset=UTF-8
> X-Powered-By	ASP.NET
> Set-Cookie	jsessionid=MXFJmqLZ2cwnR1gyFhVDvzG59zx2KcP2vKcg6lNP4Gnz5nVky4D1!1556069365;
> path=/
> [...]
> 
>> It seems like Varnish is only adding the erroneous header when it is
>> using your "caching backend", yes? (i.e. the thing on port 7676)
> 
> Yep, only with our backend.
> 
>> Would capturing the traffic Varnish sees with tcpdump and comparing
>> the two (squid and the caching backend) be feasible?
> 
> with wireshark, it's my first time using it (please, see enclosed files).

Sorry, I was more interested in the actual data of the response.
Presumably Varnish is interpreting the response from squid
differently to the response from the caching backend.

Or, is there a way for us to see the response from the backend? Is
the site http://www.groupama.fr/ currently served by the caching
backend, i.e. if I point a Varnish at it I should see the same bug?



From martin.boer at netclever.nl  Mon Aug  9 08:34:30 2010
From: martin.boer at netclever.nl (Martin Boer)
Date: Mon, 09 Aug 2010 10:34:30 +0200
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
Message-ID: <4C5FBD96.70900@netclever.nl>

I think that Pound is more suitable for what you are trying to achieve. 
Although I don't know if Pound can handle a 1000 domain/backend 
combinations.
No matter what you throw at it, 1000 stays a large number.

Martin


On 08/05/2010 04:33 PM, Tony Primerano wrote:
> I suspect what I am trying to do here falls outside of Varnish's 
> intended purpose but here is what I am trying to do...
>
> Instead of using Varnish to cache content for a single site with 
> several backends,  I want to use it to allow me to serve existing 
> sites on different domains.
>
> For example I can run varnish on test.com <http://test.com> and serve 
> content from example.com <http://example.com> using this configuration.
>
> backend test {
>   .host = "example.com <http://example.com>";
>   .port = "80";
> }
>
> sub vcl_recv {
>   set req.http.host = "example.com <http://example.com>";
>   set req.backend = test;
>   return(pass);
> }
>
> But what if I have 1000s of backends and I choose them based on the 
> domain that user's hit varnish with.   Is this something Varnish 
> handles or is it only intended to work with a handful of backends?
>
> Also, it would be really cool if I could do something like this..
> sub vcl_recv {
>   set req.http.host = "example.com <http://example.com>";
>   set req.backend.host  = req.http.host;
>   return(pass);
> }
>
> I'm guessing backends are defined ahead of time for connection pooling 
> but maybe not.
>
> Anyway, this is long enough,  tell me if I would be nuts to try to do 
> this with Varnish and if so what would be better.   I'm tempted to 
> write my own proxy but that seems like such a waste.  :-)
>
> Tony
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100809/4a7d1135/attachment.html>

From zabrane3 at gmail.com  Mon Aug  9 11:24:22 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Mon, 9 Aug 2010 13:24:22 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <20100808071024.GA21508@grum.quex.org>
References: <5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
	<20100806135548.GA19410@grum.quex.org>
	<AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>
	<20100808071024.GA21508@grum.quex.org>
Message-ID: <AANLkTinkHsWGAzrBDDaGUO-a2Wq8VPRbKtwBxupc+SLF@mail.gmail.com>

Hi Michael,

I'm verry happy that the bug was fixed:
http://varnish-cache.org/ticket/733

Is there a Varnish's git/svn repository somewhere with the fix?

Thanks guys ;-)

-- 
Regards
Zabrane

2010/8/8 Michael Alger <varnish at mm.quex.org>:
> On Fri, Aug 06, 2010 at 05:43:03PM +0200, zabrane Mikael wrote:
>>
>> As I wrote it before, accessing our backend directly with "curl"
>> or any web browser worked for us for years now. Here's an example:
>>
>> $ ?curl -0 -D - ?-v -I --no-keepalive --proxy 127.0.0.1:7676 http://www.groupama.fr/
>> Date ?Tue, 15 Jun 2010 09:18:29 GMT
>> Content-Type ?text/html; charset=UTF-8
>> X-Powered-By ?ASP.NET
>> Set-Cookie ? ?jsessionid=MXFJmqLZ2cwnR1gyFhVDvzG59zx2KcP2vKcg6lNP4Gnz5nVky4D1!1556069365;
>> path=/
>> [...]
>>
>>> It seems like Varnish is only adding the erroneous header when it is
>>> using your "caching backend", yes? (i.e. the thing on port 7676)
>>
>> Yep, only with our backend.
>>
>>> Would capturing the traffic Varnish sees with tcpdump and comparing
>>> the two (squid and the caching backend) be feasible?
>>
>> with wireshark, it's my first time using it (please, see enclosed files).
>
> Sorry, I was more interested in the actual data of the response.
> Presumably Varnish is interpreting the response from squid
> differently to the response from the caching backend.
>
> Or, is there a way for us to see the response from the backend? Is
> the site http://www.groupama.fr/ currently served by the caching
> backend, i.e. if I point a Varnish at it I should see the same bug?
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



From rtshilston at gmail.com  Mon Aug  9 11:27:44 2010
From: rtshilston at gmail.com (Rob S)
Date: Mon, 09 Aug 2010 12:27:44 +0100
Subject: Varnish computes Content-Length and set it to 0 for one of my URL!
In-Reply-To: <AANLkTinkHsWGAzrBDDaGUO-a2Wq8VPRbKtwBxupc+SLF@mail.gmail.com>
References: <5324.1278941802@critter.freebsd.dk>	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>	<20100806135548.GA19410@grum.quex.org>	<AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>	<20100808071024.GA21508@grum.quex.org>
	<AANLkTinkHsWGAzrBDDaGUO-a2Wq8VPRbKtwBxupc+SLF@mail.gmail.com>
Message-ID: <4C5FE630.5070606@gmail.com>

Michael,

 From the ticket page, you'll see phk's comment with the link [5075].  
That's the commit that fixed this issue, and can be seen at:

http://varnish-cache.org/changeset/5075

You can browse all the source at http://varnish-cache.org/browser/trunk


Rob

zabrane Mikael wrote:
> Hi Michael,
>
> I'm verry happy that the bug was fixed:
> http://varnish-cache.org/ticket/733
>
> Is there a Varnish's git/svn repository somewhere with the fix?
>
> Thanks guys ;-)
>
>   




From zabrane3 at gmail.com  Mon Aug  9 12:03:54 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Mon, 9 Aug 2010 14:03:54 +0200
Subject: Varnish computes Content-Length and set it to 0 for one of my 
	URL!
In-Reply-To: <4C5FE630.5070606@gmail.com>
References: <5324.1278941802@critter.freebsd.dk>
	<AANLkTimQJjrwNa5HuuEBmz2g6AkT_iJrlnAAUQ-m0ed2@mail.gmail.com>
	<7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net>
	<AANLkTikPqyQS2emQ3xv2dY8GyfoBmTaZA3W8QnukqTyB@mail.gmail.com>
	<AANLkTi=xr0Fu0YF7=9F_frZiviv0xxhK2niUs_uyttPw@mail.gmail.com>
	<AANLkTimRHVMpChBRxwM--ivsGztuNfBdMQV3dJA_nZrd@mail.gmail.com>
	<C3EF52C7-D300-4977-B3BF-453304DCB4DC@develooper.com>
	<AANLkTimYD4pu+2o-fqW8oTWja70uGKwaWxgLf9dT1pkZ@mail.gmail.com>
	<20100806135548.GA19410@grum.quex.org>
	<AANLkTimGjROn_ScU-fW57J_gdSxuCqPP_EtNsBQ-91FL@mail.gmail.com>
	<20100808071024.GA21508@grum.quex.org>
	<AANLkTinkHsWGAzrBDDaGUO-a2Wq8VPRbKtwBxupc+SLF@mail.gmail.com>
	<4C5FE630.5070606@gmail.com>
Message-ID: <AANLkTimJQcqSAVOK3vO0HFPH_b6iXMatGKZUiLVbn+NR@mail.gmail.com>

Thanks !

2010/8/9 Rob S <rtshilston at gmail.com>:
> Michael,
>
> From the ticket page, you'll see phk's comment with the link [5075]. ?That's
> the commit that fixed this issue, and can be seen at:
>
> http://varnish-cache.org/changeset/5075
>
> You can browse all the source at http://varnish-cache.org/browser/trunk
>
>
> Rob



From ccastro at altavoz.net  Mon Aug  9 20:28:42 2010
From: ccastro at altavoz.net (Claudio Castro)
Date: Mon, 09 Aug 2010 16:28:42 -0400
Subject: pipe even if i dont set any pipe on vcl
Message-ID: <4C6064FA.7080300@altavoz.net>

I finally comment out any pipe of my vcl, but still see some pipes on 
varnishstats, this is what i get from varnishlog:

   78 ReqStart     c 163.247.80.5 1061 569291673
   78 RxRequest    c <?xml
   78 RxURL        c version="1.0"/
   78 RxProtocol   c HTTP/1.1
   78 RxHeader     c <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 
Strict//EN"  "http: //www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   78 RxHeader     c <html>   <head>     <title>304 Not 
Modified</title>   </head>   <body>     <h1>Error 304 Not 
Modified</h1>     <p>Not Modified</p>   </body>
   78 RxHeader     c </html>
   78 RxHeader     c GET 
/prontus_multimedia/site/artic/20100808/imag/FOTO_1120100808191600.jpg 
HTTP/1.1
   78 RxHeader     c Host: foo.bar.cl
   78 RxHeader     c Connection: keep-alive
   78 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 
5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.125 
Safari/533.4
   78 RxHeader     c Referer: http://www.foobar.cl/matriz/index.html
   78 RxHeader     c Accept: */*
   78 RxHeader     c Accept-Encoding: gzip,deflate,sdch
   78 RxHeader     c Accept-Language: es-ES,es;q=0.8
   78 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
   78 VCL_call     c recv pipe
   78 VCL_call     c hash hash
   78 VCL_call     c pipe pipe
   78 Backend      c 783 default default
   78 ReqEnd       c 569291673 1281385631.073717594 1281385631.212797165 
0.140153170 0.000685453 0.138394117

Any hint?

-- 
Claudio Castro N.
Ingeniero Jefe Area de Plataforma
AltaVoz S.A.
  http://www.altavoz.net
Vi?a del Mar:
  1 Norte 461 of 208
  +56 32 276 8060
Santiago:
  Pedro de Valdivia 555 of 315
  +56 2 585 4264




From checker at d6.com  Mon Aug  9 21:06:57 2010
From: checker at d6.com (Chris Hecker)
Date: Mon, 09 Aug 2010 14:06:57 -0700
Subject: pipe even if i dont set any pipe on vcl
In-Reply-To: <4C6064FA.7080300@altavoz.net>
References: <4C6064FA.7080300@altavoz.net>
Message-ID: <4C606DF1.3020007@d6.com>


The default.vcl (or its equivalent) is appended to your vcl, so if you 
fall through anywhere you're getting its behavior.

Chris


On 2010/08/09 13:28, Claudio Castro wrote:
> I finally comment out any pipe of my vcl, but still see some pipes on
> varnishstats, this is what i get from varnishlog:
>
> 78 ReqStart c 163.247.80.5 1061 569291673
> 78 RxRequest c <?xml
> 78 RxURL c version="1.0"/
> 78 RxProtocol c HTTP/1.1
> 78 RxHeader c <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
> "http: //www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
> 78 RxHeader c <html> <head> <title>304 Not Modified</title> </head>
> <body> <h1>Error 304 Not Modified</h1> <p>Not Modified</p> </body>
> 78 RxHeader c </html>
> 78 RxHeader c GET
> /prontus_multimedia/site/artic/20100808/imag/FOTO_1120100808191600.jpg
> HTTP/1.1
> 78 RxHeader c Host: foo.bar.cl
> 78 RxHeader c Connection: keep-alive
> 78 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
> en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.125
> Safari/533.4
> 78 RxHeader c Referer: http://www.foobar.cl/matriz/index.html
> 78 RxHeader c Accept: */*
> 78 RxHeader c Accept-Encoding: gzip,deflate,sdch
> 78 RxHeader c Accept-Language: es-ES,es;q=0.8
> 78 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
> 78 VCL_call c recv pipe
> 78 VCL_call c hash hash
> 78 VCL_call c pipe pipe
> 78 Backend c 783 default default
> 78 ReqEnd c 569291673 1281385631.073717594 1281385631.212797165
> 0.140153170 0.000685453 0.138394117
>
> Any hint?
>



From tony.primerano at gmail.com  Mon Aug  9 23:47:50 2010
From: tony.primerano at gmail.com (Tony Primerano)
Date: Mon, 9 Aug 2010 19:47:50 -0400
Subject: Using Varnish to Proxy 1000s of different sites
In-Reply-To: <20100806021454.GA22378@grum.quex.org>
References: <AANLkTin1dpewNZUf1i9FUb7-FxKkE-JivNm_+WkjPbwX@mail.gmail.com>
	<AANLkTinZeSP1=cgk=YO+4hoEbRUqUDruCrgFyxg7Lwjt@mail.gmail.com>
	<AANLkTinRVdpeOTT4Sd+6e7TVq0FCexCt2hNE0dSek6gq@mail.gmail.com>
	<20100806021454.GA22378@grum.quex.org>
Message-ID: <AANLkTind+v1h6CP_O+XfscSNcQekvC=vQ_VMTysUszhM@mail.gmail.com>

Thanks for all the pointers in this thread...  for now I've been pulled to
work on something else but I'll be back in the coming weeks.

A few people mentioned Apache, the reason I don't do this with Apache is I
want reloading of the configuration to be simple,  it seems Varnish reloads
are quick and easy.

I'll experiment with Varnish inline C  next and research some of the other
proxies mentioned.

Thanks for the pointers..  I'll be back.  ;-)

Tony

On Thu, Aug 5, 2010 at 10:14 PM, Michael Alger <varnish at mm.quex.org> wrote:

> On Thu, Aug 05, 2010 at 12:21:07PM -0400, Tony Primerano wrote:
> > On Thu, Aug 5, 2010 at 10:50 AM, Per Buer <perbu at varnish-software.com
> >wrote:
> >
> > > On Thu, Aug 5, 2010 at 4:33 PM, Tony Primerano <
> tony.primerano at gmail.com>
> > > wrote:
> > >
> > > > But what if I have 1000s of backends and I choose them based on
> > > > the domain that user's hit varnish with.   Is this something
> > > > Varnish handles or is it only intended to work with a handful of
> > > > backends?
> > >
> > > It's not built for that.  Kristians dns director might help you out
> > > a bit, but it just entered trunk (will be in 2.1.4).
> >
> >
> http://kristianlyng.wordpress.com/2010/08/02/varnish-backend-selection-through-dns/
> >
> > Looks promising but in the end I think I end up with the same amount of
> > code. (a vcl_fetch with 1000s of ifs)
>
> I don't think there's any avoiding that unless there's some kind of
> logical, programmable pattern for mapping the requested host to the
> destination host. The examples you gave sound fairly arbitrary, which
> just means you'll have to have a great big list.
>
> How many actual servers (as opposed to domains) are serving as
> destinations? If it's still in the thousands, then you're probably
> better off looking at a forward-proxy for your solution, although as
> mentioned you could put Varnish in front of it to provide caching.
>
> If the number of servers you connect to is more reasonable, then it
> might be plausible to use varnish directly. The Host: header can be
> changed by VCL and is independent of the backend definition.
>
> The main performance issue you'd have, I think, is from the sheer number
> of rewrite rules you'd have to have. This is especially the case if
> you're having to run a string comparison or regular expression match
> over every rule until you find a match. If you control the DNS for the
> proxied domain, you can mitigate this a bit by spreading them across
> multiple IP addresses.
>
> For a better solution, I think you'd want to have something that can
> look up the requested domain name using a hash table or similar, so you
> get essentially constant-time lookups. I don't know of anything that can
> do that "out of the box", but you could do it with Varnish using inline
> C. It might be worthwhile doing a small-scale test with regular VCL for
> the rewrites on a few dozen domains as a proof-of-concept, then if
> you're happy with the results look at creating a more scalable rewriter.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100809/f567f6aa/attachment.html>

From tfheen at varnish-software.com  Tue Aug 10 07:28:26 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue, 10 Aug 2010 09:28:26 +0200
Subject: "Total body bytes" not actually bytes delivered?
In-Reply-To: <4C515EF5.70904@d6.com> (Chris Hecker's message of "Thu, 29 Jul
	2010 03:59:01 -0700")
References: <4C515EF5.70904@d6.com>
Message-ID: <87y6cfhqp1.fsf@qurzaw.linpro.no>

]] Chris Hecker 

| Is that by design, or should it be the number of bytes successfully
| delivered, which seems more useful?

It should be the bytes delivered, so please file a ticket in trac.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From rtshilston at gmail.com  Tue Aug 10 20:05:48 2010
From: rtshilston at gmail.com (Rob S)
Date: Tue, 10 Aug 2010 21:05:48 +0100
Subject: ESI and search engine spiders
Message-ID: <4C61B11C.2090606@gmail.com>

Hi,

On one site we run behind varnish, we've got a "most popular" widget 
displayed on every page (much like http://www.bbc.co.uk/news/).  
However, we have difficulties where this pollutes search engines, as 
searches for a specific popular headline tend not to link directly to 
the article itself, but to one of the index pages with high Google 
pagerank or similar.

What I'd like to know is how other Varnish users might have served 
different ESI content based on whether it's a bot or not.

My initial idea was to set an "X-Not-For-Bots: 1" header on the URL that 
generates the most-popular fragment, then do something like (though 
untested):

sub vcl_deliver {
    if (req.http.header.user-agent ~ "bot" && resp.http.X-Not-For-Bots 
== "1") {
       error 752 "Not for bots";
    } else {
       deliver;
    }
}
...
sub vcl_error {
    if (obj.status==750) {
        set obj.status = 200;
        synthetic {"<!-- not for bots -->"};
        deliver;
    }
}

However, such an approach doesn't work as the req object isn't available 
in vcl_deliver.  We'd prefer to use a header such as X-Not-For-Bots or 
similar, rather than hard-coding a list of ESI fragments to be 
suppressed from bots into Varnish.

Has anyone any good suggestions?



Rob



From checker at d6.com  Tue Aug 10 20:16:04 2010
From: checker at d6.com (Chris Hecker)
Date: Tue, 10 Aug 2010 13:16:04 -0700
Subject: "Total body bytes" not actually bytes delivered?
In-Reply-To: <87y6cfhqp1.fsf@qurzaw.linpro.no>
References: <4C515EF5.70904@d6.com> <87y6cfhqp1.fsf@qurzaw.linpro.no>
Message-ID: <4C61B384.9060203@d6.com>


I don't have time right now to write a test program or 100% verify that 
this was going on, but I'm pretty sure it was from watching varnishstat. 
  Would you rather I file a bug now with just the info I've got, or wait 
until I have time to make it more rigorous?

Thanks,
Chris

On 2010/08/10 00:28, Tollef Fog Heen wrote:
> ]] Chris Hecker
>
> | Is that by design, or should it be the number of bytes successfully
> | delivered, which seems more useful?
>
> It should be the bytes delivered, so please file a ticket in trac.
>



From mloftis at wgops.com  Tue Aug 10 21:12:11 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Tue, 10 Aug 2010 15:12:11 -0600
Subject: ESI and search engine spiders
In-Reply-To: <4C61B11C.2090606@gmail.com>
References: <4C61B11C.2090606@gmail.com>
Message-ID: <56C368A5B9B57484AFBB7DF2@[192.168.1.100]>



--On Tuesday, August 10, 2010 9:05 PM +0100 Rob S <rtshilston at gmail.com> 
wrote:

> Hi,
>
> On one site we run behind varnish, we've got a "most popular" widget
> displayed on every page (much like http://www.bbc.co.uk/news/).  However,
> we have difficulties where this pollutes search engines, as searches for
> a specific popular headline tend not to link directly to the article
> itself, but to one of the index pages with high Google pagerank or
> similar.
>
> What I'd like to know is how other Varnish users might have served
> different ESI content based on whether it's a bot or not.
>
> My initial idea was to set an "X-Not-For-Bots: 1" header on the URL that
> generates the most-popular fragment, then do something like (though
> untested):
>

ESI goes through all the normal steps, so a <esi:include src="/esi/blargh"> 
is fired off starting with vcl_receive looking just exactly like the 
browser had hit the cache with that as the req.url -- the entire req object 
is the same -- i am *not* certain that headers you've added get propogated 
as I've not tested that (and all of my rules are built on the assumption 
that is not the case, just to be sure)

So there's no need to do it in vcl_deliver, in fact, you're far better 
handling it in vcl_recv and/or vcl_hash (actually you really SHOULD handle 
it in vcl_hash and change the hash for these search engine specific objects 
else you'll serve them to regular users)...


for example -- assume vcl_recv sets X-BotDetector in the req header... (not 
tested)::


sub vcl_hash {
  // always take into account the url and host
  set req.hash += req.url;
  if (req.http.host) {
    set req.hash += req.http.host;
  } else {
    set req.hash += server.ip;
  }

  if(req.http.X-BotDetector == "1") {
    set req.hash += "bot detector";
  }
}


You still have to do the detection inside of varnish, I don't see any way 
around that.  The reason is that only varnish knows who it's talking to, 
and varnish needs to decide which object to spit out.  Working properly 
what happens is essentially the webserver sends back a 'template' for the 
page containing the page specific stuff, and pointers to a bunch of ESI 
fragments.  The ESI fragments are also cache objects/requests...So what 
happens is the cache takes this template, fills in ESI fragments (from 
cache if it can, fetching them if it needs to, treating them just as if the 
web browser had run to the ESI url)


This is actually exactly how I handle menu's that change based on a users 
authentication status.  The browser gets a cookie.  The ESI URL is formed 
as either 'authenticated' 'personalized' or 'global' -- authenticated means 
it varies only on the clients login state, personalized takes into account 
the actual session we're working with.  And global means everyone gets the 
same cache regardless (we strip cookies going into these ESI URLs and 
coming from these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch 
code looks for some special headers set that indicate that the recv has 
decided it needs to ditch set-cookies -- this is mostly a safety measure to 
prevent a session sticking to a client it shouldn't due to any bugs in code)

The basic idea is borrowed from 
<http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers> and 
<http://varnish-cache.org/wiki/VCLExampleCacheCookies>

HTH!




From rtshilston at gmail.com  Wed Aug 11 15:20:32 2010
From: rtshilston at gmail.com (Rob S)
Date: Wed, 11 Aug 2010 16:20:32 +0100
Subject: ESI and search engine spiders
In-Reply-To: <56C368A5B9B57484AFBB7DF2@[192.168.1.100]>
References: <4C61B11C.2090606@gmail.com>
	<56C368A5B9B57484AFBB7DF2@[192.168.1.100]>
Message-ID: <4C62BFC0.1020302@gmail.com>


Michael Loftis wrote:
>
>
> --On Tuesday, August 10, 2010 9:05 PM +0100 Rob S 
> <rtshilston at gmail.com> wrote:
>
>> Hi,
>>
>> On one site we run behind varnish, we've got a "most popular" widget
>> displayed on every page (much like http://www.bbc.co.uk/news/).  
>> However,
>> we have difficulties where this pollutes search engines, as searches for
>> a specific popular headline tend not to link directly to the article
>> itself, but to one of the index pages with high Google pagerank or
>> similar.
>>
>> What I'd like to know is how other Varnish users might have served
>> different ESI content based on whether it's a bot or not.
>>
>> My initial idea was to set an "X-Not-For-Bots: 1" header on the URL that
>> generates the most-popular fragment, then do something like (though
>> untested):
>>
>
> ESI goes through all the normal steps, so a <esi:include 
> src="/esi/blargh"> is fired off starting with vcl_receive looking just 
> exactly like the browser had hit the cache with that as the req.url -- 
> the entire req object is the same -- i am *not* certain that headers 
> you've added get propogated as I've not tested that (and all of my 
> rules are built on the assumption that is not the case, just to be sure)
>
> So there's no need to do it in vcl_deliver, in fact, you're far better 
> handling it in vcl_recv and/or vcl_hash (actually you really SHOULD 
> handle it in vcl_hash and change the hash for these search engine 
> specific objects else you'll serve them to regular users)...
>
>
> for example -- assume vcl_recv sets X-BotDetector in the req header... 
> (not tested)::
>
>
> sub vcl_hash {
>  // always take into account the url and host
>  set req.hash += req.url;
>  if (req.http.host) {
>    set req.hash += req.http.host;
>  } else {
>    set req.hash += server.ip;
>  }
>
>  if(req.http.X-BotDetector == "1") {
>    set req.hash += "bot detector";
>  }
> }
>
>
> You still have to do the detection inside of varnish, I don't see any 
> way around that.  The reason is that only varnish knows who it's 
> talking to, and varnish needs to decide which object to spit out.  
> Working properly what happens is essentially the webserver sends back 
> a 'template' for the page containing the page specific stuff, and 
> pointers to a bunch of ESI fragments.  The ESI fragments are also 
> cache objects/requests...So what happens is the cache takes this 
> template, fills in ESI fragments (from cache if it can, fetching them 
> if it needs to, treating them just as if the web browser had run to 
> the ESI url)
>
>
> This is actually exactly how I handle menu's that change based on a 
> users authentication status.  The browser gets a cookie.  The ESI URL 
> is formed as either 'authenticated' 'personalized' or 'global' -- 
> authenticated means it varies only on the clients login state, 
> personalized takes into account the actual session we're working 
> with.  And global means everyone gets the same cache regardless (we 
> strip cookies going into these ESI URLs and coming from these ESI URLs 
> in the vcl_recv/vcl_fetch code, the vcl_fetch code looks for some 
> special headers set that indicate that the recv has decided it needs 
> to ditch set-cookies -- this is mostly a safety measure to prevent a 
> session sticking to a client it shouldn't due to any bugs in code)
>
> The basic idea is borrowed from 
> <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers> and 
> <http://varnish-cache.org/wiki/VCLExampleCacheCookies>
>
> HTH!

Thanks.  We've proved this works with a simple setup:

sub vcl_recv {
        ....
        // Establish if the visitor is a search engine:
        set req.http.X-IsABot = "0";
        if (req.http.user-agent ~ "Yahoo! Slurp") { set 
req.http.X-IsABot = "1"; }
        if (req.http.X-IsABot == "0" && req.http.user-agent ~ 
"Googlebot") { set req.http.X-IsABot = "1"; }
        if (req.http.X-IsABot == "0" && req.http.user-agent ~ "msnbot") 
{ set req.http.X-IsABot = "1"; }
        ....

}
...
sub vcl_hash {
        set req.hash += req.url;
        if (req.http.host) {
                set req.hash += req.http.host;
        } else {
                set req.hash += server.ip;
        }

        if (req.http.X-IsABot == "1") {
                set req.hash += "for-bot";
        } else {
                set req.hash += "for-non-bot";
        }
        hash;
}

The main HTML has a simple ESI, which loads a page fragment whose PHP reads:

if ($_SERVER["HTTP_X_ISABOT"]) {

        echo "<!-- The list of popular posts is not displayed to search 
engines -->";
} else {
       
        // calculate most popular
        echo "The most popular article is XYZ";
}



Thanks again.



From stewsnooze at gmail.com  Wed Aug 11 15:25:52 2010
From: stewsnooze at gmail.com (Stewart Robinson)
Date: Wed, 11 Aug 2010 16:25:52 +0100
Subject: ESI and search engine spiders
In-Reply-To: <4C62BFC0.1020302@gmail.com>
References: <4C61B11C.2090606@gmail.com>
	<56C368A5B9B57484AFBB7DF2@192.168.1.100>
	<4C62BFC0.1020302@gmail.com>
Message-ID: <AANLkTikg4kS8548RBUXjYxfJTZ_oOa8uOLXRNNxO3Jft@mail.gmail.com>

Hi,

Whilst this looks excellent and I may use it to serve different
content to other types of users I think you should read, if you
haven't already, this URL which discourages this sort of behaviour.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66355

Great VCL though!
Stew


On 11 August 2010 16:20, Rob S <rtshilston at gmail.com> wrote:
>
> Michael Loftis wrote:
>>
>>
>> --On Tuesday, August 10, 2010 9:05 PM +0100 Rob S <rtshilston at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On one site we run behind varnish, we've got a "most popular" widget
>>> displayed on every page (much like http://www.bbc.co.uk/news/). ?However,
>>> we have difficulties where this pollutes search engines, as searches for
>>> a specific popular headline tend not to link directly to the article
>>> itself, but to one of the index pages with high Google pagerank or
>>> similar.
>>>
>>> What I'd like to know is how other Varnish users might have served
>>> different ESI content based on whether it's a bot or not.
>>>
>>> My initial idea was to set an "X-Not-For-Bots: 1" header on the URL that
>>> generates the most-popular fragment, then do something like (though
>>> untested):
>>>
>>
>> ESI goes through all the normal steps, so a <esi:include
>> src="/esi/blargh"> is fired off starting with vcl_receive looking just
>> exactly like the browser had hit the cache with that as the req.url -- the
>> entire req object is the same -- i am *not* certain that headers you've
>> added get propogated as I've not tested that (and all of my rules are built
>> on the assumption that is not the case, just to be sure)
>>
>> So there's no need to do it in vcl_deliver, in fact, you're far better
>> handling it in vcl_recv and/or vcl_hash (actually you really SHOULD handle
>> it in vcl_hash and change the hash for these search engine specific objects
>> else you'll serve them to regular users)...
>>
>>
>> for example -- assume vcl_recv sets X-BotDetector in the req header...
>> (not tested)::
>>
>>
>> sub vcl_hash {
>> ?// always take into account the url and host
>> ?set req.hash += req.url;
>> ?if (req.http.host) {
>> ? set req.hash += req.http.host;
>> ?} else {
>> ? set req.hash += server.ip;
>> ?}
>>
>> ?if(req.http.X-BotDetector == "1") {
>> ? set req.hash += "bot detector";
>> ?}
>> }
>>
>>
>> You still have to do the detection inside of varnish, I don't see any way
>> around that. ?The reason is that only varnish knows who it's talking to, and
>> varnish needs to decide which object to spit out. ?Working properly what
>> happens is essentially the webserver sends back a 'template' for the page
>> containing the page specific stuff, and pointers to a bunch of ESI
>> fragments. ?The ESI fragments are also cache objects/requests...So what
>> happens is the cache takes this template, fills in ESI fragments (from cache
>> if it can, fetching them if it needs to, treating them just as if the web
>> browser had run to the ESI url)
>>
>>
>> This is actually exactly how I handle menu's that change based on a users
>> authentication status. ?The browser gets a cookie. ?The ESI URL is formed as
>> either 'authenticated' 'personalized' or 'global' -- authenticated means it
>> varies only on the clients login state, personalized takes into account the
>> actual session we're working with. ?And global means everyone gets the same
>> cache regardless (we strip cookies going into these ESI URLs and coming from
>> these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch code looks for
>> some special headers set that indicate that the recv has decided it needs to
>> ditch set-cookies -- this is mostly a safety measure to prevent a session
>> sticking to a client it shouldn't due to any bugs in code)
>>
>> The basic idea is borrowed from
>> <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers> and
>> <http://varnish-cache.org/wiki/VCLExampleCacheCookies>
>>
>> HTH!
>
> Thanks. ?We've proved this works with a simple setup:
>
> sub vcl_recv {
> ? ? ? ....
> ? ? ? // Establish if the visitor is a search engine:
> ? ? ? set req.http.X-IsABot = "0";
> ? ? ? if (req.http.user-agent ~ "Yahoo! Slurp") { set req.http.X-IsABot =
> "1"; }
> ? ? ? if (req.http.X-IsABot == "0" && req.http.user-agent ~ "Googlebot") {
> set req.http.X-IsABot = "1"; }
> ? ? ? if (req.http.X-IsABot == "0" && req.http.user-agent ~ "msnbot") { set
> req.http.X-IsABot = "1"; }
> ? ? ? ....
>
> }
> ...
> sub vcl_hash {
> ? ? ? set req.hash += req.url;
> ? ? ? if (req.http.host) {
> ? ? ? ? ? ? ? set req.hash += req.http.host;
> ? ? ? } else {
> ? ? ? ? ? ? ? set req.hash += server.ip;
> ? ? ? }
>
> ? ? ? if (req.http.X-IsABot == "1") {
> ? ? ? ? ? ? ? set req.hash += "for-bot";
> ? ? ? } else {
> ? ? ? ? ? ? ? set req.hash += "for-non-bot";
> ? ? ? }
> ? ? ? hash;
> }
>
> The main HTML has a simple ESI, which loads a page fragment whose PHP reads:
>
> if ($_SERVER["HTTP_X_ISABOT"]) {
>
> ? ? ? echo "<!-- The list of popular posts is not displayed to search
> engines -->";
> } else {
> ? ? ? ? ? ? // calculate most popular
> ? ? ? echo "The most popular article is XYZ";
> }
>
>
>
> Thanks again.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



From checker at d6.com  Wed Aug 11 15:32:58 2010
From: checker at d6.com (Chris Hecker)
Date: Wed, 11 Aug 2010 08:32:58 -0700
Subject: ESI and search engine spiders
In-Reply-To: <AANLkTikg4kS8548RBUXjYxfJTZ_oOa8uOLXRNNxO3Jft@mail.gmail.com>
References: <4C61B11C.2090606@gmail.com>	<56C368A5B9B57484AFBB7DF2@192.168.1.100>	<4C62BFC0.1020302@gmail.com>
	<AANLkTikg4kS8548RBUXjYxfJTZ_oOa8uOLXRNNxO3Jft@mail.gmail.com>
Message-ID: <4C62C2AA.3000906@d6.com>


On that note, why not use robots.txt and a clear path name to turn off 
bots for the lists?

Chris

On 2010/08/11 08:25, Stewart Robinson wrote:
> Hi,
>
> Whilst this looks excellent and I may use it to serve different
> content to other types of users I think you should read, if you
> haven't already, this URL which discourages this sort of behaviour.
>
> http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66355
>
> Great VCL though!
> Stew
>
>
> On 11 August 2010 16:20, Rob S<rtshilston at gmail.com>  wrote:
>>
>> Michael Loftis wrote:
>>>
>>>
>>> --On Tuesday, August 10, 2010 9:05 PM +0100 Rob S<rtshilston at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On one site we run behind varnish, we've got a "most popular" widget
>>>> displayed on every page (much like http://www.bbc.co.uk/news/).  However,
>>>> we have difficulties where this pollutes search engines, as searches for
>>>> a specific popular headline tend not to link directly to the article
>>>> itself, but to one of the index pages with high Google pagerank or
>>>> similar.
>>>>
>>>> What I'd like to know is how other Varnish users might have served
>>>> different ESI content based on whether it's a bot or not.
>>>>
>>>> My initial idea was to set an "X-Not-For-Bots: 1" header on the URL that
>>>> generates the most-popular fragment, then do something like (though
>>>> untested):
>>>>
>>>
>>> ESI goes through all the normal steps, so a<esi:include
>>> src="/esi/blargh">  is fired off starting with vcl_receive looking just
>>> exactly like the browser had hit the cache with that as the req.url -- the
>>> entire req object is the same -- i am *not* certain that headers you've
>>> added get propogated as I've not tested that (and all of my rules are built
>>> on the assumption that is not the case, just to be sure)
>>>
>>> So there's no need to do it in vcl_deliver, in fact, you're far better
>>> handling it in vcl_recv and/or vcl_hash (actually you really SHOULD handle
>>> it in vcl_hash and change the hash for these search engine specific objects
>>> else you'll serve them to regular users)...
>>>
>>>
>>> for example -- assume vcl_recv sets X-BotDetector in the req header...
>>> (not tested)::
>>>
>>>
>>> sub vcl_hash {
>>>   // always take into account the url and host
>>>   set req.hash += req.url;
>>>   if (req.http.host) {
>>>    set req.hash += req.http.host;
>>>   } else {
>>>    set req.hash += server.ip;
>>>   }
>>>
>>>   if(req.http.X-BotDetector == "1") {
>>>    set req.hash += "bot detector";
>>>   }
>>> }
>>>
>>>
>>> You still have to do the detection inside of varnish, I don't see any way
>>> around that.  The reason is that only varnish knows who it's talking to, and
>>> varnish needs to decide which object to spit out.  Working properly what
>>> happens is essentially the webserver sends back a 'template' for the page
>>> containing the page specific stuff, and pointers to a bunch of ESI
>>> fragments.  The ESI fragments are also cache objects/requests...So what
>>> happens is the cache takes this template, fills in ESI fragments (from cache
>>> if it can, fetching them if it needs to, treating them just as if the web
>>> browser had run to the ESI url)
>>>
>>>
>>> This is actually exactly how I handle menu's that change based on a users
>>> authentication status.  The browser gets a cookie.  The ESI URL is formed as
>>> either 'authenticated' 'personalized' or 'global' -- authenticated means it
>>> varies only on the clients login state, personalized takes into account the
>>> actual session we're working with.  And global means everyone gets the same
>>> cache regardless (we strip cookies going into these ESI URLs and coming from
>>> these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch code looks for
>>> some special headers set that indicate that the recv has decided it needs to
>>> ditch set-cookies -- this is mostly a safety measure to prevent a session
>>> sticking to a client it shouldn't due to any bugs in code)
>>>
>>> The basic idea is borrowed from
>>> <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers>  and
>>> <http://varnish-cache.org/wiki/VCLExampleCacheCookies>
>>>
>>> HTH!
>>
>> Thanks.  We've proved this works with a simple setup:
>>
>> sub vcl_recv {
>>        ....
>>        // Establish if the visitor is a search engine:
>>        set req.http.X-IsABot = "0";
>>        if (req.http.user-agent ~ "Yahoo! Slurp") { set req.http.X-IsABot =
>> "1"; }
>>        if (req.http.X-IsABot == "0"&&  req.http.user-agent ~ "Googlebot") {
>> set req.http.X-IsABot = "1"; }
>>        if (req.http.X-IsABot == "0"&&  req.http.user-agent ~ "msnbot") { set
>> req.http.X-IsABot = "1"; }
>>        ....
>>
>> }
>> ...
>> sub vcl_hash {
>>        set req.hash += req.url;
>>        if (req.http.host) {
>>                set req.hash += req.http.host;
>>        } else {
>>                set req.hash += server.ip;
>>        }
>>
>>        if (req.http.X-IsABot == "1") {
>>                set req.hash += "for-bot";
>>        } else {
>>                set req.hash += "for-non-bot";
>>        }
>>        hash;
>> }
>>
>> The main HTML has a simple ESI, which loads a page fragment whose PHP reads:
>>
>> if ($_SERVER["HTTP_X_ISABOT"]) {
>>
>>        echo "<!-- The list of popular posts is not displayed to search
>> engines -->";
>> } else {
>>              // calculate most popular
>>        echo "The most popular article is XYZ";
>> }
>>
>>
>>
>> Thanks again.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



From rtshilston at gmail.com  Wed Aug 11 15:48:41 2010
From: rtshilston at gmail.com (Rob S)
Date: Wed, 11 Aug 2010 16:48:41 +0100
Subject: ESI and search engine spiders
In-Reply-To: <4C62C2AA.3000906@d6.com>
References: <4C61B11C.2090606@gmail.com>	<56C368A5B9B57484AFBB7DF2@192.168.1.100>	<4C62BFC0.1020302@gmail.com>	<AANLkTikg4kS8548RBUXjYxfJTZ_oOa8uOLXRNNxO3Jft@mail.gmail.com>
	<4C62C2AA.3000906@d6.com>
Message-ID: <4C62C659.9030505@gmail.com>

Chris, Stew,

This was always going to be controversial.  It's a very tough balancing 
act.  In our view, we're not serving additional content to seed the 
search engines, and so it's reasonable.  We are removing content that 
users find useful, but which might make it harder for the search engine 
to make a good judgement about the site overall.  Yahoo explains why 
this is desirable:
> Webpages often include headers, footers, navigational sections, 
> repeated boilerplate text, copyright notices, ad sections, or dynamic 
> content that is useful to users, but not to search engines. Webmasters 
> can apply the "robots-nocontent" attribute to indicate to search 
> engines any content that is extraneous to the main unique content of 
> the page.
A few blogs have picked up on examples of prominent sites implementing 
cloaking to different extents, such as 
http://www.seroundtable.com/archives/021504.html and 
http://www.seoegghead.com/blog/seo/the-google-cloaking-hypocrisy-p32.html.  
Then there's also the First Click Free approach 
(http://www.google.com/support/webmasters/bin/answer.py?answer=74536), 
which many people might feel is a bit borderline.

I agree many people could use the code below to try to boost their 
search engine results by including lots of keywords or links, but I'm 
confident that there are many legitimate reasons to do this.  I'd love 
not to do this in Varnish / HTTP, but there don't appear to be other 
widely supported solutions.  In this case (and addressing Chris' point) 
it's not possible to use robots.txt as we're not trying to block the 
entire page, just a subset of it.   There are ways of hinting to a 
Google Search Appliance to turn off indexing of a portion of the page 
(googleon/off tags, see 
http://perishablepress.com/press/2009/08/23/tell-google-to-not-index-certain-parts-of-your-page/), 
but these aren't supported by the normal googlebot, nor other search 
engines.  Yahoo has a robots-nocontent class that can be added to HTML 
elements, but again, it's a single solution for just one search engine 
(http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-14.html).  
I've heard of (but can't find a link) a discussion to add a specific 
attribute to any HTML element to the new HTML5 standard, but that this 
wasn't adopted.

Someone reading this might know a magic answer, but in the meantime, 
we'll be making minor page alterations to help ensure users find 
relevant results when searching Google, even if that involves a little 
cloaking to suppress a small portion of the page.


Rob


Chris Hecker wrote:
>
> On that note, why not use robots.txt and a clear path name to turn off 
> bots for the lists?
>
> Chris
>
> On 2010/08/11 08:25, Stewart Robinson wrote:
>> Hi,
>>
>> Whilst this looks excellent and I may use it to serve different
>> content to other types of users I think you should read, if you
>> haven't already, this URL which discourages this sort of behaviour.
>>
>> http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66355 
>>
>>
>> Great VCL though!
>> Stew
>>
>>
>> On 11 August 2010 16:20, Rob S<rtshilston at gmail.com>  wrote:
>>>
>>> Michael Loftis wrote:
>>>>
>>>>
>>>> --On Tuesday, August 10, 2010 9:05 PM +0100 Rob 
>>>> S<rtshilston at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On one site we run behind varnish, we've got a "most popular" widget
>>>>> displayed on every page (much like http://www.bbc.co.uk/news/).  
>>>>> However,
>>>>> we have difficulties where this pollutes search engines, as 
>>>>> searches for
>>>>> a specific popular headline tend not to link directly to the article
>>>>> itself, but to one of the index pages with high Google pagerank or
>>>>> similar.
>>>>>
>>>>> What I'd like to know is how other Varnish users might have served
>>>>> different ESI content based on whether it's a bot or not.
>>>>>
>>>>> My initial idea was to set an "X-Not-For-Bots: 1" header on the 
>>>>> URL that
>>>>> generates the most-popular fragment, then do something like (though
>>>>> untested):
>>>>>
>>>>
>>>> ESI goes through all the normal steps, so a<esi:include
>>>> src="/esi/blargh">  is fired off starting with vcl_receive looking 
>>>> just
>>>> exactly like the browser had hit the cache with that as the req.url 
>>>> -- the
>>>> entire req object is the same -- i am *not* certain that headers 
>>>> you've
>>>> added get propogated as I've not tested that (and all of my rules 
>>>> are built
>>>> on the assumption that is not the case, just to be sure)
>>>>
>>>> So there's no need to do it in vcl_deliver, in fact, you're far better
>>>> handling it in vcl_recv and/or vcl_hash (actually you really SHOULD 
>>>> handle
>>>> it in vcl_hash and change the hash for these search engine specific 
>>>> objects
>>>> else you'll serve them to regular users)...
>>>>
>>>>
>>>> for example -- assume vcl_recv sets X-BotDetector in the req header...
>>>> (not tested)::
>>>>
>>>>
>>>> sub vcl_hash {
>>>>   // always take into account the url and host
>>>>   set req.hash += req.url;
>>>>   if (req.http.host) {
>>>>    set req.hash += req.http.host;
>>>>   } else {
>>>>    set req.hash += server.ip;
>>>>   }
>>>>
>>>>   if(req.http.X-BotDetector == "1") {
>>>>    set req.hash += "bot detector";
>>>>   }
>>>> }
>>>>
>>>>
>>>> You still have to do the detection inside of varnish, I don't see 
>>>> any way
>>>> around that.  The reason is that only varnish knows who it's 
>>>> talking to, and
>>>> varnish needs to decide which object to spit out.  Working properly 
>>>> what
>>>> happens is essentially the webserver sends back a 'template' for 
>>>> the page
>>>> containing the page specific stuff, and pointers to a bunch of ESI
>>>> fragments.  The ESI fragments are also cache objects/requests...So 
>>>> what
>>>> happens is the cache takes this template, fills in ESI fragments 
>>>> (from cache
>>>> if it can, fetching them if it needs to, treating them just as if 
>>>> the web
>>>> browser had run to the ESI url)
>>>>
>>>>
>>>> This is actually exactly how I handle menu's that change based on a 
>>>> users
>>>> authentication status.  The browser gets a cookie.  The ESI URL is 
>>>> formed as
>>>> either 'authenticated' 'personalized' or 'global' -- authenticated 
>>>> means it
>>>> varies only on the clients login state, personalized takes into 
>>>> account the
>>>> actual session we're working with.  And global means everyone gets 
>>>> the same
>>>> cache regardless (we strip cookies going into these ESI URLs and 
>>>> coming from
>>>> these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch code 
>>>> looks for
>>>> some special headers set that indicate that the recv has decided it 
>>>> needs to
>>>> ditch set-cookies -- this is mostly a safety measure to prevent a 
>>>> session
>>>> sticking to a client it shouldn't due to any bugs in code)
>>>>
>>>> The basic idea is borrowed from
>>>> <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers>  and
>>>> <http://varnish-cache.org/wiki/VCLExampleCacheCookies>
>>>>
>>>> HTH!
>>>
>>> Thanks.  We've proved this works with a simple setup:
>>>
>>> sub vcl_recv {
>>>        ....
>>>        // Establish if the visitor is a search engine:
>>>        set req.http.X-IsABot = "0";
>>>        if (req.http.user-agent ~ "Yahoo! Slurp") { set 
>>> req.http.X-IsABot =
>>> "1"; }
>>>        if (req.http.X-IsABot == "0"&&  req.http.user-agent ~ 
>>> "Googlebot") {
>>> set req.http.X-IsABot = "1"; }
>>>        if (req.http.X-IsABot == "0"&&  req.http.user-agent ~ 
>>> "msnbot") { set
>>> req.http.X-IsABot = "1"; }
>>>        ....
>>>
>>> }
>>> ...
>>> sub vcl_hash {
>>>        set req.hash += req.url;
>>>        if (req.http.host) {
>>>                set req.hash += req.http.host;
>>>        } else {
>>>                set req.hash += server.ip;
>>>        }
>>>
>>>        if (req.http.X-IsABot == "1") {
>>>                set req.hash += "for-bot";
>>>        } else {
>>>                set req.hash += "for-non-bot";
>>>        }
>>>        hash;
>>> }
>>>
>>> The main HTML has a simple ESI, which loads a page fragment whose 
>>> PHP reads:
>>>
>>> if ($_SERVER["HTTP_X_ISABOT"]) {
>>>
>>>        echo "<!-- The list of popular posts is not displayed to search
>>> engines -->";
>>> } else {
>>>              // calculate most popular
>>>        echo "The most popular article is XYZ";
>>> }
>>>
>>>
>>>
>>> Thanks again.
>>>




From checker at d6.com  Wed Aug 11 16:25:05 2010
From: checker at d6.com (Chris Hecker)
Date: Wed, 11 Aug 2010 09:25:05 -0700
Subject: ESI and search engine spiders
In-Reply-To: <4C62C659.9030505@gmail.com>
References: <4C61B11C.2090606@gmail.com>	<56C368A5B9B57484AFBB7DF2@192.168.1.100>	<4C62BFC0.1020302@gmail.com>	<AANLkTikg4kS8548RBUXjYxfJTZ_oOa8uOLXRNNxO3Jft@mail.gmail.com>
	<4C62C2AA.3000906@d6.com> <4C62C659.9030505@gmail.com>
Message-ID: <4C62CEE1.3010307@d6.com>


Ah, right, I've never used ESI, so I forgot it's already glommed 
together when the client gets it.

Chris


On 2010/08/11 08:48, Rob S wrote:
> Chris, Stew,
>
> This was always going to be controversial. It's a very tough balancing
> act. In our view, we're not serving additional content to seed the
> search engines, and so it's reasonable. We are removing content that
> users find useful, but which might make it harder for the search engine
> to make a good judgement about the site overall. Yahoo explains why this
> is desirable:
>> Webpages often include headers, footers, navigational sections,
>> repeated boilerplate text, copyright notices, ad sections, or dynamic
>> content that is useful to users, but not to search engines. Webmasters
>> can apply the "robots-nocontent" attribute to indicate to search
>> engines any content that is extraneous to the main unique content of
>> the page.
> A few blogs have picked up on examples of prominent sites implementing
> cloaking to different extents, such as
> http://www.seroundtable.com/archives/021504.html and
> http://www.seoegghead.com/blog/seo/the-google-cloaking-hypocrisy-p32.html.
> Then there's also the First Click Free approach
> (http://www.google.com/support/webmasters/bin/answer.py?answer=74536),
> which many people might feel is a bit borderline.
>
> I agree many people could use the code below to try to boost their
> search engine results by including lots of keywords or links, but I'm
> confident that there are many legitimate reasons to do this. I'd love
> not to do this in Varnish / HTTP, but there don't appear to be other
> widely supported solutions. In this case (and addressing Chris' point)
> it's not possible to use robots.txt as we're not trying to block the
> entire page, just a subset of it. There are ways of hinting to a Google
> Search Appliance to turn off indexing of a portion of the page
> (googleon/off tags, see
> http://perishablepress.com/press/2009/08/23/tell-google-to-not-index-certain-parts-of-your-page/),
> but these aren't supported by the normal googlebot, nor other search
> engines. Yahoo has a robots-nocontent class that can be added to HTML
> elements, but again, it's a single solution for just one search engine
> (http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-14.html). I've
> heard of (but can't find a link) a discussion to add a specific
> attribute to any HTML element to the new HTML5 standard, but that this
> wasn't adopted.
>
> Someone reading this might know a magic answer, but in the meantime,
> we'll be making minor page alterations to help ensure users find
> relevant results when searching Google, even if that involves a little
> cloaking to suppress a small portion of the page.
>
>
> Rob
>
>
> Chris Hecker wrote:
>>
>> On that note, why not use robots.txt and a clear path name to turn off
>> bots for the lists?
>>
>> Chris
>>
>> On 2010/08/11 08:25, Stewart Robinson wrote:
>>> Hi,
>>>
>>> Whilst this looks excellent and I may use it to serve different
>>> content to other types of users I think you should read, if you
>>> haven't already, this URL which discourages this sort of behaviour.
>>>
>>> http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66355
>>>
>>>
>>> Great VCL though!
>>> Stew
>>>
>>>
>>> On 11 August 2010 16:20, Rob S<rtshilston at gmail.com> wrote:
>>>>
>>>> Michael Loftis wrote:
>>>>>
>>>>>
>>>>> --On Tuesday, August 10, 2010 9:05 PM +0100 Rob
>>>>> S<rtshilston at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On one site we run behind varnish, we've got a "most popular" widget
>>>>>> displayed on every page (much like http://www.bbc.co.uk/news/).
>>>>>> However,
>>>>>> we have difficulties where this pollutes search engines, as
>>>>>> searches for
>>>>>> a specific popular headline tend not to link directly to the article
>>>>>> itself, but to one of the index pages with high Google pagerank or
>>>>>> similar.
>>>>>>
>>>>>> What I'd like to know is how other Varnish users might have served
>>>>>> different ESI content based on whether it's a bot or not.
>>>>>>
>>>>>> My initial idea was to set an "X-Not-For-Bots: 1" header on the
>>>>>> URL that
>>>>>> generates the most-popular fragment, then do something like (though
>>>>>> untested):
>>>>>>
>>>>>
>>>>> ESI goes through all the normal steps, so a<esi:include
>>>>> src="/esi/blargh"> is fired off starting with vcl_receive looking just
>>>>> exactly like the browser had hit the cache with that as the req.url
>>>>> -- the
>>>>> entire req object is the same -- i am *not* certain that headers
>>>>> you've
>>>>> added get propogated as I've not tested that (and all of my rules
>>>>> are built
>>>>> on the assumption that is not the case, just to be sure)
>>>>>
>>>>> So there's no need to do it in vcl_deliver, in fact, you're far better
>>>>> handling it in vcl_recv and/or vcl_hash (actually you really SHOULD
>>>>> handle
>>>>> it in vcl_hash and change the hash for these search engine specific
>>>>> objects
>>>>> else you'll serve them to regular users)...
>>>>>
>>>>>
>>>>> for example -- assume vcl_recv sets X-BotDetector in the req header...
>>>>> (not tested)::
>>>>>
>>>>>
>>>>> sub vcl_hash {
>>>>> // always take into account the url and host
>>>>> set req.hash += req.url;
>>>>> if (req.http.host) {
>>>>> set req.hash += req.http.host;
>>>>> } else {
>>>>> set req.hash += server.ip;
>>>>> }
>>>>>
>>>>> if(req.http.X-BotDetector == "1") {
>>>>> set req.hash += "bot detector";
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> You still have to do the detection inside of varnish, I don't see
>>>>> any way
>>>>> around that. The reason is that only varnish knows who it's talking
>>>>> to, and
>>>>> varnish needs to decide which object to spit out. Working properly
>>>>> what
>>>>> happens is essentially the webserver sends back a 'template' for
>>>>> the page
>>>>> containing the page specific stuff, and pointers to a bunch of ESI
>>>>> fragments. The ESI fragments are also cache objects/requests...So what
>>>>> happens is the cache takes this template, fills in ESI fragments
>>>>> (from cache
>>>>> if it can, fetching them if it needs to, treating them just as if
>>>>> the web
>>>>> browser had run to the ESI url)
>>>>>
>>>>>
>>>>> This is actually exactly how I handle menu's that change based on a
>>>>> users
>>>>> authentication status. The browser gets a cookie. The ESI URL is
>>>>> formed as
>>>>> either 'authenticated' 'personalized' or 'global' -- authenticated
>>>>> means it
>>>>> varies only on the clients login state, personalized takes into
>>>>> account the
>>>>> actual session we're working with. And global means everyone gets
>>>>> the same
>>>>> cache regardless (we strip cookies going into these ESI URLs and
>>>>> coming from
>>>>> these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch code
>>>>> looks for
>>>>> some special headers set that indicate that the recv has decided it
>>>>> needs to
>>>>> ditch set-cookies -- this is mostly a safety measure to prevent a
>>>>> session
>>>>> sticking to a client it shouldn't due to any bugs in code)
>>>>>
>>>>> The basic idea is borrowed from
>>>>> <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers> and
>>>>> <http://varnish-cache.org/wiki/VCLExampleCacheCookies>
>>>>>
>>>>> HTH!
>>>>
>>>> Thanks. We've proved this works with a simple setup:
>>>>
>>>> sub vcl_recv {
>>>> ....
>>>> // Establish if the visitor is a search engine:
>>>> set req.http.X-IsABot = "0";
>>>> if (req.http.user-agent ~ "Yahoo! Slurp") { set req.http.X-IsABot =
>>>> "1"; }
>>>> if (req.http.X-IsABot == "0"&& req.http.user-agent ~ "Googlebot") {
>>>> set req.http.X-IsABot = "1"; }
>>>> if (req.http.X-IsABot == "0"&& req.http.user-agent ~ "msnbot") { set
>>>> req.http.X-IsABot = "1"; }
>>>> ....
>>>>
>>>> }
>>>> ...
>>>> sub vcl_hash {
>>>> set req.hash += req.url;
>>>> if (req.http.host) {
>>>> set req.hash += req.http.host;
>>>> } else {
>>>> set req.hash += server.ip;
>>>> }
>>>>
>>>> if (req.http.X-IsABot == "1") {
>>>> set req.hash += "for-bot";
>>>> } else {
>>>> set req.hash += "for-non-bot";
>>>> }
>>>> hash;
>>>> }
>>>>
>>>> The main HTML has a simple ESI, which loads a page fragment whose
>>>> PHP reads:
>>>>
>>>> if ($_SERVER["HTTP_X_ISABOT"]) {
>>>>
>>>> echo "<!-- The list of popular posts is not displayed to search
>>>> engines -->";
>>>> } else {
>>>> // calculate most popular
>>>> echo "The most popular article is XYZ";
>>>> }
>>>>
>>>>
>>>>
>>>> Thanks again.
>>>>
>
>



From pubcrawler.com at gmail.com  Wed Aug 11 22:59:11 2010
From: pubcrawler.com at gmail.com (pub crawler)
Date: Wed, 11 Aug 2010 18:59:11 -0400
Subject: Varnish analysis of content - do not cache conditionally based on 
	content received
Message-ID: <AANLkTinWjftHNUQ3K2ad6CztQuaJ7OKkBHgLSO6=c4td@mail.gmail.com>

We have a situation where Varnish is caching things we don't want cached.

For instance,  sometimes we may throw an error and that ends up in the
page and stuck in Varnish.  We have common language / code we can say
will always be in such page.

Another example is we sometimes ban malicious folks as-needed on our
app servers.   When they get another page it simply says BANNED! #IP#.

Both conditions are able to be detected in theory if we can have
Varnish look at the pass-through page data it gets.

Anyone have any ideas of how best to accomplish this non-cache of
these conditions within Varnish?

Thanks!



From ask at develooper.com  Wed Aug 11 23:06:06 2010
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Wed, 11 Aug 2010 16:06:06 -0700
Subject: Varnish analysis of content - do not cache conditionally based on
	content received
In-Reply-To: <AANLkTinWjftHNUQ3K2ad6CztQuaJ7OKkBHgLSO6=c4td@mail.gmail.com>
References: <AANLkTinWjftHNUQ3K2ad6CztQuaJ7OKkBHgLSO6=c4td@mail.gmail.com>
Message-ID: <E43A55E2-0357-4AED-A3C0-8014981FCB20@develooper.com>


On Aug 11, 2010, at 15:59, pub crawler wrote:

> Anyone have any ideas of how best to accomplish this non-cache of
> these conditions within Varnish?

Make the app server set appropriate cache headers (or some extra header that you check for in the VCL).


 - ask


From ahooper at bmjgroup.com  Thu Aug 12 08:49:01 2010
From: ahooper at bmjgroup.com (Alex Hooper)
Date: Thu, 12 Aug 2010 09:49:01 +0100
Subject: Backend connections not attempted -- definition
Message-ID: <4C63B57D.5070405@bmjgroup.com>

Hi,

Would someone mind explaining what "Backend connections not attempted" means?
(Or point me at a good explanation; I don't seem to be able to find one).

Many thanks,

Alex.
-- 
Alex Hooper
Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR
Tel: +44 (0) 20 7383 6049
http://group.bmj.com/

_______________________________________________________________________
The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com.  This email and any attachments are confidential.  If you have received this email in error, please delete it and kindly notify us.  If the email contains personal views then the BMJ Group accepts no responsibility for these statements.  The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses.  Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content.  BMJ Publishing Group Limited trading as BMJ Group.  A private limited company, registered in England and Wales under registration number 03102371.  Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK.
_______________________________________________________________________



From phk at phk.freebsd.dk  Thu Aug 12 09:05:25 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 12 Aug 2010 09:05:25 +0000
Subject: Backend connections not attempted -- definition 
In-Reply-To: Your message of "Thu, 12 Aug 2010 09:49:01 +0100."
	<4C63B57D.5070405@bmjgroup.com> 
Message-ID: <24746.1281603925@critter.freebsd.dk>

In message <4C63B57D.5070405 at bmjgroup.com>, Alex Hooper writes:

>Would someone mind explaining what "Backend connections not attempted" means?
>(Or point me at a good explanation; I don't seem to be able to find one).

It means that the backend was unhealthy and we did not even try to
contact it.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From harald at webmeisterei.com  Mon Aug 16 09:04:32 2010
From: harald at webmeisterei.com (Harald Friessnegger)
Date: Mon, 16 Aug 2010 11:04:32 +0200
Subject: munin plugin varnish_ - howto add support for multiple instances
Message-ID: <i4auv2$1mf$1@dough.gmane.org>

hi all

i was asking to support multiple instances in a previous post 
(http://www.gossamer-threads.com/lists/varnish/misc/14871) earlier this 
year.

in the meantime i was pointed to a similar request that solved the backward 
compatiblity problem:
http://lists.varnish-cache.org/pipermail/varnish-dev/2009-
December/002347.html

i added the patch to the current revision of the varnish_  munin plugin
http://varnish-cache.org/browser/trunk/varnish-tools/munin/varnish_?rev=4439

since i felt using the varnish instance name in the graph title is not the 
safest bet (since it might be an absolute path too) if modified the original 
suggestion a little.

see http://dev.plone.org/collective/changeset/123489 for the changelog


now you can safely install sylinks in the plugins directory and configure 
multiple instances

ln -s varnish_ /etc/munin/plugins/varnish_proj1__expunge
ln -s varnish_ /etc/munin/plugins/varnish_proj2__expunge

and add a configuration for both of them

[varnish_proj1_*]
env.varnishstat varnishstat
env.name instance1
env.graphname Project One

[varnish_proj2_*]
env.varnishstat varnishstat
env.name instance2
env.graphname Project Two



maybe you wanna add this to http://varnish-cache.org/browser/trunk/varnish-
tools/munin/varnish_?rev=4439


kind regards, 
  harald

-- 
Webmeisterei GmbH - B?ro f?r Netzfragen
Tel: +43 5572 908877,  Fax: +43 5572 908877-66
Steinebach 18, A-6850 Dornbirn
http://www.webmeisterei.com




From kristian at varnish-software.com  Mon Aug 16 12:40:12 2010
From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=)
Date: Mon, 16 Aug 2010 14:40:12 +0200
Subject: munin plugin varnish_ - howto add support for multiple instances
In-Reply-To: <i4auv2$1mf$1@dough.gmane.org>
References: <i4auv2$1mf$1@dough.gmane.org>
Message-ID: <AANLkTin+Nix4x81iP5344xsw9xzELTOU38yuOWH48Qdr@mail.gmail.com>

On Mon, Aug 16, 2010 at 11:04 AM, Harald Friessnegger
<harald at webmeisterei.com> wrote:
> hi all
(excsessive snipping)

Just want to ACK this mail. I'll review the details when I'm back in
Oslo next week, then answer fully.

- Kristian



From ingvar at redpill-linpro.com  Tue Aug 17 12:39:17 2010
From: ingvar at redpill-linpro.com (Ingvar Hagelund)
Date: Tue, 17 Aug 2010 14:39:17 +0200
Subject: RPMS of varnish-2.1.3 available
Message-ID: <4C6A82F5.80900@redpill-linpro.com>

  I built RPMS of varnish-2.1.3 for RHEL4 and 5 and compatible 
derivates. Source and binary packages for x86_64 are available at 
http://sourceforge.net/projects/varnish/files/

If anybody needs packages for other arches or RHEL versions, have a look at
http://users.linpro.no/ingvar/varnish/ , or contact me.

These packages have been tested in production, and should prove quite 
stable, but feedback is as usual always very welcome.

Ingvar




From david.birdsong at gmail.com  Tue Aug 17 20:13:35 2010
From: david.birdsong at gmail.com (David Birdsong)
Date: Tue, 17 Aug 2010 13:13:35 -0700
Subject: varnish fetching, stroring and serving cached 404 from backend server
Message-ID: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>

I've been struggling to understand why and how varnish is deciding to
cache a 404 response from a backend.  It's very tough to reproduce.
I've thrown millions of test URLs at varnish where my testing
application will respond back randomly with a 404 and I can't trigger
it.  Only user traffic is able to trigger this weird behavior.

Anybody have any experience with this?

Here's a varnishlog of a 404.
http://pastebin.com/386VU4rC

I'm working to capture the varnishlog of the resource actually getting
fetched and stored.



From mloftis at wgops.com  Wed Aug 18 02:59:45 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Tue, 17 Aug 2010 20:59:45 -0600
Subject: varnish fetching, stroring and serving cached 404 from backend
	server
In-Reply-To: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>
References: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>
Message-ID: <5AAF0084CB9EC7621B7AB3E0@[192.168.1.100]>

Varnish caching, or not caching, is completely controlled by the VCL.  Not 
having yours noone can help you.  However, the *default* VCL makes anything 
with a Cookie or HTTP Authentication header from the client uncacheable.

--On Tuesday, August 17, 2010 1:13 PM -0700 David Birdsong 
<david.birdsong at gmail.com> wrote:

> I've been struggling to understand why and how varnish is deciding to
> cache a 404 response from a backend.  It's very tough to reproduce.
> I've thrown millions of test URLs at varnish where my testing
> application will respond back randomly with a 404 and I can't trigger
> it.  Only user traffic is able to trigger this weird behavior.
>
> Anybody have any experience with this?
>
> Here's a varnishlog of a 404.
> http://pastebin.com/386VU4rC
>
> I'm working to capture the varnishlog of the resource actually getting
> fetched and stored.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc







From david.birdsong at gmail.com  Wed Aug 18 07:50:08 2010
From: david.birdsong at gmail.com (David Birdsong)
Date: Wed, 18 Aug 2010 00:50:08 -0700
Subject: varnish fetching,
	stroring and serving cached 404 from backend server
In-Reply-To: <5AAF0084CB9EC7621B7AB3E0@192.168.1.100>
References: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>
	<5AAF0084CB9EC7621B7AB3E0@192.168.1.100>
Message-ID: <AANLkTim05GVyW9m_5BmWSV=ByUMUpusD1BLarxdh3oCh@mail.gmail.com>

On Tue, Aug 17, 2010 at 7:59 PM, Michael Loftis <mloftis at wgops.com> wrote:
> Varnish caching, or not caching, is completely controlled by the VCL. ?Not
> having yours noone can help you. ?However, the *default* VCL makes anything
> with a Cookie or HTTP Authentication header from the client uncacheable.

Fair enough, I should have known vcl would be helpful, *but I didn't
think that statement was entirely true.
"completely controlled by the VCL"...I thought there were boundaries
that varnish operated in and no amount of vcl magic would override
it...such as beresp.cacheable, if that boolean evaluated to false, I
thought that varnish will never cache.  Is that incorrect?

Anyway, here's the vcl:
http://pastebin.com/egAfyT7U

>
> --On Tuesday, August 17, 2010 1:13 PM -0700 David Birdsong
> <david.birdsong at gmail.com> wrote:
>
>> I've been struggling to understand why and how varnish is deciding to
>> cache a 404 response from a backend. ?It's very tough to reproduce.
>> I've thrown millions of test URLs at varnish where my testing
>> application will respond back randomly with a 404 and I can't trigger
>> it. ?Only user traffic is able to trigger this weird behavior.
>>
>> Anybody have any experience with this?
>>
>> Here's a varnishlog of a 404.
>> http://pastebin.com/386VU4rC
>>
>> I'm working to capture the varnishlog of the resource actually getting
>> fetched and stored.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>
>
>
>
>



From piotr.teodorowski at contium.pl  Wed Aug 18 09:00:15 2010
From: piotr.teodorowski at contium.pl (Piotr Teodorowski)
Date: Wed, 18 Aug 2010 11:00:15 +0200
Subject: Problems with ACL and some prefixes
Message-ID: <201008181100.16307.piotr.teodorowski@contium.pl>

Hey,

I've noticed some problems with ACL's (which doesn't work for me for most 
subnet prefixes)

my config:
acl prd {
    "192.168.0.0"/22;
    ! "192.168.1.110";
}

varnishlog -i VCL_acl,ReqStart
   12 ReqStart     c 192.168.0.12 48855 1353135783
   12 VCL_acl      c MATCH prd 192.168.0.0/22
   12 ReqStart     c 192.168.1.91 52266 1353135784
   12 VCL_acl      c NO_MATCH prd

acl prd works only for subnet 192.168.0.0/24 not /22

if I change my configuration to
acl prd {
    "192.168.0.0"/24;
    "192.168.1.0"/24;
    "192.168.2.0"/24;
    "192.168.3.0"/24;
    ! "192.168.1.110";
}
it seems to work fine (also it works, if I use prefix /16).

I've varnish from debian squeeze:
varnishd -V
varnishd (varnish-2.1.2 SVN b8c9904)
Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS

Am I doing something wrong?

Piotr Teodorowski



From piotr.teodorowski at contium.pl  Wed Aug 18 09:45:01 2010
From: piotr.teodorowski at contium.pl (Piotr Teodorowski)
Date: Wed, 18 Aug 2010 11:45:01 +0200
Subject: Problems with ACL and some prefixes
In-Reply-To: <4C6BA88B.6060405@rcgear.co.za>
References: <201008181100.16307.piotr.teodorowski@contium.pl>
	<4C6BA88B.6060405@rcgear.co.za>
Message-ID: <201008181145.01259.piotr.teodorowski@contium.pl>

On Wednesday 18 of August 2010 11:31:55 Liaan vd Merewe wrote:
>   Piotr
> According to strict IP rules, you not allowed to supernet a 192.168.0.0
> range(its class C range).. so /22 on 192.168.0.0 is prohibited.
> 
> I don't know if that is the cause of your problem, can you maybe test on
> a 10.x.x.x range?

Yes - that's it
It works with range 10.0.0.0/22
Thanks.
Piotr Teodorowski


> 
> cheers
> L:
> 
> On 18/08/2010 11:00 AM, Piotr Teodorowski wrote:
> > Hey,
> >
> > I've noticed some problems with ACL's (which doesn't work for me for most
> > subnet prefixes)
> >
> > my config:
> > acl prd {
> >      "192.168.0.0"/22;
> >      ! "192.168.1.110";
> > }
> >
> > varnishlog -i VCL_acl,ReqStart
> >     12 ReqStart     c 192.168.0.12 48855 1353135783
> >     12 VCL_acl      c MATCH prd 192.168.0.0/22
> >     12 ReqStart     c 192.168.1.91 52266 1353135784
> >     12 VCL_acl      c NO_MATCH prd
> >
> > acl prd works only for subnet 192.168.0.0/24 not /22
> >
> > if I change my configuration to
> > acl prd {
> >      "192.168.0.0"/24;
> >      "192.168.1.0"/24;
> >      "192.168.2.0"/24;
> >      "192.168.3.0"/24;
> >      ! "192.168.1.110";
> > }
> > it seems to work fine (also it works, if I use prefix /16).
> >
> > I've varnish from debian squeeze:
> > varnishd -V
> > varnishd (varnish-2.1.2 SVN b8c9904)
> > Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >
> > Am I doing something wrong?
> >
> > Piotr Teodorowski
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
> 



From varnish at rcgear.co.za  Wed Aug 18 09:36:10 2010
From: varnish at rcgear.co.za (Liaan vd Merewe)
Date: Wed, 18 Aug 2010 11:36:10 +0200
Subject: Problems with ACL and some prefixes
In-Reply-To: <201008181100.16307.piotr.teodorowski@contium.pl>
References: <201008181100.16307.piotr.teodorowski@contium.pl>
Message-ID: <4C6BA98A.40508@rcgear.co.za>

  Piotr
According to strict IP rules, you not allowed to supernet a 192.168.0.0 
range(its class C range).. so /22 on 192.168.0.0 is prohibited.

I don't know if that is the cause of your problem, can you maybe test on 
a 10.x.x.x range?

cheers
L:


On 18/08/2010 11:00 AM, Piotr Teodorowski wrote:
> Hey,
>
> I've noticed some problems with ACL's (which doesn't work for me for most
> subnet prefixes)
>
> my config:
> acl prd {
>      "192.168.0.0"/22;
>      ! "192.168.1.110";
> }
>
> varnishlog -i VCL_acl,ReqStart
>     12 ReqStart     c 192.168.0.12 48855 1353135783
>     12 VCL_acl      c MATCH prd 192.168.0.0/22
>     12 ReqStart     c 192.168.1.91 52266 1353135784
>     12 VCL_acl      c NO_MATCH prd
>
> acl prd works only for subnet 192.168.0.0/24 not /22
>
> if I change my configuration to
> acl prd {
>      "192.168.0.0"/24;
>      "192.168.1.0"/24;
>      "192.168.2.0"/24;
>      "192.168.3.0"/24;
>      ! "192.168.1.110";
> }
> it seems to work fine (also it works, if I use prefix /16).
>
> I've varnish from debian squeeze:
> varnishd -V
> varnishd (varnish-2.1.2 SVN b8c9904)
> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
>
> Am I doing something wrong?
>
> Piotr Teodorowski
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



From varnish at rcgear.co.za  Wed Aug 18 09:31:55 2010
From: varnish at rcgear.co.za (Liaan vd Merewe)
Date: Wed, 18 Aug 2010 11:31:55 +0200
Subject: Problems with ACL and some prefixes
In-Reply-To: <201008181100.16307.piotr.teodorowski@contium.pl>
References: <201008181100.16307.piotr.teodorowski@contium.pl>
Message-ID: <4C6BA88B.6060405@rcgear.co.za>

  Piotr
According to strict IP rules, you not allowed to supernet a 192.168.0.0 
range(its class C range).. so /22 on 192.168.0.0 is prohibited.

I don't know if that is the cause of your problem, can you maybe test on 
a 10.x.x.x range?

cheers
L:


On 18/08/2010 11:00 AM, Piotr Teodorowski wrote:
> Hey,
>
> I've noticed some problems with ACL's (which doesn't work for me for most
> subnet prefixes)
>
> my config:
> acl prd {
>      "192.168.0.0"/22;
>      ! "192.168.1.110";
> }
>
> varnishlog -i VCL_acl,ReqStart
>     12 ReqStart     c 192.168.0.12 48855 1353135783
>     12 VCL_acl      c MATCH prd 192.168.0.0/22
>     12 ReqStart     c 192.168.1.91 52266 1353135784
>     12 VCL_acl      c NO_MATCH prd
>
> acl prd works only for subnet 192.168.0.0/24 not /22
>
> if I change my configuration to
> acl prd {
>      "192.168.0.0"/24;
>      "192.168.1.0"/24;
>      "192.168.2.0"/24;
>      "192.168.3.0"/24;
>      ! "192.168.1.110";
> }
> it seems to work fine (also it works, if I use prefix /16).
>
> I've varnish from debian squeeze:
> varnishd -V
> varnishd (varnish-2.1.2 SVN b8c9904)
> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
>
> Am I doing something wrong?
>
> Piotr Teodorowski
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



From svein-listmail at stillbilde.net  Wed Aug 18 10:13:42 2010
From: svein-listmail at stillbilde.net (Svein Skogen (Listmail account))
Date: Wed, 18 Aug 2010 12:13:42 +0200
Subject: Problems with ACL and some prefixes
In-Reply-To: <4C6BA88B.6060405@rcgear.co.za>
References: <201008181100.16307.piotr.teodorowski@contium.pl>
	<4C6BA88B.6060405@rcgear.co.za>
Message-ID: <4C6BB256.7000405@stillbilde.net>

On 18.08.2010 11:31, Liaan vd Merewe wrote:
>  Piotr
> According to strict IP rules, you not allowed to supernet a 192.168.0.0
> range(its class C range).. so /22 on 192.168.0.0 is prohibited.
> 
> I don't know if that is the cause of your problem, can you maybe test on
> a 10.x.x.x range?

How long ago was the conversion to classless IP again?

//Svein

-- 
--------+-------------------+-------------------------------
  /"\   |Svein Skogen       | svein at d80.iso100.no
  \ /   |Solberg ?stli 9    | PGP Key:  0xE5E76831
   X    |2020 Skedsmokorset | svein at jernhuset.no
  / \   |Norway             | PGP Key:  0xCE96CE13
        |                   | svein at stillbilde.net
 ascii  |                   | PGP Key:  0x58CD33B6
 ribbon |System Admin       | svein-listmail at stillbilde.net
Campaign|stillbilde.net     | PGP Key:  0x22D494A4
        +-------------------+-------------------------------
        |msn messenger:     | Mobile Phone: +47 907 03 575
        |svein at jernhuset.no | RIPE handle:    SS16503-RIPE
--------+-------------------+-------------------------------
         If you really are in a hurry, mail me at
               svein-mobile at stillbilde.net
 This mailbox goes directly to my cellphone and is checked
        even when I'm not in front of my computer.
------------------------------------------------------------
                     Picture Gallery:
          https://gallery.stillbilde.net/v/svein/
------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100818/5db12bc1/attachment.bin>

From ssm at redpill-linpro.com  Wed Aug 18 10:55:45 2010
From: ssm at redpill-linpro.com (Stig Sandbeck Mathisen)
Date: Wed, 18 Aug 2010 12:55:45 +0200
Subject: Problems with ACL and some prefixes
In-Reply-To: <4C6BB256.7000405@stillbilde.net> (Svein Skogen's message of
	"Wed, 18 Aug 2010 12:13:42 +0200")
References: <201008181100.16307.piotr.teodorowski@contium.pl>
	<4C6BA88B.6060405@rcgear.co.za> <4C6BB256.7000405@stillbilde.net>
Message-ID: <7x39uckx5a.fsf@fsck.linpro.no>

"Svein Skogen (Listmail account)" <svein-listmail at stillbilde.net>
writes:

> On 18.08.2010 11:31, Liaan vd Merewe wrote:
>> According to strict IP rules, you not allowed to supernet a
>> 192.168.0.0 range(its class C range).. so /22 on 192.168.0.0 is
>> prohibited.
>> 
>> I don't know if that is the cause of your problem, can you maybe test
>> on a 10.x.x.x range?
>
> How long ago was the conversion to classless IP again?

The CIDR RFC dates 1993.  :)

However, the VCL compiler does not complain.  If the newest varnish
still acts up on that CIDR, I will hazard a guess that it's not caused
by excessive and unwarranted nostalgia.

-- 
Stig Sandbeck Mathisen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100818/3aee7220/attachment.bin>

From mloftis at wgops.com  Wed Aug 18 17:49:50 2010
From: mloftis at wgops.com (Michael Loftis)
Date: Wed, 18 Aug 2010 11:49:50 -0600
Subject: varnish fetching, stroring and serving cached 404 from backend
	server
In-Reply-To: <AANLkTim05GVyW9m_5BmWSV=ByUMUpusD1BLarxdh3oCh@mail.gmail.com>
References: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>
	<5AAF0084CB9EC7621B7AB3E0@192.168.1.100>
	<AANLkTim05GVyW9m_5BmWSV=ByUMUpusD1BLarxdh3oCh@mail.gmail.com>
Message-ID: <66B6F285CB8D2E5C2D03BDD8@[192.168.1.100]>



--On Wednesday, August 18, 2010 12:50 AM -0700 David Birdsong 
<david.birdsong at gmail.com> wrote:

> On Tue, Aug 17, 2010 at 7:59 PM, Michael Loftis <mloftis at wgops.com> wrote:
>> Varnish caching, or not caching, is completely controlled by the VCL.
>> ?Not having yours noone can help you. ?However, the *default* VCL
>> makes anything with a Cookie or HTTP Authentication header from the
>> client uncacheable.
>
> Fair enough, I should have known vcl would be helpful, *but I didn't
> think that statement was entirely true.
> "completely controlled by the VCL"...I thought there were boundaries
> that varnish operated in and no amount of vcl magic would override
> it...such as beresp.cacheable, if that boolean evaluated to false, I
> thought that varnish will never cache.  Is that incorrect?

<http://www.gossamer-threads.com/lists/varnish/misc/14286> i think that 
thread should answer your ?'s on beresp.cacheable, in short, it is not 
authoritative.  Further more 404's are perfectly cacheable.


You're caching all requests for a big list of extensions on line 91...

I'm not sure about case sensitivity in the req.http.cookie statement but 
since you're passing everything to lookup anyway in the end you *should* be 
caching 404s.  I'm actually reading this and trying to figure out why a 404 
*wouldn't* cache.  In fact the only places I don't see you caching are if 
it has an auth header, a cookie AND that cookie looks like one of your auth 
cookies.  That's the only reason I could see your code NOT caching. 
Expect, Auth headers, and a cookie that looks like your auth cookie.

So I'm guessing that once in a while a miss is resulting in a 404 which 
then (appropriately) gets cached.  If you don't want 404's getting cached 
set their TTL to 0 in vcl_fetch.  Just remember that'll leave your backends 
open to that sort of mischief.

It looks like tour vcl_fetch does do a fall through to the default 
vcl_fetch, so unless there are overriding expires or cache-control headers 
coming back on 404's, they'll be cached.  If you don't want them cached you 
need to pass on them in vcl_fetch.

>
> Anyway, here's the vcl:
> http://pastebin.com/egAfyT7U




From harri.paivaniemi at tieto.com  Thu Aug 19 08:08:03 2010
From: harri.paivaniemi at tieto.com (=?iso-8859-1?q?H=2EP=E4iv=E4niemi?=)
Date: Thu, 19 Aug 2010 11:08:03 +0300
Subject: How to make a virtual host?
Message-ID: <201008191108.03423.harri.paivaniemi@tieto.com>

Hi,

I'm a varnish-newbie.

How on the earth can I make a virtual host in varnish?

I would like to, for example:

* make a default virtual host with a backend A

* make a virtual host "host1.foo.bar" with a backend B

* make a virtual host "host2.foo.bar" with a backend C

...and of course there would be different caching rules in all of those.

But how?

Thanks,

-hjp




From bjorn at ruberg.no  Thu Aug 19 08:35:58 2010
From: bjorn at ruberg.no (=?ISO-8859-1?Q?Bj=F8rn_Ruberg?=)
Date: Thu, 19 Aug 2010 10:35:58 +0200
Subject: How to make a virtual host?
In-Reply-To: <201008191108.03423.harri.paivaniemi@tieto.com>
References: <201008191108.03423.harri.paivaniemi@tieto.com>
Message-ID: <4C6CECEE.4030400@ruberg.no>

On 08/19/2010 10:08 AM, H.P?iv?niemi wrote:
> Hi,
>
> I'm a varnish-newbie.
>
> How on the earth can I make a virtual host in varnish?
>
> I would like to, for example:
>
> * make a default virtual host with a backend A
>
> * make a virtual host "host1.foo.bar" with a backend B
>
> * make a virtual host "host2.foo.bar" with a backend C
>
> ...and of course there would be different caching rules in all of those.
>
> But how?

Varnish doesn't need any particular definitions of virtual hosts, but 
will look for the "Host:" header in the HTTP requests and can make 
decisions based on that.

After you have predefined your backends A, B and C, you can do something 
like this:

sub vcl_recv {

     set req.backend = A; # This is the default backend unless
			 # another backend is set

     if (req.http.host == "host1.foo.bar") {
         set req.backend = B;
	# Add your caching rules for host1.foo.bar here

     } elsif (req.http.host )) "host2.foo.bar") {
	set req.backend = C;
	# Add your caching rules for host2.foo.bar here

     }
}

You may need to specify different caching rules in vcl_fetch as well, YMMV.

-- 
Bj?rn





From frank.gruellich at mapsolute.com  Thu Aug 19 12:48:35 2010
From: frank.gruellich at mapsolute.com (Frank Gruellich)
Date: Thu, 19 Aug 2010 14:48:35 +0200
Subject: "now" in VCL
Message-ID: <4C6D2823.6060604@mapsolute.com>

Hi,

according to "man vcl" there is a variable "now":

   Variables
     [...]

     The following variables are always available:

     now   The current time, in seconds since the epoch.

However:

root at floor [~] $ cat foo.vcl
sub vcl_recv {
        if (now)
        {
                set req.backend = localhost;
        } else {
                set req.backend = localhost;
        }
}
root at floor [~] $ varnishd -C -f ./foo.vcl
Message from VCC-compiler:
Syntax error in condition, expected '(', '!' or variable name, found 'now'
(input Line 2 Pos 13)
        if (now)
------------###-
Running VCC-compiler failed, exit 1root at floor [~] $

Hm.  What am I doing wrong?  Or was it renamed?  Removed?

Thanks in advance.

Kind regards,
-- 
Navteq (DE) GmbH
Frank Gruellich
Map24 Systems and Networks

Duesseldorfer Strasse 40a
65760 Eschborn
Germany

Phone:      +49 6196 77756-414
Fax:        +49 6196 77756-100

HRB 46215, Local Court Frankfurt am Main
Managing Directors: Thomas Golob, Hans Pieter Gieszen, Martin Robert Stockman
USt-ID-No.: DE 197947163

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100819/65cae858/attachment.bin>

From phk at phk.freebsd.dk  Sun Aug 22 20:46:51 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 22 Aug 2010 20:46:51 +0000
Subject: PHK's talk at DrupalCon10@CPH and T3con10@Frankfurt
Message-ID: <79300.1282510011@critter.freebsd.dk>


Just a quick note that I'll be giving a Varnish talk right after the
keynote on DrupalCon10 in Copenhagen on tuesday (23rd aug)

I will stick around after the talk, but I will not be attending the
entire conference.

I'm also scheduled to give what is more or less the same presentation
at T3CON in Frankfurt in October, but don't have the scheduling
information for that yet.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From joao.lisanti at locaweb.com.br  Mon Aug 23 15:27:22 2010
From: joao.lisanti at locaweb.com.br (=?iso-8859-1?Q?Jo=E3o_Gabriel?=)
Date: Mon, 23 Aug 2010 12:27:22 -0300
Subject: Error 5** - Guru Meditation
Message-ID: <65062089-776F-40C8-B89B-219894CE6BB1@locaweb.com.br>

Hi guys,
I want to know if when it is possible when an error occurs 500, instead of display page "guru meditation" of Varnish really show the error in Apache, because many times the error is on
implementation and debugging becomes more complicated.

Thanx in advance!
Jo?o Gabriel
CT-Linux




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100823/aac9874c/attachment.html>

From zabrane3 at gmail.com  Mon Aug 23 23:30:51 2010
From: zabrane3 at gmail.com (zabrane Mikael)
Date: Tue, 24 Aug 2010 01:30:51 +0200
Subject: v2.1.4 ?
Message-ID: <AANLkTin+P_MFSFeRJ7TVkaeVsvnfP1tzWMnzaO1J47D=@mail.gmail.com>

Hi guys,

When the new Varnish v2.1.4 will be released?

Thanks
-- 
Regards
Zabrane



From phk at phk.freebsd.dk  Tue Aug 24 06:42:19 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Aug 2010 06:42:19 +0000
Subject: v2.1.4 ? 
In-Reply-To: Your message of "Tue, 24 Aug 2010 01:30:51 +0200."
	<AANLkTin+P_MFSFeRJ7TVkaeVsvnfP1tzWMnzaO1J47D=@mail.gmail.com> 
Message-ID: <90300.1282632139@critter.freebsd.dk>

In message <AANLkTin+P_MFSFeRJ7TVkaeVsvnfP1tzWMnzaO1J47D=@mail.gmail.com>, zabr
ane Mikael writes:

>When the new Varnish v2.1.4 will be released?

I have heard of neither plans nor good reason to.

The next release I am working on is 3.0

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From jonathan.demello at itp.com  Tue Aug 24 06:57:06 2010
From: jonathan.demello at itp.com (Jonathan DeMello)
Date: Tue, 24 Aug 2010 10:57:06 +0400
Subject: v2.1.4 ?
In-Reply-To: <90300.1282632139@critter.freebsd.dk>
References: <AANLkTin+P_MFSFeRJ7TVkaeVsvnfP1tzWMnzaO1J47D=@mail.gmail.com>
	<90300.1282632139@critter.freebsd.dk>
Message-ID: <AANLkTin4WOmihwpknm9Uz63Czox--Loez1xjbzj=FxDc@mail.gmail.com>

Hey Paul,

Is 3.0 going to include a workaround for gzip while using ESI?

Regards,

Jonathan

On Tue, Aug 24, 2010 at 10:42 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <AANLkTin+P_MFSFeRJ7TVkaeVsvnfP1tzWMnzaO1J47D=@mail.gmail.com>, zabr
> ane Mikael writes:
>
>>When the new Varnish v2.1.4 will be released?
>
> I have heard of neither plans nor good reason to.
>
> The next release I am working on is 3.0
>
> --
> Poul-Henning Kamp ? ? ? | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG ? ? ? ? | TCP/IP since RFC 956
> FreeBSD committer ? ? ? | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>
>




From harald at webmeisterei.com  Tue Aug 24 10:27:05 2010
From: harald at webmeisterei.com (Harald Friessnegger)
Date: Tue, 24 Aug 2010 12:27:05 +0200
Subject: Error 5** - Guru Meditation
References: <65062089-776F-40C8-B89B-219894CE6BB1@locaweb.com.br>
Message-ID: <i506pq$pl6$1@dough.gmane.org>

hi 

varnish is sending a 500 status code so it's up to you to configure apache 
accordingly to handle the error itself instead of passing it.

i don't know the way to do it with apache, for nginx it would be

  proxy_intercept_errors    on; #replace varnish or nginx error pages

  error_page   502 503 =503  /status/502.html;
  location /status {
    root /some/path/status/directory/resides/in/;
  }

regards, harald

Jo?o Gabriel wrote:

> Hi guys,
> I want to know if when it is possible when an error occurs 500, instead of
> display page "guru meditation" of Varnish really show the error in Apache,
> because many times the error is on implementation and debugging becomes
> more complicated.
> 
> Thanx in advance!
> Jo?o Gabriel
> CT-Linux

-- 
Webmeisterei GmbH - B?ro f?r Netzfragen
Tel: +43 5572 908877,  Fax: +43 5572 908877-66
Steinebach 18, A-6850 Dornbirn
http://www.webmeisterei.com




From angie.tawfik at gmail.com  Wed Aug 25 01:07:26 2010
From: angie.tawfik at gmail.com (Angie T. Muhammad)
Date: Wed, 25 Aug 2010 04:07:26 +0300
Subject: RPMS of varnish-2.1.3 available
In-Reply-To: <4C6A82F5.80900@redpill-linpro.com>
References: <4C6A82F5.80900@redpill-linpro.com>
Message-ID: <AANLkTin780c1Q-_CY=ZSsBGJxCrjy7px5aHki4OzuWa5@mail.gmail.com>

In test on a staging server. Will feed you back in case any unusual issue
takes place.
Thanks a lot :)

On Tue, Aug 17, 2010 at 3:39 PM, Ingvar Hagelund
<ingvar at redpill-linpro.com>wrote:

>  I built RPMS of varnish-2.1.3 for RHEL4 and 5 and compatible derivates.
> Source and binary packages for x86_64 are available at
> http://sourceforge.net/projects/varnish/files/
>
> If anybody needs packages for other arches or RHEL versions, have a look at
> http://users.linpro.no/ingvar/varnish/ , or contact me.
>
> These packages have been tested in production, and should prove quite
> stable, but feedback is as usual always very welcome.
>
> Ingvar
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



-- 
All the best,
Angie T. Muhammad
Linux Systems Engineer
Al Masry Al Youm
http://www.almasryalyoum.com/en
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100825/8ba6b99f/attachment.html>

From angie.tawfik at gmail.com  Wed Aug 25 02:13:20 2010
From: angie.tawfik at gmail.com (Angie T. Muhammad)
Date: Wed, 25 Aug 2010 05:13:20 +0300
Subject: RPMS of varnish-2.1.3 available
In-Reply-To: <AANLkTin780c1Q-_CY=ZSsBGJxCrjy7px5aHki4OzuWa5@mail.gmail.com>
References: <4C6A82F5.80900@redpill-linpro.com>
	<AANLkTin780c1Q-_CY=ZSsBGJxCrjy7px5aHki4OzuWa5@mail.gmail.com>
Message-ID: <AANLkTik4A2B=-kD5wXhd4GRuivmRQ34KvKiLFXSDMi2d@mail.gmail.com>

Hello,
 I am not sure if this bug is reproducable, but it is ironic because Varnish
starts successfully but the INIT script reports a FAIL.

# /etc/init.d/varnish start
# TTP accelerator:                         [FAILED]

# pidof varnishd
3383 3378

# cat /var/run/varnish.pid
3378

# /etc/init.d/varnish status
varnishd (pid  3378) is running...

# netstat -antpue | grep -i varnish
tcp        0      0 127.0.0.1:6082              0.0.0.0:*
LISTEN      0          13017      3378/varnishd
tcp        0      0 0.0.0.0:80                  0.0.0.0:*
LISTEN      0          13018      3383/varnishd

# telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
200 154
-----------------------------
Varnish HTTP accelerator CLI.
-----------------------------
Type 'help' for command list.
Type 'quit' to close CLI session.


# /etc/init.d/varnish stop
Stopping varnish HTTP accelerator:                         [  OK  ]

# /etc/init.d/varnish start
# TTP accelerator:                         [FAILED]

I guarantee that after this fake FAILED varnish is working properly, but the
FAILED on the init script start is a bit scary at first sight. I tried
varnish packages on other machines and they are working fine. Nothing
special about the prompt of this machine that reports a fake failure. Its
specs are:

* Kernel 2.6.18-164.11.1.el5
* x86_64
* 4G RAMs
* CentOS 5.4

Is it reproducable with anyone? Any ideas ? It is not a big deal though,
simply wanted to report it.
Thanks.
------------------------------------------------------------------------------------------------------------
On Wed, Aug 25, 2010 at 4:07 AM, Angie T. Muhammad
<angie.tawfik at gmail.com>wrote:

> In test on a staging server. Will feed you back in case any unusual issue
> takes place.
> Thanks a lot :)
>
>
> On Tue, Aug 17, 2010 at 3:39 PM, Ingvar Hagelund <
> ingvar at redpill-linpro.com> wrote:
>
>>  I built RPMS of varnish-2.1.3 for RHEL4 and 5 and compatible derivates.
>> Source and binary packages for x86_64 are available at
>> http://sourceforge.net/projects/varnish/files/
>>
>> If anybody needs packages for other arches or RHEL versions, have a look
>> at
>> http://users.linpro.no/ingvar/varnish/ , or contact me.
>>
>> These packages have been tested in production, and should prove quite
>> stable, but feedback is as usual always very welcome.
>>
>> Ingvar
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>>
>
>
>
> --
> All the best,
> Angie T. Muhammad
> Linux Systems Engineer
> Al Masry Al Youm
> http://www.almasryalyoum.com/en
>



-- 
All the best,
Angie T. Muhammad
Linux Systems Engineer
Al Masry Al Youm
http://www.almasryalyoum.com/en
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100825/69c2b9e5/attachment.html>

From alessandro.ronchi at soasi.com  Thu Aug 26 07:21:54 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Thu, 26 Aug 2010 09:21:54 +0200
Subject: Duplicate caching of zippend and unzipped CSS and JS files
Message-ID: <AANLkTinohMqYiaCo3-J5U66nK3qr72r915zyEvk768xP@mail.gmail.com>

In my project I have both gzip and plain version of CSS and JS files, based
on user browser preferences.
How can I make varnish cache both versions and give the user the correct
version?

Thank you in advance.

-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100826/cf87f2c5/attachment.html>

From perbu at varnish-software.com  Thu Aug 26 07:35:39 2010
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 26 Aug 2010 09:35:39 +0200
Subject: Duplicate caching of zippend and unzipped CSS and JS files
In-Reply-To: <AANLkTinohMqYiaCo3-J5U66nK3qr72r915zyEvk768xP@mail.gmail.com>
References: <AANLkTinohMqYiaCo3-J5U66nK3qr72r915zyEvk768xP@mail.gmail.com>
Message-ID: <AANLkTi=DFs-6OA+P-5Kx1Xj5+WFKbaff+aBrV6ZcmGU5@mail.gmail.com>

On Thu, Aug 26, 2010 at 9:21 AM, Alessandro Ronchi <
alessandro.ronchi at soasi.com> wrote:

> In my project I have both gzip and plain version of CSS and JS files, based
> on user browser preferences. How can I make varnish cache both versions and
> give the user the correct version?
>

Varnish will cache both versions if your back end server sends a "Vary:
Content-Encoding" header. You probably want to normalize this header. See
http://varnish-cache.org/docs/trunk/tutorial/increasing_your_hitrate.html#vary
for
details on how to do that.

-- 
Per Buer,  Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100826/6c93d2d2/attachment.html>

From alessandro.ronchi at soasi.com  Thu Aug 26 12:27:26 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Thu, 26 Aug 2010 14:27:26 +0200
Subject: Set page last-modified
Message-ID: <AANLkTimH3hZ4-YV432OWwscxacg7=06ag6Vpeasm8Goa@mail.gmail.com>

Is it possible to set the last-modified for cached objects?
it seem varnish strips them in my project:
http://hobbygiochi.com



-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100826/2f45a391/attachment.html>

From jdixon at omniti.com  Fri Aug 27 19:34:03 2010
From: jdixon at omniti.com (Jason Dixon)
Date: Fri, 27 Aug 2010 15:34:03 -0400
Subject: Surge 2010 Early Registration ends Tuesday!
Message-ID: <20100827193403.GL1736@omniti.com>

Early Bird Registration for Surge Scalability Conference 2010 ends next
Tuesday, August 31.  We have a killer lineup of speakers and architects
from across the Internet.  Listen to experts talk about the newest
methods and technologies for scaling your Web presence.

http://omniti.com/surge/2010/register

This year's event is all about the challenges faced (and overcome) in
real-life production architectures.  Meet the engineering talent from
some of the best and brightest throughout the Internet:

John Allspaw, Etsy
Theo Schlossnagle, OmniTI
Bryan Cantrill, Joyent
Rasmus Lerdorf, creator of PHP
Tom Cook, Facebook
Benjamin Black, fast_ip
Christopher Brown, Opscode
Artur Bergman, Wikia
Baron Schwartz, Percona
Paul Querna, Cloudkick

Surge 2010 takes place at the Tremont Grand Historic Venue on Sept 30
and Oct 1, 2010 in Baltimore, MD.  Register NOW for the Early Bird
discount and guarantee your seat to this year's event!


-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241



From alessandro.ronchi at soasi.com  Sat Aug 28 18:13:08 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Sat, 28 Aug 2010 20:13:08 +0200
Subject: Limit varnish RAM usage
Message-ID: <AANLkTimu5-nBs-DidDE01b5XQX9iC6BmDvP3-Od4efa1@mail.gmail.com>

I've a virtual server with a very little RAM capacity.
I've no problems with hard disk usage.

Is it possible to limit RAM usage and increase disk use?

-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100828/547144ea/attachment.html>

From alessandro.ronchi at soasi.com  Sat Aug 28 19:46:35 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Sat, 28 Aug 2010 21:46:35 +0200
Subject: help with migration to 2.1
Message-ID: <AANLkTinTQOdirgta0sJ6nj-Ohe+s8u5kvvXKbkmAYAGP@mail.gmail.com>

with 2.0 I used this configuration:
http://dpaste.com/235429/

now i changed few things to make it work on fetch:
http://dpaste.com/235428/

it caches something (like minimized css) but it doesn't cache my pages.
One example:
http://hobbygiochi.com

please help me, maybe  I forgot some changes.

I also tried disabling cookie in firefox, but it's seems the problem is not
there...

-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100828/1f2bfccd/attachment.html>

From alessandro.ronchi at soasi.com  Sat Aug 28 22:31:53 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Sun, 29 Aug 2010 00:31:53 +0200
Subject: help with migration to 2.1
In-Reply-To: <AANLkTinTQOdirgta0sJ6nj-Ohe+s8u5kvvXKbkmAYAGP@mail.gmail.com>
References: <AANLkTinTQOdirgta0sJ6nj-Ohe+s8u5kvvXKbkmAYAGP@mail.gmail.com>
Message-ID: <AANLkTimwZ1JTak4NuLQXzYj8baNvEGfeG2LrvDFx9g8B@mail.gmail.com>

I solved this problem. It was ignoring default TTL settings.

2010/8/28 Alessandro Ronchi <alessandro.ronchi at soasi.com>

> with 2.0 I used this configuration:
>



-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100829/02fc95e6/attachment.html>

From tfheen at varnish-software.com  Mon Aug 30 06:19:43 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 30 Aug 2010 08:19:43 +0200
Subject: "Total body bytes" not actually bytes delivered?
In-Reply-To: <4C61B384.9060203@d6.com> (Chris Hecker's message of "Tue, 10 Aug
	2010 13:16:04 -0700")
References: <4C515EF5.70904@d6.com> <87y6cfhqp1.fsf@qurzaw.linpro.no>
	<4C61B384.9060203@d6.com>
Message-ID: <87iq2s4o74.fsf@qurzaw.linpro.no>

]] Chris Hecker 

| I don't have time right now to write a test program or 100% verify
| that this was going on, but I'm pretty sure it was from watching
| varnishstat. Would you rather I file a bug now with just the info I've
| got, or wait until I have time to make it more rigorous?

Just file the ticket with what you have; if it's wrong, we can just
close it.  :-)

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From tfheen at varnish-software.com  Mon Aug 30 06:22:26 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 30 Aug 2010 08:22:26 +0200
Subject: varnish fetching,
	stroring and serving cached 404 from backend server
In-Reply-To: <AANLkTim05GVyW9m_5BmWSV=ByUMUpusD1BLarxdh3oCh@mail.gmail.com>
	(David Birdsong's message of "Wed, 18 Aug 2010 00:50:08 -0700")
References: <AANLkTimLvxv0qiMkzZ1exFZ6PC4gCfw3HKRGSRQ04Jrn@mail.gmail.com>
	<5AAF0084CB9EC7621B7AB3E0@192.168.1.100>
	<AANLkTim05GVyW9m_5BmWSV=ByUMUpusD1BLarxdh3oCh@mail.gmail.com>
Message-ID: <87eidg4o2l.fsf@qurzaw.linpro.no>

]] David Birdsong 

| [...] ..such as beresp.cacheable, if that boolean evaluated to false, I
| thought that varnish will never cache.  Is that incorrect?

beresp.cacheable is settable from VCL, so even if it's false when you
enter vcl_fetch, that's easy to change.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From tfheen at varnish-software.com  Mon Aug 30 06:25:35 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 30 Aug 2010 08:25:35 +0200
Subject: "now" in VCL
In-Reply-To: <4C6D2823.6060604@mapsolute.com> (Frank Gruellich's message of
	"Thu, 19 Aug 2010 14:48:35 +0200")
References: <4C6D2823.6060604@mapsolute.com>
Message-ID: <87aao44nxc.fsf@qurzaw.linpro.no>

]] Frank Gruellich 

| root at floor [~] $ cat foo.vcl
| sub vcl_recv {
|         if (now)
|         {
|                 set req.backend = localhost;
|         } else {
|                 set req.backend = localhost;
|         }
| }

This VCL snippet doesn't make any sense; what are you trying to
accomplish?

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From alessandro.ronchi at soasi.com  Mon Aug 30 12:05:41 2010
From: alessandro.ronchi at soasi.com (Alessandro Ronchi)
Date: Mon, 30 Aug 2010 14:05:41 +0200
Subject: varnishlog to show txurl with domain name
Message-ID: <AANLkTinTyk0burg6YFm7gw0zOYKfJKu8Ra1OkoM2Xv6G@mail.gmail.com>

is it possible to ask varnishlog
varnishlog -b -i TxUrl

to tell also the domain name of every request?
I've multiple sites and I need to know where are those requests made...

-- 
Alessandro Ronchi
http://www.soasi.com

Hobby & Giochi
http://hobbygiochi.com
http://www.facebook.com/hobbygiochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100830/b791f653/attachment.html>

From frank.gruellich at mapsolute.com  Mon Aug 30 12:14:58 2010
From: frank.gruellich at mapsolute.com (Frank Gruellich)
Date: Mon, 30 Aug 2010 14:14:58 +0200
Subject: "now" in VCL
In-Reply-To: <87aao44nxc.fsf@qurzaw.linpro.no>
References: <4C6D2823.6060604@mapsolute.com> <87aao44nxc.fsf@qurzaw.linpro.no>
Message-ID: <4C7BA0C2.1000508@mapsolute.com>

Hi,

On 08/30/10 08:25, Tollef Fog Heen wrote:
> ]] Frank Gruellich 
> | root at floor [~] $ cat foo.vcl
> | sub vcl_recv {
> |         if (now)
> |         {
> |                 set req.backend = localhost;
> |         } else {
> |                 set req.backend = localhost;
> |         }
> | }
> 
> This VCL snippet doesn't make any sense; what are you trying to
> accomplish?

Yes, I'm aware of this.  I tried to strip down my real problem VCL, but
I just noticed that the real configuration isn't much bigger.  This is
the offending part:

sub vcl_miss {
	log "MISS: " now req.url;
	return (fetch);
}

sub vcl_hit {
	if (!obj.cacheable) {
		return (pass);
	}
	log "HIT: " now req.url;
	return (deliver);
}

I'm still getting a similar error message:

Message from VCC-compiler:
Expected ';' got 'now'
(program line 491), at
(input Line 123 Pos 22)
        log "MISS: " now req.url;
---------------------###---------
Running VCC-compiler failed, exit 1

"now" seems to be gone...

Kind regards,
-- 
Navteq (DE) GmbH
Frank Gruellich
Map24 Systems and Networks

Duesseldorfer Strasse 40a
65760 Eschborn
Germany

Phone:      +49 6196 77756-414
Fax:        +49 6196 77756-100

HRB 46215, Local Court Frankfurt am Main
Managing Directors: Thomas Golob, Hans Pieter Gieszen, Martin Robert Stockman
USt-ID-No.: DE 197947163

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100830/d1f24577/attachment.bin>

From bjorn at ruberg.no  Mon Aug 30 12:57:01 2010
From: bjorn at ruberg.no (=?UTF-8?B?QmrDuHJuIFJ1YmVyZw==?=)
Date: Mon, 30 Aug 2010 14:57:01 +0200
Subject: varnishlog to show txurl with domain name
In-Reply-To: <AANLkTinTyk0burg6YFm7gw0zOYKfJKu8Ra1OkoM2Xv6G@mail.gmail.com>
References: <AANLkTinTyk0burg6YFm7gw0zOYKfJKu8Ra1OkoM2Xv6G@mail.gmail.com>
Message-ID: <4C7BAA9D.6090400@ruberg.no>

On 08/30/2010 02:05 PM, Alessandro Ronchi wrote:
> is it possible to ask varnishlog
> varnishlog -b -i TxUrl
>
> to tell also the domain name of every request?
> I've multiple sites and I need to know where are those requests made...

You need to add TxHeader to the request, and then grep for the Host: header.

I often use something like this:

  varnishlog -i TxURL,TxHeader -b -o | egrep 'TxURL|Host:|^$'

-- 
Bj?rn




From schmidt at ze.tum.de  Mon Aug 30 13:21:17 2010
From: schmidt at ze.tum.de (Gerhard Schmidt)
Date: Mon, 30 Aug 2010 15:21:17 +0200
Subject: predefine probes and maximum lenght of vcl.inline configs
Message-ID: <4C7BB04D.6030901@ze.tum.de>

Hi,

we a running varnish as reversproxy for our university portal. We've 19
backendservers each running 4 Zope processes. So we have 76 backends.
so defining for each of this backends a probe is quite a job, not to mention
changing some settings.

Is there a way to define probs once and reuse them in the backend definitions.
I've tried

probe portal {...} and use it via .probe = portal; compiling failes at probe
portal.

<---- snip ---->
Message from VCC-compiler:
Expected one of
        'acl', 'sub', 'backend',  or 'director'
Found: 'probe' at
(input Line 1 Pos 1)
probe portal {
#####---------
Running VCC-compiler failed, exit 1
VCL compilation failed
<---- snip ---->

The problem increases because we update the config via vcl.inline over the cli
interface and there seam to be a maximum length for the config.
defining the probe only once would reduce the config size greatly.

Kind regard,
Estartu


-- 
-------------------------------------------------
Gerhard Schmidt       | E-Mail: schmidt at ze.tum.de
TU-M?nchen	      |
WWW & Online Services |
Tel: 089/289-25270    |
Fax: 089/289-25257    | PGP-Publickey auf Anfrage


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 544 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100830/8b4bfe2e/attachment.bin>

From david.birdsong at gmail.com  Tue Aug 31 22:42:37 2010
From: david.birdsong at gmail.com (David Birdsong)
Date: Tue, 31 Aug 2010 15:42:37 -0700
Subject: help understanding backend_health varnishlog line
Message-ID: <AANLkTikbP5APFGiocaOqPPKX2wUe_=Z6yAf-Oqat_M-P@mail.gmail.com>

I'm trying to understand why varnish's health checks marks my backend
down at random intervals.  Could somebody explain the columns in these
lines?

varnishlog  -i backend_health  | grep -i sick
    0 Backend_health - mogfe Went sick 4--X--- 2 3 3 0.000000 0.000387

This is a healthy line:
  0 Backend_health - mogfe Still healthy 4--X-RH 3 3 3 0.000267
0.000315 HTTP/1.1 200 OK


My backend is nginx on the same machine.  Varnish connects over
127.0.0.1.  The check stanza in nginx simply returns a 1x1 gif that
nginx keeps around in memory so I'm really confused what could be
causing the bad response.

I captured one of the failed checks in a pcap file and searched on TCP
RST flag set and only found one session which I dont know whether it
corresponds to the check or not.  It went:

varnish: SYN time: 15.55156 seq=0
nginx: ACK   time: 15.551577 seq=1 ack=3034931510
varnish: RST time: 15.551589 seq=3034931510

...no http check after the RST obviously.



From anand at rediff-inc.com  Mon Aug  9 07:16:53 2010
From: anand at rediff-inc.com (Anand Shah)
Date: Mon, 09 Aug 2010 07:16:53 -0000
Subject: Varnish on SSD
Message-ID: <4433C0C09D2441E397B23EF6066803B5@rediff.net>

Hi Guys,

 

This is my first post on mailing list and would like to have your
suggestions on this. 

 

 

I'm contemplating setting up a varnish cache on a system with SSD drives.

 

The obvious benefit is that these systems have great READ speeds and I
expect my hit ratios to be fairly high.

 

Let's assume I can put 7 SSDs in to a RAID configuration. (there are some
cases that will let me pack in much much more)

 

Implementation questions:

 

Should I use RAID0? (I expect a drive to fail eventually, so this seems
dangerous.)

 

Should I use RAID10? (This halves my disk footprint, which is costly.)

 

Should I use RAID5? (SSDs are known to have "bad" write performance and
write limits, and all the extra parity writes may slow this down
considerably.)

 

Should I just treat each disk as it's own squid datastore? (how well does
squid handle multiple data stores? and what happens if/when one fails?)

 

Should I ignore datastores and just make the SSDs in to large SWAP
partitions and let the linux VM do it's thing? (seems sloppy)

 

Any advice from you using SSDs in production environments would be greatly
appreciated. (esp if you're using them for HTTP caches)

 

Also how difficult it is for a varnish cache to allow users manage data life
cycle policy from SSD/NAND flash drives to disk. There is an open source
project on Online Hierarchical Storage Manager (OHSM) (e.g.
http://code.google.com/p/fscops/)  which i am on verge of trying bt havent
succeeded yet.

 

 

Awaiting for your reply on this!!!!

 

 

 

 

Regards,

   Anand Shah

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20100809/d5d1529d/attachment.html>