From indranilc at rediff-inc.com Fri Oct 1 03:06:39 2010 From: indranilc at rediff-inc.com (Indranil Chakravorty) Date: 1 Oct 2010 03:06:39 -0000 Subject: =?utf-8?B?TG9va2luZyB0byBpbXBsZW1lbnQgIklmIG1vZGlmaWVkIHNpbmNlIiBiZXR3ZWVuIGJyb3dzZXIgYW5kIHZhcm5pc2ggY2FjaGU=?= Message-ID: <20101001030639.24101.qmail@pro-147-138.rediffmailpro.com> Hi,  I have been mulling over the feasibility of implementing conditional requests between the client browser and varnish cache. My plan is to add a "If modified since" header while fetching the object from the backend for the client's first request. Subsequently, if the client requests the same object, then at varnish cache, comparison would be made between the "If-modified-since" header and the date time of the object in the cache. Incase, the date time is not newer then a 304 response would be returned by the cache and the client's browser would pick up the request from its own cache. My concerns:1. Do all the common browsers support reading from their own cache on receiving a 304 response?2. Is there a way to read the date time stamp of the object/ data in the cache using vcl? Thanks in advance for any suggestion, pointers. -Neel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ask at develooper.com Fri Oct 1 04:16:16 2010 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Thu, 30 Sep 2010 21:16:16 -0700 Subject: Looking to implement "If modified since" between browser and varnish cache In-Reply-To: <20101001030639.24101.qmail@pro-147-138.rediffmailpro.com> References: <20101001030639.24101.qmail@pro-147-138.rediffmailpro.com> Message-ID: On Sep 30, 2010, at 20:06, Indranil Chakravorty wrote: > I have been mulling over the feasibility of implementing conditional requests between the client browser and varnish cache. You are in luck - that already works. :-) And yes - all common browsers support this. You can combine a short-ish Cache-Control maxage header with a longer cache time in Varnish to get some control of how long content is cached for. - ask First request (get Last-Modified for the next request) ============ [ask at embla ~]$ curl -svo/dev/null http://www.pool.ntp.org/en/ * About to connect() to www.pool.ntp.org port 80 (#0) * Trying 207.171.3.6... connected * Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) > GET /en/ HTTP/1.1 > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 > Host: www.pool.ntp.org > Accept: */* > * HTTP 1.0, assume close after body < HTTP/1.0 200 OK < Server: Apache/2.2.3 (Red Hat) mod_apreq2-20051231/2.6.1 mod_perl/2.0.4 Perl/v5.8.8 < X-UA-Compatible: IE=8 < P3P: CP="NOI DEVo TAIo PSAo PSDo OUR IND UNI NAV", policyref="/w3c/p3p.xml" < Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT < Vary: Accept-Encoding < Keep-Alive: timeout=30, max=100 < Content-Type: text/html; charset=utf-8 < Content-Length: 14214 < Date: Fri, 01 Oct 2010 04:14:27 GMT < X-Varnish: 946925222 946924396 < Age: 81 < Via: 1.1 varnish * HTTP/1.0 connection set to keep alive! < Connection: keep-alive < X-Served-By: ntplax2.ntppool.net < X-Cache: HIT < X-Cache-Hits: 298 < { [data not shown] * Connection #0 to host www.pool.ntp.org left intact * Closing connection #0 IMS request ============== [ask at embla ~]$ curl -H 'If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT' -svo/dev/null http://www.pool.ntp.org/en/ * About to connect() to www.pool.ntp.org port 80 (#0) * Trying 207.171.3.6... connected * Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) > GET /en/ HTTP/1.1 > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 > Host: www.pool.ntp.org > Accept: */* > If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT > < HTTP/1.1 304 Not Modified < Date: Fri, 01 Oct 2010 04:14:31 GMT < Via: 1.1 varnish < X-Varnish: 946925256 < Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT < Vary: Accept-Encoding < Connection: keep-alive < X-Served-By: ntplax2.ntppool.net < X-Cache: HIT < X-Cache-Hits: 313 < * Connection #0 to host www.pool.ntp.org left intact * Closing connection #0 From indranilc at rediff-inc.com Fri Oct 1 05:40:27 2010 From: indranilc at rediff-inc.com (Indranil Chakravorty) Date: 1 Oct 2010 05:40:27 -0000 Subject: =?utf-8?B?UmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBtb2RpZmllZCBzaW5jZSIgYmV0d2VlbiBicm93c2VyIGFuZCB2YXJuaXNoIGNhY2hl?= In-Reply-To: Message-ID: <1285906505.S.3864.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.15628.pro-237-55.old.1285911627.2278@webmail.rediffmail.com> Thanks,  I am now trying to figure out how to get the date time stamp from an object in cache, so that I could use it to compare with the IMS header of the request from browser. Any idea/ pointers? Thanks. - Neel On Fri, 01 Oct 2010 09:45:05 +0530 Ask Bj?rn Hansen wrote > >On Sep 30, 2010, at 20:06, Indranil Chakravorty wrote: > >> I have been mulling over the feasibility of implementing conditional requests between the client browser and varnish cache. > >You are in luck - that already works. :-) > >And yes - all common browsers support this. You can combine a short-ish Cache-Control maxage header with a longer cache time in Varnish to get some control of how long content is cached for. > > > - ask > >First request (get Last-Modified for the next request) ============ > >[ask at embla ~]$ curl -svo/dev/null http://www.pool.ntp.org/en/ >* About to connect() to www.pool.ntp.org port 80 (#0) >* Trying 207.171.3.6... connected >* Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) >> GET /en/ HTTP/1.1 >> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 >> Host: www.pool.ntp.org >> Accept: */* >> >* HTTP 1.0, assume close after body >< HTTP/1.0 200 OK >< Server: Apache/2.2.3 (Red Hat) mod_apreq2-20051231/2.6.1 mod_perl/2.0.4 Perl/v5.8.8 >< X-UA-Compatible: IE=8 >< P3P: CP="NOI DEVo TAIo PSAo PSDo OUR IND UNI NAV", policyref="/w3c/p3p.xml" >< Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT >< Vary: Accept-Encoding >< Keep-Alive: timeout=30, max=100 >< Content-Type: text/html; charset=utf-8 >< Content-Length: 14214 >< Date: Fri, 01 Oct 2010 04:14:27 GMT >< X-Varnish: 946925222 946924396 >< Age: 81 >< Via: 1.1 varnish >* HTTP/1.0 connection set to keep alive! >< Connection: keep-alive >< X-Served-By: ntplax2.ntppool.net >< X-Cache: HIT >< X-Cache-Hits: 298 >< >{ [data not shown] >* Connection #0 to host www.pool.ntp.org left intact >* Closing connection #0 > > >IMS request ============== > > >[ask at embla ~]$ curl -H 'If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT' -svo/dev/null http://www.pool.ntp.org/en/ >* About to connect() to www.pool.ntp.org port 80 (#0) >* Trying 207.171.3.6... connected >* Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) >> GET /en/ HTTP/1.1 >> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 >> Host: www.pool.ntp.org >> Accept: */* >> If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT >> >< HTTP/1.1 304 Not Modified >< Date: Fri, 01 Oct 2010 04:14:31 GMT >< Via: 1.1 varnish >< X-Varnish: 946925256 >< Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT >< Vary: Accept-Encoding >< Connection: keep-alive >< X-Served-By: ntplax2.ntppool.net >< X-Cache: HIT >< X-Cache-Hits: 313 >< >* Connection #0 to host www.pool.ntp.org left intact >* Closing connection #0 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From indranilc at rediff-inc.com Fri Oct 1 05:40:27 2010 From: indranilc at rediff-inc.com (Indranil Chakravorty) Date: 1 Oct 2010 05:40:27 -0000 Subject: =?utf-8?B?UmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBtb2RpZmllZCBzaW5jZSIgYmV0d2VlbiBicm93c2VyIGFuZCB2YXJuaXNoIGNhY2hl?= In-Reply-To: Message-ID: <1285906505.S.3864.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.15628.pro-237-55.old.1285911627.48515@webmail.rediffmail.com> Thanks,  I am now trying to figure out how to get the date time stamp from an object in cache, so that I could use it to compare with the IMS header of the request from browser. Any idea/ pointers? Thanks. - Neel On Fri, 01 Oct 2010 09:45:05 +0530 Ask Bj?rn Hansen wrote > >On Sep 30, 2010, at 20:06, Indranil Chakravorty wrote: > >> I have been mulling over the feasibility of implementing conditional requests between the client browser and varnish cache. > >You are in luck - that already works. :-) > >And yes - all common browsers support this. You can combine a short-ish Cache-Control maxage header with a longer cache time in Varnish to get some control of how long content is cached for. > > > - ask > >First request (get Last-Modified for the next request) ============ > >[ask at embla ~]$ curl -svo/dev/null http://www.pool.ntp.org/en/ >* About to connect() to www.pool.ntp.org port 80 (#0) >* Trying 207.171.3.6... connected >* Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) >> GET /en/ HTTP/1.1 >> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 >> Host: www.pool.ntp.org >> Accept: */* >> >* HTTP 1.0, assume close after body >< HTTP/1.0 200 OK >< Server: Apache/2.2.3 (Red Hat) mod_apreq2-20051231/2.6.1 mod_perl/2.0.4 Perl/v5.8.8 >< X-UA-Compatible: IE=8 >< P3P: CP="NOI DEVo TAIo PSAo PSDo OUR IND UNI NAV", policyref="/w3c/p3p.xml" >< Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT >< Vary: Accept-Encoding >< Keep-Alive: timeout=30, max=100 >< Content-Type: text/html; charset=utf-8 >< Content-Length: 14214 >< Date: Fri, 01 Oct 2010 04:14:27 GMT >< X-Varnish: 946925222 946924396 >< Age: 81 >< Via: 1.1 varnish >* HTTP/1.0 connection set to keep alive! >< Connection: keep-alive >< X-Served-By: ntplax2.ntppool.net >< X-Cache: HIT >< X-Cache-Hits: 298 >< >{ [data not shown] >* Connection #0 to host www.pool.ntp.org left intact >* Closing connection #0 > > >IMS request ============== > > >[ask at embla ~]$ curl -H 'If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT' -svo/dev/null http://www.pool.ntp.org/en/ >* About to connect() to www.pool.ntp.org port 80 (#0) >* Trying 207.171.3.6... connected >* Connected to www.pool.ntp.org (207.171.3.6) port 80 (#0) >> GET /en/ HTTP/1.1 >> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 >> Host: www.pool.ntp.org >> Accept: */* >> If-Modified-Since: Fri, 01 Oct 2010 04:13:06 GMT >> >< HTTP/1.1 304 Not Modified >< Date: Fri, 01 Oct 2010 04:14:31 GMT >< Via: 1.1 varnish >< X-Varnish: 946925256 >< Last-Modified: Fri, 01 Oct 2010 04:13:06 GMT >< Vary: Accept-Encoding >< Connection: keep-alive >< X-Served-By: ntplax2.ntppool.net >< X-Cache: HIT >< X-Cache-Hits: 313 >< >* Connection #0 to host www.pool.ntp.org left intact >* Closing connection #0 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ask at develooper.com Fri Oct 1 07:44:50 2010 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 1 Oct 2010 00:44:50 -0700 Subject: Looking to implement "If modified since" between browser and varnish cache In-Reply-To: <1285906505.S.3864.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.15628.pro-237-55.old.1285911627.48515@webmail.rediffmail.com> References: <1285906505.S.3864.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.15628.pro-237-55.old.1285911627.48515@webmail.rediffmail.com> Message-ID: <288E8114-2485-4309-A1EC-7712DEA6024C@develooper.com> On Sep 30, 2010, at 22:40, Indranil Chakravorty wrote: > Thanks, > I am now trying to figure out how to get the date time stamp from an object in cache, so that I could use it to compare with the IMS header of the request from browser. Any idea/ pointers? Thanks. What are you trying to do? - ask From indranilc at rediff-inc.com Fri Oct 1 10:08:25 2010 From: indranilc at rediff-inc.com (Indranil Chakravorty) Date: 1 Oct 2010 10:08:25 -0000 Subject: =?utf-8?B?UmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBtb2RpZmllZCBzaW5jZSIgYmV0d2VlbiBicm93c2VyIGFuZCB2YXJuaXNoIGNhY2hl?= In-Reply-To: <288E8114-2485-4309-A1EC-7712DEA6024C@develooper.com> Message-ID: <1285919018.S.1783.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.59611.pro-237-175.old.1285927705.24346@webmail.rediffmail.com> If the browser sends a req for the same page again to the cache, then the "if modified since" header of the req is compared with the "date" of the object in cache. In case the cache object is not newer than the IMS header, then the cache returns 304, so that we avoid further data transfer between cache and browser. The browser in this case would simply read from its own cache.  I have come up with this solution. My basic idea is the following addition to vcl in sub vcl_hit().         if (req.http.If-Modified-Since && obj.http.Date) {                //if (obj.http.Date <= req.http.If-Modified-Since) {                //    error 304 "Not Modified";                //}        } I understand I can not use "<=", so I have gone ahead and used the corresponding inline C code to convert the http headers into string using VRT_GetHdr and strprintf/scanf funcs. However, now I am trying to figure out a way to implement the VCL error using inline C. Is that possible? Thanks. Thanks, Neel On Fri, 01 Oct 2010 13:13:38 +0530 Ask Bj?rn Hansen wrote > >On Sep 30, 2010, at 22:40, Indranil Chakravorty wrote: > >> Thanks, >> I am now trying to figure out how to get the date time stamp from an object in cache, so that I could use it to compare with the IMS header of the request from browser. Any idea/ pointers? Thanks. > >What are you trying to do? > > > - ask > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Fri Oct 1 12:24:00 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 1 Oct 2010 08:24:00 -0400 Subject: Retries after X amount of time Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C0659BF@TMG-EVS02.torstar.net> Define a different director with the same backends. Count restarts. Select a different backend in each incrementation of restart. -------------------------- Sent using BlackBerry 416 561 4871 ________________________________ From: varnish-dev-bounces at varnish-cache.org To: varnish-dev at varnish-cache.org Sent: Wed Jul 28 03:22:25 2010 Subject: Retries after X amount of time Hello all, Is it possible to have Varnish retry another backend if a certain backend doesn't response after a certain timeout when using a director? If I understand correctly, the .retries option in the random director is used to find a healthy director and doesn't account of a request to a certain director that died after a timeout. Is it a correct statement? Thanks, Eran -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Fri Oct 1 12:20:52 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 1 Oct 2010 08:20:52 -0400 Subject: Ignoring health checks in statistics Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C0659BE@TMG-EVS02.torstar.net> You can handle it in error, or use a tcp alive check from the LB. -------------------------- Sent using BlackBerry 416 561 4871 ----- Original Message ----- From: varnish-dev-bounces at varnish-cache.org To: varnish-dev at varnish-cache.org Sent: Tue Mar 16 14:56:22 2010 Subject: Ignoring health checks in statistics Hi, we have two Varnish instances and a load balancer in front. The load balancer periodically polls the varnishd's for a static file (lying also at all the backend servers) to check if varnishd is still working. It does that once per second. This completely messes up statistics like hit ratio because that static file is served from cache all the time, so currently our hit ratio is close to 99.9%. I would like to have counted this specific file neither as miss nor hit, but just serve it and ignore it otherwise. It doesn't even need to be logged. haproxy has an option monitor-net where you can specify that it does not need to log requests from that subnet. Does varnish offer a similar option? Or is there some cool VCL snippet I could add? Thanks in advance. Kind regards, -- Navteq (DE) GmbH Frank Gruellich Map24 Systems and Networks Duesseldorfer Strasse 40a 65760 Eschborn Germany Phone: +49 6196 77756-414 Fax: +49 6196 77756-100 HRB 46215, Local Court Frankfurt am Main Managing Directors: Thomas Golob, Hans Pieter Gieszen, Martin Robert Stockman USt-ID-No.: DE 197947163 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oogali at gmail.com Fri Oct 1 13:51:07 2010 From: oogali at gmail.com (Omachonu Ogali) Date: Fri, 1 Oct 2010 09:51:07 -0400 Subject: Test failure In-Reply-To: References: Message-ID: I guess a moderator just [auto-]approved this message (but not the 2nd message asking to ignore it)... Anyhow, 15 minutes after I sent this test failure, I saw that it failed because the netmask on my test box was incorrect -- I corrected the netmask, re-ran the test, and it succeeded. oo On Tue, Sep 21, 2010 at 7:30 PM, Omachonu Ogali wrote: > I received the following test failure during the build: > > ---- v1 ? FAIL CLI response 200 expected 300 > #### top ?macro def tmpdir=/tmp/vtc.13335.6b8b4567 > #### top ?macro def bad_ip=10.255.255.255 > # ? ?top ?TEST ././tests/c00003.vtc starting > # ? ?top ?TEST Check that we start if at least one listen address works > ## ? s1 ? Starting server > #### s1 ? macro def s1_addr=127.0.0.1 > #### s1 ? macro def s1_port=33848 > #### s1 ? macro def s1_sock=127.0.0.1:33848 > # ? ?s1 ? Listen on 127.0.0.1:33848 > ## ? s1 ? Started on 127.0.0.1:33848 > ## ? v1 ? Launch > ### ?v1 ? CMD: cd ../varnishd && ./varnishd -d -d -n > /tmp/vtc.13335.6b8b4567/v1 -p auto_restart=off -p > syslog_cli_traffic=off -a '127.0.0.1:0' -S > /tmp/vtc.13335.6b8b4567/v1/_S -M 127.0.0.1:47953 -P > /tmp/vtc.13335.6b8b4567/v1/varnishd.pid > -sfile,/tmp/vtc.13335.6b8b4567/v1,10M > ### ?v1 ? debug| storage_file: filename: > /tmp/vtc.13335.6b8b4567/v1/varnish.v5O0ho size 10 MB.\n > ### ?v1 ? debug| Creating new SHMFILE\n > ### ?v1 ? debug| Varnish on Linux,2.6.18-194.el5,x86_64,-sfile,-hcritbit\n > ### ?v1 ? debug| 200 193 ? ? \n > ### ?v1 ? debug| -----------------------------\n > ### ?v1 ? debug| Varnish HTTP accelerator CLI.\n > ### ?v1 ? debug| -----------------------------\n > ### ?v1 ? debug| Type 'help' for command list.\n > ### ?v1 ? debug| Type 'quit' to close CLI session.\n > ### ?v1 ? debug| Type 'start' to launch worker process.\n > ### ?v1 ? debug| \n > ### ?v1 ? CLI connection fd = 6 > ### ?v1 ? CLI RX ?107 > #### v1 ? CLI RX| vkqhwykqxioyyfcbhxrtjeupeydufenb\n > #### v1 ? CLI RX| \n > #### v1 ? CLI RX| Authentication required.\n > #### v1 ? CLI TX| auth > 2cdbd91ae8754abdc65434d020632ce6a50c54e9646160b56274e81b62f72592\n > ### ?v1 ? CLI RX ?200 > #### v1 ? CLI RX| -----------------------------\n > #### v1 ? CLI RX| Varnish HTTP accelerator CLI.\n > #### v1 ? CLI RX| -----------------------------\n > #### v1 ? CLI RX| Type 'help' for command list.\n > #### v1 ? CLI RX| Type 'quit' to close CLI session.\n > #### v1 ? CLI RX| Type 'start' to launch worker process.\n > #### v1 ? CLI TX| param.set listen_address 10.255.255.255:0 > ### ?v1 ? CLI RX ?200 > ## ? v1 ? CLI 200 > #### v1 ? CLI TX| vcl.inline vcl1 "backend s1 { .host = \"127.0.0.1\"; > .port = \"33848\"; }\n" > ### ?v1 ? CLI RX ?200 > #### v1 ? CLI RX| VCL compiled. > #### v1 ? CLI TX| vcl.use vcl1 > ### ?v1 ? CLI RX ?200 > #### v1 ? CLI TX| start > ### ?v1 ? debug| child (13380) Started\n > ### ?v1 ? CLI RX ?200 > ## ? v1 ? CLI 200 > ---- v1 ? FAIL CLI response 200 expected 300 > ### ?v1 ? debug| Child (13380) said \n > ### ?v1 ? debug| Child (13380) said Child starts\n > ### ?v1 ? debug| Child (13380) said managed to mmap 10485760 bytes of 10485760\n > # ? ?top ?Test timed out > # ? ?top ?TEST ././tests/c00003.vtc FAILED > > > If this helps out at all regarding my environment... > > # uname -a > Linux box1.testbench 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > > # gcc --version > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) > Copyright (C) 2006 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. ?There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > # rpm -qa | grep glibc > glibc-headers-2.5-49.el5_5.4 > glibc-common-2.5-49.el5_5.4 > glibc-devel-2.5-49.el5_5.4 > glibc-2.5-49.el5_5.4 > glibc-2.5-49.el5_5.4 > > > Let me know if you need any other information from my side. > > oo > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-dev > From ask at develooper.com Fri Oct 1 22:29:54 2010 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 1 Oct 2010 15:29:54 -0700 Subject: Looking to implement "If modified since" between browser and varnish cache In-Reply-To: <1285919018.S.1783.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.59611.pro-237-175.old.1285927705.24346@webmail.rediffmail.com> References: <1285919018.S.1783.H.TkFzayBCavhybiBIYW5zZW4AUmU6IExvb2tpbmcgdG8gaW1wbGVtZW50ICJJZiBt.59611.pro-237-175.old.1285927705.24346@webmail.rediffmail.com> Message-ID: <37A1C250-D414-48D8-A530-6DF75A33B2E4@develooper.com> On Oct 1, 2010, at 3:08, Indranil Chakravorty wrote: > If the browser sends a req for the same page again to the cache, then the "if modified since" header of the req is compared with the "date" of the object in cache. Did you read my first reply at all? In my relatively standard configuration this seems to work fine by default. Does it not work for you? If so, send your configuration and log snippets and the list will help you sort that out. - ask From tfheen at varnish-software.com Mon Oct 4 08:47:23 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 04 Oct 2010 10:47:23 +0200 Subject: setting obj.prefetch In-Reply-To: (Eran Sandler's message of "Tue, 13 Apr 2010 11:57:47 +0300") References: Message-ID: <87vd5icpk4.fsf@qurzaw.linpro.no> ]] Eran Sandler | Prior to Varnish 2.1 I was able to set the obj.prefetch in the vcl_fetch | function so that for different items being fetched I have a different | prefetch time. | I'm not sure if that was the correct place for Varnish prior to 2.1 but it | worked. obj.prefetch has never actually done anything, so just remove the setting of it. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From indranilc at rediff-inc.com Mon Oct 4 13:13:29 2010 From: indranilc at rediff-inc.com (Indranil Chakravorty) Date: 4 Oct 2010 13:13:29 -0000 Subject: =?utf-8?B?RXJyb3IgY29tcGlsaW5nIFZDTCB1c2luZyB2YXJuaXNoLTIuMS4yIChSZWdleHAgY29tcGlsYXRpb24gZXJyb3I6ICBub3RoaW5nIHRvIHJlcGVhdCk=?= Message-ID: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> Hi,   I am getting the following errro while compiling my vcl file. Would anyone be able to help me out? Thanks. Message from VCC-compiler:Regexp compilation error: nothing to repeat (input Line 497 Pos 69)                if (req.url ~ "^/location-staticmap/*" || req.url ~ "^/staticmap%3F*" || req.url ~ "^/wms/*") {--------------------------------------------------------------------#################-------------------------- The line is copied below, position 69 happens to be the third double quote (") in the line, the one which is at the beginning of  staticmap.         if (req.http.host ~ "^example.sample.com") {                if (req.url ~ "^/location-staticmap/*" || req.url ~ "^/staticmap%3F*" || req.url ~ "^/wms/*") {                        set obj.http.Cache-Control = "max-age=432000";                        set obj.ttl = 5d;                }                return(deliver);        } Thanks, Neel -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 4 13:18:38 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 Oct 2010 13:18:38 +0000 Subject: Proposal/specs for backend conditional requests / aka "GET If-Modified-Since" (GET IMS)) In-Reply-To: Your message of "Thu, 23 Sep 2010 12:28:14 +0200." <4C9B2BBE.5060309@schokola.de> Message-ID: <37265.1286198318@critter.freebsd.dk> I had hoped to ponder this on the train from Frankfurt to Hamburg saturday but got sidetracked by other issues and code, so bear in mind that this is not thought out as much as I would like. First issue: If-Modified-Since requests to backend. ---------------------------------------------------- I don't see any benefit from adding two new vcl-subs for this, in my world it should work the following way: Cache lookup. Find valid in-ttl object -> vcl_hit Find valid in-grace object -> vcl_miss (with obj.*) Find nothing usable -> vcl_miss (without obj.*) Before vcl_miss is called, we prepare the bereq.* as today but if we have a graced object, we make it an If-Modified-Since request by default. (... unless parameter (per backend flag ?) tells us not to). If people don't want to IMS certain objects, they remove the IMS header in vcl_miss{}, and things automatically DTRT. If we get a "304 Not Modified" back, we always instantiate a new objcore+obj and if the stevedore insists (-spersistent) we will copy the body as well. We change the beresp.status to 200 to hide the IMS action from vcl_fetch{} At this point we let go of the ref on the graced object as we have the new one fully usable at this point. And apart from all the nitty gritty, that is it as far as I can tell. Second issue: deliver graced content under VCL control ------------------------------------------------------ Now that we have the graced object in vcl_miss{}, we might as well allow return(deliver) on it, should people desire to do so, and one obvious case would be: sub vcl_miss { if (!req.backend.healty && obj.status) { return (deliver); } } Third issue: how does that work with saint mode ----------------------------------------------- Saint mode consists of two things: A) Per backend list of unhealty URL's which affect backend health status. Forget about this, it's not relevant here. B) Ability to deliver graced content if backend fetch fails. Since we have extended the graced-obj ref into vcl_miss{} above, it is not a long stretch to hold unto it until vcl_fetch{} also, and avoid the currently necessary restart in order to return graced content on backend failure. In order to not confuse the syntax more than necessary, we should probably introduce "return (deliver_graced)" both in vcl_miss{} and vcl_fetch{}. Comments ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon Oct 4 13:34:20 2010 From: slink at schokola.de (Nils Goroll) Date: Mon, 04 Oct 2010 15:34:20 +0200 Subject: Elaborating on proposed vcl_stale() and vcl_refresh() // Proposal/specs for backend conditional requests / aka "GET If-Modified-Since" (GET IMS)) In-Reply-To: <4C9B2BBE.5060309@schokola.de> References: <4C9B2BBE.5060309@schokola.de> Message-ID: <4CA9D7DC.1070000@schokola.de> Hi, I've had a first discussion with phk on IRC. IIUC, his main concern was that introducing the new states & VCL procedures would increase complexity and raise the entry barrier, in particular with regard to users wanting to (easily) customize the default vcl. I will elaborate on the reasons for the proposed solution. a) vcl_stale() -------------- The alternative we've discussed on IRC is to fall into vcl_miss(): If an obj eligible for refresh was found, default backend conditional request headers were added to bereq, and vcl_miss() would see obj. It wouldn't for the current case (a real miss). To avoid a backend conditional request, admins could remove the conditional headers (If-...) from bereq. return(refresh) wouldn't exist. Here's my subjective perspective on why vcl_stale() would make sense: - Without vcl_stale(), the semantics of vcl_miss() would mean "I have not found an object eligible for delivery, but might have found one for refresh", which I believe will confuse users more than help. Mostly, this does open up a whole lot of oddities and questions: - vcl_miss would see obj only if it was a "stale miss" Odd: What would obj.status return for a "real miss"? - cnt_miss could prepare bereq.http.If-... headers only if it was a "stale miss" - We would be able to return (deliver) for a "stale miss", but not for a real miss: What do we do here? We can't decide on the correctness of the VCL at compile time. Would return (deliver) fall back to a return (fetch) is it's a "real miss"? - Users would need to put more conditionals into vcl_miss for special handling of "stale" cases, ? la: if (obj) { # do stale case handling } else { # existing code } Depending on whether we want to default to the new or current behavior, a default vcl_stale() would look like: vcl_stale() { return (refresh); } or vcl_stale() { call vcl_miss; } If we put the mechanics into vcl_miss, retaining current behavior would need something like this with a full backend conditional: vcl_miss() { if (obj) { unset bereq.http.If-Match; unset bereq.http.If-Modified-Since; unset bereq.http.If-None-Match; unset bereq.http.If-Range; unset bereq.http.If-Unmodified-Since; } } My personal impression is that vcl_stale would help retain clarity rather than complicate things, in particular because it would help to limit additional processing to the cases when it's actually needed. b) vcl_refresh() ---------------- The alternative to vcl_refresh() would be to put additional logic into vcl_fetch(). Similar to the stale case, we would want to have access to the original object in the cache and the beresp from the refresh, so obj would be available only if vcl_refresh was entered with (beresp.status == 304). We really want to retain clarity and not introduce unneeded additional complexity. I believe that introducing more VCL states and procedures would help to achieve this, in particular because the preconditions for each case were clear and visibility of a cache object was well defined. Thanks, Nils From slink at schokola.de Mon Oct 4 13:59:07 2010 From: slink at schokola.de (Nils Goroll) Date: Mon, 04 Oct 2010 15:59:07 +0200 Subject: Proposal/specs for backend conditional requests / aka "GET If-Modified-Since" (GET IMS)) In-Reply-To: <37265.1286198318@critter.freebsd.dk> References: <37265.1286198318@critter.freebsd.dk> Message-ID: <4CA9DDAB.9040402@schokola.de> Hi phk, first of all, thank you for taking the time to respond. I agree with what you wrote about the basic logic. I could elaborate on some of the details you mentioned, but that would mostly repeat what I've already posted. What it really boils down to, IMHO, is that it can be done both ways. I've laid out why I'd prefer the two new states, but if you decide you don't want them after consideration of the reasons we've brought up, that's fine. I can live with decisions which are suboptimal from my point of view. Does anyone else have an opinion on this? Thanks, Nils From slink at schokola.de Mon Oct 4 14:09:43 2010 From: slink at schokola.de (Nils Goroll) Date: Mon, 04 Oct 2010 16:09:43 +0200 Subject: Error compiling VCL using varnish-2.1.2 (Regexp compilation error: nothing to repeat) In-Reply-To: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> References: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> Message-ID: <4CA9E027.1040302@schokola.de> > if (req.url ~ "^/location-staticmap/*" || req.url ~ > "^/staticmap%3F*" || req.url ~ "^/wms/*") { What happens if you put your matches in additional parenthesis? Also, you don't need the asterisks. E.g. in the first regexp, this means "end with and number of slashes or none at all". Nils From slink at schokola.de Mon Oct 4 14:24:21 2010 From: slink at schokola.de (Nils Goroll) Date: Mon, 04 Oct 2010 16:24:21 +0200 Subject: Error compiling VCL using varnish-2.1.2 (Regexp compilation error: nothing to repeat) In-Reply-To: <4CA9E027.1040302@schokola.de> References: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> <4CA9E027.1040302@schokola.de> Message-ID: <4CA9E395.1000505@schokola.de> > Also, you don't need the asterisks. E.g. in the first regexp, this means "end > with and number of slashes or none at all". any From sky at crucially.net Mon Oct 4 20:34:29 2010 From: sky at crucially.net (Artur Bergman) Date: Mon, 4 Oct 2010 13:34:29 -0700 Subject: Ignoring health checks in statistics In-Reply-To: <4B9FD456.7070502@navteq.com> References: <4B9FD456.7070502@navteq.com> Message-ID: <3FFFB0B8-6836-4260-AE4A-CAEB79785ABD@crucially.net> Just use vcl_error Artur On Mar 16, 2010, at 11:56 AM, Frank Gruellich wrote: > Hi, > > we have two Varnish instances and a load balancer in front. The load > balancer periodically polls the varnishd's for a static file (lying > also > at all the backend servers) to check if varnishd is still working. It > does that once per second. This completely messes up statistics like > hit ratio because that static file is served from cache all the > time, so > currently our hit ratio is close to 99.9%. I would like to have > counted > this specific file neither as miss nor hit, but just serve it and > ignore > it otherwise. It doesn't even need to be logged. > > haproxy has an option monitor-net where you can specify that it does > not > need to log requests from that subnet. Does varnish offer a similar > option? Or is there some cool VCL snippet I could add? > > Thanks in advance. > > Kind regards, > -- > Navteq (DE) GmbH > Frank Gruellich > Map24 Systems and Networks > > Duesseldorfer Strasse 40a > 65760 Eschborn > Germany > > Phone: +49 6196 77756-414 > Fax: +49 6196 77756-100 > > HRB 46215, Local Court Frankfurt am Main > Managing Directors: Thomas Golob, Hans Pieter Gieszen, Martin Robert > Stockman > USt-ID-No.: DE 197947163 > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-dev From tfheen at varnish-software.com Tue Oct 5 06:33:48 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 05 Oct 2010 08:33:48 +0200 Subject: Error compiling VCL using varnish-2.1.2 (Regexp compilation error: nothing to repeat) In-Reply-To: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> (Indranil Chakravorty's message of "4 Oct 2010 13:13:29 -0000") References: <20101004131329.47879.qmail@pro-147-138.rediffmailpro.com> Message-ID: <87aamtcfn7.fsf@qurzaw.linpro.no> Hi, as this doesn't seem relevant to Varnish development, would you mind asking user questions on the -misc list, please? Also, it would be appreciated if you could get your email client to not send HTML escapes in the plain text part. Thanks! -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Tue Oct 5 06:38:42 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 05 Oct 2010 08:38:42 +0200 Subject: Proposal/specs for backend conditional requests / aka "GET If-Modified-Since" (GET IMS)) In-Reply-To: <37265.1286198318@critter.freebsd.dk> (Poul-Henning Kamp's message of "Mon, 04 Oct 2010 13:18:38 +0000") References: <37265.1286198318@critter.freebsd.dk> Message-ID: <8762xhcff1.fsf@qurzaw.linpro.no> ]] "Poul-Henning Kamp" | Cache lookup. | Find valid in-ttl object -> vcl_hit | Find valid in-grace object -> vcl_miss (with obj.*) | Find nothing usable -> vcl_miss (without obj.*) How would this look in the VCL and VCC? Do we have a skeleton obj.* that's just marked as invalid in the case where we don't have anything usable? | Comments ? Sounds reasonable to me. I'm sure there are details and snags and all that, but the high-level picture looks fairly reasonable. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From sky at crucially.net Thu Oct 7 18:19:15 2010 From: sky at crucially.net (Artur Bergman) Date: Thu, 7 Oct 2010 11:19:15 -0700 Subject: Quorum support Message-ID: <56972B68-A2F6-43F9-A02D-60ACC1C51C63@crucially.net> Attached is a patch that implements quorum support. You specify the % of servers that need to be up, and if less than that is up it marks the entire director as down. This avoids the whack-a-mole problem when bringing up servers. Comments please. Artur ---------- diff -u -r varnish-2.1.3/bin/varnishd/cache_dir_random.c varnish- quorum/bin/varnishd/cache_dir_random.c --- varnish-2.1.3/bin/varnishd/cache_dir_random.c 2010-03-24 09:44:13.000000000 +0000 +++ varnish-quorum/bin/varnishd/cache_dir_random.c 2010-10-07 18:07:37.126411696 +0000 @@ -75,6 +75,7 @@ enum crit_e criteria; unsigned retries; + unsigned quorum_weight; double tot_weight; struct vdi_random_host *hosts; unsigned nhosts; @@ -155,7 +156,9 @@ if (VBE_Healthy_sp(sp, d2)) s1 += vs->hosts[i].weight; } - + if (s1 < vs->quorum_weight) + return (NULL); + if (s1 == 0.0) return (NULL); @@ -258,6 +261,7 @@ vh->backend = bp[te->host]; AN(vh->backend); } + vs->quorum_weight = (vs->tot_weight * t->quorum / 100); vs->nhosts = t->nmember; bp[idx] = &vs->dir; } diff -u -r varnish-2.1.3/include/vrt.h varnish-quorum/include/vrt.h --- varnish-2.1.3/include/vrt.h 2010-07-13 11:07:03.000000000 +0000 +++ varnish-quorum/include/vrt.h 2010-10-07 17:42:13.476408083 +0000 @@ -91,6 +91,7 @@ const char *name; unsigned retries; unsigned nmember; + unsigned quorum; const struct vrt_dir_random_entry *members; }; diff -u -r varnish-2.1.3/lib/libvcl/vcc_dir_random.c varnish-quorum/ lib/libvcl/vcc_dir_random.c --- varnish-2.1.3/lib/libvcl/vcc_dir_random.c 2010-07-28 11:11:58.000000000 +0000 +++ varnish-quorum/lib/libvcl/vcc_dir_random.c 2010-10-07 18:01:23.526408498 +0000 @@ -55,13 +55,14 @@ struct token *t_field, *t_be; int nelem; struct fld_spec *fs, *mfs; - unsigned u, retries; + unsigned u, retries, quorum; const char *first; char *p; - fs = vcc_FldSpec(tl, "?retries", NULL); + fs = vcc_FldSpec(tl, "?retries","?quorum", NULL); retries = 0; + quorum = 0; while (tl->t->tok != '{') { vcc_IsField(tl, &t_field, fs); ERRCHK(tl); @@ -70,6 +71,11 @@ retries = vcc_UintVal(tl); ERRCHK(tl); SkipToken(tl, ';'); + } else if (vcc_IdIs(t_field, "quorum")) { + ExpectErr(tl, CNUM); + quorum = vcc_UintVal(tl); + ERRCHK(tl); + SkipToken(tl, ';'); } else { ErrInternal(tl); } @@ -134,6 +140,7 @@ PF(tl->t_dir)); Fc(tl, 0, "\t.name = \"%.*s\",\n", PF(tl->t_dir)); Fc(tl, 0, "\t.retries = %u,\n", retries); + Fc(tl, 0, "\t.quorum = %u,\n", quorum); Fc(tl, 0, "\t.nmember = %d,\n", nelem); Fc(tl, 0, "\t.members = vdre_%.*s,\n", PF(tl->t_dir)); Fc(tl, 0, "};\n"); diff -u -r varnish-2.1.3/lib/libvcl/vcc_fixed_token.c varnish-quorum/ lib/libvcl/vcc_fixed_token.c --- varnish-2.1.3/lib/libvcl/vcc_fixed_token.c 2010-07-28 09:48:29.000000000 +0000 +++ varnish-quorum/lib/libvcl/vcc_fixed_token.c 2010-10-07 17:42:14.888909071 +0000 @@ -258,8 +258,9 @@ vsb_cat(sb, "\tint\t\t\t\t\thost;\n\tdouble\t\t\t\t\tweight;\n"); vsb_cat(sb, "};\n\nstruct vrt_dir_random {\n"); vsb_cat(sb, "\tconst char\t\t\t\t*name;\n\tunsigned\t\t\t \tretries;"); - vsb_cat(sb, "\n\tunsigned\t\t\t\tnmember;\n\tconst struct vrt_dir_r"); - vsb_cat(sb, "andom_entry\t*members;\n};\n\n/*\n"); + vsb_cat(sb, "\n\tunsigned\t\t\t\tnmember;\n\tunsigned "); + vsb_cat(sb, " quorum;\n\tconst struct vrt_dir_rand"); + vsb_cat(sb, "om_entry\t*members;\n};\n\n/*\n"); vsb_cat(sb, " * A director with round robin selection\n"); vsb_cat(sb, " */\n\nstruct vrt_dir_round_robin_entry {\n"); vsb_cat(sb, "\tint\t\t\t\t\thost;\n};\n\nstruct vrt_dir_round_robin"); -------------- next part -------------- An HTML attachment was scrubbed... URL: From andru.garman at gmail.com Thu Oct 7 22:18:52 2010 From: andru.garman at gmail.com (Andrew Garman) Date: Thu, 7 Oct 2010 18:18:52 -0400 Subject: Fwd: gzip & esi In-Reply-To: References: Message-ID: Varnish developers, If I add gzip support to varnish deliver phase, what steps would I have to go through to get it included in the main branch? Thanks, Andrew From martin at varnish-software.com Fri Oct 15 11:24:03 2010 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 15 Oct 2010 13:24:03 +0200 Subject: Request for code review of varnish-cache.org ticket #780 (Read CRLF in fetch_chunked) Message-ID: Hi, Attached is code proposed to resolve ticket #780 on vanish-cache.org. This ticket is about the final CRLF in a chunked fetch not being read by varnish. I have added the code to read 2 more bytes after finishing with the chunks and test that these are in fact CRLF. If they are not, it returns a fetch failed (-1). I was a bit puzzled that this behaviour could exist for so long in varnish without it being noticed. The reason for that is probably that in most cases this would still work as the code in http_splitline skips over leading CRLF when looking at the next response on a reused backend. This would skip the lingering CRLF in the byte stream at that moment. Because of the reading of CRLF in http_splitline, I have been unable to create a test case that would reliable fail without the fix. There was however a number of test cases that produced non-compliant chunked encoding data that would fail with the fix applied. (These tests were not adding the mandatory final CRLF to the byte stream). I have updated these test cases in the patch to add a final CRLF. Discussion of trailing headers: With this patch, if a backend was to send us trailing headers the connection will fail. Without the patch, I would expect the next connection to fail instead, as the http response line processing would not match. I think that is worse behaviour than failing on the connection that has the traling headers. The patch could be extended to skip over trailing headers to ignore them but keep the connection reusable and don't fail the fetch, but I am not sure if it is worth the bother as noone is actually using trailing headers. Regards, Martin Blix Grydeland The patch: Index: varnish-cache/bin/varnishtest/tests/r00776.vtc =================================================================== --- varnish-cache/bin/varnishtest/tests/r00776.vtc (revision 5430) +++ varnish-cache/bin/varnishtest/tests/r00776.vtc (working copy) @@ -6,6 +6,7 @@ rxreq txresp -nolen -hdr "Transfer-encoding: chunked" chunkedlen 4096 + send "\r\n" } -start varnish v1 \ Index: varnish-cache/bin/varnishtest/tests/r00387.vtc =================================================================== --- varnish-cache/bin/varnishtest/tests/r00387.vtc (revision 5430) +++ varnish-cache/bin/varnishtest/tests/r00387.vtc (working copy) @@ -10,6 +10,7 @@ send "004\r\n1234\r\n" send "000000000000000000001\r\n@\r\n" send "00000000\r\n" + send "\r\n" } -start varnish v1 -vcl+backend {} -start Index: varnish-cache/bin/varnishtest/tests/r00694.vtc =================================================================== --- varnish-cache/bin/varnishtest/tests/r00694.vtc (revision 5430) +++ varnish-cache/bin/varnishtest/tests/r00694.vtc (working copy) @@ -9,6 +9,7 @@ send "\r\n" # This is chunksize (128k) + 2 to force to chunks to be allocated chunkedlen 131074 + send "\r\n" } -start varnish v1 -vcl+backend { Index: varnish-cache/bin/varnishtest/tests/b00007.vtc =================================================================== --- varnish-cache/bin/varnishtest/tests/b00007.vtc (revision 5430) +++ varnish-cache/bin/varnishtest/tests/b00007.vtc (working copy) @@ -10,6 +10,7 @@ send "\r\n" send "00000004\r\n1234\r\n" send "00000000\r\n" + send "\r\n" rxreq expect req.url == "/foo" @@ -19,6 +20,7 @@ send "00000004\r\n1234\r\n" chunked "1234" chunked "" + send "\r\n" } -start varnish v1 -vcl+backend {} -start Index: varnish-cache/bin/varnishd/cache_fetch.c =================================================================== --- varnish-cache/bin/varnishd/cache_fetch.c (revision 5430) +++ varnish-cache/bin/varnishd/cache_fetch.c (working copy) @@ -207,6 +207,11 @@ q = bp = buf + v; } + /* Expect a CRLF to trail the chunks */ + i = HTC_Read(htc, buf, 2); + if (i != 2 || buf[0] != '\r' || buf[1] != '\n') + return -1; + if (st != NULL && st->len == 0) { VTAILQ_REMOVE(&sp->obj->store, st, list); STV_free(st); -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 18 14:52:55 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 18 Oct 2010 14:52:55 +0000 Subject: Request for code review of varnish-cache.org ticket #780 (Read CRLF in fetch_chunked) In-Reply-To: Your message of "Fri, 15 Oct 2010 13:24:03 +0200." Message-ID: <6535.1287413575@critter.freebsd.dk> In message , Mart in Blix Grydeland writes: >Attached is code proposed to resolve ticket #780 on vanish-cache.org. Looks good, but I'm worried if it is too strict. RFC2616 p166 "19.3 Tolerant Applications" suggest that we don't mandate exact CR+LF, but rather: The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR. So I think the right logic is more like: c = Rx one char if (c == CR) Rx one char if (c != LF) accept fetch log error notice close connection to prevent reuse -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From hefengcui at gmail.com Fri Oct 22 05:34:14 2010 From: hefengcui at gmail.com (hefeng cui) Date: Fri, 22 Oct 2010 13:34:14 +0800 Subject: How to add more information in varnishncsa log entry? Message-ID: Hi, varnish dev team! I am a new user of varnish and my customer and my dev team are really like Varnish Cache! Great job! I have a task to do some analytics stuff based on varnish log information. We want to add few fields in log entry so that a meaningful report can be generated. 1. How long a request was served (start from receive the request and end at before sending back the response ) 2. Cache side action information : Hit Miss or Refresh etc 3. MIME-Type client side requested. I am looking at varnishncsa.c and varnishlog.c and want to customize current logging a little bit to do that. Since this is quite urgent task for me, could any of you point me a direction how to do it ? Any help is highly appreciated! Best regard, Hefeng -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Sat Oct 23 14:43:06 2010 From: slink at schokola.de (Nils Goroll) Date: Sat, 23 Oct 2010 16:43:06 +0200 Subject: [patch] use pthread_timedjoin_np() only where it exists Message-ID: <4CC2F47A.3010404@schokola.de> r5452 breaks compilation for systems which don't have pthread_timedjoin_np(). This patch brings back the old code for such systems with one modification: pthread_join() is only called if pthread_cancel() does not return an error. Also, it changes exec_file_thread to exit with pthread_exit rather than return(). With this patch and http://www.varnish-cache.org/trac/ticket/663, make check succeeds without error on Opensolaris snv_134, except for m00001.vtc, which did already fail before for another reason I haven't yet understood (vmod doesn't get unloaded after vcl destroy). I haven't tested it on other platforms. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: varnish_pthread_timedjoin_np.patch URL: From martin at varnish-software.com Mon Oct 25 12:27:15 2010 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 25 Oct 2010 14:27:15 +0200 Subject: Fwd: Request for code review of varnish-cache.org ticket #780 (Read CRLF in fetch_chunked) In-Reply-To: References: <6535.1287413575@critter.freebsd.dk> Message-ID: Forgot to add varnish-dev on this. -Martin ---------- Forwarded message ---------- From: Martin Blix Grydeland Date: Mon, Oct 25, 2010 at 14:21 Subject: Re: Request for code review of varnish-cache.org ticket #780 (Read CRLF in fetch_chunked) To: Poul-Henning Kamp On Mon, Oct 18, 2010 at 16:52, Poul-Henning Kamp wrote: > So I think the right logic is more like: > > c = Rx one char > if (c == CR) > Rx one char > if (c != LF) > accept fetch > log error notice > close connection to prevent reuse > > Second go at this. I've kept the changes to the test cases for them to be RFC compliant, and implemented your pseudo-code. I leave the closing of the connection to the calling code, but return a value of 1 to indicate the connection is not safe to reuse. Martin Index: bin/varnishtest/tests/r00776.vtc =================================================================== --- bin/varnishtest/tests/r00776.vtc (revision 5463) +++ bin/varnishtest/tests/r00776.vtc (working copy) @@ -6,6 +6,7 @@ rxreq txresp -nolen -hdr "Transfer-encoding: chunked" chunkedlen 4096 + send "\r\n" } -start varnish v1 \ Index: bin/varnishtest/tests/r00387.vtc =================================================================== --- bin/varnishtest/tests/r00387.vtc (revision 5463) +++ bin/varnishtest/tests/r00387.vtc (working copy) @@ -10,6 +10,7 @@ send "004\r\n1234\r\n" send "000000000000000000001\r\n@\r\n" send "00000000\r\n" + send "\r\n" } -start varnish v1 -vcl+backend {} -start Index: bin/varnishtest/tests/r00694.vtc =================================================================== --- bin/varnishtest/tests/r00694.vtc (revision 5463) +++ bin/varnishtest/tests/r00694.vtc (working copy) @@ -9,6 +9,7 @@ send "\r\n" # This is chunksize (128k) + 2 to force to chunks to be allocated chunkedlen 131074 + send "\r\n" } -start varnish v1 -vcl+backend { Index: bin/varnishtest/tests/b00007.vtc =================================================================== --- bin/varnishtest/tests/b00007.vtc (revision 5463) +++ bin/varnishtest/tests/b00007.vtc (working copy) @@ -10,6 +10,7 @@ send "\r\n" send "00000004\r\n1234\r\n" send "00000000\r\n" + send "\r\n" rxreq expect req.url == "/foo" @@ -19,6 +20,7 @@ send "00000004\r\n1234\r\n" chunked "1234" chunked "" + send "\r\n" } -start varnish v1 -vcl+backend {} -start Index: bin/varnishd/cache_fetch.c =================================================================== --- bin/varnishd/cache_fetch.c (revision 5463) +++ bin/varnishd/cache_fetch.c (working copy) @@ -207,6 +207,16 @@ q = bp = buf + v; } + /* Expect a CRLF to trail the chunks */ + i = HTC_Read(htc, buf, 1); + if (i == 1 && buf[0] == '\r') + i = HTC_Read(htc, buf, 1); + if (i != 1 || (i==1 && buf[0] != '\n')) { + WSP(sp, SLT_FetchError, + "chunked missing trailing crlf"); + return 1; /* Accept fetch, but do not reuse connection */ + } + if (st != NULL && st->len == 0) { VTAILQ_REMOVE(&sp->obj->store, st, list); STV_free(st); -- Martin Blix Grydeland Varnish Software AS -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Oct 26 07:47:19 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 26 Oct 2010 09:47:19 +0200 Subject: r00663 tests failing Message-ID: <87iq0pfla0.fsf@qurzaw.linpro.no> Hi Nils, it seems like the r00663 test fails on quite a few of the build systems we have: See http://buildbot.varnish-cache.org/waterfall, particularly http://buildbot.varnish-cache.org/builders/buildbot-debian-ppc/builds/1357/steps/test/logs/stdio Any chance you could have a look at this? -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From slink at schokola.de Thu Oct 28 11:23:52 2010 From: slink at schokola.de (Nils Goroll) Date: Thu, 28 Oct 2010 13:23:52 +0200 Subject: r00663 tests failing In-Reply-To: <87iq0pfla0.fsf@qurzaw.linpro.no> References: <87iq0pfla0.fsf@qurzaw.linpro.no> Message-ID: <4CC95D48.1000207@schokola.de> Hi Tollef and all, apologies that this test was still contained in the patch. I had discussed this with phk a long time ago already and we had agreed that it is unnecessary due to the test phk had added: http://www.varnish-cache.org/trac/changeset/4569 Thank you, Nils From slink at schokola.de Thu Oct 28 11:24:15 2010 From: slink at schokola.de (Nils Goroll) Date: Thu, 28 Oct 2010 13:24:15 +0200 Subject: [patch] use pthread_timedjoin_np() only where it exists In-Reply-To: <4CC2F47A.3010404@schokola.de> References: <4CC2F47A.3010404@schokola.de> Message-ID: <4CC95D5F.7020007@schokola.de> Any feedback on this? From slink at schokola.de Thu Oct 28 12:34:00 2010 From: slink at schokola.de (Nils Goroll) Date: Thu, 28 Oct 2010 14:34:00 +0200 Subject: [patch] vmod_digest (message digests and encoders for VCL) Message-ID: <4CC96DB8.202@schokola.de> Hi, I have now finished a first version of vmod_digest. From the .rst file: --- 8< --- SYNOPSIS ======== import digest; digest.DIGEST_ENCODER(STRING) digest.encode_ENCODER(STRING) Common use cases: digest.sha256_hexlc(STRING) digest.md5_hexlc(STRING) digest.encode_base64(STRING) DESCRIPTION =========== The digest module makes message digest (hashing) functions available for use from VCLs. As a side-product, it also provides various string encoders. The provided functions are named according to the scheme given in the synopsis. --- 8< --- I'd appreciate reviews, constructive criticism and suggestions for improvements. Implementation notes: * Byte order for sha256 and md5 is little endian. I wanted to verify the implementation on a SPARC machine but had to realize that at this point varnish does not at all work on this platform, probably due to memory alignment errors. I will try to find time to look into this, make varnish work on SPARC and check the implementation as a side effect. I can't give a time frame for this. * I've added a base64 encoder and also modified the base64 decoder by phk to return the correct length of the decoded byte stream. base64_encode went into libvarnishapi because I thought it would be a good idea to have it next to base64_decode * the crc32 code is phk's implementation originally used for hashing of cache objects in Varnish, md5 is from FreeBSD At this point, only vmod_digests uses crc32 and md5, but should other components need those as well, it should be easy to move the code (into something like libvmd) * All of the VCC glue code is generated by a simplistic python script which evolved from vcc.py. I wanted to retain vcc.py, so this script generates imput to vcc.py. * The patch contains my first ever rst file. Any improvements welcome. Thanks, Nils -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: vmod_digest.patch URL: