From chenxy.china at gmail.com Wed Jul 2 09:45:23 2008 From: chenxy.china at gmail.com (chen xiaoyong) Date: Wed, 2 Jul 2008 17:45:23 +0800 Subject: n_smf_large so large ?? Message-ID: <7ab169e90807020245w2f770725le7e6d9b55c510532@mail.gmail.com> Hey guys, Varnish trunk from 2008.06.30 . os environment: amd64-unknown-freebsd7.0 please see n_smf_large , it = 18446744073709551614 . maybe something is wrong . ------------------------------------------------------------------------------------------------------------------------------------- $ varnishstat -1 client_conn 1808282 343.78 Client connections accepted client_req 5559793 1056.99 Client requests received cache_hit 5381528 1023.10 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 178075 33.85 Cache misses backend_conn 178075 33.85 Backend connections success backend_fail 0 0.00 Backend connections failures backend_reuse 177625 33.77 Backend connections reuses backend_recycle 177784 33.80 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 4325 . N struct srcaddr n_srcaddr_act 981 . N active struct srcaddr n_sess_mem 3890 . N struct sess_mem n_sess 2642 . N struct sess n_object 152337 . N struct object n_objecthead 152215 . N struct objecthead n_smf 304323 . N struct smf n_smf_frag 7 . N small free smf n_smf_large 18446744073709551614 . N large free smf n_vbe_conn 14 . N struct vbe_conn n_bereq 92 . N struct bereq n_wrk 0 . N worker threads n_wrk_create 316 0.06 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 0 0.00 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 2 . N backends n_expired 26049 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 314163 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 3705775 704.52 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 1808092 343.74 Total Sessions s_req 5559753 1056.99 Total Requests s_pipe 0 0.00 Total pipe s_pass 0 0.00 Total pass s_fetch 178070 33.85 Total fetch s_hdrbytes 1556619286 295935.23 Total header bytes s_bodybytes 21531176400 4093379.54 Total body bytes sess_closed 64849 12.33 Session Closed sess_pipeline 444 0.08 Session Pipeline sess_readahead 177 0.03 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 5522363 1049.88 Session herd shm_records 211470782 40203.57 SHM records shm_writes 16601166 3156.12 SHM writes shm_flushes 363 0.07 SHM flushes due to overflow shm_cont 25302 4.81 SHM MTX contention sm_nreq 356414 67.76 allocator requests sm_nobj 304310 . outstanding allocations sm_balloc 3945529344 . bytes allocated sm_bfree 64773947392 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free backend_req 178075 33.85 Backend requests made -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at crucially.net Wed Jul 2 16:19:23 2008 From: sky at crucially.net (Artur Bergman) Date: Wed, 2 Jul 2008 18:19:23 +0200 Subject: gzip/esi Message-ID: <193C2411-BE8A-4D3D-9754-CA1F7F01B5C3@crucially.net> Hi, So there is a problem with gzip and ESI. If the content is gziped, you can't ESI it. If it isn't you waste bandwidth and go slower. Once you have the esibits tree up and running, if each individual chunk is gziped, you can just blast it away. So here is a proposal. Add a new action named encode; to be used after esi; . encode will take the current object, clone it and encode it to match your accept- encoding, insert it into the objhead and then replace the current working object. VCL example sub recv { lookup; } sub miss { # save away the accept-encoding and make sure # we request it uncompressed so we can ESI it set req.http.tmp_accept_encoding = req.http.Accept-Encoding; set req.http.Accept-Encoding = ""; } sub fetch { # xxx need a nicer way for this? esi; if(req.http.tmp_accept_encoding ~ /gzip/) { #client wants it gziped encode("gzip"); set req.http.Accept-Encoding = "gzip"; set obj.http.Content-Encoding = "gzip"; } insert; # inserts both variants of the object - in the future lookup will get the right version } The cloned object basically gets a copy of he store and the esibits, and then compresses each individual chunk. Cheers Artur From chenxy.china at gmail.com Fri Jul 11 04:39:11 2008 From: chenxy.china at gmail.com (chen xiaoyong) Date: Fri, 11 Jul 2008 12:39:11 +0800 Subject: " n_smf_large so large" -- varnish-dev Digest, Vol 28, Issue 1 Message-ID: <7ab169e90807102139n5253a54du4e3f8ef54f69cfb2@mail.gmail.com> I found that there are reasons for this phenomenon because it is negative. Not just n_smf_large, including n_smf_frag have this phenomenon too. please see struct varnish_stats { time_t start_time; #define MAC_STAT(n,t,f,e) t n; #include "stat_field.h" #undef MAC_STAT }; stat_field.h: ....... MAC_STAT(n_smf_frag, uint64_t, 'i', "N small free smf") MAC_STAT(n_smf_large, uint64_t, 'i', "N large free smf") ...... Perhaps some code of storage_file.c should be some more stringent 2008/7/2, varnish-dev-request at projects.linpro.no < varnish-dev-request at projects.linpro.no>: > > Send varnish-dev mailing list submissions to > varnish-dev at projects.linpro.no > > To subscribe or unsubscribe via the World Wide Web, visit > http://projects.linpro.no/mailman/listinfo/varnish-dev > or, via email, send a message with subject or body 'help' to > varnish-dev-request at projects.linpro.no > > You can reach the person managing the list at > varnish-dev-owner at projects.linpro.no > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of varnish-dev digest..." > > > Today's Topics: > > 1. n_smf_large so large ?? (chen xiaoyong) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 2 Jul 2008 17:45:23 +0800 > From: "chen xiaoyong" > Subject: n_smf_large so large ?? > To: varnish-dev at projects.linpro.no > Message-ID: > <7ab169e90807020245w2f770725le7e6d9b55c510532 at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hey guys, > > Varnish trunk from 2008.06.30 . os environment: > amd64-unknown-freebsd7.0 > > please see n_smf_large , it = 18446744073709551614 . maybe something is > wrong . > > > ------------------------------------------------------------------------------------------------------------------------------------- > $ varnishstat -1 > > client_conn 1808282 343.78 Client connections accepted > client_req 5559793 1056.99 Client requests received > cache_hit 5381528 1023.10 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 178075 33.85 Cache misses > backend_conn 178075 33.85 Backend connections success > backend_fail 0 0.00 Backend connections failures > backend_reuse 177625 33.77 Backend connections reuses > backend_recycle 177784 33.80 Backend connections recycles > backend_unused 0 0.00 Backend connections unused > n_srcaddr 4325 . N struct srcaddr > n_srcaddr_act 981 . N active struct srcaddr > n_sess_mem 3890 . N struct sess_mem > n_sess 2642 . N struct sess > n_object 152337 . N struct object > n_objecthead 152215 . N struct objecthead > n_smf 304323 . N struct smf > n_smf_frag 7 . N small free smf > n_smf_large 18446744073709551614 . N large free smf > n_vbe_conn 14 . N struct vbe_conn > n_bereq 92 . N struct bereq > n_wrk 0 . N worker threads > n_wrk_create 316 0.06 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_queue 0 0.00 N queued work requests > n_wrk_overflow 0 0.00 N overflowed work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 2 . N backends > n_expired 26049 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_saved 0 . N LRU saved objects > n_lru_moved 314163 . N LRU moved objects > n_deathrow 0 . N objects on deathrow > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 3705775 704.52 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 1808092 343.74 Total Sessions > s_req 5559753 1056.99 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 0 0.00 Total pass > s_fetch 178070 33.85 Total fetch > s_hdrbytes 1556619286 295935.23 Total header bytes > s_bodybytes 21531176400 4093379.54 Total body bytes > sess_closed 64849 12.33 Session Closed > sess_pipeline 444 0.08 Session Pipeline > sess_readahead 177 0.03 Session Read Ahead > sess_linger 0 0.00 Session Linger > sess_herd 5522363 1049.88 Session herd > shm_records 211470782 40203.57 SHM records > shm_writes 16601166 3156.12 SHM writes > shm_flushes 363 0.07 SHM flushes due to overflow > shm_cont 25302 4.81 SHM MTX contention > sm_nreq 356414 67.76 allocator requests > sm_nobj 304310 . outstanding allocations > sm_balloc 3945529344 . bytes allocated > sm_bfree 64773947392 . bytes free > sma_nreq 0 0.00 SMA allocator requests > sma_nobj 0 . SMA outstanding allocations > sma_nbytes 0 . SMA outstanding bytes > sma_balloc 0 . SMA bytes allocated > sma_bfree 0 . SMA bytes free > backend_req 178075 33.85 Backend requests made > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://projects.linpro.no/pipermail/varnish-dev/attachments/20080702/28c95e25/attachment.html > > ------------------------------ > > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > > End of varnish-dev Digest, Vol 28, Issue 1 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier at qalpit.com Wed Jul 16 13:17:05 2008 From: olivier at qalpit.com (Olivier Beau) Date: Wed, 16 Jul 2008 15:17:05 +0200 Subject: varnish version question Message-ID: <1216214225.487df4d12e8ec@www.qalpit.com> Hi, (i'm posting on this list because i havent found a varnish-user list.. hope that's ok) I discovered varnish a couple days, did some pretty good testing of 1.1.2 using the rpm available on sourceforge; i was peretty impressed by the performance and by the stability under heavy load.. (though there seems to be at least one little bug: after 2 hours of production, varnish just hangs; at least the master thread; didnt have time to do any debugging yet..) Anyway.. -> Which version of varnish is good for production ? (maybe this information should be made clear on varnish's website ?) Concerning my case : i will probably be installing varnish on ~100 servers (currently running apache, with or without squid..); I (and my team) will have time for testing, patching and mailling on this list for bugs and features requests.. -> should i consider using trunk ? Best Regards, Olivier Beau ps: is there a "who's using varnish" page with number of requests per second, etc.. ? (the kind of page my management would like seeing ;) From p.millar at physics.gla.ac.uk Wed Jul 16 17:15:11 2008 From: p.millar at physics.gla.ac.uk (Paul Millar) Date: Wed, 16 Jul 2008 19:15:11 +0200 Subject: Monitoring varnish Message-ID: <200807161915.11416.p.millar@physics.gla.ac.uk> Hi all, Just to introduce myself, I'm working on a project called MonAMI. It does monitoring: aiming to be plumbing between the thing(s) you want to monitor and the monitoring system itself. More details here: http://monami.sourceforge.net/ A little while ago, someone asked for a plugin for Varnish, which turned out to be very simple to implement. I noticed the wiki page has a few links for monitoring (under Performance page). I have a few questions: would it be appropriate to add a link to MonAMI? would anyone be interested in checking whether the plugin is gathering all the correct information and presenting it correctly? For the second part, it would be really good to get feedback from the Varnish developers as I don't run Varnish myself beyond a simple toy setup, (just to verify the plugin works correctly). Cheers, Paul. From phk at phk.freebsd.dk Thu Jul 17 08:40:15 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Jul 2008 08:40:15 +0000 Subject: varnish version question In-Reply-To: Your message of "Wed, 16 Jul 2008 15:17:05 +0200." <1216214225.487df4d12e8ec@www.qalpit.com> Message-ID: <53529.1216284015@critter.freebsd.dk> In message <1216214225.487df4d12e8ec at www.qalpit.com>, Olivier Beau writes: >-> Which version of varnish is good for production ? Hi Olivier, We are getting very close to a 2.0 release, so right now the best code we have is the trunk version from subversion. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 21 20:46:46 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 21 Jul 2008 20:46:46 +0000 Subject: Monitoring varnish In-Reply-To: Your message of "Wed, 16 Jul 2008 19:15:11 +0200." <200807161915.11416.p.millar@physics.gla.ac.uk> Message-ID: <5096.1216673206@critter.freebsd.dk> Hi Paul, I have no problems with providing likns on the wiki page if you suggest some text. Poul-Henning In message <200807161915.11416.p.millar at physics.gla.ac.uk>, Paul Millar writes: >Hi all, > >Just to introduce myself, I'm working on a project called MonAMI. It does >monitoring: aiming to be plumbing between the thing(s) you want to monitor >and the monitoring system itself. More details here: > http://monami.sourceforge.net/ > >A little while ago, someone asked for a plugin for Varnish, which turned out >to be very simple to implement. > >I noticed the wiki page has a few links for monitoring (under Performance >page). > >I have a few questions: > > would it be appropriate to add a link to MonAMI? > > would anyone be interested in checking whether the plugin is gathering all >the correct information and presenting it correctly? > >For the second part, it would be really good to get feedback from the Varnish >developers as I don't run Varnish myself beyond a simple toy setup, (just to >verify the plugin works correctly). > >Cheers, > >Paul. > > > >_______________________________________________ >varnish-dev mailing list >varnish-dev at projects.linpro.no >http://projects.linpro.no/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 21 21:37:31 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 21 Jul 2008 21:37:31 +0000 Subject: " n_smf_large so large" -- varnish-dev Digest, Vol 28, Issue 1 In-Reply-To: Your message of "Fri, 11 Jul 2008 12:39:11 +0800." <7ab169e90807102139n5253a54du4e3f8ef54f69cfb2@mail.gmail.com> Message-ID: <5411.1216676251@critter.freebsd.dk> In message <7ab169e90807102139n5253a54du4e3f8ef54f69cfb2 at mail.gmail.com>, "chen xiaoyong" writes: > >I found that there are reasons for this phenomenon because it is negative. >Not just n_smf_large, including n_smf_frag have this phenomenon too. > > Perhaps some code of storage_file.c should be some more stringent This is a know problem, and basically we decided that perfect statistics were not as important as fast speed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From moha at mohanjith.net Thu Jul 24 10:17:00 2008 From: moha at mohanjith.net (Sudirikku Mohanjith) Date: Thu, 24 Jul 2008 15:47:00 +0530 Subject: Custom Error pages Message-ID: Hi, Varnish helped me reduce my server load by at least 1/10. Thank you guys for creating such a wonderful HTTP accelerator. Is it possible to customize the Varnish error pages (50x)? I have tried googling, but only found this [1]. Any help is appreciated. Cheers, Mohanjith [1] http://varnish.projects.linpro.no/ticket/274 From p.millar at physics.gla.ac.uk Thu Jul 24 22:51:50 2008 From: p.millar at physics.gla.ac.uk (Paul Millar) Date: Fri, 25 Jul 2008 00:51:50 +0200 Subject: Monitoring varnish In-Reply-To: <5096.1216673206@critter.freebsd.dk> References: <5096.1216673206@critter.freebsd.dk> Message-ID: <200807250051.50167.p.millar@physics.gla.ac.uk> On Monday 21 July 2008 22:46:46 Poul-Henning Kamp wrote: > I have no problems with providing likns on the wiki page if you suggest > some text. Thanks! Currently the paragraph says: Tools here can be [Nagios http://www.nagios.org/], or [Munin http://munin.projects.linpro.no/] (see also [Muninexchange http://muninexchange.projects.linpro.no/ and http://anders.fupp.net/plugins/] for plugins). How about appending the following: There's also a [MonAMI http://monami.sourceforge.net/] monitoring plugin for gathering Varnish statistics. or a slightly longer version: There's also a [MonAMI http://monami.sourceforge.net/] monitoring plugin for gathering Varnish statistics that can deliver the metrics to any of its reporting plugins. ? Cheers, Paul.