From make.it.easy at gmail.com Mon Sep 3 14:53:06 2012 From: make.it.easy at gmail.com (F) Date: Mon, 3 Sep 2012 21:53:06 +0700 Subject: check response from varnish cache In-Reply-To: <9EF39A7D-5EF0-412C-B2AE-FBF6E6811175@gmail.com> References: <9EF39A7D-5EF0-412C-B2AE-FBF6E6811175@gmail.com> Message-ID: anyone can help provide me? On 1 Sep 2555 BE, at 00:56, F wrote: > Can anyone confirm and show me the response in both cases? > > I have not yet using Varnish so I cannot try. > > F > > On 31 Aug 2555 BE, at 22:24, John Adams wrote: > >> Just look at the X-Cache header. I believe this is in the default VCL, and it will return HIT or MISS depending if you hit the cache or not. >> >> -j >> >> >> On Fri, Aug 31, 2012 at 6:48 AM, Atanai Wuttisetpaiboon wrote: >> Hi, >> >> I am now start learning and thinking to use Varnish cache. I would like to know how can we know from the response whether the content is retrieved from the Varnish cache or from the web server? >> >> I have seen some cache devices, they returns HTTP AGE in HTTP header: >> - If Age is 0, it means the content is from web server >> - If Age is not 0 (greater than 0), means the content is from the cache device. >> >> I am not sure if Varnish cache has this ability? >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at bsdchicks.com Mon Sep 3 15:13:06 2012 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Mon, 3 Sep 2012 17:13:06 +0200 (CEST) Subject: check response from varnish cache In-Reply-To: References: <9EF39A7D-5EF0-412C-B2AE-FBF6E6811175@gmail.com> Message-ID: <20120903171138.E37583@ishtar.drwilco.net> Hi, This mailing list is purely for development discussions, and not for helping users. Please see either the IRC channel or the varnish-misc mailing list. https://www.varnish-cache.org/trac/wiki/MailingLists Cheers, Doc On Mon, 3 Sep 2012, F wrote: > anyone can help provide me? > > > > On 1 Sep 2555 BE, at 00:56, F wrote: > >> Can anyone confirm and show me the response in both cases? >> >> I have not yet using Varnish so I cannot try. >> >> F >> >> On 31 Aug 2555 BE, at 22:24, John Adams wrote: >> >>> Just look at the X-Cache header. I believe this is in the default VCL, and it will return HIT or MISS depending if you hit the cache or not. >>> >>> -j >>> >>> >>> On Fri, Aug 31, 2012 at 6:48 AM, Atanai Wuttisetpaiboon wrote: >>> Hi, >>> >>> I am now start learning and thinking to use Varnish cache. I would like to know how can we know from the response whether the content is retrieved from the Varnish cache or from the web server? >>> >>> I have seen some cache devices, they returns HTTP AGE in HTTP header: >>> - If Age is 0, it means the content is from web server >>> - If Age is not 0 (greater than 0), means the content is from the cache device. >>> >>> I am not sure if Varnish cache has this ability? >>> >>> _______________________________________________ >>> varnish-dev mailing list >>> varnish-dev at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>> > From huozhe at gmail.com Sat Sep 8 00:47:25 2012 From: huozhe at gmail.com (Zheng Liu) Date: Fri, 7 Sep 2012 17:47:25 -0700 Subject: can not load more VCL with error "cannot allocate memory", but with plenty memory left Message-ID: Hi, I have seen people report exactly the same error, but there is no solution to it. I hope some one here can shed some light on the issue. I saw a very high SMA.s0.c_fail "allocator failures". I think it is related. Is there a way to free some memory to just fit in the VCL config? I hate to restart the instance, since filling it up (160G) takes 1 day. thanks, Zheng vcl.load newconfig /etc/varnish/default.vcl 106 VCL compiled.dlopen(./vcl.lA3o8iw_.so): ./vcl.lA3o8iw_.so: cannot apply additional memory protection after relocation: Cannot allocate memory # /usr/sbin/varnishd -V varnishd (varnish-3.0.3 revision 9e6a70f) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2011 Varnish Software AS # free -g total used free shared buffers cached Mem: 189 176 12 0 0 0 -/+ buffers/cache: 175 13 Swap: 3 0 3 # uname -a Linux hcache6.houzz.com 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1550517 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1550517 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited # varnishstat -1 client_conn 520291099 520291099.00 Client connections accepted client_drop 52 52.00 Connection dropped, no sess/wrk client_req 520311099 520311099.00 Client requests received cache_hit 479753215 479753215.00 Cache hits cache_hitpass 5396 5396.00 Cache hits for pass cache_miss 40571613 40571613.00 Cache misses backend_conn 18647472 18647472.00 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 84912 84912.00 Backend conn. failures backend_reuse 21937169 21937169.00 Backend conn. reuses backend_toolate 35368 35368.00 Backend conn. was closed backend_recycle 21973014 21973014.00 Backend conn. recycles backend_retry 1529 1529.00 Backend conn. retry fetch_head 0 0.00 Fetch head fetch_length 40567091 40567091.00 Fetch with Length fetch_chunked 0 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 1924 1924.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 1119 . N struct sess_mem n_sess 18 . N struct sess n_object 4173835 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 4174219 . N struct objectcore n_objecthead 4190726 . N struct objecthead n_waitinglist 12437 . N struct waitinglist n_vbc 7 . N struct vbc n_wrk 400 . N worker threads n_wrk_create 465 465.00 N worker threads created n_wrk_failed 256 256.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 16153 16153.00 N queued work requests n_wrk_drop 53 53.00 N dropped work requests n_backend 2 . N backends n_expired 191407 . N expired objects n_lru_nuked 36119853 . N LRU nuked objects n_lru_moved 424530660 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 529243867 529243867.00 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 520314370 520314370.00 Total Sessions s_req 520311099 520311099.00 Total Requests s_pipe 2677 2677.00 Total pipe s_pass 37582 37582.00 Total pass s_fetch 40511177 40511177.00 Total fetch s_hdrbytes 203065278649 203065278649.00 Total header bytes s_bodybytes 23783551235305 23783551235305.00 Total body bytes sess_closed 520303774 520303774.00 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 72892 72892.00 Session Linger sess_herd 16001 16001.00 Session herd shm_records 24046733247 24046733247.00 SHM records shm_writes 2162622737 2162622737.00 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 13971844 13971844.00 SHM MTX contention shm_cycles 10341 10341.00 SHM cycles through buffer sms_nreq 44028 44028.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 18403704 . SMS bytes allocated sms_bfree 18403704 . SMS bytes freed backend_req 40582821 40582821.00 Backend requests made n_vcl 1 1.00 N vcl total n_vcl_avail 1 1.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 1.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 519442684 519442684.00 HCB Lookups without lock hcb_lock 40524771 40524771.00 HCB Lookups with lock hcb_insert 40524740 40524740.00 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 1 1.00 Connection dropped late uptime 755101 755101.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 77505 77505.00 Gunzip operations LCK.sms.creat 1 1.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 132084 132084.00 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 2.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 200409429 200409429.00 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 1.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 76879907 76879907.00 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 1.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 26598 26598.00 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 1.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 520315541 520315541.00 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 1.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 521596676 521596676.00 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 1.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 1611720 1611720.00 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 1.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 2884 2884.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 8 8.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 1046638906 1046638906.00 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 40523726 40523726.00 Created locks LCK.objhdr.destroy 36334480 36334480.00 Destroyed locks LCK.objhdr.locks 2153405517 2153405517.00 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 1.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 77551306 77551306.00 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 2.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 76604938 76604938.00 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 1.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 65 65.00 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 1.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 77555176 77555176.00 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 1.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 1.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 37466927 37466927.00 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 3 3.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 100145698 100145698.00 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 123055964 123055964.00 Allocator requests SMA.s0.c_fail 3551561563273 3551561563273.00 Allocator failures SMA.s0.c_bytes 1615116324709 1615116324709.00 Bytes allocated SMA.s0.c_freed 1447612608403 1447612608403.00 Bytes freed SMA.s0.g_alloc 8654138 . Allocations outstanding SMA.s0.g_bytes 167503716306 . Bytes outstanding SMA.s0.g_space 8238 . Bytes available SMA.Transient.c_req 43934 43934.00 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 646003217 646003217.00 Bytes allocated SMA.Transient.c_freed 646003217 646003217.00 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.node_master(192.168.0.21,,80).vcls 1 . VCL references VBE.node_master(192.168.0.21,,80).happy 0 . Happy health probes VBE.node_slave(192.168.0.39,,80).vcls 1 . VCL references VBE.node_slave(192.168.0.39,,80).happy 0 . Happy health probes -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Sat Sep 8 08:53:00 2012 From: perbu at varnish-software.com (Per Buer) Date: Sat, 8 Sep 2012 10:53:00 +0200 Subject: can not load more VCL with error "cannot allocate memory", but with plenty memory left In-Reply-To: References: Message-ID: Hi Zheng, This question belongs on varnish-misc. This list is for development issues. On Sat, Sep 8, 2012 at 2:47 AM, Zheng Liu wrote: > Hi, > > I have seen people report exactly the same error, but there is no solution > to it. I hope some one here can shed some light on the issue. I saw a very > high SMA.s0.c_fail "allocator failures". I think it is related. Is there a > way to free some memory to just fit in the VCL config? I hate to restart > the instance, since filling it up (160G) takes 1 day. > I'm guessing this is the somewhat screwed up overcommit logic on Linux. Could you try to set it to "1" and try again? echo 1 > /proc/sys/vm/overcommit_memory or something. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Sep 17 07:33:41 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Sep 2012 07:33:41 +0000 Subject: RFC: new vcl_lookup{} proposal Message-ID: <20314.1347867221@critter.freebsd.dk> Right now we go directly from vcl_hash{} to either vcl_hit{}, vcl_miss{} or vcl_pass{} depending on the outcome, and that means, amongst other things, that a VCL implementation of "PURGE" as to add code to all those. Furthermore we have some semi-magic things going on behind the scenes, such as grace and saint modes. Assume: * Backend fetches will be done by a different thread than the one processing the clients request. * vcl_hash{} always is followed by vcl_lookup{} and that the above mentioned logic gets moved to VCL, rather than C code. * vcl_hit{} goes away. Then a plausible default vcl_lookup{} could then look like: sub vcl_lookup { if (!obj) { // Miss: Start a background fetch in another thread // We come back here, when the status of that fetch // is available. (ie: we go on the waiting list) return (miss); } if (obj.ttl <= 0s && obj.ttl > -req.grace && obj.ttl > -obj.grace) { // We have a grace situation if (!obj.busy) { // Nobody is fetching yet, so start a // background fetch background_fetch(); } // NB: First client does not hang on the fetch return (deliver); } if (obj.ttl <= 0s) { // object is too old, fetch new one (see miss) return (miss); } return (deliver); } Possible customizations: Purging: if (req.request == "PURGE") { // We pressume access control was in vcl_recv{} return (purge); return (purge_all); } Prefetching: if (obj.ttl < 2s && !obj.busy) { // Close to deadline, pre-fetch background_fetch(); } Don't stream, only deliver complete objects: if (!obj.complete) { wait_complete(obj); } I'm seriously pondering if saintmode should become a vmod at the same time, in which case it would probably look something like: sub vcl_lookup { if (obj.ttl <= 0s && !saint.ok(req.backend, req.url)) { return (deliviver) } } sub vcl_fetch { if (beresp.status >= 400) { saint.bad(bereq.backend, bereq.url); } } -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kylem at cavecreek.net Tue Sep 18 13:00:43 2012 From: kylem at cavecreek.net (Kyle Morgan) Date: Tue, 18 Sep 2012 13:00:43 +0000 Subject: Cookie Normalization Message-ID: <5FCA0B4D0B956A43BC91305B07A90DEDD70E97@exch10-mb2.ccbill-hq.local> I'm writing in reference to the blog post: https://www.varnish-software.com/blog/validating-cookies-varnish. We've been attempting to configure content caching for a paid members area on a client's site. The content does not change per member, but our implementation is creating separate caches for each authenticated user. Is there a way to normalize the cookie for authenticated users so that only 1 cache is created for all members? So far, our trial/error allows anyone (authenticated or not) to view the cached contents once any content from the members area is cached. Below is the relevant contents of .vcl: # Piped Directly to the webserver if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") { # return (pipe); unset req.http.cookie; } # if labeled fja or auth pass through to backend if (req.http.Cookie ~ "(auth|fja)") { return (pipe); } # if labeled userdata or concept pass to cache if (req.http.Cookie ~ "(userdata|concept)") { return (lookup); } #KyleM bypass cookies from ####### if (req.http.host ~ "^messages\.#########\.com$") { return (pass); } set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", ""); set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__atu.=[^;]+;? *", ""); set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *OA.*=[^;]+;? *", ""); #set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *fja=[^;]+;? *", "fja=member"); if (req.http.cookie ~ "^ *$") { remove req.http.cookie; } return(lookup); } sub vcl_hash { hash_data(req.http.cookie); } sub vcl_pass { if (req.http.Authorization) { return(pass); } } sub vcl_fetch { unset beresp.http.Server; set beresp.http.Server = "#####"; set beresp.grace = 30s; Any help is greatly appreciated. Thanks! Kyle M. kylem at cavecreek.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Wed Sep 19 19:50:46 2012 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 19 Sep 2012 20:50:46 +0100 Subject: Cookie Normalization In-Reply-To: <5FCA0B4D0B956A43BC91305B07A90DEDD70E97@exch10-mb2.ccbill-hq.local> References: <5FCA0B4D0B956A43BC91305B07A90DEDD70E97@exch10-mb2.ccbill-hq.local> Message-ID: On 18 September 2012 14:00, Kyle Morgan wrote: > I'm writing in reference to the blog post: > https://www.varnish-software.com/blog/validating-cookies-varnish. > > > > We've been attempting to configure content caching for a paid members area > on a client's site. The content does not change per member, but our > implementation is creating separate caches for each authenticated user. > > > > Is there a way to normalize the cookie for authenticated users so that only > 1 cache is created for all members? So far, our trial/error allows anyone > (authenticated or not) to view the cached contents once any content from the > members area is cached. Below is the relevant contents of .vcl: Your email should really be directed to varnish-misc, but my approach here would be to leave the hash alone and add a Vary based on a request header recording if a user is anonymous or logged in. This is for the inverse use case, but something like it should work: sub vcl_recv { if (!(req.http.Authorization || req.http.cookie ~ "(^|; )__ac=")) { set req.http.X-Anonymous = "true"; } } sub vcl_fetch { if (!req.http.X-Anonymous && !beresp.http.Cache-Control ~ "public") { return(pass); } if (!beresp.http.Cache-Control ~ "public") { set beresp.http.Vary = "X-Anonymous"; } } Laurence From martin at varnish-software.com Fri Sep 28 13:34:14 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:34:14 +0200 Subject: [PATCH 01/13] Add a EXP_NukeLRU() function to nuke an entire LRU structure at a time. Message-ID: Comments from review: + struct objcore *oc_array[10]; > [...] > + while (n < 10) { > That's a no-go, #define or sizeof please. Changes: Changed to use a #define for the buffer size Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-a-EXP_NukeLRU-function-to-nuke-an-entire-LRU-str.patch Type: application/octet-stream Size: 2714 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 13:42:48 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:42:48 +0200 Subject: [PATCH 02/13] Create and use a smp_signspace structure to have range checking on the growing signed data structures. Message-ID: Review comments: ... I guess this is really a manifestation of the fact that smp_sign should > not be used as a space-handle in the first place, so what we really > should do is add: > struct smp_space { > uint64_t first; > uint64_t len; > uint64_t last; > struct smp_sign sign; > } Changes done: Added a smp_signspace struct that contains a sign and has ranges. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Create-and-use-a-smp_signspace-structure-to-have-ran.patch Type: application/octet-stream Size: 10272 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 13:46:24 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:46:24 +0200 Subject: [PATCH 06/13] Free the LRU object and set free_offset when dropping empty segments in smp_close_seg() Message-ID: Review comments: Looks correct. > (When do we close an empty segment, other than shutdown ?) We do it by explicit CLI commands. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Free-the-LRU-object-and-set-free_offset-when-droppin.patch Type: application/octet-stream Size: 1031 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 13:47:12 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:47:12 +0200 Subject: [PATCH 03/13] Add smp_copy_signspace() and smp_trunc_signspace() utility functions Message-ID: Add some utility functions for signspaces. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Add-smp_copy_signspace-and-smp_trunc_signspace-utili.patch Type: application/octet-stream Size: 2779 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 13:47:18 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:47:18 +0200 Subject: [PATCH 05/13] Round to page sizes on signature syncs Message-ID: Review comments: We should wrap this in a "smp_msync" function, rather than > pollute all msync calls with it. smp_msync function added -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Round-to-page-sizes-on-signature-syncs.patch Type: application/octet-stream Size: 2499 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 13:47:15 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 15:47:15 +0200 Subject: [PATCH 04/13] Sync the complete sign area on smp_sync_sign Message-ID: Review comments: We should clarify the meaning of the smp_sign->length field in include/persistent.h: Does it include struct smp_sign or not ? Comments about this added. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Sync-the-complete-sign-area-on-smp_sync_sign.patch Type: application/octet-stream Size: 1407 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:28 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:28 +0200 Subject: [PATCH 07/13] Don't exit on full silo Message-ID: This then also breaks the previous expectation that cur_seg would always be non-NULL. Change the code to take this into account. Review comments: + ALLOC_OBJ(sg, SMP_SEG_MAGIC); > + AN(sg); > Instead of AN, you should just return failure. We should handle > malloc failures gracefully where we can easily do so. Returning failure in this case now. In both the next two cases, you should consider truncation > of the new segment, it's silly to waste a full segment if > the overflow is just one page: Added XXX-comments on truncation suggestion. For clarity, always keep left in sync with cur_seg: Done -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Don-t-exit-on-full-silo.patch Type: application/octet-stream Size: 5266 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:30 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:30 +0200 Subject: [PATCH 08/13] Add an assert that segment rounding doesn't overstep our previous allocation calculations. Message-ID: -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0008-Add-an-assert-that-segment-rounding-doesn-t-overstep.diff Type: application/octet-stream Size: 944 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:36 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:36 +0200 Subject: [PATCH 10/13] smp_thread() stopping Message-ID: Review comments: All things considered, we should probably VSL these and live with > it. SLT_PersistentEvent or something. Agreed, logging from persistent needs to be overhauled as a whole. Putting on todo list. We should avoid whole-second sleeps, they have a bad tendency to > get things to run in lockstep. Ideally each silo should sleep > for 1+random-small-delta, but at least make it different from 1.0 Changed You should simply let the tread die, and pthread_join() it from > smp_close(). Implemented -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0010-smp_thread-stopping.diff Type: application/octet-stream Size: 3393 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:33 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:33 +0200 Subject: [PATCH 09/13] Add a signal_close callback method to stevedores, which can be used to signal background threads to stop in preparation for the coming close callback. Message-ID: Review comments: In either case, it is silly that we close the silos serially, you > should add a "signal_close()" method to stevedore, and call > any which is non-NULL as the first thing in STV_close(), so that > we can sed the SC_STOP on all silos before we wait for the first > one to do so. Implemented this -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0009-Add-a-signal_close-callback-method-to-stevedores-whi.diff Type: application/octet-stream Size: 1976 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:41 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:41 +0200 Subject: [PATCH 12/13] Also report dropped bans to the stevedores. Message-ID: -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0012-Also-report-dropped-bans-to-the-stevedores.diff Type: application/octet-stream Size: 3158 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:39 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:39 +0200 Subject: [PATCH 11/13] Generalize the ban reporting to the stevedores using their API. Message-ID: This way any stevedore interested in new bans can request to be notified (not just the persistent). Review comments: > I appreciate the generality of STV_NewBan(), but what is the point > of calling BAN_Spec, rather than passing the ban+len as arguments ? Reasoning behind BAN_Spec was to preserve the opacity of the ban data structures. Changed to pass ban+len as arguments. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0011-Generalize-the-ban-reporting-to-the-stevedores-using.diff Type: application/octet-stream Size: 4447 bytes Desc: not available URL: From martin at varnish-software.com Fri Sep 28 14:03:51 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Sep 2012 16:03:51 +0200 Subject: [PATCH 13/13] Add consistency checks between the ban lists at startup Message-ID: Review comments: I wonder if it would make more sense to not write here, and instead > wait until everything is loaded and then write new (and possibly shorter) > ban lists to both segments ? (Ref: recent talk about pruning bans) I think we have to make sure that we are in a consistent state before we start running, as the ban thread is going to start adding the placeholder ban and everything the moment that starts. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0013-Add-consistency-checks-between-the-ban-lists-at-star.diff Type: application/octet-stream Size: 1329 bytes Desc: not available URL: From tfheen at varnish-software.com Fri Sep 28 14:10:05 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 28 Sep 2012 16:10:05 +0200 Subject: Tracking patches In-Reply-To: <500EE113.2090100@schokola.de> References: <500EE113.2090100@schokola.de> Message-ID: <20120928141005.GB7902@err.no> ]] Nils Goroll Hi, > at VUG5, we briefly discussed the fact that it could be helpful to track patches > and IIUC, Tollef had a good idea on this which he planned to integrate. Yes, I've now finally set up patchwork so we can track patches submitted to the mailing list. I'm not sure I want to move the project to github and their pull ideas, but we should now at least be able to track patches on https://www.varnish-cache.org/patchwork/project/varnish-cache/list/ Sorry about this taking a very long time to get implemented. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64