From wxz19861013 at gmail.com Mon Apr 1 08:20:45 2013 From: wxz19861013 at gmail.com (Xianzhe Wang) Date: Mon, 1 Apr 2013 16:20:45 +0800 Subject: varnish-3.0.2-streaming crash issue (revision cd0ccbf) Message-ID: I use varnish-3.0.2-streaming for my application. I found that the object with "Cache-Control:max-age=31536000" will miss in couple days sometimes. And then I notice that varnish child process crash sometimes. This is panic log: panic.show 200 Last panic at: Sun, 31 Mar 2013 22:16:07 GMT Assert error in Tcheck(), cache.h line 1004: Condition(t.b <= t.e) not true. thread = (cache-worker) ident = Linux,2.6.32.59-0.7-xen,x86_64,-smalloc,-smalloc,-hcritbit,epoll Backtrace: 0x42ffb3: pan_ic+d3 0x42cd35: http_IsHdr+65 0x42d311: http_FilterFields+3e1 0x43417c: RES_BuildHttp+9c 0x417248: cnt_prepresp+218 0x41998d: CNT_Session+4ad 0x4327c3: wrk_do_cnt_sess+93 0x431a0a: wrk_thread_real+3ea 0x7f0a266a56a6: _end+7f0a2602938e 0x7f0a26414f7d: _end+7f0a25d98c65 sp = 0x7f0a1369a008 { fd = 260, id = 260, xid = 752636266, client = xx.xxx.xxx.xxx xxxx, step = STP_PREPRESP, handling = hit_for_pass, err_code = 200, err_reason = (null), restarts = 0, esi_level = 0 flags = bodystatus = 3 ws = 0x7f0a1369a080 { id = "sess", {s,f,r,e} = {0x7f0a1369c628,+608,(nil),+65536}, }, http[req] = { ws = 0x7f0a1369a080[sess] "GET", " http://g.example.com/fcg-bin/cgi_emotion_list.fcg?uin=xxxx&loginUin=x&s=xxxx&num=xx&noflower=xx&g_tk=xxx ", "HTTP/1.1", "Accept: */*", "Referer: http://user.example.com/xxxxx", "Accept-Language: Zh-cn", "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; SE 2.X MetaSr 1.0", "Host: g.example.com", "Connection: close", "Cache-Control: no-cache", "Cookie: randomSeed=774464;", "X-Forwarded-For: xx.xxx.xxx.xxx", }, worker = 0x7f0a037e5a90 { ws = 0x7f0a037e5ce0 { id = "wrk", {s,f,r,e} = {0x7f0a037d3a20,0x7f0a037d3a20,(nil),+65536}, }, http[resp] = { ws = 0x7f0a037e5ce0[wrk] "HTTP/1.1", "OK", "Server: QZHTTP-2.35.2", "Via: 1.1 localhost", "X-Accelerate: 2.20", "Vary: Accept-Encoding", }, }, vcl = { srcname = { "input", "Default", }, }, obj = 0x7f0a0ddf6000 { xid = 752636266, ws = 0x7f0a0ddf6018 { id = "obj", {s,f,r,e} = {0x7f0a0ddf6200,+240,(nil),+272}, }, http[obj] = { ws = 0x7f0a0ddf6018[obj] "HTTP/1.1", "OK", "Date: Sun, 31 Mar 2013 22:16:07 GMT", "Server: QZHTTP-2.35.2", "Via: 1.1 localhost", "X-Accelerate: 2.20", "Vary: Accept-Encoding", "Content-Type: text/html", "Content-Length: 211", }, len = 211, store = { 211 { 76 69 73 69 74 43 6f 75 6e 74 43 61 6c 6c 42 61 |visitCountCallBa| 63 6b 28 7b 22 72 65 74 63 6f 64 65 22 3a 31 2c |ck({"retcode":1,| 22 76 69 73 69 74 63 6f 75 6e 74 22 3a 30 2c 22 |"visitcount":0,"| 64 61 79 76 69 73 69 74 22 3a 30 2c 22 73 70 61 |dayvisit":0,"spa| [147 more] }, }, }, }, This is my starup order: ./varnishd -f varnish-stream.vcl -s malloc,16G -T 127.0.0.1:2000 -a 0.0.0.0:8080 -p thread_pool_min=200 -p thread_pool_max=4000 -p thread_pool_add_delay=2 -p session_linger=100 -p thread_pools=2 -p http_req_hdr_len=32768 -p http_resp_hdr_len=32768 -p http_max_hdr=256 This is my varnish-stream.vcl : # This is a basic VCL configuration file for varnish. See the vcl(7) # man page for details on VCL syntax and semantics. # # Default backend definition. Set this to point to your content # server. # probe healthcheck { .url = "/"; .interval = 30s; .timeout = 0.5 s; .window = 8; .threshold = 3; .initial = 3; } backend proxy1 { .host = "x.x.x.x1"; .port = "8080"; .probe = healthcheck; } backend proxy2 { .host = "x.x.x.x2"; .port = "8080"; .probe = healthcheck; } backend proxy3 { .host = "x.x.x.x3"; .port = "8080"; .probe = healthcheck; } backend proxy3 { .host = "x.x.x.x4"; .port = "8080"; .probe = healthcheck; } director proxy client { { .backend = proxy1; .weight = 1; } { .backend = proxy2; .weight = 1; } { .backend = proxy3; .weight = 1; } } acl refresh { "x.x.x.x5"; } # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. sub vcl_recv { if(req.http.X-Real-IP){ set client.identity = req.http.X-Real-IP; }else if (req.http.referer) { set client.identity = req.http.referer; }else{ set client.identity = req.url; } # set req.backend = proxy; if(client.ip == "x.x.x.x6"){ set req.backend = proxy; }else{ set req.backend = proxy4; if (client.ip ~ refresh) { set req.hash_always_miss = true; } } #set grace if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 30m; } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { # /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.http.x-pipe && req.restarts > 0) { remove req.http.x-pipe; return (pipe); } if(req.request != "GET" && req.request != "HEAD") { # /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Cache-Control ~ "no-cache") { return (pass); } if (req.http.Accept-Encoding) { if (req.url ~ "\.(webp|jpeg|png|mid|mp3|gif|sql|jpg|nth|thm|utz|mtf|sdt|hme|tsk|zip|rar|sx|pxl|cab|mbm|app|exe|apk)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unknown algorithm remove req.http.Accept-Encoding; } } if (req.http.Authorization) { return (pass); } return (lookup); } sub vcl_pipe { set bereq.http.connection = "close"; return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { if (req.url ~ ".(jpeg|jpg|png|gif|ico|js|css)\?.*") { hash_data(regsub(req.url, "\?[^\?]*$", "")); } else{ hash_data(req.url); } if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (hash); } sub vcl_hit { return (deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { set beresp.grace = 30m; set beresp.do_stream = true; if (beresp.http.Content-Length && beresp.http.Content-Length ~ "[0-9]{8,}") { set req.http.x-pipe = "1"; return (restart); } if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "no-cache" || beresp.http.Cache-Control ~ "private"){ return (hit_for_pass); } if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120 s; return (hit_for_pass); } return (deliver); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT from varnish"; set resp.http.X-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS from varnish"; } remove resp.http.Via; remove resp.http.X-Varnish; return (deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" "} + obj.status + " " + obj.response + {"

Error "} + obj.status + " " + obj.response + {"

"} + obj.response + {"

Guru Meditation:

XID: "} + req.xid + {"

Varnish cache server

varnish

"}; return (deliver); } sub vcl_init { return (ok); } sub vcl_fini { return (ok); } This is varnishstat in 0401: ./varnishstat -1 client_conn 3769680 342.17 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 3881633 352.33 Client requests received cache_hit 26574 2.41 Cache hits cache_hitpass 6072 0.55 Cache hits for pass cache_miss 155003 14.07 Cache misses backend_conn 304379 27.63 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 41 0.00 Backend conn. failures backend_reuse 3551465 322.36 Backend conn. reuses backend_toolate 1104 0.10 Backend conn. was closed backend_recycle 3552586 322.46 Backend conn. recycles backend_retry 780 0.07 Backend conn. retry fetch_head 0 0.00 Fetch head fetch_length 2308273 209.52 Fetch with Length fetch_chunked 1285891 116.72 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_streamed 3620982 328.67 Fetch streamed fetch_bad 0 0.00 Fetch had bad headers fetch_close 26761 2.43 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 62 0.01 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 49 0.00 Fetch no body (204) fetch_304 10 0.00 Fetch no body (304) n_sess_mem 3103 . N struct sess_mem n_sess 317 . N struct sess n_object 16518 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 16878 . N struct objectcore n_objecthead 13535 . N struct objecthead n_waitinglist 3049 . N struct waitinglist n_vbc 68 . N struct vbc n_wrk 400 . N worker threads n_wrk_create 618 0.06 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 2081 0.19 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 3 . N backends n_expired 138436 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 21504 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 25702 2.33 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 3769636 342.17 Total Sessions s_req 3881633 352.33 Total Requests s_pipe 233231 21.17 Total pipe s_pass 3466827 314.68 Total pass s_fetch 0 0.00 Total fetch s_stream 3620813 328.66 Total streamed requests s_hdrbytes 986265856 89522.18 Total header bytes s_bodybytes 28556419 2592.03 Total body bytes sess_closed 3657484 331.99 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 262709 23.85 Session Linger sess_herd 262383 23.82 Session herd shm_records 304070226 27600.09 SHM records shm_writes 26571905 2411.90 SHM writes shm_flushes 2 0.00 SHM flushes due to overflow shm_cont 49165 4.46 SHM MTX contention shm_cycles 130 0.01 SHM cycles through buffer sms_nreq 846 0.08 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 384930 . SMS bytes allocated sms_bfree 384930 . SMS bytes freed backend_req 3621062 328.68 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 187649 17.03 HCB Lookups without lock hcb_lock 80799 7.33 HCB Lookups with lock hcb_insert 80799 7.33 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 11017 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 271725 24.66 Gunzip operations LCK.sms.creat 4 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 8550 0.78 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 8 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 38262423 3473.03 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 4 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 418452 37.98 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 4 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 9462038 858.86 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 4 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 9840 0.89 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 4 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 28114307 2551.90 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 4 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 212775 19.31 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 4 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1209 0.11 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 8 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 38953507 3535.76 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 270862 24.59 Created locks LCK.objhdr.destroy 141466 12.84 Destroyed locks LCK.objhdr.locks 4746962 430.88 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 4 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 856264 77.72 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 8 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 480694 43.63 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 4 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 22068 2.00 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 4 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 857486 77.83 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 4 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 6642 0.60 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 4 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 1723928 156.48 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 12 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 20863568 1893.76 Lock Operations LCK.backend.colls 0 0.00 Collisions LCK.busyobj.creat 2153 0.20 Created locks LCK.busyobj.destroy 658 0.06 Destroyed locks LCK.busyobj.locks 51597325 4683.43 Lock Operations LCK.busyobj.colls 0 0.00 Collisions SMA.s0.c_req 240044 21.79 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 14874337772 1350125.97 Bytes allocated SMA.s0.c_freed 14770447168 1340695.94 Bytes freed SMA.s0.g_alloc 30924 . Allocations outstanding SMA.s0.g_bytes 103890604 . Bytes outstanding SMA.s0.g_space 17075978580 . Bytes available SMA.Transient.c_req 6620156 600.90 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 173497152178 15748130.36 Bytes allocated SMA.Transient.c_freed 173487087827 15747216.83 Bytes freed SMA.Transient.g_alloc 1383 . Allocations outstanding SMA.Transient.g_bytes 10064351 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.proxy1(xx.xx.xx.xx1,,8080).vcls 4 . VCL references VBE.proxy1(xx.xx.xx.xx1,,8080).happy18446744073709551615 . Happy health probes VBE.proxy2(xx.xx.xx.xx2,,8080).vcls 4 . VCL references VBE.proxy2(xx.xx.xx.xx2,,8080).happy18446744073709551615 . Happy health probes VBE.proxy3(xx.xx.xx.xx3,,8080).vcls 4 . VCL references VBE.proxy3(xx.xx.xx.xx3,,8080).happy18446744073709551615 . Happy health probes VBE.proxy4(xx.xx.xx.xx4,,8081).vcls 4 . VCL references VBE.proxy4(xx.xx.xx.xx4,,8081).happy18446744073709551615 . Happy health probes This is my parameter: param.show 200 acceptor_sleep_decay 0.900000 [] acceptor_sleep_incr 0.001000 [s] acceptor_sleep_max 0.050000 [s] auto_restart on [bool] ban_dups on [bool] ban_lurker_sleep 0.010000 [s] between_bytes_timeout 60.000000 [s] cc_command "exec gcc -std=gnu99 -pthread -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 10 [seconds] clock_skew 10 [s] connect_timeout 0.700000 [s] critbit_cooloff 180.000000 [s] default_grace 10.000000 [seconds] default_keep 0.000000 [seconds] default_ttl 120.000000 [seconds] diag_bitmap 0x0 [bitmap] esi_syntax 0 [bitmap] expiry_sleep 1.000000 [seconds] fetch_chunksize 128 [kilobytes] fetch_maxchunksize 262144 [kilobytes] first_byte_timeout 60.000000 [s] group nobody (65533) gzip_level 6 [] gzip_memlevel 8 [] gzip_stack_buffer 32768 [Bytes] gzip_tmp_space 0 [] gzip_window 15 [] http_gzip_support on [bool] http_max_hdr 256 [header lines] http_range_support on [bool] http_req_hdr_len 32768 [bytes] http_req_size 32768 [bytes] http_resp_hdr_len 32768 [bytes] http_resp_size 32768 [bytes] listen_address 0.0.0.0:8080 listen_depth 1024 [connections] log_hashstring on [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_depth 5 [levels] max_restarts 4 [restarts] nuke_limit 50 [allocations] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] queue_max 100 [%] rush_exponent 3 [requests per request] saintmode_threshold 10 [objects] send_timeout 60 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 100 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] shortlived 10.000000 [s] stream_maxchunksize 256 [kilobytes] stream_tokens 10 [] syslog_cli_traffic on [bool] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 4000 [threads] thread_pool_min 200 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 300 [seconds] thread_pool_workspace 65536 [bytes] thread_pools 2 [pools] thread_stats_rate 10 [requests] user nobody (65534) vcc_err_unref on [bool] vcl_dir /opt/varnish-3.0.2-streaming/etc/varnish vcl_trace off [bool] vmod_dir /opt/varnish-3.0.2-streaming/lib/varnish/vmods waiter default (epoll, poll) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliezer at ngtech.co.il Mon Apr 1 19:00:50 2013 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Mon, 01 Apr 2013 22:00:50 +0300 Subject: Caching MS updates using varnish. In-Reply-To: <1364504704-sup-5990@geror.local> References: <5153D5B3.5040100@ngtech.co.il> <1364504704-sup-5990@geror.local> Message-ID: <5159D962.4060104@ngtech.co.il> On 03/28/2013 11:06 PM, James Pearson wrote: > Excerpts from Eliezer Croitoru's message of 2013-03-27 22:31:31 -0700: >> I want to cache MS updates using varnish. > Varnish is designed from the start as a reverse-proxy (well, http accelerator, > I guess). The kind of thing you're describing is much more suited to tools > developed to solve that problem, e.g. Squid. > > - P Squid dosn't cache 206 responses.. Eliezer > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From luky-37 at hotmail.com Tue Apr 2 07:48:12 2013 From: luky-37 at hotmail.com (Lukas Tribus) Date: Tue, 2 Apr 2013 09:48:12 +0200 Subject: Caching MS updates using varnish. In-Reply-To: <5159D962.4060104@ngtech.co.il> References: <5153D5B3.5040100@ngtech.co.il> <1364504704-sup-5990@geror.local>,<5159D962.4060104@ngtech.co.il> Message-ID: > Squid dosn't cache 206 responses.. Well then use polipo [1]; still, this is not a varnish topic. Thanks, Lukas [1] http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/ From itxx00 at gmail.com Tue Apr 2 08:46:47 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Tue, 2 Apr 2013 16:46:47 +0800 Subject: number of thread pools In-Reply-To: References: Message-ID: 2013/3/30 Sean Allen > Why is it advised for Varnish 3 to only run with 2 thread pools rather > than a thread pool per cpu core? > > I think it's because we need more worker threads by default to handle requests. -------------- next part -------------- An HTML attachment was scrubbed... URL: From enno at tty.nl Tue Apr 2 19:18:11 2013 From: enno at tty.nl (Enno van Amerongen) Date: Tue, 02 Apr 2013 21:18:11 +0200 Subject: =?iso-8859-1?Q?increase_sess=5Fworkspace_to_fix_assert_error=3F?= Message-ID: <20130402191811.3b46f376@groupware.tty.nl> Dear List, After deployment of a vmod we've written (that enables us to select a director by id in url), we've been experiencing assert errors. Below are the stack traces, but to not waste your time, here's what I think it boils down to: { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 } 4k as session workspace has worked fine for us for years, but our vmod sharding adds hundreds of directors (as opposed to 4), while the number of backends stayed the same (about 25). So, my question is, do I just need to increase sess_workspace to fix this? If not, what else ;) Regards, Enno van Amerongen Both stacktraces Apr 2 18:43:32 obfuscated /var/www/varnish[12750]: Child (20498) Panic message: Assert error in vh_get_var_alloc(), vmod_var.c line 77:#012 Condition((v) != 0) not true.#012errno = 32 (Broken pipe)#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x7fe137efe347: _end+7fe13786898f#012 0x7fe137efe862: _end+7fe137868eaa#012 0x7fecc5b93e73: _end+7fecc54fe4bb#012 0x7fecc5b939ea: _end+7fecc54fe032#012 0x7fecc5ba24f1: _end+7fecc550cb39#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012sp = 0x7fed81293008 {#012 fd = 2849, id = 2849, xid = 918440385,#012 client = 10.1.1.1 51854,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed81293080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed81293c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed81293080[sess]#012 "GET",#012 "/f/146057654_2.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14",#012 "Host: img.obfuscated.com",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: fr-FR,fr;q=0.9,en;q=0.8",#012 "Referer: http://obfuscated",#012 "Connection: Keep-Alive",#012 },#012 worker = 0x7feece7efaa0 {#012 ws = 0x7feece7efcd8 { #012 id = "wrk",#012 {s,f,r,e} = {0x7feece7dd9f0,+16,(nil),+65536},#012 },#012 },#012 vcl = {#012 srcname = {#012 "input",#012 "Default",#012 "/obfuscated/img_backends.vcl",#012 "/obfuscated/img_directors.vcl",#012 So we removed the vmod_var from our configuration as it wasn't really required anyway. Then 13 minutes later, Varnish crashed again, now with a different, but similar assert error: Apr 2 19:07:22 obfuscated /var/www/varnish[12750]: Child (18818) Panic message: Assert error in VRT_int_string(), cache_vrt.c line 324:#012 Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x4435f7: VRT_int_string+88#012 0x7fe686b9ce1f: _end+7fe686507467#012 0x7fe686b9c99a: _end+7fe686506fe2#012 0x7fe686ba7424: _end+7fe686511a6c#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012 0x43aa0a: wrk_thread_real+8cc#012sp = 0x7fed76034008 {#012 fd = 2357, id = 2357, xid = 2009537706,#012 client = 80.239.243.196 59852,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed76034080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed76034080[sess]#012 "GET",#012 "/f/148682988.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera Mini/7.5.32193/29.3134; U; nl) Presto/2.8.119 Version/11.10",#012 "Host: img.obfuscated.com",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: nl,en;q=0.9",#012 "Referer: obfuscated",#012 "Connection: Keep-Alive",#012 "X-OperaMini-Phone: Android #",#012 "X-OperaMini-Phone-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "x-forwarded-for: obfuscated",#012 "Device-Stock-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "X -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliezer at ngtech.co.il Wed Apr 3 06:58:30 2013 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Wed, 03 Apr 2013 09:58:30 +0300 Subject: Caching MS updates using varnish. In-Reply-To: References: <5153D5B3.5040100@ngtech.co.il> <1364504704-sup-5990@geror.local>, <5159D962.4060104@ngtech.co.il> Message-ID: <515BD316.5020604@ngtech.co.il> On 04/02/2013 10:48 AM, Lukas Tribus wrote: >> Squid dosn't cache 206 responses.. > Well then use polipo [1]; still, this is not a varnish topic. > > > Thanks, > > Lukas > > [1] http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/ INdeed. Thanks anyway. From m.ramanna at ymail.com Thu Apr 4 01:34:36 2013 From: m.ramanna at ymail.com (Madhusudan Ramanna) Date: Wed, 3 Apr 2013 18:34:36 -0700 (PDT) Subject: IMS question Message-ID: <1365039276.48445.YahooMailNeo@web121706.mail.ne1.yahoo.com> Hello, I'm a newbie trying Varnish 3.0.3 setup. ?We need IMS support from varnish and it looks like IMS changes didn't make it to the release. ?Could someone point me to any doc on building from source for IMS branch ? We have both centos and ubuntu machines Thanks for your help ! Madhu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Thu Apr 4 02:38:59 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 3 Apr 2013 21:38:59 -0500 Subject: IMS question In-Reply-To: <1365039276.48445.YahooMailNeo@web121706.mail.ne1.yahoo.com> References: <1365039276.48445.YahooMailNeo@web121706.mail.ne1.yahoo.com> Message-ID: <20130404023859.GE21122@nat.myhome> On Wed, Apr 03, 2013 at 06:34:36PM -0700, Madhusudan Ramanna wrote: > Hello, > > I'm a newbie trying Varnish 3.0.3 setup. ?We need IMS support from varnish and it looks like IMS changes didn't make it to the release. ?Could someone point me to any doc on building from source for IMS branch ? We have both centos and ubuntu machines > > Thanks for your help ! > > Madhu > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ?IMS = If-Modified-Since? https://www.varnish-cache.org/docs/trunk/installation/install.html The documentation provides information on how to download the development branch of varnish. The docs also cover howto build from source if you are unfamiliar with that. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From m.ramanna at ymail.com Thu Apr 4 07:18:22 2013 From: m.ramanna at ymail.com (Madhusudan Ramanna) Date: Thu, 4 Apr 2013 00:18:22 -0700 (PDT) Subject: IMS question In-Reply-To: <20130404023859.GE21122@nat.myhome> References: <1365039276.48445.YahooMailNeo@web121706.mail.ne1.yahoo.com> <20130404023859.GE21122@nat.myhome> Message-ID: <1365059902.60549.YahooMailNeo@web121701.mail.ne1.yahoo.com> Thanks Paul -? Yes IMS == If-Modified-Since I couldn't find IMS branch in?repo.varnish-cache.org. ?I saw source tarball for streaming but not for IMS branch. ?Is the IMS branch merged into trunk ?? thanks, Madhu ________________________________ From: Paul A. Procacci To: Madhusudan Ramanna Cc: "varnish-misc at varnish-cache.org" Sent: Wednesday, April 3, 2013 7:38 PM Subject: Re: IMS question On Wed, Apr 03, 2013 at 06:34:36PM -0700, Madhusudan Ramanna wrote: > Hello, > > I'm a newbie trying Varnish 3.0.3 setup. ?We need IMS support from varnish and it looks like IMS changes didn't make it to the release. ?Could someone point me to any doc on building from source for IMS branch ? We have both centos and ubuntu machines > > Thanks for your help ! > > Madhu > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ?IMS = If-Modified-Since? https://www.varnish-cache.org/docs/trunk/installation/install.html The documentation provides information on how to download the development branch of varnish.? The docs also cover howto build from source if you are unfamiliar with that. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Thu Apr 4 09:21:11 2013 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Thu, 4 Apr 2013 11:21:11 +0200 Subject: backend.set_health bug In-Reply-To: <6DD1CF2A31C9724E9425CDB5511A92FC1C683EF2@DUB-EXDDA-S32.corp.ebay.com> References: <6DD1CF2A31C9724E9425CDB5511A92FC1C683DC7@DUB-EXDDA-S32.corp.ebay.com> <6DD1CF2A31C9724E9425CDB5511A92FC1C683EF2@DUB-EXDDA-S32.corp.ebay.com> Message-ID: <20130404092111.GB21660@immer.varnish-software.com> > On 27.03.13 15:21, "Hettwer, Marian" wrote: > >seems like the parser for specifying the backend is buggy. Thank you for reporting this. Can you please file a bug on the official bugtracker: https://www.varnish-cache.org/trac Thanks. -- With regards, Lasse Karstensen Varnish Software AS From lkarsten at varnish-software.com Thu Apr 4 10:42:12 2013 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Thu, 4 Apr 2013 12:42:12 +0200 Subject: Why "pass" object goes to vcl_hash? In-Reply-To: References: Message-ID: <20130404104209.GC21660@immer.varnish-software.com> On Thu, Mar 28, 2013 at 03:07:57PM +0800, Kelvin Loke wrote: > This was bothering me for quite some time. I looked through the Varnish > Book detailed request flow diagram, an object that is "pass" in vcl_recv() > will go straight to vcl_pass() -> vcl_fetch() -> vcl_deliver(). > But the varnishlog below tells me otherwise, why an object that is "pass" > in vcl_recv still needs to go to vcl_hash()? Hi Kelvin. Great question. In vcl_fetch we need a hash value for the directors that do load balancing on backends. -- With regards, Lasse Karstensen Varnish Software AS From ruben at varnish-software.com Sat Apr 6 00:27:37 2013 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Sat, 6 Apr 2013 02:27:37 +0200 Subject: Welcome to VUG7 in New York City, USA - May 30th & 31st 2013 Message-ID: Howdy, Once again it is my pleasure to announce, in behalf of the Varnish Team, our next Varnish User Group meeting. Under the Seventh VUG (Second time we hold it in the USA we will be joining the vibrant Varnish community in the Big Apple at the very end of May. The meeting is kindly sponsored by Vimeo and 10gen. *** Registration is open, so get your ticket before it is too late: < http://vug7.eventbrite.co.uk> *** * The Varnish User Day will be on Friday 31st of May at Vimeo. For agenda and general information [1] * The Varnish Developer meeting will on Thursday 30th of May at 10gen. The exact location of the event will only be communicated to the hardcore C developers working with Varnish Core and VMOD hacking that are allowed to attend ;-) See and use the wiki for coordinating details [2] * For those interested in training, there is a session on June 3rd & 4th in Manhattan. More information, pricing and registration [3] A cool thing we will have during the User Day at this VUG is the *VCL speed-dating session* where everyone can share their best tricks, propose puzzles for other to solve and/or even changes for VCL in 4.0. Hopefully, everyone can take something to try at home under this session. Please start thinking about what you are going to show ;-) As usual, presentation are welcome, and they can be added to the agenda by replying to this email or directly in the wiki planning page [4]. Otherwise, I will eventually reach to some you and ask you if can hold a presentation :-) Feel free to tell your colleagues and friends about the event! Links: [1] [2] [3] [4] All the best, -- *Rub?n Romero* Global Sales Executive & Community Hygiene Keeper | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype & Twitter: ruben_varnish We Make Websites Fly! PD: For everyone in the Bay Area and the West Coast: There is hope! Varnish Software will visit your neighborhood as we will be at Velocity 2013 (yes, we have a booth). We expect to have an event there the day after Velocity (soon to be announce). Stay tuned! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkoelle at gmx.de Mon Apr 8 06:49:45 2013 From: mkoelle at gmx.de (Markus Koelle) Date: Mon, 8 Apr 2013 08:49:45 +0200 Subject: Problem with persistent storage Message-ID: <22ECB25A-1E1C-4014-897D-A423CCDC74E2@gmx.de> Hi, I've a out-of-space problem with persistent storage (varnish 3.0.3 an Debian wheezy)) . My storage is 64 GB and it's only half full. /var/log/syslog says all 3 hours: Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) said Out of space in persistent silo Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) said Committing suicide, restart will make space Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) ended Apr 8 08:13:45 varnish varnishd[6908]: Child cleanup complete Apr 8 08:13:45 varnish varnishd[6908]: child (17637) Started Apr 8 08:13:45 varnish varnishd[6908]: Child (17637) said Child starts Apr 8 08:13:45 varnish varnishd[6908]: Child (17637) said Dropped 10 segments to make free_reserve Apr 8 08:13:48 varnish varnishd[6908]: Child (17637) said Silo completely loaded Any hints? Best regards Markus -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulojjs at gmail.com Mon Apr 8 07:27:43 2013 From: paulojjs at gmail.com (Paulo Silva) Date: Mon, 8 Apr 2013 08:27:43 +0100 Subject: Problem with persistent storage In-Reply-To: <22ECB25A-1E1C-4014-897D-A423CCDC74E2@gmx.de> References: <22ECB25A-1E1C-4014-897D-A423CCDC74E2@gmx.de> Message-ID: Are you sure the persistent cache file is being written in the correct partition? Using ls -lha /proc//fd/ should be able to show you what file is being used. 2013/4/8 Markus Koelle : > Hi, > I've a out-of-space problem with persistent storage (varnish 3.0.3 an Debian > wheezy)) . My storage is 64 GB and it's only half full. > > /var/log/syslog says all 3 hours: > > Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) said Out of space in > persistent silo > Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) said Committing > suicide, restart will make space > Apr 8 08:13:45 varnish varnishd[6908]: Child (6909) ended > Apr 8 08:13:45 varnish varnishd[6908]: Child cleanup complete > Apr 8 08:13:45 varnish varnishd[6908]: child (17637) Started > Apr 8 08:13:45 varnish varnishd[6908]: Child (17637) said Child starts > Apr 8 08:13:45 varnish varnishd[6908]: Child (17637) said Dropped 10 > segments to make free_reserve > Apr 8 08:13:48 varnish varnishd[6908]: Child (17637) said Silo completely > loaded > > Any hints? > > Best regards Markus > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Paulo Silva From connor.walls at skillpages.com Mon Apr 8 13:05:32 2013 From: connor.walls at skillpages.com (Connor Walls) Date: Mon, 8 Apr 2013 14:05:32 +0100 Subject: Varnishncsa, Varnish:time_firstbyte, and nan Message-ID: Hello, We use varnishncsa for logging all the requests that we serve - one of the fields we log is %{Varnish:time_firstbyte}x, measuring the "Time to the first byte from the backend arrived". Now, sometimes we see this value being reported as "nan". Initially I believe this to be the case whenever an object was being served from cache, however, it seems that that is not entirely the case. Here's an example of varnishncsa -F "%{Varnish:time_firstbyte}x %{Varnish:handling}x" 0.012956619 miss 0.009663820 pass nan hit nan miss nan hit 0.223460436 pass 0.000072479 miss 0.000100851 hit 0.000062227 miss So here I see passes and misses with a value (which I expect), hits with "nan" (which I expect), hits with a value (which I don't expect) and misses with "nan" (which I also don't expect). I would just like some clarification as to what it means when "nan" is logged, and under what scenarios this occurs? Could it be related to ESI processing or similar, where some parts of the request are served from cache and other parts aren't? Thanks, Connor Walls From pbrown at nbmedia.com Mon Apr 8 19:10:11 2013 From: pbrown at nbmedia.com (Peter Brown) Date: Mon, 8 Apr 2013 19:10:11 +0000 Subject: Looking for Varnish Consultant Message-ID: <116A5A222B28B84E92C959C5866857C13B249C73@NBNYMAIL2.nbmedia.com> Hi Folks, I hope that this is appropriate to post on this forum. The Varnish Software site pointed me to this list. (https://www.varnish-cache.org/support) I'm looking for a high level Varnish Consultant. I read one web testimony of a person who used Varnish to scale a site up to almost 200 million page views a month. I'd like to find someone who has that level of expertise. Thanks, Peter Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues at betabrand.com Mon Apr 8 20:02:45 2013 From: hugues at betabrand.com (Hugues Alary) Date: Mon, 8 Apr 2013 13:02:45 -0700 Subject: IMS question In-Reply-To: <1365059902.60549.YahooMailNeo@web121701.mail.ne1.yahoo.com> References: <1365039276.48445.YahooMailNeo@web121706.mail.ne1.yahoo.com> <20130404023859.GE21122@nat.myhome> <1365059902.60549.YahooMailNeo@web121701.mail.ne1.yahoo.com> Message-ID: Hi Madhusudan, I believe you can get the IMS branch from github: https://github.com/varnish/Varnish-Cache/tree/experimental-ims -H On Thu, Apr 4, 2013 at 12:18 AM, Madhusudan Ramanna wrote: > Thanks Paul - > > Yes IMS == If-Modified-Since > > I couldn't find IMS branch in repo.varnish-cache.org. I saw source > tarball for streaming but not for IMS branch. Is the IMS branch merged > into trunk ? > > thanks, > Madhu > > ------------------------------ > *From:* Paul A. Procacci > *To:* Madhusudan Ramanna > *Cc:* "varnish-misc at varnish-cache.org" > *Sent:* Wednesday, April 3, 2013 7:38 PM > *Subject:* Re: IMS question > > On Wed, Apr 03, 2013 at 06:34:36PM -0700, Madhusudan Ramanna wrote: > > Hello, > > > > I'm a newbie trying Varnish 3.0.3 setup. ?We need IMS support from > varnish and it looks like IMS changes didn't make it to the release. ?Could > someone point me to any doc on building from source for IMS branch ? We > have both centos and ubuntu machines > > > > Thanks for your help ! > > > > Madhu > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > ?IMS = If-Modified-Since? > > https://www.varnish-cache.org/docs/trunk/installation/install.html > > The documentation provides information on how to download the > development branch of varnish. The docs also cover howto > build from source if you are unfamiliar with that. > > ~Paul > > ________________________________ > > This message may contain confidential or privileged information. If you > are not the intended recipient, please advise us immediately and delete > this message. See http://www.datapipe.com/legal/email_disclaimer/ for > further information on confidentiality and the risks of non-secure > electronic communication. If you cannot access these links, please notify > us by reply message and we will send the contents to you. > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Tue Apr 9 19:05:53 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Tue, 9 Apr 2013 15:05:53 -0400 Subject: Size a file backed cache Message-ID: I know when using malloc storage, that we end up based on our object size, about another 100% percent memory to store additional booking info. So where we have a -s malloc, 6g We need a total of 12g available to the OS. If I switch to -s file, 10g Do I need 20g of HD space available on that drive? I'm unclear how the issues with malloc map to file storage backend. For example, why does file default to 50% sizing of the volume? -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.crosby at cbc.ca Tue Apr 9 23:15:05 2013 From: blake.crosby at cbc.ca (Blake Crosby) Date: Tue, 9 Apr 2013 19:15:05 -0400 Subject: Keeping object in cache regardless of max-age Message-ID: Hello, I'm noticing the following behavior with Varnish. When an object is fetched from the Origin with a TTL of 30 seconds (Cache-Control: max-age=30) the object is immediately flushed from the cache once the 30 second time elapses. This results in Varnish having to issue a "full" HTTP GET to the origin for the object (even if it hasn't changed) when time > 30 seconds. Is it possible for Varnish to keep the object in cache past its TTL and if a future request is made for that object (which is now expired) that varnish will issue a If-Modified-Since (IMS) request. If the origin returns a 304 Not-Modified Varnish will serve the object from cache and reset the TTL counter? For Example: 00:00:00 GET /index.html (TTL 30 seconds) 00:00:30 object expired, but remains in cache 00:00:45 GET /index.html, varnish makes a IMS request to origin. Origin returns 304 00:01:15 object expired, but remains in cache ,,, The LRU rules will still apply for purging objects as the cache fills. This, by the way, is how the caching works with Akamai (which we are currently using). Our origin is specifically tuned to handle a lot of IMS requests (and thus 304s). However when fronting the origin with Varnish, the majority of the requests were 200s as the objects in the cache always expired in 30 seconds. If you need any specific details (VCLs, etc..) let me know. Thanks in advance. -- Blake Crosby Team Lead / Chef D'?quipe Media Production Support / Soutien ? la production des m?dias Canadian Broadcasting Corporation / Societ? Radio-Canada Office/Bureau: 9B122-K * Tel.: (416) 205-3103 * DID: 1-151-3103 From smwood4 at gmail.com Wed Apr 10 00:43:59 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Tue, 9 Apr 2013 17:43:59 -0700 Subject: Size a file backed cache In-Reply-To: References: Message-ID: Ultimately the amount of overhead space needed is determined by the number of objects you have. The overhead is about 1k per object. You can read more at *Sizing Your Cache*on the varnish doc. On Tue, Apr 9, 2013 at 12:05 PM, Sean Allen wrote: > I know when using malloc storage, that we end up based on our object size, > about another 100% percent memory to store additional booking info. So > where we have a > > -s malloc, 6g > > We need a total of 12g available to the OS. > > If I switch to > > -s file, 10g > > Do I need 20g of HD space available on that drive? > I'm unclear how the issues with malloc map to file storage backend. > For example, why does file default to 50% sizing of the volume? > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders at comoyo.com Wed Apr 10 06:41:40 2013 From: anders at comoyo.com (Anders Daljord Morken) Date: Wed, 10 Apr 2013 08:41:40 +0200 Subject: Keeping object in cache regardless of max-age In-Reply-To: References: Message-ID: On Wed, Apr 10, 2013 at 1:15 AM, Blake Crosby wrote: > Is it possible for Varnish to keep the object in cache past its TTL > and if a future request is made for that object (which is now expired) > that varnish will issue a If-Modified-Since (IMS) request. If the > origin returns a 304 Not-Modified Varnish will serve the object from > cache and reset the TTL counter? > Possible, well, sort of. Some work has been made on this, but it hasn't been merged into mainline, let alone released. See https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests for more information. The Varnish devs may have more information, but I don't think we can expect a timeline commitment yet. - at M -- "Sooner or later we're going to run out of things to go wrong, and then it'll work great." - Jamie Hyneman, The Mythbusters -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.fletcher at replyltd.co.uk Wed Apr 10 10:53:22 2013 From: a.fletcher at replyltd.co.uk (Fletcher Andrew) Date: Wed, 10 Apr 2013 10:53:22 +0000 Subject: setting cookies from within esi included content Message-ID: <668EAF2DA3FC334AB0824787BF2FDFD707704000@CED01MBXS05.replynet.prv> Hi, I'm using Varnish to cache a shopping site that has a mini-basket on every page. I'm caching the parent page and including the basket via ESI so that it is never cached and always gets retrieved from the backend. The mini basket is what sets the session id cookie but Varnish's ESI implementation doesn't merge the response headers of the included page with the parent before sending back to the browser. Ideally what I'd like is to cache the parent page without any session id cookies, retrieve that from the cache, retrieve the included content and add the Set-Cookie headers of the included response to the main/parent response before delivering to the browser. Is there a roadmap for Varnish to add support for merging cookies from included responses? As a workaround I'm including a file in the page (as a css file but could be 1px image, js file etc) that is never cached by Varnish (sets Cache-Control: max-age=0) and returns the required Set-Cookie header for the session id cookie. So the browser requests a page from Varnish, loads the resources of the page (the first of which is my special css) and sets the session id cookie. On subsequent requests it sends the cookie in the request which gets passed to the ESI request for the mini-basket and the correct content is included into the page before being passed back to the browser. Can anyone see any problems with this workaround? It appears to be working well for us at the moment, however I can see it creating multiple redundant sessions. The first request includes the basket via ESI, which creates a new session (that is never used again because the session id is not passed back to the browser) and then the browser makes another request for cookies.css that creates another new session (which is used from then on) I'm also concerned that some browsers and other proxy servers may not honour the Cache-Control header and attempt to always cache my cookies.css file. Thanks for any input, Andrew ________________________________ This is an electronic communication from Reply Limited, any opinions expressed in this email are those of the individual and not necessarily those of Reply Limited. The information in this email and any attachments is confidential and may be subject to legal professional privilege. it is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, please notify the sender immediately. Unless you are the intended recipient or his/her representative you are not authorised to, and must not, read, copy, distribute, use or retain this message or any part of it. At present the integrity of e-mail across the internet cannot be guaranteed and message and documents sent via this medium are potentially at risk. Please note that neither the sender nor Reply Limited accepts any responsibility for viruses and it is your responsibility to scan any attachments. All liability is excluded to the extent permitted by the law for any claims arising from the use of this medium by Reply Limited. Registered address: Reply Ltd, 38 Grosvenor Gardens, London, SW1W 0EB [Registered in England and Wales No: 3847202]. VAT No: 742468814. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Wed Apr 10 11:50:26 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Wed, 10 Apr 2013 07:50:26 -0400 Subject: Size a file backed cache In-Reply-To: References: Message-ID: I understand that. Let me rephrase that question. When using file storage, am I correct in assuming, that the overhead uses additional space on the volume that the file itself is on? On Tue, Apr 9, 2013 at 8:43 PM, Stephen Wood wrote: > Ultimately the amount of overhead space needed is determined by the number > of objects you have. The overhead is about 1k per object. You can read more > at *Sizing Your Cache*on the varnish doc. > > > On Tue, Apr 9, 2013 at 12:05 PM, Sean Allen wrote: > >> I know when using malloc storage, that we end up based on our object >> size, about another 100% percent memory to store additional booking info. >> So where we have a >> >> -s malloc, 6g >> >> We need a total of 12g available to the OS. >> >> If I switch to >> >> -s file, 10g >> >> Do I need 20g of HD space available on that drive? >> I'm unclear how the issues with malloc map to file storage backend. >> For example, why does file default to 50% sizing of the volume? >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Apr 10 12:04:51 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 10 Apr 2013 14:04:51 +0200 Subject: Size a file backed cache In-Reply-To: References: Message-ID: No. The overhead is in the form of memory structures. On Wed, Apr 10, 2013 at 1:50 PM, Sean Allen wrote: > I understand that. Let me rephrase that question. When using file storage, > am I correct in assuming, that the overhead uses additional space on the > volume that the file itself is on? > > > On Tue, Apr 9, 2013 at 8:43 PM, Stephen Wood wrote: > >> Ultimately the amount of overhead space needed is determined by the >> number of objects you have. The overhead is about 1k per object. You can >> read more at *Sizing Your Cache*on the varnish doc. >> >> >> On Tue, Apr 9, 2013 at 12:05 PM, Sean Allen wrote: >> >>> I know when using malloc storage, that we end up based on our object >>> size, about another 100% percent memory to store additional booking info. >>> So where we have a >>> >>> -s malloc, 6g >>> >>> We need a total of 12g available to the OS. >>> >>> If I switch to >>> >>> -s file, 10g >>> >>> Do I need 20g of HD space available on that drive? >>> I'm unclear how the issues with malloc map to file storage backend. >>> For example, why does file default to 50% sizing of the volume? >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > > > -- > > Ce n'est pas une signature > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdgraff at gmail.com Wed Apr 10 12:36:39 2013 From: cdgraff at gmail.com (Alejandro) Date: Wed, 10 Apr 2013 09:36:39 -0300 Subject: Looking for Varnish Consultant In-Reply-To: <116A5A222B28B84E92C959C5866857C13B249C73@NBNYMAIL2.nbmedia.com> References: <116A5A222B28B84E92C959C5866857C13B249C73@NBNYMAIL2.nbmedia.com> Message-ID: Peter, Please email me, I have 3 year of experience with Varnish in multiple HighTraffic sites, I have server 1.5Millon pageviews in 1 Hours, this was only possible using Varnish. here some prof about this: http://bit.ly/12JqQwu (SPANISH) Thanks, Ale 2013/4/8 Peter Brown > Hi Folks,**** > > ** ** > > I hope that this is appropriate to post on this forum.**** > > The Varnish Software site pointed me to this list.**** > > (https://www.varnish-cache.org/support)**** > > ** ** > > I'm looking for a high level Varnish Consultant. **** > > ** ** > > I read one web testimony of a person who used Varnish to scale a site up > to almost 200 million page views a month.**** > > I'd like to find someone who has that level of expertise.**** > > ** ** > > Thanks,**** > > ** ** > > Peter Brown**** > > ** ** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.crosby at cbc.ca Wed Apr 10 13:02:34 2013 From: blake.crosby at cbc.ca (Blake Crosby) Date: Wed, 10 Apr 2013 09:02:34 -0400 Subject: Keeping object in cache regardless of max-age In-Reply-To: References: Message-ID: <516562EA.2090101@cbc.ca> Anders, Thanks for the link. This helps. I'm going to put my vote in for this to be at least discussed at next months User Group meeting. I just took a look through the list archives, and it seems like I'm not the only one looking for this functionality :) Blake On 10/04/2013 02:41, Anders Daljord Morken wrote: > On Wed, Apr 10, 2013 at 1:15 AM, Blake Crosby > wrote: > > Is it possible for Varnish to keep the object in cache past its TTL > and if a future request is made for that object (which is now expired) > that varnish will issue a If-Modified-Since (IMS) request. If the > origin returns a 304 Not-Modified Varnish will serve the object from > cache and reset the TTL counter? > > > Possible, well, sort of. Some work has been made on this, but it hasn't > been merged into mainline, let alone released. > See > https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests for > more information. The Varnish devs may have more information, but I > don't think we can expect a timeline commitment yet. > From Raul.Rangel at disney.com Wed Apr 10 14:19:42 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Wed, 10 Apr 2013 07:19:42 -0700 Subject: setting cookies from within esi included content In-Reply-To: <668EAF2DA3FC334AB0824787BF2FDFD707704000@CED01MBXS05.replynet.prv> References: <668EAF2DA3FC334AB0824787BF2FDFD707704000@CED01MBXS05.replynet.prv> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2610CEE610@SM-CALA-VXMB04A.swna.wdpr.disney.com> I don't think merging headers is on the road map. It would require the parent request block until all the ESIs have loaded which would be bad if you have a bunch of ESIs below the fold. One work around that would work for your cookie issue is to make the parent page be the page that sends the Set-Cookie header and then have the body of it only include an esi. i.e) Set-Cookie: abc=123 This way you can cache the /full_page ESI object but still set unique cookies for each user. We make heavy use of this technique and have not had any issues with it. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Fletcher Andrew Sent: Wednesday, April 10, 2013 4:53 AM To: varnish-misc at varnish-cache.org Subject: setting cookies from within esi included content Hi, I'm using Varnish to cache a shopping site that has a mini-basket on every page. I'm caching the parent page and including the basket via ESI so that it is never cached and always gets retrieved from the backend. The mini basket is what sets the session id cookie but Varnish's ESI implementation doesn't merge the response headers of the included page with the parent before sending back to the browser. Ideally what I'd like is to cache the parent page without any session id cookies, retrieve that from the cache, retrieve the included content and add the Set-Cookie headers of the included response to the main/parent response before delivering to the browser. Is there a roadmap for Varnish to add support for merging cookies from included responses? As a workaround I'm including a file in the page (as a css file but could be 1px image, js file etc) that is never cached by Varnish (sets Cache-Control: max-age=0) and returns the required Set-Cookie header for the session id cookie. So the browser requests a page from Varnish, loads the resources of the page (the first of which is my special css) and sets the session id cookie. On subsequent requests it sends the cookie in the request which gets passed to the ESI request for the mini-basket and the correct content is included into the page before being passed back to the browser. Can anyone see any problems with this workaround? It appears to be working well for us at the moment, however I can see it creating multiple redundant sessions. The first request includes the basket via ESI, which creates a new session (that is never used again because the session id is not passed back to the browser) and then the browser makes another request for cookies.css that creates another new session (which is used from then on) I'm also concerned that some browsers and other proxy servers may not honour the Cache-Control header and attempt to always cache my cookies.css file. Thanks for any input, Andrew ________________________________ This is an electronic communication from Reply Limited, any opinions expressed in this email are those of the individual and not necessarily those of Reply Limited. The information in this email and any attachments is confidential and may be subject to legal professional privilege. it is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, please notify the sender immediately. Unless you are the intended recipient or his/her representative you are not authorised to, and must not, read, copy, distribute, use or retain this message or any part of it. At present the integrity of e-mail across the internet cannot be guaranteed and message and documents sent via this medium are potentially at risk. Please note that neither the sender nor Reply Limited accepts any responsibility for viruses and it is your responsibility to scan any attachments. All liability is excluded to the extent permitted by the law for any claims arising from the use of this medium by Reply Limited. Registered address: Reply Ltd, 38 Grosvenor Gardens, London, SW1W 0EB [Registered in England and Wales No: 3847202]. VAT No: 742468814. -------------- next part -------------- An HTML attachment was scrubbed... URL: From themadindian at yahoo.com Wed Apr 10 14:36:04 2013 From: themadindian at yahoo.com (Mad Indian) Date: Wed, 10 Apr 2013 07:36:04 -0700 (PDT) Subject: Keeping object in cache regardless of max-age In-Reply-To: <516562EA.2090101@cbc.ca> References: <516562EA.2090101@cbc.ca> Message-ID: <1365604564.10318.YahooMailNeo@web163905.mail.gq1.yahoo.com> unless I'm misunderstanding in the vcl_fetch section you should be able to for instance if ttl is less and 2 mins make it 2 mins if (beresp.ttl < 120s) { set beresp.ttl = 120s; } or if you want to say cache any url that contains members in it for ?3h if (req.url ~ "(/members/)") { ?set beresp.ttl = 3h; } ________________________________ From: Blake Crosby To: varnish-misc at varnish-cache.org Sent: Wednesday, April 10, 2013 9:02 AM Subject: Re: Keeping object in cache regardless of max-age Anders, Thanks for the link. This helps. I'm going to put my vote in for this to be at least discussed at next months User Group meeting. I just took a look through the list archives, and it seems like I'm not the only one looking for this functionality :) Blake On 10/04/2013 02:41, Anders Daljord Morken wrote: > On Wed, Apr 10, 2013 at 1:15 AM, Blake Crosby > wrote: > >? ? Is it possible for Varnish to keep the object in cache past its TTL >? ? and if a future request is made for that object (which is now expired) >? ? that varnish will issue a If-Modified-Since (IMS) request. If the >? ? origin returns a 304 Not-Modified Varnish will serve the object from >? ? cache and reset the TTL counter? > > > Possible, well, sort of. Some work has been made on this, but it hasn't > been merged into mainline, let alone released. > See > https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests for > more information. The Varnish devs may have more information, but I > don't think we can expect a timeline commitment yet. > _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.fletcher at replyltd.co.uk Wed Apr 10 14:47:58 2013 From: a.fletcher at replyltd.co.uk (Fletcher Andrew) Date: Wed, 10 Apr 2013 14:47:58 +0000 Subject: setting cookies from within esi included content In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F2610CEE610@SM-CALA-VXMB04A.swna.wdpr.disney.com> References: <668EAF2DA3FC334AB0824787BF2FDFD707704000@CED01MBXS05.replynet.prv> <2465AAEEC8B8A242B26ED5F44BCA805F2610CEE610@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: <668EAF2DA3FC334AB0824787BF2FDFD70770417A@CED01MBXS05.replynet.prv> Hi Raul, Thanks for the suggestion. That sounds like a much better solution and doesn't rely on the browser issuing a separate request to set cookies. When you say that it would require the parent request to block does that mean that Varnish starts sending the response from the cache until it hits an esi tag, at which point it waits for the included content from the backend but then carries on sending the content after that? Most of our dynamic content that is included via ESI is right at the top of the page (e.g. mini-basket, welcome 'user' message etc.) so it's going to be waiting for that whatever but I can see why for ESI lower down it wouldn't be optimal. Thanks again, Andrew From: Rangel, Raul [mailto:Raul.Rangel at disney.com] Sent: 10 April 2013 15:20 To: Fletcher Andrew; 'varnish-misc at varnish-cache.org' Subject: RE: setting cookies from within esi included content I don't think merging headers is on the road map. It would require the parent request block until all the ESIs have loaded which would be bad if you have a bunch of ESIs below the fold. One work around that would work for your cookie issue is to make the parent page be the page that sends the Set-Cookie header and then have the body of it only include an esi. i.e) Set-Cookie: abc=123 This way you can cache the /full_page ESI object but still set unique cookies for each user. We make heavy use of this technique and have not had any issues with it. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Fletcher Andrew Sent: Wednesday, April 10, 2013 4:53 AM To: varnish-misc at varnish-cache.org Subject: setting cookies from within esi included content Hi, I'm using Varnish to cache a shopping site that has a mini-basket on every page. I'm caching the parent page and including the basket via ESI so that it is never cached and always gets retrieved from the backend. The mini basket is what sets the session id cookie but Varnish's ESI implementation doesn't merge the response headers of the included page with the parent before sending back to the browser. Ideally what I'd like is to cache the parent page without any session id cookies, retrieve that from the cache, retrieve the included content and add the Set-Cookie headers of the included response to the main/parent response before delivering to the browser. Is there a roadmap for Varnish to add support for merging cookies from included responses? As a workaround I'm including a file in the page (as a css file but could be 1px image, js file etc) that is never cached by Varnish (sets Cache-Control: max-age=0) and returns the required Set-Cookie header for the session id cookie. So the browser requests a page from Varnish, loads the resources of the page (the first of which is my special css) and sets the session id cookie. On subsequent requests it sends the cookie in the request which gets passed to the ESI request for the mini-basket and the correct content is included into the page before being passed back to the browser. Can anyone see any problems with this workaround? It appears to be working well for us at the moment, however I can see it creating multiple redundant sessions. The first request includes the basket via ESI, which creates a new session (that is never used again because the session id is not passed back to the browser) and then the browser makes another request for cookies.css that creates another new session (which is used from then on) I'm also concerned that some browsers and other proxy servers may not honour the Cache-Control header and attempt to always cache my cookies.css file. Thanks for any input, Andrew ________________________________ This is an electronic communication from Reply Limited, any opinions expressed in this email are those of the individual and not necessarily those of Reply Limited. The information in this email and any attachments is confidential and may be subject to legal professional privilege. it is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, please notify the sender immediately. Unless you are the intended recipient or his/her representative you are not authorised to, and must not, read, copy, distribute, use or retain this message or any part of it. At present the integrity of e-mail across the internet cannot be guaranteed and message and documents sent via this medium are potentially at risk. Please note that neither the sender nor Reply Limited accepts any responsibility for viruses and it is your responsibility to scan any attachments. All liability is excluded to the extent permitted by the law for any claims arising from the use of this medium by Reply Limited. Registered address: Reply Ltd, 38 Grosvenor Gardens, London, SW1W 0EB [Registered in England and Wales No: 3847202]. VAT No: 742468814. ________________________________ This is an electronic communication from Reply Limited, any opinions expressed in this email are those of the individual and not necessarily those of Reply Limited. The information in this email and any attachments is confidential and may be subject to legal professional privilege. it is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, please notify the sender immediately. Unless you are the intended recipient or his/her representative you are not authorised to, and must not, read, copy, distribute, use or retain this message or any part of it. At present the integrity of e-mail across the internet cannot be guaranteed and message and documents sent via this medium are potentially at risk. Please note that neither the sender nor Reply Limited accepts any responsibility for viruses and it is your responsibility to scan any attachments. All liability is excluded to the extent permitted by the law for any claims arising from the use of this medium by Reply Limited. Registered address: Reply Ltd, 38 Grosvenor Gardens, London, SW1W 0EB [Registered in England and Wales No: 3847202]. VAT No: 742468814. -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.crosby at cbc.ca Wed Apr 10 14:53:15 2013 From: blake.crosby at cbc.ca (Blake Crosby) Date: Wed, 10 Apr 2013 10:53:15 -0400 Subject: Keeping object in cache regardless of max-age In-Reply-To: <1365604564.10318.YahooMailNeo@web163905.mail.gq1.yahoo.com> References: <516562EA.2090101@cbc.ca> <1365604564.10318.YahooMailNeo@web163905.mail.gq1.yahoo.com> Message-ID: <51657CDB.9030504@cbc.ca> The problem is not the cache time. but the fact that Varnish immediately purges the object from cache after the TTL has expired. This means that regardless if the object has changed or not, another request must be made to the backend to get a new copy. Instead of just marking the object as invalidated and using an if-modified-since (conditional GET) to see if it should update the cache copy or not. Blake On 10/04/2013 10:36, Mad Indian wrote: > unless I'm misunderstanding in the vcl_fetch section you should be able > to for instance if ttl is less and 2 mins make it 2 mins > > if (beresp.ttl < 120s) { > set beresp.ttl = 120s; > } > > or if you want to say cache any url that contains members in it for 3h > > if (req.url ~ "(/members/)") { > set beresp.ttl = 3h; > } > > > ------------------------------------------------------------------------ > *From:* Blake Crosby > *To:* varnish-misc at varnish-cache.org > *Sent:* Wednesday, April 10, 2013 9:02 AM > *Subject:* Re: Keeping object in cache regardless of max-age > > Anders, > > Thanks for the link. This helps. > > I'm going to put my vote in for this to be at least discussed at next > months User Group meeting. I just took a look through the list archives, > and it seems like I'm not the only one looking for this functionality :) > > Blake > > > On 10/04/2013 02:41, Anders Daljord Morken wrote: > > On Wed, Apr 10, 2013 at 1:15 AM, Blake Crosby > > >> wrote: > > > > Is it possible for Varnish to keep the object in cache past its TTL > > and if a future request is made for that object (which is now expired) > > that varnish will issue a If-Modified-Since (IMS) request. If the > > origin returns a 304 Not-Modified Varnish will serve the object from > > cache and reset the TTL counter? > > > > > > Possible, well, sort of. Some work has been made on this, but it hasn't > > been merged into mainline, let alone released. > > See > > https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests for > > more information. The Varnish devs may have more information, but I > > don't think we can expect a timeline commitment yet. > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > From itxx00 at gmail.com Thu Apr 11 01:25:33 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Thu, 11 Apr 2013 09:25:33 +0800 Subject: varnishd child panic Message-ID: Hello,lists. I an using a varnish for testing on a CentOS6.3 box, some basic info: [root at cdn001 ~]# uname -r 2.6.32-358.2.1.el6.x86_64 [root at cdn001 ~]# rpm -q varnish varnish-3.0.3-3.el6.art.x86_64 [root at cdn001 ~]# free -m total used free shared buffers cached Mem: 24023 1054 22968 0 137 240 -/+ buffers/cache: 676 23347 Swap: 12079 0 12079 uptime: 09:05:24 up 6 days, 11:29, 1 user, load average: 0.00, 0.00, 0.00 and my /etc/sysconfig/varnish looks like below: WORKER_STACK_SIZE=512 ... ... VARNISH_STORAGE_SIZE=16G VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" VARNISH_TTL=900 thread_pools=2 thread_pool_min=500 thread_pool_max=4000 thread_pool_timeout=120 thread_pool_add_delay=2 thread_pool_fail_delay=100 sess_workspace=32768 session_max=500000 thread_pool_stack=16384 connect_timeout=10 first_byte_timeout=60 between_bytes_timeout=60 DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -u varnish -g varnish \ -S ${VARNISH_SECRET_FILE} \ -p thread_pools=${thread_pools} \ -p thread_pool_min=${thread_pool_min} \ -p thread_pool_max=${thread_pool_max} \ -p thread_pool_timeout=${thread_pool_timeout} \ -p thread_pool_add_delay=${thread_pool_add_delay} \ -p thread_pool_fail_delay=${thread_pool_fail_delay} \ -p sess_workspace=${sess_workspace} \ -p session_max=${session_max} \ -p connect_timeout=${connect_timeout} \ -p first_byte_timeout=${first_byte_timeout} \ -p between_bytes_timeout=${between_bytes_timeout} \ -s ${VARNISH_STORAGE}" as my backend web servers runs in many vms,so I use ' dns director': director dnsdomain dns { .list = { .port = "80"; "10.0.0.0"/24; } .ttl = 12h; } and the connect_timeout in /etc/sysconfig/varnish has been set to a large number(10 s). everything works good since last night, I got a panic message in system log. By default I have 1000 workers on startup, but when this panic occur,I can just say 500+ workers using varnishstat command. bellow is the panic message in syslog: Apr 10 22:12:58 cdn001 varnishd[25730]: Child (25731) Panic message: Assert error in VRT_IP_string(), cache_vrt.c line 312:#012 Condition((p = WS_Alloc(sp->http->ws, len)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x42ee88: /usr/sbin/varnishd() [0x42ee88]#012 0x436dc5: /usr/sbin/varnishd(VRT_IP_string+0x135) [0x436dc5]#012 0x7f56bc4b922f: ./ vcl.xlWKkvTA.so(+0xb622f) [0x7f56bc4b922f]#012 0x436203: /usr/sbin/varnishd(VCL_recv_method+0x43) [0x436203]#012 0x418eaf: /usr/sbin/varnishd(CNT_Session+0xb7f) [0x418eaf]#012 0x430bd1: /usr/sbin/varnishd() [0x430bd1]#012 0x7f56c388f851: /lib64/libpthread.so.0(+0x7851) [0x7f56c388f851]#012 0x7f56c35dd90d: /lib64/libc.so.6(clone+0x6d) [0x7f56c35dd90d]#012sp = 0x7f569b089008 {#012 fd = 320, id = 320, xid = 1298044854,#012 client = 10.0.0.170 33991,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 404, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 3#012 ws = 0x7f569b089080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7f569b089c78,+32768,(nil),+32768},#012 },#012 http[req] = {#012 ws = 0x7f569b089080[sess]#012 "GET",#012 "/images/ico_hots2.gif",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera Mini/6.7.30171/29.3222; U; zh) Presto/2.8.119 Version/11.10",#012 "Host: www.example.com",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: zh-cn,en;q=0.9",#012 "Accept-Encoding: gzip, deflate",#012 "Referer: http://www.example.com/",#012 "Connection: Keep-Alive",#012 "clientip: 117.149.35.78",#012 "X-OperaMini-Features: advanced, file_system, camera, touch, folding, viewport",#012 "Device-Stock-UA: Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; BOWAY I5 Build/MocorDroid2.3.5) AppleWebKit/533.1 Apr 10 22:12:58 cdn001 varnishd[25730]: child (7270) Started Apr 10 22:12:59 cdn001 kernel: varnishd[7270]: segfault at 0 ip 000000000041067d sp 00007fff4f027a40 error 6 in varnishd[400000+70000] Apr 10 22:12:59 cdn001 varnishd[25730]: Pushing vcls failed:#012CLI communication error (hdr) Apr 10 22:12:59 cdn001 varnishd[25730]: Child (7270) died signal=11 Apr 10 22:12:59 cdn001 varnishd[25730]: Child (-1) said Child starts I have restarted varnish daemon and things looks fine now, but I am thinking that how could this error happen? is there something wrong in my varnish configuration ? or something else ? thanks . -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctoomey at gmail.com Thu Apr 11 02:18:43 2013 From: ctoomey at gmail.com (Chris Toomey) Date: Wed, 10 Apr 2013 19:18:43 -0700 Subject: Stale-while-revalidate feature Message-ID: Hi, We're using varnish and also would very much like to have the stale-while-revalidate feature that's been discussed on this list, wherein the first request for an expired object during the grace period gets served the stale content vs. waiting for the fresh content, and a background thread is spawned to asynchronous refresh the cache. Like others, we've got some slow backend servers that really penalize the poor users who have to wait for them when the cache is expired, and the various workaround options we've explored are quite hacky. I saw in https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-June/022104.html that this feature might be coming. Is this looking like it's going to make it into the next stable release? thanks much, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Thu Apr 11 05:59:12 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 11 Apr 2013 07:59:12 +0200 Subject: varnishd child panic In-Reply-To: References: Message-ID: Hi, According to the log you provided, I understand that you have a failed allocation in your worker thread (WS_Alloc returns NULL). Check the n_objoverflow counter in varnishstat. You can control the size of your workspace with the sess_workspace. The more you do in your VCL, the more memory you'll likely need for workers. Best Regards, Dridi On Thu, Apr 11, 2013 at 3:25 AM, Gao Yongwei wrote: > Hello,lists. > I an using a varnish for testing on a CentOS6.3 box, some basic info: > [root at cdn001 ~]# uname -r > 2.6.32-358.2.1.el6.x86_64 > [root at cdn001 ~]# rpm -q varnish > varnish-3.0.3-3.el6.art.x86_64 > [root at cdn001 ~]# free -m > total used free shared buffers cached > Mem: 24023 1054 22968 0 137 240 > -/+ buffers/cache: 676 23347 > Swap: 12079 0 12079 > uptime: > 09:05:24 up 6 days, 11:29, 1 user, load average: 0.00, 0.00, 0.00 > > and my /etc/sysconfig/varnish looks like below: > > WORKER_STACK_SIZE=512 > ... ... > VARNISH_STORAGE_SIZE=16G > VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" > VARNISH_TTL=900 > thread_pools=2 > thread_pool_min=500 > thread_pool_max=4000 > thread_pool_timeout=120 > thread_pool_add_delay=2 > thread_pool_fail_delay=100 > sess_workspace=32768 > session_max=500000 > thread_pool_stack=16384 > connect_timeout=10 > first_byte_timeout=60 > between_bytes_timeout=60 > DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ > -f ${VARNISH_VCL_CONF} \ > -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} > \ > -t ${VARNISH_TTL} \ > -u varnish -g varnish \ > -S ${VARNISH_SECRET_FILE} \ > -p thread_pools=${thread_pools} \ > -p thread_pool_min=${thread_pool_min} \ > -p thread_pool_max=${thread_pool_max} \ > -p thread_pool_timeout=${thread_pool_timeout} \ > -p thread_pool_add_delay=${thread_pool_add_delay} \ > -p thread_pool_fail_delay=${thread_pool_fail_delay} \ > -p sess_workspace=${sess_workspace} \ > -p session_max=${session_max} \ > -p connect_timeout=${connect_timeout} \ > -p first_byte_timeout=${first_byte_timeout} \ > -p between_bytes_timeout=${between_bytes_timeout} \ > -s ${VARNISH_STORAGE}" > > as my backend web servers runs in many vms,so I use ' dns director': > > director dnsdomain dns { > .list = { > .port = "80"; > "10.0.0.0"/24; > } > .ttl = 12h; > } > > and the connect_timeout in /etc/sysconfig/varnish has been set to a large > number(10 s). > everything works good since last night, I got a panic message in system log. > By default I have 1000 workers on startup, but when this panic occur,I can > just say 500+ workers > using varnishstat command. bellow is the panic message in syslog: > > Apr 10 22:12:58 cdn001 varnishd[25730]: Child (25731) Panic message: Assert > error in VRT_IP_string(), cache_vrt.c line 312:#012 Condition((p = > WS_Alloc(sp->http->ws, len)) != 0) not true.#012thread = > (cache-worker)#012ident = > Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 > 0x42ee88: /usr/sbin/varnishd() [0x42ee88]#012 0x436dc5: > /usr/sbin/varnishd(VRT_IP_string+0x135) [0x436dc5]#012 0x7f56bc4b922f: > ./vcl.xlWKkvTA.so(+0xb622f) [0x7f56bc4b922f]#012 0x436203: > /usr/sbin/varnishd(VCL_recv_method+0x43) [0x436203]#012 0x418eaf: > /usr/sbin/varnishd(CNT_Session+0xb7f) [0x418eaf]#012 0x430bd1: > /usr/sbin/varnishd() [0x430bd1]#012 0x7f56c388f851: > /lib64/libpthread.so.0(+0x7851) [0x7f56c388f851]#012 0x7f56c35dd90d: > /lib64/libc.so.6(clone+0x6d) [0x7f56c35dd90d]#012sp = 0x7f569b089008 {#012 > fd = 320, id = 320, xid = 1298044854,#012 client = 10.0.0.170 33991,#012 > step = STP_RECV,#012 handling = deliver,#012 err_code = 404, err_reason = > (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = > 3#012 ws = 0x7f569b089080 { overflow#012 id = "sess",#012 {s,f,r,e} = > {0x7f569b089c78,+32768,(nil),+32768},#012 },#012 http[req] = {#012 ws = > 0x7f569b089080[sess]#012 "GET",#012 "/images/ico_hots2.gif",#012 > "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera > Mini/6.7.30171/29.3222; U; zh) Presto/2.8.119 Version/11.10",#012 > "Host: www.example.com",#012 "Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, > image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: zh-cn,en;q=0.9",#012 > "Accept-Encoding: gzip, deflate",#012 "Referer: > http://www.example.com/",#012 "Connection: Keep-Alive",#012 > "clientip: 117.149.35.78",#012 "X-OperaMini-Features: advanced, > file_system, camera, touch, folding, viewport",#012 "Device-Stock-UA: > Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; BOWAY I5 Build/MocorDroid2.3.5) > AppleWebKit/533.1 > Apr 10 22:12:58 cdn001 varnishd[25730]: child (7270) Started > Apr 10 22:12:59 cdn001 kernel: varnishd[7270]: segfault at 0 ip > 000000000041067d sp 00007fff4f027a40 error 6 in varnishd[400000+70000] > Apr 10 22:12:59 cdn001 varnishd[25730]: Pushing vcls failed:#012CLI > communication error (hdr) > Apr 10 22:12:59 cdn001 varnishd[25730]: Child (7270) died signal=11 > Apr 10 22:12:59 cdn001 varnishd[25730]: Child (-1) said Child starts > > I have restarted varnish daemon and things looks fine now, but I am > thinking that how could this error happen? is there something wrong in my > varnish configuration ? or something else ? > > thanks . > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From itxx00 at gmail.com Thu Apr 11 10:40:49 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Thu, 11 Apr 2013 18:40:49 +0800 Subject: varnishd child panic In-Reply-To: References: Message-ID: Thanks for your reply Dridi,I will try to increase the sess_workerspace to see what will happen in next day, thanks. 2013/4/11 Dridi Boukelmoune > Hi, > > According to the log you provided, I understand that you have a failed > allocation in your worker thread (WS_Alloc returns NULL). Check the > n_objoverflow counter in varnishstat. You can control the size of your > workspace with the sess_workspace. The more you do in your VCL, the > more memory you'll likely need for workers. > > Best Regards, > Dridi > > On Thu, Apr 11, 2013 at 3:25 AM, Gao Yongwei wrote: > > Hello,lists. > > I an using a varnish for testing on a CentOS6.3 box, some basic info: > > [root at cdn001 ~]# uname -r > > 2.6.32-358.2.1.el6.x86_64 > > [root at cdn001 ~]# rpm -q varnish > > varnish-3.0.3-3.el6.art.x86_64 > > [root at cdn001 ~]# free -m > > total used free shared buffers cached > > Mem: 24023 1054 22968 0 137 240 > > -/+ buffers/cache: 676 23347 > > Swap: 12079 0 12079 > > uptime: > > 09:05:24 up 6 days, 11:29, 1 user, load average: 0.00, 0.00, 0.00 > > > > and my /etc/sysconfig/varnish looks like below: > > > > WORKER_STACK_SIZE=512 > > ... ... > > VARNISH_STORAGE_SIZE=16G > > VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" > > VARNISH_TTL=900 > > thread_pools=2 > > thread_pool_min=500 > > thread_pool_max=4000 > > thread_pool_timeout=120 > > thread_pool_add_delay=2 > > thread_pool_fail_delay=100 > > sess_workspace=32768 > > session_max=500000 > > thread_pool_stack=16384 > > connect_timeout=10 > > first_byte_timeout=60 > > between_bytes_timeout=60 > > DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ > > -f ${VARNISH_VCL_CONF} \ > > -T > ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} > > \ > > -t ${VARNISH_TTL} \ > > -u varnish -g varnish \ > > -S ${VARNISH_SECRET_FILE} \ > > -p thread_pools=${thread_pools} \ > > -p thread_pool_min=${thread_pool_min} \ > > -p thread_pool_max=${thread_pool_max} \ > > -p thread_pool_timeout=${thread_pool_timeout} \ > > -p thread_pool_add_delay=${thread_pool_add_delay} \ > > -p thread_pool_fail_delay=${thread_pool_fail_delay} \ > > -p sess_workspace=${sess_workspace} \ > > -p session_max=${session_max} \ > > -p connect_timeout=${connect_timeout} \ > > -p first_byte_timeout=${first_byte_timeout} \ > > -p between_bytes_timeout=${between_bytes_timeout} \ > > -s ${VARNISH_STORAGE}" > > > > as my backend web servers runs in many vms,so I use ' dns director': > > > > director dnsdomain dns { > > .list = { > > .port = "80"; > > "10.0.0.0"/24; > > } > > .ttl = 12h; > > } > > > > and the connect_timeout in /etc/sysconfig/varnish has been set to a > large > > number(10 s). > > everything works good since last night, I got a panic message in system > log. > > By default I have 1000 workers on startup, but when this panic occur,I > can > > just say 500+ workers > > using varnishstat command. bellow is the panic message in syslog: > > > > Apr 10 22:12:58 cdn001 varnishd[25730]: Child (25731) Panic message: > Assert > > error in VRT_IP_string(), cache_vrt.c line 312:#012 Condition((p = > > WS_Alloc(sp->http->ws, len)) != 0) not true.#012thread = > > (cache-worker)#012ident = > > > Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 > > 0x42ee88: /usr/sbin/varnishd() [0x42ee88]#012 0x436dc5: > > /usr/sbin/varnishd(VRT_IP_string+0x135) [0x436dc5]#012 0x7f56bc4b922f: > > ./vcl.xlWKkvTA.so(+0xb622f) [0x7f56bc4b922f]#012 0x436203: > > /usr/sbin/varnishd(VCL_recv_method+0x43) [0x436203]#012 0x418eaf: > > /usr/sbin/varnishd(CNT_Session+0xb7f) [0x418eaf]#012 0x430bd1: > > /usr/sbin/varnishd() [0x430bd1]#012 0x7f56c388f851: > > /lib64/libpthread.so.0(+0x7851) [0x7f56c388f851]#012 0x7f56c35dd90d: > > /lib64/libc.so.6(clone+0x6d) [0x7f56c35dd90d]#012sp = 0x7f569b089008 > {#012 > > fd = 320, id = 320, xid = 1298044854,#012 client = 10.0.0.170 33991,#012 > > step = STP_RECV,#012 handling = deliver,#012 err_code = 404, > err_reason = > > (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = > > 3#012 ws = 0x7f569b089080 { overflow#012 id = "sess",#012 > {s,f,r,e} = > > {0x7f569b089c78,+32768,(nil),+32768},#012 },#012 http[req] = {#012 > ws = > > 0x7f569b089080[sess]#012 "GET",#012 > "/images/ico_hots2.gif",#012 > > "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera > > Mini/6.7.30171/29.3222; U; zh) Presto/2.8.119 Version/11.10",#012 > > "Host: www.example.com",#012 "Accept: text/html, > application/xml;q=0.9, > > application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, > > image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: > zh-cn,en;q=0.9",#012 > > "Accept-Encoding: gzip, deflate",#012 "Referer: > > http://www.example.com/",#012 "Connection: Keep-Alive",#012 > > "clientip: 117.149.35.78",#012 "X-OperaMini-Features: advanced, > > file_system, camera, touch, folding, viewport",#012 > "Device-Stock-UA: > > Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; BOWAY I5 > Build/MocorDroid2.3.5) > > AppleWebKit/533.1 > > Apr 10 22:12:58 cdn001 varnishd[25730]: child (7270) Started > > Apr 10 22:12:59 cdn001 kernel: varnishd[7270]: segfault at 0 ip > > 000000000041067d sp 00007fff4f027a40 error 6 in varnishd[400000+70000] > > Apr 10 22:12:59 cdn001 varnishd[25730]: Pushing vcls failed:#012CLI > > communication error (hdr) > > Apr 10 22:12:59 cdn001 varnishd[25730]: Child (7270) died signal=11 > > Apr 10 22:12:59 cdn001 varnishd[25730]: Child (-1) said Child starts > > > > I have restarted varnish daemon and things looks fine now, but I am > > thinking that how could this error happen? is there something wrong in my > > varnish configuration ? or something else ? > > > > thanks . > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lnsano at bol.com.br Thu Apr 11 20:04:57 2013 From: lnsano at bol.com.br (lnsano at bol.com.br) Date: Thu, 11 Apr 2013 17:04:57 -0300 Subject: No subject Message-ID: <516717696ced9_18abbacac7848@a4-winter15.tmail> An HTML attachment was scrubbed... URL: From chadaustin at outlook.com Fri Apr 12 00:25:47 2013 From: chadaustin at outlook.com (Chad Austin) Date: Thu, 11 Apr 2013 17:25:47 -0700 Subject: purge ACL not being enforced Message-ID: We're running 3.0.3 and our config is set up to enforce an ACL for purges, but I recently discovered that it has no effect - purges are successful regardless of the origin IP. The config is using the example from the documentation and I've been unable to determine why it's not working. Any help would be appreciated; VCL is below. Thanks, Chad acl purge { "localhost"; "10.0.0.0"/16; } backend sc { .host = "39.22.194.41"; .port = "80"; } backend scstatic { .host = "10.0.2.109"; .port = "80"; } backend ecommerce_ext { .host = "39.22.194.40"; .port = "80"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "PURGE" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } } else if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ #return (pass); error 500 "Unknown method."; } #if (req.http.Authorization || req.http.Cookie) { # /* Not cacheable by default */ # return (pass); #} if (req.url == "/sc_status.php") { error 200 "okay."; } # remove cookies for all static content unset req.http.Cookie; if (req.http.Host == "static.pub-ecommerce.somecompany.com") { set req.backend = ecommerce_ext; set req.http.Host = "pub-ecommerce.somecompany.com"; } else if (req.http.Host ~ "static.(.*\.)?somecompany.com") { set req.backend = scstatic; set req.http.Host = "www.somecompany.com"; } else if (req.http.Host ~ "somecompany.com(:[0-9]+)?$") { set req.backend = sc; set req.http.Host = "www.somecompany.com"; } else { error 404 "Unknown virtual host."; } return (lookup); } sub vcl_fetch { unset beresp.http.Set-Cookie; # cache 404's for 2 minutes if (beresp.status >= 400 && beresp.status < 500) { set beresp.ttl = 30s; } else if (beresp.status >=500 && beresp.status < 600) { set beresp.ttl = 30s; } } sub vcl_deliver { set resp.http.X-Backend = server.identity; } sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged HIT."; } } sub vcl_miss { if (req.request == "PURGE") { purge; error 200 "Purged MISS."; } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From chadaustin at outlook.com Fri Apr 12 01:19:05 2013 From: chadaustin at outlook.com (Chad Austin) Date: Thu, 11 Apr 2013 18:19:05 -0700 Subject: purge ACL not being enforced In-Reply-To: References: Message-ID: Urg, can't believe I didn't figure this out originally... my varnish server is behind an LB and it wasn't seeing the true IP of the client request. -- Chad From: chadaustin at outlook.com To: varnish-misc at varnish-cache.org Subject: purge ACL not being enforced Date: Thu, 11 Apr 2013 17:25:47 -0700 We're running 3.0.3 and our config is set up to enforce an ACL for purges, but I recently discovered that it has no effect - purges are successful regardless of the origin IP. The config is using the example from the documentation and I've been unable to determine why it's not working. Any help would be appreciated; VCL is below. Thanks, Chad acl purge { "localhost"; "10.0.0.0"/16; } backend sc { .host = "39.22.194.41"; .port = "80"; } backend scstatic { .host = "10.0.2.109"; .port = "80"; } backend ecommerce_ext { .host = "39.22.194.40"; .port = "80"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "PURGE" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } } else if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ #return (pass); error 500 "Unknown method."; } #if (req.http.Authorization || req.http.Cookie) { # /* Not cacheable by default */ # return (pass); #} if (req.url == "/sc_status.php") { error 200 "okay."; } # remove cookies for all static content unset req.http.Cookie; if (req.http.Host == "static.pub-ecommerce.somecompany.com") { set req.backend = ecommerce_ext; set req.http.Host = "pub-ecommerce.somecompany.com"; } else if (req.http.Host ~ "static.(.*\.)?somecompany.com") { set req.backend = scstatic; set req.http.Host = "www.somecompany.com"; } else if (req.http.Host ~ "somecompany.com(:[0-9]+)?$") { set req.backend = sc; set req.http.Host = "www.somecompany.com"; } else { error 404 "Unknown virtual host."; } return (lookup); } sub vcl_fetch { unset beresp.http.Set-Cookie; # cache 404's for 2 minutes if (beresp.status >= 400 && beresp.status < 500) { set beresp.ttl = 30s; } else if (beresp.status >=500 && beresp.status < 600) { set beresp.ttl = 30s; } } sub vcl_deliver { set resp.http.X-Backend = server.identity; } sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged HIT."; } } sub vcl_miss { if (req.request == "PURGE") { purge; error 200 "Purged MISS."; } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From pbrown at nbmedia.com Fri Apr 12 17:16:47 2013 From: pbrown at nbmedia.com (Peter Brown) Date: Fri, 12 Apr 2013 17:16:47 +0000 Subject: Looking for Varnish Consultant In-Reply-To: <116A5A222B28B84E92C959C5866857C13B249C73@NBNYMAIL2.nbmedia.com> References: <116A5A222B28B84E92C959C5866857C13B249C73@NBNYMAIL2.nbmedia.com> Message-ID: <116A5A222B28B84E92C959C5866857C13B24CEDE@NBNYMAIL2.nbmedia.com> Hi Folks, Just wanted to let people know that I found a consultant, so we don't need any more queries. Kudos to this list! I got a good number of very qualified responses. Best regards, Peter Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From lnsano at bol.com.br Fri Apr 12 21:46:25 2013 From: lnsano at bol.com.br (lnsano at bol.com.br) Date: Fri, 12 Apr 2013 18:46:25 -0300 Subject: patch on ticket 1285 In-Reply-To: <516717696ced9_18abbacac7848@a4-winter15.tmail> References: <516717696ced9_18abbacac7848@a4-winter15.tmail> Message-ID: <516880b1b7a18_1f941aa647c15d@a4-winter5.tmail> An HTML attachment was scrubbed... URL: From tomlobato at gmail.com Sat Apr 13 00:28:28 2013 From: tomlobato at gmail.com (Tom Lobato) Date: Fri, 12 Apr 2013 21:28:28 -0300 Subject: Dynamic ACLs Message-ID: Hi All! Is there a way to change ACL content dynamically? I need to declare something like... acl alternative_backend { "domain1.com"; "domain2.com; } and change its content (add/del itens) in vcl_recv while Varnish is running. Thanks, Tom Lobato From pprocacci at datapipe.com Sat Apr 13 06:32:51 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Sat, 13 Apr 2013 01:32:51 -0500 Subject: Dynamic ACLs In-Reply-To: References: Message-ID: <20130413063251.GA87828@nat.myhome> On Fri, Apr 12, 2013 at 09:28:28PM -0300, Tom Lobato wrote: > > Hi All! > > Is there a way to change ACL content dynamically? > > I need to declare something like... > > acl alternative_backend { > "domain1.com"; > "domain2.com; > } > > and change its content (add/del itens) in vcl_recv while Varnish is running. The closest you will be able to get to this is to generate the definition using a scripting language of your choice to generate the necessary acl's; then reload the vcl via the management interface (varnishadm) to pick up the changes. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From tomlobato at gmail.com Sat Apr 13 07:26:33 2013 From: tomlobato at gmail.com (Tom Lobato) Date: Sat, 13 Apr 2013 04:26:33 -0300 Subject: Dynamic ACLs In-Reply-To: <20130413063251.GA87828@nat.myhome> References: <20130413063251.GA87828@nat.myhome> Message-ID: <551B6E79-0C9E-43F2-BCDC-3A22CDDFFFCA@gmail.com> Hi Paul, thank you very much for the help. Does reloading vcl changes something in varnish, as reset cache or something that impact performance? Regards, Tom Lobato Em Apr 13, 2013, ?s 3:32 AM, "Paul A. Procacci" escreveu: > On Fri, Apr 12, 2013 at 09:28:28PM -0300, Tom Lobato wrote: >> >> Hi All! >> >> Is there a way to change ACL content dynamically? >> >> I need to declare something like... >> >> acl alternative_backend { >> "domain1.com"; >> "domain2.com; >> } >> >> and change its content (add/del itens) in vcl_recv while Varnish is running. > > The closest you will be able to get to this is to generate the definition using > a scripting language of your choice to generate the necessary acl's; then reload > the vcl via the management interface (varnishadm) to pick up the changes. > > ~Paul > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From me at ibotty.net Sat Apr 13 08:09:38 2013 From: me at ibotty.net (me at ibotty.net) Date: Sat, 13 Apr 2013 10:09:38 +0200 Subject: Dynamic ACLs In-Reply-To: <551B6E79-0C9E-43F2-BCDC-3A22CDDFFFCA@gmail.com> References: <20130413063251.GA87828@nat.myhome> <551B6E79-0C9E-43F2-BCDC-3A22CDDFFFCA@gmail.com> Message-ID: <516912C2.6070007@ibotty.net> hi, > thank you very much for the help. > Does reloading vcl changes something in varnish, as reset cache or something that impact performance? in short: no. there sure are some management structures that will have to be rebuild (you are loading a new configuration after all), but you will hardly notice that. the cache is _not_ invalidated! good luck, tobi From tomlobato at gmail.com Sat Apr 13 08:36:38 2013 From: tomlobato at gmail.com (Tom Lobato) Date: Sat, 13 Apr 2013 05:36:38 -0300 Subject: Dynamic ACLs In-Reply-To: <516912C2.6070007@ibotty.net> References: <20130413063251.GA87828@nat.myhome> <551B6E79-0C9E-43F2-BCDC-3A22CDDFFFCA@gmail.com> <516912C2.6070007@ibotty.net> Message-ID: Thank you very much, Tobi! You saved my week :) Em Apr 13, 2013, ?s 5:09 AM, me at ibotty.net escreveu: > hi, > >> thank you very much for the help. >> Does reloading vcl changes something in varnish, as reset cache or something that impact performance? > > in short: no. > > there sure are some management structures that will have to be rebuild > (you are loading a new configuration after all), but you will hardly > notice that. the cache is _not_ invalidated! > > good luck, > tobi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From numard at gmail.com Sun Apr 14 11:06:14 2013 From: numard at gmail.com (Norberto Meijome) Date: Sun, 14 Apr 2013 21:06:14 +1000 Subject: Bug #1083 Message-ID: Hi everyone, is there any progress being made to fix #1083 ? any known workaround ? https://www.varnish-cache.org/trac/ticket/1083 I'm very keen to see this solved, and obviously will help where needed. cheers, -- Norberto 'Beto' Meijome -------------- next part -------------- An HTML attachment was scrubbed... URL: From idu.angela at gmail.com Tue Apr 16 03:02:15 2013 From: idu.angela at gmail.com (Angela Wang) Date: Tue, 16 Apr 2013 11:02:15 +0800 Subject: Can't reach hostname or port 80 Message-ID: <516CBF37.6030405@gmail.com> I'm using Varnish on Nginx. I have varnish configured to run on port 80 and Nginx on port 8080. Nginx is also configured on a hostname. However, I can not reach IP_ADD:80 or hostname, but I can reach IP_ADD:8080. Paste is available here http://paste.ubuntu.com/5712101/ Angela From pprocacci at datapipe.com Tue Apr 16 04:00:52 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Mon, 15 Apr 2013 23:00:52 -0500 Subject: Can't reach hostname or port 80 In-Reply-To: <516CBF37.6030405@gmail.com> References: <516CBF37.6030405@gmail.com> Message-ID: <20130416040052.GG87828@nat.myhome> On Tue, Apr 16, 2013 at 11:02:15AM +0800, Angela Wang wrote: > I'm using Varnish on Nginx. I have varnish configured to run on port 80 > and Nginx on port 8080. Nginx is also configured on a hostname. However, > I can not reach IP_ADD:80 or hostname, but I can reach IP_ADD:8080. > Paste is available here http://paste.ubuntu.com/5712101/ "Can not reach" is kinda vague. Can you telnet to the port at all? If you can, what happens during an http exchange? What log files do you have that you can share? etc., etc. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From idu.angela at gmail.com Tue Apr 16 04:01:46 2013 From: idu.angela at gmail.com (Angela Wang) Date: Tue, 16 Apr 2013 12:01:46 +0800 Subject: Can't reach hostname or port 80 In-Reply-To: <20130416040052.GG87828@nat.myhome> References: <516CBF37.6030405@gmail.com> <20130416040052.GG87828@nat.myhome> Message-ID: <516CCD2A.9040503@gmail.com> Problem solved, it was PHP app port issue. Nothing to do with Nginx. Angela On 4/16/13 12:00 PM, Paul A. Procacci wrote: > On Tue, Apr 16, 2013 at 11:02:15AM +0800, Angela Wang wrote: >> I'm using Varnish on Nginx. I have varnish configured to run on port 80 >> and Nginx on port 8080. Nginx is also configured on a hostname. However, >> I can not reach IP_ADD:80 or hostname, but I can reach IP_ADD:8080. >> Paste is available here http://paste.ubuntu.com/5712101/ > "Can not reach" is kinda vague. Can you telnet to the port at all? > If you can, what happens during an http exchange? > What log files do you have that you can share? > etc., etc. > > ~Paul > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From werner.randelshofer at fibermail.ch Tue Apr 16 08:08:21 2013 From: werner.randelshofer at fibermail.ch (Werner Randelshofer) Date: Tue, 16 Apr 2013 10:08:21 +0200 Subject: Varnish only working with ports 80, 8080 and 9080 Message-ID: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> Dear list members, I am new to Varnish. I am trying to configure it so that I can run a dozen or so node.js services as backends. For some reason, Varnish only seems to work with ports 80, 8080 and 9080 either as frontend (listen) or backend ports. When I use a different port number in VARNISH_LISTEN_PORT, then varnishd does not start. I made sure that the port number is not taken using the command netstat -nltp Also when I use a different port number as backend, then Varnish tells me that the service is unavailable, and tells me that probing fails. However the backend is fully functional, when I connect to its port number. Is there a configuration option that I am missing, or a way how I can debug this? I am running varnish-3.0.3 revision 9e6a70f on CentOs 6.4. With best regards, Werner From pprocacci at datapipe.com Tue Apr 16 13:33:09 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Tue, 16 Apr 2013 08:33:09 -0500 Subject: Varnish only working with ports 80, 8080 and 9080 In-Reply-To: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> References: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> Message-ID: <20130416133309.GI87828@nat.myhome> > When I use a different port number in VARNISH_LISTEN_PORT, then varnishd does not start. > I made sure that the port number is not taken using the command netstat -nltp Do you have a firewall that is preventing tcp syn's from making it to the ports that you specify in your config? (Either a hardware or software firewall) If you don't, when you change the VARNISH_LISTEN_PORT and start varnish, does the service show as bound via the netstat command that you ran? If it is bound, can you telnet to the port and issue http requests? If it isn't bound, they'll be logs in your system log detailing why it couldn't be bound. What do those logs say? > Also when I use a different port number as backend, then Varnish tells me that the service is unavailable, and tells me that probing fails. However the backend is fully functional, when I connect to its port number. Are you sure your attempting to connect to the right backend? Specifically, 127.0.0.1:8080 is not :8080 where is the primary ip address of the machine. Is there a software/hardware firewall preventing the tcp syn's from making it to their destination? Can you telent to the port from the varnish machine and induce an http conversation? ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From enno at tty.nl Tue Apr 16 13:54:11 2013 From: enno at tty.nl (Enno van Amerongen) Date: Tue, 16 Apr 2013 15:54:11 +0200 Subject: =?iso-8859-1?Q?Re=3A_increase_sess=5Fworkspace_to_fix_assert_error?= =?iso-8859-1?Q?=3F?= In-Reply-To: Message-ID: <20130416135411.95869875@groupware.tty.nl> Hi Rub?n, Apparently we made a mistake a while ago by not confirming the varnish 3.0 defaults compared to 2.1. So we were happily still running with a sess_workspace of 16k, instead of 64k. This worked fine with our previous config, but since we added our sharding vmod, 16k is not enough anymore. So, the fixed config with 64k as sess_workspace has been running on one of our varnish servers for almost a week now. And it's been working fine. Even when a Justin Bieber link went viral yesterday, which put our systems to the real test. Not to mention this happened at the same time it's already peak-hour. And Varnish (of course;) happily served thousands of requests per second. So yes, I can now confirm: increasing sess_workspace fixed the assertion errors in my case. regards, Enno van Amerongen _____ From: Rub?n Romero [mailto:ruben at varnish-software.com] To: Enno van Amerongen [mailto:enno at tty.nl] Sent: Sun, 14 Apr 2013 12:52:48 +0200 Subject: Re: increase sess_workspace to fix assert error? Hi Enno, Increasing it will likely help. Not sure to how much, but do give it as much as you can and let us know how it went :-) Best, - Rub?n Romero Varnish Software Den 2. apr. 2013 21.18 skrev "Enno van Amerongen" f?lgende: Dear List, After deployment of a vmod we've written (that enables us to select a director by id in url), we've been experiencing assert errors. Below are the stack traces, but to not waste your time, here's what I think it boils down to: { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 } 4k as session workspace has worked fine for us for years, but our vmod sharding adds hundreds of directors (as opposed to 4), while the number of backends stayed the same (about 25). So, my question is, do I just need to increase sess_workspace to fix this? If not, what else ;) Regards, Enno van Amerongen Both stacktraces Apr 2 18:43:32 obfuscated /var/www/varnish[12750]: Child (20498) Panic message: Assert error in vh_get_var_alloc(), vmod_var.c line 77:#012 Condition((v) != 0) not true.#012errno = 32 (Broken pipe)#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x7fe137efe347: _end+7fe13786898f#012 0x7fe137efe862: _end+7fe137868eaa#012 0x7fecc5b93e73: _end+7fecc54fe4bb#012 0x7fecc5b939ea: _end+7fecc54fe032#012 0x7fecc5ba24f1: _end+7fecc550cb39#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012sp = 0x7fed81293008 {#012 fd = 2849, id = 2849, xid = 918440385,#012 client = 10.1.1.1 51854,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed81293080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed81293c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed81293080[sess]#012 "GET",#012 "/f/146057654_2.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14",#012 "Host: img.obfuscated.com",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: fr-FR,fr;q=0.9,en;q=0.8",#012 "Referer: http://obfuscated",#012 "Connection: Keep-Alive",#012 },#012 worker = 0x7feece7efaa0 {#012 ws = 0x7feece7efcd8 { #012 id = "wrk",#012 {s,f,r,e} = {0x7feece7dd9f0,+16,(nil),+65536},#012 },#012 },#012 vcl = {#012 srcname = {#012 "input",#012 "Default",#012 "/obfuscated/img_backends.vcl",#012 "/obfuscated/img_directors.vcl",#012 So we removed the vmod_var from our configuration as it wasn't really required anyway. Then 13 minutes later, Varnish crashed again, now with a different, but similar assert error: Apr 2 19:07:22 obfuscated /var/www/varnish[12750]: Child (18818) Panic message: Assert error in VRT_int_string(), cache_vrt.c line 324:#012 Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x4435f7: VRT_int_string+88#012 0x7fe686b9ce1f: _end+7fe686507467#012 0x7fe686b9c99a: _end+7fe686506fe2#012 0x7fe686ba7424: _end+7fe686511a6c#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012 0x43aa0a: wrk_thread_real+8cc#012sp = 0x7fed76034008 {#012 fd = 2357, id = 2357, xid = 2009537706,#012 client = 80.239.243.196 59852,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed76034080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed76034080[sess]#012 "GET",#012 "/f/148682988.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera Mini/7.5.32193/29.3134; U; nl) Presto/2.8.119 Version/11.10",#012 "Host: img.obfuscated.com",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: nl,en;q=0.9",#012 "Referer: obfuscated",#012 "Connection: Keep-Alive",#012 "X-OperaMini-Phone: Android #",#012 "X-OperaMini-Phone-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "x-forwarded-for: obfuscated",#012 "Device-Stock-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "X _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner.randelshofer at fibermail.ch Wed Apr 17 09:49:53 2013 From: werner.randelshofer at fibermail.ch (Werner Randelshofer) Date: Wed, 17 Apr 2013 11:49:53 +0200 Subject: Varnish only working with ports 80, 8080 and 9080 In-Reply-To: <20130416133309.GI87828@nat.myhome> References: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> <20130416133309.GI87828@nat.myhome> Message-ID: <8AE993EB-5FF6-4DC4-9583-022BE4BFB51B@fibermail.ch> Hi Paul, I have stopped the iptables service, so there is no firewall running. When I start the varnish service, say with port 9000, then the service will not launch, and thus the port is not bound. I have tried now starting the varnish service with port 80, and setting the port to 9000 with varnishadm. I get an error 300 "Could not open sockets" as shown below. Port 80 works fine though. $ varnishadm param.set listen_address :9000 200 start 300 Could not open sockets param.set listen_address :80 200 start 200 My backends work fine. I can access them with Konqueror with any port number that I want. Telnet also works fine. I am stuck. Thanks, Werner On 16.04.2013, at 15:33, Paul A. Procacci wrote: >> When I use a different port number in VARNISH_LISTEN_PORT, then varnishd does not start. >> I made sure that the port number is not taken using the command netstat -nltp > > Do you have a firewall that is preventing tcp syn's from > making it to the ports that you specify in your config? > (Either a hardware or software firewall) > > If you don't, when you change the VARNISH_LISTEN_PORT and > start varnish, does the service show as bound via the > netstat command that you ran? > > If it is bound, can you telnet to the port and issue > http requests? > > If it isn't bound, they'll be logs in your system log > detailing why it couldn't be bound. What do those logs say? > > >> Also when I use a different port number as backend, then Varnish tells me that the service is unavailable, and tells me that probing fails. However the backend is fully functional, when I connect to its port number. > > Are you sure your attempting to connect to the right backend? > Specifically, 127.0.0.1:8080 is not :8080 where is > the primary ip address of the machine. > > Is there a software/hardware firewall preventing the tcp syn's > from making it to their destination? > > Can you telent to the port from the varnish machine and induce > an http conversation? > > ~Paul > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. > From perbu at varnish-software.com Wed Apr 17 11:24:56 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 17 Apr 2013 13:24:56 +0200 Subject: Varnish only working with ports 80, 8080 and 9080 In-Reply-To: <8AE993EB-5FF6-4DC4-9583-022BE4BFB51B@fibermail.ch> References: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> <20130416133309.GI87828@nat.myhome> <8AE993EB-5FF6-4DC4-9583-022BE4BFB51B@fibermail.ch> Message-ID: SElinux or similar? On Wed, Apr 17, 2013 at 11:49 AM, Werner Randelshofer < werner.randelshofer at fibermail.ch> wrote: > Hi Paul, > > I have stopped the iptables service, so there is no firewall running. > > When I start the varnish service, say with port 9000, then the service > will not launch, and thus the port is not bound. > > I have tried now starting the varnish service with port 80, and setting > the port to 9000 with varnishadm. > I get an error 300 "Could not open sockets" as shown below. Port 80 works > fine though. > > > $ varnishadm > > param.set listen_address :9000 > 200 > > start > 300 > Could not open sockets > > param.set listen_address :80 > 200 > > start > 200 > > > My backends work fine. I can access them with Konqueror with any port > number that I want. > Telnet also works fine. > > I am stuck. > > Thanks, > Werner > > > On 16.04.2013, at 15:33, Paul A. Procacci wrote: > > >> When I use a different port number in VARNISH_LISTEN_PORT, then > varnishd does not start. > >> I made sure that the port number is not taken using the command netstat > -nltp > > > > Do you have a firewall that is preventing tcp syn's from > > making it to the ports that you specify in your config? > > (Either a hardware or software firewall) > > > > If you don't, when you change the VARNISH_LISTEN_PORT and > > start varnish, does the service show as bound via the > > netstat command that you ran? > > > > If it is bound, can you telnet to the port and issue > > http requests? > > > > If it isn't bound, they'll be logs in your system log > > detailing why it couldn't be bound. What do those logs say? > > > > > >> Also when I use a different port number as backend, then Varnish tells > me that the service is unavailable, and tells me that probing fails. > However the backend is fully functional, when I connect to its port number. > > > > Are you sure your attempting to connect to the right backend? > > Specifically, 127.0.0.1:8080 is not :8080 where is > > the primary ip address of the machine. > > > > Is there a software/hardware firewall preventing the tcp syn's > > from making it to their destination? > > > > Can you telent to the port from the varnish machine and induce > > an http conversation? > > > > ~Paul > > > > ________________________________ > > > > This message may contain confidential or privileged information. If you > are not the intended recipient, please advise us immediately and delete > this message. See http://www.datapipe.com/legal/email_disclaimer/ for > further information on confidentiality and the risks of non-secure > electronic communication. If you cannot access these links, please notify > us by reply message and we will send the contents to you. > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner.randelshofer at fibermail.ch Wed Apr 17 12:09:12 2013 From: werner.randelshofer at fibermail.ch (Werner Randelshofer) Date: Wed, 17 Apr 2013 14:09:12 +0200 Subject: Varnish only working with ports 80, 8080 and 9080 In-Reply-To: References: <9DACDCA4-D189-4E94-80D8-C84A9237A12C@fibermail.ch> <20130416133309.GI87828@nat.myhome> <8AE993EB-5FF6-4DC4-9583-022BE4BFB51B@fibermail.ch> Message-ID: Hi Per, Yes, SELinux was it!! I have now installed setroubleshoot and when I run seaudit, it tells me how to fix the problem. That is, by executing the following command. sudo setsebool varnishd_connect_any 1 Thank you very much for your help. :-) And also thanks to everyone else who replied! Best, Werner On 17.04.2013, at 13:24, Per Buer wrote: > SElinux or similar? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brudolph at admantx.com Thu Apr 18 16:54:49 2013 From: brudolph at admantx.com (Bruce Rudolph) Date: Thu, 18 Apr 2013 12:54:49 -0400 Subject: Keeping Sessions Open after vcl_error Message-ID: I have Varnish 3.0.3 and it is in an environment that a single session can send hundreds to thousands of requests (and more) as long as the session remains open. There is a hard response threshold that needs to be met. If it is not met the client closes the session and opens another. Excessive timeouts on the requests means many session open and close events which reduces throughput. I have the following defined in the default.vcl file: Added to backend definition: .connect_timeout = 0.7s; .first_byte_timeout = 4ms; .between_bytes_timeout = 4ms; A timeout will occur after 4ms if no response from the backend is received after the request is forwarded. When a timeout occurs a synthesized response is added to an override of the vcl_error function in the default.vcl file. The synthesized response is supposed to keep the session open. Based on the documentation and from my observations in the environment when an error occurs in Varnish, as in a timeout, it causes a SessionClose. Is there a way to keep the session open while sending a synthesized response? Regards, Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From brudolph at admantx.com Thu Apr 18 20:21:08 2013 From: brudolph at admantx.com (Bruce Rudolph) Date: Thu, 18 Apr 2013 16:21:08 -0400 Subject: Always a SessionClose after a Cache Hit Message-ID: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> Running Varnish 3.0.3 on CentOS 5.8. I am wondering why there is a SessionClose after a cache hit and delivery. The example below shows the varnishlog output of the request/response. There is a keep-alive header in the request and response. Normally the client calling my app would still keep the session open and is supposed to send more requests. Any suggestions? 10773 SessionOpen c 204.13.194.63 53590 :80 10773 ReqStart c 204.13.194.63 53590 134881832 10773 RxRequest c GET 10773 RxURL c /ourapp?url=http%3A%2F%2Fjsonline.com 10773 RxProtocol c HTTP/1.1 10773 RxHeader c Host: 8.12.230.136 10773 RxHeader c Connection: keep-alive 10773 VCL_call c recv lookup 10773 VCL_call c hash 10773 Hash c jsonline.com 10773 VCL_return c hash 10773 Hit c 134858969 10773 VCL_call c hit deliver 10773 VCL_call c deliver deliver 10773 TxProtocol c HTTP/1.1 10773 TxStatus c 200 10773 TxResponse c OK 10773 TxHeader c Cache-Control: s-maxage=86400 10773 TxHeader c Content-Type: text/plain 10773 TxHeader c Content-Length: 13 10773 TxHeader c Accept-Ranges: bytes 10773 TxHeader c Date: Thu, 18 Apr 2013 19:55:17 GMT 10773 TxHeader c X-Varnish: 134881832 134858969 10773 TxHeader c Age: 10 10773 TxHeader c Via: 1.1 varnish 10773 TxHeader c Connection: keep-alive 10773 Length c 13 10773 ReqEnd c 134881832 1366314917.734391928 1366314917.734510899 1.976155043 0.000053167 0.000065804 10773 SessionClose c EOF Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Wed Apr 24 14:27:52 2013 From: cosimo at streppone.it (Cosimo Streppone) Date: Wed, 24 Apr 2013 16:27:52 +0200 Subject: vlogdump Message-ID: <5177EBE8.50909@streppone.it> Hi all, vlogdump is a small quick and dirty tool I wrote to parse the raw varnishlog output because: * I'm still using 2.1, so my varnishlog doesn't have "-m" * I wanted to quickly filter transactions by different criterias, possibly playing together * I had never used awk beyond oneliners ;) Here's the README with a few examples: https://github.com/cosimo/vlogdump#readme If you use it, don't leave it running for a long time. {{ include "ymmv", "standard_disclaimer" }} -- Cosimo From graham.lyons at bbc.co.uk Thu Apr 25 10:16:01 2013 From: graham.lyons at bbc.co.uk (Graham Lyons) Date: Thu, 25 Apr 2013 10:16:01 +0000 Subject: Varnishlog and Splunk Message-ID: Hello, Has anyone had any experience of putting output from varnishlog into Splunk? My experience of Splunk so far has involved access log type sources with events on separate lines, which is obviously quite different to what comes out of varnishlog. If there's any prior art it would interesting to hear. Thanks, Graham. ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. --------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From brudolph at admantx.com Thu Apr 25 15:51:18 2013 From: brudolph at admantx.com (Bruce Rudolph) Date: Thu, 25 Apr 2013 11:51:18 -0400 Subject: Always a SessionClose after a Cache Hit In-Reply-To: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> References: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> Message-ID: <54ECC6E7-4278-4FB8-AA28-7987386CC304@admantx.com> I verified this happens also with Varnish 2.1.5. I expect to see a lot of requests on a persistent connection. However I am only getting one request per connection. The performance to the app driving the requests "tanks" because a new session needs to be opened. Hopefully someone can shed some light on this problem. Bruce On Apr 18, 2013, at 4:21 PM, Bruce Rudolph wrote: > Running Varnish 3.0.3 on CentOS 5.8. > > I am wondering why there is a SessionClose after a cache hit and delivery. The example below shows the varnishlog output of the request/response. There is a keep-alive header in the request and response. Normally the client calling my app would still keep the session open and is supposed to send more requests. > > Any suggestions? > > 10773 SessionOpen c 204.13.194.63 53590 :80 > 10773 ReqStart c 204.13.194.63 53590 134881832 > 10773 RxRequest c GET > 10773 RxURL c /ourapp?url=http%3A%2F%2Fjsonline.com > 10773 RxProtocol c HTTP/1.1 > 10773 RxHeader c Host: 8.12.230.136 > 10773 RxHeader c Connection: keep-alive > 10773 VCL_call c recv lookup > 10773 VCL_call c hash > 10773 Hash c jsonline.com > 10773 VCL_return c hash > 10773 Hit c 134858969 > 10773 VCL_call c hit deliver > 10773 VCL_call c deliver deliver > 10773 TxProtocol c HTTP/1.1 > 10773 TxStatus c 200 > 10773 TxResponse c OK > 10773 TxHeader c Cache-Control: s-maxage=86400 > 10773 TxHeader c Content-Type: text/plain > 10773 TxHeader c Content-Length: 13 > 10773 TxHeader c Accept-Ranges: bytes > 10773 TxHeader c Date: Thu, 18 Apr 2013 19:55:17 GMT > 10773 TxHeader c X-Varnish: 134881832 134858969 > 10773 TxHeader c Age: 10 > 10773 TxHeader c Via: 1.1 varnish > 10773 TxHeader c Connection: keep-alive > 10773 Length c 13 > 10773 ReqEnd c 134881832 1366314917.734391928 1366314917.734510899 1.976155043 0.000053167 0.000065804 > 10773 SessionClose c EOF > > > Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Fri Apr 26 08:08:19 2013 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Fri, 26 Apr 2013 10:08:19 +0200 Subject: Always a SessionClose after a Cache Hit In-Reply-To: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> References: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> Message-ID: <20130426080818.GA8186@immer.varnish-software.com> On Thu, Apr 18, 2013 at 04:21:08PM -0400, Bruce Rudolph wrote: > Running Varnish 3.0.3 on CentOS 5.8. > I am wondering why there is a SessionClose after a cache hit and delivery. The example below shows the varnishlog output of the request/response. There is a keep-alive header in the request and response. Normally the client calling my app would still keep the session open and is supposed to send more requests. [..] > 10773 TxHeader c Connection: keep-alive > 10773 Length c 13 > 10773 ReqEnd c 134881832 1366314917.734391928 1366314917.734510899 1.976155043 0.000053167 0.000065804 > 10773 SessionClose c EOF Hi Bruce. I had a look at the source. This message (on Linux) originates from cnt_wait() in cache_center.c. (line 122) It means pretty much what it says. After session_linger has passed Varnish attempts to read a new request, and the read from the socket fails because the other end has disconnected. I'd break out tcpdump/wireshark and check if your client is behaving in a different way than you are expecting. -- With regards, Lasse Karstensen Varnish Software AS From lkarsten at varnish-software.com Fri Apr 26 08:35:43 2013 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Fri, 26 Apr 2013 10:35:43 +0200 Subject: Keeping Sessions Open after vcl_error In-Reply-To: References: Message-ID: <20130426083541.GB8186@immer.varnish-software.com> On Thu, Apr 18, 2013 at 12:54:49PM -0400, Bruce Rudolph wrote: [..] > A timeout will occur after 4ms if no response from the backend is received after the request is forwarded. When a timeout occurs a synthesized response is added to an override of the vcl_error function in the default.vcl file. The synthesized response is supposed to keep the session open. Based on the documentation and from my observations in the environment when an error occurs in Varnish, as in a timeout, it causes a SessionClose. Any request fulfilled by vcl_error will lead to a closed connection. > Is there a way to keep the session open while sending a synthesized response? There is a workaround in return(restart) from vcl_error. Set a req.http.x-foo marker first, so you have something to check for in recv. In recv you change the backend to something that produces your error page, this can of course have infinite TTL to get a cached/quick response. Keepalive works as expected from then on. -- With regards, Lasse Karstensen Varnish Software AS From marc.fournier at camptocamp.com Fri Apr 26 09:11:24 2013 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Fri, 26 Apr 2013 11:11:24 +0200 Subject: Varnishlog and Splunk In-Reply-To: References: Message-ID: <1366966392-sup-8590@lonquimay.wrk.lsn.camptocamp.com> Hello, Excerpts from Graham Lyons's message of 2013-04-25 12:16:01 +0200: > > Has anyone had any experience of putting output from varnishlog into > Splunk? My experience of Splunk so far has involved access log type > sources with events on separate lines, which is obviously quite different > to what comes out of varnishlog. > > If there's any prior art it would interesting to hear. I have no experience with splunk, but I know such a plugin has been added to logstash recently: https://github.com/logstash/logstash/pull/422 Leaving the splunk vs logstash debate for another time, what I'd like to point out is that the way it was done for logstash is probably the way to go for splunk too: grab the varnish API bindings for your favorite language and directly push the logs to splunk as structured data, instead of trying to parse and recompose the multi-line output of varnishlog. Cheers, Marc From straightflush at gmail.com Fri Apr 26 19:25:28 2013 From: straightflush at gmail.com (AD) Date: Fri, 26 Apr 2013 15:25:28 -0400 Subject: file vs malloc performance Message-ID: Hey, I was reading some tuning guides and came across Krystian's blog post http://kly.no/posts/2010_01_26__Varnish_best_practices__.html which notes the following: On Linux, -sfile performs far better than -smalloc once you start hitting disk Assuming we have a dataset that we know is going to be larger than RAM, can I confirm, even with non-SSD's that Varnish is more performant with -sfile over -smalloc b/c swap is going to be slower than mmap() against a file ? What happens if the dataset is larger than the file ? We are looking to host multiple sites through our varnish cluster which is certainly not going to all fit into RAM so we want to choose the right option here and we dont have an SSD option. Thanks _AD -------------- next part -------------- An HTML attachment was scrubbed... URL: From howachen at gmail.com Sat Apr 27 03:33:26 2013 From: howachen at gmail.com (howard chen) Date: Sat, 27 Apr 2013 11:33:26 +0800 Subject: Confusing with storage backends Message-ID: >From the doc: https://www.varnish-cache.org/docs/trunk/users-guide/storage-backends.html In the "Intro"... >>"The default configuration is to use the malloc backend with a limited size." And under the "file" >>This is the default storage backend and unless you specify another storage this one will used along with Transient storage. Seems contradicting? -------------- next part -------------- An HTML attachment was scrubbed... URL: From aashisn at hotmail.com Sat Apr 27 11:51:51 2013 From: aashisn at hotmail.com (Ashish) Date: Sat, 27 Apr 2013 12:51:51 +0100 Subject: Varnish Cache In-Reply-To: <20130426080818.GA8186@immer.varnish-software.com> References: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> <20130426080818.GA8186@immer.varnish-software.com> Message-ID: Hi , I am quite new on varnish, so far its been really impressive doing test and quite everything that i wanted to achieve with it, I did some test with single server, and everything looked good, interms of cache hit ratio and all. but now i am unable to see anything as such as cache at all. Could anyone please show me pointer what in my rule is causing not to cache? ( Things not working so far) # Cache as wished: /json/showcase 1s , robot.txt 4 days, CSS/JPG/JS/ 3600s # If everything is busted serve whatever we got for 6 h => vcl ################## Define Backend Servers ########################################## backend box1 { .host = "xx.xx.xx.xx"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 3s; .window = 5; .threshold = 3; } } backend box2 { .host = "xx.xx.xx.xx"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 3s; .window = 5; .threshold = 3; } } acl unwanted { "188.223.122.81"; } sub vcl_recv { if (client.ip ~ unwanted) { error 403 "You seem to be blocked man.";} if (req.http.user-agent ~ "^someuseragent") { error 403 "If you think This is Mistake Please use different User-Agent..";} } ####################################### Start of Load Balancing ######################## director my_cluster random { { .backend = box1; .weight = 2; } { .backend = box2; .weight = 2; } } ###################################### ENd of Load Balancing ############################ ###################################### Start of Request Handling ######################## #sub vcl_recv { if (req.http.host ~ "^(.*\.)?myweb\.com$") { set req.backend = my_cluster; } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } if (req.request == "GET" || req.request == "HEAD") { return (lookup); } } ####################################### DO not cache anything on probe ############## sub vcl_recv { # This box fqdn is diff then above actual domains if (req.http.host ~ "^(.*\.)?\.myboxes\.com$") { return(pass); } } ######################################### Start of forced TTL ############################ sub vcl_fetch { if (req.url ~ "/json/showcase") { set beresp.ttl = 1s; } if (beresp.http.cache-control !~ "s-maxage" && req.url ~ "\.(png|gif|jpg)$") { set beresp.ttl = 120s; } if(req.url == "/robots.txt") { # Robots.txt is updated rarely and should be cached for 4 days # Purge manually as required set beresp.ttl = 96h; } # Set TTL for these product if (req.url ~ "\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)$") { unset beresp.http.set-cookie; set beresp.ttl = 3600s; } # Allow items to be stale if needed (i.e. if we are down server wise). set beresp.grace = 6h; } From aashisn at hotmail.com Sat Apr 27 20:36:01 2013 From: aashisn at hotmail.com (Ashish) Date: Sat, 27 Apr 2013 21:36:01 +0100 Subject: Varnish Cache In-Reply-To: References: <95072A8D-4092-498C-ADC8-D5D89BEE2166@admantx.com> <20130426080818.GA8186@immer.varnish-software.com> Message-ID: Hi All, I think it was due to the nature of the website, (i.e. cookies + json updates) were playing role not to cache. I did some changes and it started working (for forced cache rule.) e.g. sub vcl_recv -> added some lookup for certain conditions. and sub vcl_fetch -> added set beresp.ttl But going into another type of trouble: i occasionally get 503 thrown without any reason. I have 2 server, i intentionally turned one off, health checkup shows its sick (great), and sends request to alive one, but 1 out of 20 times it throws 503. _[I have added ulimit -n 10000 and restart for 4 times in 503, however is there any particular reason behind this behaviour?] _ 4 RxURL c /json/showcase/ 4 RxProtocol c HTTP/1.1 4 RxHeader c Host: www.example.com 4 RxHeader c X-Real-IP: 188.223.122.81 4 RxHeader c X-Forwarded-For: 188.223.122.81 4 RxHeader c X-Forwarded-Proto: http 4 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 4 RxHeader c Accept: */* 4 RxHeader c Referer: http://www.example.com/ 4 RxHeader c Accept-Encoding: gzip,deflate,sdch 4 RxHeader c Accept-Language: en-US,en;q=0.8 4 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 4 VCL_call c recv 4 VCL_acl c NO_MATCH unwanted 4 VCL_return c pass 4 VCL_call c hash 4 Hash c /json/showcase/ 4 Hash c www.example.com 4 VCL_return c hash 4 VCL_call c pass pass 4 FetchError c no backend connection 4 VCL_call c error deliver 4 VCL_call c deliver deliver 4 TxProtocol c HTTP/1.1 4 TxStatus c 503 4 TxResponse c Service Unavailable 4 TxHeader c Server: Varnish 4 TxHeader c Content-Type: text/html; charset=utf-8 4 TxHeader c Content-Length: 715 4 TxHeader c Accept-Ranges: bytes 4 TxHeader c Date: Sat, 27 Apr 2013 15:51:25 GMT 4 TxHeader c X-Varnish: 619632706 4 TxHeader c Age: 0 4 TxHeader c Via: 1.1 varnish 4 TxHeader c Connection: close 4 Length c 715 On 27/04/2013 12:51, Ashish wrote: > Hi , I am quite new on varnish, so far its been really impressive > doing test and quite everything that i wanted to achieve with it, > I did some test with single server, and everything looked good, > interms of cache hit ratio and all. but now i am unable to see > anything as such as cache at all. > > Could anyone please show me pointer what in my rule is causing not to > cache? > > ( Things not working so far) > # Cache as wished: /json/showcase 1s , robot.txt 4 days, CSS/JPG/JS/ > 3600s > # If everything is busted serve whatever we got for 6 h > > > => vcl > > ################## Define Backend Servers > ########################################## > backend box1 { .host = "xx.xx.xx.xx"; .probe = { .url = "/"; .timeout > = 34 ms; .interval = 3s; .window = 5; .threshold = 3; } } > backend box2 { .host = "xx.xx.xx.xx"; .probe = { .url = "/"; .timeout > = 34 ms; .interval = 3s; .window = 5; .threshold = 3; } } > > acl unwanted { "188.223.122.81"; } > > sub vcl_recv { if (client.ip ~ unwanted) { error 403 "You seem to be > blocked man.";} > > if (req.http.user-agent ~ > "^someuseragent") { > error 403 "If you think This is Mistake Please use > different User-Agent..";} > } > > > ####################################### Start of Load Balancing > ######################## > > director my_cluster random { > > { .backend = box1; .weight = 2; } > { .backend = box2; .weight = 2; } > } > > ###################################### ENd of Load Balancing > ############################ > > ###################################### Start of Request Handling > ######################## > #sub vcl_recv { > if (req.http.host ~ "^(.*\.)?myweb\.com$") { > set req.backend = my_cluster; > } > > if (req.request != "GET" && > req.request != "HEAD" && > req.request != "PUT" && > req.request != "POST" && > req.request != "TRACE" && > req.request != "OPTIONS" && > req.request != "DELETE") { > /* Non-RFC2616 or CONNECT which is weird. */ > return (pipe); > } > if (req.request != "GET" && req.request != "HEAD") { > /* We only deal with GET and HEAD by default */ > return (pass); > } > if (req.http.Authorization || req.http.Cookie) { > /* Not cacheable by default */ > return (pass); > } > if (req.request == "GET" || req.request == "HEAD") { > return (lookup); > } > } > > ####################################### DO not cache anything on > probe ############## > > > sub vcl_recv { > # This box fqdn is diff then above actual domains > if (req.http.host ~ "^(.*\.)?\.myboxes\.com$") { > return(pass); > } > } > > ######################################### Start of forced TTL > ############################ > sub vcl_fetch { > if (req.url ~ "/json/showcase") { > set beresp.ttl = 1s; > } > > > if (beresp.http.cache-control !~ "s-maxage" && > req.url ~ "\.(png|gif|jpg)$") { > set beresp.ttl = 120s; > } > > if(req.url == "/robots.txt") { > # Robots.txt is updated rarely and should be cached > for 4 days > # Purge manually as required > set beresp.ttl = 96h; > } > > # Set TTL for these product > if (req.url ~ > "\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)$") { > unset beresp.http.set-cookie; > set beresp.ttl = 3600s; > } > > # Allow items to be stale if needed (i.e. if we are > down server wise). > set beresp.grace = 6h; > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Apr 29 07:46:13 2013 From: perbu at varnish-software.com (Per Buer) Date: Mon, 29 Apr 2013 09:46:13 +0200 Subject: Confusing with storage backends In-Reply-To: References: Message-ID: Hi Howard, On Sat, Apr 27, 2013 at 5:33 AM, howard chen wrote: > From the doc: > https://www.varnish-cache.org/docs/trunk/users-guide/storage-backends.html > > In the "Intro"... > > >>"The default configuration is to use the malloc backend with a limited > size." > > And under the "file" > > >>This is the default storage backend and unless you specify another > storage this one will used along with Transient storage. > > Seems contradicting? > Yes. -sfile used to be the default. Now malloc is. I've fixed it in master. Per -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihamina at rktmb.org Mon Apr 29 08:41:08 2013 From: mihamina at rktmb.org (Mihamina Rakotomandimby) Date: Mon, 29 Apr 2013 11:41:08 +0300 Subject: varnish 2x over Solaris10 on Sparc IIIi Message-ID: <517E3224.8090703@rktmb.org> Hi all, For an emergency, I just setup a varnish 2 over Solaris 10 on a Sparc IIIi. 1?) Taking v 2.1.5 was a fail as the varnish parent process didnt want to start 2?) Looking arround the Web, I saw that two versions of Varnish were available for Sparc (I mean packaged by some other system/distributions): 2.1.3 and 2.0.4 a - 2.1.3 parent launched, but couldn't connect to the background Apache b - 2.0.4 gave me satisfaction as it compiles, runs and connects to the backend. My application configuration is: - 1 front varnish on 0.0.0.0:80, configured to connect to 1 Apache located on the same machine listening on 127.0.0.1:8080 ** Varnish is _for the moment_ default configured and the VCL is about one section declaring the backend - Application is a Drupal 7 one - The database is on another machine. The hardware is: Sun v440, with 1 Sparc IIIi CPU and 8GB RAM Now, the question is: Has someone experienced varnish 2.0.4 with Solaris10 on Sparc, and what about the reliability of that mix? (I ask because Sparc doesnt seem to be the prior target for support, as stated in https://www.varnish-cache.org/trac/ticket/1177 and http://goo.gl/MxZqb) Thank you for your answers. -- RMA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Apr 29 08:55:37 2013 From: perbu at varnish-software.com (Per Buer) Date: Mon, 29 Apr 2013 10:55:37 +0200 Subject: varnish 2x over Solaris10 on Sparc IIIi In-Reply-To: <517E3224.8090703@rktmb.org> References: <517E3224.8090703@rktmb.org> Message-ID: Hi, On Mon, Apr 29, 2013 at 10:41 AM, Mihamina Rakotomandimby < mihamina at rktmb.org> wrote: > > For an emergency, I just setup a varnish 2 over Solaris 10 on a Sparc IIIi. > I would be seriously surprised if Varnish works on Sparc. It will work fine on Solaris/X86, but Sparc is pretty strict when it comes to memory alignment and the last time we checked we weren't even close. -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Mon Apr 29 14:08:57 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Mon, 29 Apr 2013 10:08:57 -0400 Subject: Varnishlog and Splunk In-Reply-To: References: Message-ID: Our ops team set it up fairly easily. Not sure exactly what they did. On Thu, Apr 25, 2013 at 6:16 AM, Graham Lyons wrote: > Hello, > > Has anyone had any experience of putting output from varnishlog into > Splunk? My experience of Splunk so far has involved access log type sources > with events on separate lines, which is obviously quite different to what > comes out of varnishlog. > > If there's any prior art it would interesting to hear. > > Thanks, > Graham. > > > > ---------------------------- > > http://www.bbc.co.uk > This e-mail (and any attachments) is confidential and may contain personal > views which are not the views of the BBC unless specifically stated. > If you have received it in error, please delete it from your system. > Do not use, copy or disclose the information in any way nor act in > reliance on it and notify the sender immediately. > Please note that the BBC monitors e-mails sent or received. > Further communication will signify your consent to this. > > --------------------- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Mon Apr 29 14:11:50 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Mon, 29 Apr 2013 10:11:50 -0400 Subject: varnishncsa Message-ID: >From reading the man page, it doesn't seem as though varnishncsa is vital to varnish running. Is there any reason to have it running if we don't need apache/ncsa style logs? -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at darkmere.gen.nz Mon Apr 29 22:02:05 2013 From: simon at darkmere.gen.nz (Simon Lyall) Date: Tue, 30 Apr 2013 10:02:05 +1200 (NZST) Subject: varnishncsa In-Reply-To: References: Message-ID: On Mon, 29 Apr 2013, Sean Allen wrote: > From reading the man page, it doesn't seem as though varnishncsa is vital to varnish running. > Is there any reason to have it running if we don't need apache/ncsa style logs? No, you only need to run varnishd. Neither varnishlog nor varnishncsa needs to be running. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From jonathan.huot at thomsonreuters.com Tue Apr 30 13:29:30 2013 From: jonathan.huot at thomsonreuters.com (jonathan.huot at thomsonreuters.com) Date: Tue, 30 Apr 2013 13:29:30 +0000 Subject: Varnishlog and Splunk In-Reply-To: References: Message-ID: <8E656B642592B942AE317E2AFAE0ABA13F9BE324@UK2P-ERFMMBX10.ERF.thomson.com> Hi Graham, Splunk didn?t care with separate lines or not, it?s all about regexp. You can setup your Splunk events by adding any separator you want. It can be a line feeds or any separator (ReqStart/ReqEnd) Currently, we?re fetching records (about 10 lines for each record) using Splunk without any issues. However, I will suggest you to use varnishncsa instead of varnishlog because the main purpose of ncsa is to write one line for each requests. You can setup the ?-F ? option to add more HTTP headers if needed. Jonathan Huot Phone: +33(0)1.47.62.78.65 From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Graham Lyons Sent: jeudi 25 avril 2013 12:16 To: varnish-misc at varnish-cache.org Subject: Varnishlog and Splunk Hello, Has anyone had any experience of putting output from varnishlog into Splunk? My experience of Splunk so far has involved access log type sources with events on separate lines, which is obviously quite different to what comes out of varnishlog. If there's any prior art it would interesting to hear. Thanks, Graham. ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. --------------------- This email was sent to you by Thomson Reuters, the global news and information company. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Thomson Reuters. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aashisn at hotmail.com Sun Apr 14 10:47:05 2013 From: aashisn at hotmail.com (Ashish) Date: Sun, 14 Apr 2013 10:47:05 -0000 Subject: varnish ssl Message-ID: I am setting up varnish as caching+entry point for public traffic. Public => varnish(x2) => loadbalancer => Web servers (x4) We have around 15 domains with ssl support on login/payment pages. I am not quite getting done here. 1) i could point all domins to varnish IP and it could route accordingly, but dont think i can make ssl workout to be sent st. to loadbalancer and then webserver question: Does ssl request gets untouched and sent directly to end server? 2) Can i somehow configure varnish to be stand alone, but point dns to loadbalancer IP's and somehow still manage to get varnish serve cached objects? Please guide me From kbagga at adobe.com Mon Apr 15 12:57:08 2013 From: kbagga at adobe.com (Karishma Bagga) Date: Mon, 15 Apr 2013 12:57:08 -0000 Subject: Varnish sending 304 back to client on cache miss Message-ID: <3875B19FE5897C41A819CB218B0BB300085B6E5A51@indiambx01.corp.adobe.com> Hi I am facing an issue where Varnish sends a "If-Modified-Since" header for a cache-miss object. I will briefly describe the system architecture and the problem for your reference. I have varnish logs saved and can share them as well. 1) We have 4 varnish instances running on machine each fetching requests from one backend server. We also have a load balancer in front of these 4 varnishes. 2) We start Varnish processes using : varnishd -f /etc/varnish/default.vcl -s malloc,2G -T 127.0.0.1:2001 -a 0.0.0.0:8081 -n v1 (similarly for v2, v3 and v4). This should make sure that the caches are in memory and exclusive to each varnish instance. 3) We get 200 OK for a URL, and it's there on delivered from the varnish cache; suddenly we get a 304 Not modified in one of the clients and that messes up our playback (this is usually when varnish is under load) 4) On analysis, I figured out that If-Modified-Since header was received at the server end (I can see the header in the Apache logs), since the object was not modified, server returned a 304. But till then the object was available in the varnish cache. Did Varnish try to invalidate it's cache (as the object ttl was set to 2 seconds). If it did so, shouldn't it keep the object in cache if the server returns a 304? 5) I don't see client adding this header 6) If I do varnishlog -b, I see a request for this URL from varnish(say v1) to backend and also this was a cache miss. My doubt here is, how does Varnish add If-Modified-Since header if it was a cache miss? Thanks Karishma -------------- next part -------------- An HTML attachment was scrubbed... URL: From kbagga at adobe.com Mon Apr 15 13:09:32 2013 From: kbagga at adobe.com (Karishma Bagga) Date: Mon, 15 Apr 2013 13:09:32 -0000 Subject: Varnish sending 304 back to client on cache miss Message-ID: <3875B19FE5897C41A819CB218B0BB300085B6E5A54@indiambx01.corp.adobe.com> My problem is the same as in https://www.varnish-cache.org/lists/pipermail/varnish-misc/2013-April/022991.html From: Karishma Bagga Sent: Monday, April 15, 2013 6:27 PM To: 'varnish-misc at varnish-cache.org' Subject: Varnish sending 304 back to client on cache miss Hi I am facing an issue where Varnish sends a "If-Modified-Since" header for a cache-miss object. I will briefly describe the system architecture and the problem for your reference. I have varnish logs saved and can share them as well. 1) We have 4 varnish instances running on machine each fetching requests from one backend server. We also have a load balancer in front of these 4 varnishes. 2) We start Varnish processes using : varnishd -f /etc/varnish/default.vcl -s malloc,2G -T 127.0.0.1:2001 -a 0.0.0.0:8081 -n v1 (similarly for v2, v3 and v4). This should make sure that the caches are in memory and exclusive to each varnish instance. 3) We get 200 OK for a URL, and it's there on delivered from the varnish cache; suddenly we get a 304 Not modified in one of the clients and that messes up our playback (this is usually when varnish is under load) 4) On analysis, I figured out that If-Modified-Since header was received at the server end (I can see the header in the Apache logs), since the object was not modified, server returned a 304. But till then the object was available in the varnish cache. Did Varnish try to invalidate it's cache (as the object ttl was set to 2 seconds). If it did so, shouldn't it keep the object in cache if the server returns a 304? 5) I don't see client adding this header 6) If I do varnishlog -b, I see a request for this URL from varnish(say v1) to backend and also this was a cache miss. My doubt here is, how does Varnish add If-Modified-Since header if it was a cache miss? Thanks Karishma -------------- next part -------------- An HTML attachment was scrubbed... URL: