From on at cs.ait.ac.th Wed Apr 1 02:31:16 2009 From: on at cs.ait.ac.th (Olivier Nicole) Date: Wed, 1 Apr 2009 09:31:16 +0700 (ICT) Subject: No subject In-Reply-To: (message from Koteswara Rao on Tue, 31 Mar 2009 17:35:41 +0530) References: Message-ID: <200904010231.n312VG3j072724@banyan.cs.ait.ac.th> Hi, > I am using Varnish on Free BSD. I am using varanishncsa to output apache st= > yle logs from Varnish. My question is=2C how do I get varnishncsa to work w= > ith cronolog to rotate logs? I am using native newsyslog from FreeBSD and simply send a SIGHUP to varnishncsa process: in /etc/newsyslog.conf /var/log/varnish-ncsa.log 644 7 * @T00 JC /var/run/varnishncsa.pid Olivier From on at cs.ait.ac.th Wed Apr 1 04:13:20 2009 From: on at cs.ait.ac.th (Olivier Nicole) Date: Wed, 1 Apr 2009 11:13:20 +0700 (ICT) Subject: Varnish on FreeBSD In-Reply-To: <86wsa6x85e.fsf@ds4.des.no> (message from =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= on Tue, 31 Mar 2009 13:07:41 +0200) References: <200903310321.n2V3LYlU022699@banyan.cs.ait.ac.th> <86wsa6x85e.fsf@ds4.des.no> Message-ID: <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> Hi, > Why use both malloc and file? Varnish will spread objects equally > between both storage backends. You'd be better off using just the file. OK, I removed that :) > > The file system used for storage has 407 GB available and Varnish > > plans to use up to 1626 GB out of that, that is far more than the 50%. > > Try instrumenting str2bytes() in lib/libvarnish/num.c to show the > details of the calculations. In str2bytes(), p is a pointer to the size > specification ("50%" in your case), fval is the numeric value (should be > 50), rel is the size of the file system in bytes (should be > approximately 407*2^30). If I am not mistaken, adding printf("%ju\n", rel); at the begining of the function str2bytes should be enough, well I am sorry to say that it prints: 3492003971072, about 3 TB, which is compatible with the proposed 1665117 MB for the file size. So I went back one step further and looked at smf_fsspace; the values in fsst.f_bsize and fsst.f_bavail are consistent with the value of rel, but are not corresponding to the size available in the file system. I will dig a bit further and keep you posted. Best regads, Olivier From plfgoa at gmail.com Wed Apr 1 06:20:54 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 1 Apr 2009 11:50:54 +0530 Subject: Syntax change Message-ID: <75cf5800903312320k5be61b3ck5cc2b1e856990d9@mail.gmail.com> Hi, I downloaded varnish from trunk and installed it but when tried to restart it using the previous version .vcl file it gave following error. Message from VCC-compiler: Variable 'obj.cacheable' not accessible in method 'vcl_fetch'. Where can I get syntax change . Is it listed on varnish site ? Thank you -Paras From plfgoa at gmail.com Wed Apr 1 07:00:56 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 1 Apr 2009 12:30:56 +0530 Subject: Syntax change : obj.grace Message-ID: <75cf5800904010000pfa36af3sfc93c8c0123180e@mail.gmail.com> Hi, Is "obj.grace" now changed to "beresp.grace" for the code in varnish trunk ? Also is it necessary to have "return " in the .vcl confg file ? Example return(pass) or just "pass" would be sufficient ? Thank you. -Paras From on at cs.ait.ac.th Wed Apr 1 07:47:50 2009 From: on at cs.ait.ac.th (Olivier Nicole) Date: Wed, 1 Apr 2009 14:47:50 +0700 (ICT) Subject: Varnish on FreeBSD In-Reply-To: <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> (message from Olivier Nicole on Wed, 1 Apr 2009 11:13:20 +0700 (ICT)) References: <200903310321.n2V3LYlU022699@banyan.cs.ait.ac.th> <86wsa6x85e.fsf@ds4.des.no> <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> Message-ID: <200904010747.n317loMx049447@banyan.cs.ait.ac.th> Hi again, After some diging I have the answer. On FreeBSD, from the man page of fstatvfs: f_bsize The preferred length of I/O requests for files on this file system. (Corresponds to the f_iosize member of struct statfs.) It is not said that f_bsize is the block size, while f_bavail is given in blocks. So in the function smf_fsspace: return (fsst.f_bsize * fsst.f_bavail); is over estimating. On Linux it seems that f_bsize is the block size, but I have no man fstatvfs available at hand. I ran a simple test: #include #include #include #include main () { int fp; int r; struct statvfs buf; r=statvfs("/tmp/on", &buf); if (r==0) { printf("%lu %llu\n", buf.f_bsize, buf.f_bavail); } else { perror("me"); } } $ a.out 16384 2297104 $ df / Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mfid0s1a 5077038 76668 4594208 2% / $ If the block size of 2KB, the results are consistant. Best regards, Olivier From des at des.no Wed Apr 1 14:16:22 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 01 Apr 2009 16:16:22 +0200 Subject: Varnish on FreeBSD In-Reply-To: <200904010747.n317loMx049447@banyan.cs.ait.ac.th> (Olivier Nicole's message of "Wed, 1 Apr 2009 14:47:50 +0700 (ICT)") References: <200903310321.n2V3LYlU022699@banyan.cs.ait.ac.th> <86wsa6x85e.fsf@ds4.des.no> <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> <200904010747.n317loMx049447@banyan.cs.ait.ac.th> Message-ID: <864ox831e1.fsf@ds4.des.no> Olivier Nicole writes: > After some diging I have the answer. > > On FreeBSD, from the man page of fstatvfs: > > f_bsize The preferred length of I/O requests for files on this > file system. (Corresponds to the f_iosize member of > struct statfs.) Yes, we should use f_frsize instead. Please try the attached patch. > $ a.out > 16384 2297104 I imagine this is the stripe size of your array? DES -- Dag-Erling Sm?rgrav - des at des.no -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishd_bsize.diff Type: text/x-patch Size: 971 bytes Desc: not available URL: From tnguyen at bleacherreport.com Wed Apr 1 14:24:03 2009 From: tnguyen at bleacherreport.com (Tung Nguyen) Date: Wed, 1 Apr 2009 07:24:03 -0700 Subject: requests keep timing out Message-ID: Hey guys, Our requests seem to be getting timed out. I set all the timeout parameters pretty high, so wondering what could be causing it. Also, how do I check out the number of current threads being used in varnishstat? Thanks, Tung # ps auxww | grep varnish root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:00 varnishlog -w varnish.log root 4301 0.0 0.0 110552 1136 ? S From tnguyen at bleacherreport.com Wed Apr 1 14:47:51 2009 From: tnguyen at bleacherreport.com (Tung Nguyen) Date: Wed, 1 Apr 2009 07:47:51 -0700 Subject: requests keep timing out In-Reply-To: References: Message-ID: Here's the full varnishstat -1 varnishstat -1 uptime 212 . Child uptime client_conn 1518 7.16 Client connections accepted client_req 5662 26.71 Client requests received cache_hit 3230 15.24 Cache hits cache_hitpass 8 0.04 Cache hits for pass cache_miss 561 2.65 Cache misses backend_conn 2307 10.88 Backend connections success backend_unhealthy 0 0.00 Backend connections not attempted backend_busy 0 0.00 Backend connections too many backend_fail 0 0.00 Backend connections failures backend_reuse 1920 9.06 Backend connections reuses backend_recycle 2038 9.61 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 365 . N struct srcaddr n_srcaddr_act 0 . N active struct srcaddr n_sess_mem 952 . N struct sess_mem n_sess 1 . N struct sess n_object 750 . N struct object n_objecthead 1079 . N struct objecthead n_smf 1272 . N struct smf n_smf_frag 98 . N small free smf n_smf_large 2 . N large free smf n_vbe_conn 4 . N struct vbe_conn n_bereq 88 . N struct bereq n_wrk 1600 . N worker threads n_wrk_create 1600 7.55 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 980 4.62 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 6 . N backends n_expired 54 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 833 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 4589 21.65 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 1517 7.16 Total Sessions s_req 5662 26.71 Total Requests s_pipe 0 0.00 Total pipe s_pass 1871 8.83 Total pass s_fetch 2307 10.88 Total fetch s_hdrbytes 1673491 7893.83 Total header bytes s_bodybytes 26560497 125285.36 Total body bytes sess_closed 249 1.17 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 5428 25.60 Session herd shm_records 308891 1457.03 SHM records shm_writes 20426 96.35 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 14 0.07 SHM MTX contention shm_cycles 0 0.00 SHM cycles through buffer sm_nreq 5117 24.14 allocator requests sm_nobj 1172 . outstanding allocations sm_balloc 10076160 . bytes allocated sm_bfree 5983641600 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 125 0.59 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 58250 . SMS bytes allocated sms_bfree 58250 . SMS bytes freed backend_req 2307 10.88 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) Tung On Wed, Apr 1, 2009 at 7:24 AM, Tung Nguyen wrote: > Hey guys, > > Our requests seem to be getting timed out. I set all the timeout > parameters pretty high, so wondering what could be causing it. > > Also, how do I check out the number of current threads being used in > varnishstat? > > Thanks, > Tung > > # ps auxww | grep varnish > root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:00 varnishlog > -w varnish.log > root 4301 0.0 0.0 110552 1136 ? S /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p > sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p > listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f > /etc/varnish/default.vcl > nobody 4302 0.0 1.1 7781688 39468 ? S /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p > sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p > listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f > /etc/varnish/default.vcl > root 4872 0.0 0.0 3876 556 pts/0 R<+ 14:22 0:00 grep > --colour=auto varnish > # > > > 0 VCL_call - prefetch > 0 VCL_return - fetch > 0 Debug - "Attempt Prefetch 1804102565" > 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 > HTTP/1.1 200 OK > 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001500 > HTTP/1.1 200 OK > 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 > HTTP/1.1 200 OK > 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 > HTTP/1.1 200 OK > 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000042 > HTTP/1.1 200 OK > 0 ExpPick - 1804096341 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1804096341 -30 > 0 ExpPick - 1804096350 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1804096350 -30 > 0 ExpPick - 1804096354 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1804096354 -30 > 0 ExpPick - 1804096376 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1804096376 -30 > 0 ExpPick - 1804096384 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1804096384 -30 > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1238593700 1.0 > 0 Backend_health - A5 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 > HTTP/1.1 200 OK > 0 ExpPick - 1804102829 prefetch > 0 VCL_call - prefetch > 0 VCL_return - fetch > 0 Debug - "Attempt Prefetch 1804102829" > 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 > HTTP/1.1 200 OK > 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001125 > HTTP/1.1 200 OK > 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 > HTTP/1.1 200 OK > 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 > HTTP/1.1 200 OK > 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000032 > HTTP/1.1 200 OK > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1238593703 1.0 > 0 ExpPick - 1803925354 ttl > 0 VCL_call - timeout > > > param.show > 200 2137 > accept_fd_holdoff 50 [ms] > acceptor default (epoll, poll) > auto_restart on [bool] > backend_http11 on [bool] > between_bytes_timeout 60.000000 [s] > cache_vbe_conns off [bool] > cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" > cli_buffer 8192 [bytes] > cli_timeout 5 [seconds] > client_http11 off [bool] > clock_skew 10 [s] > connect_timeout 1.000000 [s] > default_grace 10 > default_ttl 120 [seconds] > diag_bitmap 0x0 [bitmap] > err_ttl 0 [seconds] > esi_syntax 0 [bitmap] > fetch_chunksize 128 [kilobytes] > first_byte_timeout 60.000000 [s] > group nobody (65534) > listen_address :6081 > listen_depth 8192 [connections] > log_hashstring off [bool] > log_local_address off [bool] > lru_interval 2 [seconds] > max_esi_includes 5 [includes] > max_restarts 4 [restarts] > obj_workspace 8192 [bytes] > overflow_max 100 [%] > ping_interval 3 [seconds] > pipe_timeout 60 [seconds] > prefer_ipv6 off [bool] > purge_dups off [bool] > purge_hash on [bool] > rush_exponent 3 [requests per request] > send_timeout 600 [seconds] > sess_timeout 10 [seconds] > sess_workspace 32768 [bytes] > session_linger 0 [ms] > shm_reclen 255 [bytes] > shm_workspace 8192 [bytes] > srcaddr_hash 1049 [buckets] > srcaddr_ttl 30 [seconds] > thread_pool_add_delay 20 [milliseconds] > thread_pool_add_threshold 2 [requests] > thread_pool_fail_delay 200 [milliseconds] > thread_pool_max 500 [threads] > thread_pool_min 100 [threads] > thread_pool_purge_delay 1000 [milliseconds] > thread_pool_timeout 300 [seconds] > thread_pools 2 [pools] > user nobody (65534) > vcl_trace off [bool] > > > > 0+00:01:23 ey03-s00419 > Hitrate ratio: 4 4 4 > Hitrate avg: 0.7154 0.7154 0.7154 > > > 2111 0.00 25.43 Client connections accepted > > 8315 0.00 100.18 Client requests received > > 5019 0.00 60.47 Cache hits > > 6 0.00 0.07 Cache hits for pass > > 805 0.00 9.70 Cache misses > > 2848 0.00 34.31 Backend connections success > > 2471 0.00 29.77 Backend connections reuses > > 2691 0.00 32.42 Backend connections recycles > > 456 . . N struct srcaddr > > 27 . . N active struct srcaddr > > 995 . . N struct sess_mem > > 55 . . N struct sess > > 584 . . N struct object > > 670 . . N struct objecthead > > 1147 . . N struct smf > > 1 . . N small free smf > > 1 . . N large free smf > > 6 . . N struct vbe_conn > > 22 . . N struct bereq > > 200 . . N worker threads > > 200 0.00 2.41 N worker threads created > > 57 0.00 0.69 N overflowed work requests > > 6 . . N backends > > 2046 . . N LRU moved objects > > 6277 0.00 75.63 Objects sent with write > > 2110 0.00 25.42 Total Sessions > > 8314 0.00 100.17 Total Requests > > 2491 0.00 30.01 Total pass > > 2847 0.00 34.30 Total fetch > > 2410204 0.00 29038.60 Total header bytes > > 38160409 0.00 459763.96 Total body bytes > > 605 0.00 7.29 Session Closed > > 2 0.00 0.02 Session Pipeline > > 6 0.00 0.07 Session Read Ahead > > 7734 0.00 93.18 Session herd > > 428358 12.00 5160.94 SHM records > > 26190 12.00 315.54 SHM writes > > 2 0.00 0.02 SHM MTX contention > > 6378 0.00 76.84 allocator requests > > 1145 . . outstanding allocations > > 12103680 . . bytes allocated > > 6028206080 . . bytes free > > 448 0.00 5.40 SMS allocator requests > > 208768 . . SMS bytes allocated > > 208768 . . SMS bytes freed > > 2848 0.00 34.31 Backend requests made > > 1 0.00 0.01 N vcl total > > 1 0.00 0.01 N vcl available > > 1 . . N total active purges > > 1 0.00 0.01 N new purges added > > > > > Thanks, > Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From tnguyen at bleacherreport.com Wed Apr 1 15:29:33 2009 From: tnguyen at bleacherreport.com (Tung Nguyen) Date: Wed, 1 Apr 2009 08:29:33 -0700 Subject: requests keep timing out In-Reply-To: References: Message-ID: And here's the backtrace I tried to capture from one the hunged varnishes. # ps auxww | grep varnish root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:01 varnishlog -w varnish.log root 25199 0.0 0.0 110556 1140 ? S This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/libthread_db.so.1". (no debugging symbols found) Attaching to program: /usr/sbin/varnishd, process 25200 Error while mapping shared library sections: ./vcl.1P9zoqAU.so: No such file or directory. Reading symbols from /usr/lib64/libvarnish.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvarnish.so.1.0.0.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libvarnish.so.1 Reading symbols from /usr/lib64/libvarnishcompat.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvarnishcompat.so.1.0.0.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libvarnishcompat.so.1 Reading symbols from /usr/lib64/libvcl.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvcl.so.1.0.0.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libvcl.so.1 Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib64/debug/lib64/libdl-2.6.1.so.debug...done. done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib64/debug/lib64/libpthread-2.6.1.so.debug...done. [Thread debugging using libthread_db enabled] [New Thread 0x2ac6f560c440 (LWP 25200)] [New Thread 0x2aaf18807950 (LWP 26852)] ..... Loaded symbols for /lib/libnss_files.so.2 Symbol file not found for ./vcl.1P9zoqAU.so 0x00002ac6f5179c46 in poll () from /lib/libc.so.6 (gdb) bt #0 0x00002ac6f5179c46 in poll () from /lib/libc.so.6 #1 0x0000000000411e95 in CLI_Run () #2 0x000000000041a5e3 in child_main () #3 0x000000000042954c in start_child () #4 0x000000000042985c in mgt_run () #5 0x000000000043316a in main () (gdb) cont Continuing. Program received signal SIGQUIT, Quit. [Switching to Thread 0x40800950 (LWP 25201)] 0x00002ac6f5153a51 in nanosleep () from /lib/libc.so.6 (gdb) exit Undefined command: "exit". Try "help". (gdb) quit The program is running. Quit anyway (and detach it)? (y or n) y Quitting: Can't detach LWP 25949: No such process Any help is appreciated. Tung On Wed, Apr 1, 2009 at 7:47 AM, Tung Nguyen wrote: > Here's the full varnishstat -1 > > varnishstat -1 > uptime 212 . Child uptime > client_conn 1518 7.16 Client connections accepted > client_req 5662 26.71 Client requests received > cache_hit 3230 15.24 Cache hits > cache_hitpass 8 0.04 Cache hits for pass > cache_miss 561 2.65 Cache misses > backend_conn 2307 10.88 Backend connections success > backend_unhealthy 0 0.00 Backend connections not attempted > backend_busy 0 0.00 Backend connections too many > backend_fail 0 0.00 Backend connections failures > backend_reuse 1920 9.06 Backend connections reuses > backend_recycle 2038 9.61 Backend connections recycles > backend_unused 0 0.00 Backend connections unused > n_srcaddr 365 . N struct srcaddr > n_srcaddr_act 0 . N active struct srcaddr > n_sess_mem 952 . N struct sess_mem > n_sess 1 . N struct sess > n_object 750 . N struct object > n_objecthead 1079 . N struct objecthead > n_smf 1272 . N struct smf > n_smf_frag 98 . N small free smf > n_smf_large 2 . N large free smf > n_vbe_conn 4 . N struct vbe_conn > n_bereq 88 . N struct bereq > n_wrk 1600 . N worker threads > n_wrk_create 1600 7.55 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_queue 0 0.00 N queued work requests > n_wrk_overflow 980 4.62 N overflowed work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 6 . N backends > n_expired 54 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_saved 0 . N LRU saved objects > n_lru_moved 833 . N LRU moved objects > n_deathrow 0 . N objects on deathrow > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 4589 21.65 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 1517 7.16 Total Sessions > s_req 5662 26.71 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 1871 8.83 Total pass > s_fetch 2307 10.88 Total fetch > s_hdrbytes 1673491 7893.83 Total header bytes > s_bodybytes 26560497 125285.36 Total body bytes > sess_closed 249 1.17 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 0 0.00 Session Linger > sess_herd 5428 25.60 Session herd > shm_records 308891 1457.03 SHM records > shm_writes 20426 96.35 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 14 0.07 SHM MTX contention > shm_cycles 0 0.00 SHM cycles through buffer > sm_nreq 5117 24.14 allocator requests > sm_nobj 1172 . outstanding allocations > sm_balloc 10076160 . bytes allocated > sm_bfree 5983641600 . bytes free > sma_nreq 0 0.00 SMA allocator requests > sma_nobj 0 . SMA outstanding allocations > sma_nbytes 0 . SMA outstanding bytes > sma_balloc 0 . SMA bytes allocated > sma_bfree 0 . SMA bytes free > sms_nreq 125 0.59 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 58250 . SMS bytes allocated > sms_bfree 58250 . SMS bytes freed > backend_req 2307 10.88 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_purge 1 . N total active purges > n_purge_add 1 0.00 N new purges added > n_purge_retire 0 0.00 N old purges deleted > n_purge_obj_test 0 0.00 N objects tested > n_purge_re_test 0 0.00 N regexps tested against > n_purge_dups 0 0.00 N duplicate purges removed > hcb_nolock 0 0.00 HCB Lookups without lock > hcb_lock 0 0.00 HCB Lookups with lock > hcb_insert 0 0.00 HCB Inserts > esi_parse 0 0.00 Objects ESI parsed (unlock) > esi_errors 0 0.00 ESI parse errors (unlock) > > > Tung > > > On Wed, Apr 1, 2009 at 7:24 AM, Tung Nguyen wrote: > >> Hey guys, >> >> Our requests seem to be getting timed out. I set all the timeout >> parameters pretty high, so wondering what could be causing it. >> >> Also, how do I check out the number of current threads being used in >> varnishstat? >> >> Thanks, >> Tung >> >> # ps auxww | grep varnish >> root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:00 >> varnishlog -w varnish.log >> root 4301 0.0 0.0 110552 1136 ? S> /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p >> sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p >> listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f >> /etc/varnish/default.vcl >> nobody 4302 0.0 1.1 7781688 39468 ? S> /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p >> sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p >> listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f >> /etc/varnish/default.vcl >> root 4872 0.0 0.0 3876 556 pts/0 R<+ 14:22 0:00 grep >> --colour=auto varnish >> # >> >> >> 0 VCL_call - prefetch >> 0 VCL_return - fetch >> 0 Debug - "Attempt Prefetch 1804102565" >> 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 >> HTTP/1.1 200 OK >> 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001500 >> HTTP/1.1 200 OK >> 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 >> HTTP/1.1 200 OK >> 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 >> HTTP/1.1 200 OK >> 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000042 >> HTTP/1.1 200 OK >> 0 ExpPick - 1804096341 ttl >> 0 VCL_call - timeout >> 0 VCL_return - discard >> 0 ExpKill - 1804096341 -30 >> 0 ExpPick - 1804096350 ttl >> 0 VCL_call - timeout >> 0 VCL_return - discard >> 0 ExpKill - 1804096350 -30 >> 0 ExpPick - 1804096354 ttl >> 0 VCL_call - timeout >> 0 VCL_return - discard >> 0 ExpKill - 1804096354 -30 >> 0 ExpPick - 1804096376 ttl >> 0 VCL_call - timeout >> 0 VCL_return - discard >> 0 ExpKill - 1804096376 -30 >> 0 ExpPick - 1804096384 ttl >> 0 VCL_call - timeout >> 0 VCL_return - discard >> 0 ExpKill - 1804096384 -30 >> 0 CLI - Rd ping >> 0 CLI - Wr 0 200 PONG 1238593700 1.0 >> 0 Backend_health - A5 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 >> HTTP/1.1 200 OK >> 0 ExpPick - 1804102829 prefetch >> 0 VCL_call - prefetch >> 0 VCL_return - fetch >> 0 Debug - "Attempt Prefetch 1804102829" >> 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 >> HTTP/1.1 200 OK >> 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001125 >> HTTP/1.1 200 OK >> 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 >> HTTP/1.1 200 OK >> 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 >> HTTP/1.1 200 OK >> 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000032 >> HTTP/1.1 200 OK >> 0 CLI - Rd ping >> 0 CLI - Wr 0 200 PONG 1238593703 1.0 >> 0 ExpPick - 1803925354 ttl >> 0 VCL_call - timeout >> >> >> param.show >> 200 2137 >> accept_fd_holdoff 50 [ms] >> acceptor default (epoll, poll) >> >> auto_restart on [bool] >> backend_http11 on [bool] >> between_bytes_timeout 60.000000 [s] >> cache_vbe_conns off [bool] >> cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" >> >> cli_buffer 8192 [bytes] >> cli_timeout 5 [seconds] >> client_http11 off [bool] >> clock_skew 10 [s] >> connect_timeout 1.000000 [s] >> default_grace 10 >> default_ttl 120 [seconds] >> diag_bitmap 0x0 [bitmap] >> err_ttl 0 [seconds] >> esi_syntax 0 [bitmap] >> fetch_chunksize 128 [kilobytes] >> first_byte_timeout 60.000000 [s] >> group nobody (65534) >> listen_address :6081 >> listen_depth 8192 [connections] >> log_hashstring off [bool] >> log_local_address off [bool] >> lru_interval 2 [seconds] >> max_esi_includes 5 [includes] >> max_restarts 4 [restarts] >> obj_workspace 8192 [bytes] >> overflow_max 100 [%] >> ping_interval 3 [seconds] >> pipe_timeout 60 [seconds] >> prefer_ipv6 off [bool] >> purge_dups off [bool] >> purge_hash on [bool] >> rush_exponent 3 [requests per request] >> >> send_timeout 600 [seconds] >> sess_timeout 10 [seconds] >> sess_workspace 32768 [bytes] >> session_linger 0 [ms] >> shm_reclen 255 [bytes] >> shm_workspace 8192 [bytes] >> srcaddr_hash 1049 [buckets] >> srcaddr_ttl 30 [seconds] >> thread_pool_add_delay 20 [milliseconds] >> thread_pool_add_threshold 2 [requests] >> thread_pool_fail_delay 200 [milliseconds] >> thread_pool_max 500 [threads] >> thread_pool_min 100 [threads] >> thread_pool_purge_delay 1000 [milliseconds] >> thread_pool_timeout 300 [seconds] >> thread_pools 2 [pools] >> user nobody (65534) >> vcl_trace off [bool] >> >> >> >> >> 0+00:01:23 ey03-s00419 >> >> Hitrate ratio: 4 4 4 >> Hitrate avg: 0.7154 0.7154 0.7154 >> >> >> >> 2111 0.00 25.43 Client connections accepted >> >> >> 8315 0.00 100.18 Client requests received >> >> >> 5019 0.00 60.47 Cache hits >> >> >> 6 0.00 0.07 Cache hits for pass >> >> >> 805 0.00 9.70 Cache misses >> >> >> 2848 0.00 34.31 Backend connections success >> >> >> 2471 0.00 29.77 Backend connections reuses >> >> >> 2691 0.00 32.42 Backend connections recycles >> >> >> 456 . . N struct srcaddr >> >> >> 27 . . N active struct srcaddr >> >> >> 995 . . N struct sess_mem >> >> >> 55 . . N struct sess >> >> >> 584 . . N struct object >> >> >> 670 . . N struct objecthead >> >> >> 1147 . . N struct smf >> >> >> 1 . . N small free smf >> >> >> 1 . . N large free smf >> >> >> 6 . . N struct vbe_conn >> >> >> 22 . . N struct bereq >> >> >> 200 . . N worker threads >> >> >> 200 0.00 2.41 N worker threads created >> >> >> 57 0.00 0.69 N overflowed work requests >> >> >> 6 . . N backends >> >> >> 2046 . . N LRU moved objects >> >> >> 6277 0.00 75.63 Objects sent with write >> >> >> 2110 0.00 25.42 Total Sessions >> >> >> 8314 0.00 100.17 Total Requests >> >> >> 2491 0.00 30.01 Total pass >> >> >> 2847 0.00 34.30 Total fetch >> >> >> 2410204 0.00 29038.60 Total header bytes >> >> >> 38160409 0.00 459763.96 Total body bytes >> >> >> 605 0.00 7.29 Session Closed >> >> >> 2 0.00 0.02 Session Pipeline >> >> >> 6 0.00 0.07 Session Read Ahead >> >> >> 7734 0.00 93.18 Session herd >> >> >> 428358 12.00 5160.94 SHM records >> >> >> 26190 12.00 315.54 SHM writes >> >> >> 2 0.00 0.02 SHM MTX contention >> >> >> 6378 0.00 76.84 allocator requests >> >> >> 1145 . . outstanding allocations >> >> >> 12103680 . . bytes allocated >> >> >> 6028206080 . . bytes free >> >> >> 448 0.00 5.40 SMS allocator requests >> >> >> 208768 . . SMS bytes allocated >> >> >> 208768 . . SMS bytes freed >> >> >> 2848 0.00 34.31 Backend requests made >> >> >> 1 0.00 0.01 N vcl total >> >> >> 1 0.00 0.01 N vcl available >> >> >> 1 . . N total active purges >> >> >> 1 0.00 0.01 N new purges added >> >> >> >> >> >> Thanks, >> Tung > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From on at cs.ait.ac.th Thu Apr 2 04:46:17 2009 From: on at cs.ait.ac.th (Olivier Nicole) Date: Thu, 2 Apr 2009 11:46:17 +0700 (ICT) Subject: Varnish on FreeBSD In-Reply-To: <864ox831e1.fsf@ds4.des.no> (message from =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= on Wed, 01 Apr 2009 16:16:22 +0200) References: <200903310321.n2V3LYlU022699@banyan.cs.ait.ac.th> <86wsa6x85e.fsf@ds4.des.no> <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> <200904010747.n317loMx049447@banyan.cs.ait.ac.th> <864ox831e1.fsf@ds4.des.no> Message-ID: <200904020446.n324kHE8001615@banyan.cs.ait.ac.th> Hello, > Yes, we should use f_frsize instead. Please try the attached patch. It worked, but I installed it by hand, I am not sure what version of Varnish it applies too, neither 2.0.3 (installed) nor 2.0.4 (downloaded for testing) have a a file bin/varnishd/stevedore_utils.c or a function stv_fsspace(). I applied the patch to bin/varnishd/storage_file.c and the function smf_fsspace() and it worked just fine. > > $ a.out > > 16384 2297104 > I imagine this is the stripe size of your array? I am not sure what you mean, the values represents f_bsize and f_bavail. Best regards, Olivier From tical.net at gmail.com Thu Apr 2 04:01:55 2009 From: tical.net at gmail.com (Ray Barnes) Date: Thu, 2 Apr 2009 00:01:55 -0400 Subject: varnish stopped answering :80 Message-ID: Ahoy, Today a site that I'm caching with Varnish got hit with a massive amount of traffic after being plugged on a syndicated radio show in the US. What happened is that it stopped answering port 80 after a while. The daemon was still running, and I did not think to telnet in and look at any of the stats - I simply restarted it to get the site back online. The daemon is being run like this: varnishd -a :80 -b ip.ip.ip.ip:80 -T ip.ip.ip.ip:6082 -s malloc -w200,1200,300 . Platform is Linux 2.6 (RHEL 5.2). Is there anything blatantly wrong with this syntax? Anything I should add/fix/whatever? Any help is appreciated. Thanks! -Ray From tfheen at redpill-linpro.com Thu Apr 2 09:50:35 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Thu, 02 Apr 2009 11:50:35 +0200 Subject: Syntax change : obj.grace In-Reply-To: <75cf5800904010000pfa36af3sfc93c8c0123180e@mail.gmail.com> (Paras Fadte's message of "Wed, 1 Apr 2009 12:30:56 +0530") References: <75cf5800904010000pfa36af3sfc93c8c0123180e@mail.gmail.com> Message-ID: <87eiwbpeok.fsf@qurzaw.linpro.no> ]] Paras Fadte | Is "obj.grace" now changed to "beresp.grace" for the code in varnish | trunk ? Yes. | Also is it necessary to have "return " in the .vcl confg file | ? Example return(pass) or just "pass" would be sufficient ? ?pass? alone is still supported, but the recommended syntax is to use an explicit return. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From des at des.no Thu Apr 2 11:37:09 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 02 Apr 2009 13:37:09 +0200 Subject: Varnish on FreeBSD In-Reply-To: <200904020446.n324kHE8001615@banyan.cs.ait.ac.th> (Olivier Nicole's message of "Thu, 2 Apr 2009 11:46:17 +0700 (ICT)") References: <200903310321.n2V3LYlU022699@banyan.cs.ait.ac.th> <86wsa6x85e.fsf@ds4.des.no> <200904010413.n314DK9Z052279@banyan.cs.ait.ac.th> <200904010747.n317loMx049447@banyan.cs.ait.ac.th> <864ox831e1.fsf@ds4.des.no> <200904020446.n324kHE8001615@banyan.cs.ait.ac.th> Message-ID: <86ocvfwal6.fsf@ds4.des.no> Olivier Nicole writes: > "Dag-Erling Sm?rgrav" writes: > > Olivier Nicole writes: > > > $ a.out > > > 16384 2297104 > > I imagine this is the stripe size of your array? > I am not sure what you mean, the values represents f_bsize and > f_bavail. Yes. I imagine that in this case, f_bsize corresponds to the stripe size of your array. DES -- Dag-Erling Sm?rgrav - des at des.no From suran007 at sina.com Fri Apr 3 02:39:56 2009 From: suran007 at sina.com (suran007 at sina.com) Date: Fri, 03 Apr 2009 10:39:56 +0800 Subject: How to let varnishncsa to log the realserver ip Message-ID: <20090403023956.BFDA5A4980D@mail3-51.sinamail.sina.com.cn> hello! I have a questtion, that is How to let varnishncsa log the backend realserver ip to varnish.log?I search in google,but no result,anyone can help me ? thanks!! ??????????????????? ??????????????????? ------------------------------------------------------------------- ???????????????????(http://space.sina.com.cn/ ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishek.rhce at gmail.com Fri Apr 3 10:11:13 2009 From: abhishek.rhce at gmail.com (Abhishek Singh) Date: Fri, 3 Apr 2009 15:41:13 +0530 Subject: varnish-misc Digest, Vol 37, Issue 6 In-Reply-To: References: Message-ID: <25daf17a0904030311v4b933a29r9142e2f13490ed4@mail.gmail.com> On Fri, Apr 3, 2009 at 3:30 PM, wrote: > Send varnish-misc mailing list submissions to > varnish-misc at projects.linpro.no > > To subscribe or unsubscribe via the World Wide Web, visit > http://projects.linpro.no/mailman/listinfo/varnish-misc > or, via email, send a message with subject or body 'help' to > varnish-misc-request at projects.linpro.no > > You can reach the person managing the list at > varnish-misc-owner at projects.linpro.no > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of varnish-misc digest..." > > > Today's Topics: > > 1. Re: Varnish on FreeBSD (Dag-Erling Sm?rgrav) > 2. How to let varnishncsa to log the realserver ip > (suran007 at sina.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 02 Apr 2009 13:37:09 +0200 > From: Dag-Erling Sm?rgrav > Subject: Re: Varnish on FreeBSD > To: Olivier Nicole > Cc: varnish-misc at projects.linpro.no > Message-ID: <86ocvfwal6.fsf at ds4.des.no> > Content-Type: text/plain; charset=utf-8 > > Olivier Nicole writes: > > "Dag-Erling Sm?rgrav" writes: > > > Olivier Nicole writes: > > > > $ a.out > > > > 16384 2297104 > > > I imagine this is the stripe size of your array? > > I am not sure what you mean, the values represents f_bsize and > > f_bavail. > > Yes. I imagine that in this case, f_bsize corresponds to the stripe > size of your array. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no > > > ------------------------------ > > Message: 2 > Date: Fri, 03 Apr 2009 10:39:56 +0800 > From: suran007 at sina.com > Subject: How to let varnishncsa to log the realserver ip > To: varnish-misc > Message-ID: <20090403023956.BFDA5A4980D at mail3-51.sinamail.sina.com.cn> > Content-Type: text/plain; charset="gbk" > > >hello! > >I have a questtion, that is How to let varnishncsa log the backend > realserver ip to varnish.log?I search in google,but no >result,anyone can > help me ? > >thanks!! > put following line vcl_recv VCL. and load your new config file from varnishadmin. remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; > > ??????????????????? > ??????????????????? > > ------------------------------------------------------------------- > ???????????????????(http://space.sina.com.cn/ ) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://projects.linpro.no/pipermail/varnish-misc/attachments/20090403/1d343358/attachment-0001.htm > > ------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > > End of varnish-misc Digest, Vol 37, Issue 6 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaute at pht.no Sat Apr 4 10:25:27 2009 From: gaute at pht.no (Gaute Amundsen) Date: Sat, 4 Apr 2009 12:25:27 +0200 Subject: VCL syntax - unset and remove Message-ID: <200904041225.27400.gaute@pht.no> Various usages: remove req.http.cookie; unset obj.http.Set-Cookie; unset resp.http.magicmarker; Unset is not in man vcl (v2.0.3) but remove is. A clarification would be nice :) Regards Gaute Amundsen From shahar at mindu.co.il Mon Apr 6 14:33:47 2009 From: shahar at mindu.co.il (Shahar Fleischman) Date: Mon, 6 Apr 2009 17:33:47 +0300 Subject: a question Message-ID: <33A2E234C0D4474086DEE805DBD0DC4A8579CA@mindu03.mindu.co.il> Hi, Can anyone explain to me those varnishlog entries mean ? 0 ExpPick - 1369849794 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 1369849794 -10 They always come in this order. Thanks J -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Mon Apr 6 14:42:33 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Mon, 06 Apr 2009 16:42:33 +0200 Subject: a question In-Reply-To: <33A2E234C0D4474086DEE805DBD0DC4A8579CA@mindu03.mindu.co.il> (Shahar Fleischman's message of "Mon, 6 Apr 2009 17:33:47 +0300") References: <33A2E234C0D4474086DEE805DBD0DC4A8579CA@mindu03.mindu.co.il> Message-ID: <86eiw5lu7a.fsf@ds4.des.no> "Shahar Fleischman" writes: > Can anyone explain to me those varnishlog entries mean ? > > 0 ExpPick - 1369849794 ttl > 0 VCL_call - timeout > 0 VCL_return - discard > 0 ExpKill - 1369849794 -10 Object 1369849794 was selected as a candidate for expiry on the basis of its ttl; vcl_timeout was called and returned "discard"; the object was then destroyed. DES -- Dag-Erling Sm?rgrav - des at des.no From alecshenry at gmail.com Tue Apr 7 01:11:53 2009 From: alecshenry at gmail.com (Alecs Henry) Date: Mon, 6 Apr 2009 22:11:53 -0300 Subject: Weird log entries In-Reply-To: <3c54843f0902091229w2914b616sac6806f9c326167c@mail.gmail.com> References: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> <49908FF5.4080501@loman.net> <3c54843f0902091229w2914b616sac6806f9c326167c@mail.gmail.com> Message-ID: <3c54843f0904061811i5653a820ob11216c881f277d@mail.gmail.com> So, after a while digging through other stuff I realized what the "(null)" entries are. They are cache misses. For (apparently) every miss there's an entry on varnishncsa for that miss, and it's logged as a bunch of nulls. I haven't figured out what the other entries are though. I still see some entries (not as many as the null ones) with the date field set to "[00/Jan/1900:00:00:00 +0000]", with all the other information valid (URL, Referer, BackendName, etc..) except for the object size, which is set to "(null)" and ther server response code (also "(null)"). There are also entries that the only field set is the server response (200 or 404 -- these have the date field set correctly) and some entries with the host (or where the backend name should go) set to 127.0.0.1 and the other fields set correctly. One thing though, in none of the entries (correct, broken or plain weird) the object size is set, all I see are "-", except for one entry that had it set (strangely enough, favicon.ico!) and even then, only once. If anybody else has seen this stuff or know anything that might help explain, I'd greatly appreciate some help. I can also send you any information you might deem necessary in order to figure this stuff out. This is the varnish package for debian (2.0.3-2_amd64) all with HTTP/1.1 requests. Thanks! Alecs On Mon, Feb 9, 2009 at 5:29 PM, Alecs Henry wrote: > Hi Nick, > > I forgot to say... It's varnish-trunk. > But I saw it on both varnish 2.0.1 and 2.0.2. > > Alecs > > > On Mon, Feb 9, 2009 at 6:20 PM, Nick Loman wrote: > >> Alecs Henry wrote: >> >> EDITED TO INCLUDE THE ORIGINAL MESSAGE... >>> Hi guys, >>> >>> I was wondering if any of you has seen the following log entries when >>> using varnishncsa: >>> 127.0.0.1 - - [00/Jan/1900:00:00:00 +0000] "(null) (null) (null)" (null) >>> - "-" "-" >>> >>> Those are coupled with: >>> 127.0.0.1 - - [09/Feb/2009:19:39:46 +0000] "(null) (null) (null)" 200 >>> 39678 "-" "-" >>> I can see an object in the page that has that size (image) -- through >>> firebug, but the object didn't load into the browser until I hit reload. >>> >>> And there are also many entries like the following: >>> 127.0.0.1 - - [09/Feb/2009:19:39:52 +0000] "GET >>> http://www.example.com/img/mod.gif HTTP/1.1" 200 951 " >>> http://www.example.com/style.css" "Mozilla/5.0 (X11; U; Linux x86_64; >>> en; rv:1.9.0.5) Gecko/2008121623 Ubuntu/8.10 (intrepid) Firefox/3.0.5" >>> >>> These entries are from communication with the backend server >>> >>> I expected it to show the backend name instead of 127.0.0.1. >>> >>> Is there an explanation? >>> >>> Thanks! >>> >>> Alecs >>> >> >>> >> I've seen log entries like this for HTTP 1.0 requests (using Varnish 1.2), >> as that version of Varnish did not have support for parsing HTTP 1.0 >> headers. >> >> Cheers, >> >> Nick. >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kitai at ya.com Tue Apr 7 07:35:35 2009 From: kitai at ya.com (Kitai) Date: Tue, 7 Apr 2009 09:35:35 +0200 Subject: Weird log entries In-Reply-To: <3c54843f0904061811i5653a820ob11216c881f277d@mail.gmail.com> References: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> <49908FF5.4080501@loman.net> <3c54843f0902091229w2914b616sac6806f9c326167c@mail.gmail.com> <3c54843f0904061811i5653a820ob11216c881f277d@mail.gmail.com> Message-ID: <48d8853d0904070035m733c6aa2n5e6059a283bfa53c@mail.gmail.com> Morning. Same behaviour, same lines, same "nulls" over here, with the same amd64 arch. I just discarded those lines in the filter i use to split varnishncsa logs. Not really a big problem, but something else to take into account apart of crashes :-) David (Kitai) Cruz E-Comm Factory 2009/4/7 Alecs Henry : > So, after a while digging through other stuff I realized what the "(null)" > entries are. > They are cache misses. > For (apparently) every miss there's an entry on varnishncsa for that miss, > and it's logged as a bunch of nulls. > I haven't figured out what the other entries are though. > I still see some entries (not as many as the null ones) with the date field > set to "[00/Jan/1900:00:00:00 +0000]", with all the other information valid > (URL, Referer, BackendName, etc..) except for the object size, which is set > to "(null)" and ther server response code (also "(null)"). > There are also entries that the only field set is the server response (200 > or 404 -- these have the date field set correctly) and some entries with the > host (or where the backend name should go) set to 127.0.0.1 and the other > fields set correctly. > > One thing though, in none of the entries (correct, broken or plain weird) > the object size is set, all I see are "-", except for one entry that had it > set (strangely enough, favicon.ico!) and even then, only once. > > If anybody else has seen this stuff or know anything that might help > explain, I'd greatly appreciate some help. > I can also send you any information you might deem necessary in order to > figure this stuff out. > > This is the varnish package for debian (2.0.3-2_amd64) all with HTTP/1.1 > requests. > > Thanks! > > Alecs > > > On Mon, Feb 9, 2009 at 5:29 PM, Alecs Henry wrote: >> >> Hi Nick, >> >> I forgot to say... It's varnish-trunk. >> But I saw it on both varnish 2.0.1 and 2.0.2. >> >> Alecs >> >> On Mon, Feb 9, 2009 at 6:20 PM, Nick Loman wrote: >>> >>> Alecs Henry wrote: >>> >>>> EDITED TO INCLUDE THE ORIGINAL MESSAGE... >>>> Hi guys, >>>> >>>> I was wondering if any of you has seen the following log entries when >>>> using varnishncsa: >>>> 127.0.0.1 - - [00/Jan/1900:00:00:00 +0000] "(null) (null) (null)" (null) >>>> - "-" "-" >>>> >>>> Those are coupled with: >>>> 127.0.0.1 - - [09/Feb/2009:19:39:46 +0000] "(null) (null) (null)" 200 >>>> 39678 "-" "-" >>>> I can see an object in the page that has that size (image) -- through >>>> firebug, but the object didn't load into the browser until I hit reload. >>>> >>>> And there are also many entries like the following: >>>> 127.0.0.1 - - [09/Feb/2009:19:39:52 +0000] "GET >>>> http://www.example.com/img/mod.gif HTTP/1.1" 200 951 >>>> "http://www.example.com/style.css" "Mozilla/5.0 (X11; U; Linux x86_64; en; >>>> rv:1.9.0.5) Gecko/2008121623 Ubuntu/8.10 (intrepid) Firefox/3.0.5" >>>> >>>> These entries are from communication with the backend server >>>> >>>> I expected it to show the backend name instead of 127.0.0.1. >>>> >>>> Is there an explanation? >>>> >>>> Thanks! >>>> >>>> Alecs >>>> >>> >>> I've seen log entries like this for HTTP 1.0 requests (using Varnish >>> 1.2), as that version of Varnish did not have support for parsing HTTP 1.0 >>> headers. >>> >>> Cheers, >>> >>> Nick. >>> >>> >>> >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > From kristian at redpill-linpro.com Tue Apr 7 11:46:42 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Tue, 7 Apr 2009 13:46:42 +0200 Subject: varnish 2.0.4 and new config changes In-Reply-To: References: <20090407101937.GC5267@kjeks.linpro.no> Message-ID: <20090407114641.GE5267@kjeks.linpro.no> On Tue, Apr 07, 2009 at 04:11:04AM -0700, Jauder Ho wrote: > On Tue, Apr 7, 2009 at 3:19 AM, Kristian Lyngstol > wrote: > > On Tue, Apr 07, 2009 at 02:55:13AM -0700, Jauder Ho wrote: > > > I just downloaded the new 2.0.4 release and noticed that a couple of > > > things had changed causing my existing config to stop working. > > > Therefore, I had a couple of questions that I'm hoping that you can > > > help answer. > > > > > > a) Is beresp.ttl basically replacing obj.ttl? If so, I should be able > > > to do the following right in vcl_fetch? > > > > > > if (beresp.ttl < 300s) { > > > set beresp.ttl = 300s; > > > } > > > > Yes, for all intents and purposes, beresp is obj. The name change is > > reflecting some underlying changes that doesn't really affect how you write > > VCL - yet. > > FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk > though. It would seem I was mistaken. It doesn't look like the obj to beresp commits made it into 2.0.4 after a quick check. So that's only relevant to trunk. > > > c) Lastly, I have a config of user > nginx1 > varnish > nginx2 > > > > php-fpm and noticed that the IP being logged on nginx2 is the > > > internal IP. nginx1 has X-Forwarded-For set. (...) > > If your first nginx sets X-Forwarded-For, then setting > > req.http.X-Forwarded-For = client.ip; will overwrite it with nginx1's IP. > > So don't set X-Forwarded-For in vcl_recv and it will pass straight through > > Varnish. 2.0.4 does not affect this compared to 2.0.3. > > Right now, only nginx1 sets X-Forwarded-For; varnish does not set > X-Forwarded-For; nginx2 is a fastcgi frontend and does not set > X-Forwarded-For. > > With this config (and using -trunk), nginx2 still logs the gateway IP so > something strange is going on here. I'm not familiar with how nginx logs, but I'd start by verifying: 1. That the X-Forwarded-For does indeed reach the relevant server. 2. That the entity that logs honors X-Forwarded-For. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From s.kain at eraffe-media.de Tue Apr 7 13:02:04 2009 From: s.kain at eraffe-media.de (Sascha Kain) Date: Tue, 07 Apr 2009 15:02:04 +0200 Subject: 503 Service Unavailable + Stop Responding Message-ID: <49DB4ECC.1050504@eraffe-media.de> Hi im getting folloing error when accessing a Picture (jpg), delivered by Varnish-Cache. ==================== Error 503 Service Unavailable Service Unavailable Guru Meditation: XID: 502211958 Varnish ================ This happens sporadically. When i access the picture from the Backend directly, its working. My varnishd is running on a Debian Server proxycache2:~# uname -a Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 x86_64 GNU/Linux proxycache2:~# varnishd -V varnishd (varnish-2.0.3) Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M It also happens, that the daemon just stop answering on Port 80, i have to manually restart it! proxycache2:~# varnishstat -1 uptime 691288 . Child uptime client_conn 13757134 19.90 Client connections accepted client_req 62501336 90.41 Client requests received cache_hit 59008654 85.36 Cache hits cache_hitpass 35 0.00 Cache hits for pass cache_miss 3454784 5.00 Cache misses backend_conn 3491596 5.05 Backend connections success backend_unhealthy 0 0.00 Backend connections not attempted backend_busy 0 0.00 Backend connections too many backend_fail 1094 0.00 Backend connections failures backend_reuse 3190621 4.62 Backend connections reuses backend_recycle 3300087 4.77 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 1082 . N struct srcaddr n_srcaddr_act 69 . N active struct srcaddr n_sess_mem 4061 . N struct sess_mem n_sess 325 . N struct sess n_object 727166 . N struct object n_objecthead 412871 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 25 . N struct vbe_conn n_bereq 173 . N struct bereq n_wrk 46 . N worker threads n_wrk_create 6590 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 40391 0.06 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 498 . N expired objects n_lru_nuked 2717782 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 50761824 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 43664646 63.16 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 13757122 19.90 Total Sessions s_req 62501369 90.41 Total Requests s_pipe 10 0.00 Total pipe s_pass 37897 0.05 Total pass s_fetch 3477236 5.03 Total fetch s_hdrbytes 19737092290 28551.19 Total header bytes s_bodybytes 681892000484 986407.98 Total body bytes sess_closed 544021 0.79 Session Closed sess_pipeline 164402 0.24 Session Pipeline sess_readahead 65588 0.09 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 61809876 89.41 Session herd shm_records 2591397419 3748.65 SHM records shm_writes 170178899 246.18 SHM writes shm_flushes 65 0.00 SHM flushes due to overflow shm_cont 5567 0.01 SHM MTX contention shm_cycles 931 0.00 SHM cycles through buffer sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq 9708485 14.04 SMA allocator requests sma_nobj 1442487 . SMA outstanding allocations sma_nbytes 16106012162 . SMA outstanding bytes sma_balloc 81856959440 . SMA bytes allocated sma_bfree 65750947278 . SMA bytes free sms_nreq 15433 0.02 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 18446744073709546966 . SMS outstanding bytes sms_balloc 7172160 . SMS bytes allocated sms_bfree 7176345 . SMS bytes freed backend_req 3491587 5.05 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) Sascha Kain IT / Administration eraffe media GmbH & Co. KG Marketing - Consulting - Software Sch?nfeldstr. 17 - 83022 Rosenheim Fon: + 49 (0)8031 - 941 41 -46 Fax: + 49 (0)8031 - 941 41 -59 E-Mail: s.kain at eraffe-media.de www.eraffe-media.de - www.eraffe.de eraffe media GmbH & Co. KG, Sitz: Rosenheim, Registergericht: AG Traunstein HR A Nr. 9104, St-Nr. 156/157/58806, FA Rosenheim, USt.-ID: DE250117972 Pers?nlich haftende Gesellschafterin: eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, Registergericht: AG Traunstein HR B 16956 St-Nr. 156/116/90247, FA Rosenheim Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser From kitai at ya.com Tue Apr 7 14:22:52 2009 From: kitai at ya.com (Kitai) Date: Tue, 7 Apr 2009 16:22:52 +0200 Subject: 503 Service Unavailable + Stop Responding In-Reply-To: <49DB4ECC.1050504@eraffe-media.de> References: <49DB4ECC.1050504@eraffe-media.de> Message-ID: <48d8853d0904070722y60d80fc5w670a5b2194c775b3@mail.gmail.com> Search in the "/var/log/messages" log if varnish it's restarting. After every crash, varnish starts answering with a 503 to every request. DAvid (Kitai) Cruz 2009/4/7 Sascha Kain : > Hi > im getting folloing error when accessing a Picture (jpg), delivered by > Varnish-Cache. > > ==================== > > > ?Error 503 Service Unavailable > > Service Unavailable > > > ? ? ?Guru Meditation: > > XID: 502211958 > > Varnish > > ================ > > > This happens sporadically. > When i access the picture from the Backend directly, its working. > > > My varnishd is running on a Debian Server > proxycache2:~# uname -a > Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 > x86_64 GNU/Linux > proxycache2:~# varnishd -V > varnishd (varnish-2.0.3) > Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS > > varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M > > > It also happens, that the daemon just stop answering on Port 80, i have > to manually restart it! > > > proxycache2:~# varnishstat -1 > uptime ? ? ? ? ? ? ? ? 691288 ? ? ? ? ?. ? Child uptime > client_conn ? ? ? ? ?13757134 ? ? ? ?19.90 Client connections accepted > client_req ? ? ? ? ? 62501336 ? ? ? ?90.41 Client requests received > cache_hit ? ? ? ? ? ?59008654 ? ? ? ?85.36 Cache hits > cache_hitpass ? ? ? ? ? ? ?35 ? ? ? ? 0.00 Cache hits for pass > cache_miss ? ? ? ? ? ?3454784 ? ? ? ? 5.00 Cache misses > backend_conn ? ? ? ? ?3491596 ? ? ? ? 5.05 Backend connections success > backend_unhealthy ? ? ? ? ? ?0 ? ? ? ? 0.00 Backend connections not attempted > backend_busy ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 Backend connections too many > backend_fail ? ? ? ? ? ? 1094 ? ? ? ? 0.00 Backend connections failures > backend_reuse ? ? ? ? 3190621 ? ? ? ? 4.62 Backend connections reuses > backend_recycle ? ? ? 3300087 ? ? ? ? 4.77 Backend connections recycles > backend_unused ? ? ? ? ? ? ?0 ? ? ? ? 0.00 Backend connections unused > n_srcaddr ? ? ? ? ? ? ? ?1082 ? ? ? ? ?. ? N struct srcaddr > n_srcaddr_act ? ? ? ? ? ? ?69 ? ? ? ? ?. ? N active struct srcaddr > n_sess_mem ? ? ? ? ? ? ? 4061 ? ? ? ? ?. ? N struct sess_mem > n_sess ? ? ? ? ? ? ? ? ? ?325 ? ? ? ? ?. ? N struct sess > n_object ? ? ? ? ? ? ? 727166 ? ? ? ? ?. ? N struct object > n_objecthead ? ? ? ? ? 412871 ? ? ? ? ?. ? N struct objecthead > n_smf ? ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? ?. ? N struct smf > n_smf_frag ? ? ? ? ? ? ? ? ?0 ? ? ? ? ?. ? N small free smf > n_smf_large ? ? ? ? ? ? ? ? 0 ? ? ? ? ?. ? N large free smf > n_vbe_conn ? ? ? ? ? ? ? ? 25 ? ? ? ? ?. ? N struct vbe_conn > n_bereq ? ? ? ? ? ? ? ? ? 173 ? ? ? ? ?. ? N struct bereq > n_wrk ? ? ? ? ? ? ? ? ? ? ?46 ? ? ? ? ?. ? N worker threads > n_wrk_create ? ? ? ? ? ? 6590 ? ? ? ? 0.01 N worker threads created > n_wrk_failed ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 N worker threads not created > n_wrk_max ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 N worker threads limited > n_wrk_queue ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 N queued work requests > n_wrk_overflow ? ? ? ? ?40391 ? ? ? ? 0.06 N overflowed work requests > n_wrk_drop ? ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 N dropped work requests > n_backend ? ? ? ? ? ? ? ? ? 1 ? ? ? ? ?. ? N backends > n_expired ? ? ? ? ? ? ? ? 498 ? ? ? ? ?. ? N expired objects > n_lru_nuked ? ? ? ? ? 2717782 ? ? ? ? ?. ? N LRU nuked objects > n_lru_saved ? ? ? ? ? ? ? ? 0 ? ? ? ? ?. ? N LRU saved objects > n_lru_moved ? ? ? ? ?50761824 ? ? ? ? ?. ? N LRU moved objects > n_deathrow ? ? ? ? ? ? ? ? ?0 ? ? ? ? ?. ? N objects on deathrow > losthdr ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 HTTP header overflows > n_objsendfile ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Objects sent with sendfile > n_objwrite ? ? ? ? ? 43664646 ? ? ? ?63.16 Objects sent with write > n_objoverflow ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Objects overflowing workspace > s_sess ? ? ? ? ? ? ? 13757122 ? ? ? ?19.90 Total Sessions > s_req ? ? ? ? ? ? ? ?62501369 ? ? ? ?90.41 Total Requests > s_pipe ? ? ? ? ? ? ? ? ? ? 10 ? ? ? ? 0.00 Total pipe > s_pass ? ? ? ? ? ? ? ? ?37897 ? ? ? ? 0.05 Total pass > s_fetch ? ? ? ? ? ? ? 3477236 ? ? ? ? 5.03 Total fetch > s_hdrbytes ? ? ? ?19737092290 ? ? 28551.19 Total header bytes > s_bodybytes ? ? ?681892000484 ? ?986407.98 Total body bytes > sess_closed ? ? ? ? ? ?544021 ? ? ? ? 0.79 Session Closed > sess_pipeline ? ? ? ? ?164402 ? ? ? ? 0.24 Session Pipeline > sess_readahead ? ? ? ? ?65588 ? ? ? ? 0.09 Session Read Ahead > sess_linger ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Session Linger > sess_herd ? ? ? ? ? ?61809876 ? ? ? ?89.41 Session herd > shm_records ? ? ? ?2591397419 ? ? ?3748.65 SHM records > shm_writes ? ? ? ? ?170178899 ? ? ? 246.18 SHM writes > shm_flushes ? ? ? ? ? ? ? ?65 ? ? ? ? 0.00 SHM flushes due to overflow > shm_cont ? ? ? ? ? ? ? ? 5567 ? ? ? ? 0.01 SHM MTX contention > shm_cycles ? ? ? ? ? ? ? ?931 ? ? ? ? 0.00 SHM cycles through buffer > sm_nreq ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 allocator requests > sm_nobj ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? ?. ? outstanding allocations > sm_balloc ? ? ? ? ? ? ? ? ? 0 ? ? ? ? ?. ? bytes allocated > sm_bfree ? ? ? ? ? ? ? ? ? ?0 ? ? ? ? ?. ? bytes free > sma_nreq ? ? ? ? ? ? ?9708485 ? ? ? ?14.04 SMA allocator requests > sma_nobj ? ? ? ? ? ? ?1442487 ? ? ? ? ?. ? SMA outstanding allocations > sma_nbytes ? ? ? ?16106012162 ? ? ? ? ?. ? SMA outstanding bytes > sma_balloc ? ? ? ?81856959440 ? ? ? ? ?. ? SMA bytes allocated > sma_bfree ? ? ? ? 65750947278 ? ? ? ? ?. ? SMA bytes free > sms_nreq ? ? ? ? ? ? ? ?15433 ? ? ? ? 0.02 SMS allocator requests > sms_nobj ? ? ? ? ? ? ? ? ? ?0 ? ? ? ? ?. ? SMS outstanding allocations > sms_nbytes ? ? ? 18446744073709546966 ? ? ? ? ?. ? SMS outstanding bytes > sms_balloc ? ? ? ? ? ?7172160 ? ? ? ? ?. ? SMS bytes allocated > sms_bfree ? ? ? ? ? ? 7176345 ? ? ? ? ?. ? SMS bytes freed > backend_req ? ? ? ? ? 3491587 ? ? ? ? 5.05 Backend requests made > n_vcl ? ? ? ? ? ? ? ? ? ? ? 1 ? ? ? ? 0.00 N vcl total > n_vcl_avail ? ? ? ? ? ? ? ? 1 ? ? ? ? 0.00 N vcl available > n_vcl_discard ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 N vcl discarded > n_purge ? ? ? ? ? ? ? ? ? ? 1 ? ? ? ? ?. ? N total active purges > n_purge_add ? ? ? ? ? ? ? ? 1 ? ? ? ? 0.00 N new purges added > n_purge_retire ? ? ? ? ? ? ?0 ? ? ? ? 0.00 N old purges deleted > n_purge_obj_test ? ? ? ? ? ?0 ? ? ? ? 0.00 N objects tested > n_purge_re_test ? ? ? ? ? ? 0 ? ? ? ? 0.00 N regexps tested against > n_purge_dups ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 N duplicate purges removed > hcb_nolock ? ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 HCB Lookups without lock > hcb_lock ? ? ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 HCB Lookups with lock > hcb_insert ? ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 HCB Inserts > esi_parse ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Objects ESI parsed (unlock) > esi_errors ? ? ? ? ? ? ? ? ?0 ? ? ? ? 0.00 ESI parse errors (unlock) > > > > Sascha Kain > IT / Administration > eraffe media GmbH & Co. KG Marketing - Consulting - Software > Sch?nfeldstr. 17 - 83022 Rosenheim > > Fon: + 49 (0)8031 - 941 41 -46 > Fax: + 49 (0)8031 - 941 41 -59 > E-Mail: s.kain at eraffe-media.de > www.eraffe-media.de - www.eraffe.de > > eraffe media GmbH & Co. KG, Sitz: Rosenheim, > Registergericht: AG Traunstein HR A Nr. 9104, > St-Nr. 156/157/58806, FA Rosenheim, > USt.-ID: DE250117972 > > Pers?nlich haftende Gesellschafterin: > eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, > Registergericht: AG Traunstein HR B 16956 > St-Nr. 156/116/90247, FA Rosenheim > > Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From s.kain at eraffe-media.de Tue Apr 7 15:23:34 2009 From: s.kain at eraffe-media.de (Sascha Kain) Date: Tue, 07 Apr 2009 17:23:34 +0200 Subject: 503 Service Unavailable + Stop Responding In-Reply-To: <48d8853d0904070722y60d80fc5w670a5b2194c775b3@mail.gmail.com> References: <49DB4ECC.1050504@eraffe-media.de> <48d8853d0904070722y60d80fc5w670a5b2194c775b3@mail.gmail.com> Message-ID: <49DB6FF6.5080908@eraffe-media.de> Hi, its running correct. no such messages in logfiles. Im still having the second problem, that the varnishd just stops responding on port 80 and all requests die. maybe ill switch to 2.0.4, and check it out on high load. Kitai wrote: > Search in the "/var/log/messages" log if varnish it's restarting. > > After every crash, varnish starts answering with a 503 to every request. > > > DAvid (Kitai) Cruz > > > 2009/4/7 Sascha Kain : > >> Hi >> im getting folloing error when accessing a Picture (jpg), delivered by >> Varnish-Cache. >> >> ==================== >> >> >> Error 503 Service Unavailable >> >> Service Unavailable >> >> >> Guru Meditation: >> >> XID: 502211958 >> >> Varnish >> >> ================ >> >> >> This happens sporadically. >> When i access the picture from the Backend directly, its working. >> >> >> My varnishd is running on a Debian Server >> proxycache2:~# uname -a >> Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 >> x86_64 GNU/Linux >> proxycache2:~# varnishd -V >> varnishd (varnish-2.0.3) >> Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS >> >> varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M >> >> >> It also happens, that the daemon just stop answering on Port 80, i have >> to manually restart it! >> >> >> proxycache2:~# varnishstat -1 >> uptime 691288 . Child uptime >> client_conn 13757134 19.90 Client connections accepted >> client_req 62501336 90.41 Client requests received >> cache_hit 59008654 85.36 Cache hits >> cache_hitpass 35 0.00 Cache hits for pass >> cache_miss 3454784 5.00 Cache misses >> backend_conn 3491596 5.05 Backend connections success >> backend_unhealthy 0 0.00 Backend connections not attempted >> backend_busy 0 0.00 Backend connections too many >> backend_fail 1094 0.00 Backend connections failures >> backend_reuse 3190621 4.62 Backend connections reuses >> backend_recycle 3300087 4.77 Backend connections recycles >> backend_unused 0 0.00 Backend connections unused >> n_srcaddr 1082 . N struct srcaddr >> n_srcaddr_act 69 . N active struct srcaddr >> n_sess_mem 4061 . N struct sess_mem >> n_sess 325 . N struct sess >> n_object 727166 . N struct object >> n_objecthead 412871 . N struct objecthead >> n_smf 0 . N struct smf >> n_smf_frag 0 . N small free smf >> n_smf_large 0 . N large free smf >> n_vbe_conn 25 . N struct vbe_conn >> n_bereq 173 . N struct bereq >> n_wrk 46 . N worker threads >> n_wrk_create 6590 0.01 N worker threads created >> n_wrk_failed 0 0.00 N worker threads not created >> n_wrk_max 0 0.00 N worker threads limited >> n_wrk_queue 0 0.00 N queued work requests >> n_wrk_overflow 40391 0.06 N overflowed work requests >> n_wrk_drop 0 0.00 N dropped work requests >> n_backend 1 . N backends >> n_expired 498 . N expired objects >> n_lru_nuked 2717782 . N LRU nuked objects >> n_lru_saved 0 . N LRU saved objects >> n_lru_moved 50761824 . N LRU moved objects >> n_deathrow 0 . N objects on deathrow >> losthdr 0 0.00 HTTP header overflows >> n_objsendfile 0 0.00 Objects sent with sendfile >> n_objwrite 43664646 63.16 Objects sent with write >> n_objoverflow 0 0.00 Objects overflowing workspace >> s_sess 13757122 19.90 Total Sessions >> s_req 62501369 90.41 Total Requests >> s_pipe 10 0.00 Total pipe >> s_pass 37897 0.05 Total pass >> s_fetch 3477236 5.03 Total fetch >> s_hdrbytes 19737092290 28551.19 Total header bytes >> s_bodybytes 681892000484 986407.98 Total body bytes >> sess_closed 544021 0.79 Session Closed >> sess_pipeline 164402 0.24 Session Pipeline >> sess_readahead 65588 0.09 Session Read Ahead >> sess_linger 0 0.00 Session Linger >> sess_herd 61809876 89.41 Session herd >> shm_records 2591397419 3748.65 SHM records >> shm_writes 170178899 246.18 SHM writes >> shm_flushes 65 0.00 SHM flushes due to overflow >> shm_cont 5567 0.01 SHM MTX contention >> shm_cycles 931 0.00 SHM cycles through buffer >> sm_nreq 0 0.00 allocator requests >> sm_nobj 0 . outstanding allocations >> sm_balloc 0 . bytes allocated >> sm_bfree 0 . bytes free >> sma_nreq 9708485 14.04 SMA allocator requests >> sma_nobj 1442487 . SMA outstanding allocations >> sma_nbytes 16106012162 . SMA outstanding bytes >> sma_balloc 81856959440 . SMA bytes allocated >> sma_bfree 65750947278 . SMA bytes free >> sms_nreq 15433 0.02 SMS allocator requests >> sms_nobj 0 . SMS outstanding allocations >> sms_nbytes 18446744073709546966 . SMS outstanding bytes >> sms_balloc 7172160 . SMS bytes allocated >> sms_bfree 7176345 . SMS bytes freed >> backend_req 3491587 5.05 Backend requests made >> n_vcl 1 0.00 N vcl total >> n_vcl_avail 1 0.00 N vcl available >> n_vcl_discard 0 0.00 N vcl discarded >> n_purge 1 . N total active purges >> n_purge_add 1 0.00 N new purges added >> n_purge_retire 0 0.00 N old purges deleted >> n_purge_obj_test 0 0.00 N objects tested >> n_purge_re_test 0 0.00 N regexps tested against >> n_purge_dups 0 0.00 N duplicate purges removed >> hcb_nolock 0 0.00 HCB Lookups without lock >> hcb_lock 0 0.00 HCB Lookups with lock >> hcb_insert 0 0.00 HCB Inserts >> esi_parse 0 0.00 Objects ESI parsed (unlock) >> esi_errors 0 0.00 ESI parse errors (unlock) >> >> >> >> Sascha Kain >> IT / Administration >> eraffe media GmbH & Co. KG Marketing - Consulting - Software >> Sch?nfeldstr. 17 - 83022 Rosenheim >> >> Fon: + 49 (0)8031 - 941 41 -46 >> Fax: + 49 (0)8031 - 941 41 -59 >> E-Mail: s.kain at eraffe-media.de >> www.eraffe-media.de - www.eraffe.de >> >> eraffe media GmbH & Co. KG, Sitz: Rosenheim, >> Registergericht: AG Traunstein HR A Nr. 9104, >> St-Nr. 156/157/58806, FA Rosenheim, >> USt.-ID: DE250117972 >> >> Pers?nlich haftende Gesellschafterin: >> eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, >> Registergericht: AG Traunstein HR B 16956 >> St-Nr. 156/116/90247, FA Rosenheim >> >> Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> -- Sascha Kain IT / Administration eraffe media GmbH & Co. KG Marketing - Consulting - Software Sch?nfeldstr. 17 - 83022 Rosenheim Fon: + 49 (0)8031 - 941 41 -46 Fax: + 49 (0)8031 - 941 41 -59 E-Mail: s.kain at eraffe-media.de www.eraffe-media.de - www.eraffe.de eraffe media GmbH & Co. KG, Sitz: Rosenheim, Registergericht: AG Traunstein HR A Nr. 9104, St-Nr. 156/157/58806, FA Rosenheim, USt.-ID: DE250117972 Pers?nlich haftende Gesellschafterin: eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, Registergericht: AG Traunstein HR B 16956 St-Nr. 156/116/90247, FA Rosenheim Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser From tnguyen at bleacherreport.com Tue Apr 7 20:01:51 2009 From: tnguyen at bleacherreport.com (Tung Nguyen) Date: Tue, 7 Apr 2009 13:01:51 -0700 Subject: varnish killing off the child process after a few minutes Message-ID: Hi guys, We're on varnish 2.0.3 It looks like varnish restarts the child process for us every so often, causing 503s :(. Was wondering if this is a known issue. ###################################################################### # ps auxww | grep varnishd nobody 3385 0.0 0.3 8098024 7560 ? Sl 19:49 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 10 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 1s -p thread_pool_min 100 -f /etc/varnish/default.vcl root 3938 0.0 0.0 111584 904 ? Ss Apr03 0:01 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 10 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 1s -p thread_pool_min 100 -f /etc/varnish/default.vcl root 3961 0.0 0.0 3884 672 pts/2 S<+ 19:55 0:00 grep --colour=auto varnishd # ###################################################################### # varnishstat -1 uptime 472 . Child uptime client_conn 40 0.08 Client connections accepted client_req 100 0.21 Client requests received cache_hit 1 0.00 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 23 0.05 Cache misses backend_conn 99 0.21 Backend connections success backend_unhealthy 0 0.00 Backend connections not attempted backend_busy 0 0.00 Backend connections too many backend_fail 0 0.00 Backend connections failures backend_reuse 82 0.17 Backend connections reuses backend_recycle 99 0.21 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 1 . N struct srcaddr n_srcaddr_act 0 . N active struct srcaddr n_sess_mem 13 . N struct sess_mem n_sess 1 . N struct sess n_object 23 . N struct object n_objecthead 29 . N struct objecthead n_smf 47 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 1 . N large free smf n_vbe_conn 6 . N struct vbe_conn n_bereq 6 . N struct bereq n_wrk 200 . N worker threads n_wrk_create 200 0.42 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 13326 28.23 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 0 0.00 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 2 . N backends n_expired 0 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 1 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 65 0.14 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 40 0.08 Total Sessions s_req 100 0.21 Total Requests s_pipe 0 0.00 Total pipe s_pass 76 0.16 Total pass s_fetch 99 0.21 Total fetch s_hdrbytes 35654 75.54 Total header bytes s_bodybytes 206284 437.04 Total body bytes sess_closed 0 0.00 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 100 0.21 Session herd shm_records 9594 20.33 SHM records shm_writes 1884 3.99 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 10 0.02 SHM MTX contention shm_cycles 0 0.00 SHM cycles through buffer sm_nreq 186 0.39 allocator requests sm_nobj 46 . outstanding allocations sm_balloc 425984 . bytes allocated sm_bfree 6431973376 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 0 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 0 . SMS bytes allocated sms_bfree 0 . SMS bytes freed backend_req 99 0.21 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) ###################################################################### # I can see that the threads are starting back up in varnishlog ###################################################################### 8 RxHeader c Accept-Language: en-us,en;q=0.5 8 RxHeader c Accept-Encoding: gzip,deflate 8 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 8 RxHeader c Keep-Alive: 300 8 RxHeader c Connection: keep-alive 8 RxHeader c Referer: http://stagingbleacherreport.com/nascar 8 RxHeader c Cookie: __utma=58205160.3272659617253579300.1238530464.1239125976.1239133102.21; __utmz=58205160.1238530464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __qca=1238530467-23581915-71188578; auth_token=4c078f7b1d3097b170260c6978c9862c; pod_151314=Mon+ 8 RxHeader c If-None-Match: "004e69938cfe6ed16863cc156a0e9b34" 8 VCL_call c recv 8 VCL_return c pass 8 VCL_call c pass 8 VCL_return c pass 8 VCL_call c error 8 VCL_return c deliver 8 Length c 465 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 503 8 TxResponse c Service Unavailable 8 TxHeader c Server: Varnish 8 TxHeader c Retry-After: 0 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c Content-Length: 465 8 TxHeader c Date: Tue, 07 Apr 2009 19:49:47 GMT 8 TxHeader c X-Varnish: 171071333 8 TxHeader c Age: 0 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Connection: close 8 TxHeader c X-Cache: MISS 8 ReqEnd c 171071333 1239133787.063603163 1239133787.063603163 0.000000000 0.000000000 0.000000000 8 SessionClose c error 8 StatSess c 67.164.99.232 59479 0 1 1 0 1 0 249 465 0 StatAddr - 67.164.99.232 0 0 1 1 0 1 0 249 465 0 WorkThread - 0x55028bd0 start 0 WorkThread - 0x55829bd0 start 0 WorkThread - 0x5602abd0 start 0 WorkThread - 0x5682bbd0 start 0 WorkThread - 0x5702cbd0 start 0 WorkThread - 0x5782dbd0 start 0 WorkThread - 0x5802ebd0 start 0 WorkThread - 0x5882fbd0 start 0 WorkThread - 0x59030bd0 start 0 WorkThread - 0x59831bd0 start 0 WorkThread - 0x5a032bd0 start 0 WorkThread - 0x5a833bd0 start 0 WorkThread - 0x5b034bd0 start 0 WorkThread - 0x5b835bd0 start 0 WorkThread - 0x5c036bd0 start 0 WorkThread - 0x5c837bd0 start 0 WorkThread - 0x5d038bd0 start 0 Backend_health - A1 Back healthy 4--X-S-RH 2 2 5 0.000000 0.000000 HTTP/1.1 200 OK 0 Backend_health - A2 Back healthy 4--X-S-RH 2 2 5 0.000000 0.000000 HTTP/1.1 200 OK 0 WorkThread - 0x5d839bd0 start 0 WorkThread - 0x5e03abd0 start 0 WorkThread - 0x5e83bbd0 start 0 WorkThread - 0x5f03cbd0 start 0 WorkThread - 0x5f83dbd0 start 0 WorkThread - 0x6003ebd0 start 0 WorkThread - 0x6083fbd0 start 0 WorkThread - 0x61040bd0 start # ###################################################################### # heres are default.vcl cat /etc/varnish/default.vcl backend A1 { .host = "ey03-s00344"; .port = "80"; .probe = { .interval = 1s; .timeout = 3 s; .window = 5; .threshold = 2; .url = "http://stagingbleacherreport.com/images/blank.gif"; } } backend A2 { .host = "ey03-s00344"; .port = "80"; .probe = { .interval = 1s; .timeout = 3 s; .window = 5; .threshold = 2; .url = "http://stagingbleacherreport.com/images/blank.gif"; } } director www_director round-robin { { .backend = A1; } { .backend = A2; } } acl local { "localhost"; } C{ #include }C sub vcl_recv { C{ syslog(LOG_INFO, "@@@@ vcl_recv."); }C C{ syslog(LOG_INFO, VRT_r_req_url(sp)); }C set req.backend = www_director ; # Add a unique header containing the client address remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; /* clear cookies on feeds */ if (req.url ~ "^/articles[;/]feed") { C{ syslog(LOG_INFO, "@@@@ feed."); }C unset req.http.Cookie; lookup; } # cache assets if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets") { C{ syslog(LOG_INFO, "@@@@ assets."); }C unset req.http.Cookie; lookup; } # do not cache images_root if (req.url ~ "^/images_root") { C{ syslog(LOG_INFO, "@@@@ images_root."); }C pass; } if (req.request == "PURGE") { if (!client.ip ~ local) { error 405 "Not allowed."; } lookup; } C{ syslog(LOG_INFO, "@@@@ fallthrough."); }C } #sub vcl_pipe { # # If we don't set the Connection: close header, any following # # requests from the client will also be piped through and # # left untouched by varnish. We don't want that. # set bereq.http.connection = "close"; #} sub vcl_fetch { if (req.url ~ "^/articles[;/]feed") { unset obj.http.Set-Cookie; unset obj.http.expires; set obj.ttl = 60m; deliver; } if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets") { unset obj.http.Set-Cookie; set obj.ttl = 30m; } # similar to what twitter uses if (!obj.cacheable) { set obj.http.X-Cacheable = "NO:Not-Cacheable"; pass; } if (obj.http.Cache-Control ~ "private") { if(req.http.Cookie ~"(UserID|_session)") { set obj.http.X-Cacheable = "NO:Got Session"; } else { set obj.http.X-Cacheable = "NO:Cache-Control=private"; } pass; } if (obj.http.Set-Cookie ~ "(UserID|_session)") { set obj.http.X-Cacheable = "NO:Set-Cookie"; pass; } set obj.grace = 30s; # only one slow requests during cache misses } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } Thanks, Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From ross at trademe.co.nz Tue Apr 7 23:41:24 2009 From: ross at trademe.co.nz (Ross Brown) Date: Wed, 8 Apr 2009 11:41:24 +1200 Subject: 503 Service Unavailable + Stop Responding In-Reply-To: <49DB6FF6.5080908@eraffe-media.de> References: <49DB4ECC.1050504@eraffe-media.de> <48d8853d0904070722y60d80fc5w670a5b2194c775b3@mail.gmail.com> <49DB6FF6.5080908@eraffe-media.de> Message-ID: <1FF67D7369ED1A45832180C7C1109BCA0D35FE462C@tmmail0.trademe.local> I'm seeing similar issues (running Trunk). How much load is your Varnish server under when it stops responding? -----Original Message----- From: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] On Behalf Of Sascha Kain Sent: Wednesday, 8 April 2009 3:24 a.m. To: kitai at ya.com Cc: varnish-misc at projects.linpro.no Subject: Re: 503 Service Unavailable + Stop Responding Hi, its running correct. no such messages in logfiles. Im still having the second problem, that the varnishd just stops responding on port 80 and all requests die. maybe ill switch to 2.0.4, and check it out on high load. Kitai wrote: > Search in the "/var/log/messages" log if varnish it's restarting. > > After every crash, varnish starts answering with a 503 to every request. > > > DAvid (Kitai) Cruz > > > 2009/4/7 Sascha Kain : > >> Hi >> im getting folloing error when accessing a Picture (jpg), delivered by >> Varnish-Cache. >> >> ==================== >> >> >> Error 503 Service Unavailable >> >> Service Unavailable >> >> >> Guru Meditation: >> >> XID: 502211958 >> >> Varnish >> >> ================ >> >> >> This happens sporadically. >> When i access the picture from the Backend directly, its working. >> >> >> My varnishd is running on a Debian Server >> proxycache2:~# uname -a >> Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 >> x86_64 GNU/Linux >> proxycache2:~# varnishd -V >> varnishd (varnish-2.0.3) >> Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS >> >> varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M >> >> >> It also happens, that the daemon just stop answering on Port 80, i have >> to manually restart it! >> >> >> proxycache2:~# varnishstat -1 >> uptime 691288 . Child uptime >> client_conn 13757134 19.90 Client connections accepted >> client_req 62501336 90.41 Client requests received >> cache_hit 59008654 85.36 Cache hits >> cache_hitpass 35 0.00 Cache hits for pass >> cache_miss 3454784 5.00 Cache misses >> backend_conn 3491596 5.05 Backend connections success >> backend_unhealthy 0 0.00 Backend connections not attempted >> backend_busy 0 0.00 Backend connections too many >> backend_fail 1094 0.00 Backend connections failures >> backend_reuse 3190621 4.62 Backend connections reuses >> backend_recycle 3300087 4.77 Backend connections recycles >> backend_unused 0 0.00 Backend connections unused >> n_srcaddr 1082 . N struct srcaddr >> n_srcaddr_act 69 . N active struct srcaddr >> n_sess_mem 4061 . N struct sess_mem >> n_sess 325 . N struct sess >> n_object 727166 . N struct object >> n_objecthead 412871 . N struct objecthead >> n_smf 0 . N struct smf >> n_smf_frag 0 . N small free smf >> n_smf_large 0 . N large free smf >> n_vbe_conn 25 . N struct vbe_conn >> n_bereq 173 . N struct bereq >> n_wrk 46 . N worker threads >> n_wrk_create 6590 0.01 N worker threads created >> n_wrk_failed 0 0.00 N worker threads not created >> n_wrk_max 0 0.00 N worker threads limited >> n_wrk_queue 0 0.00 N queued work requests >> n_wrk_overflow 40391 0.06 N overflowed work requests >> n_wrk_drop 0 0.00 N dropped work requests >> n_backend 1 . N backends >> n_expired 498 . N expired objects >> n_lru_nuked 2717782 . N LRU nuked objects >> n_lru_saved 0 . N LRU saved objects >> n_lru_moved 50761824 . N LRU moved objects >> n_deathrow 0 . N objects on deathrow >> losthdr 0 0.00 HTTP header overflows >> n_objsendfile 0 0.00 Objects sent with sendfile >> n_objwrite 43664646 63.16 Objects sent with write >> n_objoverflow 0 0.00 Objects overflowing workspace >> s_sess 13757122 19.90 Total Sessions >> s_req 62501369 90.41 Total Requests >> s_pipe 10 0.00 Total pipe >> s_pass 37897 0.05 Total pass >> s_fetch 3477236 5.03 Total fetch >> s_hdrbytes 19737092290 28551.19 Total header bytes >> s_bodybytes 681892000484 986407.98 Total body bytes >> sess_closed 544021 0.79 Session Closed >> sess_pipeline 164402 0.24 Session Pipeline >> sess_readahead 65588 0.09 Session Read Ahead >> sess_linger 0 0.00 Session Linger >> sess_herd 61809876 89.41 Session herd >> shm_records 2591397419 3748.65 SHM records >> shm_writes 170178899 246.18 SHM writes >> shm_flushes 65 0.00 SHM flushes due to overflow >> shm_cont 5567 0.01 SHM MTX contention >> shm_cycles 931 0.00 SHM cycles through buffer >> sm_nreq 0 0.00 allocator requests >> sm_nobj 0 . outstanding allocations >> sm_balloc 0 . bytes allocated >> sm_bfree 0 . bytes free >> sma_nreq 9708485 14.04 SMA allocator requests >> sma_nobj 1442487 . SMA outstanding allocations >> sma_nbytes 16106012162 . SMA outstanding bytes >> sma_balloc 81856959440 . SMA bytes allocated >> sma_bfree 65750947278 . SMA bytes free >> sms_nreq 15433 0.02 SMS allocator requests >> sms_nobj 0 . SMS outstanding allocations >> sms_nbytes 18446744073709546966 . SMS outstanding bytes >> sms_balloc 7172160 . SMS bytes allocated >> sms_bfree 7176345 . SMS bytes freed >> backend_req 3491587 5.05 Backend requests made >> n_vcl 1 0.00 N vcl total >> n_vcl_avail 1 0.00 N vcl available >> n_vcl_discard 0 0.00 N vcl discarded >> n_purge 1 . N total active purges >> n_purge_add 1 0.00 N new purges added >> n_purge_retire 0 0.00 N old purges deleted >> n_purge_obj_test 0 0.00 N objects tested >> n_purge_re_test 0 0.00 N regexps tested against >> n_purge_dups 0 0.00 N duplicate purges removed >> hcb_nolock 0 0.00 HCB Lookups without lock >> hcb_lock 0 0.00 HCB Lookups with lock >> hcb_insert 0 0.00 HCB Inserts >> esi_parse 0 0.00 Objects ESI parsed (unlock) >> esi_errors 0 0.00 ESI parse errors (unlock) >> >> >> >> Sascha Kain >> IT / Administration >> eraffe media GmbH & Co. KG Marketing - Consulting - Software >> Sch?nfeldstr. 17 - 83022 Rosenheim >> >> Fon: + 49 (0)8031 - 941 41 -46 >> Fax: + 49 (0)8031 - 941 41 -59 >> E-Mail: s.kain at eraffe-media.de >> www.eraffe-media.de - www.eraffe.de >> >> eraffe media GmbH & Co. KG, Sitz: Rosenheim, >> Registergericht: AG Traunstein HR A Nr. 9104, >> St-Nr. 156/157/58806, FA Rosenheim, >> USt.-ID: DE250117972 >> >> Pers?nlich haftende Gesellschafterin: >> eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, >> Registergericht: AG Traunstein HR B 16956 >> St-Nr. 156/116/90247, FA Rosenheim >> >> Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> -- Sascha Kain IT / Administration eraffe media GmbH & Co. KG Marketing - Consulting - Software Sch?nfeldstr. 17 - 83022 Rosenheim Fon: + 49 (0)8031 - 941 41 -46 Fax: + 49 (0)8031 - 941 41 -59 E-Mail: s.kain at eraffe-media.de www.eraffe-media.de - www.eraffe.de eraffe media GmbH & Co. KG, Sitz: Rosenheim, Registergericht: AG Traunstein HR A Nr. 9104, St-Nr. 156/157/58806, FA Rosenheim, USt.-ID: DE250117972 Pers?nlich haftende Gesellschafterin: eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, Registergericht: AG Traunstein HR B 16956 St-Nr. 156/116/90247, FA Rosenheim Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From jauderho at gmail.com Wed Apr 8 00:03:58 2009 From: jauderho at gmail.com (Jauder Ho) Date: Tue, 7 Apr 2009 17:03:58 -0700 Subject: 503 Service Unavailable + Stop Responding In-Reply-To: <1FF67D7369ED1A45832180C7C1109BCA0D35FE462C@tmmail0.trademe.local> References: <49DB4ECC.1050504@eraffe-media.de> <48d8853d0904070722y60d80fc5w670a5b2194c775b3@mail.gmail.com> <49DB6FF6.5080908@eraffe-media.de> <1FF67D7369ED1A45832180C7C1109BCA0D35FE462C@tmmail0.trademe.local> Message-ID: I saw this while testing using ab on 2.0.3 a number of times. I was just doing a ab -n 1000 -c 50 (x3) when this happened (repeatably). Retesting on -trunk seems to be okay. --Jauder On Tue, Apr 7, 2009 at 4:41 PM, Ross Brown wrote: > I'm seeing similar issues (running Trunk). How much load is your Varnish > server under when it stops responding? > > > -----Original Message----- > From: varnish-misc-bounces at projects.linpro.no [mailto: > varnish-misc-bounces at projects.linpro.no] On Behalf Of Sascha Kain > Sent: Wednesday, 8 April 2009 3:24 a.m. > To: kitai at ya.com > Cc: varnish-misc at projects.linpro.no > Subject: Re: 503 Service Unavailable + Stop Responding > > Hi, its running correct. no such messages in logfiles. > > Im still having the second problem, that the varnishd just stops > responding on port 80 and all requests die. > > maybe ill switch to 2.0.4, and check it out on high load. > > > Kitai wrote: > > Search in the "/var/log/messages" log if varnish it's restarting. > > > > After every crash, varnish starts answering with a 503 to every request. > > > > > > DAvid (Kitai) Cruz > > > > > > 2009/4/7 Sascha Kain : > > > >> Hi > >> im getting folloing error when accessing a Picture (jpg), delivered by > >> Varnish-Cache. > >> > >> ==================== > >> > >> > >> Error 503 Service Unavailable > >> > >> Service Unavailable > >> > >> > >> Guru Meditation: > >> > >> XID: 502211958 > >> > >> Varnish > >> > >> ================ > >> > >> > >> This happens sporadically. > >> When i access the picture from the Backend directly, its working. > >> > >> > >> My varnishd is running on a Debian Server > >> proxycache2:~# uname -a > >> Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 > >> x86_64 GNU/Linux > >> proxycache2:~# varnishd -V > >> varnishd (varnish-2.0.3) > >> Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS > >> > >> varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M > >> > >> > >> It also happens, that the daemon just stop answering on Port 80, i have > >> to manually restart it! > >> > >> > >> proxycache2:~# varnishstat -1 > >> uptime 691288 . Child uptime > >> client_conn 13757134 19.90 Client connections accepted > >> client_req 62501336 90.41 Client requests received > >> cache_hit 59008654 85.36 Cache hits > >> cache_hitpass 35 0.00 Cache hits for pass > >> cache_miss 3454784 5.00 Cache misses > >> backend_conn 3491596 5.05 Backend connections success > >> backend_unhealthy 0 0.00 Backend connections not > attempted > >> backend_busy 0 0.00 Backend connections too many > >> backend_fail 1094 0.00 Backend connections failures > >> backend_reuse 3190621 4.62 Backend connections reuses > >> backend_recycle 3300087 4.77 Backend connections recycles > >> backend_unused 0 0.00 Backend connections unused > >> n_srcaddr 1082 . N struct srcaddr > >> n_srcaddr_act 69 . N active struct srcaddr > >> n_sess_mem 4061 . N struct sess_mem > >> n_sess 325 . N struct sess > >> n_object 727166 . N struct object > >> n_objecthead 412871 . N struct objecthead > >> n_smf 0 . N struct smf > >> n_smf_frag 0 . N small free smf > >> n_smf_large 0 . N large free smf > >> n_vbe_conn 25 . N struct vbe_conn > >> n_bereq 173 . N struct bereq > >> n_wrk 46 . N worker threads > >> n_wrk_create 6590 0.01 N worker threads created > >> n_wrk_failed 0 0.00 N worker threads not created > >> n_wrk_max 0 0.00 N worker threads limited > >> n_wrk_queue 0 0.00 N queued work requests > >> n_wrk_overflow 40391 0.06 N overflowed work requests > >> n_wrk_drop 0 0.00 N dropped work requests > >> n_backend 1 . N backends > >> n_expired 498 . N expired objects > >> n_lru_nuked 2717782 . N LRU nuked objects > >> n_lru_saved 0 . N LRU saved objects > >> n_lru_moved 50761824 . N LRU moved objects > >> n_deathrow 0 . N objects on deathrow > >> losthdr 0 0.00 HTTP header overflows > >> n_objsendfile 0 0.00 Objects sent with sendfile > >> n_objwrite 43664646 63.16 Objects sent with write > >> n_objoverflow 0 0.00 Objects overflowing workspace > >> s_sess 13757122 19.90 Total Sessions > >> s_req 62501369 90.41 Total Requests > >> s_pipe 10 0.00 Total pipe > >> s_pass 37897 0.05 Total pass > >> s_fetch 3477236 5.03 Total fetch > >> s_hdrbytes 19737092290 28551.19 Total header bytes > >> s_bodybytes 681892000484 986407.98 Total body bytes > >> sess_closed 544021 0.79 Session Closed > >> sess_pipeline 164402 0.24 Session Pipeline > >> sess_readahead 65588 0.09 Session Read Ahead > >> sess_linger 0 0.00 Session Linger > >> sess_herd 61809876 89.41 Session herd > >> shm_records 2591397419 3748.65 SHM records > >> shm_writes 170178899 246.18 SHM writes > >> shm_flushes 65 0.00 SHM flushes due to overflow > >> shm_cont 5567 0.01 SHM MTX contention > >> shm_cycles 931 0.00 SHM cycles through buffer > >> sm_nreq 0 0.00 allocator requests > >> sm_nobj 0 . outstanding allocations > >> sm_balloc 0 . bytes allocated > >> sm_bfree 0 . bytes free > >> sma_nreq 9708485 14.04 SMA allocator requests > >> sma_nobj 1442487 . SMA outstanding allocations > >> sma_nbytes 16106012162 . SMA outstanding bytes > >> sma_balloc 81856959440 . SMA bytes allocated > >> sma_bfree 65750947278 . SMA bytes free > >> sms_nreq 15433 0.02 SMS allocator requests > >> sms_nobj 0 . SMS outstanding allocations > >> sms_nbytes 18446744073709546966 . SMS outstanding bytes > >> sms_balloc 7172160 . SMS bytes allocated > >> sms_bfree 7176345 . SMS bytes freed > >> backend_req 3491587 5.05 Backend requests made > >> n_vcl 1 0.00 N vcl total > >> n_vcl_avail 1 0.00 N vcl available > >> n_vcl_discard 0 0.00 N vcl discarded > >> n_purge 1 . N total active purges > >> n_purge_add 1 0.00 N new purges added > >> n_purge_retire 0 0.00 N old purges deleted > >> n_purge_obj_test 0 0.00 N objects tested > >> n_purge_re_test 0 0.00 N regexps tested against > >> n_purge_dups 0 0.00 N duplicate purges removed > >> hcb_nolock 0 0.00 HCB Lookups without lock > >> hcb_lock 0 0.00 HCB Lookups with lock > >> hcb_insert 0 0.00 HCB Inserts > >> esi_parse 0 0.00 Objects ESI parsed (unlock) > >> esi_errors 0 0.00 ESI parse errors (unlock) > >> > >> > >> > >> Sascha Kain > >> IT / Administration > >> eraffe media GmbH & Co. KG Marketing - Consulting - Software > >> Sch?nfeldstr. 17 - 83022 Rosenheim > >> > >> Fon: + 49 (0)8031 - 941 41 -46 > >> Fax: + 49 (0)8031 - 941 41 -59 > >> E-Mail: s.kain at eraffe-media.de > >> www.eraffe-media.de - www.eraffe.de > >> > >> eraffe media GmbH & Co. KG, Sitz: Rosenheim, > >> Registergericht: AG Traunstein HR A Nr. 9104, > >> St-Nr. 156/157/58806, FA Rosenheim, > >> USt.-ID: DE250117972 > >> > >> Pers?nlich haftende Gesellschafterin: > >> eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, > >> Registergericht: AG Traunstein HR B 16956 > >> St-Nr. 156/116/90247, FA Rosenheim > >> > >> Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser > >> > >> _______________________________________________ > >> varnish-misc mailing list > >> varnish-misc at projects.linpro.no > >> http://projects.linpro.no/mailman/listinfo/varnish-misc > >> > >> > > > -- > Sascha Kain > IT / Administration > eraffe media GmbH & Co. KG Marketing - Consulting - Software > Sch?nfeldstr. 17 - 83022 Rosenheim > > Fon: + 49 (0)8031 - 941 41 -46 > Fax: + 49 (0)8031 - 941 41 -59 > E-Mail: s.kain at eraffe-media.de > www.eraffe-media.de - www.eraffe.de > > eraffe media GmbH & Co. KG, Sitz: Rosenheim, > Registergericht: AG Traunstein HR A Nr. 9104, > St-Nr. 156/157/58806, FA Rosenheim, > USt.-ID: DE250117972 > > Pers?nlich haftende Gesellschafterin: > eraffe media Verwaltungs-GmbH, Sitz: Rosenheim, > Registergericht: AG Traunstein HR B 16956 > St-Nr. 156/116/90247, FA Rosenheim > > Gesch?ftsf?hrer: Maximilian Kuss, Oliver D?ser > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jauderho at gmail.com Wed Apr 8 00:29:23 2009 From: jauderho at gmail.com (Jauder Ho) Date: Tue, 7 Apr 2009 17:29:23 -0700 Subject: varnish 2.0.4 and new config changes In-Reply-To: <20090407114641.GE5267@kjeks.linpro.no> References: <20090407101937.GC5267@kjeks.linpro.no> <20090407114641.GE5267@kjeks.linpro.no> Message-ID: On Tue, Apr 7, 2009 at 4:46 AM, Kristian Lyngstol < kristian at redpill-linpro.com> wrote: > On Tue, Apr 07, 2009 at 04:11:04AM -0700, Jauder Ho wrote: > > > FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk > > though. > > It would seem I was mistaken. It doesn't look like the obj to beresp > commits made it into 2.0.4 after a quick check. So that's only relevant to > trunk. > That could be a problem for some people as obj.ttl is not available in vcl_fetch in 2.0.4 and beresp.ttl is not available until trunk. > > > > Right now, only nginx1 sets X-Forwarded-For; varnish does not set > > X-Forwarded-For; nginx2 is a fastcgi frontend and does not set > > X-Forwarded-For. > > > > With this config (and using -trunk), nginx2 still logs the gateway IP so > > something strange is going on here. > > I'm not familiar with how nginx logs, but I'd start by verifying: > 1. That the X-Forwarded-For does indeed reach the relevant server. > 2. That the entity that logs honors X-Forwarded-For. > Looking at varnishlog (which covers the incoming request from nginx1 and response from nginx2. It looks like X-Forwarded-For is properly set on nginx1. However on the response, it looks like there are 2(?) X-Forwarded-For headers instead of being chained as in http://en.wikipedia.org/wiki/X-Forwarded-For#Format 10 SessionOpen c 192.168.1.20 33406 192.168.1.20:7777 10 ReqStart c 192.168.1.20 33406 1768153962 10 RxRequest c HEAD 10 RxURL c / 10 RxProtocol c HTTP/1.0 10 RxHeader c X-Real-IP: 208.69.40.136 10 RxHeader c X-Forwarded-For: 208.69.40.136 10 RxHeader c Host: shop.carumba.org 10 RxHeader c Connection: close 10 RxHeader c User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.10 libssh2/0.18 10 RxHeader c Accept: */* 10 VCL_call c recv 10 VCL_return c lookup 10 VCL_call c hash 10 VCL_return c hash 10 VCL_call c miss 10 VCL_return c fetch 11 BackendClose - default 11 BackendOpen b default 192.168.1.20 54829 67.180.237.183 9999 10 Backend c 11 default default 11 TxRequest b GET 11 TxURL b / 11 TxProtocol b HTTP/1.1 11 TxHeader b X-Real-IP: 208.69.40.136 11 TxHeader b X-Forwarded-For: 208.69.40.136 11 TxHeader b Host: shop.carumba.org 11 TxHeader b User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.10 libssh2/0.18 11 TxHeader b Accept: */* 11 TxHeader b X-Varnish: 1768153962 11 TxHeader b X-Forwarded-For: 192.168.1.20 11 RxProtocol b HTTP/1.1 11 RxStatus b 200 11 RxResponse b OK 11 RxHeader b Server: nginx 11 RxHeader b Date: Wed, 08 Apr 2009 00:25:35 GMT 11 RxHeader b Content-Type: text/html; charset=UTF-8 11 RxHeader b Transfer-Encoding: chunked 11 RxHeader b Connection: keep-alive 11 RxHeader b Set-Cookie: frontend=c4b996baf5465ad3c2cce532fe0af656; expires=Wed, 08 Apr 2009 01:25:35 GMT; path=/; domain=shop.carumba.org; HttpOnly 11 RxHeader b Expires: Thu, 19 Nov 1981 08:52:00 GMT 11 RxHeader b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 11 RxHeader b Pragma: no-cache 10 TTL c 1768153962 RFC 0 1239150335 1239150335 375007920 0 0 10 VCL_call c fetch 10 TTL c 1768153962 VCL 300 1239150335 10 VCL_return c pass 10 ObjProtocol c HTTP/1.1 10 ObjStatus c 200 10 ObjResponse c OK 10 ObjHeader c Server: nginx 10 ObjHeader c Date: Wed, 08 Apr 2009 00:25:35 GMT 10 ObjHeader c Content-Type: text/html; charset=UTF-8 10 ObjHeader c Set-Cookie: frontend=c4b996baf5465ad3c2cce532fe0af656; expires=Wed, 08 Apr 2009 01:25:35 GMT; path=/; domain=shop.carumba.org; HttpOnly 10 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT 10 ObjHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 10 ObjHeader c Pragma: no-cache 10 ObjHeader c X-Varnish-IP: 192.168.1.20 11 BackendReuse b default 10 Length c 8090 10 VCL_call c deliver 10 VCL_return c deliver 10 TxProtocol c HTTP/1.1 10 TxStatus c 200 10 TxResponse c OK 10 TxHeader c Server: nginx 10 TxHeader c Content-Type: text/html; charset=UTF-8 10 TxHeader c Set-Cookie: frontend=c4b996baf5465ad3c2cce532fe0af656; expires=Wed, 08 Apr 2009 01:25:35 GMT; path=/; domain=shop.carumba.org; HttpOnly 10 TxHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT 10 TxHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 10 TxHeader c Pragma: no-cache 10 TxHeader c X-Varnish-IP: 192.168.1.20 10 TxHeader c Content-Length: 8090 10 TxHeader c Date: Wed, 08 Apr 2009 00:25:35 GMT 10 TxHeader c X-Varnish: 1768153962 10 TxHeader c Age: 0 10 TxHeader c Via: 1.1 varnish 10 TxHeader c Connection: close 10 TxHeader c X-Cache: MISS 10 ReqEnd c 1768153962 1239150335.318713903 1239150335.470498323 0.000048637 0.151728630 0.000055790 10 SessionClose c Connection: close 10 StatSess c 192.168.1.20 33406 0 1 1 0 0 1 518 0 --Jauder -------------- next part -------------- An HTML attachment was scrubbed... URL: From darryl.dixon at winterhouseconsulting.com Wed Apr 8 01:02:04 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 8 Apr 2009 13:02:04 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory Message-ID: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> Hi All, I have an odd problem that I have only noticed happening since moving from 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child process. For example, I have a varnish instance with a 256MB cache allocated, that is currently consuming 4.9GB of resident memory (6.5GB virtual). The instance has only been running for 4 days and has only got 25MB of objects in the cache. This is clearly excessive and is causing us some serious problems in terms of memory pressure on the machine. Our VCL is largely unchanged from our 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, and the introduction of request restarts under certain limited scenarios. Can anyone shed some light? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From darryl.dixon at winterhouseconsulting.com Wed Apr 8 01:27:34 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 8 Apr 2009 13:27:34 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> Message-ID: <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> > Hi All, > > I have an odd problem that I have only noticed happening since moving from > 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child > process. For example, I have a varnish instance with a 256MB cache > allocated, that is currently consuming 4.9GB of resident memory (6.5GB > virtual). The instance has only been running for 4 days and has only got > 25MB of objects in the cache. > > This is clearly excessive and is causing us some serious problems in terms > of memory pressure on the machine. Our VCL is largely unchanged from our > 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, and > the introduction of request restarts under certain limited scenarios. Can > anyone shed some light? > One further footnote to this. I have a second varnish instance running on the same machine which talks to different backend servers (still primarily the same sort of content though), with the VCL only fractionally different - it does not seem to suffer from the runaway memory consumption of the first instance. The only difference in the VCL between the two is that in the one with runaway memory this is present in vcl_recv(): + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ ".*no-cache.*") { + purge_url(regsub(req.url, "[?].*$", ".*$")); + } + Is there possibly something in the regsub engine being triggered that is very expensive and would cause it to consume and hold on to large amounts of memory? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From michael at dynamine.net Wed Apr 8 02:45:10 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Tue, 7 Apr 2009 19:45:10 -0700 Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> Message-ID: Not that I have an answer, but I'd be curious to see the differences in 'pmap -x ' output for the different children. --Michael On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote: >> Hi All, >> >> I have an odd problem that I have only noticed happening since >> moving from >> 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child >> process. For example, I have a varnish instance with a 256MB cache >> allocated, that is currently consuming 4.9GB of resident memory >> (6.5GB >> virtual). The instance has only been running for 4 days and has >> only got >> 25MB of objects in the cache. >> >> This is clearly excessive and is causing us some serious problems >> in terms >> of memory pressure on the machine. Our VCL is largely unchanged >> from our >> 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, >> and >> the introduction of request restarts under certain limited >> scenarios. Can >> anyone shed some light? >> > > One further footnote to this. I have a second varnish instance > running on > the same machine which talks to different backend servers (still > primarily > the same sort of content though), with the VCL only fractionally > different > - it does not seem to suffer from the runaway memory consumption of > the > first instance. The only difference in the VCL between the two is > that in > the one with runaway memory this is present in vcl_recv(): > > + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ > ".*no-cache.*") { > + purge_url(regsub(req.url, "[?].*$", ".*$")); > + } > + > > Is there possibly something in the regsub engine being triggered > that is > very expensive and would cause it to consume and hold on to large > amounts > of memory? > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From darryl.dixon at winterhouseconsulting.com Wed Apr 8 03:05:36 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 8 Apr 2009 15:05:36 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> Message-ID: <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> > Not that I have an answer, but I'd be curious to see the differences > in 'pmap -x ' output for the different children. > > --Michael > That's a good thought - I'd almost forgotten about pmap. Anyhow, the 'normal' process is set up with a 2GB storage cache and pmap shows pretty much what you would expect - the largest anonymous segment is 41MB, with the vast majority of allocated memory booked against storage.cache (2GB). The pmap for the 'runaway' process shows 256MB booked against storage.cache, and then one of the an anonymous allocations is 2.3GB. For example: Normal process: ------------------------------------------------------ 3896: /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080 web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f /u01/data/prd/varnish/vcl.conf -s file,/u01/data/prd/varnish/storage.cache,2048M -n /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid Address Kbytes RSS Anon Locked Mode Mapping 0000000000400000 312 - - - r-x-- varnishd 000000000064d000 8 - - - rw--- varnishd 000000000064f000 68 - - - rw--- [ anon ] 0000000040000000 4 - - - ----- [ anon ] 0000000040001000 10240 - - - rw--- [ anon ] [...snip...] 00002aaab0900000 2097152 - - - rw-s- storage.cache 00002aab30900000 28 - - - r-x-- vcl.1P9zoqAU.so 00002aab30907000 2044 - - - ----- vcl.1P9zoqAU.so 00002aab30b06000 8 - - - rw--- vcl.1P9zoqAU.so 00002aab30c00000 2056 - - - rw--- [ anon ] 00002aab30f00000 2056 - - - rw--- [ anon ] 00002aab31200000 41984 - - - rw--- [ anon ] 00007fffae8f2000 88 - - - rw--- [ stack ] ffffffffff600000 8192 - - - ----- [ anon ] ---------------- ------ ------ ------ ------ total kB 2468512 - - - ------------------------------------------------------ Runaway process: ------------------------------------------------------ 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 -f /u01/data/prd-edit/varnish/vcl.conf -s file,/u01/data/prd-edit/varnish/storage.cache,256M -n /u01/data/prd-edit/varnish/state -P /u01/data/prd-edit/varnish/varnishd.pid Address Kbytes RSS Anon Locked Mode Mapping 0000000000400000 312 - - - r-x-- varnishd 000000000064d000 8 - - - rw--- varnishd 000000000064f000 68 - - - rw--- [ anon ] 0000000040000000 4 - - - ----- [ anon ] 0000000040001000 10240 - - - rw--- [ anon ] [...snip...] 00002aaab0900000 262144 - - - rw-s- storage.cache 00002aaac0900000 28 - - - r-x-- vcl.1P9zoqAU.so 00002aaac0907000 2048 - - - ----- vcl.1P9zoqAU.so 00002aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so 00002aaac0c00000 2056 - - - rw--- [ anon ] 00002aaac0f00000 2052 - - - rw--- [ anon ] 00002aaac1200000 233476 - - - rw--- [ anon ] 00002aaacf700000 2337792 - - - rw--- [ anon ] 00007fff4b619000 84 - - - rw--- [ stack ] ffffffffff600000 8192 - - - ----- [ anon ] ---------------- ------ ------ ------ ------ total kB 3121808 - - - ------------------------------------------------------ Both processes have been running for the same amount of time - 1day 17hours. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com > On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote: > >>> Hi All, >>> >>> I have an odd problem that I have only noticed happening since >>> moving from >>> 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child >>> process. For example, I have a varnish instance with a 256MB cache >>> allocated, that is currently consuming 4.9GB of resident memory >>> (6.5GB >>> virtual). The instance has only been running for 4 days and has >>> only got >>> 25MB of objects in the cache. >>> >>> This is clearly excessive and is causing us some serious problems >>> in terms >>> of memory pressure on the machine. Our VCL is largely unchanged >>> from our >>> 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, >>> and >>> the introduction of request restarts under certain limited >>> scenarios. Can >>> anyone shed some light? >>> >> >> One further footnote to this. I have a second varnish instance >> running on >> the same machine which talks to different backend servers (still >> primarily >> the same sort of content though), with the VCL only fractionally >> different >> - it does not seem to suffer from the runaway memory consumption of >> the >> first instance. The only difference in the VCL between the two is >> that in >> the one with runaway memory this is present in vcl_recv(): >> >> + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ >> ".*no-cache.*") { >> + purge_url(regsub(req.url, "[?].*$", ".*$")); >> + } >> + >> >> Is there possibly something in the regsub engine being triggered >> that is >> very expensive and would cause it to consume and hold on to large >> amounts >> of memory? >> From darryl.dixon at winterhouseconsulting.com Wed Apr 8 03:21:10 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 8 Apr 2009 15:21:10 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> Message-ID: <59033.58.28.153.120.1239160870.squirrel@services.directender.co.nz> Further to this, I just ran the pmap on the runaway process again (roughly 15 minutes since last time), and the large ~2GB allocation has jumped by about 60MB in that time, eg: ------------------------------------------------------ 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 -f /u01/data/prd-edit/varnish/vcl.conf -s file,/u01/data/prd-edit/varnish/storage.cache,256M -n /u01/data/prd-edit/varnish/state -P /u01/data/prd-edit/varnish/varnishd.pid Address Kbytes RSS Anon Locked Mode Mapping 0000000000400000 312 - - - r-x-- varnishd 000000000064d000 8 - - - rw--- varnishd 000000000064f000 68 - - - rw--- [ anon ] 0000000040000000 4 - - - ----- [ anon ] 0000000040001000 10240 - - - rw--- [ anon ] [...snip...] 00002aaab0900000 262144 - - - rw-s- storage.cache 00002aaac0900000 28 - - - r-x-- vcl.1P9zoqAU.so 00002aaac0907000 2048 - - - ----- vcl.1P9zoqAU.so 00002aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so 00002aaac0c00000 2056 - - - rw--- [ anon ] 00002aaac0f00000 2052 - - - rw--- [ anon ] 00002aaac1200000 233476 - - - rw--- [ anon ] 00002aaacf700000 2403328 - - - rw--- [ anon ] 00007fff4b619000 84 - - - rw--- [ stack ] ffffffffff600000 8192 - - - ----- [ anon ] ---------------- ------ ------ ------ ------ total kB 3187344 - - - ------------------------------------------------------ ...And in the time it's taken me to compose this mail it's grown by another 20MB: ------------------------------------------------------ 00002aaacf700000 2418688 - - - rw--- [ anon ] ------------------------------------------------------ ...so it is a pretty consistent, continuous upwards trend in terms of memory usage... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com >> Not that I have an answer, but I'd be curious to see the differences >> in 'pmap -x ' output for the different children. >> >> --Michael >> > > That's a good thought - I'd almost forgotten about pmap. Anyhow, the > 'normal' process is set up with a 2GB storage cache and pmap shows pretty > much what you would expect - the largest anonymous segment is 41MB, with > the vast majority of allocated memory booked against storage.cache (2GB). > The pmap for the 'runaway' process shows 256MB booked against > storage.cache, and then one of the an anonymous allocations is 2.3GB. For > example: > > > Normal process: > ------------------------------------------------------ > 3896: /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080 > web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f > /u01/data/prd/varnish/vcl.conf -s > file,/u01/data/prd/varnish/storage.cache,2048M -n > /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid > Address Kbytes RSS Anon Locked Mode Mapping > 0000000000400000 312 - - - r-x-- varnishd > 000000000064d000 8 - - - rw--- varnishd > 000000000064f000 68 - - - rw--- [ anon ] > 0000000040000000 4 - - - ----- [ anon ] > 0000000040001000 10240 - - - rw--- [ anon ] > [...snip...] > 00002aaab0900000 2097152 - - - rw-s- storage.cache > 00002aab30900000 28 - - - r-x-- vcl.1P9zoqAU.so > 00002aab30907000 2044 - - - ----- vcl.1P9zoqAU.so > 00002aab30b06000 8 - - - rw--- vcl.1P9zoqAU.so > 00002aab30c00000 2056 - - - rw--- [ anon ] > 00002aab30f00000 2056 - - - rw--- [ anon ] > 00002aab31200000 41984 - - - rw--- [ anon ] > 00007fffae8f2000 88 - - - rw--- [ stack ] > ffffffffff600000 8192 - - - ----- [ anon ] > ---------------- ------ ------ ------ ------ > total kB 2468512 - - - > ------------------------------------------------------ > > > > Runaway process: > ------------------------------------------------------ > 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 > web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 > -f /u01/data/prd-edit/varnish/vcl.conf -s > file,/u01/data/prd-edit/varnish/storage.cache,256M -n > /u01/data/prd-edit/varnish/state -P > /u01/data/prd-edit/varnish/varnishd.pid > Address Kbytes RSS Anon Locked Mode Mapping > 0000000000400000 312 - - - r-x-- varnishd > 000000000064d000 8 - - - rw--- varnishd > 000000000064f000 68 - - - rw--- [ anon ] > 0000000040000000 4 - - - ----- [ anon ] > 0000000040001000 10240 - - - rw--- [ anon ] > [...snip...] > 00002aaab0900000 262144 - - - rw-s- storage.cache > 00002aaac0900000 28 - - - r-x-- vcl.1P9zoqAU.so > 00002aaac0907000 2048 - - - ----- vcl.1P9zoqAU.so > 00002aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so > 00002aaac0c00000 2056 - - - rw--- [ anon ] > 00002aaac0f00000 2052 - - - rw--- [ anon ] > 00002aaac1200000 233476 - - - rw--- [ anon ] > 00002aaacf700000 2337792 - - - rw--- [ anon ] > 00007fff4b619000 84 - - - rw--- [ stack ] > ffffffffff600000 8192 - - - ----- [ anon ] > ---------------- ------ ------ ------ ------ > total kB 3121808 - - - > ------------------------------------------------------ > > > Both processes have been running for the same amount of time - 1day > 17hours. > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > > > >> On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote: >> >>>> Hi All, >>>> >>>> I have an odd problem that I have only noticed happening since >>>> moving from >>>> 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child >>>> process. For example, I have a varnish instance with a 256MB cache >>>> allocated, that is currently consuming 4.9GB of resident memory >>>> (6.5GB >>>> virtual). The instance has only been running for 4 days and has >>>> only got >>>> 25MB of objects in the cache. >>>> >>>> This is clearly excessive and is causing us some serious problems >>>> in terms >>>> of memory pressure on the machine. Our VCL is largely unchanged >>>> from our >>>> 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, >>>> and >>>> the introduction of request restarts under certain limited >>>> scenarios. Can >>>> anyone shed some light? >>>> >>> >>> One further footnote to this. I have a second varnish instance >>> running on >>> the same machine which talks to different backend servers (still >>> primarily >>> the same sort of content though), with the VCL only fractionally >>> different >>> - it does not seem to suffer from the runaway memory consumption of >>> the >>> first instance. The only difference in the VCL between the two is >>> that in >>> the one with runaway memory this is present in vcl_recv(): >>> >>> + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ >>> ".*no-cache.*") { >>> + purge_url(regsub(req.url, "[?].*$", ".*$")); >>> + } >>> + >>> >>> Is there possibly something in the regsub engine being triggered >>> that is >>> very expensive and would cause it to consume and hold on to large >>> amounts >>> of memory? >>> > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From doug at facebook.com Wed Apr 8 06:19:48 2009 From: doug at facebook.com (Doug Beaver) Date: Tue, 7 Apr 2009 23:19:48 -0700 Subject: populating cache backends out-of-band Message-ID: hi, i've searched the mailing lists and looked over the developer section of the wiki, and haven't found an answer for this yet. is it possible to preemptively store the response for a given uri without making a http request to the backend host(s)? i have a varnish tier with 100 hosts that is insulating a media serving tier, and i know ahead of time which objects users are going to request, since the objects are not requested until at least N seconds after they are written to the backend storage tier. i would like the initial user requests to always be cache hits. when new objects are written to the backend storage tier, i can make http requests for them to the varnish tier and force the objects to get loaded before users ask for them. but... all that does is time shift the i/o load on the backend, i.e. users won't see a delay when they ask for an object since it's already in the cache, but the amount of i/o work done on the backend is still the same. it would be optimal if there was a way for me to stick data into the cache without making the request to the backend media serving tier. for example, perhaps something like this on the admin cli: backend.store $uri $ttl $len $data or perhaps a way to send a POST to varnishd that contains the uri, response headers, and the content body. the previous caching solution i used was developed in-house and has a small embedded http server that stores its cached objects in a separate memcached process. the key is the uri and the value is the cached data, plus some minimal headers so we could reconstitute a valid response. that made it straightforward to do a memcache set and skip the backend requests until the object had fallen out of the cache, at which time its request rate is going to be much lower (after 7-10 days, the request rates for these objects drop 20x). i realize this is a bit of an edge case, but i would appreciate any advice or thoughts you guys have on this. if this is not supported in varnish currently, how disruptive of a change would it be to the caching model? thanks, doug -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at redpill-linpro.com Wed Apr 8 08:53:53 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 8 Apr 2009 10:53:53 +0200 Subject: varnish 2.0.4 and new config changes In-Reply-To: References: <20090407101937.GC5267@kjeks.linpro.no> <20090407114641.GE5267@kjeks.linpro.no> Message-ID: <20090408085352.GA4934@kjeks.linpro.no> On Tue, Apr 07, 2009 at 05:29:23PM -0700, Jauder Ho wrote: > On Tue, Apr 7, 2009 at 4:46 AM, Kristian Lyngstol < > kristian at redpill-linpro.com> wrote: > > > On Tue, Apr 07, 2009 at 04:11:04AM -0700, Jauder Ho wrote: > > > > > FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk > > > though. > > > > It would seem I was mistaken. It doesn't look like the obj to beresp > > commits made it into 2.0.4 after a quick check. So that's only relevant to > > trunk. > > > > That could be a problem for some people as obj.ttl is not available in > vcl_fetch in 2.0.4 and beresp.ttl is not available until trunk. That's not correct. Obj.ttl is available in hit, fetch, discard, timeout and error in 2.0.4. It's only available in hit, discard, timeout and error in trunk as beresp.ttl replaces it in fetch. > > > Right now, only nginx1 sets X-Forwarded-For; varnish does not set > > > X-Forwarded-For; nginx2 is a fastcgi frontend and does not set > > > X-Forwarded-For. > > > > > > With this config (and using -trunk), nginx2 still logs the gateway IP so > > > something strange is going on here. > > > > I'm not familiar with how nginx logs, but I'd start by verifying: > > 1. That the X-Forwarded-For does indeed reach the relevant server. > > 2. That the entity that logs honors X-Forwarded-For. > > Looking at varnishlog (which covers the incoming request from nginx1 and > response from nginx2. > > It looks like X-Forwarded-For is properly set on nginx1. However on the > response, it looks like there are 2(?) X-Forwarded-For headers instead of > being chained as in http://en.wikipedia.org/wiki/X-Forwarded-For#Format Varnish doesn't touch these headers unless you tell it to, so it looks like nginx is confused. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From kristian at redpill-linpro.com Wed Apr 8 09:27:43 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 8 Apr 2009 11:27:43 +0200 Subject: varnish killing off the child process after a few minutes In-Reply-To: References: Message-ID: <20090408092743.GB4934@kjeks.linpro.no> On Tue, Apr 07, 2009 at 01:01:51PM -0700, Tung Nguyen wrote: > Hi guys, > > We're on varnish 2.0.3 > > It looks like varnish restarts the child process for us every so often, > causing 503s :(. Was wondering if this is a known issue. Can you check the syslog for any more information? -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From des at des.no Wed Apr 8 09:41:59 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 08 Apr 2009 11:41:59 +0200 Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> (Darryl Dixon's message of "Wed, 8 Apr 2009 13:27:34 +1200 (NZST)") References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> Message-ID: <86ocv7a3dk.fsf@ds4.des.no> "Darryl Dixon - Winterhouse Consulting" writes: > + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ > ".*no-cache.*") { > + purge_url(regsub(req.url, "[?].*$", ".*$")); > + } > + It would be interesting to see how often this condition is true... > Is there possibly something in the regsub engine being triggered that is > very expensive and would cause it to consume and hold on to large amounts > of memory? I don't think so. However, your code will add an entry to the ban list every time a request that matches the above condition is received, and this entry will remain on the list until every object that was in the cache before the entry was added has been requested at least once. If you really want no-cache to purge the requested object, you should do it in vcl_hit (set obj.ttl to 0 and pass). Otherwise, you should simply pass directly from vcl_recv. DES -- Dag-Erling Sm?rgrav - des at des.no From harald at webmeisterei.com Wed Apr 8 11:42:59 2009 From: harald at webmeisterei.com (Harald Friessnegger) Date: Wed, 8 Apr 2009 13:42:59 +0200 Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> Message-ID: <200904081342.59789.harald@webmeisterei.com> hi darryl i had a simlar problem (varnish child process consuming lots of memory) a week ago (see the thread "make varnish don't start a subprocess") the solution for my problem seems to be to use a high-enough cache-size: starting varnish with -s file,"/tmp/storage",300M will make the child process use *about* 800M RAM (the RES column in top) after some hours. after that i needed to restart varnish to free some ram for my webserver. starting with a bigger cache -s file,"/tmp/storage",600M makes it using about 500M RAM (the RES column in top after 5 days of uptime) and happily coexist with zope ;-) maybe this helps you too. regards, fRiSi Am Mittwoch, 8. April 2009 05:05:36 schrieb Darryl Dixon - Winterhouse Consulting: > > Not that I have an answer, but I'd be curious to see the differences > > in 'pmap -x ' output for the different children. > > > > --Michael > > That's a good thought - I'd almost forgotten about pmap. Anyhow, the > 'normal' process is set up with a 2GB storage cache and pmap shows pretty > much what you would expect - the largest anonymous segment is 41MB, with > the vast majority of allocated memory booked against storage.cache (2GB). > The pmap for the 'runaway' process shows 256MB booked against > storage.cache, and then one of the an anonymous allocations is 2.3GB. For > example: > > > Normal process: > ------------------------------------------------------ > 3896: /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080 > web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f > /u01/data/prd/varnish/vcl.conf -s > file,/u01/data/prd/varnish/storage.cache,2048M -n > /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid > Address Kbytes RSS Anon Locked Mode Mapping > 0000000000400000 312 - - - r-x-- varnishd > 000000000064d000 8 - - - rw--- varnishd > 000000000064f000 68 - - - rw--- [ anon ] > 0000000040000000 4 - - - ----- [ anon ] > 0000000040001000 10240 - - - rw--- [ anon ] > [...snip...] > 00002aaab0900000 2097152 - - - rw-s- storage.cache > 00002aab30900000 28 - - - r-x-- vcl.1P9zoqAU.so > 00002aab30907000 2044 - - - ----- vcl.1P9zoqAU.so > 00002aab30b06000 8 - - - rw--- vcl.1P9zoqAU.so > 00002aab30c00000 2056 - - - rw--- [ anon ] > 00002aab30f00000 2056 - - - rw--- [ anon ] > 00002aab31200000 41984 - - - rw--- [ anon ] > 00007fffae8f2000 88 - - - rw--- [ stack ] > ffffffffff600000 8192 - - - ----- [ anon ] > ---------------- ------ ------ ------ ------ > total kB 2468512 - - - > ------------------------------------------------------ > > > > Runaway process: > ------------------------------------------------------ > 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 > web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 > -f /u01/data/prd-edit/varnish/vcl.conf -s > file,/u01/data/prd-edit/varnish/storage.cache,256M -n > /u01/data/prd-edit/varnish/state -P > /u01/data/prd-edit/varnish/varnishd.pid > Address Kbytes RSS Anon Locked Mode Mapping > 0000000000400000 312 - - - r-x-- varnishd > 000000000064d000 8 - - - rw--- varnishd > 000000000064f000 68 - - - rw--- [ anon ] > 0000000040000000 4 - - - ----- [ anon ] > 0000000040001000 10240 - - - rw--- [ anon ] > [...snip...] > 00002aaab0900000 262144 - - - rw-s- storage.cache > 00002aaac0900000 28 - - - r-x-- vcl.1P9zoqAU.so > 00002aaac0907000 2048 - - - ----- vcl.1P9zoqAU.so > 00002aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so > 00002aaac0c00000 2056 - - - rw--- [ anon ] > 00002aaac0f00000 2052 - - - rw--- [ anon ] > 00002aaac1200000 233476 - - - rw--- [ anon ] > 00002aaacf700000 2337792 - - - rw--- [ anon ] > 00007fff4b619000 84 - - - rw--- [ stack ] > ffffffffff600000 8192 - - - ----- [ anon ] > ---------------- ------ ------ ------ ------ > total kB 3121808 - - - > ------------------------------------------------------ > > > Both processes have been running for the same amount of time - 1day > 17hours. > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > > > On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote: > >>> Hi All, > >>> > >>> I have an odd problem that I have only noticed happening since > >>> moving from > >>> 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child > >>> process. For example, I have a varnish instance with a 256MB cache > >>> allocated, that is currently consuming 4.9GB of resident memory > >>> (6.5GB > >>> virtual). The instance has only been running for 4 days and has > >>> only got > >>> 25MB of objects in the cache. > >>> > >>> This is clearly excessive and is causing us some serious problems > >>> in terms > >>> of memory pressure on the machine. Our VCL is largely unchanged > >>> from our > >>> 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, > >>> and > >>> the introduction of request restarts under certain limited > >>> scenarios. Can > >>> anyone shed some light? > >> > >> One further footnote to this. I have a second varnish instance > >> running on > >> the same machine which talks to different backend servers (still > >> primarily > >> the same sort of content though), with the VCL only fractionally > >> different > >> - it does not seem to suffer from the runaway memory consumption of > >> the > >> first instance. The only difference in the VCL between the two is > >> that in > >> the one with runaway memory this is present in vcl_recv(): > >> > >> + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ > >> ".*no-cache.*") { > >> + purge_url(regsub(req.url, "[?].*$", ".*$")); > >> + } > >> + > >> > >> Is there possibly something in the regsub engine being triggered > >> that is > >> very expensive and would cause it to consume and hold on to large > >> amounts > >> of memory? > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Webmeisterei GmbH - B?ro f?r Netzfragen Tel: +43 5572 908877, ?Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com From tnguyen at bleacherreport.com Wed Apr 8 16:53:42 2009 From: tnguyen at bleacherreport.com (Tung Nguyen) Date: Wed, 8 Apr 2009 09:53:42 -0700 Subject: varnish killing off the child process after a few minutes In-Reply-To: <20090408092743.GB4934@kjeks.linpro.no> References: <20090408092743.GB4934@kjeks.linpro.no> Message-ID: Sure, Here's the syslog around one of the times when the child looks like its crashing. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ assets. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ vcl_recv. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: /javascripts/responders_1239147022.js?1239147050 Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ assets. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ vcl_recv. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: /images/shared/nav/dropdown-arrow.jpg Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ assets. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ vcl_recv. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: /images/shared/nav/rnd-left.jpg Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ assets. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ vcl_recv. Apr 8 01:19:00 ey03-s00344 varnishd[2628]: /images/shared/nav/rnd-right.jpg Apr 8 01:19:00 ey03-s00344 varnishd[2628]: @@@@ assets. Apr 8 01:19:01 ey03-s00344 cron[3661]: (bleacherreport) CMD (ruby /var/bleacherreport/current/bin/hit_counts.rb every_minute RAILS_ENV=staging 1>> /var/bleacherreport/current/log/cron_wrapper.log 2>&1) Apr 8 01:19:15 ey03-s00344 varnishd[2628]: @@@@ vcl_recv. Apr 8 01:19:15 ey03-s00344 kernel: [30326523.602151] varnishd[2740] general protection rip:2ac1df7c5b26 rsp:75062798 error:0 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (2628) died signal=11 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child cleanup complete Apr 8 01:19:15 ey03-s00344 varnishd[3938]: child (3670) Started Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Closed fds: 3 5 6 7 12 13 15 16 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Child starts Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said managed to mmap 6432399360 bytes of 6432399360 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Ready Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Host: ey03-s00344 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Connection: close Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 last message repeated 2 times Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said ", 3, 1) Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Host: ey03-s00344 Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Connection: close Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said Apr 8 01:19:15 ey03-s00344 last message repeated 2 times Apr 8 01:19:15 ey03-s00344 varnishd[3938]: Child (3670) said ", 3, 1) Apr 8 01:19:18 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:18 ey03-s00344 varnishd[3670]: /mlb/games/85 Apr 8 01:19:18 ey03-s00344 varnishd[3670]: @@@@ fallthrough. Apr 8 01:19:19 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:19 ey03-s00344 varnishd[3670]: /javascripts/base_1239147022.js?1239147049 Apr 8 01:19:19 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /javascripts/core_1239147022.js?1239147049 Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/hat/br_icon.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/shared/bkg-content.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/sprites/forms.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/search_indicator.gif?1239147022 Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/search_bg.png Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/logo_new.gif Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/shared/nav/bkg-nav.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/shared/nav/bkg-subnav.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/brui/tabs/bkg-selected.jpg Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:21 ey03-s00344 varnishd[3670]: /images/bkg-scoreboard.png Apr 8 01:19:21 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: /javascripts/responders_1239147022.js?1239147050 Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: /images/shared/nav/dropdown-arrow.jpg Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: /images/shared/nav/rnd-left.jpg Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: /images/shared/nav/rnd-right.jpg Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ assets. Apr 8 01:19:22 ey03-s00344 varnishd[3670]: @@@@ vcl_recv. Apr 8 01:19:22 ey03-s00344 kernel: [30326530.772097] varnishd[3785] general protection rip:2ac1df7c5b26 rsp:75863798 error:0 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3670) died signal=11 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child cleanup complete Apr 8 01:19:22 ey03-s00344 varnishd[3938]: child (3889) Started Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Closed fds: 3 5 6 7 12 13 15 16 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Child starts Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said managed to mmap 6432399360 bytes of 6432399360 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Ready Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Host: ey03-s00344 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Connection: close Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 last message repeated 2 times Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said ", 3, 1) Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Host: ey03-s00344 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Connection: close Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 last message repeated 2 times Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said ", 3, 1) Apr 8 01:19:24 ey03-s00344 varnishd[3889]: @@@@ vcl_recv. Apr 8 01:19:24 ey03-s00344 kernel: [30326532.799290] varnishd[4001] general protection rip:2ac1df7c5b26 rsp:76064798 error:0 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (3889) died signal=11 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child cleanup complete Apr 8 01:19:24 ey03-s00344 varnishd[3938]: child (4002) Started Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Closed fds: 3 5 6 7 12 13 15 16 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Child starts Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said managed to mmap 6432399360 bytes of 6432399360 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Ready Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Host: ey03-s00344 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Connection: close Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 last message repeated 2 times Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said ", 3, 1) Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Host: ey03-s00344 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Connection: close Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 last message repeated 2 times Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said ", 3, 1) Apr 8 01:20:01 ey03-s00344 cron[4249]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Apr 8 01:20:01 ey03-s00344 cron[4251]: (bleacherreport) CMD (ruby /var/bleacherreport/current/bin/hit_counts.rb every_minute RAILS_ENV=staging 1>> /var/bleacherreport/current/log/cron_wrapper.log 2>&1) Apr 8 01:20:03 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:03 ey03-s00344 varnishd[4002]: /mlb/scores Apr 8 01:20:03 ey03-s00344 varnishd[4002]: @@@@ fallthrough. Apr 8 01:20:04 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:04 ey03-s00344 varnishd[4002]: /javascripts/base_1239147022.js?1239147049 Apr 8 01:20:04 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /javascripts/core_1239147022.js?1239147049 Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/hat/br_icon.jpg Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/captchas/222_54_222_217_74_61_29_137_123_63_1_78_134_28_129_0.jpg Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/shared/bkg-content.jpg Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/sprites/forms.jpg Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/search_indicator.gif?1239147022 Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/logo_new.gif Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/search_bg.png Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Thanks, Tung Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Ready Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Host: ey03-s00344 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Connection: close Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 last message repeated 2 times Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said ", 3, 1) Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Host: ey03-s00344 Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Connection: close Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said Apr 8 01:19:22 ey03-s00344 last message repeated 2 times Apr 8 01:19:22 ey03-s00344 varnishd[3938]: Child (3889) said ", 3, 1) Apr 8 01:19:24 ey03-s00344 varnishd[3889]: @@@@ vcl_recv. Apr 8 01:19:24 ey03-s00344 kernel: [30326532.799290] varnishd[4001] general protection rip:2ac1df7c5b26 rsp:76064798 error:0 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (3889) died signal=11 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child cleanup complete Apr 8 01:19:24 ey03-s00344 varnishd[3938]: child (4002) Started Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Closed fds: 3 5 6 7 12 13 15 16 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Child starts Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said managed to mmap 6432399360 bytes of 6432399360 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Ready Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Host: ey03-s00344 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Connection: close Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 last message repeated 2 times Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said ", 3, 1) Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Probe("GET http://stagingbleacherreport.com/images/blank.gif HTTP/1.1 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Host: ey03-s00344 Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Connection: close Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said Apr 8 01:19:24 ey03-s00344 last message repeated 2 times Apr 8 01:19:24 ey03-s00344 varnishd[3938]: Child (4002) said ", 3, 1) Apr 8 01:20:01 ey03-s00344 cron[4249]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Apr 8 01:20:01 ey03-s00344 cron[4251]: (bleacherreport) CMD (ruby /var/bleacherreport/current/bin/hit_counts.rb every_minute RAILS_ENV=staging 1>> /var/bleacherreport/current/log/cron_wrapper.log 2>&1) Apr 8 01:20:03 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:03 ey03-s00344 varnishd[4002]: /mlb/scores Apr 8 01:20:03 ey03-s00344 varnishd[4002]: @@@@ fallthrough. Apr 8 01:20:04 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:04 ey03-s00344 varnishd[4002]: /javascripts/base_1239147022.js?1239147049 Apr 8 01:20:04 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /javascripts/core_1239147022.js?1239147049 Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: /images/hat/br_icon.jpg Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ assets. Apr 8 01:20:06 ey03-s00344 varnishd[4002]: @@@@ vcl_recv. Tung On Wed, Apr 8, 2009 at 2:27 AM, Kristian Lyngstol < kristian at redpill-linpro.com> wrote: > On Tue, Apr 07, 2009 at 01:01:51PM -0700, Tung Nguyen wrote: > > Hi guys, > > > > We're on varnish 2.0.3 > > > > It looks like varnish restarts the child process for us every so often, > > causing 503s :(. Was wondering if this is a known issue. > > Can you check the syslog for any more information? > > -- > Kristian Lyngst?l > Redpill Linpro AS > Tlf: +47 21544179 > Mob: +47 99014497 > -- Tung Nguyen, Lead Developer Bleacher Report, The Open Source Sports Network (510) 928-0475 -------------- next part -------------- An HTML attachment was scrubbed... URL: From darryl.dixon at winterhouseconsulting.com Wed Apr 8 20:37:53 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Thu, 9 Apr 2009 08:37:53 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <86ocv7a3dk.fsf@ds4.des.no> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> <86ocv7a3dk.fsf@ds4.des.no> Message-ID: <50436.58.28.153.120.1239223073.squirrel@services.directender.co.nz> Hi DES, > "Darryl Dixon - Winterhouse Consulting" > writes: >> + if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~ >> ".*no-cache.*") { >> + purge_url(regsub(req.url, "[?].*$", ".*$")); >> + } >> + > > It would be interesting to see how often this condition is true... > Shouldn't be too often in this environment - I have watched with varnishlog and the rate is very low... >> Is there possibly something in the regsub engine being triggered that is >> very expensive and would cause it to consume and hold on to large >> amounts >> of memory? > > I don't think so. However, your code will add an entry to the ban list > every time a request that matches the above condition is received, and > this entry will remain on the list until every object that was in the > cache before the entry was added has been requested at least once. > That sounds like a pretty good candidate then, as I can envisage that there are possibly many items that may fit that criteria in my scenario. However it definitely seems out of proportion to the actual cache utilisation (only 25MB of objects in the cache, but memory growing to multiple GB) > If you really want no-cache to purge the requested object, you should do > it in vcl_hit (set obj.ttl to 0 and pass). Otherwise, you should simply > pass directly from vcl_recv. > My understanding of using 'pass' in vcl_hit is that it causes the object to be marked as 'hit for pass' and therefore is uncacheable from that point onwards. In fact, pass'ing out of vcl_hit was what I used to do a while ago, but it caused that exact behaviour (items would no longer be cached by varnish) so I swapped it for purge_url... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From darryl.dixon at winterhouseconsulting.com Wed Apr 8 20:40:39 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Thu, 9 Apr 2009 08:40:39 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <200904081342.59789.harald@webmeisterei.com> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <55404.58.28.153.120.1239159936.squirrel@services.directender.co.nz> <200904081342.59789.harald@webmeisterei.com> Message-ID: <60335.58.28.153.120.1239223239.squirrel@services.directender.co.nz> > hi darryl > > i had a simlar problem (varnish child process consuming lots of memory) a > week > ago (see the thread "make varnish don't start a subprocess") > > the solution for my problem seems to be to use a high-enough cache-size: > > starting varnish with > > -s file,"/tmp/storage",300M > > will make the child process use *about* 800M RAM (the RES column in top) > after > some hours. after that i needed to restart varnish to free some ram for my > webserver. > > > starting with a bigger cache > > -s file,"/tmp/storage",600M > > makes it using about 500M RAM (the RES column in top after 5 days of > uptime) > and happily coexist with zope ;-) > > > > maybe this helps you too. > Hi Harald, That is a very interesting observation - I will give it a try. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From svein-listmail at stillbilde.net Fri Apr 10 10:59:47 2009 From: svein-listmail at stillbilde.net (Svein Skogen (listmail accont)) Date: Fri, 10 Apr 2009 12:59:47 +0200 Subject: Can VCL files use includes? Message-ID: <49DF26A3.7030208@stillbilde.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Can a VCL file use includes in blocks? Reason I'm asking, is that among the things I am using my varnish setup to do, is to block a long list of user-agents (of known address-scraper bots for spammers) by giving them a nice 403. However, having this list in my main config makes the main config file very long, thus less readable, and as we all know: Configfiles that are unreadable may cause security grief. :p So, what I want to do, is to move such blocklists out to their own files, and simply use include-statements (preferrably withing the respective blocks) to read them. Is this possible? //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknfJqIACgkQODUnwSLUlKS3ogCgmd1q6+6vsU9tQMhaUdPZchcB 8XQAoJdDXUn3Fl9eC5Op+3b+UZ2WcMoZ =1FmM -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Fri Apr 10 16:04:14 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 10 Apr 2009 16:04:14 +0000 Subject: Can VCL files use includes? In-Reply-To: Your message of "Fri, 10 Apr 2009 12:59:47 +0200." <49DF26A3.7030208@stillbilde.net> Message-ID: <18363.1239379454@critter.freebsd.dk> In message <49DF26A3.7030208 at stillbilde.net>, "Svein Skogen (listmail accont)" writes: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Can a VCL file use includes in blocks? Yes, in fact you can use it everywhere, just put include "filename" ; This was done so that for instance ACLs could be included etc. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jauderho at gmail.com Tue Apr 7 11:11:04 2009 From: jauderho at gmail.com (Jauder Ho) Date: Tue, 7 Apr 2009 04:11:04 -0700 Subject: varnish 2.0.4 and new config changes In-Reply-To: <20090407101937.GC5267@kjeks.linpro.no> References: <20090407101937.GC5267@kjeks.linpro.no> Message-ID: Kristian, Sorry about that. I'm cc'ing to varnish-misc. On Tue, Apr 7, 2009 at 3:19 AM, Kristian Lyngstol < kristian at redpill-linpro.com> wrote: > On Tue, Apr 07, 2009 at 02:55:13AM -0700, Jauder Ho wrote: > > I just downloaded the new 2.0.4 release and noticed that a couple of > things > > had changed causing my existing config to stop working. Therefore, I had > a > > couple of questions that I'm hoping that you can help answer. > > > > a) Is beresp.ttl basically replacing obj.ttl? If so, I should be able to > do > > the following right in vcl_fetch? > > > > if (beresp.ttl < 300s) { > > set beresp.ttl = 300s; > > } > > Yes, for all intents and purposes, beresp is obj. The name change is > reflecting some underlying changes that doesn't really affect how you write > VCL - yet. FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk though. Error msg from varnish-2.0.4 upon startup: Message from VCC-compiler: Unknown variable 'beresp.ttl' At: (input Line 128 Pos 7) if (beresp.ttl < 300s) { ------##########---------- Running VCC-compiler failed, exit 1 VCL compilation failed > > > > b) With the change above, it looks like > > http://varnish.projects.linpro.no/wiki/FAQ#HowdoIaddaHTTPheader is > invalid > > as obj is not valid in vcl_fetch. Just wanted to confirm that > > beresp.http.X-Varnish-IP; works as expected. > > See the above answer (yes, it's valid). > > > c) Lastly, I have a config of user > nginx1 > varnish > nginx2 > php-fpm > and > > noticed that the IP being logged on nginx2 is the internal IP. nginx1 has > > X-Forwarded-For set. Using the wiki's suggestion at > > > http://varnish.projects.linpro.no/wiki/FAQ#HowcanIlogtheclientIPaddressonthebackend > , > > > > > > sub vcl_recv { > > # Add a unique header containing the client address > > remove req.http.X-Forwarded-For; > > set req.http.X-Forwarded-For = client.ip; > > # [...] > > } > > > > This does not seem to work to show the user IP on nginx2 so any > suggestions > > would be appreciated as I'm wondering if the changes 2.0.4 affect this. > > Thanks! > > If your first nginx sets X-Forwarded-For, then setting > req.http.X-Forwarded-For = client.ip; will overwrite it with nginx1's IP. > So don't set X-Forwarded-For in vcl_recv and it will pass straight through > Varnish. 2.0.4 does not affect this compared to 2.0.3. > Right now, only nginx1 sets X-Forwarded-For; varnish does not set X-Forwarded-For; nginx2 is a fastcgi frontend and does not set X-Forwarded-For. With this config (and using -trunk), nginx2 still logs the gateway IP so something strange is going on here. --Jauder > > By the way, this discussion belongs on one of the mailing lists. > > -- > Kristian Lyngst?l > Redpill Linpro AS > Tlf: +47 21544179 > Mob: +47 99014497 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at redpill-linpro.com Tue Apr 14 11:09:10 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 14 Apr 2009 13:09:10 +0200 Subject: How to let varnishncsa to log the realserver ip In-Reply-To: <20090403023956.BFDA5A4980D@mail3-51.sinamail.sina.com.cn> (suran's message of "Fri, 03 Apr 2009 10:39:56 +0800") References: <20090403023956.BFDA5A4980D@mail3-51.sinamail.sina.com.cn> Message-ID: <87d4bfmqzt.fsf@qurzaw.linpro.no> ]] | I have a questtion, that is How to let varnishncsa log the backend | realserver ip to varnish.log?I search in google,but no result,anyone | can help me ? varnishncsa currently does not support custom log formats, so this is not possible yet. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Tue Apr 14 11:10:14 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 14 Apr 2009 13:10:14 +0200 Subject: VCL syntax - unset and remove In-Reply-To: <200904041225.27400.gaute@pht.no> (Gaute Amundsen's message of "Sat, 4 Apr 2009 12:25:27 +0200") References: <200904041225.27400.gaute@pht.no> Message-ID: <878wm3mqy1.fsf@qurzaw.linpro.no> ]] Gaute Amundsen | Various usages: | | remove req.http.cookie; | unset obj.http.Set-Cookie; | unset resp.http.magicmarker; | | Unset is not in man vcl (v2.0.3) but remove is. They are synonyms, but unset is deprecated. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Tue Apr 14 11:14:43 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 14 Apr 2009 13:14:43 +0200 Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <50436.58.28.153.120.1239223073.squirrel@services.directender.co.nz> (Darryl Dixon's message of "Thu, 9 Apr 2009 08:37:53 +1200 (NZST)") References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> <86ocv7a3dk.fsf@ds4.des.no> <50436.58.28.153.120.1239223073.squirrel@services.directender.co.nz> Message-ID: <874owrmqqk.fsf@qurzaw.linpro.no> ]] "Darryl Dixon - Winterhouse Consulting" | My understanding of using 'pass' in vcl_hit is that it causes the object | to be marked as 'hit for pass' and therefore is uncacheable from that | point onwards. In fact, pass'ing out of vcl_hit was what I used to do a | while ago, but it caused that exact behaviour (items would no longer be | cached by varnish) so I swapped it for purge_url... You can try setting obj.ttl = 0s and then restart in vcl_hit. I believe that should work. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From darryl.dixon at winterhouseconsulting.com Tue Apr 14 19:37:16 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 15 Apr 2009 07:37:16 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <874owrmqqk.fsf@qurzaw.linpro.no> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> <86ocv7a3dk.fsf@ds4.des.no> <50436.58.28.153.120.1239223073.squirrel@services.directender.co.nz> <874owrmqqk.fsf@qurzaw.linpro.no> Message-ID: <42136.118.93.187.84.1239737836.squirrel@services.directender.co.nz> > | My understanding of using 'pass' in vcl_hit is that it causes the object > | to be marked as 'hit for pass' and therefore is uncacheable from that > | point onwards. In fact, pass'ing out of vcl_hit was what I used to do a > | while ago, but it caused that exact behaviour (items would no longer be > | cached by varnish) so I swapped it for purge_url... > > You can try setting obj.ttl = 0s and then restart in vcl_hit. I believe > that should work. > Thanks Tollef - I will try this today and report back. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From mperham at gmail.com Wed Apr 15 20:49:57 2009 From: mperham at gmail.com (Mike Perham) Date: Wed, 15 Apr 2009 15:49:57 -0500 Subject: bus error in VCL processing Message-ID: Varnish gives a bus error (2.0.2 on OSX) when I start it up with this VCL: sub vcl_miss { if (!(bereq.url ~ req.http.host)) { set bereq.url = "/" req.http.host req.url; } } I'm trying to add the Host header to the front of the URL before passing to our S3 backend. Since relative ESIs use the modified URL, I need to verify the URL has not already been mangled before. I believe the issue is due to the use of req.http.host as the match substring. if I replace it with a hardcoded string, the bus error goes away: if (!(bereq.url ~ 'foo')) { How can I do what I'm trying to do? mike From darryl.dixon at winterhouseconsulting.com Thu Apr 16 03:15:45 2009 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Thu, 16 Apr 2009 15:15:45 +1200 (NZST) Subject: Varnish 2.0.3 consuming excessive memory In-Reply-To: <42136.118.93.187.84.1239737836.squirrel@services.directender.co.nz> References: <55047.58.28.153.120.1239152524.squirrel@services.directender.co.nz> <56423.58.28.153.120.1239154054.squirrel@services.directender.co.nz> <86ocv7a3dk.fsf@ds4.des.no> <50436.58.28.153.120.1239223073.squirrel@services.directender.co.nz> <874owrmqqk.fsf@qurzaw.linpro.no> <42136.118.93.187.84.1239737836.squirrel@services.directender.co.nz> Message-ID: <60952.58.28.124.90.1239851745.squirrel@services.directender.co.nz> >> | My understanding of using 'pass' in vcl_hit is that it causes the >> object >> | to be marked as 'hit for pass' and therefore is uncacheable from that >> | point onwards. In fact, pass'ing out of vcl_hit was what I used to do >> a >> | while ago, but it caused that exact behaviour (items would no longer >> be >> | cached by varnish) so I swapped it for purge_url... >> >> You can try setting obj.ttl = 0s and then restart in vcl_hit. I believe >> that should work. >> > > Thanks Tollef - I will try this today and report back. This seems to correct the problem - thanks Tollef! Further to root cause of the original issue: using purge_url seems to result inside EXP_Rearm with the objexp (object expire?) being added unconditionally to the exp_heap binheap, and it's not clear to me where (if ever) this heap is pruned. For example, I see in varnishlog multiple calls to the same URL from the same client with the same headers resulting in multiple unique entries being added to exp_heap, eg: ExpBan c 981217322 was banned [...] ExpBan c 981217323 was banned [...etc...] Where, if anywhere, is exp_heap pruned for entries like this? I couldn't quite make sense of exp_timer() to be certain of under what conditions it actually removes things from this heap - it looks like that function is mainly concerned with items expiring from the Expires http header or similar... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From iiminar at gmail.com Fri Apr 17 21:54:25 2009 From: iiminar at gmail.com (Igor Minar) Date: Fri, 17 Apr 2009 14:54:25 -0700 Subject: Frequent Varnish 2.0.4 Panics Message-ID: <64D52F5B-584E-45A3-8880-6511B43A7107@gmail.com> Hi there, I see frequent (~every 30 seconds) Varnish panics when running a light load tests on our webapp which is proxied by varnish. Here are some details: Varnish 2.0.4 on Solaris 10 Both Varnish and the backend are on the same host Varnish is started as: varnishd -a 0.0.0.0:8080 -f foo.vcl -T 0.0.0.0:9998 -t 300 -s malloc,2G -u webservd Has anyone else experienced the same issue? Let me know if you need more info. Thanks, Igor The error messages are: Error #1: Apr 17 14:26:31 host1 varnishd[13698]: [ID 679826 local0.error] Manager got SIGINT Apr 17 14:31:31 host1 varnishd[28730]: [ID 540784 local0.error] Child (28731) not responding to ping, killing it. Apr 17 14:31:34 host1 last message repeated 2 times Apr 17 14:31:34 host1 varnishd[28730]: [ID 733861 local0.error] Child (28731) Panic message: Assert error in WS_Release(), cache_ws.c line 172: Apr 17 14:31:34 host1 Condition(ws->r != NULL) not true. errno = 9 (Bad file number) thread = (cache-worker)sp = 82963f4 { Apr 17 14:31:34 host1 fd = 4, id = 4, xid = 1278321417, Apr 17 14:31:34 host1 client = 192.18.43.225:12811, Apr 17 14:31:34 host1 step = STP_HIT, Apr 17 14:31:34 host1 handling = error, Apr 17 14:31:34 host1 ws = 829643c { Apr 17 14:31:34 host1 id = "sess", Apr 17 14:31:34 host1 {s,f,r,e} = {8296a24,,+316,0,+16384}, Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 worker = f93adef4 { Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 vcl = { Apr 17 14:31:34 host1 srcname = { Apr 17 14:31:34 host1 "input", Apr 17 14:31:34 host1 "Default", Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 obj = 8300fd8 { Apr 17 14:31:34 host1 refcnt = 5, xid = 1278321415, Apr 17 14:31:34 host1 ws = 8300ff0 { Apr 17 14:31:34 host1 id = "obj", Apr 17 14:31:34 host1 {s,f,r,e} = {83011c4,,+555,0,+7700}, Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 http = { Apr 17 14:31:34 host1 ws = 8300ff0 { Apr 17 14:31:34 host1 id = "obj", Apr 17 14:31:34 host1 {s,f,r,e} = {83011c4,,+555,0,+7700}, Apr 17 14:31:34 host1 }, Apr 17 14:31:34 host1 hd = { Apr 17 14:31:34 host1 "Server: Sun-Java-System-Web-Server/7.0", Apr 17 14:31:34 host1 "Date: Fri, 17 Apr 2009 21:31:26 GMT", Apr 17 14:31:34 host1 "Cache-control: no-cache, must- revalidate", Apr 17 14:31:34 host1 "Pragma: no-cache", Apr 17 14:31:34 host1 "Expires: Thu, 01 Jan 1 Error #2: Apr 17 14:32:12 host1 varnishd[28730]: [ID 540784 local0.error] Child (28817) not responding to ping, killing it. Apr 17 14:32:13 host1 last message repeated 1 time Apr 17 14:32:13 host1 varnishd[28730]: [ID 733861 local0.error] Child (28817) Panic message: Assert error in WS_Reserve(), cache_ws.c line 156: Apr 17 14:32:13 host1 Condition(ws->r == NULL) not true. errno = 9 (Bad file number) thread = (cache-worker)sp = 834d30c { Apr 17 14:32:13 host1 fd = 41, id = 41, xid = 742183585, Apr 17 14:32:13 host1 client = 192.18.43.225:8624, Apr 17 14:32:13 host1 step = STP_HIT, Apr 17 14:32:13 host1 handling = error, Apr 17 14:32:13 host1 ws = 834d354 { Apr 17 14:32:13 host1 id = "sess", Apr 17 14:32:13 host1 {s,f,r,e} = {834d93c,,+287,0,+16384}, Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 worker = f8fadef4 { Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 vcl = { Apr 17 14:32:13 host1 srcname = { Apr 17 14:32:13 host1 "input", Apr 17 14:32:13 host1 "Default", Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 obj = 82a0a30 { Apr 17 14:32:13 host1 refcnt = 5, xid = 742183268, Apr 17 14:32:13 host1 ws = 82a0a48 { Apr 17 14:32:13 host1 id = "obj", Apr 17 14:32:13 host1 {s,f,r,e} = {82a0c1c,,+3939,0,+7700}, Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 http = { Apr 17 14:32:13 host1 ws = 82a0a48 { Apr 17 14:32:13 host1 id = "obj", Apr 17 14:32:13 host1 {s,f,r,e} = {82a0c1c,,+3939,0,+7700}, Apr 17 14:32:13 host1 }, Apr 17 14:32:13 host1 hd = { Apr 17 14:32:13 host1 "Server: Sun-Java-System-Web-Server/7.0", Apr 17 14:32:13 host1 "Date: Fri, 17 Apr 2009 21:31:38 GMT", Apr 17 14:32:13 host1 "Cache-control: no-cache, must- revalidate", Apr 17 14:32:13 host1 "Pragma: no-cache", Apr 17 14:32:13 host1 "Expires: Thu, 01 Jan From varnish-misc at projects.linpro.no Sat Apr 18 02:37:53 2009 From: varnish-misc at projects.linpro.no (VIAGRA ® Pfizer Inc.) Date: Sat, 18 Apr 2009 04:37:53 +0200 Subject: WJ: Message72647 Message-ID: <20090418063753.8053.qmail@maris> An HTML attachment was scrubbed... URL: -------------- next part -------------- New from WebMD: Dear varnish-misc at projects.linpro.no!Show her who the REAL man is.. Sign-up today! You are subscribed as varnish-misc at projects.linpro.no. View and manage your WebMD newsletter preferences. Subscribe to more newsletters. Change/update your email address. WebMD Privacy Policy WebMD Office of Privacy 1175 Peachtree Street, Suite 2400, Atlanta, GA 30361 ? 2009 WebMD, LLC. All rights reserved. From carlos.oliva at igloo.cl Mon Apr 20 03:39:23 2009 From: carlos.oliva at igloo.cl (Carlos Oliva G.) Date: Sun, 19 Apr 2009 23:39:23 -0400 Subject: Varnish as load balancer Message-ID: <69C3BA0F-2591-484A-9B72-DEF696CDB3EC@igloo.cl> Hi, I've been following the Varnih development for the last few months. I'm using it as a front-end cache for a fairly large Website which runs on Zope/Plone, with a Zeo Cluster architecture of 1 Zeo Server and 3 Zeo Clients sitting behind Lighttpd acting as a load balancer, which in turn is right behind Varnish. I'm currently implementing Varnish over a similar setup, but I began working with the 2.0 version of the software. After discovering and playing around with all the new options I upgraded the original Varnish installation. In the current setup I'm skipping Lighttpd altogether and began using Varnish as a load balancer altogether as well, with the 'directors' feature. However I'm still missing the last bit of the requested functionality that is to evenly balance all the requests from 'anonymous' users that aren't registered/logged into the system (which I am already identifying in the sub_rcvd function of my VCL), but for registered users to preserve their session by redirecting them always to the same backend server. I know I can create a 'hash' using some relatively unique values, as the originating IP address + the User Agent and a session-dependent cookie value, but what I don't know is (if possible), how to use this hash to determinate to what backend server to direct the request to. If what I'm asking is not possible to do with Varnish, any suggestions on using another tool as a load balancer behind Varnish would be greatly appreciated. Best regards, -- ? Carlos Oliva G. Igloo Sistemas Ltda. carlos.oliva at igloo.cl - http://www.igloo.cl/ From kristian at redpill-linpro.com Mon Apr 20 08:08:42 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Mon, 20 Apr 2009 10:08:42 +0200 Subject: Frequent Varnish 2.0.4 Panics In-Reply-To: <64D52F5B-584E-45A3-8880-6511B43A7107@gmail.com> References: <64D52F5B-584E-45A3-8880-6511B43A7107@gmail.com> Message-ID: <20090420080842.GE1762@kjeks.linpro.no> On Fri, Apr 17, 2009 at 02:54:25PM -0700, Igor Minar wrote: > Hi there, > > I see frequent (~every 30 seconds) Varnish panics when running a light > load tests on our webapp which is proxied by varnish. What sort of load are you testing with? > Has anyone else experienced the same issue? Let me know if you need > more info. By the looks of it, the child is killed of because it's unresponsive, and it seems to me that it assert-error in the child is a result of this. Have you tried simply increasing the client timeout? (-p cli_timeout). The default is very agressive, and I've seen it kill off healthy childs that are bussy delivering content even with a cli_timeout as high as 21 seconds. Definitely something to test at least. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From samcrawford at gmail.com Mon Apr 20 13:54:33 2009 From: samcrawford at gmail.com (Sam Crawford) Date: Mon, 20 Apr 2009 14:54:33 +0100 Subject: Cookie handling Message-ID: Afternoon all, I've been a keen user of Varnish for the past year or so and have just come across a use case that I'm having trouble implementing. I'm familiar with stripping the cookie upon insert and lookup in order to cache cookie-based content, but the following is a slight twist on this. We've got an SSO (single sign-on) service that requires an authentication cookie to be presented for any protected resource. Our current setup (which we're not able to change for a number of reasons) means that every URL under http://portal.company.com/* is SSO protected - including requests for static images, javascript, css, etc. Naturally we'd like to put Varnish in front of this portal to cache static content. The issue arises when we write our custom definitions of vcl_fetch. If we find a static URL we'd do some like "remove req.http.cookie" and then "lookup". However, if the content was not found in memory then it goes to the backend. But by this point the incoming SSO cookie of the user has been removed, so their request to the backend is denied. So, is there a way to lookup a query in the hashtable *without* cookies, but then if the request has to go to the backend *keep* the cookies? I appreciate that this is conceptually not an ideal solution (as we'll be caching another user's permissions to static content, etc etc), but any suggestions would be greatly appreciated. Thanks, Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at redpill-linpro.com Mon Apr 20 14:07:20 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Mon, 20 Apr 2009 16:07:20 +0200 Subject: Cookie handling In-Reply-To: References: Message-ID: <20090420140720.GF1762@kjeks.linpro.no> On Mon, Apr 20, 2009 at 02:54:33PM +0100, Sam Crawford wrote: > The issue arises when we write our custom definitions of vcl_fetch. If we > find a static URL we'd do some like "remove req.http.cookie" and then > "lookup". However, if the content was not found in memory then it goes to > the backend. But by this point the incoming SSO cookie of the user has been > removed, so their request to the backend is denied. > > So, is there a way to lookup a query in the hashtable *without* cookies, but > then if the request has to go to the backend *keep* the cookies? This is actually the default behavior. By default, only the url and hostname is part of the hash. However, the default vcl also explicitly passes any request with a cookie, which is why stripping the cookie is necessary. If you replicate the behavior you want from the default vcl in your own VCL, then end vcl_recv with lookup with a cookie, the object will be fetched using the cookie, then stored (as long as vcl_fetch allows it to be stored). Keep in mind, however, that any protection is lost, as cached data won't have their cookie confirmed. But I guess you're already aware of that. Hope this can point you in the right direction. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From andreas at atela.net Tue Apr 21 10:46:15 2009 From: andreas at atela.net (Andreas Kaiser - Atela CC) Date: Tue, 21 Apr 2009 12:46:15 +0200 Subject: multiple domains in one backend & 2 varnish servers Message-ID: <49EDA3F7.6070308@atela.net> Hello, I'm testing varnish 2.0.4 for a project and struggling trough the documentation. In my scenario I have a backend server "cms-server" with multiple domains/sites (for example domain1.com, domain2.com ...) and 2 varnish server (one in Europe and one in USA). Because some domains should be served by Europe server and others by the usa server I think I need to configure each varnish install with a specific vcl config file. What I don't have clear is if I need to use as backend only the IP of the CMS server or multiple backends for every domain, also all domains would have the same configuration related with cookies, compression, etc: backend cmsserver{ .host = "xxx.xxx.xxx.xxx"; .port = "80"; } My experience with varnish is related with one webserver with one domain that also has varnish installed... If I'm not wrong, varnish would create a cache for each hostname/domain but I'm lost how varnish and cmsserver should be configured for showing the specific content for each domain if the servers are different ones... The reason of this configuration is that the cms-server will be only accessible from a private network and the 2 varnish server (which will be used as public frontend servers) Would varnish be a good choice in this situation? Thanks in advance... From sergei.kononov at gmail.com Tue Apr 21 16:03:31 2009 From: sergei.kononov at gmail.com (Sergei Kononov) Date: Tue, 21 Apr 2009 20:03:31 +0400 Subject: X-Accel-Redirect support in varnish Message-ID: If someone is interested, here is a snippet of vcl_fetch for X-Accel-Redirect header handling: ... if ( obj.http.x-accel-redirect ~ ".*" ) { set req.url = obj.http.x-accel-redirect; restart; } ... -- Sergei Kononov From augustin at waw.com Wed Apr 22 06:41:58 2009 From: augustin at waw.com (Augustin Amann) Date: Wed, 22 Apr 2009 08:41:58 +0200 Subject: Internal redirecition with dynamic backend (fetch 302 and return content) Message-ID: <49EEBC36.3070001@waw.com> Hi, I'm looking for a reverse-proxy system with internal redirection, and I wonder if varnish can do the job. I need the reverse proxy to act as a User-Agent with 302, and fetch content instead of returning 302 to the client, to mimic 303 with 302 I think. 1) RP do GET to the backend 2) backend reply a 302 to another server 3) RP do a GET to this new location 4) RP return content and store it. Maybe i can use restart, like [1]. I have no idea where the new location (gived by 302) is, this could be a problem for a reverse proxy ! So I think it's not realy the varnish way to deal with "dynamic backend", but vcl is powerfull, and if I can define backend host and url on the fly, it could save my day ! Or if there is another way, that's fine to me ... Best regards, Augustin. [1] http://varnish.projects.linpro.no/ticket/411 From kristian at redpill-linpro.com Wed Apr 22 08:45:01 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 22 Apr 2009 10:45:01 +0200 Subject: Internal redirecition with dynamic backend (fetch 302 and return content) In-Reply-To: <49EEBC36.3070001@waw.com> References: <49EEBC36.3070001@waw.com> Message-ID: <20090422084500.GB5154@kjeks.linpro.no> On Wed, Apr 22, 2009 at 08:41:58AM +0200, Augustin Amann wrote: > 1) RP do GET to the backend > 2) backend reply a 302 to another server > 3) RP do a GET to this new location > 4) RP return content and store it. > > Maybe i can use restart, like [1]. > I have no idea where the new location (gived by 302) is, this could be a > problem for a reverse proxy ! > So I think it's not realy the varnish way to deal with "dynamic > backend", but vcl is powerfull, > and if I can define backend host and url on the fly, it could save my day ! In vcl_fetch: if (obj.status == 302) { set req.url = obj.http.Location; restart; } The normal routines in vcl_recv will have to determine which backend to use based on the new req.url. This will have to be pre-defined backends, so no random redirects I'm afraid. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From kristian at redpill-linpro.com Wed Apr 22 08:54:11 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 22 Apr 2009 10:54:11 +0200 Subject: multiple domains in one backend & 2 varnish servers In-Reply-To: <49EDA3F7.6070308@atela.net> References: <49EDA3F7.6070308@atela.net> Message-ID: <20090422085410.GC5154@kjeks.linpro.no> On Tue, Apr 21, 2009 at 12:46:15PM +0200, Andreas Kaiser - Atela CC wrote: > should be served by Europe server and others by the usa server I think I > need to configure each varnish install with a specific vcl config file. Why? They'll be working against the same backend server, and the Host-header differentiates between each domain. > What I don't have clear is if I need to use as backend only the IP of > the CMS server or multiple backends for every domain, also all domains > would have the same configuration related with cookies, compression, etc: You only need to define one backend if you only have one backend. > My experience with varnish is related with one webserver with one domain > that also has varnish installed... If I'm not wrong, varnish would > create a cache for each hostname/domain but I'm lost how varnish and > cmsserver should be configured for showing the specific content for each > domain if the servers are different ones... No configuration needed. Your webserver reads the Host: header which is part of HTTP 1.1, and Varnish, by default, uses this header combined with the URL as part of the cache hash. In essence: There's no difference in configuration from just having one domain and having multiple domains. There are some differences when you start talking purging, since the old purge.url doesn't supply a hostname, but with http-based PURGE in vcl_hit, this isn't an issue. And with purge.regexp or the new purge system, it's not an issue either. > The reason of this configuration is that the cms-server will be only > accessible from a private network and the 2 varnish server (which will > be used as public frontend servers) Would varnish be a good choice in > this situation? From the information you've given, it sounds like a reasonable choice, yes. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From augustin at waw.com Wed Apr 22 09:44:43 2009 From: augustin at waw.com (Augustin Amann) Date: Wed, 22 Apr 2009 11:44:43 +0200 Subject: Internal redirecition with dynamic backend (fetch 302 and return content) In-Reply-To: <20090422084500.GB5154@kjeks.linpro.no> References: <49EEBC36.3070001@waw.com> <20090422084500.GB5154@kjeks.linpro.no> Message-ID: <49EEE70B.6040808@waw.com> Kristian Lyngstol a ?crit : > On Wed, Apr 22, 2009 at 08:41:58AM +0200, Augustin Amann wrote: > >> 1) RP do GET to the backend >> 2) backend reply a 302 to another server >> 3) RP do a GET to this new location >> 4) RP return content and store it. >> >> Maybe i can use restart, like [1]. >> I have no idea where the new location (gived by 302) is, this could be a >> problem for a reverse proxy ! >> So I think it's not realy the varnish way to deal with "dynamic >> backend", but vcl is powerfull, >> and if I can define backend host and url on the fly, it could save my day ! >> > > In vcl_fetch: > > if (obj.status == 302) { > set req.url = obj.http.Location; > restart; > } > > The normal routines in vcl_recv will have to determine which backend to use > based on the new req.url. This will have to be pre-defined backends, so no > random redirects I'm afraid. > > Hi Kristian, Thank you for your reponse. That's what I supected ... For my needs, I combined squid to varnish, like this: # Varnish VCL: backend squid { .host = "127.0.0.1"; .port = "3128"; } sub vcl_fetch { if (obj.status == 302 || obj.status == 301) { #set req.backend = squid; set req.url = obj.http.Location; restart; } } # squid.conf http_port 3128 transparent It works like a charm ... :) Best Regards, Augustin. From s.kain at eraffe-media.de Fri Apr 24 11:28:36 2009 From: s.kain at eraffe-media.de (Kain Sascha) Date: Fri, 24 Apr 2009 13:28:36 +0200 Subject: best varnishd config for static files Message-ID: <200904241328.36519.s.kain@eraffe-media.de> Hi, we run a big picture portal. We used to cache our files with squid3, but now we want to try varnishd to deliver our static files (jpg css and js). we have 2 varnish servers, caching files from one big static server. The static server doesnt have the fastest hdds. The varnishd Servers have 24GB of ram each, and they should use around 20GB of them for storing files using LRU, cause the .css and .js should always be delivered from RAM. i compiled and started varnishd like this: /usr/local/sbin/varnishd -a :80 -b 89.250.xx.xx:80 -s malloc,15360M I didnt use a config file, because i think i dont need it for static files only. is this correct? The Loadbalancing for the two varnish-servers does our software do. Is this a good solution? Sometimes im getting a 503 error when the static server has high load. Also sometimes i am waiting a long time for some files that has to be on RAM allready. best Sascha From kristian at redpill-linpro.com Fri Apr 24 12:21:52 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Fri, 24 Apr 2009 14:21:52 +0200 Subject: best varnishd config for static files In-Reply-To: <200904241328.36519.s.kain@eraffe-media.de> References: <200904241328.36519.s.kain@eraffe-media.de> Message-ID: <20090424122151.GB8200@kjeks.linpro.no> On Fri, Apr 24, 2009 at 01:28:36PM +0200, Kain Sascha wrote: > /usr/local/sbin/varnishd -a :80 -b 89.250.xx.xx:80 -s malloc,15360M You may want to supply a -w. The default is really whimpy which could create problems during spikes of traffic. Try something like -w 400,800. Varnishstat n_wrk_limited and n_wrk_drop are values you don't want to increase after the initial startup. n_wrk_limited can increase a little every once in a while, but it should only happen on significant load spikes. > I didnt use a config file, because i think i dont need it for static files only. > is this correct? The default TTL is 120s if I am not much mistaken. You probably want it to be something closer to 500h or some other really long period of time. This has to be done in VCL if your backends don't deliver s-maxage/max-age/Expires: headers. For instance: vcl_fetch: set obj.ttl = 500h; > Is this a good solution? Yes, though you should make sure varnish caches as much as you want it to and watch the hitrate in varnishstat and check what is actually going to the backends most frequently with varnishtop -i TxURL. With varnishtop -i TxURL, you really shouldn't see many values above 1-3 for static content. > Sometimes im getting a 503 error when the static server has high load. > Also sometimes i am waiting a long time for some files that has to be on RAM > allready. The X-Varnish HTTP header can tell you if it is a hit or not. It gives you one or two numbers: One is a transaction id, and the last identifies the original transaction that fetched the object. One number: MISS; two numbers: HIT. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From s.kain at eraffe-media.de Fri Apr 24 13:28:22 2009 From: s.kain at eraffe-media.de (Kain Sascha) Date: Fri, 24 Apr 2009 15:28:22 +0200 Subject: best varnishd config for static files In-Reply-To: <20090424122151.GB8200@kjeks.linpro.no> References: <200904241328.36519.s.kain@eraffe-media.de> <20090424122151.GB8200@kjeks.linpro.no> Message-ID: <200904241528.22815.s.kain@eraffe-media.de> Thanks a lot for the great Hints. First of all, varnishtop -i TxURL hangs on my system, it shows 3-5 Entries, then freez and uses nearly 100%cpu on one of 8. Have to kill the pid. varnishtop with no args works. System: 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 x86_64 GNU/Linux varnish: varnishd (varnish-2.0.4) Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS Here is our Answer-Header, so we provide the max-age. Cache-Control max-age=2592000 Vary Accept-Encoding Expires Wed, 29 Apr 2009 21:14:47 GMT Content-Type image/jpeg Etag "857407780" Last-Modified Mon, 30 Mar 2009 20:35:52 GMT Server lighttpd/1.4.20 Content-Length 1392 Date Fri, 24 Apr 2009 13:19:59 GMT X-Varnish 623951818 451344757 Age 2131515 Via 1.1 varnish Connection keep-alive Thanks again for your hints best Am Freitag, 24. April 2009 14:21:52 schrieben Sie: > On Fri, Apr 24, 2009 at 01:28:36PM +0200, Kain Sascha wrote: > > /usr/local/sbin/varnishd -a :80 -b 89.250.xx.xx:80 -s malloc,15360M > > You may want to supply a -w. The default is really whimpy which could > create problems during spikes of traffic. Try something like -w 400,800. > Varnishstat n_wrk_limited and n_wrk_drop are values you don't want to > increase after the initial startup. n_wrk_limited can increase a little > every once in a while, but it should only happen on significant load > spikes. > > > I didnt use a config file, because i think i dont need it for static > > files only. is this correct? > > The default TTL is 120s if I am not much mistaken. You probably want it to > be something closer to 500h or some other really long period of time. This > has to be done in VCL if your backends don't deliver > s-maxage/max-age/Expires: headers. > > For instance: > > vcl_fetch: > set obj.ttl = 500h; > > > Is this a good solution? > > Yes, though you should make sure varnish caches as much as you want it to > and watch the hitrate in varnishstat and check what is actually going to > the backends most frequently with varnishtop -i TxURL. With varnishtop -i > TxURL, you really shouldn't see many values above 1-3 for static content. > > > Sometimes im getting a 503 error when the static server has high load. > > Also sometimes i am waiting a long time for some files that has to be on > > RAM allready. > > The X-Varnish HTTP header can tell you if it is a hit or not. It gives you > one or two numbers: One is a transaction id, and the last identifies the > original transaction that fetched the object. One number: MISS; two > numbers: HIT. From brian.pan at light-mc.com Fri Apr 24 14:21:24 2009 From: brian.pan at light-mc.com (Brian Pan) Date: Fri, 24 Apr 2009 09:21:24 -0500 Subject: Varnish Cache and Page Load Time Message-ID: <6616347254C43B4F98EB5C7C421BCAA174E97F@mail-33ps.atlarge.net> I have a two websites on a server running behind varnish. One of the websites is a Wordpress blog (with php, mysql etc..). The other is a simple html (without php or mysql) website. Using Firefox's Yslow add-in to check page load times, I have found that the simple html page is served in 0.5 seconds, whereas it takes 1.4-1.7 seconds to load the Wordpress site. \ With the varnish cache working, shouldn't the time to load both sites be approximately the same? Or am I missing something? Any thoughts or ideas about how I could decrease the page load time of the Wordpress site? Thanks for your help, Brian Pan brian.pan at light-mc.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfoutrel at bcstechno.com Fri Apr 24 15:11:57 2009 From: sfoutrel at bcstechno.com (=?iso-8859-1?Q?S=E9bastien_FOUTREL?=) Date: Fri, 24 Apr 2009 17:11:57 +0200 Subject: Varnish Cache and Page Load Time In-Reply-To: <6616347254C43B4F98EB5C7C421BCAA174E97F@mail-33ps.atlarge.net> References: <6616347254C43B4F98EB5C7C421BCAA174E97F@mail-33ps.atlarge.net> Message-ID: As you might know, Varnish needs some tuning to cache efficiently your website. In your case, do your cache lasts long enough to stay between 2 visits from you ? Did you made some tuning to exclude some pages, cookies from cache ? By default varnish use cookies as part of the hash so if each visitors has a unique cookie then varnish will cache one object for each visitor... Not so efficient isn't it ? Regards De : varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] De la part de Brian Pan Envoy? : vendredi 24 avril 2009 16:21 ? : varnish-misc at projects.linpro.no Objet : Varnish Cache and Page Load Time I have a two websites on a server running behind varnish. One of the websites is a Wordpress blog (with php, mysql etc..). The other is a simple html (without php or mysql) website. Using Firefox's Yslow add-in to check page load times, I have found that the simple html page is served in 0.5 seconds, whereas it takes 1.4-1.7 seconds to load the Wordpress site. \ With the varnish cache working, shouldn't the time to load both sites be approximately the same? Or am I missing something? Any thoughts or ideas about how I could decrease the page load time of the Wordpress site? Thanks for your help, Brian Pan brian.pan at light-mc.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at redpill-linpro.com Mon Apr 27 07:16:08 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Mon, 27 Apr 2009 09:16:08 +0200 Subject: best varnishd config for static files In-Reply-To: <200904241528.22815.s.kain@eraffe-media.de> References: <200904241328.36519.s.kain@eraffe-media.de> <20090424122151.GB8200@kjeks.linpro.no> <200904241528.22815.s.kain@eraffe-media.de> Message-ID: <20090427071607.GA6369@kjeks.linpro.no> On Fri, Apr 24, 2009 at 03:28:22PM +0200, Kain Sascha wrote: > Thanks a lot for the great Hints. > > First of all, varnishtop -i TxURL hangs on my system, it shows 3-5 Entries, > then freez and uses nearly 100%cpu on one of 8. Have to kill the pid. > varnishtop with no args works. Ok, that's not good... I've seen varnishtop hang like that before, but not with CPU usage. I've yet to be able to properly debug this as I can't reproduce it myself... What GNU+Linux distribution are you using? If your miss-rate is low enough, you may be able to get some sense out of varnishlog -b instead of varnishtop, but varnishtop really is the tool you want... At any rate, with static content, varnishstat should show you a hitrate in the 0.99-range (99% hit-rate, for static content, I'd say 0.999 is well within reach), unless you have a huge dataset with very even distribution of hits. > Here is our Answer-Header, so we provide the max-age. > > Cache-Control max-age=2592000 > Vary Accept-Encoding > Expires Wed, 29 Apr 2009 21:14:47 GMT > Content-Type image/jpeg > Etag "857407780" > Last-Modified Mon, 30 Mar 2009 20:35:52 GMT > Server lighttpd/1.4.20 > Content-Length 1392 > Date Fri, 24 Apr 2009 13:19:59 GMT > X-Varnish 623951818 451344757 > Age 2131515 > Via 1.1 varnish > Connection keep-alive Looks good. Age: confirms that this object is 592 hours old too. On long-duration caching like this, keep in mind that these headers affect client-cache too. If that doesn't bother you, you can leave them as is, but if you want varnish to cache for, say 720 hours, but clients should check back every 24 hours (in case you force purges), you can modify the max-age header before sending it out. > > > Sometimes im getting a 503 error when the static server has high load. > > > Also sometimes i am waiting a long time for some files that has to be on > > > RAM allready. Can you check if the X-Varnish header has one or two numbers on these errors/slow pages? The Age: header can tell you some of the same information, but on slow gets, it's not uncommon to see a positive age on cache misses. -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From varnish-misc at projects.linpro.no Mon Apr 27 22:37:32 2009 From: varnish-misc at projects.linpro.no (VIAGRA Inc.) Date: Tue, 28 Apr 2009 06:37:32 +0800 Subject: Pharmacy Online Sale 85% OFF! Message-ID: <20090428143732.2694.qmail@20081214-1330> An HTML attachment was scrubbed... URL: -------------- next part -------------- New from WebMD: Dear varnish-misc at projects.linpro.no!. Sign-up today! You are subscribed as varnish-misc at projects.linpro.no. View and manage your WebMD newsletter preferences. Subscribe to more newsletters. Change/update your email address. WebMD Privacy Policy WebMD Office of Privacy 1175 Peachtree Street, Suite 2400, Atlanta, GA 30361 ? 2009 WebMD, LLC. All rights reserved. From itlj at gyldendal.dk Wed Apr 29 13:06:03 2009 From: itlj at gyldendal.dk (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Wed, 29 Apr 2009 15:06:03 +0200 Subject: best varnishd config for static files In-Reply-To: <20090427071607.GA6369@kjeks.linpro.no> References: <200904241328.36519.s.kain@eraffe-media.de> <20090424122151.GB8200@kjeks.linpro.no> <200904241528.22815.s.kain@eraffe-media.de> <20090427071607.GA6369@kjeks.linpro.no> Message-ID: >> First of all, varnishtop -i TxURL hangs on my system, it shows 3-5 >> Entries, >> then freez and uses nearly 100%cpu on one of 8. Have to kill the pid. >> varnishtop with no args works. > > Ok, that's not good... I've seen varnishtop hang like that before, > but not > with CPU usage. I've yet to be able to properly debug this as I can't > reproduce it myself... > > What GNU+Linux distribution are you using? Same here, hangs at 100% cpu with "-i TxURL". We're running Debian Etch. -- Lars From nick at loman.net Wed Apr 29 16:18:44 2009 From: nick at loman.net (Nick Loman) Date: Wed, 29 Apr 2009 17:18:44 +0100 Subject: Theoretical connections/second limit using Varnish Message-ID: <49F87DE4.3040300@loman.net> Hi there, Has anyone come to a satisfactory solution to the issue of running out of local port numbers when Varnish makes a connection to the backend server? Under Linux, my understanding is the number of available port numbers can be increased to a maximum of 64511 by setting /proc/sys/net/ipv4/ip_local_port_range to 1024 - 65535. Assuming sockets are left in TIME_WAIT for 60 seconds that would limit the number of backend connections Varnish can make to 64511/minute or 1075/second. It seems to be acceptable to reduce TIME_WAIT to perhaps 30 seconds, doubling that to 2150/second. A solution often proposed is to use time wait recycling, or tw_reuse, but my understanding is that under Linux these settings are global and therefore can break NAT for user connections (all connections are conntracked and DNATted on our setup). 2150 requests/second is not an impossible number to achieve, especially with backend KeepAlive off. Has Varnish got a solution to this problem which does not involve time-wait recycling? One thing I've thought of is perhaps SO_REUSEADDR is used or could be used when Varnish makes connections to the backend? Regards, Nick. From phk at phk.freebsd.dk Wed Apr 29 16:22:34 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 29 Apr 2009 16:22:34 +0000 Subject: Theoretical connections/second limit using Varnish In-Reply-To: Your message of "Wed, 29 Apr 2009 17:18:44 +0100." <49F87DE4.3040300@loman.net> Message-ID: <15661.1241022154@critter.freebsd.dk> In message <49F87DE4.3040300 at loman.net>, Nick Loman writes: >Has Varnish got a solution to this problem which does not involve >time-wait recycling? One thing I've thought of is perhaps SO_REUSEADDR >is used or could be used when Varnish makes connections to the backend? Varnish tries as hard as reasonable to reuse backend connections, so you should be able to get multiple requests per backend connection. If this is not the case for you, you should find out why backend connections are not reused. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From michael at dynamine.net Wed Apr 29 16:27:49 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Wed, 29 Apr 2009 09:27:49 -0700 Subject: Theoretical connections/second limit using Varnish In-Reply-To: <15661.1241022154@critter.freebsd.dk> References: <15661.1241022154@critter.freebsd.dk> Message-ID: On Apr 29, 2009, at 9:22 AM, Poul-Henning Kamp wrote: > In message <49F87DE4.3040300 at loman.net>, Nick Loman writes: > >> Has Varnish got a solution to this problem which does not involve >> time-wait recycling? One thing I've thought of is perhaps >> SO_REUSEADDR >> is used or could be used when Varnish makes connections to the >> backend? > > Varnish tries as hard as reasonable to reuse backend connections, > so you should be able to get multiple requests per backend connection. > > If this is not the case for you, you should find out why backend > connections > are not reused. The OP said he turned backend Keep-Alive off. That's his problem. --Michael From loco at d0pefish.de Wed Apr 29 16:20:44 2009 From: loco at d0pefish.de (Peter Hinse) Date: Wed, 29 Apr 2009 18:20:44 +0200 Subject: Different TTLs dependent upon req.http.host? Message-ID: <49F87E5C.5010709@d0pefish.de> Hi all, I have a varnish config looking roughly like this: backend be1 { .host = "10.10.10.10"; .port = "80"; } sub vcl_recv { elseif (req.http.host ~ "^x.example.com") { set req.http.Host = "host.example.com"; set req.backend = be1; } elseif (req.http.host ~ "^xi.example.com") { set req.http.Host = "host.example.com"; set req.backend = be1; } } The backend server delivers dynamically rendered images I need to cache. If the original requests comes in for Host x.example.com, the TTL can be 12h, if the request is for xi.example.com, the TTL is 15m. Since I cannot use "req.http.host" in vcl_fetch and the req.url look the same, how can I distinguish the two cases and set the obj.ttl differently? Regards, Peter From nick at loman.net Wed Apr 29 16:30:04 2009 From: nick at loman.net (Nick Loman) Date: Wed, 29 Apr 2009 17:30:04 +0100 Subject: Theoretical connections/second limit using Varnish In-Reply-To: References: <15661.1241022154@critter.freebsd.dk> Message-ID: <49F8808C.7090300@loman.net> Michael S. Fischer wrote: > On Apr 29, 2009, at 9:22 AM, Poul-Henning Kamp wrote: > >> In message <49F87DE4.3040300 at loman.net>, Nick Loman writes: >> >>> Has Varnish got a solution to this problem which does not involve >>> time-wait recycling? One thing I've thought of is perhaps SO_REUSEADDR >>> is used or could be used when Varnish makes connections to the backend? >> >> Varnish tries as hard as reasonable to reuse backend connections, >> so you should be able to get multiple requests per backend connection. >> >> If this is not the case for you, you should find out why backend >> connections >> are not reused. Hi Poul-Henning, Michael, I've configured Apache with KeepAlive off, so I expect the TCP connection to be closed after each request and Varnish won't be able to use it. I've done that for a specific reason relating to backend PHP processes. Is that what you mean? I typically have thousands of connections in TIME_WAIT mode as a result, which is expected, but I wonder what the solution could be if I ever hit more connections than local ports available. If I turned Keep Alive on with Apache, could Varnish multiplex different requests from different TCP sockets to the same backend connection? > The OP said he turned backend Keep-Alive off. That's his problem. Right, exactly, but I want it this way assuming that leaving it on means that each user connection translates to one backend connection to Apache - not desirable - but perhaps this is not how Varnish operates? Cheers, Nick. From michael at dynamine.net Wed Apr 29 16:36:00 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Wed, 29 Apr 2009 09:36:00 -0700 Subject: Theoretical connections/second limit using Varnish In-Reply-To: <49F8808C.7090300@loman.net> References: <15661.1241022154@critter.freebsd.dk> <49F8808C.7090300@loman.net> Message-ID: <1CF4804A-3746-44CA-A555-CB02E87EA239@dynamine.net> On Apr 29, 2009, at 9:30 AM, Nick Loman wrote: > Michael S. Fischer wrote: >> On Apr 29, 2009, at 9:22 AM, Poul-Henning Kamp wrote: >>> In message <49F87DE4.3040300 at loman.net>, Nick Loman writes: >>> >>>> Has Varnish got a solution to this problem which does not involve >>>> time-wait recycling? One thing I've thought of is perhaps >>>> SO_REUSEADDR >>>> is used or could be used when Varnish makes connections to the >>>> backend? >>> >>> Varnish tries as hard as reasonable to reuse backend connections, >>> so you should be able to get multiple requests per backend >>> connection. >>> >>> If this is not the case for you, you should find out why backend >>> connections >>> are not reused. > > Hi Poul-Henning, Michael, > > I've configured Apache with KeepAlive off, so I expect the TCP > connection to be closed after each request and Varnish won't be able > to use it. > > I've done that for a specific reason relating to backend PHP > processes. I don't dispute your reasoning; my employer does this as well. KeepAlive with Apache/PHP can be a recipe for resource starvation on your origin servers. > I typically have thousands of connections in TIME_WAIT mode as a > result, which is expected, but I wonder what the solution could be > if I ever hit more connections than local ports available. I think SO_REUSEADDR is the answer - I'm somewhat surprised that Varnish doesn't set it by default for the backend connections. --Michael From jeff at funnyordie.com Wed Apr 29 22:38:41 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Wed, 29 Apr 2009 15:38:41 -0700 Subject: Handling graced objects between varnish servers. Message-ID: <6D6E4F5A-B9D6-41EB-9B08-27AFEC5D49DB@funnyordie.com> I'm experimenting with a large varnish cache as a central master to multiple front-end caches. The master having a long object grace and a slower LRU interval. How long can I cache objects and keep them available as graced objects? If a front end varnish cache receives an object from a downstream varnish cache that was served as a graced object will the front end varnish cache declare it invalid? If so is there a way to bypass that invalidation? --Jeff jeff at funnyordie.com From itlj at gyldendal.dk Thu Apr 30 06:59:00 2009 From: itlj at gyldendal.dk (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 30 Apr 2009 08:59:00 +0200 Subject: Putting the site in maintenance mode? Message-ID: <60B9E91D-8D9D-4056-8CC9-1240D7F38B6F@gyldendal.dk> Hi, New to varnish so forgive me if this has been answered before. I tried searching but mainly got instructions on how to maintain a boat :-) I have defined a vcl_error stanza in my vcl that puts a "sorry, we're currently maintaining the site" sign up, when all backends are sick. I had a chance to test it out today because I had to disable all backends to do some maintenance. The sign didn't go up and I got some half-baked version of my site in the browser. I restarted varnish and of course immediately invalidated all backends and put the sign up. Is there any way to put the whole site in maintenance mode in a nice way? Like telling varnish "don't serve any content, just the error page"? -- Lars J?rgensen Gyldendal IT tlf. 33 75 57 95 From phk at phk.freebsd.dk Thu Apr 30 07:14:42 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 30 Apr 2009 07:14:42 +0000 Subject: Putting the site in maintenance mode? In-Reply-To: Your message of "Thu, 30 Apr 2009 08:59:00 +0200." <60B9E91D-8D9D-4056-8CC9-1240D7F38B6F@gyldendal.dk> Message-ID: <22872.1241075682@critter.freebsd.dk> In message <60B9E91D-8D9D-4056-8CC9-1240D7F38B6F at gyldendal.dk>, =?ISO-8859-1?Q? Lars_J=F8rgensen?= writes: >Is there any way to put the whole site in maintenance mode in a nice = >way? Like telling varnish "don't serve any content, just the error = >page"? Varnish allows you to have multiple VCL programs loaded, but only one of them is active at any one time. What you could do, is therefore to have VCL for maintenance, something along the lines of; sub vcl_recv { error 503 } sub vcl_error { ... } Load it with the cli commands "vcl.load maint " When you want to go into maintenance mode, you switch to that VCL, using "vcl.use maint" when you want to get back to production, you switch back to your normal VCL with "vcl.use boot" -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Apr 30 07:16:35 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 30 Apr 2009 07:16:35 +0000 Subject: Handling graced objects between varnish servers. In-Reply-To: Your message of "Wed, 29 Apr 2009 15:38:41 MST." <6D6E4F5A-B9D6-41EB-9B08-27AFEC5D49DB@funnyordie.com> Message-ID: <22886.1241075795@critter.freebsd.dk> In message <6D6E4F5A-B9D6-41EB-9B08-27AFEC5D49DB at funnyordie.com>, Jeff Anderson writes: >How long can I cache objects and keep them >available as graced objects? As long as you want. >If a front end varnish cache receives an >object from a downstream varnish cache that was served as a graced >object will the front end varnish cache declare it invalid? If so is >there a way to bypass that invalidation? There is no indication passed along that an object is graced, so the downstream varnish sees it just like any other object and will interpret its validity based on the headers. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nick at loman.net Thu Apr 30 15:10:15 2009 From: nick at loman.net (Nick Loman) Date: Thu, 30 Apr 2009 16:10:15 +0100 Subject: Theoretical connections/second limit using Varnish In-Reply-To: <1CF4804A-3746-44CA-A555-CB02E87EA239@dynamine.net> References: <15661.1241022154@critter.freebsd.dk> <49F8808C.7090300@loman.net> <1CF4804A-3746-44CA-A555-CB02E87EA239@dynamine.net> Message-ID: <49F9BF57.4020909@loman.net> Michael S. Fischer wrote: >> I've done that for a specific reason relating to backend PHP processes. > > I don't dispute your reasoning; my employer does this as well. > KeepAlive with Apache/PHP can be a recipe for resource starvation on > your origin servers. Hi Michael, Precisely, we only have perhaps 50 PHP children serving requests, so if these are kept open to serve idle keep-alive connections, that severely limits the numbers of dynamic page requests we can serve. >> I typically have thousands of connections in TIME_WAIT mode as a >> result, which is expected, but I wonder what the solution could be if >> I ever hit more connections than local ports available. > > I think SO_REUSEADDR is the answer - I'm somewhat surprised that Varnish > doesn't set it by default for the backend connections. I've had a browse through the Varnish 2.0.4 source now and have convinced myself that SO_REUSEADDR isn't set. Can you think of any specific gotchas I might encounter if I added setsockopt() before the connect() when getting a backend connection? Cheers, Nick.