From rajat.toshniwal at tekmindz.com Wed Jan 2 05:43:04 2013 From: rajat.toshniwal at tekmindz.com (Rajat) Date: Wed, 02 Jan 2013 11:13:04 +0530 Subject: Want to know about some of the varnish features Message-ID: <30c8-50e3c900-3-381da500@144782867> I am new to varnish and running a heavy education portal , so we want to deploy varnish as cache engine. But before using it, I want to be cleared about these point, Please help me in understanding these points a>Is it possible to expire a particular file/content from cache so that if any changes are made to a file/content it can be reloaded into cache from the app server. I have read about ban (varnish 3.x), but in that case I need to restart the engine to reload the configuration, but that will purge the whole cache. which I do not want b>Can we instruct varnish not to flush the cache on reboot. c> What are the rule preferences e.g. if there are two rules applicable for a particular request which rule would get the preference. Regards Rajat Toshniwal ---------------------------------------------------------------------------------- Disclaimer: The information contained in this communication is confidential, private, proprietary, or otherwise privileged and is intended only for the use of the addressee. Unauthorized use, disclosure, distribution or copying is strictly prohibited and may be unlawful. If you have received this communication in error, please delete this message and notify the sender immediately - Samin TekMindz India Pvt. Ltd. ---------------------------------------------------------------------------------- From nick.tailor at gmail.com Wed Jan 2 06:46:51 2013 From: nick.tailor at gmail.com (nick tailor) Date: Tue, 1 Jan 2013 22:46:51 -0800 Subject: Want to know about some of the varnish features In-Reply-To: <30c8-50e3c900-3-381da500@144782867> References: <30c8-50e3c900-3-381da500@144782867> Message-ID: Hey Raj, How I did it was, as long as my backend is up, the cache updated every 5 minutes from, and if my backend was down the content it served was up to 12 hours. You can specify the ttl for objects within VCL. Cheers Nick tailor On Tue, Jan 1, 2013 at 9:43 PM, Rajat wrote: > I am new to varnish and running a heavy education portal , so we want to > deploy varnish as cache engine. > But before using it, I want to be cleared about these point, Please help > me in understanding these points > a>Is it possible to expire a particular file/content from cache so that if > any changes are made to a file/content it can be reloaded into cache from > the app server. > I have read about ban (varnish 3.x), but in that case I need to restart > the engine to reload the configuration, but that will purge the whole > cache. which I do not want > b>Can we instruct varnish not to flush the cache on reboot. > c> What are the rule preferences e.g. if there are two rules applicable > for a particular request which rule would get the preference. > > > Regards > Rajat Toshniwal > > ---------------------------------------------------------------------------------- > Disclaimer: The information contained in this communication is > confidential, private, proprietary, or otherwise privileged and is intended > only for the use of the addressee. Unauthorized use, disclosure, > distribution or copying is strictly prohibited and may be unlawful. If you > have received this communication in error, please delete this message and > notify the sender immediately - Samin TekMindz India Pvt. Ltd. > > ---------------------------------------------------------------------------------- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From JSmall at daraco.com.au Thu Jan 3 02:19:15 2013 From: JSmall at daraco.com.au (Joshua Small) Date: Thu, 3 Jan 2013 02:19:15 +0000 Subject: Varnish on a pi Message-ID: <6223D6171835FF46A542643BF7A54A203C0BC5D5@DARAEX01.business.ad> Hi, I've trying to install Varnish on a Raspberry pi. This has been discussed on their own forums with a general view of "it simply doesn't work", so I set about finding why. Some info: pi at raspberrypi ~ $ uname -a Linux raspberrypi 3.2.27+ #4 PREEMPT Fri Dec 28 16:14:38 EST 2012 armv6l GNU/Linux pi at raspberrypi ~ $ free total used free shared buffers cached Mem: 497132 284352 212780 0 63616 180780 -/+ buffers/cache: 39956 457176 Swap: 102396 0 102396 Image is the Debian port "raspbian", updated from repositories as of today. I initially assumed the repositories were the issue and obtained varnish-3.0.3 from source. The install is fairly default, documented here: sudo apt-get install automake libtool sh autogen.sh sh configure make sudo make install sudo vi /usr/local/etc/varnish/default.vcl Setup backend servers.. sudo ldconfig -n /usr/local/lib/ Varnish however, appears to crash and burn: pi at raspberrypi ~ $ sudo /usr/local/sbin/varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,16M -a 172.31.1.50:80 -d Platform: Linux,3.2.27+,armv6l,-sfile,-smalloc,-hcritbit 200 233 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,3.2.27+,armv6l,-sfile,-smalloc,-hcritbit Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. start child (19755) Started Pushing vcls failed: CLI communication error (hdr) Stopping Child 200 0 Child (19755) died signal=11 Child cleanup complete quit 500 22 Closing CLI connection In order to follow the recommended process here https://www.varnish-cache.org/trac/wiki/DebuggingVarnish, I reconfigured with --enable-debugging-symbols --enable-diagnostics. I went back and forth between a "default" install and "diagnostic" install three different times to confirm this, because I appreciate it doesn't seem plausible. Regrettably, the situation experienced is that I have a fix: Compile in diagnostic mode. With the above configure options (I ran make clean between each new build) the Varnish service appeared to operate fine. Whilst I could just leave diagnostics on, I'm assuming there's load associated with this. I've pasted below an strace of the crashed system, but I'm just not sure how else I can debug in this situation. Is there anything else that the diagnostics mode does that could possibly get involved here? [pid 18164] clock_gettime(CLOCK_REALTIME, [pid 18166] <... nanosleep resumed> NULL) = 0 [pid 18166] mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x45e00000 [pid 18166] mprotect(0x45e00000, 4096, PROT_NONE) = 0 [pid 18166] clone(Process 18173 attached (waiting for parent) Process 18173 resumed (parent 18166 ready) child_stack=0x45e0efa8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x45e0f4d8, tls=0x45e0f930, child_tidptr=0x45e0f4d8) = 18173 [pid 18173] set_robust_list(0x45e0f4e0, 0xc) = 0 [pid 18173] clock_gettime(CLOCK_REALTIME, {1357176395, 646464764}) = 0 [pid 18173] futex(0x45e0ebd4, FUTEX_WAIT_PRIVATE, 1, NULL [pid 18166] nanosleep({0, 2000000}, [pid 18164] <... clock_gettime resumed> {1357176395, 642222934}) = 0 [pid 18164] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 18164 detached [pid 18173] +++ killed by SIGSEGV +++ Below you can see the syslog that covers me starting varnish twice, once with a default configure, once with the diagnostics enabled. Jan 3 12:54:27 raspberrypi varnishd[26807]: Platform: Linux,3.2.27+,armv6l,-smalloc,-smalloc,-hcritbit Jan 3 12:54:27 raspberrypi varnishd[26807]: child (26808) Started Jan 3 12:54:27 raspberrypi varnishd[26807]: Child (26808) died signal=11 Jan 3 12:54:27 raspberrypi varnishd[26807]: Child (-1) said Child starts Jan 3 13:05:04 raspberrypi varnishd[31536]: Platform: Linux,3.2.27+,armv6l,-smalloc,-smalloc,-hcritbit Jan 3 13:05:04 raspberrypi varnishd[31536]: child (31537) Started Jan 3 13:05:04 raspberrypi varnishd[31536]: Child (31537) said Child starts Finally, some info from the config log. pi at raspberrypi ~/varnish-3.0.3 $ grep -i warn config.log cc1: warning: command line option '-fno-rtti' is valid for C++/ObjC++ but not for C [enabled by default] configure:13219: WARNING: xsltproc not found - not building documentation configure:13279: WARNING: rst2man not found - not building man pages configure:13342: WARNING: rst2html not found - not building changelog conftest.c:40:6: warning: conflicting types for built-in function 'cos' [enabled by default] configure:14290: WARNING: libedit not found, disabling libedit support conftest.c:72:6: warning: conflicting types for built-in function 'vprintf' [enabled by default] configure:15220: WARNING: won't look for sendfile() on armv6l-unknown-linux-gnueabihf conftest.c:68:3: warning: left shift count >= width of type [enabled by default] conftest.c:68:3: warning: left shift count >= width of type [enabled by default] conftest.c:69:10: warning: left shift count >= width of type [enabled by default] conftest.c:69:10: warning: left shift count >= width of type [enabled by default] Sorry for the long post - but hopefully it gives us something to go on. Any information would be appreciated. -Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From JSmall at daraco.com.au Thu Jan 3 06:17:08 2013 From: JSmall at daraco.com.au (Joshua Small) Date: Thu, 3 Jan 2013 06:17:08 +0000 Subject: FW: Varnish on a pi In-Reply-To: <6223D6171835FF46A542643BF7A54A203C0BC5D5@DARAEX01.business.ad> References: <6223D6171835FF46A542643BF7A54A203C0BC5D5@DARAEX01.business.ad> Message-ID: <6223D6171835FF46A542643BF7A54A203C0BC7E2@DARAEX01.business.ad> Hi, Just to add to below, I've replicated this on the latest trunk. I've also confirmed that the flag that the -enable-debugging-symbols doesn't matter - it's the -enable-diagnostics that makes the difference between a working installation and segfaulting installation. I can see that this appends a CFLAG -DDIAGNOSTICS to the build - but can't for the life of me see what this actually does to the operation of the service. If anyone can shed some light on where exactly this flag is used (I've done a lot of grepping to not much avail) I'd be keen to investigate it. From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Joshua Small Sent: Thursday, 3 January 2013 1:19 PM To: varnish-misc at varnish-cache.org Subject: Varnish on a pi Hi, I've trying to install Varnish on a Raspberry pi. This has been discussed on their own forums with a general view of "it simply doesn't work", so I set about finding why. Some info: pi at raspberrypi ~ $ uname -a Linux raspberrypi 3.2.27+ #4 PREEMPT Fri Dec 28 16:14:38 EST 2012 armv6l GNU/Linux pi at raspberrypi ~ $ free total used free shared buffers cached Mem: 497132 284352 212780 0 63616 180780 -/+ buffers/cache: 39956 457176 Swap: 102396 0 102396 Image is the Debian port "raspbian", updated from repositories as of today. I initially assumed the repositories were the issue and obtained varnish-3.0.3 from source. The install is fairly default, documented here: sudo apt-get install automake libtool sh autogen.sh sh configure make sudo make install sudo vi /usr/local/etc/varnish/default.vcl Setup backend servers.. sudo ldconfig -n /usr/local/lib/ Varnish however, appears to crash and burn: pi at raspberrypi ~ $ sudo /usr/local/sbin/varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,16M -a 172.31.1.50:80 -d Platform: Linux,3.2.27+,armv6l,-sfile,-smalloc,-hcritbit 200 233 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,3.2.27+,armv6l,-sfile,-smalloc,-hcritbit Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. start child (19755) Started Pushing vcls failed: CLI communication error (hdr) Stopping Child 200 0 Child (19755) died signal=11 Child cleanup complete quit 500 22 Closing CLI connection In order to follow the recommended process here https://www.varnish-cache.org/trac/wiki/DebuggingVarnish, I reconfigured with --enable-debugging-symbols --enable-diagnostics. I went back and forth between a "default" install and "diagnostic" install three different times to confirm this, because I appreciate it doesn't seem plausible. Regrettably, the situation experienced is that I have a fix: Compile in diagnostic mode. With the above configure options (I ran make clean between each new build) the Varnish service appeared to operate fine. Whilst I could just leave diagnostics on, I'm assuming there's load associated with this. I've pasted below an strace of the crashed system, but I'm just not sure how else I can debug in this situation. Is there anything else that the diagnostics mode does that could possibly get involved here? [pid 18164] clock_gettime(CLOCK_REALTIME, [pid 18166] <... nanosleep resumed> NULL) = 0 [pid 18166] mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x45e00000 [pid 18166] mprotect(0x45e00000, 4096, PROT_NONE) = 0 [pid 18166] clone(Process 18173 attached (waiting for parent) Process 18173 resumed (parent 18166 ready) child_stack=0x45e0efa8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x45e0f4d8, tls=0x45e0f930, child_tidptr=0x45e0f4d8) = 18173 [pid 18173] set_robust_list(0x45e0f4e0, 0xc) = 0 [pid 18173] clock_gettime(CLOCK_REALTIME, {1357176395, 646464764}) = 0 [pid 18173] futex(0x45e0ebd4, FUTEX_WAIT_PRIVATE, 1, NULL [pid 18166] nanosleep({0, 2000000}, [pid 18164] <... clock_gettime resumed> {1357176395, 642222934}) = 0 [pid 18164] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 18164 detached [pid 18173] +++ killed by SIGSEGV +++ Below you can see the syslog that covers me starting varnish twice, once with a default configure, once with the diagnostics enabled. Jan 3 12:54:27 raspberrypi varnishd[26807]: Platform: Linux,3.2.27+,armv6l,-smalloc,-smalloc,-hcritbit Jan 3 12:54:27 raspberrypi varnishd[26807]: child (26808) Started Jan 3 12:54:27 raspberrypi varnishd[26807]: Child (26808) died signal=11 Jan 3 12:54:27 raspberrypi varnishd[26807]: Child (-1) said Child starts Jan 3 13:05:04 raspberrypi varnishd[31536]: Platform: Linux,3.2.27+,armv6l,-smalloc,-smalloc,-hcritbit Jan 3 13:05:04 raspberrypi varnishd[31536]: child (31537) Started Jan 3 13:05:04 raspberrypi varnishd[31536]: Child (31537) said Child starts Finally, some info from the config log. pi at raspberrypi ~/varnish-3.0.3 $ grep -i warn config.log cc1: warning: command line option '-fno-rtti' is valid for C++/ObjC++ but not for C [enabled by default] configure:13219: WARNING: xsltproc not found - not building documentation configure:13279: WARNING: rst2man not found - not building man pages configure:13342: WARNING: rst2html not found - not building changelog conftest.c:40:6: warning: conflicting types for built-in function 'cos' [enabled by default] configure:14290: WARNING: libedit not found, disabling libedit support conftest.c:72:6: warning: conflicting types for built-in function 'vprintf' [enabled by default] configure:15220: WARNING: won't look for sendfile() on armv6l-unknown-linux-gnueabihf conftest.c:68:3: warning: left shift count >= width of type [enabled by default] conftest.c:68:3: warning: left shift count >= width of type [enabled by default] conftest.c:69:10: warning: left shift count >= width of type [enabled by default] conftest.c:69:10: warning: left shift count >= width of type [enabled by default] Sorry for the long post - but hopefully it gives us something to go on. Any information would be appreciated. -Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Jan 3 07:56:55 2013 From: perbu at varnish-software.com (Per Buer) Date: Thu, 3 Jan 2013 08:56:55 +0100 Subject: Want to know about some of the varnish features In-Reply-To: <30c8-50e3c900-3-381da500@144782867> References: <30c8-50e3c900-3-381da500@144782867> Message-ID: Hi, On Wed, Jan 2, 2013 at 6:43 AM, Rajat wrote: > a>Is it possible to expire a particular file/content from cache so that if > any changes are made to a file/content it can be reloaded into cache from > the app server. > I have read about ban (varnish 3.x), but in that case I need to restart > the engine to reload the configuration, but that will purge the whole > cache. which I do not want > This is well covered in the docs. https://www.varnish-cache.org/docs/3.0/tutorial/index.html b>Can we instruct varnish not to flush the cache on reboot. No. c> What are the rule preferences e.g. if there are two rules applicable for > a particular request which rule would get the preference. > N/A - https://www.varnish-cache.org/docs/3.0/tutorial/vcl.html -- *Per Buer* Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaohui.zheng at gmail.com Mon Jan 7 18:15:33 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Tue, 8 Jan 2013 02:15:33 +0800 Subject: 503 Error was met frequently Message-ID: Hello, After a few days efforts, I did not get my problem fixed, I almost exhaust every possible methods which I could do, so I am trying to get help from the community. I use varnish as web cache and load balancer to manage 3 web nodes, but recently, I get 503 errors frequently, My varnish configuration file: ======================================================= backend nanjing { .host = "10.80.125.66"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend hangzhou { .host = "10.80.125.68"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend chongqing { .host = "10.80.125.76"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } director proxy random { { .backend = chongqing; .weight = 2; } { .backend = nanjing; .weight = 4; } { .backend = hangzhou; .weight = 4; } } acl purge { "localhost"; "10.80.125.0"/24; } sub vcl_recv { set req.backend = proxy; if (req.request != "GET" && req.request != "HEAD") { # POST - Logins and edits if (req.request == "POST") { return(pass); } # PURGE - The CacheFu product can invalidate updated URLs if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return(lookup); } } # Don't cache authenticated requests if (req.http.Cookie && req.http.Cookie ~ "__ac(|_(name|password|persistent))=") { # Force lookup of specific urls unlikely to need protection if (req.url ~ "\.(js|css)") { remove req.http.cookie; return(lookup); } return(pass); } # The default vcl_recv is used from here. } sub vcl_hit { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } sub vcl_miss { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } # Enforce a minimum TTL, since we can PURGE changed objects actively # from Zope by using the CacheFu product sub vcl_fetch { if (beresp.ttl < 3600s) { set beresp.ttl = 3600s; } } Varnish boots up script ========================================== varnishd -f /etc/varnish/my.vcl -s malloc,8192M -a $ip:80 \ -T $ip:2048 \ -n vcache-my\ -p thread_pools=2 \ -p thread_pool_max=15000\ -p thread_pool_min=500\ -p listen_depth=2048 \ -p lru_interval=1800 \ -h classic,169313 \ -p connect_timeout=1800 \ -p http_max_hdr=8192\ -p http_resp_hdr_len=18192\ -p max_restarts=6 I try to the backend status: [root at hongkong varnish]# varnishadm -n vcache-my backend.list ============================================== Backend name Refs Admin Probe nanjing(10.80.125.66,,80) 68 probe Healthy 8/10 hangzhou(10.80.125.68,,80) 66 probe Healthy 7/10 chongqing(10.80.125.76,,80) 23 probe Healthy 9/10 I already downgrade the .threshold from 8 to 2, so it can make sure the all the node is in Healthy status, if I set the .threshold to 8, most of the node will be Sick. I try to use a script to wget the probe page every 2 seconds, there is no failure, but it is always have failure in the command 'backend.list', I have to script to watch the status of my website: ---------------------------------------------------------------------------------- #!/bin/bash pass=0 fail=0 while [ 1 ] do wget http://mysite/live.html -O /dev/null if [ $? -eq 0 ];then pass=$(expr $pass + 1) else fail=$(expr $fail + 1) fi echo -e "pass: $pass\n fail: $fail" sleep 5 done 25% failed, it is very strange thing, I have no clue about it, Example result about the varnish log: ======================================= varnishlog -n vcache-my| tee -a /var/log/varnish.log 977 RxHeader c Connection: keep-alive 977 RxHeader c User-Agent: Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 977 VCL_call c recv pass 977 VCL_call c hash 977 Hash c / 977 Hash c www.mywebsite.com 977 VCL_return c hash 977 VCL_call c pass pass 977 FetchError c no backend connection 977 VCL_call c error deliver 977 VCL_call c deliver deliver 977 TxProtocol c HTTP/1.1 977 TxStatus c 503 977 TxResponse c Service Unavailable 977 TxHeader c Server: Varnish 977 TxHeader c Content-Type: text/html; charset=utf-8 977 TxHeader c Retry-After: 5 977 TxHeader c Content-Length: 419 977 TxHeader c Accept-Ranges: bytes 977 TxHeader c Date: Mon, 07 Jan 2013 18:03:02 GMT 977 TxHeader c X-Varnish: 2122413499 more varnish log: shaohui dot org/downloads/varnish.tgz this 503 error make my website trap in troubles, my customers can not access my site, I did not have any clue, can some body provide some advices, thanks so much. -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From Raul.Rangel at disney.com Mon Jan 7 18:52:32 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Mon, 7 Jan 2013 10:52:32 -0800 Subject: 503 Error was met frequently In-Reply-To: References: Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> What is the load on your backend servers? It seems like they are under heavy load. One thing I did notice in your vcl_recv is the cookie code. You are checking the existence of a specific cookie and passing if it exists. If that cookie does not exist you fall back to the default vcl_recv. Well the default vcl_recv also does a check on req.http.Cookie. If it finds one then it also does a pass. My guess would be that most of your incoming requests have some sort of cookie and thus all the requests are getting passed to the backend. Have you done a varnishstat and looked at your cache hit ratios? If I were you I would add a unset req.http.cookie to the end of vcl_recv. This way the default vcl_recv won't pass the request. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Shaohui Zheng Sent: Monday, January 07, 2013 11:16 AM To: varnish-misc at varnish-cache.org Subject: 503 Error was met frequently Hello, After a few days efforts, I did not get my problem fixed, I almost exhaust every possible methods which I could do, so I am trying to get help from the community. I use varnish as web cache and load balancer to manage 3 web nodes, but recently, I get 503 errors frequently, My varnish configuration file: ======================================================= backend nanjing { .host = "10.80.125.66"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend hangzhou { .host = "10.80.125.68"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend chongqing { .host = "10.80.125.76"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } director proxy random { { .backend = chongqing; .weight = 2; } { .backend = nanjing; .weight = 4; } { .backend = hangzhou; .weight = 4; } } acl purge { "localhost"; "10.80.125.0"/24; } sub vcl_recv { set req.backend = proxy; if (req.request != "GET" && req.request != "HEAD") { # POST - Logins and edits if (req.request == "POST") { return(pass); } # PURGE - The CacheFu product can invalidate updated URLs if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return(lookup); } } # Don't cache authenticated requests if (req.http.Cookie && req.http.Cookie ~ "__ac(|_(name|password|persistent))=") { # Force lookup of specific urls unlikely to need protection if (req.url ~ "\.(js|css)") { remove req.http.cookie; return(lookup); } return(pass); } # The default vcl_recv is used from here. } sub vcl_hit { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } sub vcl_miss { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } # Enforce a minimum TTL, since we can PURGE changed objects actively # from Zope by using the CacheFu product sub vcl_fetch { if (beresp.ttl < 3600s) { set beresp.ttl = 3600s; } } Varnish boots up script ========================================== varnishd -f /etc/varnish/my.vcl -s malloc,8192M -a $ip:80 \ -T $ip:2048 \ -n vcache-my\ -p thread_pools=2 \ -p thread_pool_max=15000\ -p thread_pool_min=500\ -p listen_depth=2048 \ -p lru_interval=1800 \ -h classic,169313 \ -p connect_timeout=1800 \ -p http_max_hdr=8192\ -p http_resp_hdr_len=18192\ -p max_restarts=6 I try to the backend status: [root at hongkong varnish]# varnishadm -n vcache-my backend.list ============================================== Backend name Refs Admin Probe nanjing(10.80.125.66,,80) 68 probe Healthy 8/10 hangzhou(10.80.125.68,,80) 66 probe Healthy 7/10 chongqing(10.80.125.76,,80) 23 probe Healthy 9/10 I already downgrade the .threshold from 8 to 2, so it can make sure the all the node is in Healthy status, if I set the .threshold to 8, most of the node will be Sick. I try to use a script to wget the probe page every 2 seconds, there is no failure, but it is always have failure in the command 'backend.list', I have to script to watch the status of my website: ---------------------------------------------------------------------------------- #!/bin/bash pass=0 fail=0 while [ 1 ] do wget http://mysite/live.html -O /dev/null if [ $? -eq 0 ];then pass=$(expr $pass + 1) else fail=$(expr $fail + 1) fi echo -e "pass: $pass\n fail: $fail" sleep 5 done 25% failed, it is very strange thing, I have no clue about it, Example result about the varnish log: ======================================= varnishlog -n vcache-my| tee -a /var/log/varnish.log 977 RxHeader c Connection: keep-alive 977 RxHeader c User-Agent: Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 977 VCL_call c recv pass 977 VCL_call c hash 977 Hash c / 977 Hash c www.mywebsite.com 977 VCL_return c hash 977 VCL_call c pass pass 977 FetchError c no backend connection 977 VCL_call c error deliver 977 VCL_call c deliver deliver 977 TxProtocol c HTTP/1.1 977 TxStatus c 503 977 TxResponse c Service Unavailable 977 TxHeader c Server: Varnish 977 TxHeader c Content-Type: text/html; charset=utf-8 977 TxHeader c Retry-After: 5 977 TxHeader c Content-Length: 419 977 TxHeader c Accept-Ranges: bytes 977 TxHeader c Date: Mon, 07 Jan 2013 18:03:02 GMT 977 TxHeader c X-Varnish: 2122413499 more varnish log: shaohui dot org/downloads/varnish.tgz this 503 error make my website trap in troubles, my customers can not access my site, I did not have any clue, can some body provide some advices, thanks so much. -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Jacobs at icts.kuleuven.be Mon Jan 7 19:14:01 2013 From: Peter.Jacobs at icts.kuleuven.be (Peter Jacobs) Date: Mon, 7 Jan 2013 19:14:01 +0000 Subject: 503 Error was met frequently Message-ID: <851DE1A04402094ABD2EB2C47C31C2AD108A31C0@ICTS-S-MBX3.luna.kuleuven.be> The probes have a short timeout... You could try changing the probes temporarily to e.g.: .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } kind regards, Peter From shaohui.zheng at gmail.com Tue Jan 8 02:38:46 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Tue, 8 Jan 2013 10:38:46 +0800 Subject: 503 Error was met frequently In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: Yes, when the 503 error is happened, all my backend is in heavy load, the load is more than 40. the very strange thing is that I can wget the probe file hundreds times, and there is no fail. I can not explain it. We use cookie to save customers information, when guest come back to our website, it helps to do auto login. our varnish server work load is very low, under 1.0 in most time, but web server is in heavy load. Your answer can explain it. the hit raito sees terrible. that is the output of varnishstat: varnishstat -n vcache-my ========================================================================= 0+09:20:26 vcache-my Hitrate ratio: 8 8 8 Hitrate avg: 0.6145 0.6145 0.6145 4195411 61.92 124.77 client_conn - Client connections accepted 5371966 65.92 159.76 client_req - Client requests received 372757 7.99 11.09 cache_hit - Cache hits 6820 1.00 0.20 cache_hitpass - Cache hits for pass 63206 3.00 1.88 cache_miss - Cache misses 4240160 71.91 126.10 backend_conn - Backend conn. success 2277628 0.00 67.73 backend_fail - Backend conn. failures 84 0.00 0.00 fetch_head - Fetch head 3387280 52.93 100.73 fetch_length - Fetch with Length 417030 3.99 12.40 fetch_chunked - Fetch chunked 8669 1.00 0.26 fetch_close - Fetch wanted close 4 0.00 0.00 fetch_failed - Fetch failed 428006 0.00 12.73 fetch_304 - Fetch no body (304) 919 . . n_sess_mem - N struct sess_mem 339 . . n_sess - N struct sess 7159 . . n_object - N struct object 7633 . . n_objectcore - N struct objectcore 7635 . . n_objecthead - N struct objecthead 875 . . n_waitinglist - N struct waitinglist 62 . . n_vbc - N struct vbc 1000 . . n_wrk - N worker threads 1000 0.00 0.03 n_wrk_create - N worker threads created 219 0.00 0.01 n_wrk_queued - N queued work requests 3 . . n_backend - N backends 48739 . . n_expired - N expired objects 2567 . . n_lru_moved - N LRU moved objects 143 0.00 0.00 losthdr - HTTP header overflows That is great, thanks Rangel, Raul, your information is import for me, I will rewrite the vcl_recv function. On Tue, Jan 8, 2013 at 2:52 AM, Rangel, Raul wrote: > What is the load on your backend servers? It seems like they are under > heavy load. One thing I did notice in your vcl_recv is the cookie code. You > are checking the existence of a specific cookie and passing if it exists. > If that cookie does not exist you fall back to the default vcl_recv. Well > the default vcl_recv also does a check on req.http.Cookie. If it finds one > then it also does a pass. My guess would be that most of your incoming > requests have some sort of cookie and thus all the requests are getting > passed to the backend. Have you done a varnishstat and looked at your cache > hit ratios?**** > > ** ** > > If I were you I would add a unset req.http.cookie to the end of vcl_recv. > This way the default vcl_recv won?t pass the request.**** > > ** ** > > Raul**** > > ** ** > > *From:* varnish-misc-bounces at varnish-cache.org [mailto: > varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Shaohui Zheng > *Sent:* Monday, January 07, 2013 11:16 AM > *To:* varnish-misc at varnish-cache.org > *Subject:* 503 Error was met frequently**** > > ** ** > > Hello,**** > > ** ** > > After a few days efforts, I did not get my problem fixed, I almost > exhaust every possible methods which I could do, so I am trying to get help > from the community. **** > > ** ** > > I use varnish as web cache and load balancer to manage 3 web nodes, but > recently, I get 503 errors frequently, **** > > ** ** > > My varnish configuration file:**** > > =======================================================**** > > backend nanjing {**** > > .host = "10.80.125.66";**** > > .port = "80";**** > > .connect_timeout = 1800s;**** > > .first_byte_timeout = 1800s;**** > > .between_bytes_timeout = 1800s;**** > > .probe = {**** > > .url = "/live.html";**** > > .interval = 1s;**** > > .timeout = 3s;**** > > .window = 10;**** > > .threshold = 2;**** > > }**** > > }**** > > ** ** > > backend hangzhou {**** > > .host = "10.80.125.68";**** > > #.host = "10.36.146.202";**** > > .port = "80";**** > > .connect_timeout = 1800s;**** > > .first_byte_timeout = 1800s;**** > > .between_bytes_timeout = 1800s;**** > > ** ** > > .probe = {**** > > .url = "/live.html";**** > > .interval = 1s;**** > > .timeout = 3s;**** > > .window = 10;**** > > .threshold = 2;**** > > }**** > > }**** > > backend chongqing {**** > > .host = "10.80.125.76";**** > > .port = "80";**** > > .connect_timeout = 1800s;**** > > .first_byte_timeout = 1800s;**** > > .between_bytes_timeout = 1800s;**** > > ** ** > > .probe = {**** > > .url = "/live.html";**** > > .interval = 1s;**** > > .timeout = 3s;**** > > .window = 10;**** > > .threshold = 2;**** > > }**** > > }**** > > ** ** > > ** ** > > ** ** > > director proxy random {**** > > {**** > > .backend = chongqing;**** > > .weight = 2;**** > > }**** > > {**** > > .backend = nanjing;**** > > .weight = 4;**** > > }**** > > {**** > > .backend = hangzhou;**** > > .weight = 4;**** > > }**** > > }**** > > ** ** > > acl purge {**** > > "localhost";**** > > "10.80.125.0"/24;**** > > }**** > > ** ** > > sub vcl_recv {**** > > set req.backend = proxy;**** > > ** ** > > if (req.request != "GET" && req.request != "HEAD") {**** > > ** ** > > # POST - Logins and edits**** > > if (req.request == "POST") {**** > > return(pass);**** > > }**** > > **** > > # PURGE - The CacheFu product can invalidate updated URLs* > *** > > if (req.request == "PURGE") {**** > > if (!client.ip ~ purge) {**** > > error 405 "Not allowed.";**** > > }**** > > return(lookup);**** > > }**** > > }**** > > ** ** > > # Don't cache authenticated requests**** > > if (req.http.Cookie && req.http.Cookie ~ > "__ac(|_(name|password|persistent))=") {**** > > ** ** > > # Force lookup of specific urls unlikely to need > protection**** > > if (req.url ~ "\.(js|css)") {**** > > remove req.http.cookie;**** > > return(lookup);**** > > }**** > > return(pass);**** > > }**** > > ** ** > > # The default vcl_recv is used from here.**** > > }**** > > ** ** > > sub vcl_hit {**** > > # if (req.request == "PURGE") {**** > > # purge('');**** > > # error 200 "Purged";**** > > # }**** > > }**** > > sub vcl_miss {**** > > # if (req.request == "PURGE") {**** > > # purge('');**** > > # error 200 "Purged";**** > > # }**** > > }**** > > ** ** > > # Enforce a minimum TTL, since we can PURGE changed objects actively**** > > # from Zope by using the CacheFu product**** > > ** ** > > sub vcl_fetch {**** > > if (beresp.ttl < 3600s) {**** > > set beresp.ttl = 3600s;**** > > }**** > > }**** > > ** ** > > ** ** > > Varnish boots up script**** > > ==========================================**** > > varnishd -f /etc/varnish/my.vcl -s malloc,8192M > -a $ip:80 \**** > > -T $ip:2048 \**** > > -n vcache-my\**** > > -p thread_pools=2 \**** > > -p thread_pool_max=15000\**** > > -p thread_pool_min=500\**** > > -p listen_depth=2048 \**** > > -p lru_interval=1800 \**** > > -h classic,169313 \**** > > -p connect_timeout=1800 \**** > > -p http_max_hdr=8192\**** > > -p http_resp_hdr_len=18192\**** > > -p max_restarts=6 **** > > ** ** > > I try to the backend status:**** > > [root at hongkong varnish]# varnishadm -n vcache-my backend.list**** > > ==============================================**** > > Backend name Refs Admin Probe**** > > nanjing(10.80.125.66,,80) 68 probe Healthy 8/10**** > > hangzhou(10.80.125.68,,80) 66 probe Healthy 7/10**** > > chongqing(10.80.125.76,,80) 23 probe Healthy 9/10**** > > ** ** > > ** ** > > I already downgrade the .threshold from 8 to 2, so it can make sure the > all the node is in Healthy status, if I set the .threshold to 8,**** > > most of the node will be Sick.**** > > ** ** > > I try to use a script to wget the probe page every 2 seconds, there is no > failure, but it is always have failure in the command 'backend.list', **** > > ** ** > > I have to script to watch the status of my website:**** > > > ---------------------------------------------------------------------------------- > **** > > #!/bin/bash**** > > pass=0**** > > fail=0**** > > ** ** > > while [ 1 ]**** > > do**** > > wget http://mysite/live.html -O /dev/null**** > > if [ $? -eq 0 ];then**** > > pass=$(expr $pass + 1)**** > > else**** > > fail=$(expr $fail + 1)**** > > fi**** > > ** ** > > echo -e "pass: $pass\n fail: $fail" **** > > sleep 5**** > > done**** > > ** ** > > 25% failed, it is very strange thing, I have no clue about it, **** > > ** ** > > Example result about the varnish log:**** > > =======================================**** > > varnishlog -n vcache-my| tee -a /var/log/varnish.log**** > > ** ** > > 977 RxHeader c Connection: keep-alive**** > > 977 RxHeader c User-Agent: Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac > OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 > Safari/8536.25**** > > 977 VCL_call c recv pass**** > > 977 VCL_call c hash**** > > 977 Hash c /**** > > 977 Hash c www.mywebsite.com**** > > 977 VCL_return c hash**** > > 977 VCL_call c pass pass**** > > 977 FetchError c no backend connection**** > > 977 VCL_call c error deliver**** > > 977 VCL_call c deliver deliver**** > > 977 TxProtocol c HTTP/1.1**** > > 977 TxStatus c 503**** > > 977 TxResponse c Service Unavailable**** > > 977 TxHeader c Server: Varnish**** > > 977 TxHeader c Content-Type: text/html; charset=utf-8**** > > 977 TxHeader c Retry-After: 5**** > > 977 TxHeader c Content-Length: 419**** > > 977 TxHeader c Accept-Ranges: bytes**** > > 977 TxHeader c Date: Mon, 07 Jan 2013 18:03:02 GMT**** > > 977 TxHeader c X-Varnish: 2122413499**** > > ** ** > > more varnish log:**** > > shaohui dot org/downloads/varnish.tgz**** > > ** ** > > this 503 error make my website trap in troubles, my customers can not > access my site, I did not have any clue, can some body provide some > advices, thanks so much. **** > > ** ** > > -- > Best regards > Shaohui **** > -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaohui.zheng at gmail.com Tue Jan 8 02:44:08 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Tue, 8 Jan 2013 10:44:08 +0800 Subject: 503 Error was met frequently In-Reply-To: <851DE1A04402094ABD2EB2C47C31C2AD108A31C0@ICTS-S-MBX3.luna.kuleuven.be> References: <851DE1A04402094ABD2EB2C47C31C2AD108A31C0@ICTS-S-MBX3.luna.kuleuven.be> Message-ID: It is in an internal network, so I set the timeout only 3 secs, I wil try to set as 30s. Thanks Peter.Jacobs. On Tue, Jan 8, 2013 at 3:14 AM, Peter Jacobs wrote: > The probes have a short timeout... You could try changing the probes > temporarily to e.g.: > > .probe = { > .url = "/live.html"; > .interval = 1s; > .timeout = 30s; > .window = 10; > .threshold = 2; > } > > > kind regards, > Peter > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaohui.zheng at gmail.com Wed Jan 9 07:53:12 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Wed, 9 Jan 2013 15:53:12 +0800 Subject: 503 Error was met frequently In-Reply-To: References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: Thanks for Rangel Raul's suggestion, after did some home work, You did me a great favor, I draft an new VCL configure file ======================================= backend nanjing { .host = "10.80.125.66"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } backend hangzhou { .host = "10.80.125.68"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } backend chongqing { .host = "10.80.125.76"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } director proxy random { { .backend = chongqing; .weight = 2; } { .backend = nanjing; .weight = 4; } { .backend = hangzhou; .weight = 4; } } #acl purge { # "localhost"; # "10.80.125.0"/24; #} # sub vcl_recv { set req.backend = proxy; # remove Cookies ?ingore cookie for static page? if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|zip)(\?.*|)$") { unset req.http.cookie; } # ignore cookie for product page, it might pesudo static page if (req.url ~ "^/[^?]+-product-?+\.(html)(\?.*|)$") { unset req.http.cookie; } # remove Cookies on home page if (req.url ~ "^/$") { unset req.http.cookie; } # Allow a grace period for offering "stale" data in case backend lags set req.grace = 5m; remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; # Properly handle different encoding types if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } # Force lookup if the request is a no-cache request from the client # if (req.http.Cache-Control ~ "no-cache") { # return (pass); # } ## Default request checks if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { # Non-RFC2616 or CONNECT which is weird. return (pipe); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } ## Modified from default to allow caching if cookies are set, but not http auth if (req.http.Authorization) { /* Not cacheable by default */ return (pass); } # ORVSD tweaks ## Remove has_js and Google Analytics cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); ## Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ## Remove empty cookies. if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } if (req.url ~ ".*/server-status$") { return (pass); } ## Pass php page if (req.url ~ ".php") { return (pass); } ## Pass admin if (req.url ~ "(admin|cs_department)") { return (pass); } if (req.url ~ ".*/server-status$") { return (pass); } # The default vcl_recv is used from here. } # Do the PURGE thing sub vcl_hit { #if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; #} } sub vcl_miss { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } sub vcl_fetch { if (beresp.ttl < 3600s) { set beresp.ttl = 3600s; } } the data of varnishstat ======================================= 0+00:48:47 vcache-my Hitrate ratio: 10 81 81 Hitrate avg: 0.9317 0.9485 0.9485 257260 82.90 87.89 client_conn - Client connections accepted 367561 147.82 125.58 client_req - Client requests received 74714 41.95 25.53 cache_hit - Cache hits 35 0.00 0.01 cache_hitpass - Cache hits for pass 4438 2.00 1.52 cache_miss - Cache misses 292546 113.86 99.95 backend_conn - Backend conn. success 1090 0.00 0.37 backend_fail - Backend conn. failures 8 0.00 0.00 fetch_head - Fetch head 262473 89.89 89.67 fetch_length - Fetch with Length 27652 14.98 9.45 fetch_chunked - Fetch chunked 2095 1.00 0.72 fetch_close - Fetch wanted close 243 0.00 0.08 fetch_304 - Fetch no body (304) 837 . . n_sess_mem - N struct sess_mem 601 . . n_sess - N struct sess 3748 . . n_object - N struct object 4169 . . n_objectcore - N struct objectcore 4169 . . n_objecthead - N struct objecthead 455 . . n_waitinglist - N struct waitinglist 127 . . n_vbc - N struct vbc the backend workload is much better than before, I want to know which page is not cached,it can help me to improve the cache hit ratio, can I get some debug information from varnishstat? thanks. Do you have some more advice for my VCL configuration file.? On Tue, Jan 8, 2013 at 10:38 AM, Shaohui Zheng wrote: > Yes, when the 503 error is happened, all my backend is in heavy load, the > load is more than 40. the very strange thing is that I can wget the probe > file hundreds times, and there is no fail. I can not explain it. > > We use cookie to save customers information, when guest come back to our > website, it helps to do auto login. our varnish server work load is very > low, under 1.0 in most time, but web server is in heavy load. Your answer > can explain it. > > the hit raito sees terrible. that is the output of varnishstat: > varnishstat -n vcache-my > ========================================================================= > 0+09:20:26 > vcache-my > Hitrate ratio: 8 8 8 > Hitrate avg: 0.6145 0.6145 0.6145 > > 4195411 61.92 124.77 client_conn - Client connections > accepted > 5371966 65.92 159.76 client_req - Client requests > received > 372757 7.99 11.09 cache_hit - Cache hits > 6820 1.00 0.20 cache_hitpass - Cache hits for pass > 63206 3.00 1.88 cache_miss - Cache misses > 4240160 71.91 126.10 backend_conn - Backend conn. success > 2277628 0.00 67.73 backend_fail - Backend conn. > failures > 84 0.00 0.00 fetch_head - Fetch head > 3387280 52.93 100.73 fetch_length - Fetch with Length > 417030 3.99 12.40 fetch_chunked - Fetch chunked > 8669 1.00 0.26 fetch_close - Fetch wanted close > 4 0.00 0.00 fetch_failed - Fetch failed > 428006 0.00 12.73 fetch_304 - Fetch no body (304) > 919 . . n_sess_mem - N struct sess_mem > 339 . . n_sess - N struct sess > 7159 . . n_object - N struct object > 7633 . . n_objectcore - N struct objectcore > 7635 . . n_objecthead - N struct objecthead > 875 . . n_waitinglist - N struct waitinglist > 62 . . n_vbc - N struct vbc > 1000 . . n_wrk - N worker threads > 1000 0.00 0.03 n_wrk_create - N worker threads > created > 219 0.00 0.01 n_wrk_queued - N queued work > requests > 3 . . n_backend - N backends > 48739 . . n_expired - N expired objects > 2567 . . n_lru_moved - N LRU moved objects > 143 0.00 0.00 losthdr - HTTP header overflows > > That is great, thanks Rangel, Raul, your information is import for me, I > will rewrite the vcl_recv function. > > > On Tue, Jan 8, 2013 at 2:52 AM, Rangel, Raul wrote: > >> What is the load on your backend servers? It seems like they are under >> heavy load. One thing I did notice in your vcl_recv is the cookie code. You >> are checking the existence of a specific cookie and passing if it exists. >> If that cookie does not exist you fall back to the default vcl_recv. Well >> the default vcl_recv also does a check on req.http.Cookie. If it finds one >> then it also does a pass. My guess would be that most of your incoming >> requests have some sort of cookie and thus all the requests are getting >> passed to the backend. Have you done a varnishstat and looked at your cache >> hit ratios?**** >> >> ** ** >> >> If I were you I would add a unset req.http.cookie to the end of vcl_recv. >> This way the default vcl_recv won?t pass the request.**** >> >> ** ** >> >> Raul**** >> >> ** ** >> >> *From:* varnish-misc-bounces at varnish-cache.org [mailto: >> varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Shaohui Zheng >> *Sent:* Monday, January 07, 2013 11:16 AM >> *To:* varnish-misc at varnish-cache.org >> *Subject:* 503 Error was met frequently**** >> >> ** ** >> >> Hello,**** >> >> ** ** >> >> After a few days efforts, I did not get my problem fixed, I almost >> exhaust every possible methods which I could do, so I am trying to get help >> from the community. **** >> >> ** ** >> >> I use varnish as web cache and load balancer to manage 3 web nodes, but >> recently, I get 503 errors frequently, **** >> >> ** ** >> >> My varnish configuration file:**** >> >> =======================================================**** >> >> backend nanjing {**** >> >> .host = "10.80.125.66";**** >> >> .port = "80";**** >> >> .connect_timeout = 1800s;**** >> >> .first_byte_timeout = 1800s;**** >> >> .between_bytes_timeout = 1800s;**** >> >> .probe = {**** >> >> .url = "/live.html";**** >> >> .interval = 1s;**** >> >> .timeout = 3s;**** >> >> .window = 10;**** >> >> .threshold = 2;**** >> >> }**** >> >> }**** >> >> ** ** >> >> backend hangzhou {**** >> >> .host = "10.80.125.68";**** >> >> #.host = "10.36.146.202";**** >> >> .port = "80";**** >> >> .connect_timeout = 1800s;**** >> >> .first_byte_timeout = 1800s;**** >> >> .between_bytes_timeout = 1800s;**** >> >> ** ** >> >> .probe = {**** >> >> .url = "/live.html";**** >> >> .interval = 1s;**** >> >> .timeout = 3s;**** >> >> .window = 10;**** >> >> .threshold = 2;**** >> >> }**** >> >> }**** >> >> backend chongqing {**** >> >> .host = "10.80.125.76";**** >> >> .port = "80";**** >> >> .connect_timeout = 1800s;**** >> >> .first_byte_timeout = 1800s;**** >> >> .between_bytes_timeout = 1800s;**** >> >> ** ** >> >> .probe = {**** >> >> .url = "/live.html";**** >> >> .interval = 1s;**** >> >> .timeout = 3s;**** >> >> .window = 10;**** >> >> .threshold = 2;**** >> >> }**** >> >> }**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> director proxy random {**** >> >> {**** >> >> .backend = chongqing;**** >> >> .weight = 2;**** >> >> }**** >> >> {**** >> >> .backend = nanjing;**** >> >> .weight = 4;**** >> >> }**** >> >> {**** >> >> .backend = hangzhou;**** >> >> .weight = 4;**** >> >> }**** >> >> }**** >> >> ** ** >> >> acl purge {**** >> >> "localhost";**** >> >> "10.80.125.0"/24;**** >> >> }**** >> >> ** ** >> >> sub vcl_recv {**** >> >> set req.backend = proxy;**** >> >> ** ** >> >> if (req.request != "GET" && req.request != "HEAD") {**** >> >> ** ** >> >> # POST - Logins and edits**** >> >> if (req.request == "POST") {**** >> >> return(pass);**** >> >> }**** >> >> **** >> >> # PURGE - The CacheFu product can invalidate updated URLs >> **** >> >> if (req.request == "PURGE") {**** >> >> if (!client.ip ~ purge) {**** >> >> error 405 "Not allowed.";**** >> >> }**** >> >> return(lookup);**** >> >> }**** >> >> }**** >> >> ** ** >> >> # Don't cache authenticated requests**** >> >> if (req.http.Cookie && req.http.Cookie ~ >> "__ac(|_(name|password|persistent))=") {**** >> >> ** ** >> >> # Force lookup of specific urls unlikely to need >> protection**** >> >> if (req.url ~ "\.(js|css)") {**** >> >> remove req.http.cookie;**** >> >> return(lookup);**** >> >> }**** >> >> return(pass);**** >> >> }**** >> >> ** ** >> >> # The default vcl_recv is used from here.**** >> >> }**** >> >> ** ** >> >> sub vcl_hit {**** >> >> # if (req.request == "PURGE") {**** >> >> # purge('');**** >> >> # error 200 "Purged";**** >> >> # }**** >> >> }**** >> >> sub vcl_miss {**** >> >> # if (req.request == "PURGE") {**** >> >> # purge('');**** >> >> # error 200 "Purged";**** >> >> # }**** >> >> }**** >> >> ** ** >> >> # Enforce a minimum TTL, since we can PURGE changed objects actively**** >> >> # from Zope by using the CacheFu product**** >> >> ** ** >> >> sub vcl_fetch {**** >> >> if (beresp.ttl < 3600s) {**** >> >> set beresp.ttl = 3600s;**** >> >> }**** >> >> }**** >> >> ** ** >> >> ** ** >> >> Varnish boots up script**** >> >> ==========================================**** >> >> varnishd -f /etc/varnish/my.vcl -s malloc,8192M >> -a $ip:80 \**** >> >> -T $ip:2048 \**** >> >> -n vcache-my\**** >> >> -p thread_pools=2 \**** >> >> -p thread_pool_max=15000\**** >> >> -p thread_pool_min=500\**** >> >> -p listen_depth=2048 \**** >> >> -p lru_interval=1800 \**** >> >> -h classic,169313 \**** >> >> -p connect_timeout=1800 \**** >> >> -p http_max_hdr=8192\**** >> >> -p http_resp_hdr_len=18192\**** >> >> -p max_restarts=6 **** >> >> ** ** >> >> I try to the backend status:**** >> >> [root at hongkong varnish]# varnishadm -n vcache-my backend.list**** >> >> ==============================================**** >> >> Backend name Refs Admin Probe**** >> >> nanjing(10.80.125.66,,80) 68 probe Healthy 8/10**** >> >> hangzhou(10.80.125.68,,80) 66 probe Healthy 7/10**** >> >> chongqing(10.80.125.76,,80) 23 probe Healthy 9/10**** >> >> ** ** >> >> ** ** >> >> I already downgrade the .threshold from 8 to 2, so it can make sure the >> all the node is in Healthy status, if I set the .threshold to 8,**** >> >> most of the node will be Sick.**** >> >> ** ** >> >> I try to use a script to wget the probe page every 2 seconds, there is no >> failure, but it is always have failure in the command 'backend.list', *** >> * >> >> ** ** >> >> I have to script to watch the status of my website:**** >> >> >> ---------------------------------------------------------------------------------- >> **** >> >> #!/bin/bash**** >> >> pass=0**** >> >> fail=0**** >> >> ** ** >> >> while [ 1 ]**** >> >> do**** >> >> wget http://mysite/live.html -O /dev/null**** >> >> if [ $? -eq 0 ];then**** >> >> pass=$(expr $pass + 1)**** >> >> else**** >> >> fail=$(expr $fail + 1)**** >> >> fi**** >> >> ** ** >> >> echo -e "pass: $pass\n fail: $fail" **** >> >> sleep 5**** >> >> done**** >> >> ** ** >> >> 25% failed, it is very strange thing, I have no clue about it, **** >> >> ** ** >> >> Example result about the varnish log:**** >> >> =======================================**** >> >> varnishlog -n vcache-my| tee -a /var/log/varnish.log**** >> >> ** ** >> >> 977 RxHeader c Connection: keep-alive**** >> >> 977 RxHeader c User-Agent: Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac >> OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 >> Safari/8536.25**** >> >> 977 VCL_call c recv pass**** >> >> 977 VCL_call c hash**** >> >> 977 Hash c /**** >> >> 977 Hash c www.mywebsite.com**** >> >> 977 VCL_return c hash**** >> >> 977 VCL_call c pass pass**** >> >> 977 FetchError c no backend connection**** >> >> 977 VCL_call c error deliver**** >> >> 977 VCL_call c deliver deliver**** >> >> 977 TxProtocol c HTTP/1.1**** >> >> 977 TxStatus c 503**** >> >> 977 TxResponse c Service Unavailable**** >> >> 977 TxHeader c Server: Varnish**** >> >> 977 TxHeader c Content-Type: text/html; charset=utf-8**** >> >> 977 TxHeader c Retry-After: 5**** >> >> 977 TxHeader c Content-Length: 419**** >> >> 977 TxHeader c Accept-Ranges: bytes**** >> >> 977 TxHeader c Date: Mon, 07 Jan 2013 18:03:02 GMT**** >> >> 977 TxHeader c X-Varnish: 2122413499**** >> >> ** ** >> >> more varnish log:**** >> >> shaohui dot org/downloads/varnish.tgz**** >> >> ** ** >> >> this 503 error make my website trap in troubles, my customers can not >> access my site, I did not have any clue, can some body provide some >> advices, thanks so much. **** >> >> ** ** >> >> -- >> Best regards >> Shaohui **** >> > > > > -- > Best regards > Shaohui > -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From Raul.Rangel at disney.com Wed Jan 9 15:14:14 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Wed, 9 Jan 2013 07:14:14 -0800 Subject: 503 Error was met frequently In-Reply-To: References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2605E3A909@SM-CALA-VXMB04A.swna.wdpr.disney.com> If you are using Varnish 3 you can remove the Accept-Encoding logic. It is handled automatically now. I?m also assuming you copied this from the default vcl_recv. ## Default request checks if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { # Non-RFC2616 or CONNECT which is weird. return (pipe); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } ## Modified from default to allow caching if cookies are set, but not http auth if (req.http.Authorization) { /* Not cacheable by default */ return (pass); } I would personally get rid of it and let the default vcl_recv handle it. You also have a comment ## Modified from default to allow caching if cookies are set, but not http auth This is not actually happening. At the end of your vcl_recv you don?t do a return (lookup) so the default vcl_recv runs anyway and it will pass if any cookie is present. If you really want to cache responses with a cookie you should take a look at https://www.varnish-cache.org/trac/wiki/VCLExampleCachingLoggedInUsers. But be careful of what you cache. Also if you know your backend only responds to specific cookies you can look at the regexp at https://www.varnish-cache.org/docs/3.0/tutorial/cookies.html and strip off all the cookies except the ones you care about. If you want to see which urls are getting sent to the backend you can do a varnishlog -b -i TxURL. Raul From: Shaohui Zheng [mailto:shaohui.zheng at gmail.com] Sent: Wednesday, January 09, 2013 12:53 AM To: Rangel, Raul Cc: varnish-misc at varnish-cache.org Subject: Re: 503 Error was met frequently Thanks for Rangel Raul's suggestion, after did some home work, You did me a great favor, I draft an new VCL configure file ======================================= backend nanjing { .host = "10.80.125.66"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } backend hangzhou { .host = "10.80.125.68"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } backend chongqing { .host = "10.80.125.76"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .max_connections = 500; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 30s; .window = 10; .threshold = 2; } } director proxy random { { .backend = chongqing; .weight = 2; } { .backend = nanjing; .weight = 4; } { .backend = hangzhou; .weight = 4; } } #acl purge { # "localhost"; # "10.80.125.0"/24; #} # sub vcl_recv { set req.backend = proxy; # remove Cookies ?ingore cookie for static page? if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|zip)(\?.*|)$") { unset req.http.cookie; } # ignore cookie for product page, it might pesudo static page if (req.url ~ "^/[^?]+-product-?+\.(html)(\?.*|)$") { unset req.http.cookie; } # remove Cookies on home page if (req.url ~ "^/$") { unset req.http.cookie; } # Allow a grace period for offering "stale" data in case backend lags set req.grace = 5m; remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; # Properly handle different encoding types if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } # Force lookup if the request is a no-cache request from the client # if (req.http.Cache-Control ~ "no-cache") { # return (pass); # } ## Default request checks if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { # Non-RFC2616 or CONNECT which is weird. return (pipe); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } if (req.request != "GET" && req.request != "HEAD") { # We only deal with GET and HEAD by default return (pass); } ## Modified from default to allow caching if cookies are set, but not http auth if (req.http.Authorization) { /* Not cacheable by default */ return (pass); } # ORVSD tweaks ## Remove has_js and Google Analytics cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); ## Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ## Remove empty cookies. if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } if (req.url ~ ".*/server-status$") { return (pass); } ## Pass php page if (req.url ~ ".php") { return (pass); } ## Pass admin if (req.url ~ "(admin|cs_department)") { return (pass); } if (req.url ~ ".*/server-status$") { return (pass); } # The default vcl_recv is used from here. } # Do the PURGE thing sub vcl_hit { #if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; #} } sub vcl_miss { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } sub vcl_fetch { if (beresp.ttl < 3600s) { set beresp.ttl = 3600s; } } the data of varnishstat ======================================= 0+00:48:47 vcache-my Hitrate ratio: 10 81 81 Hitrate avg: 0.9317 0.9485 0.9485 257260 82.90 87.89 client_conn - Client connections accepted 367561 147.82 125.58 client_req - Client requests received 74714 41.95 25.53 cache_hit - Cache hits 35 0.00 0.01 cache_hitpass - Cache hits for pass 4438 2.00 1.52 cache_miss - Cache misses 292546 113.86 99.95 backend_conn - Backend conn. success 1090 0.00 0.37 backend_fail - Backend conn. failures 8 0.00 0.00 fetch_head - Fetch head 262473 89.89 89.67 fetch_length - Fetch with Length 27652 14.98 9.45 fetch_chunked - Fetch chunked 2095 1.00 0.72 fetch_close - Fetch wanted close 243 0.00 0.08 fetch_304 - Fetch no body (304) 837 . . n_sess_mem - N struct sess_mem 601 . . n_sess - N struct sess 3748 . . n_object - N struct object 4169 . . n_objectcore - N struct objectcore 4169 . . n_objecthead - N struct objecthead 455 . . n_waitinglist - N struct waitinglist 127 . . n_vbc - N struct vbc the backend workload is much better than before, I want to know which page is not cached,it can help me to improve the cache hit ratio, can I get some debug information from varnishstat? thanks. Do you have some more advice for my VCL configuration file.? On Tue, Jan 8, 2013 at 10:38 AM, Shaohui Zheng > wrote: Yes, when the 503 error is happened, all my backend is in heavy load, the load is more than 40. the very strange thing is that I can wget the probe file hundreds times, and there is no fail. I can not explain it. We use cookie to save customers information, when guest come back to our website, it helps to do auto login. our varnish server work load is very low, under 1.0 in most time, but web server is in heavy load. Your answer can explain it. the hit raito sees terrible. that is the output of varnishstat: varnishstat -n vcache-my ========================================================================= 0+09:20:26 vcache-my Hitrate ratio: 8 8 8 Hitrate avg: 0.6145 0.6145 0.6145 4195411 61.92 124.77 client_conn - Client connections accepted 5371966 65.92 159.76 client_req - Client requests received 372757 7.99 11.09 cache_hit - Cache hits 6820 1.00 0.20 cache_hitpass - Cache hits for pass 63206 3.00 1.88 cache_miss - Cache misses 4240160 71.91 126.10 backend_conn - Backend conn. success 2277628 0.00 67.73 backend_fail - Backend conn. failures 84 0.00 0.00 fetch_head - Fetch head 3387280 52.93 100.73 fetch_length - Fetch with Length 417030 3.99 12.40 fetch_chunked - Fetch chunked 8669 1.00 0.26 fetch_close - Fetch wanted close 4 0.00 0.00 fetch_failed - Fetch failed 428006 0.00 12.73 fetch_304 - Fetch no body (304) 919 . . n_sess_mem - N struct sess_mem 339 . . n_sess - N struct sess 7159 . . n_object - N struct object 7633 . . n_objectcore - N struct objectcore 7635 . . n_objecthead - N struct objecthead 875 . . n_waitinglist - N struct waitinglist 62 . . n_vbc - N struct vbc 1000 . . n_wrk - N worker threads 1000 0.00 0.03 n_wrk_create - N worker threads created 219 0.00 0.01 n_wrk_queued - N queued work requests 3 . . n_backend - N backends 48739 . . n_expired - N expired objects 2567 . . n_lru_moved - N LRU moved objects 143 0.00 0.00 losthdr - HTTP header overflows That is great, thanks Rangel, Raul, your information is import for me, I will rewrite the vcl_recv function. On Tue, Jan 8, 2013 at 2:52 AM, Rangel, Raul > wrote: What is the load on your backend servers? It seems like they are under heavy load. One thing I did notice in your vcl_recv is the cookie code. You are checking the existence of a specific cookie and passing if it exists. If that cookie does not exist you fall back to the default vcl_recv. Well the default vcl_recv also does a check on req.http.Cookie. If it finds one then it also does a pass. My guess would be that most of your incoming requests have some sort of cookie and thus all the requests are getting passed to the backend. Have you done a varnishstat and looked at your cache hit ratios? If I were you I would add a unset req.http.cookie to the end of vcl_recv. This way the default vcl_recv won?t pass the request. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Shaohui Zheng Sent: Monday, January 07, 2013 11:16 AM To: varnish-misc at varnish-cache.org Subject: 503 Error was met frequently Hello, After a few days efforts, I did not get my problem fixed, I almost exhaust every possible methods which I could do, so I am trying to get help from the community. I use varnish as web cache and load balancer to manage 3 web nodes, but recently, I get 503 errors frequently, My varnish configuration file: ======================================================= backend nanjing { .host = "10.80.125.66"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend hangzhou { .host = "10.80.125.68"; #.host = "10.36.146.202"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } backend chongqing { .host = "10.80.125.76"; .port = "80"; .connect_timeout = 1800s; .first_byte_timeout = 1800s; .between_bytes_timeout = 1800s; .probe = { .url = "/live.html"; .interval = 1s; .timeout = 3s; .window = 10; .threshold = 2; } } director proxy random { { .backend = chongqing; .weight = 2; } { .backend = nanjing; .weight = 4; } { .backend = hangzhou; .weight = 4; } } acl purge { "localhost"; "10.80.125.0"/24; } sub vcl_recv { set req.backend = proxy; if (req.request != "GET" && req.request != "HEAD") { # POST - Logins and edits if (req.request == "POST") { return(pass); } # PURGE - The CacheFu product can invalidate updated URLs if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return(lookup); } } # Don't cache authenticated requests if (req.http.Cookie && req.http.Cookie ~ "__ac(|_(name|password|persistent))=") { # Force lookup of specific urls unlikely to need protection if (req.url ~ "\.(js|css)") { remove req.http.cookie; return(lookup); } return(pass); } # The default vcl_recv is used from here. } sub vcl_hit { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } sub vcl_miss { # if (req.request == "PURGE") { # purge(''); # error 200 "Purged"; # } } # Enforce a minimum TTL, since we can PURGE changed objects actively # from Zope by using the CacheFu product sub vcl_fetch { if (beresp.ttl < 3600s) { set beresp.ttl = 3600s; } } Varnish boots up script ========================================== varnishd -f /etc/varnish/my.vcl -s malloc,8192M -a $ip:80 \ -T $ip:2048 \ -n vcache-my\ -p thread_pools=2 \ -p thread_pool_max=15000\ -p thread_pool_min=500\ -p listen_depth=2048 \ -p lru_interval=1800 \ -h classic,169313 \ -p connect_timeout=1800 \ -p http_max_hdr=8192\ -p http_resp_hdr_len=18192\ -p max_restarts=6 I try to the backend status: [root at hongkong varnish]# varnishadm -n vcache-my backend.list ============================================== Backend name Refs Admin Probe nanjing(10.80.125.66,,80) 68 probe Healthy 8/10 hangzhou(10.80.125.68,,80) 66 probe Healthy 7/10 chongqing(10.80.125.76,,80) 23 probe Healthy 9/10 I already downgrade the .threshold from 8 to 2, so it can make sure the all the node is in Healthy status, if I set the .threshold to 8, most of the node will be Sick. I try to use a script to wget the probe page every 2 seconds, there is no failure, but it is always have failure in the command 'backend.list', I have to script to watch the status of my website: ---------------------------------------------------------------------------------- #!/bin/bash pass=0 fail=0 while [ 1 ] do wget http://mysite/live.html -O /dev/null if [ $? -eq 0 ];then pass=$(expr $pass + 1) else fail=$(expr $fail + 1) fi echo -e "pass: $pass\n fail: $fail" sleep 5 done 25% failed, it is very strange thing, I have no clue about it, Example result about the varnish log: ======================================= varnishlog -n vcache-my| tee -a /var/log/varnish.log 977 RxHeader c Connection: keep-alive 977 RxHeader c User-Agent: Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 977 VCL_call c recv pass 977 VCL_call c hash 977 Hash c / 977 Hash c www.mywebsite.com 977 VCL_return c hash 977 VCL_call c pass pass 977 FetchError c no backend connection 977 VCL_call c error deliver 977 VCL_call c deliver deliver 977 TxProtocol c HTTP/1.1 977 TxStatus c 503 977 TxResponse c Service Unavailable 977 TxHeader c Server: Varnish 977 TxHeader c Content-Type: text/html; charset=utf-8 977 TxHeader c Retry-After: 5 977 TxHeader c Content-Length: 419 977 TxHeader c Accept-Ranges: bytes 977 TxHeader c Date: Mon, 07 Jan 2013 18:03:02 GMT 977 TxHeader c X-Varnish: 2122413499 more varnish log: shaohui dot org/downloads/varnish.tgz this 503 error make my website trap in troubles, my customers can not access my site, I did not have any clue, can some body provide some advices, thanks so much. -- Best regards Shaohui -- Best regards Shaohui -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From eschefter at web.de Thu Jan 10 19:17:51 2013 From: eschefter at web.de (eric schefter) Date: Thu, 10 Jan 2013 20:17:51 +0100 Subject: New user account named "varnish" appeared out of the blue Message-ID: <436581A3-FDBB-4E79-AC25-45562B16D013@web.de> Derar usres, I'm running OS X 10.6.8 I installed Bitnami, and then uninstalled it. One day later, a new user account appeared on my Mac with the name varnish. Is it possible that that account was somehow created by the BitNami installation? I'm worried that I've been hacked, and can't find any info on this. Thank you for your help. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at ifixit.com Thu Jan 10 23:48:20 2013 From: james at ifixit.com (James Pearson) Date: Thu, 10 Jan 2013 15:48:20 -0800 Subject: New user account named "varnish" appeared out of the blue In-Reply-To: <436581A3-FDBB-4E79-AC25-45562B16D013@web.de> References: <436581A3-FDBB-4E79-AC25-45562B16D013@web.de> Message-ID: <1357861571-sup-8774@geror.local> Excerpts from eric schefter's message of 2013-01-10 11:17:51 -0800: > Derar usres, > > I'm running OS X 10.6.8 > > I installed Bitnami, and then uninstalled it. > One day later, a new user account appeared on my Mac with the name varnish. > Is it possible that that account was somehow created by the BitNami installation? > I'm worried that I've been hacked, and can't find any info on this. > > Thank you for your help. > > Eric BitNami includes an installer for Varnish[0]. If you installed, say, Wordpress[1] or Rails[2], you would've gotten Varnish installed as well as other stack components. I can think of no reason why anyone maliciously accessing your machine would install Varnish on it. HTH, - P [0]: http://wiki.bitnami.org/Components/Varnish [1]: http://blog.bitnami.org/2012/07/bitnami-wordpress-now-with-varnish.html [2]: http://blog.bitnami.org/2012/08/rubystack-now-with-rails-327-and-varnish.html From shaohui.zheng at gmail.com Fri Jan 11 03:24:11 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Fri, 11 Jan 2013 11:24:11 +0800 Subject: 503 Error was met frequently In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F2605E3A909@SM-CALA-VXMB04A.swna.wdpr.disney.com> References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> <2465AAEEC8B8A242B26ED5F44BCA805F2605E3A909@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: I refers other people's configure file on internet, it seems that it is out of date :). the command *varnishlog -b -i TxURL is a great for debugging.* Thanks Rangel Raul & Peter Jacobs. the new VCL configure file works well, and the 503 error has been gone. You guys really solve me a big problem. I have other question, I know many people select LVS(Linux Virtual Server), but I prefer Varnish. I hear from some guys said Varnish can only take 3 ~ 5 backends, if it takes more backends, the performance will downgrade too much. Do you have some experience? how many backends can varnish maintain with best performance, just according to the experience. I already have 5 backends according to the workloads, but it seems that the backends workload is still too high, I want to add more nodes. Waiting for your advice, thanks. On Wed, Jan 9, 2013 at 11:14 PM, Rangel, Raul wrote: > *varnishlog -b -i TxURL*. -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Fri Jan 11 08:03:58 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 11 Jan 2013 09:03:58 +0100 Subject: 503 Error was met frequently In-Reply-To: References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> <2465AAEEC8B8A242B26ED5F44BCA805F2605E3A909@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: Hi, About the 5 backends limit, you might be interested by this link: https://www.varnish-cache.org/docs/trunk/phk/backends.html PHK basically states that 1000 backends is not an unreasonable number. I assume that backends are limited according to available resources and other factors that can't be established without proper measurements. Best regards, Dridi On Fri, Jan 11, 2013 at 4:24 AM, Shaohui Zheng wrote: > I refers other people's configure file on internet, it seems that it is out > of date :). > > the command varnishlog -b -i TxURL is a great for debugging. > > Thanks Rangel Raul & Peter Jacobs. > > the new VCL configure file works well, and the 503 error has been gone. > > You guys really solve me a big problem. > > I have other question, I know many people select LVS(Linux Virtual Server), > but I prefer Varnish. I hear from some guys said Varnish can only take 3 ~ 5 > backends, if it takes more backends, the performance will downgrade too > much. Do you have some experience? how many backends can varnish maintain > with best performance, just according to the experience. > > > I already have 5 backends according to the workloads, but it seems that the > backends workload is still too high, I want to add more nodes. Waiting for > your advice, thanks. > > > On Wed, Jan 9, 2013 at 11:14 PM, Rangel, Raul > wrote: >> >> varnishlog -b -i TxURL. > > > > > > -- > Best regards > Shaohui > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From radecki.rafal at gmail.com Fri Jan 11 09:16:01 2013 From: radecki.rafal at gmail.com (=?ISO-8859-2?Q?Rafa=B3_Radecki?=) Date: Fri, 11 Jan 2013 10:16:01 +0100 Subject: Varnish 3 - choose backend based on header field. Message-ID: Hi all. I currently use varnish 3 in my environment. I use client director with two backends. I use set client.identity = client.ip to connect client session with backend. I would like to have the possibility to choose specific backend (first or second) based on the presence of a header in http request. If my request would have a header X-AppServer = app1 varnish should forward the request to the first application server if X-AppServer = app2 to the second and if there would be no such header it should use default client director. Is it possible in varnish 3? Best regards, Rafal. From gc at ants.eu Fri Jan 11 10:00:50 2013 From: gc at ants.eu (Gianni Carabelli) Date: Fri, 11 Jan 2013 11:00:50 +0100 Subject: Varnish 3 - choose backend based on header field. In-Reply-To: References: Message-ID: <50EFE2D2.4060406@ants.eu> On 01/11/2013 10:16 AM, Rafa? Radecki wrote: > Hi all. > > I currently use varnish 3 in my environment. I use client director > with two backends. I use set client.identity = client.ip to connect > client session with backend. > I would like to have the possibility to choose specific backend (first > or second) based on the presence of a header in http request. If my > request would have a header X-AppServer = app1 varnish should forward > the request to the first application server if X-AppServer = app2 to > the second and if there would be no such header it should use default > client director. Is it possible in varnish 3? > > Best regards, > Rafal. > in vcl_recv Something like: if (req.http.header ~ "X-AppServer = app1") { set req.backend = myBackend1 } Have a look here: https://www.varnish-cache.org/docs/3.0/reference/vcl.html#variables Bye JohnnyRun From zs at enternewmedia.com Fri Jan 11 20:36:37 2013 From: zs at enternewmedia.com (Zachary Stern) Date: Fri, 11 Jan 2013 15:36:37 -0500 Subject: monitor varnish load balancing? Message-ID: Is there a way to monitor varnish's load balancing functionality and see what it's doing, and which backends are marked as up, down, etc? E.g. with haproxy, you go to http://haproxyhost/haproxy?stats and it gives you detailed info. -- zachary alex stern I systems architect o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com 60-62 e. 11th street, 4th floor | new york, ny | 10003 www.enternewmedia.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.sanmartin at unix.cl Fri Jan 11 21:02:58 2013 From: luis.sanmartin at unix.cl (Luis San Martin) Date: Fri, 11 Jan 2013 18:02:58 -0300 Subject: monitor varnish load balancing? In-Reply-To: References: Message-ID: try varnishtat :) On 11 Jan 2013 17:40, "Zachary Stern" wrote: > Is there a way to monitor varnish's load balancing functionality and see > what it's doing, and which backends are marked as up, down, etc? > > E.g. with haproxy, you go to http://haproxyhost/haproxy?stats and it > gives you detailed info. > > -- > > zachary alex stern I systems architect > > o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com > > 60-62 e. 11th street, 4th floor | new york, ny | 10003 > > www.enternewmedia.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zs at enternewmedia.com Fri Jan 11 21:04:39 2013 From: zs at enternewmedia.com (Zachary Stern) Date: Fri, 11 Jan 2013 16:04:39 -0500 Subject: monitor varnish load balancing? In-Reply-To: References: Message-ID: Thanks Luis! I see 2 . . n_backend - N backends But there's no indication of up or down. On Fri, Jan 11, 2013 at 4:02 PM, Luis San Martin wrote: > try varnishtat :) > On 11 Jan 2013 17:40, "Zachary Stern" wrote: > >> Is there a way to monitor varnish's load balancing functionality and see >> what it's doing, and which backends are marked as up, down, etc? >> >> E.g. with haproxy, you go to http://haproxyhost/haproxy?stats and it >> gives you detailed info. >> >> -- >> >> zachary alex stern I systems architect >> >> o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com >> >> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >> >> www.enternewmedia.com >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -- zachary alex stern I systems architect o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com 60-62 e. 11th street, 4th floor | new york, ny | 10003 www.enternewmedia.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.sanmartin at unix.cl Fri Jan 11 21:06:58 2013 From: luis.sanmartin at unix.cl (Luis San Martin) Date: Fri, 11 Jan 2013 18:06:58 -0300 Subject: monitor varnish load balancing? In-Reply-To: References: Message-ID: Well you should also get used to varnishlog[0]. i.e. health checking: varnishlog | grep Backend_health [0] https://www.varnish-cache.org/docs/3.0/reference/varnishlog.html On Fri, Jan 11, 2013 at 6:04 PM, Zachary Stern wrote: > Thanks Luis! > > I see > > 2 . . n_backend - N backends > > But there's no indication of up or down. > > > On Fri, Jan 11, 2013 at 4:02 PM, Luis San Martin wrote: > >> try varnishtat :) >> On 11 Jan 2013 17:40, "Zachary Stern" wrote: >> >>> Is there a way to monitor varnish's load balancing functionality and see >>> what it's doing, and which backends are marked as up, down, etc? >>> >>> E.g. with haproxy, you go to http://haproxyhost/haproxy?stats and it >>> gives you detailed info. >>> >>> -- >>> >>> zachary alex stern I systems architect >>> >>> o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com >>> >>> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >>> >>> www.enternewmedia.com >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> > > > -- > > zachary alex stern I systems architect > > o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com > > 60-62 e. 11th street, 4th floor | new york, ny | 10003 > > www.enternewmedia.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leiwang at rhapsody.com Fri Jan 11 21:07:25 2013 From: leiwang at rhapsody.com (Lei Wang) Date: Fri, 11 Jan 2013 13:07:25 -0800 Subject: Varnish 3 client_drop problem Message-ID: Hi, Need help on one problem. After few days, Varnish suddenly shows many client_drop error. Then response very slowly. I have to restart Varnish to fix it. Vanish command as following: /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f /etc/varnish/varnish80.vcl -T 127.0.0.1:6080 -t 120 -w 1,1000,120 -u varnish -g varnish -i varnish80 -S /etc/varnish/secret -p thread_pool_add_delay 2 -p thread_pools 2 -p thread_pool_min 100 -p thread_pool_max 2000 -p session_linger 50 -p sess_workspace 65536 -p http_req_hdr_len 8192 -p http_gzip_support off -n varnish80 -s malloc,24G Below is the varnishstat result. client_conn 1050088 2.58 Client connections accepted *client_drop 481044 1.18 Connection dropped, no sess/wrk * client_req 53255991 130.93 Client requests received cache_hit 49915997 122.71 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 3098734 7.62 Cache misses backend_conn 23622 0.06 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 49 0.00 Backend conn. failures backend_reuse 3076713 7.56 Backend conn. reuses backend_toolate 4640 0.01 Backend conn. was closed backend_recycle 3081374 7.58 Backend conn. recycles backend_retry 1858 0.00 Backend conn. retry fetch_head 63 0.00 Fetch head fetch_length 1122064 2.76 Fetch with Length fetch_chunked 1959269 4.82 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 3641 0.01 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 100000 . N struct sess_mem n_sess 99996 . N struct sess n_object 384255 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 384362 . N struct objectcore n_objecthead 386502 . N struct objecthead n_waitinglist 22165 . N struct waitinglist n_vbc 15 . N struct vbc n_wrk 800 . N worker threads n_wrk_create 800 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 0 0.00 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 10 . N backends n_expired 2700698 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 6742713 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 33131257 81.45 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 569070 1.40 Total Sessions s_req 53255991 130.93 Total Requests s_pipe 12 0.00 Total pipe s_pass 84 0.00 Total pass s_fetch 3085037 7.58 Total fetch s_hdrbytes 23233920070 57118.78 Total header bytes s_bodybytes 511406013002 1257251.76 Total body bytes sess_closed 268273 0.66 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 52902973 130.06 Session Linger sess_herd 19131516 47.03 Session herd shm_records 2474975941 6084.54 SHM records shm_writes 80791017 198.62 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 23501 0.06 SHM MTX contention shm_cycles 889 0.00 SHM cycles through buffer sms_nreq 154946 0.38 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 65340505 . SMS bytes allocated sms_bfree 65340505 . SMS bytes freed backend_req 3100325 7.62 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 53113990 130.58 HCB Lookups without lock hcb_lock 2909158 7.15 HCB Lookups with lock hcb_insert 2909154 7.15 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 406765 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 1 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 0 0.00 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 464838 1.14 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 13844221 34.03 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 5436356 13.36 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 165593 0.41 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 569072 1.40 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 1173472 2.88 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 1334044 3.28 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1 0.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 1 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 39531997 97.19 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 2909259 7.15 Created locks LCK.objhdr.destroy 2522760 6.20 Destroyed locks LCK.objhdr.locks 217895431 535.68 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 6191913 15.22 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 3084953 7.58 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 135560 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 6191997 15.22 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 47327 0.12 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 10 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 6233731 15.33 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 6321249 15.54 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 300614422863 739037.09 Bytes allocated SMA.s0.c_freed 289845281665 712562.00 Bytes freed SMA.s0.g_alloc 768495 . Allocations outstanding SMA.s0.g_bytes 10769141198 . Bytes outstanding SMA.s0.g_space 15000662578 . Bytes available SMA.Transient.c_req 3656 0.01 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 43886078 107.89 Bytes allocated SMA.Transient.c_freed 43882627 107.88 Bytes freed SMA.Transient.g_alloc 4 . Allocations outstanding SMA.Transient.g_bytes 3451 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.rdsmetadata(10.150.16.2,,80).vcls 1 . VCL references VBE.rdsmetadata(10.150.16.2,,80).happy 0 . Happy health probes Thanks, Lei -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.sanmartin at unix.cl Fri Jan 11 21:10:22 2013 From: luis.sanmartin at unix.cl (Luis San Martin) Date: Fri, 11 Jan 2013 18:10:22 -0300 Subject: monitor varnish load balancing? In-Reply-To: References: Message-ID: Also you may want to check this[0]. It some kind of approach for stats. [0] http://eot.cl/python/stats.py On Fri, Jan 11, 2013 at 6:06 PM, Luis San Martin wrote: > Well you should also get used to varnishlog[0]. > > i.e. health checking: > > varnishlog | grep Backend_health > > [0] https://www.varnish-cache.org/docs/3.0/reference/varnishlog.html > > > On Fri, Jan 11, 2013 at 6:04 PM, Zachary Stern wrote: > >> Thanks Luis! >> >> I see >> >> 2 . . n_backend - N backends >> >> But there's no indication of up or down. >> >> >> On Fri, Jan 11, 2013 at 4:02 PM, Luis San Martin wrote: >> >>> try varnishtat :) >>> On 11 Jan 2013 17:40, "Zachary Stern" wrote: >>> >>>> Is there a way to monitor varnish's load balancing functionality and >>>> see what it's doing, and which backends are marked as up, down, etc? >>>> >>>> E.g. with haproxy, you go to http://haproxyhost/haproxy?stats and it >>>> gives you detailed info. >>>> >>>> -- >>>> >>>> zachary alex stern I systems architect >>>> >>>> o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com >>>> >>>> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >>>> >>>> www.enternewmedia.com >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >> >> >> -- >> >> zachary alex stern I systems architect >> >> o: 212.363.1654 x106 | f: 212.202.6488 | zs at enternewmedia.com >> >> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >> >> www.enternewmedia.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From themadindian at yahoo.com Fri Jan 11 21:50:35 2013 From: themadindian at yahoo.com (Keith Grefski) Date: Fri, 11 Jan 2013 16:50:35 -0500 Subject: monitor varnish load balancing? Message-ID: Try this to check all health varnish top -i Backend_health Or to only get sick varnishtop -i Backend_health -I sick To check on load balancing use varnishtop -i BackendReuse Luis San Martin wrote: >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From themadindian at yahoo.com Fri Jan 11 22:01:30 2013 From: themadindian at yahoo.com (themadindian) Date: Fri, 11 Jan 2013 17:01:30 -0500 Subject: =?utf-8?B?UmU6IFZhcm5pc2ggMyBjbGllbnRfZHJvcCBwcm9ibGVt?= Message-ID: <1979248734.3258858.1357941694342.JavaMail.webspher@njbbicssmp03> What do you see in your varnishlog? Sent from my Verizon Wireless 4G LTE Smartphone ----- Reply message ----- From: "Lei Wang" To: Subject: Varnish 3 client_drop problem Date: Fri, Jan 11, 2013 4:07 PM Hi, Need help on one problem. After few days, Varnish suddenly shows many client_drop error. Then response very slowly. I have to restart Varnish to fix it. Vanish command as following: /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f /etc/varnish/varnish80.vcl -T 127.0.0.1:6080 -t 120 -w 1,1000,120 -u varnish -g varnish -i varnish80 -S /etc/varnish/secret -p thread_pool_add_delay 2 -p thread_pools 2 -p thread_pool_min 100 -p thread_pool_max 2000 -p session_linger 50 -p sess_workspace 65536 -p http_req_hdr_len 8192 -p http_gzip_support off -n varnish80 -s malloc,24G Below is the varnishstat result. client_conn??????????? 1050088???????? 2.58 Client connections accepted client_drop???????????? 481044???????? 1.18 Connection dropped, no sess/wrk client_req??????????? 53255991?????? 130.93 Client requests received cache_hit???????????? 49915997?????? 122.71 Cache hits cache_hitpass??????????????? 0???????? 0.00 Cache hits for pass cache_miss???????????? 3098734???????? 7.62 Cache misses backend_conn???????????? 23622???????? 0.06 Backend conn. success backend_unhealthy??????????? 0???????? 0.00 Backend conn. not attempted backend_busy???????????????? 0???????? 0.00 Backend conn. too many backend_fail??????????????? 49???????? 0.00 Backend conn. failures backend_reuse????????? 3076713???????? 7.56 Backend conn. reuses backend_toolate?????????? 4640???????? 0.01 Backend conn. was closed backend_recycle??????? 3081374???????? 7.58 Backend conn. recycles backend_retry???????????? 1858???????? 0.00 Backend conn. retry fetch_head????????????????? 63???????? 0.00 Fetch head fetch_length?????????? 1122064???????? 2.76 Fetch with Length fetch_chunked????????? 1959269???????? 4.82 Fetch chunked fetch_eof??????????????????? 0???????? 0.00 Fetch EOF fetch_bad??????????????????? 0???????? 0.00 Fetch had bad headers fetch_close?????????????? 3641???????? 0.01 Fetch wanted close fetch_oldhttp??????????????? 0???????? 0.00 Fetch pre HTTP/1.1 closed fetch_zero?????????????????? 0???????? 0.00 Fetch zero len fetch_failed???????????????? 0???????? 0.00 Fetch failed fetch_1xx??????????????????? 0???????? 0.00 Fetch no body (1xx) fetch_204??????????????????? 0???????? 0.00 Fetch no body (204) fetch_304??????????????????? 0???????? 0.00 Fetch no body (304) n_sess_mem????????????? 100000????????? .?? N struct sess_mem n_sess?????????????????? 99996????????? .?? N struct sess n_object??????????????? 384255????????? .?? N struct object n_vampireobject????????????? 0????????? .?? N unresurrected objects n_objectcore??????????? 384362????????? .?? N struct objectcore n_objecthead??????????? 386502????????? .?? N struct objecthead n_waitinglist??????????? 22165????????? .?? N struct waitinglist n_vbc?????????????????????? 15????????? .?? N struct vbc n_wrk????????????????????? 800????????? .?? N worker threads n_wrk_create?????????????? 800???????? 0.00 N worker threads created n_wrk_failed???????????????? 0???????? 0.00 N worker threads not created n_wrk_max??????????????????? 0???????? 0.00 N worker threads limited n_wrk_lqueue???????????????? 0???????? 0.00 work request queue length n_wrk_queued???????????????? 0???????? 0.00 N queued work requests n_wrk_drop?????????????????? 0???????? 0.00 N dropped work requests n_backend?????????????????? 10????????? .?? N backends n_expired????????????? 2700698????????? .?? N expired objects n_lru_nuked????????????????? 0????????? .?? N LRU nuked objects n_lru_moved??????????? 6742713????????? .?? N LRU moved objects losthdr????????????????????? 0???????? 0.00 HTTP header overflows n_objsendfile??????????????? 0???????? 0.00 Objects sent with sendfile n_objwrite??????????? 33131257??????? 81.45 Objects sent with write n_objoverflow??????????????? 0???????? 0.00 Objects overflowing workspace s_sess????????????????? 569070???????? 1.40 Total Sessions s_req???????????????? 53255991?????? 130.93 Total Requests s_pipe????????????????????? 12???????? 0.00 Total pipe s_pass????????????????????? 84???????? 0.00 Total pass s_fetch??????????????? 3085037???????? 7.58 Total fetch s_hdrbytes???????? 23233920070???? 57118.78 Total header bytes s_bodybytes?????? 511406013002?? 1257251.76 Total body bytes sess_closed???????????? 268273???????? 0.66 Session Closed sess_pipeline??????????????? 0???????? 0.00 Session Pipeline sess_readahead?????????????? 0???????? 0.00 Session Read Ahead sess_linger?????????? 52902973?????? 130.06 Session Linger sess_herd???????????? 19131516??????? 47.03 Session herd shm_records???????? 2474975941????? 6084.54 SHM records shm_writes??????????? 80791017?????? 198.62 SHM writes shm_flushes????????????????? 0???????? 0.00 SHM flushes due to overflow shm_cont???????????????? 23501???????? 0.06 SHM MTX contention shm_cycles???????????????? 889???????? 0.00 SHM cycles through buffer sms_nreq??????????????? 154946???????? 0.38 SMS allocator requests sms_nobj???????????????????? 0????????? .?? SMS outstanding allocations sms_nbytes?????????????????? 0????????? .?? SMS outstanding bytes sms_balloc??????????? 65340505????????? .?? SMS bytes allocated sms_bfree???????????? 65340505????????? .?? SMS bytes freed backend_req??????????? 3100325???????? 7.62 Backend requests made n_vcl??????????????????????? 1???????? 0.00 N vcl total n_vcl_avail????????????????? 1???????? 0.00 N vcl available n_vcl_discard??????????????? 0???????? 0.00 N vcl discarded n_ban??????????????????????? 1????????? .?? N total active bans n_ban_gone?????????????????? 1????????? .?? N total gone bans n_ban_add??????????????????? 1???????? 0.00 N new bans added n_ban_retire???????????????? 0???????? 0.00 N old bans deleted n_ban_obj_test?????????????? 0???????? 0.00 N objects tested n_ban_re_test??????????????? 0???????? 0.00 N regexps tested against n_ban_dups?????????????????? 0???????? 0.00 N duplicate bans removed hcb_nolock??????????? 53113990?????? 130.58 HCB Lookups without lock hcb_lock?????????????? 2909158???????? 7.15 HCB Lookups with lock hcb_insert???????????? 2909154???????? 7.15 HCB Inserts esi_errors?????????????????? 0???????? 0.00 ESI parse errors (unlock) esi_warnings???????????????? 0???????? 0.00 ESI parse warnings (unlock) accept_fail????????????????? 0???????? 0.00 Accept failures client_drop_late???????????? 0???????? 0.00 Connection dropped late uptime????????????????? 406765???????? 1.00 Client uptime dir_dns_lookups????????????? 0???????? 0.00 DNS director lookups dir_dns_failed?????????????? 0???????? 0.00 DNS director failed lookups dir_dns_hit????????????????? 0???????? 0.00 DNS director cached lookups hit dir_dns_cache_full?????????? 0???????? 0.00 DNS director full dnscache vmods??????????????????????? 1????????? .?? Loaded VMODs n_gzip?????????????????????? 0???????? 0.00 Gzip operations n_gunzip???????????????????? 0???????? 0.00 Gunzip operations LCK.sms.creat??????????????? 1???????? 0.00 Created locks LCK.sms.destroy????????????? 0???????? 0.00 Destroyed locks LCK.sms.locks?????????? 464838???????? 1.14 Lock Operations LCK.sms.colls??????????????? 0???????? 0.00 Collisions LCK.smp.creat??????????????? 0???????? 0.00 Created locks LCK.smp.destroy????????????? 0???????? 0.00 Destroyed locks LCK.smp.locks??????????????? 0???????? 0.00 Lock Operations LCK.smp.colls??????????????? 0???????? 0.00 Collisions LCK.sma.creat??????????????? 2???????? 0.00 Created locks LCK.sma.destroy????????????? 0???????? 0.00 Destroyed locks LCK.sma.locks???????? 13844221??????? 34.03 Lock Operations LCK.sma.colls??????????????? 0???????? 0.00 Collisions LCK.smf.creat??????????????? 0???????? 0.00 Created locks LCK.smf.destroy????????????? 0???????? 0.00 Destroyed locks LCK.smf.locks??????????????? 0???????? 0.00 Lock Operations LCK.smf.colls??????????????? 0???????? 0.00 Collisions LCK.hsl.creat??????????????? 0???????? 0.00 Created locks LCK.hsl.destroy????????????? 0???????? 0.00 Destroyed locks LCK.hsl.locks??????????????? 0???????? 0.00 Lock Operations LCK.hsl.colls??????????????? 0???????? 0.00 Collisions LCK.hcb.creat??????????????? 1???????? 0.00 Created locks LCK.hcb.destroy????????????? 0???????? 0.00 Destroyed locks LCK.hcb.locks????????? 5436356??????? 13.36 Lock Operations LCK.hcb.colls??????????????? 0???????? 0.00 Collisions LCK.hcl.creat??????????????? 0???????? 0.00 Created locks LCK.hcl.destroy????????????? 0???????? 0.00 Destroyed locks LCK.hcl.locks??????????????? 0???????? 0.00 Lock Operations LCK.hcl.colls??????????????? 0???????? 0.00 Collisions LCK.vcl.creat??????????????? 1???????? 0.00 Created locks LCK.vcl.destroy????????????? 0???????? 0.00 Destroyed locks LCK.vcl.locks?????????? 165593???????? 0.41 Lock Operations LCK.vcl.colls??????????????? 0???????? 0.00 Collisions LCK.stat.creat?????????????? 1???????? 0.00 Created locks LCK.stat.destroy???????????? 0???????? 0.00 Destroyed locks LCK.stat.locks????????? 569072???????? 1.40 Lock Operations LCK.stat.colls?????????????? 0???????? 0.00 Collisions LCK.sessmem.creat??????????? 1???????? 0.00 Created locks LCK.sessmem.destroy??????????? 0???????? 0.00 Destroyed locks LCK.sessmem.locks??????? 1173472???????? 2.88 Lock Operations LCK.sessmem.colls????????????? 0???????? 0.00 Collisions LCK.wstat.creat??????????????? 1???????? 0.00 Created locks LCK.wstat.destroy????????????? 0???????? 0.00 Destroyed locks LCK.wstat.locks????????? 1334044???????? 3.28 Lock Operations LCK.wstat.colls??????????????? 0???????? 0.00 Collisions LCK.herder.creat?????????????? 1???????? 0.00 Created locks LCK.herder.destroy???????????? 0???????? 0.00 Destroyed locks LCK.herder.locks?????????????? 1???????? 0.00 Lock Operations LCK.herder.colls?????????????? 0???????? 0.00 Collisions LCK.wq.creat?????????????????? 1???????? 0.00 Created locks LCK.wq.destroy???????????????? 0???????? 0.00 Destroyed locks LCK.wq.locks??????????? 39531997??????? 97.19 Lock Operations LCK.wq.colls?????????????????? 0???????? 0.00 Collisions LCK.objhdr.creat???????? 2909259???????? 7.15 Created locks LCK.objhdr.destroy?????? 2522760???????? 6.20 Destroyed locks LCK.objhdr.locks?????? 217895431?????? 535.68 Lock Operations LCK.objhdr.colls?????????????? 0???????? 0.00 Collisions LCK.exp.creat????????????????? 1???????? 0.00 Created locks LCK.exp.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.exp.locks??????????? 6191913??????? 15.22 Lock Operations LCK.exp.colls????????????????? 0???????? 0.00 Collisions LCK.lru.creat????????????????? 2???????? 0.00 Created locks LCK.lru.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.lru.locks??????????? 3084953???????? 7.58 Lock Operations LCK.lru.colls????????????????? 0???????? 0.00 Collisions LCK.cli.creat????????????????? 1???????? 0.00 Created locks LCK.cli.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.cli.locks???????????? 135560???????? 0.33 Lock Operations LCK.cli.colls????????????????? 0???????? 0.00 Collisions LCK.ban.creat????????????????? 1???????? 0.00 Created locks LCK.ban.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.ban.locks??????????? 6191997??????? 15.22 Lock Operations LCK.ban.colls????????????????? 0???????? 0.00 Collisions LCK.vbp.creat????????????????? 1???????? 0.00 Created locks LCK.vbp.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.vbp.locks????????????????? 0???????? 0.00 Lock Operations LCK.vbp.colls????????????????? 0???????? 0.00 Collisions LCK.vbe.creat????????????????? 1???????? 0.00 Created locks LCK.vbe.destroy??????????????? 0???????? 0.00 Destroyed locks LCK.vbe.locks????????????? 47327???????? 0.12 Lock Operations LCK.vbe.colls????????????????? 0???????? 0.00 Collisions LCK.backend.creat???????????? 10???????? 0.00 Created locks LCK.backend.destroy??????????? 0???????? 0.00 Destroyed locks LCK.backend.locks??????? 6233731??????? 15.33 Lock Operations LCK.backend.colls????????????? 0???????? 0.00 Collisions SMA.s0.c_req???????????? 6321249??????? 15.54 Allocator requests SMA.s0.c_fail????????????????? 0???????? 0.00 Allocator failures SMA.s0.c_bytes????? 300614422863??? 739037.09 Bytes allocated SMA.s0.c_freed????? 289845281665??? 712562.00 Bytes freed SMA.s0.g_alloc??????????? 768495????????? .?? Allocations outstanding SMA.s0.g_bytes?????? 10769141198????????? .?? Bytes outstanding SMA.s0.g_space?????? 15000662578????????? .?? Bytes available SMA.Transient.c_req???????? 3656???????? 0.01 Allocator requests SMA.Transient.c_fail?????????? 0???????? 0.00 Allocator failures SMA.Transient.c_bytes???? 43886078?????? 107.89 Bytes allocated SMA.Transient.c_freed???? 43882627?????? 107.88 Bytes freed SMA.Transient.g_alloc??????????? 4????????? .?? Allocations outstanding SMA.Transient.g_bytes???????? 3451????????? .?? Bytes outstanding SMA.Transient.g_space??????????? 0????????? .?? Bytes available VBE.rdsmetadata(10.150.16.2,,80).vcls??????????? 1????????? .?? VCL references VBE.rdsmetadata(10.150.16.2,,80).happy?????????? 0????????? .?? Happy health probes ? Thanks, Lei -------------- next part -------------- An HTML attachment was scrubbed... URL: From leiwang at rhapsody.com Sat Jan 12 00:00:16 2013 From: leiwang at rhapsody.com (Lei Wang) Date: Fri, 11 Jan 2013 16:00:16 -0800 Subject: Varnish 3 client_drop problem In-Reply-To: <1979248734.3258858.1357941694342.JavaMail.webspher@njbbicssmp03> References: <1979248734.3258858.1357941694342.JavaMail.webspher@njbbicssmp03> Message-ID: The only thing interesting in varnishlog is a "Write Error". 885 Debug - "Write error, retval = -1, len = 677, errno = Connection reset by peer" Thanks, Lei On Fri, Jan 11, 2013 at 2:01 PM, themadindian wrote: > What do you see in your varnishlog? > > Sent from my Verizon Wireless 4G LTE Smartphone > > > ----- Reply message ----- > From: "Lei Wang" > To: > Subject: Varnish 3 client_drop problem > Date: Fri, Jan 11, 2013 4:07 PM > > > Hi, > > Need help on one problem. > > After few days, Varnish suddenly shows many client_drop error. Then > response very slowly. I have to restart Varnish to fix it. > Vanish command as following: > /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f > /etc/varnish/varnish80.vcl -T 127.0.0.1:6080 -t 120 -w 1,1000,120 -u > varnish -g varnish -i varnish80 -S /etc/varnish/secret -p > thread_pool_add_delay 2 -p thread_pools 2 -p thread_pool_min 100 -p > thread_pool_max 2000 -p session_linger 50 -p sess_workspace 65536 -p > http_req_hdr_len 8192 -p http_gzip_support off -n varnish80 -s malloc,24G > > Below is the varnishstat result. > client_conn 1050088 2.58 Client connections accepted > *client_drop 481044 1.18 Connection dropped, no > sess/wrk* > client_req 53255991 130.93 Client requests received > cache_hit 49915997 122.71 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 3098734 7.62 Cache misses > backend_conn 23622 0.06 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 49 0.00 Backend conn. failures > backend_reuse 3076713 7.56 Backend conn. reuses > backend_toolate 4640 0.01 Backend conn. was closed > backend_recycle 3081374 7.58 Backend conn. recycles > backend_retry 1858 0.00 Backend conn. retry > fetch_head 63 0.00 Fetch head > fetch_length 1122064 2.76 Fetch with Length > fetch_chunked 1959269 4.82 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 3641 0.01 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 0 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 0 0.00 Fetch no body (204) > fetch_304 0 0.00 Fetch no body (304) > n_sess_mem 100000 . N struct sess_mem > n_sess 99996 . N struct sess > n_object 384255 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 384362 . N struct objectcore > n_objecthead 386502 . N struct objecthead > n_waitinglist 22165 . N struct waitinglist > n_vbc 15 . N struct vbc > n_wrk 800 . N worker threads > n_wrk_create 800 0.00 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 0 0.00 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 10 . N backends > n_expired 2700698 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_moved 6742713 . N LRU moved objects > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 33131257 81.45 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 569070 1.40 Total Sessions > s_req 53255991 130.93 Total Requests > s_pipe 12 0.00 Total pipe > s_pass 84 0.00 Total pass > s_fetch 3085037 7.58 Total fetch > s_hdrbytes 23233920070 57118.78 Total header bytes > s_bodybytes 511406013002 1257251.76 Total body bytes > sess_closed 268273 0.66 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 52902973 130.06 Session Linger > sess_herd 19131516 47.03 Session herd > shm_records 2474975941 6084.54 SHM records > shm_writes 80791017 198.62 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 23501 0.06 SHM MTX contention > shm_cycles 889 0.00 SHM cycles through buffer > sms_nreq 154946 0.38 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 65340505 . SMS bytes allocated > sms_bfree 65340505 . SMS bytes freed > backend_req 3100325 7.62 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_gone 1 . N total gone bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 53113990 130.58 HCB Lookups without lock > hcb_lock 2909158 7.15 HCB Lookups with lock > hcb_insert 2909154 7.15 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 406765 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 1 . Loaded VMODs > n_gzip 0 0.00 Gzip operations > n_gunzip 0 0.00 Gunzip operations > LCK.sms.creat 1 0.00 Created locks > LCK.sms.destroy 0 0.00 Destroyed locks > LCK.sms.locks 464838 1.14 Lock Operations > LCK.sms.colls 0 0.00 Collisions > LCK.smp.creat 0 0.00 Created locks > LCK.smp.destroy 0 0.00 Destroyed locks > LCK.smp.locks 0 0.00 Lock Operations > LCK.smp.colls 0 0.00 Collisions > LCK.sma.creat 2 0.00 Created locks > LCK.sma.destroy 0 0.00 Destroyed locks > LCK.sma.locks 13844221 34.03 Lock Operations > LCK.sma.colls 0 0.00 Collisions > LCK.smf.creat 0 0.00 Created locks > LCK.smf.destroy 0 0.00 Destroyed locks > LCK.smf.locks 0 0.00 Lock Operations > LCK.smf.colls 0 0.00 Collisions > LCK.hsl.creat 0 0.00 Created locks > LCK.hsl.destroy 0 0.00 Destroyed locks > LCK.hsl.locks 0 0.00 Lock Operations > LCK.hsl.colls 0 0.00 Collisions > LCK.hcb.creat 1 0.00 Created locks > LCK.hcb.destroy 0 0.00 Destroyed locks > LCK.hcb.locks 5436356 13.36 Lock Operations > LCK.hcb.colls 0 0.00 Collisions > LCK.hcl.creat 0 0.00 Created locks > LCK.hcl.destroy 0 0.00 Destroyed locks > LCK.hcl.locks 0 0.00 Lock Operations > LCK.hcl.colls 0 0.00 Collisions > LCK.vcl.creat 1 0.00 Created locks > LCK.vcl.destroy 0 0.00 Destroyed locks > LCK.vcl.locks 165593 0.41 Lock Operations > LCK.vcl.colls 0 0.00 Collisions > LCK.stat.creat 1 0.00 Created locks > LCK.stat.destroy 0 0.00 Destroyed locks > LCK.stat.locks 569072 1.40 Lock Operations > LCK.stat.colls 0 0.00 Collisions > LCK.sessmem.creat 1 0.00 Created locks > LCK.sessmem.destroy 0 0.00 Destroyed locks > LCK.sessmem.locks 1173472 2.88 Lock Operations > LCK.sessmem.colls 0 0.00 Collisions > LCK.wstat.creat 1 0.00 Created locks > LCK.wstat.destroy 0 0.00 Destroyed locks > LCK.wstat.locks 1334044 3.28 Lock Operations > LCK.wstat.colls 0 0.00 Collisions > LCK.herder.creat 1 0.00 Created locks > LCK.herder.destroy 0 0.00 Destroyed locks > LCK.herder.locks 1 0.00 Lock Operations > LCK.herder.colls 0 0.00 Collisions > LCK.wq.creat 1 0.00 Created locks > LCK.wq.destroy 0 0.00 Destroyed locks > LCK.wq.locks 39531997 97.19 Lock Operations > LCK.wq.colls 0 0.00 Collisions > LCK.objhdr.creat 2909259 7.15 Created locks > LCK.objhdr.destroy 2522760 6.20 Destroyed locks > LCK.objhdr.locks 217895431 535.68 Lock Operations > LCK.objhdr.colls 0 0.00 Collisions > LCK.exp.creat 1 0.00 Created locks > LCK.exp.destroy 0 0.00 Destroyed locks > LCK.exp.locks 6191913 15.22 Lock Operations > LCK.exp.colls 0 0.00 Collisions > LCK.lru.creat 2 0.00 Created locks > LCK.lru.destroy 0 0.00 Destroyed locks > LCK.lru.locks 3084953 7.58 Lock Operations > LCK.lru.colls 0 0.00 Collisions > LCK.cli.creat 1 0.00 Created locks > LCK.cli.destroy 0 0.00 Destroyed locks > LCK.cli.locks 135560 0.33 Lock Operations > LCK.cli.colls 0 0.00 Collisions > LCK.ban.creat 1 0.00 Created locks > LCK.ban.destroy 0 0.00 Destroyed locks > LCK.ban.locks 6191997 15.22 Lock Operations > LCK.ban.colls 0 0.00 Collisions > LCK.vbp.creat 1 0.00 Created locks > LCK.vbp.destroy 0 0.00 Destroyed locks > LCK.vbp.locks 0 0.00 Lock Operations > LCK.vbp.colls 0 0.00 Collisions > LCK.vbe.creat 1 0.00 Created locks > LCK.vbe.destroy 0 0.00 Destroyed locks > LCK.vbe.locks 47327 0.12 Lock Operations > LCK.vbe.colls 0 0.00 Collisions > LCK.backend.creat 10 0.00 Created locks > LCK.backend.destroy 0 0.00 Destroyed locks > LCK.backend.locks 6233731 15.33 Lock Operations > LCK.backend.colls 0 0.00 Collisions > SMA.s0.c_req 6321249 15.54 Allocator requests > SMA.s0.c_fail 0 0.00 Allocator failures > SMA.s0.c_bytes 300614422863 739037.09 Bytes allocated > SMA.s0.c_freed 289845281665 712562.00 Bytes freed > SMA.s0.g_alloc 768495 . Allocations outstanding > SMA.s0.g_bytes 10769141198 . Bytes outstanding > SMA.s0.g_space 15000662578 . Bytes available > SMA.Transient.c_req 3656 0.01 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 43886078 107.89 Bytes allocated > SMA.Transient.c_freed 43882627 107.88 Bytes freed > SMA.Transient.g_alloc 4 . Allocations outstanding > SMA.Transient.g_bytes 3451 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > VBE.rdsmetadata(10.150.16.2,,80).vcls 1 . VCL > references > VBE.rdsmetadata(10.150.16.2,,80).happy 0 . Happy > health probes > > > Thanks, > > Lei > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaohui.zheng at gmail.com Sat Jan 12 15:45:28 2013 From: shaohui.zheng at gmail.com (Shaohui Zheng) Date: Sat, 12 Jan 2013 23:45:28 +0800 Subject: 503 Error was met frequently In-Reply-To: References: <2465AAEEC8B8A242B26ED5F44BCA805F2605CCD21A@SM-CALA-VXMB04A.swna.wdpr.disney.com> <2465AAEEC8B8A242B26ED5F44BCA805F2605E3A909@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: Thanks Dridi Boukelmoune. In my oppion, if the varnish is not the botterneck, it can take more backends. I set 8 backends, and it seems works. And more, I see many connection failed in my varnishstat 2277628 0.00 67.73 backend_fail - Backend conn. failures After use ulimit to change configure, the issue was fixed. On Fri, Jan 11, 2013 at 4:03 PM, Dridi Boukelmoune < dridi.boukelmoune at zenika.com> wrote: > Hi, > > About the 5 backends limit, you might be interested by this link: > https://www.varnish-cache.org/docs/trunk/phk/backends.html > > PHK basically states that 1000 backends is not an unreasonable number. > > I assume that backends are limited according to available resources > and other factors that can't be established without proper > measurements. > > Best regards, > Dridi > > On Fri, Jan 11, 2013 at 4:24 AM, Shaohui Zheng > wrote: > > I refers other people's configure file on internet, it seems that it is > out > > of date :). > > > > the command varnishlog -b -i TxURL is a great for debugging. > > > > Thanks Rangel Raul & Peter Jacobs. > > > > the new VCL configure file works well, and the 503 error has been gone. > > > > You guys really solve me a big problem. > > > > I have other question, I know many people select LVS(Linux Virtual > Server), > > but I prefer Varnish. I hear from some guys said Varnish can only take 3 > ~ 5 > > backends, if it takes more backends, the performance will downgrade too > > much. Do you have some experience? how many backends can varnish maintain > > with best performance, just according to the experience. > > > > > > I already have 5 backends according to the workloads, but it seems that > the > > backends workload is still too high, I want to add more nodes. Waiting > for > > your advice, thanks. > > > > > > On Wed, Jan 9, 2013 at 11:14 PM, Rangel, Raul > > wrote: > >> > >> varnishlog -b -i TxURL. > > > > > > > > > > > > -- > > Best regards > > Shaohui > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Best regards Shaohui -------------- next part -------------- An HTML attachment was scrubbed... URL: From eschefter at web.de Sun Jan 13 14:32:42 2013 From: eschefter at web.de (eric schefter) Date: Sun, 13 Jan 2013 15:32:42 +0100 Subject: New user account named "varnish" appeared out of the blue In-Reply-To: References: Message-ID: <7531BDA4-EB2D-487D-BAA3-9F0202DA33EC@web.de> On 11.01.2013, at 12:00, varnish-misc-request at varnish-cache.org wrote: > Message: 2 > Date: Thu, 10 Jan 2013 15:48:20 -0800 > From: James Pearson > To: varnish-misc > Subject: Re: New user account named "varnish" appeared out of the blue > Message-ID: <1357861571-sup-8774 at geror.local> > Content-Type: text/plain; charset=UTF-8 > > Excerpts from eric schefter's message of 2013-01-10 11:17:51 -0800: >> Derar usres, >> >> I'm running OS X 10.6.8 >> >> I installed Bitnami, and then uninstalled it. >> One day later, a new user account appeared on my Mac with the name varnish. >> Is it possible that that account was somehow created by the BitNami installation? >> I'm worried that I've been hacked, and can't find any info on this. >> >> Thank you for your help. >> >> Eric > > BitNami includes an installer for Varnish[0]. If you installed, say, > Wordpress[1] or Rails[2], you would've gotten Varnish installed as well as > other stack components. > > I can think of no reason why anyone maliciously accessing your machine would > install Varnish on it. > > HTH, > - P > > [0]: http://wiki.bitnami.org/Components/Varnish > [1]: http://blog.bitnami.org/2012/07/bitnami-wordpress-now-with-varnish.html > [2]: http://blog.bitnami.org/2012/08/rubystack-now-with-rails-327-and-varnish.html > Thank you for the links. I can find no information that says that a user account would be created automatically with the name "varnish", and without the administrator's (my) knowledege. In addition, Bitnami and Wordpress state that varnish is disabled by default, and I didn't enable it. I did install Wordpress as well. Sorry to ask the question again in a different manner: Does varnish create a new user account in os x 10.6.8 (when enabled) which one sees when logging into the computer? If not, I can only think that I'm dealing with another source of this mysterious user account (hacker or other software). Thank you so much for your time, Sincerely, Eric From checker at d6.com Sun Jan 13 19:21:09 2013 From: checker at d6.com (Chris Hecker) Date: Sun, 13 Jan 2013 11:21:09 -0800 Subject: New user account named "varnish" appeared out of the blue In-Reply-To: <7531BDA4-EB2D-487D-BAA3-9F0202DA33EC@web.de> References: <7531BDA4-EB2D-487D-BAA3-9F0202DA33EC@web.de> Message-ID: <50F30925.3020607@d6.com> The CentOS varnish installation makes a varnish user, I would assume it'd be the same on every platform. Chris On 2013-01-13 06:32, eric schefter wrote: > > On 11.01.2013, at 12:00, varnish-misc-request at varnish-cache.org wrote: > >> Message: 2 >> Date: Thu, 10 Jan 2013 15:48:20 -0800 >> From: James Pearson >> To: varnish-misc >> Subject: Re: New user account named "varnish" appeared out of the blue >> Message-ID: <1357861571-sup-8774 at geror.local> >> Content-Type: text/plain; charset=UTF-8 >> >> Excerpts from eric schefter's message of 2013-01-10 11:17:51 -0800: >>> Derar usres, >>> >>> I'm running OS X 10.6.8 >>> >>> I installed Bitnami, and then uninstalled it. >>> One day later, a new user account appeared on my Mac with the name varnish. >>> Is it possible that that account was somehow created by the BitNami installation? >>> I'm worried that I've been hacked, and can't find any info on this. >>> >>> Thank you for your help. >>> >>> Eric >> >> BitNami includes an installer for Varnish[0]. If you installed, say, >> Wordpress[1] or Rails[2], you would've gotten Varnish installed as well as >> other stack components. >> >> I can think of no reason why anyone maliciously accessing your machine would >> install Varnish on it. >> >> HTH, >> - P >> >> [0]: http://wiki.bitnami.org/Components/Varnish >> [1]: http://blog.bitnami.org/2012/07/bitnami-wordpress-now-with-varnish.html >> [2]: http://blog.bitnami.org/2012/08/rubystack-now-with-rails-327-and-varnish.html >> > > Thank you for the links. > > I can find no information that says that a user account would be created automatically with the name "varnish", and without the administrator's (my) knowledege. In addition, Bitnami and Wordpress state that varnish is disabled by default, and I didn't enable it. > I did install Wordpress as well. > > Sorry to ask the question again in a different manner: > > Does varnish create a new user account in os x 10.6.8 (when enabled) which one sees when logging into the computer? > > If not, I can only think that I'm dealing with another source of this mysterious user account (hacker or other software). > > Thank you so much for your time, > > Sincerely, > > Eric > > > > > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From m.streuhofer at FAZ.DE Sun Jan 13 19:52:57 2013 From: m.streuhofer at FAZ.DE (Streuhofer, Manuel) Date: Sun, 13 Jan 2013 19:52:57 +0000 Subject: Varnish 3 client_drop problem In-Reply-To: References: Message-ID: I'm gonna guess here that the exact Version of Varnish you are using is 3.0.0 This version has a bug which causes your sessions to be counted upwards only. Once you hit 100.000 sessions it will start dropping clients. I'm surprised you didn't run into problems way before that. Each session means one open file handle. Here's your problem: n_sess_mem 100000 . N struct sess_mem n_sess 99996 . N struct sess There are two solutions. The better one would be to upgrade since this was not the only ugly bug in this version. You'll be better of. The quick one is to turn off hyper threading. See: https://www.varnish-cache.org/trac/ticket/897 ________________________________ From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] on behalf of Lei Wang [leiwang at rhapsody.com] Sent: Friday, January 11, 2013 10:07 PM To: varnish-misc at varnish-cache.org Subject: Varnish 3 client_drop problem Hi, Need help on one problem. After few days, Varnish suddenly shows many client_drop error. Then response very slowly. I have to restart Varnish to fix it. Vanish command as following: /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f /etc/varnish/varnish80.vcl -T 127.0.0.1:6080 -t 120 -w 1,1000,120 -u varnish -g varnish -i varnish80 -S /etc/varnish/secret -p thread_pool_add_delay 2 -p thread_pools 2 -p thread_pool_min 100 -p thread_pool_max 2000 -p session_linger 50 -p sess_workspace 65536 -p http_req_hdr_len 8192 -p http_gzip_support off -n varnish80 -s malloc,24G Below is the varnishstat result. client_conn 1050088 2.58 Client connections accepted client_drop 481044 1.18 Connection dropped, no sess/wrk client_req 53255991 130.93 Client requests received cache_hit 49915997 122.71 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 3098734 7.62 Cache misses backend_conn 23622 0.06 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 49 0.00 Backend conn. failures backend_reuse 3076713 7.56 Backend conn. reuses backend_toolate 4640 0.01 Backend conn. was closed backend_recycle 3081374 7.58 Backend conn. recycles backend_retry 1858 0.00 Backend conn. retry fetch_head 63 0.00 Fetch head fetch_length 1122064 2.76 Fetch with Length fetch_chunked 1959269 4.82 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 3641 0.01 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 100000 . N struct sess_mem n_sess 99996 . N struct sess n_object 384255 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 384362 . N struct objectcore n_objecthead 386502 . N struct objecthead n_waitinglist 22165 . N struct waitinglist n_vbc 15 . N struct vbc n_wrk 800 . N worker threads n_wrk_create 800 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 0 0.00 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 10 . N backends n_expired 2700698 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 6742713 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 33131257 81.45 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 569070 1.40 Total Sessions s_req 53255991 130.93 Total Requests s_pipe 12 0.00 Total pipe s_pass 84 0.00 Total pass s_fetch 3085037 7.58 Total fetch s_hdrbytes 23233920070 57118.78 Total header bytes s_bodybytes 511406013002 1257251.76 Total body bytes sess_closed 268273 0.66 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 52902973 130.06 Session Linger sess_herd 19131516 47.03 Session herd shm_records 2474975941 6084.54 SHM records shm_writes 80791017 198.62 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 23501 0.06 SHM MTX contention shm_cycles 889 0.00 SHM cycles through buffer sms_nreq 154946 0.38 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 65340505 . SMS bytes allocated sms_bfree 65340505 . SMS bytes freed backend_req 3100325 7.62 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 53113990 130.58 HCB Lookups without lock hcb_lock 2909158 7.15 HCB Lookups with lock hcb_insert 2909154 7.15 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 406765 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 1 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 0 0.00 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 464838 1.14 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 13844221 34.03 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 5436356 13.36 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 165593 0.41 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 569072 1.40 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 1173472 2.88 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 1334044 3.28 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1 0.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 1 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 39531997 97.19 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 2909259 7.15 Created locks LCK.objhdr.destroy 2522760 6.20 Destroyed locks LCK.objhdr.locks 217895431 535.68 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 6191913 15.22 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 3084953 7.58 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 135560 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 6191997 15.22 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 47327 0.12 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 10 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 6233731 15.33 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 6321249 15.54 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 300614422863 739037.09 Bytes allocated SMA.s0.c_freed 289845281665 712562.00 Bytes freed SMA.s0.g_alloc 768495 . Allocations outstanding SMA.s0.g_bytes 10769141198 . Bytes outstanding SMA.s0.g_space 15000662578 . Bytes available SMA.Transient.c_req 3656 0.01 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 43886078 107.89 Bytes allocated SMA.Transient.c_freed 43882627 107.88 Bytes freed SMA.Transient.g_alloc 4 . Allocations outstanding SMA.Transient.g_bytes 3451 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.rdsmetadata(10.150.16.2,,80).vcls 1 . VCL references VBE.rdsmetadata(10.150.16.2,,80).happy 0 . Happy health probes Thanks, Lei -------------- next part -------------- An HTML attachment was scrubbed... URL: From nick.tailor at gmail.com Mon Jan 14 00:29:01 2013 From: nick.tailor at gmail.com (nick tailor) Date: Sun, 13 Jan 2013 16:29:01 -0800 Subject: New user account named "varnish" appeared out of the blue In-Reply-To: <436581A3-FDBB-4E79-AC25-45562B16D013@web.de> References: <436581A3-FDBB-4E79-AC25-45562B16D013@web.de> Message-ID: Oi On 2013-01-10 11:19 AM, "eric schefter" wrote: > Derar usres, > > I'm running OS X 10.6.8 > > I installed Bitnami, and then uninstalled it. > One day later, a new user account appeared on my Mac with the name * > varnish*. > Is it possible that that account was somehow created by the BitNami > installation? > I'm worried that I've been hacked, and can't find any info on this. > > Thank you for your help. > > Eric > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Mon Jan 14 21:40:58 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Mon, 14 Jan 2013 16:40:58 -0500 Subject: banning configuration question Message-ID: Hi all, I'm trying to figure out how to do the following and I'm slightly confused by some of the documentation around banning. on a PUT that returns a 200 to a url matching /foo/{\d+}/an_action or /foo/{\d+}/another_action I want to ban /foo/{\d+} in a smart ban friendly way in my vcl. I'm unsure as to where vcl_fetch, vcl_recv that I want to hook into and how to ban in a smart ban friendly fashion just the /foo/{\d+} Any help is greatly appreciated. Thanks, Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From leiwang at rhapsody.com Mon Jan 14 22:58:23 2013 From: leiwang at rhapsody.com (Lei Wang) Date: Mon, 14 Jan 2013 14:58:23 -0800 Subject: Varnish 3 client_drop problem In-Reply-To: References: Message-ID: Thanks for the information. Looks like there are memory leak at some place. Varnish version is 3.0.3. Lei On Sun, Jan 13, 2013 at 11:52 AM, Streuhofer, Manuel wrote: > I'm gonna guess here that the exact Version of Varnish you are using is > 3.0.0 > This version has a bug which causes your sessions to be counted upwards > only. > Once you hit 100.000 sessions it will start dropping clients. > I'm surprised you didn't run into problems way before that. Each session > means one open file handle. > > Here's your problem: > > > n_sess_mem 100000 . N struct sess_mem > n_sess 99996 . N struct sess > > There are two solutions. The better one would be to upgrade since this was > not the only ugly bug in this version. You'll be better of. The quick one > is to turn off hyper threading. See: > https://www.varnish-cache.org/trac/ticket/897 > > > ------------------------------ > *From:* varnish-misc-bounces at varnish-cache.org [ > varnish-misc-bounces at varnish-cache.org] on behalf of Lei Wang [ > leiwang at rhapsody.com] > *Sent:* Friday, January 11, 2013 10:07 PM > > *To:* varnish-misc at varnish-cache.org > *Subject:* Varnish 3 client_drop problem > > Hi, > > Need help on one problem. > > After few days, Varnish suddenly shows many client_drop error. Then > response very slowly. I have to restart Varnish to fix it. > Vanish command as following: > /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f > /etc/varnish/varnish80.vcl -T 127.0.0.1:6080 -t 120 -w 1,1000,120 -u > varnish -g varnish -i varnish80 -S /etc/varnish/secret -p > thread_pool_add_delay 2 -p thread_pools 2 -p thread_pool_min 100 -p > thread_pool_max 2000 -p session_linger 50 -p sess_workspace 65536 -p > http_req_hdr_len 8192 -p http_gzip_support off -n varnish80 -s malloc,24G > > Below is the varnishstat result. > client_conn 1050088 2.58 Client connections accepted > *client_drop 481044 1.18 Connection dropped, no > sess/wrk* > client_req 53255991 130.93 Client requests received > cache_hit 49915997 122.71 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 3098734 7.62 Cache misses > backend_conn 23622 0.06 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 49 0.00 Backend conn. failures > backend_reuse 3076713 7.56 Backend conn. reuses > backend_toolate 4640 0.01 Backend conn. was closed > backend_recycle 3081374 7.58 Backend conn. recycles > backend_retry 1858 0.00 Backend conn. retry > fetch_head 63 0.00 Fetch head > fetch_length 1122064 2.76 Fetch with Length > fetch_chunked 1959269 4.82 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 3641 0.01 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 0 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 0 0.00 Fetch no body (204) > fetch_304 0 0.00 Fetch no body (304) > n_sess_mem 100000 . N struct sess_mem > n_sess 99996 . N struct sess > n_object 384255 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 384362 . N struct objectcore > n_objecthead 386502 . N struct objecthead > n_waitinglist 22165 . N struct waitinglist > n_vbc 15 . N struct vbc > n_wrk 800 . N worker threads > n_wrk_create 800 0.00 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 0 0.00 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 10 . N backends > n_expired 2700698 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_moved 6742713 . N LRU moved objects > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 33131257 81.45 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 569070 1.40 Total Sessions > s_req 53255991 130.93 Total Requests > s_pipe 12 0.00 Total pipe > s_pass 84 0.00 Total pass > s_fetch 3085037 7.58 Total fetch > s_hdrbytes 23233920070 57118.78 Total header bytes > s_bodybytes 511406013002 1257251.76 Total body bytes > sess_closed 268273 0.66 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 52902973 130.06 Session Linger > sess_herd 19131516 47.03 Session herd > shm_records 2474975941 6084.54 SHM records > shm_writes 80791017 198.62 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 23501 0.06 SHM MTX contention > shm_cycles 889 0.00 SHM cycles through buffer > sms_nreq 154946 0.38 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 65340505 . SMS bytes allocated > sms_bfree 65340505 . SMS bytes freed > backend_req 3100325 7.62 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_gone 1 . N total gone bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 53113990 130.58 HCB Lookups without lock > hcb_lock 2909158 7.15 HCB Lookups with lock > hcb_insert 2909154 7.15 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 406765 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 1 . Loaded VMODs > n_gzip 0 0.00 Gzip operations > n_gunzip 0 0.00 Gunzip operations > LCK.sms.creat 1 0.00 Created locks > LCK.sms.destroy 0 0.00 Destroyed locks > LCK.sms.locks 464838 1.14 Lock Operations > LCK.sms.colls 0 0.00 Collisions > LCK.smp.creat 0 0.00 Created locks > LCK.smp.destroy 0 0.00 Destroyed locks > LCK.smp.locks 0 0.00 Lock Operations > LCK.smp.colls 0 0.00 Collisions > LCK.sma.creat 2 0.00 Created locks > LCK.sma.destroy 0 0.00 Destroyed locks > LCK.sma.locks 13844221 34.03 Lock Operations > LCK.sma.colls 0 0.00 Collisions > LCK.smf.creat 0 0.00 Created locks > LCK.smf.destroy 0 0.00 Destroyed locks > LCK.smf.locks 0 0.00 Lock Operations > LCK.smf.colls 0 0.00 Collisions > LCK.hsl.creat 0 0.00 Created locks > LCK.hsl.destroy 0 0.00 Destroyed locks > LCK.hsl.locks 0 0.00 Lock Operations > LCK.hsl.colls 0 0.00 Collisions > LCK.hcb.creat 1 0.00 Created locks > LCK.hcb.destroy 0 0.00 Destroyed locks > LCK.hcb.locks 5436356 13.36 Lock Operations > LCK.hcb.colls 0 0.00 Collisions > LCK.hcl.creat 0 0.00 Created locks > LCK.hcl.destroy 0 0.00 Destroyed locks > LCK.hcl.locks 0 0.00 Lock Operations > LCK.hcl.colls 0 0.00 Collisions > LCK.vcl.creat 1 0.00 Created locks > LCK.vcl.destroy 0 0.00 Destroyed locks > LCK.vcl.locks 165593 0.41 Lock Operations > LCK.vcl.colls 0 0.00 Collisions > LCK.stat.creat 1 0.00 Created locks > LCK.stat.destroy 0 0.00 Destroyed locks > LCK.stat.locks 569072 1.40 Lock Operations > LCK.stat.colls 0 0.00 Collisions > LCK.sessmem.creat 1 0.00 Created locks > LCK.sessmem.destroy 0 0.00 Destroyed locks > LCK.sessmem.locks 1173472 2.88 Lock Operations > LCK.sessmem.colls 0 0.00 Collisions > LCK.wstat.creat 1 0.00 Created locks > LCK.wstat.destroy 0 0.00 Destroyed locks > LCK.wstat.locks 1334044 3.28 Lock Operations > LCK.wstat.colls 0 0.00 Collisions > LCK.herder.creat 1 0.00 Created locks > LCK.herder.destroy 0 0.00 Destroyed locks > LCK.herder.locks 1 0.00 Lock Operations > LCK.herder.colls 0 0.00 Collisions > LCK.wq.creat 1 0.00 Created locks > LCK.wq.destroy 0 0.00 Destroyed locks > LCK.wq.locks 39531997 97.19 Lock Operations > LCK.wq.colls 0 0.00 Collisions > LCK.objhdr.creat 2909259 7.15 Created locks > LCK.objhdr.destroy 2522760 6.20 Destroyed locks > LCK.objhdr.locks 217895431 535.68 Lock Operations > LCK.objhdr.colls 0 0.00 Collisions > LCK.exp.creat 1 0.00 Created locks > LCK.exp.destroy 0 0.00 Destroyed locks > LCK.exp.locks 6191913 15.22 Lock Operations > LCK.exp.colls 0 0.00 Collisions > LCK.lru.creat 2 0.00 Created locks > LCK.lru.destroy 0 0.00 Destroyed locks > LCK.lru.locks 3084953 7.58 Lock Operations > LCK.lru.colls 0 0.00 Collisions > LCK.cli.creat 1 0.00 Created locks > LCK.cli.destroy 0 0.00 Destroyed locks > LCK.cli.locks 135560 0.33 Lock Operations > LCK.cli.colls 0 0.00 Collisions > LCK.ban.creat 1 0.00 Created locks > LCK.ban.destroy 0 0.00 Destroyed locks > LCK.ban.locks 6191997 15.22 Lock Operations > LCK.ban.colls 0 0.00 Collisions > LCK.vbp.creat 1 0.00 Created locks > LCK.vbp.destroy 0 0.00 Destroyed locks > LCK.vbp.locks 0 0.00 Lock Operations > LCK.vbp.colls 0 0.00 Collisions > LCK.vbe.creat 1 0.00 Created locks > LCK.vbe.destroy 0 0.00 Destroyed locks > LCK.vbe.locks 47327 0.12 Lock Operations > LCK.vbe.colls 0 0.00 Collisions > LCK.backend.creat 10 0.00 Created locks > LCK.backend.destroy 0 0.00 Destroyed locks > LCK.backend.locks 6233731 15.33 Lock Operations > LCK.backend.colls 0 0.00 Collisions > SMA.s0.c_req 6321249 15.54 Allocator requests > SMA.s0.c_fail 0 0.00 Allocator failures > SMA.s0.c_bytes 300614422863 739037.09 Bytes allocated > SMA.s0.c_freed 289845281665 712562.00 Bytes freed > SMA.s0.g_alloc 768495 . Allocations outstanding > SMA.s0.g_bytes 10769141198 . Bytes outstanding > SMA.s0.g_space 15000662578 . Bytes available > SMA.Transient.c_req 3656 0.01 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 43886078 107.89 Bytes allocated > SMA.Transient.c_freed 43882627 107.88 Bytes freed > SMA.Transient.g_alloc 4 . Allocations outstanding > SMA.Transient.g_bytes 3451 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > VBE.rdsmetadata(10.150.16.2,,80).vcls 1 . VCL > references > VBE.rdsmetadata(10.150.16.2,,80).happy 0 . Happy > health probes > > > Thanks, > > Lei > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tharanga.abeyseela at gmail.com Thu Jan 17 04:24:44 2013 From: tharanga.abeyseela at gmail.com (Tharanga Abeyseela) Date: Thu, 17 Jan 2013 15:24:44 +1100 Subject: F5 loadbalancer between varnish and web servers Message-ID: Hi Guys, I'm configuring a varnish instance to send traffic to a f5 loadbalancer Virtual Address, and F5 will do the loadbalancing work. (im not using varnish to do loadbalancing) my domains are www.stage-abc.com www.stage.xyz.com im running two web servers.(name based virtual hosting on both nodes). node1, node2. varnish instance will send traffic to loadbalancer and loadbalancer will send traffic to either node1 or node2. my question is how should i define the backend configuration and send host headers to f5 loadbalancer. (i need to remove 80,443 or www prefixes this is my config - varnish version 3 /etc/sysconfig/varnish (im running centos) i have changed the below section. (to listen to 80,443) DAEMON_OPTS="-a :80,:443 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -u varnish -g varnish \ -S /etc/varnish/secret \ -s file,/var/lib/varnish/varnish_storage.bin,1G" LISTEN ADDRESS - X.X.X.X others are default settings. . ============================== My VCL looks like this /etc/varnish/default.vcl backend f5 { .host = "10.111.112.136"; .} if (! req.http.Host) { error 404 "Need a host header"; } set req.http.Host = regsub(req.http.Host, "^www\.", ""); set req.http.Host = regsub(req.http.Host, ":80$", ""); set req.http.Host = regsub(req.http.Host, ":443$", ""); if (req.http.Host == "stage-abc.com") { include "/etc/varnish/abc.com.vcl"; } if (req.http.Host == "stage-xyz.com") { include "/etc/varnish/xyz.com.vcl"; } do i actually need a abc.com.vcl /xyz.com.vcl or do i need to forward the host header to F5 VIP If i need to forward the header how should i do it inside vcl please let me know. Cheers, Tharanga From pprocacci at datapipe.com Thu Jan 17 04:37:09 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 16 Jan 2013 22:37:09 -0600 Subject: F5 loadbalancer between varnish and web servers In-Reply-To: References: Message-ID: <20130117043709.GD2564@nat.myhome> Firstly, you cannot terminate ssl connections with varnish. You need to consider using a web server or ssl wrapper like stunnel/pound/etc. You don't need to explicitly forward anything. Your modifications of the header will be seen by the receiving device. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From tharanga.abeyseela at gmail.com Thu Jan 17 04:52:51 2013 From: tharanga.abeyseela at gmail.com (Tharanga Abeyseela) Date: Thu, 17 Jan 2013 15:52:51 +1100 Subject: F5 loadbalancer between varnish and web servers In-Reply-To: <20130117043709.GD2564@nat.myhome> References: <20130117043709.GD2564@nat.myhome> Message-ID: actually i have a f5 loadbalncer infront of varnish, thats the ssl termination point . thanks a lot. On Thu, Jan 17, 2013 at 3:37 PM, Paul A. Procacci wrote: > Firstly, you cannot terminate ssl connections with varnish. You need > to consider using a web server or ssl wrapper like stunnel/pound/etc. > > You don't need to explicitly forward anything. Your modifications of > the header will be seen by the receiving device. > > ~Paul > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From info at massivescale.net Sat Jan 19 13:34:30 2013 From: info at massivescale.net (MassiveScale.net) Date: Sat, 19 Jan 2013 14:34:30 +0100 Subject: Editing Varnish wiki Message-ID: <50FAA0E6.2090106@massivescale.net> Hello, I wanted to update a VCL snippet at Varnish wiki. It looks like I'm not able to do it after registration and logging in. Is it a policy that you don't accept edits from outside or am I doing something wrong? -- Andrzej Godziuk http://massivescale.net/ From phk at phk.freebsd.dk Sat Jan 19 14:06:54 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 19 Jan 2013 14:06:54 +0000 Subject: Editing Varnish wiki In-Reply-To: <50FAA0E6.2090106@massivescale.net> References: <50FAA0E6.2090106@massivescale.net> Message-ID: <1781.1358604414@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1 -------- In message <50FAA0E6.2090106 at massivescale.net>, "MassiveScale.net" writes: >Hello, > >I wanted to update a VCL snippet at Varnish wiki. It looks like I'm not >able to do it after registration and logging in. Is it a policy that you >don't accept edits from outside or am I doing something wrong? It's to avoid spam. Mail me your username and I'll give you a wiki-bit -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From themadindian at yahoo.com Sun Jan 20 15:54:15 2013 From: themadindian at yahoo.com (Mad Indian) Date: Sun, 20 Jan 2013 07:54:15 -0800 (PST) Subject: Varnish 3 - choose backend based on header field. In-Reply-To: References: Message-ID: <1358697255.55866.YahooMailNeo@web163902.mail.gq1.yahoo.com> You could try with two backends named app1 and app2 and app_dir as the default director if (req.http.X-AppServer?== "app1") { ? ? ? ? ? ? ? ? set req.backend = app1; ? ? ? ? } else if (req.http.X-AppServer?== "app2") { ? ? ? ? ? ? ? ? set req.backend = app2; ? ? ? ? } else { ? ? ? ? ? ? ? ? set req.backend = app_dir; ? ? ? ? } ________________________________ From: Rafa? Radecki To: varnish-misc at varnish-cache.org Sent: Friday, January 11, 2013 4:16 AM Subject: Varnish 3 - choose backend based on header field. Hi all. I currently use varnish 3 in my environment. I use client director with two backends. I use set client.identity = client.ip to connect client session with backend. I would like to have the possibility to choose specific backend (first or second) based on the presence of a header in http request. If my request would have a header X-AppServer = app1 varnish should forward the request to the first application server if X-AppServer = app2 to the second and if there would be no such header it should use default client director. Is it possible in varnish 3? Best regards, Rafal. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Sun Jan 20 20:15:02 2013 From: bedis9 at gmail.com (Baptiste) Date: Sun, 20 Jan 2013 21:15:02 +0100 Subject: F5 loadbalancer between varnish and web servers In-Reply-To: References: <20130117043709.GD2564@nat.myhome> Message-ID: So why not using F5 caching ability???? Just not make any sense unless you have a good reason I'm curious to know. cheers On Thu, Jan 17, 2013 at 5:52 AM, Tharanga Abeyseela wrote: > actually i have a f5 loadbalncer infront of varnish, thats the ssl > termination point . > > thanks a lot. > > On Thu, Jan 17, 2013 at 3:37 PM, Paul A. Procacci > wrote: >> Firstly, you cannot terminate ssl connections with varnish. You need >> to consider using a web server or ssl wrapper like stunnel/pound/etc. >> >> You don't need to explicitly forward anything. Your modifications of >> the header will be seen by the receiving device. >> >> ~Paul >> >> ________________________________ >> >> This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From emilio at antispam.it Sun Jan 20 20:58:49 2013 From: emilio at antispam.it (emilio brambilla) Date: Sun, 20 Jan 2013 21:58:49 +0100 Subject: F5 loadbalancer between varnish and web servers In-Reply-To: References: <20130117043709.GD2564@nat.myhome> Message-ID: <50FC5A89.2010106@antispam.it> hello, On 2013/01/20 9:15 PM, Baptiste wrote: > So why not using F5 caching ability???? > Just not make any sense unless you have a good reason I'm curious to know. maybe because caching on the F5 costs at least 10 to 100 times more than on the varnish? if you have a small site (with small I mean a site that will never use all the power of the F5s you own) caching on the F5 may be a good solution, but if your site is huge, buying more F5s (or bigger F5) is surely more expensive than buying the number of servers you need for the varnish. maybe the next step will be to replace F5 with other opensource software (like vls) even for the load balancing... this will happends when you will realize the F5 yearly maintenance costs more than the hardware (+3years of maintenance) where to run lvs on... but that's another story :-) -- bye, emilio brambilla From friedrich.locke at gmail.com Wed Jan 23 12:37:20 2013 From: friedrich.locke at gmail.com (Friedrich Locke) Date: Wed, 23 Jan 2013 10:37:20 -0200 Subject: simple configuration Message-ID: Hi folks, i have just installed varnish and this is my first message to the list. I did i very simple configuration. I edited /etc/varnish/default.vcl. I included the following lines: backend default { .host = "aton.cpd.edu.br"; .port = "80"; } I stress loaded the my site, now being served by varnish, but i noticed no performance improvement. I would like to do a minimal configuration just to see how fast varnish could do. Is there something more i need to configure in order to get the taste of varnish ? Thanks in advance. From shib4u at gmail.com Wed Jan 23 13:41:32 2013 From: shib4u at gmail.com (Shibashish) Date: Wed, 23 Jan 2013 19:11:32 +0530 Subject: simple configuration In-Reply-To: References: Message-ID: On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke wrote: > Hi folks, > > i have just installed varnish and this is my first message to the list. > I did i very simple configuration. I edited /etc/varnish/default.vcl. > > I included the following lines: > > backend default { > .host = "aton.cpd.edu.br"; > .port = "80"; > } > > I stress loaded the my site, now being served by varnish, but i > noticed no performance improvement. > I would like to do a minimal configuration just to see how fast > varnish could do. > Is there something more i need to configure in order to get the taste > of varnish ? > > Thanks in advance. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > You need to describe more... - What test tool are you using? - Do you know if the pages are being served by Varnish or being passed on to your webserver? - If varnish is being MISSed, then you have to find out reasons for the same and rectify ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From enno at tty.nl Wed Jan 23 13:57:30 2013 From: enno at tty.nl (Enno van Amerongen) Date: Wed, 23 Jan 2013 14:57:30 +0100 Subject: simple configuration In-Reply-To: Message-ID: <20130123135730.35dd8fd2@groupware.tty.nl> From: Shibashish [mailto:shib4u at gmail.com] Subject: Re: simple configuration On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke wrote: Hi folks, i have just installed varnish and this is my first message to the list. I did i very simple configuration. I edited /etc/varnish/default.vcl. I included the following lines: backend default { .host = "aton.cpd.edu.br"; .port = "80"; } I stress loaded the my site, now being served by varnish, but i noticed no performance improvement. I would like to do a minimal configuration just to see how fast varnish could do. Is there something more i need to configure in order to get the taste of varnish ? Yes, you need to check and fix the http headers your backend sends, by default, Varnish honors them. https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration Enno -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrich.locke at gmail.com Wed Jan 23 16:50:14 2013 From: friedrich.locke at gmail.com (Friedrich Locke) Date: Wed, 23 Jan 2013 14:50:14 -0200 Subject: simple configuration In-Reply-To: <20130123135730.35dd8fd2@groupware.tty.nl> References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: Sorry, but i am getting this error too: Message from VCC-compiler: Invalid return "pass" ('input' Line 173 Pos 25) return (pass); ------------------------####-- ...in subroutine "vcl_fetch" ('input' Line 164 Pos 5) sub vcl_fetch { ----#########-- ...which is the "vcl_fetch" method Legal returns are: "deliver" "error" "hit_for_pass" "restart" Running VCC-compiler failed, exit 1 VCL compilation failed # Have any ideia ? On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: > From: Shibashish [mailto:shib4u at gmail.com] > Subject: Re: simple configuration > > > On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke > wrote: >> >> Hi folks, >> >> i have just installed varnish and this is my first message to the list. >> I did i very simple configuration. I edited /etc/varnish/default.vcl. >> >> I included the following lines: >> >> backend default { >> .host = "aton.cpd.edu.br"; >> .port = "80"; >> } >> >> I stress loaded the my site, now being served by varnish, but i >> noticed no performance improvement. >> I would like to do a minimal configuration just to see how fast >> varnish could do. >> Is there something more i need to configure in order to get the taste >> of varnish ? > > Yes, you need to check and fix the http headers your backend sends, by > default, Varnish honors them. > > https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration > > Enno From dridi.boukelmoune at zenika.com Wed Jan 23 16:57:03 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Wed, 23 Jan 2013 17:57:03 +0100 Subject: simple configuration In-Reply-To: References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: Hi Friedrich, You cannot issue a pass in vcl_fetch, are you looking for hit_for_pass ? https://raw.github.com/varnish/Varnish-Book/master/ui/img/vcl.png Regards, Dridi On Wed, Jan 23, 2013 at 5:50 PM, Friedrich Locke wrote: > Sorry, > > but i am getting this error too: > > Message from VCC-compiler: > Invalid return "pass" > ('input' Line 173 Pos 25) > return (pass); > ------------------------####-- > > > ...in subroutine "vcl_fetch" > ('input' Line 164 Pos 5) > sub vcl_fetch { > ----#########-- > > > ...which is the "vcl_fetch" method > Legal returns are: "deliver" "error" "hit_for_pass" "restart" > Running VCC-compiler failed, exit 1 > > VCL compilation failed > # > > > Have any ideia ? > > On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: >> From: Shibashish [mailto:shib4u at gmail.com] >> Subject: Re: simple configuration >> >> >> On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke >> wrote: >>> >>> Hi folks, >>> >>> i have just installed varnish and this is my first message to the list. >>> I did i very simple configuration. I edited /etc/varnish/default.vcl. >>> >>> I included the following lines: >>> >>> backend default { >>> .host = "aton.cpd.edu.br"; >>> .port = "80"; >>> } >>> >>> I stress loaded the my site, now being served by varnish, but i >>> noticed no performance improvement. >>> I would like to do a minimal configuration just to see how fast >>> varnish could do. >>> Is there something more i need to configure in order to get the taste >>> of varnish ? >> >> Yes, you need to check and fix the http headers your backend sends, by >> default, Varnish honors them. >> >> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration >> >> Enno > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From friedrich.locke at gmail.com Wed Jan 23 17:11:37 2013 From: friedrich.locke at gmail.com (Friedrich Locke) Date: Wed, 23 Jan 2013 15:11:37 -0200 Subject: simple configuration In-Reply-To: References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: What i am trying is below (based on https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration) : sub vcl_recv { set req.http.host = "www.centev.ufv.br"; set req.backend = centev; if (req.request == "GET" && req.url ~ "^/static/") { unset req.http.cookie; unset req.http.Authorization; return (lookup); } if (req.request == "POST") { return (pass); } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE" ) { # Non-RFC2616 or CONNECT which is weird # return (pass); } if (req.http.Authorization) { # Not cacheable by default # return (pass); } return (lookup); } sub vcl_fetch { if (req.request == "GET" && req.url ~ "^/static" ) { unset obj.http.Set-Cookie; set obj.ttl = 30m; return (deliver); } if (obj.status >= 300) { return (hit_for_pass); } if (!obj.cacheable) { return (hit_for_pass); } if (obj.http.Set-Cookie) { return (hit_for_pass); } if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private") { return (hit_for_pass); } if (obj.http.Cache-Control ~ "max-age") { unset obj.http.Set-Cookie; return (deliver); } return (hit_for_pass); } Message from VCC-compiler: Symbol not found: 'obj.cacheable' (expected type BOOL): ('input' Line 176 Pos 14) if (!obj.cacheable) { -------------#############--- Running VCC-compiler failed, exit 1 VCL compilation failed # Any advice ? PS: varnish-3.0.2 On Wed, Jan 23, 2013 at 2:57 PM, Dridi Boukelmoune wrote: > Hi Friedrich, > > You cannot issue a pass in vcl_fetch, are you looking for hit_for_pass ? > https://raw.github.com/varnish/Varnish-Book/master/ui/img/vcl.png > > Regards, > Dridi > > On Wed, Jan 23, 2013 at 5:50 PM, Friedrich Locke > wrote: >> Sorry, >> >> but i am getting this error too: >> >> Message from VCC-compiler: >> Invalid return "pass" >> ('input' Line 173 Pos 25) >> return (pass); >> ------------------------####-- >> >> >> ...in subroutine "vcl_fetch" >> ('input' Line 164 Pos 5) >> sub vcl_fetch { >> ----#########-- >> >> >> ...which is the "vcl_fetch" method >> Legal returns are: "deliver" "error" "hit_for_pass" "restart" >> Running VCC-compiler failed, exit 1 >> >> VCL compilation failed >> # >> >> >> Have any ideia ? >> >> On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: >>> From: Shibashish [mailto:shib4u at gmail.com] >>> Subject: Re: simple configuration >>> >>> >>> On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke >>> wrote: >>>> >>>> Hi folks, >>>> >>>> i have just installed varnish and this is my first message to the list. >>>> I did i very simple configuration. I edited /etc/varnish/default.vcl. >>>> >>>> I included the following lines: >>>> >>>> backend default { >>>> .host = "aton.cpd.edu.br"; >>>> .port = "80"; >>>> } >>>> >>>> I stress loaded the my site, now being served by varnish, but i >>>> noticed no performance improvement. >>>> I would like to do a minimal configuration just to see how fast >>>> varnish could do. >>>> Is there something more i need to configure in order to get the taste >>>> of varnish ? >>> >>> Yes, you need to check and fix the http headers your backend sends, by >>> default, Varnish honors them. >>> >>> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration >>> >>> Enno >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Wed Jan 23 17:14:25 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Wed, 23 Jan 2013 18:14:25 +0100 Subject: simple configuration In-Reply-To: References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: It looks like a varnish 2 VCL. Regards, Dridi On Wed, Jan 23, 2013 at 6:11 PM, Friedrich Locke wrote: > What i am trying is below (based on > https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration) > : > > sub vcl_recv { > > set req.http.host = "www.centev.ufv.br"; > > set req.backend = centev; > > if (req.request == "GET" && req.url ~ "^/static/") { > unset req.http.cookie; > unset req.http.Authorization; > return (lookup); > } > > if (req.request == "POST") { > return (pass); > } > > if (req.request != "GET" && req.request != "HEAD" && > req.request != "PUT" && req.request != "POST" && > req.request != "TRACE" && req.request != "OPTIONS" && > req.request != "DELETE" ) { > # Non-RFC2616 or CONNECT which is weird # > return (pass); > } > > if (req.http.Authorization) { > # Not cacheable by default # > return (pass); > } > > return (lookup); > } > > sub vcl_fetch { > > if (req.request == "GET" && req.url ~ "^/static" ) { > unset obj.http.Set-Cookie; > set obj.ttl = 30m; > return (deliver); > } > > if (obj.status >= 300) { > return (hit_for_pass); > } > > if (!obj.cacheable) { > return (hit_for_pass); > } > > if (obj.http.Set-Cookie) { > return (hit_for_pass); > } > > if (obj.http.Pragma ~ "no-cache" || > obj.http.Cache-Control ~ "no-cache" || > obj.http.Cache-Control ~ "private") { > return (hit_for_pass); > } > > if (obj.http.Cache-Control ~ "max-age") { > unset obj.http.Set-Cookie; > return (deliver); > } > > return (hit_for_pass); > } > > Message from VCC-compiler: > Symbol not found: 'obj.cacheable' (expected type BOOL): > ('input' Line 176 Pos 14) > if (!obj.cacheable) { > -------------#############--- > > Running VCC-compiler failed, exit 1 > > VCL compilation failed > # > > > Any advice ? > > PS: varnish-3.0.2 > > > On Wed, Jan 23, 2013 at 2:57 PM, Dridi Boukelmoune > wrote: >> Hi Friedrich, >> >> You cannot issue a pass in vcl_fetch, are you looking for hit_for_pass ? >> https://raw.github.com/varnish/Varnish-Book/master/ui/img/vcl.png >> >> Regards, >> Dridi >> >> On Wed, Jan 23, 2013 at 5:50 PM, Friedrich Locke >> wrote: >>> Sorry, >>> >>> but i am getting this error too: >>> >>> Message from VCC-compiler: >>> Invalid return "pass" >>> ('input' Line 173 Pos 25) >>> return (pass); >>> ------------------------####-- >>> >>> >>> ...in subroutine "vcl_fetch" >>> ('input' Line 164 Pos 5) >>> sub vcl_fetch { >>> ----#########-- >>> >>> >>> ...which is the "vcl_fetch" method >>> Legal returns are: "deliver" "error" "hit_for_pass" "restart" >>> Running VCC-compiler failed, exit 1 >>> >>> VCL compilation failed >>> # >>> >>> >>> Have any ideia ? >>> >>> On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: >>>> From: Shibashish [mailto:shib4u at gmail.com] >>>> Subject: Re: simple configuration >>>> >>>> >>>> On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke >>>> wrote: >>>>> >>>>> Hi folks, >>>>> >>>>> i have just installed varnish and this is my first message to the list. >>>>> I did i very simple configuration. I edited /etc/varnish/default.vcl. >>>>> >>>>> I included the following lines: >>>>> >>>>> backend default { >>>>> .host = "aton.cpd.edu.br"; >>>>> .port = "80"; >>>>> } >>>>> >>>>> I stress loaded the my site, now being served by varnish, but i >>>>> noticed no performance improvement. >>>>> I would like to do a minimal configuration just to see how fast >>>>> varnish could do. >>>>> Is there something more i need to configure in order to get the taste >>>>> of varnish ? >>>> >>>> Yes, you need to check and fix the http headers your backend sends, by >>>> default, Varnish honors them. >>>> >>>> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration >>>> >>>> Enno >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From friedrich.locke at gmail.com Wed Jan 23 17:45:35 2013 From: friedrich.locke at gmail.com (Friedrich Locke) Date: Wed, 23 Jan 2013 15:45:35 -0200 Subject: simple configuration In-Reply-To: References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: May i replace obj by beresp in vcl_fetch ? Thanks. On Wed, Jan 23, 2013 at 3:14 PM, Dridi Boukelmoune wrote: > It looks like a varnish 2 VCL. > > Regards, > Dridi > > On Wed, Jan 23, 2013 at 6:11 PM, Friedrich Locke > wrote: >> What i am trying is below (based on >> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration) >> : >> >> sub vcl_recv { >> >> set req.http.host = "www.centev.ufv.br"; >> >> set req.backend = centev; >> >> if (req.request == "GET" && req.url ~ "^/static/") { >> unset req.http.cookie; >> unset req.http.Authorization; >> return (lookup); >> } >> >> if (req.request == "POST") { >> return (pass); >> } >> >> if (req.request != "GET" && req.request != "HEAD" && >> req.request != "PUT" && req.request != "POST" && >> req.request != "TRACE" && req.request != "OPTIONS" && >> req.request != "DELETE" ) { >> # Non-RFC2616 or CONNECT which is weird # >> return (pass); >> } >> >> if (req.http.Authorization) { >> # Not cacheable by default # >> return (pass); >> } >> >> return (lookup); >> } >> >> sub vcl_fetch { >> >> if (req.request == "GET" && req.url ~ "^/static" ) { >> unset obj.http.Set-Cookie; >> set obj.ttl = 30m; >> return (deliver); >> } >> >> if (obj.status >= 300) { >> return (hit_for_pass); >> } >> >> if (!obj.cacheable) { >> return (hit_for_pass); >> } >> >> if (obj.http.Set-Cookie) { >> return (hit_for_pass); >> } >> >> if (obj.http.Pragma ~ "no-cache" || >> obj.http.Cache-Control ~ "no-cache" || >> obj.http.Cache-Control ~ "private") { >> return (hit_for_pass); >> } >> >> if (obj.http.Cache-Control ~ "max-age") { >> unset obj.http.Set-Cookie; >> return (deliver); >> } >> >> return (hit_for_pass); >> } >> >> Message from VCC-compiler: >> Symbol not found: 'obj.cacheable' (expected type BOOL): >> ('input' Line 176 Pos 14) >> if (!obj.cacheable) { >> -------------#############--- >> >> Running VCC-compiler failed, exit 1 >> >> VCL compilation failed >> # >> >> >> Any advice ? >> >> PS: varnish-3.0.2 >> >> >> On Wed, Jan 23, 2013 at 2:57 PM, Dridi Boukelmoune >> wrote: >>> Hi Friedrich, >>> >>> You cannot issue a pass in vcl_fetch, are you looking for hit_for_pass ? >>> https://raw.github.com/varnish/Varnish-Book/master/ui/img/vcl.png >>> >>> Regards, >>> Dridi >>> >>> On Wed, Jan 23, 2013 at 5:50 PM, Friedrich Locke >>> wrote: >>>> Sorry, >>>> >>>> but i am getting this error too: >>>> >>>> Message from VCC-compiler: >>>> Invalid return "pass" >>>> ('input' Line 173 Pos 25) >>>> return (pass); >>>> ------------------------####-- >>>> >>>> >>>> ...in subroutine "vcl_fetch" >>>> ('input' Line 164 Pos 5) >>>> sub vcl_fetch { >>>> ----#########-- >>>> >>>> >>>> ...which is the "vcl_fetch" method >>>> Legal returns are: "deliver" "error" "hit_for_pass" "restart" >>>> Running VCC-compiler failed, exit 1 >>>> >>>> VCL compilation failed >>>> # >>>> >>>> >>>> Have any ideia ? >>>> >>>> On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: >>>>> From: Shibashish [mailto:shib4u at gmail.com] >>>>> Subject: Re: simple configuration >>>>> >>>>> >>>>> On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke >>>>> wrote: >>>>>> >>>>>> Hi folks, >>>>>> >>>>>> i have just installed varnish and this is my first message to the list. >>>>>> I did i very simple configuration. I edited /etc/varnish/default.vcl. >>>>>> >>>>>> I included the following lines: >>>>>> >>>>>> backend default { >>>>>> .host = "aton.cpd.edu.br"; >>>>>> .port = "80"; >>>>>> } >>>>>> >>>>>> I stress loaded the my site, now being served by varnish, but i >>>>>> noticed no performance improvement. >>>>>> I would like to do a minimal configuration just to see how fast >>>>>> varnish could do. >>>>>> Is there something more i need to configure in order to get the taste >>>>>> of varnish ? >>>>> >>>>> Yes, you need to check and fix the http headers your backend sends, by >>>>> default, Varnish honors them. >>>>> >>>>> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration >>>>> >>>>> Enno >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From friedrich.locke at gmail.com Thu Jan 24 13:33:16 2013 From: friedrich.locke at gmail.com (Friedrich Locke) Date: Thu, 24 Jan 2013 11:33:16 -0200 Subject: updating a file Message-ID: Hi folks, I am running varnish now. A doubt arose: how does varnish handle the scenario: You update the content of a image file but not the url of it. If varnished has cached that image before how does it know that image has been updated ? How much time does it take to varnish realize that image has been changed ? Thanks in advance. From bjorn at ruberg.no Thu Jan 24 13:41:30 2013 From: bjorn at ruberg.no (=?ISO-8859-1?Q?Bj=F8rn_Ruberg?=) Date: Thu, 24 Jan 2013 14:41:30 +0100 Subject: updating a file In-Reply-To: References: Message-ID: <51013A0A.7040805@ruberg.no> On 01/24/2013 02:33 PM, Friedrich Locke wrote: > Hi folks, > > I am running varnish now. A doubt arose: how does varnish handle the scenario: > > You update the content of a image file but not the url of it. If > varnished has cached that image before how does it know that image has > been updated ? > How much time does it take to varnish realize that image has been changed ? Cache expiration is related to the objects' TTL defined in the VCL, i.e. what *you* decide. The default TTL is 120 seconds. Varnish does not check for updated backend resources all by itself. If the TTL has expired, the next request for it will update the cached object in Varnish. If the TTL has not expired, Varnish will not know about it and you will need to make Varnish aware of it. This is often managed by using some kind of PURGE mechanism. -- Bj?rn From dridi.boukelmoune at zenika.com Thu Jan 24 14:26:47 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 24 Jan 2013 15:26:47 +0100 Subject: simple configuration In-Reply-To: References: <20130123135730.35dd8fd2@groupware.tty.nl> Message-ID: Hi, You are right indeed. Regards, Dridi On Wed, Jan 23, 2013 at 6:45 PM, Friedrich Locke wrote: > May i replace obj by beresp in vcl_fetch ? > > Thanks. > > On Wed, Jan 23, 2013 at 3:14 PM, Dridi Boukelmoune > wrote: >> It looks like a varnish 2 VCL. >> >> Regards, >> Dridi >> >> On Wed, Jan 23, 2013 at 6:11 PM, Friedrich Locke >> wrote: >>> What i am trying is below (based on >>> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration) >>> : >>> >>> sub vcl_recv { >>> >>> set req.http.host = "www.centev.ufv.br"; >>> >>> set req.backend = centev; >>> >>> if (req.request == "GET" && req.url ~ "^/static/") { >>> unset req.http.cookie; >>> unset req.http.Authorization; >>> return (lookup); >>> } >>> >>> if (req.request == "POST") { >>> return (pass); >>> } >>> >>> if (req.request != "GET" && req.request != "HEAD" && >>> req.request != "PUT" && req.request != "POST" && >>> req.request != "TRACE" && req.request != "OPTIONS" && >>> req.request != "DELETE" ) { >>> # Non-RFC2616 or CONNECT which is weird # >>> return (pass); >>> } >>> >>> if (req.http.Authorization) { >>> # Not cacheable by default # >>> return (pass); >>> } >>> >>> return (lookup); >>> } >>> >>> sub vcl_fetch { >>> >>> if (req.request == "GET" && req.url ~ "^/static" ) { >>> unset obj.http.Set-Cookie; >>> set obj.ttl = 30m; >>> return (deliver); >>> } >>> >>> if (obj.status >= 300) { >>> return (hit_for_pass); >>> } >>> >>> if (!obj.cacheable) { >>> return (hit_for_pass); >>> } >>> >>> if (obj.http.Set-Cookie) { >>> return (hit_for_pass); >>> } >>> >>> if (obj.http.Pragma ~ "no-cache" || >>> obj.http.Cache-Control ~ "no-cache" || >>> obj.http.Cache-Control ~ "private") { >>> return (hit_for_pass); >>> } >>> >>> if (obj.http.Cache-Control ~ "max-age") { >>> unset obj.http.Set-Cookie; >>> return (deliver); >>> } >>> >>> return (hit_for_pass); >>> } >>> >>> Message from VCC-compiler: >>> Symbol not found: 'obj.cacheable' (expected type BOOL): >>> ('input' Line 176 Pos 14) >>> if (!obj.cacheable) { >>> -------------#############--- >>> >>> Running VCC-compiler failed, exit 1 >>> >>> VCL compilation failed >>> # >>> >>> >>> Any advice ? >>> >>> PS: varnish-3.0.2 >>> >>> >>> On Wed, Jan 23, 2013 at 2:57 PM, Dridi Boukelmoune >>> wrote: >>>> Hi Friedrich, >>>> >>>> You cannot issue a pass in vcl_fetch, are you looking for hit_for_pass ? >>>> https://raw.github.com/varnish/Varnish-Book/master/ui/img/vcl.png >>>> >>>> Regards, >>>> Dridi >>>> >>>> On Wed, Jan 23, 2013 at 5:50 PM, Friedrich Locke >>>> wrote: >>>>> Sorry, >>>>> >>>>> but i am getting this error too: >>>>> >>>>> Message from VCC-compiler: >>>>> Invalid return "pass" >>>>> ('input' Line 173 Pos 25) >>>>> return (pass); >>>>> ------------------------####-- >>>>> >>>>> >>>>> ...in subroutine "vcl_fetch" >>>>> ('input' Line 164 Pos 5) >>>>> sub vcl_fetch { >>>>> ----#########-- >>>>> >>>>> >>>>> ...which is the "vcl_fetch" method >>>>> Legal returns are: "deliver" "error" "hit_for_pass" "restart" >>>>> Running VCC-compiler failed, exit 1 >>>>> >>>>> VCL compilation failed >>>>> # >>>>> >>>>> >>>>> Have any ideia ? >>>>> >>>>> On Wed, Jan 23, 2013 at 11:57 AM, Enno van Amerongen wrote: >>>>>> From: Shibashish [mailto:shib4u at gmail.com] >>>>>> Subject: Re: simple configuration >>>>>> >>>>>> >>>>>> On Wed, Jan 23, 2013 at 6:07 PM, Friedrich Locke >>>>>> wrote: >>>>>>> >>>>>>> Hi folks, >>>>>>> >>>>>>> i have just installed varnish and this is my first message to the list. >>>>>>> I did i very simple configuration. I edited /etc/varnish/default.vcl. >>>>>>> >>>>>>> I included the following lines: >>>>>>> >>>>>>> backend default { >>>>>>> .host = "aton.cpd.edu.br"; >>>>>>> .port = "80"; >>>>>>> } >>>>>>> >>>>>>> I stress loaded the my site, now being served by varnish, but i >>>>>>> noticed no performance improvement. >>>>>>> I would like to do a minimal configuration just to see how fast >>>>>>> varnish could do. >>>>>>> Is there something more i need to configure in order to get the taste >>>>>>> of varnish ? >>>>>> >>>>>> Yes, you need to check and fix the http headers your backend sends, by >>>>>> default, Varnish honors them. >>>>>> >>>>>> https://www.varnish-cache.org/trac/wiki/Introduction#Ourconfiguration >>>>>> >>>>>> Enno >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From perbu at varnish-software.com Thu Jan 24 21:53:44 2013 From: perbu at varnish-software.com (Per Buer) Date: Thu, 24 Jan 2013 22:53:44 +0100 Subject: simple configuration In-Reply-To: References: Message-ID: On Wed, Jan 23, 2013 at 1:37 PM, Friedrich Locke wrote: > Hi folks, > > i have just installed varnish and this is my first message to the list. > I did i very simple configuration. I edited /etc/varnish/default.vcl. > > I included the following lines: > > backend default { > .host = "aton.cpd.edu.br"; > .port = "80"; > } > > I stress loaded the my site, now being served by varnish, but i > noticed no performance improvement. > I would like to do a minimal configuration just to see how fast > varnish could do. > Is there something more i need to configure in order to get the taste > of varnish ? > Yes. Most likely there is an issue with some cookie being set by the backend thereby disabling caching. You might want to have a look at one of the following: - Using Varnish, a tutorial - https://www.varnish-cache.org/docs/3.0/tutorial/index.html - it has a chapter on "why doesn't Varnish cache my site" - The Varnish Book - https://www.varnish-software.com/static/book/ -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpotter-varnish at codepuppy.com Thu Jan 24 22:41:13 2013 From: jpotter-varnish at codepuppy.com (Jeff Potter) Date: Thu, 24 Jan 2013 17:41:13 -0500 Subject: varnishncsa, regex, and ReqEnd? Message-ID: <6523745E-C45B-4AB6-A3BA-7B8739E79582@codepuppy.com> Howdy, I can't seem to get varnishlog or varnishncsa to run with a -m regex on ReqEnd -- even a .* match results in nothing: varnishlog -c -m 'ReqEnd:.*' (nothing) But it's definitely there: varnishlog -c | grep ReqEnd 29 ReqEnd c 1305611473 1359067120.610778332 1359067120.660271168 0.052320004 0.049444437 0.000048399 29 ReqEnd c 1305611475 1359067120.874516726 1359067120.912814140 0.115943432 0.038046122 0.000251293 42 ReqEnd c 1305611477 1359067120.964497328 1359067120.964612961 0.338384867 0.000071764 0.000043869 29 ReqEnd c 1305611479 1359067121.222486973 1359067121.230084419 0.005004644 0.007555723 0.000041723 29 ReqEnd c 1305611480 1359067121.264626026 1359067121.370830059 -0.106092930 nan nan 47 ReqEnd c 1305611482 1359067121.299682856 1359067121.487748384 3.938856840 0.188021898 0.000043631 This is on Varnish 3.0.3. Am I missing something? Background: I'm trying to get varnishncsa to output lines where the time to first byte -- or backend time, i.e. col 5 in ReqEnd -- is greater than a certain value. If there's a better / different way of doing this, that'd work too. Thanks! -Jeff From hsc at nostromo.joeh.org Fri Jan 25 10:04:09 2013 From: hsc at nostromo.joeh.org (hsc) Date: Fri, 25 Jan 2013 11:04:09 +0100 (CET) Subject: varnish (varnish-2.1.5 SVN 0843d7a) crashes on FreeBSD (8.3-RELEASE-p3) Message-ID: Hello list, I experience regular, crashes with my varnish setup. Unfortunately I cannot reproduce or trigger them. I tried to get clues on what causes the crashes by debugging the core files, but I am not an expert on gdb and debugging. I followed the guidelines on "https://www.varnish-cache.org\ /trac/wiki/DebuggingVarnish". Here's the information on the setup: 1) Include the exact version of Varnish you are using, and on which OS/OS version. varnishd (varnish-2.1.5 SVN 0843d7a) running on "FreeBSD 8.3-RELEASE-p3" with a "GENERIC amd64" kernel. $ ldd `which varnishd` /usr/local/sbin/varnishd: libvarnish.so.1 => /usr/local/lib/libvarnish.so.1 (0x80069c000) librt.so.1 => /usr/lib//librt.so.1 (0x8007ae000) libpcre.so.0 => /usr/local/lib/compat/pkg/libpcre.so.0 (0x8008b3000) libvarnishcompat.so.1 => /usr/local/lib/libvarnishcompat.so.1 (0x8009e5000) libvcl.so.1 => /usr/local/lib/libvcl.so.1 (0x800af3000) libthr.so.3 => /lib/libthr.so.3 (0x800c0d000) libm.so.5 => /lib/libm.so.5 (0x800d26000) libc.so.7 => /lib/libc.so.7 (0x800e46000) $ ulimit -c unlimited 2) Include the VCL used. The VCL config is quite huge, around 1200 lines of configuration. Which sections of the config files are of specific interest so I can provide them here? I think narrowing it down makes more sense than pasting it all. 3) Include the assert message. No assert message was given. But in syslog i noticed this every time varnishd crashed: Some usual system messages logged to the syslog, with a timestamp of 14:00. Then the timestamps jump back one hour (CEST vs. CET maybe?), followed by varnish log- entries, resulting in a crash with a core-dump. As soon as varnishd has crashed, log-entries show the correct timestamps of 14:00 again, which is the "normal" time. 13:59:02 cache [some usual and generic system log-entries] 13:59:56 cache last message repeated 2 times 14:00:04 cache kernel: pid 5421 (varnishd), uid 80: exited on signal 11 13:00:05 cache varnishd[1455]: child (19494) Started 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Child starts 13:00:05 cache varnishd[1455]: Child (19494) said Probe("GET /ping HTTP/1.1 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Host: mysite.com 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Connection: close 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Accept-Encoding: foo/bar 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache last message repeated 2 times 13:00:05 cache varnishd[1455]: Child (19494) said ", 15, 1) 13:00:05 cache varnishd[1455]: Child (19494) said Probe("GET /ping HTTP/1.1 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Host: mysite.com 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Connection: close 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Accept-Encoding: foo/bar 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache last message repeated 2 times 13:00:05 cache varnishd[1455]: Child (19494) said ", 15, 1) 13:00:05 cache varnishd[1455]: Child (19494) said Probe("GET /ping HTTP/1.1 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Host: mysite.com 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Connection: close 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache varnishd[1455]: Child (19494) said Accept-Encoding: foo/bar 13:00:05 cache varnishd[1455]: Child (19494) said 13:00:05 cache last message repeated 2 times [ ...the ping stuff syslog-entries show up here repeatedly... ] 13:00:30 cache last message repeated 2 times 13:00:30 cache varnishd[1455]: Child (19925) said ", 65, 60) 13:00:30 cache varnishd[1455]: Child (19925) said Probe("GET / HTTP/1.1 13:00:30 cache varnishd[1455]: Child (19925) said 13:00:30 cache varnishd[1455]: Child (19925) said Host: mysite.com 13:00:30 cache varnishd[1455]: Child (19925) said 13:00:30 cache varnishd[1455]: Child (19925) said Connection: close 13:00:30 cache varnishd[1455]: Child (19925) said 13:00:30 cache varnishd[1455]: Child (19925) said Accept-Encoding: foo/bar 13:00:30 cache varnishd[1455]: Child (19925) said 13:00:30 cache last message repeated 2 times 13:00:30 cache varnishd[1455]: Child (19925) said ", 65, 60) 14:00:46 cache last message repeated 3 times 14:01:19 cache last message repeated 2 times The "/ping" URL is for varnishd's happiness (via "varnishadm -T :81 debug.health"). 4) Include a backtrace of the core-dump. Steps I have done so far: compile an unstripped varnishd, collect core-dumps. So currently there are five core-dumps I can examine. Looking at them I do see that "cache_ws.c" pops up every time a crash occurs. Thus I tried having a closer look at it via "frame " and "print *ws", but I have no idea how to read the output. Also 'shm_workspace=Variable "shm_workspace" is not available.' is another message that pops up. A search for these things though did not give me clues on what might be wrong so far. Maybe you could give me some hints on what to take a closer look reading those backtraces. - core::1 ------------------------------------------------------------ Core was generated by `varnishd'. Program terminated with signal 11, Segmentation fault. #0 0x0000000800a1a39b in ?? () (gdb) bt #0 0x0000000800a1a39b in ?? () #1 0x00000000004451c3 in __func__.5737 () #2 0x0000000000000003 in ?? () #3 0x0000000000000000 in ?? () #4 0x00007fff84223e58 in ?? () #5 0x00000008012144c0 in ?? () #6 0x00000008153182e0 in ?? () #7 0x0000000000425c66 in pan_ic (func=Variable "func" is not available. ) at cache_panic.c:273 #8 0x00000000004214f8 in http_SetH (to=0x8153182e0, n=3, fm=0x0) at cache_http.c:656 #9 0x000000000042f49a in VRT_l_obj_status (sp=0x818248008, num=302) at cache_vrt.c:282 #10 0x000000080720d39e in ?? () #11 0x0000000815318220 in ?? () #12 0x00000000000001c8 in ?? () #13 0x00000008182493d9 in ?? () #14 0x0000000000000004 in ?? () #15 0x0000000000000270 in ?? () #16 0x00000000004323b5 in WS_Alloc (ws=0x7fff84210570, bytes=355566304) at cache_ws.c:130 #17 0x00000008153182e0 in ?? () #18 0x00007fff84223d00 in ?? () #19 0x00007fff84223e58 in ?? () #20 0x0000000818248008 in ?? () #21 0x000000000042aafb in VCL_error_method (sp=0x818248008) at vcl_returns.h:61 #22 0x000000000041425a in cnt_error (sp=0x818248008) at cache_center.c:370 #23 0x0000000000414d79 in CNT_Session (sp=0x818248008) at steps.h:43 #24 0x00000000004282c1 in wrk_do_cnt_sess (w=0x7fff84223d00, priv=Variable "priv" is not available. ) at cache_pool.c:294 #25 0x0000000000427547 in wrk_thread_real (qp=0x8012144c0, shm_workspace=Variable "shm_workspace" is not available. ) at cache_pool.c:183 #26 0x0000000800c3d511 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x00007fffff9fdf90 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () Cannot access memory at address 0x7fff84224000 (gdb) frame 16 #16 0x00000000004323b5 in WS_Alloc (ws=0x7fff84210570, bytes=355566304) at cache_ws.c:130 130 WS_Assert(ws); (gdb) print *ws $1 = {magic = 405045256, id = 0x42aafb "\213?@\001", s = 0x7fff84223d00 "?\221c", f = 0x7fff84223e58 "T5", r = 0x818248008 "Z\234/,p\002", e = 0x41425a "\213\205@\001", overflow = 745434957} ---------------------------------------------------------------------- - core::2 ------------------------------------------------------------ Core was generated by `varnishd'. Program terminated with signal 11, Segmentation fault. #0 0x0000000800a1a39b in ?? () (gdb) bt #0 0x0000000800a1a39b in ?? () #1 0x00000000004451c3 in __func__.5737 () #2 0x00000000000054a8 in ?? () #3 0x00000000000054a8 in ?? () #4 0x0000000811872cc8 in ?? () #5 0x0000000000000000 in ?? () #6 0x0000000811862080 in ?? () #7 0x0000000000425c66 in pan_ic (func=Variable "func" is not available. ) at cache_panic.c:273 #8 0x000000000043208c in WS_Release (ws=0x811862080, bytes=21672) at cache_ws.c:193 #9 0x000000000042b84e in vrt_assemble_string (hp=0x8118623b8, h=0x0, p=Variable "p" is not available. ) at cache_vrt.c:182 #10 0x000000000042f575 in vrt_do_string (w=0x7ffe86a3bd00, fd=395, hp=0x8118623b8, fld=1, err=0x44a7b0 "resp.response", p=0x811868348 "/newsfeed/detail/nachrichten_nachrichten_rss.xml?0.8671897399282649?0.7050130921254134?0.5526925072390788?0.04943594061651574?0.2816845032576461?0.2989058246531162?0.5712539436218447?0.798410399952849"..., ap=0x7ffe86a28490) at cache_vrt.c:224 #11 0x0000000000430200 in VRT_l_req_url (sp=0x811862008, p=0x811868348 "/newsfeed/detail/nachrichten_nachrichten_rss.xml?0.8671897399282649?0.7050130921254134?0.5526925072390788?0.04943594061651574?0.2816845032576461?0.2989058246531162?0.5712539436218447?0.798410399952849"...) at cache_vrt.c:255 #12 0x0000000806c626c4 in ?? () #13 0x00000000000000d0 in ?? () #14 0x0000000811862008 in ?? () #15 0x00007ffe86a3be58 in ?? () #16 0x0000000000000000 in ?? () #17 0x00007ffe86a285d0 in ?? () #18 0x0000000806c627a6 in ?? () #19 0x0000000811862008 in ?? () #20 0x0000000811862008 in ?? () #21 0x00007ffe86a3be58 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000811862008 in ?? () #24 0x000000000042b378 in VCL_recv_method (sp=0x7ffe86a285a0) at vcl_returns.h:21 #25 0x0000000000415143 in CNT_Session (sp=0x811862008) at cache_center.c:1070 #26 0x00000000004282c1 in wrk_do_cnt_sess (w=0x7ffe86a3bd00, priv=Variable "priv" is not available. ) at cache_pool.c:294 #27 0x0000000000427547 in wrk_thread_real (qp=0x801214740, shm_workspace=Variable "shm_workspace" is not available. ) at cache_pool.c:183 #28 0x0000000800c3d511 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x00007fffff9fdf90 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () Cannot access memory at address 0x7ffe86a3c000 ---------------------------------------------------------------------- - core::3 ------------------------------------------------------------ Core was generated by `varnishd'. Program terminated with signal 11, Segmentation fault. #0 0x0000000800a1a39b in ?? () (gdb) bt #0 0x0000000800a1a39b in ?? () #1 0x00000000004451c3 in __func__.5737 () #2 0x0000000000000003 in ?? () #3 0x0000000000000000 in ?? () #4 0x00007fffca855e58 in ?? () #5 0x00000008012e1380 in ?? () #6 0x00000008216e2ce0 in ?? () #7 0x0000000000425c66 in pan_ic (func=Variable "func" is not available. ) at cache_panic.c:273 #8 0x00000000004214f8 in http_SetH (to=0x8216e2ce0, n=3, fm=0x0) at cache_http.c:656 #9 0x000000000042f49a in VRT_l_obj_status (sp=0x81b8bd008, num=302) at cache_vrt.c:282 #10 0x00000008010bd35e in ?? () #11 0x00000008216e2c20 in ?? () #12 0x00000000000001c8 in ?? () #13 0x000000081b8be541 in ?? () #14 0x0000000000000004 in ?? () #15 0x00000000000000bf in ?? () #16 0x00000000004323b5 in WS_Alloc (ws=0x7fffca840570, bytes=560868576) at cache_ws.c:130 #17 0x00000008216e2ce0 in ?? () #18 0x00007fffca855d00 in ?? () #19 0x00007fffca855e58 in ?? () #20 0x000000081b8bd008 in ?? () #21 0x000000000042aafb in VCL_error_method (sp=0x81b8bd008) at vcl_returns.h:61 #22 0x000000000041425a in cnt_error (sp=0x81b8bd008) at cache_center.c:370 #23 0x0000000000414d79 in CNT_Session (sp=0x81b8bd008) at steps.h:43 #24 0x00000000004282c1 in wrk_do_cnt_sess (w=0x7fffca855d00, priv=Variable "priv" is not available. ) at cache_pool.c:294 #25 0x0000000000427547 in wrk_thread_real (qp=0x8012e1380, shm_workspace=Variable "shm_workspace" is not available. ) at cache_pool.c:183 #26 0x0000000800c3d511 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x00007fffff9fdf90 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffca856000 ---------------------------------------------------------------------- - core::4 ------------------------------------------------------------ Core was generated by `varnishd'. Program terminated with signal 11, Segmentation fault. #0 0x0000000800a13d5f in ?? () (gdb) bt #0 0x0000000800a13d5f in ?? () #1 0x00007fff43408430 in ?? () #2 0x0000000800a13d9c in ?? () #3 0x00000000004451c3 in __func__.5737 () #4 0x0000000000000000 in ?? () #5 0x00000000000001f7 in ?? () #6 0x00007fff4341de58 in ?? () #7 0x00000008012e14c0 in ?? () #8 0x0000000000000004 in ?? () #9 0x0000000000425c66 in pan_ic (func=Variable "func" is not available. ) at cache_panic.c:273 #10 0x000000000042e3df in VRT_int_string (sp=Variable "sp" is not available. ) at cache_vrt.c:900 #11 0x00000008010bd4de in ?? () #12 0x0000000823264420 in ?? () #13 0x0000000000000018 in ?? () #14 0x0000000000448a88 in __func__.6447 () #15 0x0000000000000004 in ?? () #16 0x000000000000001d in ?? () #17 0x00000000004323b5 in WS_Alloc (ws=0x7fff43408570, bytes=589711816) at cache_ws.c:130 #18 0x00000008232644e0 in ?? () #19 0x00007fff4341dd00 in ?? () #20 0x00007fff4341de58 in ?? () #21 0x000000080871b008 in ?? () #22 0x000000000042aafb in VCL_error_method (sp=0x80871b008) at vcl_returns.h:61 #23 0x000000000041425a in cnt_error (sp=0x80871b008) at cache_center.c:370 #24 0x0000000000414d79 in CNT_Session (sp=0x80871b008) at steps.h:43 #25 0x00000000004282c1 in wrk_do_cnt_sess (w=0x7fff4341dd00, priv=Variable "priv" is not available. ) at cache_pool.c:294 #26 0x0000000000427547 in wrk_thread_real (qp=0x8012e14c0, shm_workspace=Variable "shm_workspace" is not available. ) at cache_pool.c:183 #27 0x0000000800c3d511 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x00007fffff9fdf90 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () Cannot access memory at address 0x7fff4341e000 ---------------------------------------------------------------------- - core::5 ------------------------------------------------------------ #0 0x0000000800a1a39b in ?? () #1 0x00000000004451c3 in __func__.5737 () #2 0x0000000000000003 in ?? () #3 0x0000000000000000 in ?? () #4 0x00007fff03e23e58 in ?? () #5 0x00000008012e1560 in ?? () #6 0x00000008264deae0 in ?? () #7 0x0000000000425c66 in pan_ic (func=Variable "func" is not available. ) at cache_panic.c:273 #8 0x00000000004214f8 in http_SetH (to=0x8264deae0, n=3, fm=0x0) at cache_http.c:656 #9 0x000000000042f49a in VRT_l_obj_status (sp=0x80b925008, num=302) at cache_vrt.c:282 #10 0x00000008010bd47e in ?? () #11 0x00000008264dea20 in ?? () #12 0x00000000000001c8 in ?? () #13 0x000000080b9262e1 in ?? () #14 0x0000000000000004 in ?? () #15 0x0000000000000054 in ?? () #16 0x00000000004323b5 in WS_Alloc (ws=0x7fff03e0e570, bytes=642640608) at cache_ws.c:130 #17 0x00000008264deae0 in ?? () #18 0x00007fff03e23d00 in ?? () #19 0x00007fff03e23e58 in ?? () #20 0x000000080b925008 in ?? () #21 0x000000000042aafb in VCL_error_method (sp=0x80b925008) at vcl_returns.h:61 #22 0x000000000041425a in cnt_error (sp=0x80b925008) at cache_center.c:370 #23 0x0000000000414d79 in CNT_Session (sp=0x80b925008) at steps.h:43 #24 0x00000000004282c1 in wrk_do_cnt_sess (w=0x7fff03e23d00, priv=Variable "priv" is not available. ) at cache_pool.c:294 #25 0x0000000000427547 in wrk_thread_real (qp=0x8012e1560, shm_workspace=Variable "shm_workspace" is not available. ) at cache_pool.c:183 #26 0x0000000800c3d511 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x00007fffff9fdf90 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () Cannot access memory at address 0x7fff03e24000 ---------------------------------------------------------------------- I am happy to provide more information if needed. Please let me know. Thanks. br, hsc From GuyB at spc.int Mon Jan 28 01:57:59 2013 From: GuyB at spc.int (Guy Brodie) Date: Mon, 28 Jan 2013 01:57:59 +0000 Subject: varnisnsca , centralised logging, conditional logging of X-Fowarded-For Message-ID: <17DEC0FA12D68446AE105B6A3C7A2D7701733B14@NOUEXCH01.corp.spc.int> Hi Everyone, I'm sure this must be answered somewhere, so I do apologise if this has been answered before. Problem: How to get _either_ X-Forwarded-For _or_ client.ip (in the absence of X-Forwarded-For) into a single NSCA formatted log file using varnish / varnishnsca. Situation: We host many sites through varnish, some of which are served via an upstream CDN - i.e. a proxy that supplies X-Forwarded-For, and some which don't - ie. There is no X-Forwarded-For and just a direct connection. I have tried the -f flag to varnishnsca, however it seems to only log an IP address for X-Forwarded-For if it exists, or '-' if it doesn't, meaning when I specify -f I lose all ip addresses for direct connections in the varnishnsca log file. I have followed tips to set X-Forwarded-For from client-ip using VCL, and that works for the downstream connections (the webservers can produce good logs), however I'm trying to get a single log file from varnish. (FWIW - this log file is to be parsed by AWStats) Varnish version: varnish-3.0.0 revision cbf1284 I think I'm missing something obvious: * a custom log format or something? * Or the -f functionality is not implemented conditionally as per the description in the bug report "log X-Forwarded-For instead of clientip if it exists" ( see https://www.varnish-cache.org/trac/ticket/335 ) * Or my X-Forwarded-For is present but blank (how could that be?) * Question: Does modifying the X-Forwarded-for fields using VCL (in sub vcl_recv ) have any impact on what would be logged using varnishnsca ? Any help appreciated! Thanks for reading : -) GB (Actually just found this post: https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-August/022381.html - exact same problem I think). From wxz19861013 at gmail.com Mon Jan 28 06:51:19 2013 From: wxz19861013 at gmail.com (Xianzhe Wang) Date: Mon, 28 Jan 2013 14:51:19 +0800 Subject: How to make multiple clients can get the response at the same time by stream. Message-ID: Hello everyone, My varnish version is 3.0.2-streaming release.And I set "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object to the client directly without fetching the whole object into varnish"; This is a part of my *.vcl file: sub vcl_fetch { set beresp.grace = 30m; set beresp.do_stream = true; if (beresp.http.Content-Length && beresp.http.Content-Length ~ "[0-9]{8,}") { return (hit_for_pass); } if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "no-cache" || beresp.http.Cache-Control ~ "private") { return (hit_for_pass); } if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120 s; return (hit_for_pass); } return (deliver); } Then I request a big file(about 100M+) like "xxx.zip" from clients.There is only one client can access the object.because "the object will marked as busy as it is delivered." But if the request goes directly to ?pass? ,multiple clients can get the response at the same time. Also if I remove if (beresp.http.Content-Length && beresp.http.Content-Length ~ "[0-9]{8,}") { return (hit_for_pass); } to make the file cacheable,multiple clients can get the response at the same time. Now I want "multiple clients can get the response at the same time." in all situations("pass","hit","hit_for_pass"). What can I do for it? Any suggestions will be appreciate. Thank you. -Shawn Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.carrillo at gmail.com Mon Jan 28 09:08:34 2013 From: daniel.carrillo at gmail.com (Daniel Carrillo) Date: Mon, 28 Jan 2013 10:08:34 +0100 Subject: varnisnsca , centralised logging, conditional logging of X-Fowarded-For In-Reply-To: <17DEC0FA12D68446AE105B6A3C7A2D7701733B14@NOUEXCH01.corp.spc.int> References: <17DEC0FA12D68446AE105B6A3C7A2D7701733B14@NOUEXCH01.corp.spc.int> Message-ID: 2013/1/28 Guy Brodie : > Hi Everyone, Hi. > I'm sure this must be answered somewhere, so I do apologise if this has been answered before. > > Problem: How to get _either_ X-Forwarded-For _or_ client.ip (in the absence of X-Forwarded-For) into a single NSCA formatted log file using varnish / varnishnsca. > > Situation: We host many sites through varnish, some of which are served via an upstream CDN - i.e. a proxy that supplies X-Forwarded-For, and some which don't - ie. There is no X-Forwarded-For and just a direct connection. > > I have tried the -f flag to varnishnsca, however it seems to only log an IP address for X-Forwarded-For if it exists, or '-' if it doesn't, meaning when I specify -f I lose all ip addresses for direct connections in the varnishnsca log file. > > I have followed tips to set X-Forwarded-For from client-ip using VCL, and that works for the downstream connections (the webservers can produce good logs), however I'm trying to get a single log file from varnish. > > (FWIW - this log file is to be parsed by AWStats) > Varnish version: varnish-3.0.0 revision cbf1284 > > I think I'm missing something obvious: > * a custom log format or something? You could use -F flag as follows: -F format Specify the log format used. If no format is specified the default log format is used. Currently it is: ... %{X}i The contents of request header X. .... AFAIK you can't log conditionally as you ask, but in order to use AWStats, a custom log seems enough for you. Kind regards. From jhayter at manta.com Mon Jan 28 21:38:02 2013 From: jhayter at manta.com (Jim Hayter) Date: Mon, 28 Jan 2013 21:38:02 +0000 Subject: Question of use of ACLs Message-ID: I am running Varnish 3.02 under Ubuntu 12.04. I am familiar with the ACL example of setting up a list of IPs and testing client.ip against it to limit the ability to purge. I have an environment where I would like to check the IP of the originating request against an ACL to decide which backend to use. The problem is that with a CDN and nginx in front of varnish, client.ip does not contain the appropriate IP. The originating IP is available in one of the request headers. Is there a way to convince varnish to compare the originating IP with the ACL? Regards, Jim -- Jim Hayter Systems Engineer Office: 614.807.5145 Fax: 614.807.5104 http://Manta.com Where Small Business Grows From james at ifixit.com Mon Jan 28 22:04:14 2013 From: james at ifixit.com (James Pearson) Date: Mon, 28 Jan 2013 14:04:14 -0800 Subject: Question of use of ACLs In-Reply-To: References: Message-ID: <1359410597-sup-5418@geror.local> Excerpts from Jim Hayter's message of 2013-01-28 13:38:02 -0800: > I am running Varnish 3.02 under Ubuntu 12.04. > > I am familiar with the ACL example of setting up a list of IPs and testing client.ip against it to limit the ability to purge. > > I have an environment where I would like to check the IP of the originating request against an ACL to decide which backend to use. The problem is that with a CDN and nginx in front of varnish, client.ip does not contain the appropriate IP. The originating IP is available in one of the request headers. Is there a way to convince varnish to compare the originating IP with the ACL? Here's something I found on the subject: http://zcentric.com/2012/03/16/varnish-acl-with-x-forwarded-for-header/ HTH, - P From perbu at varnish-software.com Tue Jan 29 11:52:55 2013 From: perbu at varnish-software.com (Per Buer) Date: Tue, 29 Jan 2013 12:52:55 +0100 Subject: Question of use of ACLs In-Reply-To: References: Message-ID: If you search around you'll find a few helpful tools here and there. https://github.com/smunaut/libvmod-whatever does implement this. This probably belongs in the std vmod as this is a often requested feature. Per. On Mon, Jan 28, 2013 at 10:38 PM, Jim Hayter wrote: > I am running Varnish 3.02 under Ubuntu 12.04. > > I am familiar with the ACL example of setting up a list of IPs and testing > client.ip against it to limit the ability to purge. > > I have an environment where I would like to check the IP of the > originating request against an ACL to decide which backend to use. The > problem is that with a CDN and nginx in front of varnish, client.ip does > not contain the appropriate IP. The originating IP is available in one of > the request headers. Is there a way to convince varnish to compare the > originating IP with the ACL? > > Regards, > Jim > -- > Jim Hayter > Systems Engineer > Office: 614.807.5145 Fax: 614.807.5104 > http://Manta.com Where Small Business Grows > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuba at ovh.net Tue Jan 29 12:13:40 2013 From: kuba at ovh.net (Jakub =?utf-8?B?U8WCb2NpxYRza2k=?=) Date: Tue, 29 Jan 2013 13:13:40 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: References: Message-ID: <20130129121340.GC19688@kuba.local.lp> Hi Xianzhe Wang, you should try option "set req.hash_ignore_busy = true;" in vlc_recv. Regards, -- Jakub S. Xianzhe Wang napisa?(a): > Hello everyone, > My varnish version is 3.0.2-streaming release.And I set > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object to > the client directly without fetching the whole object into varnish"; > > This is a part of my *.vcl file: > > sub vcl_fetch { > set beresp.grace = 30m; > > set beresp.do_stream = true; > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > "[0-9]{8,}") { > return (hit_for_pass); > } > > if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ > "no-cache" || beresp.http.Cache-Control ~ "private") { > return (hit_for_pass); > } > > if (beresp.ttl <= 0s || > beresp.http.Set-Cookie || > beresp.http.Vary == "*") { > > set beresp.ttl = 120 s; > return (hit_for_pass); > } > > return (deliver); > } > > Then I request a big file(about 100M+) like "xxx.zip" from clients.There is > only one client can access the object.because "the object will marked as > busy as it is delivered." > > But if the request goes directly to ?pass? ,multiple clients can get the > response at the same time. > > Also if I remove > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > "[0-9]{8,}") { > return (hit_for_pass); > } > to make the file cacheable,multiple clients can get the response at the > same time. > > Now I want "multiple clients can get the response at the same time." in all > situations("pass","hit","hit_for_pass"). > > What can I do for it? > Any suggestions will be appreciate. > Thank you. > > -Shawn Wang > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From perbu at varnish-software.com Tue Jan 29 12:50:25 2013 From: perbu at varnish-software.com (Per Buer) Date: Tue, 29 Jan 2013 13:50:25 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: <20130129121340.GC19688@kuba.local.lp> References: <20130129121340.GC19688@kuba.local.lp> Message-ID: Hi. On Tue, Jan 29, 2013 at 1:13 PM, Jakub S?oci?ski wrote: > Hi Xianzhe Wang, > you should try option > "set req.hash_ignore_busy = true;" > in vlc_recv. > No. This won't work. Plain vanilla varnish doesn't support streaming properly. You should use the "plus" branch of Varnish if you need it. > > Regards, > -- > Jakub S. > > > Xianzhe Wang napisa?(a): > > Hello everyone, > > My varnish version is 3.0.2-streaming release.And I set > > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object > to > > the client directly without fetching the whole object into varnish"; > > > > This is a part of my *.vcl file: > > > > sub vcl_fetch { > > set beresp.grace = 30m; > > > > set beresp.do_stream = true; > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > "[0-9]{8,}") { > > return (hit_for_pass); > > } > > > > if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ > > "no-cache" || beresp.http.Cache-Control ~ "private") { > > return (hit_for_pass); > > } > > > > if (beresp.ttl <= 0s || > > beresp.http.Set-Cookie || > > beresp.http.Vary == "*") { > > > > set beresp.ttl = 120 s; > > return (hit_for_pass); > > } > > > > return (deliver); > > } > > > > Then I request a big file(about 100M+) like "xxx.zip" from clients.There > is > > only one client can access the object.because "the object will marked as > > busy as it is delivered." > > > > But if the request goes directly to ?pass? ,multiple clients can get the > > response at the same time. > > > > Also if I remove > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > "[0-9]{8,}") { > > return (hit_for_pass); > > } > > to make the file cacheable,multiple clients can get the response at the > > same time. > > > > Now I want "multiple clients can get the response at the same time." in > all > > situations("pass","hit","hit_for_pass"). > > > > What can I do for it? > > Any suggestions will be appreciate. > > Thank you. > > > > -Shawn Wang > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhayter at manta.com Tue Jan 29 15:47:54 2013 From: jhayter at manta.com (Jim Hayter) Date: Tue, 29 Jan 2013 15:47:54 +0000 Subject: Question of use of ACLs In-Reply-To: <1359410597-sup-5418@geror.local> References: <1359410597-sup-5418@geror.local> Message-ID: Thanks to both James Pearson and Per Buer for their suggestions. I implemented the one below and it is working as desired. Regards, Jim -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of James Pearson Sent: Monday, January 28, 2013 5:04 PM To: varnish-misc Subject: Re: Question of use of ACLs Excerpts from Jim Hayter's message of 2013-01-28 13:38:02 -0800: > I am running Varnish 3.02 under Ubuntu 12.04. > > I am familiar with the ACL example of setting up a list of IPs and testing client.ip against it to limit the ability to purge. > > I have an environment where I would like to check the IP of the originating request against an ACL to decide which backend to use. The problem is that with a CDN and nginx in front of varnish, client.ip does not contain the appropriate IP. The originating IP is available in one of the request headers. Is there a way to convince varnish to compare the originating IP with the ACL? Here's something I found on the subject: http://zcentric.com/2012/03/16/varnish-acl-with-x-forwarded-for-header/ HTH, - P _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From wxz19861013 at gmail.com Wed Jan 30 02:32:49 2013 From: wxz19861013 at gmail.com (Xianzhe Wang) Date: Wed, 30 Jan 2013 10:32:49 +0800 Subject: Fwd: How to make multiple clients can get the response at the same time by stream. In-Reply-To: References: <20130129121340.GC19688@kuba.local.lp> Message-ID: Hi, Thanks a lot. I tried option "set req.hash_ignore_busy = true;" in vlc_recv. I think it works. But there are side effects: it would increase backend load. I have an idea about it in my previous email. what do you think about it? Another question is that where can I find the "plus" branch of Varnish which matches this issue. Any suggestions will be appreciate. Thanks again for help. Regards, -- Shawn Wang ---------- Forwarded message ---------- From: Xianzhe Wang Date: 2013/1/30 Subject: Re: How to make multiple clients can get the response at the same time by stream. To: Jakub S?oci?ski Hi Jakub S. Thank you very much. I tried, and take a simple test, two client request the big file at the same time, they get the response stream immediately, so it works. In that case, multiple requests will go directly to "pass", they do not need to wait, but it would increase backend load. We need to balance the benefits and drawbacks. I wanna is that: Client 1 requests url /foo Client 2..N request url /foo Varnish tasks a worker to fetch /foo for Client 1 Client 2..N are now queued pending response from the worker Worker fetch response header(just header not include body) from backend, and find it non-cacheable, then make the remaining requests(Client 2..N) go directly to "pass". And creat the hit_for_pass object synchronously in the first request(Client 1). Subsequent requests are now given the hit_for_pass object instructing them to go to the backend as long as the hit_for_pass object exists. As I mentioned below, is it feasible? Or do you have any Suggestions? Thanks again for help. Regards, -- Shawn Wang 2013/1/29 Jakub S?oci?ski > Hi Xianzhe Wang, > you should try option > "set req.hash_ignore_busy = true;" > in vlc_recv. > > Regards, > -- > Jakub S. > > > Xianzhe Wang napisa?(a): > > Hello everyone, > > My varnish version is 3.0.2-streaming release.And I set > > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object > to > > the client directly without fetching the whole object into varnish"; > > > > This is a part of my *.vcl file: > > > > sub vcl_fetch { > > set beresp.grace = 30m; > > > > set beresp.do_stream = true; > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > "[0-9]{8,}") { > > return (hit_for_pass); > > } > > > > if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ > > "no-cache" || beresp.http.Cache-Control ~ "private") { > > return (hit_for_pass); > > } > > > > if (beresp.ttl <= 0s || > > beresp.http.Set-Cookie || > > beresp.http.Vary == "*") { > > > > set beresp.ttl = 120 s; > > return (hit_for_pass); > > } > > > > return (deliver); > > } > > > > Then I request a big file(about 100M+) like "xxx.zip" from clients.There > is > > only one client can access the object.because "the object will marked as > > busy as it is delivered." > > > > But if the request goes directly to ?pass? ,multiple clients can get the > > response at the same time. > > > > Also if I remove > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > "[0-9]{8,}") { > > return (hit_for_pass); > > } > > to make the file cacheable,multiple clients can get the response at the > > same time. > > > > Now I want "multiple clients can get the response at the same time." in > all > > situations("pass","hit","hit_for_pass"). > > > > What can I do for it? > > Any suggestions will be appreciate. > > Thank you. > > > > -Shawn Wang > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuba at ovh.net Wed Jan 30 10:22:31 2013 From: kuba at ovh.net (Jakub =?utf-8?B?U8WCb2NpxYRza2k=?=) Date: Wed, 30 Jan 2013 11:22:31 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: References: <20130129121340.GC19688@kuba.local.lp> Message-ID: <20130130102231.GF19688@kuba.local.lp> Hi, I would bet that hash_ignore_busy works under the version Xianzhe Wang mentioned - 3.0.2-streaming :) -- Regards, Jakub S?oci?ski Per Buer napisa?(a): > Hi. > > On Tue, Jan 29, 2013 at 1:13 PM, Jakub S?oci?ski wrote: > > > Hi Xianzhe Wang, > > you should try option > > "set req.hash_ignore_busy = true;" > > in vlc_recv. > > > > No. This won't work. > > Plain vanilla varnish doesn't support streaming properly. You should use > the "plus" branch of Varnish if you need it. > > > > > > > > > Regards, > > -- > > Jakub S. > > > > > > Xianzhe Wang napisa?(a): > > > Hello everyone, > > > My varnish version is 3.0.2-streaming release.And I set > > > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object > > to > > > the client directly without fetching the whole object into varnish"; > > > > > > This is a part of my *.vcl file: > > > > > > sub vcl_fetch { > > > set beresp.grace = 30m; > > > > > > set beresp.do_stream = true; > > > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > "[0-9]{8,}") { > > > return (hit_for_pass); > > > } > > > > > > if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ > > > "no-cache" || beresp.http.Cache-Control ~ "private") { > > > return (hit_for_pass); > > > } > > > > > > if (beresp.ttl <= 0s || > > > beresp.http.Set-Cookie || > > > beresp.http.Vary == "*") { > > > > > > set beresp.ttl = 120 s; > > > return (hit_for_pass); > > > } > > > > > > return (deliver); > > > } > > > > > > Then I request a big file(about 100M+) like "xxx.zip" from clients.There > > is > > > only one client can access the object.because "the object will marked as > > > busy as it is delivered." > > > > > > But if the request goes directly to ?pass? ,multiple clients can get the > > > response at the same time. > > > > > > Also if I remove > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > "[0-9]{8,}") { > > > return (hit_for_pass); > > > } > > > to make the file cacheable,multiple clients can get the response at the > > > same time. > > > > > > Now I want "multiple clients can get the response at the same time." in > > all > > > situations("pass","hit","hit_for_pass"). > > > > > > What can I do for it? > > > Any suggestions will be appreciate. > > > Thank you. > > > > > > -Shawn Wang > > > > > _______________________________________________ > > > varnish-misc mailing list > > > varnish-misc at varnish-cache.org > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > -- > *Per Buer* > CEO | Varnish Software AS > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! From adiedler at tecracer.de Wed Jan 30 10:51:01 2013 From: adiedler at tecracer.de (Alexander Diedler) Date: Wed, 30 Jan 2013 11:51:01 +0100 Subject: Varnish 3.0.3 Typo3 and Amazon ELB Message-ID: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09517@DC01.tecracerde.local> Hello, We have an HA-Setup with 3 seperated Typo3 Hostingservers, that are fronted by Varnish 3.0.3 and a Loadbalancer in Front of the two Varnish servers. Now we are thinking, the Varnish does not cache enough, more content will be loaded from the webservers. Some of the problems is about the Typo3 Backend users, that must set a cookie, but for the "stateless" loadbalancing we unset the cookies. So we need some help for the configuration for stable environment. Can somebody help us? Example Page : http://www.taxofit.de/produkte.html Out Cache Settings: backend nginx { .host = "10.0.0.10"; .port = "80"; } sub vcl_recv { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } set req.backend = nginx; if (req.request == "POST") { ban("req.url == " + req.url); set req.http.X-Test = req.url; return (pass); } if (req.http.X-Tx-Solr-Iq) { return(pipe); } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { return (pipe); } if (req.request != "GET" && req.request != "HEAD") { return (pass); } if (req.http.Authorization || req.http.Cookie) { return (pass); } if (req.url ~ "/typo3" || req.url ~ "/typo3/install/") { ##TYPO3-Backend nicht cachen if (req.http.cookie ~ "be_typo_user"){ ##Inhalten l??schen wenn Shift+reload gedr??ckt wird,aber nur bei eingeloggtem user (Backend-Cookie) if (req.http.Cache-Control ~ "no-cache") { set req.ttl = 0s; ban("req.url == " + req.url); return (pass); } } else{ ##Cookies von TYPO3-Seiten erase unset req.http.Cookie; } } else { unset req.http.Cookie; } return (lookup); } sub vcl_fetch { set beresp.ttl = 12h; set req.grace = 24h; if (req.url ~ "\.(jpeg|jpg|png|gif|ico|swf|js|css|txt|gz|zip|rar|bz2|tgz|tbz|html|htm|pdf|pls|torrent)$") { set beresp.ttl = 48h; } if (req.url ~ "/typo3" || req.url ~ "/typo3/install/") { if (beresp.http.set-cookie ~ "be_typo_user"){ } else{ unset beresp.http.set-cookie; } } return (deliver); } Best regards Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Wed Jan 30 13:04:23 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Wed, 30 Jan 2013 08:04:23 -0500 Subject: memory usage/cache size Message-ID: i've been reading the wiki and varnish book plus looking at varnishstat output but can't figure out how to determine what feels like it should be something fairly easy. our first live in production varnish is set to malloc 1g of memory, how can i tell when it is evicted times based on memory running low rather than ttl expiring? -- Sent from BeMail -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Wed Jan 30 13:11:08 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Wed, 30 Jan 2013 08:11:08 -0500 Subject: memory usage/cache size In-Reply-To: References: Message-ID: I just found 'N LRU nuked objects' in varnish stat. Not sure how I missed that previously. I'm assuming that is what I'm looking for. On Wed, Jan 30, 2013 at 8:04 AM, Sean Allen wrote: > i've been reading the wiki and varnish book plus looking at varnishstat > output but can't figure out how to determine what feels like it should be > something fairly easy. > > our first live in production varnish is set to malloc 1g of memory, how > can i tell when it is evicted times based on memory running low rather than > ttl expiring? > > -- > > Sent from BeMail > -- Sent from BeMail -------------- next part -------------- An HTML attachment was scrubbed... URL: From adiedler at tecracer.de Wed Jan 30 14:44:49 2013 From: adiedler at tecracer.de (Alexander Diedler) Date: Wed, 30 Jan 2013 15:44:49 +0100 Subject: Remove PHPSESSID from Header Message-ID: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09545@DC01.tecracerde.local> Hello, How Can I remove the Cookie PHPSESSID from header send by client in Varnish 3.0.3? Greetings Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Jan 30 15:07:41 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 30 Jan 2013 16:07:41 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: <20130130102231.GF19688@kuba.local.lp> References: <20130129121340.GC19688@kuba.local.lp> <20130130102231.GF19688@kuba.local.lp> Message-ID: No. It doesn't work. You'll end with multiple versions of the same object in memory and you'll disable caching when multiple requests are taking place. Sort of like a braindamaged "pass" that doesn't free up the memory when the request is done. On Wed, Jan 30, 2013 at 11:22 AM, Jakub S?oci?ski wrote: > > Hi, > I would bet that hash_ignore_busy works under the version Xianzhe Wang > mentioned - 3.0.2-streaming :) > > -- > Regards, > Jakub S?oci?ski > > Per Buer napisa?(a): > > Hi. > > > > On Tue, Jan 29, 2013 at 1:13 PM, Jakub S?oci?ski wrote: > > > > > Hi Xianzhe Wang, > > > you should try option > > > "set req.hash_ignore_busy = true;" > > > in vlc_recv. > > > > > > > No. This won't work. > > > > Plain vanilla varnish doesn't support streaming properly. You should use > > the "plus" branch of Varnish if you need it. > > > > > > > > > > > > > > > > Regards, > > > -- > > > Jakub S. > > > > > > > > > Xianzhe Wang napisa?(a): > > > > Hello everyone, > > > > My varnish version is 3.0.2-streaming release.And I set > > > > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the > object > > > to > > > > the client directly without fetching the whole object into varnish"; > > > > > > > > This is a part of my *.vcl file: > > > > > > > > sub vcl_fetch { > > > > set beresp.grace = 30m; > > > > > > > > set beresp.do_stream = true; > > > > > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > > "[0-9]{8,}") { > > > > return (hit_for_pass); > > > > } > > > > > > > > if (beresp.http.Pragma ~ "no-cache" || > beresp.http.Cache-Control ~ > > > > "no-cache" || beresp.http.Cache-Control ~ "private") { > > > > return (hit_for_pass); > > > > } > > > > > > > > if (beresp.ttl <= 0s || > > > > beresp.http.Set-Cookie || > > > > beresp.http.Vary == "*") { > > > > > > > > set beresp.ttl = 120 s; > > > > return (hit_for_pass); > > > > } > > > > > > > > return (deliver); > > > > } > > > > > > > > Then I request a big file(about 100M+) like "xxx.zip" from > clients.There > > > is > > > > only one client can access the object.because "the object will > marked as > > > > busy as it is delivered." > > > > > > > > But if the request goes directly to ?pass? ,multiple clients can > get the > > > > response at the same time. > > > > > > > > Also if I remove > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > > "[0-9]{8,}") { > > > > return (hit_for_pass); > > > > } > > > > to make the file cacheable,multiple clients can get the response at > the > > > > same time. > > > > > > > > Now I want "multiple clients can get the response at the same time." > in > > > all > > > > situations("pass","hit","hit_for_pass"). > > > > > > > > What can I do for it? > > > > Any suggestions will be appreciate. > > > > Thank you. > > > > > > > > -Shawn Wang > > > > > > > _______________________________________________ > > > > varnish-misc mailing list > > > > varnish-misc at varnish-cache.org > > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > > > > _______________________________________________ > > > varnish-misc mailing list > > > varnish-misc at varnish-cache.org > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > > > > > -- > > *Per Buer* > > CEO | Varnish Software AS > > Phone: +47 958 39 117 | Skype: per.buer > > We Make Websites Fly! > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuba at ovh.net Wed Jan 30 15:19:23 2013 From: kuba at ovh.net (Jakub =?utf-8?B?U8WCb2NpxYRza2k=?=) Date: Wed, 30 Jan 2013 16:19:23 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: References: <20130129121340.GC19688@kuba.local.lp> <20130130102231.GF19688@kuba.local.lp> Message-ID: <20130130151923.GI19688@kuba.local.lp> Ok, thanks for clarification, Per. Per Buer napisa?(a): > No. It doesn't work. > > You'll end with multiple versions of the same object in memory and you'll > disable caching when multiple requests are taking place. Sort of like a > braindamaged "pass" that doesn't free up the memory when the request is > done. > > > > On Wed, Jan 30, 2013 at 11:22 AM, Jakub S?oci?ski wrote: > > > > > Hi, > > I would bet that hash_ignore_busy works under the version Xianzhe Wang > > mentioned - 3.0.2-streaming :) > > > > -- > > Regards, > > Jakub S?oci?ski > > > > Per Buer napisa?(a): > > > Hi. > > > > > > On Tue, Jan 29, 2013 at 1:13 PM, Jakub S?oci?ski wrote: > > > > > > > Hi Xianzhe Wang, > > > > you should try option > > > > "set req.hash_ignore_busy = true;" > > > > in vlc_recv. > > > > > > > > > > No. This won't work. > > > > > > Plain vanilla varnish doesn't support streaming properly. You should use > > > the "plus" branch of Varnish if you need it. > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > -- > > > > Jakub S. > > > > > > > > > > > > Xianzhe Wang napisa?(a): > > > > > Hello everyone, > > > > > My varnish version is 3.0.2-streaming release.And I set > > > > > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the > > object > > > > to > > > > > the client directly without fetching the whole object into varnish"; > > > > > > > > > > This is a part of my *.vcl file: > > > > > > > > > > sub vcl_fetch { > > > > > set beresp.grace = 30m; > > > > > > > > > > set beresp.do_stream = true; > > > > > > > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > > > "[0-9]{8,}") { > > > > > return (hit_for_pass); > > > > > } > > > > > > > > > > if (beresp.http.Pragma ~ "no-cache" || > > beresp.http.Cache-Control ~ > > > > > "no-cache" || beresp.http.Cache-Control ~ "private") { > > > > > return (hit_for_pass); > > > > > } > > > > > > > > > > if (beresp.ttl <= 0s || > > > > > beresp.http.Set-Cookie || > > > > > beresp.http.Vary == "*") { > > > > > > > > > > set beresp.ttl = 120 s; > > > > > return (hit_for_pass); > > > > > } > > > > > > > > > > return (deliver); > > > > > } > > > > > > > > > > Then I request a big file(about 100M+) like "xxx.zip" from > > clients.There > > > > is > > > > > only one client can access the object.because "the object will > > marked as > > > > > busy as it is delivered." > > > > > > > > > > But if the request goes directly to ?pass? ,multiple clients can > > get the > > > > > response at the same time. > > > > > > > > > > Also if I remove > > > > > if (beresp.http.Content-Length && beresp.http.Content-Length ~ > > > > > "[0-9]{8,}") { > > > > > return (hit_for_pass); > > > > > } > > > > > to make the file cacheable,multiple clients can get the response at > > the > > > > > same time. > > > > > > > > > > Now I want "multiple clients can get the response at the same time." > > in > > > > all > > > > > situations("pass","hit","hit_for_pass"). > > > > > > > > > > What can I do for it? > > > > > Any suggestions will be appreciate. > > > > > Thank you. > > > > > > > > > > -Shawn Wang > > > > > > > > > _______________________________________________ > > > > > varnish-misc mailing list > > > > > varnish-misc at varnish-cache.org > > > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > > > > > > > _______________________________________________ > > > > varnish-misc mailing list > > > > varnish-misc at varnish-cache.org > > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > > > > > > > > > > -- > > > *Per Buer* > > > CEO | Varnish Software AS > > > Phone: +47 958 39 117 | Skype: per.buer > > > We Make Websites Fly! > > > > > > -- > *Per Buer* > CEO | Varnish Software AS > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! From perbu at varnish-software.com Wed Jan 30 15:36:15 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 30 Jan 2013 16:36:15 +0100 Subject: How to make multiple clients can get the response at the same time by stream. In-Reply-To: References: <20130129121340.GC19688@kuba.local.lp> Message-ID: Hi, I was a bit quick and I didn't read the whole email the first time. Sorry about that. You're actually using the streaming branch, already I see. What you're writing is really, really odd. There is a slight lock while the "first" object is being fetched where other requests will be put on the waiting list. However, when the hit-for-pass object is created these should be released and pass'ed to the clients. If the backend takes forever coming back with the response headers then the situation would be something like what you describe. However, that would be odd and doesn't make much sense. PS: The streaming branch was renamed "plus" when it got other experimental features. You'll find source https://github.com/mbgrydeland/varnish-cacheand packages at repo.varnish-cache.org/test if I recall correctly. On Wed, Jan 30, 2013 at 3:32 AM, Xianzhe Wang wrote: > > Hi, > Thanks a lot. > > I tried option > "set req.hash_ignore_busy = true;" > in vlc_recv. > I think it works. But there are side effects: it would increase backend > load. > > I have an idea about it in my previous email. what do you think about it? > > Another question is that where can I find the "plus" branch of Varnish > which matches this issue. > > Any suggestions will be appreciate. > Thanks again for help. > > Regards, > -- > Shawn Wang > > > ---------- Forwarded message ---------- > From: Xianzhe Wang > Date: 2013/1/30 > Subject: Re: How to make multiple clients can get the response at the same > time by stream. > To: Jakub S?oci?ski > > > Hi Jakub S. > Thank you very much. > I tried, and take a simple test, two client request the big file at the > same time, they get the response stream immediately, so it works. > In that case, multiple requests will go directly to "pass", they do not > need to wait, but it would increase backend load. > We need to balance the benefits and drawbacks. > > I wanna is that: > Client 1 requests url /foo > Client 2..N request url /foo > Varnish tasks a worker to fetch /foo for Client 1 > Client 2..N are now queued pending response from the worker > Worker fetch response header(just header not include body) from > backend, and find it non-cacheable, then make the remaining > requests(Client 2..N) go directly to "pass". And creat the hit_for_pass > object synchronously in the first request(Client 1). > Subsequent requests are now given the hit_for_pass object instructing > them to go to the backend as long as the hit_for_pass object exists. > > As I mentioned below, is it feasible? Or do you have any Suggestions? > > Thanks again for help. > > Regards, > -- > Shawn Wang > > > > 2013/1/29 Jakub S?oci?ski > >> Hi Xianzhe Wang, >> you should try option >> "set req.hash_ignore_busy = true;" >> in vlc_recv. >> >> Regards, >> -- >> Jakub S. >> >> >> Xianzhe Wang napisa?(a): >> > Hello everyone, >> > My varnish version is 3.0.2-streaming release.And I set >> > "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object >> to >> > the client directly without fetching the whole object into varnish"; >> > >> > This is a part of my *.vcl file: >> > >> > sub vcl_fetch { >> > set beresp.grace = 30m; >> > >> > set beresp.do_stream = true; >> > >> > if (beresp.http.Content-Length && beresp.http.Content-Length ~ >> > "[0-9]{8,}") { >> > return (hit_for_pass); >> > } >> > >> > if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ >> > "no-cache" || beresp.http.Cache-Control ~ "private") { >> > return (hit_for_pass); >> > } >> > >> > if (beresp.ttl <= 0s || >> > beresp.http.Set-Cookie || >> > beresp.http.Vary == "*") { >> > >> > set beresp.ttl = 120 s; >> > return (hit_for_pass); >> > } >> > >> > return (deliver); >> > } >> > >> > Then I request a big file(about 100M+) like "xxx.zip" from >> clients.There is >> > only one client can access the object.because "the object will marked as >> > busy as it is delivered." >> > >> > But if the request goes directly to ?pass? ,multiple clients can get >> the >> > response at the same time. >> > >> > Also if I remove >> > if (beresp.http.Content-Length && beresp.http.Content-Length ~ >> > "[0-9]{8,}") { >> > return (hit_for_pass); >> > } >> > to make the file cacheable,multiple clients can get the response at the >> > same time. >> > >> > Now I want "multiple clients can get the response at the same time." in >> all >> > situations("pass","hit","hit_for_pass"). >> > >> > What can I do for it? >> > Any suggestions will be appreciate. >> > Thank you. >> > >> > -Shawn Wang >> >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> > > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From daghf at varnish-software.com Wed Jan 30 15:46:53 2013 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Wed, 30 Jan 2013 16:46:53 +0100 Subject: Remove PHPSESSID from Header In-Reply-To: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09545@DC01.tecracerde.local> References: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09545@DC01.tecracerde.local> Message-ID: On Wed, Jan 30, 2013 at 3:44 PM, Alexander Diedler wrote: > Hello,**** > > How Can I remove the Cookie PHPSESSID from header send by client in > Varnish 3.0.3? > Hi there There are examples for this on the wiki. Please see https://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies -- *Dag Haavi Finstad* Developer | Varnish Software AS Phone: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Wed Jan 30 23:45:51 2013 From: damon at huddler-inc.com (Damon Snyder) Date: Wed, 30 Jan 2013 15:45:51 -0800 Subject: Feedback on session_linger (high load/n_wrk_overflow issues) Message-ID: When you 'varnishadm -T localhost:port param.show session_linger' it indicates at the bottom that "we don't know if this is a good idea... and feeback is welcome." We found that setting session_linger pulled us out of a bind. I wanted to add my feedback to the list in the hope that someone else might benefit from what we experienced. We recently increased the number of esi includes on pages that get ~60-70 req/s on our platform. Some of those modules were being rendered with s-maxage set to zero so that they would be refreshed on every page load (this is so we could insert a non-cached partial into the page) which further increased the request load on varnish. What we found is that after a few hours the load on a varnish box went from < 1 to > 10 or more and n_wkr_overflow started incrementing. After investigating further we noticed that the context switching went from ~10k/s to > 100k/s. We are running Linux specifically Centos. No adjusting of threads or thread pools had any impact on the thrashing. After reading Kristian's post about high-end varnish tuning we decided to try out session_linger. We started by doubling the default from 50 to 100 to test the theory ('varnishadm -T localhost:port param.set session_linger 100'). Once we did that we saw a gradual settling of the context switching (using dstat or sar -w) and a stabilizing of the load. It's such a great feature to be able to change this parameter via the admin interface. We have 50GB malloc'ed and some nuking on our boxes so restarting varnish doesn't come without some impact to the platform. Intuitively increasing session_linger makes sense. If you have several esi modules rendered within a page and the gap between them is > 50ms then they'll be reallocated elsewhere. What is not clear to me is how we should tune session_linger. We started by setting it to the 3rd quantile of render times for the esi module taken from a sampling of backend requests. This turned out to be 110ms. Damon -------------- next part -------------- An HTML attachment was scrubbed... URL: From adiedler at tecracer.de Thu Jan 31 09:23:49 2013 From: adiedler at tecracer.de (Alexander Diedler) Date: Thu, 31 Jan 2013 10:23:49 +0100 Subject: AW: Remove PHPSESSID from Header In-Reply-To: References: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09545@DC01.tecracerde.local> Message-ID: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09573@DC01.tecracerde.local> Hello, This https://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies does not work or all my other examples in the default.vcl # set req.http.cookie = regsub(req.http.cookie, "(; )?PHPSESSID=[^;]+", ""); # if (req.http.cookie ~ "^ *$") { # remove req.http.cookie; # } # # if (req.http.Cookie) { # set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "\1"); # removes all cookies named __utm? (utma, utmb...) - tracking thing # # if (req.http.Cookie == "") { # remove req.http.Cookie; # } # } # if (req.http.Cookie ~ "PHPSESSID") # { # remove req.http.Cookie; # } We need urgent help, the production system take down if the frontned servers are overload+ Best regards Alexander Von: Dag Haavi Finstad [mailto:daghf at varnish-software.com] Gesendet: Mittwoch, 30. Januar 2013 16:47 An: Alexander Diedler Cc: varnish-misc at varnish-cache.org Betreff: Re: Remove PHPSESSID from Header On Wed, Jan 30, 2013 at 3:44 PM, Alexander Diedler > wrote: Hello, How Can I remove the Cookie PHPSESSID from header send by client in Varnish 3.0.3? Hi there There are examples for this on the wiki. Please see https://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies -- [http://www.varnish-software.com/static/media/logo-email.png] Dag Haavi Finstad Developer | Varnish Software AS Phone: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Thu Jan 31 12:42:02 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 31 Jan 2013 06:42:02 -0600 Subject: Remove PHPSESSID from Header In-Reply-To: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09573@DC01.tecracerde.local> References: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09545@DC01.tecracerde.local> <739CCB2D4BF8FD48985D6E158AD327B83A5ED09573@DC01.tecracerde.local> Message-ID: <20130131124202.GQ62018@nat.myhome> >> # if (req.http.Cookie ~ "PHPSESSID") >> # { >> # remove req.http.Cookie; >> # } Are you sure PHPSESSID is in fact the name of your cookie? The snippet above on its own would remove the cookie header entirely if this was the case. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From Jim.Avery at cityofdenton.com Wed Jan 9 22:56:17 2013 From: Jim.Avery at cityofdenton.com (Avery, Jim) Date: Wed, 09 Jan 2013 22:56:17 -0000 Subject: swf not being delivered Message-ID: <60553E926D9B0C4F803B070D24FC3C0D0A032657A5@codvexdsa.codad.cityofdenton.com> Greetings: I've managed to muddle through and set up a Varnish server to deliver a few of our Public Safety web pages to external browsers, but for the life of me, I cannot get a handle on why a simple .swf doesn't appear to be making its way to an external user. Internally to our network, it works just fine, but connect to the same webpage from outside, and all I get is a blank gray background, with no content delivered, and it appears that it's still trying to download something. Running version 3 on Debian 2.6.32. Tried adding this sub to my .vcl file: sub donotgzipflash { if (bereq.http.host == "yourwebserverdomainname.com") { if (bereq.url ~ "\.swf($|?)") { set bereq.http.Accept-Encoding = "identity"; } } } sub vcl_miss { call donotgzipflash; } sub vcl_pass { call donotgzipflash; } to see if it would accomplish anything, but get this when I try to compile it: ================= root at revproxy:~# varnishd -f /etc/varnish/dpdandmedia.vcl -s malloc,1G -T 172.16.64.52:2000 -a :80 Message from VCC-compiler: Regexp compilation error: nothing to repeat ('input' Line 38 Pos 24) if (bereq.url ~ "\.swf($|?)") { -----------------------############---- Running VCC-compiler failed, exit 1 VCL compilation failed ================== And so far have found nothing to remedy the compile error. Any assistance will be appreciated. Thanks, Jim Avery City of Denton, TX Application Architect Database, Exchange, Backup, and Anti-Malware Administrator Ph: 940.349.7760 Cell: 214.850.4543 -------------- next part -------------- An HTML attachment was scrubbed... URL: From adiedler at tecracer.de Fri Jan 11 13:31:03 2013 From: adiedler at tecracer.de (Alexander Diedler) Date: Fri, 11 Jan 2013 13:31:03 -0000 Subject: 503 for CSV File download Message-ID: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09010@DC01.tecracerde.local> Hello, We have a Typo3 Installation fronted with Varnish 3.0.2 out of the Ubuntu Packages. If we try to load a CSV File, there ist an error: Error 503 Service Unavailable Service Unavailable Guru Meditation: XID: 1113729206 Varnish cache server The script create a statistc file, that will be placed in typo3temp. The header of the file for transfer to the client is : Pragma: public Cache-Control: must-revalidate, post-check=0, pre-check=0 Expires: 0 Content-Disposition: attachment; filename= res_GewinnspielDaten_090113-1113.csv Content-type: x-application/octet-stream Content-Transfer-Encoding: binary Content-length:... Can anybody help us with this issue? Best regards AlexD1979 From antonsoradoi at gmail.com Mon Jan 14 15:53:06 2013 From: antonsoradoi at gmail.com (Anton Soradoi) Date: Mon, 14 Jan 2013 15:53:06 -0000 Subject: How does varnish cache determine if a resource needs to be refreshed Message-ID: Hi, I've found a link to the site on HN and got very curious in what you have to offer. My only concern is how does Varnish deal with frequent resource updates? (The company I work for has a website that it is being actively developed, with it's resources being updated multiple times a week. I.e js files, css files, images often need refreshing) Disclaimer, I have not read up too much about Varnish, so a quick answer would suffice. Thanks! Anton -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan at acceleration.net Thu Jan 17 16:01:52 2013 From: nathan at acceleration.net (Nathan Bird) Date: Thu, 17 Jan 2013 16:01:52 -0000 Subject: An updated emacs vcl-mode.el for Varnish 3. Message-ID: <50F82064.9070009@acceleration.net> When I went to edit vcl files I didn't see much with respect to emacs. There was a mailing list post[1] from several years back that was incomplete and not targeted at varnish 3 so I tried to update it. I used the docs for varnish 3 to get a more complete list of keywords/functions/etc. Hope it helps someone else. Updated version: https://gist.github.com/3982757 Cheers, Nathan Bird [1] https://www.varnish-cache.org/lists/pipermail/varnish-misc/2008-February/015760.html From yao.yang at autonavi.com Thu Jan 24 02:37:46 2013 From: yao.yang at autonavi.com (=?gb2312?B?0e7Sog==?=) Date: Thu, 24 Jan 2013 02:37:46 -0000 Subject: Varnish Error Message Message-ID: Hi all? My Varnish was reset by itself this morning?the following is the message in /var/log/message file log. And my Varnish boot parameters is: ./varnishd -f /opt/varnish/etc/varnish/default.vcl -s malloc,40G -T 127.0.0.1:2000 -a 0.0.0.0:80 \ -p thread_pool_max=4000 \ -p thread_pools=14 \ -p thread_pool_min=40 \ -p thread_pool_add_delay=2 \ -p session_linger=50 \ -p sess_workspace=5000000 The Resarting of Varnish affects the hitrate, and how to avoild this to happen again? I think the reason which may be the size of memory is not suitable. [cid:_Foxmail.0 at 191AF9D2-9CFE-4FD9-9CA7-D80F89C34264] [cid:_Foxmail.1 at 0E1CC645-2BBA-44B2-AA32-E37FDA325F78] ________________________________ Jan 24 09:33:06 kernel: varnishd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Jan 24 09:33:06 kernel: varnishd cpuset=/ mems_allowed=0-1 Jan 24 09:33:06 kernel: Pid: 13655, comm: varnishd Not tainted 2.6.32-220.el6.x86_64 #1 Jan 24 09:33:06 kernel: Call Trace: Jan 24 09:33:06 kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 Jan 24 09:33:06 kernel: [] ? dump_header+0x90/0x1b0 Jan 24 09:33:06 kernel: [] ? __delayacct_freepages_end+0x2e/0x30 Jan 24 09:33:06 kernel: [] ? security_real_capable_noaudit+0x3c/0x70 Jan 24 09:33:06 kernel: [] ? oom_kill_process+0x8a/0x2c0 Jan 24 09:33:06 kernel: [] ? select_bad_process+0xe1/0x120 Jan 24 09:33:06 kernel: [] ? out_of_memory+0x220/0x3c0 Jan 24 09:33:06 kernel: [] ? __alloc_pages_nodemask+0x89e/0x940 Jan 24 09:33:06 kernel: [] ? alloc_pages_current+0xaa/0x110 Jan 24 09:33:06 kernel: [] ? __page_cache_alloc+0x87/0x90 Jan 24 09:33:06 kernel: [] ? __do_page_cache_readahead+0xdb/0x210 Jan 24 09:33:06 kernel: [] ? ra_submit+0x21/0x30 Jan 24 09:33:06 kernel: [] ? filemap_fault+0x4c3/0x500 Jan 24 09:33:06 kernel: [] ? __do_fault+0x54/0x510 Jan 24 09:33:06 kernel: [] ? handle_pte_fault+0xf7/0xb50 Jan 24 09:33:06 kernel: [] ? sock_aio_read+0x181/0x190 Jan 24 09:33:06 kernel: [] ? do_sync_readv_writev+0xfb/0x140 Jan 24 09:33:06 kernel: [] ? handle_mm_fault+0x1e4/0x2b0 Jan 24 09:33:06 kernel: [] ? do_sync_read+0xfa/0x140 Jan 24 09:33:06 kernel: [] ? __do_page_fault+0x139/0x480 Jan 24 09:33:06 kernel: [] ? lock_sock_nested+0xac/0xc0 Jan 24 09:33:06 kernel: [] ? _spin_unlock_bh+0x1b/0x20 Jan 24 09:33:06 kernel: [] ? release_sock+0xce/0xe0 Jan 24 09:33:06 kernel: [] ? sock_setsockopt+0x16d/0x6f0 Jan 24 09:33:06 kernel: [] ? do_page_fault+0x3e/0xa0 Jan 24 09:33:06 kernel: [] ? page_fault+0x25/0x30 Jan 24 09:33:06 kernel: Mem-Info: Jan 24 09:33:06 kernel: Node 0 DMA per-cpu: Jan 24 09:33:06 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 2: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 3: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 4: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 5: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 6: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 7: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 8: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 9: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 10: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 11: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 12: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 13: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 14: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: CPU 15: hi: 0, btch: 1 usd: 0 Jan 24 09:33:06 kernel: Node 0 DMA32 per-cpu: Jan 24 09:33:06 kernel: CPU 0: hi: 186, btch: 31 usd: 68 Jan 24 09:33:06 kernel: CPU 1: hi: 186, btch: 31 usd: 36 Jan 24 09:33:06 kernel: CPU 2: hi: 186, btch: 31 usd: 139 Jan 24 09:33:06 kernel: CPU 3: hi: 186, btch: 31 usd: 37 Jan 24 09:33:06 kernel: CPU 4: hi: 186, btch: 31 usd: 115 Jan 24 09:33:06 kernel: CPU 5: hi: 186, btch: 31 usd: 17 Jan 24 09:33:06 kernel: CPU 6: hi: 186, btch: 31 usd: 82 Jan 24 09:33:06 kernel: CPU 7: hi: 186, btch: 31 usd: 32 Jan 24 09:33:06 kernel: CPU 8: hi: 186, btch: 31 usd: 18 Jan 24 09:33:06 kernel: CPU 9: hi: 186, btch: 31 usd: 26 Jan 24 09:33:06 kernel: CPU 10: hi: 186, btch: 31 usd: 150 Jan 24 09:33:06 kernel: CPU 11: hi: 186, btch: 31 usd: 49 Jan 24 09:33:06 kernel: CPU 12: hi: 186, btch: 31 usd: 115 Jan 24 09:33:06 kernel: CPU 13: hi: 186, btch: 31 usd: 55 Jan 24 09:33:06 kernel: CPU 14: hi: 186, btch: 31 usd: 70 Jan 24 09:33:06 kernel: CPU 15: hi: 186, btch: 31 usd: 0 Jan 24 09:33:06 kernel: Node 0 Normal per-cpu: Jan 24 09:33:06 kernel: CPU 0: hi: 186, btch: 31 usd: 169 Jan 24 09:33:06 kernel: CPU 1: hi: 186, btch: 31 usd: 29 Jan 24 09:33:06 kernel: CPU 2: hi: 186, btch: 31 usd: 91 Jan 24 09:33:06 kernel: CPU 3: hi: 186, btch: 31 usd: 42 Jan 24 09:33:06 kernel: CPU 4: hi: 186, btch: 31 usd: 121 Jan 24 09:33:06 kernel: CPU 5: hi: 186, btch: 31 usd: 47 Jan 24 09:33:06 kernel: CPU 6: hi: 186, btch: 31 usd: 93 Jan 24 09:33:06 kernel: CPU 7: hi: 186, btch: 31 usd: 38 Jan 24 09:33:06 kernel: CPU 8: hi: 186, btch: 31 usd: 43 Jan 24 09:33:06 kernel: CPU 9: hi: 186, btch: 31 usd: 26 Jan 24 09:33:06 kernel: CPU 10: hi: 186, btch: 31 usd: 139 Jan 24 09:33:06 kernel: CPU 11: hi: 186, btch: 31 usd: 0 Jan 24 09:33:06 kernel: CPU 12: hi: 186, btch: 31 usd: 96 Jan 24 09:33:06 kernel: CPU 13: hi: 186, btch: 31 usd: 1 Jan 24 09:33:06 kernel: CPU 14: hi: 186, btch: 31 usd: 79 Jan 24 09:33:06 kernel: CPU 15: hi: 186, btch: 31 usd: 0 Jan 24 09:33:06 kernel: Node 1 Normal per-cpu: Jan 24 09:33:06 kernel: CPU 0: hi: 186, btch: 31 usd: 35 Jan 24 09:33:06 kernel: CPU 1: hi: 186, btch: 31 usd: 26 Jan 24 09:33:06 kernel: CPU 2: hi: 186, btch: 31 usd: 84 Jan 24 09:33:06 kernel: CPU 3: hi: 186, btch: 31 usd: 38 Jan 24 09:33:06 kernel: CPU 4: hi: 186, btch: 31 usd: 116 Jan 24 09:33:06 kernel: CPU 5: hi: 186, btch: 31 usd: 24 Jan 24 09:33:06 kernel: CPU 6: hi: 186, btch: 31 usd: 112 Jan 24 09:33:06 kernel: CPU 7: hi: 186, btch: 31 usd: 36 Jan 24 09:33:06 kernel: CPU 8: hi: 186, btch: 31 usd: 70 Jan 24 09:33:06 kernel: CPU 9: hi: 186, btch: 31 usd: 96 Jan 24 09:33:06 kernel: CPU 10: hi: 186, btch: 31 usd: 178 Jan 24 09:33:06 kernel: CPU 11: hi: 186, btch: 31 usd: 53 Jan 24 09:33:06 kernel: CPU 12: hi: 186, btch: 31 usd: 111 Jan 24 09:33:06 kernel: CPU 13: hi: 186, btch: 31 usd: 29 Jan 24 09:33:06 kernel: CPU 14: hi: 186, btch: 31 usd: 88 Jan 24 09:33:06 kernel: CPU 15: hi: 186, btch: 31 usd: 0 Jan 24 09:33:06 kernel: active_anon:11312555 inactive_anon:800769 isolated_anon:0 Jan 24 09:33:06 kernel: active_file:390 inactive_file:2546 isolated_file:0 Jan 24 09:33:06 kernel: unevictable:0 dirty:234 writeback:9 unstable:0 Jan 24 09:33:06 kernel: free:46381 slab_reclaimable:4695 slab_unreclaimable:19412 Jan 24 09:33:06 kernel: mapped:865 shmem:1 pagetables:27687 bounce:0 Jan 24 09:33:06 kernel: Node 0 DMA free:15740kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 24 09:33:06 kernel: lowmem_reserve[]: 0 3502 24207 24207 Jan 24 09:33:06 kernel: Node 0 DMA32 free:89316kB min:6512kB low:8140kB high:9768kB active_anon:2469592kB inactive_anon:492884kB active_file:68kB inactive_file:1768kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3586464kB mlocked:0kB dirty:60kB writeback:16kB mapped:52kB shmem:0kB slab_reclaimable:4092kB slab_unreclaimable:9460kB kernel_stack:16kB pagetables:4960kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? yes Jan 24 09:33:06 kernel: lowmem_reserve[]: 0 0 20705 20705 Jan 24 09:33:06 kernel: Node 0 Normal free:37136kB min:38496kB low:48120kB high:57744kB active_anon:19667276kB inactive_anon:1405088kB active_file:132kB inactive_file:1008kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21201920kB mlocked:0kB dirty:320kB writeback:20kB mapped:188kB shmem:4kB slab_reclaimable:9180kB slab_unreclaimable:41676kB kernel_stack:5696kB pagetables:61332kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:71 all_unreclaimable? yes Jan 24 09:33:06 kernel: lowmem_reserve[]: 0 0 0 0 Jan 24 09:33:06 kernel: Node 1 Normal free:43332kB min:45072kB low:56340kB high:67608kB active_anon:23113352kB inactive_anon:1305104kB active_file:1360kB inactive_file:7408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:24821756kB mlocked:0kB dirty:556kB writeback:0kB mapped:3220kB shmem:0kB slab_reclaimable:5508kB slab_unreclaimable:26512kB kernel_stack:3888kB pagetables:44456kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:42944 all_unreclaimable? yes Jan 24 09:33:06 kernel: lowmem_reserve[]: 0 0 0 0 Jan 24 09:33:06 kernel: Node 0 DMA: 3*4kB 2*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB Jan 24 09:33:06 kernel: Node 0 DMA32: 681*4kB 876*8kB 752*16kB 565*32kB 233*64kB 150*128kB 56*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 89316kB Jan 24 09:33:06 kernel: Node 0 Normal: 682*4kB 661*8kB 264*16kB 144*32kB 89*64kB 50*128kB 22*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 37136kB Jan 24 09:33:06 kernel: Node 1 Normal: 762*4kB 638*8kB 577*16kB 347*32kB 122*64kB 16*128kB 1*256kB 1*512kB 4*1024kB 0*2048kB 0*4096kB = 43208kB Jan 24 09:33:06 kernel: 6158 total pagecache pages Jan 24 09:33:06 kernel: 3086 pages in swap cache Jan 24 09:33:06 kernel: Swap cache stats: add 4892481, delete 4889395, find 4972726/5258977 Jan 24 09:33:06 kernel: Free swap = 0kB Jan 24 09:33:06 kernel: Total swap = 4194296kB Jan 24 09:33:06 kernel: 12582910 pages RAM Jan 24 09:33:06 kernel: 229730 pages reserved Jan 24 09:33:06 kernel: 5799 pages shared Jan 24 09:33:06 kernel: 12293308 pages non-shared Jan 24 09:33:06 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jan 24 09:33:06 kernel: [ 654] 0 654 2708 0 0 -17 -1000 udevd Jan 24 09:33:06 kernel: [ 1810] 0 1810 62714 71 0 0 0 rsyslogd Jan 24 09:33:06 kernel: [ 1855] 0 1855 1031 30 1 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1864] 0 1864 1031 30 2 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1865] 0 1865 2301 38 1 0 0 irqbalance Jan 24 09:33:06 kernel: [ 1874] 0 1874 1031 30 3 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1875] 0 1875 1031 30 3 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1878] 0 1878 1031 30 7 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1880] 0 1880 1031 30 7 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1883] 0 1883 1031 30 4 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1884] 0 1884 1031 30 6 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1892] 0 1892 1031 30 8 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1910] 0 1910 1031 30 12 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1912] 0 1912 1031 30 3 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1913] 0 1913 1031 31 2 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1914] 0 1914 1031 30 12 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1916] 0 1916 1031 30 14 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1917] 0 1917 1031 30 2 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1918] 0 1918 1031 30 10 0 0 cpuspeed Jan 24 09:33:06 kernel: [ 1919] 0 1919 6859 1 10 0 0 rpc.idmapd Jan 24 09:33:06 kernel: [ 5615] 81 5615 5397 55 4 0 0 dbus-daemon Jan 24 09:33:06 kernel: [ 5626] 0 5626 47285 2 3 0 0 cupsd Jan 24 09:33:06 kernel: [ 5653] 0 5653 1033 0 9 0 0 acpid Jan 24 09:33:06 kernel: [ 5662] 68 5662 6339 167 1 0 0 hald Jan 24 09:33:06 kernel: [ 5663] 0 5663 4539 1 1 0 0 hald-runner Jan 24 09:33:06 kernel: [ 5692] 0 5692 5068 1 8 0 0 hald-addon-inpu Jan 24 09:33:06 kernel: [ 5706] 68 5706 4464 1 1 0 0 hald-addon-acpi Jan 24 09:33:06 kernel: [ 5725] 0 5725 113079 36 1 0 0 automount Jan 24 09:33:06 kernel: [ 5742] 0 5742 49251 166 9 0 0 snmpd Jan 24 09:33:06 kernel: [ 5790] 0 5790 29709 1 0 0 0 abrtd Jan 24 09:33:06 kernel: [ 5798] 0 5798 2304 19 0 0 0 abrt-dump-oops Jan 24 09:33:06 kernel: [ 5809] 496 5809 348349 267 1 0 0 qpidd Jan 24 09:33:06 kernel: [ 5856] 0 5856 29320 78 0 0 0 crond Jan 24 09:33:06 kernel: [ 5867] 0 5867 5373 4 1 0 0 atd Jan 24 09:33:06 kernel: [ 5927] 0 5927 16004 24 2 0 0 sshd Jan 24 09:33:06 kernel: [ 6296] 0 6296 43628 16 0 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6298] 0 6298 7732 1 0 0 0 rotatelogs Jan 24 09:33:06 kernel: [ 6299] 0 6299 7732 1 15 0 0 rotatelogs Jan 24 09:33:06 kernel: [ 6301] 501 6301 43628 11 0 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6302] 501 6302 43628 3 12 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6303] 501 6303 43628 3 13 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6304] 501 6304 43628 3 14 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6305] 501 6305 43628 3 11 0 0 hpsmhd Jan 24 09:33:06 kernel: [ 6315] 0 6315 1015 1 3 0 0 mingetty Jan 24 09:33:06 kernel: [ 6317] 0 6317 1015 1 11 0 0 mingetty Jan 24 09:33:06 kernel: [ 6319] 0 6319 1015 2 15 0 0 mingetty Jan 24 09:33:06 kernel: [ 6321] 0 6321 1015 1 2 0 0 mingetty Jan 24 09:33:06 kernel: [ 6323] 0 6323 2707 0 2 -17 -1000 udevd Jan 24 09:33:06 kernel: [ 6324] 0 6324 2707 0 4 -17 -1000 udevd Jan 24 09:33:06 kernel: [ 6325] 0 6325 1015 1 1 0 0 mingetty Jan 24 09:33:06 kernel: [ 6327] 0 6327 1015 1 10 0 0 mingetty Jan 24 09:33:06 kernel: [23214] 0 23214 24355 676 8 0 0 varnishncsa Jan 24 09:33:06 kernel: [23215] 0 23215 1029 35 14 0 0 cronolog Jan 24 09:33:06 kernel: [27346] 0 27346 35250 693 1 0 0 munin-node Jan 24 09:33:06 kernel: [12832] 0 12832 5301081 676327 0 0 0 java Jan 24 09:33:06 kernel: [13112] 0 13112 28072 56 8 0 0 varnishd Jan 24 09:33:06 kernel: [13113] 99 13113 15328552 11431890 2 0 0 varnishd Jan 24 09:33:06 kernel: [29597] 0 29597 35034 139 1 0 0 crond Jan 24 09:33:06 kernel: [29598] 0 29598 35250 939 0 0 0 munin-node Jan 24 09:33:06 kernel: [29599] 0 29599 2307 147 0 0 0 sh Jan 24 09:33:06 kernel: [29601] 0 29601 1551 116 0 0 0 tsar Jan 24 09:33:06 kernel: [29602] 0 29602 35834 985 13 0 0 munin-node Jan 24 09:33:06 kernel: [29603] 0 29603 2307 187 0 0 0 sh Jan 24 09:33:06 kernel: [29604] 0 29604 1164 55 9 0 0 ls Jan 24 09:33:06 kernel: Out of memory: Kill process 13113 (varnishd) score 906 or sacrifice child Jan 24 09:33:06 kernel: Killed process 13113, UID 99, (varnishd) total-vm:61314208kB, anon-rss:45725988kB, file-rss:1572kB Jan 24 09:33:11 varnishd[13112]: Child (13113) not responding to CLI, killing it. Jan 24 09:33:11 varnishd[13112]: Child (13113) not responding to CLI, killing it. Jan 24 09:33:11 varnishd[13112]: Child (13113) died signal=9 Jan 24 09:33:11 varnishd[13112]: child (29620) Started Jan 24 09:33:11 varnishd[13112]: Child (29620) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Catch.jpg Type: image/jpeg Size: 93930 bytes Desc: Catch.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Catch6F31.jpg Type: image/jpeg Size: 123268 bytes Desc: Catch6F31.jpg URL: From revirii at googlemail.com Thu Jan 31 14:02:09 2013 From: revirii at googlemail.com (=?ISO-8859-1?Q?Andreas_G=F6tzfried?=) Date: Thu, 31 Jan 2013 14:02:09 -0000 Subject: add req.http.x-forwarded-for header Message-ID: Hello, i use varnish (3.0.2) and nginx (1.2.1), and i have a special setup: http: varnish (listens on *.80) -> nginx-backend (127.0.0.1:81) https: nginx (public ip:443) -> proxy_pass to same varnish instance -> nginx-backend (127.0.0.1:81) When varnish receives the requests proxied by nginx (https), varnish sees 127.0.0.1 as source, and there seems to be no solution getting varnish see the real ip. But, as you might guess, i want the public ip (need it for performance reasons) of the user. For https i found a solution. nginx is able to pass the real ip in a header: proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; By customizing the varnishncsa log format i'm able to see the users' ip address passed by nginx. So far, so good. But i'm not able to set this header when nginx isn't involved. I tried this: At the beginning of 'sub vcl_recv': if (!req.http.X-Forwarded-For) { set req.http.X-Forwarded-For = client.ip; } My intention was: if this header isn't set (and it shouldn't when varnish is accessed directly via https), set it with the value of the client ip. I've tried a couple of variations, but in the end the value in the varnishncsa log is always empty. Well... what am i doing wrong? Where's the error? thx Andreas