From geoff at uplex.de Tue Mar 1 16:49:19 2011 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Mar 2011 17:49:19 +0100 Subject: [PATCH] backend conditional requests first release Message-ID: <4D6D238F.2030908@uplex.de> Hello all, Here's our first release of the "If-Modified-Since" function (backend conditional requests). I'll be posting up a Wiki page about how it works, and what's still to be done, and open issues. Best, Geoff Simmons -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-backend-conditional-requests-first-release.patch URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From dmitry.panov at yahoo.co.uk Tue Mar 1 19:00:56 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Tue, 01 Mar 2011 19:00:56 +0000 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6D238F.2030908@uplex.de> References: <4D6D238F.2030908@uplex.de> Message-ID: <4D6D4268.9080709@yahoo.co.uk> Hi Geoff, Thanks for the patch. Unfortunately it does not compile for me: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../include -I../../lib/libvgz -I../../include -DVARNISH_STATE_DIR='"/opt/varnish/var/varnish"' -DVARNISH_VMOD_DIR='"/opt/varnish/lib/varnish/vmods"' -DVARNISH_VCL_DIR='"/opt/varnish/etc/varnish"' -g -O2 -pthread -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wformat -O0 -g -fno-inline -DDIAGNOSTICS -Wextra -Wno-missing-field-initializers -Wno-sign-compare -fstack-protector-all -Werror -MT varnishd-cache_vrt_var.o -MD -MP -MF .deps/varnishd-cache_vrt_var.Tpo -c -o varnishd-cache_vrt_var.o `test -f 'cache_vrt_var.c' || echo './'`cache_vrt_var.c cc1: warnings being treated as errors cache_vrt_var.c:112: error: no previous prototype for ?VRT_l_stale_obj_proto? cache_vrt_var.c:113: error: no previous prototype for ?VRT_l_stale_obj_response? cache_vrt_var.c:143: error: no previous prototype for ?VRT_l_stale_obj_status? cache_vrt_var.c:388: error: no previous prototype for ?vrt_r_obj_ttl? cache_vrt_var.c:416: error: no previous prototype for ?vrt_r_obj_conditional_timeout? cache_vrt_var.c:548: error: no previous prototype for ?VRT_l_stale_obj_grace? make[3]: *** [varnishd-cache_vrt_var.o] Error 1 I haven't got time to have for closer look, hope it's something easy to fix. On 01/03/2011 16:49, Geoff Simmons wrote: > Hello all, > > Here's our first release of the "If-Modified-Since" function (backend > conditional requests). > > I'll be posting up a Wiki page about how it works, and what's still to > be done, and open issues. > > > Best, > Geoff Simmons > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -- Dmitry Panov -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Mar 1 22:55:37 2011 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Mar 2011 23:55:37 +0100 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6D4268.9080709@yahoo.co.uk> References: <4D6D238F.2030908@uplex.de> <4D6D4268.9080709@yahoo.co.uk> Message-ID: <4D6D7969.10302@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/1/11 8:00 PM, Dmitry Panov wrote: > > Thanks for the patch. Unfortunately it does not compile for me: > > gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../include > -I../../lib/libvgz -I../../include > -DVARNISH_STATE_DIR='"/opt/varnish/var/varnish"' > -DVARNISH_VMOD_DIR='"/opt/varnish/lib/varnish/vmods"' > -DVARNISH_VCL_DIR='"/opt/varnish/etc/varnish"' -g -O2 -pthread -Wall > -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type > -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align > -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs > -Wredundant-decls -Wformat -O0 -g -fno-inline -DDIAGNOSTICS -Wextra > -Wno-missing-field-initializers -Wno-sign-compare -fstack-protector-all > -Werror -MT varnishd-cache_vrt_var.o -MD -MP -MF > .deps/varnishd-cache_vrt_var.Tpo -c -o varnishd-cache_vrt_var.o `test -f > 'cache_vrt_var.c' || echo './'`cache_vrt_var.c > cc1: warnings being treated as errors > cache_vrt_var.c:112: error: no previous prototype for > ?VRT_l_stale_obj_proto? > cache_vrt_var.c:113: error: no previous prototype for > ?VRT_l_stale_obj_response? > cache_vrt_var.c:143: error: no previous prototype for > ?VRT_l_stale_obj_status? > cache_vrt_var.c:388: error: no previous prototype for ?vrt_r_obj_ttl? > cache_vrt_var.c:416: error: no previous prototype for > ?vrt_r_obj_conditional_timeout? > cache_vrt_var.c:548: error: no previous prototype for > ?VRT_l_stale_obj_grace? > make[3]: *** [varnishd-cache_vrt_var.o] Error 1 You need to compile the patch without -Werror, i.e. without - --enable-werror in the configure invocation. This brings up an implementation issue, a little complex and probably only interesting to hackers. We've added stale_obj to VCL, and in cache_vrt_var.c, we tried to avoid duplicating code by re-using macros and functions for accessing fields of stale_obj that other VCL objects also have. For example, we adapted VRT_DO_HDR() to implement access to stale_obj.response and stale_obj.proto, as it does for req, bereq and so on. But that doesn't seem to be working out, because unlike the other objects, stale_obj may be NULL, and it is read-only. So in the read functions, it has to be checked for NULL, and if so, a warning is issued and a "default" value is returned. The other objects may never be NULL, and that was enforced with CHECK_OBJ_NOTNULL. We had to remove this check, check for NULL, call CHECK_OBJ and return the value if it isn't, and return the "default" value if it is. But since the macro is used for all of the other VCL objects, they aren't being enforced as non-NULL any more. Also, a macro like VRT_DO_HDR() generates the "left-hand" functions for write access, so these functions get generated for stale_obj as well, even though write access to stale_obj is syntactically illegal. This was the cause of Dmitry's problem. A function like VRT_l_stale_obj_proto() is generated in cache_vrt_var.c by VRT_DO_HDR(), but its prototype is not generated by generate.py. A missing prototype is a warning, -Werror makes it an error, so the compile fails. Without -Werror, an implementation for VRT_l_stale_obj_proto() is created in the shared object. It never gets called, since the VCL compiler prevents write access to stale_obj, it just takes up space. It may be possible to write even more complex macros with more parameters like "nullable" and "writeable", with a lot of #if's that depend on them. But I'm beginning to think that it would be better to live with a little duplication and let have stale_obj have its own functions. BTW, I couldn't work with -Werror switched on, because there's some other code in the current trunk that won't compile with it. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNbXloAAoJEOUwvh9pJNURCCAQALGDO0CLB+J6Gpee2Vk5/bpK 8ZtzTmJGhB30WxBORoscfMQmkeE4BaU+b3EroQRCMbjE4XSBDILBoLxdYBmtIudE xh69Eqf9V88KIHnGwpxEr5A9fFpwGau35ZQTCFR1E0tznWCrnTcjkvb7SkfTjBPH p4t5XC9hEHXjeXSyWM5jniLvoRJACGARniIB7fCosPejhDsWAM1UFK6jbm99B/Nl THWTdTEW2I38oqVtGXcCZR4suVd9/ZBwfWCtiq1cb2rkWftOWtDFxQMXs///brua iDlbV0Wzi2Vs9Plfxdvnrh1yz/50Jbq1w0H9thiEQRogPhRttcMK8v//qtGHBowk RpfETUrHcPgs/h1qvQW7PoR3BED0BKiY1dZvVd1oowys6oTV+7f+RpBUYVpTrVoB S3jCT8VWyvEIUyb3H5n76dz2ICL7oX/r3u1Hhs4p6V+YHIgVGhoUrs59e+/OxRgz FXb3jj1JOPI8ZKcpt+kQGaXcP6eU+ot1Zx1EWDEd0XxR+ZURHpffY3ooGNIR+pwg rZDS+9p8X837L/8LXteU/zXSaRn6lzOr1XRk8vsK9y7SbwCK7yBdYrAHbnw60dOG xMxkLi3NaHgp4s3+6QZxOpNwi0ecpP+P+YJ1OuX4zA/0dGe57KFbopmT+hTZLUpZ HiJZr/ANMKc2ylOqCPok =2qXh -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Tue Mar 1 23:30:25 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Tue, 01 Mar 2011 23:30:25 +0000 Subject: Chunked gzip encoding problem in trunk Message-ID: <4D6D8191.1060205@yahoo.co.uk> Hi guys, If the backend returns chunked gzip data it currently results in an error response from varnish. This is because both vfp_testgzip_bytes() and vfp_gunzip_bytes() assume that the gzipped stream ends at the end of the block being read. This obviously is not happening in case of chunked gzip, because chunking is done after gzipping. I solved the problem by adding Z_OK to the condition (patch attached). Best regards, -- Dmitry Panov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: chunked_gzip.patch URL: From bzeeb-lists at lists.zabbadoz.net Wed Mar 2 00:54:17 2011 From: bzeeb-lists at lists.zabbadoz.net (Bjoern A. Zeeb) Date: Wed, 2 Mar 2011 00:54:17 +0000 (UTC) Subject: varnishnsca from 3.0-pre dumps core Message-ID: Hi, not much info yet on when, why or how it happens. It's varnish 3-pre from 2011-02-21. #0 0x0000000801755a2c in thr_kill () from /lib/libc.so.7 [New Thread 801c07400 (LWP 100141/varnishncsa)] (gdb) bt #0 0x0000000801755a2c in thr_kill () from /lib/libc.so.7 #1 0x00000008017f332b in abort () from /lib/libc.so.7 #2 0x0000000800850cfc in vas_fail_default () from /usr/local/lib/libvarnish.so.1 #3 0x00000008010977e0 in VSL_NextLog () from /usr/local/lib/libvarnishapi.so.1 #4 0x0000000801097c2b in VSL_Dispatch () from /usr/local/lib/libvarnishapi.so.1 #5 0x0000000000401990 in ?? () #6 0x00000000004014de in ?? () ... /bz -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family. From geoff at uplex.de Wed Mar 2 10:40:59 2011 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 02 Mar 2011 11:40:59 +0100 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6D238F.2030908@uplex.de> References: <4D6D238F.2030908@uplex.de> Message-ID: <4D6E1EBB.4050105@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/1/11 5:49 PM, Geoff Simmons wrote: > > Here's our first release of the "If-Modified-Since" function (backend > conditional requests). > > I'll be posting up a Wiki page about how it works, and what's still to > be done, and open issues. http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests We'd appreciate any feedback, especially about the open issues listed at the bottom of the page. Thanks, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNbh66AAoJEOUwvh9pJNURgB4P/1NrmCMCRpK9DG2S6PA/x8Yn 0DE3kS7NEdGDGI8Q9wZNF2XJ8PNigLTY/Ey6uupxW7v/MM3/9cfH05y5vAB3yhHf /sYjoKV6kLx41tQlW2ihSKQIQ7CJTcJAEqOunsydR2q3CY1Bd9xkiQCjYTZHHO80 ARg8ewRxf6otIgnDzBcsisMTl3ot5L8pelUHEjKMUJuQ08JwIqXL04xFVWKJeUPf bGYnpJB6MlKwvGkXvRS8RjtHsbpKGVKykFZYNUhsDYlkkvHQ3BrmedWAeoGgAjq9 As9oQEJrKeIH+RVr1Jal878op9zV84gzkF6p+JXWZY9kPBYBEloI/iCdzKT8UqCw Qwy3u4BiuqcJKz1STpSLpBNCGmcW6bfigZvtTHFNn1Po+j1VgY8JBMQQ2Y3lOZba Tyqdy6Kz0LL91J1w+J48SiDoUrsldvU/6V05AwoORIjBHloCQN5nsAVo97MBZwqi wwaI/Bapm0sZuSNPWrVzwNmoY1rwwFs0TPnrkB4HiacSU3N7FR1nDuZZ6+JtG1Y5 6H3zhbsnD97tEDf18vIO1mlCkoVIWDlrNRWd0/EabdSJEE9D+Q/QXUjmvvWgD7ci A/orZgyHnwf/hsEA7umDpUN+2wk52g27B8R4RbgoKtK9+3SCA7ybVJu+vP9yvSER gzepfFd4KX/h9HWzw2We =hFKQ -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Wed Mar 2 11:56:55 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 02 Mar 2011 11:56:55 +0000 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6E1EBB.4050105@uplex.de> References: <4D6D238F.2030908@uplex.de> <4D6E1EBB.4050105@uplex.de> Message-ID: <4D6E3087.9050004@yahoo.co.uk> Hi Geoff, Thanks for the patch. I have tried it and it appears to be working. I've just spotted one problem so far: it looks like the gziped flag is not copied from the stale object it is possible to serve gziped object to a client that doesn't support it. I'm new to varnish and I'm not familiar with the code yet, but the following patch solved the problem for me: --- cache_http.c.orig 2011-03-02 11:51:10.000000000 +0000 +++ cache_http.c 2011-03-02 11:05:20.000000000 +0000 @@ -988,6 +988,7 @@ http_SetResp(o->http, "HTTP/1.1", 200, "OK Not Modified"); http_SetH(o->http, HTTP_HDR_REQ, "GET"); http_copyh(o->http, sp->wrk->bereq, HTTP_HDR_URL); + sp->obj->gziped = o_stale->gziped; break; On 02/03/2011 10:40, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 3/1/11 5:49 PM, Geoff Simmons wrote: >> Here's our first release of the "If-Modified-Since" function (backend >> conditional requests). >> >> I'll be posting up a Wiki page about how it works, and what's still to >> be done, and open issues. > http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > > We'd appreciate any feedback, especially about the open issues listed at > the bottom of the page. > > > Thanks, > Geoff > - -- > UPLEX Systemoptimierung > Schwanenwik 24 > 22087 Hamburg > http://uplex.de/ > Mob: +49-176-63690917 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.14 (Darwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQIcBAEBCAAGBQJNbh66AAoJEOUwvh9pJNURgB4P/1NrmCMCRpK9DG2S6PA/x8Yn > 0DE3kS7NEdGDGI8Q9wZNF2XJ8PNigLTY/Ey6uupxW7v/MM3/9cfH05y5vAB3yhHf > /sYjoKV6kLx41tQlW2ihSKQIQ7CJTcJAEqOunsydR2q3CY1Bd9xkiQCjYTZHHO80 > ARg8ewRxf6otIgnDzBcsisMTl3ot5L8pelUHEjKMUJuQ08JwIqXL04xFVWKJeUPf > bGYnpJB6MlKwvGkXvRS8RjtHsbpKGVKykFZYNUhsDYlkkvHQ3BrmedWAeoGgAjq9 > As9oQEJrKeIH+RVr1Jal878op9zV84gzkF6p+JXWZY9kPBYBEloI/iCdzKT8UqCw > Qwy3u4BiuqcJKz1STpSLpBNCGmcW6bfigZvtTHFNn1Po+j1VgY8JBMQQ2Y3lOZba > Tyqdy6Kz0LL91J1w+J48SiDoUrsldvU/6V05AwoORIjBHloCQN5nsAVo97MBZwqi > wwaI/Bapm0sZuSNPWrVzwNmoY1rwwFs0TPnrkB4HiacSU3N7FR1nDuZZ6+JtG1Y5 > 6H3zhbsnD97tEDf18vIO1mlCkoVIWDlrNRWd0/EabdSJEE9D+Q/QXUjmvvWgD7ci > A/orZgyHnwf/hsEA7umDpUN+2wk52g27B8R4RbgoKtK9+3SCA7ybVJu+vP9yvSER > gzepfFd4KX/h9HWzw2We > =hFKQ > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev Best regards, -- Dmitry Panov From phk at phk.freebsd.dk Wed Mar 2 14:30:25 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 02 Mar 2011 14:30:25 +0000 Subject: Chunked gzip encoding problem in trunk In-Reply-To: Your message of "Tue, 01 Mar 2011 23:30:25 GMT." <4D6D8191.1060205@yahoo.co.uk> Message-ID: <23986.1299076225@critter.freebsd.dk> In message <4D6D8191.1060205 at yahoo.co.uk>, Dmitry Panov writes: >I solved the problem by adding Z_OK to the condition (patch attached). You're right. Committed. Many Thanks! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Wed Mar 2 15:15:01 2011 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 02 Mar 2011 16:15:01 +0100 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6E3087.9050004@yahoo.co.uk> References: <4D6D238F.2030908@uplex.de> <4D6E1EBB.4050105@uplex.de> <4D6E3087.9050004@yahoo.co.uk> Message-ID: <4D6E5EF5.2090609@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/2/11 12:56 PM, Dmitry Panov wrote: > > Thanks for the patch. I have tried it and it appears to be working. I've > just spotted one problem so far: it looks like the gziped flag is not > copied from the stale object it is possible to serve gziped object to a > client that doesn't support it. I'm new to varnish and I'm not familiar > with the code yet, but the following patch solved the problem for me: > > --- cache_http.c.orig 2011-03-02 11:51:10.000000000 +0000 > +++ cache_http.c 2011-03-02 11:05:20.000000000 +0000 > @@ -988,6 +988,7 @@ > http_SetResp(o->http, "HTTP/1.1", 200, "OK Not Modified"); > http_SetH(o->http, HTTP_HDR_REQ, "GET"); > http_copyh(o->http, sp->wrk->bereq, HTTP_HDR_URL); > + sp->obj->gziped = o_stale->gziped; > > break; Good catch, thanks, I've extended the test cases to make sure this works right from now on. Will be available in the next version. This is a good point, we may have overlooked fields from struct object that have to be copied over from stale_obj. We'll take a closer look; and anyone who's testing the patch might want to keep an eye out for something similar. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNbl70AAoJEOUwvh9pJNUR4RsP/3kgbnXhOaqxo+9pY/2KcMbb 30xoDjAa0gNa7HyKcv7lXVfoagS1ZFe6bZ99/iFXXBpJydWkOchjiz50rI3fL5n4 x+8Wbf/nTaCvoVNguuDIt96qSW37wSl7L3gUN4PotFzykn7Rhq+p4gKjjcQQmfsu IeHUw0+07KvgbPyaEtWTOBer27cK16GTyokvshwGMkJaaaKSBccj/AjP17+1SHu1 mxqUZmsx2/qUejK+2h9+w0G7/WBPuJXv3ZUf56ul/k8xQfAaOBeIMiwqgMJEibUE Vuh3UHFszOU0CiTo5A7cyzr1k2WzlTT54oEwGJJm6Y1+wmhLejtTnmseagJDgJfU plumAIKaJM/7csqvu5KAbg9MfuYqMQPxcjd1TOoQOFI6I7j3I5ZgoL4JWMeqQx8l XFaZYmuzlK2ir0NO5MW+kYWbTgzz4+fBVz49V2bRH5JveNSttP5mKROHTuWbprFb PZgyZD+07sdBmMuDSFOQh1yBHHtUNdMfkFXpDtHXAg8r5isJugEiM0isOCMnEmmC 0UrMrOnTYrS+93P8tTG9M315BhiV+jFmHvwhSO5ogcmCsu8MbbzR8xDhJEAct9/M HlEvM/6D6zPqDTqoO/Nj88OqzrXdZp+Rp/4haDoQ+sBEDH5M/sxoJInq3bAjZ/Rg KRPGvHcYHAIa5DlfAfRa =4kLh -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Wed Mar 2 20:55:48 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 02 Mar 2011 20:55:48 +0000 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6E5EF5.2090609@uplex.de> References: <4D6D238F.2030908@uplex.de> <4D6E1EBB.4050105@uplex.de> <4D6E3087.9050004@yahoo.co.uk> <4D6E5EF5.2090609@uplex.de> Message-ID: <4D6EAED4.1070705@yahoo.co.uk> Geoff, Another thing I've noticed is that beresp.ttl appears to be -1 in vcl_fetch after a successful validation request (i.e. the one responded with 304). With my vcl (which I took from the distribution) it causes the object to be marked as hit_for_pass. I've added beresp.status != 304 to prevent this but I think it would be better if ttl was set as if the object was retrieved in full. In my case it would be the default ttl as the backend does not provide any. On 02/03/2011 15:15, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 3/2/11 12:56 PM, Dmitry Panov wrote: >> Thanks for the patch. I have tried it and it appears to be working. I've >> just spotted one problem so far: it looks like the gziped flag is not >> copied from the stale object it is possible to serve gziped object to a >> client that doesn't support it. I'm new to varnish and I'm not familiar >> with the code yet, but the following patch solved the problem for me: >> >> --- cache_http.c.orig 2011-03-02 11:51:10.000000000 +0000 >> +++ cache_http.c 2011-03-02 11:05:20.000000000 +0000 >> @@ -988,6 +988,7 @@ >> http_SetResp(o->http, "HTTP/1.1", 200, "OK Not Modified"); >> http_SetH(o->http, HTTP_HDR_REQ, "GET"); >> http_copyh(o->http, sp->wrk->bereq, HTTP_HDR_URL); >> + sp->obj->gziped = o_stale->gziped; >> >> break; > Good catch, thanks, I've extended the test cases to make sure this works > right from now on. Will be available in the next version. > > This is a good point, we may have overlooked fields from struct object > that have to be copied over from stale_obj. We'll take a closer look; > and anyone who's testing the patch might want to keep an eye out for > something similar. > > > Best, > Geoff > - -- Best regards, -- Dmitry Panov From phk at phk.freebsd.dk Wed Mar 2 22:46:42 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 02 Mar 2011 22:46:42 +0000 Subject: [PATCH] backend conditional requests first release In-Reply-To: Your message of "Wed, 02 Mar 2011 11:40:59 +0100." <4D6E1EBB.4050105@uplex.de> Message-ID: <34200.1299106002@critter.freebsd.dk> (I talked with Geoff on IRC about one aspect of the patch which didn't look nice, the tendency for the ttl/grace stuff to leak all over the place.) I've spent the evening trying to untangle ttl/grace and generalize it in a way that will make adding the conditiona_timeout (that is sort of a long word, isn't it ?) easier to fit in. You should now be able to add the field to struct exp, add the variables to the VCC python file, macro-calls in cache_vrt_var.c and hide the rest of it in cache_expire.c Note the call to EXP_Grace() in update_object_when(), you'll need to update that as well. (I'll take care of -spersistence, please don't touch that) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fil at rezo.net Thu Mar 3 07:48:16 2011 From: fil at rezo.net (Fil) Date: Thu, 3 Mar 2011 08:48:16 +0100 Subject: a few difficulties with trunk Message-ID: Hi, I wanted to try the new gzip support. My remarks so far, and a few questions, as I haven't been able to run it yet. Building trunk is not very easy; with the help of the irc # I was able to finally do it though. And now I just learned about http://repo.varnish-cache.org/snapshots/ :^) I tried to compile my current vcl file, and got a few errors. 1. set req.http.X-Forwarded-For = (req.http.X-Forwarded-For ", " client.ip); => fails on '", "', Expected an action, 'if', '{' or '}' => OK I just learned on irc that you need a "+" 2. purge("req.http.host == " req.http.host); => fails on 'purge', Expected an action, 'if', '{' or '}' => ? 3. beresp.cacheable = false; => fails, can I safely replace it with beresp.ttl=0s; ? 4. set req.hash += req.http.host => fails, I am correct in changing it to hash_data(req.http.host) ? 5. I use a trick given by http://open.blogs.nytimes.com/2010/09/15/using-varnish-so-news-doesnt-break-your-server/ if (beresp.http.X-VARNISH-TTL) { C{ char *ttl; ttl = VRT_GetHdr(sp, HDR_BERESP, "\016X-VARNISH-TTL:"); VRT_l_beresp_ttl(sp, atoi(ttl)); }C remove beresp.http.X-VARNISH-TTL; } => fails to compile: ./vcl.1P9zoqAU.c: In function 'VGC_function_vcl_fetch': ./vcl.1P9zoqAU.c:879: warning: implicit declaration of function 'atoi' I'm currently stuck with 2 (purge) and 5 (C+atoi) thanks -- Fil From tfheen at varnish-software.com Thu Mar 3 08:06:21 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 03 Mar 2011 09:06:21 +0100 Subject: varnishnsca from 3.0-pre dumps core In-Reply-To: (Bjoern A. Zeeb's message of "Wed, 2 Mar 2011 00:54:17 +0000 (UTC)") References: Message-ID: <87k4ggljb6.fsf@qurzaw.varnish-software.com> ]] "Bjoern A. Zeeb" Hi, | not much info yet on when, why or how it happens. It's varnish 3-pre | from 2011-02-21. If you could compile varnish with debugging symbols and get us a complete backtrace that'd be useful. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Thu Mar 3 09:50:18 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 03 Mar 2011 10:50:18 +0100 Subject: a few difficulties with trunk In-Reply-To: (fil@rezo.net's message of "Thu, 3 Mar 2011 08:48:16 +0100") References: Message-ID: <87wrkgjzxh.fsf@qurzaw.varnish-software.com> ]] Fil | 2. purge("req.http.host == " req.http.host); | => fails on 'purge', Expected an action, 'if', '{' or '}' | => ? Use ban, not purge. | 3. beresp.cacheable = false; | => fails, can I safely replace it with beresp.ttl=0s; ? .cacheable is gone, setting the ttl to 0s will make it not be cached. | 5. I use a trick given by | http://open.blogs.nytimes.com/2010/09/15/using-varnish-so-news-doesnt-break-your-server/ | if (beresp.http.X-VARNISH-TTL) { | C{ | char *ttl; | ttl = VRT_GetHdr(sp, HDR_BERESP, "\016X-VARNISH-TTL:"); | VRT_l_beresp_ttl(sp, atoi(ttl)); | }C | remove beresp.http.X-VARNISH-TTL; | } | | => fails to compile: | ./vcl.1P9zoqAU.c: In function 'VGC_function_vcl_fetch': | ./vcl.1P9zoqAU.c:879: warning: implicit declaration of function 'atoi' You probably need C{ #include } near the top of the vcl file, to make sure you have the atoi function declaration around. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From geoff at uplex.de Thu Mar 3 23:41:12 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 04 Mar 2011 00:41:12 +0100 Subject: [PATCH] backend conditional requests first release In-Reply-To: <4D6EAED4.1070705@yahoo.co.uk> References: <4D6D238F.2030908@uplex.de> <4D6E1EBB.4050105@uplex.de> <4D6E3087.9050004@yahoo.co.uk> <4D6E5EF5.2090609@uplex.de> <4D6EAED4.1070705@yahoo.co.uk> Message-ID: <4D702718.9070503@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/2/11 9:55 PM, Dmitry Panov wrote: > > Another thing I've noticed is that beresp.ttl appears to be -1 in > vcl_fetch after a successful validation request (i.e. the one responded > with 304). With my vcl (which I took from the distribution) it causes > the object to be marked as hit_for_pass. > > I've added beresp.status != 304 to prevent this but I think it would be > better if ttl was set as if the object was retrieved in full. In my case > it would be the default ttl as the backend does not provide any. I quite agree with you, beresp.ttl after a 304 response should be the TTL calculated for the newly created object. @phk: The problem here is that cnt_fetch() sets the TTL to -1 just before the call to VCL_fetch_method() if the response code was 304. The code is closely following the RFC here, and only retains the computed TTL for response codes listed in Ch. 13.4. I think the best solution is just to add 304 to the cases that don't cause TTL to be set back to -1. I've tested that, and all of the regression tests pass, including the new ones for IMS. Varnish would only get a 304 from the backend if it submitted a conditional request (unless the backend is broken), so we're in control of what's happening. Would you go along with that? Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNcCcYAAoJEOUwvh9pJNURHpcQAI5pwuaWcvveMKB+EQ/6hhDe wNlAoDDm2PDtUMBbv/wvB8PHTyroTNvJf+rY3FATcCV4YQwPzDzDQ0VruRd3K4WW mdkt35d2KiTLBwi5Ufmo5WrdPKZMyFzzBndWWuIZfkmALJL17uKPtqcM05ZkAglI GyYEyrEC0/009ojuqJm5U1t0hRlwkjrfqU2felOryvn6FDYPcNog61zPl+7f1CdR QVC5AQm1TfZVxeOxJdAbRNP/H1wRCJf7DpXR6VVDqlkFIjQyNdr1aMZbXeZSztNq xJOkMViLongpxE+IhIccHcx5PvcbmW5JQDo6gixqic0ntLnzmAHst6Pig8KET/go +EH0hAVTTgvbBpWtn8oxTObVPtqj71j2YBbPGk1qnRisQ7XaxavHDN6PcDi21Xr1 Xp7qo4iIiGX0HD5Qe2TKZ6h+MW5HEnOwxPrA5Xe5IMc2SNthqSbimND4zWxjWDuS RB7wemhQgBhPeENVOdyCAnllXwv1mLJSPLnOt0ok5dH6+3nd8uZhlAUUlhlz9Zhm 2EBOiyV/yoVUZCdJEOLZL7GvFhN9VfrARp6pWlwN/mTwUan7WH3uxHn/XY1naI/2 uFXA8UvQzlxVJHPFyc/BONkQquUIsHtEh1JYHC8YUhrNYWbzRHEegHeyReLqOfHG MJeQ7SYPGOHA+uHohr51 =GQMC -----END PGP SIGNATURE----- From geoff at uplex.de Fri Mar 4 00:24:13 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 04 Mar 2011 01:24:13 +0100 Subject: [PATCH] backend conditional requests first release In-Reply-To: <34200.1299106002@critter.freebsd.dk> References: <34200.1299106002@critter.freebsd.dk> Message-ID: <4D70312D.8010102@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/2/11 11:46 PM, Poul-Henning Kamp wrote: > (I talked with Geoff on IRC about one aspect of the patch which > didn't look nice, the tendency for the ttl/grace stuff to leak > all over the place.) > > I've spent the evening trying to untangle ttl/grace and generalize it > in a way that will make adding the conditiona_timeout (that is sort > of a long word, isn't it ?) easier to fit in. > > You should now be able to add the field to struct exp, add the > variables to the VCC python file, macro-calls in cache_vrt_var.c > and hide the rest of it in cache_expire.c > > Note the call to EXP_Grace() in update_object_when(), you'll need > to update that as well. (I'll take care of -spersistence, please > don't touch that) I've integrated the update and added the functionality for IMS. Two issues have come up, one that I think is minor, the other a troubling suspicion that I haven't been able to prove. We want to be able to do this: sub vcl_fetch { set beresp.ttl = 0s; set beresp.cond_ttl = 1m; return (deliver); } ... to decide that an object should not be used to answer any more requests, but could be used for IMS. EXP_set_ttl(0) sets all of the timers to -1, including cond_ttl, as we discussed. If cond_ttl then gets set to non-zero, HSH_Lookup() should be able to find it as a possible stale_obj. The problem was that HSH_Lookup() couldn't get past this line in the loop: if (o->exp.ttl <= 0.0) continue; So I changed that to: if (o->exp.ttl <= 0. && o->exp.grace <= 0. && o->exp.cond_ttl <= 0.) continue; Then it works as expected, all test cases pass. We also want to be able to say: sub vcl_hit { set obj.ttl = 0s; set obj.cond_ttl = 1m; } ... to do the same thing with the object in cache. EXP_set_ttl(0) also sets all the timers to -1, and EXP_set_ttl() calls EXP_Rearm(). Now I have EXP_set_cond_ttl() doing both of these as well. I worry about a race condition here. After EXP_set_ttl(0) calls EXP_Rearm(), it seems to me that the object may be cleared from cache before EXP_set_cond_ttl() gets a chance to change cond_ttl and call EXP_Rearm() again. I've tried to make that happen by having VCL busily evaluate a lot of expensive regexes between the set obj.ttl and set obj.cond_ttl calls. So far, all of the test cases I've tried do the right thing anyway. But don't we have a risk here? Or am I missing something that would prevent the problem? Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNcDEtAAoJEOUwvh9pJNUR4j4P/2tIyCMj6apj0zIRRYJcJyoI BfKcE2in/t5zkYPuP3GfiH+cPB+qM8QsjiVi/pj1vGrCYhJBUaPJmjNwR6TyTnF/ b/F9GwWkdbbQXmbEGHKWg1RYA977TSZ4SLCzyr5aszwnoHaDjS0ZCsYjLquczFyF wkim9sknJnuBAfDMNKO+JZ1DWP1HczE7KJXXW1Kv0zYIPFdoUzQNfJpu45YUYn6/ WWgzYVQKbU15q2CEF60dFDYqMDed7AJ1NXZI2KuMSd4gjBylZR8+YO0wHSs3REL7 QWakbOBGWgraot6/t6EUuJtZMNNM5ZKa4ZumdAyUUTGJF8UktDXIm3JdDJV9/hRC 5LajAtd5dtq9ebHzn73bNdMPXajbV+tjmyq3EDpYEdLJUFGWpxU3oZgpg6appZtx KFYpqj+PW26ZxoGLfzpRFD8wvfjt75YLsS0llkTq0t7/su44QmfchW7nwp53fJji MWByIdir03KFPZQhZgQksKzvW4XFRLxBUF/16Al9/46BOkYnAW/soPWzGOxMHpZe 02a6q5Us1ZhVf6CjvdixhXU8bxuqTC++mVmJrNhedCqLRbDsjqQrSrjqPi2ar7N8 9CPq356NIEps6bh6wcKt4msKCNMalfX2vwZq6yodEBDPjsqaCFUGiuYyh1cBodIZ ADoqSw51EGQ3lyV3rSxB =l233 -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Fri Mar 4 14:35:49 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Fri, 04 Mar 2011 14:35:49 +0000 Subject: Date parsing/formatting and type coercion functions in VCL Message-ID: <4D70F8C5.7020508@yahoo.co.uk> Hi guys, From RFC2616: Also, if the response does have a Last-Modified time, the heuristic expiration value SHOULD be no more than some fraction of the interval since that time. A typical setting of this fraction might be 10%. Some browsers (at least Firefox -- see https://developer.mozilla.org/en/HTTP_Caching_FAQ) seem to follow this recommendation. Vernish does not implement it which is fair enough and if someone wanted it they could use an inline C like this: if ((beresp.ttl == 120s || !beresp.http.Cache-Control) && !beresp.http.Expires && beresp.http.Date && beresp.http.Last-Modified) { C{ char *lm = VRT_GetHdr(sp, HDR_BERESP, "\016last-modified:"); char *d = VRT_GetHdr(sp, HDR_BERESP, "\005date:"); if (lm && d) { time_t lm_t, d_t; lm_t = TIM_parse(lm); d_t = TIM_parse(d); if (d_t > 0 && d_t > lm_t) { double ma = (d_t - lm_t) / 10; VRT_l_beresp_ttl(sp, ma); char buf[16]; snprintf(buf, sizeof(buf), "max-age=%d", (int)ma); VRT_SetHdr(sp, HDR_BERESP, "\016Cache-Control:", buf, vrt_magic_string_end); } } }C } However I just thought, wouldn't it be better if someone could write something like this: if (beresp.default_ttl_used && beresp.http.Date && beresp.http.Last-Modified) { beresp.ttl = (ParseDate(beresp.http.Date) - ParseDate(beresp.http.Last-Modified))/10; beresp.http.Cache-Control = "max-age="+int(beresp.ttl); } What's missing here: - default_ttl_used flag which should be set to true if no cache validators were found in the backend response (this should be easy to implement with the new exp structure). - ParseDate() function which could be just a wrapper around TIM_parse(). - int() function which should convert the argument to integer. All these don't seem very hard to implement and it would be really helpful in many cases. What do you think? Best regards, -- Dmitry Panov From geoff at uplex.de Fri Mar 4 14:39:10 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 04 Mar 2011 15:39:10 +0100 Subject: [PATCH] backend conditional requests second release In-Reply-To: <4D70312D.8010102@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> Message-ID: <4D70F98E.4020902@uplex.de> Hello all, Here's the second release of the "If-Modified-Since" function (backend conditional requests). The main changes since the version from March 1st: - now using phk's EXP_* interface for managing the timers, and extends it to manage conditional timeout - the name "conditional_timeout" is shortened to "cond_ttl", as discussed on IRC - compiles without warnings (does not fail on -Werror) - fixed the problem that beresp.ttl was always -1 immediately after a 304 validation - now properly delivers validated gzipped content to clients (the last three issues spotted by Dmitry Panov, thanks Dmitry) - eliminated the stats counter cache_cond_not_modified, which was redundant because Varnish already has fetch_304 - re-named the other new counter to cond_not_validated, which is now incremented for any non-304 backend response to a conditional request We'd be grateful for any more testing and feedback! Best, Geoff Simmons -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: backend_conditionals_2nd_release.patch URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Sat Mar 5 18:00:52 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sat, 05 Mar 2011 19:00:52 +0100 Subject: [PATCH] backend conditional requests second release In-Reply-To: <4D70F98E.4020902@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> Message-ID: <4D727A54.6040005@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/4/11 3:39 PM, Geoff Simmons wrote: > > Here's the second release of the "If-Modified-Since" function (backend > conditional requests). I've also updated the Wiki page: http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNcnpTAAoJEOUwvh9pJNURAmcP/34nI68NwhNuKZVMOaeURkly 6QT8tholzy1KiBZ7eqIZlSWmQmNcw0852OPpacBpMGlKyMPToWQyVWD79I3E+HXy AI6f83KpnG2QAyH5MVe9Mfx46M5w23Ex3Y28WJworQrTAm7OqhhhdPlo8qFz7FvB hfwgCB9A/o1rkSEiQedQzh23xn1PBgDiVodWId4tkBVI2kieTKKvyzsv5SxDgjeG 8nQk+2nydG2yl2N+FtEQX2RMcm3ZBsM1JTZzUtc2tlAHMZapJXHRCEKDnyMP5Tii 1JKddjg5+5u3VzanYFyGZ94FbTwKQPd6RP/nBm6gs2teoDaJRzBBaiRYib6ko1VE HCnyTtjyERZIRdU7Y3jUiZz+yBCv0UvSlQlewUco7WxOGPhRdvhN/HlbL5EtJ3Bk cu2UNv49qEkyIMQdeqJ0fOU7zAR8xPzmZdqhCxgt5L2Ewqy0ZKCJqweinAcnL1fl IdZ+MjMqTj4QRrUkH6RaUe3nYejO08uI02o4W9AeLGazL+hk8uyfISqm60vL9bN8 ae3C+XaHenDxtlIZ5BzHwZKON2Vf+HVl1AfRwBV+YkhQDf+WsxTkRLWuYY+CRKBh 5W0I4rmco6v13kZKCz29jiUsY0p0yYBUNzU15n7kLkxqMMyse1uJjNf8cjDv4jwD 39zp2R7w8r/RIZCut0G+ =jXQu -----END PGP SIGNATURE----- From geoff at uplex.de Sun Mar 6 20:47:31 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 06 Mar 2011 21:47:31 +0100 Subject: [PATCH] backend conditional requests second release In-Reply-To: <4D727A54.6040005@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> Message-ID: <4D73F2E3.9050506@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello, A head's up for any of you who may be trying out this patch -- I've been testing it under load with httperf, and have encountered some assertion failures. It's definitely caused by the patch -- I can see that in the backtraces, and the crash doesn't happen in the trunk version to which I added the extension. I believe there's a memory leak, related either to the duplication of the stale object's storage into the new object after validation, or not properly discarding the stale object after it's no longer needed. Will post a fix ASAP. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNc/LjAAoJEOUwvh9pJNURFfEP/iGGHPqYv2sxvR+4STPshM6p K5ExzNviGM3NTJ3EiqJ1TsrpTKL2/b/2p7djmjcDZmUWm+NsFqHz+L2aL5SeADXv bXETJw1Yx8KYMbrDA4jj47dWNeWF9MkjrX5GhYEKqCkGajsSc3pVTo+4LwzMPsAz Fc2ZPpJw+Vm+X/4cl0EPzE2Sf2/2ZMNyr8dyaavAFXMDLkGV00yiX9WA6D6RihWL SuRHjW6PJmNSGdgFEb9uwtTfOqt6JTNK0cGanv8qV/sqm/p8CfB051ShbuH0O45w cLKgW1g50H9BghdJ/igNyHo6OZjBUF9QZWE9Af6dbwPnn1fGug7cfS2TX56qwkGo 25UP/P2EGB9hr0q37NIb3cL0PuPaS2H1VqlKnpQexlAc7tMT72rHMyfqYGhlELhB SQDSBlE7oq9jeUd89ySNPn1UgTt2Bzuathn0TKwACnlMWvWZzy4inRrDVlApno4l RS4oaSxqISzA2i9p/nFSOLjQOEdRDu3E3FQd90uBJoQafLPtK6/l2FydTa8ixTl8 DztscathlkCJnFtSQ4s4lw5j+3n/5cYPK/VwdPXOVuMZLDCcHZBQF+GgrNySMLuV hMTfj1QoETRIOQ+Ca6Nrc+PhxKQm2ej12nlm6WaVrAdTx5A2H0xwWfhdD9RnIMYb o/Etzj63W6RZrW8ZQJUW =hO5w -----END PGP SIGNATURE----- From geoff at uplex.de Mon Mar 7 10:35:49 2011 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 07 Mar 2011 11:35:49 +0100 Subject: [PATCH] backend conditional requests third release In-Reply-To: <4D73F2E3.9050506@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> Message-ID: <4D74B505.7080709@uplex.de> Hello everyone, This is the third version of the conditional backend request feature, which fixes the bug I mentioned on the -dev list yesterday -- a memory leak and assertion failures during load tests. With this change I can now run load tests with httperf without any such errors. The load test isn't much of a performance test for Varnish -- I'm running httperf, Varnish and an Apache backend all on my machine, getting a maximum client request rate of about 18000/s, no challenge for Varnish. The point was to find out errors that only show themselves under load, and to see the differences with and without the IMS feature. So if any of you are able to test the patch under a heavy load, it would certainly be worthwhile. For those who are interested in what the problem was: I was arranging for removal of the stale_obj after it's no longer needed by doing this (simplified a bit): EXP_Clr(&stale_obj->exp); HSH_Deref(&stale_obj); if (stale_obj == NULL) EXP_Rearm(stale_obj); But I had overlooked that HSH_Deref(o) always sets o to NULL. So EXP_Rearm() was never called, and the stale_obj's continued to take up space in storage, enough so that malloc storage was eventually full. Eventually when STV_NewObject() was called to create a new object from the backend response during fetch, the allocation call sma_alloc() hit the limit sma_max, causing it to return NULL, and that leads to assert failures. The fix was just to call EXP_Rearm() before HSH_Deref(). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.conditional_backend_reqs URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Mon Mar 7 17:19:50 2011 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 07 Mar 2011 18:19:50 +0100 Subject: [PATCH] backend conditional requests 4th release In-Reply-To: <4D74B505.7080709@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> Message-ID: <4D7513B6.2040002@uplex.de> Hello all, Here's yet another version of the patch for backend conditional requests (IMS). There's no change of function, it's just necessary because some updates have been committed to the trunk since this morning, so you'll need this version of the patch to try it with the current trunk. This patch applies to version 25c5f2ed3229e41e99eadff57374c3a93b41a356 (Mon Mar 7 11:09:03 2011 +0000). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.conditional_backend_reqs_4 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From jdzstz at gmail.com Mon Mar 7 21:56:34 2011 From: jdzstz at gmail.com (jdzstz - gmail dot com) Date: Mon, 7 Mar 2011 22:56:34 +0100 Subject: Updated varnish patches for cygwin 1.7.8 Message-ID: I have updated my varnish patches for new cygwin versi?n 1.7.8 Cygwin 1.7.8 has some improvements and bugfixes, so, I have removed some of the workarounds of previous patches. The changes affects following files: - bin/varnishd/Makefile.am - configure.ac - lib/libvarnishapi/Makefile.am - lib/libvcl/Makefile.am - lib/libvmod_std/Makefile.am * Version 2.1.5: http://www.varnish-cache.org/trac/attachment/wiki/VarnishOnCygwinWindows/varnish-2.1.5-cygwin.patch * Trunk: http://www.varnish-cache.org/trac/attachment/wiki/VarnishOnCygwinWindows/varnish-githead-cygwin.patch -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-githead-cygwin.patch Type: application/octet-stream Size: 5438 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-2.1.5-cygwin.patch Type: application/octet-stream Size: 5034 bytes Desc: not available URL: From tfheen at varnish-software.com Tue Mar 8 10:07:37 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 08 Mar 2011 11:07:37 +0100 Subject: Updated varnish patches for cygwin 1.7.8 In-Reply-To: (jdzstz@gmail.com's message of "Mon, 7 Mar 2011 22:56:34 +0100") References: Message-ID: <877hca2ady.fsf@qurzaw.varnish-software.com> ]] jdzstz - gmail dot com Hi, | I have updated my varnish patches for new cygwin versi?n 1.7.8 Thanks, | * Version 2.1.5: | http://www.varnish-cache.org/trac/attachment/wiki/VarnishOnCygwinWindows/varnish-2.1.5-cygwin.patch While you're of course free to keep this around, I'm not going to merge any cygwin support to the 2.1 branch. | * Trunk: http://www.varnish-cache.org/trac/attachment/wiki/VarnishOnCygwinWindows/varnish-githead-cygwin.patch In general, I'm a bit unhappy with most of this patch as it leaks too much cygwin stuff across the build system. I'll comment some issues inline. | diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am | index b8db6e0..f920b42 100644 | --- a/bin/varnishd/Makefile.am | +++ b/bin/varnishd/Makefile.am | + | +EXTRA_SCRIPTS = | + | +sbin_SCRIPTS = ${CYGWIN_libvarnishd_OBJECTS} | + | + | DISTCLEANFILES = default_vcl.h Why the extra EXTRA_SCRIPTS? I'd also like to not have to care about gcc on cygwin having problems linking a DLL with unresolved symbols in it. | # | diff --git a/configure.ac b/configure.ac | index f574764..9f7753e 100644 | --- a/configure.ac | +++ b/configure.ac This bit has been merged. | @@ -457,6 +488,19 @@ else | *-*-darwin*) | VCC_CC="exec cc $OCFLAGS -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" | ;; | + *-*-cygwin*) | + #calculate ${exec_prefix}/lib, because it is asigned later, and we need now | + prefix_aux=`eval echo ${prefix}` | + test "x$prefix_aux" = xNONE && prefix_aux=`eval echo ${ac_default_prefix}` | + # Let make expand exec_prefix. | + exec_prefix_aux=`eval echo ${exec_prefix}` | + test "x$exec_prefix_aux" = xNONE && exec_prefix_aux=`eval echo ${prefix_aux}` | + # Remove "/" if we are in Root | + test "x$exec_prefix_aux" = "x/" && exec_prefix_aux="" | + #calculate compilation path | + comp_prefix=`pwd` | + VCC_CC="$PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS -shared %s -L${comp_prefix}/lib/libvarnish/.libs -L${comp_prefix}/bin/varnishd -L${exec_prefix_aux}/lib -L${exec_prefix_aux}/sbin -L${exec_prefix_aux}/bin -lvarnish -lvarnishd -o %o" | + ;; Adding those -L bits here looks a bit odd. Doesn't libtool just add the necessary rpath if that's needed? | +# Add library dependencies with libvarnish (linker needs it in CYGWIN during compilation time) | +# Generate fake library libvarnishd.dll.a in order to successful linking DLL when resolving varnishd.exe simbols | +case $target in | +*-*-cygwin*) | + CYGWIN_varnishd_LDFLAGS=' -Wl,-out-implib,libvarnishd.dll.a' | + CYGWIN_libvarnishd_OBJECTS='libvarnishd.dll.a' | + CYGWIN_libvmod_std_la_LIBADD='$(top_builddir)/lib/libvarnish/libvarnish.la -lvarnishd -L$(top_builddir)/bin/varnishd' | + AC_SUBST(CYGWIN_varnishd_LDFLAGS) | + AC_SUBST(CYGWIN_libvarnishd_OBJECTS) | + AC_SUBST(CYGWIN_libvmod_std_la_LIBADD) | + cygwin=true | + ;; | +*) | + cygwin=false | + ;; | +esac | + | +AM_CONDITIONAL([CYGWIN_COMPILATION], test x$cygwin = xtrue) Can we get rid of that fake library? Also, doesn't cygwin support inter-library dependencies properly? | --- a/lib/libvarnishapi/Makefile.am | +++ b/lib/libvarnishapi/Makefile.am | @@ -26,4 +26,6 @@ libvarnishapi_la_SOURCES = \ | libvarnishapi_la_CFLAGS = \ | -DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' | | -libvarnishapi_la_LIBADD = @PCRE_LIBS@ | +libvarnishapi_la_LIBADD = \ | + $(top_builddir)/lib/libvarnish/libvarnish.la \ | + @PCRE_LIBS@ Why is this? | diff --git a/lib/libvcl/Makefile.am b/lib/libvcl/Makefile.am | index aab8749..4645bba 100644 | --- a/lib/libvcl/Makefile.am | +++ b/lib/libvcl/Makefile.am | @@ -33,6 +33,8 @@ libvcl_la_SOURCES = \ | vcc_vmod.c \ | vcc_xref.c | | +libvcl_la_LIBADD = $(top_builddir)/lib/libvarnish/libvarnish.la | + | EXTRA_DIST = \ | generate.py What does this solve for you? | diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am | index 36147fe..41c32c8 100644 | --- a/lib/libvmod_std/Makefile.am | +++ b/lib/libvmod_std/Makefile.am | @@ -13,9 +13,16 @@ libvmod_std_la_SOURCES = \ | vmod_std.c \ | vmod_std_fileread.c | | +libvmod_std_la_LIBADD = ${CYGWIN_libvmod_std_la_LIBADD} | + | vcc_if.c vcc_if.h: $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc | @PYTHON@ $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc | | EXTRA_DIST = vmod.py vmod.vcc | | CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h | + | +if CYGWIN_COMPILATION | +install-exec-hook: | + $(LN_S) -f cygvmod_std-1.dll .libs/libvmod_std.so.1 | +endif What? You shouldn't symlink stuff into .libs directories. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From geoff at uplex.de Wed Mar 9 10:57:38 2011 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 09 Mar 2011 11:57:38 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D7513B6.2040002@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> Message-ID: <4D775D22.2070202@uplex.de> Hello again, And yet another version (v5) of the patch for conditional backend requests. Like the last version, this is one is to keep pace with recent commits to the trunk. The patch applies to: commit 13c60b522a6c7abe9cc1052c2a244f25f2ba5227 Date: Tue Mar 8 14:47:57 2011 +0100 phk committed some changes related to this feature, mainly the addition of the new timer variable to VCL. There are also some changes in the way that previous version of the patch worked. - The name of the timer for conditional requests changed again, now to "keep". The management parameter is "default_keep". - In addition to beresp.keep, obj.keep and stale_obj.keep, there is now req.keep, which sets an upper bound for the keep time within a session. (Like req.ttl, which phk added yesterday.) - The keep interval now begins after ttl and grace have elapsed, so an object is available for conditional requests after obj.ttl + obj.grace, and is scheduled for eviction after obj.ttl + obj.grace + obj.keep. (In previous versions of the patch, the keep interval began after ttl elapses, independent of grace, and object were evicted after obj.ttl + max(obj.grace, obj.keep).) - The default value default_keep is now 0, so you need to set a non-zero default_keep, or set keep in VCL, to make the conditional requests happen. The Wiki page has been updated to reflect all of this: http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests @phk: I made one change in the commits from yesterday. Since default_keep = 0 and EXP_get_*() was returning -1 for any timer = 0, *.keep was always -1 unless it was changed in VCL. I think VCL authors wouldn't expect that; I got feedback about an earlier version in which beresp.keep was -1 after a backend validation, which both the reporter and I thought was wrong, so I fixed it in the next version. Now I have this in cache_expire.c: double \ EXP_Get_##fld(const struct exp *e) \ { \ - return (e->fld > 0. ? e->fld : -1.); \ + return (e->fld > 0. ? e->fld : 0.); \ } \ This required a change in test case b00040.vtc, which expects req.grace == -1, but that doesn't seem to be a critical value there. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.conditional_backend_reqs_5 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From dmitry.panov at yahoo.co.uk Wed Mar 9 14:17:50 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 09 Mar 2011 14:17:50 +0000 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D775D22.2070202@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> Message-ID: <4D778C0E.60303@yahoo.co.uk> Hi Geoff, Just a heads up, I'm getting assertion failures when running a rather simple testcase: using local apache that serves /user/share/doc as the backend and running wget -r http://localhost:6802/doc Shortly after that the following errors start to appear: Child (11125) Panic message: Assert error in http_Write(), cache_http.c line 1181: Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. thread = (cache-worker) I have been able to reproduce it on 2 different machines with very different hardware configurations which makes hardware problem quite unlikely. Also httperf --server localhost --port 6802 --uri / --num-conns 1 --num-calls 4000 runs without a problem. These 2 machines both run 32bit linux kernel. I haven't been able to reproduce the problem in a 64bit environment. Unfortunately I haven't got time to try the unpatched trunk (I tried it with revisions 3 and 4 of the patch) or do any further experiments but I'll try to do so in the next couple of days and then post more details. On 09/03/2011 10:57, Geoff Simmons wrote: > Hello again, > > And yet another version (v5) of the patch for conditional backend > requests. Like the last version, this is one is to keep pace with recent > commits to the trunk. The patch applies to: > > commit 13c60b522a6c7abe9cc1052c2a244f25f2ba5227 > Date: Tue Mar 8 14:47:57 2011 +0100 > > phk committed some changes related to this feature, mainly the addition > of the new timer variable to VCL. There are also some changes in the way > that previous version of the patch worked. > > - The name of the timer for conditional requests changed again, now to > "keep". The management parameter is "default_keep". > > - In addition to beresp.keep, obj.keep and stale_obj.keep, there is now > req.keep, which sets an upper bound for the keep time within a session. > (Like req.ttl, which phk added yesterday.) > > - The keep interval now begins after ttl and grace have elapsed, so an > object is available for conditional requests after obj.ttl + obj.grace, > and is scheduled for eviction after obj.ttl + obj.grace + obj.keep. (In > previous versions of the patch, the keep interval began after ttl > elapses, independent of grace, and object were evicted after obj.ttl + > max(obj.grace, obj.keep).) > > - The default value default_keep is now 0, so you need to set a non-zero > default_keep, or set keep in VCL, to make the conditional requests happen. > > The Wiki page has been updated to reflect all of this: > > http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > > @phk: I made one change in the commits from yesterday. Since > default_keep = 0 and EXP_get_*() was returning -1 for any timer = 0, > *.keep was always -1 unless it was changed in VCL. I think VCL authors > wouldn't expect that; I got feedback about an earlier version in which > beresp.keep was -1 after a backend validation, which both the reporter > and I thought was wrong, so I fixed it in the next version. > > Now I have this in cache_expire.c: > > double \ > EXP_Get_##fld(const struct exp *e) \ > { \ > - return (e->fld> 0. ? e->fld : -1.); \ > + return (e->fld> 0. ? e->fld : 0.); \ > } \ > > This required a change in test case b00040.vtc, which expects req.grace > == -1, but that doesn't seem to be a critical value there. > > Best regards, -- Dmitry Panov -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Mar 9 14:51:33 2011 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 09 Mar 2011 15:51:33 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D778C0E.60303@yahoo.co.uk> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D778C0E.60303@yahoo.co.uk> Message-ID: <4D7793F5.1080705@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/ 9/11 03:17 PM, Dmitry Panov wrote: > > Just a heads up, I'm getting assertion failures when running a rather > simple testcase: using local apache that serves /user/share/doc as the > backend and running wget -r http://localhost:6802/doc Shortly after that > the following errors start to appear: > > Child (11125) Panic message: Assert error in http_Write(), cache_http.c > line 1181: > Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. > thread = (cache-worker) Thanks for the heads up. Can you send over the whole stack trace? > I have been able to reproduce it on 2 different machines with very > different hardware configurations which makes hardware problem quite > unlikely. Also > > httperf --server localhost --port 6802 --uri / --num-conns 1 > --num-calls 4000 > > runs without a problem. > > These 2 machines both run 32bit linux kernel. I haven't been able to > reproduce the problem in a 64bit environment. Could be running out of workspace. I fixed a similar error during the course of development, which had to do with the fact that sufficient workspace has to be allocated for the both backend response *and* the stale object; you might have found something related. Also, I've only been testing with 64 bit; looks like I better test 32 bit as well. Is there any way you can send the request & response that are being processed when the error happens? And what if you set --num-conns high and --num-calls low, say 400 connections and 10 calls per connection? Or keep setting --num-conns higher, to see if you can provoke the error? I've been running httperf with 25,000 connections and 1000 calls per connection, found a memory leak that way. > Unfortunately I haven't got time to try the unpatched trunk (I tried it > with revisions 3 and 4 of the patch) or do any further experiments but > I'll try to do so in the next couple of days and then post more details. It's a good idea to test on the unpatched trunk as well, to make sure that the bug really comes from the patch. Thanks very much for the feedback! Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNd5P1AAoJEOUwvh9pJNURmhYP/1euNCn2s9EjMTWcTu8dzLiC oWKqBa6CaifqetEhrpQKh+ow8lcIGMPkIEoa8y9GDkdF7p68GGKaJmahGWAPcml7 uM/aFZFINWho52KRQoPA/iS1sdjHovIxmArIOovQV5TIo/p/YiPoEQw06+EMm08p Q9y1OHdK0XA87S3vYyJ0rcpkg6XcEPxfVKmtp/vzq2Zgz2pK35aisx7vYdLFPQyB J6tyJlseQ3ut2TVlZFSD29Z+C4HX0zvvxByM3LUUeKA4PtFB4ItaPu4Yk10utshX lAeZa3I77H1IC3k3cIFbsSnRKJCeBC71M//L0ysn1lL/7824f6BZmtyYPvtfpGWI 3Pl75RHgAdbUxDtyEQnncyn/A3aIHnUbDdwEwv3L6DW7FrrCNdrscG0mlHDnwQP3 EWm/hsGQGkz3F30CYHeu54sRVNPCbTGrq5VybyNFIJvA13leJU0WCWG+WvRqjX5T CPWgw9jfncFQ2/xdbV9adP+R/Mky/XaYovL2RKXJ7Nc/M1xa056iqW6l/chxRT55 cXhCt3cmskT33nYR8Oqw9JoUWoaRajqMf20ReBK30a3WB2gLwnIrwR3DIQE+BlMW gk2ywQuTINaqDubzJVsqhFNVH7sU42bi+dqHBNNu7SxN9+Ft9ILt4zKNJjZwrC84 yUg7xha6a9a3sEqNL4P4 =WtW9 -----END PGP SIGNATURE----- From slink at schokola.de Wed Mar 9 17:48:35 2011 From: slink at schokola.de (Nils Goroll) Date: Wed, 09 Mar 2011 18:48:35 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D775D22.2070202@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> Message-ID: <4D77BD73.7080200@schokola.de> Hi, > - The keep interval now begins after ttl and grace have elapsed, so an > object is available for conditional requests after obj.ttl + obj.grace, > and is scheduled for eviction after obj.ttl + obj.grace + obj.keep. (In > previous versions of the patch, the keep interval began after ttl > elapses, independent of grace, and object were evicted after obj.ttl + > max(obj.grace, obj.keep).) I would like to add that I found obj.ttl + max(obj.grace, obj.keep) more intuitive. But, again, we all know about bikeshed paint. Nils From dmitry.panov at yahoo.co.uk Wed Mar 9 19:03:24 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 09 Mar 2011 19:03:24 +0000 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D7793F5.1080705@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D778C0E.60303@yahoo.co.uk> <4D7793F5.1080705@uplex.de> Message-ID: <4D77CEFC.3040906@yahoo.co.uk> Ok, I have reproduced the bug on the unpatched trunk (revision 25c5f2ed3229e41e99eadff57374c3a93b41a356) without using custom vcl (the only section I have there is the backend specification). Command to run varnish was: /opt/varnish/sbin/varnishd \ -a 0.0.0.0:6802 \ -f /opt/varnish/etc/varnish/my.vcl \ -P /var/run/varnishd.pid \ -T 127.0.0.1:2000 \ -d \ -s file,/opt/varnish/var/varnish/storage.bin,1G The system is running Debian with 32 bit kernel. As I mentioned earlier I was able to reproduce the problem on another machine with significantly different hardware configuration. The only common thing was that they were running debian with 32bit kernel. Also I used the same binaries on both machines. I could not reproduce the problem in 64 bit environment. I'm attaching the stack trace and the log file. Please let me know if I can provide any more info. On 09/03/2011 14:51, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 03/ 9/11 03:17 PM, Dmitry Panov wrote: >> Just a heads up, I'm getting assertion failures when running a rather >> simple testcase: using local apache that serves /user/share/doc as the >> backend and running wget -r http://localhost:6802/doc Shortly after that >> the following errors start to appear: >> >> Child (11125) Panic message: Assert error in http_Write(), cache_http.c >> line 1181: >> Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. >> thread = (cache-worker) > Thanks for the heads up. Can you send over the whole stack trace? > >> I have been able to reproduce it on 2 different machines with very >> different hardware configurations which makes hardware problem quite >> unlikely. Also >> >> httperf --server localhost --port 6802 --uri / --num-conns 1 >> --num-calls 4000 >> >> runs without a problem. >> >> These 2 machines both run 32bit linux kernel. I haven't been able to >> reproduce the problem in a 64bit environment. > Could be running out of workspace. I fixed a similar error during the > course of development, which had to do with the fact that sufficient > workspace has to be allocated for the both backend response *and* the > stale object; you might have found something related. Also, I've only > been testing with 64 bit; looks like I better test 32 bit as well. > > Is there any way you can send the request& response that are being > processed when the error happens? > > And what if you set --num-conns high and --num-calls low, say 400 > connections and 10 calls per connection? Or keep setting --num-conns > higher, to see if you can provoke the error? I've been running httperf > with 25,000 connections and 1000 calls per connection, found a memory > leak that way. > >> Unfortunately I haven't got time to try the unpatched trunk (I tried it >> with revisions 3 and 4 of the patch) or do any further experiments but >> I'll try to do so in the next couple of days and then post more details. > It's a good idea to test on the unpatched trunk as well, to make sure > that the bug really comes from the patch. > > Thanks very much for the feedback! > > Best regards, -- Dmitry Panov -------------- next part -------------- A non-text attachment was scrubbed... Name: logfile.txt.gz Type: application/x-gzip Size: 12618 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: trace.txt URL: From dmitry.panov at yahoo.co.uk Wed Mar 9 20:18:34 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 09 Mar 2011 20:18:34 +0000 Subject: Workspace overflow on ia32 In-Reply-To: <4D77CEFC.3040906@yahoo.co.uk> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D778C0E.60303@yahoo.co.uk> <4D7793F5.1080705@uplex.de> <4D77CEFC.3040906@yahoo.co.uk> Message-ID: <4D77E09A.6090902@yahoo.co.uk> Oops, my bad, I didn't realise the unpatched trunk installed itself in a different directory, so I was in fact running a patched version. However after I fixed this, I could still reproduce the problem. Here is the updated stack trace: Child (15077) died signal=6 Child (15077) Panic message: Assert error in http_Write(), cache_http.c line 1063: Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. thread = (cache-worker) ident = Linux,2.6.26-2-686,i686,-sfile,-smalloc,-hcritbit,epoll Backtrace: 0x807eaa5: pan_backtrace+24 0x807ed4e: pan_ic+193 0x807b79f: http_Write+e6 0x8083beb: RES_WriteObj+1cb 0x805ec3f: cnt_deliver+5e6 0x8062dd6: CNT_Session+6ae 0x8081221: wrk_do_cnt_sess+160 0x80809af: wrk_thread_real+d36 0x8080e1c: wrk_thread+109 0xb76df955: _end+af60df25 sp = 0xb7493004 { fd = 11, id = 11, xid = 1045926360, client = 127.0.0.1 51657, step = STP_DELIVER, handling = deliver, err_code = 200, err_reason = (null), restarts = 0, esi_level = 0 ws = 0xb7493054 { id = "sess", {s,f,r,e} = {0xb74937f4,+220,(nil),+16384}, }, http[req] = { ws = 0xb7493054[sess] "GET", "/doc/dvd+rw-tools/", "HTTP/1.0", "Referer: http://localhost:6802/doc/", "User-Agent: Wget/1.11.4", "Accept: */*", "Host: localhost:6802", "Connection: Keep-Alive", "X-Forwarded-For: 127.0.0.1", }, worker = 0x6e28c0ec { ws = 0x6e28c220 { overflow id = "wrk", {s,f,r,e} = {0x6e285fc0,+16384,(nil),+16384}, }, http[resp] = { ws = 0x6e28c220[wrk] "HTTP/1.1", "OK", "Server: Apache/2.2.9 (Debian) proxy_html/3.0.1", "Last-Modified: Mon, 23 Jun 2008 14:32:23 GMT", "ETag: "2222-fc35-450564fcbabc0"", "Content-Type: text/html", "Content-Length: 64565", "Accept-Ranges: bytes", "Via: 1.1 varnish", }, }, vcl = { srcname = { "input", "Default", }, }, obj = 0x8fc91000 { xid = 1045926360, ws = 0x8fc91010 { id = "obj", {s,f,r,e} = {0x8fc91140,+228,(nil),+248}, }, http[obj] = { ws = 0x8fc91010[obj] "HTTP/1.1", "OK", "Date: Wed, 09 Mar 2011 20:12:14 GMT", "Server: Apache/2.2.9 (Debian) proxy_html/3.0.1", "Last-Modified: Mon, 23 Jun 2008 14:32:23 GMT", "ETag: "2222-fc35-450564fcbabc0"", "Content-Type: text/html", "Content-Length: 64565", }, len = 64565, store = { 64565 { 3c 48 54 4d 4c 3e 0a 0a 3c 48 45 41 44 3e 0a 3c |...<| 42 41 53 45 20 48 52 45 46 3d 22 68 74 74 70 3a |BASE HREF="http:| 2f 2f 66 79 2e 63 68 61 6c 6d 65 72 73 2e 73 65 |//fy.chalmers.se| 2f 7e 61 70 70 72 6f 2f 6c 69 6e 75 78 2f 44 56 |/~appro/linux/DV| [64501 more] }, }, }, }, I've also disabled gzip support on the server which made no difference. On 09/03/2011 19:03, Dmitry Panov wrote: > Ok, I have reproduced the bug on the unpatched trunk (revision > 25c5f2ed3229e41e99eadff57374c3a93b41a356) without using custom vcl > (the only section I have there is the backend specification). > > Command to run varnish was: > > /opt/varnish/sbin/varnishd \ > -a 0.0.0.0:6802 \ > -f /opt/varnish/etc/varnish/my.vcl \ > -P /var/run/varnishd.pid \ > -T 127.0.0.1:2000 \ > -d \ > -s file,/opt/varnish/var/varnish/storage.bin,1G > > The system is running Debian with 32 bit kernel. As I mentioned > earlier I was able to reproduce the problem on another machine with > significantly different hardware configuration. The only common thing > was that they were running debian with 32bit kernel. Also I used the > same binaries on both machines. I could not reproduce the problem in > 64 bit environment. > > I'm attaching the stack trace and the log file. Please let me know if > I can provide any more info. > > On 09/03/2011 14:51, Geoff Simmons wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> On 03/ 9/11 03:17 PM, Dmitry Panov wrote: >>> Just a heads up, I'm getting assertion failures when running a rather >>> simple testcase: using local apache that serves /user/share/doc as the >>> backend and running wget -r http://localhost:6802/doc Shortly after >>> that >>> the following errors start to appear: >>> >>> Child (11125) Panic message: Assert error in http_Write(), cache_http.c >>> line 1181: >>> Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. >>> thread = (cache-worker) >> Thanks for the heads up. Can you send over the whole stack trace? >> >>> I have been able to reproduce it on 2 different machines with very >>> different hardware configurations which makes hardware problem quite >>> unlikely. Also >>> >>> httperf --server localhost --port 6802 --uri / --num-conns 1 >>> --num-calls 4000 >>> >>> runs without a problem. >>> >>> These 2 machines both run 32bit linux kernel. I haven't been able to >>> reproduce the problem in a 64bit environment. >> Could be running out of workspace. I fixed a similar error during the >> course of development, which had to do with the fact that sufficient >> workspace has to be allocated for the both backend response *and* the >> stale object; you might have found something related. Also, I've only >> been testing with 64 bit; looks like I better test 32 bit as well. >> >> Is there any way you can send the request& response that are being >> processed when the error happens? >> >> And what if you set --num-conns high and --num-calls low, say 400 >> connections and 10 calls per connection? Or keep setting --num-conns >> higher, to see if you can provoke the error? I've been running httperf >> with 25,000 connections and 1000 calls per connection, found a memory >> leak that way. >> >>> Unfortunately I haven't got time to try the unpatched trunk (I tried it >>> with revisions 3 and 4 of the patch) or do any further experiments but >>> I'll try to do so in the next couple of days and then post more >>> details. >> It's a good idea to test on the unpatched trunk as well, to make sure >> that the bug really comes from the patch. >> >> Thanks very much for the feedback! >> >> > Best regards, -- Dmitry Panov From jdzstz at gmail.com Thu Mar 10 00:25:50 2011 From: jdzstz at gmail.com (jdzstz - gmail dot com) Date: Thu, 10 Mar 2011 01:25:50 +0100 Subject: Updated varnish patches for cygwin 1.7.8 In-Reply-To: <877hca2ady.fsf@qurzaw.varnish-software.com> References: <877hca2ady.fsf@qurzaw.varnish-software.com> Message-ID: Hello, Trunk: http://www.varnish-cache.org/trac/attachment/wiki/VarnishOnCygwinWindows/varnish-githead-cygwin.patch I have made some minor changes to varnish patches for cygwin 1.7.8, I reply to your questions. 2011/3/8 Tollef Fog Heen : > In general, I'm a bit unhappy with most of this patch as it leaks too > much cygwin stuff across the build system. ?I'll comment some issues > inline. > > | diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am > | index b8db6e0..f920b42 100644 > | --- a/bin/varnishd/Makefile.am > | +++ b/bin/varnishd/Makefile.am > | + > | +EXTRA_SCRIPTS = > | + > | +sbin_SCRIPTS = ${CYGWIN_libvarnishd_OBJECTS} > | + > | + > | ?DISTCLEANFILES = default_vcl.h > > Why the extra EXTRA_SCRIPTS? ?I'd also like to not have to care about > gcc on cygwin having problems linking a DLL with unresolved symbols in it. EXTRA_SCRIPTS and sbin_SCRIPTS is a workaround in order to be able to install fake library in system when "make install" is executed. > > | ?# > | diff --git a/configure.ac b/configure.ac > | index f574764..9f7753e 100644 > | --- a/configure.ac > | +++ b/configure.ac > > This bit has been merged. I have tested this change with last trunk version and It works fine at cygwin. > > | @@ -457,6 +488,19 @@ else > | ? ? ? *-*-darwin*) > | ? ? ? ? ? ? ? VCC_CC="exec cc $OCFLAGS -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" > | ? ? ? ? ? ? ? ;; > | + ? ? *-*-cygwin*) > | + ? ? ? ? ? ? #calculate ${exec_prefix}/lib, because it is asigned later, and we need now > | + ? ? ? ? ? ? prefix_aux=`eval echo ${prefix}` > | + ? ? ? ? ? ? test "x$prefix_aux" = xNONE && prefix_aux=`eval echo ${ac_default_prefix}` > | + ? ? ? ? ? ? # Let make expand exec_prefix. > | + ? ? ? ? ? ? exec_prefix_aux=`eval echo ${exec_prefix}` > | + ? ? ? ? ? ? test "x$exec_prefix_aux" = xNONE && exec_prefix_aux=`eval echo ${prefix_aux}` > | + ? ? ? ? ? ? # Remove "/" if we are in Root > | + ? ? ? ? ? ? test "x$exec_prefix_aux" = "x/" && exec_prefix_aux="" > | + ? ? ? ? ? ? #calculate compilation path > | + ? ? ? ? ? ? comp_prefix=`pwd` > | + ? ? ? ? ? ? VCC_CC="$PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS -shared %s -L${comp_prefix}/lib/libvarnish/.libs -L${comp_prefix}/bin/varnishd -L${exec_prefix_aux}/lib -L${exec_prefix_aux}/sbin -L${exec_prefix_aux}/bin -lvarnish -lvarnishd -o %o" > | + ? ? ? ? ? ? ;; > > Adding those -L bits here looks a bit odd. ?Doesn't libtool just add the > necessary rpath if that's needed? Those lines defines VCC_CC value that is used by varnish only at execution time in order to compile VCL file at startime, and in order to search DLL libvarnishd falke DLL so libtools cannot be used. Also a comment appears at configuration.ac file that says that libtool is not used: # Command line for compiling VCL code. I wish there were a simple way # to figure this out dynamically without introducing a run-time # dependency on libtool. > > | +# Add library dependencies with libvarnish (linker needs it in CYGWIN during compilation time) > | +# Generate fake library libvarnishd.dll.a in order to successful linking DLL when resolving varnishd.exe simbols > | +case $target in > | +*-*-cygwin*) > | + ? ? ? ? ? ? CYGWIN_varnishd_LDFLAGS=' -Wl,-out-implib,libvarnishd.dll.a' > | + ? ? ? ? ? ? CYGWIN_libvarnishd_OBJECTS='libvarnishd.dll.a' > | + ? ? ? ? ? ? CYGWIN_libvmod_std_la_LIBADD='$(top_builddir)/lib/libvarnish/libvarnish.la -lvarnishd -L$(top_builddir)/bin/varnishd' > | + ? ? ? ? ? ? AC_SUBST(CYGWIN_varnishd_LDFLAGS) > | + ? ? ? ? ? ? AC_SUBST(CYGWIN_libvarnishd_OBJECTS) > | + ? ? ? ? ? ? AC_SUBST(CYGWIN_libvmod_std_la_LIBADD) > | + ? ? ? ? ? ? cygwin=true > | + ? ? ? ? ? ? ;; > | +*) > | + ? ? ? ? ? ? cygwin=false > | + ? ? ? ? ? ? ;; > | +esac > | + > | +AM_CONDITIONAL([CYGWIN_COMPILATION], test x$cygwin = xtrue) > > Can we get rid of that fake library? ?Also, doesn't cygwin support > inter-library dependencies properly? Cygwin compilation generate objects in PE/COFF format (win32 executable files) , the problem is that with COFF format, all DLL dependencies must be defined at compilation time. If some DLL (shared ojects) uses objects from binary files (EXE) then, at cygwin you have to specify a fake DLL that is generated from original EXE file, in order to be able to link against something: (from http://www.cygwin.com/ml/cygwin/2006-12/msg00592.html) The particular question of having a DLL that imports a function exported from the main .exe comes up from time to time. It is possible, but not necessarily pretty. If it is only a one-way dependency then it's rather simple to mark the symbol as dllexport in the .exe, create an import library for the .exe (yes the .exe) during linking, and then link the .dll against that import library. Linkage problems ("Undefined reference" errors). The undefined reference is because fundamentally ELF (UNIX) and PE/COFF (Windows) are very different in terms of how linking works under the hood. The short explanation is that PE/COFF requires all references to be resolved at link-time. See: http://www.mail-archive.com/cygwin at cygwin.com/msg81837.html http://www.cygwin.com/ml/cygwin/2007-06/msg00450.html http://www.cygwin.com/ml/cygwin/2005-07/msg00675.html http://www.cygwin.com/ml/cygwin/2006-12/msg00592.html > > | --- a/lib/libvarnishapi/Makefile.am > | +++ b/lib/libvarnishapi/Makefile.am > | @@ -26,4 +26,6 @@ libvarnishapi_la_SOURCES = \ > | ?libvarnishapi_la_CFLAGS = \ > | ? ? ? -DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' > | > | -libvarnishapi_la_LIBADD = @PCRE_LIBS@ > | +libvarnishapi_la_LIBADD = \ > | + ? ? $(top_builddir)/lib/libvarnish/libvarnish.la \ > | + ? ? @PCRE_LIBS@ > > Why is this? As said before, Cygwin compilation generate objects in PE/COFF format (win32 DLL files) , the problem is that with COFF format, all DLL dependencies must be defined at compilation time. http://cygwin.com/ml/cygwin/2007-07/msg00774.html Linkage problems ("Undefined reference" errors). The undefined reference is because fundamentally ELF (UNIX) and PE/COFF (Windows) are very different in terms of how linking works under the hood. The short explanation is that PE/COFF requires all references to be resolved at link-time. So only in cygwin there is a dependency of libvarnishapi with libvarnish, at compilation type. On Linux the dependency is only at execution time, it is not compulsory to add the dependency, but if it is added, there is no problem with it, so I have added it for all platforms. If remove this line DLL import, following error appears: /bin/sh ../../libtool --tag=CC --mode=link gcc -std=gnu99 -I/usr/include/ncurses -avoid-version -no-undefined -o libvcl.la -rpath /usr/local/lib/varnish vcc_acl.lo vcc_action.lo vcc_backend.lo vcc_backend_util.lo vcc_compile.lo vcc_dir_random.lo vcc_dir_round_robin.lo vcc_dir_dns.lo vcc_expr.lo vcc_parse.lo vcc_fixed_token.lo vcc_obj.lo vcc_storage.lo vcc_string.lo vcc_symb.lo vcc_token.lo vcc_var.lo vcc_vmod.lo vcc_xref.lo libtool: link: gcc -std=gnu99 -shared .libs/vcc_acl.o .libs/vcc_action.o .libs/vcc_backend.o .libs/vcc_backend_util.o .libs/vcc_compile.o .libs/vcc_dir_random.o .libs/vcc_dir_round_robin.o .libs/vcc_dir_dns.o .libs/vcc_expr.o .libs/vcc_parse.o .libs/vcc_fixed_token.o .libs/vcc_obj.o .libs/vcc_storage.o .libs/vcc_string.o .libs/vcc_symb.o .libs/vcc_token.o .libs/vcc_var.o .libs/vcc_vmod.o .libs/vcc_xref.o -o .libs/cygvcl.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libvcl.dll.a .libs/vcc_acl.o:vcc_acl.c:(.text+0x14d): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x19c): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x209): undefined reference to `_vas_fail' .libs/vcc_acl.o:vcc_acl.c:(.text+0x25f): undefined reference to `_vas_fail' .libs/vcc_acl.o:vcc_acl.c:(.text+0x2f7): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x325): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x461): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x5a1): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x6e6): undefined reference to `_vsb_printf' .libs/vcc_acl.o:vcc_acl.c:(.text+0x747): more undefined references to `_vsb_printf' follow .libs/vcc_acl.o:vcc_acl.c:(.text+0x8d3): undefined reference to `_vas_fail' .libs/vcc_acl.o:vcc_acl.c:(.text+0xd02): undefined reference to `_vas_fail' .libs/vcc_acl.o:vcc_acl.c:(.text+0x1167): undefined reference to `_vas_fail' .libs/vcc_acl.o:vcc_acl.c:(.text+0x12d2): undefined reference to `_vas_fail' .libs/vcc_action.o:vcc_action.c:(.text+0x31a): undefined reference to `_vas_fail' .libs/vcc_action.o:vcc_action.c:(.text+0x51c): more undefined references to `_vas_fail' follow .libs/vcc_action.o:vcc_action.c:(.text+0x549): undefined reference to `_vsb_printf' .libs/vcc_action.o:vcc_action.c:(.text+0x6bc): undefined reference to `_vsb_printf' .libs/vcc_action.o:vcc_action.c:(.text+0x71f): undefined reference to `_vsb_printf' .libs/vcc_action.o:vcc_action.c:(.text+0x809): undefined reference to `_vsb_printf' .libs/vcc_action.o:vcc_action.c:(.text+0xb0c): undefined reference to `_vsb_printf' .libs/vcc_action.o:vcc_action.c:(.text+0x1053): more undefined references to `_vsb_printf' follow .libs/vcc_action.o:vcc_action.c:(.text+0x11ce): undefined reference to `_vas_fail' .libs/vcc_backend.o:vcc_backend.c:(.text+0x3f): undefined reference to `_vas_fail' > > | diff --git a/lib/libvcl/Makefile.am b/lib/libvcl/Makefile.am > | index aab8749..4645bba 100644 > | --- a/lib/libvcl/Makefile.am > | +++ b/lib/libvcl/Makefile.am > | @@ -33,6 +33,8 @@ libvcl_la_SOURCES = \ > | ? ? ? vcc_vmod.c \ > | ? ? ? vcc_xref.c > | > | +libvcl_la_LIBADD = $(top_builddir)/lib/libvarnish/libvarnish.la > | + > | ?EXTRA_DIST = \ > | ? ? ? generate.py > > What does this solve for you? REMOVED: This line was similar to libvarnishapi change, but I have removed the patch, for libvcl it is not necesary. > > | diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am > | index 36147fe..41c32c8 100644 > | --- a/lib/libvmod_std/Makefile.am > | +++ b/lib/libvmod_std/Makefile.am > | @@ -13,9 +13,16 @@ libvmod_std_la_SOURCES = \ > | ? ? ? vmod_std.c \ > | ? ? ? vmod_std_fileread.c > | > | +libvmod_std_la_LIBADD = ${CYGWIN_libvmod_std_la_LIBADD} > | + > | ?vcc_if.c vcc_if.h: $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc > | ? ? ? @PYTHON@ $(top_srcdir)/lib/libvmod_std/vmod.py $(top_srcdir)/lib/libvmod_std/vmod.vcc > | > | ?EXTRA_DIST = vmod.py vmod.vcc > | > | ?CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h > | + > | +if CYGWIN_COMPILATION > | +install-exec-hook: > | + ? ? $(LN_S) -f cygvmod_std-1.dll .libs/libvmod_std.so.1 > | +endif > > What? ?You shouldn't symlink stuff into .libs directories. On cygwin system, all DLL name starts with "cyg*" instead "lib*" prefix, so all libraries names are similar to "cygvmod_std.dll" . The problem is that varnishtests vtc tests: ./tests/m00000.vtc ./tests/m00001.vtc ./tests/m00002.vtc, etc..., searchs a file named libvmod_std.so, so this symlink generate a link from created cygvmod_std.dll (cygwin file) to libvmod_std.so expected file. This change is only used in order to pass all varnishtest tests. > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-githead-cygwin.patch Type: application/octet-stream Size: 4258 bytes Desc: not available URL: From tfheen at varnish-software.com Thu Mar 10 07:09:36 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 10 Mar 2011 08:09:36 +0100 Subject: Workspace overflow on ia32 In-Reply-To: <4D77E09A.6090902@yahoo.co.uk> (Dmitry Panov's message of "Wed, 09 Mar 2011 20:18:34 +0000") References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D778C0E.60303@yahoo.co.uk> <4D7793F5.1080705@uplex.de> <4D77CEFC.3040906@yahoo.co.uk> <4D77E09A.6090902@yahoo.co.uk> Message-ID: <87wrk7v4cv.fsf@qurzaw.varnish-software.com> ]] Dmitry Panov | Oops, my bad, I didn't realise the unpatched trunk installed itself in | a different directory, so I was in fact running a patched | version. However after I fixed this, I could still reproduce the | problem. Here is the updated stack trace: I fixed this on Tuesday with: commit 5f8041ac0745ba30e9190a69ec7b5d8ac2d4c1ec Author: Tollef Fog Heen Date: Tue Mar 8 13:06:01 2011 +0100 Reduce http_resp_size to 8k on 32 bit http_resp_size is allocated on the thread_pool_workspace. This means we will hit assert errors if http_resp_size is larger than the thread_pool_workspace. 8k should be enough in most cases, from a light sample of web sites in the wild. Please try that and verify it fixes the problem for you. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Thu Mar 10 12:37:55 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 10 Mar 2011 13:37:55 +0100 Subject: lastpanic CLI command Message-ID: <1299760676-6647-1-git-send-email-tfheen@varnish-software.com> The attached patch implements a lastpanic CLI command that dumps the contents of the panic string for the last panic. Review is most welcome. From tfheen at varnish-software.com Thu Mar 10 12:37:56 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 10 Mar 2011 13:37:56 +0100 Subject: [PATCH] Add lastpanic CLI command, dumping the last panic string In-Reply-To: <1299760676-6647-1-git-send-email-tfheen@varnish-software.com> References: <1299760676-6647-1-git-send-email-tfheen@varnish-software.com> Message-ID: <1299760676-6647-2-git-send-email-tfheen@varnish-software.com> --- bin/varnishd/cache_cli.c | 1 + bin/varnishd/mgt_child.c | 31 +++++++++++++++++++++++++++++++ bin/varnishd/mgt_cli.c | 1 + bin/varnishd/mgt_cli.h | 1 + include/cli.h | 6 ++++++ 5 files changed, 40 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c index dae6af5..056a63c 100644 --- a/bin/varnishd/cache_cli.c +++ b/bin/varnishd/cache_cli.c @@ -1,6 +1,7 @@ /*- * Copyright (c) 2006 Verdens Gang AS * Copyright (c) 2006-2010 Redpill Linpro AS + * Copyright (c) 2011 Varnish Software AS * All rights reserved. * * Author: Poul-Henning Kamp diff --git a/bin/varnishd/mgt_child.c b/bin/varnishd/mgt_child.c index d8cbc59..02d2ff5 100644 --- a/bin/varnishd/mgt_child.c +++ b/bin/varnishd/mgt_child.c @@ -91,6 +91,8 @@ static struct vev *ev_poker; static struct vev *ev_listen; static struct vlu *vlu; +static struct vsb *child_panic = NULL; + /*-------------------------------------------------------------------- * Track the highest file descriptor the parent knows is being used. * @@ -433,6 +435,24 @@ mgt_report_panic(pid_t r) (intmax_t)r, vsm_head->panicstr); } +static void +mgt_save_panic(void) +{ + char time_str[30]; + if (vsm_head->panicstr[0] == '\0') + return; + + if (child_panic) + vsb_delete(child_panic); + child_panic = vsb_newauto(); + XXXAN(child_panic); + TIM_format(TIM_real(), time_str); + vsb_printf(child_panic, "Last panic at: %s\n", time_str); + vsb_printf(child_panic, "%s", vsm_head->panicstr); + vsb_finish(child_panic); + AZ(vsb_overflowed(child_panic)); +} + /*--------------------------------------------------------------------*/ static int @@ -478,6 +498,7 @@ mgt_sigchld(const struct vev *e, int what) vsb_delete(vsb); mgt_report_panic(r); + mgt_save_panic(); child_pid = -1; @@ -614,3 +635,13 @@ mcf_server_status(struct cli *cli, const char * const *av, void *priv) (void)priv; cli_out(cli, "Child in state %s", ch_state[child_state]); } + +void +mcf_lastpanic(struct cli *cli, const char * const *av, void *priv) +{ + (void)av; + (void)priv; + + if (child_panic) + cli_out(cli, "%s\n", vsb_data(child_panic)); +} diff --git a/bin/varnishd/mgt_cli.c b/bin/varnishd/mgt_cli.c index 7f719a0..48b7295 100644 --- a/bin/varnishd/mgt_cli.c +++ b/bin/varnishd/mgt_cli.c @@ -127,6 +127,7 @@ static struct cli_proto cli_proto[] = { { CLI_VCL_SHOW, "", mcf_config_show, NULL }, { CLI_PARAM_SHOW, "", mcf_param_show, NULL }, { CLI_PARAM_SET, "", mcf_param_set, NULL }, + { CLI_LASTPANIC, "", mcf_lastpanic, NULL }, { NULL } }; diff --git a/bin/varnishd/mgt_cli.h b/bin/varnishd/mgt_cli.h index c4bfd5a..e3f61cc 100644 --- a/bin/varnishd/mgt_cli.h +++ b/bin/varnishd/mgt_cli.h @@ -32,6 +32,7 @@ /* mgt_child.c */ cli_func_t mcf_server_startstop; cli_func_t mcf_server_status; +cli_func_t mcf_lastpanic; /* mgt_param.c */ cli_func_t mcf_param_show; diff --git a/include/cli.h b/include/cli.h index e16ff60..5dc5e80 100644 --- a/include/cli.h +++ b/include/cli.h @@ -230,6 +230,12 @@ "\tAuthenticate.", \ 1, 1 +#define CLI_LASTPANIC \ + "lastpanic", \ + "lastpanic", \ + "\tReturn the last panic, if any.", \ + 0, 0 + #define CLI_HIDDEN(foo, min_arg, max_arg) \ foo, NULL, NULL, min_arg, max_arg, -- 1.7.2.3 From dmitry.panov at yahoo.co.uk Thu Mar 10 17:43:23 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Thu, 10 Mar 2011 17:43:23 +0000 Subject: Workspace overflow on ia32 In-Reply-To: <87wrk7v4cv.fsf@qurzaw.varnish-software.com> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D778C0E.60303@yahoo.co.uk> <4D7793F5.1080705@uplex.de> <4D77CEFC.3040906@yahoo.co.uk> <4D77E09A.6090902@yahoo.co.uk> <87wrk7v4cv.fsf@qurzaw.varnish-software.com> Message-ID: <4D790DBB.90107@yahoo.co.uk> Hi Tollef, Yes, it has fixed the problem. Many thanks! On 10/03/2011 07:09, Tollef Fog Heen wrote: > ]] Dmitry Panov > > | Oops, my bad, I didn't realise the unpatched trunk installed itself in > | a different directory, so I was in fact running a patched > | version. However after I fixed this, I could still reproduce the > | problem. Here is the updated stack trace: > > I fixed this on Tuesday with: > > commit 5f8041ac0745ba30e9190a69ec7b5d8ac2d4c1ec > Author: Tollef Fog Heen > Date: Tue Mar 8 13:06:01 2011 +0100 > > Reduce http_resp_size to 8k on 32 bit > > http_resp_size is allocated on the thread_pool_workspace. This means > we will hit assert errors if http_resp_size is larger than the > thread_pool_workspace. 8k should be enough in most cases, from a > light sample of web sites in the wild. > > Please try that and verify it fixes the problem for you. > > Regards, Best regards, -- Dmitry Panov From mnot at yahoo-inc.com Thu Mar 10 17:55:47 2011 From: mnot at yahoo-inc.com (Mark Nottingham) Date: Thu, 10 Mar 2011 09:55:47 -0800 Subject: FYI: HTTP timeout headers Message-ID: <267A90CA-B714-4C2C-A019-E5E9685C5755@yahoo-inc.com> In a nutshell, this draft introduces two new headers: Request-Timeout, which is an end-to-end declaration of how quickly the client wants the response, and Connection-Timeout, which is a hop-by-hop declaration of how long an idle conn can stay open. I'm going to give feedback to Martin about this (I can see a few places where there may be issues, e.g., it doesn't differentiate between a read timeout on an open request and an idle timeout), but I wanted to get a sense of what Varnish developers thought; in particular - 1) is this interesting enough that you'd implement if it came out? 2) if someone submitted a patch for this, would you include it? 3) do you see any critical issues, esp. regarding performance impact? Thanks, -- Mark Nottingham mnot at yahoo-inc.com From dmitry.panov at yahoo.co.uk Thu Mar 10 18:18:35 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Thu, 10 Mar 2011 18:18:35 +0000 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D775D22.2070202@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> Message-ID: <4D7915FB.3060407@yahoo.co.uk> Hi Geoff, I have finally been able to test the patched version under some proper load. I was using tsung (http://tsung.erlang-projects.org/). I recorded a browser session in which I accessed a couple of pages, then tried to replay it with tsung. It looks like there is a memory leak in the patched version because the virtual memory size grew at a constant pace until eventually varnishd filled up the whole available space. I tried exactly the same scenario with an unpatched trunk (same revision that I applied the patch to) and it worked fine. Note there were no stale objects or conditional requests during the test because I set ttl to a high value so it's either a hit or a miss for an uncacheable object. I have a rather complex setup which includes apache with mod_proxy_html which I use to reverse proxy a real Internet web site (www.telegraph.co.uk). If there is anything I can do to help you diagnose the problem, please let me know. On 09/03/2011 10:57, Geoff Simmons wrote: > Hello again, > > And yet another version (v5) of the patch for conditional backend > requests. Like the last version, this is one is to keep pace with recent > commits to the trunk. The patch applies to: > > commit 13c60b522a6c7abe9cc1052c2a244f25f2ba5227 > Date: Tue Mar 8 14:47:57 2011 +0100 > > phk committed some changes related to this feature, mainly the addition > of the new timer variable to VCL. There are also some changes in the way > that previous version of the patch worked. > > - The name of the timer for conditional requests changed again, now to > "keep". The management parameter is "default_keep". > > - In addition to beresp.keep, obj.keep and stale_obj.keep, there is now > req.keep, which sets an upper bound for the keep time within a session. > (Like req.ttl, which phk added yesterday.) > > - The keep interval now begins after ttl and grace have elapsed, so an > object is available for conditional requests after obj.ttl + obj.grace, > and is scheduled for eviction after obj.ttl + obj.grace + obj.keep. (In > previous versions of the patch, the keep interval began after ttl > elapses, independent of grace, and object were evicted after obj.ttl + > max(obj.grace, obj.keep).) > > - The default value default_keep is now 0, so you need to set a non-zero > default_keep, or set keep in VCL, to make the conditional requests happen. > > The Wiki page has been updated to reflect all of this: > > http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > > @phk: I made one change in the commits from yesterday. Since > default_keep = 0 and EXP_get_*() was returning -1 for any timer = 0, > *.keep was always -1 unless it was changed in VCL. I think VCL authors > wouldn't expect that; I got feedback about an earlier version in which > beresp.keep was -1 after a backend validation, which both the reporter > and I thought was wrong, so I fixed it in the next version. > > Now I have this in cache_expire.c: > > double \ > EXP_Get_##fld(const struct exp *e) \ > { \ > - return (e->fld> 0. ? e->fld : -1.); \ > + return (e->fld> 0. ? e->fld : 0.); \ > } \ > > This required a change in test case b00040.vtc, which expects req.grace > == -1, but that doesn't seem to be a critical value there. > > > Best, > Geoff > Best regards, -- Dmitry Panov From geoff at uplex.de Thu Mar 10 19:41:34 2011 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 10 Mar 2011 20:41:34 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D7915FB.3060407@yahoo.co.uk> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> Message-ID: <4D79296E.8010502@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/10/11 7:18 PM, Dmitry Panov wrote: > > I have finally been able to test the patched version under some proper > load. I was using tsung (http://tsung.erlang-projects.org/). I recorded > a browser session in which I accessed a couple of pages, then tried to > replay it with tsung. > > It looks like there is a memory leak in the patched version because the > virtual memory size grew at a constant pace until eventually varnishd > filled up the whole available space. I tried exactly the same scenario > with an unpatched trunk (same revision that I applied the patch to) and > it worked fine. Note there were no stale objects or conditional requests > during the test because I set ttl to a high value so it's either a hit > or a miss for an uncacheable object. Thanks as always, Dmitry. That last part is interesting, if there are no stale objects or conditional requests, then Varnish should behave exactly as if the patch weren't there. I assume you tested the most recent version of the patch (v5, attached to the mail that you quoted)? I had fixed a memory leak in an earlier version; but that leak only could have happened if there had been stale objects. Did you notice whether there were more cache misses with the patched version? And did you by any chance run varnishstat -1 after load testing the two versions? If so, I'd be interested if there were any noticeable difference in stats about memory usage, such as n_object, s_hdrbytes, s_bodybytes or the stevedore stats. BTW, with default settings (default_keep = 0), the patched version won't do any conditional requests, so that should be enough to get it to behave the same as the unpatched version. > I have a rather complex setup which includes apache with mod_proxy_html > which I use to reverse proxy a real Internet web site > (www.telegraph.co.uk). If there is anything I can do to help you > diagnose the problem, please let me know. I'll see if I can get the same effect with my own load tests, if not I'll get in touch. Thanks again, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNeSltAAoJEOUwvh9pJNUREn4P/0HLdY1rZxAhbP6ltoxcuoLB 3++3xbNJhZ58cMGOkRdDt6ZPT7rfp/JXmTSST40i8tPpzA2M2aSaz3fhsczPYpkH c8AXxVahLVIYIffgRO6jlb+5omcx5DbGCLUwvLeSmGaJa7mFyfeJ0HH2tsCCo1+K mMWvU+txb/KCt2H61m/pNL/tgaPs6moalVPPwXVapbLb7AUmq9Wm8J626R2lKjg8 wAxIF/fO3B04IZM9ezMtEwu3XsIzi7kZpSyoyeDsSr4IZO0P9ZmZtWESBOV0K5KQ 3S/5L9aABWJntfW2ruWb3TWCRa6HDYvsI/VsIW9Y8bTIqMvMbM3yvE2Ga7wl0wAY 3BervDi2erR5iMskQqsyQ+W5daqfB3VE4hBsYhcUZ6WM+6BziIMYgcZ+zxS2l0D5 B+YM+iFKud1OvmNam+1ezbld+a2v+9P8TeD078fWLJ2CXtXpQn4CA2SaFUVXDkAR jOjzd4Q4yg2R7dUYGuLIZ5SE4QQ4WnBfgwWbIvYkJi9w2Wh6cdcDoBNxZzVoIUIq nP4Sl8Jo0v3vTrvCr15Ej+Z4p8H3uBu6B4snfZMp1LopzBz5G6ogCFLRXbeGFoil hCWqdTKwAVU2Qek2rOBMmWdW+QWgZscoklNNOwhRGsJ5omYheZawaykQf4Ycw6IE GMFyFcqUB83VifmXxJba =IqWf -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Fri Mar 11 01:30:47 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Fri, 11 Mar 2011 01:30:47 +0000 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D79296E.8010502@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> <4D79296E.8010502@uplex.de> Message-ID: <4D797B47.8000703@yahoo.co.uk> Hi Geoff, You're right, the patch didn't make any difference. The problem was that I was running an earlier version of the unpatched trunk which contained the ws overflow bug, so the reason memory usage was dropping from time to time was because one of the threads was dying. I've installed the latest trunk and it behaved exactly like the patched version: after about 80k requests the virtual memory size was about 2G (I have 1G storage configured). I ran varnishstat -1 after that (attached). So it looks like there is a memory leak in the current trunk. On 10/03/2011 19:41, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 3/10/11 7:18 PM, Dmitry Panov wrote: >> I have finally been able to test the patched version under some proper >> load. I was using tsung (http://tsung.erlang-projects.org/). I recorded >> a browser session in which I accessed a couple of pages, then tried to >> replay it with tsung. >> >> It looks like there is a memory leak in the patched version because the >> virtual memory size grew at a constant pace until eventually varnishd >> filled up the whole available space. I tried exactly the same scenario >> with an unpatched trunk (same revision that I applied the patch to) and >> it worked fine. Note there were no stale objects or conditional requests >> during the test because I set ttl to a high value so it's either a hit >> or a miss for an uncacheable object. > Thanks as always, Dmitry. That last part is interesting, if there are no > stale objects or conditional requests, then Varnish should behave > exactly as if the patch weren't there. > > I assume you tested the most recent version of the patch (v5, attached > to the mail that you quoted)? I had fixed a memory leak in an earlier > version; but that leak only could have happened if there had been stale > objects. > > Did you notice whether there were more cache misses with the patched > version? And did you by any chance run varnishstat -1 after load testing > the two versions? If so, I'd be interested if there were any noticeable > difference in stats about memory usage, such as n_object, s_hdrbytes, > s_bodybytes or the stevedore stats. > > BTW, with default settings (default_keep = 0), the patched version won't > do any conditional requests, so that should be enough to get it to > behave the same as the unpatched version. > >> I have a rather complex setup which includes apache with mod_proxy_html >> which I use to reverse proxy a real Internet web site >> (www.telegraph.co.uk). If there is anything I can do to help you >> diagnose the problem, please let me know. > I'll see if I can get the same effect with my own load tests, if not > I'll get in touch. > > > Thanks again, > Geoff -- Dmitry Panov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: stat.txt URL: From geoff at uplex.de Fri Mar 11 07:30:59 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 11 Mar 2011 08:30:59 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D797B47.8000703@yahoo.co.uk> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> <4D79296E.8010502@uplex.de> <4D797B47.8000703@yahoo.co.uk> Message-ID: <4D79CFB3.1070402@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/11/11 2:30 AM, Dmitry Panov wrote: > > You're right, the patch didn't make any difference. The problem was that > I was running an earlier version of the unpatched trunk which contained > the ws overflow bug, so the reason memory usage was dropping from time > to time was because one of the threads was dying. > > I've installed the latest trunk and it behaved exactly like the patched > version: after about 80k requests the virtual memory size was about 2G > (I have 1G storage configured). I ran varnishstat -1 after that (attached). > > So it looks like there is a memory leak in the current trunk. I tried a load test, but couldn't reproduce a memory leak. With the latest unpatched trunk using -s malloc,1G, and running httperf and an Apache backend all on my machine, I ran a load of about 12,500 reqs/s for over a half hour. varnishd's virtual memory size expanded to 911 MB after 4 minutes, but no further. The stevedore stats show 415 MB outstanding bytes after the run, which is about how much data I have in the test site. The stevedore had allocated 7 GB and freed 6.6 GB during the course of the run. All of that looks right to me. Since this is about the trunk itself rather than the patch, and if you're sure you can establish that there's a leak, maybe you should file a bug report with the evidence. Thanks again for your help with testing. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNec+yAAoJEOUwvh9pJNUR40IP/A9L5IDd7OYAocgdX096DO/b 8rwSr90kK9gwhpOaFIWf8/uUJ0kz14Aqq3xDTbEzetFNC7dvNNKzymUXmvbFyVh5 n+IMhMSBRB0evtQA9ZEOMsAOISh+/+pXifFctskO1RF8HL7tntXJIASW078AsMFJ 9Hj+3KAVl+P8dMhBGk8NXh5g6KL5c0kHw7QmLHw+C26Ixgi+ZPGwE287oKR3DvjV qwGuQ6UyIQWKEKdm8YGjXGRqSv9C1G25SoAm5UQh/cc9WcXzI7Vw7BWTQEfwqwRI 4yH26U2TRtPkNyxX561T2amya1RVANwBtFurFnz4M3KvfEyS+2FCh6yp5mnv1WWP Qr9E6Vt/eR4tnssy1Aob96AZS4JUKP6UMI+OySZ40Wh87wLzxgJieHMd/uyoyUQA knRUrH2rJ27/rpTOMwYuOjbEJ6jzBdcbLpRWZeyi0Ymw2Dd6D52+CiRLu9Vm6/vI xSCKyrwqCgAcRqlTyEt/AnJAcDBjtgS8fGXvgv4G/1IePBSw/hkHnZ2uFDPmcYoX bZoKiMm3VmkU7i6pIwoppGIDwaLrNrFb54RBnw9lLOB/5Ag0kNRs0Gfgpnt/sfhi Qgj/WwX3bS/H8HB4raKcvve9kDCr9y4EI8/rpAn/adwNJ7jK7brZUa/Oc3EsURSj QcQ9/o73ZB0qYJxFMI4m =TQkA -----END PGP SIGNATURE----- From geoff at uplex.de Fri Mar 11 11:05:25 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 11 Mar 2011 12:05:25 +0100 Subject: [PATCH] backend conditional requests 5th release In-Reply-To: <4D79CFB3.1070402@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> <4D79296E.8010502@uplex.de> <4D797B47.8000703@yahoo.co.uk> <4D79CFB3.1070402@uplex.de> Message-ID: <4D7A01F5.2030601@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello all, Another heads up on this patch, I think I've found a bug where conditional requests are being sent when they shouldn't be, according to the keep settings. Will post up a fix ASAP. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNegH0AAoJEOUwvh9pJNURHBYQAJ8NWipXmjyqqUkPcH80hKyG +CyWgM6VhbrpPRkaSno7PFLH//wVOmxZixO8JBtYLT3kXYR5P9DDweMnM9kNk0EU a+LFRSHSx0Trte++Mz5LytT65dFoouD0Vh9OxGC/EOt9uTytPtlXIZ2/V6LusUUc HTx0VTnbr18Aa5CgDIdw5OTcyo4qGFLBsGpGI63CFcvtyauFowmUiEPMiP1yQ4Yq rqCEssIJo2NeZtz+t6HXjHMPEt3Vk465N9oQiGwY7OnjJOJBdc5FObWjJTgV4Dp0 7YTPFD9xCAthkK+bw7/gpAzpW9ePb1jaX5yGwWScCRVThU2ZgSQ9DZfay73hy1Ml Bl6AeNxeD+w5T5qXXpo1z79oYULx/16qI6zf27roiyMQFkcRWc2BCxFhwsW34oMI xohXG0WDqS/QCF2Gk1Yx7mCJKX0WSWlxYZcvEzbl6viuIdQ8JtHc+HyAgV79EsjD lmAb20WoGvMpe5AZLfN/arZ45+GhjUOdjHgjiZjMKSVnC178wvjZ/BMs0lgI+VEq V4IUXg5PaGB7l15fMMpbYh3ozevPLmI3mcrQQ/xTgJvV45QcqNfWvW1q+6l9yVSB PWgEFdU2CseAZGipB1ci25LWP6KbuoOo82vhb6oDt9YZp5r1cgH4gmweDNkqN7Af djpx9qxGwpUnFRPba8UJ =lnXL -----END PGP SIGNATURE----- From dmitry.panov at yahoo.co.uk Fri Mar 11 12:54:15 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Fri, 11 Mar 2011 12:54:15 +0000 Subject: Memory leak in trunk? In-Reply-To: <4D79CFB3.1070402@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> <4D79296E.8010502@uplex.de> <4D797B47.8000703@yahoo.co.uk> <4D79CFB3.1070402@uplex.de> Message-ID: <4D7A1B77.1090009@yahoo.co.uk> Hi Geoff, and all, Well, to me it looks like a memory leak, the virtual memory consumed by varnishd grew twice as much as the size of the configured storage and continued to grow until I stopped the test. It doesn't look normal, does it? Especially considering that there were only a few pages involved. I tried the same scenario with the latest stable release and it showed a completely different behavior: Virtual Memory size grew at a much slower rate and then remained steady even after 300k requests were served. There is definitely a problem with the current trunk. Does the output from varnishstat give any clue? What should I post as an evidence? On 11/03/2011 07:30, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 3/11/11 2:30 AM, Dmitry Panov wrote: >> You're right, the patch didn't make any difference. The problem was that >> I was running an earlier version of the unpatched trunk which contained >> the ws overflow bug, so the reason memory usage was dropping from time >> to time was because one of the threads was dying. >> >> I've installed the latest trunk and it behaved exactly like the patched >> version: after about 80k requests the virtual memory size was about 2G >> (I have 1G storage configured). I ran varnishstat -1 after that (attached). >> >> So it looks like there is a memory leak in the current trunk. > I tried a load test, but couldn't reproduce a memory leak. With the > latest unpatched trunk using -s malloc,1G, and running httperf and an > Apache backend all on my machine, I ran a load of about 12,500 reqs/s > for over a half hour. varnishd's virtual memory size expanded to 911 MB > after 4 minutes, but no further. The stevedore stats show 415 MB > outstanding bytes after the run, which is about how much data I have in > the test site. The stevedore had allocated 7 GB and freed 6.6 GB during > the course of the run. All of that looks right to me. > > Since this is about the trunk itself rather than the patch, and if > you're sure you can establish that there's a leak, maybe you should file > a bug report with the evidence. > > Thanks again for your help with testing. > > > Best, > Geoff Best regards, -- Dmitry Panov From phk at phk.freebsd.dk Fri Mar 11 13:03:00 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 11 Mar 2011 13:03:00 +0000 Subject: Memory leak in trunk? In-Reply-To: Your message of "Fri, 11 Mar 2011 12:54:15 GMT." <4D7A1B77.1090009@yahoo.co.uk> Message-ID: <90030.1299848580@critter.freebsd.dk> In message <4D7A1B77.1090009 at yahoo.co.uk>, Dmitry Panov writes: >There is definitely a problem with the current trunk. is this with or without geoffs ims-patch ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dmitry.panov at yahoo.co.uk Fri Mar 11 13:04:05 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Fri, 11 Mar 2011 13:04:05 +0000 Subject: Memory leak in trunk? In-Reply-To: <90030.1299848580@critter.freebsd.dk> References: <90030.1299848580@critter.freebsd.dk> Message-ID: <4D7A1DC5.7030405@yahoo.co.uk> Both. The patch doesn't make any difference. On 11/03/2011 13:03, Poul-Henning Kamp wrote: > In message<4D7A1B77.1090009 at yahoo.co.uk>, Dmitry Panov writes: > >> There is definitely a problem with the current trunk. > is this with or without geoffs ims-patch ? > Best regards, -- Dmitry Panov From phk at phk.freebsd.dk Fri Mar 11 14:06:05 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 11 Mar 2011 14:06:05 +0000 Subject: Memory leak in trunk? In-Reply-To: Your message of "Fri, 11 Mar 2011 13:04:05 GMT." <4D7A1DC5.7030405@yahoo.co.uk> Message-ID: <90355.1299852365@critter.freebsd.dk> In message <4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: >Both. The patch doesn't make any difference. Ok, that makes it my department :-) Any indication what the leak depends on ? number of requests ? size of requests ? hit rate ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Fri Mar 11 16:41:58 2011 From: slink at schokola.de (Nils Goroll) Date: Fri, 11 Mar 2011 17:41:58 +0100 Subject: [PATCH] Solaris: Test for SO_{RCV,SND}TIMEO needs NET_LIBS Message-ID: <4D7A50D6.4040601@schokola.de> I've revisited the issue of missing socket timeouts on Solaris and noticed that they actually exist in OpenSolaris since integration of the Volo framework in fall 2008 - but the autoconf tests failed due to missing libs like in this case: configure:28453: checking whether SO_RCVTIMEO works configure:28496: gcc -o conftest -m64 -D_REENTRANT -pthreads conftest.c >&5 Undefined first referenced symbol in file socket /var/tmp//ccXeai7q.o setsockopt /var/tmp//ccXeai7q.o Nils -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-Solaris-Test-for-SO_-RCV-SND-TIMEO-needs-NET_LIBS.patch URL: From slink at schokola.de Fri Mar 11 19:38:55 2011 From: slink at schokola.de (Nils Goroll) Date: Fri, 11 Mar 2011 20:38:55 +0100 Subject: [PATCH] Normalizing the Host: header In-Reply-To: <4D5E44FF.5060302@uplex.de> References: <20000.1297968458@critter.freebsd.dk> <4D5E44FF.5060302@uplex.de> Message-ID: <4D7A7A4F.3030109@schokola.de> Hi phk, can we come to a conclusion of this discussion? My two cent: - where the host header does not undergo site-specific normalization, case folding it will once - avoid multiplication of cache content for case variations of the host header - avoid case-insensitive matching (so matches using the existing (case sensitive) ~ operator would be correct) - lead to many existing, currently wrong, VCL examples become correct - case-folding once in-place does not use any additional session space as case-folding in VCL would - it's cheaper than any alternative in a vmod/VCL can be - host header normalization is a recommendation in rfc3986 Nils From phk at phk.freebsd.dk Fri Mar 11 22:39:24 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 11 Mar 2011 22:39:24 +0000 Subject: [PATCH] Normalizing the Host: header In-Reply-To: Your message of "Fri, 11 Mar 2011 20:38:55 +0100." <4D7A7A4F.3030109@schokola.de> Message-ID: <57746.1299883164@critter.freebsd.dk> In message <4D7A7A4F.3030109 at schokola.de>, Nils Goroll writes: >can we come to a conclusion of this discussion? What bothers me is the "magic" aspect of this kind of stuff. Where does the magic intuitively begin and end ? Should we also case fold anything compared to or matched to the host header ? What about host part of Location headers ? Where does the magic end ? I very much prefer to make these things explict and consistent, so that people see them happen and know where they happen. ... but I also don't want to clutter up default.vcl with "mandatory stuff". >- case-folding once in-place does not use any additional session space > as case-folding in VCL would In my mind, this is probably the best argument for doing it, I am just not sure it convinces me. Next issue which comes right behind is: Should we also normalize URL's by reducing pointless %xx and other escapes ? Are there any other kinds of request normalization we should do ? And should we also normalize backend responses ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From aotto at mosso.com Fri Mar 11 23:05:45 2011 From: aotto at mosso.com (Adrian Otto) Date: Fri, 11 Mar 2011 15:05:45 -0800 Subject: [PATCH] Normalizing the Host: header In-Reply-To: <57746.1299883164@critter.freebsd.dk> References: <57746.1299883164@critter.freebsd.dk> Message-ID: <10ABE5A2-05CF-47CF-8FA7-FC5A0ECB6C91@mosso.com> Some advice in-line: On Mar 11, 2011, at 2:39 PM, Poul-Henning Kamp wrote: > In message <4D7A7A4F.3030109 at schokola.de>, Nils Goroll writes: > >> can we come to a conclusion of this discussion? > > What bothers me is the "magic" aspect of this kind of stuff. > > Where does the magic intuitively begin and end ? > > Should we also case fold anything compared to or matched to the > host header ? > > What about host part of Location headers ? > > Where does the magic end ? In this case, I offer the advice that all host related headers should be case folded, because DNS naming is case insensitive. So essentially anywhere Varnish handles a hostname for any comparison, it should follow the same rules. > I very much prefer to make these things explict and consistent, > so that people see them happen and know where they happen. > > ... but I also don't want to clutter up default.vcl with "mandatory stuff". That's why it's a good idea to build this in as default behavior. If it costs any meaningful performance penalty to do it, then consider using a run-time configuration variable. >> - case-folding once in-place does not use any additional session space >> as case-folding in VCL would > > In my mind, this is probably the best argument for doing it, > I am just not sure it convinces me. > > Next issue which comes right behind is: Should we also > normalize URL's by reducing pointless %xx and other escapes ? Yes, escapes should also be reduced to the lowest common denominator. > Are there any other kinds of request normalization we should do ? > > And should we also normalize backend responses ? Yes, in the same way, as those may be used for comparisons also. Adrian From dmitry.panov at yahoo.co.uk Fri Mar 11 23:45:36 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Fri, 11 Mar 2011 23:45:36 +0000 Subject: Memory leak in trunk? In-Reply-To: <90355.1299852365@critter.freebsd.dk> References: <90355.1299852365@critter.freebsd.dk> Message-ID: <4D7AB420.3030709@yahoo.co.uk> Hi Poul, I think I found the reason. I tried the same test but with http_gzip_support=off and it worked fine. tsung does not support gzip so varnish had to gunzip objects for it and this is where the leak appears to be. Hope it helps. On 11/03/2011 14:06, Poul-Henning Kamp wrote: > In message<4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: > >> Both. The patch doesn't make any difference. > Ok, that makes it my department :-) > > Any indication what the leak depends on ? > > number of requests ? > > size of requests ? > > hit rate ? > -- Dmitry Panov From perbu at varnish-software.com Sat Mar 12 08:00:43 2011 From: perbu at varnish-software.com (Per Buer) Date: Sat, 12 Mar 2011 09:00:43 +0100 Subject: Memory leak in trunk? In-Reply-To: <4D7AB420.3030709@yahoo.co.uk> References: <90355.1299852365@critter.freebsd.dk> <4D7AB420.3030709@yahoo.co.uk> Message-ID: Hi. I've been running trunk on varnish-cache.org since yesterday (14:00) and there a leak there. Turning off http_gzip_support seems to eliminate the leak. Reproducing the leak is quite simple. Just hammer varnish with a client that doesn't send accept-encoding: gzip. Per. On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov wrote: > Hi Poul, > > I think I found the reason. I tried the same test but with > http_gzip_support=off and it worked fine. tsung does not support gzip so > varnish had to gunzip objects for it and this is where the leak appears to > be. Hope it helps. > > > > On 11/03/2011 14:06, Poul-Henning Kamp wrote: > >> In message<4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: >> >> Both. The patch doesn't make any difference. >>> >> Ok, that makes it my department :-) >> >> Any indication what the leak depends on ? >> >> number of requests ? >> >> size of requests ? >> >> hit rate ? >> >> > > -- > Dmitry Panov > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Want to learn more about Varnish? http://www.varnish-software.com/whitepapers -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Sat Mar 12 14:47:51 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 12 Mar 2011 14:47:51 +0000 Subject: Memory leak in trunk? In-Reply-To: Your message of "Sat, 12 Mar 2011 09:00:43 +0100." Message-ID: <1425.1299941271@critter.freebsd.dk> In message , Per Buer writes: I think I've fixed this, a stupid oversight on my part. Please test again, when possible. Poul-Henning >--0015174c3d4addc4d1049e447a1e >Content-Type: text/plain; charset=ISO-8859-1 > >Hi. > >I've been running trunk on varnish-cache.org since yesterday (14:00) and >there a leak there. Turning off http_gzip_support seems to eliminate the >leak. > >Reproducing the leak is quite simple. Just hammer varnish with a client that >doesn't send accept-encoding: gzip. > > >Per. > >On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov wrote: > >> Hi Poul, >> >> I think I found the reason. I tried the same test but with >> http_gzip_support=off and it worked fine. tsung does not support gzip so >> varnish had to gunzip objects for it and this is where the leak appears to >> be. Hope it helps. >> >> >> >> On 11/03/2011 14:06, Poul-Henning Kamp wrote: >> >>> In message<4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: >>> >>> Both. The patch doesn't make any difference. >>>> >>> Ok, that makes it my department :-) >>> >>> Any indication what the leak depends on ? >>> >>> number of requests ? >>> >>> size of requests ? >>> >>> hit rate ? >>> >>> >> >> -- >> Dmitry Panov >> >> >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > > > >-- >Per Buer, Varnish Software >Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer >Varnish makes websites fly! >Want to learn more about Varnish? >http://www.varnish-software.com/whitepapers > >--0015174c3d4addc4d1049e447a1e >Content-Type: text/html; charset=ISO-8859-1 >Content-Transfer-Encoding: quoted-printable > >Hi.

I've been running trunk on nish-cache.org" target=3D"_blank">varnish-cache.org since yesterday (14= >:00) and there a leak there. Turning off http_gzip_support seems to elimina= >te the leak.
> >

Reproducing the leak is quite simple. Just hammer varni= >sh with a client that doesn't send accept-encoding: gzip.
>

Per.

On Sat, M= >ar 12, 2011 at 12:45 AM, Dmitry Panov <to:dmitry.panov at yahoo.co.uk" target=3D"_blank">dmitry.panov at yahoo.co.uk= >> wrote:
> > >
x #ccc solid;padding-left:1ex">Hi Poul,
>
>I think I found the reason. I tried the same test but with http_gzip_suppor= >t=3Doff and it worked fine. tsung does not support gzip so varnish had to g= >unzip objects for it and this is where the leak appears to be. Hope it help= >s.
> > >
>
>
>On 11/03/2011 14:06, Poul-Henning Kamp wrote:
>
x #ccc solid;padding-left:1ex"> >In message<ank">4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes:
>
>
x #ccc solid;padding-left:1ex"> >Both. The patch doesn't make any difference.
>
>Ok, that makes it my department :-)
>
>Any indication what the leak depends on ?
>
>number of requests ?
>
>size of requests ?
>
>hit rate ?
>
>
>
>
>--
>Dmitry Panov

>
>
>_______________________________________________
>varnish-dev mailing list
>varnish-= >dev at varnish-cache.org
> target=3D"_blank">http://www.varnish-cache.org/lists/mailman/listinfo/varn= >ish-dev
>



--
Per Buer,= >=A0Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Sk= >ype: per.buer
Varnish makes websites fly!
Want to learn more about Va= >rnish? lank">http://www.varnish-software.com/whitepapers
> > >
>
> >--0015174c3d4addc4d1049e447a1e-- > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From perbu at varnish-software.com Sat Mar 12 15:08:00 2011 From: perbu at varnish-software.com (Per Buer) Date: Sat, 12 Mar 2011 16:08:00 +0100 Subject: Memory leak in trunk? In-Reply-To: <1425.1299941271@critter.freebsd.dk> References: <1425.1299941271@critter.freebsd.dk> Message-ID: Tested. The leak is gone. www.varnish-cache.org updated, recompiled and restarted. I've added some ESI processing to the mix for good measure. :-) Per. On Sat, Mar 12, 2011 at 3:47 PM, Poul-Henning Kamp wrote: > In message , > Per > Buer writes: > > I think I've fixed this, a stupid oversight on my part. > > Please test again, when possible. > > Poul-Henning > > >--0015174c3d4addc4d1049e447a1e > >Content-Type: text/plain; charset=ISO-8859-1 > > > >Hi. > > > >I've been running trunk on varnish-cache.org since yesterday (14:00) and > >there a leak there. Turning off http_gzip_support seems to eliminate the > >leak. > > > >Reproducing the leak is quite simple. Just hammer varnish with a client > that > >doesn't send accept-encoding: gzip. > > > > > >Per. > > > >On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov >wrote: > > > >> Hi Poul, > >> > >> I think I found the reason. I tried the same test but with > >> http_gzip_support=off and it worked fine. tsung does not support gzip so > >> varnish had to gunzip objects for it and this is where the leak appears > to > >> be. Hope it helps. > >> > >> > >> > >> On 11/03/2011 14:06, Poul-Henning Kamp wrote: > >> > >>> In message<4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: > >>> > >>> Both. The patch doesn't make any difference. > >>>> > >>> Ok, that makes it my department :-) > >>> > >>> Any indication what the leak depends on ? > >>> > >>> number of requests ? > >>> > >>> size of requests ? > >>> > >>> hit rate ? > >>> > >>> > >> > >> -- > >> Dmitry Panov > >> > >> > >> > >> _______________________________________________ > >> varnish-dev mailing list > >> varnish-dev at varnish-cache.org > >> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > >> > > > > > > > >-- > >Per Buer, Varnish Software > >Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > >Varnish makes websites fly! > >Want to learn more about Varnish? > >http://www.varnish-software.com/whitepapers > > > >--0015174c3d4addc4d1049e447a1e > >Content-Type: text/html; charset=ISO-8859-1 > >Content-Transfer-Encoding: quoted-printable > > > >Hi.

I've been running trunk on http://var= > >nish-cache.org" target=3D"_blank">varnish-cache.org since yesterday > (14= > >:00) and there a leak there. Turning off http_gzip_support seems to > elimina= > >te the leak.
> > > >

Reproducing the leak is quite simple. Just hammer > varni= > >sh with a client that doesn't send accept-encoding: > gzip.
>>

Per.

On Sat, > M= > >ar 12, 2011 at 12:45 AM, Dmitry Panov < href=3D"mail= > >to:dmitry.panov at yahoo.co.uk" target=3D"_blank">dmitry.panov at yahoo.co.uk > = > >> wrote:
> > > > > >
.8ex;border-left:1p= > >x #ccc solid;padding-left:1ex">Hi Poul,
> >
> >I think I found the reason. I tried the same test but with > http_gzip_suppor= > >t=3Doff and it worked fine. tsung does not support gzip so varnish had to > g= > >unzip objects for it and this is where the leak appears to be. Hope it > help= > >s.
> > > > > >
> >
> >
> >On 11/03/2011 14:06, Poul-Henning Kamp wrote:
> >
.8ex;border-left:1p= > >x #ccc solid;padding-left:1ex"> > >In message< target=3D"_bl= > >ank">4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes:
> >
> >
.8ex;border-left:1p= > >x #ccc solid;padding-left:1ex"> > >Both. The patch doesn't make any difference.
> >
> >Ok, that makes it my department :-)
> >
> >Any indication what the leak depends on ?
> >
> >number of requests ?
> >
> >size of requests ?
> >
> >hit rate ?
> >
> >
> >
> >
> >--
> >Dmitry Panov



--
Per Buer,= > >=A0Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / > Sk= > >ype: per.buer
Varnish makes websites fly!
Want to learn more about > Va= > >rnish? target=3D"_b= > >lank">http://www.varnish-software.com/whitepapers
> > > > > >
> >
> > > >--0015174c3d4addc4d1049e447a1e-- > > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Want to learn more about Varnish? http://www.varnish-software.com/whitepapers -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Sat Mar 12 17:22:07 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sat, 12 Mar 2011 18:22:07 +0100 Subject: IMS: grace & keep Message-ID: <4D7BABBF.7030106@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello Poul-Henning, I'd like to make sure that we have a common understanding about the grace and keep intervals for backend conditionals. In particular, IIUC an object that is still in grace can be a candidate for an IMS request. Shortly after VUG you wrote: > ttl=0, grace>0 > This is a special case, where the object is expired, but still > available for grace (you have to first set obj.ttl = 0 and then set > obj.grace to nonzero, as obj.ttl = 0 also clears obj.grace) > I think in this case it is a candidate, the admin clearly considers > the content valid enough to use in an emergency, so it should be > fair game for IMS also. So: if TTL has elapsed, but Varnish finds an object in grace, and decides as it does now to serve it from cache, then it does so; otherwise Varnish uses it to attempt the conditional request (if the object has the headers for it). That would mean that setting keep = 0 does *not* switch off backend conditionals, if grace is non-zero. keep = 0 just means that the interval in which IMS can take place is not extended any longer than grace. The only way to prevent backend conditionals altogether would be in VCL, by removing the IMS and INM headers from bereq. Is that what you hand in mind? That last part tripped me up, because I was testing with keep = 0, expecting IMS requests to be switched off, but they were happening anyway (which is why I thought I had a bug). It turns out that it was the objects in grace. I think it would be fine for Varnish to work that way (and it's nice to not have a bug for once), but it should be documented clearly, so that others are not as confused as I was. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNe6u+AAoJEOUwvh9pJNURS8QP/RnLLqUdAPBTDFkNjz2jEOY/ khCW37md+T5Lu8GkL5y6+ueX00AUDEzsWFNkzL6nmk525UyUURxCUWISYXKNTpeT lm8TDarzH1HHTuZHhZqJ0Z2Gzy/5dp7UevxzY6d53iN/mjQt/4x3hL4S/EiEngBX YvwDaKS5xlfEWlUZ4xI3X5s6hWTiK2GdkNwhAQwnLox43NwuU/yoCeuN+tMkYWhU GV/X65ueUCwJ9e/CtP6P7ij+Sk/eWUx3KSdMqKB9Oi914gI3CZFmpdV+WLdDe4SP dtJYuqgxiuwHEokRsdC6VjhnxoKXBUwo0QGNY9Gmi4rDo+LhM0qNxKiRnTeBvu4t Eyhwe4HpkESdbh8KvwMCnQzMmSC+zbIMnuTYQ6Y92B6/JI8HgWKDLt2DPObl0O1j W3mgoqJCz2NDN2tWFt/K9EN92upIYquISxEges24mMRZUXWyw9oIcnFT0ITW6Jlf ogXaJBF+cluILFQRO1ubv6XYEZF6qq0jM3Otav8pVtwTBWpFpRLN/qVVdGF1Fy5D E+SO+YOyHcmFuZwJSSlYpHEBZECPxTdIwOcxqfzyQB3bhEt+ANySE71PmWErKC98 h8hQrA6HEMSp4f1SjZO/zwiD7jIWpLtrVNahml+hdl5Tk2qrxqhcUD9xkG3RfGoi d76Zk2gi3AUqHPn8LbP2 =Dd9Q -----END PGP SIGNATURE----- From jesus at omniti.com Sat Mar 12 23:58:34 2011 From: jesus at omniti.com (Theo Schlossnagle) Date: Sat, 12 Mar 2011 18:58:34 -0500 Subject: [PATCH] Solaris: Test for SO_{RCV,SND}TIMEO needs NET_LIBS In-Reply-To: <4D7A50D6.4040601@schokola.de> References: <4D7A50D6.4040601@schokola.de> Message-ID: Do you have evidence that they work? both setsockopt and the SO_RCVTIMEO define have existed for 10+ years, but anytime setsockopt is called with that, it will simply fail. On Fri, Mar 11, 2011 at 11:41 AM, Nils Goroll wrote: > I've revisited the issue of missing socket timeouts on Solaris and noticed that > they actually exist in OpenSolaris since integration of the Volo framework in > fall 2008 - but the autoconf tests failed due to missing libs like in this case: > > configure:28453: checking whether SO_RCVTIMEO works > configure:28496: gcc -o conftest -m64 -D_REENTRANT -pthreads ? conftest.c ? >&5 > Undefined ? ? ? ? ? ? ? ? ? ? ? first referenced > ?symbol ? ? ? ? ? ? ? ? ? ? ? ? ? ? in file > socket ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/tmp//ccXeai7q.o > setsockopt ? ? ? ? ? ? ? ? ? ? ? ? ?/var/tmp//ccXeai7q.o > > Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle From phk at phk.freebsd.dk Sun Mar 13 08:51:24 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 13 Mar 2011 08:51:24 +0000 Subject: IMS: grace & keep In-Reply-To: Your message of "Sat, 12 Mar 2011 18:22:07 +0100." <4D7BABBF.7030106@uplex.de> Message-ID: <88058.1300006284@critter.freebsd.dk> In message <4D7BABBF.7030106 at uplex.de>, Geoff Simmons writes: >So: if TTL has elapsed, but Varnish finds an object in grace, and >decides as it does now to serve it from cache, then it does so; >otherwise Varnish uses it to attempt the conditional request (if the >object has the headers for it). Well, this sort of goes to the max(grace,keep) vs (grace+keep) issue. If we make it max(grace,keep), then only keep should be considered for IMS, and the two timers are 100% independent. If we make it (grace+keep) they act in tandem and any object in grace is automatically subject to IMS and you can not totally disable IMS on an object. Which is more intuitive ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Sun Mar 13 09:37:59 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 13 Mar 2011 10:37:59 +0100 Subject: IMS: grace & keep In-Reply-To: <88058.1300006284@critter.freebsd.dk> References: <88058.1300006284@critter.freebsd.dk> Message-ID: <4D7C9077.4020703@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/13/11 9:51 AM, Poul-Henning Kamp wrote: > >> So: if TTL has elapsed, but Varnish finds an object in grace, and >> decides as it does now to serve it from cache, then it does so; >> otherwise Varnish uses it to attempt the conditional request (if the >> object has the headers for it). > > Well, this sort of goes to the max(grace,keep) vs (grace+keep) issue. > > If we make it max(grace,keep), then only keep should be considered > for IMS, and the two timers are 100% independent. > > If we make it (grace+keep) they act in tandem and any object in grace > is automatically subject to IMS and you can not totally disable > IMS on an object. I agree. (In the latter case, you could disable IMS, but only in VCL.) > Which is more intuitive ? We had originally gone with max(grace,keep) because it gives admins all of the options. The difference is that with max(grace,keep), it's possible to have keep < grace. In this configuration, there is a period (keep < t < grace) in which the cache can retain an object that it could serve in grace mode, but if it decides not to, then the backend fetch will not be attempted with IMS. I think it's not out of the question that an admin might want that. A site may need to have a long grace period, due to many concurrent sessions and occasionally unhealthy backends. They might also want to guard against the "PHP bug" scenario that you talked about -- the backend can't always determine that a certain resource has changed, so it might return 304 for IMS when it actually shouldn't. For that, it would make sense to have the long grace but short keep period. With (grace+keep), ie the keep interval extends strictly after the grace interval, as in the trunk right now, I don't think you can do that, not even in VCL, if I'm not mistaken. (Maybe there's some very clever VCL that I'm overlooking.) So I think that with max(grace,keep), and default_keep == default_grace, we have the widest range of options and simplest defaults. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNfJB2AAoJEOUwvh9pJNURsZcP/iPIogLHuIzm0gnqxiIEYsvK w41XPjsm+cGTQX670qC37vHDiBzMhF54ZU2e9Pw/zGjlgXN2vMXmw7jKF7jIn7+v r58R4s18vkf8YRUSLyru3+YJoU1iaR0wVg+KjcjLrknDTXNM4g+dwg5cj/BNXsAE a9MyL8HkvrMl5XfeoyZ+oGY+cdUF32fIIQ6VHfuyW95HZ51M+FSDkM4qHDOyNc5+ NeDkCrrF7rCXj2O88H/7D6ERNSmdZKn8d3emrSkTb9+m0Q3M+3+RLZ6xX+MMAjQc z8QNwrcZHyczhb3xsYisn+vqeLjRLXr1oboPit2iOlFoEO4g9Wrg57jl/z9t+VT6 ol0w8lYD9/gCuhVFmjmJQBrq/M2m7xqw9X2xBgnzUQp820wHJ1Y0T/qaKWm6gPs2 QzawFxerNWv5KhqAqK4z3dIoL052aBt1n8BDnOBwvZ/UwunCemerdKHkvlboZAoD PcR25nkwcPah7tz0sersvpTAlMqkDLB6zwAv7vYlUp+dBFFolk5oRiwQhoPSUVpC WIpgOealAp6iKKrfnFZsGtkeYviVvfaBFv6byH+A2dokzjhGm2CEojw78rkTMI77 lyFswEbHYc8CYiRN3wZDpgW4BpUuy3ROTGa06g9ufHgzlNRn1ArGMMngkGNbAv1s llKNDC2ijY/PtxYMv09m =LBFb -----END PGP SIGNATURE----- From slink at schokola.de Sun Mar 13 12:45:00 2011 From: slink at schokola.de (Nils Goroll) Date: Sun, 13 Mar 2011 13:45:00 +0100 Subject: [PATCH] Solaris: Test for SO_{RCV,SND}TIMEO needs NET_LIBS In-Reply-To: References: <4D7A50D6.4040601@schokola.de> Message-ID: <4D7CBC4C.2070307@schokola.de> Hi Theo and all, On 03/13/11 12:58 AM, Theo Schlossnagle wrote: > Do you have evidence that they work? both setsockopt and the > SO_RCVTIMEO define have existed for 10+ years, but anytime setsockopt > is called with that, it will simply fail. Yes. tests/b0002[25].vtc pass on my snv_134 machine. Before, they were always skipped due to the missing feature ("SKIPPING test, missing feature SO_RCVTIMEO_WORKS"), now they actually do something. All other tests fail sometimes, but not reproducibly, with an "Assert error in TCP_blocking(), tcp.c line 184: Condition(TCP_Check(j)) not true.". I am still trying to understand this (see also http://www.varnish-cache.org/trac/ticket/865 ) I will report on experience with production systems when I have it. Nils From phk at phk.freebsd.dk Sun Mar 13 13:00:38 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 13 Mar 2011 13:00:38 +0000 Subject: [PATCH] Solaris: Test for SO_{RCV,SND}TIMEO needs NET_LIBS In-Reply-To: Your message of "Sun, 13 Mar 2011 13:45:00 +0100." <4D7CBC4C.2070307@schokola.de> Message-ID: <9754.1300021238@critter.freebsd.dk> In message <4D7CBC4C.2070307 at schokola.de>, Nils Goroll writes: >All other tests fail sometimes, but not reproducibly, with an "Assert error in >TCP_blocking(), tcp.c line 184: Condition(TCP_Check(j)) not true.". Keep an eye on errno... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sun Mar 13 13:02:43 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 13 Mar 2011 13:02:43 +0000 Subject: IMS: grace & keep In-Reply-To: Your message of "Sun, 13 Mar 2011 10:37:59 +0100." <4D7C9077.4020703@uplex.de> Message-ID: <9774.1300021363@critter.freebsd.dk> In message <4D7C9077.4020703 at uplex.de>, Geoff Simmons writes: >So I think that with max(grace,keep), and default_keep == default_grace, >we have the widest range of options and simplest defaults. Works for me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Sun Mar 13 14:12:06 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 13 Mar 2011 15:12:06 +0100 Subject: IMS: grace & keep In-Reply-To: <9774.1300021363@critter.freebsd.dk> References: <9774.1300021363@critter.freebsd.dk> Message-ID: <4D7CD0B6.8070706@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/13/11 2:02 PM, Poul-Henning Kamp wrote: > >> So I think that with max(grace,keep), and default_keep == default_grace, >> we have the widest range of options and simplest defaults. > > Works for me. OK, I'll change this back in the next version of the patch. That will include some changes to things you've already committed to cache_expire.c -- EXP_Keep() and the computation of when in update_object_when(). Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNfNC1AAoJEOUwvh9pJNURETUQAJjb3WyRrbhN20mAG745enyf LzecpEvmlIWrvHUJsRMOUAmCATogTsk+2EFIh620PPhDMcw120X8rfa5Mmj7va5K DJveS5f4pEuJwXoWf/o8b/Qjqq7KXyL1JTQtwUfl82C+elwKkohloMtR6NUJCrg9 1JGizLa97vaS8yd3TpuGfiChgkJw5R0EeuImXAqnymXcsA0m6pORnGzDOUBJKbhq qMq244nmvYqoY4wNQEFZm3pvwsIyPDJXp8BjRT6Sv6iWXyTXfTpIofOxEMbR8+7H TSxjSMs3/oEmfPQiqNP6qjFqJWEuax4tmov+kFfnVUX42iPC6/q3yKKgAquBvb2m t1Lh5NQ8aKaLqomZCLnDD98/6N1DCmQD+HMSUvrRDXRk4HlgUBM8B3RRLmQyVRFb GfY2sit6VFeN2hMjk9uNEentQzMdsrs9y+UzsuEHY4Xt0FUamKaYQ5uZUpVH/ZBs VcO6TOPsET5Wz2F7x6G3zHt1k0VMqyuIwwADV4HrVmkZHANI+1MOMoUdAjoBstQ6 U6Hj4g7tYZNRtJRssAKSUrszO7l/4jWGHAuNC0uu64yveK/LIgK5TDEiag3tXySw 3J+feUsqHJtWIBBncAO39v5eAfoKvKZzAU4hb+X6daF/axK4RS5+d6jfBYUT31j3 C422zFZqSml/la07QyJE =f2s+ -----END PGP SIGNATURE----- From slink at schokola.de Sun Mar 13 16:06:08 2011 From: slink at schokola.de (Nils Goroll) Date: Sun, 13 Mar 2011 17:06:08 +0100 Subject: Solaris: TCP_blocking(), tcp.c line 184: Condition(TCP_Check(j)) not true In-Reply-To: <9754.1300021238@critter.freebsd.dk> References: <9754.1300021238@critter.freebsd.dk> Message-ID: <4D7CEB70.9060100@schokola.de> >> All other tests fail sometimes, but not reproducibly, with an "Assert error in >> TCP_blocking(), tcp.c line 184: Condition(TCP_Check(j)) not true.". > > Keep an eye on errno... See http://www.varnish-cache.org/trac/ticket/865 : That's the confusing part, at least for me: ECONNREFUSED Any ideas? Nils From geoff at uplex.de Mon Mar 14 08:08:08 2011 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 14 Mar 2011 09:08:08 +0100 Subject: [PATCH] backend conditional requests 6th release In-Reply-To: <4D7A01F5.2030601@uplex.de> References: <34200.1299106002@critter.freebsd.dk> <4D70312D.8010102@uplex.de> <4D70F98E.4020902@uplex.de> <4D727A54.6040005@uplex.de> <4D73F2E3.9050506@uplex.de> <4D74B505.7080709@uplex.de> <4D7513B6.2040002@uplex.de> <4D775D22.2070202@uplex.de> <4D7915FB.3060407@yahoo.co.uk> <4D79296E.8010502@uplex.de> <4D797B47.8000703@yahoo.co.uk> <4D79CFB3.1070402@uplex.de> <4D7A01F5.2030601@uplex.de> Message-ID: <4D7DCCE8.6080108@uplex.de> Hello all, This is the 6th version of the patch for backend conditional requests (usually If-Modified-Since). It applies to; commit c3c23079c171ee0049005fe335c041fdd2addcf1 Date: Sun Mar 13 16:48:18 2011 +0000 After discussion on varnish-dev, we've changed the meaning of the keep interval back to what it had been in earlier versions: both keep and grace are counted down simultaneously after TTL elapses, an object is ready for eviction after ttl + max(grace, keep), and default_keep == default_grace. This has been reflected in the Wiki page: http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests As always, please test it if you can and let us know how it goes. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.conditional_backend_reqs_6 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Mon Mar 14 16:08:40 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 14 Mar 2011 16:08:40 +0000 Subject: [PATCH] Normalizing the Host: header In-Reply-To: Your message of "Fri, 11 Mar 2011 15:05:45 PST." <10ABE5A2-05CF-47CF-8FA7-FC5A0ECB6C91@mosso.com> Message-ID: <2786.1300118920@critter.freebsd.dk> In message <10ABE5A2-05CF-47CF-8FA7-FC5A0ECB6C91 at mosso.com>, Adrian Otto writes : >In this case, I offer the advice that all host related headers should be >case folded, because DNS naming is case insensitive. So essentially >anywhere Varnish handles a hostname for any comparison, it should follow >the same rules. Yeah, and that is where the trouble start, short at guessing, we have no way of knowing which strings are hostnames and which are not. (think X-My-Secret and cookies...) Rather than venture into guessing, my attitude so far has been to take a hands off aproach and force people to think about this themselves. Obviously that is nor particularly practical either. What I'm looking for is the sensible middle ground... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dmitry.panov at yahoo.co.uk Wed Mar 16 14:10:47 2011 From: dmitry.panov at yahoo.co.uk (Dmitry Panov) Date: Wed, 16 Mar 2011 14:10:47 +0000 Subject: Memory leak in trunk? In-Reply-To: <1425.1299941271@critter.freebsd.dk> References: <1425.1299941271@critter.freebsd.dk> Message-ID: <4D80C4E7.2090800@yahoo.co.uk> Hi, I can also confirm that the leak is gone. Many thanks! On 12/03/2011 14:47, Poul-Henning Kamp wrote: > In message, Per > Buer writes: > > I think I've fixed this, a stupid oversight on my part. > > Please test again, when possible. > > Poul-Henning > >> --0015174c3d4addc4d1049e447a1e >> Content-Type: text/plain; charset=ISO-8859-1 >> >> Hi. >> >> I've been running trunk on varnish-cache.org since yesterday (14:00) and >> there a leak there. Turning off http_gzip_support seems to eliminate the >> leak. >> >> Reproducing the leak is quite simple. Just hammer varnish with a client that >> doesn't send accept-encoding: gzip. >> >> >> Per. >> >> On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panovwrote: >> >>> Hi Poul, >>> >>> I think I found the reason. I tried the same test but with >>> http_gzip_support=off and it worked fine. tsung does not support gzip so >>> varnish had to gunzip objects for it and this is where the leak appears to >>> be. Hope it helps. >>> >>> >>> >>> On 11/03/2011 14:06, Poul-Henning Kamp wrote: >>> >>>> In message<4D7A1DC5.7030405 at yahoo.co.uk>, Dmitry Panov writes: >>>> >>>> Both. The patch doesn't make any difference. >>>> Ok, that makes it my department :-) >>>> >>>> Any indication what the leak depends on ? >>>> >>>> number of requests ? >>>> >>>> size of requests ? >>>> >>>> hit rate ? >>>> >>>> Best regards, -- Dmitry Panov From martin at varnish-software.com Mon Mar 21 12:16:53 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 21 Mar 2011 13:16:53 +0100 Subject: Patch review for adding command line macro definitions to varnishtest Message-ID: Please find a patch for allowing command line option macro definitions to varnishtest, which could then be used to replace text in the test scripts. The syntax is '-D name=value', which would then expand ${name} into value when executing the test script. The specific use case for adding this is to have the ability to pass the build directory of the .so library for a varnish mod to varnishtest when building modules out-of-tree. -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishtest-dopt.patch Type: application/octet-stream Size: 3240 bytes Desc: not available URL: From tfheen at varnish-software.com Mon Mar 21 13:49:18 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 21 Mar 2011 14:49:18 +0100 Subject: Patch review for adding command line macro definitions to varnishtest In-Reply-To: (Martin Blix Grydeland's message of "Mon, 21 Mar 2011 13:16:53 +0100") References: Message-ID: <87aago1t41.fsf@qurzaw.varnish-software.com> ]] Martin Blix Grydeland | Please find a patch for allowing command line option macro definitions to | varnishtest, which could then be used to replace text in the test scripts. | The syntax is '-D name=value', which would then expand ${name} into value | when executing the test script. The specific use case for adding this is to | have the ability to pass the build directory of the .so library for a | varnish mod to varnishtest when building modules out-of-tree. Looks reasonable to me, with a minor stylistic nit: + exit (2); shouldn't this be exit(2); with no space? -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From thierry.magnien at sfr.com Tue Mar 22 13:14:55 2011 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 22 Mar 2011 14:14:55 +0100 Subject: Patch proposal: handling multiple Set-Cookie headers from backend Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA18@EXCV001.encara.local.ads> Hi, I've read the discussion on IRC channel about this: http://varnish-cache.org/trac/ticket/432 I had a look into the source code and it seemed to me that the following patch would be enough to get it working. However, I find it so simple that I'm afraid I missed something ;-) Regards, Thierry MAGNIEN DGRE/DT/TPS/PFD SFR thierry.magnien at sfr.com diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c index e3cc4e2..a7a4e50 100644 --- a/bin/varnishd/cache_center.c +++ b/bin/varnishd/cache_center.c @@ -506,6 +506,7 @@ cnt_fetch(struct sess *sp) */ http_CollectHdr(sp->wrk->beresp, H_Cache_Control); http_CollectHdr(sp->wrk->beresp, H_Vary); + http_CollectHdr(sp->wrk->beresp, H_Set_Cookie); /* * Figure out how the fetch is supposed to happen, before the diff --git a/include/http_headers.h b/include/http_headers.h index b27966d..664641f 100644 --- a/include/http_headers.h +++ b/include/http_headers.h @@ -92,6 +92,7 @@ HTTPH("Range", H_Range, 1, 0, HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* HTTPH("Referer", H_Referer, 1, 0, 0, 0, 0) /* RFC2616 14.36 */ HTTPH("Retry-After", H_Retry_After, 2, 0, 0, 0, 0) /* RFC2616 14.37 */ HTTPH("Server", H_Server, 2, 0, 0, 0, 0) /* RFC2616 14.38 */ +HTTPH("Set-Cookie", H_Set_Cookie, 2, 0, 0, 0, 0) /* RFC2109 */ HTTPH("TE", H_TE, 1, 3, HTTPH_R_PASS | HTTPH_A_PASS | HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.39 */ HTTPH("Trailer", H_Trailer, 1, 3, HTTPH_R_PASS | HTTPH_A_PASS | HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.40 */ HTTPH("Transfer-Encoding", H_Transfer_Encoding, 2, 3, HTTPH_R_PASS | HTTPH_A_PASS | HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.41 */ From phk at phk.freebsd.dk Tue Mar 22 16:16:32 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 22 Mar 2011 16:16:32 +0000 Subject: Patch proposal: handling multiple Set-Cookie headers from backend In-Reply-To: Your message of "Tue, 22 Mar 2011 14:14:55 +0100." <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA18@EXCV001.encara.local.ads> Message-ID: <49473.1300810592@critter.freebsd.dk> In message <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA18 at EXCV001.encara.local.ads >, "MAGNIEN, Thierry" writes: >I had a look into the source code and it seemed to me that the >following patch would be enough to get it working. However, I find >it so simple that I'm afraid I missed something ;-) You did :-) There is a vmod_std function for doing this in -trunk now. import std from ... sub vcl_fetch { std.collect(resp.http.set-cookie); } I'm still hoping that we'll find time/hands to produce a vmod_cookie which would give a sensible API, so people can clearly express what they want: import cookie from ... sub vcl_recv { cookie.keep_only("This_cookie", "That_cookie"); if (cookie.get("This_cookie") == "mumblemumble") { ... } } sub vcl_fetch { cookie.keep_only("This_cookie", "That_cookie"); } -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From thierry.magnien at sfr.com Tue Mar 22 16:25:45 2011 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 22 Mar 2011 17:25:45 +0100 Subject: Patch proposal: handling multiple Set-Cookie headers from backend In-Reply-To: <49473.1300810592@critter.freebsd.dk> References: Your message of "Tue, 22 Mar 2011 14:14:55 +0100." <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA18@EXCV001.encara.local.ads> <49473.1300810592@critter.freebsd.dk> Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA1B@EXCV001.encara.local.ads> OK, thanks for your response. However, does this mean my patch would be OK for a version 2.1.5 if trunk is not "production-stable" enough ? Thierry -----Message d'origine----- De?: phk at critter.freebsd.dk [mailto:phk at critter.freebsd.dk] De la part de Poul-Henning Kamp Envoy??: mardi 22 mars 2011 17:17 ??: MAGNIEN, Thierry Cc?: varnish-dev at varnish-cache.org Objet?: Re: Patch proposal: handling multiple Set-Cookie headers from backend In message <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA18 at EXCV001.encara.local.ads >, "MAGNIEN, Thierry" writes: >I had a look into the source code and it seemed to me that the >following patch would be enough to get it working. However, I find >it so simple that I'm afraid I missed something ;-) You did :-) There is a vmod_std function for doing this in -trunk now. import std from ... sub vcl_fetch { std.collect(resp.http.set-cookie); } I'm still hoping that we'll find time/hands to produce a vmod_cookie which would give a sensible API, so people can clearly express what they want: import cookie from ... sub vcl_recv { cookie.keep_only("This_cookie", "That_cookie"); if (cookie.get("This_cookie") == "mumblemumble") { ... } } sub vcl_fetch { cookie.keep_only("This_cookie", "That_cookie"); } -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Mar 22 17:31:48 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 22 Mar 2011 17:31:48 +0000 Subject: Patch proposal: handling multiple Set-Cookie headers from backend In-Reply-To: Your message of "Tue, 22 Mar 2011 17:25:45 +0100." <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA1B@EXCV001.encara.local.ads> Message-ID: <49765.1300815108@critter.freebsd.dk> In message <4A029B1A60B8E340A50D654D2F130DAA2FCEA3AA1B at EXCV001.encara.local.ads >, "MAGNIEN, Thierry" writes: >OK, thanks for your response. However, does this mean my patch would be OK = >for a version 2.1.5 if trunk is not "production-stable" enough ? I do not want that change committed, if that's what you're asking, it would cost serious workspace eatage for some sites. But for your own private use it should be fine. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From perbu at varnish-software.com Wed Mar 23 17:45:11 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 23 Mar 2011 18:45:11 +0100 Subject: New man page suggested: varnish-cli Message-ID: I've ripped out the commands, syntax and descriptions from "man varnishd" and put it into varnish-cli (7) and added a lot of general explanations. It started because I wanted to detail out the bans and purges docs (I catch myself explaining bans and purges quite often). Comments? It doesn't actually create any man pages on build yet, but should show up on the web RSN. Unless somebody objects I'll see if I can get to build like the rest. -- Per Buer,?Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Want to learn more about Varnish? http://www.varnish-software.com/whitepapers From geoff at uplex.de Thu Mar 24 06:17:12 2011 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 24 Mar 2011 07:17:12 +0100 Subject: New man page suggested: varnish-cli In-Reply-To: References: Message-ID: <4D8AE1E8.40603@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/23/11 6:45 PM, Per Buer wrote: > I've ripped out the commands, syntax and descriptions from "man > varnishd" and put it into varnish-cli (7) and added a lot of general > explanations. It started because I wanted to detail out the bans and > purges docs (I catch myself explaining bans and purges quite often). > > Comments? What's the rationale for separating this from the documentation for varnishadm? I'd think that's the one place where most Varnish users ever encounter the CLI. I remember that when I was first working with Varnish, several times I opened the varnishadm doc to find that information, and then was stumped about where it is. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNiuHnAAoJEOUwvh9pJNURnKsP+wVjJc73E2OcZMdKuhpbwHUM OcZgO3VKbOK5tLdkyaQnsnEkgPVUlvSIWyZ2M0KanItaGTjK6p615n12x5gteLEm cVGDVBzVH6WOmqKsfAQfhjzDynwTM+mah/BtzH/8CLceZRdpnz6/zYy5OsCQSLpw 2rz0p+AC/6eCui5MqmEwXFr+R4C2F6r8LYDirMWWTHOI64I3c8QZBi4T9sbEQTSk 22y5AA0rD0PdodrsLUSKxsYNzbNdYC0YzGMjwcEbF6RpGxieajED7NzHWiiJjNCe xVt7kPJayEKgWkUMN2suuH0+KkmOUAZFPZxGpetMFb283GtyalDyJaIpv38nZlt5 e1TXSagXIoZjv8q/bGV1UWXWUxduTq7SeLgPOI8CdwmCydFHypjXeiW/f7CTqGGt tbfq/o1F+7TJUCr04dsEElxIsi8d5Duv2kNQZmN2h+fhxzO/jUYdYFlpM7MOAKUD E6k8Ff4KrdVDOyfOaH0ZUEONRyE61pkHA4qRoC+qWROZGeyve4B/6NhKefmLXeqq Fs32Rn70y+2fSpn2jVI5r0Ks0XGF/q5Cmz1HuGZX/ZV+csKa2iEkezQkQuORHo5A lVXj91Ge72RIcIcsdfkk+xUPbW8UMzegmHuH/Qdn8T9eKVvwWNv4iLwm4vzDLplp CEmLcXwApTRrHPux1pJR =sA2g -----END PGP SIGNATURE----- From perbu at varnish-software.com Thu Mar 24 07:14:57 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 24 Mar 2011 08:14:57 +0100 Subject: New man page suggested: varnish-cli In-Reply-To: <4D8AE1E8.40603@uplex.de> References: <4D8AE1E8.40603@uplex.de> Message-ID: On Thu, Mar 24, 2011 at 7:17 AM, Geoff Simmons wrote: > > What's the rationale for separating this from the documentation for > varnishadm? Because the CLI is available a lot of places. It's in varnishd (with -d), varnishadm and potentially a whole bunch of other tools. > I'd think that's the one place where most Varnish users ever > encounter the CLI. I remember that when I was first working with > Varnish, several times I opened the varnishadm doc to find that > information, and then was stumped about where it is. Good point. I'll make sure to put in pointers there. -- Per Buer,?Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Want to learn more about Varnish? http://www.varnish-software.com/whitepapers