From des at linpro.no Sat Jul 8 10:35:42 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 12:35:42 +0200 Subject: setting the port number Message-ID: The attached VCL file trigges the following assertion: root at dma /opt/varnish# sh -x varnish.sh + realpath varnish.sh + dirname /home/varnish/varnish.sh + VARNISH_HOME=/home/varnish + export LD_LIBRARY_PATH=/home/varnish/lib + export PATH=/root/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/opt/varnish/bin:/home/varnish/bin + varnishd -p 80 -f /home/varnish/default.vcl -s file,/home/varnish/storage -w 10 Assertion failed: (t->tok == CSTR), function EncString, file vcl_compile.c, line 389. Abort trap (core dumped) This is basically the default config, with backend.host set to "localhost" and backend.port set to "79". The code is unmodified top-of-tree. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: default.vcl URL: From andersb at vgnett.no Sat Jul 8 14:53:54 2006 From: andersb at vgnett.no (Anders Berg) Date: Sat, 8 Jul 2006 16:53:54 +0200 Subject: Features in Varnish In-Reply-To: References: Message-ID: <1300C52E-0DC1-47AD-948A-1098DF295559@vgnett.no> I have had the time to mock up a featurelist for Varnish. This is not the list that we need to get done pretty soon concering the features that Varnish v 1.0 should have, but rather a marketing featurelist that will be present on the website. It can of cource be used as a base for the aforementioned list. I have tried to make the list as sexy as possible but may have forgotten things we feel are good features. Also I might have taken on the marketing hat to hard and overstated the features. So feedback is required. Featurelist Varnish: - Pure HTTP accelerator Varnish will deliver cached documents received from your backend(s), and do this fast - Very fast Varnish is buildt for speed, and should out-perform any other HTTP accelerator - Multiprocessor/core support Buildt with the future in mind, Varnish supports modern hardware technology - 64-bit computing support While it can run on a 32-bit platform, Varnish was buildt with 64-bit computing in mind - Modern operatingsystem technology To be able to deliver content as efficient as possible Varnish takes in use the latest operatingsystem features like Kqueue/Epoll and Sendfile. Varnish is also IPv6 compatible - Powerful configuration possibilities Varnish has a powerful configurationlanguage (VCL) which gives you endless possibilities and power over your content. You can reference another configurationfile if say your backend is acting slow - Runtime configuration deployment Changing of configurationfiles are done in runtime, no need to stop/ restart your service(s) - Protection of backend(s) Varnish is easily configured to not just stand around and see your backend(s) die if you are getting flooded with unwanted traffic. Give crawlers the content you feel they deserve. If your backend(s) should experience planned or unplanned downtime configure Varnish to never expire or fetch new content. - Ease of managment Your Varnish server(s) can be centrally controlled from a managment program for quick deployment of changes to your system. Different interfaces are available for those that don't like the commandline. - Central and flexible logging Logs are written to shared-memory, so logging is cheap. A centrally controlled logger program connects to the servers and collects these data so you can easily customize what, and how, you wanna logg to either display or file. This is done securely. - Realtime statistics As with logging, we provide you with a centrally controlled statistics collector to aggregate numbers from each server or present them individially. Again this is done in a securely manner. - Instant purging No datastructure traversing will accur, Varnish will instantly purge your document(s) - Wildcard purging Again, no datastructure traversing is done and all documents (even millions) matching your wildcard or regexp will intsantly be rendered obsolete and not delivered - Plugable storage backend(s) We give you memory, file or tar-file (Read-Only) backend out of the box, if you wanna store/read from another backend we leave you with an API - Varnish is free! Distributed under the two-clause BSD license, Varnish is free of charge and anybody can download and modify the sourcecode From des at linpro.no Sat Jul 8 15:32:43 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 17:32:43 +0200 Subject: Features in Varnish References: <1300C52E-0DC1-47AD-948A-1098DF295559@vgnett.no> Message-ID: Anders Berg writes: > I have had the time to mock up a featurelist for Varnish. This is > not the list that we need to get done pretty soon concering the > features that Varnish v 1.0 should have, but rather a marketing > featurelist that will be present on the website. It can of cource > be used as a base for the aforementioned list. I can't think of anything to add off the top of my head. BTW, I'd love to get a copy of the new logo and the CSS files for Trac. The sooner we get them in place, the better. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sat Jul 8 16:21:06 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 16:21:06 +0000 Subject: Future plans (was: setting the port number) In-Reply-To: Your message of "Sat, 08 Jul 2006 12:35:42 +0200." Message-ID: <54580.1152375666@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >The attached VCL file trigges the following assertion: Yes, VCL is only just able to handle the default config, pretty much all other variables than those you see in the default code are not implemented yet. But I would like us to spend at least a few emails on project management at this point. I am approximately 60% through the 400.000 NOK, so that leaves something like 300 hours on my part. Knowing the (100-X)/X rule for whatever X is applicable, that means that we will come up short somewhere on some subset of our ambitions. I often use a little excercise to prioritize my work that goes like "If you had to deliver one hour from now, what would you do ?, what if you had two hours ? What if you had 1 day ?" etc. The conclusion I reached on the way back home was quite different from the one I reached on the way up to Oslo. On the way to Oslo, I had expected performance to be better than squid by maybe a factor of two, and I therefore saw VCL as a major product selling point relative to Squid, and I expected us to put a fair bit of effort into VCL before the release date. However, considering that we totally smoked Squid performance wise in the tests, and did so with a stability which if not exactly rivals Squid, then at least seems to have more potential I think we should maybe tone VCL ambitions down a bit, and instead aim at being vastly more reliable and a lot faster than Squid. Despite the results of the livetest, I am painfully aware that there is quite a bit of work yet in the HTTP processing department, and some work in the rest of the "basic mechanics", optimizing the storage allocation, making the worker thread pool dynamic etc. We have the VCL compiler, we have the compiled code for the default config running and working. The major actions (pipe, pass, deliver, fetch, lookup etc) needs to be made to work correctly and reliably because they influence the architecture of the cache process. But other than that I am inclined to leave VCL alone for now. Incrementally adding stuff to VCL later on is very easy, because changes can be trivially tested with telnet as client. In comparison changing the worker thread pool, buffer manangement or HTTP header fritzing code requires testing in a live environment which is slightly more involved. So my proposed plan from now on would be: Analyse logfiles from live test Review and clean up changes made during live test. Implement all VCL actions {pipe, pass, fetch ...} and get cache process structure finally nailed down. Make worker thread pool dynamic. (start running live in VG.no ?) (Invite alpha testers ?) Implement management process more functions (start/stop etc). Improve HTTP header handling & compliance. Improve storage_file allocation policy. (Invite beta testers ?) And then for the remaining time make it buffet work, picking whatever we desire from: Adding more statistics Logfile handling Add VCL features Until we run out of time & money and (Release Varnish 1.0) In addition to all of this, there is the Linux portability, documentation and test-harness tasks, but I sort of hope they will be mostly in Dag-Erlings area ? Does this make sense ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat Jul 8 16:29:21 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 16:29:21 +0000 Subject: Features in Varnish In-Reply-To: Your message of "Sat, 08 Jul 2006 16:53:54 +0200." <1300C52E-0DC1-47AD-948A-1098DF295559@vgnett.no> Message-ID: <54727.1152376161@critter.freebsd.dk> In message <1300C52E-0DC1-47AD-948A-1098DF295559 at vgnett.no>, Anders Berg writes : >- Very fast >Varnish is buildt for speed, and should out-perform any other HTTP >accelerator I'm never keen on such absolutes, at least not until I've convinced my self that they are true :-) >- Modern operatingsystem technology >To be able to deliver content as efficient as possible Varnish takes >in use the latest operatingsystem features like Kqueue/Epoll and >Sendfile. Varnish is also IPv6 compatible The last bit may need a bit of verification. >- Powerful configuration possibilities >Varnish has a powerful configurationlanguage (VCL) which gives you >endless possibilities and power over your content. You can reference >another configurationfile if say your backend is acting slow I think the second statement rather detracts from the first by being very specific. >- Runtime configuration deployment >Changing of configurationfiles are done in runtime, no need to stop/ >restart your service(s) ... for the policy part of the configuration. >- Instant purging >No datastructure traversing will accur, Varnish will instantly purge >your document(s) > >- Wildcard purging >Again, no datastructure traversing is done and all documents (even >millions) matching your wildcard or regexp will intsantly be rendered >obsolete and not delivered Since these are basically the same feature, I would combine them: - Instant wildcard purging of documents A purge command of specifying a regular expression instantly stops delivery of any cached documents matching the expression without wasting time traversing all the cached documents. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sat Jul 8 16:43:14 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 18:43:14 +0200 Subject: Features in Varnish References: <54727.1152376161@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Anders Berg writes: > > - Very fast > > Varnish is buildt for speed, and should out-perform any other HTTP > > accelerator > I'm never keen on such absolutes, at least not until I've convinced > my self that they are true :-) I think we can safely say that Varnish out-performs all other *open source* HTTP accelerators (at least the ones I know of). > > - Modern operatingsystem technology > > To be able to deliver content as efficient as possible Varnish > > takes in use the latest operatingsystem features like Kqueue/Epoll > > and Sendfile. Varnish is also IPv6 compatible > The last bit may need a bit of verification. It is trivial to test, and if it doesn't work, it should be trivial to fix. > > - Powerful configuration possibilities > > Varnish has a powerful configurationlanguage (VCL) which gives you > > endless possibilities and power over your content. You can > > reference another configurationfile if say your backend is acting > > slow > I think the second statement rather detracts from the first by > being very specific. Agreed. > > - Instant purging > > No datastructure traversing will accur, Varnish will instantly > > purge your document(s) > > > > - Wildcard purging > > Again, no datastructure traversing is done and all documents (even > > millions) matching your wildcard or regexp will intsantly be > > rendered obsolete and not delivered > Since these are basically the same feature, I would combine them: Agreed. There are also minor issues of style and typography, but we can deal with them when we actually have a web site on which to stick the list. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Jul 8 16:59:52 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 18:59:52 +0200 Subject: Future plans References: <54580.1152375666@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > The attached VCL file trigges the following assertion: > Yes, VCL is only just able to handle the default config, pretty much > all other variables than those you see in the default code are not > implemented yet. OK. I consider "set backend.port" a priority for reasons which I'll explain below; I'll try to fix it myself during the weekend. > So my proposed plan from now on would be: > > Analyse logfiles from live test > > Review and clean up changes made during live test. > > Implement all VCL actions {pipe, pass, fetch ...} > and get cache process structure finally nailed down. > > Make worker thread pool dynamic. > > (start running live in VG.no ?) When I spoke with Anders earlier today, he expressed interest in having Varnish running continously with a low load. This will allow us to assess the stability of the code and identify any memory leaks. I'd like to run Varnish with valgrind / electric fence / jemalloc with debugging turned on / something similar for at least a few hours and see what comes up. BTW, feel free to instrument Varnish with FlexeLint comments. I'll also contact Coverity to see if they're interested in scanning Varnish; they were very gracious about OpenPAM. > (Invite alpha testers ?) As regards testing, I would like to roll a release of the live-test-1 tag (or something very close to it, with backend.port working properly and debugging mode off by default) and stick it in ports to attract testers. On the downside, testers will generate traffic on the mailing lists, and right now I don't want you to waste time answering support requests. > In addition to all of this, there is the Linux portability, documentation > and test-harness tasks, but I sort of hope they will be mostly in > Dag-Erlings area ? All of that is my responsibility, and I still have about half my 1.0 budget left (not sure about the exact numbers because our timesheet software sucks, I'll have to figure it out manually) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sat Jul 8 17:08:05 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 17:08:05 +0000 Subject: Future plans In-Reply-To: Your message of "Sat, 08 Jul 2006 18:59:52 +0200." Message-ID: <54895.1152378485@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: > >OK. I consider "set backend.port" a priority for reasons which I'll >explain below; I'll try to fix it myself during the weekend. Yes, obviously this is so basic that it needs to work, I was not suggesting abandonning VCL :-) >> (start running live in VG.no ?) > >When I spoke with Anders earlier today, he expressed interest in >having Varnish running continously with a low load. This will allow >us to assess the stability of the code and identify any memory leaks. Yes, this is obviously a good idea, but we need a program to sort the logfiles into "everything looks sane" and "something looks out of pattern" before we can really benefit. >I'd like to run Varnish with valgrind / electric fence / jemalloc with >debugging turned on / something similar for at least a few hours and >see what comes up. Yes, we should do that, but considering how little memory management we actually do, we should not spend too much time. There are there are less than 50 places where memory is actually allocated in the cache process. >BTW, feel free to instrument Varnish with FlexeLint comments. I generally prefer not to splatter flexelint comments in source and just live with the few warnings I cannot silence. >> (Invite alpha testers ?) > >As regards testing, I would like to roll a release of the live-test-1 >tag (or something very close to it, with backend.port working properly >and debugging mode off by default) and stick it in ports to attract >testers. I would prefer to not do until a bit more work (as indicated in my list) has been done, but then I certainly think it is a good idea. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sat Jul 8 17:17:20 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 19:17:20 +0200 Subject: Future plans References: <54895.1152378485@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgav writes: > > OK. I consider "set backend.port" a priority for reasons which > > I'll explain below; I'll try to fix it myself during the weekend. > Yes, obviously this is so basic that it needs to work, I was not > suggesting abandonning VCL :-) Of course not :) > > As regards testing, I would like to roll a release of the > > live-test-1 tag (or something very close to it, with backend.port > > working properly and debugging mode off by default) and stick it > > in ports to attract testers. > I would prefer to not do until a bit more work (as indicated in my > list) has been done, but then I certainly think it is a good idea. I agree that we can't ship it the way it looks right now, but I think we're pretty close to being able to call it alpha. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Sat Jul 8 18:29:31 2006 From: andersb at vgnett.no (Anders Berg) Date: Sat, 8 Jul 2006 20:29:31 +0200 Subject: Future plans (was: setting the port number) In-Reply-To: <54580.1152375666@critter.freebsd.dk> References: <54580.1152375666@critter.freebsd.dk> Message-ID: <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE@vgnett.no> Hi, On Jul 8, 2006, at 18:21, Poul-Henning Kamp wrote: > In message , Dag-Erling =?iso-8859-1? > Q?Sm=F8rgra > v?= writes: > >> The attached VCL file trigges the following assertion: > I am approximately 60% through the 400.000 NOK, so that leaves > something like 300 hours on my part. We have estimated 500.000 NOK on you Poul-Henning :) , check the email dated 20. dec. And 150k NOK on Linpro. Yes, I agree we have to prioritize soon, and I agree with what you have written Poul-Henning with regards to that. If we can attach another 250h we could cover more ground, without breaking budget. Great feedback on the featurelist guys, I'll fix'em up even if I think we should aim for "fastest" period :) Anders Berg From andersb at vgnett.no Sat Jul 8 18:41:19 2006 From: andersb at vgnett.no (Anders Berg) Date: Sat, 8 Jul 2006 20:41:19 +0200 Subject: Future plans (was: setting the port number) In-Reply-To: <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE@vgnett.no> References: <54580.1152375666@critter.freebsd.dk> <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE@vgnett.no> Message-ID: On Jul 8, 2006, at 20:29, Anders Berg wrote: > Hi, > > On Jul 8, 2006, at 18:21, Poul-Henning Kamp wrote: > >> In message , Dag-Erling =? >> iso-8859-1?Q?Sm=F8rgra >> v?= writes: >> >>> The attached VCL file trigges the following assertion: >> I am approximately 60% through the 400.000 NOK, so that leaves >> something like 300 hours on my part. > > We have estimated 500.000 NOK on you Poul-Henning :) , check the > email dated 20. dec. > And 150k NOK on Linpro. Sorry, typo. 500.000 DKK approx. 535k NOK. Then the 250h I talk about later on makes sense. Anders Berg > Yes, I agree we have to prioritize soon, and I agree with what you > have written Poul-Henning with regards to that. > If we can attach another 250h we could cover more ground, without > breaking budget. > > Great feedback on the featurelist guys, I'll fix'em up even if I > think we should aim for "fastest" period :) > > Anders Berg > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From phk at phk.freebsd.dk Sat Jul 8 18:43:22 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 18:43:22 +0000 Subject: Future plans In-Reply-To: Your message of "Sat, 08 Jul 2006 19:17:20 +0200." Message-ID: <55157.1152384202@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >I agree that we can't ship it the way it looks right now, but I think >we're pretty close to being able to call it alpha. Pretty close, but not quite. Lets aim for next weekend or so. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat Jul 8 18:58:31 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 18:58:31 +0000 Subject: Future plans (was: setting the port number) In-Reply-To: Your message of "Sat, 08 Jul 2006 20:29:31 +0200." <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE@vgnett.no> Message-ID: <55260.1152385111@critter.freebsd.dk> In message <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE at vgnett.no>, Anders Berg writes : >> I am approximately 60% through the 400.000 NOK, so that leaves >> something like 300 hours on my part. > >We have estimated 500.000 NOK on you Poul-Henning :) , check the >email dated 20. dec. >And 150k NOK on Linpro. Ohh goodie, I misremembered! :-) The other thing to decide is when to put the August meeting ? The kids have summer vacation until Aug 15, so until then I have relatively large freedom, after that I'd prefer it to be thursday (and friday) The NUUG meeting on August 15 is particularly inconvenient because it is the exact day the kids start school. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat Jul 8 20:05:08 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 20:05:08 +0000 Subject: Future plans In-Reply-To: Your message of "Sat, 08 Jul 2006 18:59:52 +0200." Message-ID: <56237.1152389108@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >> In addition to all of this, there is the Linux portability, documentation >> and test-harness tasks, but I sort of hope they will be mostly in >> Dag-Erlings area ? > >All of that is my responsibility, and I still have about half my 1.0 >budget left (not sure about the exact numbers because our timesheet >software sucks, I'll have to figure it out manually) Speaking of this, how hard would it be to build a simple test setup which uses netcat as both client and backend ? Something like: testone () { varnishd -b localhost:8081 -T one_req & nc -l 8081 < $server_resp > $server_req nc localhost 8080 < $client_req > $client_resp wait diff $server_req $server_req_ref diff $client_resp $client_resp.ref } Since we control the server document, we don't get into trouble comparing Date: headers. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sat Jul 8 20:13:42 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 22:13:42 +0200 Subject: Future plans References: <55157.1152384202@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > I agree that we can't ship it the way it looks right now, but I > > think we're pretty close to being able to call it alpha. > Pretty close, but not quite. Lets aim for next weekend or so. Yep. I did some more testing (fixed the backend.port bug!) and came across at least one important issue which we discussed earlier this week but didn't fix: hashing does not take the host into account, so you can't stick Varnish in front of a server with multiple name-based virtual hosts. I believe that the way we want to fix this is by URL normalization: the URL we store in struct http should be a cleaned-up absolute URI. This can either be done unconditionally in http_Dissect() or CacheWorker(), or as a matter of policy in vcl_recv(). (BTW, I don't like using the rr argument to http_Dissect() to distinguish between request and response; I'd rather have separate functions for each, with a common helper for the headers; but that's picking nits) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Jul 8 20:17:53 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 08 Jul 2006 22:17:53 +0200 Subject: Future plans References: <56237.1152389108@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Speaking of this, how hard would it be to build a simple test setup > which uses netcat as both client and backend ? > > Something like: > > testone () { > varnishd -b localhost:8081 -T one_req & > nc -l 8081 < $server_resp > $server_req > nc localhost 8080 < $client_req > $client_resp > wait > diff $server_req $server_req_ref > diff $client_resp $client_resp.ref > } Not very hard, you just did :) It wouldn't be very hard to write such a test framework in C either; varnish-tools/fetcher would be a good starting point. I'll see if I can tackle it tomorrow. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sat Jul 8 20:36:58 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 08 Jul 2006 20:36:58 +0000 Subject: Future plans In-Reply-To: Your message of "Sat, 08 Jul 2006 22:13:42 +0200." Message-ID: <58526.1152391018@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >Yep. I did some more testing (fixed the backend.port bug!) and came >across at least one important issue which we discussed earlier this >week but didn't fix: hashing does not take the host into account, so >you can't stick Varnish in front of a server with multiple name-based >virtual hosts. > >I believe that the way we want to fix this is by URL normalization: >the URL we store in struct http should be a cleaned-up absolute URI. I don't want to get into URI cleanup at all. My preference would be to do the opposite: hash on $host+" "+$url_without_host because trimming the host from absolute URIs is practically free whereas normalizing them is text-processing. If for some reason people need to do normalization, we can add it as a VCL operation. >(BTW, I don't like using the rr argument to http_Dissect() to >distinguish between request and response; I'd rather have separate >functions for each, with a common helper for the headers; but that's >picking nits) A lot of the HTTP functions evolved and some amount of cleanup is in order, but in this particular case I'm not sure if I agree. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sun Jul 9 04:21:22 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 09 Jul 2006 06:21:22 +0200 Subject: Future plans References: <58526.1152391018@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I don't want to get into URI cleanup at all. My preference would > be to do the opposite: hash on $host+" "+$url_without_host because > trimming the host from absolute URIs is practically free whereas > normalizing them is text-processing. Well, we still want to: - convert the Request-URI from absolute URI to absolute path when the client specifies an absolute URI (RFC2616 5.1.2) - compare the Request-URI to the Host: header if both were specified, and reject the request if they don't match - recognize when the Request-URI includes authentication information (http://user:password at host:port/path/to/document) > If for some reason people need to do normalization, we can add > it as a VCL operation. Judging by how many of my colleagues ask me the "does Varnish replace mod_security" question, I can pretty much guarantee you that URL normalization will be a popular feature. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sun Jul 9 08:56:32 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 09 Jul 2006 10:56:32 +0200 Subject: Orphaned pages Message-ID: We have a couple of orphaned pages on the wiki, which I've gathered in a "Developer resources" page which is now linked from the front page. One of them is a page about event library issues which I just created; I don't want to start a big debate, but feel free to air any ideas here or add them to the wiki. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From anders at fupp.net Mon Jul 10 08:29:55 2006 From: anders at fupp.net (Anders Nordby) Date: Mon, 10 Jul 2006 10:29:55 +0200 Subject: Logging Message-ID: <20060710082955.GA30842@totem.fix.no> Hi, Will it be possible to log cache misses? And would it be possible to log them separately? Cache misses are of particular interest to us here in Aftenposten & Finn. We control what will be cached and how long using the Cache-Control header. Sometimes people forget to add that in some page/code/whatever, and we want to see it when it happens. PS: Performance counters? :-) Cheers, -- Anders. From anders at fupp.net Mon Jul 10 08:33:45 2006 From: anders at fupp.net (Anders Nordby) Date: Mon, 10 Jul 2006 10:33:45 +0200 Subject: NUUG meeting (was: Future plans) In-Reply-To: <55260.1152385111@critter.freebsd.dk> References: <2FFA420E-1CEE-4714-B594-01BBC3DDCCBE@vgnett.no> <55260.1152385111@critter.freebsd.dk> Message-ID: <20060710083345.GB30842@totem.fix.no> On Sat, Jul 08, 2006 at 06:58:31PM +0000, Poul-Henning Kamp wrote: > The other thing to decide is when to put the August meeting ? > > The kids have summer vacation until Aug 15, so until then I have > relatively large freedom, after that I'd prefer it to be thursday > (and friday) > > The NUUG meeting on August 15 is particularly inconvenient because > it is the exact day the kids start school. If august 15 is not possible, please suggest one or (preferably two) other dates in august. Then I can check if it is OK with them. Cheers, -- Anders. From andersb at vgnett.no Mon Jul 10 15:03:08 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 10 Jul 2006 17:03:08 +0200 Subject: Logging In-Reply-To: <20060710082955.GA30842@totem.fix.no> References: <20060710082955.GA30842@totem.fix.no> Message-ID: On Jul 10, 2006, at 10:29, Anders Nordby wrote: > Hi, > > Will it be possible to log cache misses? And would it be possible > to log > them separately? Hi Anders, while I might not be the right one to answer I'll give it a shot. Right now Varnish is logging alot of data involved in the request handling to shared memory on each of the Varnish servers. It's a shared memory buffer at a fixed size (default 8M right now I think), and Varnish writes to it in a FIFO manner. Logging is about attaching itself to the SHM log and "tailing" it. We have a client for tailing the SHM log on one machine, but in the end product we are aiming for central log managment. One machine connects itself through the logging API (secured) to all the servers and gathers data to a central place. There you would write a filter that can read and present whatever data you'd like to view. From what I understand, even if a pure cachemiss logger is not present at launch we will have at least some common loggers and writing a cachemiss logger is what DES and PHK refer to as a SMOP :) Logging to SHM is extremely cheap and we will not bog the servers down with logparsing and such. We will have to wait and see what this is like networkwise, but I guess there is always a way around that that's easier. I am uncertain if we are gonnna implement a way to tell the Varnish daemon itself what we want it to logg in V 1.0. Here is some info about the logging: http://projects.linpro.no/pipermail/varnish-dev/2006-March/000096.html Please correct me if I am wrong :) Anders Berg > > Cache misses are of particular interest to us here in Aftenposten & > Finn. > We control what will be cached and how long using the Cache-Control > header. > Sometimes people forget to add that in some page/code/whatever, and we > want to see it when it happens. > > PS: Performance counters? :-) > > Cheers, > > -- > Anders. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From des at linpro.no Mon Jul 10 15:24:09 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 10 Jul 2006 17:24:09 +0200 Subject: Logging References: <20060710082955.GA30842@totem.fix.no> Message-ID: Anders Berg writes: > while I might not be the right one to answer I'll give it a shot. > Right now Varnish is logging alot of data involved in the request > handling to shared memory on each of the Varnish servers. It's a > shared memory buffer at a fixed size (default 8M right now I think), > and Varnish writes to it in a FIFO manner. It's a circular buffer, actually; varnish doesn't care if anyone reads it. > I am uncertain if we are gonnna implement a way to tell the Varnish > daemon itself what we want it to logg in V 1.0. I don't see any point in doing so. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Mon Jul 10 15:33:47 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 15:33:47 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 10:29:55 +0200." <20060710082955.GA30842@totem.fix.no> Message-ID: <75760.1152545627@critter.freebsd.dk> In message <20060710082955.GA30842 at totem.fix.no>, Anders Nordby writes: >Hi, > >Will it be possible to log cache misses? And would it be possible to log >them separately? This is one of those questions that can either be answered with "yes" (which go you right there) or a quite long explanation, which takes up the rest of the email :-) While it runs, Varnish logs anything of even moderate relevance to the shared-memory circular buffer. Any number of processes can attached to that shared memory and keep an eye on whatever they find relevant. To give an you an idea what is logged, here is a commented sample from a very simple run I just did. (Please note that we have not quite nailed these messages down yet, but all but "Debug" they will be part of the defined API to Varnish) The first column is the numeric type-tag (== column 3) Column 2 is the length of any variable part (the stuff in column 4) max is 255 bytes. Column 3 is a filedescriptor number (or zero) and is used to match entries to the session or backend connection. The fourth column is the variable text. 01 25 0 Debug [Starting 1 worker threads] The minimum number of threads specified was 1. These threads are started as permanent and will never die. 03 4 0 CLI This is the management process checking of the cache process is alive. It happens every few seconds. If the management process doesn't get a reply, it kills the cache process and forks it again. 1f 16 0 WorkThread <1 born permanent> The first thread is created and signs in. 03 4 0 CLI 03 4 0 CLI 04 15 19 SessionOpen <127.0.0.1 53578> We get a connection from localhost 1a 10 19 XID <1243897116> This is a transaction ID which is used to cross reference things in the log. The first one after startup is random and they increment after that. When an object is inserted in the hash, it retains the XID of the transaction that created it (see more below) 0e 3 19 Request 12 1 19 URL 13 8 19 Protocol 14 15 19 Header 14 30 19 Header 14 17 19 Header This is the request we got 17 4 19 VCL_call We call the vcl_recv() function to figure out what should happen. 18 6 19 VCL_trace <1 4.14> 18 5 19 VCL_trace <2 5.9> 18 5 19 VCL_trace <5 8.5> 18 5 19 VCL_trace <6 8.9> 18 6 19 VCL_trace <7 8.34> 18 6 19 VCL_trace <9 11.5> These tell us which bits of vcl code were executed, the first number is an index into a table the second is line.char 19 6 19 VCL_return vcl_recv returns "lookup", so we do... 17 4 19 VCL_call And we didn't find anything so we call vcl_miss() 18 8 19 VCL_trace <14 21.14> 19 5 19 VCL_return Which tells us to fetch it. 07 35 20 BackendOpen <192.168.64.2 53931 193.69.165.89 80> 08 10 20 BackendXID <1243897116> 0d 10 19 Backend <20 default> Session 19 opened a connection to the default backend and got fd=20. We log both ends of the TCP connection and the XID with the backends fd. Notice that these come out of order because we don't know the "20" until we have established the connection. 15 15 20 BldHdr 15 30 20 BldHdr We build the headers to send to the backend (this is currently incompletly logged) 13 8 20 Protocol 11 3 20 Status <200> 0f 2 20 Response 14 35 20 Header 14 28 20 Header 14 26 20 Header 14 38 20 Header 14 44 20 Header 14 29 20 Header 14 20 20 Header 14 22 20 Header 14 17 20 Header 14 43 20 Header And we get a reply. Notice that we have not picked up the body of the object yet. 01 114 0 Debug [TTD: max-age 900 Age: 0 Date: 1152544040 (0) Expires 1152544940 (900) our_clock 1152544040 -> ttd 1152544940 (900)] Our TTL calculations 17 5 19 VCL_call 18 8 19 VCL_trace <15 25.15> 18 7 19 VCL_trace <16 26.9> 18 7 19 VCL_trace <18 29.5> 18 7 19 VCL_trace <19 29.9> 18 7 19 VCL_trace <21 32.5> 19 6 19 VCL_return We call vcl_fetch to tell us what to do, (the object might not be cacheable etc etc) we are told to insert the object which implies delivery. 15 35 19 BldHdr 15 28 19 BldHdr 15 38 19 BldHdr 15 44 19 BldHdr 15 29 19 BldHdr 15 43 19 BldHdr We build the headers and send the object to the client. (incompletely logged presently) 03 4 0 CLI 11 3 19 Status <200> 10 6 19 Length <156934> Status and length send back. The body is received, stored and delivered here, but that generates no log messages if all goes well. 0a 0 20 BackendClose The backend connection is closed (because of the header from the backend) 06 17 19 SessionClose We close the clients TCP also, because it asked us to. 04 15 19 SessionOpen <127.0.0.1 60180> New client connection 1a 10 19 XID <1243897117> 0e 3 19 Request 12 1 19 URL 13 8 19 Protocol 14 15 19 Header 14 30 19 Header 14 17 19 Header 17 4 19 VCL_call 18 6 19 VCL_trace <1 4.14> 18 5 19 VCL_trace <2 5.9> 18 5 19 VCL_trace <5 8.5> 18 5 19 VCL_trace <6 8.9> 18 6 19 VCL_trace <7 8.34> 18 6 19 VCL_trace <9 11.5> 19 6 19 VCL_return 1b 10 19 Hit <1243897116> This time we got a hit, and the XID of the matched object (== of the session that feched it) is recorded. 17 3 19 VCL_call 18 8 19 VCL_trace <10 14.13> 18 7 19 VCL_trace <11 15.9> 18 7 19 VCL_trace <13 18.5> 19 7 19 VCL_return vcl_hit() tells us to deliver 11 3 19 Status <200> 10 6 19 Length <156934> 06 17 19 SessionClose So we do, and close. 03 4 0 CLI 03 4 0 CLI 03 4 0 CLI 03 4 0 CLI >PS: Performance counters? :-) Performance counters are also stored in the shared memory segment and can be read by any number of processes. We're no where near done defining all these, but these are the current ones: Hitrate ratio: 10 19 19 Hitrate avg: 0.5635 0.5566 0.5566 2 0.00 Client connections accepted 2 0.00 Client requests received 1 0.00 Cache hits 1 0.00 Cache misses 1 0.00 Backend connections initiated 0 0.00 Backend connections recyles 0 0.00 N struct sess 2 0.00 N struct object 2 0.00 N struct objecthead 1 0.00 N struct header 2 0.00 N struct smf 0 0.00 N struct vbe 0 0.00 N struct vbe_conn 1 0.00 N worker threads 0 0.00 N worker threads created 0 0.00 N worker threads not created 0 0.00 N worker threads shortages 0 0.00 N busy worker threads Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 10 15:37:11 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 15:37:11 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 17:03:08 +0200." Message-ID: <75817.1152545831@critter.freebsd.dk> In message , Anders Berg writes : >while I might not be the right one to answer I'll give it a shot. It's pretty spot on. For the centralized logging, it may be necessary to do a coarse filtering on the edge-node in order to conserve bandwidth, I suspect that things like the VCL execution the built headers, possibly also the received headers may not really be of any interested in the general case. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 10 15:38:28 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 15:38:28 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 17:24:09 +0200." Message-ID: <75825.1152545908@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >> I am uncertain if we are gonnna implement a way to tell the Varnish >> daemon itself what we want it to logg in V 1.0. > >I don't see any point in doing so. Well, if we end up writing too much "uninteresting junk" to the shmlog it might make sense to add a way to suppress it, but my preference is to not do so unless it becomes a problem. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Mon Jul 10 15:37:45 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 10 Jul 2006 17:37:45 +0200 Subject: Logging References: <75760.1152545627@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Performance counters are also stored in the shared memory segment > and can be read by any number of processes. We're no where near > done defining all these, but these are the current ones: > [...] has some rather more impressive numbers :) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From Anders.Berg at vg.no Mon Jul 10 15:47:46 2006 From: Anders.Berg at vg.no (Anders Berg) Date: Mon, 10 Jul 2006 17:47:46 +0200 Subject: SV: Logging References: <20060710082955.GA30842@totem.fix.no> Message-ID: <6AD33D89A21F7B479107D712EF96FDA528CBA0@VG-EXC-VIR-1.Akersgt.local> The only reason would be to hold network traffic down, but we are probably talking a hell of a lot of traffic before we staurate any kind of modern line. But if the traffis is on the same network where Varnish delivers this could be a problem for some. We could also assume a dedicated network for admin and logging, but that might not always be the case/feasable, _then_ it has a point. VG Netts blades have 2 NIC's on each blade when they are half height. Take into account that we are aiming for network redundancy on our rigg, we might have to hack some to get a failover solution working correctly. So there is a point to it, in the future, but maybe not V 1.0. Anders Berg ________________________________ Fra: varnish-dev-bounces at projects.linpro.no p? vegne av Dag-Erling Sm?rgrav Sendt: ma 10.07.2006 17:24 Til: varnish-dev at projects.linpro.no Emne: Re: Logging Anders Berg writes: > while I might not be the right one to answer I'll give it a shot. > Right now Varnish is logging alot of data involved in the request > handling to shared memory on each of the Varnish servers. It's a > shared memory buffer at a fixed size (default 8M right now I think), > and Varnish writes to it in a FIFO manner. It's a circular buffer, actually; varnish doesn't care if anyone reads it. > I am uncertain if we are gonnna implement a way to tell the Varnish > daemon itself what we want it to logg in V 1.0. I don't see any point in doing so. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no _______________________________________________ varnish-dev mailing list varnish-dev at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev ***************************************************************** Denne fotnoten bekrefter at denne e-postmeldingen ble skannet av MailSweeper og funnet fri for virus. ***************************************************************** This footnote confirms that this email message has been swept by MailSweeper for the presence of computer viruses. ***************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at linpro.no Mon Jul 10 16:01:09 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 10 Jul 2006 18:01:09 +0200 Subject: SV: Logging References: <20060710082955.GA30842@totem.fix.no> <6AD33D89A21F7B479107D712EF96FDA528CBA0@VG-EXC-VIR-1.Akersgt.local> Message-ID: "Anders Berg" writes: > The only reason would be to hold network traffic down, but we are > probably talking a hell of a lot of traffic before we staurate any > kind of modern line. Varnishd doesn't care; whatever application is used to send logs over the network will be responsible for selecting which entries to collect. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From Anders.Berg at vg.no Mon Jul 10 17:40:38 2006 From: Anders.Berg at vg.no (Anders Berg) Date: Mon, 10 Jul 2006 19:40:38 +0200 Subject: SV: SV: Logging References: <20060710082955.GA30842@totem.fix.no><6AD33D89A21F7B479107D712EF96FDA528CBA0@VG-EXC-VIR-1.Akersgt.local> Message-ID: <6AD33D89A21F7B479107D712EF96FDA528CBA1@VG-EXC-VIR-1.Akersgt.local> That would work :) Anders Berg ________________________________ Fra: varnish-dev-bounces at projects.linpro.no p? vegne av Dag-Erling Sm?rgrav Sendt: ma 10.07.2006 18:01 Til: varnish-dev at projects.linpro.no Emne: Re: SV: Logging "Anders Berg" writes: > The only reason would be to hold network traffic down, but we are > probably talking a hell of a lot of traffic before we staurate any > kind of modern line. Varnishd doesn't care; whatever application is used to send logs over the network will be responsible for selecting which entries to collect. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no _______________________________________________ varnish-dev mailing list varnish-dev at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev ***************************************************************** Denne fotnoten bekrefter at denne e-postmeldingen ble skannet av MailSweeper og funnet fri for virus. ***************************************************************** This footnote confirms that this email message has been swept by MailSweeper for the presence of computer viruses. ***************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Jul 10 18:49:57 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 18:49:57 +0000 Subject: NUUG meeting (was: Future plans) In-Reply-To: Your message of "Mon, 10 Jul 2006 10:33:45 +0200." <20060710083345.GB30842@totem.fix.no> Message-ID: <76575.1152557397@critter.freebsd.dk> In message <20060710083345.GB30842 at totem.fix.no>, Anders Nordby writes: >On Sat, Jul 08, 2006 at 06:58:31PM +0000, Poul-Henning Kamp wrote: >> The other thing to decide is when to put the August meeting ? >> >> The kids have summer vacation until Aug 15, so until then I have >> relatively large freedom, after that I'd prefer it to be thursday >> (and friday) >> >> The NUUG meeting on August 15 is particularly inconvenient because >> it is the exact day the kids start school. > >If august 15 is not possible, please suggest one or (preferably two) >other dates in august. Then I can check if it is OK with them. So now that my authoritative calender came home, I have resynchronized and it looks like it fits our calender in the aug7-aug11 week. How many days do we need anyway ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Mon Jul 10 18:53:52 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 10 Jul 2006 20:53:52 +0200 (CEST) Subject: Logging In-Reply-To: <75817.1152545831@critter.freebsd.dk> References: Your message of "Mon, 10 Jul 2006 17:03:08 +0200." <75817.1152545831@critter.freebsd.dk> Message-ID: <1398.193.213.34.102.1152557632.squirrel@denise.vg.no> > In message , Anders Berg > writes > : > >>while I might not be the right one to answer I'll give it a shot. > > It's pretty spot on. Good :) >... possibly > also the received headers may not really be of any interested > in the general case. Actually that one could be useful in production. Lets say you get swamped by a bot and wanna stop it, User-Agent would help alot in a case like that. Anders Berg > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > From phk at phk.freebsd.dk Mon Jul 10 19:09:47 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 19:09:47 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 20:53:52 +0200." <1398.193.213.34.102.1152557632.squirrel@denise.vg.no> Message-ID: <76749.1152558587@critter.freebsd.dk> In message <1398.193.213.34.102.1152557632.squirrel at denise.vg.no>, "Anders Berg " writes: >>... possibly >> also the received headers may not really be of any interested >> in the general case. > >Actually that one could be useful in production. Lets say you get swamped >by a bot and wanna stop it, User-Agent would help alot in a case like >that. That's not the kind of scenario I meant with "the general case" :-) For situations like that, I expect us to have a log-tailer that gives a top(1) like display of whatever we have selected. I should add here I envision a generic set of arguments to logtailers which the libvarnishapi should handle for programs: -i tag[,tag...] -- include these tags -x tag[,tag...] -- exclude these tags If no -i option is given a '-i all' is implict. -I regexp -- only data matching regexp -X regexp -- exclude data matching regexp -C match regexps case-insensitive So for this case your you present, the command would be: varnish-top -i header -C -I user-agent -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 10 19:14:53 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 19:14:53 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 19:09:47 GMT." <76749.1152558587@critter.freebsd.dk> Message-ID: <76814.1152558893@critter.freebsd.dk> In message <76749.1152558587 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >In message <1398.193.213.34.102.1152557632.squirrel at denise.vg.no>, "Anders Berg >" writes: > >I should add here I envision a generic set of arguments to logtailers >which the libvarnishapi should handle for programs: > > -i tag[,tag...] -- include these tags > -x tag[,tag...] -- exclude these tags > > If no -i option is given a '-i all' is implict. > > -I regexp -- only data matching regexp > -X regexp -- exclude data matching regexp > > -C match regexps case-insensitive I just realized that having these options would speed up my analyses of the logs from the live test, so I'll implement them right away... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Mon Jul 10 20:24:12 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 10 Jul 2006 22:24:12 +0200 (CEST) Subject: Logging In-Reply-To: <76814.1152558893@critter.freebsd.dk> References: Your message of "Mon, 10 Jul 2006 19:09:47 GMT." <76749.1152558587@critter.freebsd.dk> <76814.1152558893@critter.freebsd.dk> Message-ID: <1488.193.213.34.102.1152563052.squirrel@denise.vg.no> > In message <76749.1152558587 at critter.freebsd.dk>, "Poul-Henning Kamp" > writes: >>In message <1398.193.213.34.102.1152557632.squirrel at denise.vg.no>, >> "Anders Berg >>" writes: >> >>I should add here I envision a generic set of arguments to logtailers >>which the libvarnishapi should handle for programs: >> >> -i tag[,tag...] -- include these tags >> -x tag[,tag...] -- exclude these tags >> >> If no -i option is given a '-i all' is implict. >> >> -I regexp -- only data matching regexp >> -X regexp -- exclude data matching regexp >> >> -C match regexps case-insensitive > > I just realized that having these options would speed up my > analyses of the logs from the live test, so I'll implement > them right away... Looks good and sounds good. Also it concludes the answer to Anders N. :) Anders Berg > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > From phk at phk.freebsd.dk Mon Jul 10 20:51:08 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Jul 2006 20:51:08 +0000 Subject: Logging In-Reply-To: Your message of "Mon, 10 Jul 2006 19:14:53 GMT." <76814.1152558893@critter.freebsd.dk> Message-ID: <797.1152564668@critter.freebsd.dk> In message <76814.1152558893 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >> -i tag[,tag...] -- include these tags >> -x tag[,tag...] -- exclude these tags >> >> If no -i option is given a '-i all' is implict. >> >> -I regexp -- only data matching regexp >> -X regexp -- exclude data matching regexp >> >> -C match regexps case-insensitive > >I just realized that having these options would speed up my >analyses of the logs from the live test, so I'll implement >them right away... Done. Now things like: zcat /critter/_vlog21.gz | ./varnishlog -r - -i url -I % works, and it's a bit faster than grep'ing on my laptop. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Mon Jul 10 21:02:56 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 10 Jul 2006 23:02:56 +0200 (CEST) Subject: Website material In-Reply-To: <75817.1152545831@critter.freebsd.dk> References: Your message of "Mon, 10 Jul 2006 17:03:08 +0200." <75817.1152545831@critter.freebsd.dk> Message-ID: <1721.193.213.34.102.1152565376.squirrel@denise.vg.no> Hi, I have worked some on the website material: http://klikk.vg.no/varnish/ the welcome page and featurelist page are the only ones with any content so far. I am aware that the pages don't show right in Firefox and Safari, but I asure you they work great in IE :) The designer is gonna come up with some code that works better soon. I have more content available and will add more this week if I have time. Anders Berg From andersb at vgnett.no Mon Jul 10 21:06:05 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 10 Jul 2006 23:06:05 +0200 (CEST) Subject: Website material (New thread) Message-ID: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> Sorry, I used a reply and the last mail ended up in the Logging thread. Here it is again in it's own thread: Hi, I have worked some on the website material: http://klikk.vg.no/varnish/ the welcome page and featurelist page are the only ones with any content so far. I am aware that the pages don't show right in Firefox and Safari, but I asure you they work great in IE :) The designer is gonna come up with some code that works better soon. I have more content available and will add more this week if I have time. Anders Berg From des at linpro.no Tue Jul 11 06:33:31 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 11 Jul 2006 08:33:31 +0200 Subject: NUUG meeting References: <76575.1152557397@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > So now that my authoritative calender came home, I have resynchronized > and it looks like it fits our calender in the aug7-aug11 week. Isn't that a bit early? Will we be ready by then? (I assume this isn't just about NUUG, but about production deployment as well) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Jul 11 07:02:50 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 07:02:50 +0000 Subject: NUUG meeting In-Reply-To: Your message of "Tue, 11 Jul 2006 08:33:31 +0200." Message-ID: <26670.1152601370@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> So now that my authoritative calender came home, I have resynchronized >> and it looks like it fits our calender in the aug7-aug11 week. > >Isn't that a bit early? Will we be ready by then? > >(I assume this isn't just about NUUG, but about production deployment >as well) Yes, that was the idea. It's pretty much Anders' call when he feels ready for the production start. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anders at fupp.net Tue Jul 11 08:50:41 2006 From: anders at fupp.net (Anders Nordby) Date: Tue, 11 Jul 2006 10:50:41 +0200 Subject: NUUG meeting (was: Future plans) In-Reply-To: <76575.1152557397@critter.freebsd.dk> References: <20060710083345.GB30842@totem.fix.no> <76575.1152557397@critter.freebsd.dk> Message-ID: <20060711085041.GA30288@totem.fix.no> Hi, On Mon, Jul 10, 2006 at 06:49:57PM +0000, Poul-Henning Kamp wrote: >>> The NUUG meeting on August 15 is particularly inconvenient because >>> it is the exact day the kids start school. >> >>If august 15 is not possible, please suggest one or (preferably two) >>other dates in august. Then I can check if it is OK with them. > So now that my authoritative calender came home, I have resynchronized > and it looks like it fits our calender in the aug7-aug11 week. > > How many days do we need anyway ? Just one day. Usually you would have around 60 minutes. If we can suggest one day, and another day as backup, there's a bigger probability we have a match so that we can actually get the meeting set up. Info for speakers: http://nuug.no/meetings/speakerinfo.shtml Cheers, -- Anders. From andersb at vgnett.no Tue Jul 11 10:10:26 2006 From: andersb at vgnett.no (Anders Berg) Date: Tue, 11 Jul 2006 12:10:26 +0200 (CEST) Subject: NUUG meeting In-Reply-To: <26670.1152601370@critter.freebsd.dk> References: Your message of "Tue, 11 Jul 2006 08:33:31 +0200." <26670.1152601370@critter.freebsd.dk> Message-ID: <1428.193.213.34.102.1152612626.squirrel@denise.vg.no> > In message , Dag-Erling > =?iso-8859-1?Q?Sm=F8rgra > v?= writes: >>"Poul-Henning Kamp" writes: >>> So now that my authoritative calender came home, I have resynchronized >>> and it looks like it fits our calender in the aug7-aug11 week. >> >>Isn't that a bit early? Will we be ready by then? >> >>(I assume this isn't just about NUUG, but about production deployment >>as well) > > Yes, that was the idea. > > It's pretty much Anders' call when he feels ready for the production > start. Hi, today we are scheduled as following: 7.-11. aug. Live test 2 14.-18. aug. Delivery 18. aug. Goal We have also scheduled some summer vacation in the last weeks of July. I think we for the time beeing should agree on: 7.-11. aug. Live test 2 (Goal: Have a version we can run live in VG Nett's environment in a stable manner, possibly so good that we decide to run 24/7 on one server. Varnish would at this time have not have a fully implemented VCL, but the support tools should be starting to look like what we ship with. Could possibly be our alpha) if we plan to stay on budget with Poul-Henning delivery on 18. aug is too soon. I guess it also boils down to the featurelist that we have to agree on soon. I feel that the 2 other dates can not be predicted accuratly right now with the info I/we have. Could you try to estimate some timeschedule Poul-Henning? Also Dag-Erling a timeschedule of the work that Linpro needs to do, would be needed soon. Anders Berg > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From phk at phk.freebsd.dk Tue Jul 11 11:12:39 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 11:12:39 +0000 Subject: NUUG meeting In-Reply-To: Your message of "Tue, 11 Jul 2006 12:10:26 +0200." <1428.193.213.34.102.1152612626.squirrel@denise.vg.no> Message-ID: <2334.1152616359@critter.freebsd.dk> In message <1428.193.213.34.102.1152612626.squirrel at denise.vg.no>, "Anders Berg " writes: >today we are scheduled as following: > >7.-11. aug. Live test 2 Wasn't this when you were getting married ? >14.-18. aug. Delivery >18. aug. Goal I suggest we nail down the dates for the live test 2 (and NUUG talk) and set the delivery/goal when we have more firm estimates. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Tue Jul 11 11:37:47 2006 From: andersb at vgnett.no (Anders Berg) Date: Tue, 11 Jul 2006 13:37:47 +0200 (CEST) Subject: NUUG meeting In-Reply-To: <2334.1152616359@critter.freebsd.dk> References: Your message of "Tue, 11 Jul 2006 12:10:26 +0200." <1428.193.213.34.102.1152612626.squirrel@denise.vg.no> <2334.1152616359@critter.freebsd.dk> Message-ID: <1835.193.213.34.102.1152617867.squirrel@denise.vg.no> > In message <1428.193.213.34.102.1152612626.squirrel at denise.vg.no>, "Anders > Berg > " writes: > >>today we are scheduled as following: >> >>7.-11. aug. Live test 2 > > Wasn't this when you were getting married ? Deliverydate for Wife 1.0 is 26. aug. :) >>14.-18. aug. Delivery >>18. aug. Goal > > > I suggest we nail down the dates for the live test 2 (and NUUG talk) > and set the delivery/goal when we have more firm estimates. Agree. Anders Berg > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > From phk at phk.freebsd.dk Tue Jul 11 11:41:14 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 11:41:14 +0000 Subject: NUUG meeting In-Reply-To: Your message of "Tue, 11 Jul 2006 13:37:47 +0200." <1835.193.213.34.102.1152617867.squirrel@denise.vg.no> Message-ID: <17266.1152618074@critter.freebsd.dk> In message <1835.193.213.34.102.1152617867.squirrel at denise.vg.no>, "Anders Berg " writes: >> In message <1428.193.213.34.102.1152612626.squirrel at denise.vg.no>, "Anders >>>today we are scheduled as following: >>> >>>7.-11. aug. Live test 2 >> >> Wasn't this when you were getting married ? > >Deliverydate for Wife 1.0 is 26. aug. :) That's a good argument for wrapping things up in August if we can. >> I suggest we nail down the dates for the live test 2 (and NUUG talk) >> and set the delivery/goal when we have more firm estimates. So is August 7-11 firm for the live-test 2 ? Do we need a full week ? I guess if we get too much time, DES and I can work on docs etc. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Tue Jul 11 14:25:52 2006 From: andersb at vgnett.no (Anders Berg) Date: Tue, 11 Jul 2006 16:25:52 +0200 Subject: NUUG meeting In-Reply-To: <17266.1152618074@critter.freebsd.dk> References: <17266.1152618074@critter.freebsd.dk> Message-ID: <7A1A41F4-8F70-4CF6-9DDF-D8B26E32402D@vgnett.no> On Jul 11, 2006, at 13:41, Poul-Henning Kamp wrote: > In message <1835.193.213.34.102.1152617867.squirrel at denise.vg.no>, > "Anders Berg > " writes: >>> In message >>> <1428.193.213.34.102.1152612626.squirrel at denise.vg.no>, "Anders > >>>> today we are scheduled as following: >>>> >>>> 7.-11. aug. Live test 2 >>> >>> Wasn't this when you were getting married ? >> >> Deliverydate for Wife 1.0 is 26. aug. :) > > That's a good argument for wrapping things up in August if we can. For me sept. is no problem, and actually better. I will be gone the 9.-17. sept. But apart from that sept. is fine. In aug. we are receiving a SAN system also, so the week 14.-18. aug. is "installweek". The week after I will be busy under Gry's whip :) >>> I suggest we nail down the dates for the live test 2 (and NUUG talk) >>> and set the delivery/goal when we have more firm estimates. > > So is August 7-11 firm for the live-test 2 ? > > Do we need a full week ? We could probably wrap it up in 3 days, but I think we utilized a full week last time, maybe we need it again. That said, it worked out good with me at home the last 2 days, so we could run the 2 first days with you in Slagelse and on IRC. > I guess if we get too much time, DES and I can work on docs etc. That reminds me to ask for the presentation you held at Linpro. I could make some fancy Visio diagrams :) Anders Berg > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > From des at linpro.no Tue Jul 11 14:40:46 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 11 Jul 2006 16:40:46 +0200 Subject: NUUG meeting References: <17266.1152618074@critter.freebsd.dk> <7A1A41F4-8F70-4CF6-9DDF-D8B26E32402D@vgnett.no> Message-ID: Anders Berg writes: > That reminds me to ask for the presentation you held at Linpro. I > could make some fancy Visio diagrams :) http://users.linpro.no/~des/varnish-2006-07-06.pdf DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Jul 11 15:03:54 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 15:03:54 +0000 Subject: NUUG meeting In-Reply-To: Your message of "Tue, 11 Jul 2006 16:25:52 +0200." <7A1A41F4-8F70-4CF6-9DDF-D8B26E32402D@vgnett.no> Message-ID: <29673.1152630234@critter.freebsd.dk> In message <7A1A41F4-8F70-4CF6-9DDF-D8B26E32402D at vgnett.no>, Anders Berg writes : >That reminds me to ask for the presentation you held at Linpro. I >could make some fancy Visio diagrams :) DES got both the openoffice and pdf files. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Jul 11 17:16:41 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 17:16:41 +0000 Subject: NUUG meeting (was: Future plans) In-Reply-To: Your message of "Tue, 11 Jul 2006 10:50:41 +0200." <20060711085041.GA30288@totem.fix.no> Message-ID: <48642.1152638201@critter.freebsd.dk> In message <20060711085041.GA30288 at totem.fix.no>, Anders Nordby writes: >Hi, > >On Mon, Jul 10, 2006 at 06:49:57PM +0000, Poul-Henning Kamp wrote: >>>> The NUUG meeting on August 15 is particularly inconvenient because >>>> it is the exact day the kids start school. >>> >>>If august 15 is not possible, please suggest one or (preferably two) >>>other dates in august. Then I can check if it is OK with them. >> So now that my authoritative calender came home, I have resynchronized >> and it looks like it fits our calender in the aug7-aug11 week. >> >> How many days do we need anyway ? > >Just one day. Usually you would have around 60 minutes. Sorry, got the wires crossed here, I was asking how much time we needed for the Varnish live test :-) Stay tuned for a date when a talk is possible. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Tue Jul 11 20:26:01 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 11 Jul 2006 22:26:01 +0200 Subject: Via Message-ID: Varnish currently sends out a Via: header that looks like this: Via: 1.1 varnish I'd like to change it to the following, which is more in line with RFC2616: Via: 1.1 (Varnish ) Strictly speaking, we should merge this with the backend's Via: header (if any) so that if the backend sends Via: 1.0 (Apache/3.14) we should send Via: 1.1 (Varnish ), 1.0 (Apache/3.14) (we might consider moving this into VCL, BTW) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Jul 11 20:36:50 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 20:36:50 +0000 Subject: Via In-Reply-To: Your message of "Tue, 11 Jul 2006 22:26:01 +0200." Message-ID: <78022.1152650210@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: Yes, this is on my todo list. We also need to send a Via line to the server Right now I'm writing a "varnishtester" so we can script test cases efficiently. Once I have that working enough, I'll start to flesh out all the bits in the state diagram and the header handling. Poul-Henning >Varnish currently sends out a Via: header that looks like this: > >Via: 1.1 varnish > >I'd like to change it to the following, which is more in line with >RFC2616: > >Via: 1.1 (Varnish ) > >Strictly speaking, we should merge this with the backend's Via: header >(if any) so that if the backend sends > >Via: 1.0 (Apache/3.14) > >we should send > >Via: 1.1 (Varnish ), 1.0 (Apache/3.14) > >(we might consider moving this into VCL, BTW) > >DES >-- >Dag-Erling Sm?rgrav >Senior Software Developer >Linpro AS - www.linpro.no >_______________________________________________ >varnish-dev mailing list >varnish-dev at projects.linpro.no >http://projects.linpro.no/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Jul 11 21:48:00 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Jul 2006 21:48:00 +0000 Subject: varnishtester is minimally functional Message-ID: <88219.1152654480@critter.freebsd.dk> The varnishtester is minimally functional now, and understands the following commands: start start varnishd serve string [...] Send these strings when port 8081 is contacted and we see a blank line. If multiple strings specified, all but the last one are sent only once, the last (or only) one is sent as many times as requested. If first char of string is '!' close connection afterwards. open Open request connection to port 8080 close Close request connection to port 8080 req string Send string as request (open if not explicitly done already) if first char is '!' close connection afterwards. cli string Send string via varnishd CLI connection vcl name vcl_prog Load VCL program as named. exit exit test program. My minimal test-the-tester script looks like this: start serve "!HTTP/1.0 200 OK\nContent Length: 11\n\n0123456789\n" open req "!GET / HTTP/1.1\nHost: localhost\n\n" close cli ping exit And the output looks like this: ]: <> X: Pause V: <> V: <> V: <> V: <> V: <> V: <>> V: <>> V: <>> V: <>> X: Resume ]: <> ]: <> ]: <> X: Pause B: <> B: <> B: <<>> R: <> R: <> R: <> R: <> R: <> R: <> R: <<>> R: <<0123456789>> ex_req(0x82a0320, 0x11, 0x0) X: Resume ]: <> ]: <> X: Pause V: <> V: <> X: Resume ]: <> Prefix indicator: ]: read from script file X: execution information (pause/resume) V: received from Varnish stdout/stderr B: requests received by synth backend R: response received by syntg client TBD: For automated testing I plan to add a hash facility so that one can say things like: hash serve 283838383... hash req 229292929... hash cli 229292929... And it will croak if what has been received until then on the relevant connection does not generate the expected hash. Alternatively one can compare to a string: compare serve "GET / HTTP/1.1\nHost: localhost\n\n" compare req "HTTP/1.0 200 OK\nContent-Length: 11..." etc. Hopefully this will be workable to generate test cases for all relevant flows in the state-machine. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Jul 12 16:37:28 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 12 Jul 2006 18:37:28 +0200 Subject: tot dumps core Message-ID: I wanted to test the virtual host support, but the current top-of-tree just crashes when the first request comes in: > des at dma ~varnish/varnish-cache% cat varnish.sh > #!/bin/sh > > base=$(dirname $(realpath $0)) > > exec $base/bin/varnishd/varnishd \ > -b localhost:80 \ > -p 8080 \ > -s file,${base}/storage \ > -w 1 > des at dma ~varnish/varnish-cache% ./varnish.sh > file /home/des/projects/varnish/trunk/varnish-cache/storage size 8388608 bytes (16384 fs-blocks, 2048 pages) > start child pid 67665 > Child said > Child said > Child said > Child said > sig_chld(20, 8, 0) > pid = 67665 status = 0x8b > Child died :-( > des at dma ~varnish/varnish-cache% ./gvarnish.sh > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `lt-varnishd'. > Program terminated with signal 11, Segmentation fault. > Error while mapping shared library sections: > /tmp/vcl.Wjvq6wyc: No such file or directory. > Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0...done. > Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0 > Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0...done. > Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0 > Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1...done. > Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1 > Reading symbols from /lib/libpthread.so.2...done. > Loaded symbols for /lib/libpthread.so.2 > Reading symbols from /lib/libmd.so.3...done. > Loaded symbols for /lib/libmd.so.3 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Error while reading shared library symbols: > /tmp/vcl.Wjvq6wyc: No such file or directory. > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x0000000800989e9c in kse_thr_interrupt () at kse_thr_interrupt.S:2 > 2 RSYSCALL(kse_thr_interrupt) > [New Thread 0x801051c00 (sleeping)] > [New Thread 0x801051800 (sleeping)] > [New Thread 0x801051400 (runnable)] > [New Thread 0x801051000 (runnable)] > [New Thread 0x801050c00 (runnable)] > [New Thread 0x801050800 (LWP 100088)] > [New Thread 0x801050400 (runnable)] > [New LWP 100092] > Current language: auto; currently asm > (gdb) i thr > * 8 LWP 100092 0x0000000800989e9c in kse_thr_interrupt () > at kse_thr_interrupt.S:2 > 7 Thread 0x801050400 (runnable) 0x0000000800c8628c in __syscall () > at __syscall.S:2 > 6 Thread 0x801050800 (LWP 100088) 0x0000000800989e5c in kse_release () > at kse_release.S:2 > 5 Thread 0x801050c00 (runnable) 0x0000000800c6b86c in kevent () > at kevent.S:2 > 4 Thread 0x801051000 (runnable) hcl_lookup (key1=0x801132385 "/", > key2=0x801132397 "www.des.no", nobj=0x800be6c70) at hash_classic.c:151 > 3 Thread 0x801051400 (runnable) 0x0000000800c6b86c in kevent () > at kevent.S:2 > 2 Thread 0x801051800 (sleeping) _thr_sched_switch_unlocked ( > curthread=0x801051800) at pthread_md.h:226 > 1 Thread 0x801051c00 (sleeping) _thr_sched_switch_unlocked ( > curthread=0x801051c00) at pthread_md.h:226 > (gdb) thr 4 > [Switching to thread 4 (Thread 0x801051000 (runnable))]#0 hcl_lookup ( > key1=0x801132385 "/", key2=0x801132397 "www.des.no", nobj=0x800be6c70) > at hash_classic.c:151 > 151 nobj->hashpriv = he2; > Current language: auto; currently c > (gdb) where > #0 hcl_lookup (key1=0x801132385 "/", key2=0x801132397 "www.des.no", > nobj=0x800be6c70) at hash_classic.c:151 > #1 0x0000000000407972 in HSH_Lookup (w=0x7fffff7fcf30, h=0x801132178) > at cache_hash.c:71 > #2 0x000000000040614d in cnt_lookup (w=0x80106426b, sp=0x801132000) > at cache_center.c:294 > #3 0x0000000000406607 in CNT_Session (w=0x7fffff7fcf30, sp=0x801132000) > at steps.h:7 > #4 0x000000000040980c in wrk_thread (priv=0x7fffffffe6f4) at cache_pool.c:66 > #5 0x000000080097a6d9 in thread_start (curthread=0x80106426b, > start_routine=0x8011323a2, arg=0x80106426b) > at /usr/src/lib/libpthread/thread/thr_create.c:344 > #6 0x0000000800be6c94 in makectx_wrapper (ucp=0x800548870, func=0xb031c7f9, > args=0x801064260) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 > #7 0x0000000000000000 in ?? () > #8 0x0000000801051000 in ?? () > #9 0x00000000004096b0 in child_main () > #10 0x00007fffffffe6f4 in ?? () > #11 0x0000000000000000 in ?? () > #12 0x0000000000000000 in ?? () > #13 0x0000000000000000 in ?? () > Cannot access memory at address 0x7fffff7fd000 > (gdb) up 0 > #0 hcl_lookup (key1=0x801132385 "/", key2=0x801132397 "www.des.no", > nobj=0x800be6c70) at hash_classic.c:151 > 151 nobj->hashpriv = he2; > (gdb) l > 146 he2->mtx = u2; > 147 he2->key1 = strdup(key1); > 148 assert(he2->key1 != NULL); > 149 he2->key2 = strdup(key2); > 150 assert(he2->key2 != NULL); > 151 nobj->hashpriv = he2; > 152 if (he != NULL) > 153 TAILQ_INSERT_BEFORE(he, he2, list); > 154 else > 155 TAILQ_INSERT_TAIL(&hcl_head[u1], he2, list); > (gdb) p nobj > $1 = (struct objhead *) 0x800be6c70 > (gdb) p *nobj > $2 = {hashpriv = 0x48f28949fb894853, mtx = 0x8b48184a8b48d089, objects = { > tqh_first = 0x8b4c08708b481052, tqh_last = 0x8b4820408b4c2848}} The request I sent was: > HEAD / HTTP/1.1 > Host: www.des.no > Connection: close > Here's the complete log from start to crash: > 01 25 0 Debug [Starting 1 worker threads] > 20 16 0 WorkThread <1 born permanent> > 04 15 21 SessionOpen <10.0.0.12 65110> > 1a 9 21 XID <176853298> > 01 10 21 Debug [State RECV] > 0e 4 21 Request > 12 1 21 URL > 13 8 21 Protocol > 14 16 21 Header > 14 17 21 Header > 17 4 21 VCL_call > 18 5 21 VCL_trace <1 1.1> > 18 5 21 VCL_trace <2 1.1> > 18 5 21 VCL_trace <3 1.1> > 18 5 21 VCL_trace <5 1.1> > 18 5 21 VCL_trace <6 1.1> > 18 5 21 VCL_trace <8 1.1> > 18 5 21 VCL_trace <9 1.1> > 18 6 21 VCL_trace <10 1.1> > 18 6 21 VCL_trace <12 1.1> > 19 6 21 VCL_return > 01 12 21 Debug [State LOOKUP] DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Jul 12 17:04:03 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 12 Jul 2006 17:04:03 +0000 Subject: tot dumps core In-Reply-To: Your message of "Wed, 12 Jul 2006 18:37:28 +0200." Message-ID: <27700.1152723843@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >I wanted to test the virtual host support, but the current top-of-tree >just crashes when the first request comes in: Brainfart on my part. I'll fix in a sec. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Jul 12 17:07:44 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 12 Jul 2006 17:07:44 +0000 Subject: tot dumps core In-Reply-To: Your message of "Wed, 12 Jul 2006 18:37:28 +0200." Message-ID: <28080.1152724064@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >I wanted to test the virtual host support, but the current top-of-tree >just crashes when the first request comes in: >> (gdb) i thr >> * 8 LWP 100092 0x0000000800989e9c in kse_thr_interrupt () >> at kse_thr_interrupt.S:2 >> 7 Thread 0x801050400 (runnable) 0x0000000800c8628c in __syscall () >> at __syscall.S:2 >> 6 Thread 0x801050800 (LWP 100088) 0x0000000800989e5c in kse_release () >> at kse_release.S:2 >> 5 Thread 0x801050c00 (runnable) 0x0000000800c6b86c in kevent () >> at kevent.S:2 >> 4 Thread 0x801051000 (runnable) hcl_lookup (key1=0x801132385 "/", >> key2=0x801132397 "www.des.no", nobj=0x800be6c70) at hash_classic.c:151 >> 3 Thread 0x801051400 (runnable) 0x0000000800c6b86c in kevent () >> at kevent.S:2 >> 2 Thread 0x801051800 (sleeping) _thr_sched_switch_unlocked ( >> curthread=0x801051800) at pthread_md.h:226 >> 1 Thread 0x801051c00 (sleeping) _thr_sched_switch_unlocked ( >> curthread=0x801051c00) at pthread_md.h:226 >> (gdb) thr 4 Actually, that's not the trouble I think, could you check what thread 8 is doing ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Jul 12 19:51:20 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 12 Jul 2006 21:51:20 +0200 Subject: tot dumps core In-Reply-To: <28080.1152724064@critter.freebsd.dk> (Poul-Henning Kamp's message of "Wed, 12 Jul 2006 17:07:44 +0000") References: <28080.1152724064@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Actually, that's not the trouble I think, could you check what > thread 8 is doing ? (gdb) where #0 0x0000000800989e5c in kse_release () at kse_release.S:2 #1 0x0000000800977326 in sig_daemon (arg=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_sig.c:214 #2 0x00000008009813ca in kse_sched_single (kmbx=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_kern.c:886 #3 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffffbff000 DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Jul 12 21:33:01 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 12 Jul 2006 21:33:01 +0000 Subject: tot dumps core In-Reply-To: Your message of "Wed, 12 Jul 2006 21:51:20 +0200." Message-ID: <43826.1152739981@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Actually, that's not the trouble I think, could you check what >> thread 8 is doing ? > >(gdb) where >#0 0x0000000800989e5c in kse_release () at kse_release.S:2 >#1 0x0000000800977326 in sig_daemon (arg=3D0x7fffffbfef70) > at /usr/src/lib/libpthread/thread/thr_sig.c:214 >#2 0x00000008009813ca in kse_sched_single (kmbx=3D0x7fffffbfef70) > at /usr/src/lib/libpthread/thread/thr_kern.c:886 >#3 0x0000000000000000 in ?? () >Cannot access memory at address 0x7fffffbff000 Hmm, I don't spot anything obvious and I can't reproduce it. Can you reproduce it ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Thu Jul 13 11:40:43 2006 From: andersb at vgnett.no (Anders Berg) Date: Thu, 13 Jul 2006 13:40:43 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> Message-ID: <1472.193.213.34.102.1152790843.squirrel@denise.vg.no> > Sorry, I used a reply and the last mail ended up in the Logging thread. > Here it is again in it's own thread: > > Hi, > > I have worked some on the website material: > > http://klikk.vg.no/varnish/ > > the welcome page and featurelist page are the only ones with any content > so far. > > I am aware that the pages don't show right in Firefox and Safari, but I > asure you they work great in IE :) The designer is gonna come up with some > code that works better soon. A new CSS file is uploaded, and should work for the more exotic browsers. Give it a click and let me know. Anders Berg > I have more content available and will add more this week if I have time. > > Anders Berg > > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Fri Jul 14 06:19:44 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 08:19:44 +0200 Subject: tot dumps core In-Reply-To: <43826.1152739981@critter.freebsd.dk> (Poul-Henning Kamp's message of "Wed, 12 Jul 2006 21:33:01 +0000") References: <43826.1152739981@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Hmm, I don't spot anything obvious and I can't reproduce it. > > Can you reproduce it ? Yes, 100%, unfortunately. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 07:01:11 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 07:01:11 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 08:19:44 +0200." Message-ID: <3329.1152860471@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Hmm, I don't spot anything obvious and I can't reproduce it. >> >> Can you reproduce it ? > >Yes, 100%, unfortunately. Please give me a backtrace of all the threads then, I must have overlooked something. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 07:27:50 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 09:27:50 +0200 Subject: tot dumps core References: <3329.1152860471@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Please give me a backtrace of all the threads then, I must have > overlooked something. des at dma ~varnish/varnish-cache% ./varnish.sh file /home/des/projects/varnish/trunk/varnish-cache/storage size 8388608 bytes (16384 fs-blocks, 2048 pages) start child pid 85466 Child said Child said Child said Child said sig_chld(20, 8, 0) pid = 85466 status = 0x8b Child died :-( des at dma ~varnish/varnish-cache% ./gvarnish.sh GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `lt-varnishd'. Program terminated with signal 11, Segmentation fault. Error while mapping shared library sections: /tmp/vcl.6xh4auSp: No such file or directory. Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0...done. Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0 Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0...done. Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0 Reading symbols from /home/des/projects/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1...done. Loaded symbols for /home/des/projects/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1 Reading symbols from /lib/libpthread.so.2...done. Loaded symbols for /lib/libpthread.so.2 Reading symbols from /lib/libmd.so.3...done. Loaded symbols for /lib/libmd.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Error while reading shared library symbols: /tmp/vcl.6xh4auSp: No such file or directory. Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0000000800989e9c in kse_thr_interrupt () at kse_thr_interrupt.S:2 2 RSYSCALL(kse_thr_interrupt) [New Thread 0x801051c00 (sleeping)] [New Thread 0x801051800 (sleeping)] [New Thread 0x801051400 (runnable)] [New Thread 0x801051000 (runnable)] [New Thread 0x801050c00 (runnable)] [New Thread 0x801050800 (LWP 100179)] [New Thread 0x801050400 (runnable)] [New LWP 100155] Current language: auto; currently asm (gdb) i thr * 8 LWP 100155 0x0000000800989e9c in kse_thr_interrupt () at kse_thr_interrupt.S:2 7 Thread 0x801050400 (runnable) 0x0000000800c6b86c in kevent () at kevent.S:2 6 Thread 0x801050800 (LWP 100179) 0x0000000800989e5c in kse_release () at kse_release.S:2 5 Thread 0x801050c00 (runnable) 0x0000000800c6b86c in kevent () at kevent.S:2 4 Thread 0x801051000 (runnable) hcl_lookup (key1=0x801132385 "/", key2=0x801132397 "www.des.no", nobj=0x800be6c70) at hash_classic.c:151 3 Thread 0x801051400 (runnable) 0x0000000800c6b86c in kevent () at kevent.S:2 2 Thread 0x801051800 (sleeping) _thr_sched_switch_unlocked ( curthread=0x801051800) at pthread_md.h:226 1 Thread 0x801051c00 (sleeping) _thr_sched_switch_unlocked ( curthread=0x801051c00) at pthread_md.h:226 (gdb) thr 1 [Switching to thread 1 (Thread 0x801051c00 (sleeping))]#0 _thr_sched_switch_unlocked (curthread=0x801051c00) at pthread_md.h:226 226 if (ret == 0) { Current language: auto; currently c (gdb) where #0 _thr_sched_switch_unlocked (curthread=0x801051c00) at pthread_md.h:226 #1 0x000000080097bdfc in _nanosleep (time_to_sleep=0x7fffff1f9f50, time_remaining=0x7fffff1f9f40) at /usr/src/lib/libpthread/thread/thr_nanosleep.c:77 #2 0x0000000800c364f0 in __sleep (seconds=1) at /usr/src/lib/libc/gen/sleep.c:63 #3 0x00000008009726d6 in _sleep (seconds=1) at /usr/src/lib/libpthread/thread/thr_sleep.c:54 #4 0x00000000004068fc in exp_hangman (arg=0x80054a080) at cache_expire.c:67 #5 0x000000080097a6d9 in thread_start (curthread=0x80054a080, start_routine=0x801020208, arg=0x80054a080) at /usr/src/lib/libpthread/thread/thr_create.c:344 #6 0x0000000800be6c94 in makectx_wrapper (ucp=0x80054a070, func=0x801051c60, args=0x1) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 #7 0x0000000000000000 in ?? () #8 0x0000000801051c00 in ?? () #9 0x0000000000406870 in EXP_TTLchange () at cache_expire.c:35 #10 0x0000000000000000 in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () #13 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff1fa000 (gdb) thr 2 [Switching to thread 2 (Thread 0x801051800 (sleeping))]#0 _thr_sched_switch_unlocked (curthread=0x801051800) at pthread_md.h:226 226 if (ret == 0) { (gdb) where #0 _thr_sched_switch_unlocked (curthread=0x801051800) at pthread_md.h:226 #1 0x000000080097bdfc in _nanosleep (time_to_sleep=0x7fffff3fae70, time_remaining=0x7fffff3fae60) at /usr/src/lib/libpthread/thread/thr_nanosleep.c:77 #2 0x0000000800c364f0 in __sleep (seconds=1) at /usr/src/lib/libc/gen/sleep.c:63 #3 0x00000008009726d6 in _sleep (seconds=1) at /usr/src/lib/libpthread/thread/thr_sleep.c:54 #4 0x0000000000406a59 in exp_prefetch (arg=0x800549880) at cache_expire.c:99 #5 0x000000080097a6d9 in thread_start (curthread=0x800549880, start_routine=0x801020208, arg=0x800549880) at /usr/src/lib/libpthread/thread/thr_create.c:344 #6 0x0000000800be6c94 in makectx_wrapper (ucp=0x800549870, func=0x801051860, args=0x1) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 #7 0x0000000000000000 in ?? () #8 0x0000000801051800 in ?? () #9 0x00000000004069f0 in exp_hangman () at cache_expire.c:71 #10 0x0000000000000000 in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () #13 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff3fb000 (gdb) thr 3 [Switching to thread 3 (Thread 0x801051400 (runnable))]#0 0x0000000800c6b86c in kevent () at kevent.S:2 2 RSYSCALL(kevent) Current language: auto; currently asm (gdb) where #0 0x0000000800c6b86c in kevent () at kevent.S:2 #1 0x0000000800862e74 in kq_dispatch (base=0x14, arg=0x801009700, tv=0x0) at kqueue.c:220 #2 0x000000080085dd4a in event_base_loop (base=0x801120340, flags=0) at event.c:413 #3 0x00000000004049eb in vca_main (arg=0x14) at cache_acceptor.c:285 #4 0x000000080097a6d9 in thread_start (curthread=0x14, start_routine=0x8010d5800, arg=0x14) at /usr/src/lib/libpthread/thread/thr_create.c:344 #5 0x0000000800be6c94 in makectx_wrapper (ucp=0x800549070, func=0xffffffff805753c0, args=0x1) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 #6 0x0000000000000000 in ?? () #7 0x0000000801051400 in ?? () #8 0x00000000004048b0 in accept_f () at cache_acceptor.c:221 #9 0x0000000000000000 in ?? () #10 0x0000000000000000 in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff5fc000 (gdb) thr 4 [Switching to thread 4 (Thread 0x801051000 (runnable))]#0 hcl_lookup ( key1=0x801132385 "/", key2=0x801132397 "www.des.no", nobj=0x800be6c70) at hash_classic.c:151 151 nobj->hashpriv = he2; Current language: auto; currently c (gdb) where #0 hcl_lookup (key1=0x801132385 "/", key2=0x801132397 "www.des.no", nobj=0x800be6c70) at hash_classic.c:151 #1 0x0000000000407972 in HSH_Lookup (w=0x7fffff7fcf30, h=0x801132178) at cache_hash.c:71 #2 0x000000000040614d in cnt_lookup (w=0x80106429b, sp=0x801132000) at cache_center.c:294 #3 0x0000000000406607 in CNT_Session (w=0x7fffff7fcf30, sp=0x801132000) at steps.h:7 #4 0x000000000040980c in wrk_thread (priv=0x7fffffffe734) at cache_pool.c:66 #5 0x000000080097a6d9 in thread_start (curthread=0x80106429b, start_routine=0x8011323a2, arg=0x80106429b) at /usr/src/lib/libpthread/thread/thr_create.c:344 #6 0x0000000800be6c94 in makectx_wrapper (ucp=0x800548870, func=0xb031c7f9, args=0x801064290) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 #7 0x0000000000000000 in ?? () #8 0x0000000801051000 in ?? () #9 0x00000000004096b0 in child_main () #10 0x00007fffffffe734 in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () #13 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff7fd000 (gdb) thr 5 [Switching to thread 5 (Thread 0x801050c00 (runnable))]#0 0x0000000800c6b86c in kevent () at kevent.S:2 2 RSYSCALL(kevent) Current language: auto; currently asm (gdb) where #0 0x0000000800c6b86c in kevent () at kevent.S:2 #1 0x0000000800862e74 in kq_dispatch (base=0xe, arg=0x801009680, tv=0x0) at kqueue.c:220 #2 0x000000080085dd4a in event_base_loop (base=0x801120220, flags=0) at event.c:413 #3 0x0000000000405630 in vbe_main (priv=0xe) at cache_backend.c:281 #4 0x000000080097a6d9 in thread_start (curthread=0xe, start_routine=0x8010d3800, arg=0xe) at /usr/src/lib/libpthread/thread/thr_create.c:344 #5 0x0000000800be6c94 in makectx_wrapper (ucp=0x800548070, func=0x8009642e0 , args=0x0) at /usr/src/lib/libc/amd64/gen/makecontext.c:100 #6 0x0000000000000000 in ?? () #7 0x0000000801050c00 in ?? () #8 0x00000000004055b0 in vbe_rdf () at cache_backend.c:262 #9 0x0000000000000000 in ?? () #10 0x0000000000000000 in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff9fe000 (gdb) thr 6 [Switching to thread 6 (Thread 0x801050800 (LWP 100179))]#0 0x0000000800989e5c in kse_release () at kse_release.S:2 2 RSYSCALL(kse_release) (gdb) where #0 0x0000000800989e5c in kse_release () at kse_release.S:2 #1 0x0000000800977326 in sig_daemon (arg=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_sig.c:214 #2 0x00000008009813ca in kse_sched_single (kmbx=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_kern.c:886 #3 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffffbff000 (gdb) thr 7 [Switching to thread 7 (Thread 0x801050400 (runnable))]#0 0x0000000800c6b86c in kevent () at kevent.S:2 2 RSYSCALL(kevent) (gdb) where #0 0x0000000800c6b86c in kevent () at kevent.S:2 #1 0x0000000800862e74 in kq_dispatch (base=0x9, arg=0x801009660, tv=0x0) at kqueue.c:220 #2 0x000000080085dd4a in event_base_loop (base=0x8011201c0, flags=0) at event.c:413 #3 0x000000000040962e in child_main () at cache_main.c:133 #4 0x000000000040cc23 in start_child () at mgt_child.c:226 #5 0x000000000040f8ef in testme () at varnishd.c:345 #6 0x0000000000410113 in main (argc=9, argv=0x7fffffffe880) at varnishd.c:590 (gdb) thr 8 [Switching to thread 8 (LWP 100155)]#0 0x0000000800989e5c in kse_release () at kse_release.S:2 2 RSYSCALL(kse_release) (gdb) where #0 0x0000000800989e5c in kse_release () at kse_release.S:2 #1 0x0000000800977326 in sig_daemon (arg=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_sig.c:214 #2 0x00000008009813ca in kse_sched_single (kmbx=0x7fffffbfef70) at /usr/src/lib/libpthread/thread/thr_kern.c:886 #3 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffffbff000 DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 09:36:20 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 09:36:20 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 09:27:50 +0200." Message-ID: <3876.1152869780@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Please give me a backtrace of all the threads then, I must have >> overlooked something. I still don't see it... Is this with stock VCL code ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 09:47:17 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 11:47:17 +0200 Subject: tot dumps core In-Reply-To: <3876.1152869780@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 14 Jul 2006 09:36:20 +0000") References: <3876.1152869780@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Is this with stock VCL code ? Yes: des at dma ~varnish/varnish-cache% cat varnish.sh #!/bin/sh base=$(dirname $(realpath $0)) exec $base/bin/varnishd/varnishd \ -b localhost:80 \ -p 8080 \ -s file,${base}/storage \ -w 1 I thought the storage file looks weird: des at dma ~varnish/varnish-cache% ll storage -rw------- 1 des des 34365272064 Jul 14 10:01 storage des at dma ~varnish/varnish-cache% du -k storage 48 storage but I recreated it with dd (32 MB from /dev/zero) and it still crashes, also with today's code. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 11:54:33 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 11:54:33 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 11:47:17 +0200." Message-ID: <51222.1152878073@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: > >des at dma ~varnish/varnish-cache% cat varnish.sh >#!/bin/sh > >base=3D$(dirname $(realpath $0)) > >exec $base/bin/varnishd/varnishd \ > -b localhost:80 \ > -p 8080 \ > -s file,${base}/storage \ > -w 1 Works for me... Can you try to run it from the build directory ? I'm sceptical about all the weird magic that the varnishd "cheat" script does. >I thought the storage file looks weird: > >des at dma ~varnish/varnish-cache% ll storage >-rw------- 1 des des 34365272064 Jul 14 10:01 storage >des at dma ~varnish/varnish-cache% du -k storage >48 storage It's a sparse file. I think we generally allocate sequentially (it's certainly the intent that we should do) so we shouldn't need to preallocate it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 12:01:03 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 14:01:03 +0200 Subject: tot dumps core In-Reply-To: <51222.1152878073@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 14 Jul 2006 11:54:33 +0000") References: <51222.1152878073@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > des at dma ~varnish/varnish-cache% ll storage > > -rw------- 1 des des 34365272064 Jul 14 10:01 storage > It's a sparse file. I think we generally allocate sequentially > (it's certainly the intent that we should do) so we shouldn't Yes, I just don't understand why it started out at 32 GB; I though the default was half of the space available on the file system, and this file system only has ~9 GB available. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 12:06:15 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 12:06:15 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 14:01:03 +0200." Message-ID: <54225.1152878775@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Dag-Erling Sm=F8rgrav writes: >> > des at dma ~varnish/varnish-cache% ll storage >> > -rw------- 1 des des 34365272064 Jul 14 10:01 storage >> It's a sparse file. I think we generally allocate sequentially >> (it's certainly the intent that we should do) so we shouldn't > >Yes, I just don't understand why it started out at 32 GB; I though the >default was half of the space available on the file system, and this >file system only has ~9 GB available. I have a 32/64 bit problem somewhere in that code. It's on my list. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Jul 14 12:06:45 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 12:06:45 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 14:01:03 +0200." Message-ID: <54230.1152878805@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Dag-Erling Sm=F8rgrav writes: >> > des at dma ~varnish/varnish-cache% ll storage >> > -rw------- 1 des des 34365272064 Jul 14 10:01 storage >> It's a sparse file. I think we generally allocate sequentially >> (it's certainly the intent that we should do) so we shouldn't > >Yes, I just don't understand why it started out at 32 GB; I though the >default was half of the space available on the file system, and this >file system only has ~9 GB available. Actually, this is probably a good time to start using our bug-tracking thing (or wiki if nothing else) to keep track of issues) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 12:05:41 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 14:05:41 +0200 Subject: tot dumps core References: <51222.1152878073@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Can you try to run it from the build directory ? No difference. BTW, this is an Opteron 140 with 1 GB RAM running a recent -CURRENT. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Fri Jul 14 12:07:38 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 14:07:38 +0200 Subject: tot dumps core References: <51222.1152878073@critter.freebsd.dk> Message-ID: des at linpro.no (Dag-Erling Sm?rgrav) writes: > BTW, this is an Opteron 140 with 1 GB RAM running a recent -CURRENT. The build directory (and the storage file) are on NFS, but I also tried with the storage file on a local file system, and with malloc storage; no luck. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 12:14:45 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 12:14:45 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 14:07:38 +0200." Message-ID: <55435.1152879285@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >des at linpro.no (Dag-Erling Sm?rgrav) writes: >> BTW, this is an Opteron 140 with 1 GB RAM running a recent -CURRENT. > >The build directory (and the storage file) are on NFS, but I also >tried with the storage file on a local file system, and with malloc >storage; no luck. I'm (only) running on my laptop (i386) right now, so if I've done something really stupid 64bit wise that could be a vector. Trying giving an numeric IP# instead of "localhost", and please try both IPv4 and IPv6 versions to see if that's where the trouble is. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 12:30:27 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 14:30:27 +0200 Subject: tot dumps core References: <55435.1152879285@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I'm (only) running on my laptop (i386) right now, so if I've done > something really stupid 64bit wise that could be a vector. varnish-test-2 is still up and running; feel free to test there. > Trying giving an numeric IP# instead of "localhost", and please try both > IPv4 and IPv6 versions to see if that's where the trouble is. Using 127.0.0.1 instead of localhost didn't help, and using [::1]:80 doesn't work because the parser doesn't recognize it as a valid address. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 12:49:42 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 12:49:42 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 14:30:27 +0200." Message-ID: <72522.1152881382@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> I'm (only) running on my laptop (i386) right now, so if I've done >> something really stupid 64bit wise that could be a vector. > >varnish-test-2 is still up and running; feel free to test there. > >> Trying giving an numeric IP# instead of "localhost", and please try both >> IPv4 and IPv6 versions to see if that's where the trouble is. > >Using 127.0.0.1 instead of localhost didn't help, and using [::1]:80 >doesn't work because the parser doesn't recognize it as a valid >address. That's a good argument against the ':' between host and port. Should we use another separator (' ', '/', ',' or ?) or should we rather add an option for backend portnumber (and if so, which ?) ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 13:09:06 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 15:09:06 +0200 Subject: tot dumps core In-Reply-To: <72522.1152881382@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 14 Jul 2006 12:49:42 +0000") References: <72522.1152881382@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > Using 127.0.0.1 instead of localhost didn't help, and using > > [::1]:80 doesn't work because the parser doesn't recognize it as a > > valid address. > That's a good argument against the ':' between host and port. > Should we use another separator (' ', '/', ',' or ?) or should we > rather add an option for backend portnumber (and if so, which ?) ? The correct syntax for numeric IPv6 addresses is "[ip]:port". Is there a particular reason why we can't support it with slightly smarter parsing code? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 14 13:37:23 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Jul 2006 13:37:23 +0000 Subject: tot dumps core In-Reply-To: Your message of "Fri, 14 Jul 2006 15:09:06 +0200." Message-ID: <76076.1152884243@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Dag-Erling Sm=F8rgrav writes: >> > Using 127.0.0.1 instead of localhost didn't help, and using >> > [::1]:80 doesn't work because the parser doesn't recognize it as a >> > valid address. >> That's a good argument against the ':' between host and port. >> Should we use another separator (' ', '/', ',' or ?) or should we >> rather add an option for backend portnumber (and if so, which ?) ? > >The correct syntax for numeric IPv6 addresses is "[ip]:port". Is >there a particular reason why we can't support it with slightly >smarter parsing code? I'm simply using getaddrinfo() wherever I can in order to not have to open that entire kettle of fish. If "[ip]:port" is ratified, then getaddrinfo() should be taught about it. In the meantime I think I will change varnishd to use a ' ' separator so that you could either write: -b '[::1]:80' (if getaddrinfo() learns the trick) or -b '::1 80' (until then) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 14 13:40:31 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 14 Jul 2006 15:40:31 +0200 Subject: tot dumps core In-Reply-To: <76076.1152884243@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 14 Jul 2006 13:37:23 +0000") References: <76076.1152884243@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I'm simply using getaddrinfo() wherever I can in order to not > have to open that entire kettle of fish. > > If "[ip]:port" is ratified, then getaddrinfo() should be taught > about it. I believe getaddrinfo() knows about "[ip]" but expects the port number in a separate argument. The command-line parsing code needs to recognize "[ip]:port" and split it into "[ip]" and "port". DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From anders at fupp.net Mon Jul 17 11:31:14 2006 From: anders at fupp.net (Anders Nordby) Date: Mon, 17 Jul 2006 13:31:14 +0200 Subject: Controlling memory usage Message-ID: <20060717113114.GA76977@totem.fix.no> Hi, Will Varnish dynamically use RAM optimally given how much you have available? A problem with Squid today is that it's not possible to set an upper limit for how much RAM it will use, and with time it can easily grow beyond the limits of the VM causing it to get killed (that is it restarts itself, losing all cache in RAM). I take it this will not be a problem with Varnish? :-) Do you use kernel/userspace memory as needed? Cheers, -- Anders. From des at linpro.no Mon Jul 17 12:01:31 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 17 Jul 2006 14:01:31 +0200 Subject: Controlling memory usage References: <20060717113114.GA76977@totem.fix.no> Message-ID: Anders Nordby writes: > Will Varnish dynamically use RAM optimally given how much you have > available? Yes and no. With the file-based storage backend, Varnish stores documents in a single file which it mmaps, and malloc() is used only for housekeeping structures. Memory efficiency in this scenario is really up to the kernel, specifically the buffer cache. You can see from the screenshots from day 2 of the live test (online at ) that Varnish uses 400 MB RAM with ~23,000 objects cached. This later grew to ~45,000 objects, but I don't have memory usage numbers for that part of the test (I killed top at some point to look at some other numbers and didn't restart it). You can however look at the systat numbers from the last screenshot: 293100 wire 214924 act 3075272 inact 201504 cache 219632 buf Since the machine isn't swapping, "act" is more or less the sum of the stacks and heaps of all running processes, including Varnish, while "cache" and "buf" give an estimate of how much memory is used to cache disk pages, including Varnish's storage file. The most impressive number is "inact", which says that almost three quarters of the server's 4 GB of memory are completely unused. Next time, I'll try to remember to take snapshots of Varnish's /proc/$$/map to get a precise breakdown. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Mon Jul 17 13:53:18 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Jul 2006 13:53:18 +0000 Subject: Controlling memory usage In-Reply-To: Your message of "Mon, 17 Jul 2006 13:31:14 +0200." <20060717113114.GA76977@totem.fix.no> Message-ID: <90078.1153144398@critter.freebsd.dk> In message <20060717113114.GA76977 at totem.fix.no>, Anders Nordby writes: Gee, you really know how to ask "simple" questions, don't you ? :-) >Will Varnish dynamically use RAM optimally given how much you have >available? That is the intent, but we implement it by letting the kernel decide on that. >A problem with Squid today is that it's not possible to set an upper >limit for how much RAM it will use, and with time it can easily grow >beyond the limits of the VM causing it to get killed (that is it >restarts itself, losing all cache in RAM). Yes, I've seen that one already many years ago. Squids two-tier storage architecture where some stuff is in ram and the rest on disk is collapsed in varnish so that it all is on disk, but mapped into memory. It's important to understand that we are talking about at least three different kinds of memory of relevance: Virtual address room, Virtual allocated space and Virtual mapped space. The virtual address room is the amount of memory that the program can address and it is basically the width of the CPU: 32 or 64 bits. On 32 bit systems, we're limited to approx 3.5 GB. On 64bit systems it's a non-issue. (This will limit 32bit systems total cache capacity accordingly, but 64bit systems are no more expensive these days). Virtual allocated space is the varnish programs variables and malloc(3) allocated space. This is what is normally counted as "programs memory use". All of squids memory falls in this category. If you use the 'malloc" storage method of varnish it also falls here. Virtual mapped space is the backing store file which is mapped into the varnish process address space, and this is accounted differently than virtual allocated space. Finally there is the concept of "resident set size" which is the amount of actual physical RAM the process occupies with the bits of the above two which are mapped into memory at any one particular time. >I take it this will not be a problem with Varnish? :-) I hope not, but quite frankly, apart from some scientific apps I have seen very few programs that use memory this way (yet! but after all we have only had virtual memory for 20 years :-/ ) so the scope for OS bugs is certainly present. >Do you use kernel/userspace memory as needed? Well, yes and no: I leave it to the operating system to do so. I think another way to answer your question is this: Squid is written for the old UNIX/swap model before the VAX/780 computer came around: it doesn't in any sense of the word comprehend virtual memory. In may ways this gets in the way of efficiency. Reading things from a file, into userland and then writing it to a socket moves the data: from disk (with DMA) to kernel RAM buffer from kernel RAM buffer to userland RAM. from userland RAM to network buffer in kernel from network buffer in kernel to network interface (with DMA) This is just plain waste of good cycles. Varnish on the other hand, is written with the maximum comprehension of virtual memory and tries to do everything it can to avoid getting in the way of the operating systems efficient. Varnish maps the backing file into the process, so the above scenario cuts out one copy already there: from disk (with DMA) to userland mapped memory from userland mapped memory to network buffer in kernel from network buffer in kernel to network interface (with DMA) And by using the sendfile(2) system call, this even gets reduced to: from disk (with DMA) to userland mapped memory send userland mapped memory to network interface (with DMA) (Various details left out, TCP/IP checksums in particular) I hope this if not exactly answers your question, at least gives you the impression that I've thought hard about it :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Jul 17 13:59:06 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Jul 2006 13:59:06 +0000 Subject: Running a small live-test this week ? Message-ID: <90115.1153144746@critter.freebsd.dk> I would like to run a live-test later this week, and while it is not a requirement on my part, it can be done with a low volume of traffic if necessary. Is there a good time for you Anders ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Mon Jul 17 17:21:59 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 17 Jul 2006 19:21:59 +0200 Subject: Running a small live-test this week ? In-Reply-To: <90115.1153144746@critter.freebsd.dk> References: <90115.1153144746@critter.freebsd.dk> Message-ID: <2DDC1894-7C06-4789-85AF-1B0B2DB5451F@vgnett.no> Any time is good except 13-14 on thursday (meeting). Let me know by mail when you wanna do it and we'll IRC it from there. Anders Berg On Jul 17, 2006, at 15:59, Poul-Henning Kamp wrote: > > I would like to run a live-test later this week, and while it is not > a requirement on my part, it can be done with a low volume of traffic > if necessary. > > Is there a good time for you Anders ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From andersb at vgnett.no Mon Jul 17 17:49:02 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 17 Jul 2006 19:49:02 +0200 Subject: Controlling memory usage In-Reply-To: <90078.1153144398@critter.freebsd.dk> References: <90078.1153144398@critter.freebsd.dk> Message-ID: Thx for detailed answers Poul-Henning and Dag-Erling, The reason Anders N. asks about this is how Squid works today. The squid.conf file leaves you with a option to specify how much RAM you wanna use for Squid. The problem is that Squid (probably because of old design) does not really "follow" that option. If you set 256 MB RAM, you can still end up using 500 MB RAM, and if you set option close to max memory you will for certain overflow your physical RAM and start swapping/die. Your answer was detailed Poul-Henning, but what will prevent this from happpening in Varnish? Lets say you have 2 applications running on a Varnish box, and both use the memory model Varnish uses, what will happen in the long run with a lot of traffic? Will they both adjust themselvs to match pysical RAM, how would they "compete" for RAM etc? It's possible you have already answered that below, but how do we "limit" RAM usage? Also I would like to mention that while at VG neither Squid or Varnish (it seems) have any problem dealing with I/O, the traffic pattern, pysical size'es of dataset's etc. make it 100% CPU bound. My impression is that finn.no's pattern is quite different and interesting. A quick look on: http://www.tns-gallup.no/index.asp? type=tabelno_url&did=185235&sort=uv&sort_ret=desc&UgeSelect=&path_by_id= /12000/12003/12077/12266&aid=12266 will show what I mean. They have "few" users and sessions, but loads of pageviews, also at any given time many thousand "wares" are "hot" for the user, not a few popular articles. This does something to mem usage and I/O. I am looking forward to test-run Varnish on finn.no :) At the same time when a ware is "sold" purging becomes important :) Maybe not so much on finn.no today, but I can easily see auctions happen in the close future :) Anders Berg On Jul 17, 2006, at 15:53, Poul-Henning Kamp wrote: > In message <20060717113114.GA76977 at totem.fix.no>, Anders Nordby > writes: > > > Gee, you really know how to ask "simple" questions, don't you ? :-) > > >> Will Varnish dynamically use RAM optimally given how much you have >> available? > > That is the intent, but we implement it by letting the kernel > decide on that. > >> A problem with Squid today is that it's not possible to set an upper >> limit for how much RAM it will use, and with time it can easily grow >> beyond the limits of the VM causing it to get killed (that is it >> restarts itself, losing all cache in RAM). > > Yes, I've seen that one already many years ago. > > Squids two-tier storage architecture where some stuff is in ram > and the rest on disk is collapsed in varnish so that it all is > on disk, but mapped into memory. > > It's important to understand that we are talking about at least > three different kinds of memory of relevance: Virtual address room, > Virtual allocated space and Virtual mapped space. > > The virtual address room is the amount of memory that the program > can address and it is basically the width of the CPU: 32 or > 64 bits. On 32 bit systems, we're limited to approx 3.5 GB. On > 64bit systems it's a non-issue. (This will limit 32bit systems > total cache capacity accordingly, but 64bit systems are no more > expensive these days). > > Virtual allocated space is the varnish programs variables and > malloc(3) allocated space. This is what is normally counted > as "programs memory use". All of squids memory falls in this > category. If you use the 'malloc" storage method of varnish > it also falls here. > > Virtual mapped space is the backing store file which is mapped > into the varnish process address space, and this is accounted > differently than virtual allocated space. > > Finally there is the concept of "resident set size" which is > the amount of actual physical RAM the process occupies > with the bits of the above two which are mapped into memory > at any one particular time. > >> I take it this will not be a problem with Varnish? :-) > > I hope not, but quite frankly, apart from some scientific apps I > have seen very few programs that use memory this way (yet! but after > all we have only had virtual memory for 20 years :-/ ) so the scope > for OS bugs is certainly present. > >> Do you use kernel/userspace memory as needed? > > Well, yes and no: I leave it to the operating system to do so. > > I think another way to answer your question is this: > > Squid is written for the old UNIX/swap model before the VAX/780 > computer came around: it doesn't in any sense of the word comprehend > virtual memory. In may ways this gets in the way of efficiency. > Reading things from a file, into userland and then writing it to a > socket moves the data: > > from disk (with DMA) to kernel RAM buffer > from kernel RAM buffer to userland RAM. > from userland RAM to network buffer in kernel > from network buffer in kernel to network interface (with DMA) > > This is just plain waste of good cycles. > > Varnish on the other hand, is written with the maximum comprehension > of virtual memory and tries to do everything it can to avoid getting > in the way of the operating systems efficient. Varnish maps the > backing file into the process, so the above scenario cuts out > one copy already there: > > from disk (with DMA) to userland mapped memory > from userland mapped memory to network buffer in kernel > from network buffer in kernel to network interface (with DMA) > > And by using the sendfile(2) system call, this even gets reduced to: > > from disk (with DMA) to userland mapped memory > send userland mapped memory to network interface (with DMA) > > (Various details left out, TCP/IP checksums in particular) > > I hope this if not exactly answers your question, at least gives > you the impression that I've thought hard about it :-) > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From phk at phk.freebsd.dk Mon Jul 17 19:02:20 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Jul 2006 19:02:20 +0000 Subject: Controlling memory usage In-Reply-To: Your message of "Mon, 17 Jul 2006 19:49:02 +0200." Message-ID: <7746.1153162940@critter.freebsd.dk> In message , Anders Berg writes : >The reason Anders N. asks about this is how Squid works today. The >squid.conf file leaves you with a option to specify how much RAM you >wanna use for Squid. And this is right where the trouble starts. Squid is written for a machine model that has not existed since 1980 when 3BSD was released. In that model, a process has some amount of "memory" and either all of that "memory" is present in RAM or none of it is. When RAM grew short, an entire process was swapped out (hence the name: "swap out one process for another") In that environment, it gives great meaning to tell Squid how much RAM it can use, because there is some magic size where the best performance compromise for the entire machine is reached. We spent a lot of time tuning stuff like that in 1980ies, we told sort(1) how many records to sort in memory and to switch to merging temporary files if it found more etc etc. Virtual memory on the other hand, means that the kernel "fakes" things such that the process has access to the entire address-space (ie: 2^32 bytes or 2^64 bytes) and the operating. It does this by tracking which pages are used, which are modified and all that stuff. In a VM system, what you think of as "RAM" is not RAM in the hardware sense. You may in fact have all of it accessible in hardware RAM, but if the system is short of memory, you won't have, some of it will be "paged out to disk" or because we sloppily adopted the old terminology: swapped out. The real trouble starts when Squid decides that an object in its "RAM" should be purged to disk. Quite likely, the operating system already found that out earlier so the "RAM" is already on disk, somewhere in the paging- (or swap-)partition. So what happens is that first we do a disk read to pull in the RAM, then we write it to disk some other place. Twice as much I/O for no gain. The same pattern happens all over Squid, and that is responsible for the observed "once squid starts paging, it goes straight south". It doesn't help in this context that Squid stores headers and body the same place. That means that if the "RAM" of some object has been paged out, we have to page it in to see the headers, even for a conditional request which ends up not transferring the objects body. >Your answer was detailed Poul-Henning, but >what will prevent this from happpening in Varnish? Lets say you have >2 applications running on a Varnish box, and both use the memory >model Varnish uses, what will happen in the long run with a lot of >traffic? All programs running in a VM system has a function which describes how fast they reach their goal, for a given number of pages of hardware memory they have access to. Unfortunately the function also has other variables, the input to the program, the timing its interaction with the world (how long must it wait for disk-I/O etc) and the state of all sorts of kernel caches come into play. There is no way to predict the function realistically. You can measure it under some set of circumstances and get an idea how it looks. The only trick there really is to writing an VM kernel is being good at estimating this function on the fly. If two processes run at the same time, and they both need more hw-RAM pages than the system has, the kernel will be flipping some pages between them. When a program accesses a page which is not "resident", the kernel will hunt around for a page that doesn't look used (ideally: doesn't look like it _will be_ used (soon)) writes that page to disk and reassigns the page to the faulting process, possibly after filling it from a disk first. In the meantime the process (or at least: thread) cannot do anthing. If you're just one page short, there is undoubtedly some page in the process which is seldomly used, the first bit of the program which is only used during startup, some table of error messages that are only accessed when there is an error etc. As memory pressure increases, more and more such pages will be paged out. At some point, we get to pages which are infact used every so often, and then it starts hurting performance. The thing to remember in writing programs for virtual memory sytems is therefore not to be careful about how much memory you allocate, but instead be careful about how much of it you use. Something as simple as variable order in the source code can affect this: int busy_integer_variable; char seldomly_used_error_string[5000]; double often_used_number; With a pagesize of 4096, this will take up two pages, both of which will be busy. Flipping the order: int busy_integer_variable; double often_used_number; char seldomly_used_error_string[5000]; means that one page will seldomly be used, and the other will be used all the time. (The example also improves CPU-cache hitrates, but forget that for now). What Varnish does is to rely on the kernel to do this work. Instead of trying to control how much memory we use and partition our data into the fast stuff which should be in RAM and the slow stuff which we can put on the disk, we simply operate on one data set, but make sure to arrange our data such that the kernel can easily deport data which we don't use, without us needing to get involved. Therefore all object storage in Varnish is allocated on a page-aligned border. That means that entire objects can be paged out, without affecting the neighbor objects. Yes, this may waste 4095 bytes for padding, but you'd be surprised what you save in performance. >http://www.tns-gallup.no/index.asp? >type=tabelno_url&did=185235&sort=uv&sort_ret=desc&UgeSelect=&path_by_id= >/12000/12003/12077/12266&aid=12266 > >will show what I mean. They have "few" users and sessions, but loads >of pageviews, also at any given time many thousand "wares" are "hot" >for the user, not a few popular articles. This does something to mem >usage and I/O. You're thinking about memory in the oldfashioned terms here :-) Try this: Imagine the disk is the real memory and the RAM is only a cache. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Mon Jul 17 19:34:49 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 17 Jul 2006 21:34:49 +0200 Subject: Controlling memory usage References: <90078.1153144398@critter.freebsd.dk> Message-ID: Anders Berg writes: > The reason Anders N. asks about this is how Squid works today. The > squid.conf file leaves you with a option to specify how much RAM you > wanna use for Squid. I've worked with Squid quite a lot myself, and IIRC it normally uses about four times as much as you specify in the config file... > The problem is that Squid (probably because of > old design) does not really "follow" that option. If you set 256 MB > RAM, you can still end up using 500 MB RAM, and if you set option > close to max memory you will for certain overflow your physical RAM > and start swapping/die. Your answer was detailed Poul-Henning, but > what will prevent this from happpening in Varnish? Lets say you have > 2 applications running on a Varnish box, and both use the memory > model Varnish uses, what will happen in the long run with a lot of > traffic? Will they both adjust themselvs to match pysical RAM, how > would they "compete" for RAM etc? It's possible you have already > answered that below, but how do we "limit" RAM usage? We don't. The kernel takes care of it. If memory is scarce, the kernel will allocate less memory to the buffer cache and Varnish performance will decrease. (assuming you use file storage, not malloc storage) As for heap usage, which the kernel has less control over, it is mostly a function of the number of objects in the cache; we can take care of that by setting an upper limit on the number of cached objects, or even on the amount of heap used to track objects. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Mon Jul 17 20:44:45 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 17 Jul 2006 22:44:45 +0200 Subject: Controlling memory usage In-Reply-To: <7746.1153162940@critter.freebsd.dk> References: <7746.1153162940@critter.freebsd.dk> Message-ID: Okay, that made more sense to me. Thank you for another detailed explanation. Rest assured that question number 3, after "how do I start?", "how do I stop?", is "how can I allocate memory?". This is good material in the process of trying to tell old Squid users how we "use" memory. FAQ material. Anders Berg On Jul 17, 2006, at 21:02, Poul-Henning Kamp wrote: > In message , Anders > Berg writes > : > >> The reason Anders N. asks about this is how Squid works today. The >> squid.conf file leaves you with a option to specify how much RAM you >> wanna use for Squid. > > And this is right where the trouble starts. > > Squid is written for a machine model that has not existed since > 1980 when 3BSD was released. > > In that model, a process has some amount of "memory" and either all > of that "memory" is present in RAM or none of it is. When RAM grew > short, an entire process was swapped out (hence the name: "swap out > one process for another") From andersb at vgnett.no Mon Jul 17 20:47:18 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 17 Jul 2006 22:47:18 +0200 Subject: Controlling memory usage In-Reply-To: References: <90078.1153144398@critter.freebsd.dk> Message-ID: On Jul 17, 2006, at 21:34, Dag-Erling Sm?rgrav wrote: > Anders Berg writes: >> The reason Anders N. asks about this is how Squid works today. The >> squid.conf file leaves you with a option to specify how much RAM you >> wanna use for Squid. > > I've worked with Squid quite a lot myself, and IIRC it normally uses > about four times as much as you specify in the config file... > >> The problem is that Squid (probably because of >> old design) does not really "follow" that option. If you set 256 MB >> RAM, you can still end up using 500 MB RAM, and if you set option >> close to max memory you will for certain overflow your physical RAM >> and start swapping/die. Your answer was detailed Poul-Henning, but >> what will prevent this from happpening in Varnish? Lets say you have >> 2 applications running on a Varnish box, and both use the memory >> model Varnish uses, what will happen in the long run with a lot of >> traffic? Will they both adjust themselvs to match pysical RAM, how >> would they "compete" for RAM etc? It's possible you have already >> answered that below, but how do we "limit" RAM usage? > > We don't. The kernel takes care of it. If memory is scarce, the > kernel will allocate less memory to the buffer cache and Varnish > performance will decrease. > > (assuming you use file storage, not malloc storage) > > As for heap usage, which the kernel has less control over, it is > mostly a function of the number of objects in the cache; we can take > care of that by setting an upper limit on the number of cached > objects, or even on the amount of heap used to track objects. Thx for input DES. Setting an upper limit either in objects or heap is a good way to control memory usage (And the right one I guess). Are we gonna implement this? Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From phk at phk.freebsd.dk Tue Jul 18 15:40:39 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 18 Jul 2006 15:40:39 +0000 Subject: tot dumps core In-Reply-To: Your message of "Wed, 12 Jul 2006 21:33:01 GMT." <43826.1152739981@critter.freebsd.dk> Message-ID: <76711.1153237239@critter.freebsd.dk> I've seen something like it here also now. Looks like pointer-fandango :-( -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Tue Jul 18 20:47:26 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 18 Jul 2006 22:47:26 +0200 Subject: tot dumps core References: <76711.1153237239@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I've seen something like it here also now. > > Looks like pointer-fandango :-( I thought the problem might be 64-bit-specific, so I tested on a VIA Epia system, and I'm getting something completely different: root at dd ~varnish/varnish-cache# ./bin/varnishd/varnishd -p 80 -b www.des.no:80 -w 1 zsh: segmentation fault (core dumped) ./bin/varnishd/varnishd -p 80 -b www.des.no:80 -w 1 root at dd ~varnish/varnish-cache# gdb ./bin/varnishd/.libs/lt-varnishd lt-varnishd.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Core was generated by `lt-varnishd'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0 Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0 Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1 Reading symbols from /lib/libpthread.so.2...done. Loaded symbols for /lib/libpthread.so.2 Reading symbols from /lib/libmd.so.3...done. Loaded symbols for /lib/libmd.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 VCC_T_render (f=0xbfbfe3b0, info=0xbfbfd0c0, args=0xbfbfcfec) at vcl_compile.c:1894 1894 return (fprintf(f, "%*.*s", [New LWP 100098] (gdb) where #0 VCC_T_render (f=0xbfbfe3b0, info=0xbfbfd0c0, args=0xbfbfcfec) at vcl_compile.c:1894 #1 0x2817299c in __printf_out () from /lib/libc.so.7 #2 0x28172fe9 in __xvprintf () from /lib/libc.so.7 #3 0x281b3712 in __vfprintf () from /lib/libc.so.7 #4 0x28149e31 in vsnprintf () from /lib/libc.so.7 #5 0x08057d61 in sbuf_vprintf (s=0x8270180, fmt=0x2809d154 "#define VGC_backend_%T (VCL_conf.backend[%d])\n", ap=0xbfbfe47c "?\001'\b") at sbuf.c:311 #6 0x280974fa in Fh (tl=0xbfbfe8e0, indent=1, fmt=0x2809d154 "#define VGC_backend_%T (VCL_conf.backend[%d])\n") at vcl_compile.c:345 #7 0x28098bc6 in Backend (tl=0xbfbfe8e0) at vcl_compile.c:1257 #8 0x28099a1f in Parse (tl=0xbfbfe8e0) at vcl_compile.c:1378 #9 0x2809a7b6 in VCC_Compile (sb=0xbfbfcfec, b=0x8260180 "backend default {\n set backend.host = \" http\";\n set backend.port = \"\a\";\n}\n", e=0x82601d8 "") at vcl_compile.c:1805 #10 0x08056a42 in vcl_default (bflag=0xbfbfeb79 "www.des.no:80") at varnishd.c:139 #11 0x080578b3 in main (argc=7, argv=0xbfbfe9e4) at varnishd.c:598 (gdb) p *f $1 = {_p = 0x828017f "?", _r = -1077943324, _w = 0, _flags = 520, _file = -1, _bf = {_base = 0x8280175 "#define VG?", _size = 10}, _lbfsize = 671512823, _cookie = 0x28091080, _close = 0x280968bf , _read = 0x2808d000, _seek = 0x28086118 <__collate_load_error+144>, _write = 0x28086118 <__collate_load_error+144>, _ub = { _base = 0x2809f648 "\214\225", _size = 136774016}, _extra = 0xbfbfe310, _ur = 134577505, _ubuf = "a\001(", _nbuf = "\b", _lb = { _base = 0x1f
, _size = 671729773}, _blksize = -1077943224, _offset = 2884784259749380097} (gdb) up 5 #5 0x08057d61 in sbuf_vprintf (s=0x8270180, fmt=0x2809d154 "#define VGC_backend_%T (VCL_conf.backend[%d])\n", ap=0xbfbfe47c "?\001'\b") at sbuf.c:311 311 len = vsnprintf(&s->s_buf[s->s_len], SBUF_FREESPACE(s) + 1, (gdb) up #6 0x280974fa in Fh (tl=0xbfbfe8e0, indent=1, fmt=0x2809d154 "#define VGC_backend_%T (VCL_conf.backend[%d])\n") at vcl_compile.c:345 345 sbuf_vprintf(tl->fh, fmt, ap); (gdb) up #7 0x28098bc6 in Backend (tl=0xbfbfe8e0) at vcl_compile.c:1257 1257 Fh(tl, 1, "#define VGC_backend_%T (VCL_conf.backend[%d])\n", (gdb) l 1252 ExpectErr(tl, ID); 1253 t_be = tl->t; 1254 AddDef(tl, tl->t, R_BACKEND); 1255 if (tl->nbackend == 0) 1256 AddRef(tl, tl->t, R_BACKEND); 1257 Fh(tl, 1, "#define VGC_backend_%T (VCL_conf.backend[%d])\n", 1258 tl->t, tl->nbackend); 1259 Fc(tl, 0, "static void\n"); 1260 Fc(tl, 1, "VGC_init_backend_%T (void)\n", tl->t); 1261 Fc(tl, 1, "{\n"); (gdb) p tl->t $2 = (struct token *) 0x82701c0 (gdb) p tl->nbackend $3 = 0 (gdb) p *(tl->t) $4 = {tok = 166, b = 0x8260188 "default {\n set backend.host = \" http\";\n set backend.port = \"\a\";\n}\n", e = 0x826018f " {\n set backend.host = \" http\";\n set backend.port = \"\a\";\n}\n", list = {tqe_next = 0x82701e0, tqe_prev = 0x82701ac}, cnt = 0} Note that the value assigned to backend.host is completely wrong. However, I can't exclude the possibility that there may be something wrong with the machine, since: root at dd ~varnish/varnish-cache# ./bin/varnishd/varnishd -\? zsh: segmentation fault (core dumped) ./bin/varnishd/varnishd -\? root at dd ~varnish/varnish-cache# !gd gdb ./bin/varnishd/.libs/lt-varnishd lt-varnishd.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Core was generated by `lt-varnishd'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvarnish/.libs/libvarnish.so.0 Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/lib/libvcl/.libs/libvcl.so.0 Reading symbols from /usr/scratch/des/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1...done. Loaded symbols for /usr/scratch/des/varnish/trunk/varnish-cache/contrib/libevent/.libs/libevent-1.2.so.1 Reading symbols from /lib/libpthread.so.2...done. Loaded symbols for /lib/libpthread.so.2 Reading symbols from /lib/libmd.so.3...done. Loaded symbols for /lib/libmd.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x281ba145 in strlen () from /lib/libc.so.7 [New LWP 100098] (gdb) where #0 0x281ba145 in strlen () from /lib/libc.so.7 #1 0x2817059e in __printf_render_str () from /lib/libc.so.7 #2 0x28172a0f in __printf_out () from /lib/libc.so.7 #3 0x28172f75 in __xvprintf () from /lib/libc.so.7 #4 0x281b3712 in __vfprintf () from /lib/libc.so.7 #5 0x281b6942 in vfprintf () from /lib/libc.so.7 #6 0x281a558b in fprintf () from /lib/libc.so.7 #7 0x2810b5c2 in getopt () from /lib/libc.so.7 #8 0x08057845 in main (argc=673000184, argv=0xbfbfeb8a) at varnishd.c:577 DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Jul 18 20:53:05 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 18 Jul 2006 22:53:05 +0200 Subject: tot dumps core References: <76711.1153237239@critter.freebsd.dk> Message-ID: des at linpro.no (Dag-Erling Sm?rgrav) writes: > I thought the problem might be 64-bit-specific, so I tested on a VIA > Epia system, and I'm getting something completely different: > [...] > However, I can't exclude the possibility that there may be something > wrong with the machine, since: This must be the case, as I no longer have any problems on amd64: > des at xps ~% telnet dma 8080 > Trying 10.0.0.10... > Connected to dma.des.no. > Escape character is '^]'. > HEAD / HTTP/1.1 > Host: www.des.no > Connection: close > > HTTP/1.1 200 OK > Date: Tue, 18 Jul 2006 20:50:41 GMT > Server: Apache/2.0 > Last-Modified: Tue, 02 May 2006 17:07:58 GMT > ETag: "df3037-1bde-412d13675d780" > Content-Type: text/html > Content-Length: 7134 > Age: 0 > Via: 1.1 varnish > X-Varnish: xid 1694085758 > > Connection closed by foreign host. (dma is the same Opteron 140 I was testing on earlier) although I got this earlier when I closed a client connection before I finished typing the request header: > des at dma ~varnish/varnish-cache% ./varnish.sh > file /home/des/projects/varnish/trunk/varnish-cache/storage size 33554432 bytes (65536 fs-blocks, 8192 pages) > Creating new SHMFILE > start child pid 44753 > Child said > Child said > Child said > Child said > Child said <> > Child said srcaddr != NULL)> > Child said < function SES_RelSrcAddr, file cache_session.c, line 135.> > Child said < errno 22 = "Unknown error: 0"> > sig_chld(20, 8, 0) > pid = 44753 status = 0x86 > Child died :-( DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Jul 19 11:19:58 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Jul 2006 11:19:58 +0000 Subject: tot dumps core In-Reply-To: Your message of "Tue, 18 Jul 2006 22:47:26 +0200." Message-ID: <83510.1153307998@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> I've seen something like it here also now. >> >> Looks like pointer-fandango :-( > >I thought the problem might be 64-bit-specific, so I tested on a VIA >Epia system, and I'm getting something completely different: As you've seen I've found the missing star. No, I'm not proud of that taking me a day to find. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Jul 19 11:22:53 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 19 Jul 2006 13:22:53 +0200 Subject: tot dumps core In-Reply-To: <83510.1153307998@critter.freebsd.dk> (Poul-Henning Kamp's message of "Wed, 19 Jul 2006 11:19:58 +0000") References: <83510.1153307998@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > As you've seen I've found the missing star. > > No, I'm not proud of that taking me a day to find. Shit happens. The question is, what can we do to catch the next missing star faster? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Jul 19 11:42:03 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Jul 2006 11:42:03 +0000 Subject: tot dumps core In-Reply-To: Your message of "Wed, 19 Jul 2006 13:22:53 +0200." Message-ID: <83585.1153309323@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> As you've seen I've found the missing star. >> >> No, I'm not proud of that taking me a day to find. > >Shit happens. The question is, what can we do to catch the next >missing star faster? I used a little thing I have that I call "miniobj.h". It is a bunch of macros and a magic field in all structs with a unique value: struct object { + unsigned magic; +#define OBJECT_MAGIC 0x32851d42 [...] } When you allocate a structure, you have to set the magic field correctly: w->nobj = calloc(sizeof *w->nobj, 1); assert(w->nobj != NULL); + w->nobj->magic = OBJECT_MAGIC; And then you can check that all over the place, but in particular where void pointers have been used to carry your pointer: o = binheap_root(exp_heap); + if (o != NULL) + CHECK_OBJ(o, OBJECT_MAGIC); It's the kind of thing I wish I could get the compiler to do (and it's part of the 'K' language thing I'm playing with) I havn't quite decided if I want to commit this to Varnish but I may do so later today. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Jul 19 11:46:50 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Jul 2006 11:46:50 +0000 Subject: Running a small live-test this week ? In-Reply-To: Your message of "Mon, 17 Jul 2006 19:21:59 +0200." <2DDC1894-7C06-4789-85AF-1B0B2DB5451F@vgnett.no> Message-ID: <83645.1153309610@critter.freebsd.dk> In message <2DDC1894-7C06-4789-85AF-1B0B2DB5451F at vgnett.no>, Anders Berg writes : >Any time is good except 13-14 on thursday (meeting). > >Let me know by mail when you wanna do it and we'll IRC it from there. Jeg er online p? irc.linpro.no/#varnish nu, sig til n?r du er klar... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Jul 19 21:21:49 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Jul 2006 21:21:49 +0000 Subject: Todays livetest, so far... In-Reply-To: Your message of "Mon, 17 Jul 2006 19:21:59 +0200." <2DDC1894-7C06-4789-85AF-1B0B2DB5451F@vgnett.no> Message-ID: <93499.1153344109@critter.freebsd.dk> After a handful of various minor bogons, we have been running continuously for six hours (and counting). Most of the time, the 100Mbit ethernet has been running flat out: 40085 4 3845930 71505 0 98084908 0 40668 2 3769753 72112 0 98242274 0 40754 1 3921064 72026 0 98254233 0 40511 5 3630429 71832 0 98200793 0 39805 1 3561995 71755 0 98311918 0 Memory use with 31000 objects is 519MB resident, approx 17KB average. I'll head to bed soon, but we'll leave it running and see what happens... Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Thu Jul 20 07:43:24 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 20 Jul 2006 09:43:24 +0200 Subject: tot dumps core In-Reply-To: <83585.1153309323@critter.freebsd.dk> (Poul-Henning Kamp's message of "Wed, 19 Jul 2006 11:42:03 +0000") References: <83585.1153309323@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I used a little thing I have that I call "miniobj.h". > > It is a bunch of macros and a magic field in all structs with > a unique value: > [...] > I havn't quite decided if I want to commit this to Varnish > but I may do so later today. If it's done with macros and we can disable it when building production releases, I'm definitely in favor. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Thu Jul 20 07:46:00 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 20 Jul 2006 09:46:00 +0200 Subject: Todays livetest, so far... References: <93499.1153344109@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Most of the time, the 100Mbit ethernet has been running flat out: > [...] > Memory use with 31000 objects is 519MB resident, approx 17KB > average. This is very good. Do you have any measure of how well the worker thread queue worked out? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Thu Jul 20 07:54:30 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 20 Jul 2006 07:54:30 +0000 Subject: Todays livetest, so far... In-Reply-To: Your message of "Thu, 20 Jul 2006 09:46:00 +0200." Message-ID: <95524.1153382070@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Most of the time, the 100Mbit ethernet has been running flat out: >> [...] >> Memory use with 31000 objects is 519MB resident, approx 17KB >> average. > >This is very good. Do you have any measure of how well the worker >thread queue worked out? Very good, number of threads were much smaller. Their patience before suicide may be to short (10sec), but that is already a command line parameter and I will make it runtime controllable as well. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Thu Jul 20 20:32:08 2006 From: andersb at vgnett.no (Anders Berg) Date: Thu, 20 Jul 2006 22:32:08 +0200 (CEST) Subject: Purge Message-ID: <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> Hi, Poul-Henning told me (in irc before he timed-out) that it would not take long to implement either TTL per URL or PURGE in VCL. I think we should implement just that (either 2, or both). That way we can test that part of VCL in the livetesting we are doing. Some FUD developed today when our frontpage editors sensed that their Purge tool did not function correctly :) Sorry about that, and the following phonecalls. Anders Berg From phk at phk.freebsd.dk Thu Jul 20 20:39:49 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 20 Jul 2006 20:39:49 +0000 Subject: Purge In-Reply-To: Your message of "Thu, 20 Jul 2006 22:32:08 +0200." <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> Message-ID: <36315.1153427989@critter.freebsd.dk> In message <1201.193.69.165.4.1153427528.squirrel at denise.vg.no>, "Anders Berg" writes: >Poul-Henning told me (in irc before he timed-out) that it would not take >long to implement either TTL per URL or PURGE in VCL. Yes. Setting the TTL would be something as simple as: sub vcl_fetch { if (req.url = "/") { obj.ttl = 120; } } (Not sure if the vcl runtime actually allows this yet) Purge is slightly different, we can do it two ways: Accept "PURGE" in vcl_recv(); If we see it again in vcl_miss() return a 404. If we get to vcl_hit() set the obj.ttl to zero and return a suitable status code. Or alternatively implement in C that PURGE will do a regexp ban, and allow PURGE only from localhost in the default vcl_recv(). Of course that means that the argument _is_ a regexp, so "PURGE / HTTP/1.1" is not the right thing to send :-) >I think we should implement just that (either 2, or both). That way we can >test that part of VCL in the livetesting we are doing. I've started on the last nasty bit of the HTTP header storage/editing/munging stuff, and I would like to get that done first, but I'll do whichever of the other two you want afterwards. >Some FUD developed today when our frontpage editors sensed that their >Purge tool did not function correctly :) Sorry about that, and the >following phonecalls. No worries, afterall, we _did_ find a bug in Varnish expiry today. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Fri Jul 21 07:58:56 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 21 Jul 2006 09:58:56 +0200 (CEST) Subject: Purge In-Reply-To: <36315.1153427989@critter.freebsd.dk> References: Your message of "Thu, 20 Jul 2006 22:32:08 +0200." <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> <36315.1153427989@critter.freebsd.dk> Message-ID: <1213.193.69.165.4.1153468736.squirrel@denise.vg.no> > In message <1201.193.69.165.4.1153427528.squirrel at denise.vg.no>, "Anders > Berg" > writes: > >>Poul-Henning told me (in irc before he timed-out) that it would not take >>long to implement either TTL per URL or PURGE in VCL. > > Yes. > > Setting the TTL would be something as simple as: > > sub vcl_fetch { > if (req.url = "/") { > obj.ttl = 120; > } > } > > [....] > > I've started on the last nasty bit of the HTTP header > storage/editing/munging stuff, and I would like to get that done > first, but I'll do whichever of the other two you want > afterwards. Sure, do that first. The if (req.url = "/") { obj.ttl = 120; } method is enuff for now. The other way is more complex to implement at the moment I guess, so it can wait. Anders Berg > Poul-Henning > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > From des at linpro.no Fri Jul 21 08:21:43 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 21 Jul 2006 10:21:43 +0200 Subject: Purge References: <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > Some FUD developed today when our frontpage editors sensed that their > Purge tool did not function correctly :) Sorry about that, and the > following phonecalls. Is this solely a Varnish issue? I did som checking when Stein Ove called me: one of the squids (specifically, cache3.c.vgnett.no) seems to be configured differently from the others, and seems to run directly against leonora instead of going through the second level squid. I also saw that you mentioned on IRC that one of the squids wasn't on the purge list... Here's a normal response from one of the squids: <<< HTTP/1.0 200 OK <<< Date: Fri, 21 Jul 2006 07:58:19 GMT <<< Server: Apache/1.3.27 (Unix) <<< Cache-Control: max-age=900 <<< Expires: Fri, 21 Jul 2006 08:13:19 GMT <<< Last-Modified: Fri, 21 Jul 2006 07:55:01 GMT <<< ETag: "2b400e-24fef-44c08855" <<< Accept-Ranges: bytes <<< Content-Length: 151535 <<< Content-Type: text/html; charset=iso-8859-1 <<< X-Cache: HIT from www.vg.no <<< Age: 37 <<< X-Cache: HIT from www.vg.no <<< Connection: close Here's a response from cache3.c.vgnett.no: <<< HTTP/1.0 200 OK <<< Date: Fri, 21 Jul 2006 07:58:20 GMT <<< Server: Apache/1.3.27 (Unix) <<< Cache-Control: max-age=900 <<< Expires: Fri, 21 Jul 2006 08:13:20 GMT <<< Last-Modified: Fri, 21 Jul 2006 07:55:01 GMT <<< ETag: "2b400e-24fef-44c08855" <<< Accept-Ranges: bytes <<< Content-Length: 151535 <<< Content-Type: text/html; charset=iso-8859-1 <<< Age: 38 <<< X-Cache: HIT from www.vg.no <<< Via: 1.0 cache3.c.vgnett.no:80 (squid/2.6.RC1-20060621) <<< Connection: close DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Fri Jul 21 08:29:04 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 21 Jul 2006 10:29:04 +0200 Subject: Purge References: <36315.1153427989@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Setting the TTL would be something as simple as: > > sub vcl_fetch { > if (req.url = "/") { > obj.ttl = 120; > } > } We can do regexp matching there as well, right? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From steinove at vg.no Fri Jul 21 08:39:01 2006 From: steinove at vg.no (Stein Ove Rosseland) Date: Fri, 21 Jul 2006 10:39:01 +0200 Subject: Purge In-Reply-To: References: <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> Message-ID: <44C092A5.50402@vg.no> Dag-Erling Sm?rgrav wrote: > "Anders Berg" writes: > >> Some FUD developed today when our frontpage editors sensed that their >> Purge tool did not function correctly :) Sorry about that, and the >> following phonecalls. >> > > Is this solely a Varnish issue? I did som checking when Stein Ove > called me: one of the squids (specifically, cache3.c.vgnett.no) seems > to be configured differently from the others, and seems to run > directly against leonora instead of going through the second level > squid. I also saw that you mentioned on IRC that one of the squids > wasn't on the purge list... > > > <<< Via: 1.0 cache3.c.vgnett.no:80 (squid/2.6.RC1-20060621) > <<< Connection: close > And that is correct, it is different, forgot that thats the one and only running squid 2.6. Stein Ove ***************************************************************** Denne fotnoten bekrefter at denne e-postmeldingen ble skannet av MailSweeper og funnet fri for virus. ***************************************************************** This footnote confirms that this email message has been swept by MailSweeper for the presence of computer viruses. ***************************************************************** From andersb at vgnett.no Fri Jul 21 09:01:55 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 21 Jul 2006 11:01:55 +0200 (CEST) Subject: Purge In-Reply-To: References: <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> Message-ID: <1913.193.69.165.4.1153472515.squirrel@denise.vg.no> > "Anders Berg" writes: >> Some FUD developed today when our frontpage editors sensed that their >> Purge tool did not function correctly :) Sorry about that, and the >> following phonecalls. > > Is this solely a Varnish issue? I did som checking when Stein Ove > called me: one of the squids (specifically, cache3.c.vgnett.no) seems > to be configured differently from the others, and seems to run > directly against leonora instead of going through the second level > squid. I also saw that you mentioned on IRC that one of the squids > wasn't on the purge list... As Stein-Ove writes, yes it's different because it's a Squid 2.6 installation. It was not on the purgelist yesterday, but that seldom is a problem since the Expire time is seldom or never actually honored by Squid (It refreshes more often that whats set in Expires and in the config file). :) So the front editors will not notice it as a "old" frontpage. Also cache3 is handeling less traffic than Varnish, so it's hit more seldom. That explains why the Varnish page was more visible as not honoring the purge the editors sent. Anders Berg > > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Fri Jul 21 09:11:36 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 21 Jul 2006 11:11:36 +0200 Subject: Purge References: <1201.193.69.165.4.1153427528.squirrel@denise.vg.no> <1913.193.69.165.4.1153472515.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > It was not on the purgelist yesterday, but that seldom is a problem since > the Expire time is seldom or never actually honored by Squid (It refreshes > more often that whats set in Expires and in the config file). :) So the > front editors will not notice it as a "old" frontpage. Also cache3 is > handeling less traffic than Varnish, so it's hit more seldom. > That explains why the Varnish page was more visible as not honoring the > purge the editors sent. OK. I'm glad we figured it out! DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Fri Jul 21 10:15:53 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 21 Jul 2006 10:15:53 +0000 Subject: Purge In-Reply-To: Your message of "Fri, 21 Jul 2006 10:29:04 +0200." Message-ID: <80535.1153476953@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Poul-Henning Kamp" writes: >> Setting the TTL would be something as simple as: >> >> sub vcl_fetch { >> if (req.url = "/") { >> obj.ttl = 120; >> } >> } > >We can do regexp matching there as well, right? Once we tech the runtime how to, yes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ssm at linpro.no Fri Jul 21 10:44:56 2006 From: ssm at linpro.no (Stig Sandbeck Mathisen) Date: Fri, 21 Jul 2006 12:44:56 +0200 Subject: Illustrations for Varnish Message-ID: <7x4pxbme3b.fsf@linpro.no> I have re-made some of the illustrations from the Varnish presentation and added those to the repo. If you'd like more illustrations, I can make those, but I'd need to know what would be useful. Illustrations can help people understand Varnish from different perspectives. I was thinking of a few illustrations on each of the following: * Technology perspective - illustrations that describe the logic, or the flow of internal traffic. What happens to a request. * Sysadmin perspective - illustrations that describe how Varnish behaves on the server(s) it is installed on. * Network perspective - illustrations that describe how Varnish interacts with the world around it. Of course, there is some overlapping here... And then we have: * Security perspective? (service availability is "security") * Perspective of enterprise level buzzword compliance - Leveraging the potential of dedicated open-source well-modulated horizontal architecture to enable quality-focused control over mission critical business-enabled infrastructure elements. Or someting. :D -- Stig Sandbeck Mathisen, Linpro From andersb at vgnett.no Fri Jul 21 12:08:38 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 21 Jul 2006 14:08:38 +0200 Subject: Illustrations for Varnish In-Reply-To: <7x4pxbme3b.fsf@linpro.no> References: <7x4pxbme3b.fsf@linpro.no> Message-ID: On Jul 21, 2006, at 12:44, Stig Sandbeck Mathisen wrote: > > I have re-made some of the illustrations from the Varnish presentation > and added those to the repo. If you'd like more illustrations, I can > make those, but I'd need to know what would be useful. Hi Stig, and welcome :) I have already done some of the work you mention, or at least started on them. > Illustrations can help people understand Varnish from different > perspectives. I was thinking of a few illustrations on each of the > following: > > * Technology perspective - illustrations that describe the logic, or > the flow of internal traffic. What happens to a request. This one is not correct, since it was a drat to get some undertstanding of how Varnish behaves. http://klikk.vg.no/varnish/img/application_structure.png > * Sysadmin perspective - illustrations that describe how Varnish > behaves on the server(s) it is installed on. > > * Network perspective - illustrations that describe how Varnish > interacts with the world around it. http://klikk.vg.no/varnish/img/varnish_http_accelerator.png > > Of course, there is some overlapping here... > > And then we have: > > * Security perspective? (service availability is "security") > > * Perspective of enterprise level buzzword compliance - Leveraging the > potential of dedicated open-source well-modulated horizontal > architecture to enable quality-focused control over mission critical > business-enabled infrastructure elements. Or someting. :D The design of the webpage is currently at: http://klikk.vg.no/varnish I have the original Illustrator graphics for the logo. We also have a smaller logo that fits better on the bottom of a doc/page. Think of it as the bottom right logo on this page: http:// projects.linpro.no/pipermail/varnish-dev/attachments/ 20060314/30091062/VarnishLogo01-0001.jpg but designed in the spirit of the main logo now. I suggest that the docs, when parsed by a DocBook->HTML or DocBook- >PDF script will end up with a design that reflects the website design. :) Anders Berg > -- > Stig Sandbeck Mathisen, Linpro > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From andersb at vgnett.no Fri Jul 21 13:41:29 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 21 Jul 2006 15:41:29 +0200 Subject: Dates Message-ID: <1E11C537-AAFA-4069-A551-9B20D0712E09@vgnett.no> Okay, it's time to agree on some dates. I will try to follow what DES and PHK wrote earlier. **** PHK **** Mid July (phk and/or des) Ad-Hoc supervised low-volume production at VGNETT Anders need to set the Alteon to a very low setting, but otherwise his involvement is not required. Alpha release. **** /PHK **** We have passed this stage, and it is still not sane to release a Alpha. **** PHK **** Late July (phk, anders & des) Coordinated supervised high-volume production at VGNETT Same formula as last time, only I stay in Denmark and we restrict to one day. Beta release. **** /PHK **** We are in this stage now, and _maybe_ endcode will be alpha material. No changes otherwise. We keep testing Varnish in our envionment and at 100 Mbit/s traffic. PHK is coding alot, doing changes all over the design, and getting better and better performance at the same time. We are getting great value of realtime/world traffic. We are "forced" to throw in some work on VCL because of purging. DES is starting to work on doc's together with a new "member", Stig. We can try to agree on a alpha released around 30. july when this "testingperiod" is considered stable enuff for it. **** PHK **** Early August Unsupervised low-volume production at VGETT later Unsupervised high-volume production at VGETT Anders makes the call on this one. **** /PHK **** I wanna micro this a bit more for my own part, since Aug is gonna be busy for me. First week (31-4) : PHK cleanes code and codes at will. DES and Stig continue their work at need. I will be of assistance and work on webpages. Second week (7-11) : We use this week to run unsupervised live on VG Nett, PHK does this at will. I am busy with network work at VG, but can be involved at need. Third week (14-18) : This week I will be busy installing SAN at VG. I will also get Gig equipment and try to install if possible. PHK cleans up after test. Week 4 (21-26) : I am getting married and will be totaly off-line. Nothing planned, but maybe Gig test if equipment is installed. Is this a good time for DES to start Linux port? PHK finishes cleanups and starts working on VCL and tools/admin. Week 5 (28-1) : Nothing planned. DES should have started Linux port. I have to get some work done I guess. **** PHK **** September Release 1.0 **** /PHK **** Yes, we are gonne try release v 1.0 in sept. The 3 first weeks we are gonna work on docs/port/website/VCL/tools/admin/polishing. Week 3 (11-15) : I am unavailable because I am in the middle of Finnmark hunting for small flying creatures. Week 4 (18-22) : We start to wrap things up, release v 1.0 should be done this week and website should be 100%. Week 5 (25-29) : We do the official release with press and releaseparty. :) I am gonna suggest that we invite loads of people to VG: Press, Admins (large sites in norway and sweden) and show the actual switch from Squid to Varnish in realtime on a big-screen, and hold a pressconference This should make an impression :) After that we party :) This is just my suggestion. How does this sound/look? Anders Berg From phk at phk.freebsd.dk Fri Jul 21 14:49:19 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 21 Jul 2006 14:49:19 +0000 Subject: Dates In-Reply-To: Your message of "Fri, 21 Jul 2006 15:41:29 +0200." <1E11C537-AAFA-4069-A551-9B20D0712E09@vgnett.no> Message-ID: <90301.1153493359@critter.freebsd.dk> In message <1E11C537-AAFA-4069-A551-9B20D0712E09 at vgnett.no>, Anders Berg writes : >This is just my suggestion. How does this sound/look? Sounds good to me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Jul 21 21:35:09 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 21 Jul 2006 21:35:09 +0000 Subject: Purge In-Reply-To: Your message of "Fri, 21 Jul 2006 09:58:56 +0200." <1213.193.69.165.4.1153468736.squirrel@denise.vg.no> Message-ID: <35090.1153517709@critter.freebsd.dk> In message <1213.193.69.165.4.1153468736.squirrel at denise.vg.no>, "Anders Berg" writes: >> sub vcl_fetch { >> if (req.url = "/") { >> obj.ttl = 120; >> } >> } Provided the correct syntax is used, and the default rules called afterwards, this now works. The syntax error in the above example is that the obj.ttl variable is a time variable, so the 120 needs a time unit as well. >> sub vcl_fetch { >> if (req.url = "/") { >> obj.ttl = 120s; >> } >> call default_vcl_fetch; >> } or >> sub vcl_fetch { >> if (req.url = "/") { >> obj.ttl = 2m; >> } >> call default_vcl_fetch; >> } Will both work now. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Sat Jul 22 10:05:14 2006 From: andersb at vgnett.no (Anders Berg) Date: Sat, 22 Jul 2006 12:05:14 +0200 (CEST) Subject: Purge In-Reply-To: <35090.1153517709@critter.freebsd.dk> References: Your message of "Fri, 21 Jul 2006 09:58:56 +0200." <1213.193.69.165.4.1153468736.squirrel@denise.vg.no> <35090.1153517709@critter.freebsd.dk> Message-ID: <1654.193.69.165.4.1153562714.squirrel@denise.vg.no> > In message <1213.193.69.165.4.1153468736.squirrel at denise.vg.no>, "Anders > Berg" > writes: > >>> sub vcl_fetch { >>> if (req.url = "/") { >>> obj.ttl = 120; >>> } >>> } > > Provided the correct syntax is used, and the default rules called > afterwards, this now works. > > The syntax error in the above example is that the obj.ttl variable > is a time variable, so the 120 needs a time unit as well. > >>> sub vcl_fetch { >>> if (req.url = "/") { >>> obj.ttl = 120s; >>> } >>> call default_vcl_fetch; >>> } > > or > >>> sub vcl_fetch { >>> if (req.url = "/") { >>> obj.ttl = 2m; >>> } >>> call default_vcl_fetch; >>> } > > Will both work now. Great. Will you put in a obj.ttl = 120s on "/" before next test? Also I have put weight=1 on the loadbalancer, so you are ready to go at will. Anders Berg > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > From andersb at vgnett.no Sat Jul 22 16:51:25 2006 From: andersb at vgnett.no (Anders Berg) Date: Sat, 22 Jul 2006 18:51:25 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> Message-ID: <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> > Sorry, I used a reply and the last mail ended up in the Logging thread. > Here it is again in it's own thread: > > Hi, > > I have worked some on the website material: > > http://klikk.vg.no/varnish/ > > the welcome page and featurelist page are the only ones with any content > so far. > > I am aware that the pages don't show right in Firefox and Safari, but I > asure you they work great in IE :) The designer is gonna come up with some > code that works better soon. > > I have more content available and will add more this week if I have time. I have done some editing on the webpage with adding and deleting on the structure: http://klikk.vg.no/varnish/ I have also added a forum with some Categories to see how it would work out. When I spoke with Stein on the phone last week we agreed that a forum might be the right way to interact with users, and keep the mailinglist for the developers and "power-users". We wanna reach different kinds of users, even the ones that don't know how to operate in a mailinglist :) My guess (and Stein's) is that they are more likely to pay for any kind of support. At the same time a forum is easier to refere to in may ways "Ahh, I remember I saw an example VCL file on that somewhere in the forum". Anyway, the forum will eventually have the same design as the webpage, right now the Lithium-design is the one closest to Varnish. I have used PunBB as a forum before, and have learned to like it alot, it's easy to catch up with, and has decent features, no frills. I will spend more time making webiste better next week, if my back gets better... Anders Berg From des at linpro.no Sat Jul 22 21:26:13 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 22 Jul 2006 23:26:13 +0200 Subject: Dates References: <1E11C537-AAFA-4069-A551-9B20D0712E09@vgnett.no> Message-ID: Anders Berg writes: > **** PHK **** > Mid July (phk and/or des) > Ad-Hoc supervised low-volume production at VGNETT > > Anders need to set the Alteon to a very low setting, > but otherwise his involvement is not required. > > Alpha release. > > **** /PHK **** > > We have passed this stage, and it is still not sane to release a Alpha. I agree. The code is a lot more solid in many areas than I expected it to be at this stage, but there are still white spots on the map. > **** PHK **** > Late July (phk, anders & des) > Coordinated supervised high-volume production at VGNETT > > Same formula as last time, only I stay in Denmark and > we restrict to one day. > > Beta release. > > **** /PHK **** > > We are in this stage now, and _maybe_ endcode will be alpha > material. No changes otherwise. We keep testing Varnish in our > envionment and at 100 Mbit/s traffic. I want to do the beta release in August, when VG starts using Varnish in production. The code will be mostly done, but we will need time for polish and documentation in order to be ready for a 1.0 release on September 12th. > DES is starting to work on doc's together with a new "member", Stig. > We can try to agree on a alpha released around 30. july when this > "testingperiod" is considered stable enuff for it. I'd suggest July 31st (a monday) or August 1st. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Jul 22 21:30:03 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 22 Jul 2006 23:30:03 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > I have done some editing on the webpage with adding and deleting on the > structure: http://klikk.vg.no/varnish/ OK, let's try to have the static site running on varnish.linpro.no by the end of the month. > I have also added a forum with some Categories to see how it would work > out. When I spoke with Stein on the phone last week we agreed that a forum > might be the right way to interact with users, and keep the mailinglist > for the developers and "power-users". I don't like web forums; I find them an extremely inefficient form of communication. I don't mind having them... as long as nobody expects me to participate in them. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sat Jul 22 21:39:57 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 22 Jul 2006 21:39:57 +0000 Subject: Website material (New thread) In-Reply-To: Your message of "Sat, 22 Jul 2006 23:30:03 +0200." Message-ID: <81468.1153604397@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Anders Berg" writes: >> I have done some editing on the webpage with adding and deleting on the >> structure: http://klikk.vg.no/varnish/ > >OK, let's try to have the static site running on varnish.linpro.no by >the end of the month. I have started tagging doc-fodder in the commit messages, I hope you guys find it useful ? >> I have also added a forum with some Categories [...] > >I don't like web forums; I find them an extremely inefficient form of >communication. I don't mind having them... as long as nobody expects >me to participate in them. I'm sort of in the same boat. I really hate to have to add another site to my list of daily bookmarks to visit. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sat Jul 22 22:03:57 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 00:03:57 +0200 Subject: Website material (New thread) In-Reply-To: <81468.1153604397@critter.freebsd.dk> (Poul-Henning Kamp's message of "Sat, 22 Jul 2006 21:39:57 +0000") References: <81468.1153604397@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I have started tagging doc-fodder in the commit messages, I hope you > guys find it useful ? Yes. I've tagged it in my mail box, and plan to start collating it the coming week. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sat Jul 22 22:15:19 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 22 Jul 2006 22:15:19 +0000 Subject: Dates In-Reply-To: Your message of "Sat, 22 Jul 2006 23:26:13 +0200." Message-ID: <81565.1153606519@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >> We have passed this stage, and it is still not sane to release a Alpha. > >I agree. The code is a lot more solid in many areas than I expected >it to be at this stage, but there are still white spots on the map. Here is a brain-dump status report in no particular order. It should probably be in the wiki somewhere... ----------------------------------------------------------------------- What: Error handling on backend connections Situation: Right now we more or less panic on any sign of trouble. What needs to be done: We need to decide how to tie this into VCL Errors must be handled, presumably by retry. Timeouts probably desirable as well. Severity: alpha: can live with beta: must be improved 1.0: backend trouble SHALL not kill varnish ----------------------------------------------------------------------- What: Move more policy to VCL Situation: There seems to be no reason why we can't move more policy from C code to VCL code. What needs to be done: Make sure VCL support the necessary operations Move policy to default VCL functions. Severity: alpha: can live with. beta: can live with. 1.0: can live with. ----------------------------------------------------------------------- What: VCL error implementation Situation: The error keyword in VCL is not implemented What needs to be done: Find out if this is really the right strategy, it looks more and more dubious to me. Maybe a redirect to an error document loaded with storage_tar would be more useful. Severity: alpha: can live with minimal improvement beta: can live with minimal improvement 1.0: can live with minimal improvement ----------------------------------------------------------------------- What: Vary header matching Situation: We don't do it. What needs to be done: Implement it. Now that we have the http munging facilities mostly in place this is possible. Test against www.cnn.com for instance. Severity: alpha: can live with beta: can live with 1.0: can live with ----------------------------------------------------------------------- What: Performance statistics Situation: We have some What needs to be done: We need to figure out what is relevant stats We need to collect it. We need to make (some of) it available in VCL Severity: alpha: can live with beta: improvement desirable 1.0: TBD ----------------------------------------------------------------------- What: CLI management interface Situation: Only minimally functional. What needs to be done: Reevaluate the list we created initially (ie: no unlisten) Implement TCP / SSH carried CLI port Severity: alpha: can live with beta: improvement desirable 1.0: TCP/SSH mandatory ----------------------------------------------------------------------- What: VCL completeness Situation: VCL can't do much yet. What needs to be done: Go through our list, implement in proritized order Severity: alpha: can live with beta: HTTP header manipulation desirable 1.0: TBD ----------------------------------------------------------------------- What: VCL compiler error checking Situation: Errors from GCC and at load-time still happens. Failed loading of default/boot VCL is not currently fatal. What needs to be done: Check conditions better, for instance do getaddrinfo() on ACL targets and warn if non () entries fail. Severity: alpha: can live with beta: can live with 1.0: can live with ----------------------------------------------------------------------- What: (regression-)testing Situation: We need a way to build regression tests. "varnishtester" is maybe a beginning. What needs to be done: 1. (see above). 2. Think about problem 3. Write down code :-) Severity: alpha: can live with beta: would rather not live with 1.0: serious limitation ----------------------------------------------------------------------- What: Random memory corruption Situation: Most live test runs have seen one or two core dumps that look like pointer fandango after en hour or so. FlexeLint does not spot anything. What needs to be done: Run with debugging malloc of some kind, try to find. Severity: alpha: can live with if no choice beta: would really hate to live with 1.0: showstopper, we should be able to run 24h ----------------------------------------------------------------------- What: cache process restart Situation: management process does not restart automatically (for debugging convenience) What needs to be done: management process should restart cache process. Bonus: save snapshot of shmem and get traceback from gdb as well. Severity: alpha: can live with beta: can live with 1.0: should have ----------------------------------------------------------------------- What: RFC compliance audit Situation: We don't know how many small details we are missing. What needs to be done: Go through RFC(s) and sourcecode and mark up RFC(s) with "not relevant", "deliberate variance", "not implemented" and "OK". (OK preferably listing relevant testcases) Severity: alpha: can live with beta: can live with 1.0: can live with ----------------------------------------------------------------------- What: storage_file.c allocation strategy Situation: we have seen pessimisal allocation patterns with degenerate trafic patterns. Real World traffic looks basically OK, but no detailed analysis done. Missing stats an issue. What needs to be done: Carefully consider mitigation once we have analyzed the issue. Severity: alpha: no issue beta: no issue 1.0: no issue ----------------------------------------------------------------------- Nice to have: log-tailer which only shows entries which pertain to sessions from given IP# range. (possibly generic log-tailer option ?) ----------------------------------------------------------------------- Nice to have: "pigs" logtailer which shows top clients and their requests in usable format. ----------------------------------------------------------------------- Nice to have: log-tailer which writes httpd format. ----------------------------------------------------------------------- Nice to have: Better stats presentation on screen ----------------------------------------------------------------------- Nice to have: Allow purging through HTTP interface. Doesn't take much actually ----------------------------------------------------------------------- Nice to have: TAR storage method for loading static contents. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Sat Jul 22 22:35:11 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 00:35:11 +0200 (CEST) Subject: Dates In-Reply-To: References: <1E11C537-AAFA-4069-A551-9B20D0712E09@vgnett.no> Message-ID: <1524.193.213.34.102.1153607711.squirrel@denise.vg.no> > Anders Berg writes: >> **** PHK **** > > I want to do the beta release in August, when VG starts using Varnish > in production. The code will be mostly done, but we will need time > for polish and documentation in order to be ready for a 1.0 release on > September 12th. The second part differs some from my suggestion: Week 4 (18-22) : We start to wrap things up, release v 1.0 should be done this week and website should be 100%. although we could have a 1.0 release ready for the NUUG and while I am gone, we are not gonna "release it" to public before this week I hope. If we could ask NUUG to push it one week, that would conside better. >> DES is starting to work on doc's together with a new "member", Stig. >> We can try to agree on a alpha released around 30. july when this >> "testingperiod" is considered stable enuff for it. > > I'd suggest July 31st (a monday) or August 1st. Sure. What ever works for you guys. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Sat Jul 22 22:49:14 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 00:49:14 +0200 Subject: Dates References: <81565.1153606519@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Here is a brain-dump status report in no particular order. > > It should probably be in the wiki somewhere... No, this should be in the ticket system. I'll take care of it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Sat Jul 22 23:40:19 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 01:40:19 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no><1947.193.213.34.102.1153587085.squirrel@denise.vg.no> Message-ID: <1742.193.213.34.102.1153611619.squirrel@denise.vg.no> > "Anders Berg" writes: >> I have done some editing on the webpage with adding and deleting on the >> structure: http://klikk.vg.no/varnish/ > > OK, let's try to have the static site running on varnish.linpro.no by > the end of the month. That's a simple copy job. A bit worse if you mean that all the content should be done :) I was thinking of registering a domain this week, we talked about some at one stage or another, I am gonna check their awailability. Also we need to find a small CMS system for the website. Open for suggestions, remember this may also have to live on server at Linpro, and the forum software. >> I have also added a forum with some Categories to see how it would work >> out. When I spoke with Stein on the phone last week we agreed that a >> forum >> might be the right way to interact with users, and keep the mailinglist >> for the developers and "power-users". > > I don't like web forums; I find them an extremely inefficient form of > communication. I don't mind having them... as long as nobody expects > me to participate in them. Okay, I understand what, and why, you and Poul-Henning hold against forums. As developers, mailinglists are the best way to communicate. The situation is a bit different for many possible users. I see 2 ways to solve this for the benefit of both groups. 1. Forum sends daily digestes to mailinggroup, or for each post. (may have to be developed, but one can use a "subscribe" to each forum I think). 2. We get a good community going, and appropriate moderators in the forum. They again gather the stuff that needs to go to the dev group, and post it. They also function as FAQ updaters. This can be done in many ways. Any thoughts on this? Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Sun Jul 23 07:25:55 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 09:25:55 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <1742.193.213.34.102.1153611619.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > "Dag-Erling Sm?rgrav" writes: > > OK, let's try to have the static site running on varnish.linpro.no by > > the end of the month. > That's a simple copy job. A bit worse if you mean that all the content > should be done :) Don't worry about the content, that's my responsibility. > I was thinking of registering a domain this week, we > talked about some at one stage or another, I am gonna check their > awailability. I think varnish.linpro.no should do for now, don't you? > Also we need to find a small CMS system for the website. > Open for suggestions, remember this may also have to live on server at > Linpro, and the forum software. Linpro has a close relationship with eZ Systems, so eZ Publish is a natural choice from my point of view. I can set up a test site that you and your designer can play with before we go live. > Okay, I understand what, and why, you and Poul-Henning hold against > forums. As developers, mailinglists are the best way to communicate. The > situation is a bit different for many possible users. I see 2 ways to > solve this for the benefit of both groups. I understand, and I may keep track of the forums in an initial phase to help them gain momentum, but in the long run I'd prefer not to be involved. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sun Jul 23 07:48:37 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 23 Jul 2006 07:48:37 +0000 Subject: Website material (New thread) In-Reply-To: Your message of "Sun, 23 Jul 2006 09:25:55 +0200." Message-ID: <83786.1153640917@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= writes: >> I was thinking of registering a domain this week, we >> talked about some at one stage or another, I am gonna check their >> awailability. > >I think varnish.linpro.no should do for now, don't you? Which domain were you thinking of ? squid.no ? :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sun Jul 23 08:08:47 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 23 Jul 2006 08:08:47 +0000 Subject: 10 hour live-test Message-ID: <83860.1153642127@critter.freebsd.dk> Varnish has run overnight at Alteon weight=1. Here is a snapshot of relevant stats at 10 hours: 10:00:00 Hitrate ratio: 10 100 1000 Hitrate avg: 0.9604 0.9763 0.9313 99678 4.00 2.77 Client connections accepted 787265 36.96 21.87 Client requests received 748108 35.96 20.78 Cache hits 35 0.00 0.00 Cache hits for pass 36981 1.00 1.03 Cache misses 39180 1.00 1.09 Backend connections initiated 18215 0.00 0.51 N struct object 17226 0.00 0.48 N struct objecthead 19971 2.00 0.55 N struct smf 20218 0.00 0.56 N expired objects 36 0.00 0.00 HTTP header overflows I need to look into these 99499 4.00 2.76 Total Sessions 787263 35.96 21.87 Total Requests 53 0.00 0.00 Total pipe 2196 0.00 0.06 Total pass 36894 0.00 1.02 Total fetch 17262774889 8164.48 479521.52 Total header bytes 3709537800 208307.57 103042.72 Total body bytes I find it interesting that we have sent almost five times as many headerbytes as bodybytes. I wonder if somebody did a HEAD on the entire site overnight. Process RSS size was 439MB -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sun Jul 23 08:34:42 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 23 Jul 2006 08:34:42 +0000 Subject: 10 hour live-test In-Reply-To: Your message of "Sun, 23 Jul 2006 08:08:47 GMT." <83860.1153642127@critter.freebsd.dk> Message-ID: <83939.1153643682@critter.freebsd.dk> In message <83860.1153642127 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: > 36 0.00 0.00 HTTP header overflows >I need to look into these This seems to be opera doing heavy pipe-lining. I'll think about that one a bit. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Sun Jul 23 10:16:14 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 12:16:14 +0200 Subject: 10 hour live-test In-Reply-To: <83860.1153642127@critter.freebsd.dk> References: <83860.1153642127@critter.freebsd.dk> Message-ID: On Jul 23, 2006, at 10:08, Poul-Henning Kamp wrote: > > Varnish has run overnight at Alteon weight=1. > > Here is a snapshot of relevant stats at 10 hours: > > 10:00:00 > Hitrate ratio: 10 100 1000 > Hitrate avg: 0.9604 0.9763 0.9313 > > 99678 4.00 2.77 Client connections accepted > 787265 36.96 21.87 Client requests received > 748108 35.96 20.78 Cache hits > 35 0.00 0.00 Cache hits for pass > 36981 1.00 1.03 Cache misses > 39180 1.00 1.09 Backend connections initiated > 18215 0.00 0.51 N struct object > 17226 0.00 0.48 N struct objecthead > 19971 2.00 0.55 N struct smf > 20218 0.00 0.56 N expired objects > 36 0.00 0.00 HTTP header overflows > I need to look into these > > 99499 4.00 2.76 Total Sessions > 787263 35.96 21.87 Total Requests > 53 0.00 0.00 Total pipe > 2196 0.00 0.06 Total pass > 36894 0.00 1.02 Total fetch > 17262774889 8164.48 479521.52 Total header bytes > 3709537800 208307.57 103042.72 Total body bytes > > I find it interesting that we have sent almost five times as many > headerbytes as bodybytes. I wonder if somebody did a HEAD on the > entire site overnight. That was alot of head's :) Though it could be that we get browsers to only send IMS's, like they really should when the content is not new. I guess a quick look in the logfiles will show what has happend. Anders Berg > Process RSS size was 439MB > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From des at linpro.no Sun Jul 23 10:25:46 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 12:25:46 +0200 Subject: Website material (New thread) In-Reply-To: <83786.1153640917@critter.freebsd.dk> (Poul-Henning Kamp's message of "Sun, 23 Jul 2006 07:48:37 +0000") References: <83786.1153640917@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Which domain were you thinking of ? squid.no ? :-) *grin* I can easily get varnish.no - in fact, I might as well grab it right away before someone else does... but I was thinking more along the lines of varnish-cache.org (to mirror squid-cache.org), since varnish.{com,net,org} are taken. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Sun Jul 23 10:30:22 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 12:30:22 +0200 Subject: Website material (New thread) In-Reply-To: References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <1742.193.213.34.102.1153611619.squirrel@denise.vg.no> Message-ID: <730D7C67-E465-4ABE-B124-319C94D87188@vgnett.no> On Jul 23, 2006, at 9:25, Dag-Erling Sm?rgrav wrote: > "Anders Berg" writes: >> "Dag-Erling Sm?rgrav" writes: >>> OK, let's try to have the static site running on >>> varnish.linpro.no by >>> the end of the month. >> That's a simple copy job. A bit worse if you mean that all the >> content >> should be done :) > > Don't worry about the content, that's my responsibility. > >> I was thinking of registering a domain this >> week, we >> talked about some at one stage or another, I am gonna check their >> awailability. > > I think varnish.linpro.no should do for now, don't you? Sure but we will want a URL that is easy in the long run anyway. www.varnsh-cache.org is a good domain that is available. squid.no :) Would be fun, but maybe not so nice :) >> Also we need to find a small CMS system for the >> website. >> Open for suggestions, remember this may also have to live on >> server at >> Linpro, and the forum software. > > Linpro has a close relationship with eZ Systems, so eZ Publish is a > natural choice from my point of view. I can set up a test site that > you and your designer can play with before we go live. I talked to Stein about it. He meant that eZ was shooting a spurrow with a cannon, and I agree. I tried installing eZ on klikk.vg.no but for some reason I keep getting a bug that should have been solved months ago. Php 5.0 is not supported. Considering that we could get away with only having the News part of the site under a easy CMS, maybe that is the way we should go. One could even use Blogger (supports FTP) and do the design in css and autoformat it (just an example of how simple it could be done). I have not searched freshmeat or sf for a small system, but I am guessing a dime a dozen. >> Okay, I understand what, and why, you and Poul-Henning hold against >> forums. As developers, mailinglists are the best way to >> communicate. The >> situation is a bit different for many possible users. I see 2 ways to >> solve this for the benefit of both groups. > > I understand, and I may keep track of the forums in an initial phase > to help them gain momentum, but in the long run I'd prefer not to be > involved. Sure. So alternative number 2 could work. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From andersb at vgnett.no Sun Jul 23 10:40:52 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 12:40:52 +0200 Subject: Website material (New thread) In-Reply-To: References: <83786.1153640917@critter.freebsd.dk> Message-ID: On Jul 23, 2006, at 12:25, Dag-Erling Sm?rgrav wrote: > "Poul-Henning Kamp" writes: >> Which domain were you thinking of ? squid.no ? :-) > > *grin* > > I can easily get varnish.no - in fact, I might as well grab it right > away before someone else does... but I was thinking more along the > lines of varnish-cache.org (to mirror squid-cache.org), since > varnish.{com,net,org} are taken. I think we should not bother with varnish.no. It will never be used. I am just saying so because I used a couple of weeks last year cleaning up all the domains VG had registered, and alot of those "this is close, we should get that one also...", "lets take the .no also" domains went down the drain without ever beeing used. No part of Varnish is especially norwegian, but maybe Linpro wants to focus on this market segment and therefor can use the .no? Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From des at linpro.no Sun Jul 23 10:45:39 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 12:45:39 +0200 Subject: Website material (New thread) References: <83786.1153640917@critter.freebsd.dk> Message-ID: Anders Berg writes: > I think we should not bother with varnish.no. It will never be used. > I am just saying so because I used a couple of weeks last year > cleaning up all the domains VG had registered, and alot of those > "this is close, we should get that one also...", "lets take the .no > also" domains went down the drain without ever beeing used. No part > of Varnish is especially norwegian, but maybe Linpro wants to focus > on this market segment and therefor can use the .no? Not at all - Varnish is an international product. I thought of it simply because it allows us to have "varnish dot something" DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sun Jul 23 10:49:03 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 12:49:03 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <1742.193.213.34.102.1153611619.squirrel@denise.vg.no> <730D7C67-E465-4ABE-B124-319C94D87188@vgnett.no> Message-ID: Anders Berg writes: > I talked to Stein about it. He meant that eZ was shooting a spurrow > with a cannon, and I agree. I tried installing eZ on klikk.vg.no but > for some reason I keep getting a bug that should have been solved > months ago. Php 5.0 is not supported. OK, how about Drupal? It's a fairly lightweight CMS with which I have some experience. > Considering that we could get away with only having the News part of > the site under a easy CMS, maybe that is the way we should go. One > could even use Blogger (supports FTP) and do the design in css and > autoformat it (just an example of how simple it could be done). I > have not searched freshmeat or sf for a small system, but I am > guessing a dime a dozen. See cmsmatrix.org (there seems to be something wrong with their front page right now, but they're in Google's cache - top hit for "cms matrix") DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Sun Jul 23 11:31:48 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 13:31:48 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: References: <83786.1153640917@critter.freebsd.dk> Message-ID: <1175.193.213.34.102.1153654308.squirrel@denise.vg.no> > Anders Berg writes: >> I think we should not bother with varnish.no. It will never be used. >> I am just saying so because I used a couple of weeks last year >> cleaning up all the domains VG had registered, and alot of those >> "this is close, we should get that one also...", "lets take the .no >> also" domains went down the drain without ever beeing used. No part >> of Varnish is especially norwegian, but maybe Linpro wants to focus >> on this market segment and therefor can use the .no? > > Not at all - Varnish is an international product. I thought of it > simply because it allows us to have "varnish dot something" Ahh, okay. Thats a good point, but will people remember .no? Or is it better with varnish-cache.org? Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From andersb at vgnett.no Sun Jul 23 11:32:35 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 13:32:35 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no><1947.193.213.34.102.1153587085.squirrel@denise.vg.no><1742.193.213.34.102.1153611619.squirrel@denise.vg.no><730D7C67-E465-4ABE-B124-319C94D87188@vgnett.no> Message-ID: <1179.193.213.34.102.1153654355.squirrel@denise.vg.no> > Anders Berg writes: >> I talked to Stein about it. He meant that eZ was shooting a spurrow >> with a cannon, and I agree. I tried installing eZ on klikk.vg.no but >> for some reason I keep getting a bug that should have been solved >> months ago. Php 5.0 is not supported. > > OK, how about Drupal? It's a fairly lightweight CMS with which I have > some experience. Sounds good. Maybe thats the right tool for the job. >> Considering that we could get away with only having the News part of >> the site under a easy CMS, maybe that is the way we should go. One >> could even use Blogger (supports FTP) and do the design in css and >> autoformat it (just an example of how simple it could be done). I >> have not searched freshmeat or sf for a small system, but I am >> guessing a dime a dozen. > > See cmsmatrix.org (there seems to be something wrong with their front > page right now, but they're in Google's cache - top hit for "cms > matrix") Thx, good link. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Sun Jul 23 13:03:35 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 23 Jul 2006 15:03:35 +0200 Subject: Website material (New thread) References: <83786.1153640917@critter.freebsd.dk> <1175.193.213.34.102.1153654308.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > Ahh, okay. Thats a good point, but will people remember .no? > Or is it better with varnish-cache.org? We should probably go for varnish-cache.org. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Sun Jul 23 14:42:43 2006 From: andersb at vgnett.no (Anders Berg) Date: Sun, 23 Jul 2006 16:42:43 +0200 Subject: Website material (New thread) In-Reply-To: References: <83786.1153640917@critter.freebsd.dk> <1175.193.213.34.102.1153654308.squirrel@denise.vg.no> Message-ID: On Jul 23, 2006, at 15:03, Dag-Erling Sm?rgrav wrote: > "Anders Berg" writes: >> Ahh, okay. Thats a good point, but will people remember .no? >> Or is it better with varnish-cache.org? > > We should probably go for varnish-cache.org. Agree. I'll see what I can do tomorrow. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From ssm at linpro.no Mon Jul 24 05:28:38 2006 From: ssm at linpro.no (Stig Sandbeck Mathisen) Date: Mon, 24 Jul 2006 07:28:38 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> Message-ID: <7xfygrlgft.fsf@linpro.no> des at linpro.no (Dag-Erling Sm?rgrav) writes: > I don't like web forums; I find them an extremely inefficient form > of communication. I don't mind having them... as long as nobody > expects me to participate in them. phpBB, while a larger can of worms than punbb, features usenet integration. Example for no.fritid.slektsforskning.it at a genealogy site: http://www.disnorge.no/slektsforum/viewforum.php?f=2110 -- Stig Sandbeck Mathisen, Linpro From andersb at vgnett.no Mon Jul 24 08:15:33 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 24 Jul 2006 10:15:33 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: <7xfygrlgft.fsf@linpro.no> References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no><1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <7xfygrlgft.fsf@linpro.no> Message-ID: <1674.193.69.165.4.1153728933.squirrel@denise.vg.no> > des at linpro.no (Dag-Erling Sm??rgrav) writes: > >> I don't like web forums; I find them an extremely inefficient form >> of communication. I don't mind having them... as long as nobody >> expects me to participate in them. > > phpBB, while a larger can of worms than punbb, features usenet > integration. > > Example for no.fritid.slektsforskning.it at a genealogy site: > http://www.disnorge.no/slektsforum/viewforum.php?f=2110 phpBB is such a larger can of worms that I would hesitate to use it. phpBB has been defaced time after time, so much that it must have been designed wrong. I found this little add-on for punBB. http://www.punres.org/desc.php?pid=169 Maybe that could do the trick if we go for solution #1. I would rather go for solution #2, find moderators. But that will take some time to sort out. Anders Berg > -- > Stig Sandbeck Mathisen, Linpro > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Mon Jul 24 09:55:05 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 24 Jul 2006 11:55:05 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <7xfygrlgft.fsf@linpro.no> <1674.193.69.165.4.1153728933.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > phpBB is such a larger can of worms that I would hesitate to use it. phpBB > has been defaced time after time, so much that it must have been designed > wrong. > > I found this little add-on for punBB. I'd rather just stick with Drupal. It may not be very advanced, but it has a little of everything. Very basic install: DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Mon Jul 24 12:39:51 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 24 Jul 2006 14:39:51 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no><1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <7xfygrlgft.fsf@linpro.no><1674.193.69.165.4.1153728933.squirrel@denise.vg.no> Message-ID: <1183.193.213.34.102.1153744791.squirrel@denise.vg.no> > "Anders Berg" writes: >> phpBB is such a larger can of worms that I would hesitate to use it. >> phpBB >> has been defaced time after time, so much that it must have been >> designed >> wrong. >> >> I found this little add-on for punBB. > > I'd rather just stick with Drupal. It may not be very advanced, but > it has a little of everything. > > Very basic install: Thx, I'll take a look at it. The discussion over was about how to get posts in the forum to email. Unless you meant using a forum plugin to Drupal? My guess would be that such a forum would not a good and tried out like punBB og phpBB. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From des at linpro.no Mon Jul 24 13:03:49 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 24 Jul 2006 15:03:49 +0200 Subject: Website material (New thread) References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no> <1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <7xfygrlgft.fsf@linpro.no> <1674.193.69.165.4.1153728933.squirrel@denise.vg.no> <1183.193.213.34.102.1153744791.squirrel@denise.vg.no> Message-ID: "Anders Berg" writes: > The discussion over was about how to get posts in the forum to email. > Unless you meant using a forum plugin to Drupal? My guess would be that > such a forum would not a good and tried out like punBB og phpBB. Drupal has built-in discussion boards, and is one of the most widely used open source CMS systems; big enough to have its own annual conference. I trust it a lot more than I trust anything with BB in its name. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Mon Jul 24 18:06:29 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 24 Jul 2006 20:06:29 +0200 (CEST) Subject: Website material (New thread) In-Reply-To: References: <1757.193.213.34.102.1152565565.squirrel@denise.vg.no><1947.193.213.34.102.1153587085.squirrel@denise.vg.no> <7xfygrlgft.fsf@linpro.no><1674.193.69.165.4.1153728933.squirrel@denise.vg.no><1183.193.213.34.102.1153744791.squirrel@denise.vg.no> Message-ID: <2416.193.213.34.102.1153764389.squirrel@denise.vg.no> > "Anders Berg" writes: >> The discussion over was about how to get posts in the forum to email. >> Unless you meant using a forum plugin to Drupal? My guess would be that >> such a forum would not a good and tried out like punBB og phpBB. > > Drupal has built-in discussion boards, and is one of the most widely > used open source CMS systems; big enough to have its own annual > conference. I trust it a lot more than I trust anything with BB in > its name. Hmm, we are somewhat divided yet. I guess it all boils down to what lvl of involvment we want the CMS to have. I see 2 approaches. 1. Very "light" site. Mostly static content. Other sections than News is static. Links in the doc section is to a page that again links to static content. http://www.varnish-cache.org/doc/release_document.html and http://www.varnish-cache.org/doc/old/release_document_1_11.html type of content. The news section controlled by something like: http://opensourcecms.com/index.php?option=content&task=view&id=2170&Itemid=159 and a simple yet powerful forum. I have been running a punBB installation for the good part of a year (http://klikk.vg.no/blotslauget/forum/ 7100 posts), and I have grown to like it alot. Its funtionality is easy and powerful both for daily or sporadic use. I would trust it a long way since the codebase is small and mature, without any big breaches so far. Read more about it at: http://www.punbb.org/about.php 2. Very "CMS" based site. Every section is somehow connected to the main system. It delivers all aspects of the site. Forum, possible FAQ, possible issue integration. Updating any part of the site is done through a CMS system. Drupal can possibly deliver what is needed, but certain things must be adjusted to fit, and implementation is not "instant". I have checked out the forum in Drupal, and I understand why you don't like forums Dag-Erling :) While pleasant to look at, I find that it somehow lacks power and depth and is clearly a forum attached to a CMS in my pow. However it is more "integrated" into software development and release as: http://drupal.org/node/74395 and http://drupal.org/node/3060/release and http://drupal.org/node/6315 show I think using Drupal for just the newspage (overly complex) and forum part (not as good as alternatives) of varnish-cache.org would demand to much work for to little gain. Going for full integration could be done, but I fear we will have to put some (many?) houres into it, and it's updates. I read: http://drupal.org/node/73159 and it confirms that it's a framework also. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From andersb at vgnett.no Tue Jul 25 12:39:35 2006 From: andersb at vgnett.no (Anders Berg) Date: Tue, 25 Jul 2006 14:39:35 +0200 (CEST) Subject: Exciting Message-ID: <3332.193.213.34.102.1153831175.squirrel@denise.vg.no> http://blogs.zdnet.com/Google/?p=271 fun to see what Greg Stein can announce in a couple of days. Anders Berg From andersb at vgnett.no Fri Jul 28 12:04:13 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 28 Jul 2006 14:04:13 +0200 Subject: Dates In-Reply-To: <81565.1153606519@critter.freebsd.dk> References: <81565.1153606519@critter.freebsd.dk> Message-ID: On Jul 23, 2006, at 0:15, Poul-Henning Kamp wrote: > In message , Dag-Erling =?iso-8859-1? > Q?Sm=F8rgra > v?= writes: > >>> We have passed this stage, and it is still not sane to release a >>> Alpha. >> >> I agree. The code is a lot more solid in many areas than I expected >> it to be at this stage, but there are still white spots on the map. > > Here is a brain-dump status report in no particular order. > > It should probably be in the wiki somewhere... > > ---------------------------------------------------------------------- > - > What: > Error handling on backend connections > Situation: > Right now we more or less panic on any sign of trouble. > What needs to be done: > We need to decide how to tie this into VCL > Errors must be handled, presumably by retry. > Timeouts probably desirable as well. > Severity: > alpha: can live with > beta: must be improved > 1.0: backend trouble SHALL not kill varnish > Agree. I guess this is also tied to how Varnish handles "error/fault" on backend as well. > ---------------------------------------------------------------------- > - > What: > Move more policy to VCL > Situation: > There seems to be no reason why we can't move more policy > from C code to VCL code. > What needs to be done: > Make sure VCL support the necessary operations > Move policy to default VCL functions. > Severity: > alpha: can live with. > beta: can live with. > 1.0: can live with. Agree. > ---------------------------------------------------------------------- > - > What: > VCL error implementation > Situation: > The error keyword in VCL is not implemented > What needs to be done: > Find out if this is really the right strategy, it looks > more and more dubious to me. > Maybe a redirect to an error document loaded with storage_tar > would be more useful. > Severity: > alpha: can live with minimal improvement > beta: can live with minimal improvement > 1.0: can live with minimal improvement Not sure what this one means. Is this the error we return to client? Or is this the error from backend? By that I mean: if (something true){ error; } or a if (backend.error){ do something; } > ---------------------------------------------------------------------- > - > What: > Vary header matching > Situation: > We don't do it. > What needs to be done: > Implement it. > Now that we have the http munging facilities mostly > in place this is possible. > Test against www.cnn.com for instance. > Severity: > alpha: can live with > beta: can live with > 1.0: can live with Agree > ---------------------------------------------------------------------- > - > What: > Performance statistics > Situation: > We have some > What needs to be done: > We need to figure out what is relevant stats > We need to collect it. > We need to make (some of) it available in VCL > Severity: > alpha: can live with > beta: improvement desirable > 1.0: TBD Agree. Stats that should be visible in VCL would from the top of my mind be. Hits on documents. (Not sure about if we can implement this, but I am thinking in the line of 20.000 hits on /index.html last 2 min and ttl +2m. It certainly requires us to keep stats on each document/URL or a top-hit-list, and that may not be possible.) Backend responstime. Client bandwith/hits. > ---------------------------------------------------------------------- > - > What: > CLI management interface > Situation: > Only minimally functional. > What needs to be done: > Reevaluate the list we created initially (ie: no unlisten) > Implement TCP / SSH carried CLI port > Severity: > alpha: can live with > beta: improvement desirable > 1.0: TCP/SSH mandatory "Reevaluate the list we created initially (ie: no unlisten" ? Agree on the rest. > ---------------------------------------------------------------------- > - > What: > VCL completeness > Situation: > VCL can't do much yet. > What needs to be done: > Go through our list, implement in proritized order > Severity: > alpha: can live with > beta: HTTP header manipulation desirable > 1.0: TBD Agree. Should I prioritize? > ---------------------------------------------------------------------- > - > What: > VCL compiler error checking > Situation: > Errors from GCC and at load-time still happens. > Failed loading of default/boot VCL is not currently fatal. > What needs to be done: > Check conditions better, for instance do getaddrinfo() on > ACL targets and warn if non () entries fail. > Severity: > alpha: can live with > beta: can live with > 1.0: can live with > Agree. We at least need getaddrinfo on the backend IP's, but I guess that this is meant on other cases? > ---------------------------------------------------------------------- > - > What: > (regression-)testing > Situation: > We need a way to build regression tests. > "varnishtester" is maybe a beginning. > What needs to be done: > 1. (see above). 2. Think about problem 3. Write down code :-) > Severity: > alpha: can live with > beta: would rather not live with > 1.0: serious limitation Agree. The thing is that this would be a good tool for people writing code, to see that their implementation is correct. Helps debug alot I guess. > ---------------------------------------------------------------------- > - > What: > Random memory corruption > Situation: > Most live test runs have seen one or two core dumps > that look like pointer fandango after en hour or so. > FlexeLint does not spot anything. > What needs to be done: > Run with debugging malloc of some kind, try to find. > Severity: > alpha: can live with if no choice > beta: would really hate to live with > 1.0: showstopper, we should be able to run 24h Agree. > ---------------------------------------------------------------------- > - > What: > cache process restart > Situation: > management process does not restart automatically > (for debugging convenience) > What needs to be done: > management process should restart cache process. > Bonus: save snapshot of shmem and get traceback > from gdb as well. > Severity: > alpha: can live with > beta: can live with > 1.0: should have Agree. I liked the shmem snap. It would help debugging alot I guess. > ---------------------------------------------------------------------- > - > What: > RFC compliance audit > Situation: > We don't know how many small details we are missing. > What needs to be done: > Go through RFC(s) and sourcecode and mark up RFC(s) > with "not relevant", "deliberate variance", "not implemented" > and "OK". (OK preferably listing relevant testcases) > Severity: > alpha: can live with > beta: can live with > 1.0: can live with Bump to must/should have on 1.0? We are gonna need this sooner rather than later. Ppl are gonna ask for this as soon as they start using Varnish in a prod. envorionment is my guess. > ---------------------------------------------------------------------- > - > What: > storage_file.c allocation strategy > Situation: > we have seen pessimisal allocation patterns with > degenerate trafic patterns. > Real World traffic looks basically OK, but no detailed > analysis done. Missing stats an issue. > What needs to be done: > Carefully consider mitigation once we have analyzed > the issue. > Severity: > alpha: no issue > beta: no issue > 1.0: no issue Agree. 1.0 should have a way to dump the stats, so users reporting an error in that area, can give us a clue as to what is happening. > ---------------------------------------------------------------------- > - > Nice to have: > log-tailer which only shows entries which pertain to > sessions from given IP# range. (possibly generic > log-tailer option ?) Agree. > ---------------------------------------------------------------------- > - > Nice to have: > "pigs" logtailer which shows top clients and their > requests in usable format. Agree. > ---------------------------------------------------------------------- > - > Nice to have: > log-tailer which writes httpd format. I wanna bump up this one. Ppl may need it for stats. > ---------------------------------------------------------------------- > - > Nice to have: > Better stats presentation on screen Agree. > ---------------------------------------------------------------------- > - > Nice to have: > Allow purging through HTTP interface. > Doesn't take much actually Agree. > ---------------------------------------------------------------------- > - > Nice to have: > TAR storage method for loading static contents. Agree. I know that bumps upwards are gonna resolve on more work, but a httpd format log is really a must-have I think. Not for VG, but for others. I am currently looking into seeing if I can program any C at all :) httpd logg would be a good place to start I guess. A mail will follow after this one about the logging code. Anders Berg > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > From andersb at vgnett.no Fri Jul 28 12:06:16 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 28 Jul 2006 14:06:16 +0200 Subject: Varnishlog.c Message-ID: Hi, I took a look at: http://varnish.projects.linpro.no/browser/trunk/varnish-cache/bin/ varnishlog/varnishlog.c Line 132 sould be: if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 4) I guess, rather than: if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) Not sure since I have only a vague clue as to what the code does, but the pattern suggests it :) Also, I am scratching my head about the -h parameter in the code line: 214. Is this supposed to be the -r param? Anders Berg From phk at phk.freebsd.dk Fri Jul 28 12:28:30 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 28 Jul 2006 12:28:30 +0000 Subject: Dates In-Reply-To: Your message of "Fri, 28 Jul 2006 14:04:13 +0200." Message-ID: <24801.1154089710@critter.freebsd.dk> In message , Anders Berg writes : >> ---------------------------------------------------------------------- >> - >> What: >> Error handling on backend connections >> Situation: >> Right now we more or less panic on any sign of trouble. >> What needs to be done: >> We need to decide how to tie this into VCL >> Errors must be handled, presumably by retry. >> Timeouts probably desirable as well. >> Severity: >> alpha: can live with >> beta: must be improved >> 1.0: backend trouble SHALL not kill varnish >> > >Agree. > >I guess this is also tied to how Varnish handles "error/fault" on >backend as well. yes. >> ---------------------------------------------------------------------- >> - >> What: >> VCL error implementation >> Situation: >> The error keyword in VCL is not implemented >> What needs to be done: >> Find out if this is really the right strategy, it looks >> more and more dubious to me. >> Maybe a redirect to an error document loaded with storage_tar >> would be more useful. >> Severity: >> alpha: can live with minimal improvement >> beta: can live with minimal improvement >> 1.0: can live with minimal improvement > >Not sure what this one means. Is this the error we return to client? >Or is this the error from backend? > >By that I mean: > >if (something true){ > > error; > >} yes, this. >Stats that should be visible in VCL would from the top of my mind be. > >Hits on documents. (Not sure about if we can implement this, but I am >thinking in the line of 20.000 hits on /index.html last 2 min and ttl >+2m. It certainly requires us to keep stats on each document/URL or a >top-hit-list, and that may not be possible.) >Backend responstime. >Client bandwith/hits. noted, I'll get back to this. >> ---------------------------------------------------------------------- >> - >> What: >> CLI management interface >> Situation: >> Only minimally functional. >> What needs to be done: >> Reevaluate the list we created initially (ie: no unlisten) >> Implement TCP / SSH carried CLI port >> Severity: >> alpha: can live with >> beta: improvement desirable >> 1.0: TCP/SSH mandatory > >"Reevaluate the list we created initially (ie: no unlisten" ? We talked about being able to gracefully stop the cache process without destroying the listening socket, but it transpires that there is no way to do this in UNIX, there is no "unlisten()" call. >> What: >> VCL completeness >> Situation: >> VCL can't do much yet. >> What needs to be done: >> Go through our list, implement in proritized order >> Severity: >> alpha: can live with >> beta: HTTP header manipulation desirable >> 1.0: TBD > >Agree. > >Should I prioritize? Good question. I have added some fundamental bits in the meantime. >> ---------------------------------------------------------------------- >> - >> What: >> VCL compiler error checking >> Situation: >> Errors from GCC and at load-time still happens. >> Failed loading of default/boot VCL is not currently fatal. >> What needs to be done: >> Check conditions better, for instance do getaddrinfo() on >> ACL targets and warn if non () entries fail. >> Severity: >> alpha: can live with >> beta: can live with >> 1.0: can live with >> > >Agree. > >We at least need getaddrinfo on the backend IP's, but I guess that >this is meant on other cases? Also test-compiles of regexp etc. >> ---------------------------------------------------------------------- >> - >> What: >> RFC compliance audit >> Situation: >> We don't know how many small details we are missing. >> What needs to be done: >> Go through RFC(s) and sourcecode and mark up RFC(s) >> with "not relevant", "deliberate variance", "not implemented" >> and "OK". (OK preferably listing relevant testcases) >> Severity: >> alpha: can live with >> beta: can live with >> 1.0: can live with > >Bump to must/should have on 1.0? > >We are gonna need this sooner rather than later. Ppl are gonna ask >for this as soon as they start using Varnish in a prod. envorionment >is my guess. I really don't know about this one. It's a fxxking tedious job and most of what the standard allows seems to be little used in practice. >> ---------------------------------------------------------------------- >> - >> What: >> storage_file.c allocation strategy >> Situation: >> we have seen pessimisal allocation patterns with >> degenerate trafic patterns. >> Real World traffic looks basically OK, but no detailed >> analysis done. Missing stats an issue. >> What needs to be done: >> Carefully consider mitigation once we have analyzed >> the issue. >> Severity: >> alpha: no issue >> beta: no issue >> 1.0: no issue > >Agree. > >1.0 should have a way to dump the stats, so users reporting an error >in that area, can give us a clue as to what is happening. Yes. >> ---------------------------------------------------------------------- >> - >> Nice to have: >> log-tailer which writes httpd format. > >I wanna bump up this one. Ppl may need it for stats. >I know that bumps upwards are gonna resolve on more work, but a httpd >format log is really a must-have I think. Not for VG, but for others. >I am currently looking into seeing if I can program any C at all :) >httpd logg would be a good place to start I guess. A mail will follow >after this one about the logging code. I think we may want to teach the libvarnishAPI to sort things into requests and sessions so that not every program has to do this on its own. I havn't given much (enough) thought to this yet. I was hoping DES would have put these items in the ticket base by now so we could have kept track of them there ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Jul 28 12:31:46 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 28 Jul 2006 12:31:46 +0000 Subject: Varnishlog.c In-Reply-To: Your message of "Fri, 28 Jul 2006 14:06:16 +0200." Message-ID: <24821.1154089906@critter.freebsd.dk> In message , Anders Berg writes : >Hi, > >I took a look at: > >http://varnish.projects.linpro.no/browser/trunk/varnish-cache/bin/ >varnishlog/varnishlog.c > > >Line 132 sould be: if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", >4) I guess, rather than: > >if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) No, it is correct. the "p[1] == 4" checks that the argument is 4 bytes long and the memcpy checks that it has the right contents. >Also, I am scratching my head about the -h parameter in the code >line: 214. Is this supposed to be the -r param? The -h option is an (experimental) attempt to show only "not normal" goings on. It's supposed to supress any GET or HEAD request that gets a hit and delivers it with 200, or which does a fetch&insert and delivers with 200. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Fri Jul 28 12:40:43 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 28 Jul 2006 14:40:43 +0200 Subject: Varnishlog.c References: <24821.1154089906@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Anders Berg writes: > > Line 132 sould be: > > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 4) > > I guess, rather than: > > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) > No, it is correct. the "p[1] == 4" checks that the argument is > 4 bytes long and the memcpy checks that it has the right contents. Yes, the 3 should be a 4 since "HEAD" has four letters. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Fri Jul 28 12:42:29 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 28 Jul 2006 14:42:29 +0200 Subject: Dates References: <24801.1154089710@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > I was hoping DES would have put these items in the ticket base by now > so we could have kept track of them there ? Life got in the way. I've done it now, though some of the priorities / severities might be off a bit. The "nice to have" items have no milestone, everything else is set to 1.0. Let me know if you have trouble modifying tickets or if you want additional components or milestones registered. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andersb at vgnett.no Fri Jul 28 13:28:24 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 28 Jul 2006 15:28:24 +0200 (CEST) Subject: Varnishlog.c In-Reply-To: References: <24821.1154089906@critter.freebsd.dk> Message-ID: <1441.193.213.34.102.1154093304.squirrel@denise.vg.no> > "Poul-Henning Kamp" writes: >> Anders Berg writes: >> > Line 132 sould be: >> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 4) >> > I guess, rather than: >> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) >> No, it is correct. the "p[1] == 4" checks that the argument is >> 4 bytes long and the memcpy checks that it has the right contents. > > Yes, the 3 should be a 4 since "HEAD" has four letters. Thats what I also "figured" out. Anders Berg > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > From phk at phk.freebsd.dk Fri Jul 28 13:40:55 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 28 Jul 2006 13:40:55 +0000 Subject: Varnishlog.c In-Reply-To: Your message of "Fri, 28 Jul 2006 15:28:24 +0200." <1441.193.213.34.102.1154093304.squirrel@denise.vg.no> Message-ID: <25153.1154094055@critter.freebsd.dk> In message <1441.193.213.34.102.1154093304.squirrel at denise.vg.no>, "Anders Berg " writes: >> "Poul-Henning Kamp" writes: >>> Anders Berg writes: >>> > Line 132 sould be: >>> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 4) >>> > I guess, rather than: >>> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) >>> No, it is correct. the "p[1] == 4" checks that the argument is >>> 4 bytes long and the memcpy checks that it has the right contents. >> >> Yes, the 3 should be a 4 since "HEAD" has four letters. > >Thats what I also "figured" out. Duh! I looked at the wrong line... yes, "HEAD" has four characters. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From andersb at vgnett.no Fri Jul 28 14:12:28 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 28 Jul 2006 16:12:28 +0200 (CEST) Subject: Varnishlog.c In-Reply-To: <25153.1154094055@critter.freebsd.dk> References: Your message of "Fri, 28 Jul 2006 15:28:24 +0200." <1441.193.213.34.102.1154093304.squirrel@denise.vg.no> <25153.1154094055@critter.freebsd.dk> Message-ID: <1654.193.213.34.102.1154095948.squirrel@denise.vg.no> > In message <1441.193.213.34.102.1154093304.squirrel at denise.vg.no>, "Anders > Berg > " writes: >>> "Poul-Henning Kamp" writes: >>>> Anders Berg writes: >>>> > Line 132 sould be: >>>> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 4) >>>> > I guess, rather than: >>>> > if (h_opt && p[1] == 4 && !memcmp(p + 4, "HEAD", 3) >>>> No, it is correct. the "p[1] == 4" checks that the argument is >>>> 4 bytes long and the memcpy checks that it has the right contents. >>> >>> Yes, the 3 should be a 4 since "HEAD" has four letters. >> >>Thats what I also "figured" out. > > Duh! I looked at the wrong line... > > yes, "HEAD" has four characters. Wohoo! I can claim a bug-fix :) Anders Berg > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > >