From pquerna at apache.org Wed Nov 1 09:06:30 2006 From: pquerna at apache.org (Paul Querna) Date: Wed, 01 Nov 2006 01:06:30 -0800 Subject: Varnish version 2 and beyond Message-ID: <45486396.4080502@apache.org> ... > What ? > ====== > > The features I have heard people ask for so far are: > > Things we already know how to do: > Bugfixes > Docs > More VCL > Simple Server side includes. > Vary: header processing (requirement for compression etc.) > Compression. I wrote the Vary support for Apache's mod_disk_cache at one time, and would be happy to provide feedback on what worked and what didn't. To me it seems the lack of Vary (and hence enable compression in most cases) is a major blocker for larger adoption. ... > As a programmer I want to add, high on the list: > > Regression test driver > Regression test cases Have you looked at the httpd-test project, used by mod_perl and httpd? http://svn.apache.org/viewvc/httpd/test/trunk/perl-framework/ Its some crazy Perl stuff, but it makes it quite easy to build new test cases quickly. -Paul Querna From phk at phk.freebsd.dk Wed Nov 1 09:19:30 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 01 Nov 2006 09:19:30 +0000 Subject: Regression tests (was: Varnish version 2 and beyond) In-Reply-To: Your message of "Wed, 01 Nov 2006 01:06:30 PST." <45486396.4080502@apache.org> Message-ID: <14783.1162372770@critter.freebsd.dk> In message <45486396.4080502 at apache.org>, Paul Querna writes: >I wrote the Vary support for Apache's mod_disk_cache at one time, and >would be happy to provide feedback on what worked and what didn't. To >me it seems the lack of Vary (and hence enable compression in most >cases) is a major blocker for larger adoption. I'll take you up on that in due time :-) >> As a programmer I want to add, high on the list: >> >> Regression test driver >> Regression test cases > >Have you looked at the httpd-test project, used by mod_perl and httpd? > >http://svn.apache.org/viewvc/httpd/test/trunk/perl-framework/ I havn't really given this area the attention it deserves, so I have not looked at any tools at all. (There is a potential license issue if we import something, but we can probably handle that somehow.) But I think it may make sense to write our own tool for purely practical reasons: Will need to teach our tool about changing Varnish parameters and flipping VCL programs, in essence making the Varnish CLI a paramter in the equation. But also the temporal aspect of testing the expiry and timeouts has me worried, getting that right is going to be slightly involved. This is one of the areas where I would LOVE to see somebody with a good PERL clue step up and help us out, because I'm not strong on PERL and this is a very well isolated task to work on. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From pierre at queinnec.org Wed Nov 1 22:21:53 2006 From: pierre at queinnec.org (Pierre Queinnec) Date: Wed, 01 Nov 2006 23:21:53 +0100 Subject: FYI: Installation instructions updated to include Mac OS X In-Reply-To: <9CDE3BE2-853F-4C02-924D-279FBBDA53DA@anduin.net> References: <83581A1D-E1BD-4377-A97C-C0F75D3F72E8@tomster.org> <9CDE3BE2-853F-4C02-924D-279FBBDA53DA@anduin.net> Message-ID: <45491E01.4000608@queinnec.org> Hi Eirik, I've been wanting to share a quick n'dirty patch to get it to work on OS X with a clock_gettime hack, but as I updated to the current SVN version, I saw this commit: http://projects.linpro.no/pipermail/varnish-commit/2006-October/001218.html It should answer your question. BTW, I ponder wether it's better to use gettimeofday-style or something like: host_get_clock_service(mach_host_self(), REALTIME_CLOCK, &myClock); clock_get_time(myClock, &myTime); I bet the current compat hack is better, because it doesn't involve including mach headers. Anyway, thanks to the ones that made it possible and thanks for this quality software! -- Pierre Queinnec Architecte - Zenika http://www.zenika.com Eirik ?verby wrote: > On Oct 28, 2006, at 7:10 PM, Tom Lazar wrote: > >> I just installed varnish successfully on my OSX boxen and edited the >> installation instructions[1] accordingly. > > Just curious, what happened to those missing time functions on OSX? > /Eirik > >> >> it's just two additional steps: >> >> --snip-- >> 1. install `automake`, i.e. via [http://darwinports.opendarwin.org/ >> Darwin Ports]: `sudo port install automake` as the version of >> `automake` that ships with OS X is too old. >> >> 2. patch the `autogen.sh` file: >> >> {{{ >> 10a11,15 >> > if [ -e /opt/local/bin/automake ] ; then >> > PATH=/opt/local/bin:${PATH} >> > export PATH >> > fi >> > >> 30c35 >> < libtoolize --copy --force >> --- >> > glibtoolize --copy --force >> }}} >> >> ''n.b. this assumes you are using the abovementioned >> [http://darwinports.opendarwin.org/ Darwin Ports]. The key point is to >> insert the path to your newer version of `automake` '''before''' the >> system path in order to 'hide' the old version from `autogen.sh`.'' >> >> --snap-- >> >> >> cheers, >> >> tom >> >> [1] http://varnish.projects.linpro.no/wiki/Installation >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From maxime at ezx.fr Thu Nov 2 02:12:00 2006 From: maxime at ezx.fr (Maxime) Date: Thu, 2 Nov 2006 03:12:00 +0100 Subject: VCL commands Message-ID: <20061102021200.GA26673@trolleur.net> Hi, I couldn't find these infos in vcl(7), maybe they should be added there : - meaning of pipe, pass, lookup, etc (you can guess for some commands, but what is the meaning of insert_pass ?)... - meaning of obj.cacheable (and obj.valid ?). Ok, it says : "this object is cacheable", but how does varnish set this variable ? -- Maxime Ritter From des at linpro.no Thu Nov 2 12:16:14 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 02 Nov 2006 13:16:14 +0100 Subject: FYI: Installation instructions updated to include Mac OS X In-Reply-To: <45491E01.4000608@queinnec.org> (Pierre Queinnec's message of "Wed, 01 Nov 2006 23:21:53 +0100") References: <83581A1D-E1BD-4377-A97C-C0F75D3F72E8@tomster.org> <9CDE3BE2-853F-4C02-924D-279FBBDA53DA@anduin.net> <45491E01.4000608@queinnec.org> Message-ID: Pierre Queinnec writes: > I've been wanting to share a quick n'dirty patch to get it to work on OS > X with a clock_gettime hack, but as I updated to the current SVN > version, I saw this commit: > > http://projects.linpro.no/pipermail/varnish-commit/2006-October/001218.html > > It should answer your question. BTW, I ponder wether it's better to use > gettimeofday-style or something like: > host_get_clock_service(mach_host_self(), REALTIME_CLOCK, &myClock); > clock_get_time(myClock, &myTime); > > I bet the current compat hack is better, because it doesn't involve > including mach headers. The correct solution is to pester Apple until they implement clock_gettime(), which is required by POSIX. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From lists at tomster.org Thu Nov 2 16:17:25 2006 From: lists at tomster.org (Tom Lazar) Date: Thu, 2 Nov 2006 17:17:25 +0100 Subject: FYI: Installation instructions updated to include Mac OS X In-Reply-To: <9CDE3BE2-853F-4C02-924D-279FBBDA53DA@anduin.net> References: <83581A1D-E1BD-4377-A97C-C0F75D3F72E8@tomster.org> <9CDE3BE2-853F-4C02-924D-279FBBDA53DA@anduin.net> Message-ID: <0B651A17-A9F7-417E-BCA2-361B4930D903@tomster.org> On Oct 31, 2006, at 8:39 PM, Eirik ?verby wrote: > On Oct 28, 2006, at 7:10 PM, Tom Lazar wrote: > >> I just installed varnish successfully on my OSX boxen and edited >> the installation instructions[1] accordingly. > > Just curious, what happened to those missing time functions on OSX? not sure about that issue (sorry for the late response) atm varnish doesn't actually run on my mac. i don't have the error handy, but i guess it means, for the time being, that varnish compiles but doesn't run on mac os x. cheers, tom > /Eirik > >> >> it's just two additional steps: >> >> --snip-- >> 1. install `automake`, i.e. via [http:// >> darwinports.opendarwin.org/ Darwin Ports]: `sudo port install >> automake` as the version of `automake` that ships with OS X is too >> old. >> >> 2. patch the `autogen.sh` file: >> >> {{{ >> 10a11,15 >> > if [ -e /opt/local/bin/automake ] ; then >> > PATH=/opt/local/bin:${PATH} >> > export PATH >> > fi >> > >> 30c35 >> < libtoolize --copy --force >> --- >> > glibtoolize --copy --force >> }}} >> >> ''n.b. this assumes you are using the abovementioned [http:// >> darwinports.opendarwin.org/ Darwin Ports]. The key point is to >> insert the path to your newer version of `automake` '''before''' >> the system path in order to 'hide' the old version from >> `autogen.sh`.'' >> >> --snap-- >> >> >> cheers, >> >> tom >> >> [1] http://varnish.projects.linpro.no/wiki/Installation >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> > > From hema at tv2.dk Mon Nov 6 16:20:34 2006 From: hema at tv2.dk (Henrik Nicolaisen) Date: Mon, 6 Nov 2006 17:20:34 +0100 Subject: Problems using subnets in acl Message-ID: <90C76C61-FCAF-4B27-9C45-0DDB0D28398E@tv2.dk> Hi I am testing varnish, and I need to allow our internal development net to purge documents. It works perfectly if i add single ip's, but when i try to add subnets it won't work, is this a bug, or am i writing something wrong in vcl ? acl purge { "localhost"; "10.10.0.0/16"; } sub vcl_recv { if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } } I have also tried "10.10.0.0/255.255.0.0"; Any ideas ? Venlig hilsen Henrik Nicolaisen TV 2 Interaktiv -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1931 bytes Desc: not available URL: From des at linpro.no Tue Nov 7 12:19:44 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 07 Nov 2006 13:19:44 +0100 Subject: Problems using subnets in acl In-Reply-To: <90C76C61-FCAF-4B27-9C45-0DDB0D28398E@tv2.dk> (Henrik Nicolaisen's message of "Mon, 6 Nov 2006 17:20:34 +0100") References: <90C76C61-FCAF-4B27-9C45-0DDB0D28398E@tv2.dk> Message-ID: Henrik Nicolaisen writes: > acl purge { > "localhost"; > "10.10.0.0/16"; > } The correct syntax is "10.10.0.0"/16 (i.e. keep the mask length outside the quotes). DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Nov 7 12:22:20 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 07 Nov 2006 12:22:20 +0000 Subject: Problems using subnets in acl In-Reply-To: Your message of "Mon, 06 Nov 2006 17:20:34 +0100." <90C76C61-FCAF-4B27-9C45-0DDB0D28398E@tv2.dk> Message-ID: <10695.1162902140@critter.freebsd.dk> In message <90C76C61-FCAF-4B27-9C45-0DDB0D28398E at tv2.dk>, Henrik Nicolaisen wri tes: >It works perfectly if i add single ip's, but when i try to add >subnets it won't work, is this a bug, or am i writing something wrong >in vcl ? > >acl purge { > "localhost"; > "10.10.0.0/16"; >} Hmm, that should work, it's probably a bug. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From gaute at pht.no Thu Nov 9 14:25:33 2006 From: gaute at pht.no (Gaute Amundsen) Date: Thu, 9 Nov 2006 15:25:33 +0100 Subject: Whats cached right now? Message-ID: <200611091525.33941.gaute@pht.no> Hi again. I'm back from struggling with squid and squidGuard, squirm and iRedirector.. I figured varnish on it's own IP, in front of apache with good rewriterules, in front of zope, would be better after all, than the huge unwieldy squid that was not meant for the purpose, combined with these external rewriters with cryptic syntax, that where not meant for reverse proxying either... Well, anyway, I've been playing with headers, vcl and the varnishtools, but I can't quite figure out if there is a way to display what is beeing cached _right_now_. I realize I may be asking the wrong question, but to me that is the most straightforward thing I want to know when trying to make a big site cachable. Another thing. Run-Time Parameters: auto_restart Is that what it sounds like? If I put "-p auto_restart=on" in /etc/sysconfig/varnish it will restart the child prosess if it dies? Regards Gaute Amundsen From trondmm-varnish at crusaders.no Thu Nov 9 14:18:39 2006 From: trondmm-varnish at crusaders.no (Trond Michelsen) Date: Thu, 9 Nov 2006 15:18:39 +0100 Subject: Purging with regexes over HTTP Message-ID: <20061109141839.GB12694@crusaders.no> Hi. I'm having problems purging URLs with the PURGE request-method. So far, I've only been able to purge full URLs, one at a time. But I'd like to purge them with regexes instead. Shouldn't it be possible to say PURGE LAYERS=roads HTTP/1.0 Host: hostname and accomplish the same as you do when issuing the command url.purge LAYERS=roads in the varnish CLI? Using the CLI works as expected, but I'm trying to write a script that should purge URLs automatically, whenever new data is available. I assume that it'll be easier to write a script that talks to varnish over HTTP, than a script that uses Expect to control the CLI, so I'd prefer using the HTTP interface. -- Trond Michelsen From gaute at pht.no Thu Nov 9 15:21:12 2006 From: gaute at pht.no (Gaute Amundsen) Date: Thu, 9 Nov 2006 16:21:12 +0100 Subject: Purging with regexes over HTTP In-Reply-To: <20061109141839.GB12694@crusaders.no> References: <20061109141839.GB12694@crusaders.no> Message-ID: <200611091621.13893.gaute@pht.no> In http://varnish.projects.linpro.no/wiki/FAQ it says at the end of the entry: "How can I force a refresh on a object cached by varnish?" After describing how to set up the PURGE HTTP method: >However, this way of implementing purge does not support wildcard purge yet >(because of restrictions in VCL). If I don't beat you to it, I would be happy if you would share your expect script :) Regards Gaute Amundsen On Thursday 09 November 2006 15:18, Trond Michelsen wrote: > Hi. > > I'm having problems purging URLs with the PURGE request-method. So > far, I've only been able to purge full URLs, one at a time. But I'd > like to purge them with regexes instead. > > Shouldn't it be possible to say > > PURGE LAYERS=roads HTTP/1.0 > Host: hostname > > and accomplish the same as you do when issuing the command > > url.purge LAYERS=roads > > in the varnish CLI? > > Using the CLI works as expected, but I'm trying to write a script that > should purge URLs automatically, whenever new data is available. I > assume that it'll be easier to write a script that talks to varnish > over HTTP, than a script that uses Expect to control the CLI, so I'd > prefer using the HTTP interface. From regebro at gmail.com Thu Nov 9 17:10:52 2006 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 9 Nov 2006 18:10:52 +0100 Subject: Quick explanation of the VCL commands? Message-ID: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> Hi everybody. Tomorrow I'll install Varnish on a big projects preproduction site, to replace squid, mostly because I can't get Squid to actually cache anything no matter how much documentation I read. Varnish, despite the lack of documentation, I got up and modified the configuration for my needs in an hour. :-) However, I'm gonna spend tomorrow trying to optimize the configuration and would like to understand VCL a bit better. Mostly, I wonder if there is a document that explains, or if somebody can quickly explain, what all the VCL commands, pipe, pass, insert_pass and so on do. Just a quick explanation would suffice, I think. I tried to read the source, but the lack of comments meant it didn't help. ;) Thanks! -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From stonorn at giraffen.dk Thu Nov 9 20:06:20 2006 From: stonorn at giraffen.dk (Anton Stonor) Date: Thu, 09 Nov 2006 21:06:20 +0100 Subject: Quick explanation of the VCL commands? In-Reply-To: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> References: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> Message-ID: <45538A3C.701@giraffen.dk> Hi Lennart, > However, I'm gonna spend tomorrow trying to optimize the configuration > and would like to understand VCL a bit better. Mostly, I wonder if > there is a document that explains, or if somebody can quickly explain, > what all the VCL commands, pipe, pass, insert_pass and so on do. I don't thing a full documentation is there yet, but this man-page might be useful: http://varnish.projects.linpro.no/browser/trunk/varnish-cache/man/vcl.7 Also, have a look at PHK's presentation -- the diagram on page 20 is helpful in understanding of what is going on: http://phk.freebsd.dk/pubs/varnish_wip.pdf Finally, the default VCL snippet is worth a look (from my notepad, might have changed since): --- snippet --- sub default_vcl_recv { if (req.request != "GET" && req.request != "HEAD") { pipe; } if (req.http.Expect) { pipe; } if (req.http.Authenticate || req.http.Cookie) { pass; } lookup; } sub default_vcl_hit { if (!obj.cacheable) { pass; } deliver; } default_vcl_miss { fetch; } sub default_vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { insert_pass; } if (resp.http.Set-Cookie) { insert_pass; } insert; } sub default_vcl_timeout { discard; } --- snippet end --- /Anton From regebro at gmail.com Thu Nov 9 23:01:58 2006 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 10 Nov 2006 00:01:58 +0100 Subject: Quick explanation of the VCL commands? In-Reply-To: <45538A3C.701@giraffen.dk> References: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> <45538A3C.701@giraffen.dk> Message-ID: <319e029f0611091501w4d3406e6n8bbade1e87068459@mail.gmail.com> On 11/9/06, Anton Stonor wrote: > Also, have a look at PHK's presentation -- the diagram on page 20 is > helpful in understanding of what is going on: Yeah, super, thanks! Now all I need to know is what the difference between pass and pipe is... -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From regebro at gmail.com Fri Nov 10 11:39:58 2006 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 10 Nov 2006 12:39:58 +0100 Subject: How to best debug your VCL? Message-ID: <319e029f0611100339t21ee094wc8a32759040cac40@mail.gmail.com> Hi! I'm currently trying out different rule settings in VCL to make sure just the right things are cached, and if the rulesets gets a bit complex I'd like to know how to best debug it, that is how to best know which if statements was true or not. Is there for example a log statment? I couldn't find anything in the vcl man page, and "log" didn't work in any case, but maybe it's called something else? :-) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From phk at phk.freebsd.dk Fri Nov 10 13:53:36 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 10 Nov 2006 13:53:36 +0000 Subject: How to best debug your VCL? In-Reply-To: Your message of "Fri, 10 Nov 2006 12:39:58 +0100." <319e029f0611100339t21ee094wc8a32759040cac40@mail.gmail.com> Message-ID: <1228.1163166816@critter.freebsd.dk> In message <319e029f0611100339t21ee094wc8a32759040cac40 at mail.gmail.com>, "Lenna rt Regebro" writes: >Hi! > >I'm currently trying out different rule settings in VCL to make sure >just the right things are cached, and if the rulesets gets a bit >complex I'd like to know how to best debug it, that is how to best >know which if statements was true or not. There is a parameter that enables VCL logging. In your shmlog you will get a log entry for each executed "token" of the VCL code with line+char. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From gaute at pht.no Fri Nov 10 14:05:50 2006 From: gaute at pht.no (Gaute Amundsen) Date: Fri, 10 Nov 2006 15:05:50 +0100 Subject: Purging a specific host Message-ID: <200611101505.50668.gaute@pht.no> The way I have set up varnish now, it sits in front of an apache that serves a bunch of vhosts. Is there a way to purge all documents belonging to one of them? Let me guess.. only with http? Gaute. From gaute at pht.no Fri Nov 10 15:21:56 2006 From: gaute at pht.no (Gaute Amundsen) Date: Fri, 10 Nov 2006 16:21:56 +0100 Subject: Quick explanation of the VCL commands? In-Reply-To: <319e029f0611091501w4d3406e6n8bbade1e87068459@mail.gmail.com> References: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> <45538A3C.701@giraffen.dk> <319e029f0611091501w4d3406e6n8bbade1e87068459@mail.gmail.com> Message-ID: <200611101621.58054.gaute@pht.no> On Friday 10 November 2006 00:01, Lennart Regebro wrote: > On 11/9/06, Anton Stonor wrote: > > Also, have a look at PHK's presentation -- the diagram on page 20 is > > helpful in understanding of what is going on: > > Yeah, super, thanks! > > Now all I need to know is what the difference between pass and pipe is... http://varnish.projects.linpro.no/svn/trunk/varnish-doc/en/varnish-architecture/article.xml 1.3.6 Pass and Pipe modes Requests which the can not or should not be handled by Varnish can be either passed through or piped through to the backend. Passing acts on a per-request basis and tries to make the connection to both the client and the backend reusable. Piping acts as a transparent tunnel and whatever happens for the rest of the lifetime of the client and backend connection is not interpreted by Varnish. From regebro at gmail.com Fri Nov 10 16:11:55 2006 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 10 Nov 2006 17:11:55 +0100 Subject: How to best debug your VCL? In-Reply-To: <1228.1163166816@critter.freebsd.dk> References: <319e029f0611100339t21ee094wc8a32759040cac40@mail.gmail.com> <1228.1163166816@critter.freebsd.dk> Message-ID: <319e029f0611100811u5efe7e44v311a0edfefed8606@mail.gmail.com> On 11/10/06, Poul-Henning Kamp wrote: > There is a parameter that enables VCL logging. In your shmlog > you will get a log entry for each executed "token" of the VCL > code with line+char. Aha, the vcl trace setting, got it, thanks! -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From regebro at gmail.com Fri Nov 10 16:25:24 2006 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 10 Nov 2006 17:25:24 +0100 Subject: Quick explanation of the VCL commands? In-Reply-To: <200611101621.58054.gaute@pht.no> References: <319e029f0611090910h110a708dydcfd409a19c9fe41@mail.gmail.com> <45538A3C.701@giraffen.dk> <319e029f0611091501w4d3406e6n8bbade1e87068459@mail.gmail.com> <200611101621.58054.gaute@pht.no> Message-ID: <319e029f0611100825g2d649bc8p4c281d5b8dbe4496@mail.gmail.com> On 11/10/06, Gaute Amundsen wrote: > On Friday 10 November 2006 00:01, Lennart Regebro wrote: > > On 11/9/06, Anton Stonor wrote: > > > Also, have a look at PHK's presentation -- the diagram on page 20 is > > > helpful in understanding of what is going on: > > > > Yeah, super, thanks! > > > > Now all I need to know is what the difference between pass and pipe is... > > > http://varnish.projects.linpro.no/svn/trunk/varnish-doc/en/varnish-architecture/article.xml > > 1.3.6 Pass and Pipe modes > > Requests which the can not or should not be handled by > Varnish can be either passed through or piped through to > the backend. > > Passing acts on a per-request basis and tries to make the > connection to both the client and the backend reusable. > > Piping acts as a transparent tunnel and whatever happens > for the rest of the lifetime of the client and backend > connection is not interpreted by Varnish. > Aha...! Clever. Thanks for the answer! -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From gaute at pht.no Sun Nov 12 15:16:41 2006 From: gaute at pht.no (Gaute Amundsen) Date: Sun, 12 Nov 2006 16:16:41 +0100 Subject: Purge by python and expect Message-ID: <200611121616.41717.gaute@pht.no> Ok, I gott there first I guess :) Quick & dirty, more like sketches in fact, so please post your improvements, and suggestions. Regards Gaute Amundsen By expect: #!/usr/bin/expect -- set timeout 5 log_user 0 spawn -noecho telnet localhost 90 #spawn telnet localhost 90 expect { "Escape character is '^]'." { send -- "url.purge .*\r"; expect -re "(\\d+ \\d+)(.*)PURGE" puts $expect_out(1,string) close; puts "DONE\r" } "Connection refused" { puts "FAILED\r"} } by python: #!/usr/bin/python def by_http(): # not complete, need to take a list of urls, and possibly a host as paramertes import httplib #(scheme, host, path, params, query, fragment ) = urlparse.urlparse(objecturl) conn = httplib.HTTPConnection("cache.site.com") conn.request("PURGE", "/foo/bar/baf") r1 = conn.getresponse() conn.close() return r1.status, r1.reason def by_telnet(): # deletes everything from all hosts. dont know if host matching is possible import telnetlib import socket #tn.set_debuglevel(10) try: tn = telnetlib.Telnet('localhost',90) except socket.error, e: return e[1] #socket.error: (111, 'Connection refused') tn.write("url.purge .*\n") # purge documets matching regexp (a,b,c) = tn.expect(["\d+ \d+"], 2) tn.close() return b.group(0) + "\nDONE" From gaute at pht.no Sun Nov 12 15:47:52 2006 From: gaute at pht.no (Gaute Amundsen) Date: Sun, 12 Nov 2006 16:47:52 +0100 Subject: Purging. Now what? Message-ID: <200611121647.54547.gaute@pht.no> Now that I can trigger a purge when a customer presses "save" in our CMS, the next step is trying to do it somewhat smarter... Purging everything from all hosts in the cache is simple via telnet, but a bit brutish. It could get noticable as well, with maybe 50 customers saving through the day.. Purging one url at a time is more presice, but then I have to keep track of what to purge. Finding all urls in a site is not very efficient, and 95% of those would not be in the cache anyway. I could build a small daemon to tail the access logs, and keep a running buffer of recently accessed pages. Then I could easily prefetch pages as well, after purging them. But it does not feel quite right this either... Sort of buliding a shadow copy of varnish timeout mechanism. Any good suggestions? Regards Gaute Amundsen From andersb at vgnett.no Mon Nov 13 20:53:39 2006 From: andersb at vgnett.no (Anders Berg) Date: Mon, 13 Nov 2006 21:53:39 +0100 Subject: Purging. Now what? Message-ID: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> Hi Gaute, and thanks for the script examples in earlier posts. With regards to your question about purging I must admit that I am unsure what you are trying to achieve. Could we be talking corner case here? I will try to explain why and how purging is used in general, so please don't be offended if I state the obvious or can't see your challange or what I am saying is to trivial. Purging is not used to control content expiration, the HTTP headers like Expires and max-age etc. do that. Purging is used as a "oh-shit- have-to-delete-now" mechanism. Let's say you have a default cache time for your article/page on 5 min., but 5 min. is to long to wait for a update if there is important stuff that needs to replace the content in the cache. That's when you would use purging to "force" an update on that/those page(s). If it's only one page, go directly for that page (no reg-exp), if there are more that one you have to keep control over what pages need to be refreshed or use a URL schema so that a reg-exp purge will delete them. Does this answer your question? Please explain a bit deeper, with examples, if this does not. Anders Berg Sys.adm VG Nett // www.vg.no > Date: Sun, 12 Nov 2006 16:47:52 +0100 > From: Gaute Amundsen > Subject: Purging. Now what? > To: varnish-misc at projects.linpro.no > Message-ID: <200611121647.54547.gaute at pht.no> > Content-Type: text/plain; charset="iso-8859-1" > > Now that I can trigger a purge when a customer presses "save" in > our CMS, the > next step is trying to do it somewhat smarter... > > Purging everything from all hosts in the cache is simple via > telnet, but a bit > brutish. It could get noticable as well, with maybe 50 customers > saving > through the day.. > > Purging one url at a time is more presice, but then I have to keep > track of > what to purge. Finding all urls in a site is not very efficient, > and 95% of > those would not be in the cache anyway. > > I could build a small daemon to tail the access logs, and keep a > running > buffer of recently accessed pages. Then I could easily prefetch > pages as > well, after purging them. But it does not feel quite right this > either... > Sort of buliding a shadow copy of varnish timeout mechanism. > > Any good suggestions? > > Regards > Gaute Amundsen From gaute at pht.no Tue Nov 14 08:44:22 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 09:44:22 +0100 Subject: Purging. Now what? In-Reply-To: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> Message-ID: <200611140944.23359.gaute@pht.no> I am well aware of what purging is intended for. I guess at VG, the journalists dont mind of they post a story, and it takes 5 minutes to appear on the frontpage? Or more likely you can predict what will change, and purge that. Thing is, we have a whole bunch of CMS sites on different domains, and the customers are not all so tech-savy, so telling them "sometimes it may take 5 minutes", would only create confusion. Furthermore, the sites all differ enough that it is not simple to predict, what pages will be affectd by an uppdate. However, few of them would have very many pages in the cache at any time, so that flushing them all is not a problem. Flushing all the pages of all the customers everytime one of them saves the least little thing, IS a problem, since that would happen often enough during "prime time" to make our sites appear to be unpredictable, and fickle. Is that any clearer? In tecnical terms, as I tried to ask before, I need to: 1 ) be able to purge by domain on the console 2 ) purge by patern match by HTTP PURGE Or if neither of those are possible 3 ) keep track of the urls in the cache, with or without the help of varnish, so that I can purge them by HTTP PURGE. Beeing able to prefetch would be the payoff for the hassle of the last alternative I guess :) I know the docs says that 2 can't be done. An authoritative confirmation that 1 can't be either would be helpfull, as that would let me consentrate on 3. A way to list what is beeing cached at any one time would be really helpfull both in implementing 3 of course, but allso in getting the headers configured right for all the different pieces of code that live on our servers. The information is in the logs, I know, I only find it a bit cumbersome to work with. If I end up building some small log-tailers to assist me in this, would that be in line with the intentions of the architecture do you think? Gaute. On Monday 13 November 2006 21:53, Anders Berg wrote: > Hi Gaute, > > and thanks for the script examples in earlier posts. > > With regards to your question about purging I must admit that I am > unsure what you are trying to achieve. Could we be talking corner > case here? > I will try to explain why and how purging is used in general, so > please don't be offended if I state the obvious or can't see your > challange or what I am saying is to trivial. > > Purging is not used to control content expiration, the HTTP headers > like Expires and max-age etc. do that. Purging is used as a "oh-shit- > have-to-delete-now" mechanism. Let's say you have a default cache > time for your article/page on 5 min., but 5 min. is to long to wait > for a update if there is important stuff that needs to replace the > content in the cache. That's when you would use purging to "force" an > update on that/those page(s). If it's only one page, go directly for > that page (no reg-exp), if there are more that one you have to keep > control over what pages need to be refreshed or use a URL schema so > that a reg-exp purge will delete them. > > Does this answer your question? Please explain a bit deeper, with > examples, if this does not. > > Anders Berg > Sys.adm > VG Nett // www.vg.no > > > Date: Sun, 12 Nov 2006 16:47:52 +0100 > > From: Gaute Amundsen > > Subject: Purging. Now what? > > To: varnish-misc at projects.linpro.no > > Message-ID: <200611121647.54547.gaute at pht.no> > > Content-Type: text/plain; charset="iso-8859-1" > > > > Now that I can trigger a purge when a customer presses "save" in > > our CMS, the > > next step is trying to do it somewhat smarter... > > > > Purging everything from all hosts in the cache is simple via > > telnet, but a bit > > brutish. It could get noticable as well, with maybe 50 customers > > saving > > through the day.. > > > > Purging one url at a time is more presice, but then I have to keep > > track of > > what to purge. Finding all urls in a site is not very efficient, > > and 95% of > > those would not be in the cache anyway. > > > > I could build a small daemon to tail the access logs, and keep a > > running > > buffer of recently accessed pages. Then I could easily prefetch > > pages as > > well, after purging them. But it does not feel quite right this > > either... > > Sort of buliding a shadow copy of varnish timeout mechanism. > > > > Any good suggestions? > > > > Regards > > Gaute Amundsen > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From regebro at gmail.com Tue Nov 14 09:07:55 2006 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 14 Nov 2006 10:07:55 +0100 Subject: Purging. Now what? In-Reply-To: <200611140944.23359.gaute@pht.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611140944.23359.gaute@pht.no> Message-ID: <319e029f0611140107r4a79ccc0yc45be0b8cad66600@mail.gmail.com> On 11/14/06, Gaute Amundsen wrote: > 3 ) keep track of the urls in the cache, with or without the help of varnish, > so that I can purge them by HTTP PURGE. I don't see why you would need to keep track of what is cached. If a page has changed, purge it. It doesn't matter if it is in the cache or not. If it's not, then nothing gets purged. :-) This should be easily done with a HTTP call from the site system which calls a purge for the changed page, it's parent and maybe the frontpage or something (obviously dependent on your site structure). -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From gaute at pht.no Tue Nov 14 09:33:38 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 10:33:38 +0100 Subject: Purging. Now what? In-Reply-To: <319e029f0611140107r4a79ccc0yc45be0b8cad66600@mail.gmail.com> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611140944.23359.gaute@pht.no> <319e029f0611140107r4a79ccc0yc45be0b8cad66600@mail.gmail.com> Message-ID: <200611141033.38872.gaute@pht.no> On Tuesday 14 November 2006 10:07, Lennart Regebro wrote: > On 11/14/06, Gaute Amundsen wrote: > > 3 ) keep track of the urls in the cache, with or without the help of > > varnish, so that I can purge them by HTTP PURGE. > > I don't see why you would need to keep track of what is cached. If a > page has changed, purge it. It doesn't matter if it is in the cache or > not. If it's not, then nothing gets purged. :-) > This should be easily done with a HTTP call from the site system which > calls a purge for the changed page, it's parent and maybe the > frontpage or something (obviously dependent on your site structure). Mostly because, as I said, many of our sites are large, and it is hard to predict what pagees any one change wil affect, and many of our pages does not even exist as recognizable objects, just as URLs. ( If you had ever worked with zope and it's concept of acquisition, you would understand :-/ ) To be able to "watch the cache" would give me confidence that we where not caching things that should not be. Watching a "pool of cache misses" would be good as well, come to think of it. Until prefetch is implemented in varnish, an added benefit would be the ability to do that myself, by having "wget fetch the page" of what I just purged. Gaute From phk at phk.freebsd.dk Tue Nov 14 09:50:38 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 14 Nov 2006 09:50:38 +0000 Subject: Purging. Now what? In-Reply-To: Your message of "Tue, 14 Nov 2006 10:33:38 +0100." <200611141033.38872.gaute@pht.no> Message-ID: <10024.1163497838@critter.freebsd.dk> One of my friends have had "So much code to hack, so little time" in his signature for many years. Yes, man of the features you request are already talked about and planned, it's just a matter coding. But things do take time. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From trondmm-varnish at crusaders.no Tue Nov 14 10:04:41 2006 From: trondmm-varnish at crusaders.no (Trond Michelsen) Date: Tue, 14 Nov 2006 11:04:41 +0100 Subject: Purging. Now what? In-Reply-To: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> Message-ID: <20061114100441.GA24310@crusaders.no> On Mon, Nov 13, 2006 at 09:53:39PM +0100, Anders Berg wrote: > With regards to your question about purging I must admit that I am > unsure what you are trying to achieve. Could we be talking corner > case here? It's possible that I have an edge case for you :) > I will try to explain why and how purging is used in general, so > please don't be offended if I state the obvious or can't see your > challange or what I am saying is to trivial. > > Purging is not used to control content expiration, the HTTP headers > like Expires and max-age etc. do that. Right. But my problem is that I don't know in advance when something expires, I only know when it has expired. I'm using Varnish to cache maptiles from a WMS-server. We also have to layers of WMS-servers. First there's mapserver, which is used to draw general maps, then there's our own WMS-server, that's used to draw weather maps. All requests to our WMS-server passes through mapserver. Unfortunately, mapserver does not support any content expiration headers, like Expires or If-Modified-Since or anything like that. Besides, if it did - how should it behave if it needs to fetch two layers from our WMS-server, and one of them is unchanged? Mapserver can't be expected to keep a cache of its own. > Purging is used as a "oh-shit- > have-to-delete-now" mechanism. Let's say you have a default cache > time for your article/page on 5 min., but 5 min. is to long to wait > for a update if there is important stuff that needs to replace the > content in the cache. That's when you would use purging to "force" an > update on that/those page(s). > If it's only one page, go directly for that page (no reg-exp), if > there are more that one you have to keep control over what pages > need to be refreshed or use a URL schema so that a reg-exp purge > will delete them. Well, since we're using tiled images, the number of potential URLs becomes quite large very fast. At any given zoomlevel, there are 4**zoomlevel tiles. So at zoomlevel 12 (which we use for Google Earth), there are 16,7 million tiles. We have about 20 different datalayers, and every layer contains data for every hour for 48 hours. That means that we have to purge up to 16 billion tiles when a datamodel is updated (about every 12 hours)[1]. Since our internal WMS-server is fairly slow, we really, really don't want to generate any tiles more than once, so it's important for us that Varnish keeps everything in cache until it's purged. > Does this answer your question? Please explain a bit deeper, with > examples, if this does not. I hope this explains why we'd like to purge URLs regularly with a regex. [1] Of course, only a tiny fraction of this will actually be generated in the first place. We don't expect users to zoom in this close on every part of the world, but it illustrates the potential number of URLs that need to be purged if it can't be done with a regex. -- Trond Michelsen From phk at phk.freebsd.dk Tue Nov 14 10:16:32 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 14 Nov 2006 10:16:32 +0000 Subject: Purging. Now what? In-Reply-To: Your message of "Tue, 14 Nov 2006 11:04:41 +0100." <20061114100441.GA24310@crusaders.no> Message-ID: <10132.1163499392@critter.freebsd.dk> In message <20061114100441.GA24310 at crusaders.no>, Trond Michelsen writes: >Well, since we're using tiled images, the number of potential URLs >becomes quite large very fast. At any given zoomlevel, there are >4**zoomlevel tiles. So at zoomlevel 12 (which we use for Google >Earth), there are 16,7 million tiles. We have about 20 different >datalayers, and every layer contains data for every hour for 48 >hours. That means that we have to purge up to 16 billion tiles when a >datamodel is updated (about every 12 hours)[1]. Since our internal >WMS-server is fairly slow, we really, really don't want to generate >any tiles more than once, so it's important for us that Varnish keeps >everything in cache until it's purged. This is exactly what Varnish's url.purge facility is supposed to help you do. All that is required is that you can express the regexp to match what you want to purge. Right now url.purge is not available in VCL, but that is on the list of todo items. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From gaute at pht.no Tue Nov 14 10:19:05 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 11:19:05 +0100 Subject: Purging. Now what? In-Reply-To: <10024.1163497838@critter.freebsd.dk> References: <10024.1163497838@critter.freebsd.dk> Message-ID: <200611141119.07500.gaute@pht.no> On Tuesday 14 November 2006 10:50, Poul-Henning Kamp wrote: > One of my friends have had "So much code to hack, so little time" > in his signature for many years. > > Yes, man of the features you request are already talked about and > planned, it's just a matter coding. > > But things do take time. That's something of which I am very aware :) A pitty I am nowhere near good enough with C to be able to contribute anything directly. I guess some "duplication of effort in python" is unavoidable then :) I would suggest to my company to contribute a bit financially, but with the model you suggest for that in the "version 2 and beyond" mail, I'm afraid that it would only be a small drop in the bucket, and as such rather impractical. Gaute From gaute at pht.no Tue Nov 14 10:58:27 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 11:58:27 +0100 Subject: Purging. Now what? In-Reply-To: <20061114100441.GA24310@crusaders.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <20061114100441.GA24310@crusaders.no> Message-ID: <200611141158.27949.gaute@pht.no> On Tuesday 14 November 2006 11:04, Trond Michelsen wrote: > On Mon, Nov 13, 2006 at 09:53:39PM +0100, Anders Berg wrote: > > With regards to your question about purging I must admit that I am > > unsure what you are trying to achieve. Could we be talking corner > > case here? > > It's possible that I have an edge case for you :) > Following the saying that one should allways try to make ones blunders in front of people who can tell you where you whent wrong, I will hazard a guess: Your best option is to keep track of every single tile that gets generated, and then do a match against that data to find what to purge when something changes. That way you can use your own regexps, in your own data, in the language of your choice, to find all the exact urls to purge, that have actually been served in your choosen period of time. If things change only a few times a day, I think I would just parse the logs there and then. Gaute From trondmm-varnish at crusaders.no Tue Nov 14 12:25:37 2006 From: trondmm-varnish at crusaders.no (Trond Michelsen) Date: Tue, 14 Nov 2006 13:25:37 +0100 Subject: Purging. Now what? In-Reply-To: <200611141158.27949.gaute@pht.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <20061114100441.GA24310@crusaders.no> <200611141158.27949.gaute@pht.no> Message-ID: <20061114122537.GB13921@crusaders.no> On Tue, Nov 14, 2006 at 11:58:27AM +0100, Gaute Amundsen wrote: > On Tuesday 14 November 2006 11:04, Trond Michelsen wrote: >> On Mon, Nov 13, 2006 at 09:53:39PM +0100, Anders Berg wrote: >>> With regards to your question about purging I must admit that I am >>> unsure what you are trying to achieve. Could we be talking corner >>> case here? >> It's possible that I have an edge case for you :) > Following the saying that one should allways try to make ones blunders in > front of people who can tell you where you whent wrong, I will hazard a > guess: > > Your best option is to keep track of every single tile that gets generated, > and then do a match against that data to find what to purge when something > changes. > That way you can use your own regexps, in your own data, in the language of > your choice, to find all the exact urls to purge, that have actually been > served in your choosen period of time. > > If things change only a few times a day, I think I would just parse the logs > there and then. That seems unneccessary complex. I have to have some way of finding the relevant URLs from the log, and that's likely to be a regex. And I can't really see how it's easier to use a regex on a logfile to find relevant URLs, then purging those URLs individually. Instead of simply telling Varnish to purge all URLs matching the very same regex. -- Trond Michelsen From gaute at pht.no Tue Nov 14 13:48:18 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 14:48:18 +0100 Subject: Purging. Now what? In-Reply-To: <20061114122224.GA13921@crusaders.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611141158.27949.gaute@pht.no> <20061114122224.GA13921@crusaders.no> Message-ID: <200611141448.19382.gaute@pht.no> On Tuesday 14 November 2006 13:22, Trond Michelsen wrote: > On Tue, Nov 14, 2006 at 11:58:27AM +0100, Gaute Amundsen wrote: > > On Tuesday 14 November 2006 11:04, Trond Michelsen wrote: > >> On Mon, Nov 13, 2006 at 09:53:39PM +0100, Anders Berg wrote: > >>> With regards to your question about purging I must admit that I am > >>> unsure what you are trying to achieve. Could we be talking corner > >>> case here? > >> > >> It's possible that I have an edge case for you :) > > > > Following the saying that one should allways try to make ones blunders in > > front of people who can tell you where you whent wrong, I will hazard a > > guess: > > > > Your best option is to keep track of every single tile that gets > > generated, and then do a match against that data to find what to purge > > when something changes. > > That way you can use your own regexps, in your own data, in the language > > of your choice, to find all the exact urls to purge, that have actually > > been served in your choosen period of time. > > > > If things change only a few times a day, I think I would just parse the > > logs there and then. > > That seems unneccessary complex. I have to have some way of finding > the relevant URLs from the log, and that's likely to be a regex. And I > can't really see how it's easier to use a regex on a logfile to find > relevant URLs, then purging those URLs individually. Instead of simply > telling Varnish to purge all URLs matching the very same regex. Well, as I have been hashing out for the last few days, that is not implemented 100% yet. You can (1) purge by regexp on the management console, but it does not know about the hostname. or (2) you can purge individual urls by HTTP PURGE, hostname and all. In your case, perhaps you only have a few hostnames, so (1) will work? Gaute From regebro at gmail.com Tue Nov 14 14:39:03 2006 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 14 Nov 2006 15:39:03 +0100 Subject: Purging. Now what? In-Reply-To: <200611141033.38872.gaute@pht.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611140944.23359.gaute@pht.no> <319e029f0611140107r4a79ccc0yc45be0b8cad66600@mail.gmail.com> <200611141033.38872.gaute@pht.no> Message-ID: <319e029f0611140639l6ebaa663j34db245d60f57698@mail.gmail.com> On 11/14/06, Gaute Amundsen wrote: > To be able to "watch the cache" would give me confidence that we where not > caching things that should not be. Oh, you are worried about URLs disappearing when you modify stuff but still being cached... Yeah, that can be tricky. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From gaute at pht.no Tue Nov 14 15:42:00 2006 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 14 Nov 2006 16:42:00 +0100 Subject: Purging. Now what? In-Reply-To: <319e029f0611140639l6ebaa663j34db245d60f57698@mail.gmail.com> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611141033.38872.gaute@pht.no> <319e029f0611140639l6ebaa663j34db245d60f57698@mail.gmail.com> Message-ID: <200611141642.01623.gaute@pht.no> On Tuesday 14 November 2006 15:39, Lennart Regebro wrote: > On 11/14/06, Gaute Amundsen wrote: > > To be able to "watch the cache" would give me confidence that we where > > not caching things that should not be. > > Oh, you are worried about URLs disappearing when you modify stuff but > still being cached... Yeah, that can be tricky. More worried that I could be caching some obscure search result, or other "must be dynamic" page deep in a site somewhere. As it is I have to purge EVERYTHING each time something changes, so "ghost urls" are not a broblem, - yet. Controll, that is the main thing :) Gaute From regebro at gmail.com Tue Nov 14 16:58:07 2006 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 14 Nov 2006 17:58:07 +0100 Subject: Purging. Now what? In-Reply-To: <200611141642.01623.gaute@pht.no> References: <91488996-35B2-497A-ADE3-3811B10C542C@vgnett.no> <200611141033.38872.gaute@pht.no> <319e029f0611140639l6ebaa663j34db245d60f57698@mail.gmail.com> <200611141642.01623.gaute@pht.no> Message-ID: <319e029f0611140858k5d892828r4e63e8c69e63a50f@mail.gmail.com> On 11/14/06, Gaute Amundsen wrote: > More worried that I could be caching some obscure search result, or other > "must be dynamic" page deep in a site somewhere. Ah, of course, you can cache search-results as well, I didn't even think of that. :-) > As it is I have to purge EVERYTHING each time something changes, so "ghost > urls" are not a broblem, - yet. Well, it's a brutal method, but it works, assuming you update your site much less frequently than you cache times out. ;-) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From regebro at gmail.com Thu Nov 16 11:11:21 2006 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 16 Nov 2006 12:11:21 +0100 Subject: Assertion error in ev_now() Message-ID: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> Hi! Any insight into this: Assert error in ev_now(), mgt_event.c line 81: Condition(clock_gettime(CLOCK_MONOTONIC,&ts) == 0) not true. Very weird. Varnish installed fine on the preproduction machine, and this production machine is supposed to be reasonably similar... -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From gaute at pht.no Thu Nov 16 11:25:08 2006 From: gaute at pht.no (Gaute Amundsen) Date: Thu, 16 Nov 2006 12:25:08 +0100 Subject: running varnishncsa -aw Message-ID: <200611161225.09393.gaute@pht.no> Just a quick bash question. I expect I will have to run "varnishncsa -aw finename" at some point to gather logs for analysis. But how do I start and run a shell tool like this as a daemon? I could have it running in a screen session I guess, but I'm sure there is a better way.. Anyone? Gaute. From regebro at gmail.com Thu Nov 16 13:14:44 2006 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 16 Nov 2006 14:14:44 +0100 Subject: Assertion error in ev_now() In-Reply-To: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> References: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> Message-ID: <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> On 11/16/06, Lennart Regebro wrote: > Assert error in ev_now(), mgt_event.c line 81: > Condition(clock_gettime(CLOCK_MONOTONIC,&ts) == 0) not true. > > Very weird. Varnish installed fine on the preproduction machine, and > this production machine is supposed to be reasonably similar... I copied the installation from teh preproduction to the production, and got the same error, so it's not from a difference in compilation, but a difference in the environment, either OS, libraries or hardware. But I have no idea what. Any ideas welcome. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From regebro at gmail.com Thu Nov 16 16:26:40 2006 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 16 Nov 2006 17:26:40 +0100 Subject: Assertion error in ev_now() In-Reply-To: <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> References: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> Message-ID: <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> On 11/16/06, Lennart Regebro wrote: > On 11/16/06, Lennart Regebro wrote: > > Assert error in ev_now(), mgt_event.c line 81: > > Condition(clock_gettime(CLOCK_MONOTONIC,&ts) == 0) not true. > > > > Very weird. Varnish installed fine on the preproduction machine, and > > this production machine is supposed to be reasonably similar... > > I copied the installation from teh preproduction to the production, > and got the same error, so it's not from a difference in compilation, > but a difference in the environment, either OS, libraries or hardware. > But I have no idea what. Any ideas welcome. New adventures in Varnish: Turns out the machines have wildly differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the difference? 2.4... that's pretty old. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From ingvar at linpro.no Fri Nov 17 11:01:13 2006 From: ingvar at linpro.no (Ingvar Hagelund) Date: Fri, 17 Nov 2006 12:01:13 +0100 Subject: Assertion error in ev_now() In-Reply-To: <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> References: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> Message-ID: <455D9679.7070903@linpro.no> * Lennart Regebro > New adventures in Varnish: Turns out the machines have wildly > differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the > difference? 2.4... that's pretty old. Varnish is designed for FreeBSD 6 and Linux 2.6. It will not work properly with Linux 2.4. Ingvar -- N?r alt annet feiler: Symlink From regebro at gmail.com Fri Nov 17 13:24:20 2006 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 17 Nov 2006 14:24:20 +0100 Subject: Assertion error in ev_now() In-Reply-To: <455D9679.7070903@linpro.no> References: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> <455D9679.7070903@linpro.no> Message-ID: <319e029f0611170524x189f87c5qc812d14d79b8ed7c@mail.gmail.com> On 11/17/06, Ingvar Hagelund wrote: > * Lennart Regebro > > New adventures in Varnish: Turns out the machines have wildly > > differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the > > difference? 2.4... that's pretty old. > > Varnish is designed for FreeBSD 6 and Linux 2.6. It will not work > properly with Linux 2.4. OK, that's what I though. Thanks for the confirmation. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From andersb at vgnett.no Fri Nov 17 18:22:50 2006 From: andersb at vgnett.no (Anders Berg) Date: Fri, 17 Nov 2006 19:22:50 +0100 Subject: Assertion error in ev_now() Message-ID: <9F25394B-F270-4426-AA13-84EE3426D0B2@vgnett.no> Varnish does not support Linux 2.4. It might work, but we will never check if it does. The reason is that Varnish uses "modern" OS features. That's the reason it performs so well :) YS Anders Berg > Message: 4 > Date: Thu, 16 Nov 2006 17:26:40 +0100 > From: "Lennart Regebro" > Subject: Re: Assertion error in ev_now() > To: varnish-misc at projects.linpro.no > Message-ID: > <319e029f0611160826i2388b4efn4cc158a3dea34886 at mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 11/16/06, Lennart Regebro wrote: > >> On 11/16/06, Lennart Regebro wrote: >> >>> Assert error in ev_now(), mgt_event.c line 81: >>> Condition(clock_gettime(CLOCK_MONOTONIC,&ts) == 0) not true. >>> >>> Very weird. Varnish installed fine on the preproduction machine, and >>> this production machine is supposed to be reasonably similar... >>> >> >> I copied the installation from teh preproduction to the production, >> and got the same error, so it's not from a difference in compilation, >> but a difference in the environment, either OS, libraries or >> hardware. >> But I have no idea what. Any ideas welcome. >> > > New adventures in Varnish: Turns out the machines have wildly > differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the > difference? 2.4... that's pretty old. From gaute at pht.no Mon Nov 20 09:32:03 2006 From: gaute at pht.no (Gaute Amundsen) Date: Mon, 20 Nov 2006 10:32:03 +0100 Subject: running varnishncsa -aw In-Reply-To: <200611161225.09393.gaute@pht.no> References: <200611161225.09393.gaute@pht.no> Message-ID: <200611201032.03351.gaute@pht.no> FYI. man nohup G. On Thursday 16 November 2006 12:25, Gaute Amundsen wrote: > Just a quick bash question. > > I expect I will have to run "varnishncsa -aw finename" > at some point to gather logs for analysis. > > But how do I start and run a shell tool like this as a daemon? > I could have it running in a screen session I guess, but I'm sure there is > a better way.. > > Anyone? > > Gaute. > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From dwetzel at nerim.net Mon Nov 20 14:42:18 2006 From: dwetzel at nerim.net (Damien Wetzel) Date: Mon, 20 Nov 2006 15:42:18 +0100 Subject: question about number of backends Message-ID: <17761.48842.607135.164426@dwetzel@nerim.net> Hello, Have you planned to have the possibility to have several backend choices depending of the HOST http directive received. The goal would be to manage several domains from one machine . Thank, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL ("`-/")_.-'"``-._ . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' Using Linux requires patience in learning. _.- _..-_/ / ((.' Using Windows requires learning patience. ((,.-' ((,/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From phk at phk.freebsd.dk Mon Nov 20 15:01:38 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 20 Nov 2006 15:01:38 +0000 Subject: question about number of backends In-Reply-To: Your message of "Mon, 20 Nov 2006 15:42:18 +0100." <17761.48842.607135.164426@dwetzel@nerim.net> Message-ID: <4932.1164034898@critter.freebsd.dk> In message <17761.48842.607135.164426 at dwetzel@nerim.net>, Damien Wetzel writes: >Hello, >Have you planned to have the possibility to have several backend choices >depending of the HOST http directive received. >The goal would be to manage several domains from one machine . Yes, this already works. You need to assign the backend you want to use before pipe/pass/fetch is done, so do it in vcl_recv() -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Tue Nov 21 13:12:42 2006 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 21 Nov 2006 14:12:42 +0100 Subject: Assertion error in ev_now() In-Reply-To: <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> (Lennart Regebro's message of "Thu, 16 Nov 2006 17:26:40 +0100") References: <319e029f0611160311p3968affeycd0334e112a9214f@mail.gmail.com> <319e029f0611160514i12ce8f3ake04e4089733a34ae@mail.gmail.com> <319e029f0611160826i2388b4efn4cc158a3dea34886@mail.gmail.com> Message-ID: "Lennart Regebro" writes: > Assert error in ev_now(), mgt_event.c line 81: > Condition(clock_gettime(CLOCK_MONOTONIC,&ts) == 0) not true. > [...] > New adventures in Varnish: Turns out the machines have wildly > differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the > difference? 2.4... that's pretty old. Looks like 2.4 doesn't have a monotonic clock. We should check for that in configure.ac... DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Nov 21 14:10:06 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 21 Nov 2006 14:10:06 +0000 Subject: Assertion error in ev_now() In-Reply-To: Your message of "Tue, 21 Nov 2006 14:12:42 +0100." Message-ID: <1716.1164118206@critter.freebsd.dk> In message , Dag-Erling =?iso-8859-1?Q?Sm=F8rgra v?= writes: >"Lennart Regebro" writes: >> Assert error in ev_now(), mgt_event.c line 81: >> Condition(clock_gettime(CLOCK_MONOTONIC,&ts) =3D=3D 0) not true. >> [...] >> New adventures in Varnish: Turns out the machines have wildly >> differing Linux versions. 2.6.17.6 vs 2.4.31. Could that be the >> difference? 2.4... that's pretty old. > >Looks like 2.4 doesn't have a monotonic clock. We should check for >that in configure.ac... We can fall back to CLOCK_REALTIME in that case, but we suffer ill effects if the clock is stepped (by NTPD or otherwise). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From regebro at gmail.com Tue Nov 21 14:14:39 2006 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 21 Nov 2006 15:14:39 +0100 Subject: Assertion error in ev_now() In-Reply-To: <1716.1164118206@critter.freebsd.dk> References: <1716.1164118206@critter.freebsd.dk> Message-ID: <319e029f0611210614y2266a98j949f8ba2be78c084@mail.gmail.com> On 11/21/06, Poul-Henning Kamp wrote: > We can fall back to CLOCK_REALTIME in that case, but we suffer ill > effects if the clock is stepped (by NTPD or otherwise). And how necessary is it to support Linux 2.4 in the first place? Failing when you run ./configure seems to be a good option to me. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ From sten at blinkenlights.nl Wed Nov 22 15:39:06 2006 From: sten at blinkenlights.nl (Sten Spans) Date: Wed, 22 Nov 2006 16:39:06 +0100 (CET) Subject: Varnish version 2 and beyond In-Reply-To: <11162.1162323669@critter.freebsd.dk> References: <11162.1162323669@critter.freebsd.dk> Message-ID: On Tue, 31 Oct 2006, Poul-Henning Kamp wrote: > > > Varnish version 1 has been out for some time now, and it is time > to talk about varnish version 2 and the future in general. > > I told Anders that I wanted a bit of vacation before I even > contemplated the future. That is why I have been relatively silent > and inactive since the release. > > But I have also talked to a number of early Varnish users and given > a couple of talks about Varnish in various settings and the reception > have been very warm and postive. > > And then everybody asks me "What happens next ? What about version 2 ?" > > So lets me open the discussion about version 2 with a loooong email: > > > What ? > ====== > > The features I have heard people ask for so far are: > > Things we already know how to do: > Bugfixes > Docs > More VCL > Simple Server side includes. > Vary: header processing (requirement for compression etc.) > Compression. > tar storage method for static/emergency contents > The cluster controller I just now started looking at varnish, after watching your eurobsdcon presentation. I'm looking into varnish as a frontend for remote and local freebsd/linux mirrors. A small 1gb package cache reachable via l2 speeds things up quite a bit. The main thing that seems to be missing (for my purposes) is more control over the request sent to the backends, Host: and url rewriting in vcl would be nice. If some more flexibility is added then people who currently use apache+mod_rewrite should feel right at home. A trivial switch to use the backend hostname instead of the client supplied hostname would cover most cases though. Thanks for the great software. -- Sten Spans "There is a crack in everything, that's how the light gets in." Leonard Cohen - Anthem From dwetzel at nerim.net Wed Nov 22 15:52:12 2006 From: dwetzel at nerim.net (Damien Wetzel) Date: Wed, 22 Nov 2006 16:52:12 +0100 Subject: Varnish version 2 and beyond In-Reply-To: References: <11162.1162323669@critter.freebsd.dk> Message-ID: <17764.29228.144108.439121@dwetzel@nerim.net> Hello, Regarding varnish 2 or later, have you had requests to support SSL ? Damien, Sten Spans writes: > On Tue, 31 Oct 2006, Poul-Henning Kamp wrote: > > > > > > > Varnish version 1 has been out for some time now, and it is time > > to talk about varnish version 2 and the future in general. > > > > I told Anders that I wanted a bit of vacation before I even > > contemplated the future. That is why I have been relatively silent > > and inactive since the release. > > > > But I have also talked to a number of early Varnish users and given > > a couple of talks about Varnish in various settings and the reception > > have been very warm and postive. > > > > And then everybody asks me "What happens next ? What about version 2 ?" > > > > So lets me open the discussion about version 2 with a loooong email: > > > > > > What ? > > ====== > > > > The features I have heard people ask for so far are: > > > > Things we already know how to do: > > Bugfixes > > Docs > > More VCL > > Simple Server side includes. > > Vary: header processing (requirement for compression etc.) > > Compression. > > tar storage method for static/emergency contents > > The cluster controller > > I just now started looking at varnish, after watching your eurobsdcon > presentation. I'm looking into varnish as a frontend for remote and local > freebsd/linux mirrors. A small 1gb package cache reachable via l2 speeds > things up quite a bit. > > The main thing that seems to be missing (for my purposes) is more control > over the request sent to the backends, Host: and url rewriting in vcl > would be nice. If some more flexibility is added then people who currently > use apache+mod_rewrite should feel right at home. > > A trivial switch to use the backend hostname instead of the client > supplied hostname would cover most cases though. > > > Thanks for the great software. > > -- > Sten Spans > > "There is a crack in everything, that's how the light gets in." > Leonard Cohen - Anthem > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From dwetzel at nerim.net Wed Nov 22 16:01:45 2006 From: dwetzel at nerim.net (Damien Wetzel) Date: Wed, 22 Nov 2006 17:01:45 +0100 Subject: varnish + GSLB Message-ID: <17764.29801.792201.848144@dwetzel@nerim.net> Hello again, did somebody ever thought of using Varnish in several locations for one domain, using a DNS Global Server Load Balancer to route requests to the appropriate Varnish Cache. I'm thinking about the opportunity to write such a GSLB if it doesn't exist. One idea could be to add into future releases of Varnish some management agents that could be queried by this GSLB to help in routing requests accordingly. Damien, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL ("`-/")_.-'"``-._ . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' Using Linux requires patience in learning. _.- _..-_/ / ((.' Using Windows requires learning patience. ((,.-' ((,/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From regebro at gmail.com Wed Nov 22 16:15:41 2006 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 22 Nov 2006 17:15:41 +0100 Subject: Varnish version 2 and beyond In-Reply-To: <-5856718865213142486@unknownmsgid> References: <11162.1162323669@critter.freebsd.dk> <-5856718865213142486@unknownmsgid> Message-ID: <319e029f0611220815r68a85393u7e7d2b5b6121588@mail.gmail.com> For me SSL support is high on the list. Other than that, it gets to be a question of what path you want to go: More features, like URL rewrites, better virtual hosting, load balancing, etc, etc, or keeping it simple. I'd actually prefer the later. I'm used to having a very "layered" approach to the hosting, with Apache first, then Squid (but now Varnish) then Pound for load balancing and finally the actual web-server. This works fine, but I would like to put Varnish in the front so I can have just one for multiple sites, which is why I's like SSL support. That probably also would get rid of Apache completely in some cases, but not all. And to get rid of the rest you would need to implement Apaches URL rewrite and load balancing and probably a couple of more things. I'd probably prefer Varnish to just do caching, but do it really well. :) //Lennart From phk at phk.freebsd.dk Wed Nov 22 16:18:01 2006 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 22 Nov 2006 16:18:01 +0000 Subject: Varnish version 2 and beyond In-Reply-To: Your message of "Wed, 22 Nov 2006 17:15:41 +0100." <319e029f0611220815r68a85393u7e7d2b5b6121588@mail.gmail.com> Message-ID: <9118.1164212281@critter.freebsd.dk> In message <319e029f0611220815r68a85393u7e7d2b5b6121588 at mail.gmail.com>, "Lenna rt Regebro" writes: >For me SSL support is high on the list. > >Other than that, it gets to be a question of what path you want to go: >More features, like URL rewrites, better virtual hosting, load >balancing, etc, etc, or keeping it simple. The idea behind the VCL language is to be able to have features and still keep it simple. Features implemented via VCL will be design be clearly isolated (and testable!) because they are only activated if the VCL code says so. >This works fine, but I would like to put Varnish in the front so I can >have just one for multiple sites, which is why I's like SSL support. SSL support is noted, but not currently on the short list for version two. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.