From guillaume.quintard at smartjog.com Mon Dec 2 14:56:23 2013 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Mon, 2 Dec 2013 15:56:23 +0100 Subject: Varnishtop on the new logging API Message-ID: <529C9F97.60209@smartjog.com> As said on IRC, here's a shot at porting varnishtop on the new logging API. I didn't change much, maybe not enough (RB tree freeing?), but it seems to work. Comments welcome. -- Guillaume Quintard -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Port-varnishtop-to-the-new-logging-API.patch Type: text/x-patch Size: 9212 bytes Desc: not available URL: From guillaume.quintard at smartjog.com Tue Dec 3 14:43:31 2013 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Tue, 3 Dec 2013 15:43:31 +0100 Subject: PipeEnd proposal, and question on ReqEnd Message-ID: <529DEE13.8000106@smartjog.com> Hello all, To get the ball running, here's a naive approach of what we could see in the PipeEnd line (sister of ReqEnd and BereqEnd) in term of timing/accounting. ---PipeEnd--- Timestamps: t_req TS when complete HTTP request-headers received t_toclient TS when we stop sending to the client t_tobackend TS when we stop sending to the backend t_idle TS when everything is done Fields: Timestamp (since epoch) when the request started t_req Timestamp (since epoch) when the request ended t_idle Total request time t_idle - t_req Time to ready response t_resp - t_req Time to transmit response t_idle - t_resp Of course, I'd still like to have ReqHdr size ReqBody size BereqAll size Maybe we can monitor de first few buffers we receive from the backend for a "\n\n" and so be able to split BereqAll in BereqHdr and BereqBody, but I'm not sure it's useful to anybody (though it would be consistent with ReqEnd and BereqEnd). -- Guillaume Quintard From tfheen at varnish-software.com Wed Dec 4 14:07:40 2013 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 4 Dec 2013 15:07:40 +0100 Subject: More on the HAProxy proxy protocol In-Reply-To: <38901.1384197548@critter.freebsd.dk> References: <20131111153413.GA27258@immer.varnish-software.com> <38901.1384197548@critter.freebsd.dk> Message-ID: <20131204140740.GA1096@err.no> ]] Poul-Henning Kamp > I've been thinking about something like this: > > remote.ip // [IP Other end of TCP connection > remote.port // [INT Our sockets peer-address > > local.ip // [IP own end of the TCP connection > local.port // [INT sockets local address > > > client.ip // [IP] Which IP$ client to connected to our end from. > // if proto == PROXY > // set from PROXY.hdr > // else > // set from remote.ip > > server.ip // [IP] Which IP# client connected to in our end. > server.port // [INT] > // if proto == PROXY > // set from PROXY.hdr > // else > // set from our.* These work for me. > client.identity // Best case ultimate client identity > // if X-F-F: > // set from X-F-F > // else > // set from client.ip > > I'm somewhat tempted to make client.identity a STRING, rather than > an IP, to make it clear to people that running it through an ACL > is a bad idea. client.identity is already a string, and I don't think we should set it from X-F-F, but rather just client.ip. It can be trivially overridden if the sysadmin wants that. -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From phk at phk.freebsd.dk Fri Dec 13 12:27:46 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 13 Dec 2013 12:27:46 +0000 Subject: New VCL_BLOB type Message-ID: <8551.1386937666@critter.freebsd.dk> As agreed at the VDD in Berlin, I have added a "BLOB" type to the VCC. It can only be used as return value and argument to VMOD functions. I've reused the vmod_priv structure as carrier for BLOBs and added a length field to it for this purpose. Examples in libvmod_debug and m00012.vtc Please provide feedback ASAP-ish if this implementation will not work for the usecases we talked about (vmod-digest etc.) so we do not cast a bad implementation in stone for 4.0-R. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Sat Dec 14 01:22:28 2013 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sat, 14 Dec 2013 01:22:28 +0000 Subject: PATCH: fix dependency Message-ID: <20131214012228.GA30708@nitzer> Hi, The diff below ensures the dependency on vsc_fields.h is always present. f.- diff --git a/man/Makefile.am b/man/Makefile.am index 56b0a52..477d37d 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -32,11 +32,11 @@ else @false endif +varnish-counters.7: vsc2rst \ + $(top_srcdir)/include/tbl/vsc_fields.h if HAVE_RST2MAN -varnish-counters.7: vsc2rst ./vsc2rst | ${RST2MAN} - $@ else -varnish-counters.7: $(top_srcdir)/include/tbl/vsc_fields.h @echo "========================================" @echo "You need rst2man installed to make dist" @echo "========================================" From fgsch at lodoss.net Sat Dec 14 01:29:16 2013 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sat, 14 Dec 2013 01:29:16 +0000 Subject: PATCH: fix warning on systems without TCP_KEEPxxx Message-ID: <20131214012916.GB30708@nitzer> Hi, Just a reminder. Not all OS support TCP_KEEPxxx, OpenBSD being one of them. f.- diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index 6a601a8..3138f9f 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -387,7 +387,9 @@ vca_acct(void *arg) { struct listen_sock *ls; double t0, now; +#ifdef HAVE_TCP_KEEP unsigned u; +#endif int i; THR_SetName("cache-acceptor"); From varnish at syrota.com Sat Dec 21 20:02:38 2013 From: varnish at syrota.com (varnish at syrota.com) Date: Sat, 21 Dec 2013 14:02:38 -0600 Subject: varnishtest documentation Message-ID: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> Hi, I've started using varnishtest framework recently, and have an extremely hard time finding documentation on specifics. So I started writing my own. I think it would be valuable for the project to include it in the distribution, so decided to write to this list. You can check out my first attempt at reverse engineering varnishtest script commands here: http://blog.syrota.com/2013/12/varnishtest-script-file-commands.html Since there is no page (that I can find) that talks about script file structure and commands, I'm not sure where to submit a pull request at this time. In addition, there are a lot of holes that need to be filled before it's good for general public usage. What would you think be the best route to proceed with? Can someone help fill those question marks and correct any errors I might have made? Best regards, Sergey Syrota From dridi.boukelmoune at zenika.com Sat Dec 21 21:27:42 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Sat, 21 Dec 2013 22:27:42 +0100 Subject: varnishtest documentation In-Reply-To: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> Message-ID: Hi Sergey, A vtc(7) man page is on my todo-list, maybe we could work on it together ? I did too read the source code to understand it better, and wrote a blog post [1] too ! I'm glad I'm not alone thinking that varnishtest should be adopted beyond varnish and modules. I also have another blog post coming, which shows among other things how to implement it on a project. I'm bookmarking your post, it's even more useful since I haven't dug that much into varnish 4, aside from keeping my vmod up to date, and that you cover new features like logexpect :) Best Regards, Dridi [1] http://blog.zenika.com/index.php?post/2012/08/27/Introducing-varnishtest On Sat, Dec 21, 2013 at 9:02 PM, wrote: > Hi, > > I've started using varnishtest framework recently, and have an extremely > hard time finding documentation on specifics. So I started writing my own. I > think it would be valuable for the project to include it in the > distribution, so decided to write to this list. > > You can check out my first attempt at reverse engineering varnishtest script > commands here: > http://blog.syrota.com/2013/12/varnishtest-script-file-commands.html > > Since there is no page (that I can find) that talks about script file > structure and commands, I'm not sure where to submit a pull request at this > time. In addition, there are a lot of holes that need to be filled before > it's good for general public usage. > > What would you think be the best route to proceed with? Can someone help > fill those question marks and correct any errors I might have made? > > Best regards, > Sergey Syrota > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From varnish at syrota.com Sat Dec 21 21:42:57 2013 From: varnish at syrota.com (varnish at syrota.com) Date: Sat, 21 Dec 2013 15:42:57 -0600 Subject: varnishtest documentation In-Reply-To: References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> Message-ID: <20131221154257.51042usrcnlokjeo@art.yamanotecorporation.com> Yeah, your blog is one I relied on heavily to get me off the ground with my project. Nice writeup :) Yes, we can work together on it. Man page might be suitable place for it. Is this a convention? And if so, is this the place for those man pages: https://github.com/varnish/Varnish-Cache/tree/master/doc/sphinx/phk? I can submit a draft containing info in my blog post, but this will need to be amended before can be merged into the repository. Quoting Dridi Boukelmoune : > Hi Sergey, > > A vtc(7) man page is on my todo-list, maybe we could work on it together ? > > I did too read the source code to understand it better, and wrote a > blog post [1] too ! I'm glad I'm not alone thinking that varnishtest > should be adopted beyond varnish and modules. I also have another blog > post coming, which shows among other things how to implement it on a > project. I'm bookmarking your post, it's even more useful since I > haven't dug that much into varnish 4, aside from keeping my vmod up to > date, and that you cover new features like logexpect :) > > Best Regards, > Dridi > > [1] http://blog.zenika.com/index.php?post/2012/08/27/Introducing-varnishtest > From dridi.boukelmoune at zenika.com Sat Dec 21 21:54:51 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Sat, 21 Dec 2013 22:54:51 +0100 Subject: varnishtest documentation In-Reply-To: <20131221154257.51042usrcnlokjeo@art.yamanotecorporation.com> References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> <20131221154257.51042usrcnlokjeo@art.yamanotecorporation.com> Message-ID: On Sat, Dec 21, 2013 at 10:42 PM, wrote: > Yeah, your blog is one I relied on heavily to get me off the ground with my > project. Nice writeup :) Thanks ! I'm glad it helped :D > Yes, we can work together on it. Man page might be suitable place for it. Is > this a convention? And if so, is this the place for those man pages: > https://github.com/varnish/Varnish-Cache/tree/master/doc/sphinx/phk? I can > submit a draft containing info in my blog post, but this will need to be > amended before can be merged into the repository. Look at the vcl(7) source [1] that will be both turned into a man page and an html page [2]. Dridi [1] https://github.com/varnish/Varnish-Cache/blob/3.0/doc/sphinx/reference/vcl.rst [2] https://www.varnish-cache.org/docs/3.0/reference/vcl.html > Quoting Dridi Boukelmoune : > >> Hi Sergey, >> >> A vtc(7) man page is on my todo-list, maybe we could work on it together ? >> >> I did too read the source code to understand it better, and wrote a >> blog post [1] too ! I'm glad I'm not alone thinking that varnishtest >> should be adopted beyond varnish and modules. I also have another blog >> post coming, which shows among other things how to implement it on a >> project. I'm bookmarking your post, it's even more useful since I >> haven't dug that much into varnish 4, aside from keeping my vmod up to >> date, and that you cover new features like logexpect :) >> >> Best Regards, >> Dridi >> >> [1] >> http://blog.zenika.com/index.php?post/2012/08/27/Introducing-varnishtest >> > > From dridi.boukelmoune at zenika.com Sat Dec 21 22:00:08 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Sat, 21 Dec 2013 23:00:08 +0100 Subject: New VCL_REGEX type ? Message-ID: Hello, I hope this is not too late, but I'd like to see a /VCL_REGEXP?/ type for VMODs. I already use regular expressions in my main vmod, but there are two areas where it could greatly improve: - compiling them once and for all, not for each request (worst case) - having them validated at VCL load time What do you think, does this make sense ? Best Regards, Dridi From phk at phk.freebsd.dk Sat Dec 21 22:20:13 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 21 Dec 2013 22:20:13 +0000 Subject: varnishtest documentation In-Reply-To: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> Message-ID: <65579.1387664413@critter.freebsd.dk> In message <20131221140238.61593pibn6wz20w0 at art.yamanotecorporation.com>, varni sh at syrota.com writes: Hi Sergey, Yes, varnishtest is pretty undocumented. I trust it you have spotted the 347 test-descriptions in the source tree ? They're probably the main documentation for it. The main reason varnishtest has not been documented, is that it we weren't sure if anybody would ever bother read the documentation if we wrote it, and given that we have more ideas for things to do than time to do them it ended up somewhat low in the pile. Also, truth to be told, it is more than a little bit obscure, because it has been written to get out into obscure corners of varnishd behaviour, so it has all sorts of really weird magic code, which is not useful in more than one or maybe two of the official test-cases. That said, I often find myself rumaging around the test-cases and source code, to try to remember how to do X with varnishtest so documentation would absolutely be a good idea. So yes, We'd love it if you write a "rst2man" file for varnishtest :-) >script commands here: >http://blog.syrota.com/2013/12/varnishtest-script-file-commands.html A quick top-level architectural overview: The varnishtest program can run test-cases, each one written in a *.vtc file. The "main" part of the program will make a list of tests to be run (-n# + filename arguments) and schedule -j# of them in parallel. Unless -k is given, it stops on the first test-failure. Normally, it will report a single "success" line per test-case or dump the full log if the test-case failed. -q is quiet mode, only test failuers are reported and only with a single summary line. -v is verbose mode, it always dumps the full log. -i tells it that it's running in the source tree (special trickery is involved in finding the varnishd program and the vmods.) Each test-case consists of a number of independent "instances" of various kinds, clients, servers, varnishd's and so on. These are run in separate threads or even processes, to gain concurrency. Most of the instances (client/server/varnish at least) take -start = Start now -wait = Wait for completion -run = -start -wait arguments. Once the test-case is over, all started instances gets a "we're done" signal and an implict -wait behind the scenes. The most typical use case is: server ... -start varnish ... -start client ... -run This will start a server, a varnishd and a client in parallel and wait for the client to complete. Semaphores mostly used to synchronize clients and servers "around" varnish, so that the server will not send something particular until the client tells it to, but it can also be used synchronize multiple clients or servers running in parallel. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From varnish at syrota.com Sat Dec 21 22:41:29 2013 From: varnish at syrota.com (varnish at syrota.com) Date: Sat, 21 Dec 2013 16:41:29 -0600 Subject: varnishtest documentation In-Reply-To: <65579.1387664413@critter.freebsd.dk> References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> <65579.1387664413@critter.freebsd.dk> Message-ID: <20131221164129.1509754fm0tueiw4@art.yamanotecorporation.com> Quoting Poul-Henning Kamp : > I trust it you have spotted the 347 test-descriptions in the > source tree ? They're probably the main documentation for it. Yeah, saw them, and that was one of the main sources on how to use the features. Although some of the arguments I could never find in those test cases > The main reason varnishtest has not been documented, is that it > we weren't sure if anybody would ever bother read the documentation > if we wrote it, and given that we have more ideas for things to > do than time to do them it ended up somewhat low in the pile. Yeah, judging by how hard it is to come by any discussion of it, it is quite rarely used feature, so the priority is understandable :) > A quick top-level architectural overview: Thanks for the overview. That helps with a few things that I had not understood before. I'll probably put together a pull request for this architecture overview (I'll try to fit it into the existing varnishtest page) and vtc commands. And we can go from there, if anyone will be interested. Best regards, Sergey Syrota From phk at phk.freebsd.dk Sat Dec 21 22:42:03 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 21 Dec 2013 22:42:03 +0000 Subject: New VCL_REGEX type ? In-Reply-To: References: Message-ID: <65900.1387665723@critter.freebsd.dk> In message , Dridi Boukelmoune writes: >I hope this is not too late, but I'd like to see a /VCL_REGEXP?/ >type for VMODs. It's on my list from the developers meeting in Berlin and I've done some of the preliminary footwork already. I won't promise they'll be there for 4.0-RELEASE, but they will come in the 4.X series somewhere. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi.boukelmoune at zenika.com Sun Dec 22 09:06:43 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Sun, 22 Dec 2013 10:06:43 +0100 Subject: New VCL_REGEX type ? In-Reply-To: <65900.1387665723@critter.freebsd.dk> References: <65900.1387665723@critter.freebsd.dk> Message-ID: On Sat, Dec 21, 2013 at 11:42 PM, Poul-Henning Kamp wrote: > In message > , Dridi Boukelmoune writes: > >>I hope this is not too late, but I'd like to see a /VCL_REGEXP?/ >>type for VMODs. > > It's on my list from the developers meeting in Berlin and I've done > some of the preliminary footwork already. Thanks, I haven't caught up yet with VUG8 videos, slides and minutes. > I won't promise they'll be there for 4.0-RELEASE, but they will > come in the 4.X series somewhere. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. From manu at gandi.net Tue Dec 31 10:16:13 2013 From: manu at gandi.net (Emmanuel Hocdet) Date: Tue, 31 Dec 2013 11:16:13 +0100 Subject: varnish-dev@varnish-cache.org Message-ID: <267DCE0C-936D-4122-8723-8F1BF2BF5BCB@gandi.net> Hi, I have made a patch to add proxy protocol to Varnish 3.0 you can find it at http://varnish.hocdet.net Emmanuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From manu at gandi.net Tue Dec 31 11:11:38 2013 From: manu at gandi.net (Emmanuel Hocdet) Date: Tue, 31 Dec 2013 12:11:38 +0100 Subject: proxy protocol for varnish 3.0.5 In-Reply-To: <267DCE0C-936D-4122-8723-8F1BF2BF5BCB@gandi.net> References: <267DCE0C-936D-4122-8723-8F1BF2BF5BCB@gandi.net> Message-ID: with the right object... Le 31 d?c. 2013 ? 11:16, Emmanuel Hocdet a ?crit : > Hi, > > I have made a patch to add proxy protocol to Varnish 3.0 > you can find it at http://varnish.hocdet.net > > Emmanuel > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From 79841031 at qq.com Fri Dec 6 02:11:55 2013 From: 79841031 at qq.com (=?gb18030?B?oe/j0ePYIKXp?=) Date: Fri, 06 Dec 2013 02:11:55 -0000 Subject: varnish 3.0.4version may be have memory leak Message-ID: dear: i use varnish 3.0.4 version, and find that varnish eat all my memory at last. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2CFEE7E3 at 2AD4FB5F.5532A152 Type: application/octet-stream Size: 659191 bytes Desc: not available URL: From 79841031 at qq.com Fri Dec 6 02:20:36 2013 From: 79841031 at qq.com (=?gb18030?B?oe/j0ePYIKXp?=) Date: Fri, 06 Dec 2013 02:20:36 -0000 Subject: varnish 3.0.4version may be have memory leak Message-ID: dear: (1) eat my memory i use varnish 3.0.4 version, and find that varnish eat all my memory at last. taking the picture for example, the varnish eat my 3G memory now, and i find the Transient part is more than 2G(no http request sometimes and graces are default)? (2)session_herd also increase too fast. In this case, the n_sess_mem increase too fast untill the max value, then varnish can't work -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 30FCEDD8 at E286680F.6434A152 Type: application/octet-stream Size: 659191 bytes Desc: not available URL: