From phk at phk.freebsd.dk Mon Aug 1 08:26:43 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 01 Aug 2016 08:26:43 +0000 Subject: short HTTP workshop notes Message-ID: <16671.1470040003@critter.freebsd.dk> As you may remember I spent a week of my summer vacation in Stockholm participating in the HTTP workshop. I want to make absolutely clear that the HTTP workshop is a Good Thing, but at the same time it is deeply troubling. The major troubling thing is that the Internet is rapidly getting feudalized the way Bruce Schenier talkes about here: http://en.collaboratory.de/w/Power_in_the_Age_of_the_Feudal_Internet New developments in HTTP protocol is almost entirely focused on what the big feudal overlords, Google, FaceBook etc. needs. For instance, there were, as far as I can tell, not a single representative for any CMS or law enforcement present in Stockholm, but there were multiple people from Google, Microsoft, FaceBook, Mozilla, Akamai, Fastly etc. There were two people from our end of the "ecosystem": Willy(HAproxy) and me. Proxies are considered outdated/unwanted technology by the big guys. Interestingly the Google people in the workshop seemed to be unaware that Google uses load-balancers. So what happened ? A review of H2 experience. Breaking News: It's not fantastic and a lot of the issues that were pointed out before ratification had merit. Too late now. H2 _can_ be tuned to work, but it is non-trivial and most sites don't know their traffic well enough and don't have the resources. Very few FOSS projects which support HTTP supports HTTP2, they lack developer bandwidth. Newsflash! PUSH is not magic. One report: 47% of all pushes rejected. Band-aids have been proposed for that (cache-digests draft), but a (complex) band-aid on top the PUSH band-aid were not exactly cheered on. Simpler solution may be almost as effective, for instance just count of cached objects for that host. Google has dumped the QUIC UDP transport onto IETF, the same way they dumped SPDY in there: "Take it or leave it, we're doing it anyway." I cloned their github, got 600KLOC, but only about 20-30KLOC seems to be the actual protocol. Not thrilled. Q: Have you planned on standardized QUIC APIs to avoid the OpenSSL/HeartBleed lock-in problem? A: No, not our job. At least Microsoft & I disagree. HTTP3 to HTTP6 were mentioned in various levels of jest, everybody seems to recognize that H2 is not the end. Increasing desire to draw a dotted line through HTTP separating purely semantic stuff from stuff transports may have to deal with. Still N-1 consensus that "we can out-encrypt the politicians" but zero desire or intent to increase client privacy wrt. server (user-agent, cookies, fingerprinting etc.) Most TLS1.3 implementations now talk to each other. Aims for release this autumn. TLS1.3's "0-RTT" brings a new level of complexity, since the "0RTT data" does not have replay protection. FaceBook raised important detail for 0RTT requests: Difference between idempotent (can be retried/replayed and does no damage) and replay-safe (can be replayed and does neither damage nor leak information). Some effort to get a common structure for new headers going forward, but JSON has too many problems. Wrote straw-man on the way home in train, we'll see. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri Aug 5 10:54:26 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 5 Aug 2016 12:54:26 +0200 Subject: MessagePack for CLI responses In-Reply-To: References: Message-ID: Hi all, This is just a thought dump sent from my phone, because I will likely forget all about it by the time I get to my laptop again. I've come across MessagePack (mp) in the past [1] and didn't think much about it, for some reason it recently occurred to me it could be a good fit for the CLI. mp is a binary equivalent to json, and it's no secret that I really don't like json for general purpose serialization (read: unless it's consumed by JavaScript, and I don't hold JavaScript dear in my heart). However mp has some interesting boundaries for types that don't exist with json. The CLI could get structured responses from the child serialized as mp instead of plain text. It could either be the current scheme of a line containing metadata about the response followed by a payload, or a single mp response containing the status and the structured response. I like the latter better because the response could be streamed instead of built upfront. It could improve the situation when you have thousands of backends for instance. If the CLI is binary, it's no longer human-friendly. However the responsibility of rendering could be shift in varnishadm with the possibility of outputting either json or plain text. Setups (like varnish-agent) relying on -T or -M would need to understand mp, possibly with the help of libvarnishapi. Having structured responses could also mean the introduction of schemas. With the need for versioning of the CLI protocol similar to VRT (breaking vs adding). Commands handled by the manager should obviously also speak mp too. The spec [2] is very straightforward and even allows extension points, although I don't think we'd need any. Native support for blobs too, but again I don't see a use case for the CLI. varnishstat could also have mp rendering in addition to the existing XML and json. Dridi, enjoying cloudy-but-sunny Hamburg [1] http://msgpack.org [2] https://github.com/msgpack/msgpack/blob/master/spec.md -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Aug 5 11:06:49 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 05 Aug 2016 11:06:49 +0000 Subject: MessagePack for CLI responses In-Reply-To: References: Message-ID: <25665.1470395209@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >I've come across MessagePack (mp) in the past [1] and didn't think much >about it, for some reason it recently occurred to me it could be a good fit >for the CLI. I'm no particular fan of JSON either, but until a clear consensus materializes in the "binary-JSON" space, I'm not picking sides. In addition to MP, there is also CBOR and a number of $FOO-JSON efforts. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Aug 8 08:20:20 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 08 Aug 2016 08:20:20 +0000 Subject: Coverity issues in varnishhist Message-ID: <4351.1470644420@critter.freebsd.dk> They both look serious-ish... 2 new defect(s) introduced to varnish found with Coverity Scan. New defect(s) Reported-by: Coverity Scan Showing 2 of 2 defect(s) ** CID 1364600: Concurrent data access violations (MISSING_LOCK) /bin/ varnishhist/varnishhist.c: 216 in upd_vsl_ts() _____________________________________________________________________________________________________ _ * CID 1364600: Concurrent data access violations (MISSING_LOCK) /bin/ varnishhist/varnishhist.c: 216 in upd_vsl_ts() 210 if (p == NULL) 211 return; 212 213 t = strtod(p + 1, NULL); 214 215 if (t > vsl_ts) CID 1364600: Concurrent data access violations (MISSING_LOCK) Accessing ?vsl_ts? without holding lock ?mtx?. Elsewhere, ?vsl_ts? is accessed with ?mtx? held 1 out of 2 times (1 of these accesses strongly imply that it is necessary). 216 vsl_ts = t; 217 } 218 219 static int /*__match_proto__ (VSLQ_dispatch_f)*/ 220 accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], 221 void *priv) ** CID 1364599: Control flow issues (MISSING_BREAK) /bin/varnishhist/ varnishhist.c: 271 in accumulate() _____________________________________________________________________________________________________ _ * CID 1364599: Control flow issues (MISSING_BREAK) /bin/varnishhist/ varnishhist.c: 271 in accumulate() 265 !strcasecmp(VSL_CDATA(tr->c->rec.ptr), 266 "retry")) 267 skip = 1; 268 break; 269 case SLT_Timestamp: 270 tsp = VSL_CDATA(tr->c->rec.ptr); CID 1364599: Control flow issues (MISSING_BREAK) The above case falls through to this one. 271 default: 272 if (tag != match_tag) 273 break; 274 275 if (active_profile-> prefix && 276 strncmp(VSL_CDATA(tr->c->rec.ptr), -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Mon Aug 8 13:03:55 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 08 Aug 2016 13:03:55 +0000 Subject: Coverity issues in varnishhist In-Reply-To: <4351.1470644420@critter.freebsd.dk> References: <4351.1470644420@critter.freebsd.dk> Message-ID: I've pushed changes to fix these issues. Martin On Mon, 8 Aug 2016 at 10:24 Poul-Henning Kamp wrote: > They both look serious-ish... > > 2 new defect(s) introduced to varnish found with Coverity Scan. > > New defect(s) Reported-by: Coverity Scan Showing 2 of 2 defect(s) > > ** CID 1364600: Concurrent data access violations (MISSING_LOCK) /bin/ > varnishhist/varnishhist.c: 216 in upd_vsl_ts() > > > _____________________________________________________________________________________________________ > _ * CID 1364600: Concurrent data access violations (MISSING_LOCK) /bin/ > varnishhist/varnishhist.c: 216 in upd_vsl_ts() 210 if (p == NULL) 211 > return; > 212 213 t = strtod(p + 1, NULL); 214 215 if (t > vsl_ts) > > CID 1364600: Concurrent data access violations (MISSING_LOCK) > Accessing ?vsl_ts? without holding lock ?mtx?. Elsewhere, > ?vsl_ts? > is accessed with ?mtx? held 1 out of 2 times (1 of these > accesses > strongly imply that it is necessary). > > 216 vsl_ts = t; 217 } 218 219 static int /*__match_proto__ > (VSLQ_dispatch_f)*/ > 220 accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], > 221 > void *priv) > > ** CID 1364599: Control flow issues (MISSING_BREAK) /bin/varnishhist/ > varnishhist.c: 271 in accumulate() > > > _____________________________________________________________________________________________________ > _ * CID 1364599: Control flow issues (MISSING_BREAK) /bin/varnishhist/ > varnishhist.c: 271 in accumulate() 265 > !strcasecmp(VSL_CDATA(tr->c->rec.ptr), > 266 "retry")) 267 skip = 1; 268 break; 269 case SLT_Timestamp: 270 tsp = > VSL_CDATA(tr->c->rec.ptr); > > CID 1364599: Control flow issues (MISSING_BREAK) The above case > falls through to this one. > > 271 default: 272 if (tag != match_tag) 273 break; 274 275 if > (active_profile-> > prefix && 276 strncmp(VSL_CDATA(tr->c->rec.ptr), > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Tue Aug 23 11:13:20 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 23 Aug 2016 13:13:20 +0200 Subject: libvmod-example moved to /varnishcache/ Message-ID: <20160823111319.GA12002@immer.varnish-software.com> Hi all. To make collaboration easier, I've moved vmod-example to: https://github.com/varnishcache/libvmod-example/ Old location on /varnish/libvmod-example still has some open pull requests and issues, I'll try to handle those. I've added a README with the new link. -- Lasse Karstensen Varnish Software AS From slink at schokola.de Mon Aug 29 16:12:08 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 29 Aug 2016 18:12:08 +0200 Subject: [PATCH] PRIV pointers available in vmod obj methods In-Reply-To: <564F04EC.80404@uplex.de> References: <562E2362.2050104@uplex.de> <91548.1446634707@critter.freebsd.dk> <564F04EC.80404@uplex.de> Message-ID: <0ca48785-15e9-2f74-8af5-8d7dbe130794@schokola.de> I have updated https://github.com/varnishcache/varnish-cache/wiki/VIP1:-PRIV_*-visibility-and-lifetime-control with an Alternative Idea From dridi at varni.sh Tue Aug 30 06:17:12 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 30 Aug 2016 08:17:12 +0200 Subject: [PATCH] PRIV pointers available in vmod obj methods In-Reply-To: <0ca48785-15e9-2f74-8af5-8d7dbe130794@schokola.de> References: <562E2362.2050104@uplex.de> <91548.1446634707@critter.freebsd.dk> <564F04EC.80404@uplex.de> <0ca48785-15e9-2f74-8af5-8d7dbe130794@schokola.de> Message-ID: On Mon, Aug 29, 2016 at 6:12 PM, Nils Goroll wrote: > I have updated > https://github.com/varnishcache/varnish-cache/wiki/VIP1:-PRIV_*-visibility-and-lifetime-control > with an Alternative Idea I like the idea of making it both someone else's problem and making it as easy as possible for them. From slink at schokola.de Tue Aug 30 20:21:12 2016 From: slink at schokola.de (Nils Goroll) Date: Tue, 30 Aug 2016 22:21:12 +0200 Subject: [PATCH] PRIV pointers available in vmod obj methods In-Reply-To: References: Message-ID: On 26/10/15 13:32, Arianna Aondio wrote: > Hi, > as discussed at bugwash today, I'm posting a patch that makes private > pointers(PRIV_TASK, PRIV_TOP, PRIV_VCL) available in VMODS objects. > Please refer to trac ticket #1800 for more details. There was only one line missing to get this for current master, see https://github.com/varnishcache/varnish-cache/pull/2061 lib/libvcc/vcc_vmod.c Nils