From phk at phk.freebsd.dk  Mon Jan  2 17:32:59 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jan 2017 17:32:59 +0000
Subject: Experimental bugwash 19:00Z-ish
Message-ID: <54848.1483378379@critter.freebsd.dk>

As mentioned earlier I will experiment with holding a second
bugwash on mondays at a non-EU friendly time.

If you have open tickets, patches or pull requests, you're more
than welcome to join in on our IRC channel (#varnish-hacking on
irc.linpro.no) and chat about it.

ttyl...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Tue Jan  3 06:30:57 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 3 Jan 2017 07:30:57 +0100
Subject: VCL refcounting vs temprerature
Message-ID: <CABoVN9CxSDEaSpmZL_HE+enjrRv13-Q3UVGs7jiWLyjcTAUSPQ@mail.gmail.com>

Hi everyone,

A quick message to make you all aware of a small problem that is
nothing new but interesting to revisit since we have started
discussions to REFCOUNT ALL THE THINGS, starting with backends.

The way workers don't release their VCL (unless we enable the vclrel
debug flag in the CLI) means that a worker going back to the pool may
be doing so while holding a ref to a VCL that may later be discarded.
You need enough traffic to get to use those workers, otherwise there's
no way to complete the discard operation.

It has been true for quite a while, possibly from the beginning, and
AFAICT the first knob to control this behavior appeared in the commit
2023864 (from 2010).

Fast forward to Varnish 4.1, we introduce VCL temperature and instead
of blocking the discard operation it now blocks the cool-down operation.

A VCL may hang in a busy state after either:

- vcl_cooldown (if state is auto)
- vcl.state <name> cold
- vcl.discard <name>

This is nothing new, but it partially defeats the purpose of having a
COLD state with a reduced footprint (which was new). The expansion
of refcounting to more areas is a good occasion to have this case in
mind.

Cheers,
Dridi



From phk at phk.freebsd.dk  Thu Jan  5 16:08:30 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 05 Jan 2017 16:08:30 +0000
Subject: Varnish code coverage results online
Message-ID: <15686.1483632510@critter.freebsd.dk>

I have spent the day retooling the code-coverage stuff I have been
running manually since day almost-one of Varnish.

The results are now online:

	http://varnish-cache.org/gcov

(There's a link in the navbar at the bottom of the project homepage)

I have also, at the same time raised the official project goal to
"90%+ of *all* executable code tested automatically".

Your assistance reaching this goal will be appreciated.

Right now none of the curses(3) code is tested automatically.
Doing so is non-trivial, but doable.

Maybe varnishtest::process should simply (ha!) grow a -pty option ?

Another big chunk of non-tested code is in libvgz, and just
on general principles I might go in there with some lobotomizing
#ifdefs and WRONG()'s, since that is code that we don't want
to ever execute in the first place.

Poul-Henning

PS: Needless to say: The instant we hit the 90% goal I'll raise it again.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Thu Jan  5 17:31:32 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 5 Jan 2017 18:31:32 +0100
Subject: Varnish code coverage results online
In-Reply-To: <15686.1483632510@critter.freebsd.dk>
References: <15686.1483632510@critter.freebsd.dk>
Message-ID: <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>

> I have also, at the same time raised the official project goal to
> "90%+ of *all* executable code tested automatically".

Only line coverage? Nothing for branches?

90% branch coverage is even a bigger challenge ;)

Another thing: if we want proper code coverage reports, we need to
aggregate from different platforms if we hope to cover platform-specific
code. Does lcov(1) work on FreeBSD? I know the "l" stands for Linux
but at least with lcov I _know_ we can merge reports.

The problem would then be: how to collect the reports?

I personally use codecov.io which can receive and aggregate coverage
reports and pin them to commits. There are other similar services, I'm
not trying to promote a particular solution.

Dridi



From KCowan at alm.com  Fri Jan  6 01:01:04 2017
From: KCowan at alm.com (Kari Cowan)
Date: Fri, 6 Jan 2017 01:01:04 +0000
Subject: Simple redirect to correct a URL
Message-ID: <08B718B8CE98354E94D0201C94F374AD8201B381@EXMB01NYC1.nyc.amlaw.corp>

I'm sure I must be doing something hair-brained ... shouldn't this work in V5?

if (req.url~ "^/type/news?$") {
   return (synth (750,"http://" + req.http.Host + "/news/"));
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170106/b918f7cd/attachment.html>

From guillaume at varnish-software.com  Fri Jan  6 08:14:50 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 6 Jan 2017 09:14:50 +0100
Subject: Simple redirect to correct a URL
In-Reply-To: <08B718B8CE98354E94D0201C94F374AD8201B381@EXMB01NYC1.nyc.amlaw.corp>
References: <08B718B8CE98354E94D0201C94F374AD8201B381@EXMB01NYC1.nyc.amlaw.corp>
Message-ID: <CAJ6ZYQwedBQuh9CCMeEjSCqWegBdAGjMJ-zKP9mcJ+oEYojgew@mail.gmail.com>

Not to beat my own drum, but I blogged about this recently:
https://info.varnish-software.com/blog/rewriting-urls-with-varnish-redirection

TL;DR: you need a 30X return code, and to put the new URL in the correct
header (location).

On Jan 6, 2017 02:06, "Kari Cowan" <KCowan at alm.com> wrote:

> I?m sure I must be doing something hair-brained ? shouldn?t this work in
> V5?
>
>
>
> if (req.url~ "^/type/news?$") {
>
>    return (synth (750,"http://" + req.http.Host + "/news/"));
>
> }
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170106/eacda597/attachment.html>

From guillaume at varnish-software.com  Fri Jan  6 08:16:19 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 6 Jan 2017 09:16:19 +0100
Subject: Simple redirect to correct a URL
In-Reply-To: <CAJ6ZYQwedBQuh9CCMeEjSCqWegBdAGjMJ-zKP9mcJ+oEYojgew@mail.gmail.com>
References: <08B718B8CE98354E94D0201C94F374AD8201B381@EXMB01NYC1.nyc.amlaw.corp>
 <CAJ6ZYQwedBQuh9CCMeEjSCqWegBdAGjMJ-zKP9mcJ+oEYojgew@mail.gmail.com>
Message-ID: <CAJ6ZYQxT40AQ3=4xRqH5Vuzufk1n+Kx1HYy84HsuebCQ8vH_Bw@mail.gmail.com>

And I just noticed, please use varnish-misc for this kind of requests,
varnish-dev is for dev topics only.

On Jan 6, 2017 09:14, "Guillaume Quintard" <guillaume at varnish-software.com>
wrote:

> Not to beat my own drum, but I blogged about this recently: https://info.
> varnish-software.com/blog/rewriting-urls-with-varnish-redirection
>
> TL;DR: you need a 30X return code, and to put the new URL in the correct
> header (location).
>
> On Jan 6, 2017 02:06, "Kari Cowan" <KCowan at alm.com> wrote:
>
>> I?m sure I must be doing something hair-brained ? shouldn?t this work in
>> V5?
>>
>>
>>
>> if (req.url~ "^/type/news?$") {
>>
>>    return (synth (750,"http://" + req.http.Host + "/news/"));
>>
>> }
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170106/dbad8361/attachment.html>

From phk at phk.freebsd.dk  Fri Jan  6 11:05:23 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jan 2017 11:05:23 +0000
Subject: Varnish code coverage results online
In-Reply-To: <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
References: <15686.1483632510@critter.freebsd.dk>
 <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
Message-ID: <19397.1483700723@critter.freebsd.dk>

--------
In message <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg at mail.gmail.com>, Dridi Boukelmoune 
writes:

>> I have also, at the same time raised the official project goal to
>> "90%+ of *all* executable code tested automatically".
>
>Only line coverage? Nothing for branches?
>
>90% branch coverage is even a bigger challenge ;)

That's probably impossible, given the number of asserts we have.

>Another thing: if we want proper code coverage reports, [...]

Are my my l33t HTML skills not proper ?!

Seriously:  This is good enough to tell us which bits of our
code isn't covered by "make check", which is the important part.

Merging in data from other runs would not be a problem, my scripts
already has support for merging because some sourcefiles drop .gcda
in multiple directories already.  We'd just need to set up a client
to run "make check" and submit a condensed version of the raw data.

If you feel like running a Linux (or other) client, let me know and
we can get it set up, I think my current plan is to run this only
once per night.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Fri Jan  6 11:21:50 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 6 Jan 2017 12:21:50 +0100
Subject: Varnish code coverage results online
In-Reply-To: <19397.1483700723@critter.freebsd.dk>
References: <15686.1483632510@critter.freebsd.dk>
 <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
 <19397.1483700723@critter.freebsd.dk>
Message-ID: <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>

On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg at mail.gmail.com>, Dridi Boukelmoune
> writes:
>
>>> I have also, at the same time raised the official project goal to
>>> "90%+ of *all* executable code tested automatically".
>>
>>Only line coverage? Nothing for branches?
>>
>>90% branch coverage is even a bigger challenge ;)
>
> That's probably impossible, given the number of asserts we have.

Yes, my approach is to run the test suite with assertions, and disable
them for coverage.

>>Another thing: if we want proper code coverage reports, [...]
>
> Are my my l33t HTML skills not proper ?!
>
> Seriously:  This is good enough to tell us which bits of our
> code isn't covered by "make check", which is the important part.
>
> Merging in data from other runs would not be a problem, my scripts
> already has support for merging because some sourcefiles drop .gcda

Cool, I had no idea!

> in multiple directories already.  We'd just need to set up a client
> to run "make check" and submit a condensed version of the raw data.

And here comes tools/vcov.sh :p

> If you feel like running a Linux (or other) client, let me know and
> we can get it set up, I think my current plan is to run this only
> once per night.

If we come up with something, Travis CI could do that for us for each
push (so some work needs to be done to match the nightly scheduling).
We'd just need to set up a new job, and I think it would even run as
root and skip less tests.

Dridi



From phk at phk.freebsd.dk  Fri Jan  6 11:36:26 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jan 2017 11:36:26 +0000
Subject: Varnish code coverage results online
In-Reply-To: <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>
References: <15686.1483632510@critter.freebsd.dk>
 <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
 <19397.1483700723@critter.freebsd.dk>
 <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>
Message-ID: <19522.1483702586@critter.freebsd.dk>

--------
In message <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg at mail.gmail.com>, Dridi Boukelmoune 
writes:
>On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

>> If you feel like running a Linux (or other) client, let me know and
>> we can get it set up, I think my current plan is to run this only
>> once per night.
>
>If we come up with something, Travis CI could do that for us for each
>push (so some work needs to be done to match the nightly scheduling).

Doing it on each push would be silly.

In particular because it takes one and a half error to run "make
check" with gcov:  Gcov isn't parallel-safe, so all the tests has
to run serially, and the implementation sucks, so it takes forever
to update the .g??? files.

I'll look at it, I hadn't considered multiplatform when I did this.

The simplest is probably to (ab)use vtest to do this.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Fri Jan  6 12:47:46 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 6 Jan 2017 13:47:46 +0100
Subject: Varnish code coverage results online
In-Reply-To: <19522.1483702586@critter.freebsd.dk>
References: <15686.1483632510@critter.freebsd.dk>
 <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
 <19397.1483700723@critter.freebsd.dk>
 <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>
 <19522.1483702586@critter.freebsd.dk>
Message-ID: <CABoVN9Ad=R+qyAGz1+_XYXbzOvr6krHUYVoBi44PyEGUeNm7ig@mail.gmail.com>

On Fri, Jan 6, 2017 at 12:36 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg at mail.gmail.com>, Dridi Boukelmoune
> writes:
>>On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>
>>> If you feel like running a Linux (or other) client, let me know and
>>> we can get it set up, I think my current plan is to run this only
>>> once per night.
>>
>>If we come up with something, Travis CI could do that for us for each
>>push (so some work needs to be done to match the nightly scheduling).
>
> Doing it on each push would be silly.

We may be able to run the last build "with coverage enabled" using
some API. I don't know, I'd need to look at the docs, but this way it
could be done from the Varnish infra.

> In particular because it takes one and a half error to run "make
> check" with gcov:  Gcov isn't parallel-safe, so all the tests has
> to run serially, and the implementation sucks, so it takes forever
> to update the .g??? files.

I thought gcov'ed binaries could run concurrently if the underlying
file-system allowed it. I never had any issue with gcov and parallel
builds and regardless, if varnishtest spawns a varnishd, won't they
compete on coverage collection for libvarnish for instance?

> I'll look at it, I hadn't considered multiplatform when I did this.
>
> The simplest is probably to (ab)use vtest to do this.

I don't know about simplest, running nightly builds from a script that
runs pseudo-continuously doesn't sound straightforward.

Dridi



From phk at phk.freebsd.dk  Fri Jan  6 13:28:54 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jan 2017 13:28:54 +0000
Subject: Varnish code coverage results online
In-Reply-To: <CABoVN9Ad=R+qyAGz1+_XYXbzOvr6krHUYVoBi44PyEGUeNm7ig@mail.gmail.com>
References: <15686.1483632510@critter.freebsd.dk>
 <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>
 <19397.1483700723@critter.freebsd.dk>
 <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>
 <19522.1483702586@critter.freebsd.dk>
 <CABoVN9Ad=R+qyAGz1+_XYXbzOvr6krHUYVoBi44PyEGUeNm7ig@mail.gmail.com>
Message-ID: <19814.1483709334@critter.freebsd.dk>

--------
In message <CABoVN9Ad=R+qyAGz1+_XYXbzOvr6krHUYVoBi44PyEGUeNm7ig at mail.gmail.com>, Dridi Boukelmoune 
writes:

>I thought gcov'ed binaries could run concurrently if the underlying
>file-system allowed it. I never had any issue with gcov and parallel
>builds and regardless, if varnishtest spawns a varnishd, won't they
>compete on coverage collection for libvarnish for instance?

Last I tried it, I got totally screwed up numbers.

(And no, because varnishtest waits for varnishd to exit before it
exits.)

>I don't know about simplest, running nightly builds from a script that
>runs pseudo-continuously doesn't sound straightforward.

if `date +%Y%m%d` changed do gcov ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From KCowan at alm.com  Fri Jan  6 15:15:34 2017
From: KCowan at alm.com (Kari Cowan)
Date: Fri, 6 Jan 2017 15:15:34 +0000
Subject: Simple redirect to correct a URL
In-Reply-To: <CAJ6ZYQwedBQuh9CCMeEjSCqWegBdAGjMJ-zKP9mcJ+oEYojgew@mail.gmail.com>
References: <08B718B8CE98354E94D0201C94F374AD8201B381@EXMB01NYC1.nyc.amlaw.corp>
 <CAJ6ZYQwedBQuh9CCMeEjSCqWegBdAGjMJ-zKP9mcJ+oEYojgew@mail.gmail.com>
Message-ID: <08B718B8CE98354E94D0201C94F374AD8201CDC1@EXMB01NYC1.nyc.amlaw.corp>

Thanks ? I will move any followup questions over to the misc list.  That?s a great article ? just wanted to say.


From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
Sent: Friday, January 06, 2017 12:15 AM
To: Kari Cowan <KCowan at alm.com>
Cc: Varnish Development <varnish-dev at varnish-cache.org>
Subject: Re: Simple redirect to correct a URL

Not to beat my own drum, but I blogged about this recently: https://info.varnish-software.com/blog/rewriting-urls-with-varnish-redirection

TL;DR: you need a 30X return code, and to put the new URL in the correct header (location).

On Jan 6, 2017 02:06, "Kari Cowan" <KCowan at alm.com<mailto:KCowan at alm.com>> wrote:
I?m sure I must be doing something hair-brained ? shouldn?t this work in V5?

if (req.url~ "^/type/news?$") {
   return (synth (750,"http://" + req.http.Host + "/news/"));
}

_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org<mailto:varnish-dev at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170106/b0144dc0/attachment.html>

From phk at phk.freebsd.dk  Fri Jan 13 11:13:58 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 13 Jan 2017 11:13:58 +0000
Subject: Thinking about march 15 release
Message-ID: <70811.1484306038@critter.freebsd.dk>

I have opened a ticket for the march 15 release:

	https://github.com/varnishcache/varnish-cache/issues/2183

First order of the day is to decide if this will be a 5.1 or 6.0 release,
your input is kindly requested (in the ticket!)

I'd like if we can decide this question on mondays bugwash.

I have also created a "r=6.0" label, please tag tickets which would
cause API changes preventing it from going into 5.1 with this tag.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From guillaume at varnish-software.com  Tue Jan 17 09:46:52 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 17 Jan 2017 10:46:52 +0100
Subject: Age precision rounding causes premature stale object
In-Reply-To: <CAAARNrEpYUBVAG3xKCk98NCTGyFrFtJAn0hXRrz=PV9KB+n9yw@mail.gmail.com>
References: <CAAARNrEpYUBVAG3xKCk98NCTGyFrFtJAn0hXRrz=PV9KB+n9yw@mail.gmail.com>
Message-ID: <CAJ6ZYQyF7RS6Am0TWBi9He4rO0CmmikGymxhZXPPsdSzWWKf8w@mail.gmail.com>

I'm in favor but the potential issue lies with layered setups where objects
could be cached for too long (got bit by that one), to the conservative
approach makes sense.

We can probably just put that in a vmod and see how people like it? That'd
be a first step towards a vmod-rfc2616.

-- 
Guillaume
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170117/ff6607b7/attachment.html>

From dridi at varni.sh  Thu Jan 19 16:23:14 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 19 Jan 2017 17:23:14 +0100
Subject: autoconf madness
In-Reply-To: <CABoVN9C3L3RFX+Uyj+r7WvjozQHv383oBEk17fanFv1DK8MTcQ@mail.gmail.com>
References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de>
 <a10c3956-6af9-e82b-c7bc-1fa1e03451e6@schokola.de>
 <CABoVN9C3L3RFX+Uyj+r7WvjozQHv383oBEk17fanFv1DK8MTcQ@mail.gmail.com>
Message-ID: <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ@mail.gmail.com>

On Fri, Dec 30, 2016 at 5:34 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
>> Any ideas?
>
> Not yet, but I started another thread on github [1] so I suggest we
> move the discussion there. As far I'm concerned, I think I'm done
> with Varnish for this year ;)
>
> Cheers
>
> [1] https://github.com/varnishcache/varnish-cache/pull/2174

Just a quick (as in "no, I don't know how to be quick") note on my
non-progress on this issue.

I've been working on #2174 and I made significant progress but I
closed it and stopped all my autotools activities after the recent
loud and clear messages that "autocrap is the next thing to remove".
Not an exact quote, but probably close enough.

I don't care if we switch build systems, but I'm recommending one last
time to stick to autotools. I don't think I'm lost in the bazaar, and
I probably hate dealing with those tools as much as the next guy. But
most of the shit we get from autocrap is caused by ourselves, because
we are poor users on top of complaining (me included until recently
when I increased both my understanding and hatred of this mess).

I used my pet project cashpack as a test bed and managed to get CFLAGS
working with both combinations of old/recent versions of GCC or Clang,
suncc, and even pcc. Just when I thought it wasn't working, it turned
out that actually suncc 12.4 grew the -std and -pedantic flags
(properly rejected with 12.3).

My handling of warnings is very portable and is backed by autoconf.
The results can be seen on the current master [1] where basically I
throw CFLAGS at the compiler and end up only with the ones it
supports. In my case most of the shit I had to cope with was caused by
GCC and Clang being inconsistent back-stabbers. I also never had the
occasion to investigate all the CFLAGS workarounds in Varnish, but I'm
confident I can get rid of most of the cruft.

As a side note, suncc was of great help to better comply with C99.

I keep my `configure.ac` [2] as simple as possible, and even though it
may seem large in addition to `cashpack.m4` bear in mind I'm leaving
space in my code to keep it readable. Because m4 is the worst macro
language I have seen so far. Also my build system does a lot despite
[3] the small size of the project, in hindsight it's not even _that much_
code.

Same offer as last September: I can incrementally look after the build
system but after the recent (multiple) threats on autotools, I'm
putting my autoconf efforts on hold. I otherwise have the means to
test many things by myself thanks to Nils, and I can nudge every once
in a while to get a test done on FreeBSD. And in any case if I break
something on the master branch, `git revert` + angry mails at me
should be enough to fix it.

Also please note that Guillaume once submitted a patch for a
non-recursive automake build, which I think would also help contain
the complexity^Wmadness.

Cheers,
Dridi

[1] https://github.com/Dridi/cashpack/blob/b559be2/m4/cashpack.m4#L94-L137
[2] https://github.com/Dridi/cashpack/blob/b559be2/configure.ac
[3] integration with many tools, plus dependencies for interoperability tests



From phk at phk.freebsd.dk  Thu Jan 19 17:00:14 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 19 Jan 2017 17:00:14 +0000
Subject: autoconf madness
In-Reply-To: <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ@mail.gmail.com>
References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de>
 <a10c3956-6af9-e82b-c7bc-1fa1e03451e6@schokola.de>
 <CABoVN9C3L3RFX+Uyj+r7WvjozQHv383oBEk17fanFv1DK8MTcQ@mail.gmail.com>
 <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ@mail.gmail.com>
Message-ID: <1248.1484845214@critter.freebsd.dk>

--------
In message <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ at mail.gmail.com>
, Dridi Boukelmoune writes:

Yes, we are probably poor users of autocrap.

But I don't want to make excuses for that.

We're trying to build 8 small binaries, three static and four dynamic
libraries.

We should not have to even think about build infrastructure *at all*.

*It* *should* *just* *work*.

And that is why I want to get rid of autocrap:  It doesn't just work,
not even close.

>Also please note that Guillaume once submitted a patch for a
>non-recursive automake build, which I think would also help contain
>the complexity^Wmadness.

Absolutely, and I'd love to have that going in.

The only change I ask is that each of the lib/bin subdirectories
have a trivial makefile along the lines of:

	*:
		cd ../.. && make $(*)

(no, make(1) doesn't grok that, but you get the idea...


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Fri Jan 20 08:18:30 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 20 Jan 2017 09:18:30 +0100
Subject: autoconf madness
In-Reply-To: <1248.1484845214@critter.freebsd.dk>
References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de>
 <a10c3956-6af9-e82b-c7bc-1fa1e03451e6@schokola.de>
 <CABoVN9C3L3RFX+Uyj+r7WvjozQHv383oBEk17fanFv1DK8MTcQ@mail.gmail.com>
 <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ@mail.gmail.com>
 <1248.1484845214@critter.freebsd.dk>
Message-ID: <CABoVN9DA=OF0obW6UUvRkOHWnKPQ5QH4f7CRL1aJ5A+K=V-Bbg@mail.gmail.com>

On Thu, Jan 19, 2017 at 6:00 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9DKswgGkEVQ+Jb-HUvx1PNyyPc0o5+nrzZ=R732Q9gBiQ at mail.gmail.com>
> , Dridi Boukelmoune writes:
>
> Yes, we are probably poor users of autocrap.
>
> But I don't want to make excuses for that.
>
> We're trying to build 8 small binaries, three static and four dynamic
> libraries.
>
> We should not have to even think about build infrastructure *at all*.
>
> *It* *should* *just* *work*.

I'm afraid compilers don't make it easy when it comes to just working
and portability. At least on the dependency management side pkg-config
does a great job (if *.pc files are properly written).

> And that is why I want to get rid of autocrap:  It doesn't just work,
> not even close.

There are things from autotools themselves and from the
autoconf-archive that certainly don't just work, or even work.

>>Also please note that Guillaume once submitted a patch for a
>>non-recursive automake build, which I think would also help contain
>>the complexity^Wmadness.
>
> Absolutely, and I'd love to have that going in.

So what's the goal? Are we still getting rid of autocrap?

> The only change I ask is that each of the lib/bin subdirectories
> have a trivial makefile along the lines of:
>
>         *:
>                 cd ../.. && make $(*)
>
> (no, make(1) doesn't grok that, but you get the idea...

Yes, although make's design was very spot on IMO it also lacks many
useful things today. As for myself, I haven't felt the need to cd into
a sub-directory for a long time, so I would no longer hurt from having
a single Makefile.

Dridi



From phk at phk.freebsd.dk  Mon Jan 23 04:57:50 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 23 Jan 2017 04:57:50 +0000
Subject: AFK today and tomorrow
Message-ID: <91613.1485147470@critter.freebsd.dk>

I'm mostly away from the keyboard monday & tuesday, the concrete
wall elements for our house is being raised.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Fri Jan 27 20:11:51 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 27 Jan 2017 21:11:51 +0100
Subject: [master] b6d0ff8 Merge tests and reshuffle things
In-Reply-To: <E1cXBOS-0006SJ-Vp@project.varnish-software.com>
References: <E1cXBOS-0006SJ-Vp@project.varnish-software.com>
Message-ID: <CABoVN9BnPZh5TSbLiYmisCB0TyvSGb940K7fWA3njvzFkNWbSw@mail.gmail.com>

On Fri, Jan 27, 2017 at 7:37 PM, Federico G. Schwindt <fgsch at lodoss.net> wrote:
>
> commit b6d0ff84fabfce8bffad426c02421eb891ef6a7b
> Author: Federico G. Schwindt <fgsch at lodoss.net>
> Date:   Fri Jan 27 18:35:22 2017 +0000
>
>     Merge tests and reshuffle things

Instead of shuffling things *right after* I stabilized them we could
have discussed this. It now fails even more frequently. And yes I
know my fix would hide errors (not actual test failures) but at least
it was better than noise considering the hard-coded ports.

I suggest a revert of this commit and then a reshuffle without the
merge of r1813. I'm aware it is covering varnishd usage but it's also
a regression test so I can live with "have same address" being its own
test. For some systems vtest is currently reporting 100% of failures
caused by c3.vtc for 9a652c7.

Dridi



From dridi at varni.sh  Fri Jan 27 20:14:36 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 27 Jan 2017 21:14:36 +0100
Subject: [master] b6d0ff8 Merge tests and reshuffle things
In-Reply-To: <CABoVN9BnPZh5TSbLiYmisCB0TyvSGb940K7fWA3njvzFkNWbSw@mail.gmail.com>
References: <E1cXBOS-0006SJ-Vp@project.varnish-software.com>
 <CABoVN9BnPZh5TSbLiYmisCB0TyvSGb940K7fWA3njvzFkNWbSw@mail.gmail.com>
Message-ID: <CABoVN9BtZzWJFErsE+Qgofpi9xO2XEo5On4BqFYm3Pmx_NHVgQ@mail.gmail.com>

On Fri, Jan 27, 2017 at 9:11 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> On Fri, Jan 27, 2017 at 7:37 PM, Federico G. Schwindt <fgsch at lodoss.net> wrote:
>>
>> commit b6d0ff84fabfce8bffad426c02421eb891ef6a7b
>> Author: Federico G. Schwindt <fgsch at lodoss.net>
>> Date:   Fri Jan 27 18:35:22 2017 +0000
>>
>>     Merge tests and reshuffle things
>
> Instead of shuffling things *right after* I stabilized them we could
> have discussed this. It now fails even more frequently. And yes I
> know my fix would hide errors (not actual test failures) but at least
> it was better than noise considering the hard-coded ports.
>
> I suggest a revert of this commit and then a reshuffle without the
> merge of r1813. I'm aware it is covering varnishd usage but it's also
> a regression test so I can live with "have same address" being its own
> test. For some systems vtest is currently reporting 100% of failures
> caused by c3.vtc for 9a652c7.

I see you've fixed it again while I was writing this message.



From fgsch at lodoss.net  Fri Jan 27 20:21:19 2017
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 27 Jan 2017 20:21:19 +0000
Subject: [master] b6d0ff8 Merge tests and reshuffle things
In-Reply-To: <CABoVN9BtZzWJFErsE+Qgofpi9xO2XEo5On4BqFYm3Pmx_NHVgQ@mail.gmail.com>
References: <E1cXBOS-0006SJ-Vp@project.varnish-software.com>
 <CABoVN9BnPZh5TSbLiYmisCB0TyvSGb940K7fWA3njvzFkNWbSw@mail.gmail.com>
 <CABoVN9BtZzWJFErsE+Qgofpi9xO2XEo5On4BqFYm3Pmx_NHVgQ@mail.gmail.com>
Message-ID: <CAJV_h0bqgoxbq=8-ZsDT_Owg7z_UE306N0aja97-DYM0b+B7+A@mail.gmail.com>

Not happy about hiding the error but it will do it for now.

The tests are not using a hardcoded port anymore and should fail with the
expected error now.
Why this is a problem in sunos is beyond to me at the moment.

On Fri, Jan 27, 2017 at 8:14 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:

> On Fri, Jan 27, 2017 at 9:11 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> > On Fri, Jan 27, 2017 at 7:37 PM, Federico G. Schwindt <fgsch at lodoss.net>
> wrote:
> >>
> >> commit b6d0ff84fabfce8bffad426c02421eb891ef6a7b
> >> Author: Federico G. Schwindt <fgsch at lodoss.net>
> >> Date:   Fri Jan 27 18:35:22 2017 +0000
> >>
> >>     Merge tests and reshuffle things
> >
> > Instead of shuffling things *right after* I stabilized them we could
> > have discussed this. It now fails even more frequently. And yes I
> > know my fix would hide errors (not actual test failures) but at least
> > it was better than noise considering the hard-coded ports.
> >
> > I suggest a revert of this commit and then a reshuffle without the
> > merge of r1813. I'm aware it is covering varnishd usage but it's also
> > a regression test so I can live with "have same address" being its own
> > test. For some systems vtest is currently reporting 100% of failures
> > caused by c3.vtc for 9a652c7.
>
> I see you've fixed it again while I was writing this message.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20170127/8e2c199c/attachment.html>

From dridi at varni.sh  Sat Jan 28 00:19:18 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Sat, 28 Jan 2017 01:19:18 +0100
Subject: [master] b6d0ff8 Merge tests and reshuffle things
In-Reply-To: <CAJV_h0bqgoxbq=8-ZsDT_Owg7z_UE306N0aja97-DYM0b+B7+A@mail.gmail.com>
References: <E1cXBOS-0006SJ-Vp@project.varnish-software.com>
 <CABoVN9BnPZh5TSbLiYmisCB0TyvSGb940K7fWA3njvzFkNWbSw@mail.gmail.com>
 <CABoVN9BtZzWJFErsE+Qgofpi9xO2XEo5On4BqFYm3Pmx_NHVgQ@mail.gmail.com>
 <CAJV_h0bqgoxbq=8-ZsDT_Owg7z_UE306N0aja97-DYM0b+B7+A@mail.gmail.com>
Message-ID: <CABoVN9A7cAhDRCRms7rrDZXK1E3R+5P3q3t66ah6v706Vn8vfQ@mail.gmail.com>

On Fri, Jan 27, 2017 at 9:21 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
> Not happy about hiding the error but it will do it for now.
>
> The tests are not using a hardcoded port anymore and should fail with the
> expected error now.

Your test case is misleading because it pretends to be safe, but either
${v1_port} is bound when you're using it the test can't [1] possibly pass,
or it's available for anyone to bind and you run into the same
concurrency problem again.

> Why this is a problem in sunos is beyond to me at the moment.

SunOS, FreeBSD, and my Linux box. I tried to get rid of the hard-coded
port too but didn't manage, so I kept it as close as the original test.

This is very simple: varnishtest doesn't offer enough isolation to safely
run this test, especially with concurrent jobs.

Dridi

[1] Except maybe with SO_REUSEPORT, but... can of worms?



From dridi at varni.sh  Sat Jan 28 18:30:29 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Sat, 28 Jan 2017 19:30:29 +0100
Subject: [master] a4e07fc Correct this test for good
In-Reply-To: <E1cXXL6-0003de-IP@project.varnish-software.com>
References: <E1cXXL6-0003de-IP@project.varnish-software.com>
Message-ID: <CABoVN9CKuAu1n2cUG-14uAnbi4fWsyp585M2RLTCRU9Ej4kWKQ@mail.gmail.com>

On Sat, Jan 28, 2017 at 7:03 PM, Federico G. Schwindt <fgsch at lodoss.net> wrote:
>
> commit a4e07fc5a611062afddd2fb114d43d793054c15f
> Author: Federico G. Schwindt <fgsch at lodoss.net>
> Date:   Sat Jan 28 14:38:50 2017 +0000
>
>     Correct this test for good
>
>     (Ab)use the bound address used for the bad_backend macro to expose a
>     random port and avoid the dreaded collisions.

This is a topic I actually wanted to bring up to the next bugwash (I
couldn't attend last Monday) and while ${bad_backend} is convenient
for testing, it relies on brittle behavior from the way varnish does
host/port resolution. It doesn't question tests relying on this macro
and the new ${random_port}. I found both names misleading too.

As for the solution to this particular problem, I find it quite
elegant naming aside.

Dridi



From dridi at varni.sh  Sat Jan 28 19:27:08 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Sat, 28 Jan 2017 20:27:08 +0100
Subject: [master] a4e07fc Correct this test for good
In-Reply-To: <CABoVN9CKuAu1n2cUG-14uAnbi4fWsyp585M2RLTCRU9Ej4kWKQ@mail.gmail.com>
References: <E1cXXL6-0003de-IP@project.varnish-software.com>
 <CABoVN9CKuAu1n2cUG-14uAnbi4fWsyp585M2RLTCRU9Ej4kWKQ@mail.gmail.com>
Message-ID: <CABoVN9DL-kKD5tESuyP6e5LZ0mU295ZV0O29PJL39RP1WFVaQQ@mail.gmail.com>

> As for the solution to this particular problem, I find it quite
> elegant naming aside.

Except that it fails on FreeBSD and SunOS, I'm pushing a revert :(

Dridi