From ingvar at redpill-linpro.com Wed Aug 5 15:16:25 2026 From: ingvar at redpill-linpro.com (Ingvar Hagelund) Date: Wed, 5 Aug 2026 17:16:25 +0200 (CEST) Subject: random tests failing on s390x In-Reply-To: <202607080537.6685b0GO029302@critter.freebsd.dk> References: <202607080537.6685b0GO029302@critter.freebsd.dk> Message-ID: <1369558639.173270.1785942985028.JavaMail.zimbra@redpill-linpro.com> My two cents: On Fedora, official builds are done on central servers. If you don't have special hardware available (s390x, aarch64, ppc64, etc), you may also send "scratch" (test) builds to these as well. On s390x I disable some of the most troublesometime sensitive tests. In addition, I run the test suite in series, ie without any -j parameter to make to not stress the system. But as Red Hat's s390x park is limited, some builds fail just because the builders are busy. And yes, this also means all builds, including just a simple test takes quite some time to finish. So to check if a specific test goes through while debugging some problem, I do a scratch build that runs that single test instead of the whole make check. Ingvar ----- On 8 Jul, 2026, at 07:37, Poul-Henning Kamp wrote: > -------- > Marco d'Itri writes: > > Is this caused by the s390x build daemons being overloaded? > That's my conclusion. > We also have a s390x instance in our own "vtest": > https://vinyl-cache.org/vtest/ > and it shows similar random lossage, even though we run that > platform with VTEST_DURATION=180 > That said, some of our test-cases are more sensitive to timing than > others, and we try to improve that where we can, and I recoginize > the cases you mention as being on my personal "trouble-list". > We have discussed if some tests should be "advisory", in the sense > that a failure is just a "warning" not an "error", but that feels > too much like giving up :-) > > Is this something that can be worked around, either upstream or in the > > Debian package, or should I disable the test suite on s390x? > I would disable tests on s390x. > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > vinyl-dev mailing list > vinyl-dev at vinyl-cache.org > https://vinyl-cache.org/lists/mailman/listinfo/vinyl-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Aug 5 15:34:01 2026 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 05 Aug 2026 15:34:01 +0000 Subject: random tests failing on s390x In-Reply-To: <1369558639.173270.1785942985028.JavaMail.zimbra@redpill-linpro.com> References: <202607080537.6685b0GO029302@critter.freebsd.dk> <1369558639.173270.1785942985028.JavaMail.zimbra@redpill-linpro.com> Message-ID: <6a7357e9.50c34.c6e6876@critter.freebsd.dk> I spent time on s390x over summer. It transpired that our s390x VM was so old that it predated their "self-service" facilities, and therefore probably got overlooked or sidelined in some ways. After I recreated the VM with the "self-service" facility, we clearly got more resources and it works a LOT better than the old VM. It's still not on par with the rest, but the noise level is now something we can live with: https://vinyl-cache.org/vtest/ -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From os at flyingcircus.io Wed Aug 12 10:25:39 2026 From: os at flyingcircus.io (Oliver Schmidt) Date: Wed, 12 Aug 2026 12:25:39 +0200 Subject: =?utf-8?Q?VINYL=5FSTATE=5FDIR_revisited_=28=E2=80=A6and_its_creat?= =?utf-8?Q?ion=29?= Message-ID: Hi, I am one of the NixOS maintainers for the vinyl-cache package, and while packaging the 9.0.1 release a while ago I stumbled over a few details regarding the VINYL_STATE_DIR changes that confuse me. So I'd like to follow up on https://code.vinyl-cache.org/vinyl-cache/vinyl-cache/issues/4477 here. First of all, changing such important data location defaults in a minor security patch release is not great from a packager's point of view. But that happened, let's move on. ## state dir creation during install phase For packaging vinyl-cache in nixpkgs, I had to patch out the changed install-data-local phase [1], as it now tries to create that (potentially external) state dir during the install phase of the package build process. Admittedly Nix packaging happening in a sandbox is special in its own way, but even classic packaging of rpm and deb packaging nowadays happens sandboxed. So creating the statedir at _build time_ does in no way ensure the directory to be present in the actual system installing such a package later on. For backwards-compatibility I decided to keep `"--with-statedir=/run"`, so the rule attempted to mkdir within the /run/ directory which of course failed in the sandbox. But even with the new default location in /var/lib/ creating directories during packaging has the same issues, right? In the meantime, that makefile target has slightly changed and is now prefixed with DESTDIR, but the general issue remains. I'd like to understand your reasoning behind that Makefile decision and work on a solution that allows building in the Nix sandbox with a custom statedir and avoids patching the makefile. ## state dir moved to /various/lib: memory-backed? I remember that having the VINYL_STATE_DIR on a RAM-backed filesystem used to be very important for performance. Is this still the case? I could not find any concrete statement on that in the docs anymore. Due to this, the decision to move the default state dir to /var/lib/ surprises me. Citing FHS considerations only make sense as long as that directory does not need to be memory-backed, because a normal FHS-compliant /var/lib/ is seldomly on memory-backed storage. And if we have to resort to stunts like @rfc1036 proposed > My plan for the Debian vinyl-cache package is to ship a systemd unit like vinyl-cache-workdir.mount which will mount an appropriate tmpfs on /var/lib/vinyl-cache/vinyld/. then FHS and the needs of vinyld are kind of orthogonal. On NixOS we already let systemd manage the state dir presence despite keeping it in /run/ (/var/run/ is a symlink to that on systemd-based distros). The noexec considerations on Debian systems regarding /var/run/ are valid though, the question then is whether Debian is the special case here and needs to handle this downstream, or whether everyone else needs to. Best regards osnyx PS: Is anyone willing to provide me a signup token for code.vinyl-cache.org ? [1] https://github.com/flyingcircusio/nixpkgs/blob/6dfb871fdcb1e4f39bb913876cf5e2ea1b1673fe/pkgs/servers/vinyl-cache/0001-Makefile-do-not-create-VINYL_STATE_DIR.patch -- Dipl.-Inf. Oliver Schmidt ? os at flyingcircus.io ? Systems Engineer ? Pronomen er/he Flying Circus Internet Operations GmbH ? https://flyingcircus.io Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick From nils.goroll at uplex.de Wed Aug 19 11:49:37 2026 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 19 Aug 2026 13:49:37 +0200 Subject: Pondering a "threadless" VCL execution model Message-ID: <5a0955bf-d9d1-424d-b305-734afdfe054d@uplex.de> Hi, as many of you reading this will probably know, a colleague and me have been working on async I/O for Vinyl Cache for ~2 years and we are going to publish what we have this year. The following concerns the client side only. The model we are following so far is basically one I/O thread per core, which can be thought of the acceptor and waiter combined, plus doing most of the I/O. When VCL needs to run, the I/O thread would dispatch the req parsing and VCL FSM to run on a worker. This would ultimately call the transport's vtr_deliver_f, which would return VTR_D_DISEMBARK to leave the actual delivery with the I/O thread. The reason to switch to a worker thread is that VCL could do anything, in particular block on I/O, and we certainly would not want the I/O thread to halt neglect other work just because some vmod calls sleep(10). Now this is all good and well, except that the thread switching has a cost, and there are many cases where VCL never blocks. And, obviously, for these cases not dispatching VCL and running it in the I/O thread would be more efficient. Also, at the moment, we might still block waiting for a backend thread, but handling this differently does not look like a relevant problem. I have been pondering ideas on and off how we could achieve to keep the dedicated worker thread for VCL if/when we need it, and get rid of it where we can. The obvious way would be for vmod functions to announce "I will block" when they do, and core code would implement some kind of "coroutine" context swapping using swapcontext. This looks to me like the right solution, but it comes with complexity and potential pitfalls, like if Thread Local Storage is used. The other idea is just to have VMODs announce globally that they "need a thread" (or maybe the inverse, rather). This way, as soon as a VMOD using blocking I/O was imported, processing would switch from "single worker" to "dedicated thread". I am skeptical about this idea, because it might render the whole optimization useless in practice. Now my question is if anyone has any better ideas on this? Nils -- Nils Goroll (he/him) ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x1DCD8F57A3868BD7.asc Type: application/pgp-keys Size: 4845 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From nils.goroll at uplex.de Fri Aug 21 15:58:22 2026 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Aug 2026 17:58:22 +0200 Subject: =?UTF-8?Q?Re=3A_VINYL=5FSTATE=5FDIR_revisited_=28=E2=80=A6and_its_c?= =?UTF-8?Q?reation=29?= In-Reply-To: References: Message-ID: <6b6bac1e-5274-4044-ba59-6fa61580bee4@uplex.de> Hi Oliver, thank you for getting in touch. I want to say upfront that I am really not knowledgeable in software packaging and appreciate any help and advice. > First of all, changing such important data location defaults in a minor security patch release is not great from a packager's point of view. But that happened, let's move on. Sorry. See above. > I'd like to understand your reasoning behind that Makefile decision and work on a solution that allows building in the Nix sandbox with a custom statedir and avoids patching the makefile. The relevant discussion is in https://code.vinyl-cache.org/vinyl-cache/vinyl-cache/issues/4477 and linked tickets. The fundamental problem here is that, from a performance perspective, we _really_ want the state directory to reside on a tmpfs, but it can not be noexec (iow, we need to run code (= the compiled VCL) from it), see https://vinyl-cache.org/docs/trunk/installation/platformnotes.html#on-linux-use-tmpfs-for-the-workdir I had made an attempt to find a default which would work for most, but then had to learn: https://code.vinyl-cache.org/vinyl-cache/vinyl-cache/issues/4477#issuecomment-60406 Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x1DCD8F57A3868BD7.asc Type: application/pgp-keys Size: 4845 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: