Check Vinyl Cache for crashes

Usually Vinyl Cache is really, really stable and we’ve seen setups running continuously for years. But in case Vinyl crashes, you loose all cached data! [1] And you might not even notice - unless it happens often. This is because Vinyl starts two processes: The manager forks and monitors the worker as a child process, which handles all the traffic and especially holds the cache. If the worker crashes, it immediately gets restarted by the manager. Thus there is usually very little influence on your website’s availability, but all cache is lost. This might also lead to a peak in backend requests you might notice. If you have a surprisingly bad hitrate and wonder why, you should check for crashes.

[1]

If you use a persistent storage engine like fellow from vmod slash, your cache is not lost, but you are still affected, because the restart takes longer.

Let’s simulate a panic and learn how one can take notice of it.

Start Vinyl Cache and check processes:

martin@notebook:~$ vinyld -b 127.0.0.1
Child launched OK
martin@notebook:~$ ps -ef | grep vinyl
martin     39040       1  0 16:08 ?        00:00:00 vinyld -b 127.0.0.1
martin     39054   39040  0 16:08 ?        00:00:00 vinyld -b 127.0.0.1

39040 is the manager (parent) process, 39054 the worker (child) process. The child was started once:

martin@notebook:~$ vinylstat -1 -f MGT.child_start
MGT.child_start              1         0.00 Child process started

Now let’s deliberately make the worker process panic:

martin@notebook:~$ vinyladm debug.panic.worker

As you can see the old worker process 39054 vanished and a new worker process 39717 was created:

martin@notebook:~$ ps -ef | grep vinyl
martin     39040       1  0 16:08 ?        00:00:00 vinyld -b 127.0.0.1
martin     39717   39040  0 16:12 ?        00:00:00 vinyld -b 127.0.0.1

The MGT.child_start counter increased:

martin@notebook:~$ vinylstat -1 -f MGT.child_start
MGT.child_start              2         0.09 Child process started

You can also check the panic message, which is very important for bug reports and might give you an idea of what went wrong:

martin@notebook:~$ vinyladm panic.show
Child (42204) Panic at: Wed, 27 May 2026 14:26:25 GMT
Assert error in ccf_panic(), cache/cache_panic.c line 809:
  Condition((strcmp(\"\", \"You asked for it\")) == 0) not true.
version = vinyl-cache-trunk revision 1d77ba0e41609e40debc1705741c001bbf72848c, vrt api = 23.0
ident = Linux,6.8.0-117-generic,x86_64,-jnone,-sdefault,-sdefault,-ssynth,-hcritbit,epoll
now = 17701.340457 (mono), 1779891984.459089 (real)
Backtrace:
  0x556f7d8f69ea: vinyld(VBT_format+0x7a) [0x556f7d8f69ea]
  0x556f7d86994e: vinyld(+0x6994e) [0x556f7d86994e]
  0x556f7d8f5e8b: vinyld(VAS_Fail+0x1b) [0x556f7d8f5e8b]
...

Tip

If we had killed the worker for example using kill -9 39054, it would also have been restarted by the manager, but no panic message would have been logged.

You might want to monitor vinyladm panic.show and/or vinylstat -1 -f MGT.child_start with your favorite monitoring tool.

If you are in a hurry and need a quick & dirty solution you could use something like this:

[vinyl@vinyl01:~]$ cat panic_notify.sh
#!/bin/bash
tmp=/tmp/panic_notify.$$
if vinyladm panic.show >${tmp} 2>/dev/null ; then
        mailx -s "vinyl panic" <${tmp} admin@example.com && vinyladm panic.clear
fi
rm -f ${tmp}

[vinyl@vinyl01:~]$ crontab -l
*/10 * * * *    /home/vinyl/panic_notify.sh

Contributing

We welcome fixes and improvements to this tutorial! To contribute, please go to https://code.vinyl-cache.org/vinyl-cache/homepage

For how to get an account, please see https://code.vinyl-cache.org/vinyl-cache/code.vinyl-cache.org#invitations and https://vinyl-cache.org/organization/moving.html