Match or search&replace request- or response-bodies with VMOD re¶

Introduction¶

Vinyl Cache has always been very capable for manipulating headers, but left body data manipulations to the basics (gzip, esi).

There are cases when changing body data is required to achieve a task like fixing html-links in backend responses from backends not under your control.

Implementing such functionality in Vinyl Cache is as easy as setting up mod_substitute in Apache, just way more flexible. All you need is vmod_re.

Search&replace backend response bodies in vcl_backend_response¶

Let’s assume you have a misbehaving backend which is not under your control. It sends foo in its response-bodies, but you need to replace it with bar. Here’s how this can be accomplished:

import re;

sub vcl_init {
        new myregex = re.regex("foo", asfilter = true);
}

sub vcl_backend_response {
        if (beresp.http.Content-Encoding == "gzip" || beresp.do_gzip) {
                set beresp.filters += " gunzip myregex gzip";
        } else {
                set beresp.filters += " myregex";
        }
        myregex.substitute_all("bar");
}

We define a filter implementing the regex search, add it to the filter chain and instruct it to replace all matches with the .substitute_all() method.

As you can see, gzipped responses from backends need to be taken extra care of.

Important

If you use return(pass); in vcl_recv for requests whose response you want to modify, you should set set bereq.http.Accept-Encoding = "gzip"; in vcl_pass. This way the backend will only respond with gzip or no compression, but not deflate or other algorithms we can’t handle for substitution.

If you also use edge side includes (esi), please refer to beresp.filters in VCL Reference.)

Tip

To ensure all examples in this tutorial are working as expected, vtc files for use with vinyltest are provided.

Download the vtc file for this first example body_regex-beresp.vtc and run it using: vinyltest <testfile.vtc>.

Make sure your PATH contains the vinyld binary. You can easily make your own modifications to try out stuff. The reference manual for vtc is available here.

We can also use regex backreferences for replacements like this:

import re;

sub vcl_init {
        new myregex = re.regex("foo(\d+)bar", asfilter = true);
}

sub vcl_backend_response {
        if (beresp.http.Content-Encoding == "gzip" || beresp.do_gzip) {
                set beresp.filters += " gunzip myregex gzip";
        } else {
                set beresp.filters += " myregex";
        }
        myregex.substitute_all("\1");
}

Here, the pattern contains a capturing group (\d+), which matches any number of decimal digits. The substitution references the group by its running count \1.

For example the body content foo123bar will be replaced with 123. The vtc file for this example is available here: body_regex-beresp-backref.vtc

Search&replace client response bodies in vcl_deliver¶

So far we modified backend responses in vcl_backend_response.

If Cache-Control headers and/or VCL code allow the response to be cacheable, our modified response will be cached.

There might be cases where we want to store the response unmodified, but apply some individual replacements right before a response is sent out to the client.

A real world scenario for this case might be the passing of access tokens in m3u8 playlists for HLS video playback:

import re;

sub vcl_init {
        new re_hls_m3u8 = re.regex("(vs\d\.m3u8)", asfilter = true);
}

sub vcl_recv {
        set req.http.mytoken = regsub (req.url, ".*(\?.*)", "\1");   # grab url params and store it in a header for later use
        set req.url = regsub (req.url, "(.*)\?.*", "\1");   # cut params from url to allow for efficient caching
}

sub vcl_deliver {
        unset req.http.Accept-Encoding;   # the response is returned uncompressed even if the client supported compression because currently there is no gzip VDP in Vinyl-Cache
        set resp.filters += " re_hls_m3u8";
        re_hls_m3u8.substitute_all("\1" + req.http.mytoken);
}

Let’s suppose a client requests /playlist.m3u8?mytoken123abc.

After checking the token for access in VCL not shown here, ?mytoken123abc is saved and cut off the url.

Thus /playlist.m3u8 is looked up in the cache (instead of /playlist.m3u8?mytoken123abc, which would probably have a hitrate around zero because every client brings his own token…).

If not found in the cache, it is retrieved from the backend and stored in the cache. Let’s assume the body content is vs0.m3u8.

Before sending it to the client, it will be modified to vs0.m3u8?mytoken123abc.

This way the access token of the client is passed down to the subplaylists.

In a an additional step not shown here, the token can also be passed further down to individual chunks referenced in vs0.m3u8 using the same technique.

The vtc file for this example is available here: body_regex-resp.vtc

Search&replace client request bodies in vcl_recv¶

Everything we did so far aimed to modify responses from a backend server to Vinyl and further to the client. vmod_re is also capable of modifying client request bodies, for example in POST or PUT requests.

Let’s assume client requests are not under our control. For example, we need to support old apps using old request body content schemes in parallel to up-to-date apps using a new scheme:

import re;

sub vcl_init {
        new myregex = re.regex("old", asfilter = true);
}

sub vcl_recv {
        set req.filters += " myregex";
        myregex.substitute_all("new");
}

The vtc file for this example is available here: body_regex-req.vtc

Like in the response examples before, more complex regex with backreferences may be used.

By default, responses to POST-requests – the most typical case for request bodies – are not cached. But this method also works with request body hash based caching not shown here, because the request body is modified for every request.

Search&replace client requests bodies in vcl_backend_fetch¶

In other scenarios, we might want to modify the request body only when we send a request to a backend, for example to sanitize a known backend exploit.

Here’s an example of how this can be accomplished:

import re;

sub vcl_init {
        new myregex = re.regex("badexploitstring", asfilter = true);
}

sub vcl_backend_fetch {
        set bereq.filters += " myregex";
        myregex.substitute_all("");
}

The vtc file for this example is available here: body_regex-bereq.vtc. This example requires Vinyl Cache 9.1 or later.

Search client request bodies in vcl_recv¶

Instead of modifying request body content, you can also only search/match it and take action based on the result like sending redirects, errors, bypass caching and much more:

import re;
import std;

sub vcl_init {
        new myregex = re.regex("(badexploitstring)", forbody = true);
}

sub vcl_recv {
        if (req.method == "POST" && myregex.match_body(req_body)) {
                # maybe log violation
                std.log("dangerous request body content found: " + myregex.backref(1, ""));

                # maybe respond with http error
                return(synth(500, "internal server error"));

                # maybe respond with redirect
                #return(synth(301, "terms_and_conditions.html"));

                # maybe accept request but bypass caching
                #return(pass);
        }
}

The vtc file for this example is available here: body_regex-req-match.vtc

Search backend request bodies in vcl_deliver¶

Backend response bodies can be inspected similarly:

import re;

sub vcl_init {
        new myregex = re.regex("secretdatanottobeexposed", forbody = true);
}

sub vcl_deliver {
        if (myregex.match_body(resp_body)) {
                return(synth(403, "forbidden"));
        }
}

The vtc file for this example is available here: body_regex-resp-match.vtc

Search binary backend request bodies in vcl_deliver¶

It is easy to forget that regular expressions not only work for text, but also for binary content.

Let’s take a look at this real life example of checking if a response looks like it contains an image.

It is used to suppress the output of plain text rendering errors to the client and rather return an http error instead:

import re;

sub vcl_init {
        new re_image = re.regex("(?i)^(?:" +
                "\xFF\xD8\xFF\xE0[\x00-\xFF]{2}JFIF|" + # JPEG/JFIF
                "\xFF\xD8\xFF\xE1[\x00-\xFF]{2}Exif|" + # JPEG/Exif
                "\x89PNG\x0D\x0A\x1A\x0A|" +            # PNG
                "GIF8[79]a|" +                          # GIF
                "BM.{4}[\x00-\xFF]{2}\x00\x00|" +       # BMP
                "RIFF[\x00-\xFF]{4}WEBP|" +             # WebP
                "\x00\x00\x00[\x14-\x28]ftypavif" +     # AVIF
                ")", forbody = true);
}

sub vcl_deliver {
        if (! re_image.match_body(resp_body)) {
                return (synth(500, "No image data seen"));
        }
}

The vtc file for this example is available here: body_regex-resp-match-image.vtc

Documentation / Further Reading¶

The current documentation for vmod re is available using man vmod_re or online here.

Contributing¶

If you found any mistake in this tutorial, I’d like to cite Poul-Henning: “We’d absolutely love to have you help improve the project homepage, send us pull requests!” https://code.vinyl-cache.org/vinyl-cache/homepage