Yesterday, v0.12.3 of RGBMatrixEmulator released, featuring mainly non-functional changes around automated testing and a new CI pipeline that runs in GitHub Actions. For those not familiar with RGBME, here’s the full context. Long story short, it’s an emulator for Raspberry Pi HUB75 LED matrices.

In the last few RBME releases I accidentally caused a a few bugs, so I set out to write some tests to help me avoid that in the future. It ended up being a lot harder than anticipated! Today I want to talk a bit about testing an emulator like this, because unit tests are less useful than functional tests, but due to the architecture of RGBME (and ultimately the library it emulates), it ended up being more of an exercise in manipulating Python’s package loader than actually writing tests!

RGBME / rpi-rgb-led-matrix Architecture

I need to give a little background here about how these libraries are used before I talk about my tests. Let’s boil these down into a few bullet points:

• RGBME emulates rpi-rgb-led-matrix Python bindings.
• rpi-rgb-led-matrix has some sample Python scripts to show developers how to use the library.
  • To run one, you would have to execute it from the samples directory – in other words, it has relative imports!
• RGBME also supports arbitrary matrix sizes. Common ones are 32x32, 64x32, etc. but you could chain several together and make a huge one, if you wanted.
• RGBME outputs pixels to the screen via display adapters – anything from your browser, terminal, a Pygame window, etc.

Testing Approach

Rather than obsess over unit tests from all the functions I wrote from the emulator, I decided that I’d rather focus on making sure the emulator outputs the correct pixels. I also wanted to parameterize the tests – as mentioned RGBME supports arbitrary sizes, so I wanted good test coverage for common sizes.

To do this, my plan was reasonably straightforward:

1. Include the samples from rpi-rgb-led-matrix (actually already done, but prior to this release required manual testing)
2. Create a new display adapter type that simply cached the most recent frame buffer (a 2D array of RGB as a 3-tuple)
3. Capture known good reference images from each sample
4. Set up a test framework that imports each sample class and runs it with the new display adapter, then compares it to the reference image

Complexities Commence

From the approach, #1 and #2 were quite simple. Including samples as mentioned required no work. Creating a new display adapter was easy because this library is basically all dependency injection, so I just needed to build a class with the right adapter interface, and everything just works. The adapter is basically headless and does nothing but cache the last several frames. Astute reviewers will note the strange halt_after and halt_fn attributes… We will touch on that shortly.

Capturing reference images is where things get interesting. I could have done this by hand, but that would have been time-consuming and error-prone, not to mention not scalable if I ever need someone else to be able to do it. The question is how to automate it?

All About That Samplebase

At the heart of the sample scripts is a base class called Samplebase. It handles argument parsing, setting up matrix options, setting up a matrix object (!), and running the sample – all of this mostly through a god-like function called process(). Subclasses can implement run() which will contain matrix draw routines to put pixels to the screen.

Given that this code comes directly from the library I’m trying to emulate, I didn’t really want to alter it to suit a test framework since in theory the tests should behave identically to running samples in any other environment.

Here’s the guts (link to full class as well):

class SampleBase(object):
    # Other functions omitted

    def process(self):
        # An argparse.ArgumentParser instance
        self.args = self.parser.parse_args()

        options = RGBMatrixOptions()

        # A bunch of calls like this omitted for brevity...
        options.rows = self.args.led_rows
        options.cols = self.args.led_cols

        self.matrix = RGBMatrix(options = options)

        try:
            # Start loop
            print("Press CTRL-C to stop sample")
            self.run()
        except KeyboardInterrupt:
            print("Exiting\n")
            sys.exit(0)

        return True

Of the functions Samplebase performs, I really, really need to be in control of that RGBMatrix object (it’s the thing RGBME emulates). If I can’t get to it, my only other option is to write a custom sample base class that I can inject a matrix into.

A Samplebase Subclass

Now let’s look at something that actually draws to the screen. We’ll use RunText (link), a sample that just scrolls “Hello World!”:

#!/usr/bin/env python
# Display a runtext with double-buffering.
from samplebase import SampleBase
from RGBMatrixEmulator import graphics
import time


class RunText(SampleBase):
    def __init__(self, *args, **kwargs):
        super(RunText, self).__init__(*args, **kwargs)
        self.parser.add_argument("-t", "--text", help="The text to scroll on the RGB LED panel", default="Hello world!")

    def run(self):
        # Set up for canvases, fonts, colors, etc.
        offscreen_canvas = self.matrix.CreateFrameCanvas()
        font = graphics.Font()
        font.LoadFont("./fonts/7x13.bdf")
        textColor = graphics.Color(255, 255, 0)
        pos = offscreen_canvas.width
        my_text = self.args.text

        # Looping draw calls
        while True:
            offscreen_canvas.Clear()
            len = graphics.DrawText(offscreen_canvas, font, pos, 10, textColor, my_text)
            pos -= 1
            if (pos + len < 0):
                pos = offscreen_canvas.width

            time.sleep(0.05)
            offscreen_canvas = self.matrix.SwapOnVSync(offscreen_canvas)

# Run the stuff!
if __name__ == "__main__":
    run_text = RunText()
    if (not run_text.process()):
        run_text.print_help()

A few things to note here. First is the relative import:

# This is in samples/samplebase!
from samplebase import Samplebase

Next is the call to run_text.process() – that means that the base class sets up that matrix object we need before we get a chance to do so!

Solving Relative Imports

The first pressing problem is to get the samples loaded correctly at all. This took a lot of trial-and-error, as it turns out weird things break when you mess with path on the fly.

First, I created a file to house the loader code. This reference file handles a lot of the stuff we need later, like, y’know, actually running the samples, or generating those reference images that we’ll use to test against later. The important piece is this:

import sys, os, importlib

from collections import namedtuple

sys.path.append(os.path.abspath(os.path.join(__file__, "..", "..")))

import samples

sys.modules["samplebase"] = samples.samplebase

Reference = namedtuple("Reference", ("file_name", "name", "frame", "halt_fn"))

_REFERENCES = [
    Reference("canvas-brightness", "CanvasBrightness", 256, send_kb_interrupt),
    Reference("graphics", "GraphicsTest", 140, send_kb_interrupt),
    Reference("grayscale-block", "GrayscaleBlock", 256, send_kb_interrupt),
    Reference("image-brightness", "ImageBrightness", 256, send_kb_interrupt),
    Reference("image-scroller", "ImageScroller", 50, send_kb_interrupt),
    Reference("pulsing-brightness", "GrayscaleBlock", 256, send_kb_interrupt),
    Reference("pulsing-colors", "PulsingColors", 256, send_kb_interrupt),
    Reference(
        "rotating-block-generator", "RotatingBlockGenerator", 256, send_kb_interrupt
    ),
    Reference("runtext", "RunText", 264, send_kb_interrupt),
    Reference("simple-square", "SimpleSquare", 256, send_kb_interrupt),
    Reference("singleton", "MultCanvas", 256, send_kb_interrupt),
    # Not a class
    # Reference("image-draw", "ImageDraw", 256, send_kb_interrupt)
    # Needs an extra arg
    # Reference("image-viewer", "ImageViewer", 256, send_kb_interrupt)
]

REFERENCES = []

for reference in _REFERENCES:
    module = importlib.import_module(f"samples.{reference.file_name}")
    sample = getattr(module, reference.name)
    sample.name = reference.name
    sample.file_name = reference.file_name
    sample.frame = reference.frame
    sample.halt_fn = reference.halt_fn
    REFERENCES.append(sample)

So let’s walk through this.

• First, we need to import some packages to interact with paths, the OS, and importing other packages.
• We need to add the relative path to the samples directory to the Python path.
  • It’s two levels up from the current file, so append it to the path with… sys.path.append(os.path.abspath(os.path.join(__file__, "..", "..")))
• Now samples can be imported easily.
• To make from samplebase import Samplebase work in each sample, it can be added to the modules directly: sys.modules["samplebase"] = samples.samplebase
• Next, there’s _REFERENCES, an internal list of named tuples containing all the sample file paths, the class names, the frame we want to screencap, and the way we want the script to be halted (more later!).
• And finally – for each tuple, import it from samples via the file name and set all the attributes as needed.

Whew! That’s kind of a mess!

So What’s send_kb_interrupt?

If you remember back to the Samplebase.process() code, there was this try..catch block:

        try:
            # Start loop
            print("Press CTRL-C to stop sample")
            self.run()
        except KeyboardInterrupt:
            print("Exiting\n")
            sys.exit(0)

This does kinda what you would expect – wait for CTRL + C and then exit. The send_kb_interrupt simply raises a custom exception SampleExecutionHalted that is unambiguous so we can capture the exit event later. Together with the frame number, these are injected as halt_fn and halt_after to the display adapter. The adapter tracks what frame it’s on, then calls the halt function to throw the exit exception and stop execution.

We need this because this makes the tests deterministic. Luckily for me, they’re single-threaded, so halting on a specific frame number in each test will always give the same output.

Putting It All Together

Ok, so we have samples loading in and a way to deterministically test them. But how do we inject that matrix object we need?

Here’s a doozy:

def run_sample(sample_class, size, screenshot_path=None):
    sys.argv = [
        f"{sample_class.file_name}.py",
        f"--led-cols={size[0]}",
        f"--led-rows={size[1]}",
    ]

    cwd = os.getcwd()
    os.chdir(os.path.abspath(os.path.join(__file__, "..", "..", "samples")))

    sample = sample_class()
    sample.usleep = lambda _: 1 + 1

    def mockedSetAttr(inst, name, value):
        if name == "matrix":
            inst.__dict__[name] = value

            inst.matrix.CreateFrameCanvas()
            adapter = inst.matrix.canvas.display_adapter
            adapter.halt_after = sample.frame
            adapter.halt_fn = sample.halt_fn
            adapter.width = inst.matrix.width
            adapter.height = inst.matrix.height
            adapter.options = inst.matrix.options
        else:
            super(inst.__class__, inst).__setattr__(name, value)

    sample_class.__setattr__ = mockedSetAttr

    try:
        sample.process()
    except SampleExecutionHalted:
        adapter = sample.matrix.canvas.display_adapter

        if screenshot_path:
            refdir = os.path.join(screenshot_path, sample_class.file_name)

            if not os.path.exists(refdir):
                os.mkdir(refdir)

            refpath = os.path.join(refdir, f"w{adapter.width}h{adapter.height}.png")
            adapter._dump_screenshot(refpath)

        return adapter._last_frame()
    except Exception:
        traceback.print_exc()
    finally:
        sample.matrix.canvas.display_adapter._reset()
        os.chdir(cwd)

This is a custom sample run function that takes a sample and a matrix size and instantiates it, runs it, and does any extra post-run handling required.

First, we spoof the arguments to the file via the name of the sample filename + the --led-cols and --led-rows flags. Those flags are used by both libraries to determine how big of a matrix the script is executing on.

Next, theres a directory change to samples (after saving our current working directory for later). The script is technically runnable at this point, but there’s some more we need to do.

    sample = sample_class()

    # This lets us control the matrix assignment!
    # self.matrix = RGBMatrix(options=options)
    def mockedSetAttr(inst, name, value):
        if name == "matrix":
            inst.__dict__[name] = value

            inst.matrix.CreateFrameCanvas()
            adapter = inst.matrix.canvas.display_adapter
            adapter.halt_after = sample.frame
            adapter.halt_fn = sample.halt_fn
            adapter.width = inst.matrix.width
            adapter.height = inst.matrix.height
            adapter.options = inst.matrix.options
        else:
            super(inst.__class__, inst).__setattr__(name, value)

    sample_class.__setattr__ = mockedSetAttr

This guy is the meat and potatoes of this function. This is instantiating the sample, then hooking into that assignment to the matrix option we really, really need (remember?). This piece, when it’s assigned, lets us make sure there’s a display adapter with the right config to capture frames and halt at the right time. Exactly what we need!

Finally, in the try..except..finally, the custom error finally shows up (SampleExecutionHalted) allowing us to capture a screenshot here, if we want – and we do, in order to get reference images for later. The display adapter is also reset in the finally block, making sure we don’t hold onto old frames for new tests: sample.matrix.canvas.display_adapter._reset()

All in, once you have this in place, all the samples are runnable in a deterministic way!

The Easy Part

Now for the last piece – the tests!

I think this is pretty straight-forward, it’s just a comparison of two images pixel-by-pixel:

class TestSampleRunMatchesReference(TestCase):
    @parameterized.expand(TESTS)
    def test_sample(self, name, sample, size):
        expected = reference_to_nparray(sample, size)

        if expected is None:
            print(f"\n{sample.file_name} {size} has no reference image. Skipping...")
            return

        # Suppress "Press CTRL-C to stop sample" messages
        with io.StringIO() as buf, contextlib.redirect_stdout(buf):
            actual = run_sample(sample, size)

        if not np.array_equal(expected, actual):
            image = Image.fromarray(np.array(actual, dtype="uint8"), "RGB")
            image.save(
                os.path.abspath(
                    os.path.join(
                        __file__,
                        "..",
                        "result",
                        f"{sample.file_name}-w{size[0]}h{size[1]}.png",
                    )
                )
            )

            self.assertTrue(
                False,
                f"Actual results do not match reference screenshot. See test/result/{sample.file_name}-w{size[0]}h{size[1]}.png to compare",
            )

        self.assertTrue(True)

There’s a bit I glossed over here, namely that the RGBME instance reads from a config file for some extra configuration, which also needed to be spoofed. I took the easy way out and wrote that file for each test set eup – this is in line with the testing goals in that I wanted to simulate actually running each test.

I didn’t anticipate this problem being this difficult. I learned quite a bit about Python’s package loader, ended up needing to inject attributes via __setattr__ mocks, and had to implement a kind of hot reloading for my new display adapter. All in, a very interesting weekend project!

That’s all for now. Thanks for reading!