Announcing Go Interface Fuzzer

We’re excited to announce the open source release of Go Interface Fuzzer, a tool for automating the boilerplate of writing a fuzz tester to compare implementations of an interface.

Let’s jump straight into the real-world example which motivated this. We have a message store, where all the operations are expressed in an interface. Here’s a simplified version of that interface:

type Store interface {
    // Inserts an entry in the store. Returns an error if an
    // entry with greater or equal ID was
    // already inserted.
    Put(msg Message) error

    // Returns a slice of all messages in the specified
    // channel, from the specified ID to the message with
    // most recent ID.
    EntriesSince(sinceID uint64, channel Channel) []Message

    // Returns the ID of the most recently inserted message.
    MostRecentID() uint64

    // Returns the number of messages in the store.
    NumEntries() uint64

    // Returns all messages across all channels as a single
    // slice, sorted by ID.
    AsSlice() []Message

    // Returns the maximum number of messages in the store.
    MessageLimit() uint64
}

It’s nothing special, it’s even lacking some features you might expect, like the ability to retrieve a specific message. Speaking of messages, here’s what they look like:

type Message struct {
    // Each message has a unique ID.
    ID uint64

    // A message belongs to a specific channel.
    Channel Channel

    // And has a body
    Body string
}

type Channel string

We have two implementations of the Store interface. One is simple and was written such that it would be obviously correct, with no attention paid to performance. The other is faster, does less allocation, and is a bit more complex. We wanted to make sure both of these implementations behaved the same. This is a fairly simple task, it just involves a lot of boilerplate. The code goes like this:

1. Produce a new reference Store and test Store.
2. Do this some number of times:
   1. Pick a random operation in the interface.
   2. Generate random parameter values, possibly with some biassing.
   3. Perform the same operation on both stores.
   4. Complain if there is a deviation.

The code is tightly coupled to the interface, but is totally mechanical and requires very little thought. It’s all boilerplate, and we as programmers don’t like writing boilerplate.

Enter Go Interface Fuzzer

It can generate all the boilerplate for you, given the source file containing the interface and some special comments to guide the generation. A minimally marked-up Store looks like this:

/*
@fuzz interface: Store
@known correct:  makeReferenceStore
@generator:      generateChannel Channel
@generator:      generateMessage Message
*/
type Store interface {
    /* ... */
}

There are three special things here:

• @fuzz interface starts a fuzzer definition;
• @known correct gives a function to generate the reference implementation;
• @generator gives a function to generate values of a given type.

If we run go-interface-fuzzer store.go, it prints three functions to stdout:

• FuzzTestStore(makeTest func() Store, t *testing.T)\ intended to be used to construct a test case for go test
• FuzzStore(makeTest func() Store, rand *rand.Rand, max uint) error\ a version of the above which takes an explicit PRNG and number of operations to try;
• FuzzStoreWith(reference Store, test Store, rand *rand.Rand, max uint) error\ a version of the above which takes both stores.

There are flags to suppress the first two functions, if all you want is the third. There are also flags to generate a complete source file, with package name and imports; and to write out to a file.

This is just the surface

There are more facilities: invariant checking, stateful generators, and custom comparators. We’re happily using Go Interface Fuzzer internally here at Pusher and it has actually found some bugs, some even in the reference implementation! It’s easy to inadvertently write passing tests, because it’s easy to not notice edge cases. Being able to verify that a large number of random operations produce the same output greatly increases confidence.

Check out the project on GitHub, there’s also a more fully-featured version of the Store example in the repository.

If you have any comments or questions, feel free to open an issue on GitHub, or contact me on Twitter (@barrucadu).