Beautifully Simple Benchmarking with Go

I recently read David Huie's great post about Go being possibly the next great teaching language, and I wasn't so sure at first. I absolutely love Go and use it as my first choice imperative language for both hacking something together quickly and for building robust web applications, but I wondered if an absolute beginner would still be better off with a dynamically typed language like Python.

And then every so often, I run into a problem to solve with Go and it just feels so easy and intuitive while being robust, that I can't help but think that David is absolutely right. Everybody's got a pet language though, so instead of just yapping on about it, I've decided that every time I do something that seems incredibly easy or powerful in Go, I'll just document it for the world to see and judge for themselves. Today, I'll talk about beautifully simple benchmarking in Go.

Let's start with a small example function. This function can really be anything, but I'll just write up a short one that concatenates some strings together, with a function that repeats the function on an input string to give us a problem we can grow as we need. Putting it all in string_concat.go:

package string_concat

func ConcatOperator(original *string, concat string) {  
    // This could be written as 'return *original + concat'
    // but I wanted to confirm no special compiler optimizations
    // existed for concatenating a string to itself.
    *original = *original + concat
}

func SelfConcatOperator(input string, n int) string {  
    output := ""
    for i := 0; i < n; i++ {
        ConcatOperator(&output, input)
    }
    return output
}

How would we benchmark our function in Go? In Go, we create *_test.go files to specify a test file for any *.go file. Here, we just create a file string_concat_test.go file and include:

package string_concat

import "testing"

func BenchmarkSelfConcatOperator1000(b *testing.B) {  
    for n := 0; n < b.N; n++ {
        SelfConcatOperator("test", 1000)
    }
}

Go's testing package will take care of increasing the variable b.N for any function we include within the for loop, so that we'll benchmark our function SelfConcatOperator repeatedly for a minimum of 1 second. This is so that we can get statistically significant results with sufficient repetitions. For functions that take longer per run, we can increase the minimum amount of time Go allots per benchmark so that we get statistically meaningful results.

Now, go to your terminal to where your string_concat.go and string_concat_test.go files reside. All we need to type to run our tests is:

go test -bench=.  

And that's it! You'll hopefully get an output that looks like:

testing: warning: no tests to run  
PASS  
BenchmarkSelfConcatOperator1000         500       3191695 ns/op  
ok      _/path/to/your/code/string_concat   1.915s  

And there it is: 3,191,695 nanoseconds to run our function which concatenated 1000 strings together. That obviously wasn't too useful, but the steps are really that simple for absolutely any function, maybe your favourite database query or text handling behemoth.

...OK, ok, we've got to benchmark something real in Go while we're at it. Here's something a bit more interesting, comparing two possible string concatenation implementations:

  1. Using the usual, immutable strings and the '+' operator ("goo" += "goo")
  2. Writing each new string to a growing buffer of bytes.

For those from the Java world, this are the classic String vs StringBuffer implementations. The common '+' version produces O(n2) complexity since every character from both strings to be concatenated is copied one by one into a new string, while the buffer of bytes only copies in the new string into an existing array of bytes, growing the underlying dynamic array of bytes when it needs to for an amortized O(n) run time.

For string_cocat.go, we have:

package string_concat

import (  
    "bytes"
)

func ConcatOperator(original *string, concat string) {  
    // This could be written as 'return *original + concat' but wanted to confirm no special
    // compiler optimizations existed for concatenating a string to itself.
    *original = *original + concat
}

func SelfConcatOperator(input string, n int) string {  
    output := ""
    for i := 0; i < n; i++ {
        ConcatOperator(&output, input)
    }
    return output
}

func ConcatBuffer(original *bytes.Buffer, concat string) {  
    original.WriteString(concat)
}

func SelfConcatBuffer(input string, n int) string {  
    var output bytes.Buffer
    for i := 0; i < n; i++ {
        ConcatBuffer(&output, input)
    }
    return string(output.Bytes())
}

And for our testing, we've taken advantage of Go's first class functions to give us a beautifully short and readable benchmarking test:

package string_concat

import "testing"

const TEST_STRING = "test"

func benchmarkConcat(size int, SelfConcat func(string, int) string, b *testing.B) {  
    for n := 0; n < b.N; n++ {
        SelfConcat(TEST_STRING, size)
    }
}

func BenchmarkConcatOperator2(b *testing.B)      { benchmarkConcat(2, SelfConcatOperator, b) }  
func BenchmarkConcatOperator10(b *testing.B)     { benchmarkConcat(10, SelfConcatOperator, b) }  
func BenchmarkConcatOperator100(b *testing.B)    { benchmarkConcat(100, SelfConcatOperator, b) }  
func BenchmarkConcatOperator1000(b *testing.B)   { benchmarkConcat(1000, SelfConcatOperator, b) }  
func BenchmarkConcatOperator10000(b *testing.B)  { benchmarkConcat(10000, SelfConcatOperator, b) }  
func BenchmarkConcatOperator100000(b *testing.B) { benchmarkConcat(100000, SelfConcatOperator, b) }

func BenchmarkConcatBuffer2(b *testing.B)      { benchmarkConcat(2, SelfConcatBuffer, b) }  
func BenchmarkConcatBuffer10(b *testing.B)     { benchmarkConcat(10, SelfConcatBuffer, b) }  
func BenchmarkConcatBuffer100(b *testing.B)    { benchmarkConcat(100, SelfConcatBuffer, b) }  
func BenchmarkConcatBuffer1000(b *testing.B)   { benchmarkConcat(1000, SelfConcatBuffer, b) }  
func BenchmarkConcatBuffer10000(b *testing.B)  { benchmarkConcat(10000, SelfConcatBuffer, b) }  
func BenchmarkConcatBuffer100000(b *testing.B) { benchmarkConcat(100000, SelfConcatBuffer, b) }  

Running our test again with go test -bench=., we get:

testing: warning: no tests to run  
PASS  
BenchmarkConcatOperator2    20000000            98.3 ns/op  
BenchmarkConcatOperator10     1000000          1126 ns/op  
BenchmarkConcatOperator100       30000         47260 ns/op  
BenchmarkConcatOperator1000         500       3358669 ns/op  
BenchmarkConcatOperator10000          10     152739858 ns/op  
BenchmarkConcatOperator100000           1    3827856587 ns/op  
BenchmarkConcatBuffer2     5000000           338 ns/op  
BenchmarkConcatBuffer10     3000000           524 ns/op  
BenchmarkConcatBuffer100      300000          5312 ns/op  
BenchmarkConcatBuffer1000       50000         33944 ns/op  
BenchmarkConcatBuffer10000        5000        258864 ns/op  
BenchmarkConcatBuffer100000        1000       1742545 ns/op  

Woah, 3,827,856,587 ns vs 1,742,545 ns for 100,000 runs. That's a ~2000x slowdown in just 100,000 runs -- O(n2) catches up with you fast. And that's it for today folks, hopefully I've shown just how beautifully simple benchmarking in Go really is. I highly recommend Dave Cheney's more in-depth look at Go benchmarking to learn more.