Beautifully Simple Benchmarking with Go
I recently read David Huie's great post about Go being possibly the next great teaching language, and I wasn't so sure at first. I absolutely love Go and use it as my first choice imperative language for both hacking something together quickly and for building robust web applications, but I wondered if an absolute beginner would still be better off with a dynamically typed language like Python.
And then every so often, I run into a problem to solve with Go and it just feels so easy and intuitive while being robust, that I can't help but think that David is absolutely right. Everybody's got a pet language though, so instead of just yapping on about it, I've decided that every time I do something that seems incredibly easy or powerful in Go, I'll just document it for the world to see and judge for themselves. Today, I'll talk about beautifully simple benchmarking in Go.
Let's start with a small example function. This function can really be anything, but I'll just write up a short one that concatenates some strings together, with a function that repeats the function on an input string to give us a problem we can grow as we need. Putting it all in string_concat.go:
package string_concat
func ConcatOperator(original *string, concat string) {
// This could be written as 'return *original + concat'
// but I wanted to confirm no special compiler optimizations
// existed for concatenating a string to itself.
*original = *original + concat
}
func SelfConcatOperator(input string, n int) string {
output := ""
for i := 0; i < n; i++ {
ConcatOperator(&output, input)
}
return output
}
How would we benchmark our function in Go? In Go, we create *_test.go files to specify a test file for any *.go file. Here, we just create a file string_concat_test.go file and include:
package string_concat
import "testing"
func BenchmarkSelfConcatOperator1000(b *testing.B) {
for n := 0; n < b.N; n++ {
SelfConcatOperator("test", 1000)
}
}
Go's testing package will take care of increasing the variable b.N for any function we include within the for loop, so that we'll benchmark our function SelfConcatOperator repeatedly for a minimum of 1 second. This is so that we can get statistically significant results with sufficient repetitions. For functions that take longer per run, we can increase the minimum amount of time Go allots per benchmark so that we get statistically meaningful results.
Now, go to your terminal to where your string_concat.go and string_concat_test.go files reside. All we need to type to run our tests is:
go test -bench=.
And that's it! You'll hopefully get an output that looks like:
testing: warning: no tests to run
PASS
BenchmarkSelfConcatOperator1000 500 3191695 ns/op
ok _/path/to/your/code/string_concat 1.915s
And there it is: 3,191,695 nanoseconds to run our function which concatenated 1000 strings together. That obviously wasn't too useful, but the steps are really that simple for absolutely any function, maybe your favourite database query or text handling behemoth.
...OK, ok, we've got to benchmark something real in Go while we're at it. Here's something a bit more interesting, comparing two possible string concatenation implementations:
- Using the usual, immutable strings and the '+' operator ("goo" += "goo")
- Writing each new string to a growing buffer of bytes.
For those from the Java world, this are the classic String vs StringBuffer implementations. The common '+' version produces O(n2) complexity since every character from both strings to be concatenated is copied one by one into a new string, while the buffer of bytes only copies in the new string into an existing array of bytes, growing the underlying dynamic array of bytes when it needs to for an amortized O(n) run time.
For string_cocat.go, we have:
package string_concat
import (
"bytes"
)
func ConcatOperator(original *string, concat string) {
// This could be written as 'return *original + concat' but wanted to confirm no special
// compiler optimizations existed for concatenating a string to itself.
*original = *original + concat
}
func SelfConcatOperator(input string, n int) string {
output := ""
for i := 0; i < n; i++ {
ConcatOperator(&output, input)
}
return output
}
func ConcatBuffer(original *bytes.Buffer, concat string) {
original.WriteString(concat)
}
func SelfConcatBuffer(input string, n int) string {
var output bytes.Buffer
for i := 0; i < n; i++ {
ConcatBuffer(&output, input)
}
return string(output.Bytes())
}
And for our testing, we've taken advantage of Go's first class functions to give us a beautifully short and readable benchmarking test:
package string_concat
import "testing"
const TEST_STRING = "test"
func benchmarkConcat(size int, SelfConcat func(string, int) string, b *testing.B) {
for n := 0; n < b.N; n++ {
SelfConcat(TEST_STRING, size)
}
}
func BenchmarkConcatOperator2(b *testing.B) { benchmarkConcat(2, SelfConcatOperator, b) }
func BenchmarkConcatOperator10(b *testing.B) { benchmarkConcat(10, SelfConcatOperator, b) }
func BenchmarkConcatOperator100(b *testing.B) { benchmarkConcat(100, SelfConcatOperator, b) }
func BenchmarkConcatOperator1000(b *testing.B) { benchmarkConcat(1000, SelfConcatOperator, b) }
func BenchmarkConcatOperator10000(b *testing.B) { benchmarkConcat(10000, SelfConcatOperator, b) }
func BenchmarkConcatOperator100000(b *testing.B) { benchmarkConcat(100000, SelfConcatOperator, b) }
func BenchmarkConcatBuffer2(b *testing.B) { benchmarkConcat(2, SelfConcatBuffer, b) }
func BenchmarkConcatBuffer10(b *testing.B) { benchmarkConcat(10, SelfConcatBuffer, b) }
func BenchmarkConcatBuffer100(b *testing.B) { benchmarkConcat(100, SelfConcatBuffer, b) }
func BenchmarkConcatBuffer1000(b *testing.B) { benchmarkConcat(1000, SelfConcatBuffer, b) }
func BenchmarkConcatBuffer10000(b *testing.B) { benchmarkConcat(10000, SelfConcatBuffer, b) }
func BenchmarkConcatBuffer100000(b *testing.B) { benchmarkConcat(100000, SelfConcatBuffer, b) }
Running our test again with go test -bench=., we get:
testing: warning: no tests to run
PASS
BenchmarkConcatOperator2 20000000 98.3 ns/op
BenchmarkConcatOperator10 1000000 1126 ns/op
BenchmarkConcatOperator100 30000 47260 ns/op
BenchmarkConcatOperator1000 500 3358669 ns/op
BenchmarkConcatOperator10000 10 152739858 ns/op
BenchmarkConcatOperator100000 1 3827856587 ns/op
BenchmarkConcatBuffer2 5000000 338 ns/op
BenchmarkConcatBuffer10 3000000 524 ns/op
BenchmarkConcatBuffer100 300000 5312 ns/op
BenchmarkConcatBuffer1000 50000 33944 ns/op
BenchmarkConcatBuffer10000 5000 258864 ns/op
BenchmarkConcatBuffer100000 1000 1742545 ns/op
Woah, 3,827,856,587 ns vs 1,742,545 ns for 100,000 runs. That's a ~2000x slowdown in just 100,000 runs -- O(n2) catches up with you fast. And that's it for today folks, hopefully I've shown just how beautifully simple benchmarking in Go really is. I highly recommend Dave Cheney's more in-depth look at Go benchmarking to learn more.
— Soroush Pour (@soroushjp)
Member discussion