Spicing Up Your Go Test Harness with Bash

Learn how to quickly string together a set of very simple bash commands to make your go tests output stand out have an immediate understanding of failures, coverage. … it may not be the best way to do this, but it is good enough for me ;-)

Photo by Theo Crazzolara on Unsplash

Making eye contact with your go tests

Golang already has a good built-in tooling to run automation testing and gather insights test coverage metrics. Every go developer is very much familiar with the go test and the go test -cover commands. These are excellent tools that allow you to easily run a battery of tests and have additional information about coverage.

In line with the philosophy of the language the output provided by the tool is also quite terse and essential, in its default configuration. While we can configure the execution of the test to produce detailed and extensive HTML output for the coverage metrics, if we have a setup that that is meant to run tests on commit, we are more interested in quickly getting a visual cues on the status of tests without exploring a generated report. To this purpose, the standard output of the tool my result a bit bland and not immediately capturing your attention as needed.

I am a particularly visual person and my attention is more easily driven by cues. These help me quickly focus on the things that matter and without requiring a deep focus. When it comes to committing code changes I want to immediately identify which tests are being broken and so that I can move my attention to the things to fix. This becomes more and more important when you have a large number of tests to run, and to my eyes the black and white output of the shell is not that enticing!

Enter bash, and its powerful set of capabilities and related tools, often forgotten. We can use basic capabilities such as pipes, file manipulation, advanced terminal capabilities, and string searching and regular expression to improve the test output and make what matters stand out. This task can be easily done in a few hours and then become part of your tool belt for use in any go project.

Understanding the go test output

The execution of a battery of tests in a go project can be simply executed by running the command go test in the directory where the go source code and tests resides. The output looks something like this:

.... a lot of blurb generated by the execution of the tests...
PASS
ok     <path relative to $GOPATH/src/>  <time>s

In case of test failure you will see something like this:

.... a lot of blurb generated by the execution of the tests...
FAIL   <path relative to $GOPATH/src/>  <time>s

Based on the amount of output generated by the execution of the test this information may be lost and not very easy to capture at a glance. In addition, this output is difficult to consume if we want to identify which (and how many) tests have been run. Golang helps us by providing the ability of specifying a verbose mode (i.e. -test.v) that produces additional details for each test. If we then run go test -test.v we get something like this:

=== RUN: <TestFunctionName1>
... output of the single test....
--- PASS: <TestFunctionName1> ([time]s)
=== RUN: <TestFunctionName2>
... output of the single test....
--- FAIL: <TestFunctionName2>
....
...
=== RUN: <TestFunctionNameN>
... output of the single test...
--- PASS: <TestFunctionNameN> ([time]s)
FAIL
exit status 1
FAIL   <path relative to $GOPATH/src/>  <time>s

Now we can see which tests have failed, but again, such information may be lost in the debug information generated by the execution of the tests and not easy to grasp immediately.

Sifting through the noise of test logs..

The very first thing we would like to do is to eliminate the unnecessary information in the output and just print or capture the output needed to generate a summary view.

The output format of the verbose mode helps us quickly identify the outcome of each test: lines starting with === PASS: <TestFunctionName> provide information about a successful test, while lines starting with === FAIL: <TestFunctionName> provide information about failing tests. By using bash we can easily remove all other lines and simply output these lines to the console with the use of grep :

go test -test.v | grep --text "^---"

The --text passed to the grep command ensures that the input is treated as text rather than binary should the output of the test generate information that may not be interpreted naturally as text (e.g. binary dumps).

This simple filter will allow us to see something like this:

--- PASS: <TestFunctionName1> ([time]s)
--- FAIL: <TestFunctionName2>
....
--- PASS: <TestFunctionNameN> ([time]s)

This view already provides us with a summary view, as we can easily see which tests have failed and take action.

Another improvement that can be easily made is ensuring that PASS and FAIL are properly highlighted by associating them with colours that are usually mapped to success and failure: green and red. This task is easily accomplished by leveraging: 1) the support for colours that ANSI terminals have; 2) basic substitution rules with the help of sed.

A good discussion on the support for colours on the various terminals can be found in Wikipedia, but if you just want a quick reference just check here. Text formatting can be added by surrounding the text to be formatted by escape codes. The echo command interprets these codes and applies the proper transformations to render the text. The bash script below provides you with an example of how to write PASS and FAIL in the corresponding green and red colours:

#!/bin/bash
echo -e "\033[0;32mPASS\033[0m"
echo -e "\033[0;31mFAIL\033[0m"

What is happening in the above script? The -e switch passed to the echo command tells the command to interpret the escape codes that are passed in the string as escape code and apply the transformations required to the text (based on configuration of the terminal it is optional). The \033 (or \e , but not as universal as \033 ) preamble identifies an escape sequence, which starts with the open bracket [ . The sequence is composed by a list of codes separated by a ; that identifies the transformations to apply the text terminated by the m character. In particular:

• \033[0;31m indicates to set background colour to black (i.e. 0) and foreground colour to green (i.e. 32)
• \033[0;32m indicates to set background colour to black (i.e. 0 ) and foreground colour to red (i.e. 31)

Because the escape sequence sets a mode in the terminal, we need to rest the modality once we have completed the rendering of the text of interest. This is the reason why at the end of the text we find the escape sequence \033[0m which completes the string.

Now that we understand how to colour text, we can simply use sed to replace the occurrences of PASS and FAIL with the corresponding coloured version and since we are there we also strip away the initial --- preamble that we don’t really need.

go test -test.v | grep --text "^--- " > summary.test
sed -e 's/--- PASS/\\033[0;32mPASS\\033[0m/g' \
    -e 's/--- FAIL/\\033[0;31mFAIL\\033[0m/g' \
    summary.test > summary.colour

Note that the \ character has a specific meaning for sed expression and if we want to prevent that it is being interpreted by sed we need to escape by preceding it with another \ character. At this point, if we simply read the file line by line we can see our test summary coloured in a way that easily shows which test failed:

while read -r LINE; do echo -e "${LINE}"; done < summary.colour

This gives an output similar to this:

Great! we now have something easy to read and eye catching.

Adding coverage information

Another important information we may want to capture and highlight is the level of coverage of our tests. The standard tooling available with Golang provides us with the required information and it is just a matter of extracting it and massage it a little bit to put it in evidence.

Information about test coverage can be obtained if we add the -cover flag to the command line executing tests. If we run:

go test -test.v -cover

We obtain something like this:

=== RUN: <TestFunctionName1>
... output of the single test....
--- PASS: <TestFunctionName1> ([time]s)
=== RUN: <TestFunctionName2>
... output of the single test....
--- PASS: <TestFunctionName2>
....
...
=== RUN: <TestFunctionNameN>
... output of the single test...
--- PASS: <TestFunctionNameN> ([time]s)
PASS
coverage: <percentage> of statements
ok   <path relative to $GOPATH/src/>  <time>s

We can easily extract the coverage information by using the grep command and for better handling convert it into a number. The following command capture the line containing the coverage data and extract the string that represents the percentage of statements covered by tests.

go test -test.v -cover | grep --text "^coverage: " | cut -d ' ' -f 2

The cut command breaks down the string captured by grep by using the space as a delimiter and extracts the second item in the line which represents our percentage. Given that we want to do further processing on both the original test data and the percentage we can modify the line above as follows:

go test -test.v -cover > output.test
COVERAGE=$(cat output.test | grep --text "^coverage: " | cut -d ' ' -f 2)

This allows us to operate on the coverage and, for instance, providing different rendering for the percentage. To apply different rendering to the coverage, we first need to convert it into a number. The flexibility of bash here comes to the rescue again, for as long as a string represents a valid number it will be possible to treat it as such. Given that Golang represents the percentage in the form dd.d% we simply need to:

• remove the percentage at the end; and
• multiply the number by 10 (it is better to have an integral number).

This is easily done by using the expansion rules as follows:

COVERAGE_NUMBER=${COVERAGE_NUMBER//[.%]/}

This expansion rule, simply eliminates the . and % characters. The result is a string only composed by numbers, which incidentally is also the percentage multiplied by 10. With a number, we can now decided to divide the coverage value in brackets and assign different colours based on the percentage:

• [0, 49.9] => red colour
• [50.0, 79.9] => yellow colour
• [80.0, 100.0] => green colour

This can be easily done with simple regular expression and the case construct in bash:

case ${COVERAGE_NUMBER} in
  [0-4][0-9][0-9]|[0-9][0-9]) COLOR="31m" ;;
  [5-7][0-9][0-9]) COLOR="33m" ;;
  *) COLOR="32m" 
esac

Remember that a coverage of 45.6% is now expressed as 456. The case options are designed to match ranges for numbers that represent the brackets discussed before. For the rendering we can still use COVERAGE since it retains its original (and more human intelligible) form.

Bring the monkeys in …

Today’s ANSI terminal implementations have full support for the UTF-8 character set. This means we can use special UTF-8 characters codes and these will be rendered as icons in the terminal. Minikube is a great example of how icons can make more interesting a console log. We will be using the sample principle here, to further enrich our visual experience and creating more compelling test reports…. monkeys!

case ${COVERAGE_NUMBER} in
  [0-4][0-9][0-9]|[0-9][0-9]) COLOUR="31m" ; ICON="🙈" ;;
  [5-7][0-9][0-9]) COLOUR="33m" ; ICON="🙉" ;;
  *) COLOUR="32m" ; ICON="🐒"
esac

If you are on a Mac OS X, you can simply drag and drop the emoji in the bash script and these will be rendered as shown above. Similar solutions are available for any operating system, as emoji are represented as a sequence of UTF-8 character codes, which could also be typed directly. You can find here the full list of supported emojis with the corresponding UTF codes.

If we want to output the coverage we can simply add the following:

echo "Test coverage: \033[0;${COLOUR}${COVERAGE}\033[0m - ${ICON}"

When we put all together, we obtain something like this:

Obviously, if you don’t like monkeys you can choose any other emoji.

Putting it all together

While a visual test summary may be good, we would still like to retain the most important information, which is whether ALL the test passed or there is any failure. We could parse the information from the output of the tests but Golang provides us with a better way to capture this information.

The execution of the go test command returns a non-zero value if there is any test failure or any other error. We can capture this value in a variable and then control the display of our tests based on the exit code of the command.

go test -test.v -cover > output.test
RETVAL=$?
....
if [[ ${RETVAL} != 0 ]]; then
  echo "There are test failures!"
else
  echo "All tests passed"
fi
echo  "Test coverage: \033[0;${COLOUR}${COVERAGE}\033[0m - ${ICON}"

In addition, we would like to have this information as a header rather than at the end of the list of tests and perhaps decide to print the details of all the tests executed if and only if there are test failures. We can easily wrap this behaviour into a bash function that receives as an argument the path where to run the tests.

When we put all together the we have a script like this:

#!/bin/bash
# This function runs the tests via go test, captures information
# such as the coverage and the detail of the tests run and presents
# it in a more effective manner. The function takes one parameter 
# that is the path to the package to test. 
# 
function test_package() {
   PASS="👍"
   FAIL="👎"
   CURRENT_PATH=$(pwd)
   PACKAGE_PATH=$1
   cd ${PACKAGE_PATH}
   
   # run test with detail and coverage and save to file
   go test -test.v -cover > output.test
   RETVAL=$?
   # extract the coverage and massage it into a number so 
   # that we can bracket the value and render it based on
   # the range
   COVERAGE=$(cat output.test | grep --text "^coverage" | cut -d ' ' -f 2)
   COVERAGE_NUMBER=${COVERAGE_NUMBER//[.%]/}
   case ${COVERAGE_NUMBER} in
     [0-4][0-9][0-9]|[0-9][0-9]) COLOUR="31m" ; ICON="🙈" ;;
     [5-7][0-9][0-9]) COLOUR="33m" ; ICON="🙉" ;;
     *) COLOUR="32m" ; ICON="🐒"
   esac
   # determine the outcome of the test and provide total
   # summary. If there are test failures, process the 
   # output and show the detail
   echo "Package: $PACKAGE_PATH"
   if [[ ${RETVAL} != 0 ]]; then
     echo "${FAIL} - There are test failures!"
     echo "Coverage: \033[0;${COLOUR}${COVERAGE}\033[0m - ${ICON}"
     
     cat output.test | grep --text "^--- " > summary.test
     sed -e 's/--- PASS/\\033[0;32mPASS\\033[0m/g' \
         -e 's/--- FAIL/\\033[0;31mFAIL\\033[0m/g' \
         summary.test > summary.colour
     while read -r LINE; do echo -e "${LINE}"; done < summary.colour
     rm summary.colour
     rm summary.test
   else
     echo "${PASS} - All Test Passed!" 
     echo "Coverage: \033[0;${COLOUR}${COVERAGE}\033[0m - ${ICON}"
   fi
   echo ""
   rm output.test
   cd ${CURRENT_PATH}
   # we return the original value returned by  `go test`
   # so that the calling context can still perform any
   # other operation based on the outcome of the tests
   return ${RETVAL}
}
# we can now invoke test package on multiple paths and have
# a test summary for each package.
for PACKAGE in (ls ...); do
  test_package $PACKAGE
done

Once you have removed comments and spaces, this script is about 30 lines of bash commands. Very short and simple. The output for a set a successful execution of the test is something like this:

If the battery of tests run has failures you will see something like this:

A few thoughts

In this article we have explored some of the capabilities of the go test command and how to spice it up with a few bash commands to make the output of the test more readable at a glance.

Why bash? Well, the intent was to demonstrate how much you can do from the shell without the need for a full-fledged programming language and a compiler/interpreter. Bash (and other versions of the shell interpreter) is available by default in systems that even have minimal footprint and very little components.

Stringing together a few bash commands to implement the simple test report did not take more than a couple of hours and can be done without any fancy editor. We have managed to pretty up a dull console output thus making the information contained in it more readable.

The script is not perfect and there are several areas of improvement:

• we can probably avoid some of the processing and use more complex pipes;
• we can also add information about the total number of test failed versus the total number of test executed; and
• we can make the script more robust with respect to the way it captures and processes the output (at present time we are relying on the fact that the standard test output does not have lines starting with --- or coverage:).

In other words, there is room for improvement! Poke around fiddle with it and adjust it to your need. It will serve you well once it is in your tool belt.

DISCLAIMER: while I have tried to keep the script as general as possible I have run it on a Mac OS X box. Slight changes may be required if run on other operating systems, based on the implementation of some of the POSIX commands such as echo and sed . These changes are minor.