Automa Blog

Kill Bugs Dead

While our name BugFree Software certainly represents one of our core values, it goes without saying that all software of any appreciable complexity, including our products, will contain bugs. All we can do - what we have to do - is to fix any bugs we find and make sure they never return again.

We recently had a bug where Automa would fail to start without displaying an error message to the user when the installation directory was not writeable. This would for instance occur when the user installed Automa with admin privileges to a folder requiring such privileges, but then started Automa without an elevated account. Since we require Automa's installation directory to be writeable, the fix to the problem was to display an error message to the user that says that he might have to start Automa with administrator privileges:

Error message for non-writeable installation directory

In the spirit of acceptance test driven development, the first step in fixing the above problem consisted of writing an automated system test that captures the erroneous behaviour. The backbone of our system tests for Automa is a little Python class that allows sending inputs to and checking outputs from a console process:

class ApplicationConsoleTester(object):
	def send_input(self, input):
		# Implementation details hidden...
	def expect_output(self, expected_output):
		# Implementation details hidden...

You (roughly) use it like so:

cmd = ApplicationConsoleTester("cmd")
cmd.send_input("echo Hello World!")
cmd.expect_output("Hello World!")

When the expected output is not received within a given timeout, an AssertionError is raised. This makes it very easy to use ApplicationConsoleTester in conjunction with one of the unit testing frameworks available for Python.

To highlight the above bug, we wrote the following Python TestCase:

class NonWriteableInstallDirST(unittest.TestCase):
    # Some implementation details hidden...
    def test_error_message_for_non_writeable_install_dir(self):
        self.automa_install_dir.set_writeable(False)
        automa_tester = ApplicationConsoleTester("Automa.exe")
        automa_tester.expect_output(
            "Cannot write to Automa's installation directory. If you "
            "installed Automa with administrator privileges then you "
            "might also have to start Automa with those privileges.\n"
            "\nType Enter to quit."
        )
        sleep(5)
        self.assertFalse(
            automa_tester.hasTerminated(), 
            "Automa did not give the user enough time to "
            "see the error message."
        )
        automa_tester.send_input("\n")
        self.assertTrue(automa_tester.hasTerminated())

Once we had written the test and seen it fail (in the style of good (acceptance)-test driven development), it was easy to fix the bug.

Having the test automated allows us to execute it every time Automa is built. This ensures that the bug will never occur again. But there is another benefit of having such a system test: By discovering and fixing the bug, we have effectively enriched Automa's feature set. Keeping the system tests in sync with development means that the system test suite forms a comprehensive, up-to-date documentation of the required functionality. There is no more partial knowledge as to what works and what doesn't, possibly spread amongst multiple individuals. There is only one truth: that determined by the tests.

blog comments powered by Disqus