Image Recognition

In addition to specifying GUI elements by their name (or other properties), you can also refer to them via screenshots. For example, suppose you want to automate clicking a button "Cancel" on your screen:

Cancel Button

To do this using image recognition in Automa, you take a screenshot of the button and save it in a .png file, say "cancel_button.png". Then, you use the Image predicate to refer to your screenshot:

start("notepad")
write("Some text")
click("Close")
# Cancel the "Do you want to save?" dialog:
click(Image("cancel_button.png"))

A special feature of Automa's image recognition algorithms is that they are compatible with the Sikuli image automation tool. For details, please refer to this blog post.

2-Dimensional Regular Expressions

In some cases, it can happen that the application you are automating itself changes the size of its GUI elements. For example, you may have a menu element that is sometimes displayed wider than normal:

Expanding button size

Such cases present a problem for image recognition because the actual pixels that make up the GUI elements are different.

To cater for cases like the one above, Automa provides a facility called "two-dimensional regular expressions". The idea is the following: An image that expands horizontally like

Menu entry Save

can be seen as the "Save" part, followed by a vertical grey bar some number of times, followed by the "CTRL + S" part:

Menu entry Save constituents

This is somewhat similar to the wildcard pattern "*.*" sometimes used when searching for files or, more generally, to regular expressions that are a ubiquitous tool for matching strings of text in computing.

Provided the three image parts are saved in files called "save_left.png", "save_center.png" and "save_right.png", Automa lets you identify an expandable GUI element such as the one above using the expression

Image("save_left.png").followed_by(
	Image("save_center.png").once_or_more().followed_by(
		Image("save_right.png")
	)
)

In addition to once_or_more, Automa also supports the quantifiers any_times and maybe_once.

If you have an image that expands vertically rather than horizontally, you can supply the parameter VERTICALLY as an extra argument to the functions followed_by, once_or_more and any_times:

Image(...).followed_by(..., VERTICALLY)
Image(...).once_or_more(VERTICALLY)
Image(...).any_times(VERTICALLY)