What are strategies to construct reusable Sikuli screen shot libraries?

Question

I would like to use Sikuli to automate both GUI apps and Web apps running within browser on Mac OS X and Windows. My purpose is currently less for testing, and more for GUI automation of tedious, repetitive tasks for a team that unfortunately doesn't have lower-level automation access at this time.

I'm thinking that I'd like to build up one or more libraries of screen shots for the GUI apps and Web apps that I can reuse across projects. I'd often be running the same automation steps for different apps or, for Web apps, in different browser/platform combinations.

What are some good strategies for constructing reusable Sikuli screen shot libraries? Some thoughts:

• should I capture screen shots outside of Sikuli, and then slice/dice those images to pull out specific interface elements within Sikuli?
• how can I best keep track of screen shots for equivalent interface elements across similar GUI apps?
• how can I best keep track of screen shots the same Web apps as seen in different browsers or platforms?
• how can I best organize elements that are hierarchical, like menus where you must make choice 1, then choice 2, then choice 3 (but the next choice only appears after the previous one is selected)?
• should screen shots be saved as variables to be able to reference them more generically?
• should I construct Python lists or dictionaries that contain screen shots?
• should I group screen shots into separate Sikuli files based on application/platform?

I'm assuming in all of this that I could import the libraries like a Python module, which certainly seems possible from the documentation.

Thanks!

Answer 1

There is an add-on called "Robust GUI Automation Library for Sikuli".

Even if you don't end up using the library, there are some really good lessons to be learned by looking at their implementation of the problem.

A few suggestions:

should I capture screen shots outside of Sikuli, and then slice/dice those images to pull out specific interface elements within Sikuli?

1. More important than how you get your elements is how those elements are stored. I standardize how I name graphics ie: Button_OK.png rather than Sikuli's unpredictable_default_name.png

2. You can add image libraries "on the fly" in your Sikuli script. Store different browser and platform graphics in different directories.

   myImagePath = "M:\\myImageLibrary\\"
   addImagePath(myImagePath)
   

how can I best keep track of screen shots for equivalent interface elements across similar GUI apps?

Naming conventions!

\\firefox\\Button_OK.png
\\IE8\\Button_OK.png

You can also play with the "similarity" of the Pattern to get the same graphic to hit on both IE and Firefox (but without false positives). This can take a bit of trial and error.

should I construct Python lists or dictionaries that contain screen shots?

This is a really good practice and has worked well for me in certain circumstances. Sometimes though, the filename is better documentation of the script functionality than a list offset.

I'm assuming in all of this that I could import the libraries like a Python module, which certainly seems possible from the documentation.

Yes you can import libraries.

Answer 2

That looks like a great library recommended by spearson.

I would add one more concept to the list, which is calibration.

As with any testing industries, calibration of your instruments is a must.

Within the SQA/automation fields, assumption can lead to disaster.

Scenario:

On Monday, you decide on your Chrome submit button screen shots to be used in Sikuli-powered automation.

You work fast and by Tuesday, your test suite is delivering accurate pass/fails as you expect.

On Friday afternoon, just before beer'o'clock, the machine auto-upgrades to the next minor release of Chrome, which modifies the cancel buttons just enough to be matched by Sikuli as submit buttons.

You glance at your reports before leaving the office for the weekend and your reports seem to be running fine as usual, but you don't realize they're giving false-positives until dreaded Monday when things have been broken all weekend (but, hey, at least you had a good weekend!).

Totally hypothetical situation, but hopefully stresses the need for "testing your tests" or calibrating your tools in a write-once, run-many automation environment.

Solution:

To mitigate problematic situations like the one above, you could setup a web page(s) that you know behaves a certain way to interactions with the screenshots in your static library. Before each test suite/bulk automation project runs, it will call the calibration suite and make sure everything is functioning as expected, be it a browser, file manager, etc

Answer 3

As a Java developer (not sure if this applies to many sikuli users): I rolled my own implementation. I created a singleton FileDirectory object which has methods for default directories (resources, patterns, screenshots). And whenever I had sikuli take a screenshot I would save it to this default directory (which was created if it didn't exist on start-up). My application would than log the screenshot. Using an html logger I could link to the screenshot directory. So my html logs had links embeded in them to the images the application took.

I know this might seem difficult at first, but it ended up providing exactly what use case needed.

Answer 4

Maybe this will give you some idea's.
I have a file where I have multiple definition that need to look for a certain image.
Somethimes the image they need to look for is different in different browsers.
So I use something like this:

File: ImageLib.sikuli

browser_Windows_Firefox = ("windowsFox.png")
browser_Mac_Firefox = ("macFox.png")

File: Execute.sikuli

from ImageLib import *

# Variable you can get set somewhere 
operatingSystem = 'Mac'

image = ()
if (operatingSystem == 'Windows'):
    image = browser_Windows_Firefox
elif (operatingSystem == 'Mac'):
    image = browser_Mac_Firefox
else:
    # Unknow option here.... 
    pass

# Find and highlight it. 
imageLoc = find(image)
imageLoc.highlight(5)