Paul Hammant's Blog: Playing With Sikuli For Desktop Automation

Sikuli has been on the scene for a few years. It continually moves forward, but is also ready for prime time usage. Ready, not only in the QA field, but for any job that would require you to automate more than just the inner bounds of a browser frame (yay Selenium!).

I made a simple demo of it’s capabilities on Github in Java with Maven as the build-tech. You should be able to make that into a Gradle build quite easily.

My Sikuli script finds Firefox (already open in my universe), and points it to a speadsheet demo written in Flex/Flash. It then clicks overwrites some cellss, and asserts that a total cell was updated (based on the changes made).

# How to run it, after cloning from github:
mvn install 

Raimund Hocke (who is pushing Sikuli hard and would love people to get involved) forked my repo and did a number of cleanups and refactorings. It is much more sophisticated. He’s also moved the logic to a main() nethod, and gone from Firefox to Safari. You’ll be prompted to change a system permission for for this script to work well (SystemPreferences -> Security & Privacy -> tab Privacy -> check the accessibility for “terminal”. You might want to turn that off, after running the script). Here is the video showing Raimund’s script running:

# How to run it, after cloning from github:
mvn exec:java -Dexec.mainClass="my.sikuli.example.MySikuliTest" 

Of course there are other techniques for automating Flash from within Selenium, but I wanted to showcase Sikuli here. You can also use Selenium and Sikuli together.