What is Screen Automation?

Screen automation is a class of automated tasks based on visual changes on a screen.  Unlike traditional API based automation, here you are not sending commands to a backend API, but rather using a library to detect visual cues on screen (image or text/ocr) and then take decisions based on those queues.

Why not use an API?

An API would be the far better way to automate most things.  Unfortunately there are times where an API isn’t available.  For example, while MetaTrader 4 uses your broker’s API to make trades (and can be scripted using C), in the cases of Binary Options, the user is disconnected from an API.  In these cases the broker charges a fee (or a service) to make automated calls through an API.

I have found that screen automation is more valid when you are working in a closed system.  You have little to no back end access and must replicate human behavior / logic based on visual reference.


The upside to screen automation, is you can automate anything.  Sure webdriver is awesome to automate a browser, but what if you want to automate a Flash application? How about World of Warcraft?  What about making a bot that can place trades on a webpage, based on visual reference in a chart?


Unfortunately you must maintain a library of images for the script to work.  You also are moving at a pace much slower than an API call.  Consider that C can make a API call to a server in nanoseconds…. Ruby/Python/Groovy in ms.  API calls also will probably let you have more detailed access that the web interface may not allow (i.e. premium services that you pay for at the API level.)

Binary Options

Unlike regular Forex (where trades can be sent through MetaTrader), Binary Options are handled via a Web Based system.  Brokers such as Nadex, MarketsWorld, BossCapital, all use a web interface. Binary Options are a gamble that a commodity will rise or fall in a given time frame.  You put some money up for that gamble.

For example, as a trader you can say, “I wager $5 that the price of Gold will drop from it’s current price, at the end of 2min.”

Screen Shot 2015-02-06 at 11.39.16 AM


Webpage… then Webdriver, right?

At first glance you may think, “Well I can script this with Web Driver.”  There’s some trickiness here. The broker is benefited if you slip up.  The broker may change web element ID’s, or they may move values around.

There’s also another problem.  As a trader, your information on what to do (go low, go high, set how long for the expiration of the order) is all based on a separate piece of software – that charting software (NinjaTrader, MetaTrader, etc.)  That means, you would need your charting software to send a signal, that then would kick off the web automation… but the web automation is tricky. Brokers are probably on to web automation solutions and will do some tricks like:

  • remove your default commodity and replace it with something else (I.e. if you trade AUJPY currency, you could open and login with webdriver and find you are bidding on Gold.)
  • change the UI element ID’s to trip up webdriver, or remove all web id’s.
  • refresh the page while it’s open to reset values you’ve picked.

Screen Automation to the Rescue

So we come back to screen automation.  With screen automation we can do something like this:

  1. Check the screen for indicators in the charting package
  2. Have logic that tells the screen automation when to retry and validate an indicator is time for a commitment to put an order in
  3. Use screen automation to navigate the broker’s website to place the order

As you grow in confidence, you can run this more often.

Benefits of Automation

If you have logic for a market condition, and the logic is proven, then automating it takes away some failure points:

  • Emotional States
  • Stress
  • Sleepiness/Groggy
  • Confusion

The automation will constantly work to create the orders you feel are viable.

Negatives of Automation

The automation won’t check news and analyze concepts, such as the Greece and the ECB and its affect on the EUR/USD market.  You can’t just let the automation run wild.  If you do, it will move into an unstable market condition and take massive losses, unless of course you have such sophisticated logic to account for all market conditions.

Screen Automation Setup

So lets get started with the automation.  The goal of the automation is to:

  • Recognize images, patterns and text on screen
  • Make logic based decisions

We’ll need a scripting language that makes API calls to the screen automation framework.  For this purpose I prefer the SikuliX Framework. SikuliX is a MIT project that is run by Raiman.  While this framework has it’s own IDE and scripting ability, the IDE is completely limited.  We will only use the IDE to capture images, and use a scripting language to handle all our logic.

Which Language?

I’ve used Groovy/Grails and JRuby with SikuliX.  My preference is for Groovy.  The reason is that the SikuliX calls for JRuby are missing some key components.  Since SikuliX is a Java Jar, it makes more sense to utilize a Java based language.  For the purpose of this example, I’ll be using Groovy.

Installing SikuliX

Installing SikuliX isn’t that bad… you can follow the official instructions over at:

  • Download the SikuliX installer
  • Save it to the appropriate location (i.e. in Windows I believe you can not put it in Program Files), on OSX I put it in /Users/[me]/SikuliX… then run the setup jar
  • In the SikuliX Setup screen, you must have option #2 selected. This will create the Java-Jar file
  • Once all the jar’s are extracted to the install folder you chose, just take note of the path

Project Creation

I used IntelliJ to set up the project.  IntelliJ allows me to add libraries via the Project Structure window (accessed via File | Project Structure.)  This is super easy, as we can simply point to the SikuliX Java Jar file location that was installed.

In your IDE do something similar to add the SikuliX jar to your project.

Note, do NOT move the SikuliX jar from it’s install location.  It will not run standalone and must be run from the folder it installed to.

Collecting Image Data

Get Your Indicators All Ready

At this point you want to get your charting software up and running and pull out indicators that have clear YES/NO output.  It could be a Red Square, or a Green column, or a number on the screen… or the words “GOOD SPREAD.”  You will want to modify your indicators so that the output is easy for SikuiX to figure out.  Don’t expect SikuliX to know what to do with a MACD line.  Such indicators fluctuate so much SikuliX won’t be able to identify the subtle changes.

You could use the Data window in MetaTrader4, and have SikuliX pull out data values that populate (like Spread, MACD values, etc.)

What I prefer is strong visual queues.  I modify my EA’s and Indicators to give me words on the screen.  I like to use indicators that have strong visible representation (long Green Bars, or Red Bars.)

You will need to get what you expect to see on the screen.

For the purpose of this post, I’m going to provide two example visuals that can be found easily by SikuliX.

Example Chart Visuals:

GOOD green_sell_indicator





The three indicators above could mean anything.  The Green bar might be an RSI Overbought, the RED a RSI Oversold, the world “GOOD” could be the spread of a candle.

Whatever you use for an indicator, make sure you have visuals that will be easy for a computer to pick out from all the noise on the screen.

It may help to change the color themes of your charting software to better pick out these indicators from the rest of what’s on the screen.

Capturing the Images

It seems logical that you should just be able to grab screenshots, or on OSX do a CMD + SHIFT + 4… but it won’t work.  Why?  I don’t know.  I use the SikuliX IDE to capture all the images I need.

To do this, first open the SikuliX IDE.  After it launches you have to wait a min or two (it will be unresponsive for a few min.)

Once SikuliX is ready, save your SikuliX file to your Groovy project (just makes it a bit easier.)

Once it’s working, make sure you do not have a secondary screen plugged in (Like a Thunderbolt) and click the option:

Screen Shot 2015-02-06 at 12.33.46 PM



Once you click that, the screen will dim and the SikuliX IDE will disappear.  You will have your chart in the background with your indicator up.  Use your mouse and click/drag a rectangle to cover the range of your indicator.

Once done, the right side of your SikuliX IDE will now say something like:

exists() and in the parenthesis will be the image itself.

Click on the image in the parenthesis. Change it’s name to something familiar to you.

Repeat this process until you capture all the visual indicators you want.

Now you should have your indicators all saved.  On OSX the images save within the sikuliX file.  So you saved this SikuliX file as a name.sikuli file.  Hopefully you saved it in your Groovy project.

So you go to your Groovy project, click in to name.sikuli file and it will show you tons of images inside (all those you captured.) I recommend you copy these out to a folder at the root of your project… these will be the official images used in the automation… for example, putting the images into /sikuli_imgs


Create a groovy file in your new Groovy Project.


First lets add a few imports at the top:

import org.sikuli.script.App
import org.sikuli.script.Key
import org.sikuli.script.Pattern
import org.sikuli.script.Screen

I’ll explain a bit of what these classes do.

App is a SikuliX class that lets us open apps or change focus on applications (like bringing Firefox to focus, and then MetaTrader, etc.)

Key is a SikuliX class that lets us type or send keys… like clicking “tab” or entering a value into  text field.

Pattern is very important.  Without pattern, SikuliX can find images on the screen but will take a low matching threshold.  We want to make sure we have exact matches, so we will use methods from this class.  NOTE: This is a class that has a lot of missing methods in the JRuby version, which is why we’re using Groovy.

Screen is the classic SikuliX class that does all the image recognition.

There are a few others you can consider. One that is really useful is the Region class (that lets us block out a x,y coordinate of the screen to only consider.)


Let’s instantiate a few things.  First, we want to create a reference for Screen… so we can do:

sc = new Screen()
greenbar = new Pattern("sikuli_imgs/green.png")
redbar = new Pattern("sikuli_imgs/red.png")
goodspread = new Pattern("sikuli_imgs/GOOD.png")

After Screen is instantiated as sc, I created some variables here… greenbar, redbar and goodspread. notice they instantiate the Pattern class and pass into it an image.

In the examples above, greenbar might be a overbought RSI indicator, redbar might be a oversold RSI indicator and so forth. Good is the word “good” on the screen.

Check Indicators Method

Let’s make a method in our groovy file that will look for these indicators.

def checkIndicator(time, reTest = false){
                bar = “greenbar”
                println “I see a greebar”
               bar = “none”
               if (reTest == true){
                   test = “greenbar”
                   return test
                    return false

In the above code example, I’m just going to check for one indicator.

Notice I’m using sc with a  dot method of exists, which passes in the parameter greenbar and a dot method of similar, which has a parameter of a percentage (0.99) or 99%.   That bit of code says:

  • Let’s see if something exists on the screen
  • It should the the image referenced in the Pattern instantiation “greenbar”
  • we want it to be 99% similar to the image.

why didn’t I pass in 1, instead of 0.99?  Because there’s a SikuliX bug where it won’t work on 100% accuracy, so we use 99%, which is pretty much as good as we’ll get.

My method has two params (one required.)  First I pass in a time to sleep, and I have an optional parameter to check if this is a retest (I set it to false by default.)

I’ll explain those parameters in a sec.  For now… let’s test this.  put your indicator on the screen somewhere (the image.)  Add a call to this method like so:


If found it should sleep 1 second and report “I see a greenbar.”  If you remove the image from the screen it should no longer report that.

If it does, that means SikuliX is set up fine and you’re able to validate an image on screen (your indicator image.)  As long as that indicator is on screen, we can do some logic.

Timing Strategy

I take 2 required parameters (time in ms) and one optional in the checkIndicator method.  Here’s why.  I have a strategy that requires specific times to make decisions.

My strategy:

  • I work in 5min intervals (5min chart)
  • I want to make a decision in the last 30sec of a 5 min candle (4min 30sec from the start of a candle)
  • after the sleep, I check the screen to see if the indicator is visible.
  • If it is, I will want to place a bid through my Binary Option broker.

I’m not giving out all my logic. Obviously I must check for different market strategies… I’m providing enough code for you to emulate this if you desire.

Core Code

Above the checkIndicator method, I have a while loop… like so:

x = true
   sleep(1000) // loops to check current time every second
   Calendar cal = Calendar.getInstance()
   def min = cal.get(Calendar.MINUTE)
   if (min % 5 == 0) {
         println “I’m in a 5min candle”
         def ci = checkIndicator(270000)
         if (ci == “greenbar”){
            if(sc. exists(eurusd.similar(0.99)){
                  println “Placing bid on EURUSD”
                  println “Do not see Option for EURUSD”
                  x = false
In the incomplete code above, I have a while loop. It will run as long as x is true.  Inside the loop the first thing I do is sleep 1 second.  Next I check the current min.  If the current min of the hour is evenly divisible by 5, then it’s the start of a 5min candle and I’m going to start my Binary Option strategy.  However, if the min is not yet divisible by 5 (i.e. 12:22), then I loop, wait a second, and repeat till I get into the next available min that is divisible by 5 (i.e. 12:25.)

Next I call the checkIndicator method, passing in the time parameter of 270000 ms.  So I’m saying, “Ok the 5 min candle has started, lets call the checkIndicator method and tell it to wait 4min 30 sec. and then see if we have any indicators for our strategy.”

After the checkIndicator method call, I check the value returned from checkIndicator (ci) and see if it is equivalent to the word “greenbar.”

If so, I now will commence with the Browser stuff… I switch focus to Firefox… If need be I would have login logic, or else, if it’s already logged in, I would use the method to find the appropriate images on screen to click, etc.

You can make use of images and keys to get everything you need to place an order.


What is the reTest?  As mentioned, I don’t have all my code provided.  In my case, I have an indicator that can sometimes fire premature.  In those cases it can fire on a candle, then on the next and on the next.  I only care about the last candle it fires.

In this case, when the indicator is found, I call the method itself and change the reTest value to true…. like so:

return checkIndicator(reTime, reTest = true)

Now the same method is called and this time reTest is set to true… inside the method I have some logic to check if this is a reTest or not… basically, if i still see the indicator… I retest again.  however if at some point my reTest  had previously seen an indicator that I no longer see… I exist out returning a value to the while loop.

Proof of Concept

In the attached video below, a short proof of concept is run.  This is a simulated market (as waiting for the right market condition to arise would be problematic for a quick demo.)  By displaying the MT4 indicator on screen (green bar) and then hiding it once the script finds it – then the spawns a browser with the broker, sets the buy/sell options and places the bid.


No responses yet

Leave a Reply

Your email address will not be published. Required fields are marked *