Updating Automation Framework: Minimize Flakiness

Web automation often has areas of flakiness. Sometimes a site is so dynamic that tests just end up executing too soon or too late. Adding sleep is not always the solution, and often the worst solution (only as a last result). At this point in my Python – Selenium Test harness and web app, I’ve removed all flakiness. Pulling for visibility and retesting a failed case are two examples of minimizing flakiness

Pulling for Visibility

One pain point for automation tests is the dynamic response hit various next actions can take. Clicking save on a screen may take anywhere from a fraction of a second to several seconds. While a slow response can be a use case to fail a test, when functional automated regression is happening, we want to know if the core functionality works as desired.

While Selenium’s built in waits (wait for element, wait until visible) are one way of dynamic waiting, I find in my own tests it often fails. This can be for a variety of reasons, but one solution is to create a method that validates if a button is still active (such as a clicked save button) or if a modal is still visible after clicking a next action.

Waiting for something to become Visible

def check_exists_by_css(val):
     try:
         self.driver.find_element(By.CSS_SELECTOR, val)
     except NoSuchElementException:
         return False
     return True

i = 0
while description is False and i < 10:
     time.sleep(1)
     description_visible = check_exists_by_css('.description')
     print("Is description visible? " + str('.description'))
     i = i + 1

After a 1 second pause, the while loop calls the check_exists_by_css method. This method takes a parameter of the CSS element. It returns true if it exists and false if it is gone. As long as it’s false we wait until it becomes visible.

Pulling for when something is No longer Visible

In a situation where a modal is visible, but you want to wait until it’s no longer visible to move to the next action, you can do the reverse of what’s above.

    def wait_until_modal_closed(self, css_value):
        modal = self.util.check_exists_by_css(css_value)
        print("Is modal open? " + str(modal))
        i = 0
        while modal and i < 20:
            time.sleep(1)
            modal = self.util.check_exists_by_css(css_value)
            print("Is modal open? " + str(modal))
            i = i + 1
        print("modal is closed")
        return

The code above is from a Python Class that waits for modals to be closed. The method is called, passing in the css to check if it exists (the check_exists_by_css method is from a utility class in this example). We get back a False or True, and if True, we wait 1 second and loop up to 20 times (i being a counter). Once the modal is closed and the check_exists_by_css method returns false, then we know the modal is gone and we can proceed to the next step.

Why not use waits? As mentioned, Selenium Waits (wait for element or wait for something to be visible) are good, but I run into specific fail points using those methods. Manually pulling for an element to exist or no longer exists seems to work a lot better.

Retrying Tests that Fail

Another method to knock out flakiness is to re-run a failed test before logging the failure to the database.

def retry_test(function, test):
    try:
        function()
        results.__setitem__(test, 'Pass')  # saving result to data object
        test_pass = True
    except Exception as e:
        print(e)
        results.__setitem__(test, 'Fail')   # saving result to data object
        test_pass = False
    if REMOTE is not False:
        db_connection(test, test_pass)     # pushing result to db

def report_results(function, test):
    try:
        function()     
        results.__setitem__(test, 'Pass')   # saving result to data object
        test_pass = True
        if REMOTE is not False:
            db_connection(test, test_pass) # pushing result to db
    except Exception as e:
        print(e)
        print("Retrying Test: " + test)
        time.sleep(30)  # provide time for next rerun
        retry_test(function, test)

The above example is what I currently use. Test results are pushed to the report_results method which takes the name of a test (which is also the name of the test function). I turn that name into a function and if the try clause works, we push the pass to the db. If it fails, I wait 30 seconds and then the except clause pushes the test to some retry logic (retry_test()). If the retry fails, a failure is logged to the database.

With these two methods in place, I’ve eliminated flakiness in my tests. Granted there might be reasons to throw an error/failure on the first failure. Or a use case might want to fail of the next action takes too long. However, for functional testing of code, these work-arounds are very useful.