Webdriver.IO (2021)

I recently looked at the Web Automation framework, Webdriver IO, in 2021 and compared it to what I currently use. There are certainly some useful aspects to it, but there are also many drawbacks.

To start with let me mention what I currently use for automation…. I build a lot of automation tools for specific testing (log parsing, data analysis, phone call quality, etc)…. however for web automation testing I use Golem… I love it. I like Python as a language, so that’s a plus, but I also like how the system runs in a self-contained web app. You don’t have to run it that way, but that’s the default. It has a nifty reporting system built in, and comes all setup up with an easy UI for editing your configs… like setting up test environments in JSON blocks, browser support, etc. It even pulls down the latest web drivers when needed.

But I’m always looking to improve. If there are better things on the market, I’m open to learning something new, and I really would like to put my JavaScript to the test… a framework written in JS might be a nice way to go… and although we could build it from scratch, let’s take a look at one of the biggest names out there: WEBDRIVER IO.

Setting up Webdriver.IO

Getting Webdriver.IO up and running doesn’t take much effort. If you follow along the Getting Started guide, you’ll see you really just run a npm installer:

https://webdriver.io/docs/gettingstarted

Setting it up you’ll get prompted for things like, “do you want to use BDD,” “what browser do you want to use as a default…” and so on. I went with Cucumber for BDD, and Chrome as the browser default, and chose async test types.

Once installed you can create a template test with the following command:

npx wdio config

But keep in mind it’s broken. It appears their hosted demo site on Heroku is not functional, so the template automation will fail. However, if you treat it as a scaffolding generator, it does generate a sample feature test, and specifications that are useful to build from.

The Bad

Out of the box I found that Webdriver IO doesn’t really do things I’d expect. It took hours of tweaking to get it setup in a way that would be QA friendly.

Setting up Environments

Shockingly poor in implementation is lack of a built in environment system… well there is a work around… but its messy. This is something that should work out of the box… I went through various stack overflow answers that were messy… ending up with this video which I think handled it the best:

Basically you edit the config file (wdio.config.js) and change your baseUrl value to simply declare it: baseUrl, then at the top of your wdio.config.js file you add logic to check process.env.[YOUR ENVIRONMENT NAME] is true, then use the specified baseUrl. From the command line you kick off the test with [YOUR ENVIRONMENT NAME]=true npx wdio….

QA31=true npx wdio --spec=login

If your team has a lot of environments (like I do), then you end up with a convoluted IF or Switch statement at the top of your wdio.config.js file.

Messy. But it’s the best solution I could find for the framework. Compared to Golem, well Golem wins hands down in configuration. In Golem you edit a JSON file and specify key value pairs of your environments (“qa31”: “https://…”), which is a lot cleaner and easier to maintain.

Clearing Browser Sessions Between Tests

A basic premise in tests is that each test scenario (or test) will run in its own session. By default Webdriver.IO doesn’t work like this. It runs all tests using the same browser session… I found that very strange. It’s a great idea if you had a bunch of tests for a logged in user… but what about running a test of a user who logs in successfully, then a test where a user fails login. Then a test where a user resets their password… all using the same login screen. Session becomes a problem in these cases.

Googling around I found on stack overflow a solution to use browser.reloadSession(). This is added in the config file (wdio.config.js) in one of the “after blocks.”

There are several after block options:

• afterStep
• afterScenario
• afterFeature

In my case I use afterScenario… however, browser.resetSession() didn’t work for me! It certainly reset the browser session, but it caused all sorts or problems on the subsequent tests!

On a whim I tried, browser.deleteCookies() in the afterScenario and that did the trick.

Compared to Golem, Golem by default runs every browser test in an incognito type of mode. Each test has a clean session. This might get tricky when you want several tests to share data, but the most common type of problem is keeping session data clean, something that by default isn’t setup in Webdriver.Io.

Setting up Reports

The reports aren’t setup. I suppose its because they offer a dozen or more different reporting systems, but you’d think there’d be a default.

This seems like a trivial task to integrate a reporting system, but it becomes a PITA. How it works is you to the docs and look on the left nav for “reports”.

After clicking on a few report options, I liked the look of Allure, so I went ahead with that one, you can see the details on allure reports from this link: https://webdriver.io/docs/allure-reporter

In your test folder you

npm install @wdio/allure-reporter --save-dev

After you do that, the docs say you add this to the config:

 reporters: [['allure', {
        outputDir: 'allure-results',
        disableWebdriverStepsReporting: true,
        disableWebdriverScreenshotsReporting: true,
    }]],

If you were to run a test at this point, you’ll get XML files in the allure-results sub-directory of your test folder. Nothing too usable.

You now have to install Allure… I thought the npm install of it above did that, but it didn’t. You actually need to go to the official document for Allure and notice it uses special syntax for installation:

npm install -g allure-commandline --save-dev

Once that was done I could now generate a report by running:

allure generate allure-results && allure open

It was… well it was OK. It was confusing because some of the steps were out of sync… I saw the Given, Then and then the When… of each and every test. It’s not a deal breaker, but it’s odd… it’s not something I’d show to a decision maker. But for my own benefit this particular report as OK. Granted there are many other report styles to chose from.

But why not auto generate the report each time the Test concludes? Supposedly this is doable… You simply add this code into the config file:

onComplete: function() {
        const reportError = new Error('Could not generate Allure report')
        const generation = allure(['generate', 'allure-results', '--clean'])
        return new Promise((resolve, reject) => {
            const generationTimeout = setTimeout(
                () => reject(reportError),
                5000)

            generation.on('exit', function(exitCode) {
                clearTimeout(generationTimeout)

                if (exitCode !== 0) {
                    return reject(reportError)
                }

                console.log('Allure report successfully generated')
                resolve()
            })
        })
    }

However, right away I noticed that the call to allure wouldn’t work because it’s not defined, and I’m unable to define it / import it outside a module. In other words, how do I import it into a config file?

Compared to Golem, Golem is much easier.. in fact you don’t do any setup at all. Reports are built in. The reports are clean and useful.

Conclusion

I might keep using the framework… only to practice my JS skills. Overall I found it lacks a lot of features built in and turned on in frameworks like Golem, by default.