The Reason Behind my own Framework
I used to utilize a pre-made framework for web automation. I thought at the time, that such a framework would take the load off maintaining Web Automation, so I could focus more on writing tests, and creating other tools and dashboards for my company. The chosen framework I was using was Golem. Golem had everything I could want, a web based IDE, scripts written in Python and a web ui front end for reporting and maintenance tasks. Sadly Golem suddenly went into archival and was no longer maintained.
One option was to switch from Golem to some other pre-built framework. Most frameworks are lacking in reporting. At best generating a single use report for the last test. I preferred a database driven reporting system, so I could monitor the delta events (more fails) between MASTER and the current PULL REQUEST in test. It occurred to me that I could take a few evenings and weekends and create my own automation framework.
Framework Musthaves
There are certain needs that I require for an automation framework. Having written many a web based tool, I knew I wanted to collect data on each automation run. A database driven tool, would need middleware API to pull from the database and output JSON. A frontend UI would consume the JSON and present it in a meaningful manner.
Languages
Language-wise I’m agnostic. I write in Grails/Groovy often, but decided to use something more friendly to test creation. For this reason I opted for Python.
For the front end I decided on using Vue 3 as I’m familiar with it and have a preference for it.
Test Execution Options
While tests could be run locally (to debug problems), in the end they will need to run from a central system. That central system could be a 3rd party (Browser Stack or Lambda Test). A central system could also be an in-house grid-like system running tests in parallel using something like Selenium Grid or Selenoid.
Where I work we already have a relationship with LambdaTest and they have a great selection of browsers, browser versions and operating systems to run tests against (even mobile). The downside to LambdaTest is the parallel test limitation and speed of test execution.
Equally so there are downsides to Selenoid and Selenium Grid. With Selenium Grid, it feels slow (to me) and the maintenance on various nodes / vm’s can be time consuming. Selenoid is FAST. Written in Go, it’s incredibly fast. The downside of Selenoid is that it only runs specific browsers and versions (Chrome, Firefox and Opera and their latest versions).
My decision was to build a switch in the framework where I could determine the executor. For a rapid response to functional regression, I want to run against a local Selenoid server. However, for compatibility testing, I would prefer to use LambdaTest’s framework – where I could run other browsers on OSX, Windows and mobile devices. Finally, in debug mode I would want to run tests locally off my laptop.
Reporting
I want a UI that shows what the last run results are. So a web app displaying PASS and FAIL for each test. While this could be written in a variety of ways, I preferred to separate the frontend concerns from the backend. Using a framework like Angular or Vue a front end UI could be generated rapidly.
Not only does the UI reflect test results, but the above screenshot shows a UI that allows filtering and sorting of the columns. I could filter on all tests for Logging In to see each results across different browsers, environments and branches.
As I also built a monitoring system of the QA environments using the ELK stack (described in detail on my security blog FFE4.org. Linking to this monitoring system in the Navigation I can one-click monitor the SQL slow log, memory, CPU and compare against different automation runs.
Page Object Model
Page objects allow good coverage (every item on each page) as well as making tests easier to construct and maintain in the long run. DRY principles are also great. The more code that can be rolled up into a utility class, or central method, the easier it will be to correct mistakes or update the code. Rather than chasing each method in a separate class, one method called into each class makes a lot more sense.
Getting Started
My first step was to create the Python automation test framework. Using Page Factory I setup the python project into two folders/packages: Pages and Tests. But first we need to install page factory, which is done by:
import selenium-page-factoryPage Example
Let’s say we have a Login page. It could look like this:
from seleniumpagefactory.Pagefactory import PageFactory
from selenium.webdriver.common.by import By
class Loginpage(PageFactory):
def __init__(self, driver):
self.driver = driver
self.util = Utilities(driver)
locators = {
'username': ('CSS', "#username"),
'password': ('CSS', '#password'),
'login_button': ('CSS', '#authSubmit2'),
'reset': ('XPATH', '//*[@id="login-form"]/div[3]/a'),
'forgot': ('CSS', '#forgotEmailAddress'),
'forgot_pass_submit': ('CSS', '#btnForgotPassword')
}
def enter_username(self, username):
self.util.wait_until_css_displayed('#username')
self.username.send_keys(username)
def enter_password(self, password):
self.password.send_keys(password)
def click_login(self):
self.login_button.click()The Class definition inherits the PageFactory. The driver is not invoked here, but created in the test, which I’ll get to in a bit.
Notice the Locator’s object. Each element on the page is defined in the object. Notice we can define XPATH and CSS values.
Methods are the page tests, like clicking a button called login. This is handled by simply using
self.login_button.click()The login_buttin is defined in the locator object. Appending the Selenium click() action performs the action.
Test Example
The correlating tests are listed in a Class in the tests folder/package. Here’s an example of a Login test Class:
from pages.login_page import Loginpage
from pages.dashboard_page import Dashboardpage
from webdriver_update import assign_driver
class LoginTest:
def __init__(self, browser, qa, remote, project, branch, category, username, password, platform, version, geoLocation):
self.browser = browser
self.qa = qa
self.remote = remote
self.project = project
self.branch = branch
self.category = category
self.username = username
self.password = password
self.platform = platform
self.version = version
self.geo = geoLocation
def test_setup(self, test_name):
driver = assign_driver(self.browser, self.remote, self.project, self.category, test_name=test_name,platform=self.platform, version=self.version, geoLocation=self.geo)
environment = env[self.qa]
driver.get('http://' + environment + '.<your site>.com')
return driver
def test_successful_login(self):
driver = self.test_setup("Successful Login")
homepage = Homepage(driver)
login_page = Loginpage(driver)
dashboard_page = Dashboardpage(driver)
homepage.click_login()
login_page.enter_username(self.username)
login_page.enter_password(self.password)
login_page.click_login()
dashboard_page.validate_url(env[self.qa])
driver.quit()There’s a lot going on here. First I import the LoginPage class. Initializing the class are various options that might be used in the tests. The driver gets setup here in the class. This is passed through each method, such as the test_successful_login method.
Several values are not defined here, such as the QA environment. I’m constructing that URL and passing in the front of the domain (my qa environment). My environment is an object listing them as named strings with the machine value itself, like “Q1”: “machine-1” which has the full url appended here in setup. This is important because environments change all the time. You may have 5 or 10 environments, each with different code to test. You want to pass in a value for the environment at Test setup.
Main Script
A main.py (or by whatever name you wish) sets up all the variables and values required for this.
from tests.login_test import LoginTest
# Test Variables
BROWSER = 'chrome'
VERSION = '111' # Browser version
PLATFORM = 'Windows 11'
QA_ENVIRONMENT = '<your qa environemnt>'
REMOTE = "selenoid"
PROJECT = '<your project name>'
BRANCH = '<branch name or number>'
CATEGORY = QA_ENVIRONMENT.upper() + ': ' + BRANCH + str(test_run_time)
USERNAME = '<username for logging in>'
PASSWORD = '<password used for logging in>'
GEO = 'US' # LambdaTest can run against different geolocations.
These are the constants for setting up the tests.
# Loading page definitions
login_tests = LoginTest(BROWSER, QA_ENVIRONMENT, REMOTE, PROJECT, BRANCH, CATEGORY, USERNAME, PASSWORD, PLATFORM,
VERSION, geoLocation=GEO)
if __name__ == '__main__':
# List of tests to run
# ToDo: Use a DB to store tests to iterate over
tests = [
[login_tests.test_successful_login, 'Login Page - Login']
]
# total tests
print('Total Tests to Run: ' + str(len(tests)))
# Multithreading / parallelism
# Turn-on 4 worker threads.
Thread(target=worker, daemon=True).start()
Thread(target=worker, daemon=True).start()
Thread(target=worker, daemon=True).start()
Thread(target=worker, daemon=True).start()
# iterate over tests, sending to queue pool
for test in tests:
q.put(test)
# Block until all tasks are done.
q.join()
print('All work completed')In the example above I’m instantiating the LoginTest class (passing in relevant data) and assigning to login_tests. Inside the main I have a ToDo here to turn this into a pull from a database, but right in this example it iterates over a collection of tests (inside an array). You could use a map or a database call. What’s happening here is I’m looping through the array, calling each test.
As for results, you can print out the result through some mechanism like:
def worker():
while True:
item = q.get()
print(f'Working on {item[1]}')
report_results(item[0], item[1])
print(f'Finished {item[1]}')
q.task_done()
# Report generator
def mysql_connection(test, test_pass):
try:
mydb = mysql.connector.connect(
host="<db location>",
user="<db user>",
password="<password>",
database="test_results"
)
except mysql.connector.Error as e:
print(e)
mycursor = mydb.cursor()
sql = "INSERT INTO selenoid_results (run, run_key, branch, environment, test_name, browser, platform, geo, pass) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"
val = (test_run_time, test_key, BRANCH, QA_ENVIRONMENT.upper(), test, BROWSER.upper(), PLATFORM, GEO, test_pass)
mycursor.execute(sql, val)
mydb.commit()
print(mycursor.rowcount, "record inserted.")
def report_results(function, test):
try:
function()
results.__setitem__(test, 'Pass')
test_pass = True
print(f"PASS {QA_ENVIRONMENT}: {test} - {BRANCH}: {BROWSER.upper()} on {PLATFORM}")
except Exception as e:
print(e)
results.__setitem__(test, 'Fail')
test_pass = False
print(f"\nFAIL {QA_ENVIRONMENT}: {test} - {BRANCH}: {BROWSER.upper()} on {PLATFORM}\n")
if REMOTE is not False:
mysql_connection(test, test_pass)Here’s how I’m reporting results… In the report_results method, I’m outputting pass or fail per test, but I’m also taking two params. The first, test, is the name of the test from the data object (my array). I take the name of that function (function) and append () to it, this executes the name. That way I can take the string test_successful_login and turn it into a method.
The report method is called from a worker method… this how I’m threading the tests to run in parallel… In the main method I have 4 worker threads created and the worker method is what calls the reporter method.
The Worker method is calling report_results(item[0], item[1]) which is the array, first element of the array and the 2nd. The first element is the name of the method (also the test name). Back in the report function that method name is pulled in through the parameter, and then modified with () to execute it as the method. In this way the test name is passed around to invoke the method of the same name.
Right now I have one test in the array, if I had 50, it would process 4 at a time. Well it would send 4 at a time, the executor (LambdaTest, Selenoid, Selenium Grid) would take those processes and farm them to their own grid system.
Database
I’m using Mysql here, but you could use any db. To use this in Python I’m using the mysql-connector-python library. A pip install of that will pull the library down and importing it is done via import mysql.connector
The database schema for me has a database table housing columns for these columns and their respective data types:
run: varchar
run_key: int
branch: varchar
environment: varchar
test_name: varchar
browser: varchar
platform: varchar
geo: varchar
pass: BooleanMy database insert is handled via the mysql_ connection method. This is invoked after the evaluation of a test’s pass or fail status. We can do some other nifty things like pass a link to a video capture done in Selenoid into a column like “video.” If you collect error output, that too can be stored in a column, like an “exception” column.
Executor
I can use local browsers on my computer, the Selenoid framework at my job or LambdaTest as a 3rd party. This is accomplished wherever you define the setup for the the web driver. For me I do this in a separate file.
def assign_driver(browser='Chrome', remote=True, project='My Project', category='Customer App',
test_name='Generic Testing of Feature', platform='Windows 11', version='latest', geoLocation='US'):
if remote == "lambdatest":
desired_caps = {
'LT:Options': {
"user": user_name,
"accessKey": app_key,
"project": project,
"build": category,
"name": test_name,
"platformName": platform,
"selenium_version": "4.8.0",
"geoLocation": geoLocation
},
"browserName": browser,
"browserVersion": version,
}
remote_url="<remote host like Browser Stack or Lambdatest>/wd/hub"
driver = webdriver.Remote(
command_executor=remote_url,
desired_capabilities=desired_caps
)
return driver
elif remote == 'selenoid':
desired_caps = {
"browserName": browser,
"browserVersion": version,
"selenoid:options": {
"enableVideo": True,
"name": test_name,
"screenResolution": "1280x1024"
}
}
remote_url = "<server where selenoid is installed>:4444/wd/hub"
driver = webdriver.Remote(
command_executor=remote_url,
desired_capabilities=desired_caps
)
return driverAbove is the code I call to invoke the web driver with it’s capabilities. In this snippet example if the remote value is “lambdatest” it sends the tests to LambdaTest, of “selenoid” is passed in it sends to my Selenoid server.
I won’t detail installing / setting up Selenoid as they cover it very detailed in their official docs.
This remote value is set in the main.py file that launches the tests – it’s one of the constants that needs to be set with a value.
Selenoid can also be configured with Video Capabilities!
The path to the video for each test can be passed to your database/store for reference.
Middleware API
We need something to talk to the database and output the results of a query for the frontend to render. This could be done with the existing Python Project, or you could generate a middleware app in any language or framework to accomplish this (Spring Boot, Grails, Rails, Flask).
I’m going to use Flask, to stick to the Python concept. This code is pretty simple. All this does is connect to the database, and perform a SELECT query, then pass it back as JSON:
import flask
from flask import jsonify
from flask_cors import CORS
import mysql.connector
from datetime import datetime, timedelta
app = flask.Flask(__name__)
CORS(app)
app.config["DEBUG"] = True
last_24hr = datetime.today() - timedelta(days=1)
time_stamp = last_24hr.strftime('%Y-%m-%d %H:%M:%S')
@app.route('/test_results', methods=['GET'])
def home():
try:
mydb = mysql.connector.connect(
host="<database host name or ip>",
user="<database user>",
password="<database password>",
database="<database name>"
)
except mysql.connector.Error as e:
print(e)
mycursor = mydb.cursor(dictionary=True)
sql = (f"""SELECT branch, browser, environment, geo, pass, platform, run, run_key, test_name FROM test_results.selenoid_results WHERE run > '{time_stamp}' ORDER BY run DESC""")
mycursor.execute(sql)
results = mycursor.fetchall()
return jsonify(results)
if __name__ == "__main__":
app.run(host='0.0.0.0', debug=True)For the above I used Flask, so flask has be installed via PIP. I also installed flask_cors, due to various issues using a browser to access the app. This is invoked after the app setup at the top, using CORS(app).
Flask is defined to use a route of /test_results with method GET. Under that I have the database login/setup. Then the database select statement is made, where results are queried from within the past 24hr.
Note the cursor call is using dictionary=True.
At the end we return jsonify(results) so the database results are returned as JSON.
People may raise an eyebrow for the app.run(host=’0.0.0.0′, debut=True) – this is used because it’s not a production app. This is deployed probably in a QA environment somewhere and it’s used for simple internal processes. Having host=’0.0.0.0′ tells the app to take connections from anywhere. In production you wouldn’t do this, but again, this is just an example for a QA focused application (somewhere behind a firewall).
Once setup, hitting the <host ip or name>:<port>/test_results in a browser will return JSON for the query specified. Now we need to setup the UI.
Running the API Middleware
We need this to keep running. The best practice here is to use a webserver like Tomcat, NGINX, or Apache. You can look up tutorials on that. What I did for a quick fix is to run the Python server as a Linux service:
[Unit]
Description=Flask Server
After=multi-user.target
[Service]
Type=simple
Restart=always
ExecStart=/usr/bin/python3 /home/bwarner/flask_server.py
[Install]
WantedBy=multi-user.targetEditing/creating a new file called /etc/systemd/system/<something>.service is how you would create this file on Ubuntu or Debian. Once setup you would enable it using:
sudo systemctl enable <yourfile>.service
Then run sudo systemctl start <yourfile>.service
Double check the server is running!
Frontend UI
Take a snippet of JSON output from the database we configured and use it for a mock up. Using a web framework like Angular, Vue, React or an MVC app (Grails, Rails, etc), will call the API endpoint in the middleware app.
I repurposed some code I wrote last year for a web-based spreadsheet. This uses a framework called AG Grid. Grid here is not about selenium Grid, but about the spreadsheet layout.
I really don’t want to get into all the complexities of AG Grid, but will point mostly to their online documentation:
<template>
<h2 style="font-family:'Trebuchet MS', 'Lucida Sans Unicode', 'Lucida Grande', 'Lucida Sans', Arial, sans-serif;color:white;padding-left: 2px;font-size: 1em;">Selenoid Test Results (over last 24hr)</h2>
<ag-grid-vue
class="ag-theme-alpine"
style="width: 100%;height: 1800px"
:columnDefs="columnDefs.value"
:paginationAutoPageSize="true"
:pagination="true"
:rowData="rowData.value"
:defaultColDef="defaultColDef"
rowSelection="multiple"
animateRows="true"
@cell-clicked="cellWasClicked"
@grid-ready="onGridReady"
>
</ag-grid-vue>
</template>
<script>
import "ag-grid-community/styles/ag-grid.css";
import "ag-grid-community/styles/ag-theme-alpine.css";
import "./assets/override.css"
import { AgGridVue } from "ag-grid-vue3";
import { reactive, onMounted, ref } from "vue";
export default {
name: "App",
components: {
AgGridVue
},
setup() {
const gridApi = ref(null); // Optional - for accessing Grid's API
// Obtain API from grid's onGridReady event
const onGridReady = (params) => {
gridApi.value = params.api;
};
const rowData = reactive({});
const failColor = {
"fail": params => params.value == 0,
"pass": params => params.value == 1
}
const columnDefs = reactive({
value: [
{ field: "run", width: 220, maxWidth: 300},
{ field: "run_key", headerName:"Batch#" , width: 120, maxWidth: 120},
{ field: "branch", headerName:"Branch#", width:100, maxWidth: 120},
{ field: "environment", headerName:"Env.", width:100, maxWidth: 120},
{ field: "test_name", headerName:"Test", width:350, maxWidth: 450},
{ field: "browser", headerName:"Browser", width:90, maxWidth: 120},
{ field: "platform", headerName:"OS", width:90, maxWidth: 120},
{ field: "pass", headerName:"Result", width:90, maxWidth: 120, cellClassRules: failColor, cellRenderer: function(params){ if (params.value == 1){return "PASS"}else{return "FAIL"}}},
],
});
// DefaultColDef sets props common to all Columns
const defaultColDef = {
sortable: true,
filter: true,
flex: 1,
resizable: true,
};
// Example load data from sever
onMounted(() => {
fetch("<your middleware server that returns json results from the db>")
.then((result) => result.json())
.then((remoteRowData) => (rowData.value = remoteRowData));
})
return {
onGridReady,
columnDefs,
rowData,
defaultColDef,
cellWasClicked: (event) => { // Example of consuming Grid Event
console.log("cell was clicked", event);
},
deselectRows: () =>{
gridApi.value.deselectAll()
},
};
},
};
</script>The above is the basic code I have for Vue 3 using AG Grid. Towards the bottom the onMounted method is fetching from the API server. It consumes the JSON returned is parsed and defined in the columnDefs method.
I have defined the width (in pixels) of each column, defined the headers and set the “run” column as sortable. The piece of code below is doing dynamic coloring and naming:
{ field: "pass", headerName:"Result", width:90, maxWidth: 120, cellClassRules: failColor, cellRenderer: function(params){ if (params.value == 1){return "PASS"}else{return "FAIL"}}}The “pass” field is what’s returned from my database. It’s a column holding the test as pass or fail, defined as a 1 or 0. 1 being Pass and 0 being Fail. This is why I have a cellClassRules: failColor which is this method:
const failColor = {
"fail": params => params.value == 0,
"pass": params => params.value == 1
}The above “fail” and “pass” are sent to the html and used as CSS values:
.pass {
background-color: rgb(9, 237, 32);
color: rgb(0, 0, 0);
font-weight: bold;
}
.fail {
background-color: rgb(237, 9, 9);
color: white;
font-weight: bold;
}This way the field will take the dynamic CSS element value based on the data returned from the database. If it’s a 0 (fail) it is assigned the css class name “fail” and that colors the field red. Likewise if 1 is returned from the database for the test, that means “pass” and “pass” has the color coding of Green.
Finally, back at the field definition I also transform the 0 and 1 to the words “pass” and “fail” using this bit of code for the field in the columnDef:
cellRenderer: function(params){ if (params.value == 1){return "PASS"}else{return "FAIL"}}The result is this:
Columns can be filtered for values (like Env filtered for Q1, or Q30), Result for all Fails or PASS’s, and Browser filtered for certain types. We could filter for Test by “Like ‘Login%'” all Login tests across different branches would be filtered. Very useful.
No responses yet