Handmade tests quarantining for Jenkins

Intro

If you found this article, then you have probably made the same mistake as I did: you chose Jenkins CI for running automated tests.

While Jenkins is a wonderful open source instrument for building your software with a lot of brilliant  plugins, it still lacks one mandatory feature for running automated tests: quarantine. I.e. you can not mark test as ‘temporary broken’ and skip it’s result not to ruin regression. Well, you can, but with hacks and pain.

So the best solution I could think of is a storing quarantine test information in Jira (or any similar tool, but exactly my company uses the one). Which makes sense storing information on problem, that makes test to fail and quarantine information to be stored in the same place.

Overview

The algorithm is the next:

  1. Consider that we have Jira ticket JIR-1 which makes test TEST1 failing.
  2. We put information, that JIR-1 blocks TEST1 into JIR-1 itself (I used labels for it, but nothing prevents you from using e.g. links or comments). So, we put label ‘TEST1_blocked’ to JIR-1 ticket (test is added to quarantine).
  3. During pre-test preparation of automated tests, we retrieve information about the tests are expected to be ignored.
  4. During testing itself we can use acquired information e.g. for skipping the test or changing the test result from ‘failed’ to ‘skipped’ with our custom reason.
  5. After the issue has solved, jira ticket is closed and our test automation stops to ignore test result (i.e. test is removed from quarantine).

 

How to retrieve quarantined tests info from Jira (in python).

The most convenient way to retrieve information from jira is using a library. Unfortunately, this requires access to jira api, which is something hard to receive if you work in a larger company. So I’ll describe here workaround that uses usual http requests (with requests library).

import requests

# Acquiring authenticated HTTP session to jira
login = "mylogin"
password = "mypassword"
jira_server = "https://my.jira.server.com/"
post_data = {"os_username": login,
 "os_password": password,
 "login": "LogIn"}
session = requests.Session()
session.post(url=jira_server + "login.jsp", data=post_data, verify=False

# Now we have authenticated session and can issue jira JQL request
# We are searching for issues which are not closed yet and have
# labels 'XXX_blocked' which indicates quarantined tests)
request = self.session.get(url=jira_server + "rest/api/2/search",
params={"jql": "status != Closed AND issueFunction in issueFieldMatch(\"labels\", \"\*_(blocked)\") "})
json = request.json()

After this you have a dictionary, that represents JSON with all information related to quarantined tests and issues:

Screenshot_20171220_140139

From this dictionary you can easily acquire at least names of tests to be skipped, at most – anything you have in Jira related to this issue (description, reporter, date of reporting, etc.)

Usage

Having the information above, you can control your tests results. E.g. you can apply decorator to your test cases. Decoration function will run tests from list of quarantined in try-except block and in case of failure the function can change test result to ‘skipped’ and add explanation with information taken from Jira (e.g. ‘this test was skipped because of JIR-1 issue. Issue is opened by <Reporter> at <Date of report> and still not solved’).

The rest is quite easy and up to you only and your fantasy. But it’s always better not to use Jenkins for running automated tests than implementing such a workarounds.

 

Advertisements

Managing your ESX virtual machines from Python

If you write your tests in python, your test machines are virtual ones under Vmware ESX hypervisor, then you probably want to manipulate those virtual test machines from your tests (e.g. power them on/off, create/apply snapshots, add/delete networking interfaces, etc.)

Guys from Vmware created API for you (well, any hypervisor has something similar it the one is not developed by junkie). API itself is a SOAP-based and you can find documentation here.

But if you are as lazy as me, you’d prefer to use python wrapper for it: pyvmomi.

It’s important: to use Vmware api for manipulating machines (i.e. not read-only) you have to buy some, at least the cheaper ESX license. Otherwise, you will be allowed to have R/O access to ESX hypervisor: check virtual machine state, info about guest OS, etc.

Let’s see how helloworld looks. For example, we’ll power off random Centos7 machine:

import ssl
from pyVim.connect import SmartConnect
from pyVmomi import vim

# Connecting to ESX
context = ssl._create_unverified_context() # if your ESX certificate is self-signed like my one
connection = SmartConnect(host=esx_host,
                          user=esx_user,
                          pwd=esx_password,
                          port=int(esx_port),
                          sslContext=context,
                          connectionPoolTimeout=-1)
# Getting the list of VirtualMachine objects
content = connection.content
vmList = content.viewManager.CreateContainerView(content.rootFolder,
                                                 [vim.VirtualMachine],
                                                 True).view
# Now you can operate over list of powerful 'vim.VirtualMachine' objects
# Let's search through them for Centos7 machine
for vm in vmList:
if vm.guest.guestId == "centos7_64Guest":
# And now we can manipulate the vm
vm.PowerOff()

That’s it. This is short and not robustness version of this example. Other examples can be found here.