Software program Testing: What Generative AI Can—and Cannot—Do

Is chatgpt coming to do automation engineers jobs? The brief reply is “Possibly, however in all probability not.”

The lengthy reply entails understanding precisely what ChatGPT and different generative-AI instruments can and can’t do properly within the context of software program testing and take a look at automation. As seen in lots of different domains, ChatGPT can do wonderful issues that appear like a risk to conventional engineering duties. However there are additionally many issues it could’t do—or no less than do properly. Automation engineers who need to use generative AI to do their jobs higher than exchange them with AI want to grasp the distinction.

To supply steering, this text will stroll by way of the capabilities and limitations of ChatGPT as a software for accelerating take a look at automation workflows, the place it shines, and the place it requires domain-specific experience.

AI and take a look at automation, pre-ChatGPT

The concept of ​​making use of AI to software program testing shouldn’t be new. For years, numerous software program testing and deployment platforms have provided AI-powered “bots” (Robo (A preferred instance is Google Firebase) that may navigate by way of functions, robotically resolve what to check, after which run checks.

These bots assist automation engineers outline take a look at instances and run checks. However they’ve apparent drawbacks, corresponding to the issue of constructing firewalls previous functions and the tendency to get slowed down in infinite loops when testing.

For that cause, automation engineers have traditionally considered AI as a software that may cut back a number of the labor and technicality required to carry out primary testing—however not as something that may exchange human testers.

How good is ChatGPT at software program take a look at automation?

Generative-AI instruments like ChatGPT symbolize a brand new breed of options that provide extra flexibility than conventional AI instruments. They’re additionally straightforward to execute utilizing pure language queries, which makes them extra handy than many typical options.

Does this imply ChatGPT can excel the place conventional software program testing instruments fall brief? To search out out, let’s undergo a real-world situation you would possibly encounter as an automation engineer and see how properly ChatGPT handles it.

Think about you need to take a look at the UI for an internet software. If it is a browser-based app, you need to use ChatGPT to generate the automated script with the next question.

Generate a Selenium script that visits kobiton.com, selects the “Platform” menu choice, after which clicks on “Actual System Check”.

ChatGPT responds with the script I bought from it:

from selenium import webdriver 
from selenium.webdriver.widespread.by import By 
from selenium.webdriver.help.ui import WebDriverWait 
from selenium.webdriver.help import expected_conditions as EC 
 
# Arrange Chrome webdriver and navigate to kobiton.com 
driver = webdriver.Chrome() 
driver.get("https://kobiton.com/") 
 
# Look forward to the "Platform" menu choice to be clickable and click on on it 
platform_menu = WebDriverWait(driver, 10).till( 
    EC.element_to_be_clickable((By.XPATH, "//a[text()='Platform']")) 
) 
platform_menu.click on() 
 
# Look forward to the "Actual System Testing" choice to be clickable and click on on it 
real_device_testing = WebDriverWait(driver, 10).till( 
    EC.element_to_be_clickable((By.XPATH, "//a[text()='Real Device Testing']")) 
) 
real_device_testing.click on() 
 
# Shut the browser 
driver.give up()
This script comes near testing what the query requested for, however there are some points. The massive factor is that “Platform” is not truly a clickable ingredient on kobiton.com (though you need to hover over it to get the “Actual System Check” hyperlink). ChatGPT couldn’t detect this truth as a result of it analyzed kobiton.com in a easy approach.

A human automation engineer, nevertheless, will simply acknowledge this concern, and the Selenium script might be modified as wanted. Due to this fact, on this case, Generative AI is ready to do maybe 80% of the work essential to resolve a test-automatic take a look at; An automation engineer with domain-specific data ought to do the remaining.

ChatGPT and cellular testing

For instance, think about that as an alternative of testing the UI of a web site, we need to take a look at the UI of a cellular app operating on a particular system. That is the place issues get actually tough for generative AI.

As a result of ChatGPT would not have a fleet of cellular units, it could’t analyze cellular functions to write down take a look at instances the best way it does for web sites. In different phrases, there isn’t a strategy to inform chatgpt to “set up my app on Galaxy S23 and take a look at login display screen”. It is a largely handbook process, with the assistance of software program testing platforms that present automation engineers entry to cellular units.

Ideally, there are methods that generative AI may help engineers on this state of affairs. For instance, it could evaluation the automated scripts you write and counsel further take a look at instances you could have neglected—corresponding to testing situations the place customers go away passwords clean (reasonably than simply testing when customers enter the fallacious password).

However on this case, ChatGPT can deal with solely a small share of the work required to fulfill software-testing necessities. A lot of the trouble should come from human test-automation engineers, who’ve domain-specific experience and entry to the testing infrastructure that ChatGPT lacks.

Abstract

Generative AI is evolving quickly, and it is unlikely that it is going to be capable of work previous the constraints we have outlined above. Somebody might discover a strategy to give ChatGPT entry to cellular units, for instance, to generate automated scripts for cellular apps (and never only for web sites).

However even then, it is onerous to think about a world the place instruments like ChatGPT can deal with all software program take a look at automation in an end-to-end vogue. There’ll nearly all the time be loopholes and oversights that people should tackle. Automation engineers who need to excel in an AI-centric world should focus their energies on doing what AI cannot, to automate easy and tedious take a look at automation.