- Change analyzer
Change-Analyzer (CA in short) is a framework built utilizing ML techniques, leveraging OpenAI Gym library and LudwigAI framework. CA allows product teams to get feedback regarding their software product, aka SUT (System Under Test).
Essentially, is built around the following main Data features:
- Data Collection, done through Automated Exploratory Testing
- Data Reconstruction, done through Automated Regression Testing
- Data Analysis, done through Change Detection
- Data Validation, done through Verdict Generation
After installation and configuration (see sections below) Data Collection can be started. An agent is exploring the SUT without prior knowledge of it, recording testing sequence and state of the application. Once the execution is completed, the Data is saved in a csv file, available in a dedicated folder, from recordings. Currently, the Data consists of the following information regarding the steps of the executed sequence:
- Timestamp: an Integer with the execution time in epoch
- SequenceId: a String with a unique ID to identify the sequence
- PageSourceBefore: a String with the page source (before step execution)
- PageSourceAfter: a String with the page source (after step execution)
- ActionToPerform: a String with the action that will be performed next
- ActionImageBefore: a numpy.ndarray with the image of SUT (before step execution)
- ActionImageAfter: a numpy.ndarray with the image of SUT (after step execution)
Once several sequences exist, they can be used for Data Reconstruction. An agent is rerunning the specified existing sequence, recording a new state of the application. Data collected has the same structure as described in the above section.
In this phase, two sequences are analyzed. The sequences are the same, from steps perspective, but are executed against
different SUT versions (deltas).
A report is created, to showcase the differences between the two sequences, if any. The goal is to provide a visual tool
for change detection.
The report will contain the following info:
- The info regarding which is the expected sequence and which is the actual sequence
- A visual indicator for each step if it was considered passed or failed
- A description of the executed steps step
- A short info to support the step status
- Expected image of the SUT, from expected sequence
- Actual image of the SUT, from actual sequence
- Actual and expected images may contain highlighted elements, if they were detected to be part of a change
Using ML techniques, in this phase, the tool should be able to say if the detected changes are expected behavior of faults in the SUT. Data Validation will be cover in more details later, because currently is in planning phase.
- Download Selenium standalone server.
- Version 3.141.59 confirmed as working.
- Put .jar file to place where you can easily find it.
- FOR DESKTOP APP TESTING: Download Windows Application Driver.
- Version 1.2.1 confirmed as working.
- Install downloaded .msi file.
- FOR WEBSITE TESTING: Download Chrome driver.
- Version 94.0.4606.61 confirmed as working.
- Put binary file to place where you can easily find it.
- Windows logo -> developer settings -> Enable
- Official documentation if needed.
Requires Python 3.7 or newer installed and added to PATH variable.
- Run
install.batlocated in source code root directory.- This script will install necessary dependencies defined in setup.py.
If preferred, it is also possible to use a Conda environment.
- Open an Anaconda terminal and go to project's main folder
- Run
conda env create -f requirements.yml - The above command will create the conda environment
change_analyzerincluding all the needed dependencies - In order to use the Conda environment, you can either attach it to a project within your favorite IDE, or activate
it on command line with
conda activate change_analyzer
It's mandatory to tell change-analyzer how to execute testing. For this reason .ini configuration is needed. Create configure now before running the software.
- More details can be found in later part of this README.
Requires Java 8 installed and added to PATH variable.
- Start Selenium standalone server:
- Open CMD and change directory to Selenium directory (downloaded in step 1).
java -jar selenium-server-standalone-[version].jar- Version needs to be change accordingly.
- Minimize CMD and let it run on background.
- FOR DESKTOP APP TESTING: Start Windows Application Driver
- Run
WinAppDriver.exe- Default location C:\Program Files (x86)\Windows Application Driver
- Minimize CMD and let it run on background.
- Run
- FOR WEBSITE TESTING:
- Run
chromedriver.exe - Minimize CMD and let it run on background.
- Run
- Run the software:
- Open CMD and change directory to source code root.
- Activate virtual environment:
.venv\Scripts\activate - Generate new report:
ca-run --config [configuration filename].ini
- Results appear in recordings directory.
Requires two generated test results.
- Open CMD and change directory to source code root.
- Activate virtual environment:
.venv\Scripts\activate - Compare results:
ca-compare --sequence1_folder [Test folder 1] --sequence2_folder [Test folder 2]- Replace
[Test folder 1]with real results folder name (example 2021_10_05-14_15_00). - Test results are generated to recordings directory.
- Argument paths starts from recordings directory.
- Replace
It is used to explore a given application and generate new test results.
--config(mandatory) - Defines which .ini file to use. Currently, Windows and Web platforms are supported.--csv_folder(optional) - it triggers the Replay mode. Replay will run the sequence found within the csv_folder.--steps(optional) - it represents the number of total steps desired for the sequence to have (default is 10)--strategy(optional) - it defines the agent strategy, either random or rl (reinforcement learning)--model_dir(optional) - it specifies the name of the pretrained model to use from pretrained_models folder
Results can be found within recordings folder in a dedicated sub-folder.
1a. ca-run --config win.ini --steps 5 --strategy rl --model_dir model1
CA will run using the config file win.ini and will aim to have a sequence of 5 steps. It will also use reinforcement learning as strategy, with the pretrained model model1.
1b. ca-run --config web.ini --steps 15
CA will run using the config file web.ini and will aim to have a sequence of 15 steps. In this case, the strategy to get the next action is random.
It is used to compare two test results of the same test sequence, applied to two versions of the same SUT.
--sequence1_folder(optional) - recordings folder with the expected test results--sequence2_folder(optional) - recordings folder with the actual test results
Note: If both sequences are missing, CA will attempt to find the last two test results
Results can be found within sequence1_folder folder in a dedicated sub-folder.
2a. ca-compare --sequence1_folder 2022_08_26-12_14_34 --sequence2_folder 2022_08_26-12_24_45
CA will compare the test results from defined sequence1_folder (expected) and sequence2_folder (actual).
This project uses .ini files as configure files to change software's behavior. User must define own configuration files. Ini file must define following values:
[driver]
command_executor=
platform=
app=
Example .ini file using WinAppDriver and Windows platform:
[driver]
command_executor=http://127.0.0.1:4723
platform=win
app=/path/to/test/application.exe
Example .ini file using SeleniumDriver and Web platform:
[driver]
command_executor=http://127.0.0.1:4444/wd/hub
platform=web
url=https://learn.ivves.eu/
Port of command_executor needs to match the driver's (ex. WinAppDriver/chromedriver) port.
The package was developed by F-Secure Corporation in scope of IVVES project. This work was labelled by ITEA3 and funded by local authorities under grant agreement 'ITEA-2019-18022-IVVES'.