Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving the test suite #9

Open
Wenzel opened this issue May 30, 2017 · 1 comment
Open

Improving the test suite #9

Wenzel opened this issue May 30, 2017 · 1 comment

Comments

@Wenzel
Copy link
Member

Wenzel commented May 30, 2017

Our test suite can be improved.
The current design has some limitations:

Problem 1: API

To run a test, you have to use the VMTest class interface provided and run your test like the following:

        def enter_NtOpenKey(syscall):
            pass

        def enter_NtCreateKey(syscall):
            pass

        hooks = {
            'NtOpenKey': enter_NtOpenKey,
            'NtCreateKey': enter_NtCreateKey,
        }
        events, exec_time = self.vm_test.run(cdrom_iso, hooks=hooks)

This forces you to use callbacks with all the problems that goes with it.
For example it's impossible to maintain a state accross multiple callbacks.
-> What if i want to get some information from one callback and use in another one ?

The solution to this would be to let the developer use the Nitro API,and provide VMTest class just as a convenient wrapper.

Nitro API:

with Backend(domain, analyze_enabled) as backend:
        backend.nitro.set_traps(True)
        for event in backend.nitro.listen():
              syscall = backend.process_event(event)
              if syscall.name == 'NtOpenFile':
                  # test

Problem 2 : Configurability

test_nitro.py has been written just to test Windows_7_x64 and nothing more.

As a result, the vm name is hardcoded in the setUp:

class TestNitro(unittest.TestCase):

    def setUp(self):
        con = libvirt.open('qemu:///system')
        domain = con.lookupByName('nitro_win7x64')
        self.vm_test = VMTest(domain)
        self.cdrom = CDROM()
        # clean old test directory
        test_dir_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), self._testMethodName)
        shutil.rmtree(test_dir_path, ignore_errors=True)
        os.makedirs(test_dir_path, exist_ok=True)
        self.script_dir = os.path.dirname(os.path.realpath(__file__))
        # chdir into this directory for the test
        self.origin_wd = os.getcwd()
        os.chdir(test_dir_path)
        # create logging file handler
        self.f_handler = logging.FileHandler('test.log', mode='w')
        logging.getLogger().addHandler(self.f_handler)
        logging.info('Starting test at {}'.format(datetime.datetime.now()))

As we want to make tests on Windows 7 32 bits, as well as Windows 8, Windows 10 and for Linux, we have to find a may to make it configurable

Problem 3 : Split the tests

The test have only been designed for windows_7_x64 and as such, some of them will not run on a Windows_7_x86.
On the other hand, some tests are generic enough to be run accross all Windows OS, and even on Linux

We have to group the tests that are generic enough and split the other in separate test modules, that will be called according to the configuration specified (either command line or a config file)

@Wenzel
Copy link
Member Author

Wenzel commented May 31, 2017

This PR #10 solves the first problem, allowing the developer to control the Nitro loop instead of defining callbacks.

The VM name is still hardcoded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant