Skip to content

Latest commit

 

History

History
481 lines (353 loc) · 15.2 KB

developer.adoc

File metadata and controls

481 lines (353 loc) · 15.2 KB
Raw

OpenSCAP Developer Manual

This part of documentation is meant to serve mainly to developers who want to contribute to OpenSCAP, help to fix bugs, or take an advantage of the OpenSCAP library and create own projects on top of it.

Table of Contents

1. Building OpenSCAP on Linux

If you want to build the libopenscap library and the oscap tool from the source code then follow these instructions:

  1. Get the source code

    Choose 1a or 1b depending on whether you want sources from a release tarball or the git repository.

    1. Use a release tarball:

      # replace ${version} with the desired version
wget https://github.com/OpenSCAP/openscap/releases/download/${version}/openscap-${version}.tar.gz
tar -xzpf openscap-${version}.tar.gz
cd openscap-${version}
mkdir -p build

      OR

    2. Use fresh sources from git repository.

      $ git clone --recurse-submodules https://github.com/OpenSCAP/openscap.git
$ cd openscap
      Note
      		 We include yaml-filter library as a git submodule. To get more information about using git submodules, read Git Tools - Submodules.

  2. Get the build dependencies

    To build the library you will also need to install the build dependencies.

    Build dependencies may vary depending on enabled features (by the cmake command). Some of the dependencies are optional, if they are not detected, openscap will be compiled without respective optional features.

    On RHEL 7 / Fedora 23 / CentOS 7, the command to install the build dependencies is:

    sudo yum install \
cmake dbus-devel GConf2-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel xmlsec1-devel xmlsec1-openssl-devel apt-devel

    On Fedora 24+, the command to install the build dependencies is:

    sudo yum install \
cmake dbus-devel GConf2-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python3-devel python3-dbusmock rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel xmlsec1-devel xmlsec1-openssl-devel apt-devel

    On RHEL 8 / CentOS 8, the command to install the build dependencies is:

    sudo yum install \
cmake dbus-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python36-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel xmlsec1-devel xmlsec1-openssl-devel apt-devel

    On Ubuntu 16.04, Debian 8 or Debian 9, the command to install the build dependencies is:

    sudo apt-get install -y cmake libdbus-1-dev libdbus-glib-1-dev libcurl4-openssl-dev \
libgcrypt20-dev libselinux1-dev libxslt1-dev libgconf2-dev libacl1-dev libblkid-dev \
libcap-dev libxml2-dev libldap2-dev libpcre3-dev python-dev swig libxml-parser-perl \
libxml-xpath-perl libperl-dev libbz2-dev librpm-dev g++ libapt-pkg-dev libyaml-dev \
libxmlsec1-dev libxmlsec1-openssl

    Since version 1.3.9 OpenSCAP can use either PCRE or PCRE2 library to handle regular expressions. The default behaviour for 1.3.x line of versions is to try and link with PCRE. In order to switxh the build system to PCRE2 one must configure CMake with '-DWITH_PCRE2=True' parameter. Dependencies: Debian/Ubuntu — pcre2-dev, Fedora/RHEL — pcre2-devel.

    When you have all the build dependencies installed you can build the library.

  3. Build the library

    Run the following commands to build the library:

    $ cd build/
$ cmake ../
$ make

    On Ubuntu 18.04 and potentially other distro, the python3 dist-packages path is wrong. If the following command:

    $ python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

    returns "/usr/local/lib/python3/dist-packages" instead of a path like "/usr/local/lib/python3.6/dist-packages", you must override this path, otherwise you will not be able to import openscap_api.

    $ cmake ../ -DPYTHON_SITE_PACKAGES_INSTALL_DIR=/usr/local/lib/python3.6/dist-packages

  4. Build the HTML documentation

    It is possible to generate a complete API documentation, User manual, Developer manual and contribute documents in HTML format.

    If you want to build the HTML documentation you will need to install Doxygen and AsciiDoc. To install AsciiDoc, you can run dnf install asciidoc. To install Doxygen, you can run dnf install doxygen.

    Run the following command to build the documentation:

    $ cmake -DENABLE_DOCS=TRUE ../
$ make docs

    The resulting documentation is located in build/docs directory in these subdirectories:

    • html - contains the API documentation generated by Doxygen

    • manual - contains OpenSCAP User Manual

    • developer - contains OpenSCAP Developer Manual

    • contribute - contains contribute documents

  5. Run the tests

    After building the library you might want to run library self-checks. To do that you need to have these additional packages installed:

    wget lua which procps-ng initscripts chkconfig sendmail bzip2 rpm-build strace

    On Ubuntu 18.04, also install:

    rpm-common

    It is also required to have sendmail service running on the system:

    $ systemctl start sendmail.service

    Now you can execute the following command to run library self-checks:

    $ ctest

    It’s also possible to use ctest to test any other oscap binary present in the system. You just have to set the path of the binary to the CUSTOM_OSCAP variable:

    $ export CUSTOM_OSCAP=/usr/bin/oscap; ctest

    Some tests that use the so-called offline mode of probes need to chroot during the test execution. Some of those probes use the chroot syscall, which an unprivileged process is not allowed to do. This is not a problem during the scanning itself, as oscap is usually scanning as root. However, we don’t want to run oscap as root during tests, as the whole test suite would have to use root privileges to clean up.

    Instead, build the oscap-chrootable target as superuser, or build oscap-chrootable-nocap first and then grant the capability manually. This target creates the binary that the test suite will use for some of those offline tests. In offline tests, use the set_offline_test_mode [chroot directory] and unset_offline_test_mode functions from the common test module - those will set variables in such way that the unquoted $OSCAP invocation will use the chroot-capable binary, or it will exit with an error code, aborting the test. Therefore, it is recommended to run

    $ sudo make oscap-chrootable

    Not every check tests the oscap tool, however, when the CUSTOM_OSCAP variable is set, only the checks which do are executed.

    To enable the MITRE tests, use the ENABLE_MITRE flag:

    $ cmake -DENABLE_MITRE=TRUE ..

    These test require specific features of the environment to function properly; most notably, a MTA needs to be listening on port 25. We suggest using our container mitre_tests to test MITRE functionality if possible:

    $ docker build --tag openscap_mitre_tests:latest -f Dockerfiles/mitre_tests . && docker run openscap_mitre_tests:latest

  6. Install

    Run the installation procedure by executing the following command:

    $ make install

2. Running oscap

It is important to use your compiled libopenscap.so library with your oscap tool. The easiest way how to achieve that without need to install libopenscap.so to the system path, is to use a shell script called oscap_wrapper or run in the OpenSCAP build directory.

$ cd build/
$ ./oscap_wrapper xccdf eval ... whatever
$ ./run valgrind utils/oscap xccdf eval ... whatever

The run script is generated at configure time by CMake and it sets the following environment variables:

  • LD_LIBRARY_PATH - path to libopenscap.so

  • OSCAP_SCHEMA_PATH - path to XCCDF, OVAL, CPE, …​ XSD schemas and schematrons (required for correct SCAP content validation)

  • OSCAP_XSLT_PATH - path to XSLT transformations. (required if you want to generate html documents from xml)

  • OSCAP_CPE_PATH - path to the OpenSCAP CPE dictionary.

The oscap_wrapper script is a convenience shortcut for run utils/oscap call.

3. Debugging

Developers and users who intend to help find and fix possible bugs in OpenSCAP or possible bugs in their security policies have these possibilities:

3.1. Verbose mode

The verbose mode provides user additional information about process of system scanning. The mode is useful for diagnostics of SCAP content evaluation and also for debugging. It produces a detailed report log with various messages. The mode is available for xccdf eval, oval eval, oval collect and oval analyse modules. There is no need to special compilation, the feature is available for all OpenSCAP users.

To turn the verbose mode on, run oscap with this option:

  • --verbose VERBOSITY_LEVEL - Turn on verbose mode at specified verbosity level.

The VERBOSITY_LEVEL can be one of:

  1. DEVEL - the most detailed information for developers and bug hunters

  2. INFO - reports content processing and system scanning

  3. WARNING - possible failures which OpenSCAP can recover from

  4. ERROR - shows only serious errors

The verbose messages will be written on standard error output (stderr). Optionally, you can write the log into a file using --verbose-log-file FILE.

This is an example describing how to run OpenSCAP in verbose mode:

$ oscap oval eval --results results.xml --verbose INFO --verbose-log-file log.txt oval.xml

Then see the log using eg.:

$ less log.txt

3.2. Debug mode

Debug mode is useful for programmers. You need to build OpenSCAP from source code with a custom configuration to enable the debug mode. Use this command:

$ cmake -DCMAKE_BUILD_TYPE=Debug .. && make

Debug mode provides:

  • debug symbols on and optimization off - you can use gdb, every process that was run.

  • assertions are evaluated.

3.2.1. Example

$ bash ./run gdb --args utils/oscap xccdf eval \
--profile hard --results xccdf-results.xml \
--oval-results my-favourite-xccdf-checklist.xml

The --oval-results option force oscap tool to generate OVAL Result file for each OVAL session used for evaluation. It’s also very useful for debugging!

3.3. Environment variables

There are few more environment variables that control oscap tool behaviour.

  • OSCAP_FULL_VALIDATION=1 - validate all exported documents (slower)

  • SEXP_VALIDATE_DISABLE=1 - do not validate SEXP expressions (faster)

  • OSCAP_PCRE_EXEC_RECURSION_LIMIT - override default recursion limit for match in pcre_exec/pcre2_match calls in textfilecontent(54) probes.

4. Generating of code coverage

Code coverage can be useful during writing of test or performance profiling. We could separate the process into five phases.

1) Get dependencies

# dnf install lcov

2) Run CMake & make

To allow code to generate statistics, we need to compile it with specific flags.

$ CFLAGS="--coverage -ftest-coverage -fprofile-arcs" LDFLAGS=-lgcov cmake -DCMAKE_BUILD_TYPE=Debug ../
$ make

3) Run code

In this phase we should run code. We can run it directly or via test suite.

$ bash ./run utils/oscap

4) Generate and browse results

$ lcov -t "OpenSCAP coverage" -o ./coverage.info -c -d .
$ genhtml -o ./coverage ./coverage.info
$ xdg-open ./coverage/index.html # open results in browser

5) Clean stats

Every run only modify our current statistics and not rewrite them completely. If we want to generate new statistics, we should remove the old ones.

$ lcov --directory ./ --zerocounters ; find ./ -name "*.gcno" | xargs rm
$ rm -rf ./coverage

5. Building OpenSCAP on Windows using Visual Studio

Prerequisites:

1) Get dependencies

We will use Vcpkg to download libraries that are required to build OpenSCAP.

Click on Start → Windows System → Command Prompt.

mkdir c:\devel
cd c:\devel
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install curl libxml2 libxslt bzip2 pcre pthreads getopt-win32
.\vcpkg integrate install

2) Get OpenSCAP

cd c:\devel
git clone -b main https://github.com/OpenSCAP/openscap.git

3) Generate Visual Studio Solution

cd openscap
cd build
cmake -D ENABLE_PYTHON3=FALSE -D CMAKE_TOOLCHAIN_FILE=c:/devel/vcpkg/scripts/buildsystems/vcpkg.cmake ..

4) Open in Visual Studio

  1. Launch Visual Studio

  2. Click on File → Open → Project/Solution…​

  3. Locate c:\devel\openscap\build\openscap.sln

5) Build

  1. Select build type (Debug, Release, …​) in the drop-down menu in the top panel.

  2. Click on Build → Build Solution.

Built binaries and their dependencies are now located in C:\devel\openscap\build\<BUILD_TYPE>\, eg. C:\devel\openscap\build\Debug\

6. Building OpenSCAP for Windows on a Linux box (cross-compilation)

Currently it is possible to cross-compile OpenSCAP for Windows only without probes. The resulting binary is not able to perform scanning. Instructions for cross-compiling OpenSCAP for Windows:

1) Install the cross-compiler & dependencies

Note
 mingw32-pthreads needs to be version 5.0 or greater. 
# yum install mingw32-gcc mingw32-binutils mingw32-libxml2 \
mingw32-libgcrypt mingw32-pthreads mingw32-libxslt \
mingw32-curl mingw32-pcre \
mingw32-filesystem mingw32-bzip2

2) Checkout the main branch of the OpenSCAP repository

$ git clone -b main https://github.com/openscap/openscap.git
$ cd openscap

3) Prepare the build

$ mkdir build-win32
$ cd build-win32
$ mingw32-cmake -D ENABLE_PYTHON3=FALSE -D ENABLE_PROBES=FALSE -D ENABLE_OSCAP_UTIL_DOCKER=FALSE ../

4) Build!

$ make

Resulting oscap.exe can be found in the utils/ directory.

If you would like to send us a patch fixing any Windows compiling issues, please consult the page about contributing to the OpenSCAP project.

7. OpenSCAP Reference Manual

For more information about OpenSCAP library, you can refer to this online reference manual: OpenSCAP reference manual. This manual is included in a release tarball and can be regenerated from project sources by Doxygen documentation system.