Maintainer Info

Semantic PR title types

See here: https://github.com/commitizen/conventional-commit-types/blob/master/index.json

Issue filters:

• Remove work item like issues from list

Setup Nix with flakes support

1. Install
2. Enable flakes:
   • Permanently (recommended):

     mkdir -p ~/.config/nix
     echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

   • Or, per command. Add: --extra-experimental-features nix-command --extra-experimental-features flakes

Clang Tidy

Clang Tidy can be run by:

mkdir build
cd build
cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=On
cd ..
clang-tidy -p build --checks=* <file_here>

Note: --checks=* produces way too many irrelevant warnings, see below for a recommended list.

This command line works pretty well:

clang-tidy -p build --checks=-*,readability-*,modernize-*,performance-*,-modernize-use-trailing-return-type,-readability-uppercase-literal-suffix <file_here>

GDB - Pretty printing

The pretty printing script allows GDB to visualize types that it cannot by default, such as v_array. This works both with command line and using GDB in VSCode. The same thing is done in Visual Studio on Windows using the natvis file, which is included in the project and automatically loaded.

In order to use the GDB pretty printers, they need to be loaded. This can either be done automatically when GDB is started or each time.

To load the script into a running gdb session you can run:

(gdb) python exec(open('path/to/vowpalwabbit/gdb_pretty_printers.py').read())

To automatically load it, create the file if it does not exist ~/.gdbinit, and add the script load into that file:

python exec(open('path/to/vowpalwabbit/gdb_pretty_printers.py').read())

If using VSCode, ensure the launch target contains the following properties:

MIMode": "gdb",
"setupCommands": [
    {
        "description": "Enable pretty-printing for gdb",
        "text": "-enable-pretty-printing",
        "ignoreFailures": true
    }
]

Python tools

Installation

pip3 install yapf mypy pylint --user

Tools

• yapf - Code formatter
  • Usage: python3 -m yapf -i python/vowpalwabbit/*.py
  • By default uses PEP8 style
  • Any other formatter can be used
• mypy - Static type checking
  • Usage: python3 -m mypy vowpalwabbit/*.py --ignore-missing-imports
    • --ignore-missing-imports is required as the native extension, `pylibvw, does not have type stubs.
  • mypy supports mixed dynamic and static typed Python, so the code can be annotated one function at a time
• pylint - Python code linting
  • Usage: python3 -m pylint vowpalwabbit/*.py

Obtaining new CMake version

Ubuntu

Instructions from: https://apt.kitware.com/

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common wget
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -

# 16.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ xenial main'

# 18.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'

# 20.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ focal main'

sudo apt-get update

# Optional, ensure keyring stays up to date as it is rotated.
sudo apt-get install kitware-archive-keyring
sudo apt-key --keyring /etc/apt/trusted.gpg del C1F34CDD40CD72DA

sudo apt-get install cmake

Windows

Download and install from: https://cmake.org/download/

Nix

Nix allows for declarative package management (among other things) and reproducible builds across systems. To obtain a standard environment to build vw in you can simply run nix-shell --pure in the same direct as the following file. This will drop you into a new shell with all of the dependencies installed.

with import <nixpkgs> { };
{
  mystdenv ? pkgs.stdenv,
  myboost ? pkgs.boost174
 }:
mystdenv.mkDerivation {
  name = "vw-build-nix-shell";
  buildInputs = [
    cmake
    myboost
    zlib
    ninja
    flatbuffers
  ];
}

To override the compiler version:

# GCC - standard version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).gccStdenv'
# GCC - specific version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).gcc8Stdenv'

# Clang - standard version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).clangStdenv'
# Clang - specific version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).llvmPackages_9.stdenv'

To specify which Boost version to use:

nix-shell --pure --arg myboost '(import <nixpkgs> {}).boost174'

Manual Benchmark CI

1. Go here: https://github.com/VowpalWabbit/vowpal_wabbit/actions/workflows/run_benchmarks_manual.yml
2. Click "Run workflow" and input the refs you wish to test.

Generating Linux release artifacts

export VW_REPO_DIR=...
docker pull ubuntu:16.04
docker run -it --rm -v $VW_REPO_DIR:/vw ubuntu:16.04 /bin/bash

Inside the docker container:

apt update
apt install -y g++ wget dpkg-dev

# Install CMake
# Look here: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Maintainer-Info#obtaining-new-cmake-version

# Install zlib
wget -O zlib.tar.gz 'https://zlib.net/fossils/zlib-1.2.8.tar.gz' \
   && tar xvzf zlib.tar.gz \
   && cd zlib-1.2.8 \
   && ./configure --static --archs=-fPIC \
   && make -j$(nproc) \
   && make install \
   && cd .. && rm -rf zlib*

# Install boost 
wget -O boost.tar.gz 'https://sourceforge.net/projects/boost/files/boost/1.70.0/boost_1_70_0.tar.gz/download' \
   && tar -xvzf boost.tar.gz \
   && mkdir boost_output \
   && cd boost_1_70_0 \
   && ./bootstrap.sh --prefix=/boost_output --with-libraries=program_options,system,thread,test,chrono,date_time,atomic \
   && ./bjam -j$(nproc) cxxflags=-fPIC cflags=-fPIC -a install \
   && cd .. && rm -rf boost_1_70_0 boost.tar.gz

cd /vw
mkdir build
cd build
cmake .. -DBUILD_SHARED_LIBS=Off -DSTATIC_LINK_VW_JAVA=On -DBUILD_TESTS=Off -DBUILD_JAVA=Off -DBUILD_PYTHON=Off
make all -j$(nproc)

cpack -G DEB 
cpack -G TGZ

Build with vendored dependencies

To use vendored dependencies the following can be used:

cmake -S . -B build -G Ninja \
    -DCMAKE_BUILD_TYPE:STRING="Release" \
    -DFMT_SYS_DEP:BOOL="OFF" \
    -DRAPIDJSON_SYS_DEP:BOOL="OFF" \
    -DSPDLOG_SYS_DEP:BOOL="OFF" \
    -DVW_BOOST_MATH_SYS_DEP:BOOL="OFF" \
    -DVW_GTEST_SYS_DEP:BOOL="OFF" \
    -DVW_ZLIB_SYS_DEP:BOOL="OFF" \
    -DBUILD_TESTING:BOOL="OFF"
cmake --build build