Skip to content

Dev: Building on WSL & Windows

Jump to bottom
Alex Seaton edited this page Dec 4, 2024 · 28 revisions

Building using WSL

This is a quick guide for how to develop ArcticDB in a fresh WSL installation.

Install WSL

Create %USERPROFILE%/.wslconfig file with that specifies enough memory for the build, eg:

# Settings apply across all Linux distros running on WSL 2
[wsl2]

# Limits VM memory
memory=48GB

Install WSL https://learn.microsoft.com/en-us/windows/wsl/install WSL 2, Ubuntu 22.04

Fix for WSL Clock Sync Issue

The WSL clock seems to lose sync with the hardware clock. This thread describes the issue and offers a fix https://stackoverflow.com/questions/65086856/wsl2-clock-is-out-of-sync-with-windows

Setup for Linux Build (using WSL)

Open a WSL Ubuntu 22.04 shell.

# SSH key
ssh-keygen 
>>> copy ~/.ssh/id_rsa.pub key to your Github account

# Clone
mkdir ~/source && cd ~/source
git clone [email protected]:man-group/ArcticDB.git
cd ~/source/ArcticDB
git submodule update --init --recursive

# Install C++ toolchain
sudo apt update
sudo apt-get install build-essential gcc-11 cmake gdb

# Required shared libraries and tools
sudo apt-get install zip pkg-config flex bison libkrb5-dev libsasl2-dev libcurl4-openssl-dev

# Install Python and create venv
# AS the default python is 3.12 following step WILL NOT WORK (GRusev)
<~not relevant~> sudo apt-get install python3-dev python3 python3-pip python3-venv
# Use following commands:
# based on : https://docs.vultr.com/how-to-install-python-and-pip-on-ubuntu-24-04
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10
sudo apt install python3.10-dev
sudo apt install python3.10-venv

#Just in cse there is another python on the system we will make link to proper one for us
/usr/local/bin$ sudo ln -s /usr/bin/python3.10 /usr/local/bin/python

# Create python virtual environment
python -m venv ~/venvs/310

# Activate the venv
source ~/venvs/310/bin/activate

# Get all Python dependencies - very hacky, should be improved
# Remove `dataclasses` from `setup.cfg`
pip install wheel
pip install setuptools
python setup.py egg_info
# Remove `[Testing]` line and '[:python_version < "3.7"]' line from ./python/arcticdb.egg-info/requires.txt
pip install -r ./python/arcticdb.egg-info/requires.txt

# Install test dependencies
# For CLion in WSL users, please start CLion in WSL terminal
# Starting directly in Windows will run CLion in non-interactive mode,
# Which .bashrc will not be source and mess up the enviromental setting of below dependencies

# Install mongodb
# Install community edition
# Please follow https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/

# Install azurite
# 1. Install nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
source ~/.bashrc
# 2. Install npm using nvm
nvm install 16
# 3. Install azurite using npm
npm install -g azurite


# Install x-11 libs, needed for clion UI
sudo apt-get install libxi-dev libxtst6 libxrender1

# Install a browser (useful for clion to open links)
sudo apt-get install firefox

Set up CLion within WSL

sudo snap install clion --classic and run with clion &

Create `arcticdb/cpp/CMakeUserPresets.json with contents:

{
  "version": 3,
  "configurePresets": [
    {
      "name": "linux-gcc-debug",
      "cacheVariables": {
        "CMAKE_MAKE_PROGRAM": "make",
        "CMAKE_C_COMPILER": "gcc",
        "CMAKE_CXX_COMPILER": "g++"
      },
      "inherits": ["linux-debug"]
    },
    {
      "name": "linux-gcc-release",
      "cacheVariables": {
        "CMAKE_MAKE_PROGRAM": "make",
        "CMAKE_C_COMPILER": "gcc",
        "CMAKE_CXX_COMPILER": "g++"
      },
      "inherits": ["linux-release"]
    }
  ],
  "buildPresets": [
    {"name": "linux-debug-build", "configurePreset": "linux-gcc-debug", "targets": "arcticdb_ext", "jobs": <n> },
    {"name": "linux-release-build", "configurePreset": "linux-gcc-release", "targets": "arcticdb_ext", "jobs": <n> }
  ]
}

where <n> is a suitable number of cores to build on for your machine. 12 works well on the Man Group machines.

Choose the python interpreter

File -> Settings -> Build, Execution, Deployment -> Python interpreter -> Add interpreter and add the Python venv created earlier.

You can generate the protobuf Python stubs with:

python setup.py protoc --build-lib python

Right click cpp/CMakeLists.txt in the file explorer, and click Load CMake project. This will fail.

You will need to choose the correct cmake preset.

File -> Settings -> Build, Execution, Deployment -> CMake, disable the Debug project and enable the linux-gcc-debug - linux-debug-build preset. Reload the CMake project and build the arcticdb_ext target.

Create the symlink in the python folder

ln -s ../cpp/out/linux-gcc-debug-build/arcticdb/arcticdb_ext.cpython-310-x86_64-linux-gnu.so arcticdb_ext.so

Debug C++ from a pytest

Create a custom run configuration like the below. Consider saving it as a project file in your working copy. Then put a breakpoint in the C++ code and run the configuration in debug mode (select it, then Shift+F9).

image2023-8-17_14-15-40

Running VS Code on WSL

  1. Install VS Code outside of WSL
  2. After it is installed, within WSL run code . in the ArcticDB - this will download the VS Code Server on WSL
  3. To build, run the following command: ARCTIC_CMAKE_PRESET=linux-debug pip install -ve .
  • preset can also be linux-release
  • NOTE: check the version of gcc, it it 13, you will need to downgrade it to 10 or 11 otherwise some warnings will fail the build
  • IMPORTANT: if you get the following error or similar:
Could NOT find Python (missing: Development.Module)

try to delete your build directory(cpp/out) and rebuild

  1. To debug, you need to create a launch.json file in the .vscode folder and you can use the following example json:
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Launch",
            "type": "cppdbg",
            "request": "launch",
            "program": "/path/to/your/python",
            "args": [
                "python/test-arctic.py"
            ],
            "stopAtEntry": false,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "description": "Set Disassembly Flavor to Intel",
                    "text": "-gdb-set disassembly-flavor intel",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}
  1. Make sure to change program and args fields to suit your needs
  2. You can Debug with F5
  3. To setup python tests (execute and debug in vscode) do: invoke Control Palette Alt+Ctrl+P > Python: Configure Tests > Select 'python'

Building a python wheel for a specific python version

It's easiest to just download the wheel from the CI build. See how here. Still if you need/want to do it locally:

You'll need a python venv with the correct version (if you want a Python 3.10 wheel you can use the ~/venvs/310 setup earlier). If you e.g. want a 3.11 wheel you can set up the venv by:

# Install python 3.11
sudo apt-get install python3.11-dev python3.11 python3.11-venv
python3.11 -m venv ~/venvs/311

# Install the required packages in the venv (repeat the steps from above)

Then you can build the wheel with:

cd ~/source/ArcticDB
# Remove stale `CMakeCache.txt` if you're changing venv
rm cpp/out/linux-release-build/CMakeCache.txt

source ~/venvs/311/bin/activate
python setup.py bdist_wheel

Windows Setup

There are two types of windows builds setups provided via presets.

  • Full MSVS build setup which creates a Visual Studio project. Using the cl compiler and the MSBuild provided by the windows-vs-2022 preset. This is suitable for development and debugging with Visual Studio.
  • cl + Ninja builds. They don't require a full installation of Visual Studio and are run on the CI.

Make sure to use the same Visual Studio toolset as in cpp/custom-triplets/x64-windows-static-msvc.cmake. This is the only preset that we guarantee will compile successfully ArcticDB and its dependencies. To install the old 14.41 compiler launch the Visual Studio installer then,

Modify Individual components Search for "MSVC" Tick MSVC v143 ... x64/x86 ... v14.41-17.11

To use the custom triplet correct in CLion I also had to create cpp/CMakeUserPresets.json as follows:

{
  "version": 3,
  "configurePresets": [
    {
      "name": "windows1441",
      "hidden": true,
      "generator": "Ninja",
      "environment": { "cmakepreset_expected_host_system": "Windows" },
      "cacheVariables": {
        "CMAKE_C_COMPILER": "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/bin/Hostx64/x64/cl.exe",
        "CMAKE_CXX_COMPILER": "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/bin/Hostx64/x64/cl.exe",
        "VCPKG_OVERLAY_TRIPLETS": "C:/Users/Alex/source/ArcticDB/cpp/custom-triplets",
        "VCPKG_TARGET_TRIPLET": "x64-windows-static-msvc"
      }
    },
    {
      "name": "windows-cl-debug1441",
      "inherits": ["common_vcpkg", "windows1441"],
      "description": "Debug features enabled, but linked against release Python",
      "architecture": {
        "value": "x64",
        "strategy": "external"
      },
      "toolset": {
        "value": "version=14.41,host=x64",
        "strategy": "external"
      }
    },
    { "name": "windows-cl-release1441",       "inherits": "windows-cl-debug1441",       "cacheVariables": { "CMAKE_BUILD_TYPE": "Release" }}
  ],
  "buildPresets": [
    {"name": "windows-cl1441-debug",          "configurePreset": "windows-cl-debug1441",          "targets": "arcticdb_ext", "jobs": 14 },
    {"name": "windows-cl1441-release",        "configurePreset": "windows-cl-release1441",        "targets": "arcticdb_ext", "jobs": 14 }
  ]
}

Do not share the same files across Windows and WSL as the filesystem performance is rubbish.

Download and run the Visual Studio installer, select "Desktop development with C++" and click "Install".

Use the "Windows 64 Installer (64-bit)" link to install the Python versions you want to develop against from https://www.python.org/downloads/windows/. This guide will assume Python 3.10 installed in the default location.

Optional: Add Python to your PATH from Control Panel "Edit environment variables for your account" (e.g. C:\Users\<username>\AppData\Local\Programs\Python\Python310) above the .../WindowsApps variable (otherwise the Microsoft Store will open from the terminal).

Open PowerShell:

# SSH key
ssh-keygen.exe
>>> copy ~/.ssh/id_rsa.pub key to your Github account

# Clone
mkdir ~/source && cd ~/source
git clone [email protected]:man-group/ArcticDB.git
cd ~/source/ArcticDB
git submodule update --init --recursive

# Create venv - Python version can be changed in the obvious way. This step requires PowerShell to be running with admin privileges.
C:\Users\<username>\AppData\Local\Programs\Python\Python310\python.exe -m venv --symlinks ..\venvs\310

# This step requires PowerShell to be running with admin privileges
cd .\cpp\third_party\lmdbcxx\ & .\symlink_srcs.ps1

# Activate the venv
..\venvs\310\Scripts\Activate.ps1

# Get all Python dependencies - very hacky, should be improved
# Remove `dataclasses` from `setup.cfg`
pip install wheel
python.exe setup.py egg_info
# Remove `[Testing]` line from .\python\arcticdb.egg-info\requires.txt
pip install -r .\python\arcticdb.egg-info\requires.txt

# Generate the Python protobuf files
python.exe setup.py protoc --build-lib python

CLion Setup

Download and run the CLion installer.

Open the ArcticDB directory from the CLion launcher screen.

On Toolchains selection, select Visual Studio, set the architecture to amd64, and set it as the default by using the up arrow to move it above MinGW.

In cpp/CMakeLists.txt, add the lines:

set(Python_EXECUTABLE "C:/Users/<username>/AppData/Local/Programs/Python/Python310/python.exe")

immediately after the cmake_minimum_required line. Also in CMakePresets.json, update the relevant buildPresets array element:

{"name": "windows-cl-debug", "configurePreset": "windows-cl-debug", "targets": "arcticdb_ext", "jobs": <n> }

where <n> is a suitable number of cores to build on for your machine.

File -> Settings -> Build, Execution, Deployment -> Python interpreter -> Add interpreter and add the Python venv created earlier.

Right click cpp/CMakeLists.txt in the file explorer, and click Load CMake project. This will fail.

File -> Settings -> Build, Execution, Deployment -> CMake, disable the Debug project and enable the windows-cl-debug - windows-cl-debug preset. Reload the CMake project and build the arcticdb_ext target.

Run the following command in an admin PowerShell terminal for the Python tests to work :

# Modify 310 to other Python versions as appropriate
New-Item -Path .\python\arcticdb_ext.cp310-win_amd64.pyd -ItemType SymbolicLink -Value .\cpp\out\windows-cl-debug-build\arcticdb\arcticdb_ext.cp310-win_amd64.pyd

When building in debug mode I got errors like python312_d.lib not found when linking test_unit_arcticdb against Python. I also had to set our Cmake option,

ARCTICDB_MSVC_OMIT_RUNTIME_CHECKS=ON

To fix, I copied that file from the Python installation's libs folder to ${git_root}/cpp/out/windows-cl-debug-build.

Native windows build with VS 2022

Prerequisites

  1. Install git
  2. Install VS 2022. In the installer under Desktop & Mobile select "Desktop Development with C++". The defaults should work.
  3. Install CMake (check https://github.com/man-group/ArcticDB/blob/master/cpp/CMakeLists.txt for the minimal required version)
  4. Install Python. Note Prefer downloading and installing it through https://www.python.org/downloads/ rather than Microsoft Store. The latter has some know issues such as venv --symlink not working. Important when installing Python also install the debug symbols, the installer has a checkbox for this.

Setup git

Setup GitHub SSH Key

You can skip this if you are using another method to access GitHub (e.g. HTTPS + PAT) or if you have already setup SSH key for your account. Otherwise follow this GitHub tutorial.

Clone

Open GitBash and type

git clone [email protected]:man-group/ArcticDB.git
cd ArcticDB
git submodule update --init --recursive

Setup Python

  1. Open PowerShell as admin and type
# Create venv
C:\Users\<username>\AppData\Local\Programs\Python\Python310\python.exe -m venv --symlinks <path_to_venv>
  1. Open PowerShell as regular user and type
# Activate venv
<path_to_venv>/Scripts/Activate.ps1
# Get all Python dependencies - very hacky, should be improved
# Remove `dataclasses` from `setup.cfg`
pip install wheel
python.exe setup.py egg_info
# Remove `[Testing]` line from .\python\arcticdb.egg_info\requires.txt
pip install -r .\python\arcticdb.egg_info\requires.txt

# Generate the Python protobuf files
python.exe setup.py protoc --build-lib python

Setup VS 2022 project

The most straightforward way to setup VS 2022 build is to use the windows-vs-2022 preset.

Setup LMDB

This step has to be performed only once. Open PowerShell as admin and type:

cd <arctic_repo_location>/ArcticDB/cpp/third_party/lmdbcxx/ & ./symlinks_srcs.ps1

Generate VS 2022 project

# Activate venv
<path_to_venv>/Scripts\Activate.ps1
# Generate
cd <arctic_repo_location>/ArcticDB/cpp
cmake --preset windows-vs-2022 ./

This will create a project located in <arctic_repo_location>/ArcticDB/cpp/out/windows-vs-2022-build.

Note the preset enables /MT by default, you might need to lower the threads used by MSBuild in case of insufficient RAM.

Run tests with C++ breakpoints (MSVS)

After generating a project

  1. Go to View -> Solution Explorer. Right click on arcticdb_ext project and select Set as startup project.
  2. Right click on arcticdb_ext. Properties -> Debugging
    1. Command set the absolute path to python from your virtual environment
    2. Command Arguments set to -m pytest <absolute_path_to_tests>

Now running the code in Debug will be hitting breakpoints in code.

ArcticDB Wiki

Clone this wiki locally