Use vcpkg to install Windows dependencies #98
Conversation
nbyavuz
changed the title
|
Having the list of the packages to be installed in two places(.cirrus.yml and windows_install_pg_deps.ps1) is bad. I tried to save the list at top env like: .cirrus.yml: env:
...
VCPKG_PACKAGES: gettext icu libxml2 libxslt lz4 openssl pkgconf python3 tcl zlib zstdbut powershell gave this error at: .\vcpkg.exe install --debug --binarysource=files,${Env:VCPKG_CACHE},readwrite ${Env:VCPKG_PACKAGES} --triplet=x64-windowsand the error message is: Do you have any idea why? |
|
For readline, there have been a couple of threads about that: https://www.postgresql.org/message-id/20230607185458.GA1334487%40rfd.leadboat.com One of the problems mentioned there is VA_ARGS_NARGS, but we already fixed that in 3f28bd73. It is interesting that those two threads mention two different workarounds for C1026. Seems like readline is pretty close to working on Windows, but you should be aware that more work and probably code changes are probably needed! |
Why do we need it in .cirrus.yml at all? Can't we just exclusively have it in some image generation script? |
I suspect what might be happening is that powershell ends up passing all the packages as a single argument, and that then can't be interpreted as a package name by vcpkg. |
nbyavuz
force-pushed
the
windows-deps-vcpkg
branch
from
41eefd1 to
5b200fe
Compare
Rebased. However, I think these unrelated things may be caused by me 😟. |
nbyavuz
force-pushed
the
windows-deps-vcpkg
branch
3 times, most recently
from
4a7a3e2 to
0ed122d
Compare
nbyavuz
force-pushed
the
windows-deps-vcpkg
branch
from
0ed122d to
79a3d48
Compare
|
I updated the PR. Now, the PR does not remove anything. It installs packages by vcpkg and to use these packages: 1- Set PKG_CONFIG_PATH to 'c:\vcpkg\installed\x64-windows\debug\lib\pkgconfig' 2- Remove extra_[lib|include]_dirs for the openssl, otherwise there will be a conflict. If PKG_CONFIG_PATH is not set, it works as it did before. |
nbyavuz
force-pushed
the
windows-deps-vcpkg
branch
from
79a3d48 to
eda4c30
Compare
This task's aim is installing required packages to run vcpkg in next steps. - The steps to install required packages are moved from windows-ci to windows-base task. - Cleaning unnecessary files step is moved to windows_clean_unnecessary_files.ps1 script because it will be used in both tasks. - Now, windows-ci task depends on windows-base task. So, added another matrix to the .cirrus.yml file for '- depends_on'. - 'Clean up Windows base images' task is added to deleting windows-base images. These images will not be used afterwards, so it is safe to remove all of them. - Packer pre-commit check for windows-base task is added to Makefile.
'Build VCPKG Cache' task has two aims. - Firstly, generate vcpkg cache and upload it to Cirrus CI. So, it can be used in future CI runs - Secondly, export vcpkg files then zip it and upload as artifact. So, windows-ci task will download that artifact and will use that instead of doing vcpkg install again. To achieve above: - 'Build VCPKG Cache' task and 'windows_install_packages_via_vcpkg' script is created. This task runs the script and generates cache and vcpkg files. Then them to Cirrus CI. In this commit, this task does nothing. It is just for better visibility.
In this commit, all of the windows-base, build vcpkg cache and
windows-ci tasks are synced to install packages via vcpkg in end Windows
ci image. The end image includes the dependencies that are installed by
both vcpkg and installers.
To use dependencies that are installed by vcpkg:
1- Remove extra_[lib|include]_dirs for the openssl, otherwise there will
be a conflict.
2- PG_DEPS_['PATH', 'PKG_CONFIG', 'PKG_CONFIG_PATH',
'EXTRA_INCLUDE_DIRS', 'EXTRA_LIB_DIRS'] are exported as global env
variablles. Use them accordingly, for example:
* PATH: ${PG_DEPS_PATH}${PATH}
* PKG_CONFIG: ${PG_DEPS_PKG_CONFIG}
* PKG_CONFIG_PATH: ${PG_DEPS_PKG_CONFIG_PATH}
* -Dextra_include_dirs=%PG_DEPS_EXTRA_INCLUDE_DIRS%
* -Dextra_lib_dirs=%PG_DEPS_EXTRA_LIB_DIRS%
Summary of the code changes:
- windows_install_packages_via_vcpkg.ps1 script is modified to be called
from both windows.pkr.hcl and build vcpkg cache task.
- windows-ci task depends on build-vcpkg-cache task now.
nbyavuz
force-pushed
the
windows-deps-vcpkg
branch
from
eda4c30 to
7b94dc7
Compare
|
PR is updated:
Here is the example CI Run: https://cirrus-ci.com/task/4961222515228672, it is based on Andres' ci-use-vcpkg branch. The only change is last commit, it uses exported env variables from 4). |
Now
Windows - Server 2019, VS 2019 - Meson & ninjatask able to findicu, libxml, libxslt, lz4, openssl, plperl, zlib and zstd.There are 4 steps to install packages via vcpkg:
1- Create a base Windows image which has dependencies to run vcpkg.
2- Install packages via vcpkg and save & upload the vcpkg cache. Reuse that cache on the next install.
3- Create a Windows image from the base image (step #1). While installing packages via vcpkg, use cache from step #2.
4- Delete Windows base image.
Example run with vcpkg image: https://cirrus-ci.com/build/4816085688516608
Important notes:
After this PR is merged, extra_*_dirs needs to be removed from Windows task in Postgres.
Please merge freebsd: Update to 13.3 #97 first.
Some errors:
tcl is not working for now and I did not start to debug it yet. I wanted to discuss the PR first since the differences are quite big.
readline: vcpkg is able to install package but build step on Postgres fails.
libsystemd: can not be installed via vcpkg.
libuuid: can not be installed via vcpkg.