Skip to content

Commit

Permalink
cache as post action (i.e. at end of workflow) when setupOnly:true (#57)
Browse files Browse the repository at this point in the history 
* cache at end of workflow when using setupOnly:true
* update docs
  • Loading branch information
@lukka
lukka authored Nov 28, 2020
1 parent d950c94 commit 324f999
Show file tree
Hide file tree
Showing 11 changed files with 25,291 additions and 485 deletions.
125 changes: 53 additions & 72 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@

# [The **run-vcpkg** action for caching artifacts and using vcpkg on GitHub](https://github.com/marketplace/actions/run-vcpkg)

The **run-vcpkg** action restores from cache [vcpkg](https://github.com/microsoft/vcpkg) along with the previously installed ports. Briefly:
- If there is a cache miss, vpckg is fetched and installed; the cache's key is composed by hashing the hosting OS name, the command line arguments and the vcpkg's commit id.
- Restoring from cache can be skipped with `doNotCache:true`.
- Then `vcpkg` is run to install the desired ports. This is a no-op if artifacts are already restored.
- This step can be skipped with `setupOnly:true`.
- Artifacts and vcpkg are then saved in cache (if it was a 'cache miss').
- Saving to cache can be skipped with `doNotCache:true`.
The **run-vcpkg** action restores from cache [vcpkg](https://github.com/microsoft/vcpkg) along with the previously installed ports. On the other hand when there is a "cache miss":
- `vpckg` is fetched and installed; the cache's key is composed by hashing the hosting OS name, the command line arguments and the vcpkg's commit id.
- Restoring from cache can be skipped with `doNotCache: true`.
- Then `vcpkg` is run to install the desired ports.
- This step can be skipped with `setupOnly: true`.
- Artifacts and `vcpkg` are then saved in cache.
- Saving to cache can be skipped with `doNotCache: true`.
- Saving to cache happens at the end of the workflow in case `setupOnly: true`, otherwise it happens at the end of the action execution.

The provided [samples](#samples) use [GitHub hosted runners](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners).

Expand All @@ -20,12 +21,12 @@ Good companions are the [run-cmake](https://github.com/marketplace/actions/run-c
## User Manual
* [Contributing](#contributing)
* [Quickstart](#quickstart)
* [Restore cache/install/create cache](#install)
* [Restore cache/do not install/create cache](#setuponly)
* [Setup vcpkg and use CMake and a vcpkg.json to install and build](#manifest)
* [Setup vcpkg and use your own scripts](#setuponly)
* [Flowchart](#flowchart)
* [Best practices](#best-practices)
* [Use vcpkg as a submodule of your repository](#use-vcpkg-as-a-submodule-of-your-repository)
* [Use vcpkg's response file as an argument](#use-vcpkgs-response-file-as-an-argument)
* [Use vcpkg's vcpkg.json file to specify the dependencies](#vcpkgjson)
* [Action reference: all input/output parameters](#reference)
* [Samples](#samples)
* [Projects](#projects)
Expand All @@ -36,58 +37,53 @@ Read [CONTRIBUTING.md](CONTRIBUTING.md)

## <a id='quickstart'>Quickstart</a>

### <a id='install'>Setup vcpkg and install ports</a>
### <a id='manifest'>Setup vcpkg and use CMake with a vcpkg.json to install dependencies and build your project</a>

It is __highly recommended__ to [use vcpkg as a submodule](#best-practices). Here below the sample where vcpkg is a Git submodule:
It is __highly recommended__ to either [use vcpkg as a submodule](#vcpkgsubmodule) and to [use vcpkg.json](#vcpkgjson) to declaratively specify the dependencies.

```yaml
# Sample when vcpkg is a submodule of your repository (highly recommended!)

#-uses: actions/cache@v1 <===== YOU DO NOT NEED THIS!
Both suggestions are shown in the [hosted-advanced-setup-vcpkg-manifest.yml](https://github.com/lukka/CppBuildTasks-Validation/blob/master/.github/workflows/hosted-advanced-setup-vcpkg-manifest.yml) workflow, here below an excerpt:

# Install latest CMake.
- uses: lukka/get-cmake@latest

# Restore from cache the previously built ports. If "cache miss", then provision vcpkg, install desired ports, finally cache everything for the next run.
- name: Restore from cache and run vcpkg
uses: lukka/run-vcpkg@v5
with:
# Response file stored in source control, it provides the list of ports and triplet(s).
vcpkgArguments: '@${{ env.vcpkgResponseFile }}'
# Location of the vcpkg as submodule of the repository.
vcpkgDirectory: '${{ github.workspace }}/vcpkg'
# Since the cache must be invalidated when content of the response file changes, let's
# compute its hash and append this to the computed cache's key.
appendedCacheKey: ${{ hashFiles(env.vcpkgResponseFile) }}

- name: 'Build with CMake and Ninja'
uses: lukka/run-cmake@v3
with:
cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
cmakeListsTxtPath: '${{ github.workspace }}/cmakesettings.json/CMakeLists.txt'
useVcpkgToolchainFile: true
buildDirectory: '${{ runner.workspace }}/b/ninja'
cmakeAppendedArgs: '-GNinja Multi-Config'
# Or build multiple configurations out of a CMakeSettings.json file created with Visual Studio.
# cmakeListsOrSettingsJson: CMakeSettingsJson
# cmakeSettingsJsonPath: '${{ github.workspace }}/cmakesettings.json/CMakeSettings.json'
# configurationRegexFilter: '${{ matrix.configuration }}'
```yaml
#-uses: actions/cache@v1 <===== YOU DO NOT NEED THIS!

# Install latest CMake.
- uses: lukka/get-cmake@latest

# Restore from cache the previously built ports. If a "cache miss" occurs, then vcpkg is bootstrapped.
- name: Restore artifacts, or setup vcpkg (do not install any package)
uses: lukka/run-vcpkg@v6
with:
# Just install vcpkg for now, do not install any ports in this step yet.
setupOnly: true
# Location of the vcpkg as submodule of the repository.
vcpkgDirectory: '${{ github.workspace }}/vcpkg'
# Since the cache must be invalidated when content of the vcpkg.json file changes, let's
# compute its hash and append this to the computed cache's key.
appendedCacheKey: ${{ hashFiles( '**/vcpkg_manifest/vcpkg.json' ) }}
vcpkgTriplet: ${{ matrix.triplet }}
- name: Run CMake to install the dependencies with vcpkg.json manifest and build the project
uses: lukka/run-cmake@v3
with:
cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
cmakeListsTxtPath: '${{ github.workspace }}/vcpkg_manifest/CMakeLists.txt'
# This will pass to CMake the vcpkg.cmake toolchain file.
useVcpkgToolchainFile: true
buildWithCMake: true
```
### <a id='setuponly'>Setup vcpkg only</a>
### <a id='setuponly'>Setup vcpkg only and use your own scripts</a>
When `setupOnly: true`, it only setups vcpkg and set VCPKG_ROOT environment variable without installing any port. The provisioned vcpkg can then be used as follows in a subsequent step:
When `setupOnly: true`, it only setups `vcpkg` without installing any port. The provisioned `vcpkg` can then be used in a subsequent step:

```yaml
# Restore from cache the previously built ports. If cache-miss, download, build vcpkg.
# Restore from cache the previously built ports. If cache-miss, download and build vcpkg (aka "bootstrap vcpkg").
- name: Restore from cache and install vcpkg
# Download and build vcpkg, without installing any port. If content is cached already, it is a no-op.
uses: lukka/run-vcpkg@v5
uses: lukka/run-vcpkg@v6
with:
setupOnly: true
# Now that vcpkg is installed, it is being used to run desired arguments.
# Now that vcpkg is installed, it is being used to run with the desired arguments.
- run: |
$VCPKG_ROOT/vcpkg @$vcpkgResponseFile
$VCPKG_ROOT/vcpkg install boost:linux-x64
shell: bash
```
Expand All @@ -103,34 +99,19 @@ When `setupOnly: true`, it only setups vcpkg and set VCPKG_ROOT environment vari

## Best practices

### Use **vcpkg** as a submodule of your repository ###
### <a id='vcpkgsubmodule'>Use **vcpkg** as a submodule of your repository</a>

When using **vcpkg**, be aware of how it works, specifically:
- a specific version of vcpkg must be used either locally and on build servers;
- a specific version of vcpkg is identified by the commit id of the used vcpkg repository;
- it not possible to choose which version of a port to install, instead it is the used version of vcpkg that establishes which version (just one) of a port is available;
To sum up, **you need to pin the specific version of vcpkg you want to use to keep a consistent development experience between local and remote build environments.** This is accomplished by **using vcpkg as submodule of your Git repository**; this way the version of vcpkg used is implied by the commit id specified by the submodule for vcpkg.
- a specific version of `vcpkg` must be used either locally and on build servers;
- a specific version of `vcpkg` is identified by the commit id of the used vcpkg repository;
- it not possible to choose which version of a port to install, instead it is the used version of `vcpkg` that establishes which version (just one) of a port is available;

### Use vcpkg's response file as an argument
To sum up, **you need to pin the specific version of vcpkg you want to use to keep a consistent development experience between local and remote build environments.** This is accomplished by **using vcpkg as submodule of your Git repository**; this way the version of `vcpkg` used is implied by the commit id specified by the submodule for `vcpkg`.

vcpkg accepts a response file that contains the arguments, suitable to store the list of ports to be installed. **It is useful to store the response file under source control, this helps to run vcpkg the same exact way locally and remotely on the build servers.** For example if you want to run:
### <a id='vcpkgjson'>Use vcpkg's vcpkg.json file to specify the dependencies</a>

> vcpkg install boost zlib:x64 libmodbus --triplet x64
it is instead possible to run
> vcpkg install @response_file.txt
where `response_file.txt` contains (with no trailing whitespaces allowed):

```yaml
boost
zlib:x64
libmodbus
--triplet
x64
```
`vcpkg` can consume a [vcpkg.json](https://github.com/microsoft/vcpkg/blob/master/docs/specifications/manifests.md) file, that declaratively specifies the dependencies.
**Putting this manifest-like file under source control is highly recommended as this helps to run vcpkg the same exact way locally and remotely on the build servers.**

## <a id="samples">Samples</a>

Expand Down
84 changes: 31 additions & 53 deletions __tests__/basic.test.ts
View file
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
import * as process from 'process'
import * as fs from 'fs'
import * as vcpkgAction from '../src/vcpkg-action'
import * as actionLib from '@lukka/action-lib'
import * as vcpkgUtils from '../src/vcpkg-utils'
import * as vcpkgaction from '../src/vcpkg-action'
import * as actionlib from '@lukka/action-lib'
import * as cache from '@actions/cache'
import * as core from '@actions/core'
import * as runVcpkgLib from '@lukka/run-vcpkg-lib'
import * as runvcpkglib from '@lukka/run-vcpkg-lib'
import * as path from 'path'
import { BaseUtilLib } from '@lukka/base-util-lib'
import { ActionLib } from '@lukka/action-lib'
import * as baseutil from '@lukka/base-util-lib'
import * as vcpkgutils from '../src/vcpkg-utils'

jest.setTimeout(15 * 1000)
jest.setTimeout(15 * 1000);
// Mocks entire action-lib module.
jest.mock("@lukka/action-lib");
// Mocks entire run-vcpkg-lib module.
jest.mock("@lukka/run-vcpkg-lib");

function createDir(path: string) {
Expand Down Expand Up @@ -49,25 +49,24 @@ afterAll(async () => {
});

test('run-vcpkg: basic run scenario test', async () => {
const vcpkg: vcpkgAction.VcpkgAction = new vcpkgAction.VcpkgAction(
new BaseUtilLib(new actionLib.ActionLib()));
const vcpkg: vcpkgaction.VcpkgAction = new vcpkgaction.VcpkgAction(
new baseutil.BaseUtilLib(new actionlib.ActionLib()));
await vcpkg.run();
});

test('run-vcpkg: cache hit scenario test', async () => {
// Arrange.
const saveCacheSpy = jest.spyOn(cache, "saveCache").mockImplementation(
function (a, b, c): Promise<number> { return Promise.resolve(17); });
function (a, b, c): Promise<number> {
return Promise.resolve(17);
});
const restoreCacheSpy = jest.spyOn(cache, "restoreCache").mockImplementation(
function (a, b, c): Promise<string> { return Promise.resolve("hit"); });
const keyMatchMock = jest.spyOn(vcpkgUtils, "isExactKeyMatch").mockImplementation(
function (key: string, cacheKey?: string): boolean {
return true;
});
const keyMatchMock = jest.spyOn(vcpkgutils.Utils, "isExactKeyMatch").mockReturnValue(true);

// Act.
const vcpkg: vcpkgAction.VcpkgAction = new vcpkgAction.VcpkgAction(
new BaseUtilLib(new actionLib.ActionLib()));
const vcpkg: vcpkgaction.VcpkgAction = new vcpkgaction.VcpkgAction(
new baseutil.BaseUtilLib(new actionlib.ActionLib()));
await vcpkg.run();

// Asserts.
Expand All @@ -87,14 +86,10 @@ test('run-vcpkg: cache miss scenario test', async () => {
// Mock returns an underfined key since it is a cache miss.
const restoreCacheSpy = jest.spyOn(cache, "restoreCache").mockImplementation(
function (a, b, c): Promise<string | undefined> { return Promise.resolve(undefined); });
const keyMatchMock = jest.spyOn(vcpkgUtils, "isExactKeyMatch").mockImplementation(
function (key: string, cacheKey?: string): boolean {
return false;
});
const vcpkgRunnerRunSpy = jest.spyOn(runVcpkgLib.VcpkgRunner.prototype, "run");
const vcpkgRunnerRunSpy = jest.spyOn(runvcpkglib.VcpkgRunner.prototype, "run");
// Act.
const vcpkg: vcpkgAction.VcpkgAction = new vcpkgAction.VcpkgAction(
new BaseUtilLib(new actionLib.ActionLib()));
const vcpkg: vcpkgaction.VcpkgAction = new vcpkgaction.VcpkgAction(
new baseutil.BaseUtilLib(new actionlib.ActionLib()));
await vcpkg.run();

// Asserts.
Expand All @@ -104,33 +99,19 @@ test('run-vcpkg: cache miss scenario test', async () => {
expect(saveCacheSpy).toBeCalledTimes(1);
expect(vcpkgRunnerRunSpy).toBeCalledTimes(1);
expect(saveCacheSpy.mock.calls[0][1]).toBe(key);

// Clear mocks.
keyMatchMock.mockRestore();
});

test('run-vcpkg: cache must not be restored/saved when "doNotCache" is true scenario test', async () => {
// Arrange.
const saveCacheSpy = jest.spyOn(cache, "saveCache");
const restoreCacheSpy = jest.spyOn(cache, "restoreCache");
const keyMatchMock = jest.spyOn(vcpkgUtils, "isExactKeyMatch").mockImplementation(
function (key: string, cacheKey?: string): boolean {
return true;
});
const vcpkgRunnerRunSpy = jest.spyOn(runVcpkgLib.VcpkgRunner.prototype, "run");
const vcpkgRunnerRunSpy = jest.spyOn(runvcpkglib.VcpkgRunner.prototype, "run");
process.env["INPUT_DONOTCACHE"] = "true";
jest.mock("@actions/core");
const inputMock = jest.spyOn(core, "getInput").mockImplementation(
(name: string, options): string => {
if (name === vcpkgAction.doNotCacheInput) {
return "true";
} else {
return jest.requireActual("@actions/core").getInput(name, options);
}
})

// Act.
const vcpkg: vcpkgAction.VcpkgAction = new vcpkgAction.VcpkgAction(
new BaseUtilLib(new actionLib.ActionLib()));
const vcpkg: vcpkgaction.VcpkgAction = new vcpkgaction.VcpkgAction(
new baseutil.BaseUtilLib(new actionlib.ActionLib()));
await vcpkg.run();

// Asserts.
Expand All @@ -139,30 +120,27 @@ test('run-vcpkg: cache must not be restored/saved when "doNotCache" is true scen
expect(restoreCacheSpy).toBeCalledTimes(0);
expect(saveCacheSpy).toBeCalledTimes(0);
expect(vcpkgRunnerRunSpy).toBeCalledTimes(1);

// Clear mocks.
keyMatchMock.mockRestore();
});

test('isExactKeyMatch tests', () => {
const hash = vcpkgUtils.hashCode("hashCode");
const hash2 = vcpkgUtils.hashCode("hashCode2");
expect(vcpkgUtils.isExactKeyMatch(hash, hash2)).toBeFalsy();
expect(vcpkgUtils.isExactKeyMatch(hash2, hash)).toBeFalsy();
expect(vcpkgUtils.isExactKeyMatch(hash, hash)).toBeTruthy();
const hash = vcpkgutils.Utils.hashCode("hashCode");
const hash2 = vcpkgutils.Utils.hashCode("hashCode2");
expect(vcpkgutils.Utils.isExactKeyMatch(hash, hash2)).toBeFalsy();
expect(vcpkgutils.Utils.isExactKeyMatch(hash2, hash)).toBeFalsy();
expect(vcpkgutils.Utils.isExactKeyMatch(hash, hash)).toBeTruthy();
});

test('getVcpkgCommitId tests', async () => {
const p = path.resolve(path.join(__dirname, ".."));
const baseUtils = new BaseUtilLib(new ActionLib());
const baseUtils = new baseutil.BaseUtilLib(new actionlib.ActionLib());
// It must return undefined when GITHUB_WORKSPACE is not defined.
process.env.GITHUB_WORKSPACE = undefined;
expect(await vcpkgUtils.getVcpkgCommitId(baseUtils, p)).toBe(undefined);
expect(await vcpkgutils.Utils.getVcpkgCommitId(baseUtils, p)).toBe(undefined);

// It must return undefined when there is no vcpkg repository.
process.env.GITHUB_WORKSPACE = "/var/tmp/anything";
process.env.INPUT_VCPKGDIRECTORY = "fafsadfdsds";
expect(await vcpkgUtils.getVcpkgCommitId(baseUtils, p)).toBeFalsy();
expect(await vcpkgutils.Utils.getVcpkgCommitId(baseUtils, p)).toBeFalsy();

const expectedVcpkgSubmoduleCommitId = "commit_id_sha";
// A commit SHA is 40 characters long.
Expand All @@ -172,6 +150,6 @@ test('getVcpkgCommitId tests', async () => {
jest.spyOn(fs, 'existsSync').mockImplementationOnce(function (filePath) {
return true;
});
const commitId = await vcpkgUtils.getVcpkgCommitId(baseUtils, p);
const commitId = await vcpkgutils.Utils.getVcpkgCommitId(baseUtils, p);
expect(commitId).toBe(expectedVcpkgSubmoduleCommitId);
});
2 changes: 2 additions & 0 deletions action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ inputs:
runs:
using: 'node12'
main: 'dist/index.js'
post: 'dist/post/index.js'
post-if: 'success()'

branding:
icon: 'terminal'
Expand Down
Loading

0 comments on commit 324f999

Please sign in to comment.