Notation to sign and verify arbitrary data

[priteshbandi]

Is your feature request related to a problem?

We would like to use Notation to sign and verify any arbitrary data and keep using the fine grain control it provides using trust policy and plugin model.

What solution do you propose?

Notation support sign and verify any arbitrary data

notation sign-blob ./somefile.txt
OR
notation sign ./somefile.txt --blob

cc: @yizha1 @toddysm @shizhMSFT @patrickzheng200 @gokarnm @iamsamirzon

[Two-Hearts]

Thanks @priteshbandi, it's on my list, will share a PoC once it's ready, and we could start from there.

[priteshbandi]

Thanks @patrickzheng200 for proactively working on this.
Do you have ECD (Estimated Completion Date) for PoC? Also, can you share any early draft around CLI interface?

[gokarnm]

Thanks @patrickzheng200 ! There would be some spec work associated for this feature, covering changes (if any) for payload format, signature storage on disk, and trust policy.

[yizha1]

@gokarnm @priteshbandi I remember we had some discussions around local artifacts/arbitrary files/signature storage at least one year ago. At that time, we tracked the features in roadmap repo, I did some searches and find related issues like

• [User Story] - notation sign/verify Offline/Local artifacts
• Local storage of Signatures

There were also some discussions and investigations during implementing feature of "sign/verify local OCI artifact", which is currently an experimental feature. But we didn't continue the discussion recently since it is not in v1 scope any more.

This could be a good topic for community meeting, we can discuss it further on the plan. Does it make sense?

[FeynmanZhou]

I agree that this is a significant feature request for Notation and has been discussed several times. This request was also captured in Requirement (scenario-1-local-build-sign-validate) and Goals.

Before having an ECD for this issue, let's figure out the end-to-end scenarios, how it works with trust policy, have an approachable design proposal and CLI Spec, and split the work items.

Let's discuss it in the next community meeting.

[FeynmanZhou]

A follow-up question, if signing local arbitrary files is supported, will signing content from OCI Image Layout be replaced accordingly?

[Two-Hearts]

Thanks @patrickzheng200 for proactively working on this. Do you have ECD (Estimated Completion Date) for PoC? Also, can you share any early draft around CLI interface?

It's in its initial stage, so I don't have an ECD for now. But as suggested by Yi and Feynman, we could have a discussion on the feature in the next week's meeting.
(btw, @priteshbandi @gokarnm this is my current github account: @Two-Hearts, I couldn't get any notification if you ping my previous account. Thanks.)

[toddysm]

Here are a few scenarios I am aware of where this will be helpful.

Signing SBOMs for Software

We have teams that produce binaries that are not distributed via OCI registries (think of published on web sites, or package managers, etc.). They would like to produce SBOMs for those binaries and sign the SBOMs with Notation. The resulting artifacts are binary, SBOM, signature of the SBOM (.sig file for example). Those three artifacts can be published on a web site for download as separate items.

Distributing the SBOM via registry

As an expansion of the above scenario. We don't want to change the workflow of the above teams, but we would like to distribute the SBOM via registry. In this scenario the binary, the SBOM and the SBOM signature are created using the above process. However, we would like to use the SBOM and the SBOM signature and publish them to a registry for distribution. What that means is that we would like to take the SBOM file, the SBOM signature file (.sig file) and pack them as OCI artifacts and push to the registry.

In both scenarios above, we would like to use Notation to verify the signatures.

I assume this will need updates to the specification in addition to the code.

[Two-Hearts]

Thanks @toddysm for providing the user scenarios.
Here's my design doc:
https://hackmd.io/QteHaBQTS-6h-AsU04U1cQ?view

Here's a PoC implementing the above design released on my local branch:
https://github.com/Two-Hearts/notation/releases/tag/v1.0.0-arbitraryFile

It's an initial version, and please feel free to leave your comments/suggestions. Thanks.
@shizhMSFT @priteshbandi @sajayantony @toddysm @yizha1 @FeynmanZhou @JeyJeyGao

[FeynmanZhou]

Apart from signing SBOM, I can share two typical scenarios that rely on signing and verifying arbitrary files.

Sign and verify the installation binaries and release assets for GitHub releases

There are some open-source projects that require maintainers to use the GPG key to sign the installation binaries and other release assets when publishing on the release page. Users are recommended to verify the GPG signature with the GPG tool when downloading from the GitHub release page. Just take the ORAS release verification process as an example: oras-project/oras-www#69 (comment)

However, signing and verifying with the GPG key is a bit cumbersome for maintainers and users since GPG has complicated setup steps. Notation could be a good alternative for signing and verifying the installation binary package if signing arbitrary files is supported. Open-source projects like Kubernetes, Argo CD published signed binaries and suggest users verifying them when downloading from the release page.

Sign and verify the installation binaries for traditional software release process

Some free softwares and organizations also publish their executable binaries and artifacts on FTP servers, central repository (like Nexus), and object storage. This feature is also useful when signing and verifying those binaries in the traditional release process. Project organizations can distribute the signature alongside the binaries to users.

[Two-Hearts]

Apart from signing SBOM, I can share two typical scenarios that rely on signing and verifying arbitrary files.

Sign and verify the installation binaries and release assets for GitHub releases

There are some open-source projects that require maintainers to use the GPG key to sign the installation binaries and other release assets when publishing on the release page. Users are recommended to verify the GPG signature with the GPG tool when downloading from the GitHub release page. Just take the ORAS release verification process as an example: oras-project/oras-www#69 (comment)

However, signing and verifying with the GPG key is a bit cumbersome for maintainers and users since GPG has complicated setup steps. Notation could be a good alternative for signing and verifying the installation binary package if signing arbitrary files is supported. Open-source projects like Kubernetes, Argo CD published signed binaries and suggest users verifying them when downloading from the release page.

Sign and verify the installation binaries for traditional software release process

Some free softwares and organizations also publish their executable binaries and artifacts on FTP servers, central repository (like Nexus), and object storage. This feature is also useful when signing and verifying those binaries in the traditional release process. Project organizations can distribute the signature alongside the binaries to users.

Thanks @FeynmanZhou, the above design should cover the scenarios mentioned by you as well.

[yizha1]

Thanks @patrickzheng200 ! There would be some spec work associated for this feature, covering changes (if any) for payload format, signature storage on disk, and trust policy.

For the 1st scenario as Toddy described, we need specifications to cover

• signature payload
• storage on disk
• trust policy
• Any changes to the existing signing and verification workflow

For the 2nd scenario "Distributing the SBOM via registry", we need spec to cover

• storage of both SBOM and signature file in registry
• Any changes to the existing verification workflow.

We need CLI spec to cover the CLI UX for both scenarios

[Two-Hearts]

Thanks @patrickzheng200 ! There would be some spec work associated for this feature, covering changes (if any) for payload format, signature storage on disk, and trust policy.

For the 1st scenario as Toddy described, we need specifications to cover

• signature payload
• storage on disk
• trust policy
• Any changes to the existing signing and verification workflow

For the 2nd scenario "Distributing the SBOM via registry", we need spec to cover

• storage of both SBOM and signature file in registry
• Any changes to the existing verification workflow.

We need CLI spec to cover the CLI UX for both scenarios

@yizha1 Here's the PR for the CLI spec changes: #765.

[yizha1]

Created two issues for tracking the changes on Notary Project specifications:

• Add new specifications for non-OCI signatures specifications#275
• Add new specifications for distributing non-OCI artifacts and signatures via OCI compliant registries specifications#276

/cc: @priteshbandi @Two-Hearts @gokarnm @shizhMSFT @toddysm @FeynmanZhou

[priteshbandi]

Discussions are tracked in #767

[yizha1]

Added more details for these two scenarios described in #741 (comment)

@toddysm @gokarnm @iamsamirzon @priteshbandi @FeynmanZhou @Two-Hearts @shizhMSFT @sajayantony PTAL.

Procedure for signing an arbitrary file and store the signature on the file system

1. Authenticate to a KMS

2. Create a signing key or use an existing signing key, and retrieve the key id from the KMS

3. Use Notation CLI to sign the file on the file system. An example of command:

   notatoin sign --blob <filepath> --media-type <type> --output-signature <signature_path>` --id <key_id> --plugin <plugin_name>
   

   Upon successful signing, the siganture is stored on <signature_path>

Procedure for verifying the file and signature on the filesystem

1. Add the root certificate to a named trust store mystore, the type of trust store is ca
2. Create a trust policy file mypolicy.json with the following content

{
  {
    "name": "wabbit-networks-non-oci",
    "signatureVerification": {
      "level": "strict"
    },
    "trustStores": [ "ca:mystore" ],
    "trustedIdentities": ["*"],
    "scopes": [ "non-oci" ]
  },
 {
    "name": "globalPolicy",
    "signatureVerification": {
      "level": "strict"
    },
    "trustStores": [ "ca:someGlobalCA" ],
    "trustedIdentities": ["*"],
    "scopes": [ "*" ]
  }
}

3. Use notation policy command to import trust policies from mypolicy.json file

notation policy import mypolicy.json

4. Use notation verify command to verify the file against the signature

notation verify --blob <filepath> --media-type <media_type> --signature <signature_path> --policy-scope "non-oci"

Procedure for verifying the file and signature as OCI artifacts in the registry

1. Pull the file and signature from the registry to the filesystem
   <TODO: add recommended tooling>
2. The rest procedure is the same as previous verification procedure

Procedure for storing the file and signature as OCI artifacts in the registry

<TODO: It is not to be solved by this feature, but it will be good to provide recommendation tooling for users to achieve this goal from e2e point of view.>

[yizha1]

As discussed at the meeting on 1/2/2024, move this feature to next milestone 1.2.0

[yizha1]

Move this to Future due to resource issue per community discussion.

[priteshbandi]

Task List with progress

1. [Completed] Spec update in specification repo | Support arbitrary blob signing specifications#283
2. [Completed] notation-go code updates
   1. [Completed] Add support for signing of arbitrary data | feat: add support for signing blob notation-go#379
   2. [Completed] Add support for verification of arbitrary data | feat: Implements VerifyBlob functionality notation-go#394
3. [Completed] Spec update in CLI /specification repo | docs: spec updates for arbitrary blob signing #811
4. notation CLI code changes
   1. Changes to expose for signing and verification of arbitrary data as CLI.
      1. [abandoned] Add new command for notation blob sign | feat: add sign blob commands #856
      2. [abandoned] Add new command for notation blob verify | Adding verify blob commands #889
      3. [abandoned] Add new command for notation blob inspect feat: add inspect blob commands #888
      4. Update OCI commands(notation sign, notation verify, notation list, notation inspect) to make sure they dont work with blob and if user tries to do that emit error messages and redirect them to notation blob
      5. Optional, Add stdin and stdout support for sign, verify and inspect command.
   2. Write e2e tests for signing, verifying and inspecting arbitrary data using CLI.

[yizha1]

@shizhMSFT @Two-Hearts @JeyJeyGao Would you mind reviewing the task list for blob signing listed in the above comment?

[shizhMSFT]

@shizhMSFT @Two-Hearts @JeyJeyGao Would you mind reviewing the task list for blob signing listed in the above comment?

The above list LGTM. In addition to the above list, we need to discuss updating the notary project spec for the file extensions for blob signing (details see #974).

[yizha1]

We also need to cover the scenario mentioned in this comment #741 (comment)