Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ruleSet and KMS setting for CLI CSFLE during init #2942

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

channingdong
Copy link
Contributor

@channingdong channingdong commented Nov 15, 2024

Release Notes

This PR serves as a following up PR to finalize the CSFLE work on CLI, including the following changes:

Breaking Changes

  • PLACEHOLDER

New Features

  • Add ruleSet and Key Management Service (KMS) driver support to finalize the Client Side Field Level Encryption (CSFLE) feature for confluent kafka topic [produce | consume]

Bug Fixes

  • PLACEHOLDER

Checklist

  • Leave this box unchecked if features are not yet available in production

What

  • Add the KMS driver and field-level encryption executor for each schema type during serializer/deserializer init.
  • Add all the built-in schemas required as reference/extension schemas for the CSFLE features.
  • Add encryption related test cases for all 3 types of schemas.
  • Copy the built-in schemas to a local temporary folder during Kafka topic produce/consume (same folder as the main schema).
  • Update the AVRO schema serializer such that ruleSet can be extracted correctly from the native Go object.
  • Update the PROTOBUF schema deserializer process to parse the ruleSet correctly.
  • Update the confluent-kafka-go library to latest version.
  • Update the serializer/deserializer unit tests such that the temporary schema directory creation and deletion only happens once before/after all tests start/finish, instead of doing it inside each individual unit test.

References

Test & Review

The manual verification passing result can be found here:
https://docs.google.com/document/d/1GwXz9hNOkub_Br-2nssoYWCf6elZBvwo7TMhCNYinwE/edit?tab=t.0#heading=h.1xcbdvagwov4

@channingdong channingdong requested a review from a team as a code owner November 15, 2024 21:08
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link
Member

@rayokota rayokota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @channingdong , left some comments

pkg/serdes/avro_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/avro_serialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/json_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/json_serialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/protobuf_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/protobuf_serialization_provider.go Outdated Show resolved Hide resolved
@channingdong channingdong changed the title RuleSet and KMS setting for CLI CSFLE Add ruleSet and KMS setting for CLI CSFLE during init Nov 26, 2024
Comment on lines 152 to 177
currDir, err := os.Getwd()
if err != nil {
return nil, fmt.Errorf("Error getting current working directory: %v\n", err)
}
log.CliLogger.Debugf("Current working directory is: %s\n", currDir)

// Copy all the built-in proto schemas needed for CSFLE to <importPath> where the main schema is stored
// Note: folder path should be set correctly based on current working directory
// Expected working directory in CLI test is: /Users/github.com/confluentinc/cli/pkg/serdes
// Expected working directory in CLI shell execution is: /Users/github.com/confluentinc/cli
for _, folder := range builtInSchemaFoldersToCopy {
dst := importPath + "/" + folder

// Skip copying the built-in schema folders if they are present already in the temp folder
if _, err = os.Stat(dst); err == nil {
log.CliLogger.Debugf("Built-in schema folder already exists %s, skipping copy again:\n", dst)
continue
}

// Locate the source of built-in schema folders
if strings.HasSuffix(currDir, "confluentinc/cli") {
folder = "pkg/serdes/" + folder
}
if err = copy.Copy(folder, dst); err != nil {
return nil, fmt.Errorf("Error copying built-in schemas folder %s: %w\n", folder, err)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sgagniere This part is something relatively new to CLI, your review will be appreciated!

rayokota
rayokota previously approved these changes Nov 26, 2024
Copy link
Member

@rayokota rayokota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @channingdong , LGTM

sgagniere
sgagniere previously approved these changes Dec 5, 2024
Copy link
Member

@sgagniere sgagniere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After syncing w/ Channing: we need to make the proto files available even when users don't have any copy on their machine

Copy link
Member

@sgagniere sgagniere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants