Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow using empty strings as job parameters #3158

Merged
merged 4 commits into from
Feb 2, 2024

Conversation

kanterov
Copy link
Contributor

@kanterov kanterov commented Jan 26, 2024

Changes

Allow using an empty string as the "default" value for job parameters, otherwise, such jobs are rejected because "default" field is required.

Remove "omitempty" modifier because both fields are required.

Tests

Verified manually with Terraform

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

@kanterov kanterov requested review from a team as code owners January 26, 2024 10:47
@kanterov kanterov requested review from tanmay-db and removed request for a team January 26, 2024 10:47
@kanterov kanterov changed the title Allow using emtpy strings as job parameters Allow using empty strings as job parameters Jan 26, 2024
@kanterov kanterov requested a review from pietern January 26, 2024 10:48
@codecov-commenter
Copy link

codecov-commenter commented Jan 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (acc1cea) 84.29% compared to head (c5d5983) 83.14%.
Report is 23 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3158      +/-   ##
==========================================
- Coverage   84.29%   83.14%   -1.15%     
==========================================
  Files         162      168       +6     
  Lines       14405    14811     +406     
==========================================
+ Hits        12143    12315     +172     
- Misses       1565     1771     +206     
- Partials      697      725      +28     
Files Coverage Δ
jobs/resource_job.go 88.51% <ø> (ø)

... and 27 files with indirect coverage changes

@pietern
Copy link
Contributor

pietern commented Jan 26, 2024

@kanterov @gaborratky-db Isn't it valid to define a job where some job parameters are required? I.e. you have to specify the parameters when running it? If so, then this approach does not allow that use case.

I believe this fix needs to use the ForceSendFields feature to mark the default as set or not. That way it is possible to both define parameters with no default (such that specifying them for a run is mandatory) as well as parameters with the default being an empty string.

@kanterov
Copy link
Contributor Author

@pietern, we don't really have 'required' job parameters in this case. But, the 'default' field in the API is required. If it's set to an empty string, it gets skipped in the API request because of the 'omitempty' thing, and that's causing a 400 error. So, we've got to make sure we include this 'default' field every time, even if it's just an empty string.

@pietern pietern requested a review from mgyucht January 29, 2024 09:44
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kanterov Please add a unit test to confirm that empty strings are now specified. There are two cases to consider; a user explicitly passes an empty string (good), and a user doesn't specify the default at all, which would default to an empty string. In the latter case, we could see configuration drift on the TF side.

@mgyucht mgyucht added this pull request to the merge queue Feb 2, 2024
Merged via the queue into databricks:main with commit 2d5f808 Feb 2, 2024
5 checks passed
@@ -90,8 +90,22 @@ func TestAccJobTasks(t *testing.T) {

notebook_task {
notebook_path = databricks_notebook.this.path
base_parameters = {
"param_0" = "{{job.parameters.empty_default}}"
"param_0" = "{{job.parameters.non_empty_default}}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kanterov This is a map, so only one of these will end up on the backend.

Does that impact the validity of the integration test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh... my bad, we need to fix it. It shouldn't matter because we don't run it, and I've tried empty value manually. But let's fix the test.

tanmay-db added a commit that referenced this pull request Feb 5, 2024
### New Features and Improvements
* Exporter: timestamps are now added to log entries ([#3146](#3146)).
* Validate metastore id for databricks_grant and databricks_grants resources ([#3159](#3159)).
* Exporter: Skip emitting of clusters that come from more cluster sources ([#3161](#3161)).
* Fix typo in docs ([#3166](#3166)).
* Migrate cluster schema to use the go-sdk struct ([#3076](#3076)).
* Introduce Generic Settings Resource ([#2997](#2997)).
* Update actions/setup-go to v5 ([#3154](#3154)).
* Change default branch from `master` to `main` ([#3174](#3174)).
* Add .codegen.json configuration ([#3180](#3180)).
* Exporter: performance improvements for big workspaces ([#3167](#3167)).
* update ([#3192](#3192)).
* Exporter: fix generation of cluster policy resources ([#3185](#3185)).
* Fix unit test ([#3201](#3201)).
* Suppress diff should apply to new fields added in the same chained call to CustomizableSchema ([#3200](#3200)).
* Various documentation updates ([#3198](#3198)).
* Use common.Resource consistently throughout the provider ([#3193](#3193)).
* Extending customizable schema with `AtLeastOneOf`, `ExactlyOneOf`, `RequiredWith` ([#3182](#3182)).
* Fix `databricks_connection` regression when creating without owner ([#3186](#3186)).
* add test code for job task order ([#3183](#3183)).
* Allow using empty strings as job parameters ([#3158](#3158)).
* Fix notebook parameters in acceptance test ([#3205](#3205)).
* Exporter: Add retries for `Search`, `ReadContext` and `Import` operations when importing the resource ([#3202](#3202)).
* Fixed updating owners for UC resources ([#3189](#3189)).
* Adds `databricks_volumes` as data source  ([#3150](#3150)).

### Documentation Changes

### Exporter

### Internal Changes
@tanmay-db tanmay-db mentioned this pull request Feb 5, 2024
github-merge-queue bot pushed a commit that referenced this pull request Feb 6, 2024
* Release v1.35.1

### New Features and Improvements
* Exporter: timestamps are now added to log entries ([#3146](#3146)).
* Validate metastore id for databricks_grant and databricks_grants resources ([#3159](#3159)).
* Exporter: Skip emitting of clusters that come from more cluster sources ([#3161](#3161)).
* Fix typo in docs ([#3166](#3166)).
* Migrate cluster schema to use the go-sdk struct ([#3076](#3076)).
* Introduce Generic Settings Resource ([#2997](#2997)).
* Update actions/setup-go to v5 ([#3154](#3154)).
* Change default branch from `master` to `main` ([#3174](#3174)).
* Add .codegen.json configuration ([#3180](#3180)).
* Exporter: performance improvements for big workspaces ([#3167](#3167)).
* update ([#3192](#3192)).
* Exporter: fix generation of cluster policy resources ([#3185](#3185)).
* Fix unit test ([#3201](#3201)).
* Suppress diff should apply to new fields added in the same chained call to CustomizableSchema ([#3200](#3200)).
* Various documentation updates ([#3198](#3198)).
* Use common.Resource consistently throughout the provider ([#3193](#3193)).
* Extending customizable schema with `AtLeastOneOf`, `ExactlyOneOf`, `RequiredWith` ([#3182](#3182)).
* Fix `databricks_connection` regression when creating without owner ([#3186](#3186)).
* add test code for job task order ([#3183](#3183)).
* Allow using empty strings as job parameters ([#3158](#3158)).
* Fix notebook parameters in acceptance test ([#3205](#3205)).
* Exporter: Add retries for `Search`, `ReadContext` and `Import` operations when importing the resource ([#3202](#3202)).
* Fixed updating owners for UC resources ([#3189](#3189)).
* Adds `databricks_volumes` as data source  ([#3150](#3150)).

### Documentation Changes

### Exporter

### Internal Changes

* upd

* readable

* upd

* upd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants