Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for terraform private module source for git #7167

Merged
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8c41565
initial commit
vishwahiremat Feb 7, 2024
d212a66
adding changes to support private git repo
vishwahiremat Feb 13, 2024
de5399d
refactor
vishwahiremat Feb 13, 2024
ce877cc
adding comments
vishwahiremat Feb 13, 2024
d1a7ab7
Merge branch 'main' into vishwahiremat/private-repo-support
vishwahiremat Feb 13, 2024
3b29b9e
addressing comments
vishwahiremat Feb 15, 2024
70a6bfe
Merge branch 'main' into vishwahiremat/private-repo-support
vishwahiremat Feb 16, 2024
188e3d8
addressing comments
vishwahiremat Feb 16, 2024
e717d11
changing the field name secretstore->secret
vishwahiremat Feb 16, 2024
e2d4936
addressing comments
vishwahiremat Feb 20, 2024
4b6b8b9
adding change to add url config to gitconfig
vishwahiremat Feb 21, 2024
2149aad
addressing comments
vishwahiremat Feb 21, 2024
f66fcc7
adding comments
vishwahiremat Feb 21, 2024
4db184f
adding secret loader
vishwahiremat Feb 21, 2024
ec388f7
addressing comments
vishwahiremat Feb 22, 2024
89c32fa
addressing comments
vishwahiremat Feb 22, 2024
d5cefb8
merge conflicts
vishwahiremat Feb 22, 2024
c3b89df
addressing comments
vishwahiremat Feb 22, 2024
b843bc7
addressing comments
vishwahiremat Feb 23, 2024
6023202
addressing comments
vishwahiremat Feb 23, 2024
7f89d77
adding typespec examples
vishwahiremat Feb 23, 2024
81e5598
adding typespec examples
vishwahiremat Feb 23, 2024
2ada35f
Trigger Build
vishwahiremat Feb 23, 2024
fcdc510
Trigger Build
vishwahiremat Feb 23, 2024
8c178ca
adding error for write file
vishwahiremat Feb 23, 2024
c7091f0
fixing linter error
vishwahiremat Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion hack/bicep-types-radius/generated/index.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"Resources":{"Applications.Core/applications@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":58},"Applications.Core/containers@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":123},"Applications.Core/environments@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":148},"Applications.Core/extenders@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":166},"Applications.Core/gateways@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":187},"Applications.Core/httpRoutes@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":201},"Applications.Core/secretStores@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":224},"Applications.Core/volumes@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":261},"Applications.Dapr/pubSubBrokers@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":49},"Applications.Dapr/secretStores@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":66},"Applications.Dapr/stateStores@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":84},"Applications.Datastores/mongoDatabases@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":50},"Applications.Datastores/redisCaches@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":69},"Applications.Datastores/sqlDatabases@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":88},"Applications.Messaging/rabbitMQQueues@2023-10-01-preview":{"RelativePath":"applications/applications.messaging/2023-10-01-preview/types.json","Index":50}},"Functions":{"applications.core/extenders":{"2023-10-01-preview":[{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":262}]},"applications.core/secretstores":{"2023-10-01-preview":[{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":268}]},"applications.datastores/mongodatabases":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":90}]},"applications.datastores/rediscaches":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":92}]},"applications.datastores/sqldatabases":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":94}]},"applications.messaging/rabbitmqqueues":{"2023-10-01-preview":[{"RelativePath":"applications/applications.messaging/2023-10-01-preview/types.json","Index":52}]}}}
{"Resources":{"Applications.Core/applications@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":58},"Applications.Core/containers@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":123},"Applications.Core/environments@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":154},"Applications.Core/extenders@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":172},"Applications.Core/gateways@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":193},"Applications.Core/httpRoutes@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":207},"Applications.Core/secretStores@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":230},"Applications.Core/volumes@2023-10-01-preview":{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":267},"Applications.Dapr/pubSubBrokers@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":49},"Applications.Dapr/secretStores@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":66},"Applications.Dapr/stateStores@2023-10-01-preview":{"RelativePath":"applications/applications.dapr/2023-10-01-preview/types.json","Index":84},"Applications.Datastores/mongoDatabases@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":50},"Applications.Datastores/redisCaches@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":69},"Applications.Datastores/sqlDatabases@2023-10-01-preview":{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":88},"Applications.Messaging/rabbitMQQueues@2023-10-01-preview":{"RelativePath":"applications/applications.messaging/2023-10-01-preview/types.json","Index":50}},"Functions":{"applications.core/extenders":{"2023-10-01-preview":[{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":268}]},"applications.core/secretstores":{"2023-10-01-preview":[{"RelativePath":"applications/applications.core/2023-10-01-preview/types.json","Index":274}]},"applications.datastores/mongodatabases":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":90}]},"applications.datastores/rediscaches":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":92}]},"applications.datastores/sqldatabases":{"2023-10-01-preview":[{"RelativePath":"applications/applications.datastores/2023-10-01-preview/types.json","Index":94}]},"applications.messaging/rabbitmqqueues":{"2023-10-01-preview":[{"RelativePath":"applications/applications.messaging/2023-10-01-preview/types.json","Index":52}]}}}
40 changes: 40 additions & 0 deletions pkg/corerp/api/v20231001preview/environment_conversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package v20231001preview

import (
"fmt"
"reflect"
"strings"

v1 "github.com/radius-project/radius/pkg/armrpc/api/v1"
Expand Down Expand Up @@ -63,6 +64,24 @@ func (src *EnvironmentResource) ConvertTo() (v1.DataModelInterface, error) {
}
converted.Properties.Compute = *envCompute

recipeConfig := src.Properties.RecipeConfig
if recipeConfig != nil {
if recipeConfig.Terraform != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these lines are specific to Terraform, maybe we should move this logic to a helper function?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @sk593. We can have a function that takes care of the recipeConfig conversion and that function can make a call to another function which takes care of terraform related stuff conversion. With this approach we could keep things simple. It would be more readable, easier to debug and write unit tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if recipeConfig.Terraform.Authentication != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these conditions be collapsed? Or do we expect that there may be future logic that's unique to each config level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case i think we still need to check nil at every level to avoid null pointer. Let me know if you think otherwise

gitConfig := recipeConfig.Terraform.Authentication.Git
if gitConfig != nil {
pat := gitConfig.Pat
p := map[string]datamodel.Secret{}
for k, v := range pat {
p[k] = datamodel.Secret{
SecretStore: to.String(v.SecretStore),
}
}
converted.Properties.RecipeConfig.Terraform.Authentication.Git.PAT = p
}
}
}
}
if src.Properties.Recipes != nil {
envRecipes := make(map[string]map[string]datamodel.EnvironmentRecipeProperties)
for resourceType, recipes := range src.Properties.Recipes {
Expand Down Expand Up @@ -151,6 +170,27 @@ func (dst *EnvironmentResource) ConvertFrom(src v1.DataModelInterface) error {
dst.Properties.Recipes = recipes
}

if !reflect.DeepEqual(env.Properties.RecipeConfig, datamodel.RecipeConfigProperties{}) {
recipeConfig := &RecipeConfigProperties{}
if !reflect.DeepEqual(env.Properties.RecipeConfig.Terraform, datamodel.TerraformConfigProperties{}) {
Copy link
Contributor

@sk593 sk593 Feb 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, since this logic is specific to Terraform, maybe we can move it to a helper function

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can apply the same thing as I mentioned above but this time in reverse.

This function -> RecipeConfig conversion function -> Terraform related conversion function

In future, it would be very easy to add new types of provider conversions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created a separate func

Copy link
Contributor

@kachawla kachawla Feb 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the suggestion of creating a Terraform related conversion function. @vishwahiremat I know you have updated it to recipe config specific conversion, have you considered adding Terraform specific function to make it easy to extend the recipe config function in the future?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vishwa and I discussed offline, we will abstract it out into Terraform specific function once we add support for other types.

recipeConfig.Terraform = &TerraformConfigProperties{}
if !reflect.DeepEqual(env.Properties.RecipeConfig.Terraform.Authentication, datamodel.AuthConfig{}) {
recipeConfig.Terraform.Authentication = &AuthConfig{}
if !reflect.DeepEqual(env.Properties.RecipeConfig.Terraform.Authentication.Git, datamodel.GitAuthConfig{}) {
recipeConfig.Terraform.Authentication.Git = &GitAuthConfig{}
if env.Properties.RecipeConfig.Terraform.Authentication.Git.PAT != nil {
recipeConfig.Terraform.Authentication.Git.Pat = map[string]*Secret{}
}
}
}
}
for k, v := range env.Properties.RecipeConfig.Terraform.Authentication.Git.PAT {
recipeConfig.Terraform.Authentication.Git.Pat[k] = &Secret{
SecretStore: to.Ptr(v.SecretStore),
}
}
dst.Properties.RecipeConfig = recipeConfig
}
if env.Properties.Providers != (datamodel.Providers{}) {
dst.Properties.Providers = &Providers{}
if env.Properties.Providers.Azure != (datamodel.ProvidersAzure{}) {
Expand Down
14 changes: 14 additions & 0 deletions pkg/corerp/api/v20231001preview/environment_conversion_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,19 @@ func TestConvertVersionedToDataModel(t *testing.T) {
Scope: "/planes/aws/aws/accounts/140313373712/regions/us-west-2",
},
},
RecipeConfig: datamodel.RecipeConfigProperties{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a few different layers and we could run into nil pointer exception easily. We should test it thoroughly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we remove the PAT map from this test, we can see if we run into nil pointer exception above in the original conversion logic.

Terraform: datamodel.TerraformConfigProperties{
Authentication: datamodel.AuthConfig{
Git: datamodel.GitAuthConfig{
PAT: map[string]datamodel.Secret{
"dev.azure.com": datamodel.Secret{
SecretStore: "/planes/radius/local/resourcegroups/default/providers/Applications.Core/secretStores/github",
},
},
},
},
},
},
Recipes: map[string]map[string]datamodel.EnvironmentRecipeProperties{
ds_ctrl.MongoDatabasesResourceType: {
"cosmos-recipe": datamodel.EnvironmentRecipeProperties{
Expand Down Expand Up @@ -368,6 +381,7 @@ func TestConvertDataModelToVersioned(t *testing.T) {
if tt.filename == "environmentresourcedatamodel.json" {
require.Equal(t, "Azure/cosmosdb/azurerm", string(*versioned.Properties.Recipes[ds_ctrl.MongoDatabasesResourceType]["terraform-recipe"].GetRecipeProperties().TemplatePath))
require.Equal(t, recipes.TemplateKindTerraform, string(*versioned.Properties.Recipes[ds_ctrl.MongoDatabasesResourceType]["terraform-recipe"].GetRecipeProperties().TemplateKind))
require.Equal(t, "/planes/radius/local/resourcegroups/default/providers/Applications.Core/secretStores/github", string(*versioned.Properties.RecipeConfig.Terraform.Authentication.Git.Pat["dev.azure.com"].SecretStore))
switch c := recipeDetails.(type) {
case *TerraformRecipeProperties:
require.Equal(t, "1.1.0", string(*c.TemplateVersion))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,19 @@
"scope": "/planes/aws/aws/accounts/140313373712/regions/us-west-2"
}
},
"recipeConfig": {
"terraform": {
"authentication": {
"git": {
"pat": {
"dev.azure.com":{
"secretStore":"/planes/radius/local/resourcegroups/default/providers/Applications.Core/secretStores/github"
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if this part doesn't exist (by mistake of the operator), would this cause a nil pointer exception?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think it will nil pointer if pat property doesnt exist

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a unit test for this?

}
}
}
},
"recipes": {
"Applications.Datastores/mongoDatabases":{
"cosmos-recipe": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@
"scope": "/planes/aws/aws/accounts/140313373712/regions/us-west-2"
}
},
"recipeConfig": {
"terraform": {
"authentication": {
"git": {
"pat": {
"dev.azure.com":{
"secretStore":"/planes/radius/local/resourcegroups/default/providers/Applications.Core/secretStores/github"
}
}
}
}
}
},
"recipes": {
"Applications.Datastores/mongoDatabases":{
"cosmos-recipe": {
Expand Down
40 changes: 40 additions & 0 deletions pkg/corerp/api/v20231001preview/zz_generated_models.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading