Add support for internal-frontend #602

dcaputo-harmoni · 2024-10-26T17:11:57Z

What was changed

Adds support for internal-frontend which has been available since v1.20

Why?

Because it is necessary for certain auth configurations to allow internal access, also addresses #560

adamko147 · 2024-10-28T18:03:06Z

charts/temporal/values.yaml

@@ -242,6 +242,33 @@ server:
    containerSecurityContext: {}
    topologySpreadConstraints: []
    podDisruptionBudget: {}
+  internal-frontend:


seems to me the helm chart is uses camelCase convention, would it ok if we renamed internal-frontend to internalFrontend? It would also makes the usage a bit easier, we could use {{ if $.Values.server.internalFrontend.enabled }} directly.

@adamko147 I had initially done this as internalFrontend but recall these identifiers are actually used to generate the kubernetes services and it doesn't allow uppercase letters - hence internal-frontend which corresponds to what the actual temporal feature is named.

huh, thanks, did not realize that.

Just as a proposition: you can create a simple helper to use internalFrontend as a "yaml" part and internal-frontend in names when needed:

{{- define "serviceName" -}} {{- $service := index . 0 -}} {{- if eq $service "internalFrontend" }} {{- print "internal-frontend" }} {{- else }} {{- print $service }} {{- end }} {{- end -}}

Usage:

{{ $service := include "serviceName" (list "frontend" "internalFrontend" "matching" "history" "worker") }}

adamko147 · 2024-10-28T18:09:20Z

charts/temporal/values.yaml

@@ -242,6 +242,33 @@ server:
    containerSecurityContext: {}
    topologySpreadConstraints: []
    podDisruptionBudget: {}
+  internal-frontend:


also, to be backwards compatible, should we disable internal frontend by default?

@adamko147 agreed, that was meant to be disabled by default, I'll make this update

dleblanc · 2024-10-28T19:41:32Z

This seems to contain all of the work (and more) that was done in #571. I'm happy to close that one if we can get this over the line.

… services and pdbs are not created if disabled

dcaputo-harmoni · 2024-10-28T20:29:09Z

@adamko147 @dleblanc just pushed a commit to address disabling by default, let me know if there's anything else needed here

adamko147

👍looks good to me. still needs 👀 from code owners...

TheGeniesis

Thx for the MR, I'm waiting for an official implementation of feature!

TheGeniesis · 2024-10-29T10:15:07Z

charts/temporal/values.yaml

@@ -242,6 +242,33 @@ server:
    containerSecurityContext: {}
    topologySpreadConstraints: []
    podDisruptionBudget: {}
+  internal-frontend:


Just as a proposition: you can create a simple helper to use internalFrontend as a "yaml" part and internal-frontend in names when needed:

{{- define "serviceName" -}} {{- $service := index . 0 -}} {{- if eq $service "internalFrontend" }} {{- print "internal-frontend" }} {{- else }} {{- print $service }} {{- end }} {{- end -}}

Usage:

{{ $service := include "serviceName" (list "frontend" "internalFrontend" "matching" "history" "worker") }}

TheGeniesis · 2024-10-29T10:19:03Z

charts/temporal/templates/admintools-deployment.yaml

@@ -33,9 +33,17 @@ spec:
          env:
            # TEMPORAL_CLI_ADDRESS is deprecated, use TEMPORAL_ADDRESS instead
            - name: TEMPORAL_CLI_ADDRESS
+              {{- if index $.Values.server "internal-frontend" "enabled" }}


@adamko147 Just a question about security:

If we use internal frontend here then everyone who has access to the admin-tools pod will be able to execute commands without any additional auth.

Is this expected?

well, that’s how we use it in our use case. Temporal is running in dedicated cluster/namespace and only frontend is exposed to outside world. access to the k8s cluster controlled via IAM, so unless you don’t have cluster access, you don’t have admin tools access.
it’s also works better for ops person when troubleshooting, who’s logs into admin tools using kubectl and from there has access w/o need to generate access token. and as I said, kubectl access is controlled by IAM connected to our internal company IDP where we also set the auth access

@adamko147 what you described matches our use case as well

My internal "deny all" rule is screaming that it will be a problem for some people who has stricter polices related to this topic. But if code owners are ok with that then I'm ok too.

TheGeniesis · 2024-10-29T10:33:10Z

charts/temporal/templates/server-configmap.yaml

@@ -173,8 +182,10 @@ data:
      {{- toYaml . | nindent 6 }}
    {{- end }}

+    {{- if not (index $.Values.server "internal-frontend" "enabled") }}


Did you test it with authorization?

When I was preparing similar changes I had to add condition to "authorization" + "publicClient" only for frontend. Otherwise internalFrontend will try to do auth and fail. Also, if we skip auth section for frontend then users will be able to access UI without permission.

I ended up with creating a loop to make separated configmap per service

{{- range $originalService := (list "frontend" "internalFrontend" "history" "matching" "worker") }} {{ $serviceValues := index $.Values.server $originalService }} {{ $service := include "serviceName" (list $originalService) }} ... {{- if eq $service "frontend" }} {{- with $server.config.authorization }} authorization: {{- toYaml . | nindent 10 }} {{- end }} {{- end }} ... {{- if ne $service "frontend"}} internal-frontend: rpc: grpcPort: {{ $server.internalFrontend.service.port }} httpPort: {{ $server.internalFrontend.service.httpPort }} membershipPort: {{ $server.internalFrontend.service.membershipPort }} bindOnIP: "0.0.0.0" {{- end }}

@TheGeniesis We have tested it with authorization on frontend (in our case using OIDC) which is working, but I haven't specifically tested auth for internal-frontend. I was under the impression (possibly mistakenly?) that the purpose of internal-frontend was to bypass authorization for internal access, but perhaps there's a middle ground where it has a different level of auth configured. If you have insights or ideas here would appreciate any contributions to this PR as it's a bit outside of my wheelhouse.

@TheGeniesis I don't think you're mistaken about the internal frontend purpose. It was introduced at temporalio/temporal#3706

The server worker needs to make connections to the frontend with effective "admin" access (to all namespaces). With pluggable claim mappers and authorizers, we can't control the logic in them, and requiring all users to set up mTLS with special claims is a large burden. Adding internal frontends as an alternative is easier and more reliable.

In our use case we have https://docs.temporal.io/self-hosted-guide/security#plugins custom authorizer/claim mapper for JWT tokens issued by our auth server and there is no way other temporal components (e.g. server worker) could get/refresh the token required for frontend access, so internal-frontend solves exactly this case.

I've successfully tested this PR with authorization enabled. Currently we're enabling auth using the built-in authorizer and claim plugins, and it's working without issue.

I haven't leveraged the features that were failing when we didn't have the internal worker enabled (eg: scheduled jobs), but the system did come up without the errors and seemed to be working properly.

I agree with @adamko147 about the internal frontend's purpose.

I've been trying to find time in between meetings to test this, so I appreciate that confirmation, @dleblanc.

Maybe I wasn't clear enough.

About definitions
As you mentioned - internal-frontend shouldn't have any authorization and should by used by internal pods (like worker).

Frontend should have authorization and should be used by users (direct calls/web).

My auth scenario
I use Azure EntraID to manage accesses with default Temporal Authorizer + mapper.

The EntraID was configured by following a medium article. The only difference is in system namespace which in the newer version is temporal-system instead of system.

The problem which I faced
During tests services were failing, because internal-frontend had "authorization" part in the config. I had to disable it for the internal-frontend, so it stopped trying to do auth for internal services.

The Frontend needs the "authorization" part and publicClient (since it's public).

This MR
The current config uses the same configmap for "frontend" and "internal-frontend". So, it's not possible to have different setup for both (internal without authorization, frontend without internal-frontend).

How I solved the issue in my repo
I have separated configmaps for frontend and internal-frontend with enabled/disabled options described above

I see it's working for you. There might be one more factor which I'm not aware of and it was the reason why it didn't fully work for me. Unfortunately, I'm behind my working schedule and I don't have time to test deployments again with this MR, to confirm that I'm able to reproduce the same problem again :/

stephen-harris · 2024-11-15T12:02:01Z

charts/temporal/values.yaml

+      type: ClusterIP
+      port: 7233
+      membershipPort: 6933
+      httpPort: 7243


Should these ports be 6936 and 7236 by default to match https://github.com/temporalio/temporal/blob/main/docker/config_template.yaml#L283-L284

Ports 6936 and 7236 also matches the suggestion in the release notes for version 1.20 (which introduced the internal-frontend).

dcaputo-harmoni · 2024-12-07T03:54:19Z

Just trying to understand what needs to be done to get this merged. One comment referenced changing port numbers (which is a quick fix) but seems that's all that's outstanding? Could one of the code owners confirm?

tomwheeler · 2024-12-08T16:31:33Z

Just trying to understand what needs to be done to get this merged. One comment referenced changing port numbers (which is a quick fix) but seems that's all that's outstanding? Could one of the code owners confirm?

@dcaputo-harmoni, I sincerely apologize for the delay here. Testing this is a manual process, and between meetings and other projects, I haven't had a contiguous block of time to see that all the way through yet. I will do that this week.

There are two changes I would like to see:

The port numbers used here deviate from those mentioned in the 1.20 release notes and used in our docker-compose template. I'd prefer if this PR could be updated to use those port numbers.
It would be great if you could follow the suggestion from @TheGeniesis so that it would follow the camel-case convention (internalFrontend) in our Helm Chart but use the hyphenated form for the service in Kubernetes itself.

Add support for internal-frontend

02d903a

dcaputo-harmoni requested a review from a team as a code owner October 26, 2024 17:11

dleblanc mentioned this pull request Oct 28, 2024

Including an internal-frontend service with auth enabled #571

Closed

adamko147 reviewed Oct 28, 2024

View reviewed changes

Disabling internal-frontend by default and ensuring that deployments,…

37834d7

… services and pdbs are not created if disabled

adamko147 reviewed Oct 28, 2024

View reviewed changes

TheGeniesis reviewed Oct 29, 2024

View reviewed changes

robholland mentioned this pull request Nov 15, 2024

[Feature Request] Allow disabling default worker service #569

Closed

stephen-harris reviewed Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for internal-frontend #602

Add support for internal-frontend #602

dcaputo-harmoni commented Oct 26, 2024

adamko147 Oct 28, 2024

dcaputo-harmoni Oct 28, 2024

adamko147 Oct 28, 2024

TheGeniesis Oct 29, 2024

adamko147 Oct 28, 2024

dcaputo-harmoni Oct 28, 2024

dleblanc commented Oct 28, 2024

dcaputo-harmoni commented Oct 28, 2024

adamko147 left a comment

TheGeniesis left a comment

TheGeniesis Oct 29, 2024

TheGeniesis Oct 29, 2024

adamko147 Nov 1, 2024 •

edited

Loading

dcaputo-harmoni Nov 3, 2024

TheGeniesis Nov 6, 2024

TheGeniesis Oct 29, 2024

dcaputo-harmoni Nov 3, 2024

adamko147 Nov 3, 2024 •

edited

Loading

dleblanc Nov 5, 2024

tomwheeler Nov 5, 2024

TheGeniesis Nov 6, 2024

stephen-harris Nov 15, 2024

tomwheeler Nov 20, 2024

dcaputo-harmoni commented Dec 7, 2024

tomwheeler commented Dec 8, 2024 •

edited

Loading

Add support for internal-frontend #602

Are you sure you want to change the base?

Add support for internal-frontend #602

Conversation

dcaputo-harmoni commented Oct 26, 2024

What was changed

Why?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dleblanc commented Oct 28, 2024

dcaputo-harmoni commented Oct 28, 2024

adamko147 left a comment

Choose a reason for hiding this comment

TheGeniesis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamko147 Nov 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamko147 Nov 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcaputo-harmoni commented Dec 7, 2024

tomwheeler commented Dec 8, 2024 • edited Loading

adamko147 Nov 1, 2024 •

edited

Loading

adamko147 Nov 3, 2024 •

edited

Loading

tomwheeler commented Dec 8, 2024 •

edited

Loading