feat: support workload identity token #1556
Conversation
cvvz
requested review from
vibhansa-msft,
souravgupta-msft,
ashruti-msft,
syeleti-msft and
jainakanksha-msft
as code owners
|
@vibhansa-msft this PR exposes |
| EnvAzStorageSpnClientId = "AZURE_STORAGE_SPN_CLIENT_ID" | ||
| EnvAzStorageSpnClientSecret = "AZURE_STORAGE_SPN_CLIENT_SECRET" | ||
| EnvAzStorageSpnOAuthTokenFilePath = "AZURE_OAUTH_TOKEN_FILE" | ||
| EnvAzStorageSpnWorkloadIdentityToken = "WORKLOAD_IDENTITY_TOKEN" |
There was a problem hiding this comment.
if we use WORKLOAD_IDENTITY_TOKEN env var, who is responsible for token refresh?
There was a problem hiding this comment.
The token was generated by kubelet when it try to mount volume, and it will be passed to csi driver and csi driver set it as WORKLOAD_IDENTITY_TOKEN env. Kubelet is responsible for updating the token ,but I think we should only use the token during mounting.
| azspn.config.TenantID, | ||
| azspn.config.ClientID, | ||
| func(ctx context.Context) (string, error) { | ||
| return azspn.config.WorkloadIdentityToken, nil |
There was a problem hiding this comment.
Is this option something like where user is providing the token directly to blobfuse?
If so, what is the process to refresh the token or validation of token ?
There was a problem hiding this comment.
why do we need to save the token in a file? that's not safe
There was a problem hiding this comment.
My question was are we expecting a file name here in this env variable or directly the token. If directly token is what we expect then who will be responsible for the refresh. If its a file then re-reding the file (assuming the file now has the latest token) can work but giving directly the token will not work.
There was a problem hiding this comment.
The token was generated by kubelet when try to mount the blob volume and pass to blobfuse for authentication. I think we only need the token for mounting. Kubelet is responsible for updating the token ,but I think we only need the token during mounting.
There was a problem hiding this comment.
I think here the problem is when token is expired, whether blobfuse mount is still valid or not
There was a problem hiding this comment.
I created a blobfuse pod and the io was not impacted when the token expires.
If the token expired, kubelet will generate a new token when we try to mount again.
There was a problem hiding this comment.
If you remount then its fine as its a fresh start. Workflow we are discussing here is when Blobfuse remains mounted for a longer duration and in between the token expires. There will not be any remount in this case and newly generated token will not be forwarded to Blobfuse as well. In such case all storage REST call shall fail with 4xx errors. Can you validate this flow.
There was a problem hiding this comment.
I have tested that when the token expires, the blobfuse io won't be impacted. What kind of storage REST call do you mean? CSI driver does not have any storage REST call after volume mounted. Is there any REST call from blobfuse2 side? Can you show me the way to validate that?
There was a problem hiding this comment.
Blobfuse interacts with Azure Storage via REST call only. Lets assume you are trying to upload a 10GB file. Job starts and file is divided into block and each block is uploaded via a seperate REST call (few uploads going on in parallel). Now while the upload is in progress the token expires. As Blobfuse is not updated with the new token, it will continue using the expired token to upload the remaining chunks but the storage backend will fail these calls with Auth error as the token is expired. This is the part where you need to somehow convey Blobfuse to start using a new token. Generally MSI/SPN case Blobfuse acquires a token and 5 minute before its expiry it refreshes the token internally. Here as the token is provided as an input, its responsibility of the token provider to take care of the refresh and provide the updated token to Blobfuse while the job is going on. This is what Sourav pointed earlier in his comment that the function you have implemented shall take care of token update as well.
There was a problem hiding this comment.
If the token that you are sending it across to Blobfuse is not going to expire then its fine and I can approve the PR
|
@cvvz Please update PR description |
…to support-wi-token
… into support-wi-token
✅ What
Support provide workload identity token when authenticating rather than use the token file.
🤔 Why
Kubelet generate the token when mounting blob volume and we should pass the token to blobfuse2 rather than create a token file.
👩🔬 How to validate if applicable
I've already tested with blob-csi-driver
🔖 Related links