Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authentication does not work with kubernetes_asyncio as the only package installed #1008

Open
jmgilman opened this issue Feb 28, 2023 · 0 comments · May be fixed by #1010
Open

Authentication does not work with kubernetes_asyncio as the only package installed #1008

jmgilman opened this issue Feb 28, 2023 · 0 comments · May be fixed by #1010
Labels
bug Something isn't working

Comments

@jmgilman
Copy link

jmgilman commented Feb 28, 2023 •
edited
Loading

Long story short

The documentation claims that kubernetes_asyncio is supported; however, it does not appear to be used in the piggybacking code. Attempting to launch the operator with only kubernetes_asyncio installed results in authentication failures.

Kopf version

1.36.0

Kubernetes version

1.24

Python version

3.10.9

Code

@kopf.on.create(group="velero.io", version="v1", plural="backups")
async def create_fn(body, **kwargs):
    logging.info(f"A handler is called with body: {body}")

Logs

[2023-02-28 11:16:33,027] kopf._core.reactor.r [DEBUG   ] Starting Kopf 1.36.0.
[2023-02-28 11:16:33,028] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2023-02-28 11:16:33,028] kopf.activities.auth [DEBUG   ] Activity 'login_with_kubeconfig' is invoked.
[2023-02-28 11:16:33,058] kopf.activities.auth [INFO    ] Activity 'login_with_kubeconfig' succeeded.
[2023-02-28 11:16:33,058] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2023-02-28 11:16:33,767] kopf._core.reactor.r [ERROR   ] Resource observer has failed: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})
Traceback (most recent call last):
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('REDACTED/apis')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/observation.py", line 113, in resource_observer
    resources = await scanning.scan_resources(groups=group_filter, settings=settings, logger=logger)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 31, in scan_resources
    resources.update(await coro)
  File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 68, in _read_new_apis
    rsp = await api.get('/apis', settings=settings, logger=logger)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Admission insights chain is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Namespace observer is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Credentials retriever is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Admission webhook server is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Admission validating configuration manager is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.o [DEBUG   ] Streaming tasks stopping is skipped: no tasks given.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Multidimensional multitasker is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Poster of events is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Admission mutating configuration manager is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Daemon killer is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Root tasks are stopped: finishing normally; tasks left: set()
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG   ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('REDACTED/apis')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/277vy3lj0y87drkzz1g7d5lfz6vkcqdx-python3.10-kopf-1.36.0/bin/.kopf-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/cli.py", line 59, in wrapper
    return fn(*args, **kwargs)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/cli.py", line 113, in run
    return running.run(
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 58, in run
    loop.run_until_complete(operator(
  File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 135, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 416, in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 238, in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/observation.py", line 113, in resource_observer
    resources = await scanning.scan_resources(groups=group_filter, settings=settings, logger=logger)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 31, in scan_resources
    resources.update(await coro)
  File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 68, in _read_new_apis
    rsp = await api.get('/apis', settings=settings, logger=logger)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})

Additional information

The "native" authentication doesn't seem to support short-lived tokens as provided by a typical EKS cluster. The Python clients have no issue dealing with this, however, as noted above it appears that kubernetes_asyncio is never queried. The issue is resolved by adding the kubernetes package as a dependency. However, I now need to have both kubernetes and kubernetes_asyncio available for the operator to work. One reason this is undesirable is it bloats the final container image.
@jmgilman jmgilman added the bug Something isn't working label Feb 28, 2023
@jmgilman jmgilman changed the title Authenticate does not work with only kubernetes_asyncio Authentication does not work with kubernetes_asyncio as the only package installed Feb 28, 2023
@asteven asteven linked a pull request Mar 3, 2023 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

also support logging in via kubernetes_asyncio asteven/kopf
1 participant
@jmgilman