KhaaS : Bug into playbook and GRPC server doesn't works

[theoberthier]

Describe the bug

1. To have ui-jupyter i must modifed docker-compose.release.yaml to add your jupyter ui image

2. kubehound dump remote => add env on host :

   • AWS_ACCESS_KEY_ID=<>
   • AWS_SECRET_ACCESS_KEY=<>
   • AWS_DEFAULT_REGION=<>
   • AWS_ENDPOINT_URL=http://:

3. GRPC server deny connection

To Reproduce
Steps to reproduce the behavior:

1. launch all stack with :
   "docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml up -d"
   in /Kubehound/deployments/kubehound/
   this error it's raise : service "ui-jupyter" has neither an image nor a build context specified: invalid compose project

2. GRPC aren't reachable :
   add env variable describe in 2.
   when i try to reach the endpoint :9000, with grpc client or ./bin/build/kubehound dump remote --bucket s3://kh-bucket --insecure --khaas-server 10.10.20.50:9000

INFO[17:05:58] Loading application from inline command      
INFO[17:05:58] Using /home/<user>/.config/kubehound.yaml for default config 
INFO[17:05:58] Initializing application telemetry           
WARN[17:05:58] Telemetry disabled via configuration         
INFO[17:05:58] Loading Kubernetes data collector client     
WARN[17:05:58] About to dump k8s cluster: "default" - Do you want to continue ? [Yes/No] 


-----------------
INFO[17:06:01] Launching ingestion on <ip>:9000 [rundID: 01j7h0s28d1m5hckz4gbgngwaa] 
FATA[17:06:01] call Ingest (default:01j7h0s28d1m5hckz4gbgngwaa): rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp <ip>:9000: connect: connection refused"

i tried to logs a container, and execute shell inside to debug it, but i can't launch anything with docker exec -it ....

Expected behavior
When i launch ./kubehound dump remote ........ i except to push dump into s3 bucket (it's works) and i want to send RPC request to my GRPC server

[jt-dd]

Thanks for reporting the issue. I spotted some errors regarding the deployment example. We are deploying a fix #265 . Can you try redeploying with the following file:

• https://github.com/DataDog/KubeHound/blob/91a3e4a3ba1a221af1368a8b7415fba9158095a0/deployments/kubehound/docker-compose.release.yaml
• Command: docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml up

Also for easier setup, we are adding env variable to setup the ingestor/grpc image #264. Regarding your config what did you use regarding the ingestor.api.endpoint and ingestor.api.insecure ?

[jt-dd]

Everything have been updated in v1.5.1. It should work out of the box now. You can setup your environment using the env variable KH_*.

[theoberthier]

I have try to deploy v1.5.1 and in docker-compose.yaml, in ui-jupyter, the field "profile" stop the deployment of jupyter ui.
When i move profile, the deployment works or i put --profile jupyter, but the documentation don't talk about this.

• For the GRPC server, i have configured the KH_* env and i have "connection refused"
• For the bucket env i don't saw the env variable for the endpoint of bucket, to default, they s3 command like aws s3 ls s3 used the domain name of amazon , my-bucket.s3.amazon .... and i have need to specified a enpoint-url to contact my local s3 bucket.
  In version 1.4.0, AWS_ENDPOINT_URL is understood by Kubehound and I saw that the blob storage step was successful, but this is not the case in the new version.

the process blocked in blob storage step, with this error : "dump core: empty bucket name"

Thank you for your answers

[jt-dd]

For the GRPC server issue can you post:

• docker ps output
• docker logs kubehound-release-grpc-1 output (just make sure you anonymise the bucket name)

For the bucket, I am going to push a fix for it.

[theoberthier]

For sure :

$ docker ps
ghcr.io/datadog/kubehound-binary:latest   "/kubehound serve"       2 days ago   Up 41 seconds           0.0.0.0:9000->9000/tcp                                                         kubehound-release-grpc-1

$ docker logs kubehound-release-grpc-1

time="09:14:41" level=fatal msg="factory config creation: graph database client creation: E0104: no successful connections could be made: Forbidden"
time="09:14:42" level=info msg="Loading application configuration from default embedded"
time="09:14:43" level=warning msg="No local config file was found (kubehound.yaml)"
time="09:14:43" level=info msg="Using /kubehound for default config\n"
time="09:14:43" level=info msg="Initializing application telemetry"
time="09:14:43" level=warning msg="Telemetry disabled via configuration"
time="09:14:43" level=info msg="Starting KubeHound Distributed Ingestor Service"
time="09:14:43" level=info msg="Initializing providers (graph, cache, store)"
time="09:14:43" level=info msg="Loading cache provider"
time="09:14:43" level=info msg="Loaded memcache cache provider"
time="09:14:43" level=info msg="Loading store database provider"
time="09:14:43" level=info msg="Loaded mongodb store provider"
time="09:14:43" level=info msg="Loading graph database provider"
2024/09/19 09:14:43 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:14:43 Error creating new connection for connection pool: Forbidden
2024/09/19 09:14:43 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:14:43" level=warning msg="Retrying to connect [1/5]"
2024/09/19 09:14:53 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:14:53 Error creating new connection for connection pool: Forbidden
2024/09/19 09:14:53 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:14:53" level=warning msg="Retrying to connect [2/5]"
2024/09/19 09:15:03 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:03 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:03 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:03" level=warning msg="Retrying to connect [3/5]"
2024/09/19 09:15:13 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:13 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:13 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:13" level=warning msg="Retrying to connect [4/5]"
2024/09/19 09:15:23 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:23 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:23 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:23" level=warning msg="Retrying to connect [5/5]"
2024/09/19 09:15:33 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:33 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:33 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:33" level=fatal msg="factory config creation: graph database client creation: E0104: no successful connections could be made: Forbidden"

Here are the main logs that keep coming back

[jt-dd]

Did you pull the latest version using docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml pull ?

Can you post the image sha of your image ?

• docker inspect kubehound-release-grpc-1 --format='{{.Image}}'

[theoberthier]

• docker inspect : sha256:d53db372b4202989fab80f00b43abeba21ce765b4d4fb2c9195cc873a9286b95

I pulled new images and i restarted, i have same message in new release when i launch kubehound dump remote :

• bucket name are empty

in v1.4.1 binary with the same env, when i dump remote the connection to GRPC server is refused with new images.

[jt-dd]

How do you set your bucket name ? If you set it from the config file kubehound.yaml, which key is setting it up ?

It should bucket_url like that:

# Ingestor configuration (for KHaaS)
ingestor:
  blob:
    # (i.e.: s3://<your-bucket>)
    bucket_url: ""