Podman containers monitoring support

[carbolymer]

Is your feature request related to a problem? Please describe.
When using podman instead of docker, glances fails with an exception.

Describe the solution you'd like
Glances should not fail to start. Podman containers should be visible in a similar manner as docker containers. Podman supports Docker API v1.40: https://docs.podman.io/en/latest/markdown/podman-system-service.1.html

Describe alternatives you've considered
n/a

Additional context
n/a

Steps to reproduce

1. Install podman
2. Enable podman service `systemctl start --user podman.service
3. Set DOCKER_HOST environment variable:

export XDG_RUNTIME_DIR=/run/user/$(id -u)  # this should not be needed - XDG_RUNTIME_DIR should be already set on your system
export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"

4. Start glances - it hangs with the following error (needs to be ctrl-c'd):

glances                                                                                                                                                                                                            
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/images/latest/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/glances", line 33, in <module>
    sys.exit(load_entry_point('Glances==3.2.4', 'console_scripts', 'glances')())
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 150, in main
    start(config=core.get_config(), args= core.get_args())
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 109, in start
    mode = GlancesMode(config=config, args=args)
  File "/usr/lib/python3.10/site-packages/glances/standalone.py", line 80, in __init__
    self.stats.update()
  File "/usr/lib/python3.10/site-packages/glances/stats.py", line 230, in update
    self._plugins[p].update()
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1151, in wrapper
    ret = fct(self, *args, **kw)
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1166, in wrapper
    ret = fct(*args, **kw)
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_docker.py", line 250, in update
    container_stats['Image'] = container.image.tags
  File "/usr/lib/python3.10/site-packages/docker/models/containers.py", line 40, in image
    return self.client.images.get(image_id.split(':')[1])
  File "/usr/lib/python3.10/site-packages/docker/models/images.py", line 314, in get
    return self.prepare_model(self.client.api.inspect_image(name))
  File "/usr/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image
    return self._result(
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.10/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.40/images/latest/json: Not Found ("failed to find image latest: latest: No such image")
^CException ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1560, in _shutdown
    lock.acquire()
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 69, in __signal_handler
    end()
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 84, in end
    sys.exit(0)
SystemExit: 0

[nicolargo]

Hi,

please you send us the Glances and Python Docker Client version (pip freeze | grep docker) ?

[pawcykca]

I have similar problem using podman on CentOS Stream 9

# glances
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/images/v1.0.11/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/glances", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/glances/__init__.py", line 150, in main
    start(config=core.get_config(), args=core.get_args())
  File "/usr/local/lib/python3.9/site-packages/glances/__init__.py", line 109, in start
    mode = GlancesMode(config=config, args=args)
  File "/usr/local/lib/python3.9/site-packages/glances/standalone.py", line 80, in __init__
    self.stats.update()
  File "/usr/local/lib/python3.9/site-packages/glances/stats.py", line 230, in update
    self._plugins[p].update()
  File "/usr/local/lib/python3.9/site-packages/glances/plugins/glances_plugin.py", line 1158, in wrapper
    ret = fct(self, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/glances/plugins/glances_plugin.py", line 1173, in wrapper
    ret = fct(*args, **kw)
  File "/usr/local/lib/python3.9/site-packages/glances/plugins/glances_docker.py", line 250, in update
    container_stats['Image'] = container.image.tags
  File "/usr/local/lib/python3.9/site-packages/docker/models/containers.py", line 40, in image
    return self.client.images.get(image_id.split(':')[1])
  File "/usr/local/lib/python3.9/site-packages/docker/models/images.py", line 314, in get
    return self.prepare_model(self.client.api.inspect_image(name))
  File "/usr/local/lib/python3.9/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/docker/api/image.py", line 251, in inspect_image
    return self._result(
  File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.9/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.40/images/v1.0.11/json: Not Found ("failed to find image v1.0.11: v1.0.11: No such image")

^CException ignored in: <module 'threading' from '/usr/lib64/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 1470, in _shutdown
    lock.acquire()
  File "/usr/local/lib/python3.9/site-packages/glances/__init__.py", line 69, in __signal_handler
    end()
  File "/usr/local/lib/python3.9/site-packages/glances/__init__.py", line 84, in end
    sys.exit(0)
SystemExit: 0

# dnf list installed | grep podman
podman.x86_64                       2:3.4.5-0.7.el9           @appstream
podman-catatonit.x86_64             2:3.4.5-0.7.el9           @appstream
podman-docker.noarch                2:3.4.5-0.7.el9           @appstream

# glances --version
Glances v3.2.4.2 with PsUtil v5.9.0
Log file: /tmp/glances-root.log

# pip freeze | grep docker
docker==5.0.3

[daqqad]

Same problem on MicroOS

# glances
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/images/latest/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/glances", line 33, in <module>
    sys.exit(load_entry_point('Glances==3.2.5', 'console_scripts', 'glances')())
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 189, in main
    start(config=core.get_config(), args=core.get_args())
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 118, in start
    mode = GlancesMode(config=config, args=args)
  File "/usr/lib/python3.10/site-packages/glances/standalone.py", line 80, in __init__
    self.stats.update()
  File "/usr/lib/python3.10/site-packages/glances/stats.py", line 221, in update
    self._plugins[p].update()
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1157, in wrapper
    ret = fct(self, *args, **kw)
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1174, in wrapper
    ret = fct(*args, **kw)
  File "/usr/lib/python3.10/site-packages/glances/plugins/glances_docker.py", line 258, in update
    container_stats['Image'] = container.image.tags
  File "/usr/lib/python3.10/site-packages/docker/models/containers.py", line 40, in image
    return self.client.images.get(image_id.split(':')[1])
  File "/usr/lib/python3.10/site-packages/docker/models/images.py", line 314, in get
    return self.prepare_model(self.client.api.inspect_image(name))
  File "/usr/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image
    return self._result(
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.10/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.40/images/latest/json: Not Found ("failed to find image latest: docker.io/library/latest: No such image")


^CException ignored in: <module 'threading' from '/usr/lib64/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1560, in _shutdown
    lock.acquire()
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 75, in __signal_handler
    end()
  File "/usr/lib/python3.10/site-packages/glances/__init__.py", line 90, in end
    sys.exit(0)
SystemExit: 0

#  glances --version
Glances v3.2.5 with PsUtil v5.9.0

Don't have pip installed so

# rpm -qa|grep docker
python310-docker-5.0.3-2.1.noarch

Manually testing pulling data from socket returns info without problems
curl --unix-socket /var/run/docker.sock http://localhost/v1.40/images/json|jq

[RazCrimson]

@daqqad I presume you also have podman, then shouldn't you be using that unix socket?

Also I did look into the API a bit, some fields seem to be receiving fixed values(IO & Network), while memory and cpu usage were fine. Getting the images list fails for some reason. Need more research on it.

Another alternative would be to make another plugin:
If we do choose that option this might be helpful: https://github.com/containers/podman-py (https://pypi.org/project/podman/)

[daqqad]

@RazCrimson I updated service file and renamed the socket for compatibility, but I am using podman.

[nicolargo]

+1 for the @RazCrimson proposal. If the Podman API is not fully compliant with the Docker one, the best solution is to create a new Podman plugin for Glances. Under the wood, if Podman is detected, the Docker plugin should be disable automatically.
For the dev part, we should avoid duplication code between the two plugins...

[nicolargo]

Added in the Glances version 4 roadmap (because the Python Podman Lib is only available on Python 3).

[RazCrimson]

@nicolargo I think we should not automatically disable the docker plugin too. If we do, we should atleast provide the option to enable it.

There might be edge cases where someone would want to monitor both docker and podman containers. 🙂

[sunjam]

There might be edge cases where someone would want to monitor both docker and podman containers. slightly_smiling_face

Absolutely, I've found the most natural way to get started with podman is to gradually migrate and test services in case of incompatibilities. Having the option to monitor both makes sense. Could a non-required recommendation by offered to disable docker when enabling podman, or vice versa.

[nicolargo]

Error during first test: containers/podman-py#179

[nicolargo]

Quick and dirty sandbox:

Start some pod:

podman pod create --name podtest
podman pod list
podman run -dt --pod podtest ubuntu top
podman ps -a --pod

Execute the Python code (https://gist.github.com/nicolargo/b9e6bd079b07fe64cb8c9b798a219cb0):

Release:  3.4.4
Compatible API:  1.40
Podman API:  3.4.4 

<Image: 'docker.io/library/alpine:latest'> e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 

<Image: 'k8s.gcr.io/pause:3.5'> ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459 

<Image: 'docker.io/library/ubuntu:latest'> 216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc 

================================================================================
AutoRemove False
Command None
Created 2022-10-22T14:23:03.120912374+02:00
CreatedAt 
ExitCode 0
Exited False
ExitedAt -62135596800
Id 9491515251edcd5bb5dc17205d7ee573c0be96fe0b08b0a12a7e2cea874565ea
Image k8s.gcr.io/pause:3.5
ImageID ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459
IsInfra True
Labels None
Mounts []
Names ['8d0f1c783def-infra']
Namespaces {}
Networks None
Pid 181221
Pod 8d0f1c783def193085bf68d2f245971e1c33562d96a0ee6952396865de3cb4ec
PodName podtest
Ports None
Size None
StartedAt 1666441640
State running
Status 
================================================================================
AutoRemove False
Command ['top']
Created 2022-10-22T14:27:20.000770538+02:00
CreatedAt 
ExitCode 0
Exited False
ExitedAt -62135596800
Id 1da5bc154a086d39eb2fce44a0ab11b89aab7bae86f5752cac0eb3393e9de9de
Image docker.io/library/ubuntu:latest
ImageID 216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc
IsInfra False
Labels None
Mounts []
Names ['hungry_lalande']
Namespaces {}
Networks None
Pid 181228
Pod 8d0f1c783def193085bf68d2f245971e1c33562d96a0ee6952396865de3cb4ec
PodName podtest
Ports None
Size None
StartedAt 1666441640
State running
Status 

Stop the pod:

podman stop podtest

Main issue: no information concerning the CPU / RAM / DiskIO via this lib but should be exposed by the API because the podman stats command line display the following information:

[nicolargo]

Another way:

podman stats --no-stream --format=json -a

Return:

[
 {
  "id": "1da5bc154a08",
  "name": "hungry_lalande",
  "cpu_time": "675.345ms",
  "cpu_percent": "4.05%",
  "avg_cpu": "4.05%",
  "mem_usage": "876.5kB / 7.836GB",
  "mem_percent": "0.01%",
  "net_io": "-- / --",
  "block_io": "-- / --",
  "pids": "1"
 },
 {
  "id": "9491515251ed",
  "name": "8d0f1c783def-infra",
  "cpu_time": "5.15ms",
  "cpu_percent": "3.09%",
  "avg_cpu": "3.09%",
  "mem_usage": "221.2kB / 7.836GB",
  "mem_percent": "0.00%",
  "net_io": "-- / --",
  "block_io": "-- / --",
  "pids": "1"
 }
]

Could have a look on this repository https://github.com/m-erhardt/check-container-stats/blob/master/check_container_stats_podman.py for a similar approach (but JSON format is better...).

[nicolargo]

Create a new branch for the ongoing development: https://github.com/nicolargo/glances/tree/issue1985

[RazCrimson]

@nicolargo
How do you want the Podman to be supported?
The current changes in https://github.com/nicolargo/glances/tree/issue1985 seems to modify the existing plugin to support podman containers too.

So would you prefer a new podman plugin or make the current one support podman too?

In the latter case, should we have a column to indicate the container engine from which a container was found?

[nicolargo]

API corrected after a restart of the Podman service.

$ python
>>> from podman import PodmanClient
>>> uri = "unix:///run/user/1000/podman/podman.sock"
>>> PodmanClient(base_url=uri).version()
{'Platform': {'Name': 'linux/amd64/ubuntu-22.04'}, 'Components': [{'Name': 'Podman Engine', 'Version': '3.4.4', 'Details': {'APIVersion': '3.4.4', 'Arch': 'amd64', 'BuildTime': '1970-01-01T01:00:00+01:00', 'Experimental': 'false', 'GitCommit': '', 'GoVersion': 'go1.17.3', 'KernelVersion': '5.15.0-58-generic', 'MinAPIVersion': '3.1.0', 'Os': 'linux'}}, {'Name': 'Conmon', 'Version': 'conmon version 2.0.25, commit: unknown', 'Details': {'Package': 'conmon: /usr/bin/conmon'}}, {'Name': 'OCI Runtime (crun)', 'Version': 'crun version 0.17\ncommit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL', 'Details': {'Package': 'crun: /usr/bin/crun'}}], 'Version': '3.4.4', 'ApiVersion': '1.40', 'MinAPIVersion': '1.24', 'GitCommit': '', 'GoVersion': 'go1.17.3', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '5.15.0-58-generic', 'BuildTime': '1970-01-01T01:00:00+01:00'}

Get images list:

>>> for image in client.images.list():
...     print(image, image.id, "\n")
... 
<Image: 'docker.io/library/alpine:latest'> e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 

<Image: 'k8s.gcr.io/pause:3.5'> ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459 

<Image: 'docker.io/library/ubuntu:latest'> 216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc 

Images and containers infos:

>>> print(json.dumps(client.df(), indent=4))
{
    "Images": [
        {
            "Repository": "docker.io/library/alpine",
            "Tag": "latest",
            "ImageID": "e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294",
            "Created": "2022-05-23T19:19:31.970967174Z",
            "Size": 5815366,
            "SharedSize": 0,
            "UniqueSize": 5815366,
            "Containers": 1
        },
        {
            "Repository": "k8s.gcr.io/pause",
            "Tag": "3.5",
            "ImageID": "ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459",
            "Created": "2021-03-16T13:16:57.822648569Z",
            "Size": 689969,
            "SharedSize": 0,
            "UniqueSize": 689969,
            "Containers": 1
        },
        {
            "Repository": "docker.io/library/ubuntu",
            "Tag": "latest",
            "ImageID": "216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc",
            "Created": "2022-10-04T23:35:20.857335994Z",
            "Size": 80360801,
            "SharedSize": 0,
            "UniqueSize": 80360801,
            "Containers": 2
        }
    ],
    "Containers": [
        {
            "ContainerID": "1da5bc154a086d39eb2fce44a0ab11b89aab7bae86f5752cac0eb3393e9de9de",
            "Image": "216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc",
            "Command": [
                "top"
            ],
            "LocalVolumes": 0,
            "Size": 77837008,
            "RWSize": 34,
            "Created": "2022-10-22T14:27:20.000770538+02:00",
            "Status": "configured",
            "Names": "hungry_lalande"
        },
        {
            "ContainerID": "28cb9c28fef7dc64c8ef000f436fb867c355d0a85861ae5cab106ab26126ec7d",
            "Image": "e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294",
            "Command": [
                "/bin/sh"
            ],
            "LocalVolumes": 0,
            "Size": 5529096,
            "RWSize": 104,
            "Created": "2022-05-29T10:15:53.954943081+02:00",
            "Status": "exited",
            "Names": "test"
        },
        {
            "ContainerID": "9491515251edcd5bb5dc17205d7ee573c0be96fe0b08b0a12a7e2cea874565ea",
            "Image": "ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459",
            "Command": [],
            "LocalVolumes": 0,
            "Size": 682696,
            "RWSize": 12,
            "Created": "2022-10-22T14:23:03.120912374+02:00",
            "Status": "running",
            "Names": "8d0f1c783def-infra"
        },
        {
            "ContainerID": "a1206796715fc73abb1146bd0f5a4027305dcf0227bba762d660cd62462ff8ad",
            "Image": "216c552ea5ba7b0e3f6e33624e129981c39996021403518019d19b8843c27cbc",
            "Command": [
                "top"
            ],
            "LocalVolumes": 0,
            "Size": 77837008,
            "RWSize": 34,
            "Created": "2023-01-21T11:55:38.718496053+01:00",
            "Status": "running",
            "Names": "strange_lewin"
        }
    ],
    "Volumes": []
}

[nicolargo]

@RazCrimson

i just push some change in the dedicated branche: https://github.com/nicolargo/glances/tree/issue1985

But i am not satisfied of the current implementation and some features should be implemented for the Podman containers.

Here is the current status in the console UI:

(the container in the middle is a Docker, others are Podman).

What we should do:

• refactor the code in order (currently in glances_docker.py) to make the plugin compliant with multiple containers engines. So rename glances_docker.py to glances_containers.py and put the engines implementation in a subfolder. For example containers/glances_docker.py or glances_podman.py...).
• the UI (both console and WebUI) should be change to display the engine type (a new column with the engine name will be displayed only if more than one engine is used)
• some engine will have specifics informations (Pod name for Podman for example)... How to display (or not) those information (to be discussed)
• concerning Podman, some stats are not display (uptime and IO. For IO there is an open issue NET IO BLOCK IO is Not Present via Running podman stats containers/podman#11695)

@RazCrimson any free time to contribute to this enhancement request ?

[RazCrimson]

@nicolargo

Sure, I'll try taking a stab at this.

[RazCrimson]

@nicolargo
Just an architectural question for the implementation.

I was thinking if we could try moving towards more typed-style of Python and more structured data objects instead of dicts (maybe use dataclasses) as that would probably reduce lot of errors due to missing keys or having typos in keys and would also make the code more readable.

Currently, planning to use typing features that are available in python3.6(so won't be breaking compatibility for python3) for the containers plugin and maintain the existing dict interface so that nothing breaks.

What are your thoughts on this?

[nicolargo]

@RazCrimson any news ?

[RazCrimson]

Sorry for the late reply @nicolargo
Bit busy this week. 😅

I found what the issue is.
Will try to get it fixed by this weekend.

[nicolargo]

Thanks @RazCrimson !

After this issue, roadmap for the version 3.4 will be freezed.

Glances 3.4 is coming.

[nicolargo]

@RazCrimson On the develop branch after an update of the Docker lib (docker-6.1.0), Glances is broken and do not start anymore. Get back to normal when docker-6.0.1 is used. Be careful on the #1985 branch (traced in #2366).

[nicolargo]

Last but not least, Glances Alpine Image failed (ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects). See https://github.com/nicolargo/glances/actions/runs/4900769661/jobs/8751667592

Not sure this is related to the previous issue...

[nicolargo]

First problem solved (had to force stream=True in the stats() method).

[nicolargo]

Second problem here again: https://github.com/nicolargo/glances/actions/runs/4901481540/jobs/8752798884

Perhaps a Rust compiler not installed on the CI:

#25 93.68           =============================DEBUG ASSISTANCE=============================
#25 93.68           If you are seeing a compilation error please try the following steps to
#25 93.68           successfully install cryptography:
#25 93.68           1) Upgrade to the latest pip and try again. This will fix errors for most
#25 93.68              users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
#25 93.68           2) Read https://cryptography.io/en/latest/installation/ for specific
#25 93.68              instructions for your platform.
#25 93.68           3) Check our frequently asked questions for more information:
#25 93.68              https://cryptography.io/en/latest/faq/
#25 93.68           4) Ensure you have a recent Rust toolchain installed:
#25 93.68              https://cryptography.io/en/latest/installation/#rust

Strange because:

If you are using Linux, then you should upgrade pip (in a virtual environment!) and attempt to install cryptography again before trying to install the Rust toolchain. On most Linux distributions, the latest version of pip will be able to install a binary wheel, so you won’t need a Rust toolchain.

Will try: https://cryptography.io/en/latest/installation/#alpine

Not better...

#24 104.0       If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
#24 104.0       
#24 104.0       To update pip, run:
#24 104.0       
#24 104.0           pip install --upgrade pip

Not better after forcing upgrade to the latest pip version: https://github.com/nicolargo/glances/actions/runs/4901847706/jobs/8753350639

[RazCrimson]

@nicolargo
A fix for the docker issue (waiting to be upstreamed): docker/docker-py#3120

I missed one patch that needed to be upstreamed to podman-py. That is the cause of the current issue. 😅
Tracked here: containers/podman-py#266

I've added a temporary patch to the issue1985 branch to make it work with the latest podman release.
Give it a run when possible.

[nicolargo]

Perfect @RazCrimson ! The temporary patch make the job.

What do you think... Do we have to wait for the two upstreams (docker/docker-py#3120 and containers/podman-py#266) to release Glances 3.4 ?

[RazCrimson]

@nicolargo

We could probably make a release now.

We don't really depends on any feature in their newer releases, so I think we can go ahead and make a release.

[nicolargo]

Perfect, i am going to merge the #1985 branch into develop.

Last issue to solve before the 3.4 release is the #2368 concerning the Alpine image. I will try to have a look.

[nicolargo]

Before the merge, i have made a quick test using the make run-issuecommand line.

Note: on my system, i have one Docker containers and two Podman pods.

With the current develop branch:

docker        [OK]    0.01400s 

With the #1985 branch:

containers    [OK]    1.19207s

Not sure this is an issue but perhaps the time take to Grab Podman stats ?

@RazCrimson What do you think ?

[RazCrimson]

@nicolargo

Yep its probably podman.

In my tests for issue1985

Just Docker: 0.0299
Just Podman: 0.3307
Docker + Podman: 0.4041

I had 2 podman containers and 2 docker containers.

I'll look into optimising it. Removing the temp patch when those changes get upstreamed should hopefully reduce this time.

[nicolargo]

Ok, so we stay on the #1985 branch for the moment.

We wait the two upstreams:

• Fix container.stats infinite blocking on stream mode docker/docker-py#3120
• Streaming support for PodsManager.stats API containers/podman-py#266

After that, we can remove the patch: f7f4f38 and merge into develop.

[RazCrimson]

@nicolargo
Just an update, I cached the version calls to the podman server to speed up the update times.
It is a lot more reasonable now.

containers    [OK]    0.04262s

2 docker + 2 podman containers as the above

Changes are pushed to issue1985, test them when possible

[nicolargo]

A last issue when running Glances in Webserver mode:

# curl http://localhost:61208/api/3/containers

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
    <head>
        <title>Error: 404 Not Found</title>
        <style type="text/css">
          html {background-color: #eee; font-family: sans;}
          body {background-color: #fff; border: 1px solid #ddd;
                padding: 15px; margin: 15px;}
          pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
        </style>
    </head>
    <body>
        <h1>Error: 404 Not Found</h1>
        <p>Sorry, the requested URL <tt>&#039;http://localhost:61208/api/3/containers&#039;</tt>
           caused an error:</p>
        <pre>Cannot get plugin containers (datetime.datetime(2023, 1, 21, 11, 55, 38) is not JSON serializable)</pre>
    </body>
</html>

also reproduced with:

# ./venv/bin/python -m glances -C ./conf/glances.conf --stdout-json containers

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nicolargo/dev/glances/glances/__main__.py", line 18, in <module>
    glances.main()
  File "/home/nicolargo/dev/glances/glances/__init__.py", line 180, in main
    start(config=core.get_config(), args=core.get_args())
  File "/home/nicolargo/dev/glances/glances/__init__.py", line 131, in start
    mode.serve_forever()
  File "/home/nicolargo/dev/glances/glances/standalone.py", line 171, in serve_forever
    while self.__serve_once():
  File "/home/nicolargo/dev/glances/glances/standalone.py", line 155, in __serve_once
    ret = not self.screen.update(self.stats, duration=adapted_refresh)
  File "/home/nicolargo/dev/glances/glances/outputs/glances_stdout_json.py", line 47, in update
    stat = stats.get_plugin(plugin).get_json()
  File "/home/nicolargo/dev/glances/glances/plugins/glances_plugin.py", line 394, in get_json
    return self.get_stats()
  File "/home/nicolargo/dev/glances/glances/plugins/glances_plugin.py", line 390, in get_stats
    return json_dumps(self.stats)
  File "/home/nicolargo/dev/glances/glances/globals.py", line 56, in json_dumps
    return ujson.dumps(data)
TypeError: datetime.datetime(2023, 1, 21, 11, 55, 38) is not JSON serializable

===

Quick analyse: the issue came from the StartedAt type for Pod (type is good for Docker).

[{'Command': ['top'],
  'Created': '2023-01-21T11:55:38.718496053+01:00',
  'Id': 'a1206796715fc73abb1146bd0f5a4027305dcf0227bba762d660cd62462ff8ad',
  'Image': "['docker.io/library/ubuntu:latest']",
  'StartedAt': datetime.datetime(2023, 1, 21, 11, 55, 38),
  'Status': 'running',
  'Uptime': '3 months',
  'cpu_percent': 7.0788650697910715e-06,
  'engine': 'podman',
  'io_r': 0.0,
  'io_w': 0.0,
  'key': 'name',
  'memory_usage': 1310720.0,
  'name': 'strange_lewin',
  'network_rx': 0.0,
  'network_tx': 0.0,
  'pod_id': '8d0f1c783def',
  'pod_name': 'strange_lewin'},
 {'Command': [],
  'Created': '2022-10-22T14:23:03.120912374+02:00',
  'Id': '9491515251edcd5bb5dc17205d7ee573c0be96fe0b08b0a12a7e2cea874565ea',
  'Image': "['k8s.gcr.io/pause:3.5']",
  'StartedAt': datetime.datetime(2023, 1, 21, 11, 55, 38),
  'Status': 'running',
  'Uptime': '3 months',
  'cpu_percent': 3.0275666980429393e-10,
  'engine': 'podman',
  'io_r': 0.0,
  'io_w': 0.0,
  'key': 'name',
  'memory_usage': 282624.0,
  'name': '8d0f1c783def-infra',
  'network_rx': 0.0,
  'network_tx': 0.0,
  'pod_id': '8d0f1c783def',
  'pod_name': '8d0f1c783def-infra'},
 {'Command': ['/portainer'],
  'Created': '2022-10-29T14:59:10.266701439Z',
  'Id': '3abd51c615968482d9ccff5afc629f267f6dda113ed68b75b432615fae3b49fb',
  'Image': "['portainer/portainer-ce:2.9.3']",
  'StartedAt': '2023-04-08T07:56:52.939212149Z',
  'Status': 'running',
  'Uptime': '4 weeks',
  'cpu_percent': 0.0,
  'engine': 'docker',
  'io_r': None,
  'io_w': None,
  'key': 'name',
  'memory_usage': None,
  'name': 'portainer',
  'network_rx': None,
  'network_tx': None}]

===

Also the WebUI interface (VueJS) should be adapated.

===

The documentation is not up-to-date (talk about Docker and not Containers plugin).

[nicolargo]

Done in the last commit:

• Make the podman object serializable
• Update the WebUI (VueJS componant)
• Update the documentation

[nicolargo]

Last possible optimization: your patch to cache the Podman version (refreshed only every 5 minutes) makes the job but Glances is always slow to start (1 second longer than the previous version).

Is it useful to get the Docker and Podman version as it is not displayed in any UI ?

[RazCrimson]

@nicolargo

Makes sense. Let's drop the version calls in both docker and podman.

I could make the changes in another 8 hrs or if you are available now to get that done, go ahead.

Edit:

Another option would be to drop the version calls during the plugin initialisation and perform them once every 5 min during the updates.

What do you think of this?

[nicolargo]

@RazCrimson no need, i am currently working on it :)

[nicolargo]

Done in latest commit.

Every thing should be tested now :)

[RazCrimson]

@nicolargo
Everything working fine on my end.

Just one small UI inconsistency.

In 5e7cb1d the commit message says Containers status not have to be colorized but it seems to have removed the color for Uptime and this change is not applied to the webui equivalent.

Do you want to remove color for the status or uptime ? 🤔

[nicolargo]

Only for uptime.

[RazCrimson]

Cool, made the required changes on latest commit to develop!