problem deleting image or tag 16.6.1 #2867
Comments
Gabrielandre02
changed the title
|
Hi, We are facing the same issue. In addition to the reported details, the gitlab registry cleanup policy when enabled will generate plenty of logs in both sides due to images not being deleted. We got several instances, but the one we are facing this issue in is using registry 2.8.x instead of 2.7.x. It seems related to several things :
|
|
Hi, unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry. However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image. Did anyone try this? |
|
Hi @sachilles,
This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?
I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\ |
|
I'm also experiencing errors when deleting images from the container registry. I noticed the problem after upgrading to 16.6. I suppose it is connected to Gitlab forking the Docker distribution registry. In my case I'm currently using |
|
I actually stumbled on https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1471217687 that consider my setup (using In that thread is suggested to switch to the I tried the switch
registry:
- image: registry:2.7.1
+ image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.86.1-gitlab"
volumes:
+ - ./registry:/etc/docker/registry
- /data/gitlab/registry:/registry
- ./certs:/certsI experienced an issue with file system permission during deletion, but I haven't figured out the correct user to switch to. I indeed temporary solved it by allowing write to everyone. |
|
Hi @ymazzer,
I have the same problem. It doesn't matter if I try to delete container images via the GitLab web interface or via the cleanup policy. The result is the same. (I'm using the "official" self-hosted docker registry in version 2.8.1.)
Okay, I guess the proposed way is to migrate the content of the "official" self-hosted docker registry (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459). |
|
Dear @avvertix, I found somewhere (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459) that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry, as several changes were included after the container registry was forked by the GitLab maintainers. (My first attempt to replace the Docker registry with the GitLab container registry failed). However, please make sure to backup the entire Docker registry data directories before the necessary migration. Once I (or someone else) is successful, we will update the corresponding configuration files and documentation. |
|
@avvertix, Thanks |
Dear @sachilles I also saw the link you shared, but my interpretation is that the issue refers to the migration from filesystem metadata (or object storage) to the database storage for the images metadata. As far as I understood migrating to the database metadata is in beta and not yet fully complete that's why I didn't try that way. I saw a workaround following the feedback issue on the use of external registries. I totally agree that my approach is just a workaround, but probably could make easier the migration path until all official migrations tools are available. |
@Gabrielandre02 could you tell me more about why you had to clean all the images, as in the end I was able to use the same storage directory by ensuring that the user within the docker image had access to the filesystem. (btw seems that in my case the user on the host os didn't have write permission on a parent folder in the storage, after I ensured that all the folders/files are writeable by the current user everything worked without changing nothing) |
|
Hi @sachilles @avvertix, I just tried out the simple following procedure:
# docker-compose.yml
# [...]
registry:
#image: registry:2.8
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
restart: unless-stopped
expose:
- "5000"
ports:
- "127.0.0.1:5000:5000"
volumes:
- ./registry-config.yml:/etc/docker/registry/config.yml # this has changed as described before by @avvertix
- /opt/gitlab/disk/data/shared/registry:/var/lib/registry/docker/registry
- /etc/ssl/private/registry.my-registry.com:/certs
environment:
REGISTRY_LOG_LEVEL: info
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
REGISTRY_AUTH_TOKEN_REALM: https://my-gitlab.com/jwt/auth
REGISTRY_AUTH_TOKEN_SERVICE: container_registry
REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/cert.crt
REGISTRY_STORAGE_DELETE_ENABLED: "true"
networks:
- web
# [...]Hope this will help. PS: I obviously made a backup before doing the operation. |
|
Will there be an update of the docs and an “official” way for migrating to the new docker image? |
|
@etlam Yes, the project-related docs will be updated possible. |
Whenever I try to delete an image or a tag, I'm getting this error but the logs in the registry aren't showing anything, nor are those in Gitlab.
This installation was via a Linux package, I made the backup and left it in a new configuration with Docker Swarm, using Traefik as a reverse proxy
LOGS GITLAB:
LOGS REGISTRY:
{"auth.user.name":"","err.code":"digest invalid","err.message":"provided digest did not match uploaded content","go.version":"go1.20.8","http.request.host":"registry_registry:5000","http.request.id":"09c7cdce-048b-4c9d-89e0-c368efc1b13f","http.request.method":"DELETE","http.request.remoteaddr":"10.0.1.232:48710","http.request.uri":"/v2/compmon/application/panorama_tunnel_ssh/nginx/manifests/alpine","http.request.useragent":"GitLab/16.6.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"7.641504ms","http.response.status":400,"http.response.written":98,"level":"error","msg":"response completed with error","time":"2023-12-15T12:18:36.491871799-03:00","vars.name":"compmon/application/panorama_tunnel_ssh/nginx","vars.reference":"alpine"} registry_registry.1.og01zgrjx0v9@SRVDOCKER02 | 10.0.1.232 - - [15/Dec/2023:12:18:36 -0300] "DELETE /v2/compmon/application/panorama_tunnel_ssh/nginx/manifests/alpine HTTP/1.1" 400 98 "" "GitLab/16.6.1"application_json.log
{"severity":"INFO","time":"2023-12-15T15:17:23.240Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"5d8fd23d3c5c78765a575cda99fdafd7","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":85,"container_repository_path":"compmon/application/panorama_tunnel_ssh/nginx","project_id":189,"third_party_cleanup_tags_service":true} {"severity":"ERROR","time":"2023-12-15T15:17:23.298Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"5d8fd23d3c5c78765a575cda99fdafd7","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":85,"project_id":189,"message":"could not delete tags: alpine"}The text was updated successfully, but these errors were encountered: