Command duplicate, option -ignore-extension doesn't always work

[antonio-ivanovski]

I have two images being reported as duplicates from Immich but not from immich-go

Left (original)

{
  "id": "a36a22be-a149-4bf4-9382-935f84406ae2",
  "deviceAssetId": "1000006383",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "b45cc9ab602dac461c6284bdb7015c44c9f8270dee1ffbd03cab01c1005fea8a",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-22/20240422_163537.heic",
  "originalFileName": "20240422_163537.heic",
  "originalMimeType": "image/heic",
  "resized": true,
  "thumbhash": "XxgKDQJwaIaZeHd3qJiJZx9d12BX",
  "fileCreatedAt": "2024-04-22T14:35:37.525Z",
  "fileModifiedAt": "2024-04-22T14:35:38.000Z",
  "localDateTime": "2024-04-22T16:35:37.525Z",
  "updatedAt": "2024-06-19T09:32:34.001Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "Galaxy S24+",
    "exifImageWidth": 4000,
    "exifImageHeight": 3000,
    "fileSizeInByte": 4652170,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-22T14:35:37.525Z",
    "modifyDate": "2024-04-22T14:35:37.525Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 2.2,
    "focalLength": 2.2,
    "iso": 1250,
    "exposureTime": "1/50",
    "latitude": 42.0109921997222,
    "longitude": 21.3718387997222,
    "city": "Krušopek",
    "state": "Grad Skopje",
    "country": "The Republic of North Macedonia",
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": null,
  "people": [],
  "checksum": "ltAFmoRS9V/BD/17XcuGnkCRrs0=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "021ce122-723b-4461-8b95-f1cb063b1cb0"
}

Right (Google Photos)

{
  "id": "42c9d832-94aa-48e6-b4ff-45e310419855",
  "deviceAssetId": "20240422_163537.jpg-2414264",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "asus",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-22/20240422_163537.jpg",
  "originalFileName": "20240422_163537.jpg",
  "originalMimeType": "image/jpeg",
  "resized": true,
  "thumbhash": "XxgKDQJwaIaZeHd3qJiJZx9d12BX",
  "fileCreatedAt": "2024-04-22T14:35:37.525Z",
  "fileModifiedAt": "2024-04-22T16:35:37.000Z",
  "localDateTime": "2024-04-22T16:35:37.525Z",
  "updatedAt": "2024-06-19T09:36:30.906Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "Galaxy S24+",
    "exifImageWidth": 4000,
    "exifImageHeight": 3000,
    "fileSizeInByte": 2414264,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-22T14:35:37.525Z",
    "modifyDate": "2024-04-22T14:35:37.525Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 2.2,
    "focalLength": 2.2,
    "iso": 1250,
    "exposureTime": "1/50",
    "latitude": 42.0109921997222,
    "longitude": 21.3718387997222,
    "city": "Krušopek",
    "state": "Grad Skopje",
    "country": "The Republic of North Macedonia",
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": null,
  "people": [],
  "checksum": "0TxA/Xw3FsR/k6zS6LJfM+cqf1g=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "021ce122-723b-4461-8b95-f1cb063b1cb0"
}

I am running immich-go with:

./immich-go -server=http://localhost:2283 -key=<my key> duplicate -yes -ignore-tz-errors -ignore-extension

There are quite few more instances of this where duplicate is not detected. Some are videos.

Let me know if you need more samples/data.

[antonio-ivanovski]

Here is another example just for testing purposes

{
  "id": "a09d31f4-4739-4388-bb59-eb2e709f04a6",
  "deviceAssetId": "20240428_162300.heic-1098177",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "casaos",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-28/20240428_162300.heic",
  "originalFileName": "20240428_162300.heic",
  "originalMimeType": "image/heic",
  "resized": true,
  "thumbhash": "ZLYFMwpwnneMeXPleHCABzc=",
  "fileCreatedAt": "2024-04-28T14:23:00.937Z",
  "fileModifiedAt": "2024-04-28T14:23:00.000Z",
  "localDateTime": "2024-04-28T16:23:00.937Z",
  "updatedAt": "2024-06-20T14:44:12.741Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "SM-N985F",
    "exifImageWidth": 1868,
    "exifImageHeight": 4000,
    "fileSizeInByte": 1098177,
    "orientation": "1",
    "dateTimeOriginal": "2024-04-28T14:23:00.937Z",
    "modifyDate": "2024-04-28T14:23:00.937Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 1.8,
    "focalLength": 7,
    "iso": 50,
    "exposureTime": "1/229",
    "latitude": null,
    "longitude": null,
    "city": null,
    "state": null,
    "country": null,
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": "b2f9e882-ced5-4d85-be2e-7e7408811629",
  "people": [],
  "checksum": "U0STAk9xu7EyQ/7Bl7j97jtMGAA=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "0544b3d3-d141-4058-8e92-90e997011ed9"
}

{
  "id": "2b2efb7e-92e4-467b-a44b-c7cea8b8ab7c",
  "deviceAssetId": "20240428_162300.jpg-4494797",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "asus",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-28/20240428_162300.jpg",
  "originalFileName": "20240428_162300.jpg",
  "originalMimeType": "image/jpeg",
  "resized": true,
  "thumbhash": "ZLYFMwpwnneMeXPleHCABzc=",
  "fileCreatedAt": "2024-04-28T14:23:00.937Z",
  "fileModifiedAt": "2024-04-28T16:23:00.000Z",
  "localDateTime": "2024-04-28T16:23:00.937Z",
  "updatedAt": "2024-06-20T14:44:12.741Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "SM-N985F",
    "exifImageWidth": 4000,
    "exifImageHeight": 1868,
    "fileSizeInByte": 4494797,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-28T14:23:00.937Z",
    "modifyDate": "2024-04-28T14:23:00.937Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 1.8,
    "focalLength": 7,
    "iso": 50,
    "exposureTime": "1/229",
    "latitude": null,
    "longitude": null,
    "city": null,
    "state": null,
    "country": null,
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": "4843608a-fbe5-415e-bb5e-76b42f94bba5",
  "people": [],
  "checksum": "oE3/8nkj43R3QK5hvnos4RL30M4=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "0544b3d3-d141-4058-8e92-90e997011ed9"
}

One issue I see is that the JPEG image is larger in size then the HEIC image

[simulot]

The duplicate command is meant to remove duplicates caused by the compressed version of the photo stored by Google Photo. Same name, same date, but different file size. Immich-go will remove the most compressed files: the smallest file.

In your case the heic version would be discarded. Maybe you would prefer keeping the HEIC one, knowing that HEIC is better thant JPG.

One issue I see is that the JPEG image is larger in size then the HEIC image

HEIC format is reputed to produce smaller files than JPG for the same quality. That's your 1st case that is unusual.