Add HTJ2K Compressor

[palemieux]

WIP DO NOT MERGE

This patch fulfills my action item from the September 19 TSC meeting.

Introduction

This patch proposes to add support for High-Throughput JPEG 2000 (HTJ2K) compression to OpenEXR -- HTJ2K is JPEG 2000 with the HT block coder standardized in Rec. ITU-T T.814 | ISO/IEC 15444-15. As detailed at Evaluating HTJ2K as a compressor option for OpenEXR, HTJ2K demonstrates significant improvements in both speed and file size over other OpenEXR compressors. Furthermore, HTJ2K is a worldwide standard that benefits from a diversity of implementations, builds on JPEG 2000, which is in broad use in studio workflows, is appropriate for long-term preservation, has both lossy and lossless modes and supports integer and floating point samples. Finally, the unique features of HTJ2K could be powerful additions to high-performance OpenEXR workflows where the full image may not always be viewed at its full resolution.

The patch currently defines four compressors:

• HT: Lossless coding using HTJ2K. Each chunk is the full image frame and the open-source OpenJPH library is used.
• HT256: Lossless coding using HTJ2K. Each chunk is 256 lines and the open-source OpenJPH library is used.
• HTK: Lossless coding using HTJ2K. Each chunk is the full image frame and the commercial KDU library is used.
• HTK256: Lossless coding using HTJ2K. Each chunk is 256 lines and the commercial KDU library is used.

Questions

• can a compressor store configuration information once in a file, so that the same information is available for each chunk to both the compressor and decompressor?

Notes

• HT and HTK Compressors are identical and use the open-source OpenJPH library is the Kakadu SDK is not found
• Ultimately there will be a single lossless HTJ2K Compressor. The HT and HTK Compressors are currently offered to ensure interop, and facilitate comparisons, between OSS and commercial options.
• The full-frame and 256-line chunk options are currently offered to evaluate the impact of sub-frame chunking on coding efficiency performance.

Todo

• add support for 32-bit samples to OpenJPH
• explore adding a progressive decoding API to OpenEXR
• explore tradeoff of chunk-size (currently set to 256) on compression efficiency, single-threaded/multi-threaded throughput and partial decoding (low-resolution and/or region of interest)
  • can chunking can be replaced with full-frame coding and J2K accessibility features? This is more important in lossy coding to avoid edge effects.
  • select a chunk size, i.e., is 256-line a reasonable value?
  • can J2K tiles be used instead of independent codestreams to remove JPEG 2000 codestream header overhead?

[linux-foundation-easycla]

• ❌ - login: @palemieux / name: Pierre-Anthony Lemieux . The commit (ecf213f, 21a653e, af0c7e9, 2ee9f7a, dbdd615, 439cfe5, 107fbbd, a6be4cb, 7003805, 6eda711, af25ab0) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.
• ❌ - login: @michaeldsmith . The commit (3cc1729) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[cary-ilm]

The CI is failing in the validate step, which is attempting to link a test program against the library. The error message indicate the link is failing with unresolved symbols, possibly because that cmake configuration is missing the new dependency.

However, this part of the CI has been rewritten in the PR I just merged. Can you rebase your branch onto the current main branch now? It may still fail, but hopefully it will be easier to resolve then at least.

[palemieux]

Can you rebase your branch onto the current main branch now? It may still fail, but hopefully it will be easier to resolve then at least.

@cary-ilm Done.

[cary-ilm]

Do you really mean for these to be commented out? I think they're leading to a failure with the CI validate step.

[cary-ilm]

The failure in the "Validate" step of the linux build is along the lines of what we discussed in the TSC meeting today, the check to make sure "make install" installs just the right files. This output from the logs appears to indicate that the cmake configuration is causing the openjph library and headers to be installed, when that shouldn't happen, it'll cause problems for downstream projects:

Error: The following files were installed but were not expected:
  bin/ojph_compress
  bin/ojph_expand
  include/OpenEXR/ImfHTCompressor.h
  include/OpenEXR/ImfHTKCompressor.h
  include/openjph/ojph_arch.h
  include/openjph/ojph_arg.h
  include/openjph/ojph_base.h
  include/openjph/ojph_codestream.h
  include/openjph/ojph_defs.h
  include/openjph/ojph_file.h
  include/openjph/ojph_mem.h
  include/openjph/ojph_message.h
  include/openjph/ojph_params.h
  include/openjph/ojph_version.h
  lib64/libopenjph.so
  lib64/libopenjph.so.0.18
  lib64/libopenjph.so.0.18.2
  lib64/pkgconfig/openjph.pc

Try to make sure your cmake "fetch" of openjph mirror as close as possible how the current setup handles libdeflate and Imath. I had a similar issue with them originally but the current setup should handle the the fetch properly.

[palemieux]

Thanks! What is the best way to run that validate step locally?

[palemieux]

@cary-ilm I remember now... I could never fix the following errors:

[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "Iex" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "IlmThread" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXRCore" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXR" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXRUtil" which requires target "openjph" that is not in any export set.

It seems to impose some requirements on openjph, which I could not find a solution for.

[cary-ilm]

I'm not enough of a cmake expert to immediately know how to resolve this. Since this is a work in progress, you could just disable the CI "validate" step entirely for now, just edit the workflow file on your branch and add a "if: false" line, or just delete it. We'll need it resolved eventually before merging, but at least that will allow you to get on with other priorities.

I'm happy to help resolve this, but I won't have much time to look into it until mid-January.