erofs-utils
===========

erofs-utils includes user-space tools for erofs filesystem images.
Currently only mkfs.erofs is available.

mkfs.erofs
----------

It can create 2 primary kinds of erofs images: (un)compressed.

 - For compressed images, it's able to use several compression
   algorithms, but lz4(hc) are only supported due to the current
   linux kernel implementation.

 - For uncompressed images, it can decide whether the last page of
   a file should be inlined or not properly [1].

Dependencies
~~~~~~~~~~~~

 lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.0+ recommended

How to build for lz4-1.9.0 or above
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To build you can run the following commands in order:

::

	$ ./autogen.sh
	$ ./configure
	$ make

mkfs.erofs binary will be generated under mkfs folder. There are still
some issues which affect the stability of LZ4_compress_destSize()
* they have impacts on lz4 only rather than lz4HC * [3].

How to build for lz4-1.8.0~1.8.3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For these old lz4 versions, lz4hc algorithm cannot be supported without
lz4 static libary due to LZ4_compress_HC_destSize unstable api usage,
which means only lz4 algrithm is available if lz4 static library isn't found.

On Fedora, static lz4 can be installed using:

	yum install lz4-static.x86_64

However, it's not recommended to use those versions since there were bugs
in these compressors, see [2] [3] as well.

How to generate erofs images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Currently lz4 and lz4hc are available for compression, e.g.
 $ mkfs.erofs -zlz4hc foo.erofs.img foo/

Or leave all files uncompressed as a option:
 $ mkfs.erofs foo.erofs.img foo/

How to generate legacy erofs images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Decompression inplace and compacted indexes have been introduced in
linux-5.3, which are not backward-compatible with older kernels.

In order to generate _legacy_ erofs images for old kernels,
add "-E legacy-compress" to the command line, e.g.

 $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/


Known issues
~~~~~~~~~~~~

1. LZ4HC cannot compress long zeroed buffer properly with
   LZ4_compress_HC_destSize()
   lz4/lz4#784

Obsoleted erofs.mkfs
~~~~~~~~~~~~~~~~~~~~

There is an original erofs.mkfs version developped by Li Guifu,
which was replaced by the new erofs-utils implementation.

git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs

It may still be useful since new erofs-utils has not been widely used in
commercial products. However, if that happens, please report bug to us
as well.

Contribution
------------

erofs-utils is a GPLv2+ project as a part of erofs file system,
feel free to send patches or feedback to us.

To:
  linux-erofs mailing list   <[email protected]>
  Li Guifu                   <[email protected]>
  Miao Xie                   <[email protected]>
  Fang Wei                   <[email protected]>

Cc:
  Gao Xiang                  <[email protected]>
  Chao Yu                    <[email protected]>

Comments
--------

[1] According to the erofs on-disk format, the last page of files could
    be inlined aggressively with its metadata in order to reduce the I/O
    overhead and save the storage space.

[2] There was a bug until lz4-1.8.3, which can crash erofs-utils randomly.
    Fortunately bugfix by our colleague Qiuyang Sun was merged in lz4-1.9.0.

    For more details, please refer to
    https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82

[3] There are many crash fixes merged to lz4 1.9.2 for LZ4_compress_destSize(),
    and I once ran into some crashs due to those issues.
    * Again lz4HC is not effected for this section. *

    [LZ4_compress_destSize] Allow 2 more bytes of match length
    https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda

    [LZ4_compress_destSize] Fix rare data corruption bug
    https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1

    [LZ4_compress_destSize] Fix overflow condition
    https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a

    [LZ4_compress_destSize] Fix off-by-one error in fix
    https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333

    [LZ4_compress_destSize] Fix off-by-one error
    https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba

    since upstream lz4 doesn't have stable branch for old versions, it's
    preferred to use latest upstream lz4 library (although some regressions
    could happen since new features are also introduced to latest upstream
    version as well) or backport all stable bugfixes to old stable versions,
    e.g. our unoffical lz4 fork: https://github.com/erofs/lz4