v0.2.2

Fix for broken file lock on GCS.

v0.2.1

Fixed public GCS bucket issue

• For public buckets, use anonymous client instead of a client with default auth (google.auth.default()).

v0.2.0

v0.2.0

Added two methods to all URI classes.

• find_all_files(): Find all files (not sub-directories) on a directory.
• rmdir(): Delete all outputs on a directory.
  • local path: rm -rf-like deletion on the directory itself.
  • gs://, s3://: Find all files prefixed with the directory and delete them all.

Added CLI sub-command for each new method.

• See details with autouri --help.
• Sub-command find to wrap AutoURI(uri_dir).find_all_files().
• Sub-command rmdir to wrap AutoURI(uri_dir).rmdir().

Added

• pre-commit for better linting with Python Black style.

Unlocking locked files on Google Cloud Storage buckets.

If you see any failures related to timeout of a lock file (.lock) on gs://, then you can use the following command lines to unlock/delete it. Deleting a lock is generally safe if you don't have multiple processes trying to write on it.

$ gsutil retention temp release [LOCK_FILE]
$ gsutil rm -f [LOCK_FILE]

v0.1.4 (minor bug fix)

Bug fix for URL contents decoding

Traceback (most recent call last):
  File "/usr/local/bin/caper", line 13, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/caper/cli.py", line 58, in main
    c.submit()
  File "/usr/local/lib/python3.6/dist-packages/caper/caper.py", line 384, in submit
    imports_file = self.__create_imports_zip_file_from_wdl(tmp_dir)
  File "/usr/local/lib/python3.6/dist-packages/caper/caper.py", line 847, in __create_imports_zip_file_from_wdl
    if CaperWDLParser(self._wdl).zip_subworkflows(zip_file):
  File "/usr/local/lib/python3.6/dist-packages/caper/caper_wdl_parser.py", line 75, in zip_subworkflows
    root_zip_dir=tmp_d)
  File "/usr/local/lib/python3.6/dist-packages/caper/caper_wdl_parser.py", line 189, in __recurse_subworkflow
    depth=depth + 1)
  File "/usr/local/lib/python3.6/dist-packages/caper/caper_wdl_parser.py", line 146, in __recurse_subworkflow
    imports = self.find_imports()
  File "/usr/local/lib/python3.6/dist-packages/caper/caper_wdl_parser.py", line 39, in find_imports
    CaperWDLParser.RE_PATTERN_WDL_IMPORT)
  File "/usr/local/lib/python3.6/dist-packages/caper/caper_wdl_parser.py", line 85, in __find_val
    for line in u.read().split('\n'):
  File "/usr/local/lib/python3.6/dist-packages/autouri/httpurl.py", line 113, in read
    return b.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

v0.1.3

Removed --verbose since it already shows INFO level logs.
Refactored code structure and added individual logger to each module.

v0.1.2.1

Added parameters

• --debug and --verbose: better logging.
• Better directory parsing in CLI (e.g. . and ..)

Bug fixes

• Required python version: 3.5 -> 3.6 (due to variable annotation).
• Fix for dependency problem during pip install.

v0.1.1

Important notice

• To allow direct transfer between S3 and GCS, Autouri uses gsutil CLI. For this, GCSURI.USE_GSUTIL_FOR_S3 flag must be on. Otherwise Autouri will stream file transfer through a local machine. Old gsutil had a py3 compatibility issue. Users need to update gsutil to >= 4.47.

Added circleci (integrated testing)

Added CLI

• See autouri --help for details

Improved locking to prevent race condition

• Tested up to 10 (50 for local file) threads competing to write on the same file.

Others

• Soft-linking for local paths (AbsPath)
• Converting relative path into abs path (AbsPath.get_abspath_if_exists())
  • AbsPath only allows absolute path as its name says. so added a helper function for such conversion so that apps using Autouri can use it.
• Detailed logging (INFO, DEBUG)
• Can hide flag for AutoURI.cp() and AutoURI.localize() (return_flag=False).
• Can get URL from public GCS/S3 bucket (without presigning)
  • For GCSURI, S3URI only. Formatting for a public URL looks like the following:

     _GCS_PUBLIC_URL_FORMAT = 'http://storage.googleapis.com/{bucket}/{path}'
    _S3_PUBLIC_URL_FORMAT = 'http://{bucket}.s3.amazonaws.com/{path}'
    

  • This is different from a presigned URL. It's for a file on a bucket with public access. If users use this function on a private bucket file then this function will still return a URL but public access to this URL will be limited (403).

v0.1.0

First release