Command-line utilities for S3-compatible Object Storage
These utilities provide a command-line interface to an Amazon S3-compatible file/object storage services, also known as an Object Store.
pip install umobj
or
python setup.py install
For convenience, consider setting a few environmental variables that umobj will use to authenticate you.
- OBJ_ACCESS_KEY_ID - The user's access key
- OBJ_SECRET_ACCESS_KEY - The user's secret key
- OBJ_SERVER - The object server endpoint
For example, if you use the bash shell, you can add the following to your
.bashrc or .bash_profile.
export OBJ_ACCESS_KEY_ID="<YOUR_ACCESS_KEY>"
export OBJ_SECRET_ACCESS_KEY="<YOUR_SECRET_KEY>"
export OBJ_SERVER="obj.umiacs.umd.edu"You can optionally pass the --access_key,
--secret_key and --server switches to the utilities
to get the same behavior.
These utilities use a common syntax for specifying the bucket and an optional
key or path. The bucket and key are separated by a :. For example,
my_bucket:bar.txt denotes the bucket my_bucket and the key
bar.txt. For more information on a specific utility, you can pass the
flag -h or --help for help.
To list buckets and keys in the Object Store you can use the lsobj command. If given without an argument, it will list your buckets.
$ lsobj
bob
test
zetaYou can then give it a bucket name to list the contents within that bucket.
% lsobj -l test
-rwx--- 2013-10-04T15:25:09.000Z 1.8 KB UMIACSCA.pem
-rwx--- 2013-10-04T15:25:24.000Z 311.4 KB cover.jpg
-rwx--- 2013-10-04T15:27:39.000Z 0.0 b foo/
-rwx--- 2013-10-04T15:27:40.000Z 0.0 b foo/bar
-rwx--- 2013-10-04T15:25:32.000Z 31.8 KB screenshot.jpg
-rwx--- 2013-10-04T15:26:48.000Z 8.3 KB thunderbird.xpm
================================================================================
TOTAL: 353.3 KB 6 FilesIn an Object Store, there are only buckets and keys. This means that any
apparent directory structure is only emulated using the UNIX / character
in a key name. Any key in the bucket ending in a / will be interpreted
as a directory. You can also list only subdirectories with the lsobj
utility.
% lsobj -l test:foo/
foo/
-rwx--- 2013-10-04T15:27:39.000Z 0.0 b foo/
-rwx--- 2013-10-04T15:27:40.000Z 0.0 b foo/bar
================================================================================
TOTAL: 0.0 b 2 Filesmkobj creates buckets and directories in the Object Store.
Please note that bucket names are unique in the Object Store, so you may very well get an error back that the name has already been taken.
% mkobj foo
Created bucket foo.
% lsobj
bob
foo
test
zetaYou can also create directories to group your data.
% mkobj foo:bar/
% lsobj -l foo
-rwx--- 2013-10-04T15:38:38.000Z 0.0 b bar/
================================================================================
TOTAL: 0.0 b 1 FileCopying files to the Object Store can be done per-file or recursively both to and from the Object Store.
To copy a single file to the Object Store you can use cpobj and
specify a bucket with a trailing : (with an optional additional
path).
% cpobj test.png foo:
100% |#########################################################################|
% lsobj -l foo
-rwx--- 2013-10-04T15:38:38.000Z 0.0 b bar/
-rwx--- 2013-10-07T20:06:48.000Z 18.3 KB test.png
================================================================================
TOTAL: 18.3 KB 2 FilesTo copy a directory of files you will need to use the -r or
--recursive flags.
% lsobj foo
================================================================================
TOTAL: 0.0 b 0 Files
% cpobj -r /tmp/stuff foo:stuff
100% |#########################################################################|
100% |#########################################################################|
100% |#########################################################################|
% lsobj -l foo
-rwx--- 2013-10-08T00:13:52.000Z 0.0 b stuff/
-rwx--- 2013-10-08T00:13:55.000Z 26.0 KB stuff/WindowsSecurity.png
-rwx--- 2013-10-08T00:13:54.000Z 226.5 KB stuff/changepass.tiff
-rwx--- 2013-10-08T00:13:55.000Z 18.3 KB stuff/test.png
================================================================================
TOTAL: 270.8 KB 9 FilesYou can delete your buckets and keys with rmobj. It can take a bucket
and work recursively to delete all the files and the bucket itself.
It can also just delete specific files in a bucket.
% rmobj -r foo
Are you sure you want to remove all the contents of the bucket 'foo'? [yes/no] yes
Do you want to remove the bucket 'foo'? [yes/no] no% lsobj -l foo
-rwx--- 2013-10-09T18:44:20.000Z 1.0 KB setup.cfg
-rwx--- 2013-10-09T18:44:17.000Z 781.0 b setup.py
-rwx--- 2013-10-09T18:44:09.000Z 289.0 b test-requirements.txt
================================================================================
TOTAL: 2.0 KB 3 Files
% rmobj foo:setup.cfg foo:setup.py
% lsobj -l foo
-rwx--- 2013-10-09T18:44:09.000Z 289.0 b test-requirements.txt
================================================================================
TOTAL: 289.0 b 1 FileYou can also remove directories within a bucket. To do so, you will need to
pass the -r flag. This will prompt once for the removal of every
key under that directory unless the -f flag is passed as well.
Specifying -i will prompt for every file.
% lsobj foo
-rwx--- 2013-10-10T21:58:17.000Z 0.0 b research/
-rwx--- 2013-10-10T21:58:17.000Z 0.0 b research/papers/
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/papers/paper1.tex
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/papers/paper2.tex
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/posters/
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/posters/poster1.tex
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/posters/poster2.tex
================================================================================
TOTAL: 0.0 b 7 Files
% rmobj -rf foo:research/posters/
% lsobj foo
-rwx--- 2013-10-10T21:58:17.000Z 0.0 b research/
-rwx--- 2013-10-10T21:58:17.000Z 0.0 b research/papers/
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/papers/paper1.tex
-rwx--- 2013-10-10T21:58:18.000Z 0.0 b research/papers/paper2.tex
================================================================================
TOTAL: 0.0 b 4 FilesYou can use a glob character to delete all bucket contents, but not the bucket itself.
% rmobj -rf foo:*
% lsobj foo
================================================================================
TOTAL: 0.0 b 0 FilesUse chobj to set permissions/ACLs on buckets and keys.
% chobj -m clear -p liam:FULL_CONTROL -p derek:FULL_CONTROL foo:test-requirements.txtYou can use syncobj to synchronize files and directories into the Object
Store. It does this by comparing the checksum between the local and remote
versions of the key. This does not follow symlinks and will warn when it
encounters files (character, block, fifo, etc...) that it can not archive.
% syncobj -r devel/puppet test:
100% |#########################################################################|Use catobj to print the contents of a key to stdout. This may be useful
for streaming the content of a file to another command for some sort of
processing without having to write the key to an intermediary file. If any
errors are printed in the process of running this command, you can be assured
that they will be printed to stderr, and should not interfere with what's being
printed to stdout.
% catobj foo:something.txt
Fallaces sunt rerum species. Mutantur omnia nos et mutamur in illis.Use streamobj to read the contents of a data stream that has been piped from
stdout and upload it to a bucket. This allows you to send large amounts of data to
a bucket without saving it all on disk or in memory first. A name for the file
must be specified using the -n flag because a name can't be inferred from the
data stream.
% stream_source | streamobj -n filename foo:somedirectoryUse cmpobj to check the md5 sum of a key in a bucket or compare the md5 sums of a local directory to a bucket. The key is downloaded in order for the correct md5 hash to be generated. Bagit archives can also be verified, which is done by retrieving the manifest and comparing the expected checksums to the ones computed by downloading each key.
% cmpobj mybucket:foo.txt
9029668a43dfa60f1f267eea59111bbcUse webobj to configure web configurations of a bucket. Currently, this
supports setting a S3 website configuration such that bucket can be served as a
static site.
% webobj -m create -c website --index=index.html --error=error.html mybucket
% webobj -m examine -c website mybucket
Index: index.html
Error Key: error.html
% webobj -m delete -c website mybucket
These utilities and libraries all work with Python 2.7+ and Python 3.6+
They require the following packages:
- Boto > 2.49.0
- FileChunkIO
- Progressbar
- qav
The official repository for umobj is hosted in the UMIACS Gitlab service. Please open new issues in the Gitlab tracker.
umobj - Command-line utilities for S3-compatible Object Storage
Copyright (C) 2017 UMIACS
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Email:
[email protected]