OSSFS is a Python-based interface for file systems that enables interaction with OSS (Object Storage Service). Through OSSFS, users can utilize fsspec's standard API to operate on OSS objects
You can install OSSFS via pip from PyPI:
$ pip install ossfsUp-to-date package also provided through conda-forge distribution:
$ conda install -c conda-forge ossfsHere is a simple example of locating and reading an object in OSS.
import ossfs
fs = ossfs.OSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
fs.ls('/dvc-test-anonymous/LICENSE')
[{'name': '/dvc-test-anonymous/LICENSE',
'Key': '/dvc-test-anonymous/LICENSE',
'type': 'file',
'size': 11357,
'Size': 11357,
'StorageClass': 'OBJECT',
'LastModified': 1622761222}]
with fs.open('/dvc-test-anonymous/LICENSE') as f:
... print(f.readline())
b' Apache License\n'For more use case and apis please refer to the documentation of fsspec
Async OSSFS is a variant of ossfs that utilizes the third-party async OSS backend aiooss2, rather than the official sync one, oss2. Async OSSFS allows for concurrent calls within bulk operations, such as cat, put, and get etc even from normal code, and enables the direct use of fsspec in async code without blocking. The usage of async OSSFS is similar to the synchronous variant; one simply needs to replace OSSFileSystem with AioOSSFileSystem need to do is replacing the OSSFileSystem with the AioOSSFileSystem
import ossfs
fs = ossfs.AioOSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
print(fs.cat('/dvc-test-anonymous/LICENSE'))
b' Apache License\n'
...Although aiooss2 is not officially supported, there are still some features that are currently lacking. However, in tests involving the put/get of 1200 small files, the async version of ossfs ran ten times faster than the synchronous variant (depending on the pool size of the concurrency).
| Task | time cost in (seconds) |
|---|---|
| put 1200 small files via OSSFileSystem | 35.2688 (13.53) |
| put 1200 small files via AioOSSFileSystem | 2.6060 (1.0) |
| get 1200 small files via OSSFileSystem | 32.9096 (12.63) |
| get 1200 small files via AioOSSFileSystem | 3.3497 (1.29) |
Contributions are very welcome. To learn more, see the Contributor Guide.
Distributed under the terms of the Apache 2.0 license, Ossfs is free and open source software.
If you encounter any problems, please file an issue along with a detailed description.