Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python bindings #393

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
name: Build and Publish Python Package

on:
push:
tags:
- 'v*'

# Add top-level permissions block
permissions:
id-token: write
contents: read

jobs:
macos:
runs-on: macos-latest
# Add permissions to job
permissions:
id-token: write
contents: read
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
target: [x86_64, aarch64]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
args: --release --out dist
sccache: 'true'
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: dist/*.whl
if-no-files-found: error

windows:
runs-on: windows-latest
# Add permissions to job
permissions:
id-token: write
contents: read
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
target: [x64]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: ${{ matrix.target }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
args: --release --out dist
sccache: 'true'
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: dist/*.whl
if-no-files-found: error

linux:
runs-on: ubuntu-latest
# Add permissions to job
permissions:
id-token: write
contents: read
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
target: [x86_64]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --user cffi
python -m pip install --user patchelf
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
manylinux: auto
args: --release --out dist
sccache: 'true'
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: dist/*.whl
if-no-files-found: error

sdist:
runs-on: ubuntu-latest
# Add permissions to job
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
command: sdist
args: --out dist
- name: Upload sdist
uses: actions/upload-artifact@v3
with:
name: wheels
path: dist/*.tar.gz
if-no-files-found: error

release:
name: Release
runs-on: ubuntu-latest
needs: [macos, windows, linux, sdist]
# Keep existing permissions
permissions:
id-token: write
contents: read
steps:
- uses: actions/download-artifact@v3
with:
name: wheels
path: dist
- name: Display structure of downloaded files
run: ls -R dist
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: dist/
verbose: true
print-hash: true
11 changes: 10 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@ license = "GPL-3.0"
name = "self_encryption"
readme = "README.md"
repository = "https://github.com/maidsafe/self_encryption"
version = "0.30.0"
version = "0.30.263"

[features]
default = []
python = ["pyo3/extension-module"]

[dependencies]
aes = "~0.8.1"
Expand All @@ -23,6 +27,7 @@ num_cpus = "1.13.0"
itertools = "~0.10.0"
tempfile = "3.6.0"
xor_name = "5.0.0"
pyo3 = { version = "0.19", optional = true, features = ["extension-module"] }

[dependencies.brotli]
version = "~3.3.0"
Expand Down Expand Up @@ -64,3 +69,7 @@ name = "basic_encryptor"
[[bench]]
name = "lib"
harness = false

[lib]
name = "self_encryption"
crate-type = ["cdylib", "rlib"]
101 changes: 80 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,47 +11,106 @@ Self encrypting files (convergent encryption plus obfuscation)

## Overview

A version of [convergent encryption](http://en.wikipedia.org/wiki/Convergent_encryption) with an additional obfuscation step. This pattern allows secured data that can also be [de-duplicated](http://en.wikipedia.org/wiki/Data_deduplication). This library presents an API that takes a set of bytes and returns a secret key derived from those bytes, and a set of encrypted chunks.
A reverse function is provided, where the pair returned from encryption (secret key and encrypted chunks) is passed in, returning the original bytes.
There is also the possibility to seek the original bytes in the contents of the encrypted chunks, by calling the seek helper function to produce information used to locate the relevant chunks, and then call the decrypt_range api with the chunks, the secret key and seek information from the previous step.
A version of [convergent encryption](http://en.wikipedia.org/wiki/convergent_encryption) with an additional obfuscation step. This pattern allows secured data that can also be [de-duplicated](http://en.wikipedia.org/wiki/Data_deduplication). This library presents an API that takes a set of bytes and returns a secret key derived from those bytes, and a set of encrypted chunks.

There is an important aspect to note:

This library provides very secure encryption of the data, and the returned encrypted chunks can be considered as safe as if encrypted by any other modern encryption algorithm.
**However** the returned secret key **requires the same secure handling as would be necessary for any secret key**.
**Important Security Note**: While this library provides very secure encryption of the data, the returned secret key **requires the same secure handling as would be necessary for any secret key**.

![image of self encryption](https://github.com/maidsafe/self_encryption/blob/master/img/self_encryption.png?raw=true)

## Video of the process
[self_encryption process and use case video](https://www.youtube.com/watch?v=Jnvwv4z17b4)
## Documentation
- [Self Encrypting Data Whitepaper](https://docs.maidsafe.net/Whitepapers/pdf/SelfEncryptingData.pdf)
- [Process Overview Video](https://www.youtube.com/watch?v=Jnvwv4z17b4)

## Usage

The library can be used through either Rust or Python interfaces.

### Rust Usage

#### Installation

Add this to your `Cargo.toml`:
```toml
[dependencies]
self_encryption = "0.30"
```

#### Example Using Basic Encryptor

```bash
# Encrypt a file
cargo run --example basic_encryptor -- -e <full_path_to_any_file>

# Decrypt a file
cargo run --example basic_encryptor -- -d <full_path_to_secret_key> <full_destination_path_including_filename>
```

### Python Usage

#### Installation

```bash
pip install self-encryption
```

#### Basic In-Memory Example

```python
from self_encryption import encrypt_bytes, decrypt_chunks

# Create test data (must be at least 3 bytes)
data = b"Hello World" * 1024

## Whitepaper
# Encrypt the data
data_map, chunks = encrypt_bytes(data)

[Self Encrypting Data](https://docs.maidsafe.net/Whitepapers/pdf/SelfEncryptingData.pdf), David Irvine, First published September 2010, Revised June 2015.
# Decrypt and verify
decrypted = decrypt_chunks(data_map, chunks)
assert data == decrypted
```

## Examples
#### File-Based Example with Chunk Storage

### Using `self_encryptor`
```python
from self_encryption import encrypt_file, decrypt_from_files

This library splits a set of bytes into encrypted chunks and also produces a secret key for the same. This secret key allows the file to be reconstituted. Instructions to use the 'basic_encryptor' example are as follows:
# Encrypt file and store chunks
data_map, chunk_files = encrypt_file("input.txt", "chunks_dir")

##### Encrypt a file:
# Decrypt from stored chunks
decrypt_from_files("chunks_dir", data_map, "output.txt")
```

cargo run --example basic_encryptor -- -e <full_path_to_any_file>
#### Streaming Interface Example

You should now have the example binary in `../self_encryption/target/debug/examples/`. The `secret_key` for the given file and it's encrypted chunks will be written to the current directory.
```python
from self_encryption import StreamSelfEncryptor, StreamSelfDecryptor

##### Decrypt a file:
# Stream encryption
encryptor = StreamSelfEncryptor("input_file.dat", chunk_dir="chunks_dir")
chunks = []
data_map = None

cargo run --example basic_encryptor -- -d <full_path_to_secret_key> <full_destination_path_including_filename>
while True:
chunk, maybe_data_map = encryptor.next_encryption()
if chunk is None:
data_map = maybe_data_map
break
chunks.append(chunk)

This will restore the original file to the given destination path.
# Stream decryption
decryptor = StreamSelfDecryptor("output_file.dat", data_map)
for chunk in chunks:
is_complete = decryptor.next_encrypted(chunk)
if is_complete:
break
```

## License

Licensed under the General Public License (GPL), version 3 ([LICENSE](LICENSE) http://www.gnu.org/licenses/gpl-3.0.en.html).

### Linking exception
### Linking Exception

self_encryption is licensed under GPLv3 with linking exception. This means you can link to and use the library from any program, proprietary or open source; paid or gratis. However, if you modify self_encryption, you must distribute the source to your modified version under the terms of the GPLv3.

Expand Down
4 changes: 2 additions & 2 deletions examples/basic_encryptor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ async fn main() {

let result = encrypted_chunks
.par_iter()
.map(|c| (c, storage.clone()))
.enumerate()
.map(|(_, c)| (c, storage.clone()))
.map(|(c, store)| store.put(XorName::from_content(&c.content), c.content.clone()))
.collect::<Vec<_>>();

Expand Down Expand Up @@ -195,7 +196,6 @@ async fn main() {
Ok::<(_, _), Error>((
key.clone(),
EncryptedChunk {
index: key.index,
content: storage.get(key.dst_hash)?,
},
))
Expand Down
19 changes: 19 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[build-system]
requires = ["maturin>=1.0,<2.0"]
build-backend = "maturin"

[project]
name = "self_encryption"
dynamic = ["version"]
description = "Python bindings for self-encryption library"
authors = [{ name = "David Irvine", email = "david.irvine@maidsafe.net" }]
requires-python = ">=3.7"
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Rust",
"Operating System :: OS Independent",
]

[tool.maturin]
features = ["python"]
module-name = "self_encryption"
Loading
Loading