Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backup functionality #161

Merged
merged 37 commits into from
Jan 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3696d13
Add preliminary backup command
eimrek Sep 26, 2023
ec151e2
Merge branch 'main' into backup-cmd
eimrek Oct 16, 2023
a701374
interface overhaul; still wip
eimrek Oct 25, 2023
15e9982
implement keep; delete old backups
eimrek Nov 1, 2023
b5a30f2
organized into a class & improved logging
eimrek Nov 1, 2023
15bdb18
plug-in func to backup_auto_folders (for aiida-core)
eimrek Nov 2, 2023
ccf56c6
omit symlink without error, if fs doesn't support it
eimrek Nov 7, 2023
f5441cc
add cli tests
eimrek Nov 8, 2023
9de905a
ci: make 'ssh localhost' work
eimrek Nov 8, 2023
a2031d9
cli: return 1 if failure
eimrek Nov 8, 2023
c030168
fix CI macos ssh localhost
eimrek Nov 28, 2023
bd11651
skip backup tests on windows
eimrek Nov 28, 2023
c5ecf2f
reorganize
eimrek Dec 6, 2023
5a20caa
add tests
eimrek Dec 7, 2023
8a2c33f
testing to 100%
eimrek Dec 7, 2023
5da58dc
remove module name in from logging output
eimrek Dec 7, 2023
b88d0ee
update docs
eimrek Dec 7, 2023
f5476a5
fix docs
eimrek Dec 7, 2023
fd2d038
tiny docstring update
eimrek Dec 7, 2023
a195ca5
re-add dest_trailing_slash to call_rsync, as it's used in aiida-core
eimrek Dec 7, 2023
3eb7d17
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
8fa8d6e
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
37af8c0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 11, 2024
d2b7f74
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
ecc32b4
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
c80a499
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
620693c
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
60cd429
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
a699a79
Update docs/pages/backup.md
eimrek Jan 11, 2024
ae367a8
update dbackup docs
eimrek Jan 11, 2024
52b9bf2
Merge branch 'backup-cmd' of github.com:aiidateam/disk-objectstore in…
eimrek Jan 11, 2024
a33cc98
change cli docstring
eimrek Jan 11, 2024
94a37a5
remove verbosity check
eimrek Jan 11, 2024
ddd9150
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
54c1e23
move validate to constructor; use valueerror instead of backuperror
eimrek Jan 11, 2024
a79f73a
Merge branch 'backup-cmd' of github.com:aiidateam/disk-objectstore in…
eimrek Jan 11, 2024
ac10458
adapt tests
eimrek Jan 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 40 additions & 36 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,19 @@ name: Continuous integration
on: [push, pull_request]

jobs:

pre-commit:

runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: "3.8"
- uses: pre-commit/[email protected]
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: "3.8"
- uses: pre-commit/[email protected]

tests:

runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -30,30 +27,37 @@ jobs:
python-version: ["3.8", "3.9", "3.10", "3.11"]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies (including dev dependencies at frozen version)
# I'm using pip install -e to make sure that the coverage properly traces the runs
# also of the concurrent tests (maybe we can achieve this differently)
run: |
python -m pip install --upgrade pip
pip install -e .[progressbar,optionaltests]
pip install -r requirements.lock
- name: Test with pytest
# No need to run the benchmarks, they will run in a different workflow
# Also, run in very verbose mode so if there is an error we get a complete diff
run: pytest -vv --cov=disk_objectstore --benchmark-skip
env:
SQLALCHEMY_WARN_20: 1
- name: Create xml coverage
run: coverage xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
files: ./coverage.xml
name: disk-objectstore
## Commenting the following lines - if often fails, and if at least one manages to push, it should be enough
# fail_ci_if_error: true
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
# Set up 'ssh localhost' that is used in testing the backup command
# skipped for windows, as it doesn't support this setup or the backup command
- name: set up 'ssh localhost'
if: matrix.os != 'windows-latest'
run: |
.github/workflows/setup-ssh-localhost.sh
ssh -v localhost
- name: Install dependencies (including dev dependencies at frozen version)
# I'm using pip install -e to make sure that the coverage properly traces the runs
# also of the concurrent tests (maybe we can achieve this differently)
run: |
python -m pip install --upgrade pip
pip install -e .[progressbar,optionaltests]
pip install -r requirements.lock
- name: Test with pytest
# No need to run the benchmarks, they will run in a different workflow
# Also, run in very verbose mode so if there is an error we get a complete diff
run: pytest -vv --cov=disk_objectstore --benchmark-skip
env:
SQLALCHEMY_WARN_20: 1
- name: Create xml coverage
run: coverage xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
files: ./coverage.xml
name: disk-objectstore
## Commenting the following lines - if often fails, and if at least one manages to push, it should be enough
# fail_ci_if_error: true
9 changes: 9 additions & 0 deletions .github/workflows/setup-ssh-localhost.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -ev

ssh-keygen -q -t rsa -b 4096 -N "" -f "${HOME}/.ssh/id_rsa"
ssh-keygen -y -f "${HOME}/.ssh/id_rsa" >> "${HOME}/.ssh/authorized_keys"
ssh-keyscan -H localhost >> "${HOME}/.ssh/known_hosts"

chmod 700 "${HOME}/.ssh"
chmod 600 "${HOME}/.ssh"/*
Loading
Loading