Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to mount ceph temporarily #46

Merged
merged 10 commits into from
Nov 16, 2023
Merged

How to mount ceph temporarily #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ff4dbb7
add HowTo guide for mounting ceph temporarily
sfmig Nov 15, 2023
050d1c5
add link to temp mounting in permanent mounting guide
sfmig Nov 15, 2023
78807c8
add to index
sfmig Nov 15, 2023
965fb98
add permanently to title
sfmig Nov 15, 2023
1444640
fixed broken links
niksirbi Nov 14, 2023
d52ccbd
ignore problematic links during linkcheck
niksirbi Nov 14, 2023
55cf0cf
exluced an extra url from linkcheck
niksirbi Nov 16, 2023
d5ff193
small rewordings
niksirbi Nov 16, 2023
aa932df
make structure of permanent mounting guide mirror that of the tempora…
niksirbi Nov 16, 2023
dd7369e
use explicit target for cross-references
niksirbi Nov 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 31 additions & 22 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@

# -- Project information -----------------------------------------------------

project = 'HowTo'
copyright = '2022, Neuroinformatics Unit'
author = 'Neuroinformatics Unit'
project = "HowTo"
copyright = "2022, Neuroinformatics Unit"
author = "Neuroinformatics Unit"

# The full version, including alpha/beta/rc tags
release = '0.0.1'
release = "0.0.1"


# -- General configuration ---------------------------------------------------
Expand All @@ -31,16 +31,16 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.githubpages',
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.viewcode',
'sphinx.ext.intersphinx',
'sphinx.ext.napoleon',
'sphinx_design',
'myst_parser',
'numpydoc',
'nbsphinx',
"sphinx.ext.githubpages",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.viewcode",
"sphinx.ext.intersphinx",
"sphinx.ext.napoleon",
"sphinx_design",
"myst_parser",
"numpydoc",
"nbsphinx",
]

# Configure the myst parser to enable cool markdown features
Expand All @@ -63,7 +63,7 @@
myst_heading_anchors = 3

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
Expand All @@ -75,27 +75,36 @@
"**/includes/**",
]

# Ignore certain URLs from being checked
linkcheck_ignore = [
"https://neuromorpho.org/",
"https://wiki.ucl.ac.uk/", # ignore everything on the internal wiki
"https://linux.die.net/man/1/rsync",
"https://www.uclb.com/",
"https://support.zadarastorage.com",
]
linkcheck_retries = 2

# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'pydata_sphinx_theme'
html_title = 'HowTo'
html_theme = "pydata_sphinx_theme"
html_title = "HowTo"

# Redirect the webpage to another URL
# Sphinx will create the appropriate CNAME file in the build directory
# https://www.sphinx-doc.org/en/master/usage/extensions/githubpages.html
html_baseurl = 'https://howto.neuroinformatics.dev/'
html_baseurl = "https://howto.neuroinformatics.dev/"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_static_path = ["_static"]

html_css_files = [
'css/custom.css',
"css/custom.css",
]

html_favicon = "_static/logo_light.png"
Expand All @@ -113,8 +122,8 @@
# The type of image to be used (see below for details)
"type": "fontawesome",
}
],
"logo": {
],
"logo": {
"text": "HowTo",
"image_light": "logo_light.png",
"image_dark": "logo_dark.png",
Expand Down
44 changes: 22 additions & 22 deletions docs/source/open_science/Data-sharing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Currently, SWC does not have any specific code and data sharing requirements, bu


## Code
Original code can be hosted in a public GitHub/GitLab repository, hosted on an organisational account (i.e. not an individual account). This could be the [SWC organisation](https://github.com/SainsburyWellcomeCentre), or that of a particular team, lab or collaboration. To enable reuse, code should include an [appropriate license](https://howto.neuroinformatics.dev/guides/Licensing.html).
Original code can be hosted in a public GitHub/GitLab repository, hosted on an organisational account (i.e. not an individual account). This could be the [SWC organisation](https://github.com/SainsburyWellcomeCentre), or that of a particular team, lab or collaboration. To enable reuse, code should include an [appropriate license](licensing-target).

To ensure a permanent archive of the code associated with a publication a [DOI](https://www.doi.org/) should be created (e.g. using [Zenodo](https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content)).

Expand All @@ -46,7 +46,7 @@ In some cases, there may be a modality or field-specific repository that would b
In all cases, using a repository that can create a [DOI](https://www.doi.org/) will help increase citations.

### Structure
There is no single best way to structure your data, but it must be in a form that allows users to easily reuse it for their own purposes. This may be by adopting specific file formats (such as [NeuroData Without Borders](https://www.nwb.org/)). Alternatively the data could be organised within a standardised project folder structure. Many labs have their own system, but those starting from scratch may wish to use the [SWC-Blueprint data structure](https://swc-blueprint.neuroinformatics.dev/).
There is no single best way to structure your data, but it must be in a form that allows users to easily reuse it for their own purposes. This may be by adopting specific file formats (such as [NeuroData Without Borders](https://www.nwb.org/)). Alternatively the data could be organised within a standardised project folder structure. Many labs have their own system, but those starting from scratch may wish to use the [NeuroBlueprint data structure](https://neuroblueprint.neuroinformatics.dev/).

**However the data is structured, all relevant metadata must be included with the raw data.**

Expand All @@ -55,23 +55,23 @@ If you have any technical questions about sharing data or code, please contact a

(repo-table)=
## Reference of potential repositories for neuroscience data
| Repository |URL |Domain |
|------------------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| Globus | |Anything |
| figshare |https://rdr.ucl.ac.uk/The_Sainsbury_Wellcome_Centre|Anything |
| Zenodo |https://zenodo.org/ |Anything |
| Dryad |https://datadryad.org |Anything |
| GIN (German Neuroinformatics Node) |https://gin.g-node.org/ |Neuroscience |
| EBRAINS |https://ebrains.eu/service/share-data |Neuroscience |
| Open Source Brain |https://www.v2.opensourcebrain.org/ |Any neuroscience data (data must be hosted elsewhere) |
| Brain Image Library |http://www.brainimagelibrary.org/ |Large brain image datasets |
| Image data resource |https://idr.openmicroscopy.org/ |Reference image datasets |
| BioImage Archive |https://www.ebi.ac.uk/bioimage-archive/ |Biological images |
| DANDI |https://dandiarchive.org/ |Electrophysiology, optophysiology, behavioural time-series and images from immunostaining experiments|
| NeMO |https://nemoarchive.org/ |Omic data from the BRAIN Initiative (& others) |
| Openneuro |https://openneuro.org/ |BIDS-compliant MRI, PET, EEG etc |
| CRCNS |https://crcns.org/ |Computational neuroscience |
| BrainGlobe |https://gin.g-node.org/BrainGlobe/atlases |Brain Atlases |
| NeuroMorpho |https://neuromorpho.org/ |Neuronal morphologies |
| Cell Image Library |http://ccdb.ucsd.edu/home |Cell images |
| ModelDB |https://senselab.med.yale.edu/ModelDB/ |Computational neuroscience models |
| Repository | URL | Domain |
| ---------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Globus | | Anything |
| figshare | https://rdr.ucl.ac.uk/The_Sainsbury_Wellcome_Centre | Anything |
| Zenodo | https://zenodo.org/ | Anything |
| Dryad | https://datadryad.org | Anything |
| GIN (German Neuroinformatics Node) | https://gin.g-node.org/ | Neuroscience |
| EBRAINS | https://ebrains.eu/service/share-data | Neuroscience |
| Open Source Brain | https://www.v2.opensourcebrain.org/ | Any neuroscience data (data must be hosted elsewhere) |
| Brain Image Library | http://www.brainimagelibrary.org/ | Large brain image datasets |
| Image data resource | https://idr.openmicroscopy.org/ | Reference image datasets |
| BioImage Archive | https://www.ebi.ac.uk/bioimage-archive/ | Biological images |
| DANDI | https://dandiarchive.org/ | Electrophysiology, optophysiology, behavioural time-series and images from immunostaining experiments |
| NeMO | https://nemoarchive.org/ | Omic data from the BRAIN Initiative (& others) |
| Openneuro | https://openneuro.org/ | BIDS-compliant MRI, PET, EEG etc |
| CRCNS | https://crcns.org/ | Computational neuroscience |
| BrainGlobe | https://gin.g-node.org/BrainGlobe/atlases | Brain Atlases |
| NeuroMorpho | https://neuromorpho.org/ | Neuronal morphologies |
| Cell Image Library | http://ccdb.ucsd.edu/home | Cell images |
| ModelDB | https://senselab.med.yale.edu/ModelDB/ | Computational neuroscience models |
3 changes: 2 additions & 1 deletion docs/source/open_science/Licensing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(licensing-target)=
# Software licensing

## Disclaimer
Expand Down Expand Up @@ -50,4 +51,4 @@ If it is installed separately, and you simply call the library (e.g. using a Pyt
Firstly, you should ensure that the license for your code is compatible with the included code (e.g. by using a [license compatibility checking tool](https://joinup.ec.europa.eu/collection/eupl/solution/joinup-licensing-assistant/jla-compatibility-checker)). You should also adhere to any terms of the third party license and it is good practice to always include a copy of this license with the third party code.

## Acknowledgements
This document was adapted from the excellent [Imperial OSS license document](https://www.imperial.ac.uk/enterprise/staff/industry-partnerships-and-commercialisation/commercialisation/intellectual-property-guidance/open-source-software-licences/).
This document was adapted from the excellent [Imperial OSS license document](https://www.imperial.ac.uk/research-and-innovation/support-for-staff/scholarly-communication/research-data-management/sharing-data/research-software/).
54 changes: 54 additions & 0 deletions docs/source/programming/Mount-ceph-ubuntu-temp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
(target-mount-ceph-ubuntu-temp)=
# Mount ceph storage on Ubuntu temporarily
In this example, we will **temporarily** mount a partition of `ceph` in an Ubuntu machine. The mounting is temporary because it will only remain until the next reboot. To mount a partition of `ceph` permanently, see [Mount ceph storage on Ubuntu permanently](target-mount-ceph-ubuntu-perm).


## Prerequisites
- Administrator rights (`sudo`) on the local Ubuntu machine
- `cifs-utils` installed via `sudo apt-get install cifs-utils`


## Steps
### 1. Create a mount point
Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in `/mnt` or `/media`. In the example below, we will use `/mnt/ceph-neuroinformatics`.

```bash
sudo mkdir /mnt/ceph-neuroinformatics
```

### 2. Mount the `ceph` partition
To mount the desired partition on the directory we just created, run the `mount` command with the appropriate options. In our example, this would be:
```bash
sudo mount -t cifs -o user=<SWC-USERNAME>,domain=AD.SWC.UCL.AC.UK //ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics /mnt/ceph-neuroinformatics
```
:::{note}
You can check the full set of options for the `mount` command by running `mount --help`
:::

Make sure to replace the following for your particular case:
- `//ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics` with the path to your desired partition, e.g. `//ceph-gw02.hpc.swc.ucl.ac.uk/<LAB-NAME>`
- `/media/ceph-neuroinformatics` with the path to the local mount point you created in the previous step.
- `<SWC-USERNAME>` with your SWC username

If the command is executed without errors you will be prompted for your SWC password.

### 3. Check the partition is mounted correctly
It should show up in the list of mounting points when running `df -h`

:::{note}
The command `df` prints out information about the file system
:::

### 4. To unmount the storage
Run the following command
```bash
sudo umount /mnt/ceph-neuroinformatics
```
Remember that because the mounting is temporary, the `ceph` partition will be unmounted upon rebooting our machine.

You can check that the partition is correctly unmounted by running `df -h` and verifying it does not show in the output.


## References
- [Mount network drive under Linux temporarily/permanently](https://www.rz.uni-kiel.de/en/hints-howtos/connecting-a-network-share/Linux/through-temporary-permanent-mounting/mount-network-drive-under-linux-temporarily-permanently)
- [How to Mount a SMB Share in Ubuntu](https://support.zadarastorage.com/hc/en-us/articles/213024986-How-to-Mount-a-SMB-Share-in-Ubuntu)
17 changes: 11 additions & 6 deletions docs/source/programming/Mount-ceph-ubuntu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
# Mount ceph storage on Ubuntu
(target-mount-ceph-ubuntu-perm)=
# Mount ceph storage on Ubuntu permanently
In this example, we will **permanently** mount the `neuroinformatics` partition of `ceph`. The same procedure can be used to mount other partitions, such as those belonging to particular labs or projects.

To mount a partition of `ceph` only temporarily, see [Mount ceph storage on Ubuntu temporarily](target-mount-ceph-ubuntu-temp).


The following instructions have been tested on **Ubuntu 20.04 LTS**.

## Prerequisites
- Administrator rights (`sudo`) on the local Ubuntu machine
- `cifs-utils` installed via `sudo apt-get install cifs-utils`

## Store your SWC credentials
## Steps
### 1. Store your SWC credentials
First, create a file to store your SWC login credentials and save it in your home directory.
You can do that on the terminal via `nano ~/.smb_swc`.

Expand All @@ -32,14 +37,14 @@ This will ensure the file is readable and writable only by the owner (you).
However, if someone gets access to your machine with your user logged in, they will still be able to read the SWC password.
:::

## Create a mount point
### 2. Create a mount point
Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in `/mnt` or `/media`. In the example below, we will use `/media/ceph-neuroinformatics`.

```bash
sudo mkdir /media/ceph-neuroinformatics
```

## Edit the fstab file
### 3. Edit the fstab file
This will allow you to mount the storage automatically on boot. Before editing the file, make a backup of it, just in case.
```bash
sudo cp /etc/fstab /etc/fstab.bak
Expand All @@ -59,12 +64,12 @@ Make sure to replace the following:

Save the file and exit.

## Mount the storage
### 4. Mount the storage
You can now mount the storage by running `sudo mount -a`. If you get an error, check the `fstab` file for typos. If you get no error, you should be able to see the mounted storage by running `df -h`.

The next time you reboot your machine, the storage should be mounted automatically.

## Unmount the storage
### 5. Unmount the storage
To unmount the storage, run `sudo umount /media/ceph-neuroinformatics` (or whatever your mount point is).

To permanently unmount the storage, remove the lines you added to the `fstab` file and run `sudo mount -a` again.
1 change: 1 addition & 0 deletions docs/source/programming/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Small tips and tricks that do not warrant a long-form guide can be found in the
SSH-SWC-cluster
SSH-vscode
Mount-ceph-ubuntu
Mount-ceph-ubuntu-temp
Cookiecutter-cruft
Troubleshooting
```