diff --git a/.ansible-lint b/.ansible-lint index 720363f13..2a69bc008 100644 --- a/.ansible-lint +++ b/.ansible-lint @@ -1,3 +1,8 @@ --- exclude_paths: - - "~/.ansible" # Exclude external playbooks. + - '~/.ansible' # Exclude external playbooks. +skip_list: + # We explicitly use latest combined with other tech to pin versions (e.g. Spacewalk). + - '403' # "Package installs should not use latest." + - '701' # "No 'galaxy_info' found in meta/main.yml of a role." +... diff --git a/.circleci/config.yml b/.circleci/config.yml index 2d2e5455c..042a4f17c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -10,25 +10,24 @@ jobs: build: docker: - image: circleci/python:3 - working_directory: ~/repo - steps: - checkout - - run: name: install dependencies command: | python3 -m venv venv . venv/bin/activate pip install ansible-lint - - # run tests! - run: name: run tests command: | . venv/bin/activate ansible-lint cluster.yml > lint_results 2>&1 || true - errors=$(grep -c "\:" lint_results) - if (( errors > 224 )); then /bin/false; fi cat lint_results + errors=$(grep -c '^[0-9]* [A-Z].*' lint_results) + echo '###############################################' + printf 'Counted %d ansible-lint errors.\n' ${errors:-0} + echo '###############################################' + if (( errors > 1 )); then /bin/false; fi +... \ No newline at end of file diff --git a/.gitignore b/.gitignore index 11338427f..c72a9ebd0 100644 --- a/.gitignore +++ b/.gitignore @@ -7,7 +7,7 @@ *.pyc *.retry *.swp -.vault_pass.txt* +.vault* documentation/.~lock.UMCG Research IT HPC cluster technical design.docx# promtools/results/* roles/hpc-cloud diff --git a/README.md b/README.md index 756eb0fb0..daffbabc2 100644 --- a/README.md +++ b/README.md @@ -102,42 +102,44 @@ Deploying a fully functional virtual cluster from scratch involves the following ansible-galaxy install -r galaxy-requirements.yml ``` -2. Create `.vault_pass.txt`. +2. Create a `vault_pass.txt`. The vault passwd is used to encrypt/decrypt the ```secrets.yml``` file per cluster, which will be created in the next step if you do not already have one. - If you have multiple HPC clusters with their own vault passwd you can have multiple vault password files. - The pattern ```.vault_pass.txt*``` is part of ```.gitignore```, so if you use ```.vault_pass.txt.[name-of-the-cluster]``` - for your vault password files they will not accidentally get committed to the repo. - - * To generate a new Ansible vault password and put it in ```.vault_pass.txt.[name-of-the-cluster]```, use the following oneliner: + In addition a second vault passwd is used for various files in ```group_vars/all/``` and which contain settings that are the same for all clusters. + If you have multiple HPC clusters with their own vault passwd you will have multiple vault password files. + The pattern ```.vault*``` is part of ```.gitignore```, so if put the vault passwd files in the ```.vault/``` subdir, + they will not accidentally get committed to the repo. + + * To generate a new Ansible vault password and put it in ```.vault/vault_pass.txt.[name-of-the-cluster|all]```, use the following oneliner: ```bash - tr -cd '[:alnum:]' < /dev/urandom | fold -w30 | head -n1 > .vault_pass.txt.[name-of-the-cluster] + tr -cd '[:alnum:]' < /dev/urandom | fold -w60 | head -n1 > .vault_pass.txt.[name-of-the-cluster|all] ``` - * Or to use an existing Ansible vault password create ```.vault_pass.txt.[name-of-the-cluster]``` and use a text editor to add the password. - * Make sure the ```.vault_pass.txt.[name-of-the-cluster]``` is private: + * Or to use an existing Ansible vault password create ```.vault/vault_pass.txt.[name-of-the-cluster|all]``` and use a text editor to add the password. + * Make sure the ```.vault/``` subdir and it's content is private: ```bash - chmod go-rwx .vault_pass.txt.[name-of-the-cluster] + chmod -R go-rwx .vault/ ``` 3. Configure Ansible settings including the vault. - To create a new virtual cluster you will need ```group_vars``` and an inventory for that HPC cluster: + To create a new virtual cluster you will need ```group_vars``` and an static inventory for that HPC cluster: - * See the ```*_hosts.ini``` files for existing clusters for examples to create a new ```[name-of-the-cluster]*_hosts.ini```. + * See the ```static_inventories/*_hosts.ini``` files for existing clusters for examples to create a new ```[name-of-the-cluster]*_hosts.ini```. * Create a ```group_vars/[name-of-the-cluster]/``` folder with a ```vars.yml```. You'll find and example ```vars.yml``` file in ```group_vars/template/```. To generate a new ```secrets.yml``` with new random passwords for the various daemons/components and encrypt this new ```secrets.yml``` file: ```bash ./generate_secrets.py group_vars/template/secrets.yml group_vars/[name-of-the-cluster]/secrets.yml - ansible-vault --vault-password-file=.vault_pass.txt.[name-of-the-cluster] encrypt group_vars/[name-of-the-cluster]/secrets.yml + ansible-vault --vault-id [name-of-the-cluster]@.vault/vault_pass.txt.[name-of-the-cluster] encrypt group_vars/[name-of-the-cluster]/secrets.yml + ``` The encrypted ```secrets.yml``` can now safely be committed. - The ```.vault_pass.txt.[name-of-the-cluster]``` file is excluded from the repo using the ```.vault_pass.txt*``` pattern in ```.gitignore```. + The ```.vault/vault_pass.txt.[name-of-the-cluster]``` file is excluded from the repo using the ```.vault*``` pattern in ```.gitignore```. To use use an existing encrypted ```group_vars/[name-of-the-cluster]/secrets.yml```: - * Add a ```.vault_pass.txt.[name-of-the-cluster]``` file to the root folder of this repo and use a text editor to add the vault password to this file. + * Add a ```.vault/vault_pass.txt.[name-of-the-cluster]``` file to this repo and use a text editor to add the vault password to this file. 4. Configure the Certificate Authority (CA). @@ -185,7 +187,7 @@ Deploying a fully functional virtual cluster from scratch involves the following Execute: ``` dd if=/dev/urandom bs=1 count=1024 > roles/slurm-management/files/[name-of-the-cluster]_munge.key - ansible-vault --vault-password-file=.vault_pass.txt.[name-of-the-cluster] encrypt roles/slurm-management/files/[name-of-the-cluster]_munge.key + ansible-vault --vault-id [name-of-the-cluster]@.vault/vault_pass.txt.[name-of-the-cluster] encrypt roles/slurm-management/files/[name-of-the-cluster]_munge.key ``` The encrypted ```[name-of-the-cluster]_munge.key``` can now safely be committed. @@ -194,11 +196,17 @@ Deploying a fully functional virtual cluster from scratch involves the following Some examples for the *Talos* development cluster: * Configure the dynamic inventory and jumphost for the *Talos* test cluster: ```bash - export AI_INVENTORY='talos_hosts.ini' + export AI_INVENTORY='static_inventories/talos_hosts.ini' export AI_PROXY='reception' - export ANSIBLE_VAULT_PASSWORD_FILE='.vault_pass.txt.talos' + export ANSIBLE_VAULT_IDENTITY_LIST='all@.vault/vault_pass.txt.all, talos@.vault/vault_pass.txt.talos' + ``` + This can also be accomplished with less typing by sourcing an initialisation file, which provides the ```lor-config``` function + to configure these environment variables for a specific cluster/site: + ``` + . ./lor-init + lof-config talos ``` - * Firstly + * Firstly, * Create local admin accounts, which can then be used to deploy the rest of the playbook. * Deploy the signed hosts keys. Without local admin accounts we'll need to use either a ```root``` account for direct login or the default user account of the image used to create the VMs. diff --git a/cluster.yml b/cluster.yml index 01e7e2257..671820649 100644 --- a/cluster.yml +++ b/cluster.yml @@ -2,9 +2,8 @@ # Order of deployment required to prevent chicken versus the egg issues: # 0. For all deployment phases: # export AI_PROXY="${jumphost_name}" -# export AI_INVENTORY="${cluster_name}_hosts.ini" +# export AI_INVENTORY="static_inventories/${cluster_name}_hosts.ini" # ANSIBLE_VAULT_PASSWORD_FILE=".vault_pass.txt.${cluster_name}" -# # 1. Use standard CentOS cloud image user 'centos' or 'root' user and without host key checking: # export ANSIBLE_HOST_KEY_CHECKING=False # ansible-playbook -i inventory.py -u centos -l 'jumphost,cluster' single_role_playbooks/admin-users.yml @@ -17,14 +16,29 @@ # ansible-playbook -i inventory.py -u [admin_account] cluster.yml # This will configure: # A. Jumphost first as it is required to access the other machines. -# B. SAI as it is required to -# * configure layout on shared storage devices used by other machines. -# * configure Slurm control and Slurm database. -# C. DAI -# D. UI -# E. Compute nodes -# F. Documentation server +# B. Basic roles for all cluster machines part 1: +# * Roles that do NOT require regular accounts or groups to be present. +# C. An LDAP with regular user accounts, which may be required for additional roles. +# (E.g. a chmod or chgrp for a file/folder requires the corresponding user or group to be present.) +# D. Basic roles for all cluster machines part 2: +# * Roles that DO depend on regular accounts and groups. +# E. SAI as it is required to: +# * Configure layout on shared storage devices used by other machines. +# * Configure Slurm control and Slurm database. +# F. DAI +# G. UI +# H. Compute nodes +# I. Documentation server +# + # +# Dummy play to ping jumphosts and establish a persisting SSH connection +# before trying to connect to the machines behind the jumphost, +# which may otherwise fail when SSH connection multiplexing is used. +# +- name: 'Dummy play to ping jumphosts and establish a persistent SSH connection.' + hosts: jumphost + - name: 'Sanity checks before we start.' hosts: all pre_tasks: @@ -47,7 +61,7 @@ - sshd - node_exporter - {role: geerlingguy.security, become: true} - - prom_proxy + - {role: grafana_proxy, when: ansible_hostname == 'airlock'} - regular-users tasks: - name: 'Install cron job to reboot jumphost regularly to activate kernel updates.' @@ -61,31 +75,49 @@ cron_file: reboot become: true -- name: 'B. Roles for SAIs.' +- name: 'B. Basic roles for all cluster machines part 1.' hosts: - - sys-admin-interface + - cluster roles: - admin-users - ssh_host_signer - ssh_known_hosts - spacewalk_client - logins - - figlet_hostname - - mount-volume - - ldap + - figlet_motd - node_exporter - static-hostname-lookup - cluster - - sshd - resolver - - shared_storage + - coredumps + +- name: 'C. Create LDAP account server.' + hosts: + - ldap-server + roles: + - role: openldap + when: + - use_ldap | default(true, true) | bool + - create_ldap | default(false, true) | bool + +- name: 'D. Basic roles for all cluster machines part 2.' + hosts: + - cluster + roles: + - ldap # client + - sshd - regular-users + - shared_storage -- hosts: slurm-management +- name: 'E. Roles for SAIs.' + hosts: + - sys-admin-interface roles: + - mount-volume - slurm-management - prom_server - - {role: cadvisor, become: true} + - grafana + - cadvisor vars: # These variables are needed by the mariadb role. # Which is a depencency of the slurm-management role. @@ -93,74 +125,50 @@ hostname_node0: "{{ ansible_hostname }}" ip_node0: "{{ ansible_default_ipv4['address'] }}" -- name: 'C. Roles for DAIs.' +- name: 'F. Roles for DAIs.' hosts: deploy-admin-interface roles: - - admin-users - - ssh_host_signer - - ssh_known_hosts - - spacewalk_client - - logins - - figlet_hostname - mount-volume - build-environment - - ldap - - node_exporter - - static-hostname-lookup - - cluster - - sshd - - resolver - - shared_storage - - regular-users - envsync -- name: 'D. Roles for UIs.' +- name: 'G. Roles for UIs.' hosts: user-interface roles: - - admin-users - - ssh_host_signer - - ssh_known_hosts - - spacewalk_client - - logins - - figlet_hostname - - ldap - - node_exporter - - static-hostname-lookup - - cluster - - sshd - - resolver - - shared_storage + - build-environment - slurm_exporter - slurm-client - - regular-users - sudoers + - subgroup_directories + - role: fuse-layer + when: fuse_mountpoint is defined and fuse_mountpoint | length >= 1 -- name: 'E. Roles for compute nodes.' +- name: 'H. Roles for compute nodes.' hosts: compute-vm roles: - - admin-users - - ssh_host_signer - - ssh_known_hosts - - spacewalk_client - - logins - - figlet_hostname - mount-volume - - build-environment - - ldap - - node_exporter - - static-hostname-lookup - - cluster - - sshd - - resolver - - shared_storage - slurm-client - - regular-users -- name: 'F. Roles for documentation servers.' +- name: 'I. Roles for documentation servers.' hosts: - docs roles: - admin-users - ssh_host_signer + - yum-repos + - {role: geerlingguy.repo-epel, become: true} + - sshd + - {role: geerlingguy.security, become: true} - online_docs + tasks: + - name: 'Install cron job to reboot server regularly to activate kernel updates.' + cron: + name: 'Reboot to load new kernel.' + weekday: '1' + minute: '45' + hour: '11' + user: root + job: /bin/needs-restarting -r >/dev/null 2>&1 || /sbin/shutdown -r +60 "Restarting to apply updates..." + cron_file: reboot + become: true ... diff --git a/documentation/Gearshift_technical_design.md b/documentation/Gearshift_technical_design.md index 5c895e28b..02f05ee03 100644 --- a/documentation/Gearshift_technical_design.md +++ b/documentation/Gearshift_technical_design.md @@ -196,7 +196,7 @@ Both Grafana and Prometheus server will run inside Docker containers on this VM. | FileSystem Quota reporting | quota.bash
https://github.com/molgenis/cluster-utils | DAI | GCC | | SLURM cluster usage reporting | slurm_report.bash
https://github.com/molgenis/cluster-utils | DAI | GCC | | Account expiration and group memberships | COmanage | SURF SCZ COmanage | GCC | -| SLURM Job Profiling | SLURM plugin & grafana | SAI/DAI? | GCC | +| SLURM Job Profiling | SLURM plugin & grafana | SAI/DAI | GCC | #### DNS diff --git a/galaxy-requirements.yml b/galaxy-requirements.yml index 7679b9e6a..3ef758b39 100644 --- a/galaxy-requirements.yml +++ b/galaxy-requirements.yml @@ -1,7 +1,6 @@ --- - src: geerlingguy.firewall version: 2.4.0 -- src: geerlingguy.postfix - src: geerlingguy.repo-epel - src: geerlingguy.security ... \ No newline at end of file diff --git a/group_vars/all/ip-addresses.yml b/group_vars/all/ip-addresses.yml new file mode 100644 index 000000000..43e388f3d --- /dev/null +++ b/group_vars/all/ip-addresses.yml @@ -0,0 +1,358 @@ +$ANSIBLE_VAULT;1.2;AES256;all +65656631343735643036343164303061333734323338626664623035343761396265623434646536 +3566396536303562313230613162383930663233373066310a656134613262363666653133613139 +31613930656365323330323565346537333965653636376639336539336631633836336231623033 +6261613739613966340a636330623065356330653661666562316266313034363666333139393630 +39383938306134336363313038386361656533633334363262343736653964343163386665643338 +34343666376561323961623837383038323161353632643966366332323830303339326536313737 +39353731616362353430303166343639343261363330633065353135306562316561623530623364 +38623838316231353263336536616433383462623833393562366637326265346631663939613566 +65626166363363363939616565626130616461363538373531643531613034646438396531363331 +38333366653733386364333837333865346539323533326166393362303032383730626433376239 +36313964303336633439323236343334316137656166343433353962343237613365326164333831 +32623933376364636462633439663464383939373635346138353733333430623761343730626539 +30653331353838343665396165306365343336333761366364316239323361623934343465633963 +30383534663336636639633831623134396535636638306434353164636338623064653233363532 +36323933363564346637333035393535306233653435663436343738306335323835353631626363 +39643435623034393264663036313238646361386233356663343938653937346166633534333361 +35303337386336383964353661366438366432343835316238373636313030353138643335383838 +36326366333933616563366465333362616135313332303638343565333530383036333964656433 +35623035376438633334303765643661613635323334646635386232353661653834633832333630 +66376335333864356532623332616163646234306562303665386261613365393637643335626435 +35353233396136333434666233666132623331656561343838646136336365653136646533386637 +31343231343237626465313437623763636463323838313635353166343961633262373437313664 +66653332636230613730343166386132363431393836333763333536643164386235336236353431 +64623361663064616232393130333930383730303738383236353537643931323663663737646234 +62396463663962306131656462303230643330656534343663343562643832613832353132303338 +62623436333235356139616139613833626130346238333333306235656238373336353336316561 +61393431363237386138316434333136323837303835333136643563646236353361316565343337 +65313363313735306534303238386435663632326336653663343630313337383032356234336665 +32623936316263646439623930653133313834376462353535386134323530633334333664333165 +38616137313332333265653730356333616463333862356630383833323934653432303738633737 +38613239646566666631643031316430386434303636653639333461626439646266623239353361 +33613662333238656532656266383866363165316537663331396339623834343035646536636436 +35303439393739356163306261336235393762346332643534666238653463386232626331363336 +61313030303361343535363732653039346162386464313534316166343330363664303361396638 +35653838383863363463656661363434613335323431373665313633623038353036313431373136 +65623335313537313163653262356239323232613363396133373138313334616466373966353738 +37366136333365366130393938313761396535373562666137353465306339653239643563393630 +31303139383462623635623439613835633539666332386139643765313434653039396539323434 +65646261636338626563613662653630383463623561653537323633306131626165383632303762 +37323434663366616363636237356636323232363632323338326633346335333266636137303233 +65636264653536613231386434353739633637353036613662393262373465356631303734383462 +65396137653633313562616530373539383931343830653839643933633335653162643030333138 +30316435613639333466383764383231633866326334373433643635623163623433313535336433 +64663537623138633462633037306561313638633136383363333463343066663564663064646639 +33346630666363643439353131346363396131316435323630316138656231363531373737306430 +33663864646364366264646531333864656530613531396439303037386563333833643564336464 +61323135353431386238363962343037363032323130643562653735353061306339326330306337 +32626166363937626261363262363065323534666439303636613462663766663562363265373132 +31626631326161383866386161356431383161333139613536366433376532383635313633323932 +30373663653139386235663563616432663766323838316164656237646462363330613566626230 +35623431316164343238373761393163653165353365306465613936613131653930393164373032 +33626261666139343836313530336365343234633939633966376565323931656263316139373335 +36323331323430386637386466653763313166373165643730633330646632356161386365333639 +39643664366364643736326361643037303739393637356165653933666662363437353636613834 +64316636363364346235366262343730616536363631393734363762643062666133353761393134 +30303832663032303666376132346134633035643866643036303630653232383130663135386663 +61353239636638613435336537393939346566653035626336353866636537663662336665646232 +65643732613365336661366662666130343661626435663730353065666436663862376136376435 +31303531316133393764306566383839303465653631373635336666613761363464356236323965 +61386530373138643761316338633532303365353031353632663364326461363564343466383165 +65333539613336313735636537393264316633613362306538633366323464303363663065623338 +61643933393432346562656162643561326334386663626635343738373130626662623436633431 +63383635323831303736333836303761303762326331656435333739353164633834646566346564 +39666664646336366363643664636432386461393337666566313764623061303561343733313234 +61373831303334323132313566383663396634393535326233313635306366336435636631323031 +65306134313035313839636432363138373632333331366362346332333531376361653761396134 +64363236326131346438626665666334666338356366383036386362643166373439396165323231 +33353236656362343037333363313436383934383230653133373339336362396566376432646337 +66386662623133653633336536653762306664643532333063633034353834646566376462326538 +31663961396538366261353932383766366432626361306331646465393164383636613761616564 +64313537376639656465393039626536303234376134333231323331656664653466396531313433 +37346537346636666261336130373537613830666562656437356562636136663765633962376131 +35303935373066343336316636386632646432376665633962306439373434336336626434313034 +64343434303765366566636164303165623536343437656636646532356465373333653431646438 +39623262303336396663323262636335383238373732663463626434376132393330396135366632 +61323330663562383531366239393634383037636436646137346536636462383032646362326430 +35613364316665636232306630633962623538623533666466326465303263653332666664313038 +39366131626665626331663264343562313138643438626434643534323133306634643033643933 +66396238326530663061363638303866646639303733623639653730356664663136626165343565 +64646330623231353333333061663732393830316430306665323364646666376461653531306564 +39636231333735663233363635653166656636623432346266343939636233653465646361623739 +33613834336263333431613161663036626631653534643536633963353163616638346364383532 +62323865666630626136366537623264616631373165313231666663396634643365333361373735 +33623161346164663836356536303462613962666366643931323936663735316436623135626463 +62373164323465613766623564393339383638313861333436653662356136346666393965396231 +65623737326234663930323663373263363132613630363133376333373163633631303135326237 +33346230383739306562343236383433343333333435303338336435613636643037366162626163 +66363333623930656263316564366538643137373038633263363031356361353031343738356138 +39386439646466613239666439633463353332623231386163316465613038393432623738663139 +39376635613032303336313564633131373063316339326133346366653733373133663938313436 +32386161323366323064323434616465656433626364353439303235336166613335333961336165 +64343463373166363135393062613763313938636130353139636565363932363764666435636461 +39323166343034626439663434393865303831366263316639386363373139333237393762386637 +61626566396334313164303834316130336139303933316334323037333961323136616461376537 +66613031353531303962343936353533303637623866313561663133323433653330366633636634 +32353730303039613164346563623866653462653033396133616631653937613032326133666238 +33333765653461656564633631613231363766656265643863386231396664373530383237643535 +35373662346339373731353064323834316366303066306663623338613039303964346461626130 +31646337326630306166373463653039353631326338326365303238643533336233393734336636 +65326166643238333965333664383761343838366235643339366461623763656230353234303631 +66353034646664383630393665363163343132633134613061653737316566393633393335383536 +33643232656236373231356133626333363331626665623838396438353934653030386463623932 +33656437373434306362653632336266643437366436623037356631636436653263313466623930 +63653365616532646538653737333235333163316136353563623836653564393435663662333531 +32646139656162366338623662336433626531363862313565663433663938333834313965333834 +64656134316562363763633361313939346434303036633361633833373662306331663639363762 +39373133373637363039613666346130666261316665663631353265396631303438333238633433 +30356361353338313366613066326165383837623532343835333433323238633132663464383862 +38353535363761643338643235303764333539373839333661346338316332373336326665373134 +65643430646434396132316631626362623064613864666462613761303936626463643630386366 +33333531303635383763323662333361343262353432653536323136366433396334343734383735 +39333362343862393566373264643966623030373164376134316639313237633239383333616636 +30343531376532366364306163343239613330363036303434326530393230613963663239306665 +35373033653864383263303331333865313433353261303839353862303334393863303439326664 +64636431396164643962303834303138356634636432306264616137303631353162316134336430 +35353434666361356530666537616231313666633939343066636230626638393363336434616234 +61653561336666343833346435323561306663363036643362333935326461376336323137376139 +34313166663663356639303763353830303462313962313031656165323438306362616633313132 +62323938333035373234616330633162313061306237643964323531663962633036626435396436 +31373538623465343634363464613037343533643464623334626634306633363933393066656138 +63363064343036666534353963326234643939613333646538613063363937353765353965333666 +30343839336337393761396434346633336164303132613335343636363566353133643330396366 +39623932633736386635623761373632636434613733656233653436643561646137383361366537 +63386431666264633839656532376139386232666135313664646537393464623734386636316536 +37396534323262363331663265373962323032663038306231346161633734656161636561373033 +65373861643236393238353861376364316430353764306563313638643039666230313737316365 +34306166373430373263363962346534326563313134366336346666326439396662363237393362 +65383630643161363434666534303163333035613865656138663865636262363233376137656230 +62303133326135343639623436656230396664633731663830366565336231396330386636646365 +32363637363437346237343131303264366663393663643966613531333638353836623639646531 +30666465343062353966393037303837343536666565656662616533303362366338313034353663 +65653361383066366234666463306563666166636539373031313530373564383332643161386363 +37316133666532376361346132356438303431656364323437333532656463633938323962373638 +34363436373562333831313835616334346331356665313530643031623733303066383734653235 +63393130303163303736386230623839386532623838333166643837663838383265643037356130 +32623636333831333631323664633938336564333338343939616262333962643465636231366334 +37313162666432366630333338663535613436343861633138363834393531363031633534393439 +66366261383937633933613835643463383537356235313039613462333737363430643438613937 +66633365376539396136376136643262313766316539353562356239303130613637396231343366 +39323261313237306533363439613065613639333830613533386664646332336230613961356466 +34353338303661643732656234386136353236303161666262366131396366663363623862346264 +34643638373036393961386662356665643263636432396361333565613136383630353066303935 +34613037373636656237303262636333663032623932353438323230623363316534356362633961 +64336235333339613431316335363136373539623665616562636165363839323133383864633339 +35313364353038373038383863633631383732316239336332323862333738646365306636663663 +64393433336437666636626338646362663734303132393934336439346166343832353862616132 +32303838616365653332383738646264393934343330303838346133343536626362343666303036 +62643934663036666334396561646332613338313065643735666234303464353364373136633236 +61326565376233613565386235343937383930623061366236663134303261363836646365616639 +35323837666261346339633130353532653662376638373566623630316530386562376234353338 +63303163663637386437303936333536636431326562613064663938326637376566376539393266 +38383461333933336365653466303839393064363132323633613030353431613966346334383662 +34323234326565663934396133393137363466376261326161616132303566386266643563303663 +31313535633734376663666239626233326562366135653934346634363836643139386261623839 +36373339326636393864393937343766333931313437333061396236373634316638366233643834 +34313237363864376666343061613264663239393038323137626538333164643938323764396637 +66316339363666303637643162663634376166343764396534353732646561333138393363393433 +64666664333737666462336238383835623938646338626338386131613032613764313865303864 +39326631616236613163323561623336636164643231323666326131386562356634626536663463 +30646264386434343230353436313833366666613262353362316635313139643231623765646461 +66383039363361303239663531643630626233396634353665323962356638353932386333393638 +61343938663037326662303839313132353939616333353837326164353731353366376365646336 +32396635383961646461663738303463636363323862343333303865383966613363306634623537 +33353363636136323563623635663066353139313566323931613266646164633234313234653164 +31373438383764666538316138396639396431306330376132363062303961613138393732353536 +39623237303834396362376539383336633166313330366231393164336365396666356662323336 +38643462326133643466613163316436353266313931656539376561616162353338613837316262 +32386537383935616335646530343533623062316161616262626562326535393961393032303739 +30303033333962313766653933643731393066353933323733363761396231643631383765306166 +33306533636236643937616263646633623165616363356635636663373561383633306664383938 +32646638663137613737363762633936313363373034656335666666643863616334636130343265 +31306533643032636236343562326632633863623434626339626365633061383962646532373864 +65653230663138613837326638613132333330326336656165643466663862353363306337626234 +34323462366464343165393164376133626264623064383232663537316537363536616632343436 +34646532366164316137366436616537326330613330333766363032393134336562643062356330 +31373862303832303566643836613736613631646636663164663133663031363062616162303063 +39333265373861346433636638653234346666393763323565346665383037346632626162313039 +64353066643662366366303230313665613932653837396432333734303864643736616162353736 +65666665653931343363396237613466643732386131306233373835613739323366666139373932 +35666438303363313130633466363931326336343836333731373638626566663366316131363132 +38643765313034393661326138316631333265326134386335383431633538386339306233616364 +65393135643536653864386435333463373563393961323232333330316231323136666438383639 +66313831643734326562633535366562366130646364346661396430313830326365343865653530 +37396364613639303839663630636664373365663366333731653733393136613461383739613566 +63636465383461306330646430303830613963363332633330346137633130653264303334393262 +34343538346432663737323463333366363734393232303065373832316531326361323935383166 +33396634333035376433636163616230313266383135303935393434346161646132626539616262 +34613632646239666165353635373337633931656366396532343565313933633534393466303361 +31316235633538633438616434373934393337653832636565633330303235633633306538616163 +63616262623431643334623134336165383730306464636238613634316334306631613030393630 +33616639663337313238643763376162333437616534363063393439636331616132656530363061 +32643961346435663538613333353433383331636135373664373135343562633037613865653165 +64323734393566323732326138336433383663396666393363643865353765646463373235306364 +34313062666364323265363330313830393236373964393639366336313039366435356630343635 +34666564343930333436323131346235383466663762303131376130353761333132386633313564 +32663432336666656363653366336438326239303635323966393433626432376563353634366434 +65353339653862353932626266656363346562626134383064313365373434666435353738303765 +61643164333630336664303661613164666130313137623334363230333035396635656665396266 +64393334643630366136303161636563373838323933643939626561366265613462336432663932 +66653766626433346265386339653034613662313565356239613064623634373966383936646562 +38343365343037316465616235316438653262306637663039666235383830323764313665636264 +34623161343632643234346630613162383737623739393036333862386365633935653563386465 +31323461633232663931613831616332306161323562646661326435363438373436376164366432 +63323438356666616563363061316431616663613666393936303761623239663465643637303436 +61623936666533656665386137656566303438346162643765663439623333313138653337303930 +37653439376531666364323433333163363137633761333863646562343832333062623762306437 +63363065646536316663323465343838393434393864653965386432353263343234636134393037 +65613064333036316565663736616666383836346333636366393635373465396139343066326530 +64333135373931616262363435356235386133653933626338323435383535623163313633346465 +38363663343866656231386564343665613735623034613161343037396235636134366466643830 +64303062386537383334653439626565373266353432646564613863333664643861646266653330 +32613432333130393536333263303037636264336565656335666265373266373166313437626638 +62633632666435623161666337626130303835356637363363636265313136313735653732373636 +38313363353138343339303561306566373830363933393666623165363633646465663664343939 +34323663386233626362633862306633643464646664323835646530363535366535646564653366 +34336164356263653833653832323636323266323830666531333461363839366163356365353666 +32663530323538363033316132613937626238333432306338396462613234623735363363323862 +65303763383961336133336633396134356361666564646630373763623236303066393739353139 +32316666343631313236343861643361306464313266633663643461633966643936396564323566 +66643133653132633136303463313439383935306636346337343435643463373236366437376131 +63656264376661353334363535336133613130616532393339383031396332646136326334346330 +65346161633361653135623939303339386334353033306539396231666636613639363731323337 +62613664373530643364663566343661343735653430303332613234653364326133373736613135 +63383038653566643734303565323761343866336566383762653930396461616365626335623338 +62336637303536623530343136373930376139383965396135323637636135383739633635646139 +35353539653331326337643634626433353731633038343635663536626638373537626132373033 +37343837363934623839663132633564353531363132393630323231323862356562316639663537 +65373264653761316432396631323939326333306238373339616162376265653265363262626437 +37393461333637353737636537643437303964383534306533653864366363396636323663363730 +62383865303338323637646232663163343862653135356236626263663135636436313166356132 +34326363653563303636346261303264303664666563353339613238316161366134383862616666 +61333161393763313638303962343964356336313461323665663939393361366139343664373534 +34373966653133663063336462323933303933613262636137383065353965623636303137643436 +38613463336566303730613336366139666235376665303235353739373038663235623762313830 +36306431353032343265316537376330373937326339656463373333393366613466653863306435 +30363532366636306161653231653139323863613563623835656366626461333637373934643664 +61616232376164383763343865626338313032356262353861333562313737376331643564646433 +39363130663636303734313138643830653738353264353631633333353235383032363536626135 +38353934333731663564646231373733646565616336613835393965383934326261646265643666 +39323838373334306330663635633535393933333837373231363862653963313062386264386335 +38333532626236333966396632303235373036626630356237383964396333636463646237376366 +30346133373933316135353666393930333261626463653133653963313961353262653666613263 +35326632346133613233333032373738316530373363356231613761306234313636383662343265 +30633661353437333936313162626165633035633236333236643734633539653733363561623333 +30353766663336393739303337383766656438386361363637363938323739373264333161616538 +62633434363563313634636138393537643362653634623938656538343933666638323536656433 +65363237626339333666653463666335376238383465663338616364393634316634633161343539 +35393833666136356636386532376565333732666539306639366564303162656661396161666165 +30633266336132396437663936643364623634326465363465663634373264336665653930393864 +61393436363035623832643638376538636437393165643338303865656162616435323236356633 +39616662343664623732356164313866303635313163363735383938316662373636363035303064 +36643536626366663863666137623338666338613262313265356161313264313362666132623666 +37623235356262663965326661353032353861383336323662323761303466373539333866653335 +63656663376331653935353030306332666431336336633039656531623830336362353337396434 +61363234346632656162366463633462396235663239333030613764396634353762326331633835 +35396539363061386131343135306331376132313130303536333230383838343631656235356132 +64373364393731393739323131653331306563396338323265393030613532336234643762376537 +33356161366638396537393436633339373765346236303038336637613831633134666139306637 +62363331646331323531323862653231383136663163336362613263393463346363303663393438 +39656534376334303266653835656230633063633231653763393036633364626338623336656334 +63353465663438396438653864636164643565373439303162656165313561653162383232613737 +65306663343230643166653364386232656662623466393464316633363734656134373632623835 +31656563303336323762613464343261346565613064626433306230346330376664363766623231 +61303530363461313831353830613363393265303663653631643561343262393636383664633835 +36666339346465333137346133663735353465613530656664373531616664333339303466353936 +36396332646333346238666661333565343564363437376536666664643736343732363332373664 +34363166626630616362623333356339636132326466383331643636626164336262336366393535 +31333930623865313033613135646536636263613835346238633561323139326335313935386433 +39326433346131353832336661333561323931303964343563643134623766623937316265333236 +66653138643439633932353239616462633333313330623530313531376632643633383835316361 +35623732306434633465616635656262353337653563623338356439333731313162316564643564 +37396633313966353765653134636435633963353166666366346335616139303136373466663837 +65343363646630323833623466616538363463306563316663346363333066353365313139323365 +64303766623566396364383037363733323935383738313363303762336131653137373633323961 +31376336316466373164363538663739636436313933326332343664393339613031386233353233 +64616365303265666236366161633731653835323161343364323561383330333964343562336561 +30376363646435646438376564386534386536323965313762383830336538623038386664383335 +30383062333930663330386166333639393365336233376565303430333337346234663866643833 +31383639633831396165373536386133373531386537616439386562616231366532353035653439 +38656165643339353064363933343937383030396663363165623862353830386639636361356335 +38393037643261336162623337353463336336353762363336346561666165666465626338316531 +33313362353762636537333264346164633062346437643839386239616461383161376563623735 +31633935663738393639306231653539643830653331313864333238666463616638373061653235 +62653031306634376631323762356137633138396266336166306461313665643563393839383639 +35643163646233373434326230616237386338656637373837363863393462363336343363343261 +39373063613964663032323831303861656165363162373962313463613662663037623235353734 +62643838633232633833643330666638366439393162613964613766323461333664303933663562 +38616665643931343365613332353730336139663330363431333031356266646163646666356237 +31313139643938393031366531363736653038646464396363366233336163373134343931356439 +33383431613961643935353466323330663035343336393961386663663535303232393539633532 +35336236356339383765383065323635383339313538643936353737346437313234336137643861 +30633264626436636661633563333061663338356564313431363634643532333161613632373938 +35326163333930663962373732623437613132306539643937366434633761636338616537336164 +35366463373563386361353339303061373332623237303434393336643033383035343361386438 +36633737666461346434393333333864363061373639636665346439323766663638383438353839 +65643761663535316664613164623934303961383861303766346264396539643233366437643763 +65323064353137356537306637626535333962323465613334373330613332353332376633393635 +39626162623665663431656234613865333936356362336435616338643733363936626536663239 +64633136646337393036343532363761346562386337653235383065653437626632613933633261 +34373463633362646432616330356238396630353630383866633034373631653530316664643038 +63373233623263353833316261326333353731366533303634306335343737363933626266616265 +62633236366134383932343433353466656636333939623634393435396266643262383661376332 +64323936366638613763653332386665323338306538623935646365316563616232303066666538 +39363833626165303566303731363435646337623563383038396566303663613664343665333365 +30373564303463373261653666303031313532396438633631626137323337613133623062346338 +65323265613133353433303934376637336662363065316630623865313033653839303137663336 +35303933666136653866363132343138346661346532383366313562366131393036306266353934 +38333338393138376532393665386235343030363039373164356265663164393265353061333063 +39363630626266323665336464303362653464633134386134386138663336626637653132303636 +38363639383031653361383034323163393261646139633530333562383362333464623865383233 +33323437366566653932653637343335643739653437633530376531323130623034366532376262 +62373766393861356536613564366564386233383235396533366437663136363463613032356334 +34396536633566633764386431323762373237303363393331303932363530373766636330613434 +32306131653938323234623966633331663262623836646636306238393661346635396166636631 +65663834363232383030326266343633383838316638313438613861663831343037666165333538 +34623432366231623738313866646366353666323532656664653530626564376265393539333233 +66666164343662646634636561646334346361373163373362353136636134366638303634366234 +30363661623233643765623637646431396164303231323564643466353432656164666337383062 +30303939643261626562626639323236396238393266643837346464306561633266613463636430 +66626339343939383763343037656264646466643363343635303837393334646366383731393761 +66316135613030613164623166323530663263326663613237363033663261343066313630326431 +63346135343838363937666136663538616237383339376561363162333661666463633331613035 +30316336343966306339303035356466623939613739396130626239666337353332626333323963 +65653663336262643130633639326532653133386430616634323333353933336530393136353839 +31333636656235383930326461623739666230666530623163323633636635643864383732646538 +35353235613366343835623765326362623232386266646462356437316632363062653331333261 +31393961383261316464623430613430333064386236663732326432383264653133303834653634 +39633437316135306437386364636265333534343938653737626531643538663535363439303164 +30623466343863306637303538613231633265336537663733346138633233333562316336666331 +37656263326565343437633038333637306633366336643333313966343730356563613138303033 +66613231653264356336393265336163303333343932373030356238353935656264663332656666 +34343638616338633030306431626138616662393233646631333331326137396638653535326438 +31623866616237653464643635306665376166306233346239376335316663653737356534373866 +38323261383839366131653738313530643431653734373061656565343733376339346532343963 +34653234666661373565663065343164363038653937626239653033396131643332393531393864 +36333432336133386331383734636332383536383135333137353063616137653538353965393964 +34383330383434643361656535383437333931306663356163646130363964393931396136653966 +64303035323032356637643866656464383735653265396530353738383739356534663330643331 +39346464356263303638373661373632643165393065636662356539653831643462356363383330 +37623434666230373162396265313032353236386565386361363338363637376266643936306432 +64326235396564666431303532623439303730316565636461656561343631356530623234656433 +35623161396632323164663061356663323165346438646661346335336231343161383637303738 +35643965386537326130313266346565643434343937643539353932613161316363633136643261 +38663066666334396231396633663761313939366366656330303233636138386362373663323239 +32306633333166613834323661656164313066326631613833646433326664373664363333616638 +63313665353661616462323737363435313163343036393261313466396433386634303966663564 +38313330363832343436643863636663336131663633353566346564383962633465613432643930 +61383630643063306265396335396630393561616534363164616539616133303063366133663934 +34323439356166613931333563373432363231643631626236626364653839363465343733653061 +31663337353138363531613632626363336265386661333161386466616662653764643363363566 +30336533353734666239316338643437373431393937313164616565393361343662393662343032 +66656234653834316130656233623965616532353139313332623064376634633262383336363734 +65373733303034333763656331663863636436636236363263343839306134343064373333393533 +64326238343133626435366236323838616662373462633735636266393563366133666633383263 +61633531376534376365373131613032373138643566643165333739303862383161653332393436 +62663934303537363836666261303964653736316430353739623737303836633363353830356562 +37623966666366396464646432643833313932316134643966646337386535386231 diff --git a/group_vars/all/users.yml b/group_vars/all/users.yml new file mode 100644 index 000000000..8e4c3b21e --- /dev/null +++ b/group_vars/all/users.yml @@ -0,0 +1,1790 @@ +$ANSIBLE_VAULT;1.2;AES256;all +63393034343966333631396631623737653339636466373139383436616633663261356164613632 +6132323333363532636139646236373132313563663962650a626664663364653261373361343936 +39626666386165373937346265616637626166303738386634366437303033656138633064383433 +6334336330396663360a303630613765326431666631373961316330343062636331373463653966 +64643961393264666437336439663831376564333932303036326366346232366135313866623466 +33323162383365353563643362653937316532396537343935366637623936373733633739616165 +34646636656564366662666132346433373435646636613431656331373437316564383465383334 +33306166396633343636343163306537396636363564373865633464353132626533633162326161 +63633138393164616664666434306330383961643466346332303764363136343439323564336661 +66626438356135393164643135323833366530666165636332356437636564633364303734336133 +30646463656232323563646466316361353366343438353830636237333232343063343837383636 +66333634326136313065383438346263323761336363393233383338326464326532653162666634 +35616161336133356530656632353830393637343366326132376161363666373530346431323639 +66373230323138626265353839633636343833306563343064633234343338316435656464303030 +65653965306638313339633764376364393263363064323564343530643531376334333639343736 +34623565623838373233353132346330356438303831363462356663333236633435613165386531 +35653663313166643339633237643462666337316539333364383566363166303135633665326538 +37653965646463666639313137303138373439396139656332326436326337393436636336326338 +33316334323835636636643936346439333266303035646664303863663463343465303139623536 +38356462666161383266346337623361656634356532343631383661346166633765623636333733 +36303137636661333161653230386636313235303937383938396634623235636133353661363364 +63383763646330633737353763626662663237306366666365346466643161306632653861633865 +62356536613430353132363762623937663961363038636339646664386333373464653865333634 +65373435613263656335653236303662376538333863383933646266656364663437376230653433 +61396634633436386665613730626164643364386434313535663437326165353837316265383066 +66316330643236343335306466633663326336373261663937616166393530393061303835323732 +35303537613036313236626137393835643734336336353138336665626636383533396665383162 +30323932373664656438326336363232653766393233613962363136623362356636663161653232 +63326130653236623432363737343433333130316462353965613133363230653431666330663234 +39653835633238343836393465636630303366663166333762303039366562626138663131303063 +33333065353262383639363439613535343834633739623432346233623461356234336533313332 +34386166303633326362653665633630343464383539386162643833316433323035306537396364 +62626264366432653163333839373436323033333261666338663237383534393336343938663331 +63623833323964663139363437333835643562306261656165326636333531353435653563353930 +33376461643030626339323161623539613564366266343061343064646563333563396565343930 +61383861333231383831343761303735643062616231393162653365393935346364386531363964 +32366236366639366566353263386665376165346165376636343835633765363763366439613764 +33343765643961663639386339653337383266363932663330336465363765333939626130623766 +62306136346136633264666538623730643466623539653865326330613639663465363530363531 +35643132666131393331383765656162353731616137376166376665343332623261356363653063 +35636162343462663032643234353839376665383462313661373731313861303061656431626439 +66663163613132373334376339343835326332373961396632313231616137353139623637366631 +35353763636136653836396631643263656530393030373838363763306464386332653662633438 +30323536633866333539306636643965653763643262346164323764393863336236666231623765 +62616639623135306135656562393238363532326665303163333733383738633839613333633463 +35646438396339613432323838663362663835633632623466656333333766656663343432303264 +64663533346362353530643464396133333734633933623032366661393064353330316534306264 +30666662366366333932616363373866303337623465323662376433303432353965613137633635 +39393466303261373833353032653364653930656663323633623437396436613734353364396163 +33666163323337303866363661396231653566616233326630376331633730613332656661313031 +37613031366261623634346634613161303736653363623138376332303432656437363738616561 +61323830393362623831646535383564393032326437633739646165336530346637363964626533 +35656262366565393062656234326331653465383730336263383130663433623536623939316562 +65306632393433356361333265666162666662656135333564376539303261313631313438666537 +62396336323935393365653461393866636665373132386335633732303164363436326561326236 +34333736386661336335353765626661363831353430306539383430373237316635326561333162 +39656564643032336131313937316439343739386332306361356337656666333332343033616466 +65346430353130643831613934326235353639373436616335393639346666646261303939343730 +30363637663263363138326133333838633334633066653864626261396234643630656134666232 +63353736306534383663383136386336633861353365666534333634353162316261626332333737 +33323562666433393938303439666361393063383733326263653031616238316531333438353033 +35623563333465363461346561323438323939613833316564333832623936623661373536393637 +64656336346662663562363739333538613263373734303039386132623739633635643264646461 +65303165323435633332373136343839346262623639613832626430663836313434626537663430 +36343964353335383833666163316662623365386238613765303066366532353535303866333265 +66373339383035343638363063633333663439336333643262393563323438343239653262363837 +32666566623862626633316335326362393330646161376533316532366336613939623330313734 +64653761313436396161646138313635653931343964336138656631326132653564333164636436 +64306336316636396565613530343038356665626635393337383835363733343833623461373737 +31363264653962633530363132663033303634343965643363376262326237366539373534323735 +37393962653862323262356330303931626137386666366665626535383633393833653036373933 +66643661393532303439336330363666653239393464313335303331656565616562316635323865 +62333363326133333162303738326235306438386164366636636335613032653061343765343135 +37306537656431303139633561376662613633346436323635636332336236326435663932313535 +32633039353266373763303563633237626164393735353463343764376137383133363530643339 +30656437346138643064303736336362306331643930613934646337326664393834646436623564 +63333933653134303762626134613431626166643237313530666239383437643336383531373534 +36373533643930313030303362666131386235633034366134343666373932386262336338333932 +34383339643736306130333666343765613865306562376233363335396537363665643631656366 +39633635633839396337623166616162653334316330636465356464653435333362363932393362 +37353034656562636662373334633431326666363133353437306238303436616362313537373435 +66666562633335633936393630623261323064636562653336646164613036303464356330393333 +65393563363661316264326266636238373836643563303335653439373233306134346534333261 +63653561326437336266376166663732363838306361653066633330313637666434373133653365 +61663534656537343365643463313761333839373239613839343661663231323234353630333763 +38336462383237383838353934316435306564663633333232363131343261326266653263393366 +66333533376231623534643537303938396237643632333563303534623938343039366366623836 +39313237333836633561623064303239626331393438663464666463396631343634666634643165 +37326232613738643833616230646437393838393935346264373662326531363163303964316166 +38383234346533656334616262656332373733306336666434643066383030343161343239373863 +64346430356137656338303438653462646133396239643632656139383136653363323737333863 +30353265623238393434363034353738666232656238303561633836666163613263353338653362 +38303433643566613333656464376234636432633931343430383838373132363662663734333939 +62393163343938363933653932316465656365306461633131663935626132646630303734333236 +39386234633766666164663639666561643432643835323036373137333034663439353439316666 +31313165303436356532303930656461636235353638343237313865383632613834303636303637 +34663439643065653262313166663761653638376465356636356636333331376536333338356231 +35363033353236346635363539336632383330636664623633396138323865333462383238396361 +63633435623137663539656663323061316565363736336665636363313834376438633665633937 +62646463353138646434313537393464333865636431353931343336666664643961303633633930 +30663531663366613263616364646139306462316666663131313634663730396238626338613631 +66333132663837636263663763386231363631303363373434663562653061303061636334353633 +66663138356333386632346432303230656238353962353931653261333464303733363636623132 +30353137303663613332353736653737326630323661636166666339666332366539653930616138 +30303466613834643135316365663437636536636535383735383238346132303637356564353564 +33643333306639656331663763656333633733303735343530653766616665353835636236663935 +38343365353130356364663032623230633931356238326366313638353837393339663932366539 +31393132316136336533326466343935613931303362646635336331393461623838656164373337 +66633239326337303730373164373634333965663135393665616534326233633965366233383339 +38303438613561356464303033393836343837386664653532333036386666346531303731353938 +39623665323336306162306530346463646131346664333137623166656339343366353532643563 +33396538613965303662643763643734356333323066633337323563646266643930643036353339 +39633862363361383931626535363436336536356631353830333230626434346530393434653263 +38356466626233323361363464643861383037316665623162303738363338313164623038613832 +62303936336232303562363762306537653539646430316430656237633235343763376266376530 +36613862623865616635396335383937303163316132333766326433383862306437306630393037 +62373233653535663639646365373534333039346332376564306266373534313435333631666638 +33373864346435366534386632346633316462663261346433313838626232333939353638383664 +62623062346561653133363265663634313762396235356538313038336233326537313963336337 +66333163323933323536663561626538633966656562323263383563646133306161616464623064 +36366535333733343161653535616264613833386136646239326339633563303764653564366336 +38336635626232396664656137653336393639656165613832633235376535343130616162386434 +65623832356431376432613332346336353136646630363963303534633136663566626562363736 +35303564376264393931643261396662656635663936656461663833306536653135363038653938 +66346338646336343038323062313930303238333664376535653062616263653864653133623331 +61636663333730653033326633633164313034623234376664623730333662393161396336616436 +31396539633666343538646364666133353039363432613235613761656532336336353331643566 +66663161353038376332303235366236333062343364383732316434393866366561366435643166 +38376538643532363233386436623265336464356261616361666138333530643436643364386134 +33373335343838663239623262323162623639666438633239613165313435616630643134326533 +39313639343936613061363634633665323839616362616266613134636532333665396131663239 +30636136363863353064316339656236613861623530643735303939626639376530666135653338 +39393138633761373262316432363832383962633966343361333031653164386461616566613066 +38656662366338326538333832316364363466383462316432306361646137636266636435373262 +38646462656434366664656631623066383663653064343164323732306466633633376130303134 +66393661336337396234343730313731323530333034613533633430616562393032326466663230 +31353563306333383138633637353161316434646562376430323538393431313165623765373662 +36636364653364623266393363333262613066653538613866626464646664636231626236623335 +36346435643131613334303539316132366637613337663639303639643836663162383332643638 +62393062656335383933343961363639633831396431356365303333346533393530663062613264 +32623031333066343838643239623266383631616230623063343961613635626230323530343831 +61326162633632303833613464663636616165643930306533333734383231633461363963643732 +38353138653834323461326639663138373134383166343930336430376664623363666430613439 +66643935313936303565313962333132333431313363653662636666343865323530396363323835 +63623132623565653936356339356339626361626535316335346639313438323536303437326361 +38386263363238633138353435346465383439346130373331353666376335306439346264623939 +30373030633332326663396337636163633332383139646430346361396634333333336135396433 +61326664393365386334306439356463636531653866383739313130663662363730353566623031 +35373036663930653362346165383864636361343866313961386663626534353436343561373137 +30346366363762336363393738616532363939376536313234363966376336316337633235316338 +32376365396436303737656166643637323033653765343839376166666431623165656231643936 +66633030373232353862326561636430643436383738353032386666393931633832643936393061 +66666531313866393637343065376364376661303838313164666235366137313639303839363231 +61343934303834616561316634363239646433313938343366363931363738613130353236323839 +34306535623230643234613736393935323461303030353730386632323830653732356636383064 +63643038353938656466356533653933336161303132363831306664653465303163346637383535 +37353062643561643737656164303332623636663038626639623835316437323939356563393564 +37656536633931303330386363633366643936333132316361626438633235373665323932633463 +33333562326561666161356136663064363162303430353037393734393863313565363932363930 +62666135653236653533353238623536386462653563363533633265306138356661363866376638 +37356163363830656562666265313864366239663137646562623766383632653039316538653837 +37343637663335623863353733646339653832613665346362626231333066313031626637313332 +30613431336635653736396236633231326662316563666563383036313061656463663236623262 +64653966633833393566343139323837336439326532336663613534373462346265323761323462 +35643432643331323430373130356337646130623436646334306134376461343965633832303839 +36623761346630366336616266366437323836303739323534346137393365646166663830343539 +61633361323836323366343361653930663162656366356166396131393933636561326133356134 +64303530393266323831666338323838346563356436656433363834366235386637333364633664 +35613432353633323262333638653632613530373961616234326231613737336333313935663337 +31373136393932313032363130663265303235343061653530626564616362323264623966326230 +61636536393535326334653530653934343734336134326232376537373232346535323462393464 +31343739316562393761373931306632333661663736363832356263646530383435336563653232 +32343262303365353862333638616263383165663639336436323931373836636236363862346236 +35383963373031623736336666386339373766386164366462316430653235393534383562326135 +34646137643134373437646464316233336166393533663531646633373133366534353161303938 +63373639653939656536323833653334366635646139346632613932386237313932323962643562 +64633834613661646461303130636130653561323837346134313761626630356263386633663130 +63633337333165313930613535386538643761613361666462393764343666333733643765626563 +33376137396265306162343364656665663864383135643162636562333234383633666535616662 +34666631616436653939376337396133346235613735666135393834366265623234636132323766 +62383263623862643464333934373666643336663832323936623761316634323339633561643333 +37306561376166666439353033663136356132396564326333656536646562346536333765363464 +63643539373238316162313138663131306136623163346664323333643737373039366435373631 +65656332313162366531303336313532363766363736663139633831353836376363643337636330 +63646539353765343938663635306130343964666534633565643832656232633633376133396264 +64313438613534306436383631376265313830303961616162346466643438323066623630636333 +33356561623462626439623539323931613834373637646234303532303065636332633038656338 +31643136393339633364396465373166623663636436383036356632343034623064643934363430 +30653763386162373962396535383961383238336632633861386533396638643465316435633335 +65343262356136303531643063346265356263323030343032313334613634356639616532316264 +62643162343164396533666466343931396563316636633163636563653436313836613636616130 +61613334653064636136663237306636623965656465343836663836626137376433333665396535 +30323431323131666666333562313061653933376331626331623564366134323464636361333861 +65656463653565366235656139383561323665653835373939343931306665623930396439393432 +63393539316231663963633162396665383965643537666130613438356663356233363535383031 +32613535346236666162343966383934306537626162623131363935656130386136613061356234 +33346232613032333937346432656334383139623135663132383361643139383736393739376531 +65633231623936336439316235383536396235336536643764663763626230626566666630393062 +39353535346138353166336135316434663463633830636562626634653734346131373436386537 +66383865396161393434366566646134646262363362613965313133613037316238666263376665 +38646238633336346363643231626665386538656237643662313039343166356463376230646361 +64626435346530393664646233303366313533353334636334393834303861616537636231386663 +33366565353535636130343930613461633734626631373962663663303761643538663437346635 +39333732386366363333626432363838303565353234303733383537653434323539396430316536 +33386335616164353635323438316365396338633166646531656334343266613361303433666564 +61623763633639316639333536353835373963633537653637653030666663386566336538383436 +65346266363365646231653635633761616232316239393461343834643138623761303331383463 +66363132656334313561656162323233613763326632383262363833326339656638306565616137 +33653362343533633266323339326432393339636162313266643636643836396431653331656438 +36366330383636376663663532343166633038343861376531383665396465623235353661663034 +31393037653835333838356266623633633831643233346537353633396466323234313434363865 +66346361393163613338313762393262666662656535306565616534653137663436376239633732 +62383166616431396166373461333466343861323135316363303733643037303665396136313464 +34343830373535323536613465363431353937333238333939306234336133653939313763623337 +65623862643830336265333933663331353738333731383363646263636632323832653862383937 +66393332646463313933326236373337383637636534313934646435343362666463383737316565 +37633331303564616532393462643831653561613162393237303337373965346664313064336439 +34303837313130656630366635613633333635616365313638353734363033363239303332613933 +31396664643731396263643964616163343639393039383064623539333632623136353238623634 +32326138336262313164373634313961376562616665643462643261653633663930623663663533 +38323762666334333733656666373664626333376537653635356631343031633665366364613532 +39396364633433613466663166306337383135346239343739366437373862333566363735656365 +61313135343763333339313361663935303835353134323635313461303734393435373964336235 +37386665633866313364633762366536333066326366646636333934643964313732653238346434 +38346462323665643837333937653761663631373431316134393863343563313336306232333830 +37386134323662303839626435303437616363663539383232383630373061363061376532653230 +63643134306330623534633966623430636636613133333032636130306239373137326331646333 +37333965366432346330313566393063323233646231343333303737386235336263313866343064 +66653630646562313563343335366365653232626565636431386236633334363730303865626163 +38653963323237353066633862616332663936326230313638386461383837663264333164363339 +30303133633464646430633034383462636465333034303832356331366334326135363530326332 +66353934653536636639356430663935656535306364623136666635373337353366613733663134 +65633731333837626139323134656635353235373463333535613135346436323931633735626535 +33363538313233666332343337386330656461623266646564303936366634643933303362353562 +32633239333862623031313332616361383538336232613635383864393239666234336232623762 +39653939326264623465613638613131363962336237623231303336383963633433323835363432 +63656464323433633161316365333065663539626262383533633131633063366632323430653064 +36353566333138393138303138396330646131653034653034393833343631303039306661393833 +33623663323637613966643636626534633737623833636362336631623464303835303536383035 +35333130386131636262636438366662346335306432393137306633356236626664386163646234 +32303663626462383065363264653030353438353639333366343232323937383165313436376434 +33656531653933653230636437653031633465656633316535343030363564303830636631616465 +65343636656330306133396364623434333437633036393232363534393932636564316236323836 +65646339653566623934666164333730636364653531313036336165363064366633393166316465 +38663839313934376661636663373839326664376536643932323763323034363534343263333766 +62336633323732663832336137646235333064646564383761386537376137656464376565663732 +30643061313262613638616633383736303462313564643539373333383864376530346463643934 +62303132666164616261353163623731396630653230313834646363383463626565616631636136 +64633032343133623132346366623264396531306334363362363137343332653037393639643966 +36313565616464663564353739643464336639666563333631343062333561626337616437376362 +32396339303165663663613130386534383262643331633030366435646138613462303938383333 +35316236316133626430383535353935363863303939396665316630333838353735396261623934 +61616632336339393630383030646630613839316333326530656136306364646636396237636564 +35643331653964316261353931393738663863333437633735313633623362613861613133303534 +66386536663139623036393632373135356266626161373335306163303130376265336533646236 +30646230656661613565363438383064336336336530666566396365623163306331646236343063 +62326664343235363634363264323564313535373638633365646562386132663162333463633364 +65333438623839663466626238356336393838653033643330363732663031613732333339373630 +64323862643766323938376239333632396634653366623062646465663135393762303535356463 +39626665656238316333313636353337663835376632393532376462353332386532393562376262 +30376631616666353130323263353937666433616337323634396332303836323561653465356236 +39626534366136323536613739396132383863643131623834373735306338323366303361323237 +64643364653262393931356463333230626636663335613164616362333463633765393235366439 +30663862336130326333633734643965323836326337356262383437383433326236633833633661 +66623633343161366232373065636464343464383038323533353330393664633337373362613639 +61323533356135653531303432316333363234346131303938333936303561386532613236363465 +32356130636539376135323531623931656631653234616231323436643931323032303634353832 +32346335313966346663643537656232646363333933326431373239623065666162643663316532 +35353963623431616233373335393164343936393236303334396232623138353434326365636230 +38646463633736633165306530656137326365623532313735666262613132663961323232326437 +38616637346261396664383637316365323862376666653733356564393036346436663033373032 +63653337343631626539316431306630646437653563623035616636656334336138366239326463 +35316464313335313261636637663230373162346337313264323563656566383163326365663831 +37623939626532333938346666393134623830303463393865386561383464393237333230393331 +66316434653735643931353666333534386133373561616134623561393436353139613031626437 +38636433623037653235343638343865323666356439383832663034643565306635313361383566 +63386336633661336632623033323630386439323031383037303835373738643934636533616439 +64316266643236363130666336306134666338303434643366353736313833366266643531373239 +31663263326135346563313731326138646638663432393730633661373639613130346464643835 +65366131313330613764626231356365313064363738623737356261333064383130363165666264 +37343062333831373434383732323663633534323864623935396635613931396139626339623765 +39626135616139653031613838653438363731353230326661366566653534643538323431343531 +64666262323831343935376630656566633034623564376635666661396365373935303764346330 +61333563386631346638306130353635646136616666613937616663313362646233396337326439 +32663233333031653636613166323466356239316166356264373437356362633663366339376332 +31653561613461623730396265323661643532343537303932323830393334646338623031396363 +66626363383235616633383161343865653331386431646337303431613035646438363431643131 +35356638313465346236633134653137306335333831646261613162613037656633353736353837 +37613063643735323665386166646365396431306230373237313030393238366661633235633132 +65353266356662613363373136306632343632333931613834626436333765623033353636633230 +31666330623737313064343263663965306634393835633635646162356166356233383134386133 +34656236386365323434653961646334633262383431623433626233323666663935313635643466 +38383036313537376437643264643666626566616562386538373133666166366464363731316161 +64306331323838313332306133323466386463383637353461656366656134343863336331623764 +36306139333464663531323132363738336265373231653536323331333964303165636433636632 +38313231636162643732633235646430636630666439663636393232353664386330643135383732 +39306566343133663436646262346130656464303231643735393265336438366164623239396538 +62353065353166626633623538626261653762363362646161653861353664363434666433316335 +63383739656237346462396136616466313634623330383133653137616630653234373533656138 +37653531343961656432613134386334613039336461313632373135616435326266343733623339 +38363161656539316363303435313166303131623965623534353232653566316530636262656431 +61633836393836663663303135346431326338633163353362303062393733306436623932393230 +37383763316166306438626630663034366462313362326138656637373435333566333537303731 +39653935626536663934383138653732336463663465356461303130653332653538366564353832 +61646465366338376562636231316463346231336361386664666464396564323331323137343935 +61336261616635346431393064343035306261346132626136393233383031386636373138376461 +30303039396461626334366664663738303763393834383032633734343636343231363834383134 +33353461336361313931636632396166353066653237383838653632616434393839323837623636 +31633163636635363765326336626631383761376633356237393664613962313864363734646135 +34386232323166653462383335626233326362663931626635656535363534633531316233393736 +38376263656565373630386633643439393164626632376334313034343632613635356234366333 +39316433323235646235356530373564356539613539386138363032343334366334383661626330 +34623530343639393431656130303635383064656232346236323935656464373736366631353639 +62326366613465393566663431363531623730333862333630366163663937363338336461623566 +61666265313432353730393436376632666363303165663465666339623937303334366139393632 +38396634303834636539343438303633343630653833356332376331313064656534633131656662 +36373634306663626231626566396530373032616432336164626165313365366132356634393639 +61353335613034316538316239366138303138373464633633336232653830386161353932363737 +61373638656630656466313065613065666361316231623162353132373565366461633734653038 +63663866633932343965353062363239363665383031613637386338366135346631323162336664 +32663437363832666136386162363264356164643839633863333531316532643234633561323936 +39363236613835616234613339613632346164663864373562356331643834333237366464383066 +36643935613863613135623436653966636261663236373239656531363262613339363938646232 +33343566313162666236626666626533343433353138303937323066303433393064633235376534 +63396565313266303263633465363766356339383166376638336631343064386237613331316466 +35623565653631303630376232396462303163663337313564663837343031383264376465366162 +65636234316636316132343536656235316464656564363266633862616433383434616336393065 +33333462306434656136643833353761393831303438666138323034346636366464326361626532 +39366136363664376238623366303934393939373735306637333566376331643465376435663536 +36313339313630323666336233613534353735353132663433356565303130643033393263393462 +37373263616135613665636336613464396532366436666463636330363139326236663063313438 +32636235373764646637643261343530356433376361363037393330663664353866623865316237 +64386232653366353465386438303231383936366261333532663661633566326634333033356662 +62326163386533373835623439333735303439313430663135613332626537326435373235356463 +33393666643736313233343738653463363430366261383539633236393662623163323933336463 +33323035666633613962343465393236613862343866626236396566626539383564366231323830 +36303030613030643662653964366432366433663032343262303832633864393930396138666533 +32653939343163653339663465626235393038333661313231376531653432646238323262613730 +32643136393163396133643038343763343634346362303135613863373566636333653835666336 +39623638633466666430636266323865363163363765323034316539613063393961323039663134 +31323330626466623533643966626231666637653664653236626438336537346561303237326237 +66373463386361323232643665346438623766383236663037656561336233336263333931653039 +39633164663361353330623237646166333164313635366338663365623366623934666134306430 +31323365303139626162383562636366303664363833643463323132393665653830633265626465 +63396239633563396663343965663736393330323138376337356665646235666266663064626665 +65633961613361363563643632646331623862623630376632323066343761373465393930373032 +64303237643361626362306438646365393163306562393466376465396337363335326532366231 +34643630343037623739336262326137326561323133613261643164613230323131656639363334 +35663764646466323132393430353164303939663964653161363730343966326537633036626438 +33313834303435616461613930373137623433626636316638613166393932343362303366333561 +37343538376330626661623661653236343165353433633031393462643264333365616664396566 +66343738633638396639343931383536613633333134373430386564383661396366663839373463 +37373438356337306362333133643136366163313835323865626239363436303939313131613433 +61303439333862336532303063323430616331373330663962326130653936393930323630326537 +62323635313836343766613038343139653033666432613261636661396135623139323530343331 +32366235353162656535656538666461326237373236396661306630356433616666383663636562 +32643632333633313331616633663264383637323339616165373232306365396438316266623838 +32616135363161383362306365626566393037646135323231626432653863613539663661323136 +66373366626263393165303636616165383836656635653439303533346239626161326330643233 +63646664613535333665383937333634336633643937393463313836303131316264633266363637 +61343930376661333762623337383965343962323866323766363161333433366263636665623933 +61383037383737366662663531373430333535356264373831393865323132656165373135383865 +61396664356161343465353563316437306263326332646130376331373230386463336233313362 +37366166646238396465663637646230653532393537396163643235616631313139666566333231 +32376330616233353639326539313136663566353063383762386535386638396335363330396135 +37633134386138643436343361306464646337373961643034663861313864393065383965623164 +31316238623663386437363335646231633338303563393466353032333761313835643661356261 +33356235386436643939353763353138386632316538313830336638643135336337376338643063 +39333532653665333965636132396262383334353866383639363831396438623165646264323466 +65663966326564356161326661363161653434303063383031313664313563646363656637333661 +33343930343631663363376465363361313364656339313961353738393036366164643664363534 +62346634613234363437623432356336613764663164316335303638656664393663346561303762 +61336533306663336265626232323361353531353365313764383732383034353538383462653561 +65626461626266623065373461313966643136656363313665393063663430393765623330306466 +64656431303264333466353966653563316331646336626165643031353835633939323566323830 +64373464656663343232336139353432616331326130363432326438383132373264363139373530 +33343861306233646365366137643637386135353436663839356533613865326437316236666332 +63303232353630373132646266353163616330643332626232306261363531636133363837363136 +63623735356263303135323331333731633266333135623530373161633130376334656366313837 +66336234623465373837616431366164656539643461343830633130613934643163373836636536 +66303063626462323165373531383764393038323030363338386133373034653931643636313538 +37643139613863363535353161656265323337343035616363393934323962353938336333653130 +39633562313337303064626332653831646237396564313330323061653338613766316563663535 +30633564396334326362613261396138613662343365396639326166663365306461316632633964 +30343139306637373037306433386636663230663264343765313762623262303239656266376265 +38323835633133656166376330383139383165613962373437633761376262643937356232636363 +32306332343263646435616564303465666561633630343131323866336662346534333737393361 +34383635383430333066383263643765323938313533343636303665623839383361336535386431 +65383161363966366238343434386338633638396533363635386133346261643534373061393566 +39646632336563356366303162316233333833346265666530653465386230366533363839353366 +35353335636138326430313137663838303465363338313534373133616238386339613939613638 +34383137333436373366303735376431316239383362653830396636353863373533626135613738 +37386132636230313635653239343564643436346535336339346232613135653934336162633836 +32393366626566626138663666393463343766326131373337326536616562343739663037393034 +33663333393237303463613635353932376535336562366466666364343765663062653031343538 +34333833666461633938326132323865393131316239316634356335643938353662316531313639 +33386337323138666364643666623966366362633634343537373465666664346137313766393739 +34386261383564313163643866353530303734373337616237613135313132383161636336303339 +31363964356335626463353363613236356639373962643339613631326338333038323535623161 +64323862353539356164613465383965316637313436633130313165396338396232633536306331 +32316236646663643934336265633838646363663835666462623065666339666363383762613538 +62323766376561376335663330386131383965383338643435393935366338363636343466323234 +63633263343366613135383263383135656230383436383032336237336265316432666461376664 +30633666616133373230386132333863383562356235656132636237643762333465313861383936 +35303065656332393737313061306437663763343532383363623634643837343038653034663432 +65373236613736626163333636346537383032636461386131656164353366353934326361373165 +66613930356466633862616464653336393637633865626461343563383763303066393736616666 +32316632363663303663333639326565396664623530323261663837373064663965333763376534 +35316435366264316664653137303765316663393761663337623566373134353363346361356566 +32363436326161373166363463653235353139643232633737643632316133386238356561326139 +65353238383833636430323836343837356532386534653466336361303830353863393630386366 +33356332303539383935376433666362383363363833323461366661363666363865636235396332 +38333736613637316136366538356665393766653264333061323735336231623764323239333132 +61666132383836303939353430646566363361396432323661643232613639636464383164363362 +30363862656237626332383966343834393265323536633631633330386637643032393164373263 +63326433366233333239656231306466656465373137373363303266363161623661316130303231 +36363161666430663939353738653235386566366337313463656666303135643863616230376337 +63323963383735393865343733396465353833323733386132646537663331313234316361373937 +30626565643538656330656364383437663966386366363261623234663165356161323563333338 +34646532333665393263653863303761323232663765326636396166623663363539316462313963 +34663461336230343764623261376339376435393761616262633636656434303063373138623330 +65633531646535363938323766326132613061353437373763383664643539346165313833383631 +33666631613039353037376431333065366465653536353565333830323332353066343931383932 +39616463653863326634636564363638643865663062666239316337666137653637313766626439 +62653362643339623563336134353363646638313335396235643335663464383061663566343236 +66313033363538666363336564323964386239363164323731343566306363376438343339616432 +32653734663132616561616565383365663164613334376438623739363338326435343232613438 +31643565316261326439343637376264303966366537613662663336616133623664383639613263 +65376262346232333234633163306636663238323233633564353963353262333661383666383363 +63636532306164323631306539306138643036626362633939336563653433323737373363623064 +33366337366637623838363462346231653534623531623336666535393832656331363063656339 +64623239326434326632376636363833353664386532363765663739366165393461336363336533 +63626137653462383738376462383264653030636232613537656463623664663931366539333138 +63323466313062373032343633336664393134336231323039333035386137333837376663333434 +62393732343430653238653537623831396633363133376162363765616564333432393839313835 +31326361636130303138363330313365663835643365393238383666306164656636623162386137 +62346131323139303465386436656464616139363731386461383832636163383736373965343065 +33316533643334326262643164636431633135386565626139643337363339616235653037633465 +35323834346236386236303430653631316466373534343539363134393635623661303438333830 +63313139366331356436626463656162353637666666313161393336313134303163653232376333 +37613638613364653139326262376638343936306132313937636539636533343162663439646434 +32363339663862366564653032653534383438656337656630653564373163626365343136646261 +66336134346435623033316235366163656339343935633430303537646363383834636336353532 +36396632653136313834666266383663393735316133376436633664633336636265393431343836 +34356466386162313931336336363962353766396139356265366231393437653032343339356432 +62663961623638643266313235646636333264636533616466333664653761373730363663306339 +31623435366239316632333637656563366133643863313365646362663030626564613831623261 +39336466636236656461373161313430363830313439376534323730613631313561663234346564 +38653830643837613763373238313937626665646634633634326432353535376230306466646139 +31386530616563656534616463366235373530633865333637356139636339333834373639616364 +36366165353330313866306230326239393633343838623761633666633266373032386161383161 +64323639383738333238333865373131633031363762326363633037373965353639616164343466 +66316638383237316134393836356661323637386661316436353630353365393963383866373738 +63653732323032383062663061363363396264336330313431303962363134613336333536346432 +30613561316234393736336439663630313335366535376634363639653563313965333137613631 +66356631326630373930323238346538326166386336383430643865303732633564633939373163 +39323761353733386431383431653264333435363764346164393062316364656662656163393063 +37313465616239376461636535386539393035643631333638643064306435346661653333646134 +64323637376337626366313739616166346361363738333834666231646565396462343430383963 +64643337343863386335396163643363326266616537306233613665623131353965303037646466 +36303338326663623739616630626162613130623030653565643466333330376664626161646463 +30616234333764386532356564653161333634353634373437346166343961303336336563343330 +31336330303661656633653765396433323639313763316238316132633535393735353530323736 +62613266636236343366616565643230353830653237393564623764323664373161663433336636 +36353661633933363265623634383736633837343732386264336365623365636231356463346233 +37656263313231376536393333393366663163656563323833646236376437336566396134396336 +38323132393838613835653765616661633239356235643266303366376461626235393932623135 +31633064653131646530613465626463653761643830623139626532343432663631303939363836 +31386562633533383063366161353939623263383464666261346438323264343234343337363338 +33303462393235373732643238313033343664656431363432383132613530653462623732353535 +37643362333763396163623639386662653262643061646630333466313162336332333765353461 +35323336323263613464373264346162613433366135613332626136356431376461356631633038 +66643436633266626431346233623261313965336666663131396134396366646630306436326537 +63376238336336303365323338356463653630646366373235663334373566353138653364303634 +35663562396662383238326565656132396532333430333066656538376139373236303231313133 +37303435623961616530373464316532376361663837363161353666323134356334623631356264 +32353430623030663330373461326332383037343963343362663636366532666539376336656337 +36343566653837616534656638656632376435666436316366656365653830636536323764393835 +66363665333333663436383933363231343638326163343563343664383066306166363334343937 +32363332333336353033343365653261346134316265626330313731363764653463336262323034 +34663665663639343263303839313764346164303834626461366631353335376239353336313834 +30316664326531646131616430393763363530383133396530363163373832623064643066343135 +30326638376134383434663765356338383131333534336565353130396436646239643265323639 +30343131376637633939643764613864663938353861656436373233386333666333326532393138 +66383466383537346366663163663137613731326662336639366634336336363638346234326430 +38316539376133383963653535333162616430303531363066343064356437616164653862376362 +61653465326562363466326566643735663830313838306238376566386261663035343431616465 +30356432633066633964336135343639353935353964633933386265313439386265316161323230 +65326439643134323631396130366562323837326534383536633861643238366135393534656463 +61386338306336376362626131323936373630376434393238636233653037666166613530333532 +35366461336632343432613130656566373762363663653035633830623133393965613464393137 +31346465616534363131393130306338613461346438393030643733643933666230663237336136 +37303263326463383331383465356435663631323664366234303534306232333834343735646234 +66363039656561663732646535373363346436653336333637356235346662613937353330656663 +62346363386632363132643537616534366335306439366231636163313433633830643239396564 +34656130626633643134393031616635373635613063653537356531383761323636623236393965 +37303435336565653466343831323231306366333066346638643530376431303661646138376239 +35323632343066393639373462616262353835333932323534393035643938633731323032666132 +64326438623562646262633366303034303134316465613365373832356331343463396232346337 +37623162643936653833316437326263383735346234383661383436386234333736343062643733 +65613334323037333332393964306261326430646661303533353431373332626434393432363537 +38623134333764643637313636633263303532666534303535333464643965353234346531353535 +34326366303761643336323333323833653438383639383165616633386664373435616337643962 +33343065626162303265346135343166326662353136336562343761336164313531393233373437 +64313632306639626537643462666435653265386464666564623937396463383762313131626631 +65303930626531643161313461393438343965633732306337303734653039646235356166643932 +36386665376132386661323533623032333863303739363732333263616162343432313762666334 +66663961336239333036333962323764316238363466373637373265616430613531646338363034 +66633762356137663435303932346561353735343034666232646264383262666434393663336562 +33353863613362623464366637636434303834613135623865376433636531626563626261346634 +32333532326636313363323835373264353539326431333635303338316465636461633563653466 +37633832323830653065616465343639343536646632353466303563633433653930393863613232 +66623863353038616330313837373866613166666666353433383662396338313162653239353937 +64383839313665323534303330396363303238393562313762346164663164323637613730663236 +38356162346164666262306632383035343137656337613364653033393736316464393764643131 +65383836613263323065343130333437323639313530623234346534636530333833653964386634 +32303564376437626534316461326465336535386461663662326634396364393461663039346534 +32313437373562623566666437303932343033353738393833613866326132656462313530613036 +61643937646235316631373532393864643863333266333363666439653838653335356564653535 +61666638363733613462376432626266303532363664306432633638623931663233656366346434 +37343864653662663062313531396262323031336665383733336331366335666639323733343565 +37623661313736636132343435353266333134383139346431626666356137316563636265386132 +35383732366232623532626362393162636535343535313538643536653535623761373233613530 +32646365666637376666313662626137626534386631386132643839313062383064663466646234 +38363663323933303639346237353262616261326262653634303436613039643561633534343037 +39396465323566353062336335663830633264643165313832393561333532306432316239343064 +38366366326537376231636139313066666238346663633962303464393433336164313461666137 +31373938653963636262323863316238376533373539323764373836646532306139333264343037 +31343464383262303134396565393333336234313062303366663834356364626434373931326634 +39653134646163373036666533376465653830333938386363653962623130636234626334313434 +63393133626337343433383262656663373463376666656130623033376363373361626566653734 +32653034646662313239386639376530313732333166326263333831613338636437383164613938 +61633263633562326234303762633236386636353664626439623135646232393563396163633565 +33313165613930313330373639626436653032653533623163613966623365343463343263386561 +30356261343639633235663035383436323463653663363734383334613438303738386138613261 +36613761306465346266393430336436303532613131333736643732386539636365303764626261 +64626237363035666538396265353637663435633739346638373234633432306666663836383964 +31356565336439386532353634343861666434343439666662636130313038653038613731303632 +38636566313466613662316439333165633938303038623531663965643461363934363339623435 +35633530343434303336633836336535363630623930326236313565343334643537623663366138 +32303934396464626165333136336637643461396335313233393666383638336362633733633235 +33636439643237623862626338313166376462633164643934383339313461613061616361343833 +66353932663761353738623266623033393161336536623135613739323733653031643032363234 +37316366636365663237336336643733343936633935313235373536386431343637633731656336 +31633763613164376665373834313864363431366439633263363161383739636132613266366161 +64343238393937343337393333393661653261646564343935396262316232326262306362363864 +64396136316533633164333831326531333932643530656333646534623461313136313363626265 +66396530336337633330316632366565383136326666623665323464333239616564383933643563 +62616238396433343661353361393764313430313165336232663030646361343031666630353561 +64303930323665663234343638613535373261343066333065386238343133653734636666343039 +63386663343336643931653831626439653463373266373130373038653830373337346132383336 +64376662656164613138383830616430393431616633626430663535303931616261343332363439 +37633664656535346338336535373966633339303866383230383839343136333136353563313066 +32643233323731663264663061663562353835636564363065393764336435323062653162303538 +61316131663038366366346664306632643764343462633162313861313937316133343066643532 +61336332383437383733363435666230353339303831363030636133333637323235636339383666 +35383063363763363266663162373333336235303464396630326336666635323030323266663832 +62303465653230313033393362613033383030306532373263353066306437646564643938346435 +38656335333966376461313630376231306531316565653062356263663430396337373833343438 +33663730633136616664343262356165303966373165633162386537643937356363383331373632 +62333538623234613531346535353531643031356530663638306337323630353535313662393961 +39643737356536333336366663353238633337316430646165306631323632373831663636333166 +37336535623166376234363035326332653936613739336238616565613438646639393437376433 +61656162613439653835326264623666666430323065373432363961313862323533313939393631 +33323038663030613561323134393561343232313633643639623264666636353635653132306632 +36343662343762336261366137373563316365643435613735366235363562616433666662623038 +38353962636530366361643035643061353335646261323836373833653035663166333034323161 +34663366303733356563383630643066313231323034656134383837333137653665333139616339 +65386134383763373636643538643362323839653565383866356133336464336166303135656635 +36343130346239383762303663343761373031343962333534666162336233303165303262376433 +65363833616261656432373465373462396564363564303861613632346638343564333437363838 +66633831633938373861643632663932343266323561373432316331633539373264636430346330 +34613236333561323837613034633036303263343538363461356234646562613335663338666532 +65633336336464323835626565316237393033376563656333633238303865646431643632386334 +61313463653732346432636464663263356331613336393865636435306133306533313530616338 +30383138383237303639626261623136316364356237613066326566313433623937386439333335 +63633164346331663864633236656232323438653564653231353834393434386661343562323162 +30323161613632386137326262626665396334616434636562626664326562646530356135336664 +30386263303363623664633062393633353663373037326539396430613439656533363735336534 +37333866666132356235613639383866343162346434636437356139613934666263623065373236 +33383834353630396266363736633633336539323662656363623634666537383938393465333966 +65393237373135303037363531353964383333313431303535376231393934373732613561373030 +64393837323739333061653061333862376261616233636435343637343037313465633461356234 +34613361343361626238303264363537646432373338326532643636303433323764663434666639 +33316532623538376131623463633434313132363635393339323664333731313563373363383036 +66343032303835656238383237663066366530353963373163306638336135613634633932356332 +61616432356233336165313838633266643930333361356361396639313737643561643138373661 +66393532306135323533393833613538356165303734633037366330623861373165633532366466 +30316236373430653534663962666361376265303330613130666461323437663733306236383161 +63306236373539373037656162613930373835653537323366393565393632373431656164346237 +33353065373332306239386234323764623062343138396332636561663930333138393332656133 +64376162643464333962363631323463663238346430366566386335653635393266646562316135 +39626461396233636533666265373666396431636139653364643735303436333162633736656333 +66623466326464343234613465663661363335656536386263373035326230303862363362643435 +30303466666563633439663339313035663832366265373031333832636538616238623233343232 +66653239316163336234636135386238333038353934663931626434643531613135363733623836 +62306462616332633035373436626238393931336134336436363833616339353035653239613731 +65643262386566643930633661326533356635323431633932333739353966636339306436343763 +63643731346463623531376463653663626135623864393964333562343265643165643532313662 +38333566653735356530636438623562363630353634643434386435383931336539303933656435 +39306561383936326130393662363163313432663562323631343539366265316635376234623832 +34393832383166663237316137333265653765653138623866346263383561623566363138383436 +36386134633631656565363833616430346563623939613830366134646439623535393238626530 +34303763323865613961313863663363313565646633663038346262653733336266333131643439 +34366230366565326662633335666165643038653638636230613035613764353462366438353637 +30343039346665333466643362383161613038376534623635303065663463313066616233636562 +64653965363564396134656333383237633238363565353037363139326238353539656636663434 +39356662383234666639366338646138396365613361306631633461366239326430303566643233 +64636462383736383438343331393130633133313136343236353036653735643336623636303235 +64376465643364383336363839373465653936646165346232633739316232366238336436656164 +64303465633863386364323064353834656134313637646166306237643132613133343335383531 +62343063326339346534613964373862336135626430613837353535656165623065353364623530 +65666632363739366464363337363334333866346135633332663762653136663334623734643937 +62353265396631663262333562643638653433333139653662633638383531643037303964313535 +63663564383239343830393332613032653463666234346235343365396239373866333532393336 +61303336373461376334623030303664343238383337633233656337643365353536323562363239 +39316332386262633633623932623136663933353439393631643630616234663434376662663165 +34373234343665646635373362363164323333333262396630646266313939613133623134363838 +34623838343039656137333063353933303734343436626431336131386262623838373233636131 +32353062663961346261333933373866616633393332623134633762316461373236353335653739 +37666262306230396430663961383231363436393564386333383061366339366161316336613737 +37373864663965376564333836363234623064323038646530336632333132333262376636393562 +65663336326632623239373733653438323961316330363331633362356433656432313530303763 +39393039396630356663366461343562333234636638333364343937626330366638323839653432 +32663037653565353934326665653132386261386534623230373532653162383233306238376261 +38306163356366353666623238366431626533366365666236393564376336383836393835396538 +65333565323233306437303336303064336635643936653138653537316334613462393237356631 +32623230626263333638663133613033643664313239396231316665323965366539343561316133 +64636137646334653531343366376635353032313538393262376431653132393962626463383435 +65383633373265363936323363376538663566313735346235653430353537373937326436306631 +31373830626230366663653563396366613537343139353762613561653131633266393332643163 +32373534666564663531616431613936383731346238393661313462616131613230396531643430 +33373034393733613132303166383738363338633338373362323938386133323362373630633662 +63343666653437323063353331313836343131653464633563396437323232616531386539353465 +36653337626539303866356535396335646331306139646137633038633964653335363163366433 +64356132643335383934313135326330616333386235636661353933316365326363373263343539 +63643036646537636361323162633338646566653134616531663931383636343832393761376462 +37636165316162313238306462336139323737386633373132633030623837643937646661616232 +35666265353436326634363431306238646563303135306164323164343562666238386535303562 +64643231303633616138653939356265366666393162613235346132313031646533396164383263 +38623365303030376133303632306334336235333238653965396462646365613836346263613032 +61383239653561383136353930353530303465313166303237633331623662623463323632313330 +66366465363830373564356436663265383534363033626430386361386161343638396330336132 +35313335626661626563653162613465663738333439626338626237633766373939313064343632 +31393639353963663365313333623266326232306264616333333035363431316365373138643239 +65643433303935306230353763393538373830653163356333363131623033336137613064363630 +38313664653866363164303864366331353936666537393133333834333831346634643361376632 +36396530643561303232393738663763663532663230613062663837333835313037643739663637 +38396338326564333765656139336335373830383462623962326435346336366535376364356639 +36663931303437396464373539336239633433383562396634613162366335613966313537343233 +64353633363964633138336265653730396430393835333138663236366533613737363464386632 +33636132363334633434313639653936393231653432336633636366633166396639323064633831 +64663763313239646333323038396638316632653762633632353064353733393039646461633631 +66313131643631323632303139646566653661656532343834626230343736346631303933303838 +34313163636435656436343530663036303530343535356439626137393561373937653963326435 +30303166353938653336386166646465363764656134326436363135636435363839623164663263 +37333939663866393166383435316636303161346264616562363436656461323033356331353566 +65653463323231303965326665333939643462623362346433666437396232313965383766653336 +30356137346238383732303330356235663065376563376662393566306630363333613233303330 +38306639633364366530646566363863663364653133616531303135343665636335376330623338 +38306565623862633438666536636232366332383032353031353531633261306430643162366531 +31623565303637353166343130343861353736393563323830623062306539396339643639323233 +35323331626165383739616561613139363538396334623166623366323737393265363361373135 +36353636363965383165383330626361653565366231643663386436633965393466363062633962 +32306331626161366363343465646137653230313734353334393935343032663237393239303835 +31316261313564613562396330653861323865663862346637663233613732383762646236346537 +61396333333634323162306133633861353333393738316337653638616464373430313863616462 +64626536306263646438633136396636326130653336663438393033353633346563643261653431 +34356231333735303136363938373732306538386336623765663233343635663132653332663239 +66663564333239373136613430346233373137366165653833343264336538376363383436613866 +30373735333838636430366136363462643438343539303335393836643764383464313637653130 +31326436663431643737386530393730356138343632353265646261306365303362323761383237 +61373430616238316436396430626637376662366630346536643761333138613633653264663737 +32633437643966366434303865393835643833326664646237303162353164373939323734663634 +33383733396464383465666664303766326533366632643236386634343566393665316361373761 +32393532643436663535333061373337636664383233323230396161336263643638363661646539 +38373335656662626235663565373666353234613632363961303966356462303333303332633566 +66366134373230616234626466353833646639373034626238346431366464383665373231626262 +64646535306336313031313933353330336530366535306535396634313666323439396633656532 +61376630363963353432346566303765653233373530356366323666316666353938336330393632 +66616462376265393633336165643533313861623434323765343135663937383231303664316166 +32316161656637313131376261633738656266333661373364333739663737313233356265643037 +39633962383263303834623962656263303334323962353164643665633763656561333630363363 +65623237363134336262303635373033633931666534613766393539653338653535316461373338 +35303232313239383837323039656139363035656338386431333430356138353936373530383638 +30616161323263363463333633663565626362386230356264353633663863613237626361616666 +30363061333266306537643230303834666430613635303062356462383035323162373535336362 +34313436653366356463383935636331636138353433356435333633373939666231313631613465 +30326239383762636462663233623834333832613761323232646361663162316261303233383732 +61383366646132316565393337623833316536346562396639663561363330323339343831653962 +35363432663563353638623333336632333432646533666130633930353233616230316339653064 +38656437666331356538336633626233616166393339646534373233363032656164383436633033 +39646661643535333335396631313965643433353762306533336138613838303835396366333731 +35333535343066383735333264663938623066386535323536643366396239646364303066306435 +63353839303932383136386132373239396135346135666239663936383365633938613161356438 +61333862306639366535343936393639626234633466666637343734363062396466376331663939 +63373636643563313334346363656566393637623639353165663830326366616664363833653235 +61356136303131333939333130373234326166323263636534373033323961313730386534623830 +64386563343334343937326538663939376235383234313931343931316439316130306634333566 +38343637373165313435666139313530313135646366616431363833663831366464363936356433 +36653933353235633262323939396237643731386135393032346336333964393535393033333562 +61366635636136633965633761313131316366353234306130656666643363396431623238343637 +31383833616430326533373732353164656535303366393666636635336565623164616634613562 +61636230316162386635643834633130363665633362306238636637313563373033626661396461 +39636638363164356461326639313738396663323264326462353335613134633230363838626531 +33356534653530653135323537626564636330393039353264626134306530656533343533313234 +34313138323934376361623266333937396131366464346134303932353164343639363634363739 +32626661363335376331613839613132343532323435613230323265306135353966333934363165 +38313130396238623666356461653461613038333332383839623134626539653939393463393164 +34393735643339363638623762626430303630306665646364333731616234386564306339633662 +35613263383935316561383361616131303065316537353066653033373133653637653136313934 +33393863616630643833363636383334393934613136386331313539346230613338336161373931 +34303039366232643537313334313364333332626566376165643637356332336230656264316233 +66323431633931363662643563666631303764623338323331316430356134366661643965653031 +37386162623361373735353037663135646532623862346232343161376566323432356538633139 +62643136353661353565316564386131663635356532633631346335656130383566656362393732 +61636364376133353037366330343763616232396666303532376537386331353338393261633531 +30346435313536303638393933353335353734326437336262396165316164663138613934363638 +63323533306661643365346631626465623062323362313838383233316664616332623031633934 +37623037636465316239393039623939356635613261653131646135343064613132373637653530 +39356164623239303061343963343963306662396362366538653530386333313033366436336337 +35303136306531653333353561303564336634643736643865353662353466636432316665303065 +64653665343130643136323434303662373336346438623236656231316436306662626366373136 +39316262626362333731623463353137393461613561363134393037633837376261653035373530 +30613039343233313837393333376438663863383362363235336433343134373361643633633964 +61616335366634623537393733366333613033646631326238313135393463383634356664396430 +33346265353163383039636133343764323465333833316138386466616163373031643538623236 +30393532393463653936356339653232333331346239323966313931316538383034626161633635 +38656532306433316166373233653237656465313066346233333266363834366431316635646136 +64616538646532323663303239373061643930323431653233643962616432333231313730323835 +62363062343039376464663332363138323931663133393737363930643436343939333063303830 +62393630633339306331393464333639346636323534646136343064616263386339366635633730 +61326531353239633838373536323930326261373332636631613464363531343632633666356137 +30373135393266353366313034356432353065636535343532623438393037656333336664363863 +32633737653633386130386335313634353135336437366561353463663165613263336237333731 +64336666663062616231376531663736626137643334363439656436373461363862633361393366 +35636261323131373438623033373965383230623564633738643334356262643834346432306165 +33393339623063303362653236313336643763313833343066313964396562343162616530633438 +63306165636362326565616130623032613162386139613262353339613330346131336130343834 +36653663623665653434623735366433653033323430306531303632353437343333643066386437 +62636562663839636166383430363638366366383132326166613934643863353636643132316363 +34326663613030343432666630363935303462623666616533363765346466626564643362373038 +32653037386636323664353163303338326665656332333939393861626136656536393938363938 +30663134376336383831323232623936646264333334333664323063656534343233376330313061 +65383365646537323936373365393235656232353936653233343832646265306164343937616130 +62346332366537653937626239373336323337663562326631333039393433396565616335333335 +66336365396664633665353139323732626437633930353764316434653538653830316661366564 +37383734303866663565616530393262333966373833656236653938653761623732646237626466 +65636166343438623434633562643639623364623764336330643734356637323462306231363538 +33613837353135333636613635613465653530316630363439613162653139323339386438356236 +64356535356430346664373538323566623733373863623635393130343639366236656434363266 +34616330363030663434626666303339383134666562656132643738663633663233623335366232 +39623766636261323662363430393737343562383637373030366138363337316236383133313163 +37343031396432363037373139376366316131383630613034343035656232653731323331313839 +64656633643530343966346235356539323565313334636632623761663833393361643831643762 +32303030383062663834323966356531373635303336633237653430646133393738656434616132 +30616335313265356462343536373833316661663835626533656361643965393532346133656139 +38396463386663356333663633633461353137343436653834633530323762643833356163306430 +30303061313562383666613331653630376466633361633735353639613735383562373233366266 +30393635666464353335663362646630633732363662626566353033363762356663653664613365 +34326461666436323530656235623034343938646333303334633739356235626436396465303466 +32313638393130613937333865666338653738326562323930363561633965623164636530393363 +64663435643561323666303039633163343232323461356664303336366264633337396438336161 +31666431666166333630393961643363393637623238643932626337383166623165643765353737 +37636336623836346665653039313535653830636537636465633235306231326665663864346633 +62643834363965376665623263313031616131366132313336373665323363636439343238353636 +36346162373832393831343930323834346536663133623438363865366561306431613838383037 +36393436373330363663383163633162646137316561303962316436356534363232616432356535 +33343464336461656366336231363432626438663733616338616364346539383861356662666632 +35656235363165616663613633613062383335633638373633663435663931303566663630333939 +35353335633831633433326665376164383038663135623761353163373838376133303666383261 +63343737373431323037393362333362313531303734363765653638333262653863353130356666 +33633861306637653637643836343033616230353835666466386563666564663531616463666632 +62386432333765383635336436303739343734613636663634373533653436623761643237323639 +63363136653137353036396433316363613030343237386333306331643632343534303563343664 +66346231353635313965363233643538336666323136656361623831323564303734303937363539 +63653034366238333733386432653539353864353230396231386433373130663461323962366263 +34643335343366666161363665353162373437356530396665373338636137353034663733373336 +62313534353832653939663864313238363333306134373533353261373633303336363638623764 +36386633303134616236383133613361323334363539303764393936343432393431663864363164 +62373565336665356136633535383936353265306436613563303964616464326135626465616262 +63396332393534303465623234636636633966373061616665353932393063323233353163333030 +63623330393538323238326438643933616430376634363066613530626633323036306639613833 +36653963303761353439366630653735363532313034343337653030396332633939303137353238 +39343764353065633132393738366539396264313431323539383531313234623432306261333561 +30383934623864643735393064303335643238356138303135353432303134316464383239393264 +65626132656163326638633139613139643535393336626332323639643339623963636536316164 +61303563393066346234383439303166383831336635346363663239646136646464303339346666 +32306635363235613538316138363738353664663436623037666130393439346162393230653663 +65646533356332336332653862346366376632386132373835346338356431393431373735303337 +66313837363161663832376664626634353435643366396530366432373832623339356361396236 +66353634313836386365663437633437336365623032366232343262313833393636373034323037 +65313638616165343561653039613039613065646430303834363932623766633038643536663964 +39323838313564323364393662613237333333396438313465653362363032313731313833636162 +34643763663661663139616264363332333132613339343130306432316164313830316364393238 +36663636306639626631636161636266613761393165343234646639633538366563633366333833 +35663037613263656361386537386135656431306539313861656332393865373238343935636661 +39396634643165333563303361623164636365373133666562393264343564393236396431663662 +32666461313839336232613431663038333766613862376165653033386666303765373030636663 +34306631656337333433393262643234346534646464346234363266326666393766323462303230 +30613064386264353231643530623335353335363763636130366561663565643565343366366661 +38663762366266333233396330313639303733303533333932383561356230326236386363393537 +36316237343861333235656465333066623365333230373837636163633338333834663032353564 +35333730336666303162623130643364373666383761316137323834346362626263376134356666 +33316161616633653636613032393466613062363334623463613766336132623833376466626335 +36366634396234353561303538353236666232616336353337343331373663373131386434346366 +35636262323339316434653433643131653261353831323932396138326339313938363737396337 +39666164333465646265633831663335323738346331336630313330373066666333653466666635 +61383063323231386233343066363463333637633332643461396632383831383230623835313966 +37613939363563613735323862373735643762386237363538356333376133616665633336363234 +34353563376565343431313264623035363366653038383236616334636635643062663836613732 +61653263666162363732653564373163346365386330613739396433653634323065656134343963 +30323832336435363435393363653462663032336633303938336562616335313966383935373638 +33653162636432663162306161616536643961333735346436616662616631336632616665343662 +31646534333933343163363738616132373037613037626661356336666466653735656437363731 +33626130306433613063373831323863353535303066383733303031373762393035613831383935 +39353138626362393030616538353664626462323062353032363264313733356266353166613435 +34333135346231383137393936613065653436636361653436643032363236353765323834383766 +30666538393864306633356231343635636463623662653833376132626436613733363163373738 +37323539616330313066326131653931613432316362373966313433646633366561333531313365 +37346638393364303765306330643761333165353938346536633463353033363638316331663566 +37653065383531653430663065373365396139386135346237353731646264636335376137616461 +30366165333832383265623464316335313731663133383264363738393434613761356162346131 +61653635386337383363383732636533666630643265373065663737323161666433396331303638 +34643131356631343238623764376232626564396238386536373465306464323732373362336532 +37623234383462356139663766616135333963653632306166303863666531333263343533626566 +63353733656537366538613964623461366262333333646139373034326562646136666539303836 +38353030613031653734623264333837386163626238633337346535616261383065323764663438 +35623431326139663737363238316533303365313732613134353463623166343464646335386666 +62653536373433393938343234326165643862376666353437393663616536313965616137386534 +35626562313563363364323430653737633466343939373062636133633838656664623037333134 +64353230373037643161303931306239636234373039623264373362346435353865343239663762 +64663530323566636534386532326363346632316330303837316334323434316539336163633036 +66353138393131336134303665343864336264626432313865343935646235363332663264383339 +36306435666530383233396564323839646337616134343566373735316433616263306565613438 +38353862393661323962333439343233376462363265343163623535653435623834343033303536 +37313561633336313830333866353738316533343430363734663639326536383163613533313762 +36656165316661303831336362303562393633326638363434353735373830356665643634313136 +62393136313166633232633864306235646632373036333833353832326134646631613533663362 +35323061343630386232376362663662626564623865396334323064393333653435613566393261 +37356665386466363233663764373662396261336638393334346636643361643739396233613635 +64646438366464646566316463343261636365396134353336333365356238663263356266623866 +62646135393736376461643964343630343566663838373730333333306562333135663762303632 +64383333323232306237366132353262663664633633653137623964646435643332333833633138 +36663461613636623062306563313838663737336333626561353863393866363130356533316166 +32613430353966376333393833366332383833653931313330393862346664653265313930323536 +66343662396133623262353838633131633066383739303933383464333130306235663066616665 +66356261643933353766633635623263636430633734326635363239666263393739643937393064 +64303739333831306232386433656662336266663536656337343635666464666361643562326465 +65343137393234386365363337646437303865336338386437386666346463363338316566663530 +37303835323139316365373536623062646562303338303933363638643932616631316334633433 +39333364656463303561616231646435366533613033333332653963646361373337303233373833 +39333537306435626263393331613238353163636162353137646139623266313064383461666630 +63376565346662653732316335623130653535383339383539333233653839633761346632663839 +62336636663461653839313965306332636562643064646336363165313261613330376232353332 +65393062626535373835653164333132356466613563306163636630393033363937393632396663 +65386336396566353763613464323935303331313535303336643037313661613964306537346236 +63373038303565353637303962636137353837613735396366383131663430363664396636376638 +63336638396663393131626535376565616263383233316561363336393763376234613265616462 +39336339626135326638613035623939376235316232663263336236386531613461386261626632 +65303639313134616631386532383832373865666631613635313338346266366532383639643262 +31316366336162616434646464383238383339623039643238653263643665616661663234363932 +31386161376430393131663232363236656535326238623936663663353266376334623465666535 +37633634363062623435666663616263623133343561323264303033383265353065363038383834 +66396239653161393638313964346335663661623334393666326161363731356265343632363930 +66323765653664326663666133333634396330346633626637376366643765363534663166633464 +63303137336233643237353234316664336635653131656336303933353565653434623432636163 +61323230343932636266643663363034643732303638616536343530373536343534626430616639 +66333530366331643937333065363633383166343835656264383938643034343134373332396438 +38666566306339353739653539636537393034356239643436313366393138383231333133643165 +61616138343833663066646361336332326231666632663633613330343139616566313135303335 +63646561623934333166303933303033623733333833326635303530626130666438366234323831 +37616235663130366535333961626133363633383439363162613234636265663962623830373662 +62323430363666626632656634386439663265393732363361653637613063353039333237353939 +38343563323439336464343132653630326262323063336233316534326363353638623837393665 +66663766323334303732626464663363306533396239623832663637306130353163363735336330 +34633733336363386264316435383131613239373331303165323064393433353634656664653739 +38313761336162643863363932383736366535316636356633363466376466336562626432616561 +62626162376362356263326533653233393438386532373631633137643763383938626164656437 +34316632353133303535383161363036343537613766376532323836393065336565356336396235 +37396439373963376537343438303530646237393066626363383961646536363661613264306538 +38346337653564306163306134373864306131353138636532343938616162333036613933616536 +39656362316462333139373662323236633961333736616630353435623435316631663433623035 +31616434623565623066396230643832633133303139343837623536366364613262343438313764 +63326335656466386539303132626164396332313933653339633337373731353963646332336131 +34383063623564643763633130633066616432613739666135383038613665346438663330353238 +31343765336165396335386532323365363863623733303864353438376666613633303630363530 +33333834373733363663363232343066616639623064623530623734393833643633373434313735 +33663634386662663966333636383237323330366135376339646462623232353530353835633739 +37666632376164313466616631653039653462396263666639346434666563626138393732333030 +33663961313031313962636563626661623061346339363134616535356339373330373336656461 +62666661643435303666613435353231313566656665613039356266643533313065666634323438 +62396664313937623363363763356438643764323337373033363435353363343835376534643536 +64386535613365316636336133626337626336653939306265363836323530326637636333393433 +34623939666138633430613939306539306631643061616537313431393736613335326632653734 +32623732313931643863346463316633353834333931616566653233326365626233353035623631 +37356165336165386565393138623330353935316436323231313235343962623330386261653432 +66313865363361356361653363663561376337653632316230393062376561666334396562623964 +61613666366466616132613531316663656530623165653062393132363862353236323964636530 +64383938633835656462666537636666313666386538653762653235643362663437656331313830 +32346634656461376334386330626664643562666434663133356166646666313465643134653765 +63343138366230643530653762386466623862633739393933386564373735306139313138656337 +37356330316266346235616634653464623733313761663838626364376665333934636266333665 +34633536313638336632306336323266626434343932623063643064663931386434306136383462 +63383536383734376462386339363661326661346136376661346263646239633766333831323039 +62383563393461626335353639613130656232623935396436373666313162343630366634663431 +30316539323536393039306665656430376438613963616564613633653064663935613935323461 +36386339383639326639656664346465313163643363613037663566333636646363643262613134 +64336636363661336237356535336231663666636461663262616663303634393132653533323666 +64393163313432613337323036313938633135623730653838626537326537336363346666333166 +38373463626361396662656165633031356430383536626266663232396636313635656337613063 +31323038393933366436313864393038326230643930386463343430623334613634313363343933 +33373162663965643562343339383365623838643136353864616130623836616532316561633164 +62343964386565306565663162363339353965373132633336656264313932356137616364646430 +37343332343731613831356562373661343335646234383230383731356339343835316362373464 +33356530346231353165653539363839383964363937306463323630393862653530303263633636 +37666133303165306535633231646239386561343930666163633363383861343862623061666331 +32386632343235376362396633383331393263386233316337326631336336366335366633303264 +33613161616661326365326666373966646331333364373565393339323663386338323039633137 +30626363663732306439303232326534393737616635656563636436623763346530373966633734 +34643132326664333964663237623861623963386466653566383037396433353736663631646436 +38376561643833333864373961316236376265643236363933316663383635636136363064366536 +38646235376465613235653836396162363338613666376463353961323165323333623735386130 +35363735383865376466613535336239363831373661353838316537366638383963303236643661 +66303261323065623462393238363536653135323736366262306264326436333533313034376330 +39333861373438326331643465336336333665613231346138373466653934366662393166393730 +36633832356265313638373934346664366662326164363436626665396432643137366432613733 +62303235666130656563663665376332623061616532333865366132663662633263666632326531 +37643663323039633865623035363764323763303433356635333039636337386631303737656565 +65376130643563376261396264653337323166623338343065333430393663336239356436343935 +30303266643230633634383234616432386135663032366431613738623635626335313631336636 +33366162383039663836663766366663656335623531646264653137653137333832393138626136 +37623739613332613532323936383530373466363166333365656232393865393638393661363263 +39343332383832383432353363383837393734623537616139383132333162336335326131366237 +63373830313437636665663634313434396564616464366637393363376436643066663136663563 +30613231373263333636646635346239633063323831666365376233646436303764346530353862 +32633365613861646663393435623738353332663564613935663637666565303434666632333132 +30386636343833336438366539363430333161356538376165653764363561373837306565323562 +66616433353164653865346466613333653162313961633137356438613163353065373634336631 +37626463366435386666633332396339303932346433663664303935306334313830646561666338 +61356534373065623038313035623264383963376635306563656234643835653537326433353039 +32643737646430383933613136636331316237333232306435613866633737663530343739313835 +30643765333665646639336562393030306538303064366333323566626136303965313563373836 +63333133623436666362663735376263663635353766626530353236663863346464386364633537 +38366531306563316534383663666462616337613336636631626232343231333833336138616265 +63343234373734653635363263323334306238653562343461336262636637623833343236343033 +36323436353764353464316661653632373539373231373830393638616362656539356338373361 +62666331323532643262376265613232383836366233306365353737323835343235396335616437 +39653137653038656631646533393062393761616566316539316263313466623538666164386130 +36656337333031663530383262363231643931656536326363626135303863386139353736626361 +39626537356333386334646363613935313630363339613964663563613931343631613261653238 +31643238343136393566343966313230363038633663396336626265383236336364383236383063 +33623764666533363534396235383139626433393864393430333535336163633430316639646163 +66633134336635313263666130343337346339326262323633396364346531646634353366333839 +34366134323838366239363730633262393839636163653430333138653834343335313934666163 +31363130396339353338363030643239353266306635663431383463313536313130643036316632 +35363032366361363661393362326433623664616563613566383730396365613938663331323735 +63633466356464616465613031656661346261626235336338636530356566303530323762653730 +35393663393738646335396237346166343239666532383736376136316266626631663939633833 +30373533353136343738303566383265633931356639356332363130343431376562636263666436 +62303862646366613261336434373830366437626561393462326438613465633332383965323239 +61643932646632323136663662333437623863393733313236623639333232376566623638353737 +65393063633063303131353432613961326534393537643339616562666461626432366136646436 +65636335383039326438333330366337653164366366383135306530396262306538653536666539 +65663961643633353239613630623730386130616566623464393334356233323464633266373337 +36353636356438633664636431383934306234633533393730386539653664633061396432363565 +66623561303637383834663539633439666265353539363665316331653834313536356534356631 +38306230646637363366333765653632613766303465303130396461626637326334383362636566 +39356562353965653233616131666336626166373664333830356263636631636266316333653661 +37326661643861363835316532643233633538383931366266643663356434366638633463366563 +64663362636639653464353834646463323939616432663134373633333732316665336334636165 +63663639386636633936396165616661393330653733396530646132653933656639363834323664 +30343364383461316332323936323433346638663632363233366439316365633838386436313432 +38303264646563353634653531383932383431333461313761373139313034616365366531373462 +65636330353534303439393362386638396634383436333438363166393435303931346663613361 +32386539353933303963373862343966306635353364323638333566333463326434323961313533 +32333666643532656330303937353232366638313134613161376235623539363262313135316432 +61316335633261646461666339333735663638666464343739346166643335373763346234383265 +36333035393530316261643933313830336436633635663732393861356330363061396237366636 +65663534626330623237346136663130383932663933646635663834396461343634326238376363 +32653933323939373933376131633237366538333036623639373239393633373531633133383130 +34626564366532346166366666383837646162303361666639383363646139323664636233613437 +36613432336231666263326236396539346361636164656537326334653933316165336563346535 +37383466656562303536373161333761356439346531633163653265373166363866316432316236 +35393836323139646565616238376536376335303632326332653236376235353566343636393036 +30643837386139666332346238373065663962663366323737663564656534393562643366343237 +37663462393830343730313332336438316466336166343139613038303739303463643631383563 +37343835303339663730653263326364333762663932356636616465376463356363303864336264 +38383630616438616166356263663864623062616135636135643831643166316265663864303365 +65633764666263393130383130306336373762633663326231373964323932346435326136333162 +62323663333262303765346363323561353739383638646331323866323764653863643535373264 +30343563653065343632616464643164616536366139323834653432323530313030623236396135 +32616236663439363135313130383833626634346536386232306435393731376164303933306135 +38643861353266303434393066336536303932633135666231363263613537653436663564323161 +31333434333233376165356337623162316235326234623339663033613732393264353235373931 +62643362343335663065343165613634326635393061653264346633623730333131366464393963 +33623936393561643339336638376437643534353036653435306663653830306636613336626634 +33336237353735363565613164383066356633303630363664613866653366356266623530356135 +36653838303938366561313435643761366237306236306362346464313830613364636531396366 +33636338633266356236373732326165383139353739336233393236666538333964613265636132 +33383664386132643965383237626565653731306261613763393738356165363866616638303736 +35333330336535353332653137363137373635383435393537626136333732373331393565323133 +66343833356130306430633432656436316265376465363537326530363730643465386637633836 +34363132366230336264663634633665653063373136333163616134653764653730666666306234 +33373931633936643265313030323066323866333230346662396264636163333164633436313365 +66643534383166333433393237663136366534363331613039663638613538383530616130353963 +65316136613765313732353132356130326633633739313238396339623462353433353834386461 +63363566376338353264343633323530393636343538663965623832336363633936333439623432 +39326366346332323663383631616330356638333438663361393036376634356533343163646434 +30326130343634313839646137623430636137383930616362343365373333326361393939343637 +62393438643931323038336239613339333561303065326537613866353766316633366537633034 +66303064616334333264363166616531643364386564356563386163623564646435376131666438 +30626337356539316430626565306135363462383838613230663462383263613163303164643465 +65316635303536663433323836336538316162323164313861663032616330366432656164373566 +64383832376135393863323037373263633866363732386664333835623563363036353766393231 +30363865646136626633623965363036656465613230666637366262646138626262346563393961 +34373361393939343137643635663465343834353039653764373261343137646166316433366561 +39666437313634616536386165633930636435326461663130323937336134623036633234366338 +30643063346266626664623566386230343666333665386261313431663534623335393365666261 +36313061623463663136323262313664393532396161343539633861313661346463663534613462 +61383738303064333430613039383937303433636536343163356438626566356531623836396630 +65623431346166303038663166626431663436616437376636636332356139663164353739376563 +61656261616233643538306364353763666564323732376239643434616237376237346135363531 +35306365356530303231633930383561613561623839333831616539613732363933393534313563 +32303132373962353461323666656535303933396139383935663830383766333639666634306134 +63646461336262666661656562613235343138623735303337366537666163316239336233353761 +33363835336434643537373063633363313536653034656365313762316365393866666633336130 +61656163623331616634336163326137363439333835646230623937363736613933373166346432 +35616538646163656233313463613134313436633363663232666638363234326430346536663530 +37656633343030616631303739333933386237646336353734316332353534303036623837306433 +37323938323439396363356539326532643939366139386365363233643666396135623437343466 +62343733383631306537366532633436356237326364666462313161633035343032363632313762 +66613337333338333131623230363266303233383663313532396338343434393963366265363963 +34383438313063646336303861653336613934383765376262393630636237623565666561383830 +32363433636162383532613133336565323462633866333366363734646333386265323139323466 +62356336383231663138373766376134666534383036353064376363333535623862356163356561 +37376232386165396165663865393536383930343635346234363233666238616164306463306461 +63333166383661386238613339366564323233323763623339333963366463303766363663323636 +64303734383733353264353332316437653636636539306433353932323138313238326466653737 +65383166313539313832323534653061633433346136643435336166366462613236623737653138 +34343761366335393461373835353739323031646633336530303430653963646562653130666330 +61646530663133353130313931396661313435373737336330326330343136376465343261386235 +65383332343965313431363039353266343432663938353330663564626465623338396635363364 +30303862363935613533323364303765353837326130306335393462653664313338386133343437 +61336564616133336331326361333432653663656162323332396462333265303930643361346235 +32663763643336343635373664636266623663646535393062386332623262333538373036643334 +33353734393563336536313765626439353833366264393031333364633437636134333133646561 +66636538626661383830353864656462643234376138333162343963353034303633316164326165 +61626635643566323436633639643834326163313366306434316236646531343466633537623330 +30633132663864316533386338363932343366343362363536363535343436326234393131373765 +36623564306335333265366539353636353236366161323735653361646364663136363061626535 +38353936336433303465393139326339636363353538643064303935666565383064373533383164 +39616361343439363439363834646463653639343134663334656639323761313639313465363034 +36373561313935646232306231616164383966653035383738633962383438376638303361376334 +34653334666634646432333835366439356432363064386538393266386233333266656436313633 +63343263646630363439663232656137656537646233303064343066333363653138326432663063 +33623331653634643739373535303238616139356434656633633339366333316462396563333631 +32613336333132613838333161366462366436653434663430323161313930643133396562393938 +39396535333438386365626135363432653232666161626134303137396435333630613731316261 +64656266386130326435306561306433613062663937623935616137393934653634613136343835 +34323939323631376135623333376339616634373066646333636662363763313036393366373034 +34363664656633383463383435363965313931396634376137666239316437363431303432626634 +38666230663464376331653635326235383333353366306437373464353832633262316634346232 +39303231353161383462363166313839613530633537346335643462393637373136393566316438 +65653737663336643039646336653131336434626465393662653033666434333766343036396566 +63363833326133346563356433643965313834303232633435346164363437383231366332626434 +61373065666230306138343731643865373435373937386238363937663165336362303732643866 +32643430663430346135613430316430653635633266646331386532303564336464373237336634 +30333161303536313233303164373563613566336464353030316666626663666162323837646633 +62656164356136613666623065663234343437356337653739336439646461363061666262363862 +66663032613630393731366538393366616537396439306464363735646338346234333836326332 +31336562666261396363633465396466666430646361333866383937396236343166323037633637 +39336131303932616263363737366637616539333362306338656335643136346265326464373430 +33333830666239323461653966333933306261633432323030636238623134306166333734663136 +33336537323264636533636662613930323966313164633831626661313265373932336431383932 +64383465366464343230663039333465366333373133366537323330343430393438366130623534 +65626432353835326536366462336431386465663932393361323461356533376364343639363138 +31666562366531636336333731313130333633666338323466373865316132613534363832326131 +65663931613366363965626335356631666561323164633562383464303936626534633731336337 +35336566643964353831313064643665636665393632366639363538616638303764376638393765 +37613331643461663161356332633433366138613164393033353136643834636161366665313239 +31353230643332333636353935386632633132303064663533663562323262636131373764366536 +31313266313635306339636533383235623536383266666337376265366636383634393662653539 +63323962343836623262653465613064363962386434343638393034356665363665343263306234 +62653532656461653237383238386133353735616264313364323861626162653238333633363138 +62613534623461356364663539663265616238353837623437346462623830383731343066303961 +63353462393836323739303837366237633262626338613632653536366662366531386438393762 +33373466306664663662616239663363663462376133393539373135333739373334643963383636 +39356435373438373539313136333966656539343863373365623931663561323265323962323439 +36383234616636663138636166393039333135633239393236613263373965363531613966636362 +66326238326232346266356661336531353939306134653036393935323163333763326161336233 +38636538313539653765366634386138633636346635376532633163663132336637333766643365 +31323666653765323266626336303933386238653137353065376339383439653566663366643832 +32313730623033663334383436313832323762333163653534333265643938643439353832343932 +37346463663538336338303135343435633936623436383638356538356538333461623466396261 +32326535613031363365653463386132323033626239316562636533626338366636373436346230 +33643566663861646130393266343333666638383534636539666662326166383836353732336361 +63336431373930633061376333393437656539323666373664623036386465336530326336316239 +38626131316565363463636438336332653030353366636165386464363661623661613762356264 +66376534663338303161366366646162383363373664353466666233313838383164373162363663 +65313730653131313233316530323330626535373261383661363465353639633030333664353264 +36383066393030346439336661366336326336326461366138616337343561373536393634333465 +65663433363830316333373166386263326161663461353139353335643934383234306164383632 +36366164386437396163663237393038386630353462313138653863363435323134363335613766 +30343761393438316631646531376238366431306634306638373032356333616138616436376439 +30386136326137343463346231376234326532616266626263323065306633356333373839666133 +31353138653363313666316466653135316130633339643930613436653138656238376263636162 +30663665303364363362323236336363643537383363663832396262376366353730353437363231 +37316433646238643863623530373037323030666465373265326461323566333131663236333933 +38636439643432633062323263326661313562653139343333646332663937613538393338666338 +64326130366161323265643764663531626634306238613030303237626134636162343938363031 +37366639326338643631353365323637323336613238333934363836646264633433326464306465 +39336264343136353636323061623637383165613139336563303063623137616165393134663739 +36353531333138356236303063386638373334626536306563616432653536666163376665663432 +30386237336439343032663061643339656234383130626638396134313661303064643731323964 +35633033663366656233373832323131626235363761393839666237656637636631363465353936 +63383462326331633964626434313837306536663734626635383562613531663234613136633231 +34643331373162623534303635366537663937616635643462376233303633353832326537343737 +63323234343162333164316662306137346362646437313134303736313462376136646234623765 +34323865363734616338613563393539393631356133363033396537646661626638623733663262 +61366431343935653635353739346230306266333763346437623765306131643632353935346462 +62363664303731653661326230613932313763633564363831316631333164366333613665353332 +66613536376266396663353434626261353938376662333661643438633235653462386238636366 +62633737313432376330346334326537646366636438396666303932376533636439643134653433 +63396331646236666335383535326339393031353966633235643533316665373663643964336330 +36666264633330353164666436633631313438323163646237623931663835373630396539383466 +38656663633036643835643439376464646165313564366463336236323938396235663130346361 +61616432356562663265643632373334323737303131356262386537613339353963643566353061 +64376231353366313366613939333562303732343635336538623438383335366638386264373536 +38643239306332313461356434373362623031346331616632653830616562623937356232303233 +62376538383463373962373866303230643364373965373564653837393465393934306563363964 +61653765353861613434306539316166643439633538653963326561656337633763663466323538 +35326465343963366465323939656539633761323636366139663465646432643466316234353933 +66383331363835623664346363623031363833363963393739653461353833316538643961623337 +36326266383735303737313636376636376166303231646162336230363361373831333133353563 +64333166656636396339353934616332636462663763623833303437656464306139303636333137 +64623336393637363236346539646331303462393331366330613438653761656666383232373263 +63346264383136303833376639356136643638633164353461643663646336323132646435643032 +35636535626634303238656537313365656334616566393832346662323739666466343938333535 +64666261623863623663633535653263643761386539386363326539663132356565663366613932 +66616530303465643838663462303764666337313032626537636334343066393163323336646435 +38636232656664366630613935366635313965376361666233343335303764313638646262386537 +61376633396232313865373337306638623337626437366130323537663432343064626234643764 +65346439393139646334666431646433626332313335646361356136393933653864616337346438 +30353637343562383961346463633332336634346637323434636537383239376630343164363330 +34326166613137383230653163386538303933353736363738643834386633363862633363643561 +62356563386430626339393466386530326233313032313865656265656432333761303436633063 +39343431323032636435666135343436656435313665383139643966306337663334346337636534 +62386265643536376461663232383964333237646239343635363738393061626539626161663465 +64393735303334376662366663373332323632353564313264303464646163373630376337653461 +64616633313061316638303966333638646161373336343261393864363565376635633833316532 +36663563643632326461613731366439373765666266663164323961366138353138626336616665 +35393938383633396632613634383234306663356231363131386636666437613463656463393163 +62636163356362343337393265666566376437346338316165666161373234636565326636316136 +62656237343437376561366633316334313730393564306561643332316230666133333365653439 +62663739376666616665633463623937326134356438656631663964623238666663366166393931 +38623333633638616261336130613133346436363033643961663430373063373963346461323463 +34336165323466653338333462613836376365353839393830666464626465393232353861313334 +63623564323638613334653130636636353762623862333662633231643364383963336336353637 +33326363623866636637306532333932323761366362643031373639616439303232323332633730 +30376165666363626566316537333461393464613333386338303232323062353632376562663634 +38303465613132313138323365343061333634656331393262326633613133653538616538326230 +62356265373031613466633765303130323062623464633163393232616463356336653266653566 +39396335623765663339326339623335363639396264643062643936333761636663313162333233 +62653165623831636230323965636466356638356135666230373939663433333739383061623363 +30386161336166376335623066623230343862663435333363386666343531363164326239616531 +37666135383036656665353936656637666436366636633162633838663631396433343664363063 +37343130613734306361663538336465336236646630653838623734383234393662616138666332 +63626461663666333039613037313265363965363762346234666266626438656361363438626665 +31643530333837626231323662363662346330626461343934656531306237346130383462356432 +64663938633134656637353261393161646138383964656466366564343562623338326665363165 +63643637323863653561383539613063633961396533336137656539353264636363393931373766 +66333163623866346434633730323165656332626666316533356235306161356235323332623866 +31376531333138303139326337653461316530326334343036356231626331396365626438623663 +38363462663732306131663036376365636561363462323464346339376135623639373237633236 +34366266366165656634663536646636623631613036373039636535663730343234353034373262 +62323735306235643262313066333330333663653835383465653661326334313662303837366634 +34303365326435346563656262353564336633633331653066343636346438356666333761663834 +30663037383337346431663536386336333534306335626665633836633036643530616231323762 +32343536373262323266383039633364386431626337363231396239303765356538336564656132 +62376238343161313034643034313332616230316538613436636662316136316565623737383039 +38346534343734643061656563343263306631363166623937303733653666396131646231306466 +65653631643761383030303034343565343635326639316364363566643363326232373338633034 +61646565636131386361383762333830643939613633313637323038396436613961366635613666 +36356663373439343266636265313862353736386465646135363765316639393334663232663232 +35303938363839343834303139383537306563326232393462313835383032386536646161636632 +65393430376234396438633961353466656135633935326166613032363432353666386330396531 +62343830396133343035643533386439626363613062336266643266646334333538636637646366 +34313134646233666637663033303864653939623134343964363232653863333265323036636163 +65303261313037633961353436383066303239646239386130343030643131336236636665343566 +61383561366136363937666430323133323830366238396566656565633439386230653630313964 +39346331353866303837646136393338356435383663326536633930333732336336386234393936 +31373331626333346237353035663264613865656336353433336235393136613637656264346335 +31333231616137353334363061326638636566306462323636393166323932373132346463636234 +34613336626463373430666364313030636431313538623065366161353961376661366463303633 +39333937373762386164623566626537323861636535633864303234336566323638393562663333 +36333635613965323865376234636337633336643132326333376430343136636135333766353638 +65303166356533373331346363303333343165636438393665616632666661663134613339383832 +33613564363435653464393139306434613466626435306361353631383534316666616464626138 +61643131623835393134633434636433326533313031643363313033396233393031396363613431 +35363962643431633237616133623834636230643165623838636664343762303336643331303163 +33626562333134393765393438633061383439353366383436396466303661396164373239363132 +65316165666432336631353836383437636132373935383638626631346132376638396439333034 +39646461663332386339333638626161326435356161313564383539333164376238353936383938 +66383438386665356336626338616136633161626237366461353330356162336461653839326332 +38353962353534323031636638386638653066396162386261633861303132353937336266383361 +32623930613139663739623039353233623661623763663930303963316132326332363033666239 +64383364616535333065653737636634343838363661376363653931643339393566623937333363 +31613033643830653731363461633231333862383866643662653537306235356532373262663537 +66663562393133343061643363633832633638666534636366386536303035356364363237623431 +37626565643435616361643964316366306165376138616365363361313034313539323861626538 +62663665333563326436633362303466306430386262643364343966393766323961323433303135 +35333335663961653537396235376439306161363832353762353737343130626537393233353432 +34353263633666383162303162346161666430333331323536363538333466653939633766356363 +63303538333233323432376136363464386633613739633262663438316639366263636561353265 +31316637353939633666363566306235376438336432376538373339353335653964363330346635 +37633536666137663631303130366230303863366464653261373634646665313761313137343530 +39336537386333616335373836626138656636313064333932613364666562613261393831346661 +34393661313939303730656232313939616538613761373161353839366235363238653731393738 +32366439396531613139326533363063303932353832643632623934333862373365323764336532 +33646235623839323332323862316562333534633962386361663365396532643132663232636265 +63623833656638633861333962623065393761346136343163663764653763343533353164346139 +39373138323535316333313462303131633664303630303565326437376138613533643937663466 +31613063326364383639306561656639323036653930393939363164323932303563643230613237 +64343434653063313231633134643538356365366630633731356630343461333763376566626263 +34303661353964623334376136653238303431623339653732616235613934336138666564323535 +37326233663766616565313536363337343965313234323861623531313733643734653533656633 +62663531643332623765313461626461623039313239346434346331393436633365363334336665 +65393336623834633665336236383135366365333939643731623436336532643132306234373339 +35643663366139356430663664636666356436626663316136376462353838653432363735613362 +34613932306666333862613766336262366332646431663735393264386565613361396433343933 +64666532663966333066646638326563623439316539366430393830303663343135386530313866 +64656538626530306162313432373065353164353134316664356565326365316164306539323362 +61656338613863356462643166306163366534376261366465316463353936623362656466366462 +66313961626136386635363137336332663437326366326234303761323366313462643830633331 +37666431353264613630376362303835616235326663303334396361613166663630356534353538 +32613334653536633364626639663234616361363665666631323261623965346133376665646164 +33316339386465623433613230303732643536323836616436346664346630353432656539613563 +38386233363635626262383538353036373761663639346534663562643838303563323666643432 +36303765333230373637636331663162363138356661333464323965373835616237376330343830 +32326561363764363934333264393438616432656437376638323463343964363564663538646364 +63316332643638303531313838393762633564373639313761623031336532636462396132656238 +34616134663335643537303966383039353032653966646365336532386637333962613539313264 +61323961343634373534356430663730346331353166333637323035663033383564303165366333 +61373633373531613837386636396563383637363436303133343935383965623763633332343461 +61383665383362613632343166336666333934616639343234626263643666656635613732326430 +36376534636361303764326236386439613563323061666137343937316338313039396335643232 +61376334366335626436646261633566343534633436383234346234333762346638646132376464 +66643363323830613237323461346436643339366639363436396462656262653133643233316435 +36363364356534646234343661313439323038303966326436643236633437303763366337336335 +39653835326339306166353837613062313430303139373639343133643739303735656366326435 +31353664383033313163363134376363656462346633663835633334656564653731333739353763 +37373930356564343133393362373964303136326138653835303738323062626139626332623766 +31306430623632623864306337653131323336343538353036356662623061393933636263373630 +33323231373733396438333038393732373436376135646235666265613339613361353432353030 +33653236656332323036653738616235636433633965313933333565373736306165333230323863 +66303761303961663933373431633965656534646237306332323335633739373630326230343338 +35326130366661323635623439313763633534363432636261383931643565653062623962353436 +32363637303065373465333436643966656666323766653364643735353535643633383861346663 +35663435393862356439643738313264643366643130326637663233323838333961353635306332 +38316139343233326135386230383166346664343464363438653935326239616336613566303663 +36656237366264363536386662393565643937306165343132376666653766643637653135643136 +61353335663461326562366231373937343730363837353033343262656161306130353066343033 +64313563396638363038346238356231393163346131316664343963363531633965333463646135 +34643236383666303265326136316263376532643930336664643463323830346633376663643935 +36656665356531336463346530313765346436616237356661323061623532396563396631303132 +36333262373637323466623965363432386332323130656664666663613137633038396464373130 +34363066613662326638623866663766326166643132323738346464353762326336333436613762 +38626563343563313764373135303962386430653438336632663566303633303530343433303165 +63613463376539303131363963343538613234323466663630643334626261303630663264313263 +63316534373961343662623562383839623335303132303434353364326466646362616266333739 +38333237373461643336306534393163643463376533613065393062343966653434616139386231 +39333435666464306533363761626334343533656230303664383234623731633237653061323433 +31623962656637396633363032656666383237323064326363323364643564626466343233653733 +34383865636162623164306535333737663734343462316665643465643461633438323264626362 +38393664376131326563633261316663346632643139306636386265393033323161623539373063 +65666539333130306137653633616332386431393738616563643263393762363437386563373134 +31663761643335316365666432323734643832633266363564643266613563383164643731306162 +39353763313764613532663562313362343135653033343633386164363139333561393237373765 +32666534643964656263326638363236333338326361376466313761326336303466653539623937 +64663637623363386437653061633539303239333961353432353535666630323137373339633338 +33333164313564363566656336393433623062336163636532393739653062353864613964346562 +34336565633636653232353231356330636263333165383936383466353363333731323030306130 +66646162333464326531333236616464616230663764373530356465363935636137316138386662 +39373730666265663331306166353263386138663961666565666331323261363435626332323737 +37316563353464306236376563373366616237623033666363623838393736613064626431366665 +66623563386432353539336161366232633739303661393134633437643462363462393463616662 +33333461366130653432623937643963303438373638663866633437623366346661636136633134 +66613836613361303336386234646436333739636363396364643139333134396466636630633165 +34663337326539623064613366613239613232346366323331646639653165373430613438613435 +37623734666536343936626533316436323365303336393665353032366438323632613465386138 +37356265323461396637643063623631326334313830323932626234333033303337303464653938 +37666436646430396636316563376661646439623633343836343531633735636639323837383139 +39333834643765316334653436376564616338383034646663366630616565313533303966376133 +35373264393139663564663761313436383064653839323433386638373735316534313839643536 +37393563666130613561663831313438333737353933323665356330386537616438333439626461 +34303265313261326165656666636435323566333861306365353461663734613861643265313733 +64626431386163373539333737323732346166643232353034353635333531343036306638346662 +63306537343931333763313332666463313264316666333438313634366266613664626163366561 +62383735623763626235663632353934323236343566636336646661366362633466313238343931 +35316131303932616230326664643336356261333766313433343530336465313461343166373964 +30356162363131383231393236636166666636343231343666623038356339636465373462613138 +39313437383330633333636637326538366533383531333163326466623035323432313830613639 +35663064393832663930336665373661306234386365373464336362363534666365373734313734 +35363066383038386566363635653131636465313963666562336365396432366564656564323435 +63363533626638356662666262363662383362313165346331376166313331383663313431613564 +32386561303235336333393133616464386238623434653066653633383165353837613232316631 +30623761343232373334643063316161633030633736656461376262363562636466336631653166 +33353039373730353530383833643464333337333736353037643361376437333962306462393533 +31333832383363306536623032333736643630373235366233633866646233396663353132346264 +30316564386263376662346132656466613765653332363965373032626338303336366332333966 +33343062663664313233653737666261643432306161623732633936333032383634326236393630 +39303734363462656263396434623239353832396665663137386637643133623861656232653264 +37626364363766653035383265343132303166336632623861633264666233376466326530346331 +39316535633237366164303235373037386463613364633036323635636334393437303937636232 +63356639303831386231346433396563366261333261323065363736373731653361303739383461 +66623561386563313035366437623163663239336531343162636434303339306166393266363061 +33383933353333383963613935666232323035316362356630656331363035326638316533353331 +31323334643834353563356135346138333731636163323833623363343236623136336639303538 +30663232666535356333393632633139363036346236666361353430383962306239373534336636 +66303561653661393037393166636132396566306361653031353465363434613931306533343764 +31656262363530313939316337666338363432366330356430666231356362323865616235366437 +32373330623831663736323231633834643963393663376164333565373063636434323865613235 +32353638353539643065363333363164666333306434326466343234353934316163306635333763 +63656231383633363730666330326433326535373365373766383339393761313431323731633863 +34646435313332393735366366376630376334613663366463656430316136663664303163636631 +37306261376232643263633963303636666136643336653136326663623666313862363230376430 +61393666323965396564353038663961623633376333346162613066396539656530306434343931 +31653461326439646332363938303230313335326134393433616565383333316331316365353937 +63346263366533363161323630623835316264623830373865346433633439623765663735303039 +34313731623064626331353137346336306633343263376332396431653366636231303962626439 +33363764316439383066336164343361636637323335353666393433383734373766653237393339 +66323536663766313435383064356562353331623236613464643865326663663337646365623462 +61343732363661373535326537323835636465366231313632353437386330663333643730643737 +65303464383634663639616632323134323363396132323363633830346363303230306433643862 +34306536653738653533333534306536303166616163303137343764653333663538373732626161 +66383863646137663630313761316562363934393736633736623039656438643263393930393233 +30346239646630633933636465356462383562613366363163396165356631386535313036306461 +38303432313365313563346430616135636466353331333335333839633338646461633235636635 +34313363363661613861356332616136396234326262343132616432663062666536633365633738 +33323461613531363235333638663532373962396637303830343661303430363762316162646263 +33363334313836386535373338373634336135626663656363376361323562383361386163623134 +39663964386366643939643035626661313634663065656466363138386439323933373734383531 +36376236386535616437663437396236386232393739303733393162373037376265303262393134 +62666165353831373465306364393864343230333433383839636135386138323132623132303866 +62333766666234343431613264356136633265623937316331646438663466303534633434646263 +35626237363638333137323863366537353862323334613032633031653739626438663234353035 +62333036653065666235306537323664356136366264623164373931356639323434623566386164 +32343639623266643831383133313637323466373133623964346265613937633839356330343966 +33306432343764316531386532343332303263653361366435353664616365616131386130366137 +32326464363139616263363234633961353637636637326361326632393264356432363364313138 +61306464613332316662363339396231373061336231626365306531373131386634326336306666 +32353932346431323831653331306131393561336335616463616138633232383136316133373438 +39373536316639323065613039383632386439646437306237616164313361643431343034366162 +38663339316163346166623265643130666465653736623730346337383863383461326565306138 +34313738333865396137663633343636663064623165636533326265383964363233616330333431 +64373837386264396263353630616336663962366364646335376136643865653062373664313365 +37363333663361333336616130366135323166393365613235393865316663376339623161343439 +37373631316330656236643939393831316432323930396262663263353235623633313238663866 +31316130656431376436393332623061316163633261393564366431663564363237366166633962 +62643931613831363931613062393162393832373339373862316237326162343062323161313566 +36613961616265323934646337373137353465613361373565323939633332386333653938636661 +39663961306132383833646439303332313462313361666563616139353163353231613035393632 +61343463663838393166663465363933383635623265666638336531393061356230383166633662 +30343137656236633861373866656630336334633463313064316362653837663166363762646663 +33356661323561616630303037313264306133663431653338366130663432663463363163323530 +30653935353332336636363933663662316333333039316465343335653337616430336635383266 +36303235383433343965303335366566623437393131313730356630376134643161646535653936 +64393938346466303538393739363534616236366331396462316234373038373937383763313864 +65313337353831383432333030383465626231626136343464393638366662623734333861386631 +65656530323161323639306536326161336236626363306163626434353762356562343961356433 +65333931656264396565636430643537393666343931383033626462663939386137373235393835 +33616262613033313033646335336463663334303132323065623361616161393130373163643964 +32353565353661336364613937386533633064633263333261303861613563636537643863323335 +37303034383930326333656561623935373334623763386666323563633062396230386233616261 +33316236373331363432356433306635333235383336373930613062373766313332376236393139 +31316562613430633863356237386562613736386636373932323762313438313730653334383836 +65393934663038303338383165383664396636373037636237353165306538313938646335646535 +66323863323664396665663130333439383133633434373561363465633537306664396539333263 +34613931343532623861343566363733353166633639373738663431313636356133643236323136 +62386665323534636531323039356536373339616638336138646563303035373039373931613133 +30653962383331386634643565333062663931396362613533386364376261333835336661333063 +37623237656564393334643735643634623132623463623831303436373365343165363835366563 +34353538326534363837653762636361643065363637343462656333626632383232623765346261 +30373966373838663738616564653237393334386363626638613362393162653730313564306437 +37323134343761653634633931303034623233666338396335613763373261343135656163303166 +63653336363064323736613431633865633061326161366135303933653966663566343830653038 +38343361383765396331653661636563346535623137343530653639666630656639363163303438 +38333234396661333533343831336139613862393036376336323465306164366632313164663664 +33363761343335303362623439316261356462636561323166666130356632366232653335383962 +35633831373631316635313332633266646236626664356137316336653236663539653534346366 +31376363343361666437383731343238343762336535313165333462393866353661363033393065 +39326433313936363733623131616163386663653865626438366561653734353463626338323162 +37373830353937306661373861363938333563663365306162346162666165616433626164623636 +62313633623831326233343161343566303534663531623937646630643762326239336135363365 +39376665653462353737383038333862313264666334333936396265303536303963366532643936 +63336233616136663130663836386162616535393238393465326565353866363734366336323461 +64636435653735326536343563353931666138643530666664613134353939663839336363303731 +61323833393764393637633130323363653164346634306365383338343737303435356535313164 +63666136363434366639343261323835333361336635666535363731666539356562303632366235 +61333039643261643561663733373935366433646538643763306332663936323130656534626134 +38306535356134653036663333346438663464306138616666633039353236363930363234663838 +32656630633762386436616136363063303866376366313935396131343362626234646161326331 +38376432313966613264616463633335333435393362326632323238313864323030663439373564 +61386262306232396262656538663037313431343935636134306162333939633135373561653461 +36393234363734633566313739633964363730633832366436336466643130376566363538643436 +35333764346534646266386633356531623337373166333937626462383130316366613763343063 +35363735313262383136616334356262393061386330326664333031306164663263323162613764 +39376564633131323663366238303838336163623665393431303133386635353137316234383361 +31613065323564623662663632633964323738633863356134373033383261656363313930373930 +37343335336131303439646661623930316530393464626431616636656564376537323062626364 +66323465613532343939303334613030363135353634313533373238643432643266666233626235 +35393639653036353865326238656132313133663734326637316437613635323661613938623033 +32373535373465373166643434326664363635326438396361343238356163653238643131393735 +64633261313065333838633763633164643634373266656638393563353534633837366536326364 +32333737613766623639323830633966313530653433373563326331616163386263393861383637 +64646530376134623034323161383965343566613134306438613463396238623432643763616662 +33353630396161343239613962363830316330613133313731373235666236633932656537333566 +36643763636538353937626266663065663534376131346530623630613434356264326464326365 +64643464366135666336326564303331646633326539393963393735316465326632393733363734 +38323139393363636436356531663262333166626361363935313662663563383538343361353832 +31623362326132363734316635393866366332333564653236383838353862643531333234633761 +66333134313536636163313062646630336238303433663864646237383533626461333664353437 +31363537313537623236336134666336303932383563353333306339326462303064386531323233 +37636333343937636166326139366239303265316633326338373331336136653531636663613161 +31386534653331343430326436663437333139313635626131626437353664336437333836306630 +64616539623066313731333234316331353063366164323030346338613064383066626366376230 +66393437666163303038613762343766316534636430663530386536303834373831303964353133 +61303732633464643934613533333535316631306633376164653531396435386335346161356238 +37386264363664633532363862323933356233393335643362666362623965633766313137303531 +34376531653834656263326166623861616465313465346636336534633363383536323662333036 +33313332643832366231626233613038326533653162396134663334383133663264653634626637 +62613638363963646630633561656531383466346461356630656431333737363937333837663135 +34646432366335323465356462366331386636643032653661373738386261306363333062656337 +66666437636233656331613465326639396539336335613535613537623037636335653164326439 +63333162633162623830383363666637323231396561636263333131303534333966306564613331 +32383964646430303234366434363235376466356532396136323361663035323366376533313465 +31306237336134623230613135646535363232653338366464393461363565646336326334353436 +62633763343761613738333336316334626463373830626562366438383932623232363833306364 +34626331616561366333313831613739616331356435396232333064356638646230663439313563 +61396637323464353336666637653237626164316330336332323439353535326461383136336534 +38663162366432366166613037653033393133336139623332633732616538303162356334396237 +38323531313638306230343034633661613734343264626231323232633536353135653431376161 +30663863646364396432326233303934376534306364656130363535386339373930363466623839 +32363665343565323465373137643930343966656238326131656165363833636335623732313333 +33313134663431303964393035373861363439376139643862373235303138363961626162393063 +31373935373332353335346638636263666266336362396338636461316466306233333939366234 +65363039396238653837613435623331303330373430376331313266383536636239396138636132 +32306234646563363836616361393031626239613364633164393235306264613336336466386662 +36343961383632613837646238323133376536366337323566666662323537396565373339383663 +64653533313665326264616361326135616663343139386234303164383262353466623635663730 +31383930333764616164343666356264653366626335393166383933336539313434373933323632 +31353061363930616433306261323662383264393432383730356539316131323436626131666665 +36313165396334633463386432383666356333373166626365333362363866653862633430356166 +66633439616461323437616134313964646438343334313631623736613463616438363530656564 +35363537343831346633323837303736383061633966643436373338376663323665343834653832 +63356365303438323563303037366332646633303138333537633163333266666533313332623131 +35366530653039643564656462613364363339313539323835643537396131393931663065306635 +65363134303839353239393466303663623364666236646130316237626562653139316563656566 +39376266386638353839326230333233356365646161623030313637666563643061623732613933 +38663239646661343663663534366230623333613636353136356530366461646663633962623664 +63396561666238333139333937333262646236653535323464333731653965393536653765323936 +62313363393138353564346138663961356633366330316163376330343932623439643833393263 +30383734396565316636363431383063626639636462353436656563663964643032343038366133 +32343432633961323334643333623464633163386230313366306630663263623030383066373331 +34636430373336396162616535346531383533386661623138323233326566323164393334613566 +35366131313037323265653365653936383763666464396239323734613633376134643034343639 +37386663623630316335363637326636383932633836653238363137336137633931656264363830 +37383137666465623536613936346236343537323133663561616535653732396533353735633238 +66386436643862343764386565303535393938363931666561643938383662386532306562653439 +62373837626164363161376635373736663463636362313166616636303534396631363733623739 +62623065613365303266326635663164363032666631653162643231303931343132343434383232 +38653032623638393863323964653233363564363438353262306538316561623966333536616230 +65616364383835663833316139613164616263366230363734623132643433653136353236626638 +30666562333866643863363033643935346136356266303835346461613739383564666531373033 +33643662336531656435313134333631353335383032383165396661393630643030316262393435 +35636331353861343534623438363638663332316430616433623334643131343265333233333237 +66636337363230626661376364663965303833383464623162306538643865366566616637383664 +31313038393836336438653165646339343961656634623965343666363732653063313336323332 +36656439643061353764663639343237373735393266653431616139393766623230616239313636 +39316238643033383364616365393139393637363836616162303465666432653732373861336131 +31393464356464353832313765333639333532326262663866346262646339653037636532386561 +65376638626565656362633734643735613039623431633034386532626633666566336362303864 +33636534646432366466333937333362363839313530323334356662323234316261356530306132 +61643637316363646165623635353035316361636131663532346533373963313736383031333133 +38616261393330343464626330316332656435353563616633303037363865356265343535376538 +63363163313232626261353964366339323561663534336162353363653336313363396264303037 +33643635616637336134303565336630343566643637393937343838346661333563643932356365 +30633936323632613833373837316633633565313939303638366463316639616564393562623637 +31383933356163633432656265376234333934653135393662653639333662366337643530316139 +32386261663934613731613466363734333234613332353663373732326338646233376335326363 +30313063316337353632633338313139363331346130393333643938646662633936363631633161 +36386536353966666466396130396663386137386434623231636564613338613430656161623837 +31396462396339383635663865363430383933386532666632663462643266336637646434366362 +34383337343139346564613830366361383238396265303535353833306431333861396136316364 +36646265613034313965353562343061376130666161326638663765386430373965323663326534 +61393231363961343237613237623565636335323765303762373936346334653136613330316133 +32333261306634313265323530393737626233356561366631313764306332343863623338393465 +61323133623834653866333639383338346662643739383737383135663037353762666665316666 +31656165363035376630303962633661366564643633336539626265663463363036643861633765 +37373939613337393831373031303837623234313937643565633861663966663632333234616130 +62373031373261343036376638333730666333386131376234326135656438653937383735663963 +37333964636462666433646561643831376461376239366566393037383135383931333961653433 +62353633303365313464613164663338373161343963646361663666336261643461633764653332 +38343934326132316262316637323834316134626438666362353964383734386433613337366538 +66306338303761343838663765643035613533373537333737616537663331313333386534356636 +30353935353135346262346337343965653137656361396633356363633537623737313534303332 +37393262326438353161326337393236613134343632616138656232633264646635633831653465 +32333064303539393738386432343766663732636237333332313533633934663035336666663239 +34333939346532343165376262373233376564313130333036303539316639343730663036346563 +37373432316533333639313231313566363139663333623336313537333933343362643537653837 +61383532623965373734303636333638313337636639643864633063303265396331383330636163 +38333834653464336662656632323962653836306563336239376536633839336539613761353930 +66386530366631623963333334376237666466613533333266656234626633646466613664356430 +65623432333764383062316138646366656639616162386330646537633130393264353462356362 +63356264396232306237386333656138646132316431613133393032653365313664373131656436 +35626162323663653362626532333865656461636633306561303736353939373063646161343235 +33656534386466636261366330376132656137623236663263653165623333643336643665613764 +39643264643264643430316434653561616334646235396564356535666164383932326437356366 +64376166393537643930343536326165636562643538383438303038626539633666323661633965 +66633964333733326335346239666166376265643730663432373338303664356639633633316439 +64303266626436323266303765636530383062373661346363656631623562323134623466396463 +31326237646161333062326532333738346563333736616461313530363734376134633137666363 +30623862316663646137633333313933653738666533633438633834343635333934386532376262 +34333866373139343135343463663436343336303364633330643336646432616166363066333964 +37303839303534373133616632356464346239316562623536396466396438663866666539626235 +61643734636634336639636233336437393565633831326438666666363933626132643063393230 +62323734656266343737356438363766346630353036303136313239613731363435346262633633 +63303864343265613831363865666535323565663232653030336366393865663966633038323939 +62343366656361616531373962326264326436323136383231373034303561383837323531336632 +66363630633030333866303935613263336234333734643733646130336662333165663736316566 +33626436343634636133383736333761343039376332613731303237356362303962356266326138 +63633231646439383161356538323135633833636561343163373161633134353933623138323463 +65366131653534663361623234633932303937636261343665373633666335636263393030373530 +63646265393162666639663631366530323233646535356539643837346666346562626334643732 +62623961366438386637363936353663393235386566303432353864316164333730393264613539 +38336130336663343437643733366331653437393131393062383133353662373832303731653163 +39316462633435393638363566376631613831623139653365393563316331393136663134303365 +64326565633934316339666362323837613437376430393563663436333062656562636662306463 +31353865643330306434393263303435303230613734396431643066393161316433633337363264 +36653930353439306438636430616439303930616261306163626238303138646339396438353638 +38383464633963626437626531313137383766306434303535616363303935386630666633313537 +34366161613164346265376633636532346132623462653065326434643732353838663835376161 +61313665656137616662343431343730376630633230616666663331313562323936313261366165 +31346130643965643366336632306161383765363338303237646335626232323134306466376534 +61613232353336393939383634353262306239353064303463616531336333363937353562313765 +31376537366664303734613661313561636662396663326665363737323535343333393661333232 +39336264613763626233313933326231656562353066356337376265643462623834306433336434 +65626432383863303431383730336234613433373837313965656634626664326465323539373264 +37653930633933353239623635643263303766396338376630313936386561356165366132643932 +39383162356233346363613738303139313864333762643739646535323136316231343965643334 +35343736376238393237343266326636656162646666393364653331666338663031626135323532 +32346237616561336264643665653733653338653733393539643061633063656635636264326238 +32343364666133383535343031356361323365643063376530303935333131663138663731386138 +62303065656235623863363336383066623866643732666436313537376165366330656633353536 +30636434343931396437333162653335613135633138306363363061343831623430306637353265 +63623964353438656436333832666633626665306332353665636333376332643735653236613262 +66333863316464643463656630303763306139646632333535356330633038383732633861613835 +65316431376666626639303862396364333238653639663464346638363334656266363862306439 +64303761343666323862353337323265636234356132363339663533356430633564663231313833 +30313735643038353266343939643934663266326261313132653562333533333131626564666365 +61613533353963303865616262613237306262663963636136383335633536656430626165376439 +33326632663565363561633236353563626636373963646138396363306239393931653462323139 +66643837636632333962313739643832346639393065656536613938363533353137303263653362 +33383132633364303466653832383934353330383030363132393661663131386230663566636636 +62363331306530383734383532353663383637656363616430366434343538343838613835656232 +64393965636230313034326564343163373366333061623763623136653234383266613264376637 +66663035646234346162643530313238623231373966313664643930303833626536366462353530 +64383833323564663837316161656336326265373436633164353731656132633433393539643538 +64346537633137363235363831333861346538633638326264656434646331336262353534643163 +32343331353835343431356231323335303631393065383464636538333933313330623936666562 +63643739363731333432633063613833353732613538323438373536333366616336336533373361 +62353030316563653736643432383334663638643362313232316634636362346330663833373934 +34346433626132306330323037396162346662376539663339353866616531653339353532333637 +38313164633735396235363530333734393466393532643436316436316637643535303636383532 +64633961346236386430633765373564326665626662303663653936323062643436666537343030 +31666337363061646432653233656133306132323034343233356564616438653937306235346536 +32623837373164313432623637393433383235633862653866383833303939376530663033666564 +61326163666439616138653230386433323930326564353961363465383031626535626162306564 +39616331363661346264373730353163626132343334323762306465373561633533353730396231 +39643361326165316539626162326536623065343966376334643163666162363438623538663937 +33323138613136623662316263313966663061623537653863386466323039636531623036316334 +33326461396534376233633530643863376337393062313738363065343530643561363962373261 +34306436616535613037633731646332353566643233633766643234306666653630393966633138 +37303739306232303734393934643831613764353962663735346266393062626136323065613836 +36623435613033363730303537386664303931366562303232303336643963643763373363333565 +32373734376162356138633536343366336630343333666336343462653334396538306331363365 +39316161646331393134623137666161356335316332613539373935343461613965633265336362 +38366532346161323362653865653730663435633433653332363633383134386533663464633862 +31333235313764356536376163383330626265373265386565373337653265623037653235323933 +34663564306336663832323735643531643936626161383239336563363764353061343537333337 +31346430633662353136646637613963653531313264666132386564663736383337336639633035 +32356538343135643464306262366462616264636563353361623663633332666239643561666236 +62363064393366636663646139653366623235363131663639343665373261336361346435363561 +63663435373337633534356464326631336566326431646266313865333465336133343937346337 +30393633373362336466633830663534663661643535303632653339646666653732393739613635 +66303632383838366235653366333065333537326161623330356665613834383164356639313862 +39393761353762656136613930643066663363346563333461623331633737663335376539633366 +63326333356434376635363461326663623538633731363665343233383533666662393932346436 +65326430346461343134306232333866346365646632393866386639653438336139663166616439 +64616330336262336366616230613935313663633866306634346263346661343661313131316537 +36326139363233393131613365333366623763383233386463383063616430633437313065326438 +37306365316330373565316237316537636534383333383366303538343931653464646238313163 +36383564663337623935323965323462666634353933366637323139343837653230663639373334 +34306538396537313065336331306461336265636231373331613933363937326339376536616535 +65633765323061643034643433333666666139313233656638386530353632353931383135613436 +31346564666136643562623138626663616331666538356438333530326162373235363831393862 +61373838366531383365393437363233643230353465386431646633653931633738303138666138 +64633532323733376239663037396133386561323265656530306164666436656433633838623566 +35393864646138633735623061633465383934383232356263646335616266353037636234316264 +61616365383965666365353563373833356235316539343630346233393831313139383963323762 +31643066313338616432393436663137383337396238346661383234373535353634666564643565 +30346363343937616437613961623465666434386564656161663530343364646133613435643330 +64333131646434343163313636623730643461663934316263376338306235633437363330623435 +38366139313864666133393136623764613861663733313633393765336531383230663735663232 +36396365356563353536623165323063636639333866633736343235643734356139643534333330 +37346361646561363737633662633339333362306366393236316331363066303463613933313366 +64646164346665663565633235643436333666303233393766323033313062633864376533363663 +65636435336537653231373132363332316636383261393734636439633531346136346263393335 +66626631633039623036313933386662396637643162353365613831343435636539363737316138 +30303333373031643033396531393665663739333933663961613266386232333762353231633463 +30313062636531656161326562373332363832316232386139316634386361303132633335383031 +38376436393364343334323438323564336165333339336666316335633838646130383937633764 +38626233623666653732666434326332373663353439346462396664366466333234363132396566 +36646131386234613936383438633963363634326334313439373635613561346432386631343935 +31366361353861366437356366323539393632336438623536346333636233653361663265363634 +64343263313439376334393031313939623961383165393765316633646535336336326564383335 +64623330653638336539633563623233346237383039646162616166313531363139343063336634 +39306534386261656239313761306239636339306531373435636565623633313761366436323339 +37656266623830386433366566303465306636326562616262643462376538356138613766656138 +38303564653563663566373636326361636161613231373865623339333531613262303262663836 +64366366383665343765393730396563656330316432316132323163333230343434323362633762 +38653965316638326138303933666633333362313939336465326365356631613039343136363865 +64613533373032643361636437373563323463613165313963623537626161643865643364653861 +31316336356235373432643861353938636133316439643362383139373836636565633964346436 +65333662663438616338316563623733656332626231613165306332353632363030346365633937 +66666438313639336334623733623135623161353635653538323632613863346330663762633661 +32333561383461326536393436633262636332383561366130353334366237633033646437366331 +32356561643130376433306231363837383733613662396238666234653533623238646562343936 +38313737656263306265373836356531393265303665613033396565373430343836623139336262 +36656262323539303938646234653165326461616463316365376331333963323139306466333338 +31666566383132306463306533626161323266623131356135643836313033663830333432346236 +39613663343739306439663330653139343965346139336430653035386361386265383334353766 +31326461373666316139333933656365616230313339343035386437646430343464633733613234 +33396631373161346261323735313237653566383061613337303139353934633633363635363232 +34626663666433366135653362643335323631623133303831623533326138643433383465373362 +62323732326538303261343937303936666566333636346331366336616637306236336639303630 +62376630623038653861303237616162376237373932643130396431333665303262633830313132 +32376230366466303238616161336563306630386538616437333861313961383630613664393162 +34633737353363656133363036636166343965646631656162623533306230663165613939656534 +66343363383434333562303835326430373932633163623263346330386266633862363531616432 +64653537363462376562623965626334326533623163633039666663396632306563333438356338 +32623738333632623264393938346135393931636465623032613066306631656233303534616562 +38393565636264613239643932313032306537316135643930386235383762636664303464653733 +37626333616163633933396336663235353638616262623536616435633964366537396262646436 +37633931623963616637393232313031636633386231326365333739386166653332313235663632 +33313138326335613462383534616633376536663963326332393237353836383365623834393936 +32393138316263643636303936653138316633663534386432666562313331383861666633623961 +33316134356461366435666161653232316461343366613763303638343263323432626531663031 +39353638393036373139313563356433643566636333346466393838343239303636613234646237 +66623939383637623534343965316533366531316630663636353663366538303838373061326232 +34323430303365633664303439656432663364353064633738633433336236303831656135363337 +34323463393865386536393634626330646466356335653962316332613861373331303961356439 +37396334646532303861363861356532626464616262343239613239333032626430376261613665 +62353831353837303833656433353639356130373139373230613064353966333831653165653531 +62363638303932613333646166336531346461656434626335383964663435366637666465663163 +33353932396663656134653463346439373530393738346236343139303966303930636434613463 +30313739633132636339316666363439646463333439306565333461346530643234363961323462 +31373534636162366431386130656265333138383535383230383637343030666136643065373437 +65616435373364383565643839363834663339646131303263343333313335363363653537633335 +39623032356261643361343034343039623430653161336334343632613539386238353932343637 +62616565666338336535306234353834653165386262353035393666366433393536383637663638 +34643534353665363735353032623564643634656161336235653136333338353133323566663064 +35363939393461343861663137633664623035666232333563666638633061366461303335613032 +63616338323734326261306462663966306466313034323732353935666130346161333366646131 +31303231316262383266663165396462386366343936303663363737343861656130393039646236 +65643539373539373831663066366261623537366362376338313530356262623664393632323338 +34356365336134356235336539663261323565616131313333663765376334323364613361386634 +39376235326432666135653938663864313064356637653961316432396638366438383336346662 +31613333356630663238633732643936336236376334636138333764626434356565633135313937 +34313464653539326232326430396332383331653634653436343130623733356466383562616333 +31383339353837363862623161306566373165323261373365333366643038376466373337383164 +65353166353335646331346230613265373934656537346132633664613235343063373738616564 +35393465323936303564623031663830306439333833313363306534353130336164356439333566 +38636134613132333834373936303236346539623164366262616165643931326434333761323530 +38613964623230306632346235386531646161393961656137383137633330313065306332613065 +35643664616333643635323931323061313930646230663366666131386138636366393732636362 +65323064646361366133636530326433616162376534363436616631643532343339373266303762 +37646662323961396533653162346534313364653565633534326165663036353163636535643235 +63653039633234363035373230376133373934653333353361306464636664323066633661636430 +37613231333963323864376539643732653164323161313565613666336236373963373533333339 +30376333356635646433643036336532366239353266393538343839356365623862336638306538 +66653934306136633963653138313430613232313433623639616539663839396230393236366330 +61393832376664336163356666633965393632346337653461303064633035306165383361396433 +30383831623030333537633330386233643135636161336431366561636139363838336536353766 +32626537646535363230376663323366356231386466353665366336313065323330653630613139 +39323536393037626631613736613461346331623166396462336138363261346239346563333336 +62616637393537656235323866313739343535353733323135616536626364633337326633633766 +62363832653238333033376365356332633231366132366637663563356232353431333437363136 +34333062656539356563366565313930313036306236333136663365356536366336366130353835 +66626130326463393265373834353732313031303264623234373962383730626632343861373534 +64383064636635383333313339303637633962633838306566343464623630326265346334346434 +39306638643932353935396231623865626539663733396262303663316438356234323633373830 +63343461623663373162386164396235626263336438303831643966373462316539633930353630 +65376133653234306435383264396233346333366665666133393437626265343138643631363137 +62633731643063313065633261656261623561623833303033653964666238336639643363646331 +64316465386336353661633437303763313664303166303639376334346632623664373938613831 +65386461336264633834303137313965356662313131663836646539346361396139636637383137 +33666134623366383764663063323763343662623464393862353963613538353439643238626633 +63336361636639386436653338383064346533383038366135666235353735333730346364396630 +38643238316434386633613032333463666438303136303738306163333135383634616232323763 +37366361323266303333626232333438323834316537623534353536313937623761373963373732 +62346538303261643835346130313864326663333033313535393366383038393532663035616432 +37626134666466646631346230616562663136383233356638616561376466313465346334383735 +34613366323033363836306265383333346539393265623766376530616233613966323136623433 +61343439373937366538356336396131343031366166363336376431356362663564326337306563 +61303161656365346365323065323865666565613836353864616632646336363433343630393466 +64653365653335323233356437333065616233373535366439383161316237393637326138353137 +32393861346537636132653432393531323833373036663838653864363966333639393163323739 +61333836626132333631303531663431616366353133643439643134363539383737346231393361 +66643830363432613236303734616465346531373065613134373839343133336533373862373763 +37623131353864653331646239303938373031386437646533646236623236623062313161343363 +38353836336634366533623530616330373239633966643162313236613061666463383937643539 +64353366313233323139616630303932633661383733343762323062323266623064653561316136 +33663530653861616537646434613066383666653162386136316639666230666434623764373633 +38386436653366666164346139396638353731656233643665623539376131383137643037393731 +65643562653137326531353263303738653631643966313137376332633932393862356631626662 +66353935643635633761636336316136373362636664366166346365643466333163383964366566 +39386332313133353239386435376437663633316433363231333632313438316366643438633231 +38366231383237343862653039643838336237323862323932646264326239306230613737323264 +33303563386230326162356633343066373232386665653765336666376337613362316137643035 +62376633363031376432303666353039343430316565353231663533353635653064326163613831 +37376666663734643233356366346564356262366161653631323036353236313235363039306434 +38356334613762353238353264653962366165313330343262303464366535366532363638343637 +31316537313264386539623537633563383765626465353632323564306565303161656364313965 +39396332376330303134393433316265393232393735303737366430333232383938336566666666 +39313831306665393830613266626335383965313262366461666531306333363066313830373865 +32383766373036396466386461353639356130373235376434653064653631633531313465663066 +38333264643834373736623463333538326262333530383237393666393132373066393238373363 +34346632313932333434366636333131663762666164326239336266373065336436376637313834 +38396161396565383630633535343439373331343035383739346164663836316232663065356261 +39333361393739303232303935303636313831373237666537656564656333316461643937363839 +37613235626437633566383963623233666363393061346137636264366265376664306136386338 +64656363306538393730663661646632633435316464393861313633353236666561636334643766 +36333766346466343339373736383866303437623764666436343934643139623535653030653339 +65616562393039653264636338623336363437613036656637646131306262623239646334663938 +30346532363830616338633535623331373836623563656362363763376334373661323762636336 +35313163336263326437646162373734336530346330326461356336316361323861323563303563 +31356461366633643064303932383531633164626338346462373136393838613532333565303839 +66616666623063666536653235656165373765613736323864363631613735323563313136346135 +36346231313537346364393166356534366661306531646433393061363131333931323437366263 +30636431343738303836343465336565626533643665303439306336663334643631383264363336 +31373162633131623462626365343634313263323061376365313265656262363863623861626637 +65636336333465613834383735653562363630353339353630353831393363333936663638626462 +65626438366665623139303734326136653464383764653437626266623631333938366636383835 +66656361386466633932376134316664633961613331366539313732643534386262386663613162 +36666163353033333036623466616561666439393463653232373730656462396436316132356462 +39393338666335633437383662323963396531396235323666636233323261393463663061306633 +66363536386431636237386338636438373134343633396664643933346631613838653837386639 +38353764396135636435666365393635386663633031356162373437383939643762303766333038 +38636163366561376338333037666631363031313731333238316231396439326330383438663965 +30356264613138626539653036386365616235386538313435646166666139663131346535613835 +30633032366365613562323236623032666134363764646231646161653830336264626639353562 +36653161663833666137653032383364636131303031353462313335393533656138336435346262 +63626330663435653230376333393633643533393161633564303837663564356132373738613333 +31333232353937626662656262333537363464306137646135353938363434663064333963396632 +38306131323831666437333165633261366534356237353333623433363363633230613536313161 +32353837666162303133616665633733643034383937626265626366326339363162306137323166 +65373065336436663533363461313838366161313436316563323562613131336664643466356436 +33373766343639353731303733656434343563626563646262373065346137356163303533633632 +37396233393330333237626438386135393030366439396430326262323137313837303935623134 +38353137623233633161323735646261326632323261313063626662613765343664303265613530 +38333738633932376362643938363937383836396635303835616266376231333839633861353730 +30313439333235376437643736366431616264316365333837616162633731666334346630663166 +37353862663263623961656365366231613531346534373232333564356530353733313864336436 +66353464613362383633356637633363303564633231396635333331323863373538353931643564 +30643331633961373632356233313732666130613732666637353339303534646363313532306431 +34383234633561333131313436343439633063643834393935613632626537366264636132393030 +62383263393566313266386266623361396337633465633731393234366661336537373164636435 +39666339353566333734666331383364396164613461633937376631643535373136383731646535 +37643733396332306336623461343463396166663639376632386334666661643562386365323866 +32363564633566396534656338613732313263663665366439373462646265383266353436646366 +62613661373538656637316362653362653735396264636465336565376166643061616634376337 +62386538396330323336653164653837646166363164363232626263343839656130633039363830 +61633763343164393936623564636532386666666164626663656265653633666335663761303761 +63323639646230343630616537396336373831313363343932636537336233663263303733366434 +63323839316162653562663436666161393537653266383264656261303831386562656334666465 +38333530653064653936366638373933313235366165633565343233633462343361623038646137 +3731 diff --git a/group_vars/all/vars.yml b/group_vars/all/vars.yml index 6b2c3129d..3241b823e 100644 --- a/group_vars/all/vars.yml +++ b/group_vars/all/vars.yml @@ -1,6 +1,6 @@ --- admin_ranges: "129.125.249.0/24,172.23.40.1/24" -ssh_host_signer_ca_keypair_dir: "{{ inventory_dir }}/ssh-host-ca" +ssh_host_signer_ca_keypair_dir: "ssh-host-ca" ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/hpc-ca" ssh_host_signer_key_types: '.*(rsa|ed25519).*' ssh_host_signer_hostnames: "{{ ansible_fqdn }},{{ ansible_hostname }}{% for host in groups['jumphost'] %},{{ host }}+{{ ansible_hostname }}{% endfor %}" @@ -14,190 +14,6 @@ firewall_allowed_tcp_ports: - '22' # SSH - '9100' # Node Exporter # -# Local user account specs. -# Note: -# * all local users are listed here. -# * In ../[name]-cluster/vars.yml we list which users are created locally on which cluster as regular and/or admin users. -# * Never ever change nor recycle a UID value here unless you are in for a surprise... -# -auth_users: - centos: - comment: 'Cloud User' - uid: 1000 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCdOt9U8m3oa/ka8vRTOWxU9uh13hR9F5FoW7SRbrQMWX3XYCEF1mFSTU0WHqYlkOm5atbkqRnR2WUOuG2YjCDJ6KqvpYGjITqHilBCINkWuXozoT5HkGbMtcN1nYDh4b+lGhg3ttfTBKBPusLz0Mca68EL6MjmSsgbRSIceNqFrfbjcc/YhJo7Kn769RW6W/ToClVHNHqgC47ZGXDc5acUrcfiaPNFSlyUjqCMKyO7sGOm/o4TTLffznH4A4iNn+/IX+7dGZRlwcmPjsBlpMk8zjQQqDE6l/UykbwKgYBJRO02PeNg3bqDAwSGR5+e4raJ3/mN3tkQqC/cAD3h4eWaRTBJdnLltkOFFeXux4jvuMFCjLYslxHK/LH//GziarA0OQVqA+9LWkwtLx1rKtNW6OaZd45iandwUuDVzlbADxwXtqjjnoy1ZUsAR83YVyhN/fqgOe2i34Q48h27rdkwRwAINuqnoJLufaXyZdYi4QintKOScp3ps/lSXUJq+zn7yh54JCz2l/MhDNUBpBWvZevJTXxqQBszAp5gv0KE2VuPOyrmzo+QeBxKqglMSonguoVolfb9sEYT5Xhu1zR6thRtoBT813kzpeVSzMUAr/KOD+ILSjWKUNT0JuiCXsEDD7Zqx/kspTsHpi/+2irAdcXgAEA+fiJqxsNfV4cpQw== pneerincx - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBzwniHWpMcGx0Pj3rZvXuaJbZa+iNbNpIhuARXW/GV0 pneerincx ED25519 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCUfwAhBD4vCDYgsr04Kxn1e+vIcx7EzOEJrwi4Bv1Fc329TAifMTLeXXjPlehNvDvxq1Eb6I0v0CA01OwtD2QH+jnKGK7/RXwOfKHZQDsfZ1qL725So8z2rLfTOiIBn01zwSZTPoMC0NoDEj1H7RUpuSTSWazRmZJAi4S9aWU7DK+aWp0vR4UzvxWNFuzhhSJPOrHBx0O6st67oVRyhhIFo67dIfgI/fDwuT7+hAfAzGtuWAW1SI33ucDtaSSs3CT6ndPIU1jzRwrK/Xoq2vzyso6ptj9N/qJfauVUtwhQs//9hGjIP7H2m4maUDR60qDveUy4QNbRoJQuT28FrZxdYjEWyU7E3/yuBSX5Lggk9GuolpGBTj3EDLth0LUsB/hjjGNSebNL/pF5wQR9Usu9omXf4f3dPfU/X0SaWjeY1ukU4saRefn9FIu1ZV3w6TQUybM/2ZcHzbS2JDieirMTZ2uGUVZyAX4TID40Pc84bcFbfQULkqBGPmp2X3rrfJgg8GmmX92qT/OEEPQ6tsA909dxvXGMYzb/7B5MjiAjdkhhIlRzjFz8zy0dkTAMopxwHPI4Fr1z/LhP8Or7pv31HfG/RIW8pOcanvvRRzqoSohDrfxobzczce42S/qrD0sE2gQdwbnAh0JlPmB7erSrqhxEjw0pHXd8CWx4yH3oJQ== gvdvries - pieter: - comment: 'Pieter Neerincx' - uid: 1001 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCdOt9U8m3oa/ka8vRTOWxU9uh13hR9F5FoW7SRbrQMWX3XYCEF1mFSTU0WHqYlkOm5atbkqRnR2WUOuG2YjCDJ6KqvpYGjITqHilBCINkWuXozoT5HkGbMtcN1nYDh4b+lGhg3ttfTBKBPusLz0Mca68EL6MjmSsgbRSIceNqFrfbjcc/YhJo7Kn769RW6W/ToClVHNHqgC47ZGXDc5acUrcfiaPNFSlyUjqCMKyO7sGOm/o4TTLffznH4A4iNn+/IX+7dGZRlwcmPjsBlpMk8zjQQqDE6l/UykbwKgYBJRO02PeNg3bqDAwSGR5+e4raJ3/mN3tkQqC/cAD3h4eWaRTBJdnLltkOFFeXux4jvuMFCjLYslxHK/LH//GziarA0OQVqA+9LWkwtLx1rKtNW6OaZd45iandwUuDVzlbADxwXtqjjnoy1ZUsAR83YVyhN/fqgOe2i34Q48h27rdkwRwAINuqnoJLufaXyZdYi4QintKOScp3ps/lSXUJq+zn7yh54JCz2l/MhDNUBpBWvZevJTXxqQBszAp5gv0KE2VuPOyrmzo+QeBxKqglMSonguoVolfb9sEYT5Xhu1zR6thRtoBT813kzpeVSzMUAr/KOD+ILSjWKUNT0JuiCXsEDD7Zqx/kspTsHpi/+2irAdcXgAEA+fiJqxsNfV4cpQw== pneerincx - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBzwniHWpMcGx0Pj3rZvXuaJbZa+iNbNpIhuARXW/GV0 pneerincx ED25519 - gerben: - comment: 'Gerben van der Vries' - uid: 1002 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCUfwAhBD4vCDYgsr04Kxn1e+vIcx7EzOEJrwi4Bv1Fc329TAifMTLeXXjPlehNvDvxq1Eb6I0v0CA01OwtD2QH+jnKGK7/RXwOfKHZQDsfZ1qL725So8z2rLfTOiIBn01zwSZTPoMC0NoDEj1H7RUpuSTSWazRmZJAi4S9aWU7DK+aWp0vR4UzvxWNFuzhhSJPOrHBx0O6st67oVRyhhIFo67dIfgI/fDwuT7+hAfAzGtuWAW1SI33ucDtaSSs3CT6ndPIU1jzRwrK/Xoq2vzyso6ptj9N/qJfauVUtwhQs//9hGjIP7H2m4maUDR60qDveUy4QNbRoJQuT28FrZxdYjEWyU7E3/yuBSX5Lggk9GuolpGBTj3EDLth0LUsB/hjjGNSebNL/pF5wQR9Usu9omXf4f3dPfU/X0SaWjeY1ukU4saRefn9FIu1ZV3w6TQUybM/2ZcHzbS2JDieirMTZ2uGUVZyAX4TID40Pc84bcFbfQULkqBGPmp2X3rrfJgg8GmmX92qT/OEEPQ6tsA909dxvXGMYzb/7B5MjiAjdkhhIlRzjFz8zy0dkTAMopxwHPI4Fr1z/LhP8Or7pv31HfG/RIW8pOcanvvRRzqoSohDrfxobzczce42S/qrD0sE2gQdwbnAh0JlPmB7erSrqhxEjw0pHXd8CWx4yH3oJQ== gvdvries - marieke: - comment: 'Marieke Bijlsma' - uid: 1003 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDb8ulPLVGL78KJ8Egg7i2V9JLsge4m4+G6kdCuX7p7T7WRFH54DjaBl52UnkgbuTML/2r6c1gk3pXF2wlOtyHKqhD4AyvY1l/NyLSn1kkgY3XaWp64pFmmEydqOOrPX6L9cMGEyPjnfjr/GWbihzFn7E9Hc0kkp7CPbbdAlmwnKTk1m87CtKHVVV7rg7t7tI+pwoBhAGq1KpwxvNyKQT9Duwo+0eP/xZPZ/b12j7edxjjgpEtV+mCldsbXS+JyMVAScJXYV6TYcSyZhNhLnhzZIikjvV8/LcFxt4sURMeWLkiw3EqQOpDazJT6p6zo0KFfglvYG7ps8ijsnYuz4BkvMGx5bJQZVT4RdzQASisEUhJY1t0ZLGfs4bix2yMNmwCkypNZq72G2p/e2A9n1NhVSyOXfzHonQBFbL5xUX/1PNKXt027wTCbnl0OA/gLdez0NeanRzVjfDJGLOueC93rAJRIAWk+UOUBWAmHvL7XdnrgPq2puxk3sKCijUgxEkh1xqgMST5MTq3DMzese4jeuAQErhs5WnkOiythn4i4ydJ0oUwAjZhSFnGBSzol0Iar6chxfsp2U/pcl97QKXGLXkIvlZ7vMtYdbxopJ8uYQaOdkDycU1upR6pylZ6LnP8mF+iTqcHry4rmQ5rp46m2L5Cbp3eJZ7LFPXTVLUvWWw== mbijlsma - egon: - comment: 'Egon Rijpkema' - uid: 1004 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKUBdTEHUj6MxvfEU7KcI+UPAvqJ9jGJ7hHm3e7XFTb9 egon@egon-pc - morris: - comment: 'Morris Swertz' - uid: 1005 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDfKxBNTqlsoTt1DloXbsRDqUyZgYbAGFsSOKhkHfjTD7zotloUwsd7388J/Ip9dOE5xPySWMSqmjcY8FLYIsEnKaC2LKJya6ck0sOrW+kynV+H9VxLsdnErw5bh8Uga3cGeHX+NKRw9dyNkvFB5B690PidBmSXRRTvXVUBvUeYAAdaoVGSQFtgV/lri2ojWR0yVpy2oCqI/eoXO13NJZS8hyoMDTI1QmnuqarNPIIvYmrAr/bO0fNJuzLqzoAcfw6I4rOw/iE8Zuo2Tl9Erjh1J9nJ91Q+78/VY1H7etltNZe4zxtipaB0HfjkHmhTW2xNMNi5D9FkzHbPhlpShzwsajP0xRpQ8JIgsOli/OHnVU0Mzd6WQf43CliNQMj5Qh50TUYdd0IW0ypjz/h2QEmh560R0NHbvRJ6BDHACceszAMPQjj4zlJLxZJejQ2GijWtvL2Yq2XyVlE7rPH3GA1x3Fy29yBNrgkWsH5CKLMudqBiQ6Js9rHJwQx/WjMA6hLiNqxbHW8t5UHNA4C/tppT12qLWvQkAUUOh9ij/aRnT69V4DlZ/nfbtcJWSjiIToCX++GATm1JrlmzGYoqZy5OMGp5SIdd6+CT+D8E01q9nZYkWokT2EeL3r6I1b8CwIVpmDb5cx6d60tOLjh09jeQMc0PcxeRs6Jo6lQj3L4sZw== m.a.swertz@rug.nl - roan: - comment: 'Roan Kanninga' - uid: 1006 - pub_keys: | - # Revoked: key format not compliant with requirements. - wim: - comment: 'Wim Nap' - uid: 1007 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEY6mVTXsbXOfN5FSYvTMgK8JnODeR7NB50Ilvz3eDd9 w.k.nap - ger: - comment: 'Ger Strikwerda' - uid: 1008 - pub_keys: | - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvx1ebTndL/HitD30uNpvESXWUAxT3j0e0CzrBUZ8fHDv+vZTbWBRtWbnLgCnVDPa3GclA1lpnvJD9JBjBhUa8= ger@ger-pc - robin: - comment: 'Robin Teeninga' - uid: 1009 - pub_keys: | - # Revoked: key format not compliant with requirements. - kees: - comment: 'Kees Visser' - uid: 1010 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGbGIEjuURXEI2rPUzLzQSsH/OvZJQwCPFO7w0Uls9Xy stealth@operator - gvdvries: - comment: 'Gerben van der Vries' - uid: 1011 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCUfwAhBD4vCDYgsr04Kxn1e+vIcx7EzOEJrwi4Bv1Fc329TAifMTLeXXjPlehNvDvxq1Eb6I0v0CA01OwtD2QH+jnKGK7/RXwOfKHZQDsfZ1qL725So8z2rLfTOiIBn01zwSZTPoMC0NoDEj1H7RUpuSTSWazRmZJAi4S9aWU7DK+aWp0vR4UzvxWNFuzhhSJPOrHBx0O6st67oVRyhhIFo67dIfgI/fDwuT7+hAfAzGtuWAW1SI33ucDtaSSs3CT6ndPIU1jzRwrK/Xoq2vzyso6ptj9N/qJfauVUtwhQs//9hGjIP7H2m4maUDR60qDveUy4QNbRoJQuT28FrZxdYjEWyU7E3/yuBSX5Lggk9GuolpGBTj3EDLth0LUsB/hjjGNSebNL/pF5wQR9Usu9omXf4f3dPfU/X0SaWjeY1ukU4saRefn9FIu1ZV3w6TQUybM/2ZcHzbS2JDieirMTZ2uGUVZyAX4TID40Pc84bcFbfQULkqBGPmp2X3rrfJgg8GmmX92qT/OEEPQ6tsA909dxvXGMYzb/7B5MjiAjdkhhIlRzjFz8zy0dkTAMopxwHPI4Fr1z/LhP8Or7pv31HfG/RIW8pOcanvvRRzqoSohDrfxobzczce42S/qrD0sE2gQdwbnAh0JlPmB7erSrqhxEjw0pHXd8CWx4yH3oJQ== gvdvries - pneerincx: - comment: 'Pieter Neerincx' - uid: 1012 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCdOt9U8m3oa/ka8vRTOWxU9uh13hR9F5FoW7SRbrQMWX3XYCEF1mFSTU0WHqYlkOm5atbkqRnR2WUOuG2YjCDJ6KqvpYGjITqHilBCINkWuXozoT5HkGbMtcN1nYDh4b+lGhg3ttfTBKBPusLz0Mca68EL6MjmSsgbRSIceNqFrfbjcc/YhJo7Kn769RW6W/ToClVHNHqgC47ZGXDc5acUrcfiaPNFSlyUjqCMKyO7sGOm/o4TTLffznH4A4iNn+/IX+7dGZRlwcmPjsBlpMk8zjQQqDE6l/UykbwKgYBJRO02PeNg3bqDAwSGR5+e4raJ3/mN3tkQqC/cAD3h4eWaRTBJdnLltkOFFeXux4jvuMFCjLYslxHK/LH//GziarA0OQVqA+9LWkwtLx1rKtNW6OaZd45iandwUuDVzlbADxwXtqjjnoy1ZUsAR83YVyhN/fqgOe2i34Q48h27rdkwRwAINuqnoJLufaXyZdYi4QintKOScp3ps/lSXUJq+zn7yh54JCz2l/MhDNUBpBWvZevJTXxqQBszAp5gv0KE2VuPOyrmzo+QeBxKqglMSonguoVolfb9sEYT5Xhu1zR6thRtoBT813kzpeVSzMUAr/KOD+ILSjWKUNT0JuiCXsEDD7Zqx/kspTsHpi/+2irAdcXgAEA+fiJqxsNfV4cpQw== pneerincx - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBzwniHWpMcGx0Pj3rZvXuaJbZa+iNbNpIhuARXW/GV0 pneerincx ED25519 - mbijlsma: - comment: 'Marieke Bijlsma' - uid: 1013 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDb8ulPLVGL78KJ8Egg7i2V9JLsge4m4+G6kdCuX7p7T7WRFH54DjaBl52UnkgbuTML/2r6c1gk3pXF2wlOtyHKqhD4AyvY1l/NyLSn1kkgY3XaWp64pFmmEydqOOrPX6L9cMGEyPjnfjr/GWbihzFn7E9Hc0kkp7CPbbdAlmwnKTk1m87CtKHVVV7rg7t7tI+pwoBhAGq1KpwxvNyKQT9Duwo+0eP/xZPZ/b12j7edxjjgpEtV+mCldsbXS+JyMVAScJXYV6TYcSyZhNhLnhzZIikjvV8/LcFxt4sURMeWLkiw3EqQOpDazJT6p6zo0KFfglvYG7ps8ijsnYuz4BkvMGx5bJQZVT4RdzQASisEUhJY1t0ZLGfs4bix2yMNmwCkypNZq72G2p/e2A9n1NhVSyOXfzHonQBFbL5xUX/1PNKXt027wTCbnl0OA/gLdez0NeanRzVjfDJGLOueC93rAJRIAWk+UOUBWAmHvL7XdnrgPq2puxk3sKCijUgxEkh1xqgMST5MTq3DMzese4jeuAQErhs5WnkOiythn4i4ydJ0oUwAjZhSFnGBSzol0Iar6chxfsp2U/pcl97QKXGLXkIvlZ7vMtYdbxopJ8uYQaOdkDycU1upR6pylZ6LnP8mF+iTqcHry4rmQ5rp46m2L5Cbp3eJZ7LFPXTVLUvWWw== mbijlsma - mswertz: - comment: 'Morris Swertz' - uid: 1014 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDfKxBNTqlsoTt1DloXbsRDqUyZgYbAGFsSOKhkHfjTD7zotloUwsd7388J/Ip9dOE5xPySWMSqmjcY8FLYIsEnKaC2LKJya6ck0sOrW+kynV+H9VxLsdnErw5bh8Uga3cGeHX+NKRw9dyNkvFB5B690PidBmSXRRTvXVUBvUeYAAdaoVGSQFtgV/lri2ojWR0yVpy2oCqI/eoXO13NJZS8hyoMDTI1QmnuqarNPIIvYmrAr/bO0fNJuzLqzoAcfw6I4rOw/iE8Zuo2Tl9Erjh1J9nJ91Q+78/VY1H7etltNZe4zxtipaB0HfjkHmhTW2xNMNi5D9FkzHbPhlpShzwsajP0xRpQ8JIgsOli/OHnVU0Mzd6WQf43CliNQMj5Qh50TUYdd0IW0ypjz/h2QEmh560R0NHbvRJ6BDHACceszAMPQjj4zlJLxZJejQ2GijWtvL2Yq2XyVlE7rPH3GA1x3Fy29yBNrgkWsH5CKLMudqBiQ6Js9rHJwQx/WjMA6hLiNqxbHW8t5UHNA4C/tppT12qLWvQkAUUOh9ij/aRnT69V4DlZ/nfbtcJWSjiIToCX++GATm1JrlmzGYoqZy5OMGp5SIdd6+CT+D8E01q9nZYkWokT2EeL3r6I1b8CwIVpmDb5cx6d60tOLjh09jeQMc0PcxeRs6Jo6lQj3L4sZw== m.a.swertz@rug.nl - rkanninga: - comment: 'Roan Kanninga' - uid: 1015 - pub_keys: | - # Revoked: key format not compliant with requirements. - henkjan: - comment: 'Henk-Jan Zilverberg' - uid: 1016 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKOKgVel0GbF67zZaVR0TFo82e5XeZOP1e3Ld3gIdaER h.j.zilverberg - envsync: - comment: 'Environment Synchronizer' - uid: 1017 - pub_keys: | - # No direct logins required: sudo only. - solve-rd-dm: - comment: 'Datamanager user of solve-rd group' - uid: 1018 - pub_keys: - umcg-atd-dm: - comment: 'Datamanager user of umcg-atd group' - uid: 1019 - pub_keys: - ljohansson: - comment: 'Lennart Johansson' - uid: 1020 - pub_keys: | - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDWqyiEypRvLN6SzmETPpZc4sAeCZO3KFZJCAEXjTgehukD5t7785YBteTq9EF6UmE+5vTNMPVpGDS71YSoI6HmZBao2zqI/rwp6DK+TtK9qCwKXF0ye67WjSFqWgEgDj+RY2C3Hk7Kwb4lzrcWbw+pl6aj4Jzdbe4Yg0UaSvkEiiWifEbjkYNitUmZcVvYcskuUgCTvuCsKDNFtkRXh4CZ6PMC8ovQWbE1zZgDjRGXjZMPURLBQHz2pQhUGndILPoRNBot4CREUnfNUxOGiGI2RwoADSfVeHXKgFlV8r0eK4VZIJWGwwwJzmtnV4SrHMRiAe+xabCXKtppRHW73ujpjrNhuuBrEkILVE38cwbkVHGdVOfb2fS3bMsbEBDRYvR/UNvqyvMdnYp3qM5Hf8chGEcd0E37/Z/1iOeuqXYOs5fx83tVHm7f6RK1BvcUMMk4g71c72fGG6jVswrEOTmhgbOVNTJRUJJMrUPoEs3PKRg0QbQ59WcQ1ZJeMD5/JRWCssEZJbBZBrdBLdlRg4KKCcKzpcMRKFBJXy770p2GGjtAba/TmcD34H2LETVc46NJoLXDMSV8olM1z9wp0tgxJWRJUKMXzCnly2hfjtvrgu5XAnCnvDGYwI6Z58Ekzcjr0EgEmATN4NsyHEWqpYqNbAO1Z8IaAG6vLH4MxGlYbQ== ljohansson - ddanis: - comment: 'Daniel Danis' - uid: 1021 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICKam1PSlpKh2lP/UuY0rDwYMXiPEXJwUoqdwmmEMKih ddanis - ksablauskas: - comment: 'Karolis Sablauskas' - uid: 1022 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMuctiQfcAsVDbcDv7rt7jMxlTKe8eOoLwJdEmdSR4I4 ksablauskas - wsteyaert: - comment: 'Wouter steyaert' - uid: 1023 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHvtYVqBzyLk0gRL0Teq5UnIPjrkJGvbmzjHzPq2D5gw wsteyaert - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA1cD51nNVkZLmVpgnlecprPCNdf6W9IQ/O7KVE5DG3b wsteyaert - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKcTDLafk1hTH5whx4aJyFwG6fPVz9H7EjOrXHaN8ssQ wsteyaert - fokke: - comment: 'Fokke Dijkstra' - uid: 1024 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDRVddVL5RVcv2ubRTNdPQ1Z7L+llPe3jxPgLYKvl6Uo fokke@laptop-cit - slaurie: - comment: 'Steve Laurie' - uid: 1025 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILVkUC6qXGL933sVdlH2dmsTY49s9/Lb9mpP1jpQo1ca slaurie - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAxnQzwIjJEBVcHe9ezfi0YP3tvTvmbwfhfExm/lABIkurwt5Fe4uYnci1h1cN7c+sL1XFZPgNS4fsQHYhfhZF0FI5+gDZQ5ve2ltDqcMvcQ+M8ykZb4mca1pqnaKpG/qTq8cvBI84TmsfZ/99ppVhvJtBX3fs5EOEIo6Kb7aTzTOtsUULDTRvX4PQk7kNCUfAvqP5xHgzxQbEnq4VvUrTcNnrtIH3dvAoLWSAO5f6eTF1KgMXMTdgqQgafKk25cVuhug8hduPQ0k44E8Bx3/qOOU8CavSJF+0gqX+BsjryG3oy2bujfTno8kddAbwfLQgdBZcg+EN8uDrSxF2jHtJMIqvRfit8RzWjKX/W8rtTcXKvq1pfxD25K+aUE5kZeci4UQL2GSVCTbxJHaDURVpLA0YM6Bm2ewHbbQsmDR8RwH6IGocq70ukj3HfDrae8DOV93xyEsKvnupTy9eR1lvzhDrzfTA4P1m2PZxxq/ud4g15lBNOOrlOvofNIi3X+klJ9oO09EvvvJRpHXrEVWFpU5w2ceVnze6E6jcmoPX/mj+L0984lJf3TS0Zx2aLU93IM6qlQ5kHsnMHbMLry4QN6fD1V/PRcjKuuxL1+ZNTVimMXW/Tl0cOleGsXslLxwzOGXaLeOcdvekjE2OSC+V+vvt9rquGqcCqwoDWarpCjk= slaurie - sido: - comment: 'Sido Haakma' - uid: 1026 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDo/zQab61JMdNlZH1A9bKNqhmemvqimh0tNiaXc45UA shaakma - mpauper: - comment: 'Marc Pauper' - uid: 1027 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKX6eNqh9P3BNH96VDXxu/LimWGcJGfSYIpKowvoWxmm mpauper - jvdvelde: - comment: 'Joeri van der Velde' - uid: 1028 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHR3d0L9yW9LwjaqodEAWs3frNEjyj0M+fq/+jY+I9rh jvdvelde - fmusacchia: - comment: 'Francesco Musacchia' - uid: 1029 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJjTuMz1RGEWYDvBr1f9gMc71T5Gwx0txPEAsyKVcFXW fmusacchia - dspalding: - comment: 'Dylan Spalding' - uid: 1030 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINazVx99YGc98wZBtlJcP8KqtJ2LZ1K+JfVWx5wtSNjN jdylanspalding - psivakumar: - comment: 'Prasanth Sivakumar' - uid: 1031 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK2c7bI3+hjALAfsytm6rQ+VU3hPRVGgrU5fUSrVHMMY psivakumar - fandrade: - comment: 'Fernanda de Andrade' - uid: 1032 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKvNTAsJVVZrWgfuLO7Vj35vdKusWwmcWJGovGIR+S88 fdeandrade - gdemidov: - comment: 'German Demidov' - uid: 1033 - pub_keys: | - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOeknUxTyEeYoTGyDTsRV8WnLMS7tbhy/9FamgmUj5V3 gmdemidov - -# # Local group specs. # Note: # * all local groups are listed here. diff --git a/group_vars/boxy-cluster/secrets.yml b/group_vars/boxy-cluster/secrets.yml new file mode 100644 index 000000000..2cf59d5ab --- /dev/null +++ b/group_vars/boxy-cluster/secrets.yml @@ -0,0 +1,7 @@ +$ANSIBLE_VAULT;1.2;AES256;boxy +36623362373535623136633932643237393366373931343066626138356634633866643162653366 +3130303639373965363536313532326336623830636463640a353738306562646430393331373139 +35386566356434613430333633613935343431316138323631636263663862626130316636613838 +6562326139323436640a323462393263396161383338613663373032336637353762316466636234 +63636532306333613664393163303832393038653765613966613531343165363430386339323131 +6232303063313166306532356539306662663536663965313133 diff --git a/group_vars/boxy-cluster/vars.yml b/group_vars/boxy-cluster/vars.yml new file mode 100644 index 000000000..8216914ed --- /dev/null +++ b/group_vars/boxy-cluster/vars.yml @@ -0,0 +1,27 @@ +--- +slurm_cluster_name: 'boxy' +slurm_cluster_domain: 'hpc.rug.nl' +stack_prefix: 'bx' +ldap_uri: ldap://172.23.40.249 +ldap_base: ou=umcg,o=asds +ldap_binddn: cn=clusteradminumcg,o=asds +regular_groups: + - 'umcg-atd' + - 'umcg-lifelines' +regular_users: + - user: 'umcg-lifelines-dm' + groups: ['umcg-lifelines'] + sudoers: '%umcg-lifelines-dms' +pfs_mounts: [ + { pfs: 'umcgst08', + source: '172.23.34.213@tcp:172.23.34.214@tcp:', + type: 'lustre', + rw_options: 'defaults', + ro_options: 'defaults,ro' }, +] +lfs_mounts: [ + { lfs: 'tmp03', + pfs: 'umcgst08', + groups: ['umcg-atd', 'umcg-lifelines'] }, +] +... \ No newline at end of file diff --git a/group_vars/docs.yml b/group_vars/docs.yml new file mode 100644 index 000000000..2bbece7f7 --- /dev/null +++ b/group_vars/docs.yml @@ -0,0 +1,9 @@ +--- +# +# Configure allowed network ports for geerlingguy.firewall role +# +firewall_allowed_tcp_ports: + - '22' # SSH + - '80' # HTTP + - '443' # HTTPS +... diff --git a/group_vars/fender-cluster/secrets.yml b/group_vars/fender-cluster/secrets.yml index f077c6bc9..5879b1f1d 100644 --- a/group_vars/fender-cluster/secrets.yml +++ b/group_vars/fender-cluster/secrets.yml @@ -1,57 +1,56 @@ -$ANSIBLE_VAULT;1.1;AES256 -39353463643262323431356561653834313234353365363566633330333636326135336335666430 -3632386562376164613435653530366261396130366164660a303331333361393266313232313566 -66333833656463653262323135306637343761643766383565303936346363343734636433343662 -3463643464643034650a666166663030666434373431353532636437353135613464343239336136 -62336532656565646331306566373564333665336366313436316231666134336366343732653830 -32663361653462623665303866656364623834343861313936393732386462386562663833383063 -38393636313136623762643231633963383038656563366436306439353838306230383030363061 -30313939363334336538356363386136363539316431343261313362343731316237373163316533 -37656131623863653730356362383735376433643164343462633739343763633639333162663764 -35323763363433363136333232313265366433366137666665643865336265623637373334626666 -38656632346662656534653364366133626630353432616166646561306530656635326130343136 -66313436313230623837306130326435636661303566656664636464306366636361633635323363 -65626438386366303938316436386364646630663463373834326638373065343861393931633465 -31343339306635393136333361353964636361616165373536356164316361663431646136356134 -63356266653962613134363162306534396564363737366236616137653961346637663635313132 -32353063373466306461656537363336616665656666656631356234313866633263376661633262 -32623164633233653366386165666266336264623133376134336531663139383365643063333266 -35363637303038633136353532633838356661623035663762336237343338316266626335633362 -33396464363131363431363832316666336337623162363737313632313466653763386434393037 -64343632313631396562323131396562616434356238336561653931616361646462303261313731 -34323633303532386139396663353635626634313761313539356161643333306561386431623366 -32393966643232366130356234316265643366313661313230323431663165376436643738383037 -64383037366639353361303063373864333134373062666335393434363733353735633864666432 -32313034616264323661613934306561323439353332343736326531303932636430663133643634 -64316163623430623033343365343831323532393366653966316138633462366434636239373636 -39306566613330393034636537366338313532303038666331343330323330333064663365383239 -34633361303061316130393537636434666362623831363361633861316433636631353665366261 -33626338373662343539393033386333363631353365333061653632383530376264666237666630 -62643936356263383461633133633635346338323932376630323138303264336163366338386539 -65363831343632386463316662366130303461396261363238323531316462326538653434613236 -66383031396631636239316533643138313762363231343966666361363139646337363138373961 -62633132623531633431353433323631633137393030633764343866633434316538363935613761 -65363466613539313539633863346333653939383731616664643938633830353061353236623731 -63636162313163646339616438353939353964313264656236653337386166333130346666316239 -35353434306462646230343130633534373537363764366634363330316264326662313730633637 -62303964666661363637306565643136316336393531646166303138303561343035623430356538 -33336364633261333630653338313830313238666334323938643435353738313662396337333238 -32643238366433393532613436363933306633313439336466373730373461376262303166393435 -32313835353036336231366565653835373432666464626231653638303031383866366230656638 -62616637633462616634383530363161343431623335623732386639633765376536663638323134 -64623966613034343761383738316136363236376362666132303061613534383463373461383138 -64303334616561666437656366316531643738366636613365663566633038303830383163316266 -35356538656331363638333464656137333762353437316633356466313563366661353364363962 -31633931613238336537636337306265666334383862613161613663626537346336323266343166 -31643332343439386562633632343865303236383530386333653032376338613930613065316166 -37346262356138393835346431353563316130373932396161336238643861356238356131653539 -63393433306263366333646333323538393330373266656365643765633935323363666334343463 -61656133393066666530653936303432343264383265646530343239623535646637336538616434 -33313863306264633932396634656330386237353036303730623963363138633261663762643937 -65343630616436386164643732663632646265616430636262376630656436353233353137393830 -39623631613737386638323565663135653331373836613862653735613266626263313039383734 -62383237343064306531633332636334323465313366333937396130343631393163653131646431 -31373836336462343561393637366365356566363666663338313037336237643762363033343466 -66373833623062326166336235313866633436316535306137373333633131313331303533633333 -61353837613133326435306431613138323430663632333535386538323235623835653732643866 -33373862613961643339 +$ANSIBLE_VAULT;1.2;AES256;fender +36366239323230613866316336346361356133393530313739666431636165653833393862623462 +3635366335383232336561613461663966623134346365660a363235653263353434633133353563 +38383836356437666665383333643866353133373836376564383639393862383233663831343539 +3166363766303164370a303135326534396131636531623934396434373233663662633331633661 +36613830373961383438373662353365313736666663306535383531653939663432376133633163 +30376337303561653136653163313162396336396362303235373339613739313862613562623766 +34643961366339336239396135313264643064383435333030333438373739646639626663663133 +63353239393633393236336661323163313838393732313735326534356561666363393735343163 +64643161343636636639616265383238663866346338643132373230323235326361333864633363 +63343935613533323164636237396238313332633239303562333539373237383134636531666136 +65353566323161306532353762353864313530643835633965646333346162643462356266396539 +35353637303166613431323130303064363738323836623231303965303362336135326562373232 +35313838326339623637633630663765313030353164313865343330613966643031313133656639 +32373635363661373334633732623439356237366334633161623233656332663330383930653762 +66336131396362383362323363656337356363363266393835656463373032386433646262323639 +36366437623066306137336530333135643935626165613137623634363730323463313161376561 +65306135326139663231616332353564616337393338333166363964383430343562623534393962 +65646139366335323361346438376637633736363066623537386130363139663665363937306366 +65333332303966663038386563623265643465333837633832666339386361333866343362326232 +61663334383634663936656437616564363064333030326435343734313666653163353161343533 +32336361363065346162613439313439303337656332353134623536636366656430396436663436 +37643734306534366534306237336138376465646535346364303465626261653839373462306537 +36323639653066303164343962363733666631303735616238636163306337396637633534636565 +38646566653366316163326530653635323533333738303936653736386234393036353663353339 +66666463666632393537383330316666643231336636313264396136623566376330346432303335 +36363366393738303934653663666133643530636164636264363535393632346236346432326435 +64383439363935376561396632316563323234323266333834636337663832306232666631613535 +30356166646366613761613739303930656139666233643435633564313462316638323763336366 +64353334653064646335626166303037623063333062393361353266323561646361363632386130 +36633337323735646436623335616538346237313861386135313461326533373333346363653537 +33626235663163623864383430353438623835613164623136616331646234386339646335393963 +34623938613731373435326437323434306335343064346337363435306436626437633865363134 +65303561616238623666393365313662363331656430363939623764393830306331616239613766 +30613663386231343839323235373630353464383536616462633031643735396636303464353731 +39353935376332343938643736303664653132383466343237313135313635386366616336316536 +30633338663862373830633430336132306131363864656562303737316536653566656336313965 +31373039323438353739626332623961663061313639363534653265636561623138633735656461 +61396264343739366335396639303235383736323864313038656230313131373936613936356333 +65333564613636333132353466316461616238373962623131323336653733396337386534633933 +61316139633030393437386231393234653837363236643731653765663336396537663761366334 +61316133373763313537386234373663643233326666656463366264646135343738356464666661 +32353733383637303430323136373536373862363630323035373134323366343036303332373132 +38666639646363393964656161633735313964623931343439363235323732613363656433396363 +32336565303366353538333165313764333037663861336338663966366139333036616638666534 +63623264346262656535343737303835356561646566316162623439373966646130376437646532 +37393662306534666262613461396135373965336665353963363635373132343835626361323537 +32353532306135303766656266663031393433306165393061306331626230636233353736316534 +62376538323236323934333539366235666139626436326665633334656463333038323733643866 +64663830386239313636656133376234623338633437353064663064393630613032623337323338 +62353238346136653766346330636231343637386332343562326365646361646632333566306163 +64336665326634323734313763303763666336333832366539336330383466613733393866383538 +31643131646636333635353236316635333162666265363862623166366239333635386236343238 +32613066373536386130653435373835373435656264386366386364653730383664663535646131 +64633236376439643937613861316231623537616630656438386531383166386337613135386432 +626239343734393139336264323932383266 diff --git a/group_vars/fender-cluster/vars.yml b/group_vars/fender-cluster/vars.yml index a16f5be89..0ecc67af1 100644 --- a/group_vars/fender-cluster/vars.yml +++ b/group_vars/fender-cluster/vars.yml @@ -6,6 +6,7 @@ slurm_version: '17.11.9-2.umcg.el7' mailhub: '192.168.0.5' rewrite_domain: "{{ stack_prefix }}-sai{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}" motd: "To solve or not to solve, that's the question." +figlet_font: 'slant' vcompute_hostnames: "{{ stack_prefix }}-vcompute[01-12]" vcompute_sockets: 16 vcompute_cores_per_socket: 1 @@ -29,8 +30,7 @@ ui_ethernet_interfaces: ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/ca-key-production-ebi" use_ldap: yes create_ldap: yes -uri_ldap: fd-dai -uri_ldaps: fd-dai +ldap_uri: ldap://fd-dai ldap_port: 389 ldaps_port: 636 ldap_base: dc=hpc,dc=rug,dc=nl @@ -53,6 +53,7 @@ private_storage_net_id: net_provider_vlan3126 private_storage_subnet_id: subnet3126 security_group_id: SSH-and-ping-2 slurm_ldap: true +fuse_mountpoint: /groups/solve-rd/prm10/EGAD00001005352/ availability_zone: AZ_1 storage_availability_zone: nova local_volume_size: 1 @@ -72,8 +73,6 @@ local_admin_users: - 'morris' - 'pieter' - 'wim' - - 'umcg-atd-dm' - - 'solve-rd-dm' - 'gerben' envsync_user: 'envsync' envsync_group: 'depad' @@ -88,10 +87,13 @@ regular_users: groups: ['depad'] - user: 'solve-rd-dm' groups: ['solve-rd'] - sudoers: ['%solve-rd'] + sudoers: '%solve-rd' - user: 'umcg-atd-dm' groups: ['umcg-atd'] - sudoers: ['%umcg-atd'] + sudoers: '%umcg-atd' + - user: 'solve-rd-ateambot' + groups: ['solve-rd'] + sudoers: '%solve-rd' - user: 'gvdvries' groups: ['users', 'depad','umcg-atd', 'solve-rd'] - user: 'mbijlsma' @@ -106,14 +108,14 @@ regular_users: groups: ['users', 'depad','umcg-atd', 'solve-rd'] - user: 'ddanis' groups: ['users', 'solve-rd'] + - user: 'dhendriksen' + groups: ['users', 'solve-rd'] - user: 'ksablauskas' groups: ['users', 'solve-rd'] - user: 'wsteyaert' groups: ['users', 'solve-rd'] - user: 'slaurie' groups: ['users', 'solve-rd'] - - user: 'mpauper' - groups: ['users', 'solve-rd'] - user: 'jvdvelde' groups: ['users', 'solve-rd'] - user: 'fmusacchia' @@ -126,21 +128,73 @@ regular_users: groups: ['users', 'solve-rd'] - user: 'gdemidov' groups: ['users', 'solve-rd'] - + - user: 'amohan' + groups: ['users', 'solve-rd'] + - user: 'cthomas' + groups: ['users', 'solve-rd'] + - user: 'mbenjamins' + groups: ['users','depad', 'umcg-atd', 'solve-rd'] + - user: 'lmatalonga' + groups: ['users', 'solve-rd'] + - user: 'gbullich' + groups: ['users', 'solve-rd'] + - user: 'ebenetti' + groups: ['users', 'solve-rd'] + - user: 'kdelange' + groups: ['users', 'depad', 'umcg-atd', 'solve-rd'] + - user: 'sli' + groups: ['users', 'solve-rd'] + - user: 'rschuele' + groups: ['users', 'solve-rd'] + - user: 'jvandrovcova' + groups: ['users', 'solve-rd'] + - user: 'itepaske' + groups: ['users', 'solve-rd'] + - user: 'jpelaez' + groups: ['users', 'solve-rd'] + - user: 'asommer' + groups: ['users', 'solve-rd'] + - user: 'droelofs' + groups: ['users', 'solve-rd'] + - user: 'dlagorce' + groups: ['users', 'solve-rd'] + - user: 'gwarren' + groups: ['users', 'solve-rd'] + - user: 'mmehtarizadeh' + groups: ['users', 'solve-rd'] + - user: 'cveal' + groups: ['users', 'solve-rd'] + - user: 'rthompson' + groups: ['users', 'solve-rd'] + - user: 'alaner' + groups: ['users', 'solve-rd'] + - user: 'edeboer01' + groups: ['users', 'solve-rd'] + - user: 'adenommepichon' + groups: ['users', 'solve-rd'] + - user: 'avitobello' + groups: ['users', 'solve-rd'] + - user: 'yduffourd' + groups: ['users', 'solve-rd'] + - user: 'elopez02' + groups: ['users', 'solve-rd'] + - user: 'amarcegrau' + groups: ['users', 'solve-rd'] pfs_mounts: [ { pfs: 'ecst02', - source: 'em-isi-3126.ebi.ac.uk:/ifs/Solve-RD/ecst02', + source: 'em-isi-3126.ebi.ac.uk:/ifs/Solve-RD', type: 'nfs', rw_options: 'defaults,_netdev,noatime,nodiratime', ro_options: 'defaults,_netdev,noatime,nodiratime,ro' }, ] lfs_mounts: [ { lfs: 'home', - pfs: 'ecst02' }, - { lfs: 'groups/GROUP/tmp10', + pfs: 'ecst02', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp10', pfs: 'ecst02', groups: ['umcg-atd', 'solve-rd'] }, - { lfs: 'groups/GROUP/prm10', + { lfs: 'prm10', pfs: 'ecst02', groups: ['umcg-atd', 'solve-rd'] }, { lfs: 'env10', diff --git a/group_vars/gearshift-cluster/secrets.yml b/group_vars/gearshift-cluster/secrets.yml index b418faa27..939139a7a 100644 --- a/group_vars/gearshift-cluster/secrets.yml +++ b/group_vars/gearshift-cluster/secrets.yml @@ -1,4 +1,4 @@ -$ANSIBLE_VAULT;1.1;AES256 +$ANSIBLE_VAULT;1.1;AES256;gearshift 39306430383937653465313262643266626162353565393265316337633637643134373437366234 3734316636626335343130383363353465343734633263620a313133616435333236373134343261 33656164373834656366303233626432623636316633303433346466386233393638373965653037 diff --git a/group_vars/gearshift-cluster/vars.yml b/group_vars/gearshift-cluster/vars.yml index d90a9f598..ff6348a6d 100644 --- a/group_vars/gearshift-cluster/vars.yml +++ b/group_vars/gearshift-cluster/vars.yml @@ -5,21 +5,24 @@ stack_prefix: 'gs' slurm_version: '18.08.8-1.el7.umcg' mailhub: '172.23.34.34' rewrite_domain: "imperator{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}" +figlet_font: 'cyberlarge' motd: | !!! WARNING: This cluster is in beta testing and may be redeployed from scratch, which - may result in complete data loss starting - on 09:00 on the following EOS Fridays: + may result in complete data loss of home and tmp01 group folders + starting on 09:00 on the following EOS Fridays: ################################### # Date EOS # ################################### - # Friday 2019-05-03 Sprint 136 # - # Friday 2019-06-14 Sprint 138 # - # Friday 2019-07-26 Sprint 140 # - # Friday 2019-09-06 Sprint 142 # - # Friday 2019-10-18 Sprint 144 # + # Friday 2020-03-12 Sprint 152 # + # Friday 2020-04-24 Sprint 154 # + # Friday 2020-06-05 Sprint 156 # + # Friday 2020-07-16 Sprint 158 # + # Friday 2020-08-27 Sprint 160 # ################################### You have been warned!!! + This does not affect prm03 group folders, + which are on a different storage system. additional_etc_hosts: | 172.23.40.21 gs-compute01 gs-compute01.hpc.local 172.23.40.22 gs-compute02 gs-compute02.hpc.local @@ -32,7 +35,7 @@ additional_etc_hosts: | 172.23.40.29 gs-compute09 gs-compute09.hpc.local 172.23.40.30 gs-compute10 gs-compute10.hpc.local 172.23.40.31 gs-compute11 gs-compute11.hpc.local -vcompute_hostnames: "{{ stack_prefix }}-vcompute[01-11]" +vcompute_hostnames: "{{ stack_prefix }}-vcompute[01-10]" vcompute_sockets: 24 vcompute_cores_per_socket: 1 vcompute_real_memory: 221501 @@ -46,22 +49,24 @@ vcompute_ethernet_interfaces: - 'eth2' ui_hostnames: "{{ slurm_cluster_name }}" ui_sockets: 2 -ui_cores_per_socket: 1 -ui_real_memory: 3789 +ui_cores_per_socket: 12 +ui_real_memory: 221500 ui_local_disk: 0 -ui_features: 'prm01,tmp01' +ui_features: 'prm03,tmp01' ui_ethernet_interfaces: - 'eth0' - 'eth1' ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/umcg-hpc-ca" use_ldap: yes create_ldap: no -uri_ldap: 172.23.40.249 -uri_ldaps: comanage-in.id.rug.nl -ldap_port: 389 -ldaps_port: 636 -ldap_base: ou=research,o=asds -ldap_binddn: cn=clusteradminumcg,o=asds +ldap_uri: 'ldap://172.23.40.249' +ldap_port: '389' +ldaps_port: '636' +ldap_base: 'ou=research,o=asds' +ldap_binddn: 'cn=clusteradminumcg,o=asds' +ldap_group_object_class: 'groupofnames' +ldap_group_quota_soft_limit_template: 'ruggroupumcgquotaLFSsoft' +ldap_group_quota_hard_limit_template: 'ruggroupumcgquotaLFS' filter_passwd: '(|(rugpersonentitlementvalue=scz)(rugpersonentitlementvalue=umcg))' filter_shadow: '(|(rugpersonentitlementvalue=scz)(rugpersonentitlementvalue=umcg))' pam_authz_search: '(|(&(objectClass=posixGroup)(cn=co_bbmri_g-GRP_Gearshift)(memberUid=$username))(&(cn=$username)(rugpersonentitlementvalue=umcg)))' @@ -78,29 +83,214 @@ local_admin_users: - 'henkjan' - 'wim' - 'kees' + - 'robin' + - 'marlies' envsync_user: 'umcg-envsync' envsync_group: 'umcg-depad' +functional_admin_group: 'umcg-funad' hpc_env_prefix: '/apps' +regular_groups: + - 'umcg-aad' + - 'umcg-as' + - 'umcg-atd' + - 'umcg-biogen' + - 'umcg-bionic-mdd-gwama' + - 'umcg-bios' + - 'umcg-dag3' + - 'umcg-datateam' + - 'umcg-depad' + - 'umcg-endocrinology' + - 'umcg-franke-scrna' + - 'umcg-gaf' + - 'umcg-gap' + - 'umcg-gastrocol' + - 'umcg-gcc' + - 'umcg-gdio' + - 'umcg-gonl' + - 'umcg-griac' + - 'umcg-gsad' + - 'umcg-impact' + - 'umcg-lifelines' + - 'umcg-lld' + - 'umcg-llnext' + - 'umcg-micompany' + - 'umcg-mmbimg' + - 'umcg-msb' + - 'umcg-oncogenetics' + - 'umcg-pub' + - 'umcg-radicon' + - 'umcg-solve-rd' + - 'umcg-tifn' + - 'umcg-ukb' + - 'umcg-ugli' + - 'umcg-verbeek' + - 'umcg-weersma' + - 'umcg-wijmenga' +regular_users: + - user: 'umcg-aad-dm' + groups: ['umcg-aad'] + sudoers: '%umcg-aad' + - user: 'umcg-capicebot' + groups: ['umcg-as'] + sudoers: '%umcg-as' + - user: 'umcg-as-dm' + groups: ['umcg-as'] + sudoers: '%umcg-as' + - user: 'umcg-atd-ateambot' + groups: ['umcg-atd'] + sudoers: '%umcg-atd' + - user: 'umcg-atd-dm' + groups: ['umcg-atd'] + sudoers: '%umcg-atd' + - user: 'umcg-biogen-dm' + groups: ['umcg-biogen'] + sudoers: 'umcg-hwestra,umcg-ndeklein' + - user: 'umcg-bionic-mdd-gwama-dm' + groups: ['umcg-bionic-mdd-gwama'] + sudoers: 'umcg-fhuider,umcg-ifedko' + - user: 'umcg-bios-dm' + groups: ['umcg-bios'] + sudoers: '%umcg-bios' + - user: 'umcg-dag3-dm' + groups: ['umcg-dag3'] + sudoers: 'umcg-avich,umcg-akurilshchikov,umcg-rgacesa' + - user: 'umcg-datateam-dm' + groups: ['umcg-datateam'] + sudoers: 'umcg-mbijlsma' + - user: 'umcg-endocrinology-dm' + groups: ['umcg-endocrinology'] + sudoers: '%umcg-endocrinology-dms' + - user: 'umcg-franke-scrna-dm' + groups: ['umcg-franke-scrna'] + sudoers: 'umcg-ddevries,umcg-hbrugge,umcg-mvdwijst' + - user: 'umcg-gaf-ateambot' + groups: ['umcg-gaf'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-gaf-dm' + groups: ['umcg-gaf'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-gap-ateambot' + groups: ['umcg-gap'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-gap-dm' + groups: ['umcg-gap'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-gastrocol-dm' + groups: ['umcg-gastrocol'] + sudoers: 'umcg-avich,umcg-hushixian' + - user: 'umcg-gcc-dm' + groups: ['umcg-gcc'] + sudoers: '%umcg-gcc' + - user: 'umcg-gdio-dm' + groups: ['umcg-gdio'] + sudoers: 'umcg-cvandiemen' + - user: 'umcg-gonl-dm' + groups: ['umcg-gonl'] + sudoers: 'umcg-pneerincx' + - user: 'umcg-griac-dm' + groups: ['umcg-griac'] + sudoers: 'umcg-cvermeulen,umcg-cxu,umcg-ndijk,umcg-mdevries,umcg-mberg' + - user: 'umcg-gsad-dm' + groups: ['umcg-gsad'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-impact-dm' + groups: ['umcg-impact'] + sudoers: '%umcg-impact' + - user: 'umcg-lifelines-dm' + groups: ['umcg-lifelines'] + sudoers: '%umcg-lifelines-dms' + - user: 'umcg-lld-dm' + groups: ['umcg-lld'] + sudoers: 'umcg-hwestra,umcg-hbrugge' + - user: 'umcg-llnext-dm' + groups: ['umcg-llnext'] + sudoers: 'umcg-tsinha,umcg-sgarmaeva' + - user: 'umcg-micompany-dm' + groups: ['umcg-micompany'] + sudoers: 'umcg-cxu,umcg-cvermeulen' + - user: 'umcg-mmbimg-dm' + groups: ['umcg-mmbimg'] + sudoers: '%umcg-mmbimg-dms' + - user: 'umcg-msb-dm' + groups: ['umcg-msb'] + sudoers: '%umcg-msb' + - user: 'umcg-oncogenetics-dm' + groups: ['umcg-oncogenetics'] + sudoers: '%umcg-mterpstra' + - user: 'umcg-pub-dm' + groups: ['umcg-pub'] + sudoers: 'umcg-pneerincx' + - user: 'umcg-radicon-dm' + groups: ['umcg-radicon'] + sudoers: 'umcg-gvdvries,umcg-kdelange,umcg-mbenjamins,umcg-mbijlsma,umcg-pneerincx,umcg-rkanninga' + - user: 'umcg-solve-rd-dm' + groups: ['umcg-solve-rd'] + sudoers: 'umcg-gvdvries,umcg-mbijlsma,umcg-pneerincx' + - user: 'umcg-tifn-dm' + groups: ['umcg-tifn'] + sudoers: 'umcg-akurilshchikov,umcg-jfu' + - user: 'umcg-ukb-dm' + groups: ['umcg-ukb'] + sudoers: '%umcg-ukb-dms' + - user: 'umcg-ugli-dm' + groups: ['umcg-ugli'] + sudoers: 'umcg-pdeelen,umcg-mbenjamins,umcg-pneerincx' + - user: 'umcg-verbeek-dm' + groups: ['umcg-verbeek'] + sudoers: '%umcg-verbeek' + - user: 'umcg-weersma-dm' + groups: ['umcg-weersma'] + sudoers: 'umcg-avich,umcg-hushixian' + - user: 'umcg-wijmenga-dm' + groups: ['umcg-wijmenga'] + sudoers: 'umcg-jgelderloos,umcg-aclaringbould,umcg-rmodderman,umcg-hwestra,umcg-avich,umcg-obakker,umcg-ddevries' +lustre_quota_type: 'group' pfs_mounts: [ { pfs: 'umcgst10', - source: 'gcc-storage001.stor.hpc.local:/ifs/rekencluster/umcgst10', + source: 'gcc-storage001.stor.hpc.local:/ifs/rekencluster', type: 'nfs4', rw_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime', ro_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime,ro' }, + { pfs: 'umcgst02', + source: '172.23.57.203@tcp12:172.23.57.204@tcp12:/dh2', + type: 'lustre', + rw_options: 'defaults,lazystatfs,flock', + ro_options: 'defaults,lazystatfs,ro' }, ] # -# During public beta phase we only use tmp* folders/mounts for 'production' groups. -# Only anlysis team development (umcg-atd) group, which is exclusively used for testing has prm* folder/mount for now. +# Only anlysis team development (umcg-atd) group, which is exclusively used for testing has prm01 folder/mount for now. # lfs_mounts: [ { lfs: 'home', - pfs: 'umcgst10' }, - { lfs: 'groups/GROUP/tmp01', pfs: 'umcgst10', - groups: ['umcg-atd', 'umcg-biogen', 'umcg-bios', 'umcg-dag3', 'umcg-gastrocol', 'umcg-gcc', 'umcg-lld', 'umcg-tifn', 'umcg-weersma', 'umcg-wijmenga' ] }, - { lfs: 'groups/GROUP/prm01', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp01', pfs: 'umcgst10', - groups: ['umcg-atd'] }, + groups: [ + 'umcg-aad', 'umcg-as', 'umcg-atd', 'umcg-biogen', 'umcg-bionic-mdd-gwama', + 'umcg-bios', 'umcg-dag3', 'umcg-datateam', 'umcg-endocrinology', 'umcg-franke-scrna', + 'umcg-gaf', 'umcg-gap', 'umcg-gastrocol', 'umcg-gcc', 'umcg-gdio', 'umcg-gonl', + 'umcg-griac', 'umcg-gsad', 'umcg-impact', 'umcg-lifelines', 'umcg-lld', + 'umcg-llnext', 'umcg-micompany', 'umcg-mmbimg', 'umcg-msb', 'umcg-oncogenetics', + 'umcg-pub', 'umcg-radicon', 'umcg-solve-rd', 'umcg-tifn', 'umcg-ukb', 'umcg-ugli', + 'umcg-verbeek', 'umcg-weersma', 'umcg-wijmenga' + ]}, + { lfs: 'prm01', + pfs: 'umcgst10', + groups: [ + 'umcg-atd' + ]}, + { lfs: 'prm03', + pfs: 'umcgst02', + groups: [ + 'umcg-aad', 'umcg-as', 'umcg-atd', 'umcg-biogen', 'umcg-bionic-mdd-gwama', + 'umcg-bios', 'umcg-dag3', 'umcg-datateam', 'umcg-endocrinology', 'umcg-franke-scrna', + 'umcg-gaf', 'umcg-gap', 'umcg-gastrocol', 'umcg-gcc', 'umcg-gdio', 'umcg-gonl', + 'umcg-griac', 'umcg-gsad', 'umcg-impact', 'umcg-lifelines', 'umcg-lld', + 'umcg-llnext', 'umcg-micompany', 'umcg-mmbimg', 'umcg-msb', 'umcg-oncogenetics', + 'umcg-pub', 'umcg-radicon', 'umcg-solve-rd', 'umcg-tifn', 'umcg-ukb', 'umcg-ugli', + 'umcg-verbeek', 'umcg-weersma', 'umcg-wijmenga' + ]}, { lfs: 'env01', pfs: 'umcgst10', machines: "{{ groups['compute-vm'] + groups['user-interface'] }}" }, diff --git a/group_vars/hyperchicken-cluster/secrets.yml b/group_vars/hyperchicken-cluster/secrets.yml index ecb98c0c6..d72741ca3 100644 --- a/group_vars/hyperchicken-cluster/secrets.yml +++ b/group_vars/hyperchicken-cluster/secrets.yml @@ -1,38 +1,37 @@ -$ANSIBLE_VAULT;1.1;AES256 -39396536363834383637346134303432333739303439633863303336363465386639653537643537 -6338653434323233646434313434323731626264346639340a626164643631303337623838373064 -65643930383966393161333161623630393130393132623131356137396466626235313861643465 -3430626437633662650a313739333333303137303038363532336236656565396139643161393336 -31383764626133316263333265653536303133353766656666663264663463343135633865393464 -61616461626265623533326436383933326365323666336565363239643936663738643462353666 -34386631383639636538616434613633353263376336633631396366656537326463393932633261 -39343663636430643835663165613164343437343832643136653535373934643461363564653935 -62356337613165383538373330656332346663663338333166613038396461303430383763656236 -63666361616364393230663264623337636165313039386430386164303033396532373439636535 -32656636663165613763353864633538663638393161613234383765656337633437613732616431 -61366463636636363661346634393366623362653031656430356563623737303832376439303365 -37353834383830306139306239643436623030633535316634343465656235623639363533393737 -36623436616564643936663037623731393438316465393364303835393963316333626438373737 -35373363383862353836623061373362646637633634376431353838346239643933663432646539 -35323764613962346537363834353833373339363135643230396361306237313866643834663433 -33336563613037363866663364333962383933343430356638616431656539313363383763376463 -32643338316235396164326538393163386137336564636433373166616639366139343939306665 -32313730333239306335343832356663613863336637643334613232633163326366613666336366 -62363362353063383236636463653037343562363462616634633933333338343434626534323063 -34373535396461656264636436356433636138613066373466336664656666396331306434623738 -66653962623937666230646161333764616134626432353831663236633537613832396636643336 -32373139323638396263353566313161333933626366373063316533636432363166656338313565 -66316334333764336634643039663936373561333261316432333530353261323765663934323861 -32336533663638636563343066633638666434376162643861393161633337393231313931613361 -63366564616432633863333231386137363735346437313235303862373032353039643336663731 -66616134353732326436663464306562393538323432373238643465326531346430636161343665 -39626262383630386630366566383466313931316130393830343161336333313531633465616561 -39396661393831613564623462626364663231666362653663613466386532653235383437633462 -31626533386239653234623365393264326633386463393666633036353465373837376335316162 -30623539313538373934393835663331663932336266633163303636333239393661613236386633 -35393562303930343137633030303766396564613266623763326461346365653261393336613333 -33633932666636646430633563393362363666663766633531333333666365663761386130323164 -65663864356662396531316331336561383930366231643537653632623038333430346338386163 -34643431386139373864303462666335613261303832333639386639613965616365613836306430 -32666661633832653835643263633539333166613261376135336362306562313961313030623763 -3736 +$ANSIBLE_VAULT;1.2;AES256;hyperchicken +66353738346134663036333364646165363636366235343761663431353139626662376530326261 +3261623936646136643464343830623566636236326131300a626533626433643539343430663330 +30386631613831616261383136633838663565383136306234663261313365326338653730666136 +3134646135306132300a616564646237333133613234643666383561323831383434646338323166 +65653330376133303563343239313761646634313163363162336362326333613033646137616338 +34616232393065306664643961333838363864396533383761656137383232346265653032613537 +61396631623030313839323062623965353438313234386561633932313534343030613661346466 +32326230393064393133346534393932363730663238353861646133343264663938616338616231 +37303730393831623862346633373634353562303939346139376331333830633361323136363131 +65393161393334313834396639666333373465353837386432353331336137366338316139303032 +35333435303130343462393738303966376662636639656432316266653563363535313330646138 +33646136316136323463323934383630623834303033616662313662373162363864613638373638 +33623936616634383961306538643861333236623835306230343537353061373335376365356634 +61626132646237653864653061346336316262326235366533643532393036633061353431376539 +31653330353966333238653032323662393738666564346263623861313066336439313461316265 +38353364636533306133376339326334646531393964653031656639666166656432333263633432 +66316562303233303539333136323339343132316532376637363338656435376130626438633537 +36326634636136653838303363346466643730393139396232366438363962316533646562666566 +64316439386139656639653038656231643761623233633861343833653664663431636366613963 +33623732363065313561323739646537376535393661383134613737643465343432303166393234 +34616162363332306436633638633762376538616263666430623961366139656462353236333030 +64333935613665306232356138666637306337663866656236656434643835306331613161373163 +64653461646339636364643636376164363462306131353666656130316338623032313133306230 +36613236336436663830646265363364303635323037356134633437303137373234646635383133 +32346630373831353635346161646230383537656636666436356330336532643532333236393639 +63626234363634383730393335376537393932316637336266613464613261316639363034353361 +39656239643936643031333835303162623832353666343365336334323335326531356436353435 +34363938623036393938363432323262333537616530663634386664356134646434383062393537 +61633034653435393336313366653634393466656266383033643263623338393236336432366237 +32643034613233626437646430646661613961386362353237313334393633393635373733666436 +65353966376464653330623135623631306164393334636262636339393138663266656366353965 +39653835303662396332633765316464313966383334623735626266373532303436636164323630 +30356363643330393339353261316464636432383233343165646234643136363165393761303933 +31386261636237643436623034653833366537643862393165346532636639346331626666656330 +38326362366232373362373138373665386531343764613039633933376463613430393933333163 +39326632646265363163 diff --git a/group_vars/hyperchicken-cluster/vars.yml b/group_vars/hyperchicken-cluster/vars.yml index 0d33900d7..e39246d72 100644 --- a/group_vars/hyperchicken-cluster/vars.yml +++ b/group_vars/hyperchicken-cluster/vars.yml @@ -6,6 +6,7 @@ slurm_version: '18.08.8-1.el7.umcg' mailhub: '192.168.0.5' rewrite_domain: "{{ stack_prefix }}-sai{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}" motd: "To solve or not to solve, that's the question." +figlet_font: 'doh' vcompute_hostnames: "{{ stack_prefix }}-vcompute01" vcompute_sockets: 16 vcompute_cores_per_socket: 1 @@ -29,8 +30,7 @@ ui_ethernet_interfaces: ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/umcg-hpc-development-ca" use_ldap: yes create_ldap: yes -uri_ldap: hc-dai -uri_ldaps: hc-dai +ldap_uri: ldap://hc-dai ldap_port: 389 ldaps_port: 636 ldap_base: dc=hpc,dc=rug,dc=nl @@ -64,9 +64,6 @@ nameservers: [ local_admin_groups: - 'admin' - 'docker' - - 'solve-rd' - - 'umcg-atd' - - 'depad' local_admin_users: - 'centos' - 'egon' @@ -76,9 +73,6 @@ local_admin_users: - 'morris' - 'pieter' - 'wim' - - 'umcg-atd-dm' - - 'solve-rd-dm' - - 'envsync' envsync_user: 'envsync' envsync_group: 'depad' hpc_env_prefix: '/apps' @@ -92,10 +86,10 @@ regular_users: groups: ['depad'] - user: 'solve-rd-dm' groups: ['solve-rd'] - sudoers: ['%solve-rd'] + sudoers: '%solve-rd' - user: 'umcg-atd-dm' groups: ['umcg-atd'] - sudoers: ['%umcg-atd'] + sudoers: '%umcg-atd' - user: 'gvdvries' groups: ['users', 'depad', 'umcg-atd', 'solve-rd'] - user: 'mbijlsma' @@ -108,24 +102,25 @@ regular_users: groups: ['users', 'depad', 'umcg-atd', 'solve-rd'] - user: 'umcg-atd-dm' groups: ['users', 'umcg-atd'] - sudoers: ['pieter','gerben'] + sudoers: 'pieter,gerben' - user: 'solve-rd-dm' groups: ['users', 'solve-rd'] - sudoers: ['%solve-rd'] + sudoers: '%solve-rd' pfs_mounts: [ { pfs: 'ecst01', - source: 'em-isi-3126.ebi.ac.uk:/ifs/Solve-RD/ecst01', + source: 'em-isi-3126.ebi.ac.uk:/ifs/Solve-RD', type: 'nfs', rw_options: 'defaults,_netdev,noatime,nodiratime', ro_options: 'defaults,_netdev,noatime,nodiratime,ro' }, ] lfs_mounts: [ { lfs: 'home', - pfs: 'ecst01' }, - { lfs: 'groups/GROUP/tmp09', + pfs: 'ecst01', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp09', pfs: 'ecst01', groups: ['umcg-atd', 'solve-rd'] }, - { lfs: 'groups/GROUP/prm09', + { lfs: 'prm09', pfs: 'ecst01', groups: ['umcg-atd', 'solve-rd'] }, { lfs: 'env09', diff --git a/group_vars/jumphost.yml b/group_vars/jumphost.yml index bdcc2698f..db61cfe13 100644 --- a/group_vars/jumphost.yml +++ b/group_vars/jumphost.yml @@ -1,10 +1,9 @@ --- firewall_allowed_tcp_ports: - - "22" - - "80" + - "22" # SSH. + - "443" # SSH fallback when 22 is blocked. + - "3000" # Grafana server. firewall_additional_rules: - - "iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 22" - "iptables -A INPUT -i eth1 -p tcp -s 129.125.2.233,129.125.2.225,129.125.2.226 --dport 9090 -j ACCEPT -m comment --comment 'prometheus server'" - ssh_host_signer_hostnames: "{{ ansible_hostname }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %},{{ ansible_hostname }}{% if public_ip_addresses is defined and public_ip_addresses[ansible_hostname] | length %},{{ public_ip_addresses[ansible_hostname] }}{% endif %}" ... diff --git a/group_vars/marvin-cluster/secrets.yml b/group_vars/marvin-cluster/secrets.yml new file mode 100644 index 000000000..d95b1ddff --- /dev/null +++ b/group_vars/marvin-cluster/secrets.yml @@ -0,0 +1,26 @@ +$ANSIBLE_VAULT;1.1;AES256 +66646230336431313862346230623038616234626563616133363038343932373038333135356165 +6633346638383962343739623866643465396331643932330a656533306666643364613538653231 +62396435643364643066613636353230393432363932336462613830333962623338633836366432 +6461373236633639330a633630363938303663653931613331656337633061353866653631663638 +31323462363131633934336139396133333037383435666231653462626265353930643739393339 +33633431383338653265386633333039646465396334646561656137376139353732656562316365 +61666163623462656539326430363631383862383235333535383565363232626439613436363935 +63323930366664366261333733333630343239333663646430313134393366323030333637383231 +63643833303531653730656137393963346465643264623335393062373339303166336233663363 +35633762623937653064656365386462653234383630363839323434386332333838623638616366 +64343032613439346333636162653137376563313631353431643464383633353730613137656332 +38666365333465356338373965623231396363383232653235323639623665396330303066616265 +38626232663263336236653339326139343239346638393833333435343035636433396138663662 +30633835316336613331343039656665626162633736303962626130333234326564353732363636 +30376662376230613163613965346665616664303935343338353661323963373866333836613136 +62643463653039353634396131616164393465626435353338616235393134653236373262616164 +34393866633132346538333966353431653961316361363162656665663764323365336263343332 +39613163626665363530373561363662363366393266383830313637363239353137636436373330 +66343366623161633964623334316463323362386161633235343036663162623063626230633263 +33656336346334353430663266343038373633343434306338666266636366626533383264623961 +63356437336564643032336665336363623234336566353632666466643732373037373935653865 +32333936393566656134636535386363636231616662343765353331623661653132623939663334 +63656464656363303636616364353564323966313462346562303234333836613961626266616336 +37383130313331636562623237313063373332626137656336343533656334613262376536393062 +6661 diff --git a/group_vars/marvin-cluster/vars.yml b/group_vars/marvin-cluster/vars.yml new file mode 100644 index 000000000..bebf2a9fa --- /dev/null +++ b/group_vars/marvin-cluster/vars.yml @@ -0,0 +1,156 @@ +--- +slurm_cluster_name: 'marvin' +slurm_cluster_domain: '' +stack_prefix: 'mv' +slurm_version: '18.08.8-1.el7.umcg' +mailhub: '192.168.0.5' +rewrite_domain: "{{ stack_prefix }}-sai{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}" +motd: "It is rare, but not unheard of." +figlet_font: 'speed' +vcompute_hostnames: "{{ stack_prefix }}-vcompute01" +vcompute_sockets: 16 +vcompute_cores_per_socket: 1 +vcompute_real_memory: 64264 +vcompute_max_cpus_per_node: "{{ vcompute_sockets * vcompute_cores_per_socket - 2 }}" +vcompute_max_mem_per_node: "{{ vcompute_real_memory - vcompute_sockets * vcompute_cores_per_socket * 512 }}" +vcompute_local_disk: 0 +vcompute_features: 'tmp11' +vcompute_ethernet_interfaces: + - 'eth0' + - 'eth1' +ui_hostnames: "{{ slurm_cluster_name }}" +ui_sockets: 4 +ui_cores_per_socket: 1 +ui_real_memory: 7821 +ui_local_disk: 0 +ui_features: 'prm11,tmp11' +ui_ethernet_interfaces: + - 'eth0' + - 'eth1' +ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/ca-key-production-ebi" +use_ldap: yes +create_ldap: yes +ldap_uri: ldap://mv-dai +ldap_port: 389 +ldaps_port: 636 +ldap_base: dc=ejp,dc=rd,dc=nl +ldap_binddn: cn=admin,dc=ejp,dc=rd,dc=nl +ldap_domain: ejp.rd.nl +key_name: Gerben +image_cirros: cirros-0.3.4-x86_64-disk.img +image_centos7: centos7 +cloud_user: centos +flavor_ui: s1.large +flavor_vcompute: s1.gargantuan +flavor_management: s1.small +public_net_id: ext-net-37 +public_ip_addresses: + dockingport: '193.62.55.244' +private_net_id: EJP-RD-network +private_subnet_id: EJP-RD-subnet +private_storage_net_id: +private_storage_subnet_id: +security_group_id: SSH-and-ping-2 +slurm_ldap: true +availability_zone: AZ_1 +storage_availability_zone: nova +local_volume_size: 1 +nameservers: [ +# '/em-isi-3126.ebi.ac.uk/10.35.126.201', # Local DNS lookups for shared storage. + '8.8.4.4', # Google DNS. + '8.8.8.8', # Google DNS. +] +local_admin_groups: + - 'admin' + - 'docker' + - 'umcg-atd' + - 'depad' +local_admin_users: + - 'centos' + - 'egon' + - 'gerben' + - 'henkjan' + - 'marieke' + - 'morris' + - 'pieter' + - 'wim' + - 'umcg-atd-dm' + - 'envsync' +envsync_user: 'envsync' +envsync_group: 'depad' +hpc_env_prefix: '/apps' +regular_groups: + - 'users' + - 'depad' + - 'erknet' + - 'euro-nmd' + - 'ern-liver' + - 'umcg-atd' +regular_users: + - user: 'envsync' + groups: ['depad'] + + - user: 'erknet-dm' + groups: ['erknet'] + sudoers: '%erknet' + + - user: 'euro-nmd-dm' + groups: ['euro-nmd'] + sudoers: '%euro-nmd' + + - user: 'ern-liver-dm' + groups: ['ern-liver'] + sudoers: '%ern-liver' + + - user: 'umcg-atd-dm' + groups: ['umcg-atd'] + sudoers: '%umcg-atd' + - user: 'gvdvries' + groups: ['users', 'depad', 'umcg-atd', 'erknet', 'euro-nmd','ern-liver'] + - user: 'mbijlsma' + groups: ['users', 'depad', 'umcg-atd', 'erknet', 'euro-nmd','ern-liver'] + - user: 'mswertz' + groups: ['users', 'depad', 'umcg-atd', 'erknet', 'euro-nmd','ern-liver'] + - user: 'pneerincx' + groups: ['users', 'depad', 'umcg-atd', 'erknet', 'euro-nmd','ern-liver'] + - user: 'rkanninga' + groups: ['users', 'depad', 'umcg-atd', 'erknet', 'euro-nmd','ern-liver'] + - user: 'umcg-atd-dm' + groups: ['users', 'umcg-atd'] + sudoers: 'pieter,gerben' + - user: 'erknet-dm' + groups: ['users', 'erknet'] + sudoers: '%erknet' + - user: 'euro-nmd-dm' + groups: ['users', 'euro-nmd'] + sudoers: '%euro-nmd' + - user: 'ern-liver-dm' + groups: ['users', 'ern-liver'] + sudoers: '%ern-liver' +pfs_mounts: [ + { pfs: 'ecst01', + source: '', + type: 'nfs', + rw_options: 'defaults,_netdev,noatime,nodiratime', + ro_options: 'defaults,_netdev,noatime,nodiratime,ro' }, +] +lfs_mounts: [ + { lfs: 'home', + pfs: 'ecst01', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp11', + pfs: 'ecst01', + groups: ['umcg-atd', 'erknet', 'euro-nmd', 'ern-liver'] }, + { lfs: 'prm11', + pfs: 'ecst01', + groups: ['umcg-atd', 'erknet', 'euro-nmd', 'ern-liver'] }, + { lfs: 'env11', + pfs: 'ecst01', + machines: "{{ groups['compute-vm'] + groups['user-interface'] }}" }, +] +interfaces: + - device: 'eth0' + bootproto: 'dhcp' + - device: 'eth1' + bootproto: 'dhcp' +... diff --git a/group_vars/nibbler-cluster/vars.yml b/group_vars/nibbler-cluster/vars.yml index 094738b5b..6420054e4 100644 --- a/group_vars/nibbler-cluster/vars.yml +++ b/group_vars/nibbler-cluster/vars.yml @@ -27,8 +27,7 @@ ui_ethernet_interfaces: ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/umcg-hpc-development-ca" use_ldap: yes create_ldap: no -uri_ldap: ldap.pilot.scz.lab.surf.nl -uri_ldaps: ldap.pilot.scz.lab.surf.nl +ldap_uri: ldap://ldap.pilot.scz.lab.surf.nl ldap_port: 636 ldaps_port: 636 ldap_base: o=ElixirNL,dc=pilot-clients,dc=scz,dc=lab,dc=surf,dc=nl @@ -95,23 +94,24 @@ regular_users: - user: 'solve-rd-dm' groups: ['users', 'solve-rd'] pfs_mounts: [ - { pfs: 'Local-NFS', - source: 'nb-nfs:/nfs', + { pfs: 'nfs', + source: 'nb-nfs:', type: 'nfs4', rw_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime', ro_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime,ro' }, ] lfs_mounts: [ { lfs: 'home', - pfs: 'Local-NFS' }, - { lfs: 'groups/GROUP/tmp09', - pfs: 'Local-NFS', + pfs: 'nfs', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp09', + pfs: 'nfs', groups: ['umcg-atd', 'solve-rd'] }, - { lfs: 'groups/GROUP/prm09', - pfs: 'Local-NFS', + { lfs: 'prm09', + pfs: 'nfs', groups: ['umcg-atd', 'solve-rd'] }, { lfs: 'env09', - pfs: 'Local-NFS', + pfs: 'nfs', machines: "{{ groups['compute-vm'] + groups['user-interface'] }}" }, ] ... diff --git a/group_vars/talos-cluster/secrets.yml b/group_vars/talos-cluster/secrets.yml index 4a22e8d4d..114bfe578 100644 --- a/group_vars/talos-cluster/secrets.yml +++ b/group_vars/talos-cluster/secrets.yml @@ -1,4 +1,4 @@ -$ANSIBLE_VAULT;1.1;AES256 +$ANSIBLE_VAULT;1.1;AES256;talos 30373339623134663537366566656333666430313866636235646439626335333264383838663566 6239353966343130313534643037383230636436653233340a646662396165323861343261313465 39356330383536616534386230313433656339353830376632623037393238386463383563653233 diff --git a/group_vars/talos-cluster/vars.yml b/group_vars/talos-cluster/vars.yml index c4d77adbf..611126195 100644 --- a/group_vars/talos-cluster/vars.yml +++ b/group_vars/talos-cluster/vars.yml @@ -3,8 +3,10 @@ slurm_cluster_name: 'talos' slurm_cluster_domain: 'hpc.rug.nl' stack_prefix: 'tl' slurm_version: '18.08.8-1.el7.umcg' +slurm_allow_jobs_to_span_nodes: true mailhub: '172.23.34.34' rewrite_domain: "{{ stack_prefix }}-sai{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}" +figlet_font: 'ogre' motd: | !!! WARNING: This cluster is in beta testing and may be redeployed from scratch, which @@ -13,11 +15,11 @@ motd: | ################################### # Date EOS # ################################### - # Friday 2019-05-03 Sprint 136 # - # Friday 2019-06-14 Sprint 138 # - # Friday 2019-07-26 Sprint 140 # - # Friday 2019-09-06 Sprint 142 # - # Friday 2019-10-18 Sprint 144 # + # Friday 2020-03-12 Sprint 152 # + # Friday 2020-04-24 Sprint 154 # + # Friday 2020-06-05 Sprint 156 # + # Friday 2020-07-16 Sprint 158 # + # Friday 2020-08-27 Sprint 160 # ################################### You have been warned!!! vcompute_hostnames: "{{ stack_prefix }}-vcompute[01-03]" @@ -26,7 +28,7 @@ vcompute_cores_per_socket: 1 vcompute_real_memory: 7821 vcompute_max_cpus_per_node: "{{ vcompute_sockets * vcompute_cores_per_socket - 2 }}" vcompute_max_mem_per_node: "{{ vcompute_real_memory - vcompute_sockets * vcompute_cores_per_socket * 512 }}" -vcompute_local_disk: 0 +vcompute_local_disk: 270000 vcompute_features: 'tmp08' vcompute_ethernet_interfaces: - 'eth0' @@ -43,12 +45,14 @@ ui_ethernet_interfaces: ssh_host_signer_ca_private_key: "{{ ssh_host_signer_ca_keypair_dir }}/umcg-hpc-development-ca" use_ldap: yes create_ldap: no -uri_ldap: 172.23.40.249 -uri_ldaps: comanage-in.id.rug.nl -ldap_port: 389 -ldaps_port: 636 -ldap_base: ou=umcg,o=asds -ldap_binddn: cn=clusteradminumcg,o=asds +ldap_uri: 'ldap://172.23.40.249' +ldap_port: '389' +ldaps_port: '636' +ldap_base: 'ou=umcg,o=asds' +ldap_binddn: 'cn=clusteradminumcg,o=asds' +ldap_group_object_class: 'groupofnames' +ldap_group_quota_soft_limit_template: 'ruggroupumcgquotaLFSsoft' +ldap_group_quota_hard_limit_template: 'ruggroupumcgquotaLFS' nameservers: [ '/gcc-storage001.stor.hpc.local/172.23.40.244', # Local DNS lookups for shared storage. '8.8.4.4', # Google DNS. @@ -58,34 +62,152 @@ local_admin_groups: - 'admin' - 'docker' local_admin_users: + - 'alex2' - 'egon' - 'gerben' - 'henkjan' + - 'kim' - 'marieke' + - 'marlies' - 'morris' - 'pieter' + - 'robin' - 'wim' envsync_user: 'umcg-envsync' envsync_group: 'umcg-depad' +functional_admin_group: 'umcg-funad' hpc_env_prefix: '/apps' +regular_groups: + - 'umcg-atd' + - 'umcg-depad' + - 'umcg-endocrinology' + - 'umcg-gcc' + - 'umcg-lifelines' +regular_users: + - user: 'umcg-atd-ateambot' + groups: ['umcg-atd'] + sudoers: '%umcg-atd' + - user: 'umcg-atd-dm' + groups: ['umcg-atd'] + sudoers: '%umcg-atd' + - user: 'umcg-endocrinology-dm' + groups: ['umcg-endocrinology'] + sudoers: '%umcg-endocrinology-dms' + - user: 'umcg-gcc-dm' + groups: ['umcg-gcc'] + sudoers: '%umcg-gcc' + - user: 'umcg-lifelines-dm' + groups: ['umcg-lifelines'] + sudoers: '%umcg-lifelines-dms' pfs_mounts: [ { pfs: 'umcgst11', - source: 'gcc-storage001.stor.hpc.local:/ifs/rekencluster/umcgst11', + source: 'gcc-storage001.stor.hpc.local:/ifs/rekencluster', type: 'nfs4', rw_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime', ro_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime,ro' }, ] +lustre_quota_type: 'group' lfs_mounts: [ { lfs: 'home', - pfs: 'umcgst11' }, - { lfs: 'groups/GROUP/tmp08', pfs: 'umcgst11', - groups: ['umcg-atd', 'umcg-gcc'] }, - { lfs: 'groups/GROUP/prm08', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'tmp08', pfs: 'umcgst11', - groups: ['umcg-atd', 'umcg-gcc'] }, + groups: ['umcg-atd', 'umcg-endocrinology', 'umcg-gcc', 'umcg-lifelines'] }, + { lfs: 'prm08', + pfs: 'umcgst11', + groups: ['umcg-atd', 'umcg-gcc', 'umcg-lifelines'] }, { lfs: 'env08', pfs: 'umcgst11', machines: "{{ groups['compute-vm'] + groups['user-interface'] }}" }, ] -... \ No newline at end of file +iptables_allow_icmp_inbound: + - 'umcg_net1' + - 'umcg_net2' + - 'umcg_net3' + - 'rug_bwp_net' + - 'rug_operator' + - 'rug_gcc_cloud_net' + - 'foyer' + - 'boxy' + - 'bender' + - 'lobby' + - 'calculon' + - 'flexo' + - 'gate' + - 'zinc-finger' + - 'coenzyme' + - 'passage' + - 'leucine-zipper' + - 'chaperone' + - 'airlock' + - 'jenkins1' + - 'jenkins2' +iptables_allow_ssh_inbound: + - 'umcg_net1' + - 'umcg_net2' + - 'umcg_net3' + - 'rug_bwp_net' + - 'rug_operator' + - 'foyer' + - 'boxy' + - 'bender' + - 'lobby' + - 'calculon' + - 'flexo' + - 'gate' + - 'zinc-finger' + - 'coenzyme' + - 'passage' + - 'leucine-zipper' + - 'chaperone' + - 'airlock' + - 'jenkins1' + - 'jenkins2' +iptables_allow_ssh_outbound: + - 'foyer' + - 'boxy' + - 'bender' + - 'lobby' + - 'calculon' + - 'flexo' + - 'gate' + - 'peregrine' + - 'gattaca01' + - 'gattaca02' + - 'cher_ami' + - 'eriba_ds' + - 'molgenis_downloads' + - 'airlock' + - 'surfsara_grid_ui' + - 'lumc_shark_ui' + - 'cnag_sftp' + - 'erasmus_mc_net' + - 'rug_f5_net' + - 'sanger_sftp' +iptables_allow_ebi_mysql_outbound: + - 'ebi_sanger_net1' + - 'ebi_sanger_net2' +iptables_allow_ftp_outbound: + - 'ebi_sanger_net1' + - 'ebi_sanger_net2' + - 'broad_ftp' + - 'ncbi_net1' + - 'ncbi_net2' +iptables_allow_aspera_outbound: + - 'ebi_sanger_net1' + - 'ebi_sanger_net2' + - 'broad_aspera_1' + - 'broad_aspera_2' + - 'broad_aspera_3' + - 'broad_aspera_4' + - 'broad_aspera_5' + - 'broad_aspera_6' + - 'broad_aspera_7' + - 'broad_aspera_8' + - 'broad_aspera_9' + - 'ncbi_net1' + - 'ncbi_net2' +iptables_allow_globus_outbound: + - 'sanger_globus' +... diff --git a/heat/heat_cluster.yml b/heat/heat_cluster.yml index 043553a01..a3a8c8f72 100644 --- a/heat/heat_cluster.yml +++ b/heat/heat_cluster.yml @@ -15,7 +15,7 @@ resources: properties: key_name: adminkey image: {get_param: image_name} - flavor: 4C-8GB + flavor: 24C-240GB networks: - network: vlan983 fixed_ip: 172.23.40.33 diff --git a/inventory.py b/inventory.py index 12a64974a..00642d9d5 100755 --- a/inventory.py +++ b/inventory.py @@ -5,9 +5,10 @@ External inventory script for Ansible ============================================================= -Generates Ansible inventory with hostnames from a static inventory file located -in the same dir as this script. By default this script looks for an inventory named +Generates Ansible inventory with hostnames from a static inventory file. +By default this script looks for an inventory named inventory.ini +located in the same dir as this script. or alternatively for an inventory file name as defined in export AI_INVENTORY='some_inventory.ini' @@ -112,9 +113,8 @@ def __init__(self): ' is either missing or not readable: Check path and permissions.\n' + ' If your static inventory file has a different name,\n' + ' you need to export the AI_INVENTORY environment variable\n' + - ' to point to a static inventory file in the same dir as where\n' + - ' ' + os.path.realpath(__file__) + "\n" + - ' is located.') + ' to specify where your static inventory file is located relative to\n' + + ' ' + os.path.realpath(__file__) + "\n") sys.exit(1) # Read settings and parse CLI arguments. diff --git a/.lorrc b/lor-init similarity index 62% rename from .lorrc rename to lor-init index ad5b70a45..9c125b48d 100755 --- a/.lorrc +++ b/lor-init @@ -4,12 +4,12 @@ # for which the config is stored in the repo containing this script. # # This script assumes there is for each CLUSTERNAME a corresponding: -# 1. ./group_vars/CLUSTERNAME-cluster/ Ansible group variables and secrets encrypted with Ansible vault. -# 2. ./CLUSTERNAME_hosts.ini Ansible inventory file in the root of this repo. -# 3. ./.vault_pass.txt.CLUSTERNAME Ansible vault password file in the root of this repo. +# 1. ./group_vars/CLUSTERNAME-cluster/ Ansible group variables and secrets encrypted with Ansible vault. +# 2. ./CLUSTERNAME_hosts.ini Ansible inventory file in the root of this repo. +# 3. ./.vault/vault_pass.txt.CLUSTERNAME Ansible vault password file in the root of this repo. # # Once this code was sourced (not executed!) like this: -# $> . .lorrc +# $> . ./lor-init # You can call the lor-config function for a specific cluster. E.g.: # $> lor-config talos # @@ -38,8 +38,9 @@ function lor-config() { fi declare -a required_paths=( "${LOR_DIR}/group_vars/${cluster}-cluster/" - "${LOR_DIR}/${cluster}_hosts.ini" - "${LOR_DIR}/.vault_pass.txt.${cluster}" + "${LOR_DIR}/static_inventories/${cluster}_hosts.ini" + "${LOR_DIR}/.vault/vault_pass.txt.${cluster}" + "${LOR_DIR}/.vault/vault_pass.txt.all" ) for required_path in "${required_paths[@]}"; do if [[ ! -e "${required_path}" ]]; then @@ -50,17 +51,17 @@ function lor-config() { # # Get jumphost from inventory file. # - local jumphost=$(fgrep -A1 '[jumphost]' "${LOR_DIR}/${cluster}_hosts.ini" | tail -1) + local jumphost=$(fgrep -A1 '[jumphost]' "${LOR_DIR}/static_inventories/${cluster}_hosts.ini" | tail -1) # # Init and report current setup. # cd ${LOR_DIR} - export AI_INVENTORY="${cluster}_hosts.ini" + export AI_INVENTORY="static_inventories/${cluster}_hosts.ini" export AI_PROXY="${jumphost}" - export ANSIBLE_VAULT_PASSWORD_FILE=".vault_pass.txt.${cluster}" + export ANSIBLE_VAULT_IDENTITY_LIST="all@.vault/vault_pass.txt.all, ${cluster}@.vault/vault_pass.txt.${cluster}" export ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS=ignore printf '%s\n' "Current working directory is $(pwd)" printf '%s\n' "Using AI_PROXY: ${AI_PROXY}" printf '%s\n' "Using AI_INVENTORY: ${AI_INVENTORY}" - printf '%s\n' "Using ANSIBLE_VAULT_PASSWORD_FILE: ${ANSIBLE_VAULT_PASSWORD_FILE}" + printf '%s\n' "Using ANSIBLE_VAULT_IDENTITY_LIST: ${ANSIBLE_VAULT_IDENTITY_LIST}" } diff --git a/roles/build-environment/tasks/generic-build-dependencies.yml b/roles/build-environment/tasks/generic-build-dependencies.yml index d719382a9..bb8d52805 100644 --- a/roles/build-environment/tasks/generic-build-dependencies.yml +++ b/roles/build-environment/tasks/generic-build-dependencies.yml @@ -4,13 +4,18 @@ state: latest update_cache: yes name: - - "@development" + - '@development' - 'glibc-common' - 'glibc-devel' - 'glibc-headers' - 'gnutls-devel' + - 'intltool' - 'kernel-devel' - 'libselinux-devel' - 'openssl-devel' + - 'patch' + - 'qt5-qtbase-devel' + - 'qt5-qtxmlpatterns-devel' + - 'rpm-build' become: true ... \ No newline at end of file diff --git a/roles/cadvisor/handlers/main.yml b/roles/cadvisor/handlers/main.yml new file mode 100644 index 000000000..c3ab3c559 --- /dev/null +++ b/roles/cadvisor/handlers/main.yml @@ -0,0 +1,17 @@ +--- +# +# Important: maintain correct handler order. +# Handlers are executed in the order in which they are defined +# and not in the order in which they are listed in a "notify: handler_name" statement! +# +# Restart before reload: an reload after a restart may be redundant but should not fail, +# but the other way around may fail when the impact of changes was too large for a reload. +# +- name: 'Restart cadvisor service.' + systemd: + name: 'cadvisor.service' + state: restarted + daemon_reload: yes + become: true + listen: restart_cadvisor +... \ No newline at end of file diff --git a/roles/cadvisor/tasks/main.yml b/roles/cadvisor/tasks/main.yml index 982653a14..d1b60f052 100644 --- a/roles/cadvisor/tasks/main.yml +++ b/roles/cadvisor/tasks/main.yml @@ -1,25 +1,19 @@ --- -- name: Install service files. +- name: 'Install cadvisor service file.' template: - src: templates/cadvisor.service - dest: /etc/systemd/system/cadvisor.service - mode: 644 + src: 'templates/cadvisor.service' + dest: '/etc/systemd/system/cadvisor.service' + mode: '0644' owner: root group: root - tags: - - service-files + notify: restart_cadvisor + become: true -- name: install service files - command: systemctl daemon-reload - -- name: enable service at boot +- name: 'Make sure cadvisor service is enabled and started.' systemd: - name: cadvisor.service + name: 'cadvisor.service' + state: started enabled: yes - -- name: make sure servcies are started. - systemd: - name: cadvisor.service - state: restarted - tags: - - start-service + daemon_reload: yes + become: true +... \ No newline at end of file diff --git a/roles/cluster/tasks/build_lustre_client.yml b/roles/cluster/tasks/build_lustre_client.yml index 7dda01262..fbe7f6861 100644 --- a/roles/cluster/tasks/build_lustre_client.yml +++ b/roles/cluster/tasks/build_lustre_client.yml @@ -1,9 +1,11 @@ --- - -- name: Fetch the lustre client source +- name: 'Fetch the Lustre client source code.' get_url: - url: https://downloads.whamcloud.com/public/lustre/lustre-2.11.0/el7.4.1708/client/SRPMS/lustre-client-dkms-2.11.0-1.el7.src.rpm - dest: /tmp/lustre-client-dkms-2.11.0-1.el7.src.rpm + url: 'https://downloads.whamcloud.com/public/lustre/lustre-2.11.0/el7.4.1708/client/SRPMS/lustre-client-dkms-2.11.0-1.el7.src.rpm' + dest: '/tmp/lustre-client-dkms-2.11.0-1.el7.src.rpm' -- name: build the lustre client. - command: rpmbuild --rebuild --without servers /tmp/lustre-client-dkms-2.11.0-1.el7.src.rpm +- name: 'Build the Lustre client.' + command: + cmd: 'rpmbuild --rebuild --without servers /tmp/lustre-client-dkms-2.11.0-1.el7.src.rpm' + creates: '/tmp/lustre-client-dkms-2.11.0-1.el7.src.rpm.rebuild' +... \ No newline at end of file diff --git a/roles/cluster/tasks/main.yml b/roles/cluster/tasks/main.yml index 46af6253d..306a97603 100644 --- a/roles/cluster/tasks/main.yml +++ b/roles/cluster/tasks/main.yml @@ -1,19 +1,91 @@ --- +- name: Set hostname to inventory_hostname + hostname: + name: "{{ inventory_hostname | regex_replace('^' + ai_jumphost + '\\+','') }}" + become: true + +- name: Set selinux in permissive mode + selinux: + policy: targeted + state: permissive + become: true + +- name: Install some standard software + yum: + state: latest + update_cache: yes + name: + - bash-completion + - bc + - bzip2 + - curl + - dos2unix + - figlet + - fuse + - fuse-libs + - git + - git-core + - gnutls + - lsof + - nano + - ncdu + - ncurses-static + - openssl + - qt5-qtbase + - qt5-qtxmlpatterns + - readline-static + - rsync + - screen + - singularity-runtime + - singularity + - tcl-devel + - telnet + - tmux + - tree + - unzip + - vim + - wget + tags: + - software + become: true + +- name: Check if rsync >= 3.1.2 is installed on the managed hosts. + shell: + cmd: | + set -o pipefail + rsync --version 2>&1 | head -n 1 | sed 's|^rsync *version *\([0-9\.]*\).*$|\1|' | tr -d '\n' + args: + warn: no + changed_when: false + failed_when: false + check_mode: no + register: rsync_version_managed_host + +- name: Abort when modern rsync >= 3.1.2 is missing on the managed hosts. + debug: + msg: "FATAL: Need rsync >= 3.1.2 on {{ inventory_hostname }}, but detected {{ rsync_version.stdout }}." + when: 'rsync_version_managed_host is failed or (rsync_version_managed_host.stdout is version_compare("3.1.2", operator="<"))' + failed_when: 'rsync_version_managed_host is failed or (rsync_version_managed_host.stdout is version_compare("3.1.2", operator="<"))' + - name: Check if rsync >= 3.1.2 is installed on the control host. - shell: | - rsync --version 2>&1 | head -n 1 | sed 's|^rsync *version *\([0-9\.]*\).*$|\1|' | tr -d '\n' + shell: + cmd: | + set -o pipefail + rsync --version 2>&1 | head -n 1 | sed 's|^rsync *version *\([0-9\.]*\).*$|\1|' | tr -d '\n' args: warn: no changed_when: false failed_when: false + ignore_errors: true check_mode: no - register: rsync_version + register: rsync_version_control_host + delegate_to: localhost - name: Abort when modern rsync >= 3.1.2 is missing on control host. debug: msg: "FATAL: Need rsync >= 3.1.2 on {{ inventory_hostname }}, but detected {{ rsync_version.stdout }}." - when: 'rsync_version is failed or (rsync_version.stdout is version_compare("3.1.2", operator="<"))' - failed_when: 'rsync_version is failed or (rsync_version.stdout is version_compare("3.1.2", operator="<"))' + when: 'rsync_version_control_host is failed or (rsync_version_control_host.stdout is version_compare("3.1.2", operator="<"))' + failed_when: 'rsync_version_control_host is failed or (rsync_version_control_host.stdout is version_compare("3.1.2", operator="<"))' delegate_to: localhost - name: Add custom config files to /etc/skel/. @@ -54,48 +126,4 @@ - src: '.ssh' mode: 'Du=rwx,Dgo-rwx,Fu=rw,Fgo-rwx' become: true - -- name: Set hostname to inventory_hostname - hostname: - name: "{{ inventory_hostname | regex_replace('^' + ai_jumphost + '\\+','') }}" - become: true - -- name: Set selinux in permissive mode - selinux: - policy: targeted - state: permissive - become: true - -- name: Install some standard software - yum: - state: latest - update_cache: yes - name: - - bc - - bzip2 - - curl - - dos2unix - - figlet - - git - - git-core - - gnutls - - lsof - - nano - - ncdu - - ncurses-static - - openssl - - readline-static - - screen - - tcl-devel - - telnet - - tmux - - tree - - unzip - - vim - - wget - - fuse - - fuse-libs - tags: - - software - become: true ... diff --git a/roles/coredumps/defaults/main.yml b/roles/coredumps/defaults/main.yml new file mode 100644 index 000000000..93a482168 --- /dev/null +++ b/roles/coredumps/defaults/main.yml @@ -0,0 +1,10 @@ +--- +# +# For disabling coredumps: +# +coredumps_pattern: '|/bin/false' +# +# For enabling coredumps: +# +#coredumps_pattern: 'core' +... \ No newline at end of file diff --git a/roles/coredumps/tasks/main.yml b/roles/coredumps/tasks/main.yml new file mode 100644 index 000000000..df8dffb10 --- /dev/null +++ b/roles/coredumps/tasks/main.yml @@ -0,0 +1,17 @@ +--- +- name: 'Install sysctl config file to manage coredumps upon next reboot.' + template: + src: 'templates/50-ansible-managed-coredumps.conf' + dest: '/etc/sysctl.d/50-ansible-managed-coredumps.conf' + owner: 'root' + group: 'root' + mode: '0644' + become: true + +- name: 'Modify sysctl kernel parameter to manage coredumps right now.' + sysctl: + name: kernel.core_pattern + value: "{{ coredumps_pattern }}" + sysctl_set: yes + become: true +... \ No newline at end of file diff --git a/roles/coredumps/templates/50-ansible-managed-coredumps.conf b/roles/coredumps/templates/50-ansible-managed-coredumps.conf new file mode 100644 index 000000000..801f742d5 --- /dev/null +++ b/roles/coredumps/templates/50-ansible-managed-coredumps.conf @@ -0,0 +1,14 @@ +# +# This file is deployed with the coredumps role from the Ansible playbook of the league-of-robots repo. +# DO NOT EDIT MANUALLY; update source and re-deploy instead! +# +# {{ ansible_managed }} +# +# Disable creating coredump files: These can be huge, we never use them anyway and +# writing relatively large files on a file system under heavy load may be so slow +# that it causes Slurm hit the UnkillableStepTimeout limit. +# This may happen when a job exceeds its requested memory and gets killed by the Out Of Memory (OOM) Killer, +# which is perfectly normal behaviour,and writing the coredump file is so slow that it takes the node offline +# with STATE=drained and REASON="Kill task failed", which is not what we want. +# +kernel.core_pattern = {{ coredumps_pattern }} diff --git a/roles/figlet_hostname/tasks/main.yml b/roles/figlet_hostname/tasks/main.yml deleted file mode 100644 index a86268870..000000000 --- a/roles/figlet_hostname/tasks/main.yml +++ /dev/null @@ -1,24 +0,0 @@ ---- -- name: Install figlet. - yum: - state: latest - update_cache: yes - name: - - figlet - become: true - -- name: Generate figlet ascii art. - # shell: figlet -f {{ role_path }}/files/doh.flf -w 180 {{ ansible_hostname }} - shell: | - figlet {{ ansible_hostname }} - register: figlet - -- name: Insert figlet in /etc/motd - template: - src: templates/motd - dest: /etc/motd - owner: root - group: root - mode: 0644 - become: true -... \ No newline at end of file diff --git a/roles/figlet_hostname/templates/motd b/roles/figlet_hostname/templates/motd deleted file mode 100644 index 87baac192..000000000 --- a/roles/figlet_hostname/templates/motd +++ /dev/null @@ -1,4 +0,0 @@ -{{ figlet.stdout }} - -{{ motd }} - diff --git a/roles/figlet_hostname/defaults/main.yml b/roles/figlet_motd/defaults/main.yml similarity index 71% rename from roles/figlet_hostname/defaults/main.yml rename to roles/figlet_motd/defaults/main.yml index 0c0ddad64..e00e0bf8c 100644 --- a/roles/figlet_hostname/defaults/main.yml +++ b/roles/figlet_motd/defaults/main.yml @@ -1,2 +1,4 @@ --- motd: Dear users, please be careful with this system. +figlet_font: speed +... diff --git a/roles/figlet_motd/files/cyberlarge.flf b/roles/figlet_motd/files/cyberlarge.flf new file mode 100644 index 000000000..62a126a3d --- /dev/null +++ b/roles/figlet_motd/files/cyberlarge.flf @@ -0,0 +1,429 @@ +flf2a$ 4 3 10 -1 20 +Cyberfont - large +Figlet conversion by Kent Nassen, kentn@cyberspace.org, 8-10-94 +From: stock@fwi.uva.nl (Lennert Stock) +Date: 15 Jul 1994 00:04:25 GMT + +Here are some fonts. Non-figlet I'm afraid, if you wanna convert them, be +my guest. I posted the isometric fonts before. + +------------------------------------------------------------------------------ + + .x%%%%%%x. .x%%%%%%x. + ,%%%%%%%%%%. .%%%%%%%%%%. + ,%%%' )' \) :( `( `%%%. + ,%x%)________) --------- L e n n e r t S t o c k ( _ __ (%x%. + (%%%~^88P~88P| |~=> .=-~ %%%) + (%%::. .:,\ .' `. /,:. .::%%) + `;%:`\. `-' | | `-' ./':%:' + ``x`. -===.' stock@fwi.uva.nl -------- `.===- .'x'' + / `:`.__.; :.__.':' \ + .d8b. ..`. .'.. .d8b. + $ $@ + $ $@ + $ $@ + $ $@@ + /@ + / @ + . @ + $@@ + ..@ + ''@ + @ + @@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ + .@ + '@ + @ + @@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ + $@ + $@ + .@ + '@@ + $$$@ + ___@ + $$$@ + $$$@@ + $@ + $@ + .@ + $@@ + /@ + / @ + / @ + / @@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ + $@ + .@ + .@ + $@@ + $@ + .@ + .@ + '@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ + ____ @ + ___|@ + \ @ + ` @@ +@ +@ +@ +@@ + _______@ + |_____|@ + | |@ + @@ + ______ @ + |_____]@ + |_____]@ + @@ + _______@ + | $@ + |_____$@ + @@ + ______$@ + | \@ + |_____/@ + @@ + _______@ + |______@ + |______@ + @@ + _______@ + |______@ + | $@ + @@ + ______@ + | ____@ + |_____|@ + @@ + _ _@ + |_____|@ + | |@ + @@ + _____@ + | $@ + __|__@ + @@ + _____@ + | $@ + __| $@ + @@ + _ _@ + |____/$@ + | \_@ + @@ + $ $@ + | $@ + |_____@ + @@ + _______@ + | | |@ + | | |@ + @@ + __ _@ + | \ |@ + | \_|@ + @@ + _____ @ + | |@ + |_____|@ + @@ + _____ @ + |_____]@ + | $@ + @@ + _____ @ + | __|@ + |____\|@ + @@ + ______@ + |_____/@ + | \_@ + @@ + _______@ + |______@ + ______|@ + @@ + _______@ + | $@ + | $@ + @@ + _ _@ + | |@ + |_____|@ + @@ + _ _@ + \ /$@ + \/ $@ + @@ + _ _ _@ + | | |@ + |__|__|@ + @@ + _ _@ + \___/$@ + _/ \_@ + @@ + __ __@ + \_/ $@ + | $@ + @@ + ______@ + ____/@ + /_____@ + @@ +@ +@ +@ +@@ + \ @ + \ @ + \ @ + \@@ +@ +@ +@ +@@ +@ +@ +@ +@@ + @ + @ + _____@ + @@ + .@ + `@ + @ + @@ + _______@ + |_____|@ + | |@ + @@ + ______ @ + |_____]@ + |_____]@ + @@ + _______@ + | $@ + |_____$@ + @@ + ______$@ + | \@ + |_____/@ + @@ + _______@ + |______@ + |______@ + @@ + _______@ + |______@ + | $@ + @@ + ______@ + | ____@ + |_____|@ + @@ + _ _@ + |_____|@ + | |@ + @@ + _____@ + | $@ + __|__@ + @@ + _____@ + | $@ + __| $@ + @@ + _ _@ + |____/$@ + | \_@ + @@ + $ $@ + | $@ + |_____@ + @@ + _______@ + | | |@ + | | |@ + @@ + __ _@ + | \ |@ + | \_|@ + @@ + _____ @ + | |@ + |_____|@ + @@ + _____ @ + |_____]@ + | $@ + @@ + _____ @ + | __|@ + |____\|@ + @@ + ______@ + |_____/@ + | \_@ + @@ + _______@ + |______@ + ______|@ + @@ + _______@ + | $@ + | $@ + @@ + _ _@ + | |@ + |_____|@ + @@ + _ _@ + \ /$@ + \/ $@ + @@ + _ _ _@ + | | |@ + |__|__|@ + @@ + _ _@ + \___/$@ + _/ \_@ + @@ + __ __@ + \_/ $@ + | $@ + @@ + ______@ + ____/@ + /_____@ + @@ +@ +@ +@ +@@ + |@ + |@ + |@ + |@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ +@ +@ +@ +@@ diff --git a/roles/figlet_hostname/files/doh.flf b/roles/figlet_motd/files/doh.flf similarity index 100% rename from roles/figlet_hostname/files/doh.flf rename to roles/figlet_motd/files/doh.flf diff --git a/roles/figlet_motd/files/ogre.flf b/roles/figlet_motd/files/ogre.flf new file mode 100644 index 000000000..e72446f81 --- /dev/null +++ b/roles/figlet_motd/files/ogre.flf @@ -0,0 +1,626 @@ +flf2a$ 6 5 20 15 13 +Standard by Glenn Chappell & Ian Chai 3/93 -- based on .sig of Frank Sheeran +Figlet release 2.0 -- August 5, 1993 + +Explanation of first line: +flf2 - "magic number" for file identification +a - should always be `a', for now +$ - the "hardblank" -- prints as a blank, but can't be smushed +6 - height of a character +5 - height of a character, not including descenders +20 - max line length (excluding comment lines) + a fudge factor +15 - default smushmode for this font (like "-m 15" on command line) +13 - number of comment lines + +$@ +$@ +$@ +$@ +$@ +$@@ + _ @ + / \@ + / /@ +/\_/ @ +\/ @ + @@ + _ _ @ +( | )@ + V V @ + @ + @ + @@ + _ _ @ + _| || |_ @ +|_ .. _|@ +|_ _|@ + |_||_| @ + @@ + _ @ + | | @ +/ __)@ +\__ \@ +( /@ + |_| @@ + _ __@ +(_)/ /@ + / / @ + / /_ @ +/_/(_)@ + @@ + ___ @ + ( _ ) @ + / _ \/\@ +| (_> <@ + \___/\/@ + @@ + _ @ +( )@ +|/ @ + @ + @ + @@ + __@ + / /@ +| | @ +| | @ +| | @ + \_\@@ +__ @ +\ \ @ + | |@ + | |@ + | |@ +/_/ @@ + @ +__/\__@ +\ /@ +/_ _\@ + \/ @ + @@ + @ + _ @ + _| |_ @ +|_ _|@ + |_| @ + @@ + @ + @ + @ + _ @ +( )@ +|/ @@ + @ + @ + _____ @ +|_____|@ + @ + @@ + @ + @ + @ + _ @ +(_)@ + @@ + __@ + / /@ + / / @ + / / @ +/_/ @ + @@ + ___ @ + / _ \ @ +| | | |@ +| |_| |@ + \___/ @ + @@ + _ @ +/ |@ +| |@ +| |@ +|_|@ + @@ + ____ @ +|___ \ @ + __) |@ + / __/ @ +|_____|@ + @@ + _____ @ +|___ / @ + |_ \ @ + ___) |@ +|____/ @ + @@ + _ _ @ +| || | @ +| || |_ @ +|__ _|@ + |_| @ + @@ + ____ @ +| ___| @ +|___ \ @ + ___) |@ +|____/ @ + @@ + __ @ + / /_ @ +| '_ \ @ +| (_) |@ + \___/ @ + @@ + _____ @ +|___ |@ + / / @ + / / @ + /_/ @ + @@ + ___ @ + ( _ ) @ + / _ \ @ +| (_) |@ + \___/ @ + @@ + ___ @ + / _ \ @ +| (_) |@ + \__, |@ + /_/ @ + @@ + @ + _ @ +(_)@ + _ @ +(_)@ + @@ + @ + _ @ +(_)@ + _ @ +( )@ +|/ @@ + __@ + / /@ +/ / @ +\ \ @ + \_\@ + @@ + @ + _____ @ +|_____|@ +|_____|@ + @ + @@ +__ @ +\ \ @ + \ \@ + / /@ +/_/ @ + @@ + ___ @ +/ _ \@ +\// /@ + \/ @ + () @ + @@ + ____ @ + / __ \ @ + / / _` |@ +| | (_| |@ + \ \__,_|@ + \____/ @@ + _ @ + /_\ @ + //_\\ @ +/ _ \@ +\_/ \_/@ + @@ + ___ @ + / __\@ + /__\//@ +/ \/ \@ +\_____/@ + @@ + ___ @ + / __\@ + / / @ +/ /___ @ +\____/ @ + @@ + ___ @ + / \@ + / /\ /@ + / /_// @ +/___,' @ + @@ + __ @ + /__\@ + /_\ @ +//__ @ +\__/ @ + @@ + ___ @ + / __\@ + / _\ @ +/ / @ +\/ @ + @@ + ___ @ + / _ \@ + / /_\/@ +/ /_\\ @ +\____/ @ + @@ + @ + /\ /\@ + / /_/ /@ +/ __ / @ +\/ /_/ @ + @@ + _____ @ + \_ \@ + / /\/@ +/\/ /_ @ +\____/ @ + @@ + __ @ + \ \ @ + \ \@ + /\_/ /@ + \___/ @ + @@ + @ + /\ /\@ + / //_/@ +/ __ \ @ +\/ \/ @ + @@ + __ @ + / / @ + / / @ +/ /___ @ +\____/ @ + @@ + @ + /\/\ @ + / \ @ +/ /\/\ \@ +\/ \/@ + @@ + __ @ + /\ \ \@ + / \/ /@ +/ /\ / @ +\_\ \/ @ + @@ + ___ @ + /___\@ + // //@ +/ \_// @ +\___/ @ + @@ + ___ @ + / _ \@ + / /_)/@ +/ ___/ @ +\/ @ + @@ + ____ @ + /___ \@ + // / /@ +/ \_/ / @ +\___,_\ @ + @@ + __ @ + /__\ @ + / \// @ +/ _ \ @ +\/ \_/ @ + @@ + __ @ +/ _\ @ +\ \ @ +_\ \ @ +\__/ @ + @@ + _____ @ +/__ \@ + / /\/@ + / / @ + \/ @ + @@ + @ + /\ /\ @ +/ / \ \@ +\ \_/ /@ + \___/ @ + @@ + @ + /\ /\ @ + \ \ / / @ + \ V / @ + \_/ @ + @@ + __ __ @ +/ / /\ \ \@ +\ \/ \/ /@ + \ /\ / @ + \/ \/ @ + @@ +__ __@ +\ \/ /@ + \ / @ + / \ @ +/_/\_\@ + @@ + @ +/\_/\@ +\_ _/@ + / \ @ + \_/ @ + @@ + _____@ +/ _ /@ +\// / @ + / //\@ +/____/@ + @@ + __ @ +| _|@ +| | @ +| | @ +| | @ +|__|@@ +__ @ +\ \ @ + \ \ @ + \ \ @ + \_\@ + @@ + __ @ +|_ |@ + | |@ + | |@ + | |@ +|__|@@ + @ + /\ @ +|/\|@ + @ + @ + @@ + @ + @ + @ + @ + _____ @ +|_____|@@ + _ @ +( )@ + \|@ + @ + @ + @@ + @ + __ _ @ + / _` |@ +| (_| |@ + \__,_|@ + @@ + _ @ +| |__ @ +| '_ \ @ +| |_) |@ +|_.__/ @ + @@ + @ + ___ @ + / __|@ +| (__ @ + \___|@ + @@ + _ @ + __| |@ + / _` |@ +| (_| |@ + \__,_|@ + @@ + @ + ___ @ + / _ \@ +| __/@ + \___|@ + @@ + __ @ + / _|@ +| |_ @ +| _|@ +|_| @ + @@ + @ + __ _ @ + / _` |@ +| (_| |@ + \__, |@ + |___/ @@ + _ @ +| |__ @ +| '_ \ @ +| | | |@ +|_| |_|@ + @@ + _ @ +(_)@ +| |@ +| |@ +|_|@ + @@ + _ @ + (_)@ + | |@ + | |@ + _/ |@ +|__/ @@ + _ @ +| | __@ +| |/ /@ +| < @ +|_|\_\@ + @@ + _ @ +| |@ +| |@ +| |@ +|_|@ + @@ + @ + _ __ ___ @ +| '_ ` _ \ @ +| | | | | |@ +|_| |_| |_|@ + @@ + @ + _ __ @ +| '_ \ @ +| | | |@ +|_| |_|@ + @@ + @ + ___ @ + / _ \ @ +| (_) |@ + \___/ @ + @@ + @ + _ __ @ +| '_ \ @ +| |_) |@ +| .__/ @ +|_| @@ + @ + __ _ @ + / _` |@ +| (_| |@ + \__, |@ + |_|@@ + @ + _ __ @ +| '__|@ +| | @ +|_| @ + @@ + @ + ___ @ +/ __|@ +\__ \@ +|___/@ + @@ + _ @ +| |_ @ +| __|@ +| |_ @ + \__|@ + @@ + @ + _ _ @ +| | | |@ +| |_| |@ + \__,_|@ + @@ + @ +__ __@ +\ \ / /@ + \ V / @ + \_/ @ + @@ + @ +__ __@ +\ \ /\ / /@ + \ V V / @ + \_/\_/ @ + @@ + @ +__ __@ +\ \/ /@ + > < @ +/_/\_\@ + @@ + @ + _ _ @ +| | | |@ +| |_| |@ + \__, |@ + |___/ @@ + @ + ____@ +|_ /@ + / / @ +/___|@ + @@ + __@ + / /@ + | | @ +< < @ + | | @ + \_\@@ + _ @ +| |@ +| |@ +| |@ +| |@ +|_|@@ +__ @ +\ \ @ + | | @ + > >@ + | | @ +/_/ @@ + @ + /\/|@ +|/\/ @ + @ + @ + @@ + _ _ @ +(_)_(_)@ + / _ \ @ +| _ |@ +|_| |_|@ + @@ + _ _ @ +(_)_(_)@ + / _ \ @ +| |_| |@ + \___/ @ + @@ + _ _ @ +(_) (_)@ +| | | |@ +| |_| |@ + \___/ @ + @@ + _ _ @ +(_)_(_)@ + / _` |@ +| (_| |@ + \__,_|@ + @@ + _ _ @ +(_)_(_)@ + / _ \ @ +| (_) |@ + \___/ @ + @@ + _ _ @ +(_) (_)@ +| | | |@ +| |_| |@ + \__,_|@ + @@ + ____ @ +| __ \@ +| |/ /@ +| |\ \@ +|_||_/@ + @@ diff --git a/roles/figlet_motd/files/slant.flf b/roles/figlet_motd/files/slant.flf new file mode 100644 index 000000000..ea27b6eda --- /dev/null +++ b/roles/figlet_motd/files/slant.flf @@ -0,0 +1,1295 @@ +flf2a$ 6 5 16 15 10 0 18319 +Slant by Glenn Chappell 3/93 -- based on Standard +Includes ISO Latin-1 +figlet release 2.1 -- 12 Aug 1994 +Permission is hereby given to modify this font, as long as the +modifier's name is placed on a comment line. + +Modified by Paul Burton 12/96 to include new parameter +supported by FIGlet and FIGWin. May also be slightly modified for better use +of new full-width/kern/smush alternatives, but default output is NOT changed. + + $$@ + $$ @ + $$ @ + $$ @ + $$ @ +$$ @@ + __@ + / /@ + / / @ + /_/ @ +(_) @ + @@ + _ _ @ +( | )@ +|/|/ @ + $ @ +$ @ + @@ + __ __ @ + __/ // /_@ + /_ _ __/@ +/_ _ __/ @ + /_//_/ @ + @@ + __@ + _/ /@ + / __/@ + (_ ) @ +/ _/ @ +/_/ @@ + _ __@ + (_)_/_/@ + _/_/ @ + _/_/_ @ +/_/ (_) @ + @@ + ___ @ + ( _ ) @ + / __ \/|@ +/ /_/ < @ +\____/\/ @ + @@ + _ @ + ( )@ + |/ @ + $ @ +$ @ + @@ + __@ + _/_/@ + / / @ + / / @ +/ / @ +|_| @@ + _ @ + | |@ + / /@ + / / @ + _/_/ @ +/_/ @@ + @ + __/|_@ + | /@ +/_ __| @ + |/ @ + @@ + @ + __ @ + __/ /_@ +/_ __/@ + /_/ @ + @@ + @ + @ + @ + _ @ +( )@ +|/ @@ + @ + @ + ______@ +/_____/@ + $ @ + @@ + @ + @ + @ + _ @ +(_)@ + @@ + __@ + _/_/@ + _/_/ @ + _/_/ @ +/_/ @ + @@ + ____ @ + / __ \@ + / / / /@ +/ /_/ / @ +\____/ @ + @@ + ___@ + < /@ + / / @ + / / @ +/_/ @ + @@ + ___ @ + |__ \@ + __/ /@ + / __/ @ +/____/ @ + @@ + _____@ + |__ /@ + /_ < @ + ___/ / @ +/____/ @ + @@ + __ __@ + / // /@ + / // /_@ +/__ __/@ + /_/ @ + @@ + ______@ + / ____/@ + /___ \ @ + ____/ / @ +/_____/ @ + @@ + _____@ + / ___/@ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ + _____@ +/__ /@ + / / @ + / / @ +/_/ @ + @@ + ____ @ + ( __ )@ + / __ |@ +/ /_/ / @ +\____/ @ + @@ + ____ @ + / __ \@ + / /_/ /@ + \__, / @ +/____/ @ + @@ + @ + _ @ + (_)@ + _ @ +(_) @ + @@ + @ + _ @ + (_)@ + _ @ +( ) @ +|/ @@ + __@ + / /@ +/ / @ +\ \ @ + \_\@ + @@ + @ + _____@ + /____/@ +/____/ @ + $ @ + @@ +__ @ +\ \ @ + \ \@ + / /@ +/_/ @ + @@ + ___ @ + /__ \@ + / _/@ + /_/ @ +(_) @ + @@ + ______ @ + / ____ \@ + / / __ `/@ +/ / /_/ / @ +\ \__,_/ @ + \____/ @@ + ___ @ + / |@ + / /| |@ + / ___ |@ +/_/ |_|@ + @@ + ____ @ + / __ )@ + / __ |@ + / /_/ / @ +/_____/ @ + @@ + ______@ + / ____/@ + / / @ +/ /___ @ +\____/ @ + @@ + ____ @ + / __ \@ + / / / /@ + / /_/ / @ +/_____/ @ + @@ + ______@ + / ____/@ + / __/ @ + / /___ @ +/_____/ @ + @@ + ______@ + / ____/@ + / /_ @ + / __/ @ +/_/ @ + @@ + ______@ + / ____/@ + / / __ @ +/ /_/ / @ +\____/ @ + @@ + __ __@ + / / / /@ + / /_/ / @ + / __ / @ +/_/ /_/ @ + @@ + ____@ + / _/@ + / / @ + _/ / @ +/___/ @ + @@ + __@ + / /@ + __ / / @ +/ /_/ / @ +\____/ @ + @@ + __ __@ + / //_/@ + / ,< @ + / /| | @ +/_/ |_| @ + @@ + __ @ + / / @ + / / @ + / /___@ +/_____/@ + @@ + __ ___@ + / |/ /@ + / /|_/ / @ + / / / / @ +/_/ /_/ @ + @@ + _ __@ + / | / /@ + / |/ / @ + / /| / @ +/_/ |_/ @ + @@ + ____ @ + / __ \@ + / / / /@ +/ /_/ / @ +\____/ @ + @@ + ____ @ + / __ \@ + / /_/ /@ + / ____/ @ +/_/ @ + @@ + ____ @ + / __ \@ + / / / /@ +/ /_/ / @ +\___\_\ @ + @@ + ____ @ + / __ \@ + / /_/ /@ + / _, _/ @ +/_/ |_| @ + @@ + _____@ + / ___/@ + \__ \ @ + ___/ / @ +/____/ @ + @@ + ______@ + /_ __/@ + / / @ + / / @ +/_/ @ + @@ + __ __@ + / / / /@ + / / / / @ +/ /_/ / @ +\____/ @ + @@ + _ __@ +| | / /@ +| | / / @ +| |/ / @ +|___/ @ + @@ + _ __@ +| | / /@ +| | /| / / @ +| |/ |/ / @ +|__/|__/ @ + @@ + _ __@ + | |/ /@ + | / @ + / | @ +/_/|_| @ + @@ +__ __@ +\ \/ /@ + \ / @ + / / @ +/_/ @ + @@ + _____@ +/__ /@ + / / @ + / /__@ +/____/@ + @@ + ___@ + / _/@ + / / @ + / / @ + / / @ +/__/ @@ +__ @ +\ \ @ + \ \ @ + \ \ @ + \_\@ + @@ + ___@ + / /@ + / / @ + / / @ + _/ / @ +/__/ @@ + //|@ + |/||@ + $ @ + $ @ +$ @ + @@ + @ + @ + @ + @ + ______@ +/_____/@@ + _ @ + ( )@ + V @ + $ @ +$ @ + @@ + @ + ____ _@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ + __ @ + / /_ @ + / __ \@ + / /_/ /@ +/_.___/ @ + @@ + @ + _____@ + / ___/@ +/ /__ @ +\___/ @ + @@ + __@ + ____/ /@ + / __ / @ +/ /_/ / @ +\__,_/ @ + @@ + @ + ___ @ + / _ \@ +/ __/@ +\___/ @ + @@ + ____@ + / __/@ + / /_ @ + / __/ @ +/_/ @ + @@ + @ + ____ _@ + / __ `/@ + / /_/ / @ + \__, / @ +/____/ @@ + __ @ + / /_ @ + / __ \@ + / / / /@ +/_/ /_/ @ + @@ + _ @ + (_)@ + / / @ + / / @ +/_/ @ + @@ + _ @ + (_)@ + / / @ + / / @ + __/ / @ +/___/ @@ + __ @ + / /__@ + / //_/@ + / ,< @ +/_/|_| @ + @@ + __@ + / /@ + / / @ + / / @ +/_/ @ + @@ + @ + ____ ___ @ + / __ `__ \@ + / / / / / /@ +/_/ /_/ /_/ @ + @@ + @ + ____ @ + / __ \@ + / / / /@ +/_/ /_/ @ + @@ + @ + ____ @ + / __ \@ +/ /_/ /@ +\____/ @ + @@ + @ + ____ @ + / __ \@ + / /_/ /@ + / .___/ @ +/_/ @@ + @ + ____ _@ + / __ `/@ +/ /_/ / @ +\__, / @ + /_/ @@ + @ + _____@ + / ___/@ + / / @ +/_/ @ + @@ + @ + _____@ + / ___/@ + (__ ) @ +/____/ @ + @@ + __ @ + / /_@ + / __/@ +/ /_ @ +\__/ @ + @@ + @ + __ __@ + / / / /@ +/ /_/ / @ +\__,_/ @ + @@ + @ + _ __@ +| | / /@ +| |/ / @ +|___/ @ + @@ + @ + _ __@ +| | /| / /@ +| |/ |/ / @ +|__/|__/ @ + @@ + @ + _ __@ + | |/_/@ + _> < @ +/_/|_| @ + @@ + @ + __ __@ + / / / /@ + / /_/ / @ + \__, / @ +/____/ @@ + @ + ____@ +/_ /@ + / /_@ +/___/@ + @@ + __@ + _/_/@ + _/_/ @ +< < @ +/ / @ +\_\ @@ + __@ + / /@ + / / @ + / / @ + / / @ +/_/ @@ + _ @ + | |@ + / /@ + _>_>@ + _/_/ @ +/_/ @@ + /\//@ + //\/ @ + $ @ + $ @ +$ @ + @@ + _ _ @ + (_)(_)@ + / _ | @ + / __ | @ +/_/ |_| @ + @@ + _ _ @ + (_)_(_)@ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ + _ _ @ + (_) (_)@ + / / / / @ +/ /_/ / @ +\____/ @ + @@ + _ _ @ + (_)_(_)@ + / __ `/ @ +/ /_/ / @ +\__,_/ @ + @@ + _ _ @ + (_)_(_)@ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ + _ _ @ + (_) (_)@ + / / / / @ +/ /_/ / @ +\__,_/ @ + @@ + ____ @ + / __ \@ + / / / /@ + / /_| | @ + / //__/ @ +/_/ @@ +160 NO-BREAK SPACE + $$@ + $$ @ + $$ @ + $$ @ + $$ @ +$$ @@ +161 INVERTED EXCLAMATION MARK + _ @ + (_)@ + / / @ + / / @ +/_/ @ + @@ +162 CENT SIGN + __@ + __/ /@ + / ___/@ +/ /__ @ +\ _/ @ +/_/ @@ +163 POUND SIGN + ____ @ + / ,__\@ + __/ /_ @ + _/ /___ @ +(_,____/ @ + @@ +164 CURRENCY SIGN + /|___/|@ + | __ / @ + / /_/ / @ + /___ | @ +|/ |/ @ + @@ +165 YEN SIGN + ____@ + _| / /@ + /_ __/@ +/_ __/ @ + /_/ @ + @@ +166 BROKEN BAR + __@ + / /@ + /_/ @ + __ @ + / / @ +/_/ @@ +167 SECTION SIGN + __ @ + _/ _)@ + / | | @ + | || | @ + | |_/ @ +(__/ @@ +168 DIAERESIS + _ _ @ + (_) (_)@ + $ $ @ + $ $ @ +$ $ @ + @@ +169 COPYRIGHT SIGN + ______ @ + / _____\ @ + / / ___/ |@ + / / /__ / @ +| \___/ / @ + \______/ @@ +170 FEMININE ORDINAL INDICATOR + ___ _@ + / _ `/@ + _\_,_/ @ +/____/ @ + $ @ + @@ +171 LEFT-POINTING DOUBLE ANGLE QUOTATION MARK + ____@ + / / /@ +/ / / @ +\ \ \ @ + \_\_\@ + @@ +172 NOT SIGN + @ + ______@ +/___ /@ + /_/ @ + $ @ + @@ +173 SOFT HYPHEN + @ + @ + _____@ +/____/@ + $ @ + @@ +174 REGISTERED SIGN + ______ @ + / ___ \ @ + / / _ \ |@ + / / , _/ / @ +| /_/|_| / @ + \______/ @@ +175 MACRON + ______@ +/_____/@ + $ @ + $ @ +$ @ + @@ +176 DEGREE SIGN + ___ @ + / _ \@ +/ // /@ +\___/ @ + $ @ + @@ +177 PLUS-MINUS SIGN + __ @ + __/ /_@ + /_ __/@ + __/_/_ @ +/_____/ @ + @@ +178 SUPERSCRIPT TWO + ___ @ + |_ |@ + / __/ @ +/____/ @ + $ @ + @@ +179 SUPERSCRIPT THREE + ____@ + |_ /@ + _/_ < @ +/____/ @ + $ @ + @@ +180 ACUTE ACCENT + __@ + /_/@ + $ @ + $ @ +$ @ + @@ +181 MICRO SIGN + @ + __ __@ + / / / /@ + / /_/ / @ + / ._,_/ @ +/_/ @@ +182 PILCROW SIGN + _______@ + / _ /@ +/ (/ / / @ +\_ / / @ + /_/_/ @ + @@ +183 MIDDLE DOT + @ + _ @ +(_)@ + $ @ +$ @ + @@ +184 CEDILLA + @ + @ + @ + @ + _ @ +/_)@@ +185 SUPERSCRIPT ONE + ___@ + < /@ + / / @ +/_/ @ +$ @ + @@ +186 MASCULINE ORDINAL INDICATOR + ___ @ + / _ \@ + _\___/@ +/____/ @ + $ @ + @@ +187 RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK +____ @ +\ \ \ @ + \ \ \@ + / / /@ +/_/_/ @ + @@ +188 VULGAR FRACTION ONE QUARTER + ___ __ @ + < / _/_/ @ + / /_/_/___@ +/_//_// / /@ + /_/ /_ _/@ + /_/ @@ +189 VULGAR FRACTION ONE HALF + ___ __ @ + < / _/_/__ @ + / /_/_/|_ |@ +/_//_/ / __/ @ + /_/ /____/ @ + @@ +190 VULGAR FRACTION THREE QUARTERS + ____ __ @ + |_ / _/_/ @ + _/_ < _/_/___@ +/____//_// / /@ + /_/ /_ _/@ + /_/ @@ +191 INVERTED QUESTION MARK + _ @ + (_)@ + _/ / @ +/ _/_ @ +\___/ @ + @@ +192 LATIN CAPITAL LETTER A WITH GRAVE + __ @ + _\_\@ + / _ |@ + / __ |@ +/_/ |_|@ + @@ +193 LATIN CAPITAL LETTER A WITH ACUTE + __@ + _/_/@ + / _ |@ + / __ |@ +/_/ |_|@ + @@ +194 LATIN CAPITAL LETTER A WITH CIRCUMFLEX + //|@ + _|/||@ + / _ | @ + / __ | @ +/_/ |_| @ + @@ +195 LATIN CAPITAL LETTER A WITH TILDE + /\//@ + _//\/ @ + / _ | @ + / __ | @ +/_/ |_| @ + @@ +196 LATIN CAPITAL LETTER A WITH DIAERESIS + _ _ @ + (_)(_)@ + / _ | @ + / __ | @ +/_/ |_| @ + @@ +197 LATIN CAPITAL LETTER A WITH RING ABOVE + (())@ + / |@ + / /| |@ + / ___ |@ +/_/ |_|@ + @@ +198 LATIN CAPITAL LETTER AE + __________@ + / ____/@ + / /| __/ @ + / __ /___ @ +/_/ /_____/ @ + @@ +199 LATIN CAPITAL LETTER C WITH CEDILLA + ______@ + / ____/@ + / / @ +/ /___ @ +\____/ @ + /_) @@ +200 LATIN CAPITAL LETTER E WITH GRAVE + __ @ + _\_\@ + / __/@ + / _/ @ +/___/ @ + @@ +201 LATIN CAPITAL LETTER E WITH ACUTE + __@ + _/_/@ + / __/@ + / _/ @ +/___/ @ + @@ +202 LATIN CAPITAL LETTER E WITH CIRCUMFLEX + //|@ + _|/||@ + / __/ @ + / _/ @ +/___/ @ + @@ +203 LATIN CAPITAL LETTER E WITH DIAERESIS + _ _ @ + (_)(_)@ + / __/ @ + / _/ @ +/___/ @ + @@ +204 LATIN CAPITAL LETTER I WITH GRAVE + __ @ + _\_\@ + / _/@ + _/ / @ +/___/ @ + @@ +205 LATIN CAPITAL LETTER I WITH ACUTE + __@ + _/_/@ + / _/@ + _/ / @ +/___/ @ + @@ +206 LATIN CAPITAL LETTER I WITH CIRCUMFLEX + //|@ + _|/||@ + / _/ @ + _/ / @ +/___/ @ + @@ +207 LATIN CAPITAL LETTER I WITH DIAERESIS + _ _ @ + (_)(_)@ + / _/ @ + _/ / @ +/___/ @ + @@ +208 LATIN CAPITAL LETTER ETH + ____ @ + / __ \@ + __/ /_/ /@ +/_ __/ / @ + /_____/ @ + @@ +209 LATIN CAPITAL LETTER N WITH TILDE + /\//@ + _//\/ @ + / |/ / @ + / / @ +/_/|_/ @ + @@ +210 LATIN CAPITAL LETTER O WITH GRAVE + __ @ + __\_\@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +211 LATIN CAPITAL LETTER O WITH ACUTE + __@ + __/_/@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +212 LATIN CAPITAL LETTER O WITH CIRCUMFLEX + //|@ + _|/||@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +213 LATIN CAPITAL LETTER O WITH TILDE + /\//@ + _//\/ @ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ +214 LATIN CAPITAL LETTER O WITH DIAERESIS + _ _ @ + (_)_(_)@ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ +215 MULTIPLICATION SIGN + @ + @ + /|/|@ + > < @ +|/|/ @ + @@ +216 LATIN CAPITAL LETTER O WITH STROKE + _____ @ + / _// \@ + / //// /@ +/ //// / @ +\_//__/ @ + @@ +217 LATIN CAPITAL LETTER U WITH GRAVE + __ @ + __\_\_@ + / / / /@ +/ /_/ / @ +\____/ @ + @@ +218 LATIN CAPITAL LETTER U WITH ACUTE + __ @ + __/_/_@ + / / / /@ +/ /_/ / @ +\____/ @ + @@ +219 LATIN CAPITAL LETTER U WITH CIRCUMFLEX + //| @ + _|/||_@ + / / / /@ +/ /_/ / @ +\____/ @ + @@ +220 LATIN CAPITAL LETTER U WITH DIAERESIS + _ _ @ + (_) (_)@ + / / / / @ +/ /_/ / @ +\____/ @ + @@ +221 LATIN CAPITAL LETTER Y WITH ACUTE + __ @ +__/_/_@ +\ \/ /@ + \ / @ + /_/ @ + @@ +222 LATIN CAPITAL LETTER THORN + __ @ + / /_ @ + / __ \@ + / ____/@ +/_/ @ + @@ +223 LATIN SMALL LETTER SHARP S + ____ @ + / __ \@ + / / / /@ + / /_| | @ + / //__/ @ +/_/ @@ +224 LATIN SMALL LETTER A WITH GRAVE + __ @ + __\_\_@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +225 LATIN SMALL LETTER A WITH ACUTE + __ @ + __/_/_@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +226 LATIN SMALL LETTER A WITH CIRCUMFLEX + //| @ + _|/||_@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +227 LATIN SMALL LETTER A WITH TILDE + /\//@ + _//\/_@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +228 LATIN SMALL LETTER A WITH DIAERESIS + _ _ @ + (_)_(_)@ + / __ `/ @ +/ /_/ / @ +\__,_/ @ + @@ +229 LATIN SMALL LETTER A WITH RING ABOVE + __ @ + __(())@ + / __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +230 LATIN SMALL LETTER AE + @ + ____ ___ @ + / __ ` _ \@ +/ /_/ __/@ +\__,_____/ @ + @@ +231 LATIN SMALL LETTER C WITH CEDILLA + @ + _____@ + / ___/@ +/ /__ @ +\___/ @ +/_) @@ +232 LATIN SMALL LETTER E WITH GRAVE + __ @ + _\_\@ + / _ \@ +/ __/@ +\___/ @ + @@ +233 LATIN SMALL LETTER E WITH ACUTE + __@ + _/_/@ + / _ \@ +/ __/@ +\___/ @ + @@ +234 LATIN SMALL LETTER E WITH CIRCUMFLEX + //|@ + _|/||@ + / _ \ @ +/ __/ @ +\___/ @ + @@ +235 LATIN SMALL LETTER E WITH DIAERESIS + _ _ @ + (_)(_)@ + / _ \ @ +/ __/ @ +\___/ @ + @@ +236 LATIN SMALL LETTER I WITH GRAVE + __ @ + \_\@ + / / @ + / / @ +/_/ @ + @@ +237 LATIN SMALL LETTER I WITH ACUTE + __@ + /_/@ + / / @ + / / @ +/_/ @ + @@ +238 LATIN SMALL LETTER I WITH CIRCUMFLEX + //|@ + |/||@ + / / @ + / / @ +/_/ @ + @@ +239 LATIN SMALL LETTER I WITH DIAERESIS + _ _ @ + (_)_(_)@ + / / @ + / / @ +/_/ @ + @@ +240 LATIN SMALL LETTER ETH + || @ + =||=@ + ___ || @ +/ __` | @ +\____/ @ + @@ +241 LATIN SMALL LETTER N WITH TILDE + /\//@ + _//\/ @ + / __ \ @ + / / / / @ +/_/ /_/ @ + @@ +242 LATIN SMALL LETTER O WITH GRAVE + __ @ + __\_\@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +243 LATIN SMALL LETTER O WITH ACUTE + __@ + __/_/@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +244 LATIN SMALL LETTER O WITH CIRCUMFLEX + //|@ + _|/||@ + / __ \@ +/ /_/ /@ +\____/ @ + @@ +245 LATIN SMALL LETTER O WITH TILDE + /\//@ + _//\/ @ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ +246 LATIN SMALL LETTER O WITH DIAERESIS + _ _ @ + (_)_(_)@ + / __ \ @ +/ /_/ / @ +\____/ @ + @@ +247 DIVISION SIGN + @ + _ @ + __(_)_@ +/_____/@ + (_) @ + @@ +248 LATIN SMALL LETTER O WITH STROKE + @ + _____ @ + / _// \@ +/ //// /@ +\_//__/ @ + @@ +249 LATIN SMALL LETTER U WITH GRAVE + __ @ + __\_\_@ + / / / /@ +/ /_/ / @ +\__,_/ @ + @@ +250 LATIN SMALL LETTER U WITH ACUTE + __ @ + __/_/_@ + / / / /@ +/ /_/ / @ +\__,_/ @ + @@ +251 LATIN SMALL LETTER U WITH CIRCUMFLEX + //| @ + _|/||_@ + / / / /@ +/ /_/ / @ +\__,_/ @ + @@ +252 LATIN SMALL LETTER U WITH DIAERESIS + _ _ @ + (_) (_)@ + / / / / @ +/ /_/ / @ +\__,_/ @ + @@ +253 LATIN SMALL LETTER Y WITH ACUTE + __ @ + __/_/_@ + / / / /@ + / /_/ / @ + \__, / @ +/____/ @@ +254 LATIN SMALL LETTER THORN + __ @ + / /_ @ + / __ \@ + / /_/ /@ + / .___/ @ +/_/ @@ +255 LATIN SMALL LETTER Y WITH DIAERESIS + _ _ @ + (_) (_)@ + / / / / @ + / /_/ / @ + \__, / @ +/____/ @@ diff --git a/roles/figlet_motd/files/speed.flf b/roles/figlet_motd/files/speed.flf new file mode 100644 index 000000000..d294c52ff --- /dev/null +++ b/roles/figlet_motd/files/speed.flf @@ -0,0 +1,1301 @@ +flf2a$ 6 5 16 15 16 +Speed by Claude Martins 2/95 -- based on Slant +Includes ISO Latin-1 +figlet release 2.1 -- 12 Aug 1994 +Permission is hereby given to modify this font, as long as the +modifier's name is placed on a comment line. + +Explanation of first line: +flf2 - "magic number" for file identification +a - should always be `a', for now +$ - the "hardblank" -- prints as a blank, but can't be smushed +6 - height of a character +5 - height of a character, not including descenders +14 - max line length (excluding comment lines) + a fudge factor +15 - default smushmode for this font +16 - number of comment lines + + $$@ + $$ @ + $$ @ + $$ @ + $$ @ +$$ @@ +______@ +___ /@ +__ / @ + /_/ @ +(_) @ + @@ +___ _ @ +_( | )@ +_|/|/ @ + $ @ + $ @ + @@ +_______ __ @ +____/ // /_@ +_ _ _ __/@ +/_ _ __/ @ + /_//_/ @ + @@ +_______@ +____/ /@ +__ __/@ +_(_ ) @ +/ _/ @ +/_/ @@ +____ __@ +__(_)_/_/@ +____/_/ @ +__/_/_ @ +/_/ (_) @ + @@ +______ @ +__( _ ) @ +_ __ \/|@ +/ /_/ < @ +\____/\/ @ + @@ +___ @ +_( )@ +_|/ @ + $ @ +$ @ + @@ +_______@ +____/_/@ +__ / @ +_ / @ +/ / @ +|_| @@ +______ @ +____| |@ +____ /@ +___ / @ +__/_/ @ +/_/ @@ +_____ @ +____/|_@ +_| /@ +/_ __| @ + |/ @ + @@ + @ +______ @ +___/ /_@ +/_ __/@ + /_/ @ + @@ + @ + @ + @ +___ @ +_( )@ +_|/ @@ + @ + @ +________@ +_/_____/@ + $ @ + @@ + @ + @ + @ +___ @ +_(_)@ + @@ +_________@ +______/_/@ +____/_/ @ +__/_/ @ +/_/ @ + @@ +_______ @ +__ __ \@ +_ / / /@ +/ /_/ / @ +\____/ @ + @@ +______@ +__< /@ +__ / @ +_ / @ +/_/ @ + @@ +______ @ +__|__ \@ +____/ /@ +_ __/ @ +/____/ @ + @@ +________@ +__|__ /@ +___/_ < @ +____/ / @ +/____/ @ + @@ +_____ __@ +__ // /@ +_ // /_@ +/__ __/@ + /_/ @ + @@ +__________@ +___ ____/@ +______ \ @ + ____/ / @ +/_____/ @ + @@ +________@ +__ ___/@ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +______@ +/__ /@ +__ / @ +_ / @ +/_/ @ + @@ +_______ @ +__( __ )@ +_ __ |@ +/ /_/ / @ +\____/ @ + @@ +_______ @ +__ __ \@ +_ /_/ /@ +_\__, / @ +/____/ @ + @@ + @ +_____ @ +___(_)@ +___ @ +_(_) @ + @@ + @ +_____ @ +___(_)@ +___ @ +_( ) @ +_|/ @@ +____@ +_ /@ +/ / @ +\ \ @ + \_\@ + @@ + @ +_______@ +_ ____/@ +/____/ @ + $ @ + @@ +___ @ +__ \ @ +___ \@ +__ /@ +_/_/ @ + @@ +_____ @ +_ __ \@ +__/ _/@ +_/_/ @ +(_) @ + @@ +_________ @ +__ ____ \@ +_ / __ `/@ +/ / /_/ / @ +\ \__,_/ @ + \____/ @@ +_______ @ +___ |@ +__ /| |@ +_ ___ |@ +/_/ |_|@ + @@ +________ @ +___ __ )@ +__ __ |@ +_ /_/ / @ +/_____/ @ + @@ +_________@ +__ ____/@ +_ / @ +/ /___ @ +\____/ @ + @@ +________ @ +___ __ \@ +__ / / /@ +_ /_/ / @ +/_____/ @ + @@ +__________@ +___ ____/@ +__ __/ @ +_ /___ @ +/_____/ @ + @@ +__________@ +___ ____/@ +__ /_ @ +_ __/ @ +/_/ @ + @@ +_________@ +__ ____/@ +_ / __ @ +/ /_/ / @ +\____/ @ + @@ +______ __@ +___ / / /@ +__ /_/ / @ +_ __ / @ +/_/ /_/ @ + @@ +________@ +____ _/@ + __ / @ +__/ / @ +/___/ @ + @@ +_________@ +______ /@ +___ _ / @ +/ /_/ / @ +\____/ @ + @@ +______ __@ +___ //_/@ +__ ,< @ +_ /| | @ +/_/ |_| @ + @@ +______ @ +___ / @ +__ / @ +_ /___@ +/_____/@ + @@ +______ ___@ +___ |/ /@ +__ /|_/ / @ +_ / / / @ +/_/ /_/ @ + @@ +_____ __@ +___ | / /@ +__ |/ / @ +_ /| / @ +/_/ |_/ @ + @@ +_______ @ +__ __ \@ +_ / / /@ +/ /_/ / @ +\____/ @ + @@ +________ @ +___ __ \@ +__ /_/ /@ +_ ____/ @ +/_/ @ + @@ +_______ @ +__ __ \@ +_ / / /@ +/ /_/ / @ +\___\_\ @ + @@ +________ @ +___ __ \@ +__ /_/ /@ +_ _, _/ @ +/_/ |_| @ + @@ +________@ +__ ___/@ +_____ \ @ +____/ / @ +/____/ @ + @@ +________@ +___ __/@ +__ / @ +_ / @ +/_/ @ + @@ +_____ __@ +__ / / /@ +_ / / / @ +/ /_/ / @ +\____/ @ + @@ +___ __@ +__ | / /@ +__ | / / @ +__ |/ / @ +_____/ @ + @@ +___ __@ +__ | / /@ +__ | /| / / @ +__ |/ |/ / @ +____/|__/ @ + @@ +____ __@ +__ |/ /@ +__ / @ +_ | @ +/_/|_| @ + @@ +__ __@ +_ \/ /@ +__ / @ +_ / @ +/_/ @ + @@ +______@ +___ /@ +__ / @ +_ /__@ +/____/@ + @@ +________@ +____ _/@ +___ / @ +__ / @ +_ / @ +/__/ @@ +___ @ +__ \ @ +___ \ @ +____ \ @ +______\@ + @@ +________@ +____/ /@ +____ / @ +___ / @ +__/ / @ +/__/ @@ +_ //|@ +_|/||@ + $ @ + $ @ +$ @ + @@ + @ + @ + @ + @ +________@ +_/_____/@@ +___ @ +_( )@ +__V @ + $ @ +$ @ + @@ + @ +______ _@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +______ @ +___ /_ @ +__ __ \@ +_ /_/ /@ +/_.___/ @ + @@ + @ +_______@ +_ ___/@ +/ /__ @ +\___/ @ + @@ +_________@ +______ /@ +_ __ / @ +/ /_/ / @ +\__,_/ @ + @@ + @ +_____ @ +_ _ \@ +/ __/@ +\___/ @ + @@ +________@ +___ __/@ +__ /_ @ +_ __/ @ +/_/ @ + @@ + @ +_______ _@ +__ __ `/@ +_ /_/ / @ +_\__, / @ +/____/ @@ +______ @ +___ /_ @ +__ __ \@ +_ / / /@ +/_/ /_/ @ + @@ +_____ @ +___(_)@ +__ / @ +_ / @ +/_/ @ + @@ +________ @ +______(_)@ +_____ / @ +____ / @ +___ / @ +/___/ @@ +______ @ +___ /__@ +__ //_/@ +_ ,< @ +/_/|_| @ + @@ +______@ +___ /@ +__ / @ +_ / @ +/_/ @ + @@ + @ +_______ ___ @ +__ __ `__ \@ +_ / / / / /@ +/_/ /_/ /_/ @ + @@ + @ +_______ @ +__ __ \@ +_ / / /@ +/_/ /_/ @ + @@ + @ +______ @ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ + @ +________ @ +___ __ \@ +__ /_/ /@ +_ .___/ @ +/_/ @@ + @ +______ _@ +_ __ `/@ +/ /_/ / @ +\__, / @ + /_/ @@ + @ +________@ +__ ___/@ +_ / @ +/_/ @ + @@ + @ +________@ +__ ___/@ +_(__ ) @ +/____/ @ + @@ +_____ @ +__ /_@ +_ __/@ +/ /_ @ +\__/ @ + @@ + @ +____ __@ +_ / / /@ +/ /_/ / @ +\__,_/ @ + @@ + @ +___ __@ +__ | / /@ +__ |/ / @ +_____/ @ + @@ + @ +___ __@ +__ | /| / /@ +__ |/ |/ / @ +____/|__/ @ + @@ + @ +____ __@ +__ |/_/@ +__> < @ +/_/|_| @ + @@ + @ +_____ __@ +__ / / /@ +_ /_/ / @ +_\__, / @ +/____/ @@ + @ +______@ +___ /@ +__ /_@ +_____/@ + @@ +_______@ +____/_/@ +__/_/ @ +< < @ +/ / @ +\_\ @@ +_______@ +____ /@ +___ / @ +__ / @ +_ / @ +/_/ @@ +____ _ @ +____| |@ +____/ /@ +____>_>@ +__/_/ @ +/_/ @@ +__/\//@ +_//\/ @ + $ @ + $ @ +$ @ + @@ +_____ _ @ +___(_)(_)@ +__ _ | @ +_ __ | @ +/_/ |_| @ + @@ +____ _ @ +__(_)_(_)@ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +____ _ @ +__(_) (_)@ +_ / / / @ +/ /_/ / @ +\____/ @ + @@ +____ _ @ +__(_)_(_)@ +_ __ `/ @ +/ /_/ / @ +\__,_/ @ + @@ +____ _ @ +__(_)_(_)@ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +____ _ @ +__(_) (_)@ +_ / / / @ +/ /_/ / @ +\__,_/ @ + @@ +_________ @ +____ __ \@ +___ / / /@ +__ /_| | @ +_ //__/ @ +/_/ @@ +160 + $$@ + $$ @ + $$ @ + $$ @ + $$ @ +$$ @@ +161 +_____ @ +___(_)@ +__ / @ +_ / @ +/_/ @ + @@ +162 +_______@ +____/ /@ +_ ___/@ +/ /__ @ +\ _/ @ +/_/ @@ +163 +_________ @ +____ ,__\@ +___/ /_ @ +__/ /___ @ +(_,____/ @ + @@ +164 +___ /|___/|@ +___| __ / @ +__ /_/ / @ +_ ___ | @ +|/ |/ @ + @@ +165 +___ ____@ +___| / /@ +_ _ __/@ +/_ __/ @ + /_/ @ + @@ +166 +_______@ +____ /@ +_____/ @ +____ @ +_ / @ +/_/ @@ +167 +_______ @ +____/ _)@ +__ | | @ +_| || | @ +_| |_/ @ +(__/ @@ +168 +___ _ @ +_(_) (_)@ + $ $ @ + $ $ @ +$ $ @ + @@ +169 +__________ @ +___ _____\ @ +__ / ___/ |@ +_ / /__ / @ +| \___/ / @ + \______/ @@ +170 +______ _@ +__ _ `/@ +__\_,_/ @ +/____/ @ + $ @ + @@ +171 +______@ +_ / /@ +/ / / @ +\ \ \ @ + \_\_\@ + @@ +172 + @ +________@ +_/___ /@ + /_/ @ + $ @ + @@ +173 + @ + @ +_______@ +_/____/@ + $ @ + @@ +174 +__________ @ +___ ___ \ @ +__ / _ \ |@ +_ / , _/ / @ +| /_/|_| / @ + \______/ @@ +175 +________@ +_/_____/@ + $ @ + $ @ + $ @ + @@ +176 +_____ @ +_ _ \@ +/ // /@ +\___/ @ + $ @ + @@ +177 +________ @ +_____/ /_@ +____ __/@ +___/_/_ @ +/_____/ @ + @@ +178 +__ ___ @ +__|_ |@ +_ __/ @ +/____/ @ + $ @ + @@ +179 +__ ____@ +__|_ /@ +__/_ < @ +/____/ @ + $ @ + @@ +180 +____@ +_/_/@ + $ @ + $ @ +$ @ + @@ +181 + @ +______ __@ +___ / / /@ +__ /_/ / @ +_ ._,_/ @ +/_/ @@ +182 +_________@ +_ _ /@ +/ (/ / / @ +\_ / / @ + /_/_/ @ + @@ +183 + @ +___ @ +_(_)@ + $ @ + $ @ + @@ +184 + @ + @ + @ + @ +___ @ +_/_)@@ +185 +_____@ +_< /@ +_ / @ +/_/ @ +$ @ + @@ +186 +______ @ +__ _ \@ +__\___/@ +/____/ @ + $ @ + @@ +187 +_____ @ +__ \ \ @ +___ \ \@ +__ / /@ +___/_/ @ + @@ +188 +_____ __ @ +_< / _/_/ @ +_/ /_/_/___@ +/_//_// / /@ + /_/ /_ _/@ + /_/ @@ +189 +_____ __ @ +_< / _/_/__ @ +_/ /_/_/|_ |@ +/_//_/ / __/ @ + /_/ /____/ @ + @@ +190 +__ ____ __ @ +__|_ / _/_/ @ +__/_ < _/_/___@ +/____//_// / /@ + /_/ /_ _/@ + /_/ @@ +191 +___ _ @ +___(_)@ +__ / @ +/ _/_ @ +\___/ @ + @@ +192 +______ @ +____\_\@ +__ _ |@ +_ __ |@ +/_/ |_|@ + @@ +193 +_______@ +____/_/@ +__ _ |@ +_ __ |@ +/_/ |_|@ + @@ +194 +____ //|@ +____|/||@ +__ _ | @ +_ __ | @ +/_/ |_| @ + @@ +195 +_____/\//@ +____//\/ @ +__ _ | @ +_ __ | @ +/_/ |_| @ + @@ +196 +_____ _ @ +___(_)(_)@ +__ _ | @ +_ __ | @ +/_/ |_| @ + @@ +197 +____(())@ +___ |@ +__ /| |@ +_ ___ |@ +/_/ |_|@ + @@ +198 +______________@ +___ ____/@ +__ /| __/ @ +_ __ /___ @ +/_/ /_____/ @ + @@ +199 +_________@ +__ ____/@ +_ / @ +/ /___ @ +\____/ @ + /_) @@ +200 +______ @ +____\_\@ +__ __/@ +_ _/ @ +/___/ @ + @@ +201 +_______@ +____/_/@ +__ __/@ +_ _/ @ +/___/ @ + @@ +202 +____ //|@ +____|/||@ +__ __/ @ +_ _/ @ +/___/ @ + @@ +203 +_____ _ @ +___(_)(_)@ +__ __/ @ +_ _/ @ +/___/ @ + @@ +204 +______ @ +____\_\@ +__ _/@ +__/ / @ +/___/ @ + @@ +205 +_______@ +____/_/@ +__ _/@ +__/ / @ +/___/ @ + @@ +206 +____ //|@ +____|/||@ +__ _/ @ +__/ / @ +/___/ @ + @@ +207 +_____ _ @ +___(_)(_)@ +__ _/ @ +__/ / @ +/___/ @ + @@ +208 +_________ @ +____ __ \@ +___ /_/ /@ +/_ __/ / @ + /_____/ @ + @@ +209 +_____/\//@ +____//\/ @ +__ |/ / @ +_ / @ +/_/|_/ @ + @@ +210 +______ @ +____\_\@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +211 +_______@ +____/_/@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +212 +___ //|@ +___|/||@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +213 +____/\//@ +___//\/ @ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +214 +____ _ @ +__(_)_(_)@ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +215 + @ +__ @ +_/|/|@ + > < @ +|/|/ @ + @@ +216 +________ @ +__ _// \@ +_ //// /@ +/ //// / @ +\_//__/ @ + @@ +217 +______ @ +____\_\_@ +_ / / /@ +/ /_/ / @ +\____/ @ + @@ +218 +_______ @ +____/_/_@ +_ / / /@ +/ /_/ / @ +\____/ @ + @@ +219 +___ //| @ +___|/||_@ +_ / / /@ +/ /_/ / @ +\____/ @ + @@ +220 +____ _ @ +__(_) (_)@ +_ / / / @ +/ /_/ / @ +\____/ @ + @@ +221 +______ @ +___/_/_@ +__ \/ /@ +___ / @ +__/_/ @ + @@ +222 +______ @ +___ /_ @ +__ __ \@ +_ ____/@ +/_/ @ + @@ +223 +_________ @ +____ __ \@ +___ / / /@ +__ /_| | @ +_ //__/ @ +/_/ @@ +224 +______ @ +____\_\_@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +225 +_______ @ +____/_/_@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +226 +___ //| @ +___|/||_@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +227 +____/\//@ +___//\/_@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +228 +____ _ @ +__(_)_(_)@ +_ __ `/ @ +/ /_/ / @ +\__,_/ @ + @@ +229 +_______ @ +____(())@ +_ __ `/@ +/ /_/ / @ +\__,_/ @ + @@ +230 + @ +______ ___ @ +_ __ ` _ \@ +/ /_/ __/@ +\__,_____/ @ + @@ +231 + @ +_______@ +_ ___/@ +/ /__ @ +\___/ @ +/_) @@ +232 +_____ @ +___\_\@ +_ _ \@ +/ __/@ +\___/ @ + @@ +233 +______@ +___/_/@ +_ _ \@ +/ __/@ +\___/ @ + @@ +234 +___ //|@ +___|/||@ +_ _ \ @ +/ __/ @ +\___/ @ + @@ +235 +____ _ @ +__(_)(_)@ +_ _ \ @ +/ __/ @ +\___/ @ + @@ +236 +_____ @ +___\_\@ +__ / @ +_ / @ +/_/ @ + @@ +237 +______@ +___/_/@ +__ / @ +_ / @ +/_/ @ + @@ +238 +___ //|@ +___|/||@ +__ / @ +_ / @ +/_/ @ + @@ +239 +_ _ _ @ +_(_)_(_)@ +__/ / @ +_ / @ +/_/ @ + @@ +240 +____ || @ +____=||=@ +____ || @ +/ __` | @ +\____/ @ + @@ +241 +_____/\//@ +____//\/ @ +__ __ \ @ +_ / / / @ +/_/ /_/ @ + @@ +242 +______ @ +____\_\@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +243 +_______@ +____/_/@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +244 +___ //|@ +___|/||@ +_ __ \@ +/ /_/ /@ +\____/ @ + @@ +245 +____/\//@ +___//\/ @ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +246 +____ _ @ +__(_)_(_)@ +_ __ \ @ +/ /_/ / @ +\____/ @ + @@ +247 + @ +_____ @ +___(_)_@ +/_____/@ + (_) @ + @@ +248 + @ +_______ @ +_ _// \@ +/ //// /@ +\_//__/ @ + @@ +249 +______ @ +____\_\_@ +_ / / /@ +/ /_/ / @ +\__,_/ @ + @@ +250 +_______ @ +____/_/_@ +_ / / /@ +/ /_/ / @ +\__,_/ @ + @@ +251 +___ //| @ +___|/||_@ +_ / / /@ +/ /_/ / @ +\__,_/ @ + @@ +252 +____ _ @ +__(_) (_)@ +_ / / / @ +/ /_/ / @ +\__,_/ @ + @@ +253 +________ @ +_____/_/_@ +__ / / /@ +_ /_/ / @ +_\__, / @ +/____/ @@ +254 +_______ @ +____ /_ @ +___ __ \@ +__ /_/ /@ +_ .___/ @ +/_/ @@ +255 +_____ _ @ +___(_) (_)@ +__ / / / @ +_ /_/ / @ +_\__, / @ +/____/ @@ diff --git a/roles/figlet_motd/tasks/create_motd.yml b/roles/figlet_motd/tasks/create_motd.yml new file mode 100644 index 000000000..71406061e --- /dev/null +++ b/roles/figlet_motd/tasks/create_motd.yml @@ -0,0 +1,22 @@ +--- +- name: 'Generate figlet ASCII art for hostname.' + shell: | + {{ figlet_cmd }} > "/etc/{{ ansible_hostname }}.figlet.{{ figlet_cmd | checksum }}" + args: + creates: "/etc/{{ ansible_hostname }}.figlet.{{ figlet_cmd | checksum }}" + become: true + +- name: 'Get generated figlet ASCII art for hostname.' + slurp: + src: "/etc/{{ ansible_hostname }}.figlet.{{ figlet_cmd | checksum }}" + register: figlet + +- name: 'Create /etc/motd.' + template: + src: 'templates/motd' + dest: '/etc/motd' + owner: root + group: root + mode: '0644' + become: true +... \ No newline at end of file diff --git a/roles/figlet_motd/tasks/main.yml b/roles/figlet_motd/tasks/main.yml new file mode 100644 index 000000000..5d420ecbd --- /dev/null +++ b/roles/figlet_motd/tasks/main.yml @@ -0,0 +1,33 @@ +--- +- name: 'Install figlet.' + yum: + state: latest + update_cache: yes + name: figlet + become: true + +- name: 'Install custom figlet fonts.' + synchronize: + src: "files/{{ item }}" + dest: "/usr/share/figlet/{{ item }}" + owner: no + group: no + use_ssh_args: yes + rsync_opts: + - '--chmod=Fu=rw,Fgo=r' + - '--perms' + - '--force' + with_items: + - 'cyberlarge.flf' + - 'doh.flf' + - 'ogre.flf' + - 'slant.flf' + - 'speed.flf' + become: true + +- name: 'Create MOTD.' + include_tasks: + file: "{{ playbook_dir }}/roles/figlet_motd/tasks/create_motd.yml" + vars: + figlet_cmd: 'figlet -f "{{ figlet_font }}" "{{ ansible_hostname }}"' +... \ No newline at end of file diff --git a/roles/figlet_motd/templates/motd b/roles/figlet_motd/templates/motd new file mode 100644 index 000000000..cbf741b9f --- /dev/null +++ b/roles/figlet_motd/templates/motd @@ -0,0 +1,4 @@ +{{ figlet['content'] | b64decode }} + +{{ motd }} + diff --git a/roles/fuse-layer/handlers/main.yml b/roles/fuse-layer/handlers/main.yml new file mode 100644 index 000000000..366e59928 --- /dev/null +++ b/roles/fuse-layer/handlers/main.yml @@ -0,0 +1,9 @@ +--- +- name: Restart fuse-layer service. + systemd: + name: 'ega-fuse-client.service' + state: 'restarted' + daemon_reload: 'yes' + become: true + listen: restart_fuser-layer +... diff --git a/roles/fuse-layer/tasks/main.yml b/roles/fuse-layer/tasks/main.yml index b3061ed84..4d2efe438 100644 --- a/roles/fuse-layer/tasks/main.yml +++ b/roles/fuse-layer/tasks/main.yml @@ -1,53 +1,49 @@ --- -- file: +- name: Install EGA Fuse client. + yum: + state: latest + name: + - ega-fuse-client + become: true + notify: restart_fuser-layer + +- name: Create /usr/local/fuse-layer directory. + file: path: /usr/local/fuse-layer state: directory - mode: 0755 + mode: '0755' + owner: root + group: root become: true + notify: restart_fuser-layer - name: Install service files. template: - src: templates/fuse-layer.service - dest: /etc/systemd/system/fuse-layer.service - mode: 644 + src: templates/ega-fuse-client.service + dest: /etc/systemd/system/ega-fuse-client.service + mode: '0644' owner: root group: root tags: - service-files become: true + notify: restart_fuser-layer -- name: Install fuse files. - template: - src: templates/fuse.sh - dest: /usr/local/fuse-layer/fuse.sh - mode: 644 - owner: root - group: root - become: true - -- name: Install config.ini files. +- name: Install config.ini files. template: src: templates/config.ini dest: /usr/local/fuse-layer/config.ini - mode: 644 + mode: '0644' owner: root group: root become: true + notify: restart_fuser-layer -- name: install service files - command: systemctl daemon-reload - become: true - -- name: enable service at boot +- name: Make sure fuse-layer service is enabled and started. systemd: - name: fuse-layer - enabled: yes - become: true - -- name: make sure servcies are started. - systemd: - name: fuse-layer.service - state: restarted - tags: - - start-service + name: 'ega-fuse-client.service' + state: 'started' + enabled: 'yes' + daemon_reload: yes become: true + notify: restart_fuser-layer diff --git a/roles/fuse-layer/templates/ega-fuse-client.service b/roles/fuse-layer/templates/ega-fuse-client.service new file mode 100644 index 000000000..dd636f7e1 --- /dev/null +++ b/roles/fuse-layer/templates/ega-fuse-client.service @@ -0,0 +1,16 @@ +[Unit] +Description=EGA Fuse Layer + +[Service] +TimeoutStartSec=0 + +ExecStart=/apps/software/AdoptOpenJDK/8u222b10-hotspot/bin/java \ +-Xmx4g \ +-jar /usr/local/fuse-layer/ega-fuse-1.0-SNAPSHOT.jar \ +-f /usr/local/fuse-layer/config.ini \ +-m "{{ fuse_mountpoint }}" \ +-u "{{ fuse_user }}" \ +-p "{{ fuse_password }}" + +[Install] +WantedBy=multi-user.target diff --git a/roles/fuse-layer/templates/fuse-layer.service b/roles/fuse-layer/templates/fuse-layer.service deleted file mode 100644 index dc77507d1..000000000 --- a/roles/fuse-layer/templates/fuse-layer.service +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=EGA Fuse Layer - -[Service] -TimeoutStartSec=0 -Restart=always -ExecStart=/bin/bash /usr/local/fuse-layer/fuse.sh - -[Install] -WantedBy=multi-user.target diff --git a/roles/fuse-layer/templates/fuse.sh b/roles/fuse-layer/templates/fuse.sh deleted file mode 100644 index a240e15fc..000000000 --- a/roles/fuse-layer/templates/fuse.sh +++ /dev/null @@ -1,8 +0,0 @@ -source "/apps/modules//modules.bashrc" -module load Java/8-LTS - -java -Xmx2g -jar /usr/local/fuse-layer/ega-fuse-1.0-SNAPSHOT.jar \ --f /usr/local/fuse-layer/config.ini \ --m "{{ fuse_mountpoint }}" \ --u "{{ fuse_user }}" \ --p "{{ fuse_password }}" diff --git a/roles/grafana/handlers/main.yml b/roles/grafana/handlers/main.yml new file mode 100644 index 000000000..b0dfe56bc --- /dev/null +++ b/roles/grafana/handlers/main.yml @@ -0,0 +1,17 @@ +--- +# +# Important: maintain correct handler order. +# Handlers are executed in the order in which they are defined +# and not in the order in which they are listed in a "notify: handler_name" statement! +# +# Restart before reload: an reload after a restart may be redundant but should not fail, +# but the other way around may fail when the impact of changes was too large for a reload. +# +- name: Restart grafana service. + systemd: + name: 'grafana.service' + state: restarted + daemon_reload: yes + become: true + listen: restart_grafana +... diff --git a/roles/grafana/meta/main.yml b/roles/grafana/meta/main.yml new file mode 100644 index 000000000..20f53d089 --- /dev/null +++ b/roles/grafana/meta/main.yml @@ -0,0 +1,16 @@ +--- +galaxy_info: + role_name: grafana + author: Pieter Neerincx (UMCG) Egon Rijpkema (UG) + description: runs grafana in a docker container. + min_ansible_version: 2.4 + license: "license (GPLv3)" + platforms: + - name: CentOS + versions: + - all + + +dependencies: + - {role: docker} +... diff --git a/roles/grafana/tasks/main.yml b/roles/grafana/tasks/main.yml new file mode 100644 index 000000000..5e030ef01 --- /dev/null +++ b/roles/grafana/tasks/main.yml @@ -0,0 +1,36 @@ +--- +- name: Create directories for grafana. + file: + path: "{{ item }}" + state: directory + mode: 0755 + owner: '65534' + with_items: + - '/srv/grafana/lib' + notify: + - restart_grafana + become: true + +- name: Install service files. + template: + src: 'templates/grafana.service' + dest: '/etc/systemd/system/grafana.service' + mode: 0644 + owner: root + group: root + tags: + - service-files + notify: + - restart_grafana + become: true + +- name: Make sure grafana service is started and enabled on (re)boot. + systemd: + name: grafana.service + enabled: yes + state: started + daemon_reload: yes + tags: + - start-service + become: true +... diff --git a/roles/grafana/templates/grafana.service b/roles/grafana/templates/grafana.service new file mode 100644 index 000000000..10babd1fe --- /dev/null +++ b/roles/grafana/templates/grafana.service @@ -0,0 +1,14 @@ +[Unit] +Description=Grafana monitoring dashboard +After=docker.service +Requires=docker.service + +[Service] +TimeoutStartSec=0 +Restart=always +ExecStartPre=-/usr/bin/docker kill grafana +ExecStartPre=-/usr/bin/docker rm grafana +ExecStart=/usr/bin/docker run --rm --name=grafana --network host \ + -v /srv/grafana/lib:/var/lib/grafana \ + -e GF_PATHS_DATA=/var/lib/grafana \ + grafana/grafana diff --git a/roles/grafana_proxy/files/tls/cert.pem b/roles/grafana_proxy/files/tls/cert.pem new file mode 100644 index 000000000..998fc7c03 --- /dev/null +++ b/roles/grafana_proxy/files/tls/cert.pem @@ -0,0 +1,45 @@ +-----BEGIN CERTIFICATE----- +MIIFHTCCBMOgAwIBAgIRAIZRRoAnkmdok1TkINyXl/4wCgYIKoZIzj0EAwIwRDELMAkGA1UEBhMC +TkwxGTAXBgNVBAoTEEdFQU5UIFZlcmVuaWdpbmcxGjAYBgNVBAMTEUdFQU5UIE9WIEVDQyBDQSA0 +MB4XDTIwMDYxMjAwMDAwMFoXDTIyMDYxMjIzNTk1OVowgbMxCzAJBgNVBAYTAk5MMQ8wDQYDVQQR +EwY5NzEyQ1AxEjAQBgNVBAgTCUdyb25pbmdlbjESMBAGA1UEBxMJR3JvbmluZ2VuMRYwFAYDVQQJ +Ew1Ccm9lcnN0cmFhdCA1MSQwIgYDVQQKExtSaWprc3VuaXZlcnNpdGVpdCBHcm9uaW5nZW4xEDAO +BgNVBAsTB0NpVCBIUEMxGzAZBgNVBAMTEmFpcmxvY2suaHBjLnJ1Zy5ubDBZMBMGByqGSM49AgEG +CCqGSM49AwEHA0IABNryOiLPhNM1ynMD9Z29w+U9HUZEiT7dFg0wF69Yo78m3fni68/1/BJWuCT0 +Ke4kekDuZLCIQGpt6hbvwqt7kjqjggMkMIIDIDAfBgNVHSMEGDAWgBTttKAzahsIkba9+kGSvZqr +q2P0UzAdBgNVHQ4EFgQUfj39oCaeDulvoIBadw0uuufJxXUwDgYDVR0PAQH/BAQDAgeAMAwGA1Ud +EwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMEkGA1UdIARCMEAwNAYLKwYB +BAGyMQECAk8wJTAjBggrBgEFBQcCARYXaHR0cHM6Ly9zZWN0aWdvLmNvbS9DUFMwCAYGZ4EMAQIC +MD8GA1UdHwQ4MDYwNKAyoDCGLmh0dHA6Ly9HRUFOVC5jcmwuc2VjdGlnby5jb20vR0VBTlRPVkVD +Q0NBNC5jcmwwdQYIKwYBBQUHAQEEaTBnMDoGCCsGAQUFBzAChi5odHRwOi8vR0VBTlQuY3J0LnNl +Y3RpZ28uY29tL0dFQU5UT1ZFQ0NDQTQuY3J0MCkGCCsGAQUFBzABhh1odHRwOi8vR0VBTlQub2Nz +cC5zZWN0aWdvLmNvbTCCAX0GCisGAQQB1nkCBAIEggFtBIIBaQFnAHUARqVV63X6kSAwtaKJafTz +fREsQXS+/Um4havy/HD+bUcAAAFyp2RofQAABAMARjBEAiAykzfRFg6kmOHTBUBn9QctLKcmZiab +nZ/9rNe5qrOvEgIgfJy8u1FdX4zZ5tt0djKiQGNk6NJkAVcHwM0+liOz9GwAdgDfpV6raIJPH2yt +7rhfTj5a6s2iEqRqXo47EsAgRFwqcwAAAXKnZGisAAAEAwBHMEUCIQDz9kSR3BdKbl3QPkX3ZtdG +Gd4u9QYCJ5mhUadYCEZdfQIgVi91o7Qrq4MLemj32Loc85Z3lootD5147wRwrCvFqRUAdgBvU3as +MfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZEwAAAXKnZGp4AAAEAwBHMEUCIB+rRhPUQNjFh753 +r7cqP4YB7xcxpbCyGphUbUFGo0Z5AiEAzPqck56sT/2Ya6jzdKfwz5hpBiffztBrEA7Eb2fl3lAw +HQYDVR0RBBYwFIISYWlybG9jay5ocGMucnVnLm5sMAoGCCqGSM49BAMCA0gAMEUCIBCniWd8tVSH +DmktU6NsVU1H4LtEM3v9zjLGiVoziHskAiEA14jiX8F4OH8kFKoleQfWTtkFWtpS00+rGoSD5CYX +uM8= +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +MIIDeTCCAv+gAwIBAgIRAOuOgRlxKfSvZO+BSi9QzukwCgYIKoZIzj0EAwMwgYgxCzAJBgNVBAYT +AlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMV +VGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgRUNDIENlcnRpZmljYXRp +b24gQXV0aG9yaXR5MB4XDTIwMDIxODAwMDAwMFoXDTMzMDUwMTIzNTk1OVowRDELMAkGA1UEBhMC +TkwxGTAXBgNVBAoTEEdFQU5UIFZlcmVuaWdpbmcxGjAYBgNVBAMTEUdFQU5UIE9WIEVDQyBDQSA0 +MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEXYkvGrfrMs2IwdI5+IwpEwPh+igW/BOWetmOwP/Z +IXC8fNeC3/ZYPAAMyRpFS0v3/c55FDTE2xbOUZ5zeVZYQqOCAYswggGHMB8GA1UdIwQYMBaAFDrh +CYbUzxnClnZ0SXbc4DXGY2OaMB0GA1UdDgQWBBTttKAzahsIkba9+kGSvZqrq2P0UzAOBgNVHQ8B +Af8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH +AwIwOAYDVR0gBDEwLzAtBgRVHSAAMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8vc2VjdGlnby5jb20v +Q1BTMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RF +Q0NDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2BggrBgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKG +M2h0dHA6Ly9jcnQudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RFQ0NBZGRUcnVzdENBLmNydDAlBggr +BgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAKBggqhkjOPQQDAwNoADBlAjAfs9ns +M0qaJGVu6DpWVy4qojiOpwV1h/MWZ5GJxy6CKv3+RMB3STkaFh0+Hifbk24CMQDRf/ujXAQ1b4nF +pZGaSIKldygcdCDAxbAd9tlxcN/+J534CJDblzd/40REzGWwS5k= +-----END CERTIFICATE----- + diff --git a/roles/grafana_proxy/files/tls/dhparam b/roles/grafana_proxy/files/tls/dhparam new file mode 100644 index 000000000..088f9673d --- /dev/null +++ b/roles/grafana_proxy/files/tls/dhparam @@ -0,0 +1,8 @@ +-----BEGIN DH PARAMETERS----- +MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz ++8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a +87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7 +YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi +7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD +ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg== +-----END DH PARAMETERS----- \ No newline at end of file diff --git a/roles/grafana_proxy/files/tls/fullchain.pem b/roles/grafana_proxy/files/tls/fullchain.pem new file mode 100644 index 000000000..ada72c0f4 --- /dev/null +++ b/roles/grafana_proxy/files/tls/fullchain.pem @@ -0,0 +1,38 @@ +-----BEGIN CERTIFICATE----- +MIIDeTCCAv+gAwIBAgIRAOuOgRlxKfSvZO+BSi9QzukwCgYIKoZIzj0EAwMwgYgxCzAJBgNVBAYT +AlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMV +VGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgRUNDIENlcnRpZmljYXRp +b24gQXV0aG9yaXR5MB4XDTIwMDIxODAwMDAwMFoXDTMzMDUwMTIzNTk1OVowRDELMAkGA1UEBhMC +TkwxGTAXBgNVBAoTEEdFQU5UIFZlcmVuaWdpbmcxGjAYBgNVBAMTEUdFQU5UIE9WIEVDQyBDQSA0 +MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEXYkvGrfrMs2IwdI5+IwpEwPh+igW/BOWetmOwP/Z +IXC8fNeC3/ZYPAAMyRpFS0v3/c55FDTE2xbOUZ5zeVZYQqOCAYswggGHMB8GA1UdIwQYMBaAFDrh +CYbUzxnClnZ0SXbc4DXGY2OaMB0GA1UdDgQWBBTttKAzahsIkba9+kGSvZqrq2P0UzAOBgNVHQ8B +Af8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH +AwIwOAYDVR0gBDEwLzAtBgRVHSAAMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8vc2VjdGlnby5jb20v +Q1BTMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RF +Q0NDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2BggrBgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKG +M2h0dHA6Ly9jcnQudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RFQ0NBZGRUcnVzdENBLmNydDAlBggr +BgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAKBggqhkjOPQQDAwNoADBlAjAfs9ns +M0qaJGVu6DpWVy4qojiOpwV1h/MWZ5GJxy6CKv3+RMB3STkaFh0+Hifbk24CMQDRf/ujXAQ1b4nF +pZGaSIKldygcdCDAxbAd9tlxcN/+J534CJDblzd/40REzGWwS5k= +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +MIID0zCCArugAwIBAgIQVmcdBOpPmUxvEIFHWdJ1lDANBgkqhkiG9w0BAQwFADB7MQswCQYDVQQG +EwJHQjEbMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRowGAYD +VQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmljYXRlIFNlcnZpY2Vz +MB4XDTE5MDMxMjAwMDAwMFoXDTI4MTIzMTIzNTk1OVowgYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQI +EwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVT +VCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgRUNDIENlcnRpZmljYXRpb24gQXV0aG9yaXR5 +MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEGqxUWqn5aCPnetUkb1PGWthLq8bVttHmc3Gu3ZzWDGH9 +26CJA7gFFOxXzu5dP+Ihs8731Ip54KODfi2X0GHE8ZncJZFjq38wo7Rw4sehM5zzvy5cU7Ffs30y +f4o043l5o4HyMIHvMB8GA1UdIwQYMBaAFKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBQ6 +4QmG1M8ZwpZ2dEl23OA1xmNjmjAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zARBgNV +HSAECjAIMAYGBFUdIAAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v +QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhho +dHRwOi8vb2NzcC5jb21vZG9jYS5jb20wDQYJKoZIhvcNAQEMBQADggEBABns652JLCALBIAdGN5C +mXKZFjK9Dpx1WywV4ilAbe7/ctvbq5AfjJXyij0IckKJUAfiORVsAYfZFhr1wHUrxeZWEQff2Ji8 +fJ8ZOd+LygBkc7xGEJuTI42+FsMuCIKchjN0djsoTI0DQoWz4rIjQtUfenVqGtF8qmchxDM6OW1T +yaLtYiKou+JVbJlsQ2uRl9EMC5MCHdK8aXdJ5htN978UeAOwproLtOGFfy/cQjutdAFI3tZs4RmY +CV4Ks2dH/hzg1cEo70qLRDEmBDeNiXQ2Lu+lIg+DdEmSx/cQwgwp+7e9un/jX9Wf8qn0dNW44bOw +geThpWOjzOoEeJBuv/c= +-----END CERTIFICATE----- diff --git a/roles/grafana_proxy/files/tls/privkey.pem b/roles/grafana_proxy/files/tls/privkey.pem new file mode 100644 index 000000000..421af4b46 --- /dev/null +++ b/roles/grafana_proxy/files/tls/privkey.pem @@ -0,0 +1,20 @@ +$ANSIBLE_VAULT;1.2;AES256;gearshift +66323163383661366363623638373465366236653466616339396538383935666333613636663665 +3036373438656230303430383561623939393439353230350a323438626264623838353866623164 +64313637623139393534626233663532326166613734366630623065393430613662396534666233 +3465663338663833650a653865346137656433666130343232653538303536386538663134646461 +63306539643237666464363631336137323963363533646566346536363761376566383233396465 +61363334613264373230323730396534633662633665393031346438323031303535386135323235 +36643562386564643063653333646239623630373664356336343031386165333831303464623464 +62316236366531616362653863386131303037323234383666346332383465356131353538633538 +33383633373561646131653661633337363835633531663337313330393837663365303830376161 +30613861613864303562306632653938623262363638386265336262643537323230353066656636 +63613832646237656130633463306530373865633062316530326535653936326235386238616166 +32356132393961643736636661636633633930616564613535646331393564636236303836383933 +30653730313130636635636539323863333635346333663939646431613061613730333334346138 +66393236373430623639333766356566313335386265353363636638623461653033336433343938 +33653538663532626662643363363330643237336630396633326463393261333162386534303864 +38323130613035323134356535326636363365633833633635646336663135366161303862633034 +65306633373366326263316662663439303434613633653563626363316632313032386566666533 +62623636653766333537386336336561376630353036323462363666616535633033346533623661 +353862313361396331623063333362336263 diff --git a/roles/grafana_proxy/files/tls/request.csr b/roles/grafana_proxy/files/tls/request.csr new file mode 100644 index 000000000..85c6b3b2d --- /dev/null +++ b/roles/grafana_proxy/files/tls/request.csr @@ -0,0 +1,10 @@ +-----BEGIN CERTIFICATE REQUEST----- +MIIBajCCAQ8CAQAwgawxCzAJBgNVBAYTAk5MMRIwEAYDVQQIDAlHcm9uaW5nZW4x +EjAQBgNVBAcMCUdyb25pbmdlbjEgMB4GA1UECgwXVW5pdmVyc2l0eSBvZiBHcm9u +aW5nZW4xEDAOBgNVBAsMB0NpVCBIUEMxGzAZBgNVBAMMEmFpcmxvY2suaHBjLnJ1 +Zy5ubDEkMCIGCSqGSIb3DQEJARYVZS5tLmEucmlqcGtlbWFAcnVnLm5sMFkwEwYH +KoZIzj0CAQYIKoZIzj0DAQcDQgAE2vI6Is+E0zXKcwP1nb3D5T0dRkSJPt0WDTAX +r1ijvybd+eLrz/X8Ela4JPQp7iR6QO5ksIhAam3qFu/Cq3uSOqAAMAoGCCqGSM49 +BAMCA0kAMEYCIQDIE6W/UiXxxL5t/J9FA2vAHJUhNVYgt2QWwoi0cq9TKgIhAKuQ +E0Qyj9RZAl7eJR8x4EohnTU4ujQaKKNN/fhk5Gbl +-----END CERTIFICATE REQUEST----- diff --git a/roles/prom_proxy/tasks/main.yml b/roles/grafana_proxy/tasks/main.yml similarity index 52% rename from roles/prom_proxy/tasks/main.yml rename to roles/grafana_proxy/tasks/main.yml index 4e7c587e9..19523e40e 100644 --- a/roles/prom_proxy/tasks/main.yml +++ b/roles/grafana_proxy/tasks/main.yml @@ -3,7 +3,7 @@ yum: name: nginx state: latest - update_cache: yes + update_cache: true become: true - name: Deploy nginx.conf. @@ -15,13 +15,22 @@ group: root become: true -- name: Add .htpasswd +- name: create tls certs directory + file: + path: /etc/certificates/live/airlock.hpc.rug.nl + state: directory + mode: 0751 + become: true + +- name: copy certificate and chain files in place copy: - content: "{{ prom_proxy_htpasswd }}" - dest: /etc/nginx/.htpasswd - mode: 0600 - owner: nginx - group: nginx + src: "tls/{{ item }}" + dest: /etc/certificates/live/{{ item }} + with_items: + - cert.pem + - dhparam + - fullchain.pem + - privkey.pem become: true - name: Make sure nginx is enabled and restarted. diff --git a/roles/grafana_proxy/templates/domain.key b/roles/grafana_proxy/templates/domain.key new file mode 100644 index 000000000..e0bb7c061 --- /dev/null +++ b/roles/grafana_proxy/templates/domain.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDJa06GC0zGqRM3 +XlrpOCkC8sQIzsEz9Wk5aovwaiXeQM3pnv2x30/YfP/afTdycd+aGqUtJbF2rYMb +zjiLRe3h4bYazJClI71zIMEysIVWTIXjqOYSZqnvZ3WQ1vdiiDN3c4hIkFAc870y +cfEckb6nXMZhSPc/hYkV68IXKClDIfZfzNWRxDdBMseUjyjpji8pl2+8ItIWcshU +QJeRo6wSiM4VLmalN7Nl/fO4SpP//Ysd8jE3P3Xl1H9ryqVzl/YWsCp9CIpurHLS +30tPUwJLvfb6RBPNpdNfs8sSQQnlJagJvy6ox0YgK1Z1Iq2y9gtdis9rrLUddzsJ +XBM8H10jAgMBAAECggEBALdFXH27anCZB1MqF65921NDcfxRpJqSgtwrhomLDEne +zX31nQFsIErr8iHsLbeExeCCvikGUyOZXeK/EdQoHPubVQetey/IBWnsM97bn3eW +yX0NActcfuPHT8Iq145gP/RAu5DLPxoY7d4orys/siq20AtVSSQTbVj02/YWJPVf +JECUdMe7K7btWjiDZFWf3Lpa5DUWk9gSOhxksgKJeZxl5EKVYfBZ62Do0ZA4F2gz +t3bUH2gC20SFLwLNxbfImXsP8m7FemneNcTllRVQDJvXd0nt+a+TXKkED3lKfnUi +I2NAG/QWdsm3baLMUYJ1t+HIqaIYxf6tXlalWjE90QECgYEA6zE6jhq1K24skbZ5 +nwhU/kO8cEd7OGHWt1uv3S/cLh9+A3ARJydtWRPRqPQMHatgFPetMp2h8fiu+Aoj +2DknZ6v8Zrp/AnFucofnOPtZ8HPMnuFNJ/mFcy354mLQr0WqVktsW8IBTYQIAYNX +lLBDI3b2DPfuQIF6yElhnNDMq1sCgYEA2z0qIA33ZxoCOLorq23ybqxUuBbH2Gxk +fQXlW8SA1CCnMPQh99NBi3R/vL/xouhiQQIomYILrT6tYOHEtomTTpcIvhB2zwPv +xGI0nIzTE7c606lBXQlw5Oo4DWBEf3D6N9MwjU1zkU1t5TbIMb5SxFtZKHnqYYz/ +aiWNUFZ559kCgYEAhU8rDH8Q76QMOS9/8meLnW+RcObjbgru5JFgtnO5Tp9lEtfD +sTjE3ze/oA9siqin1lqo3kbBE2FM86TzukxzDG1MrIHLGfxniR2FLyQxT3yjn+6+ +iAkD3kxSFkLCXIaVTUKFEeF+UYyj6Q+kZdAezgxQ3PrvHbsGeF43qOYlDH8CgYEA +wHrouw/H/+rjvG4m3PFAgU4nayAFAMvZRbKhdXoGKTcp0l7kW7JLv2XeqBZUzVP8 +v1pnzq5QFiALFB3jGOecPsl3UXce+A1aui4GRtOUlaCusxnMWyRoy3GfGI9GVCUa +aVyvKb+wEQfLzgvUo2eXPvF8S38N+etSaxzg/oGNsbECgYBoNFcerfgN8VPfs0iJ +f65fuqX7GR+arOTTNP5gqf3RhdLqJ5Ptp82iD5CSNkB/A+LSIX2LuGHc5uIQ0iP8 +5ZfzxEqLk6VQOpSxNH2sCDnp5LHSk2IV78pNCmrnB55KHSMU/grPrUqJZ2IPRGxa +QA3E+4tkWkOaTUz7DKwwfmbItw== +-----END PRIVATE KEY----- diff --git a/roles/grafana_proxy/templates/nginx.conf b/roles/grafana_proxy/templates/nginx.conf new file mode 100644 index 000000000..2d9dd04e2 --- /dev/null +++ b/roles/grafana_proxy/templates/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ +# * Official Russian Documentation: http://nginx.org/ru/docs/ + +user nginx; +worker_processes auto; +error_log /var/log/nginx/error.log; +pid /run/nginx.pid; + +# Load dynamic modules. See /usr/share/nginx/README.dynamic. +include /usr/share/nginx/modules/*.conf; + +events { + worker_connections 1024; +} + +http { + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + + include /etc/nginx/mime.types; + default_type application/octet-stream; + + # Load modular configuration files from the /etc/nginx/conf.d directory. + # See http://nginx.org/en/docs/ngx_core_module.html#include + # for more information. + include /etc/nginx/conf.d/*.conf; + +server { + listen 443 ssl http2; + server_name airlock.hpc.rug.nl; + + ssl_certificate /etc/certificates/live/airlock.hpc.rug.nl/cert.pem; + ssl_certificate_key /etc/certificates/live/airlock.hpc.rug.nl/privkey.pem; + ssl_session_timeout 1d; + ssl_session_cache shared:MozSSL:10m; # about 40000 sessions + ssl_session_tickets off; + + # curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam + ssl_dhparam /etc/certificates/live/airlock.hpc.rug.nl/dhparam; + + # intermediate configuration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; + ssl_prefer_server_ciphers off; + + # HSTS (ngx_http_headers_module is required) (63072000 seconds) + add_header Strict-Transport-Security "max-age=63072000" always; + + # OCSP stapling + ssl_stapling on; + ssl_stapling_verify on; + + # verify chain of trust of OCSP response using Root CA and Intermediate certs + ssl_trusted_certificate /etc/certificates/live/airlock.hpc.rug.nl/fullchain.pem; + + # replace with the IP address of your resolver + resolver 127.0.0.1; + + # reverse proxy requests to imperatir + location / { + proxy_buffering off; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + proxy_pass http://imperator:3000; + proxy_pass_header X-CSRF-Token; + client_max_body_size 10M; + } + + + + + } +} + diff --git a/roles/iptables/README.md b/roles/iptables/README.md new file mode 100755 index 000000000..62bde29e4 --- /dev/null +++ b/roles/iptables/README.md @@ -0,0 +1,95 @@ +# Ansible Role: Firewall (iptables) + +https://www.digitalocean.com/community/tutorials/how-to-migrate-from-firewalld-to-iptables-on-centos-7 + +[![Build Status](https://travis-ci.org/geerlingguy/ansible-role-firewall.svg?branch=master)](https://travis-ci.org/geerlingguy/ansible-role-firewall) + +Installs an iptables-based firewall for Linux. Supports both IPv4 (`iptables`) and IPv6 (`ip6tables`). + +This firewall aims for simplicity over complexity, and only opens a few specific ports for incoming traffic (configurable through Ansible variables). If you have a rudimentary knowledge of `iptables` and/or firewalls in general, this role should be a good starting point for a secure system firewall. + +After the role is run, a `firewall` init service will be available on the server. You can use `service firewall [start|stop|restart|status]` to control the firewall. + +## Requirements + +None. + +## Role Variables + +Available variables are listed below, along with default values (see `defaults/main.yml`): + + firewall_state: started + firewall_enabled_at_boot: true + +Controls the state of the firewall service; whether it should be running (`firewall_state`) and/or enabled on system boot (`firewall_enabled_at_boot`). + + firewall_allowed_tcp_ports: + - "22" + - "80" + ... + firewall_allowed_udp_ports: [] + +A list of TCP or UDP ports (respectively) to open to incoming traffic. + + firewall_forwarded_tcp_ports: + - { src: "22", dest: "2222" } + - { src: "80", dest: "8080" } + firewall_forwarded_udp_ports: [] + +Forward `src` port to `dest` port, either TCP or UDP (respectively). + + firewall_additional_rules: [] + firewall_ip6_additional_rules: [] + +Any additional (custom) rules to be added to the firewall (in the same format you would add them via command line, e.g. `iptables [rule]`/`ip6tables [rule]`). A few examples of how this could be used: + + # Allow only the IP 167.89.89.18 to access port 4949 (Munin). + firewall_additional_rules: + - "iptables -A INPUT -p tcp --dport 4949 -s 167.89.89.18 -j ACCEPT" + + # Allow only the IP 214.192.48.21 to access port 3306 (MySQL). + firewall_additional_rules: + - "iptables -A INPUT -p tcp --dport 3306 -s 214.192.48.21 -j ACCEPT" + +See [Iptables Essentials: Common Firewall Rules and Commands](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands) for more examples. + + firewall_log_dropped_packets: true + +Whether to log dropped packets to syslog (messages will be prefixed with "Dropped by firewall: "). + + firewall_disable_firewalld: false + firewall_disable_ufw: false + +Set to `true` to disable firewalld (installed by default on RHEL/CentOS) or ufw (installed by default on Ubuntu), respectively. + +## Dependencies + +None. + +## Example Playbook + + - hosts: server + vars_files: + - vars/main.yml + roles: + - { role: geerlingguy.firewall } + +*Inside `vars/main.yml`*: + + firewall_allowed_tcp_ports: + - "22" + - "25" + - "80" + +## TODO + + - Make outgoing ports more configurable. + - Make other firewall features (like logging) configurable. + +## License + +MIT / BSD + +## Author Information + +This role was created in 2014 by [Jeff Geerling](https://www.jeffgeerling.com/), author of [Ansible for DevOps](https://www.ansiblefordevops.com/). diff --git a/roles/iptables/defaults/main.yml b/roles/iptables/defaults/main.yml new file mode 100755 index 000000000..c0739694f --- /dev/null +++ b/roles/iptables/defaults/main.yml @@ -0,0 +1,36 @@ +--- +# +# ICMP (only inbound; outbound ICMP is allowed without restrictions). +# +iptables_allow_icmp_inbound: [] +# +# SSH. +# +iptables_allow_ssh_inbound: [] +iptables_allow_ssh_outbound: [] +# +# Public MySQL databases @ EBI (only outbound). +# +iptables_allow_ebi_mysql_outbound: [] +# +# FTP (only outbound). +# +iptables_allow_ftp_outbound: [] +# +# Aspera (only outbound). +# +iptables_allow_aspera_outbound: [] +# +# Globus ToolKit (only outbound). +# +iptables_allow_globus_outbound: [] +# +# Log dropped packets only for debugging to prevent flooding logs! +# +iptables_log_dropped_packets: false +# +# Set to true to ensure other iptables management software is disabled. +# +iptables_disable_firewalld: false +iptables_disable_ufw: false +... diff --git a/roles/iptables/handlers/main.yml b/roles/iptables/handlers/main.yml new file mode 100755 index 000000000..24528bd2c --- /dev/null +++ b/roles/iptables/handlers/main.yml @@ -0,0 +1,17 @@ +--- +- name: 'Restart iptables based firewall deamons (IPv4 and IPv6).' + service: + name: "{{ item }}" + state: 'restarted' + daemon_reload: 'yes' + with_items: + - 'iptables' + - 'ip6tables' + become: true + listen: restart_iptables + +- name: 'Configure iptables based firewalls (IPv4 and IPv6).' + command: '/root/configure-iptables-firewall.bash' + become: true + listen: configure_iptables +... diff --git a/roles/iptables/tasks/disable-other-firewalls.yml b/roles/iptables/tasks/disable-other-firewalls.yml new file mode 100755 index 000000000..50aae98b9 --- /dev/null +++ b/roles/iptables/tasks/disable-other-firewalls.yml @@ -0,0 +1,34 @@ +--- +- name: Check if firewalld package is installed on RedHat. + shell: yum list installed firewalld + args: + warn: no + register: firewalld_installed + ignore_errors: true + changed_when: false + when: ansible_os_family == "RedHat" and firewall_disable_firewalld + +- name: Disable the firewalld service on RedHat. + service: + name: firewalld + state: stopped + enabled: no + when: ansible_os_family == "RedHat" and firewall_disable_firewalld and firewalld_installed.rc == 0 + +- name: Check if ufw package is installed (on Ubuntu). + shell: service ufw status + args: + warn: no + register: ufw_installed + ignore_errors: true + changed_when: false + when: ansible_distribution == "Ubuntu" and firewall_disable_ufw + notify: Disable ufw firewall. + +- name: Disable ufw firewall. + service: + name: ufw + state: stopped + enabled: no + when: ansible_distribution == "Ubuntu" and firewall_disable_ufw and ufw_installed.rc == 0 +... \ No newline at end of file diff --git a/roles/iptables/tasks/main.yml b/roles/iptables/tasks/main.yml new file mode 100644 index 000000000..a8db567b5 --- /dev/null +++ b/roles/iptables/tasks/main.yml @@ -0,0 +1,128 @@ +############################################################################################################################################### +# Query tests +#- name: 'List hostinfo 1.' +# debug: +# msg: "allNetworkInterfaces: {{ ansible_facts | dict2items | selectattr('value.ipv4', 'defined') | map(attribute='value.device') | list }}" +# +#- name: 'List hostinfo 2.' +# debug: +# msg: "allNetworkInterfaces: {{ ansible_facts | json_query(_query) }}" +# vars: +# _query: "*.ipv4.address" +# +#- name: 'List hostinfo 3.' +# debug: +# msg: "allNetworkInterfaces: {{ ansible_facts | dict2items | selectattr('value.ipv4', 'defined') | list }}" +# +#- name: 'List hostinfo 3.q' +# debug: +# msg: "allNetworkInterfaces: {{ ansible_facts | dict2items | selectattr('value.ipv4', 'defined') | list | json_query(_query) }}" +# vars: +# _query: '[].{Name: value.device, IPv4: value.ipv4, IPv6: value.ipv6, Type: value.type}' +############################################################################################################################################### + +# +# This role configures host-based firewalls using iptables. +# We first create a list of IPv4 addresses used by a host and determine if these addresses are public or private. +# Next we fetch the network interface names for the public and private addresses: +# * internal interfaces = those who use a private IPv4 address. +# * external interfaces = those who use a public IPv4 address. +# Finally we configure the IPv4 firewall to: +# * allow anything over the loopback interface. +# * allow anything over internal interfaces. +# * disable anything by default except for specific services on specific ports to/from specific subnets. +# A subnet is specified with mask as [0-9].[0-9].[0-9].[0-9]/[0-9] and may contain one or more IP addresses. +# E.g. 111.111.111.111/32 is the single machine 111.111.111.111 +# and 111.111.111.111/24 is the range from 111.111.111.1 up to and including 111.111.111.254. +# Finally we configure the IPv6 firewall to: +# * allow anything over the loopback interface. +# * disable anything else over any other interface both internal and external. +# + +--- +#- name: 'Install iptables with yum.' +# yum: +# state: latest +# update_cache: yes +# name: +# - 'iptables' +# - 'iptables-services' +# #### ToDo: fail2ban +# notify: restart_iptables +# become: true + +#- name: 'Enable netfilter kernel module for FTP connection tracking.' +# lineinfile: +# path: '/etc/sysconfig/iptables-config' +# regexp: '^IPTABLES_MODULES=' +# line: 'IPTABLES_MODULES="nf_conntrack_ftp"' +# notify: +# - restart_iptables +# - reconfigure_iptables +# become: true + +- name: 'Create lists of private and of public IP addresses.' + set_fact: + private_ip_addresses: "{{ ansible_facts | dict2items + | json_query('[*].value.ipv4.address') + | map('ipaddr', 'private') | list + | reject('==', None) | list }}" + public_ip_addresses: "{{ ansible_facts | dict2items + | json_query('[*].value.ipv4.address') + | map('ipaddr', 'public') | list + | reject('==', None) | list }}" + +- name: 'Create lists of internal and external network interfaces.' + set_fact: + internal_interfaces: "{{ ansible_facts | dict2items + | selectattr('value.ipv4.address', 'defined') + | selectattr('value.ipv4.address', 'in', private_ip_addresses) + | map(attribute='value.device') | list }}" + external_interfaces: "{{ ansible_facts | dict2items + | selectattr('value.ipv4.address', 'defined') + | selectattr('value.ipv4.address', 'in', public_ip_addresses) + | map(attribute='value.device') | list }}" + +- name: 'DEBUG: List discovered IP addressess and network interfaces.' + debug: + msg: | + Private IP addresses: {{ private_ip_addresses }}. + Public IP addresses: {{ public_ip_addresses }}. + Internal interfaces: {{ internal_interfaces }}. + External interfaces: {{ external_interfaces }}. + +- name: 'Deploy firewall configuration script.' + template: + src: 'configure-iptables-firewall.bash.j2' + dest: '/root/configure-iptables-firewall.bash' + owner: root + group: root + mode: 0740 +# notify: reconfigure_iptables + become: true + +# +#- name: 'Configure the firewall service.' +# service: +# name: "{{ item }}" +# state: 'started' +# enabled: 'yes' +# daemon_reload: 'yes' +# with_items: +# - 'iptables' +# - 'ip6tables' +# notify: reconfigure_iptables +# become: true + +#- import_tasks: disable-other-firewalls.yml +# when: iptables_disable_firewalld or iptables_disable_ufw +... + + +# systemctl disable firewalld +# yum install iptables-services +# systemctl enable iptables +# systemctl enable ip6tables +# systemctl start iptables +# systemctl start ip6tables +# /root/firewall/firewall.sh \ No newline at end of file diff --git a/roles/iptables/tasks/main.yml.disabled b/roles/iptables/tasks/main.yml.disabled new file mode 100755 index 000000000..462229346 --- /dev/null +++ b/roles/iptables/tasks/main.yml.disabled @@ -0,0 +1,45 @@ +--- +- name: Ensure iptables is present. + package: name=iptables state=present + +#- name: Flush iptables the first time playbook runs. +# command: > +# iptables -F +# creates=/etc/firewall.bash + +- name: Copy firewall configuration script into place. + template: + src: firewall.bash.j2 + dest: /etc/firewall.bash + owner: root + group: root + mode: 0744 + notify: restart firewall + +- name: Copy firewall init script into place (for System V init systems). + template: + src: firewall.init.j2 + dest: /etc/init.d/firewall + owner: root + group: root + mode: 0755 + when: "ansible_service_mgr != 'systemd'" + +- name: Copy firewall systemd unit file into place (for systemd systems). + template: + src: firewall.unit.j2 + dest: /etc/systemd/system/firewall.service + owner: root + group: root + mode: 0644 + when: "ansible_service_mgr == 'systemd'" + +- name: Configure the firewall service. + service: + name: firewall + state: "{{ iptables_state }}" + enabled: "{{ iptables_enabled }}" + +- import_tasks: disable-other-firewalls.yml + when: iptables_disable_firewalld or iptables_disable_ufw +... diff --git a/roles/iptables/templates/configure-iptables-firewall.bash.j2 b/roles/iptables/templates/configure-iptables-firewall.bash.j2 new file mode 100755 index 000000000..02b97ac47 --- /dev/null +++ b/roles/iptables/templates/configure-iptables-firewall.bash.j2 @@ -0,0 +1,430 @@ +#!/bin/bash + +# +# Script to configure an iptables based firewall using template from Ansible playbook role. +# +# This file must be located in /etc/firewall.bash +# +# Common port reference: +# 22: SSH +# 25: SMTP +# 80: HTTP +# 123: NTP +# 443: HTTPS + +# +# Bash sanity. +# +set -u + +# +# Hosts and interfaces. +# +TARGET_SERVER='{{ ansible_hostname }}' +declare -a INTERNAL_INTERFACES=( +{% for internal_interface in internal_interfaces %} + '{{ internal_interface }}' +{% endfor %} +) +declare -a EXTERNAL_INTERFACES=( +{% for external_interface in external_interfaces %} + '{{ external_interface }}' +{% endfor %} +) + +# +# Network addresses and ranges. +# +LOOPBACK='127.0.0.0/8' + +declare -a ALLOW_ICMP_INBOUND=( +{% for ip in iptables_allow_icmp_inbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_SSH_INBOUND=( +{% for ip in iptables_allow_ssh_inbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_SSH_OUTBOUND=( +{% for ip in iptables_allow_ssh_outbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_EBI_MYSQL_OUTBOUND=( +{% for ip in iptables_allow_ebi_mysql_outbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_FTP_OUTBOUND=( +{% for ip in iptables_allow_ftp_outbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_ASPERA_OUTBOUND=( +{% for ip in iptables_allow_aspera_outbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) +declare -a ALLOW_GLOBUS_OUTBOUND=( +{% for ip in iptables_allow_globus_outbound %} + '{{ ip.addr }}' # {{ ip.desc }} +{% endfor %} +) + +# +## +### Main. +## +# + +# +# Get (short) name of the server where this script is executed. +# * Ignore the domain. +# * Remove any -mgmt suffixes +# +SERVER_NAME="$(hostname -s)" +SERVER_NAME="$(echo ${SERVER_NAME} | sed 's/-mgmt//')" + +# +# Check if we have a config valid for this server. +# +if [[ "${SERVER_NAME}" == "${TARGET_SERVER}" ]]; then + echo "INFO: Hostname check passed. Will configure iptables firewall..." +else + echo "ERROR: This config file is for \"${TARGET_SERVER}\", but this is \"${SERVER_NAME}\"." + echo 'FATAL: Cannot configure firewall on this server.' + exit 1 +fi + +# +## +### Kernel tweaks. +## +# + + +############ ToDo ##################################### +# Kernel tweaks old and not yet migrated to new syntax. +# +echo '0' > /proc/sys/net/ipv4/ip_forward +####################################################### + +# +# Prevent SYNC-floods. +# +echo '1' > /proc/sys/net/ipv4/tcp_syncookies # confusing name, but applies to IP version 6 as well. + +# +# Limit response to ICMP packets. +# +echo '1' > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts +echo '1' > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses + +# +# Disable redirects and forwarding, which are only required on routers. +# +echo '0' > /proc/sys/net/ipv4/conf/all/accept_redirects +echo '0' > /proc/sys/net/ipv6/conf/all/accept_redirects +echo '0' > /proc/sys/net/ipv4/conf/all/forwarding +echo '0' > /proc/sys/net/ipv6/conf/all/forwarding + +# +# Disable logging of spoofed packets, source routed packets, +# and redirect packets to prevent flooding the logs. +# +echo '0' > /proc/sys/net/ipv4/conf/all/log_martians + +# +# Disable source routed packets, which should only be required for debugging network issues. +# +echo '0' > /proc/sys/net/ipv4/conf/all/accept_source_route +echo '0' > /proc/sys/net/ipv6/conf/all/accept_source_route + +# +# Enable IP spoofing protection. +# +echo '1' > /proc/sys/net/ipv4/conf/all/rp_filter + +# +## +### IPv6: disable all and log. +## +# + +# +# Flush and delete existing rules. +# +ip6tables -F +ip6tables -X + +# +# Set the default policies to drop everything. +# +ip6tables -P INPUT DROP +ip6tables -P OUTPUT DROP +ip6tables -P FORWARD DROP + +# +# We must accept IPv6 traffic on the loopback interface to prevent tests from failing +# during installation of verious software packages with network functionality. +# +ip6tables -A INPUT -i lo -j ACCEPT +ip6tables -A OUTPUT -o lo -j ACCEPT + +# +## +### IPv4. +## +# + +# +# Initialize and flush everything to start with a clean slate. +# +iptables -F +iptables -X +iptables -t nat -F +iptables -t nat -X +iptables -t mangle -F +iptables -t mangle -X +iptables -t raw -F +iptables -t raw -X +iptables -t security -F +iptables -t security -X +iptables -Z + +# +# Create custom chain for LOGDROP. +# +iptables -N LOGDROP +iptables -A LOGDROP -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix 'Dropped by iptables firewall: ' +iptables -A LOGDROP -j DROP + +# +# Config default policies to drop. +# +# Log EVERYTHING (ONLY for Debug). +# iptables -A INPUT -j LOG +{% if iptables_log_dropped_packets %} +iptables -P INPUT LOGDROP +{% else %} +iptables -P INPUT DROP +{% endif %} +iptables -P OUTPUT DROP +iptables -P FORWARD DROP + +# +# Refuse loopback packets incoming from external interface. +# +for EXT_INTERFACE in "${EXTERNAL_INTERFACES[@]}"; do + iptables -A INPUT -i "${EXT_INTERFACE}" -d "${LOOPBACK}" -j DROP +done + +# +# Allow loopback. +# +iptables -A INPUT -i lo -j ACCEPT +iptables -A OUTPUT -o lo -j ACCEPT + +# +# Allow anything over internal interfaces. +# +for INT_INTERFACE in "${INTERNAL_INTERFACES[@]}"; do + iptables -A INPUT -i "${INT_INTERFACE}" -j ACCEPT + iptables -A OUTPUT -o "${INT_INTERFACE}" -j ACCEPT +done + +############################################################################################################################################### + + + +# Forwarded ports. +{# Add a rule for each forwarded port #} +{% for forwarded_port in iptables_forwarded_tcp_ports %} +iptables -t nat -I PREROUTING -p tcp --dport {{ forwarded_port.src }} -j REDIRECT --to-port {{ forwarded_port.dest }} +iptables -t nat -I OUTPUT -p tcp -o lo --dport {{ forwarded_port.src }} -j REDIRECT --to-port {{ forwarded_port.dest }} +{% endfor %} +{% for forwarded_port in iptables_forwarded_udp_ports %} +iptables -t nat -I PREROUTING -p udp --dport {{ forwarded_port.src }} -j REDIRECT --to-port {{ forwarded_port.dest }} +iptables -t nat -I OUTPUT -p udp -o lo --dport {{ forwarded_port.src }} -j REDIRECT --to-port {{ forwarded_port.dest }} +{% endfor %} + +# +# Allow established connections: +# +iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT + + + + +############################################################################################################################3 + +# +# Allow all outbound ICMP. +# Allow limited inbound ICMP: +# Type 0 Echo Reply (a.k.a. pong) must be RELATED. +# Type 8 Echo Request (a.k.a. ping) +# Type 3 Destination Unreachable: +# Unable to deliver the datagram to the specified network, host, protocal, or port. +# Also sent if the datagram needs to be fragmented and the the Don't Fragment flag is on. +# Required for Path-MTU Discovery and to prevent a PMTU "black hole". +# Type 11 Time Exceeded: +# The Time To Live (TTL) for the datagram has been exceeded. +# Required for traceroute. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p icmp -m state --state ESTABLISHED,RELATED -j ACCEPT +for IP_ADDRESS in ${ALLOW_ICMP_INBOUND[@]}; do + iptables -A INPUT -i ${INTERFACE_EXT} -p icmp --icmp-type 3 -s ${IP_ADDRESS} -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT + iptables -A INPUT -i ${INTERFACE_EXT} -p icmp --icmp-type 8 -s ${IP_ADDRESS} -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT + iptables -A INPUT -i ${INTERFACE_EXT} -p icmp --icmp-type 11 -s ${IP_ADDRESS} -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT +done + +# +# Allow outbound NTP. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p udp --dport 123 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p udp --sport 123 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow outbound SMTP. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -m multiport --dport 25,587 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp -m multiport --sport 25,587 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow outbound DNS. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p udp --sport 53 -m state --state ESTABLISHED -j ACCEPT +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 53 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow outbound HTTP. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow outbound HTTPS. +# +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow SSH inbound and outbound. +# +for IP_ADDRESS in ${ALLOW_SSH_INBOUND[@]}; do + iptables -A INPUT -i ${INTERFACE_EXT} -p tcp -s ${IP_ADDRESS} --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT +done +for IP_ADDRESS in ${ALLOW_SSH_OUTBOUND[@]}; do + iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${IP_ADDRESS} --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT +done +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow MySQL outbound. +# +# Required a.o. for conncections to the public Ensembl databases via the Ensembl Perl API +# +for IP_ADDRESS in ${ALLOW_EBI_MYSQL_OUTBOUND[@]}; do + iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${IP_ADDRESS} -m multiport --dports 3306,5306,5316 -m state --state NEW,ESTABLISHED -j ACCEPT +done +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp -m multiport --sports 3306,5306,5316 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow FTP outbound. +# +# We need a default and an additional connection tracking kernel module +# in order to track FTP connections and know what is a RELATED connection. +# The extra module must be added to /etc/sysconfig/iptables-config +# IPTABLES_MODULES="nf_conntrack_ftp" +# +/sbin/modprobe nf_conntrack # Default module: should already be present, just checking here. +/sbin/modprobe nf_conntrack_ftp # Extra module: should be added to /etc/sysconfig/iptables-config, just checking here. +# +# Firstly, allow FTP control initiated by the client. +# +for IP_ADDRESS in ${ALLOW_FTP_OUTBOUND[@]}; do + iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${IP_ADDRESS} --dport 21 -m state --state NEW,ESTABLISHED -j ACCEPT +done +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 21 -m state --state ESTABLISHED -j ACCEPT +# +# Secondly, allow FTP data connections. +# * For Active Mode FTP the client must accept RELATED connections from the server on port 20 +# to the client on a port number negotiated in the FTP control connection. +# * For Passive Mode FTP the client starts a RELATED connection from a random own high port number +# to the server's fixed high port number negotiated in the FTP control connection. +# * For both Active and Passive Mode FTP, the nf_conntrack_ftp kernel module is required at the FTP client +# to pick up the negotiated port number from the FTP control packet payloads. +# +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 20 -m state --state ESTABLISHED,RELATED -j ACCEPT # Active Mode +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --dport 20 -m state --state ESTABLISHED -j ACCEPT # Active Mode +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp --sport 1024: --dport 1024: -m state --state RELATED,ESTABLISHED -j ACCEPT # Passive Mode +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp --sport 1024: --dport 1024: -m state --state ESTABLISHED -j ACCEPT # Passive Mode + +# +# Allow Aspera outbound. +# +# Aspera requires (most common configs): +# * SSH on either TCP port 22 or TCP 33001 for control and +# * Data stream on at least UDP port 33001. +# In case the server OS doesn't allow UDP port sharing a range of UDP ports is used +# where the number of ports determines the max number of concurrent connections/clients. +# +for IP_ADDRESS in ${ALLOW_ASPERA_OUTBOUND[@]}; do + iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${IP_ADDRESS} -m multiport --dports 22,33001 -m state --state NEW,ESTABLISHED -j ACCEPT + iptables -A OUTPUT -o ${INTERFACE_EXT} -p udp -d ${IP_ADDRESS} -m multiport --dports 33001:33100 -m state --state NEW,ESTABLISHED -j ACCEPT +done +iptables -A INPUT -i ${INTERFACE_EXT} -p tcp -m multiport --sports 22,33001 -m state --state ESTABLISHED -j ACCEPT +iptables -A INPUT -i ${INTERFACE_EXT} -p udp -m multiport --sports 33001:33100 -m state --state ESTABLISHED -j ACCEPT + +# +# Allow Globus outbound. +# +# Globus requires (most common configs) for "broker" service: +# * TCP port 2223 outbound for control channel with the Globus Transfer service and for obtaining certificates during initial setup. +# Normally this will be the *.globus.org subnet. +# * UDP port 19302 outbound for connecting to STUN server when setting up a session with another Globus Connect Personal endpoint. +# Normally this will be the stun.l.google.com Google STUN server. +# For each endpoint: +# * TCP ports 50000-51000 outbound for data channel for transfers with Globus Connect Server endpoints. +# * UDP ports 32768-65535 outbound for data channel for transfers with other Globus Connect Personal endpoints. + +# ToDo: GLOBUS_ORG_NET and GOOGLE_STUN +iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${GLOBUS_ORG_NET} --dport 2223 -m state --state NEW,ESTABLISHED -j ACCEPT +iptables -A OUTPUT -o ${INTERFACE_EXT} -p udp -d ${GOOGLE_STUN} --dport 19302 -m state --state NEW,ESTABLISHED -j ACCEPT +for IP_ADDRESS in ${ALLOW_GLOBUS_OUTBOUND[@]}; do + iptables -A OUTPUT -o ${INTERFACE_EXT} -p tcp -d ${IP_ADDRESS} -m multiport --dports 50000:51000 -m state --state NEW,ESTABLISHED -j ACCEPT + iptables -A OUTPUT -o ${INTERFACE_EXT} -p udp -d ${IP_ADDRESS} -m multiport --dports 32678:65535 -m state --state NEW,ESTABLISHED -j ACCEPT +done + +# +# (Re)start fail2ban, so it will (re)create a custom f2b-SSH iptables chain. +# +#service fail2ban restart + +# +# List the rules: +# +echo '#' +echo '##' +echo '### iptables --list -n ###' +echo '##' +echo '#' +iptables --list -n +echo '#' +echo '##' +echo '### ip6tables --list -n ###' +echo '##' +echo '#' +ip6tables --list -n + +# +# Save config for next boot. +# +service iptables save +service ip6tables save diff --git a/roles/iptables/templates/firewall.unit.j2 b/roles/iptables/templates/firewall.unit.j2 new file mode 100755 index 000000000..5165d88ff --- /dev/null +++ b/roles/iptables/templates/firewall.unit.j2 @@ -0,0 +1,12 @@ +[Unit] +Description=Firewall +After=syslog.target network.target + +[Service] +Type=oneshot +ExecStart=/etc/firewall.bash +ExecStop=/sbin/iptables -F +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target diff --git a/roles/ldap/defaults/main.yml b/roles/ldap/defaults/main.yml index e5a9d4ff5..61e9ade5f 100644 --- a/roles/ldap/defaults/main.yml +++ b/roles/ldap/defaults/main.yml @@ -2,8 +2,7 @@ use_ldap: yes # needed for the sshd template ldap_port: 389 ldaps_port: 636 -uri_ldap: '' -uri_ldaps: '' +ldap_uri: '' ldap_base: '' ldap_binddn: '' ... diff --git a/roles/ldap/files/ssh_ldap_wrapper.py b/roles/ldap/files/ssh_ldap_wrapper.py index a5a54c17a..dd8cb8dc0 100755 --- a/roles/ldap/files/ssh_ldap_wrapper.py +++ b/roles/ldap/files/ssh_ldap_wrapper.py @@ -3,15 +3,18 @@ Usage: ssh_ldap_wrapper.py Custom ssh-ldap-wrapper script. -Fetches public keys from LDAP using default ssh-ldap-helper and -Filters the public keys by dropping unsupported key types or short key sizes considered weak. -We accept fixed size ed25519 keys and >= 4096 bits rsa keys. +1. Fetches public keys + * For admin users from local credentials (~/.ssh/authorized_keys). + This ensures the system will be maintainable in case of a lost connection to the ldap. + * For regular users from LDAP using default ssh-ldap-helper. +2. Filters the public keys by dropping unsupported key types or short key sizes considered weak. + We accept fixed size ED25519 keys and >= 4096 bits RSA keys. +3. Optionally and only for non-admin users: prepend ForcedCommand to each public key + to limit what the key pair may be used for. E.g. rsync-only. -Admin users will be sourced from local credentials. This ensures the system will be maintainable in case of a lost connection to the ldap. - -Refactored from a original in bash, which became too obfustcated. """ +import argparse import logging import os.path import sshpubkeys @@ -19,6 +22,7 @@ import sys import yaml + class UserKeys(object): """ Class holding information about a user and her/his keys. @@ -31,10 +35,28 @@ class UserKeys(object): def __init__(self, user: str, admin_gid: int): self.user = user self.admin_gid = admin_gid + # + # Get all public keys either from local authorized_keys files or from an LDAP. + # if self.is_admin(): self.keys = self.local_keys else: self.keys = self.ldap_keys + # + # Filter keys for valid (strong) ones dropping keys based on weak algorithms. + # + self.keys = self.filtered_keys + # + # Optional post processing to restrict SSH options and use ForcedCommands + # * only for regular accounts + # * not for admins to make sure they won't get locked out + # when further processing fails. + # + if self.is_admin(): + # Stop any further processing. + return + if self.is_rsync_only(): + self.keys = self.rsync_only_keys def is_admin(self): """ @@ -50,10 +72,41 @@ def is_admin(self): except subprocess.CalledProcessError as err: logging.error(err) logging.error(err.stderr) - return False + sys.exit(0) return int(gid) == self.admin_gid + def is_rsync_only(self): + """ + It would be best if users get minimal privileges and only receive "full" shell access + if they have certain attributes. This currently does not work, so we have to do it + the other way around: limit users to rsync-only if they have certain attributes. + + Returns: + bool: whether the user is an rsync-only user. + """ + if 'guest' in self.user: + return True + + try: + groups = subprocess.run( + ['id', '-Gn', self.user], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True).stdout + except subprocess.CalledProcessError as err: + logging.error(err) + logging.error(err.stderr) + sys.exit(0) + if 'rsync-only' in str(groups): + return True + elif 'sftp-only' in str(groups): + # This is for backwards compatibility with old account management systems + # from the days we used SFTP as opposed to Rsync servers. + return True + + return False + def is_ok(self, key: str): """ Args: @@ -75,7 +128,7 @@ def is_ok(self, key: str): return False if ssh_key.key_type == b'ssh-rsa' and ssh_key.bits < self.rsa_key_size: logging.error( - "Invalid key: minimum keysize for rsa is {} bits".format( + "Invalid key: minimum key size for RSA is {} bits".format( self.rsa_key_size)) return False elif ssh_key.key_type in (b'ssh-ed25519', b'ssh-rsa'): @@ -93,7 +146,27 @@ def filtered_keys(self): Returns: str: list of keys """ - return '\n'.join(filter(self.is_ok, self.keys.split('\n'))) + if self.keys != '': + return '\n'.join(filter(self.is_ok, self.keys.split('\n'))) + else: + return '' + + @property + def rsync_only_keys(self): + """ + Return keys that are restricted to use for rsync-only. + This is enforced by using a "ForceCommand" prepended to each public key resulting in the following format: + + restrict,command="/bin/rsync --server --daemon --config=/etc/rsyncd.conf ." + + Returns: + str: list of keys prefixed with forced rsync daemon command. + """ + if self.keys != '': + return "\n".join(['restrict,command="/bin/rsync --server --daemon --config=/etc/rsyncd.conf ." {0}'.format(line) + for line in self.keys.split('\n')]) + else: + return '' @property def local_keys(self): @@ -131,8 +204,11 @@ def ldap_keys(self): if __name__ == '__main__': # Log messages will go to sys.stderr. logging.basicConfig(level=logging.INFO) - dirname = os.path.dirname(os.path.abspath(__file__)) - with open(os.path.join(dirname, 'ssh_ldap_wrapper.yml'), 'r') as f: + config_file = os.path.splitext(os.path.abspath(__file__))[0] + '.yml' + with open(os.path.join(config_file), 'r') as f: config = yaml.load(f.read(), Loader=yaml.BaseLoader) - user_keys = UserKeys(sys.argv[1], int(config['admin_gid'])) - print(user_keys.filtered_keys) + parser = argparse.ArgumentParser(description='Fetch public keys for a user.') + parser.add_argument('user') + arguments = parser.parse_args() + user_keys = UserKeys(arguments.user, int(config['admin_gid'])) + print(user_keys.keys) diff --git a/roles/ldap/handlers/main.yml b/roles/ldap/handlers/main.yml index d17a798d7..7b795f8cc 100644 --- a/roles/ldap/handlers/main.yml +++ b/roles/ldap/handlers/main.yml @@ -2,7 +2,7 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # - name: Restart nslcd service. service: diff --git a/roles/ldap/templates/ldap.conf b/roles/ldap/templates/ldap.conf index 7d7792f65..db68b379a 100644 --- a/roles/ldap/templates/ldap.conf +++ b/roles/ldap/templates/ldap.conf @@ -5,7 +5,7 @@ # See ldap.conf(5) for details. # -uri ldap://{{ uri_ldap }} +uri {{ ldap_uri }} base {{ ldap_base }} ssl no tls_cacertdir /etc/openldap/cacerts diff --git a/roles/ldap/templates/nslcd.conf b/roles/ldap/templates/nslcd.conf index 4045bdae0..34a38024c 100644 --- a/roles/ldap/templates/nslcd.conf +++ b/roles/ldap/templates/nslcd.conf @@ -2,7 +2,7 @@ uid nslcd gid ldap ssl no tls_cacertdir /etc/openldap/cacerts -uri ldap://{{ uri_ldap }} +uri {{ ldap_uri }} base {{ ldap_base }} {% if filter_passwd is defined %} filter passwd {{ filter_passwd }} diff --git a/roles/ldap/templates/ssh_ldap_wrapper.yml b/roles/ldap/templates/ssh_ldap_wrapper.yml index 12082f88b..0ef1f8179 100644 --- a/roles/ldap/templates/ssh_ldap_wrapper.yml +++ b/roles/ldap/templates/ssh_ldap_wrapper.yml @@ -1,2 +1,3 @@ --- admin_gid: "{{ auth_groups['admin'].gid }}" +... diff --git a/roles/logins/files/set_quota.sh b/roles/logins/files/set_quota.sh new file mode 100644 index 000000000..8493395d1 --- /dev/null +++ b/roles/logins/files/set_quota.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +# +# Limit disk quota for regular users, which have a home dir in /home/${PAM_USER}/. +# The admin users and root are not affected as they have their home in /admin/${PAM_USER}/ or /root/, respectively. +# This prevents users from accidentally transferring large data sets to the jumphosts with only a small local disk +# as opposed to a cluster machine with large shared storage mounts. Filling up the small OS disk on the jumphost, +# would effectively kill the ability to login for all other users. +# + +homedir="$(getent passwd "${PAM_USER}"| cut -d: -f6)" +if [[ "${homedir}" =~ ^/home/* ]]; then + setquota -u "${PAM_USER}" 512 1024 64 128 -a +fi + +exit 0 \ No newline at end of file diff --git a/roles/logins/handlers/jumphost-quota.yml b/roles/logins/handlers/jumphost-quota.yml new file mode 100644 index 000000000..36e1f40f8 --- /dev/null +++ b/roles/logins/handlers/jumphost-quota.yml @@ -0,0 +1,22 @@ +--- +# +# Important: maintain correct handler order. +# Handlers are executed in the order in which they are defined +# and not in the order in which they are listed in a "notify: handler_name" statement! +# +- name: 'Rebuild GRUB config.' + shell: | + timestamp="$(date '+%Y-%m-%dT%H:%M:%S%z')" + cp -v '/boot/grub2/grub.cfg' "/boot/grub2/grub.cfg.ansible_backup_${timestamp}" + grub2-mkconfig -o '/boot/grub2/grub.cfg' + args: + executable: '/bin/bash' + become: true + listen: rebuild_grub_config + +- name: 'Reboot to activate updated GRUB config.' + reboot: + msg: 'Reboot initiated by Ansible role logins from League of Robots playbook/repo.' + become: true + listen: rebuild_grub_config +... \ No newline at end of file diff --git a/roles/logins/handlers/main.yml b/roles/logins/handlers/main.yml index 96426abfc..eb6695662 100644 --- a/roles/logins/handlers/main.yml +++ b/roles/logins/handlers/main.yml @@ -2,10 +2,10 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # -- name: Run authconfig update. - shell: "authconfig --enablemkhomedir --update" +- name: 'Run authconfig update.' + command: "authconfig --enablemkhomedir --update" become: true listen: authconfig_update @@ -17,9 +17,9 @@ # https://bugzilla.redhat.com/show_bug.cgi?id=1532105 # Workaround for now is to always restart systemd-logind after DBus is restarted. # -- name: Restart oddjobd service and its dependencies. +- name: 'Restart oddjobd service and its dependencies.' service: - name: "{{item}}" + name: "{{ item }}" state: restarted with_items: - dbus @@ -27,4 +27,7 @@ - oddjobd become: true listen: restart_oddjobd + +- name: 'Set tiny quota for regular users on jumphosts.' + import_tasks: "{{ playbook_dir }}/roles/logins/handlers/jumphost-quota.yml" ... diff --git a/roles/logins/tasks/jumphost-quota.yml b/roles/logins/tasks/jumphost-quota.yml new file mode 100644 index 000000000..fbcb1d527 --- /dev/null +++ b/roles/logins/tasks/jumphost-quota.yml @@ -0,0 +1,79 @@ +# +# (Re)mount file systems with quota support. +# +--- +# +# Adding quota options to /etc/fstab does noting for an XFS root partition, +# but we add them anyway to get /etc/fstab in sync with what the mount or/proc/mounts will report. +# +- name: 'Enable user quota for XFS / in /etc/fstab.' + lineinfile: + path: '/etc/fstab' + backup: yes + backrefs: yes + regexp: '^(\S+\s+{{ item.mountpoint }}\s+\S+\s+)(?!(?:\S*,)?{{ item.mountoption }}(?:,\S*)?\s+)(\S+)(\s+.+)$' + line: '\1{{ item.mountoption }},\2\3' + with_items: + - mountpoint: '/' + mountoption: 'uquota' + when: inventory_hostname in groups['jumphost'] + register: fstab_status + become: true + +# +# Cannot simply remount an XFS file system to ebnable quota: +# Quota must be enabled at boot. +# +- name: 'Checking if we need to update GRUB cmdline.' + shell: "grep 'GRUB_CMDLINE_LINUX=.*rootflags.*' /etc/default/grub" + register: grub_config_grep + changed_when: false + failed_when: false + +- name: 'Enable user quota for XFS / in GRUB config.' + lineinfile: + path: '/etc/default/grub' + backup: yes + backrefs: yes + regexp: '^(GRUB_CMDLINE_LINUX=\")([^\"]*)(\s+rootflags=(\S+))([^\"]*\")$' + line: '\1\2 rootflags=uquota,\4\5' + when: inventory_hostname in groups['jumphost'] and ('rootflags' in grub_config_grep.stdout) and ('uquota' not in grub_config_grep.stdout) + notify: rebuild_grub_config + become: true + +- name: 'Enable user quota for XFS / in GRUB config.' + lineinfile: + path: '/etc/default/grub' + backup: yes + backrefs: yes + regexp: '^(GRUB_CMDLINE_LINUX=\")([^\"]*)(\")$' + line: '\1\2 rootflags=uquota\3' + when: inventory_hostname in groups['jumphost'] and ('rootflags' not in grub_config_grep.stdout) + notify: rebuild_grub_config + become: true + +# +# This depends on a correctly configured pam-script PAM plugin. +# +- name: 'Install set_quota.sh script.' + copy: + src: 'set_quota.sh' + dest: '/etc/pam-script.d/set_quota.sh' + owner: root + group: root + mode: '0744' + when: inventory_hostname in groups['jumphost'] + become: true + +- name: 'Enable set_quota.sh script for certain PAM states.' + file: + src: "{{ item }}" + dest: "/etc/pam-script.d/{{ item }}_ses_open" + owner: root + group: root + state: link + with_items: + - set_quota.sh + when: inventory_hostname in groups['jumphost'] + become: true +... \ No newline at end of file diff --git a/roles/logins/tasks/main.yml b/roles/logins/tasks/main.yml index 904f65774..b8849cc0c 100644 --- a/roles/logins/tasks/main.yml +++ b/roles/logins/tasks/main.yml @@ -1,20 +1,20 @@ --- -- name: Create cron job to restart systemd-logind regularly as workaround for known DBus bug. +- name: 'Create cron job to restart systemd-logind regularly as workaround for known DBus bug.' cron: # # Silly workaround for bug in interaction dbus <-> logind # Need DBus 1.11.10 for a fix, but CentOS 7.6 is stuck on dbus 1.10.24. # - name: Restart systemd-logind - minute: "/10" + name: 'Restart systemd-logind' + minute: '/10' user: root - job: /bin/systemctl restart systemd-logind + job: '/bin/systemctl restart systemd-logind' cron_file: restart_logind become: true -- name: Install yum dependencies. +- name: 'Install yum dependencies.' yum: - state: latest + state: 'latest' update_cache: yes name: - pam_script @@ -24,10 +24,10 @@ - restart_oddjobd become: true -- name: Deploy password-auth-ac for PAM. +- name: 'Deploy password-auth-ac for PAM.' copy: - src: password-auth-ac - dest: /etc/pam.d/password-auth-ac + src: 'password-auth-ac' + dest: '/etc/pam.d/password-auth-ac' owner: root group: root mode: '0600' @@ -36,9 +36,9 @@ - restart_oddjobd become: true -- name: Enable pam_script. +- name: 'Enable pam_script.' file: - src: pam_script + src: 'pam_script' dest: "/etc/{{ item }}" owner: root group: root @@ -51,9 +51,9 @@ - pam_script_ses_open become: true -- name: Create /etc/pam-script.d/ dir. +- name: 'Create /etc/pam-script.d/ dir.' file: - name: /etc/pam-script.d + name: '/etc/pam-script.d' state: directory owner: root group: root @@ -71,8 +71,8 @@ # #- name: Install login_checks.sh script. # copy: -# src: login_checks.sh -# dest: /etc/pam-script.d/login_checks.sh +# src: 'login_checks.sh' +# dest: '/etc/pam-script.d/login_checks.sh' # owner: root # group: root # mode: '0755' @@ -81,30 +81,37 @@ # #- name: Enable login_checks.sh script for ses_open. # file: -# src: login_checks.sh -# dest: "/etc/pam-script.d/{{ item }}" +# src: "{{ item }}" +# dest: "/etc/pam-script.d/{{ item }}_ses_open" # owner: root # group: root # state: link # with_items: -# - login_checks.sh_ses_open +# - login_checks.sh # when: inventory_hostname in groups['cluster'] # become: true # -- name: Disable login_checks.sh script as we use the Slurm job_submit.lua plugin now. +- name: 'Disable login_checks.sh script as we use the Slurm job_submit.lua plugin now.' file: - path: "{{ item }}" + path: "/etc/pam-script.d/{{ item }}" state: absent with_items: - login_checks.sh_ses_open - login_checks.sh become: true -- name: Enable services. +- name: 'Set tiny quota for regular users on jumphosts.' + include_tasks: + file: "{{ playbook_dir }}/roles/logins/tasks/jumphost-quota.yml" + when: + - inventory_hostname in groups['jumphost'] | default([]) + +- name: 'Enable services.' systemd: name: "{{ item }}" enabled: yes state: started + daemon_reload: yes with_items: - dbus.service - oddjobd.service diff --git a/roles/mariadb/files/innodb.cnf b/roles/mariadb/files/innodb.cnf new file mode 100644 index 000000000..ec1aef869 --- /dev/null +++ b/roles/mariadb/files/innodb.cnf @@ -0,0 +1,12 @@ +[mysqld] +character_set_server = utf8mb4 +collation_server = utf8mb4_unicode_ci +innodb_buffer_pool_size = 1024M +innodb_file_per_table = TRUE +innodb_log_file_size = 64M +innodb_lock_wait_timeout = 900 +max_allowed_packet = 16M +max_connections = 256 +query_cache_limit = 128K +query_cache_size = 64M +skip_name_resolve = TRUE diff --git a/roles/mariadb/files/my.cnf b/roles/mariadb/files/my.cnf deleted file mode 100644 index b14ee66f0..000000000 --- a/roles/mariadb/files/my.cnf +++ /dev/null @@ -1,68 +0,0 @@ -[client] -port = 3306 -socket = /var/run/mysqld/mysqld.sock - - -[mysqld_safe] -socket = /var/run/mysqld/mysqld.sock -nice = 0 - -[mysqld] -skip-host-cache -skip-name-resolve -pid-file = /var/run/mysqld/mysqld.pid -socket = /var/run/mysqld/mysqld.sock -port = 3306 -basedir = /usr -datadir = /var/lib/mysql -tmpdir = /tmp -lc_messages_dir = /usr/share/mysql -lc_messages = en_US -skip-external-locking -connect_timeout = 5 -wait_timeout = 600 -max_allowed_packet = 16M -thread_cache_size = 128 -sort_buffer_size = 4M -bulk_insert_buffer_size = 16M -tmp_table_size = 32M -max_heap_table_size = 32M -myisam_recover_options = BACKUP -key_buffer_size = 128M -table_open_cache = 400 -myisam_sort_buffer_size = 512M -concurrent_insert = 2 -read_buffer_size = 2M -read_rnd_buffer_size = 1M -query_cache_limit = 128K -query_cache_size = 64M -slow_query_log_file = /var/log/mysql/mariadb-slow.log -long_query_time = 10 - -expire_logs_days = 10 -max_binlog_size = 100M -default_storage_engine = InnoDB -innodb_buffer_pool_size = 256M -innodb_log_buffer_size = 16M -innodb_file_per_table = 1 -innodb_open_files = 400 -innodb_io_capacity = 400 -innodb_flush_method = O_DIRECT - -default-storage-engine = innodb -max_connections = 4096 -collation-server = utf8_general_ci -character-set-server = utf8 - -[galera] - -[mysqldump] -quick -quote-names -max_allowed_packet = 16M - -[mysql] - -[isamchk] -key_buffer = 16M - diff --git a/roles/mariadb/handlers/main.yml b/roles/mariadb/handlers/main.yml index cfb0800f4..27a5a33ac 100644 --- a/roles/mariadb/handlers/main.yml +++ b/roles/mariadb/handlers/main.yml @@ -2,18 +2,27 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # # Restart before reload: an reload after a restart may be redundant but should not fail, # but the other way around may fail when the impact of changes was too large for a reload. # # Currently restart only: mysql.service does not support reloads yet :(. # -- name: Restart MariaDB/MySQL service. +- name: 'Restart MariaDB/MySQL service.' systemd: - name: 'mysql.service' - state: restarted + name: 'mariadb.service' + state: 'restarted' daemon_reload: yes become: true - listen: restart_mysql -... \ No newline at end of file + listen: restart_mariadb + +- name: 'Start MariaDB/MySQL service.' + systemd: + name: 'mariadb.service' + state: 'started' + enabled: yes + daemon_reload: yes + become: true + listen: start_mariadb +... diff --git a/roles/mariadb/tasks/main.yml b/roles/mariadb/tasks/main.yml index eab3ebd8b..5fd4ce7be 100644 --- a/roles/mariadb/tasks/main.yml +++ b/roles/mariadb/tasks/main.yml @@ -1,93 +1,97 @@ -# Install a docker based MariaDB. +# Install MariaDB. --- -- name: Make MariaDB/MySQL settings volume. - file: - path: "{{ item }}" - state: directory - mode: '0777' # This is broken, insecure, oops. - owner: root - group: root - with_items: - - '/srv/mariadb/lib/mysql' - - '/srv/mariadb/etc/mysql' - - '/srv/mariadb/etc/mysql/conf.d' +- name: 'Install mariadb-server.' + yum: + name: + - 'mariadb-server' + state: 'latest' + update_cache: yes notify: - - restart_mysql + - 'restart_mariadb' become: true -- name: Deploy MariaDB/MySQL config file. - copy: - src: 'files/my.cnf' - dest: '/srv/mariadb/etc/mysql/conf.d/my.cnf' - mode: '0660' - owner: root - group: root - notify: - - restart_mysql +- name: 'Create/check dirs used by MariaDB/MySQL.' + file: + path: "{{ item.path }}" + state: 'directory' + mode: "{{ item.mode }}" + owner: "{{ item.og }}" + group: "{{ item.og }}" + with_items: + - path: '/var/lib/mysql' + mode: '0755' + og: 'mysql' + - path: '/etc/my.cnf.d/' + mode: '0755' + og: 'root' become: true -- name: Deploy MariaDB/MySQL galara.cnf on node if we have at least three nodes. +- name: 'Deploy MariaDB/MySQL galara.cnf on node if we have at least three nodes.' template: - src: files/galera.cnf - dest: /srv/mariadb/etc/mysql/conf.d/galera.cnf - mode: '0660' - owner: root - group: root + src: 'files/galera.cnf' + dest: '/etc/my.cnf.d/galera.cnf' + mode: '0644' + owner: 'root' + group: 'root' notify: - - restart_mysql + - restart_mariadb when: "'databases' in group_names and groups['databases'] | length >= 3" become: true - # This mimics galera_new_cluster.sh -- name: Initialize a new MariaDB/MySQL cluster. - block: - - set_fact: - mariadb_args: "--wsrep-new-cluster" - - template: - src: 'templates/mysql.service' - dest: '/etc/systemd/system/mysql.service' - mode: '0644' - owner: root - group: root - notify: - - restart_mysql - - systemd: - name: mysql.service - state: started - enabled: yes - daemon_reload: yes - when: "'databases' in group_names and groups['databases'] \ - | length >= 3 and ansible_hostname == hostname_node0" +- name: 'Deploy MariaDB/MySQL config files.' + copy: + src: "files/{{ item }}" + dest: "/etc/my.cnf.d/{{ item }}" + mode: '0644' + owner: 'root' + group: 'root' + with_items: + - 'innodb.cnf' + notify: + - 'restart_mariadb' become: true -- name: Install MariaDB/MySQL service file. - block: - - set_fact: - mariadb_args: "" - - template: - src: 'templates/mysql.service' - dest: '/etc/systemd/system/mysql.service' - mode: '0644' - owner: root - group: root - notify: - - restart_mysql +- name: 'Make sure the MariaDB/MySQL service is enabled and started.' + systemd: + name: 'mariadb.service' + state: 'started' + enabled: 'yes' + daemon_reload: 'yes' become: true -- name: Give the master node some time to initialize the MariaDB/MySQL cluster. +# This command will fail when the root password was set previously +- name: 'Check if MariaDB/MySQL root password is set.' + command: + cmd: mysqladmin -u root status + changed_when: false + failed_when: false + register: root_pwd_check + +- name: 'Set MariaDB/MySQL root user password.' + mysql_user: + name: 'root' + host: 'localhost' + password: "{{ MYSQL_ROOT_PASSWORD }}" + check_implicit_admin: true + login_unix_socket: "/var/lib/mysql/mysql.sock" + login_user: 'root' + login_password: '' + state: 'present' + when: root_pwd_check.rc == 0 + +# +# Must flush handlers explicitly, because this role is included as dependency for Slurm. +# Without explicit flush the handlers will run after the slurm rol tasks, +# which is too late resulting in Slurm unable to work with the Slurm Accounting DB. +# +- meta: flush_handlers + +- name: 'Give the MariaDB/MySQL master node some time to initialize the database cluster.' command: bash -c "sleep 60" when: "'databases' in group_names and groups['databases'] \ | length >= 3" -- name: Make sure MariaDB/MySQL service is started. - systemd: - name: mysql.service - state: started - enabled: yes - daemon_reload: yes - become: true - -- name: Give the MariaDB/MySQL cluster some time to initialize replication. +- name: 'Give the MariaDB/MySQL cluster some time to initialize replication.' command: bash -c "sleep 60 && systemctl daemon-reload" when: "'databases' in group_names and groups['databases'] | length >= 3" become: true diff --git a/roles/mariadb/files/galera.cnf b/roles/mariadb/templates/galera.cnf similarity index 100% rename from roles/mariadb/files/galera.cnf rename to roles/mariadb/templates/galera.cnf diff --git a/roles/mariadb/templates/mysql.service b/roles/mariadb/templates/mysql.service deleted file mode 100644 index 2b6410a8b..000000000 --- a/roles/mariadb/templates/mysql.service +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=Mariadb Container -After=docker.service -Requires=docker.service - -[Service] -TimeoutStartSec=0 -Restart=always -ExecStartPre=/bin/bash -c "/usr/bin/docker rm -f %n || true" -ExecStartPre=/usr/bin/docker pull mariadb:10.2 -ExecStart=/usr/bin/docker run --name %n \ - --network host \ - -v /srv/mariadb/lib/mysql:/var/lib/mysql \ - -v /srv/mariadb/etc/mysql/conf.d:/etc/mysql/conf.d \ - -e MYSQL_ROOT_PASSWORD={{ MYSQL_ROOT_PASSWORD }} mariadb:10.2 {{ mariadb_args }} - -[Install] -WantedBy=multi-user.target diff --git a/roles/mount-volume/tasks/main.yml b/roles/mount-volume/tasks/main.yml index 2bc84b398..10e94bd55 100644 --- a/roles/mount-volume/tasks/main.yml +++ b/roles/mount-volume/tasks/main.yml @@ -1,45 +1,47 @@ --- -- name: Check the local mount point. - command: mountpoint {{volume_mount_point}} +- name: 'Check the local mount point.' + command: "mountpoint {{ volume_mount_point }}" register: mount_local failed_when: false + changed_when: false -- name: Create an ext4 filesystem on /dev/vdb. +- name: 'Create an ext4 filesystem on /dev/vdb.' filesystem: fstype: 'ext4' dev: '/dev/vdb' when: mount_local.rc == 1 become: true -- name: Mount the volume. +- name: 'Mount the volume.' mount: - path: "{{volume_mount_point}}" + path: "{{ volume_mount_point }}" src: '/dev/vdb' fstype: 'ext4' opts: 'rw,relatime' - state: 'mounted' + state: mounted become: true -- name: Recheck the local mount point. - command: mountpoint {{volume_mount_point}} +- name: 'Recheck the local mount point.' + command: "mountpoint {{ volume_mount_point }}" register: mount_local failed_when: false + changed_when: false -- name: Configure permissions for root. +- name: 'Configure permissions for root.' file: - path: "{{volume_mount_point}}" - state: 'directory' - owner: 'root' - group: 'root' + path: "{{ volume_mount_point }}" + state: directory + owner: root + group: root mode: '0755' when: mount_local.rc == 0 and inventory_hostname in groups['compute-vm'] become: true -- name: Configure permissions for deploy admins. +- name: 'Configure permissions for deploy admins.' file: - path: "{{volume_mount_point}}" - state: 'directory' - owner: 'root' + path: "{{ volume_mount_point }}" + state: directory + owner: root group: "{{ envsync_group }}" mode: '2775' when: mount_local.rc == 0 and inventory_hostname in groups['deploy-admin-interface'] diff --git a/roles/node_exporter/handlers/main.yml b/roles/node_exporter/handlers/main.yml new file mode 100644 index 000000000..66cdb6157 --- /dev/null +++ b/roles/node_exporter/handlers/main.yml @@ -0,0 +1,26 @@ +--- +# +# Important: maintain correct handler order. +# Handlers are executed in the order in which they are defined +# and not in the order in which they are listed in a "notify: handler_name" statement! +# +# Restart before reload: an reload after a restart may be redundant but should not fail, +# but the other way around may fail when the impact of changes was too large for a reload. +# +- name: 'Restart node-exporter service.' + systemd: + name: 'node-exporter.service' + state: restarted + daemon_reload: yes + become: true + listen: restart_node_exporter +# +# Service reloads after restarts. +# +- name: 'Reload node-exporter service.' + systemd: + name: 'node-exporter.service' + state: reloaded + become: true + listen: reload_node_exporter +... \ No newline at end of file diff --git a/roles/node_exporter/tasks/main.yml b/roles/node_exporter/tasks/main.yml index 345e6f50b..75c50699a 100644 --- a/roles/node_exporter/tasks/main.yml +++ b/roles/node_exporter/tasks/main.yml @@ -1,42 +1,38 @@ --- -- file: - path: /usr/local/prometheus +- name: 'Create dir for Prometheus.' + file: + path: '/usr/local/prometheus' state: directory - mode: 0755 + mode: '0755' + notify: restart_node_exporter become: true -- name: Install node exporter +- name: 'Install Prometheus node exporter.' copy: src: "{{ playbook_dir }}/promtools/results/node_exporter" - dest: /usr/local/prometheus/node_exporter - mode: 0755 + dest: '/usr/local/prometheus/node_exporter' + mode: '0755' + notify: restart_node_exporter become: true -- name: Install service files. +- name: 'Install node exporter service file.' template: - src: templates/node-exporter.service - dest: /etc/systemd/system/node-exporter.service - mode: 644 + src: 'templates/node-exporter.service' + dest: '/etc/systemd/system/node-exporter.service' + mode: '0644' owner: root group: root tags: - service-files + notify: restart_node_exporter become: true -- name: install service files - command: systemctl daemon-reload - become: true - -- name: enable service at boot +- name: 'Make sure node exporter service is enabled and started.' systemd: - name: node-exporter + name: 'node-exporter.service' + state: started enabled: yes - become: true - -- name: make sure servcies are started. - systemd: - name: node-exporter.service - state: restarted tags: - start-service - become: true \ No newline at end of file + become: true +... \ No newline at end of file diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen1.png b/roles/online_docs/files/PuTTYgen/PuTTYgen1.png deleted file mode 100755 index 70d19fd2c..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen1.png and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen1.psd b/roles/online_docs/files/PuTTYgen/PuTTYgen1.psd deleted file mode 100644 index a828dcf20..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen1.psd and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen2.png b/roles/online_docs/files/PuTTYgen/PuTTYgen2.png deleted file mode 100755 index f54149bce..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen2.png and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen2.psd b/roles/online_docs/files/PuTTYgen/PuTTYgen2.psd deleted file mode 100644 index 66c35c8dc..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen2.psd and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen3.png b/roles/online_docs/files/PuTTYgen/PuTTYgen3.png deleted file mode 100755 index 7e3bdd934..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen3.png and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen3.psd b/roles/online_docs/files/PuTTYgen/PuTTYgen3.psd deleted file mode 100644 index 0958d4cab..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen3.psd and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen4.png b/roles/online_docs/files/PuTTYgen/PuTTYgen4.png deleted file mode 100755 index 908742bf3..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen4.png and /dev/null differ diff --git a/roles/online_docs/files/PuTTYgen/PuTTYgen4.psd b/roles/online_docs/files/PuTTYgen/PuTTYgen4.psd deleted file mode 100644 index 124fa86ca..000000000 Binary files a/roles/online_docs/files/PuTTYgen/PuTTYgen4.psd and /dev/null differ diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Info.plist b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Info.plist new file mode 100644 index 000000000..c3bc3715a --- /dev/null +++ b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Info.plist @@ -0,0 +1,56 @@ + + + + + CFBundleAllowMixedLocalizations + + CFBundleDevelopmentRegion + English + CFBundleExecutable + applet + CFBundleIconFile + applet + CFBundleIdentifier + com.apple.ScriptEditor.id.mount-cluster-drives + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + mount-cluster-drives + CFBundlePackageType + APPL + CFBundleShortVersionString + 1.0 + CFBundleSignature + aplt + LSMinimumSystemVersionByArchitecture + + x86_64 + 10.6 + + LSRequiresCarbon + + NSAppleEventsUsageDescription + This script needs to control other applications to run. + NSSystemAdministrationUsageDescription + This script needs access to administer this system to run. + WindowState + + bundleDividerCollapsed + + bundlePositionOfDivider + 0.0 + dividerCollapsed + + eventLogLevel + 2 + name + ScriptWindowState + positionOfDivider + 1104 + savedFrame + 402 38 1449 1379 0 0 2560 1417 + selectedTab + log + + + diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/MacOS/applet b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/MacOS/applet new file mode 100755 index 000000000..1ace4cac8 Binary files /dev/null and b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/MacOS/applet differ diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/PkgInfo b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/PkgInfo new file mode 100644 index 000000000..3253614c4 --- /dev/null +++ b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/PkgInfo @@ -0,0 +1 @@ +APPLaplt \ No newline at end of file diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/Scripts/main.scpt b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/Scripts/main.scpt new file mode 100644 index 000000000..2d072a01b Binary files /dev/null and b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/Scripts/main.scpt differ diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.icns b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.icns new file mode 100644 index 000000000..0cdd17086 Binary files /dev/null and b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.icns differ diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.rsrc b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.rsrc new file mode 100644 index 000000000..a37c62da3 Binary files /dev/null and b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/applet.rsrc differ diff --git a/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/description.rtfd/TXT.rtf b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/description.rtfd/TXT.rtf new file mode 100644 index 000000000..a38ebce84 --- /dev/null +++ b/roles/online_docs/files/attachments/mount-cluster-drives.app/Contents/Resources/description.rtfd/TXT.rtf @@ -0,0 +1,5 @@ +{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600 +{\fonttbl} +{\colortbl;\red255\green255\blue255;} +{\*\expandedcolortbl;;} +} \ No newline at end of file diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Info.plist b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Info.plist new file mode 100644 index 000000000..6f3c37e06 --- /dev/null +++ b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Info.plist @@ -0,0 +1,54 @@ + + + + + CFBundleAllowMixedLocalizations + + CFBundleDevelopmentRegion + English + CFBundleExecutable + applet + CFBundleIconFile + applet + CFBundleIdentifier + com.apple.ScriptEditor.id.LaunchShellAsApp + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + LaunchShellAsApp + CFBundlePackageType + APPL + CFBundleShortVersionString + 1.0 + CFBundleSignature + aplt + LSMinimumSystemVersionByArchitecture + + x86_64 + 10.6 + + LSRequiresCarbon + + NSAppleEventsUsageDescription + This script needs to control other applications to run. + WindowState + + bundleDividerCollapsed + + bundlePositionOfDivider + 0.0 + dividerCollapsed + + eventLogLevel + 2 + name + ScriptWindowState + positionOfDivider + 518 + savedFrame + 347 482 1194 769 0 0 2560 1417 + selectedTab + result + + + diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/MacOS/applet b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/MacOS/applet new file mode 100755 index 000000000..1ace4cac8 Binary files /dev/null and b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/MacOS/applet differ diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/PkgInfo b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/PkgInfo new file mode 100644 index 000000000..3253614c4 --- /dev/null +++ b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/PkgInfo @@ -0,0 +1 @@ +APPLaplt \ No newline at end of file diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/Scripts/main.scpt b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/Scripts/main.scpt new file mode 100644 index 000000000..86617e3a2 Binary files /dev/null and b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/Scripts/main.scpt differ diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.icns b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.icns new file mode 100644 index 000000000..0cdd17086 Binary files /dev/null and b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.icns differ diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.rsrc b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.rsrc new file mode 100644 index 000000000..534606f04 Binary files /dev/null and b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/applet.rsrc differ diff --git a/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/description.rtfd/TXT.rtf b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/description.rtfd/TXT.rtf new file mode 100644 index 000000000..a38ebce84 --- /dev/null +++ b/roles/online_docs/files/attachments/ssh-client-config.app/Contents/Resources/description.rtfd/TXT.rtf @@ -0,0 +1,5 @@ +{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600 +{\fonttbl} +{\colortbl;\red255\green255\blue255;} +{\*\expandedcolortbl;;} +} \ No newline at end of file diff --git a/roles/online_docs/files/mkdocs/docs/css/custom.css b/roles/online_docs/files/mkdocs/docs/css/custom.css new file mode 100644 index 000000000..8a03a525f --- /dev/null +++ b/roles/online_docs/files/mkdocs/docs/css/custom.css @@ -0,0 +1,3 @@ +table td > code { + white-space: nowrap; +} \ No newline at end of file diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift1.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift1.png new file mode 100644 index 000000000..ea20aec05 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift1.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift2.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift2.png new file mode 100644 index 000000000..8a8373599 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift2.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift3.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift3.png new file mode 100644 index 000000000..066690e8f Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift3.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift3b.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift3b.png new file mode 100644 index 000000000..e69e8b351 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift3b.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift4.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift4.png new file mode 100644 index 000000000..18909b589 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift4.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ForkLift5.png b/roles/online_docs/files/mkdocs/docs/img/ForkLift5.png new file mode 100644 index 000000000..67e38445b Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ForkLift5.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm1.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm1.png new file mode 100644 index 000000000..fc1edf47b Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm1.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm10.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm10.png new file mode 100644 index 000000000..ee7ae9000 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm10.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm11.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm11.png new file mode 100644 index 000000000..379cbc32b Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm11.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm2.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm2.png new file mode 100644 index 000000000..1c701155e Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm2.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm3.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm3.png new file mode 100644 index 000000000..78435b986 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm3.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm4.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm4.png new file mode 100644 index 000000000..005fb3ae9 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm4.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm5.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm5.png new file mode 100644 index 000000000..98e54affc Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm5.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm6.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm6.png new file mode 100644 index 000000000..5a4cece02 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm6.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm7.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm7.png new file mode 100644 index 000000000..cd76ebb32 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm7.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm8.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm8.png new file mode 100644 index 000000000..4d376aa2e Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm8.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm9a.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9a.png new file mode 100644 index 000000000..4bf4d92e7 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9a.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm9b.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9b.png new file mode 100644 index 000000000..145162a9e Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9b.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/MobaXterm9c.png b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9c.png new file mode 100644 index 000000000..3b2165f40 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/MobaXterm9c.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0b.png b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0b.png new file mode 100644 index 000000000..1488793fa Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0b.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0d.png b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0d.png new file mode 100644 index 000000000..d2cd90153 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-0d.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-1.png b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-1.png new file mode 100644 index 000000000..7447bb7a9 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-1.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-2.png b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-2.png new file mode 100644 index 000000000..12739f648 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/mount-cluster-drives-2.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/osxfuse-1.png b/roles/online_docs/files/mkdocs/docs/img/osxfuse-1.png new file mode 100644 index 000000000..5f137dda4 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/osxfuse-1.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/osxfuse-2.png b/roles/online_docs/files/mkdocs/docs/img/osxfuse-2.png new file mode 100644 index 000000000..4f7dc7332 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/osxfuse-2.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/puttygen1.png b/roles/online_docs/files/mkdocs/docs/img/puttygen1.png deleted file mode 100644 index b9ab45e54..000000000 Binary files a/roles/online_docs/files/mkdocs/docs/img/puttygen1.png and /dev/null differ diff --git a/roles/online_docs/files/mkdocs/docs/img/puttygen2.png b/roles/online_docs/files/mkdocs/docs/img/puttygen2.png deleted file mode 100644 index 83af348dd..000000000 Binary files a/roles/online_docs/files/mkdocs/docs/img/puttygen2.png and /dev/null differ diff --git a/roles/online_docs/files/mkdocs/docs/img/puttygen3.png b/roles/online_docs/files/mkdocs/docs/img/puttygen3.png deleted file mode 100644 index c6d798fda..000000000 Binary files a/roles/online_docs/files/mkdocs/docs/img/puttygen3.png and /dev/null differ diff --git a/roles/online_docs/files/mkdocs/docs/img/puttygen4.png b/roles/online_docs/files/mkdocs/docs/img/puttygen4.png deleted file mode 100644 index abebdf85d..000000000 Binary files a/roles/online_docs/files/mkdocs/docs/img/puttygen4.png and /dev/null differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1-error.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1-error.png new file mode 100644 index 000000000..445a55efb Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1-error.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1.png new file mode 100644 index 000000000..04a3c01c2 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-1.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-2.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-2.png new file mode 100644 index 000000000..d6dc43ac2 Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-2.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-3.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-3.png new file mode 100644 index 000000000..bf238164e Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-3.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-4.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-4.png new file mode 100644 index 000000000..35c13725d Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-4.png differ diff --git a/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-5.png b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-5.png new file mode 100644 index 000000000..1984062fb Binary files /dev/null and b/roles/online_docs/files/mkdocs/docs/img/ssh-client-config-macos-5.png differ diff --git a/roles/online_docs/files/originals/AppleScript/ssh-client-config.scpt b/roles/online_docs/files/originals/AppleScript/ssh-client-config.scpt new file mode 100644 index 000000000..da282735d Binary files /dev/null and b/roles/online_docs/files/originals/AppleScript/ssh-client-config.scpt differ diff --git a/roles/online_docs/files/Flowcharts/cluster.graffle b/roles/online_docs/files/originals/Flowcharts/cluster.graffle similarity index 100% rename from roles/online_docs/files/Flowcharts/cluster.graffle rename to roles/online_docs/files/originals/Flowcharts/cluster.graffle diff --git a/roles/online_docs/files/Flowcharts/cluster.pdf b/roles/online_docs/files/originals/Flowcharts/cluster.pdf similarity index 100% rename from roles/online_docs/files/Flowcharts/cluster.pdf rename to roles/online_docs/files/originals/Flowcharts/cluster.pdf diff --git a/roles/online_docs/files/Flowcharts/data-transfers.pdf b/roles/online_docs/files/originals/Flowcharts/data-transfers.pdf similarity index 100% rename from roles/online_docs/files/Flowcharts/data-transfers.pdf rename to roles/online_docs/files/originals/Flowcharts/data-transfers.pdf diff --git a/roles/online_docs/files/Flowcharts/icons.ai b/roles/online_docs/files/originals/Flowcharts/icons.ai similarity index 100% rename from roles/online_docs/files/Flowcharts/icons.ai rename to roles/online_docs/files/originals/Flowcharts/icons.ai diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift1.png b/roles/online_docs/files/originals/ForkLift/ForkLift1.png new file mode 100644 index 000000000..51d0a346d Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift1.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift1.psd b/roles/online_docs/files/originals/ForkLift/ForkLift1.psd new file mode 100644 index 000000000..9587deaf8 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift1.psd differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift2.png b/roles/online_docs/files/originals/ForkLift/ForkLift2.png new file mode 100644 index 000000000..5b01ca606 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift2.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift2.psd b/roles/online_docs/files/originals/ForkLift/ForkLift2.psd new file mode 100644 index 000000000..006a8be83 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift2.psd differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift3.png b/roles/online_docs/files/originals/ForkLift/ForkLift3.png new file mode 100644 index 000000000..65c67eaa1 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift3.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift3.psd b/roles/online_docs/files/originals/ForkLift/ForkLift3.psd new file mode 100644 index 000000000..650307bd4 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift3.psd differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift3b.png b/roles/online_docs/files/originals/ForkLift/ForkLift3b.png new file mode 100644 index 000000000..22ae5cf6a Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift3b.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift4.png b/roles/online_docs/files/originals/ForkLift/ForkLift4.png new file mode 100644 index 000000000..2d673ad19 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift4.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift4.psd b/roles/online_docs/files/originals/ForkLift/ForkLift4.psd new file mode 100644 index 000000000..dba131ad9 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift4.psd differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift5.png b/roles/online_docs/files/originals/ForkLift/ForkLift5.png new file mode 100644 index 000000000..268da36d2 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift5.png differ diff --git a/roles/online_docs/files/originals/ForkLift/ForkLift5.psd b/roles/online_docs/files/originals/ForkLift/ForkLift5.psd new file mode 100644 index 000000000..47dff2c50 Binary files /dev/null and b/roles/online_docs/files/originals/ForkLift/ForkLift5.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm0.png b/roles/online_docs/files/originals/MobaXterm/MobaXterm0.png new file mode 100644 index 000000000..d1206d4d3 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm0.png differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm1.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm1.PNG new file mode 100644 index 000000000..0dc8ffa23 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm1.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm1.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm1.psd new file mode 100644 index 000000000..367efa729 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm1.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm10.png b/roles/online_docs/files/originals/MobaXterm/MobaXterm10.png new file mode 100644 index 000000000..adb3c7f03 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm10.png differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm10.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm10.psd new file mode 100644 index 000000000..81872fc6d Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm10.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm11.png b/roles/online_docs/files/originals/MobaXterm/MobaXterm11.png new file mode 100644 index 000000000..118831dc8 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm11.png differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm11.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm11.psd new file mode 100644 index 000000000..6aaaa44bc Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm11.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm2.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm2.PNG new file mode 100644 index 000000000..ba6d57607 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm2.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm2.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm2.psd new file mode 100644 index 000000000..4eb3a72c5 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm2.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm3.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm3.PNG new file mode 100644 index 000000000..3b0f95802 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm3.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm3.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm3.psd new file mode 100644 index 000000000..462df0711 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm3.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm4.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm4.PNG new file mode 100644 index 000000000..18384cf89 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm4.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm4.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm4.psd new file mode 100644 index 000000000..0e0d73680 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm4.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm5.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm5.psd new file mode 100644 index 000000000..b51ce5d71 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm5.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm6.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm6.PNG new file mode 100644 index 000000000..c9b8145bf Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm6.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm6.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm6.psd new file mode 100644 index 000000000..731108c70 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm6.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm7.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm7.PNG new file mode 100644 index 000000000..16287c30d Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm7.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm7.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm7.psd new file mode 100644 index 000000000..ee76bf375 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm7.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm8.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm8.PNG new file mode 100644 index 000000000..e3e110625 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm8.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm8.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm8.psd new file mode 100644 index 000000000..08b8375a3 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm8.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.PNG new file mode 100644 index 000000000..af0759a72 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.psd new file mode 100644 index 000000000..d9b018d61 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm9a.psd differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.PNG b/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.PNG new file mode 100644 index 000000000..44849c726 Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.PNG differ diff --git a/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.psd b/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.psd new file mode 100644 index 000000000..8101d440c Binary files /dev/null and b/roles/online_docs/files/originals/MobaXterm/MobaXterm9b.psd differ diff --git a/roles/online_docs/files/Slurm/slurm_qos.graffle b/roles/online_docs/files/originals/Slurm/slurm_qos.graffle similarity index 100% rename from roles/online_docs/files/Slurm/slurm_qos.graffle rename to roles/online_docs/files/originals/Slurm/slurm_qos.graffle diff --git a/roles/online_docs/files/Slurm/slurm_qos.pdf b/roles/online_docs/files/originals/Slurm/slurm_qos.pdf similarity index 100% rename from roles/online_docs/files/Slurm/slurm_qos.pdf rename to roles/online_docs/files/originals/Slurm/slurm_qos.pdf diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0a.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0a.png new file mode 100644 index 000000000..76353bf37 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0a.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.png new file mode 100644 index 000000000..fca3b4a9e Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.psd b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.psd new file mode 100644 index 000000000..65e03dca7 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0b.psd differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0c.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0c.png new file mode 100644 index 000000000..ca76bcb94 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0c.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0d.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0d.png new file mode 100644 index 000000000..d2cd90153 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-0d.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-1.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-1.png new file mode 100644 index 000000000..7447bb7a9 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-1.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.png b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.png new file mode 100644 index 000000000..059ecec5c Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.psd b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.psd new file mode 100644 index 000000000..1dd807a7a Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/mount-cluster-drives-2.psd differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.png b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.png new file mode 100644 index 000000000..ba770934a Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.psd b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.psd new file mode 100644 index 000000000..b90c79077 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-1.psd differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.png b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.png new file mode 100644 index 000000000..8da4f7c02 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.png differ diff --git a/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.psd b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.psd new file mode 100644 index 000000000..0b1cb03e5 Binary files /dev/null and b/roles/online_docs/files/originals/mount-cluster-drives/osxfuse-2.psd differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0a.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0a.png new file mode 100644 index 000000000..006b40955 Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0a.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0b.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0b.png new file mode 100644 index 000000000..d8c2751cc Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-0b.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1-error.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1-error.png new file mode 100644 index 000000000..445a55efb Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1-error.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1.png new file mode 100644 index 000000000..fccb22a0a Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-1.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-2.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-2.png new file mode 100644 index 000000000..57684839f Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-2.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-3.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-3.png new file mode 100644 index 000000000..c14633cb9 Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-3.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-4.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-4.png new file mode 100644 index 000000000..5624b0dad Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-4.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-5.png b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-5.png new file mode 100644 index 000000000..480c2f645 Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config-macos-5.png differ diff --git a/roles/online_docs/files/originals/ssh-client-config/ssh-client-config.scpt b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config.scpt new file mode 100644 index 000000000..da282735d Binary files /dev/null and b/roles/online_docs/files/originals/ssh-client-config/ssh-client-config.scpt differ diff --git a/roles/online_docs/handlers/main.yml b/roles/online_docs/handlers/main.yml index e6b70d1aa..6ff92d6ad 100644 --- a/roles/online_docs/handlers/main.yml +++ b/roles/online_docs/handlers/main.yml @@ -2,18 +2,36 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # -- name: Restart services and their dependencies. +- name: 'Restart services and their dependencies.' service: - name: "{{item}}" + name: "{{ item }}" state: restarted with_items: - - httpd + - 'httpd' become: true - listen: restart_httpd + listen: 'restart_httpd' -- name: (Re)build webpages. +- name: 'Zip the scripted tools from the tmp subfolder and save the archives in the attachments subfolder.' + archive: + path: "/srv/mkdocs/{{ slurm_cluster_name }}/tmp/{{ item.path }}" + dest: "/srv/mkdocs/{{ slurm_cluster_name }}/docs/attachments/{{ item.dest }}" + format: 'zip' + owner: 'root' + group: 'root' + mode: '0640' + with_items: + - path: "ssh-client-config-for-{{ slurm_cluster_name }}.app" + dest: "ssh-client-config-for-{{ slurm_cluster_name }}-macos.zip" + - path: "ssh-client-config-for-{{ slurm_cluster_name }}.bash" + dest: "ssh-client-config-for-{{ slurm_cluster_name }}-linux.zip" + - path: 'mount-cluster-drives.app' + dest: 'mount-cluster-drives-macos.zip' + become: true + listen: 'zip_attachments' + +- name: '(Re)build webpages.' shell: | cd /srv/mkdocs/{{ slurm_cluster_name }}/ umask 0022 @@ -22,5 +40,5 @@ export PATH="${PATH}:/usr/local/bin/" mkdocs build become: true - listen: build_mkdocs + listen: 'build_mkdocs' ... \ No newline at end of file diff --git a/roles/online_docs/tasks/main.yml b/roles/online_docs/tasks/main.yml index 3af69a6fb..9bc721b9c 100644 --- a/roles/online_docs/tasks/main.yml +++ b/roles/online_docs/tasks/main.yml @@ -3,27 +3,35 @@ # --- - name: 'Check if rsync >= 3.1.2 is installed on the control host.' - shell: | - rsync --version 2>&1 | head -n 1 | sed 's|^rsync *version *\([0-9\.]*\).*$|\1|' | tr -d '\n' + shell: + cmd: | + set -o pipefail + rsync --version 2>&1 | head -n 1 | sed 's|^rsync *version *\([0-9\.]*\).*$|\1|' | tr -d '\n' args: warn: no changed_when: false failed_when: false check_mode: no - register: rsync_version - delegate_to: localhost + register: 'rsync_version' + delegate_to: 'localhost' + - name: 'Abort when modern rsync >= 3.1.2 is missing on control host.' debug: msg: "FATAL: Need rsync >= 3.1.2 on the control host, but detected {{ rsync_version.stdout }}." when: 'rsync_version is failed or (rsync_version.stdout is version_compare("3.1.2", operator="<"))' failed_when: 'rsync_version is failed or (rsync_version.stdout is version_compare("3.1.2", operator="<"))' - delegate_to: localhost + delegate_to: 'localhost' + +- name: 'Check OS version of target host.' + fail: + msg: 'This role requires RedHat/CentOS version >= 7.x' + when: ansible_facts['os_family'] != "RedHat" or ansible_facts['distribution_major_version'] <= "6" # # This is currently problematic when # * the default user used to login to the cluster is different from # * the default user used to login on the webserver where the docs are hosted -# as you van specify only one account name with -u on the commandline when running ansible-playbook. +# as you can specify only one account name with -u on the commandline when running ansible-playbook. # - name: 'Gather facts from hosts, so we can use them in the generated docs.' setup: @@ -34,10 +42,12 @@ - "{{ groups['user-interface'] }}" - name: 'Get Slurm version from scontrol on UI.' - shell: | - scontrol version | head -n 1 | sed 's|^slurm *\([0-9\.]*\).*$|\1|' | tr -d '\n' + shell: + cmd: | + set -o pipefail + scontrol version | head -n 1 | sed 's|^slurm *\([0-9\.]*\).*$|\1|' | tr -d '\n' args: - executable: /bin/bash + executable: '/bin/bash' warn: no environment: SOURCE_HPC_ENV: "True" # Required to source our modules.bashrc in non-interactive shells. @@ -45,7 +55,7 @@ changed_when: false failed_when: false check_mode: no - register: slurm_version + register: 'slurm_version' - name: 'Get Lmod version from UI.' # @@ -53,12 +63,14 @@ # Modules based on Lua: Version 6.5.8 2016-09-03 13:41 -05:00 (CDT) # by Robert McLay mclay@tacc.utexas.edu # - shell: | - unset MODULEPATH - source ~/.modulesrc >/dev/null 2>&1 - lmod -v | sed '/^$/d' | sed 's|^.*Version \([0-9\.]*\).*$|\1|' | head -1 + shell: + cmd: | + set -o pipefail + unset MODULEPATH + source ~/.modulesrc >/dev/null 2>&1 + lmod -v | sed '/^$/d' | sed 's|^.*Version \([0-9\.]*\).*$|\1|' | head -1 args: - executable: /bin/bash + executable: '/bin/bash' warn: no environment: SOURCE_HPC_ENV: "True" # Required to source our modules.bashrc in non-interactive shells. @@ -66,73 +78,71 @@ changed_when: false failed_when: false check_mode: no - register: lmod_version - -- name: 'Install libselinux-python.' - yum: - state: latest - update_cache: yes - name: - - libselinux-python - when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] == "6" - become: true + register: 'lmod_version' - name: 'Set selinux in permissive mode.' selinux: - policy: targeted - state: permissive + policy: 'targeted' + state: 'permissive' become: true - name: 'Install EPEL repo and rsync.' yum: - state: latest + state: 'latest' update_cache: yes name: - - epel-release - - rsync + - 'epel-release' + - 'rsync' become: true - name: 'Install webserver and php.' yum: - state: latest + state: 'latest' update_cache: yes name: - - php - - httpd + - 'php' + - 'httpd' notify: - - restart_httpd + - 'restart_httpd' become: true - name: 'Enable webserver.' service: name: "{{ item }}" enabled: yes - state: started + state: 'started' with_items: - - httpd + - 'httpd' become: true -- name: 'Install MkDocs on RedHat 6.x.' - import_tasks: mkdocs-for-redhat6.yml - when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] == "6" - - name: 'Install MkDocs on RedHat 7.x.' - import_tasks: mkdocs-for-redhat7.yml - when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] == "7" + include_tasks: + file: "{{ playbook_dir }}/roles/online_docs/tasks/mkdocs-for-redhat7.yml" + when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] >= "7" -- name: 'Create base directories for MarkDown and HTML files on webserver.' +- name: 'Create base directories for MarkDown and HTML files.' file: - path: "{{ item }}" - state: directory - owner: root - group: root - mode: '0755' + path: "{{ item.path }}" + state: 'directory' + owner: 'root' + group: 'root' + mode: "{{ item.mode }}" with_items: - - "/srv/mkdocs/{{ slurm_cluster_name }}/" - - "/var/www/html/{{ slurm_cluster_name }}/" + - path: "/srv/mkdocs/{{ slurm_cluster_name }}/" + mode: '0750' + - path: "/srv/mkdocs/{{ slurm_cluster_name }}/tmp/" + mode: '0750' + - path: "/srv/mkdocs/{{ slurm_cluster_name }}/docs/" + mode: '0750' + - path: "/srv/mkdocs/{{ slurm_cluster_name }}/docs/attachments" + mode: '0750' + - path: "/var/www/html/{{ slurm_cluster_name }}/" + mode: '0755' + - path: "/var/www/html/{{ slurm_cluster_name }}/attachments/" + mode: '0755' become: true -- name: 'Create static files for MarkDown on webserver.' +- name: 'Create static files for index in document root.' synchronize: src: "{{ playbook_dir }}/roles/online_docs/files/index//./{{ item.src }}" dest: "/var/www/html/" @@ -163,20 +173,20 @@ - src: 'sitemap.php' become: true -- name: 'Create sub directory structure for MarkDown files on webserver.' +- name: 'Create sub directory structure for MarkDown files.' file: path: "/srv/mkdocs/{{ slurm_cluster_name }}/{{ item.path }}" - state: directory - owner: root - group: root - mode: '0700' - with_filetree: 'templates/mkdocs' + state: 'directory' + owner: 'root' + group: 'root' + mode: '0750' + with_filetree: "{{ playbook_dir }}/roles/online_docs/templates/mkdocs" when: item.state == 'directory' notify: - - build_mkdocs + - 'build_mkdocs' become: true -- name: 'Create static files for MarkDown on webserver.' +- name: 'Create static files for MarkDown.' synchronize: src: "{{ playbook_dir }}/roles/online_docs/files/mkdocs/docs/./{{ item.src }}" dest: "/srv/mkdocs/{{ slurm_cluster_name }}/docs/" @@ -202,60 +212,91 @@ # IMPORTANT: src and dest paths must be absolute paths. Otherwise Ansible will expand the path itself which will remove the /./ dir. - '--relative' - '--omit-dir-times' - #- '--omit-link-times' - - '--chmod=Du=rwx,Dgo-rwx,Fu=rw,Fgo-rwx' + - '--omit-link-times' + - '--chmod=Du=rwx,Dg=rx,Do-rwx,Fu=rwX,Fug=rX,Fo-rwx' - '--perms' - '--force' with_items: + - src: 'css' - src: 'img' - when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] < "7" + notify: + - 'build_mkdocs' become: true -- name: 'Create static files for MarkDown on webserver.' +- name: 'Create skeleton for AppleScript apps.' synchronize: - src: "{{ playbook_dir }}/roles/online_docs/files/mkdocs/docs/./{{ item.src }}" - dest: "/srv/mkdocs/{{ slurm_cluster_name }}/docs/" + src: "{{ playbook_dir }}/roles/online_docs/files/attachments/{{ item.src }}/./" + dest: "/srv/mkdocs/{{ slurm_cluster_name }}/tmp/{{ item.dest }}" owner: 'no' group: 'no' use_ssh_args: 'yes' + recursive: 'yes' rsync_opts: - # --omit-dir-times Is required to prevent "sync error: some files/attrs were not transferred" - # for file systems like NFS mounts that cannot handle setting dir times properly. - # --omit-link-times Is required to prevent "sync error: some files/attrs were not transferred" - # for file systems like NFS mounts that cannot handle setting dir times properly. - # Requires rsync >= 3.1.2 (default on Enterprise Linux >= 7.x). - # --chmod Is required to prevent errors when the perms on the source are not what is required/expected on the destination. - # Fixing perms on the source would be good, but that may be out of our control. - # In that case --chmod ensures we get what we want on the destination. - # Works only when combined with --perms. - # --force Is required when symlinks have changed into dirs/files or vice versa. - # In that case the wrong outdated stuff has to be deleted on the destination first before the new stuff can be created. - # Deleting the outdated stuff may fail without --force. - # --relative In combination with a "source_server:some/path/not/created/on/destination/./path/created/on/destination/some_file" (dot dir) - # recreates a partial dir structure on the destination relative to the /./ dir, when it does not already exist. - # Without this combination of --relative and dot dir rsync will error when the path does not exist on the destination. - # IMPORTANT: src and dest paths must be absolute paths. Otherwise Ansible will expand the path itself which will remove the /./ dir. - '--relative' - '--omit-dir-times' - '--omit-link-times' - - '--chmod=Du=rwx,Dgo-rwx,Fu=rw,Fgo-rwx' + - '--chmod=Du=rwx,Dg=rx,Do-rwx,Fu=rwX,Fg=rX,Fo-rwx' - '--perms' - '--force' with_items: - - src: 'img' - when: ansible_facts['os_family'] == "RedHat" and ansible_facts['distribution_major_version'] >= "7" + - src: 'ssh-client-config.app' + dest: "ssh-client-config-for-{{ slurm_cluster_name }}.app" + - src: 'mount-cluster-drives.app' + dest: 'mount-cluster-drives.app' + notify: + - 'zip_attachments' + become: true + +- name: 'Make sure main script of AppleScript apps are executable.' + file: + path: "/srv/mkdocs/{{ slurm_cluster_name }}/tmp/{{ item }}/Contents/MacOS/applet" + state: 'file' + owner: 'root' + group: 'root' + mode: '0750' + with_items: + - "ssh-client-config-for-{{ slurm_cluster_name }}.app" + - 'mount-cluster-drives.app' + notify: + - 'zip_attachments' + become: true + +- name: 'Create files for attachments based on templates.' + template: + src: "attachments/{{ item.src }}" + dest: "/srv/mkdocs/{{ slurm_cluster_name }}/tmp/{{ item.dest }}" + owner: 'root' + group: 'root' + mode: "{{ item.mode }}" + with_items: + # + # For macOS users we inject the configuration script for this cluster into the AppleScript app skeleton: + # + # + - src: 'ssh-client-config.bash' + dest: "ssh-client-config-for-{{ slurm_cluster_name }}.app/Contents/MacOS/ssh-client-config.command" + mode: '0750' + # + # For Linux/Unix users: just generate a bare Bash script. + # + - src: 'ssh-client-config.bash' + dest: "ssh-client-config-for-{{ slurm_cluster_name }}.bash" + mode: '0750' + notify: + - 'zip_attachments' become: true -- name: 'Create MarkDown files based on templates on webserver.' +- name: 'Create MarkDown files based on templates.' template: src: "{{ item.src }}" dest: "/srv/mkdocs/{{ slurm_cluster_name }}/{{ item.path }}" - owner: root - group: root - mode: '0600' - with_filetree: 'templates/mkdocs' - when: item.state == 'file' + owner: 'root' + group: 'root' + mode: '0640' + with_filetree: "{{ playbook_dir }}/roles/online_docs/templates/mkdocs" + # Exclude temporary *.html preview files, which are also exlcuded in .gitignore and should not be transferred. + when: item.state == 'file' and '.md.html' not in item.path notify: - - build_mkdocs + - 'build_mkdocs' become: true -... \ No newline at end of file +... diff --git a/roles/online_docs/tasks/mkdocs-for-redhat6.yml b/roles/online_docs/tasks/mkdocs-for-redhat6.yml deleted file mode 100644 index 7706b2216..000000000 --- a/roles/online_docs/tasks/mkdocs-for-redhat6.yml +++ /dev/null @@ -1,27 +0,0 @@ ---- -- name: Install Python 3 for CentOS 6. - yum: - state: latest - update_cache: yes - name: - - python34 - - python34-setuptools - become: true - -- name: Download get-pip.py for CentOS 6. - get_url: - url: https://bootstrap.pypa.io/get-pip.py - dest: /root/get-pip.py - mode: 0744 - become: true - -- name: Execute get-pip.py for Python 3 on CentOS 6. - shell: | - python3 /root/get-pip.py --no-setuptools --no-wheel - become: true - -- name: Install MkDocs and deps using pip on CentOS 6. - shell: | - python3 -m pip install mkdocs - become: true -... \ No newline at end of file diff --git a/roles/online_docs/tasks/mkdocs-for-redhat7.yml b/roles/online_docs/tasks/mkdocs-for-redhat7.yml index 63ef542dc..316b2c564 100644 --- a/roles/online_docs/tasks/mkdocs-for-redhat7.yml +++ b/roles/online_docs/tasks/mkdocs-for-redhat7.yml @@ -1,16 +1,18 @@ --- -- name: Install Python 3 for CentOS 7. +- name: 'Install Python 3 for CentOS 7.' yum: - state: latest + state: 'latest' update_cache: yes name: - - python36 - - python36-setuptools - - python36-pip + - 'python36' + - 'python36-setuptools' + - 'python36-pip' become: true -- name: Install MkDocs and deps using pip on CentOS 7. - shell: | - python3 -m pip install mkdocs +- name: 'Install MkDocs and deps using pip on CentOS 7.' + pip: + name: 'mkdocs' + executable: 'pip3' + state: 'latest' become: true ... \ No newline at end of file diff --git a/roles/online_docs/templates/attachments/ssh-client-config.bash b/roles/online_docs/templates/attachments/ssh-client-config.bash new file mode 100755 index 000000000..489fedf36 --- /dev/null +++ b/roles/online_docs/templates/attachments/ssh-client-config.bash @@ -0,0 +1,518 @@ +#jinja2: trim_blocks:False +#!/bin/bash +# +# Bash sanity. +# +set -e # Exit on error. +set -u # Error on unbound variables. +# +# Set umask to ensure the file and folders for SSH are private. +# +umask 0077 +# +# Make sure to use UTF and not some funky wonky charset. +# +LANG='en_US.UTF-8' +LC_NUMERIC="${LANG}" + +SCRIPT_NAME="$(basename "${0}")" +SCRIPT_NAME="${SCRIPT_NAME%.*sh}" + +# +## +### Functions. +## +# +function showHelp() { + # + # Display commandline help on STDOUT. + # + cat <" "${SCRIPT_NAME}" "${_log_timestamp}" "${_log_level}" "${_problematic_line}" "${_problematic_function}") + _log_line="${_log_line_prefix} ${_log_message}" + if [[ -n "${mixed_stdouterr:-}" ]]; then + _log_line="${_log_line} STD[OUT+ERR]: ${mixed_stdouterr}" + fi + if [[ "${_status}" -ne 0 ]]; then + _log_line="${_log_line} (Exit status = ${_status})" + fi + # + # Log to STDOUT (low prio <= 'WARN') or STDERR (high prio >= 'ERROR'). + # + if [[ "${_log_level_prio}" -ge '4' || "${_status}" -ne 0 ]]; then + printf '%s\n' "${_log_line}" 1>&2 + else + printf '%s\n' "${_log_line}" + fi + fi + # + # Exit if this was a FATAL error. + # + if [[ "${_log_level_prio}" -ge '5' ]]; then + trap - EXIT + if [[ "${_status}" -ne 0 ]]; then + exit "${_status}" + else + exit 1 + fi + fi +} + +# +# Lock function using flock and a file descriptor (FD). +# This uses FD 200 as per flock manpage example. +# +function thereShallBeOnlyOne() { + local _lock_file + local _lock_dir + _lock_file="${1}" + _lock_dir="$(dirname "${_lock_file}")" + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Checking if flock utility is installed ..." + if command -v flock >/dev/null 2>&1; then + # + # The flock file locking utility is installed. + # + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Will use flock utility for locking." + mkdir -p "${_lock_dir}" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" "${?}" "Failed to create dir for lock file @ ${_lock_dir}." + exec 200>"${_lock_file}" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" "${?}" "Failed to create FD 200>${_lock_file} for locking." + if ! flock -n 200; then + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Lockfile ${_lock_file} already claimed by another instance of $(basename "${0}")." + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' 'Another instance is already running and there shall be only one.' + # No need for explicit exit here: log4Bash with log level FATAL will make sure we exit. + else + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Successfully got exclusive access to lock file ${_lock_file}." + fi + else + # + # The flock file locking utility is missing. + # Use a simple, but not atomic so less reliable fallback. + # Note: this is always the case on macOS that has no flock installed by default :( + # + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "The flock utility is missing; will fallback to less reliable pgrep ..." + if [[ $(pgrep "${SCRIPT_NAME}") -gt 1 ]]; then + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' 'Another instance is already running and there shall be only one.' + fi + fi +} + +function manageConfig() { + local _user="${1}" + local _private_key_file="${2}" + local _result + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Will configure SSH logins to {{ slurm_cluster_name | capitalize }} for' + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' " for user: ${_user}" + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' " using private key file: ${_private_key_file}" + # + # Create directory for SSH config if it did not already exist. + # + _result=$(mkdir -p -m 700 "${HOME}/.ssh/tmp" 2>&1) \ + || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Failed to create ${HOME}/.ssh/tmp. Result was: ${_result}" + _result=$(mkdir -p -m 700 "${HOME}/.ssh/conf.d" 2>&1) \ + || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Failed to create ${HOME}/.ssh/conf.d. Result was: ${_result}" + # + # Fix permissions recursively in case directories already existed, + # but permissions were to open: make the SSH client config private. + # + _result=$(chmod -R go-rwx "${HOME}/.ssh" 2>&1) \ + || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Failed to fix permissions recursively for ${HOME}/.ssh/. Result was: ${_result}" + # + # Create new known_hosts file and append the public key of the Certificate Authority (CA) for {{ slurm_cluster_name | capitalize }}. + # + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Appending the public key of the Certificate Authority (CA) to ${HOME}/.ssh/known_hosts ..." + printf '%s\n' \ + "@cert-authority {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}*,{% if public_ip_addresses is defined and public_ip_addresses[jumphost] | length %}{{ public_ip_addresses[jumphost] }},{% endif %}{% endfor %}{% for adminhost in groups['administration'] %}*{{ adminhost | regex_replace('^' + ai_jumphost + '\\+','')}},{% endfor %}*{{ stack_prefix }}-* {{ lookup('file', ssh_host_signer_ca_private_key+'.pub') }} for {{ slurm_cluster_name }}" \ + > "${HOME}/.ssh/known_hosts.new" + if [[ -e "${HOME}/.ssh/known_hosts" ]]; then + # + # When user already had a known_hosts file, then + # remove a potentially outdated CA public key for the same machines based on the slurm_cluster_name: {{ slurm_cluster_name }} + # and append all other lines to the new known_hosts file. + # + sed '/^\@cert-authority .* for {{ slurm_cluster_name }}$/d' \ + "${HOME}/.ssh/known_hosts" \ + | sort \ + >> "${HOME}/.ssh/known_hosts.new" + fi + # + # Make new known_hosts file the default. + # + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Replacing ${HOME}/.ssh/known_hosts with ${HOME}/.ssh/known_hosts.new ..." + mv "${HOME}/.ssh/known_hosts.new" "${HOME}/.ssh/known_hosts" + # + # Create main config file if it did not already exist. + # + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Creating or updating ${HOME}/.ssh/config ..." + touch "${HOME}/.ssh/config" + # + # Instruct main config file to include additional config files from the conf.d subdir. + # This is the only change we make to the main config file; + # all {{ slurm_cluster_name | capitalize }} specific stuff will go into a separate config file, + # so we can easily update that by replacing the {{ slurm_cluster_name | capitalize }} config file + # without affecting any other SSH configs. + # + if grep -cqi '^Include conf.d/\*$' "${HOME}/.ssh/config"; then + # + # Check the order: the Include directive must be placed before any Host or Match directives, + # otherwise the Include will only apply to a specific set of hosts. + # + if grep -cqi '^Host\|^Match' "${HOME}/.ssh/config"; then + local _first_line_include="$(grep -in '^Include conf.d/\*$' "${HOME}/.ssh/config" | head -n 1 | awk -F ':' '{print $1}')" + local _first_line_host_or_match="$(grep -in '^Host\|^Match' "${HOME}/.ssh/config" | head -n 1 | awk -F ':' '{print $1}')" + if [[ "${_first_line_include}" -lt "${_first_line_host_or_match}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Include directive for conf.d subdir already present in main ${HOME}/.ssh/config file" + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' " and located in correct order: before Host or Match directives." + else + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Include directive for conf.d subdir already present in main ${HOME}/.ssh/config file on the wrong line." + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'The order is important: the line "Include conf.d/*" must be present before any "Host ..." or "Match ..." directives.' + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Fix the order of lines in your ${HOME}/.ssh/config file manually and run this script again." + fi + else + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Include directive for conf.d subdir already present in main ${HOME}/.ssh/config file." + fi + else + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Adding Include directive for conf.d subdir to config ${HOME}/.ssh/config." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Prepending Include directive for conf.d subdir to new config ${HOME}/.ssh/config.new ..." + printf '%s\n\n' 'Include conf.d/*' > "${HOME}/.ssh/config.new" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Appending existing ${HOME}/.ssh/config to new config ${HOME}/.ssh/config.new ..." + cat "${HOME}/.ssh/config" >> "${HOME}/.ssh/config.new" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Replacing existing ${HOME}/.ssh/config with ${HOME}/.ssh/config.new ..." + mv "${HOME}/.ssh/config.new" "${HOME}/.ssh/config" + fi + # + # Create cluster specific config file + # + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Creating or updating ${HOME}/.ssh/conf.d/{{ slurm_cluster_name }} ..." + cat < "${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}" +# +# Special comment lines parsed by our mount-cluster-drives script to create sshfs mounts. +# (Will be ignored by OpenSSH.) +# {% set sshfs_jumphost = groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') %} +# {% set sshfs_ui = groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') %} +#SSHFS {{ sshfs_ui }}_groups={{ sshfs_jumphost }}+{{ sshfs_ui }}:/groups/ +#SSHFS {{ sshfs_ui }}_home={{ sshfs_jumphost }}+{{ sshfs_ui }}:/home/${_user}/ +# + +# +# Generic stuff: only for macOS clients. +# +IgnoreUnknown UseKeychain + UseKeychain yes +IgnoreUnknown AddKeysToAgent + AddKeysToAgent yes +# +# Host settings. +# +Host{% for jumphost in groups['jumphost'] %} {{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }}*{% endfor %} + # + # Default account name when not specified explicitly. + # + User ${_user} + # + # Prevent timeouts + # + ServerAliveInterval 60 + ServerAliveCountMax 5 + # + # We use public-private key pairs for authentication. + # Do not use password based authentication as fallback, + # which may be confusing and won't work anyway. + # + IdentityFile "${_private_key_file}" + PasswordAuthentication No + # + # Multiplex connections to + # * reduce lag when logging in to the same host in a second terminal + # * reduce the amount of connections that are made to prevent excessive DNS lookups + # and to prevent getting blocked by a firewall, because it thinks we are executing a DoS attack. + # + # Name/location of sockets for connection multiplexing are configured using the ControlPath directive. + # In the ControlPath directive %C expands to a hashed value of %l_%h_%p_%r, where: + # %l = local hostname + # %h = remote hostname + # %p = remote port + # %r = remote username + # This makes sure that the ControlPath is + # * a unique socket that is local to machine on which the sessions are created, + # which means it works with home dirs from a shared network file system. + # (as sockets cannot be shared by servers.) + # * not getting to long as the hash has a fixed size not matter how long %l_%h_%p_%r was. + # + ControlMaster auto + ControlPath ~/.ssh/tmp/%C + ControlPersist 1m +# +# Expand short jumphost names to FQDN or IP address. +#{% if public_ip_addresses is defined and public_ip_addresses | length %}{% for jumphost in groups['jumphost'] %} +Host {{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} + HostName {{ public_ip_addresses[jumphost | regex_replace('^' + ai_jumphost + '\\+','')] }}{% endfor %}{% else %} +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} {% endfor %}{% if slurm_cluster_domain | length %}!*.{{ slurm_cluster_domain }}{% endif %} + HostName %h{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}{% endif %} +# +# Universal jumphost settings for triple-hop SSH. +# +Host *+*+* + ProxyCommand ssh -x -q -i "${_private_key_file}" $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p +# +# Double-hop SSH settings to connect via specific jumphosts. +# +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}+* {% endfor %}{% raw %}{% endraw %} + User ${_user} + ProxyCommand ssh -x -q -i "${_private_key_file}" %r@\$(echo %h | sed 's/+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W \$(echo %h | sed 's/^[^+]*+//'):%p +# +# Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 443 as fall-back. +# Do not use port 443 by default for SSH as it is officially assigned to HTTPS traffic +# and some firewalls will cause problems with SSH traffic over port 443. +# +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}443+* {% endfor %}{% raw %}{% endraw %} + User ${_user} + ProxyCommand ssh -x -q -i "${_private_key_file}" %r@\$(echo %h | sed 's/443+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W \$(echo %h | sed 's/^[^+]*+//'):%p -p 443 + +EOF +} + +# +## +### Main. +## +# + +# +# Trap all exit signals: HUP(1), INT(2), QUIT(3), TERM(15), ERR. +# +trapSig HUP INT QUIT TERM EXIT ERR + +# +# Initialise Log4Bash logging with defaults. +# +l4b_log_level='INFO' +getLogLevelPrio "${l4b_log_level}" +l4b_log_level_prio="${resolved_log_level_prio}" +mixed_stdouterr='' # global variable to capture output from commands for reporting in custom log messages. + +# +# Get commandline arguments. +# +declare user +while getopts ":l:u:h" opt; do + case "${opt}" in + l) + l4b_log_level="${OPTARG}" + getLogLevelPrio "${l4b_log_level}" + l4b_log_level_prio="${resolved_log_level_prio}" + ;; + u) + user="${OPTARG}" + ;; + h) + showHelp + trap - EXIT + exit 0 + ;; + \?) + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Invalid option -${OPTARG}. Try $(basename "${0}") -h for help." + ;; + :) + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "Option -${OPTARG} requires an argument. Try $(basename "${0}") -h for help." + ;; + esac +done + +# +# Make sure only one copy of this script runs simultaneously to prevent messing up config files. +# +thereShallBeOnlyOne "${TMPDIR:-/tmp}/${SCRIPT_NAME}.lock" + +# +# Get account name and path to private key. +# +if [[ -z "${user:-}" ]]; then + read -e -p "Type the account name you received from the helpdesk for logins to {{ slurm_cluster_name | capitalize }} and press [ENTER]: " user +fi +if [[ -z "${user:-}" ]]; then + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' 'Your account name cannot be empty.' +fi +read -e -p "Specify the path to the private key file you want to use (or accept the default: ~/.ssh/id_ed25519) and press [ENTER]: " private_key_file +private_key_file="${private_key_file:-~/.ssh/id_ed25519}" +if [[ -e "${private_key_file/#\~/${HOME}}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "The specified private key file ${private_key_file} exists." +else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '1' "The specified private key file ${private_key_file} does not exist." +fi + +# +# Check if client has a compatible OpenSSH version. +# We need OpenSSH >= 7.3 to support the Include directive in the main ~/.ssh/config file. +# +ssh_version_string="$(ssh -V 2>&1)" +ssh_version_regex='OpenSSH_([0-9][0-9]*).([0-9][0-9]*)' +if [[ "${ssh_version_string}" =~ ${ssh_version_regex} ]]; then + ssh_version_major="${BASH_REMATCH[1]}" + ssh_version_minor="${BASH_REMATCH[2]}" + compatible_ssh='no' + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Detected OpenSSH version ${ssh_version_major}.${ssh_version_minor}." + if [[ "${ssh_version_major}" -gt '7' ]]; then + compatible_ssh='yes' + elif [[ "${ssh_version_major}" -eq '7' ]]; then + if [[ "${ssh_version_minor}" -ge '3' ]]; then + compatible_ssh='yes' + fi + fi + if [[ "${compatible_ssh}" == 'yes' ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "OpenSSH version ${ssh_version_major}.${ssh_version_minor} is compatible." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Your OpenSSH client version ${ssh_version_major}.${ssh_version_minor} is too old and incompatible." + fi +else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME[0]:-main}" '0' "Failed to determine OpenSSH client version; you may be using an incompatible version." +fi + +# +# Create/update SSH config. +# +manageConfig "${user}" "${private_key_file}" + +# +# Notify user. +# +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Finished configuring your SSH client for logins to {{ slurm_cluster_name | capitalize }}.' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'You can log in to User Interface {{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' via jumphost {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' in a terminal with the following SSH command:' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'We will now test your connection by executing the above SSH command to login and logout.' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'If this is the first time your private key will be used for an SSH session,' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' you will receive a prompt to supply the password for your private key,' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' which can be stored in your login KeyChain,' +log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' " so you won't have to retype the password again for a subsequent SSH session." +read -e -p "Press [ENTER] to test your connection." +if ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} exit; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Login was succesful.' + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Consult the online documentation for additional examples ' + log4Bash 'INFO' "${LINENO}" "${FUNCNAME[0]:-main}" '0' ' and how to transfer data with rsync over SSH.' +else + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Failed to login; check if your network either wired or using WiFi is up.' + log4Bash 'WARN' "${LINENO}" "${FUNCNAME[0]:-main}" '0' 'Consult the online documentation for debugging options.' +fi +read -e -p "Press [ENTER] to exit." + +trap - EXIT +exit 0 diff --git a/roles/online_docs/templates/mkdocs/docs/accounts.md b/roles/online_docs/templates/mkdocs/docs/accounts.md index 46001c646..cbbc8085e 100644 --- a/roles/online_docs/templates/mkdocs/docs/accounts.md +++ b/roles/online_docs/templates/mkdocs/docs/accounts.md @@ -54,7 +54,7 @@ To generate an RSA key pair with OpenSSH, type the following command: ssh-keygen -t ed25519 -C "your_comment_see_below" ``` As comment/label for your keys please use **your first initial followed by (optionally your middle name followed by) your family name** all in lowercase and without any separators like spaces, dots or underscores. -So if your name is _**Jack the Hippo**_, please use _**jthehippo**_ as comment, so we can easily identify the key as yours. +So if your name is _**Jack Peter Frank the Hippo**_, please use _**jthehippo**_ as comment, so we can easily identify the key as yours. ###### Select where to store the key pair @@ -64,11 +64,15 @@ Enter file in which to save the key (/path/to/your/home_dir/.ssh/id_ed25519): * key if you have no key in the default location. - 2. OpenSSH will by default use the key from the default location. - If you create the key in a non-default location, you will need to explicitly specify which key file to use when connecting via ssh or sftp. + + 1. Accepting the default will overwrite an existing key pair, + so only accept the default if you either do not have a default key pair yet + or if you want to replace your default key pair. + 2. If you create a key pair in a non-default location, + you will need to explicitly specify which key file to use when you start a session. + Consult the OpenSSH manual for details. ###### Secure the private key @@ -91,40 +95,44 @@ ssh-keygen -p -f ~/.ssh/id_ed25519 #### 2.B On Windows -###### Get PuTTYgen +###### Get a terminal and key generator application -You can use the ```PuTTYgen``` application, which is distributed as part of the **PuTTY** suite and also bundled with **WinSCP**, to generate a key pair. +Your OS does not come with a default terminal and key generator application, so you will need to download and install one. +There are many options all of which have their own pros and cons; we suggest you give [MobaXterm](https://mobaxterm.mobatek.net) version >= 12.3 a try +as it features a key generator, terminal and graphical user interface for data transfers all-in-one. +The following steps use the *portable* version of *MobaXterm Home Edition*, which is free and does not need to be installed with an installer; +just download, unpack and execute. +If you want to use another terminal, key generator or data transfer app please consult their manuals... + + * 0: Check your MobaXterm version is **12.3 or newer** as older ones have a known bug and won't work. + * 1: Launch MobaXterm and choose the ```MobaKeyGen (SSH key generator)``` from the tools as shown in the screenshot below. - * Install [WinSCP](http://winscp.net/eng/download.php) if you only want to transfer data to/from the cluster via a graphical user interface. - * Install [PuTTY](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) if you want to login via SSH to process data or if you want to transfer data via the commandline. - -1. Launch PuTTYgen as shown in the screenshot below. -![launch PuTTYgen](img/puttygen1.png) +![launch MobaKeyGen](img/MobaXterm1.png) ###### Configure -![Select key type](img/puttygen2.png) +![Select key type](img/MobaXterm2.png) * 2: From the **parameters** section at the bottom of the window choose: ```Type of key to generate:``` **ED25519** * 3: Click the **Generate** button... ###### Generate key pair -![Generate randomness and subsequently key pair](img/puttygen3.png) +![Generate randomness and subsequently key pair](img/MobaXterm3.png) - * 4: Yes you really have to move the mouse now: computers are pretty bad at generating random numbers and PuTTYgen uses the coordinates of your mouse movement as a seed to generate a random number. + * 4: Yes you really have to move the mouse now: computers are pretty bad at generating random numbers and MobaKeyGen uses the coordinates of your mouse movement as a seed to generate a random number. ###### Secure private key and save pair to disk Your key pair was generated. -![Save keys](img/puttygen4.png) +![Save keys](img/MobaXterm4.png) Now make sure you: * 5: Replace the comment in **Key comment** with **your first initial followed by (optionally your middle name followed by) your family name** all in lowercase and without any separators like spaces, dots or underscores. - So if your name is _**Jack the Hippo**_, please use _**jthehippo**_ as comment, so we can easily identify the key as yours. + So if your name is _**Jack Peter Frank the Hippo**_, please use _**jthehippo**_ as comment, so we can easily identify the key as yours. * 6: Secure your private key with a good password **before** saving the private key. DO NOT choose a simple password or even worse an empty one! * 7: Confirm the password * 8: Click the **Save public key** button. @@ -138,20 +146,36 @@ Now make sure you: To request an account, [contact the helpdesk via email](../contact/) and - * 3.A If on Linux / Unix / macOS: + * Either if on a **Windows** computer: + Paste the contents of the public key as displayed in MobaKeyGen's *Public key for pasting into OpenSSH authorized_keys file* field in the email. + * Or if on a **Linux** / **Unix** / **macOS** computer: Attach the id_rsa.pub public key file generated with ssh-keygen. If you cannot see / find the key file, you most likely stored the file in a folder starting with a ```.```; e.g. in your ```~/.ssh``` folder which is the default. Folders and files that start with a ```.``` are *hidden* files and not displayed by default. On macOS you can press ```++<.>``` to toggle the visibility of hidden files in *Open...* and *Save...* dialog windows. Please use a search engine for a solution to display hidden files in other situations like Finder windows or on other platforms. - * 3.B If on Windows: - Paste the contents of the public key as displayed in PuTTYgen's *Public key for pasting into OpenSSH authorized_keys file* field in the email. - * Motivate your account request by specifying the project your are working on and by adding your collaborators on CC. + * Please motivate your account request and + * For **guest** accounts to access only a data transfer machine associated with the cluster: + * Specify the project your are working on and add your collaborators on CC. + * For **regular** accounts to access the cluster: + * Specify the groups you want to become a member of in order to access specific data sets. + Put all group owners of the corresponding groups on CC and ask them to approve your request in a reply to the helpdesk. + If you do not know who the group owners are, please consult a colleague or your boss / P.I. / team lead / project lead / etc. + * Please add a staff member of the department/group where you are appointed or the project you are involved in on CC and + ask him/her to confirm your appointment/involvement and the expiration date of your contract in a reply to the helpdesk. + We will then setup your cluster account with the same expiration date. + (A staff member can be your boss, P.I., team lead, project lead or secretary.) + Please note: we cannot give you access until we have received both approval from at least one group owner and a confirmation for the expiration date of your contract/collaboration. * Never ever email/give anyone your private key! If you do, the key is no longer private and useless for security: trash the key pair and start over by generating a new pair. * If you ever suspect that your private key may have been compromised (laptop got stolen, computer got infected with a virus/trojan/malware, etc.): * [notify the helpdesk](../contact/) immediately, so we can revoke the public key for the compromised private key - * and start over by generating a new pair + * and start over by generating a new pair. ## 4. Start using servers/services -Once you get notified by email that your account is ready you can proceed to [login](../logins/) \ No newline at end of file + * Once you get notified by email that your account is ready you can proceed to [login](../logins/) + * If you want to request access to an additional group, send your request by email to the helpdesk and with the corresponding group owners on CC. + You can lookup the group owners yourself on the cluster using: + + module load cluster-utils + colleagues -g \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/analysis.md b/roles/online_docs/templates/mkdocs/docs/analysis.md index 948b7b8d0..77302063f 100644 --- a/roles/online_docs/templates/mkdocs/docs/analysis.md +++ b/roles/online_docs/templates/mkdocs/docs/analysis.md @@ -39,7 +39,7 @@ If the data you want to work with is not already available on the cluster, you will need to transfer data to the cluster, which is covered by [instructions on a separate data transfers page](../datatransfers/). -## Jobs +## Job types Simple tasks that require little time, little computational power and little memory can be executed directly on a User Interface (UI) server. Please think twice though before you hit enter: if you crash the UI, others can no longer submit nor monitor their jobs, which is pretty annoying. @@ -50,7 +50,7 @@ There are 2 kind of jobs: 1. _batch_ jobs a.k.a. non-interactive jobs 2. _interactive_ jobs -### 1. Batch jobs +## 1. Batch jobs For large data sets or long running commands you will need to create a (bash) job script, which can be submitted to the Slurm scheduler. When resources are available the scheduler will copy the job at the top of the queue to a compute node where the job script will run. @@ -225,14 +225,23 @@ Deleting all your jobs in one go: scancel -u [your account] ``` -### 2. Interactive jobs +## 2. Interactive jobs When you need to interact with a running job you can start an interactive session with the [srun](http://slurm.schedmd.com/srun.html) command. -Just like for the ```sbatch``` command you will need to request resources like amount of cores, amount of memory, work allocation time (walltime), etc. E.g.: +This creates a shell on a compute node, which works the same as a shell on the User Interface except that the shell is restricted to the requested resources. +This is ideal for debugging/testing and prevents your work from running out of control and crashing processes from other users or vice versa. +Just like for the ```sbatch``` command you will need to request resources like amount of cores, amount of memory, work allocation time (walltime), etc. +E.g. to request a session for one hour: ``` -srun -cpus-per-task=1 --mem=1gb --nodes=1 --qos=dev --time=00:01:00 --pty bash -i +srun --cpus-per-task=1 --mem=1gb --nodes=1 --qos=priority --time=01:00:00 --pty bash -i ``` When the requested resources are available the interactive session will start immediately. +To increase the chance your interactive session will start quickly, even when the cluster is relatively busy, you can request _Quality of Service_ level _priority_ with ```--qos=priority```. + +**Essential**: the order of ```srun``` arguments is not important except that ```--pty bash -i``` must be last. +Any options after that are interpreted as arguments for the requested shell and not for the ```srun``` command. +Hence the ```-i``` in the example is an argument for the ```bash``` shell. + When you exit the bash shell using either the ```exit``` command or by pressing ```CTRL+d``` the interactive job will be cancelled automagically and the corresponding resources released. diff --git a/roles/online_docs/templates/mkdocs/docs/datatransfers-macos.md b/roles/online_docs/templates/mkdocs/docs/datatransfers-macos.md new file mode 100644 index 000000000..fe4707d84 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/datatransfers-macos.md @@ -0,0 +1,120 @@ +#jinja2: trim_blocks:False +# Data transfers - Push to or pull from {{ slurm_cluster_name | capitalize }} User Interface via Jumphost with GUI on macOS + +On macOS you can use + + * either _**SSHFS**_ (SSH File System) to browse the file systems of the {{ slurm_cluster_name | capitalize }} cluster directly in the _**Finder**_ + * or a dedicated _SFTP_ client app like _**ForkLift2**_. + +Both use the _**S**ecure **F**ile **T**ransfer **P**rotocol_, which is basically _FTP_ tunneled over an _SSH_ connection. + +## SFTP with SSHFS in the Finder + +_SSHFS_ simulates a file system over a normal SSH connection and depends on the _FUSE libraries_, which extend the native macOS file handling capabilities via third-party file systems. +**Important**: SSHFS is great for browsing file systems and transferring data. +You may also open a remote file read-only in an application on your Mac for a quick peak at it's content, +but you should not open a remote file in read-write mode and start to make changes unless you want to get surprising results; +When you loose the network connection, for example due to unstable WiFi or when you put your Mac to sleep, +you may loose unsaved changes or worse end up with "half modified" corrupt files. +So when you want to modify a remote file, transfer a copy from the remote server to your local Mac, +make changes to the copy, save them and transfer the updated file to the server. + +### 1. Install FUSE for macOS + + * Download and install _**FUSE for macOS**_ from [https://osxfuse.github.io/](https://osxfuse.github.io/) + * Note: when you are on _macOS Catalina_ or newer, you may get a security warning like this: + ![InstallFuseForMacOS](img/osxfuse-1.png) + Open the _Security Preferences_ pane from _System Preferences_ and + ![InstallFuseForMacOS](img/osxfuse-2.png) + 1: Open the lock to make changes and + 2: Allow the system to load the Fuse for macOS kernel extensions developed by _Benjamin Fleischer_ + +### 2. Install SSHFS for macOS + + * Download and install _**SSHFS**_ from [https://osxfuse.github.io/](https://osxfuse.github.io/) + +### 3. Download and run mount-cluster-drives app + + * Download and unzip the [mount-cluster-drives](../attachments/mount-cluster-drives-macos.zip) AppleScript application. + * Locate the ```mount-cluster-drives``` app, **right click** in the ```Finder``` application and select _**Open**_ from the contextual pop-up menu: + ![Launch mount-cluster-drives.app](img/mount-cluster-drives-0b.png) + Note: when you are on macOS _Catalina_ or newer, you may get a security warning like this: + ![Launch mount-cluster-drives.app](img/mount-cluster-drives-0d.png) + Select _**Open**_ to continue. + (If you started the app by double clicking as opposed to choosing _Open_ from the contextual pop-up menu, + the window will look similar, but will lack to _Open_ button allowing you only only to _Cancel_ or _Move to Bin_) + * Depending on your macOS version, you may receive a pop-up requesting permission to allow access to the ```Finder``` application: + ![Allow access to the Finder.app](img/mount-cluster-drives-1.png) + Click _Ok_ to allow access to the ```Finder```. + If you want to revoke this permission or change it back to allow later on, you can do so in + _System Preferences_ -> _Security & Privacy_ prefs -> _Privacy_ tab -> _Automation_ + * The ```mount-cluster-drives``` app will mount the file systems of all configured clusters in a sub directory of your home dir named ```ClusterDrives```. + ![ClusterDrivesInFinder](img/mount-cluster-drives-2.png) + You can now drag and drop files in the ```Finder``` to transfer to / from {{ slurm_cluster_name | capitalize }}. + * To unmount the _SSHFS_ shares click the eject button behind the name of the share. + +##### Technical Details + +Some technical details for the curious who like to know how this works or need to debug connection issues: + +The ```mount-cluster-drives``` app parses special comment lines like this: +``` +# +# Special comment lines parsed by our mount-cluster-drives script to create sshfs mounts. +# (Will be ignored by OpenSSH.) +# {% set sshfs_jumphost = groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') %} +# {% set sshfs_ui = groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') %} +#SSHFS {{ sshfs_ui }}_groups={{ sshfs_jumphost }}+{{ sshfs_ui }}:/groups/ +#SSHFS {{ sshfs_ui }}_home={{ sshfs_jumphost }}+{{ sshfs_ui }}:/home// +# +``` +in the OpenSSH client configuration file for {{ slurm_cluster_name | capitalize }} ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}```, +which was created by the ```ssh-client-config-for-{{ slurm_cluster_name }}``` app +from the OpenSSH client configuration [instructions for macOS clients](../logins-macos/) page. + +The parsed comment lines result in the following mount commands: +``` +sshfs -o "defer_permissions,follow_symlinks,noappledouble,noapplexattr,reconnect,auto_xattr,auto_cache,connecttimeout=10,volname={{ sshfs_ui }}_groups" \ + "{{ sshfs_jumphost }}+{{ sshfs_ui }}:/groups/" \ + "~/ClusterDrives/{{ sshfs_ui }}_groups" +sshfs -o "defer_permissions,follow_symlinks,noappledouble,noapplexattr,reconnect,auto_xattr,auto_cache,connecttimeout=10,volname={{ sshfs_ui }}_home" \ + "{{ sshfs_jumphost }}+{{ sshfs_ui }}:/home/" + "~/ClusterDrives/{{ sshfs_ui }}_home" +``` + +If you have access to multiple clusters, which were configured in a similar way, you may have multiple _SSHFS_ mounts, +which are all mounted with the same ```mount-cluster-drives``` app. + +## SFTP with dedicated ForkLift2 client + +If you prefer a dedicated Graphical User Interface that is both free and supports multi-hop SSH via a jumphost using your OpenSSH config, we suggest you give _ForkLift 2_ a try. +You can get _ForkLift 2_ from the [App store](https://apps.apple.com/app/forklift-file-manager-and-ftp-sftp-webdav-amazon-s3-client/id412448059). +Please note that there is a newer version _ForkLift 3_, but this one is not available from the App store neither is it free. +There are various other options, but those are either paid apps or they don't support multi-hop SSH using your OpenSSH config. + +To start a session with _ForkLift 2_: + + * Launch the app; You will see two file browser columns next to each other. + Both will initially show the same contents of your local home dir. + ![Allow access to the Terminal.app](img/ForkLift1.png) + To configure one of the columns to show the contents of the cluster, click on the **star symbol** at the beginning of the path at the top of a column. + * Click the **+** button to create a new _favorite_ + ![Allow access to the Terminal.app](img/ForkLift2.png) + * Provide the connection details: + ![Allow access to the Terminal.app](img/ForkLift3b.png) + * _Protocol:_ **SFTP** + * _Name_: **{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}** + * _Server_: **{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}** + * _Username_: your account name as you received it from the helpdesk + * Leave the _Password_ field empty. + * Optionally you can specify a default encoding and remote path to start browsing on the cluster. + Click the **Save** button to store the new favorite. + * Your favorite should now be listed under _Favorites_. + ![Allow access to the Terminal.app](img/ForkLift4.png) + * Click on your new favorite to connect to the server and start a session. + ![Allow access to the Terminal.app](img/ForkLift5.png) + Note that if you did not specify an explicit _remote path_ you will start by default in your remote home dir on the cluster, which may be empty. + +----- + +Back to operating system independent [instructions for data transfers](../datatransfers/) \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/datatransfers-windows.md b/roles/online_docs/templates/mkdocs/docs/datatransfers-windows.md new file mode 100644 index 000000000..895e04d53 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/datatransfers-windows.md @@ -0,0 +1,12 @@ +#jinja2: trim_blocks:False +# Data transfers - Push to or pull from {{ slurm_cluster_name | capitalize }} User Interface via Jumphost with GUI on Windwos + +On Windows you can use the built-in _**SFTP**_ or _**SCP**_ client in the corresponding tab of MobaXterm. +Once logged in using the SSH client configuration instructions for [Windows clients](../logins-windows/), +you can simply drag and drop from _**MobaXterm**_ to _**Windows Explorer**_ or vice versa to transfer files. + +![MobaXtermBrowser](img/MobaXterm9c.png) + +----- + +Back to operating system independent [instructions for data transfers](../datatransfers/) \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/datatransfers.md b/roles/online_docs/templates/mkdocs/docs/datatransfers.md index 9fa006bb5..7ecd2e0d0 100644 --- a/roles/online_docs/templates/mkdocs/docs/datatransfers.md +++ b/roles/online_docs/templates/mkdocs/docs/datatransfers.md @@ -10,20 +10,24 @@ Your options to move data to/from the {{ slurm_cluster_name | capitalize }} HPC 1. Push data from an external machine to the cluster UI via the jumphost or Pull data on an external machine from the cluster UI via the jumphost. - Supported protocols: - * rsync over ssh + Supported protocol: + * SSH 2. Push data from the cluster UI to an external server or Pull data on the cluster UI from an external server. Supported protocols: - * rsync over ssh - * http(s) + * SSH + * HTTP(S) {# jinja2 comment: firewall requires improvements for aspera. - * aspera + * Aspera #} ## 1. Push to or pull from cluster UI via jumphost -#### Using rsync over ssh + * via [GUI on Windows](../datatransfers-windows/) + * via [GUI on macOS](../datatransfers-macos/) + * via the commandline: see below for _rsync_ over SSH + +#### Using rsync over SSH * You can transfer data with ```rsync``` over _SSH_ to copy files to for example your home dir on the cluster with something like the command below. @@ -40,51 +44,50 @@ Your options to move data to/from the {{ slurm_cluster_name | capitalize }} HPC * Swap source and destination to pull data from the cluster as opposed to pushing data to the cluster. -## 2. Push to or pull from another server - -#### Using rsync over ssh - -When you login from your local computer (optionally via a jumphost) to a server of the {{ slurm_cluster_name | capitalize }} HPC cluster -and next need to transfer data from {{ slurm_cluster_name | capitalize }} to another server or vice versa, -you will need to temporarily forward your private key to the server from the {{ slurm_cluster_name | capitalize }} HPC cluster. -This is known as _SSH agent forwarding_ and can be accomplished with the ```-A``` argument on the commandline. +## 2. Push to or pull from another (SSH) server -* _**Note**_: You **cannot** accomplish this by configuring a ```ProxyCommand``` directive in the ```${HOME}/.ssh/config``` file on your local computer. -* _**Note**_: Do **not** use SSH with _agent forwarding_ by default for all your sessions as it is less secure. -* If you do need _agent forwarding_, then login with ```-A``` like this: +#### Using rsync over SSH - $your_client> ssh -A {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} +When you login from your local computer (via a jumphost) to a server of the {{ slurm_cluster_name | capitalize }} HPC cluster +and next need to transfer data from {{ slurm_cluster_name | capitalize }} to another SSH server or vice versa, +you will need: -* Execute the following command to verify that _agent forwarding_ worked and to list the identities (private keys) available to the SSH agent: + 1. A private key on {{ slurm_cluster_name | capitalize }} and + 2. A corresponding public key on the other server. - ${{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}> ssh-add -l +To get a private key on {{ slurm_cluster_name | capitalize }} you can - * You should get a response with at least one key fingerprint, which means you can now transfer data with ```rsync``` to/from the other server - assuming you are allowed to access the other server, are allowed to transfer the data and that no firewalls are blocking the connection. - * If you get ```The agent has no identities.``` or ```Could not open a connection to your authentication agent.``` instead then the key forwarding failed. - This may happen when you were already logged in to the same server without the ```-A``` option in another active SSH session; - make sure you logout from the server of the {{ slurm_cluster_name | capitalize }} HPC cluster in all terminals and try login with ```-A``` again. - If that did not help, then the next step depends on the OS of the machine where you are running your SSH client and/or the SSH client itself. - * MacOS/Linux/Unix (and !MobaXterm on Windows): Use the ```ssh-add -l``` command on your _**client**_ - When you also get the message ```The agent has no identities.``` or ''Could not open a connection to your authentication agent'' on your SSH client, - you need to add your private key. If your private key is located in the default path (~/.ssh/id_ed25519) you can use the following command: +* either create a new key pair on {{ slurm_cluster_name | capitalize }} +* or temporarily forward your private key with _SSH agent forwarding_ to {{ slurm_cluster_name | capitalize }} - $your_client> ssh-add +###### Configure SSH agent forwarding - If your key is not located in the default path, you will have to specify which private key file to add: +First, configure _SSH agent forwarding_ using one of: - $your_client> ssh-add /path/to/my/private.key +* [Instructions for MobaXterm on Windows](../ssh-agent-forwarding-mobaxterm/) +* [Instructions for OpenSSH on macOS / Linux / Unix](../ssh-agent-forwarding-openssh/) - * PuTTY on Windows: Check if _**Pageant**_ (part of the PuTTY Suite) is running and if your private key was loaded in _Pageant_. - When _Pageant_ is running, the app will have an icon in the system tray on the bottom right corner of your screen. - Double click the _Pageant_ icon in the system try to open a window with the list of loaded keys; - load your private key when it is not yet in the list. +Next, login to {{ slurm_cluster_name | capitalize }} and verify that _agent forwarding_ worked +by executing the following command to list the identities (private keys) available to your _SSH agent_: +``` +${{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}> ssh-add -l +``` -* Use rsync to pull data from the other cluster: +* You should get a response with at least one key fingerprint, which means you can now transfer data with ```rsync``` to/from the other server + assuming you have an enabled account with public key on the other server and that no firewalls are blocking the connection. +* If instead you get ```The agent has no identities``` or ```Could not open a connection to your authentication agent```, + then the key forwarding failed. + This may happen when you were already logged in to the same server without _agent forwarding_ in another active SSH session; + make sure you logout from all {{ slurm_cluster_name | capitalize }} servers in all terminals and try login with _agent forwarding_ again. - ${{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}> rsync -av your-account@other-server.some.domain:/path/to/source_folder /path/to/destination_folder/ +###### Transfer data with rsync -* Swap source and destination to push data to the other server as opposed to pulling data from the other sever. +Once you have a private key on {{ slurm_cluster_name | capitalize }} and can login to the other server using ssh, +you can use rsync (over ssh) to pull data from the other server like this: +``` +${{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}> rsync -av your-account@other-server.some.domain:/path/to/source_folder /path/to/destination_folder/ +``` +Swap source and destination to push data to the other server as opposed to pulling data from the other sever. #### Using http(s) diff --git a/roles/online_docs/templates/mkdocs/docs/logins-linux.md b/roles/online_docs/templates/mkdocs/docs/logins-linux.md new file mode 100644 index 000000000..c01aa8858 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/logins-linux.md @@ -0,0 +1,218 @@ +#jinja2: trim_blocks:False +# SSH config and login to UI via Jumphost for users on Linux/Unix + +The following assumes: + + * you have a ```${HOME}/.ssh``` folder with SSH keys (as generated using the instructions for requesting accounts) + * and that you received a notification that your account has been activated + * and that you are on the machine from which you want to connect to the cluster. + * and that this machine has **OpenSSH 7.3p1 or newer**. + Older versions lack several OpenSSH features we need and are incompatible. + +##### 1. Create required directories and files if they do not exist yet + +``` +mkdir -p -m 700 "${HOME}/.ssh/" +mkdir -p -m 700 "${HOME}/.ssh/tmp/" +mkdir -p -m 700 "${HOME}/.ssh/conf.d/" +touch "${HOME}/.ssh/known_hosts" +touch "${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}" +chmod -R go-rwx "${HOME}/.ssh" +``` + +##### 2. Configure Certificate Authority's (CA) public key to verify the identity of cluster servers + +Append the public key from the Certificate Authority we used to sign the host keys of our machines to your ```${HOME}/.ssh/known_hosts``` file. +Open a terminal and copy paste the following commands: +``` +# +# Create new known_hosts file and append the UMCG HPC CA's public key. +# +printf '%s\n' \ + "@cert-authority {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}*,{% if public_ip_addresses is defined and public_ip_addresses[jumphost] | length %}{{ public_ip_addresses[jumphost] }},{% endif %}{% endfor %}{% for adminhost in groups['administration'] %}*{{ adminhost | regex_replace('^' + ai_jumphost + '\\+','')}},{% endfor %}*{{ stack_prefix }}-* {{ lookup('file', ssh_host_signer_ca_private_key+'.pub') }} for {{ slurm_cluster_name }}" \ + > "${HOME}/.ssh/known_hosts.new" +if [[ -e "${HOME}/.ssh/known_hosts" ]]; then + # + # When user already had a known_hosts file, then + # remove a potentially outdated CA public key for the same machines based on the slurm_cluster_name: {{ slurm_cluster_name }} + # and append all other lines to the new known_hosts file. + # + sed '/^\@cert-authority .* for {{ slurm_cluster_name }}$/d' "${HOME}/.ssh/known_hosts" \ +| sort >> "${HOME}/.ssh/known_hosts.new" +fi +# +# Make new known_hosts file the default. +# +mv "${HOME}/.ssh/known_hosts.new" "${HOME}/.ssh/known_hosts" +``` + +##### 3. Add include directive to main SSH config file + +Use a text editor to add the following line +``` +Include conf.d/* +``` +to the beginning of your ```${HOME}/.ssh/config``` file. +Important: this _Include_ directive must precede any lines containing _Host_ or _Match_ directives, +otherwise the _Include_ will only apply to a specific set of hosts. + +##### 4. Create SSH config file for {{ slurm_cluster_name | capitalize }} + +Now we need to configure transparent multi-hop SSH for {{ slurm_cluster_name | capitalize }}. +Open your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` file in a text editor and add the lines below +replacing all occurrences of _**youraccount**_ with the account name you received from the helpdesk. +``` +# +# Generic stuff: only for macOS clients. +# +IgnoreUnknown UseKeychain + UseKeychain yes +IgnoreUnknown AddKeysToAgent + AddKeysToAgent yes +# +# Host settings. +# +Host reception* + # + # Default account name when not specified explicitly. + # + User youraccount + # + # Prevent timeouts + # + ServerAliveInterval 60 + ServerAliveCountMax 5 + # + # We use public-private key pairs for authentication. + # Do not use password based authentication as fallback, + # which may be confusing and won't work anyway. + # + PasswordAuthentication No + # + # Multiplex connections to + # * reduce lag when logging in to the same host in a second terminal + # * reduce the amount of connections that are made to prevent excessive DNS lookups + # and to prevent getting blocked by a firewall, because it thinks we are executing a DoS attack. + # + # Name/location of sockets for connection multiplexing are configured using the ControlPath directive. + # In the ControlPath directive %C expands to a hashed value of %l_%h_%p_%r, where: + # %l = local hostname + # %h = remote hostname + # %p = remote port + # %r = remote username + # This makes sure that the ControlPath is + # * a unique socket that is local to machine on which the sessions are created, + # which means it works with home dirs from a shared network file system. + # (as sockets cannot be shared by servers.) + # * not getting to long as the hash has a fixed size not matter how long %l_%h_%p_%r was. + # + ControlMaster auto + ControlPath ~/.ssh/tmp/%C + ControlPersist 1m +# +# Expand short jumphost names to FQDN or IP address. +#{% if public_ip_addresses is defined and public_ip_addresses | length %}{% for jumphost in groups['jumphost'] %} +Host {{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} + HostName {{ public_ip_addresses[jumphost | regex_replace('^' + ai_jumphost + '\\+','')] }}{% endfor %}{% else %} +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} {% endfor %}{% if slurm_cluster_domain | length %}!*.{{ slurm_cluster_domain }}{% endif %} + HostName %h{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}{% endif %} +# +# Universal jumphost settings for triple-hop SSH. +# +Host *+*+* + ProxyCommand ssh -x -q $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p +# +# Double-hop SSH settings to connect via Jumphosts{% if slurm_cluster_domain | length %}{{ slurm_cluster_domain }}{% endif %}. +# +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}+* {% endfor %}{% raw %}{% endraw %} + User youraccount + ProxyCommand ssh -x -q %r@$(echo %h | sed 's/+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W $(echo %h | sed 's/^[^+]*+//'):%p +# +# Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 443 as fall-back. +# Do not use port 443 by default for SSH as it officially assigned to HTTPS traffic +# and some firewalls will cause problems when trying to route SSH over port 443. +# +Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}443+* {% endfor %}{% raw %}{% endraw %} + User youraccount + ProxyCommand ssh -x -q %r@$(echo %h | sed 's/443+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W $(echo %h | sed 's/^[^+]*+//'):%p -p 443 +``` + +##### 5. Login via Jumphost + + * You can now login to the _UI_ named ```{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` with the account as specified in your ```${HOME}/.ssh/config``` + via the _Jumphost_ named ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}``` + using the alias ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}```. + Type the following command in a terminal: + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * In order to override the account name specified in your ```${HOME}/.ssh/config``` you can use: + + ssh some_other_account@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * If you also added the ```Host *+*+*``` code block from the example ```${HOME}/.ssh/config``` you can do tripple hops starting with a _Jumphost_ like this: + + ssh jumphost+intermediate_server+destination_server + + * In case you are on a network where the default port for _SSH_ (22) is blocked by a firewall you can try to setup _SSH_ over port 443, which is the default for HTTPS and almost always allowed, using an alias like this: + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}443+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + +#### Frequent Asked Questions (FAQs) and trouble shooting + + * Q: Why do I get the error ```muxserver_listen bind(): No such file or directory.```? + A: You may have failed to create the ```${HOME}/.ssh/tmp``` folder or the permissions on this folder are wrong. + * Q: Why do I get the error ```ControlPath too long```? + A: The ```ControlPath ~/.ssh/tmp/%C``` line in your ```${HOME}/.ssh/config``` file expands to a path that is too long. + Change the ```ControlPath``` line in your ```${HOME}/.ssh/config``` file to create a shorter path for the automagically created sockets. + * Q: Why do I get the error ```ssh_exchange_identification: Connection closed by remote host```? + A: Either this server does not exist (anymore). You may have a typo in the name of the server you are trying to connect to. + Check both the command you typed as well as your ```${HOME}/.ssh/config``` for typos in server names. + Or you are using the wrong private key. If your private key is not saved with the default name in the default location, + check if you specified the correct private file both for the ```ProxyCommand``` in your ```${HOME}/.ssh/config``` + as well as with the ```-i``` option for the ```ssh``` command. + * Q: Why do I get the error ```Permission denied (publickey).```? + A: This error can be caused by various configuration issues: + * Either you are using the wrong account name + * or you are using the wrong private key file + * or the permissions on your ```${HOME}/.ssh/``` dir and/or on its content are wrong + * or your account is misconfigured on our account server. + Firstly, check your account name, private key and permissions. + Secondly, check if you can login to the _Jumphost_ with a single hop using + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * If you can login to the _Jumphost_, but cannot use double hop SSH to login to the _UI_ via the _Jumphost_, + you may have to add your private key to the SSH agent on you local machine. + To check which private key(s) are available to your SSH agent you can list them with on your local computer with: + + ssh-add -l + + * If you cannot login and get: + + The agent has no identities. + + then you have to add your private key with the ```ssh-add``` command, which should return output like this: + + Identity added: /path/to/your/home/dir/.ssh/id_ed25519 (key_comment) + + Your private key should now be listed when you check with ```ssh-add -l```, which should look like this: + + 256 SHA256:j/ZNnUvHYW3U$wgIapHw73SnhojjxlWkAcGZ6qDX6Lw key_comment (ED25519) + + If that did not resolve the issue, then increase the verbosity to debug connection problems (see below). + + * Q: Can I increase the verbosity to debug connection problems? + A: Yes try adding ```-vvv``` like this: + ```ssh -vvv youraccount@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` + If that does not help to figure out what is wrong please [contact the helpdesk](../contact/) and + * Do include: + 1. The command you used for your failed login attempt + 2. The output of that failed login attempt with ```-vvv``` debugging enabled + 3. A copy of your ```${HOME}/.ssh/config``` file. + * **Never ever send us your private key**; It does not help to debug your connection problems, but will render the key useless as it is no longer private. + +----- + +Back to operating system independent [instructions for logins](../logins/) + diff --git a/roles/online_docs/templates/mkdocs/docs/logins-macos.md b/roles/online_docs/templates/mkdocs/docs/logins-macos.md new file mode 100644 index 000000000..fa648a5c6 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/logins-macos.md @@ -0,0 +1,150 @@ +#jinja2: trim_blocks:False +# SSH config and login to UI via Jumphost for users on macOS + +The following assumes: + + * you have a ```${HOME}/.ssh``` folder with SSH keys (as generated using the instructions for requesting accounts) + * and that you received a notification with your account name and that your account has been activated + * and that you are on the machine from which you want to connect to the cluster + * and that this machine runs **macOS _Sierra_ 10.12.2 or newer**, which includes OpenSSH 7.3p1 or newer. + Older versions lack several OpenSSH features we need and are incompatible. + +## 1. Configure your SSH client + +We've compiled an AppleScript app to configure your SSH client, which will: + + * Create an SSH client config from scratch if none exists. + * Append to an existing one leaving the config for other servers/machines untouched. + * Update the config for {{ slurm_cluster_name | capitalize }} if the app is executed again. + +#### Quick Install + + * Download the zipped [ssh-client-config-for-{{ slurm_cluster_name }}](../attachments/ssh-client-config-for-{{ slurm_cluster_name }}-macos.zip) AppleScript application. + * Locate and unzip the downloaded archive, which will result in an ```ssh-client-config-for-{{ slurm_cluster_name }}``` application + (optionally with ```.app``` extension depending on your display preferences). + * Start the ```ssh-client-config-for-{{ slurm_cluster_name }}``` app by double clicking in the ```Finder``` application. + * Follow the instructions ... + Check the _Detailed Walkthrough_ below if you experience problems or skip to the *Log in to {{ slurm_cluster_name | capitalize }}* section. + +#### Detailed Walkthrough + +The ```ssh-client-config-for-{{ slurm_cluster_name }}``` app is a wrapper for an installation script that will be executed in the ```Terminal``` application. +It will configure your SSH client by: + + * Adding an ```Include conf.d/*``` directive to your main ```${HOME}/.ssh/config``` file + * Adding a ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` config file for the {{ slurm_cluster_name | capitalize }} specific code. + +The ```ssh-client-config-for-{{ slurm_cluster_name }}``` app will guide you through the following steps: + + * Depending on your macOS version, you may receive a pop-up requesting permission to allow access to the ```Terminal``` application: + ![Allow access to the Terminal.app](img/ssh-client-config-macos-1.png) + Click _Ok_ to allow access to the ```Terminal```. + If you want to revoke this permission or change it back to allow later on, you can do so in + _System Preferences_ -> _Security & Privacy_ prefs -> _Privacy_ tab -> _Automation_ + * The ```ssh-client-config-for-{{ slurm_cluster_name }}``` app will open the configuration script in the ```Terminal``` application and prompt for your account name. + ![Type your account name](img/ssh-client-config-macos-2.png) + Type your account name as you received it from the helpdesk and hit the \[ENTER\] key on your keyboard. + Optionally you can specify an alternative location for your private key file. + (Just hit the \[ENTER\] key to use the default private key file path.) + * Your SSH client will now be configured for logins to {{ slurm_cluster_name }} via the corresponding jumphost + followed by a connection test: the script will try to login using the created config with the account you supplied and the ssh command + ```ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` + Make sure you are connected to the internet and hit the \[ENTER\] key on your keyboard to start the connection test. + ![Test SSH connection](img/ssh-client-config-macos-3.png) + * If this was the first time you use your private key for an SSH session, you will get prompted to supply the password for your private key. + ![Enter password for your private key](img/ssh-client-config-macos-4.png) + Note that this is the password you chose yourself when you created the public private key pair. + We have no backup whatsoever; If you forgot the password, you will have to start over by creating a new key pair. + * Done! Hit the \[ENTER\] key on your keyboard to exit the configuration script. + ![Done](img/ssh-client-config-macos-5.png) + * If you made a mistake, you can simply run the ```ssh-client-config-for-{{ slurm_cluster_name }}``` app again to update/fix your config. + +## 2 Log in to {{ slurm_cluster_name | capitalize }} on the commandline in a Terminal + +Note: If you only need to transfer data and prefer a Graphical User Interface (GUI), you can skip the instructions for working on the commandline below and go straight to +[Keep - What is stored where on {{ slurm_cluster_name | capitalize }}](../storage/) and [Data transfers - How to move data to / from {{ slurm_cluster_name | capitalize }}](../datatransfers/) + +If you want to transfer data using the commandline or analyze data on the cluster using jobs: + + * You can now login to the _UI_ named ```{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` + with the account as specified in your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` + via the _Jumphost_ named ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}``` + using the alias ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}```. + Type the following command in a terminal: + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * In order to override the account name specified in your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` you can use: + + ssh some_other_account@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * If you also added the ```Host *+*+*``` code block from the example ```${HOME}/.ssh/config``` you can do tripple hops starting with a _Jumphost_ like this: + + ssh jumphost+intermediate_server+destination_server + + * In case you are on a network where the default port for _SSH_ (22) is blocked by a firewall you can try to setup _SSH_ over port 443, which is the default for HTTPS and almost always allowed, using an alias like this: + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}443+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + +## Frequent Asked Questions (FAQs) and trouble shooting + + * Q: Why do I get the error ```Bad configuration option: IgnoreUnknown```? + A: Your OpenSSH client is an older one that does not understand the ```IgnoreUnknown``` configuration option. + You have to comment/disable the + ```# Generic stuff: only for macOS clients``` + section listed at the top of the ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` config file. + * Q: Why do I get the error ```muxserver_listen bind(): No such file or directory.```? + A: You may have failed to create the ```${HOME}/.ssh/tmp``` folder or the permissions on this folder are wrong. + * Q: Why do I get the error ```ControlPath too long```? + A: The ```ControlPath ~/.ssh/tmp/%C``` line in your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` file expands to a path that is too long. + Change the ```ControlPath``` line in your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` file to create a shorter path for the automagically created sockets. + * Q: Why do I get the error ```ssh_exchange_identification: Connection closed by remote host```? + A: Either this server does not exist (anymore). You may have a typo in the name of the server you are trying to connect to. + Check both the command you typed as well as your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}``` for typos in server names. + Or you are using the wrong private key. If your private key is not saved with the default name in the default location, + check if the correct private key file is specified both for the ```ProxyCommand``` and ```IdentityFile``` directives + in your ```${HOME}/.ssh/conf.d/{{ slurm_cluster_name }}```. + * Q: Why do I get the error ```Permission denied (publickey).```? + A: This error can be caused by various configuration issues: + * Either you are using the wrong account name + * or you are using the wrong private key file + * or the permissions on your ```${HOME}/.ssh/``` dir and/or on its content are wrong + * or your account is misconfigured on our account server. + Firstly, check your account name, private key and permissions. + Secondly, check if you can login to the _Jumphost_ with a single hop using + + ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} + + * If you can login to the _Jumphost_, but cannot use double hop SSH to login to the _UI_ via the _Jumphost_, + you may have to add your private key to the SSH agent on you local machine. + To check which private key(s) are available to your SSH agent you can list them with on your local computer with: + + ssh-add -l + + * If you cannot login and get: + + The agent has no identities. + + then you have to add your private key with the ```ssh-add``` command, which should return output like this: + + Identity added: /path/to/your/home/dir/.ssh/id_ed25519 (key_comment) + + Your private key should now be listed when you check with ```ssh-add -l```, which should look like this: + + 256 SHA256:j/ZNnUvHYW3U$wgIapHw73SnhojjxlWkAcGZ6qDX6Lw key_comment (ED25519) + + If that did not resolve the issue, then increase the verbosity to debug connection problems (see below). + + * Q: Can I increase the verbosity to debug connection problems? + A: Yes try adding ```-vvv``` like this: + ```ssh -vvv youraccount@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` + If that does not help to figure out what is wrong please [contact the helpdesk](../contact/) and + * Do include: + 1. The command you used for your failed login attempt + 2. The output of that failed login attempt with ```-vvv``` debugging enabled + 3. A copy of your ```${HOME}/.ssh/config``` file. + * **Never ever send us your private key**; It does not help to debug your connection problems, but will render the key useless as it is no longer private. + +----- + +Back to operating system independent [instructions for logins](../logins/) diff --git a/roles/online_docs/templates/mkdocs/docs/logins-windows.md b/roles/online_docs/templates/mkdocs/docs/logins-windows.md new file mode 100644 index 000000000..5e5009ef9 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/logins-windows.md @@ -0,0 +1,74 @@ +#jinja2: trim_blocks:False +# SSH config and login to UI via Jumphost for users on Windows + +The instructions below assume: + + * you've already downloaded _**[MobaXterm](https://mobaxterm.mobatek.net)**_ to generate a pair of SSH keys (using the instructions for requesting accounts) + * and will now use _**MobaXterm**_ to login to the cluster + * and that you received a notification with your account name and that your account has been activated + * and that you are on the machine from which you want to connect to the cluster. + +If you prefer another terminal application consult the corresponding manual. + +###### Launch MobaXterm and create a new session + +![launch MobaXterm](img/MobaXterm5.png) + + * Launch _**MobaXterm**_ and click the _**Session**_ button from the top left of the window. + * A _**Session settings**_ window will popup. + +###### Configure a new session + +![Configure MobaXterm session](img/MobaXterm6.png) + + * Session type + * 1: Select _**SSH**_. + * Basic SSH settings tab + * 2: _Remote host_ field: Use the name of the User Interface (UI) _**{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ . + * 3: _Specify username_ field: Use your _**account name**_ as you received it by email from the helpdesk. + * Advanced SSH settings tab: + * 4: _Use private key_ field: Select the _**private key file**_ you generated previously. + +![Configure MobaXterm session](img/MobaXterm7.png) + + * Network settings tab + * 5: _Gateway SSH server_ field: Use the _Jumphost_ {% if public_ip_addresses is defined and public_ip_addresses | length %}IP address _**{{ public_ip_addresses[groups['jumphost'] | first] }}**_{% else %}address _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}**_{% endif %}. + * Optional: _Port_ field: The default port for SSH is 22 and this is usually fine. + However if you encounter a network where port 22 is blocked, you can try port 443. (Normally used for HTTPS, but our Jumposts can use it for SSH too.) + * 6: _User_ field: Use your _**account name**_ as you received it by email from the helpdesk (same as for 3). + * 7: _Use private key_ field: Select the _**private key file**_ you generated previously (same as for 4). + * 8: Click _**OK**_ + +###### Password (popup) + +![Configure MobaXterm session](img/MobaXterm8.png) + + * MobaXterm should now produce a popup window where you can enter the _**password**_ to decrypt the private key. + * Note this is the password you chose yourself when you created the key pair. + * You are the only one that ever knew this password; we have no copy/backup whatsoever on the server side. + If you forgot the password, the private key is useless and you will have to start over by creating a new key pair. + +###### Password again (prompt) + +![Configure MobaXterm session](img/MobaXterm9a.png) + +MobaXterm should now start a session and login to the _Jumphost_ resulting in + + * a session tab (left part of the window with white background) and + * a terminal where you can type commands (right part of the screen with black background). + +In the terminal tab _**MobaXterm**_ will try to login from the _Jumphost_ to the _User Interface (UI)_ with the same private key file. +This may require retyping the password to decrypt the private key a second time, this time in the terminal tab. + +###### Session established + +You have now logged in to the UI {{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}. + +![Configure MobaXterm session](img/MobaXterm9b.png) + +The left part of the window with white background switched to a file browser, +while the right part remains a terminal where you can type commands. + +----- + +Back to operating system independent [instructions for logins](../logins/) diff --git a/roles/online_docs/templates/mkdocs/docs/logins.md b/roles/online_docs/templates/mkdocs/docs/logins.md index ee15848f6..8903d5f67 100644 --- a/roles/online_docs/templates/mkdocs/docs/logins.md +++ b/roles/online_docs/templates/mkdocs/docs/logins.md @@ -1,297 +1,49 @@ #jinja2: trim_blocks:False # How to start a session and connect to a User Interface server -## User Interface (UI) and jumphost servers +## User Interface (UI) and Jumphost servers To submit jobs, check their status, test scripts, etc. you need to login on a _**User Interface (UI)**_ server using SSH. Each cluster has its own _**UI**_ and the one for the {{ slurm_cluster_name | capitalize }} HPC cluster is named _**{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_. The UI and various other servers that make up the cluster receive updates during scheduled maintenance, -but as this disrupts the processing of jobs scheduled maintenance is planned only ~twice a year. +but as this disrupts the processing of jobs, scheduled maintenance is planned only ~twice a year. Not applying updates for several months could become a serious security risk for machines that are directly accessible via the internet. Therefore the servers of the {{ slurm_cluster_name | capitalize }} cluster are on an internal network that is not directly accessible from the internet. -In order to access the UI you will need to hop via a _**jumphost**_, +In order to access the UI you will need to hop via a _**Jumphost**_, which is a security hardened machine that is not in any way involved in the processing of jobs nor in storing data and does receive daily (security) updates. -In order to apply/activate security patches the jumphost may be temporarily unavailable, which means you cannot login to the _UI_ and hence cannot manage jobs nor create new ones, +In order to apply/activate security patches the _Jumphost_ may be temporarily unavailable, which means you cannot login to the _UI_ and hence cannot manage jobs nor create new ones, but existing jobs (running or queued) won't be affected and the cluster will continue to process those. -The _**jumphost**_ for the {{ slurm_cluster_name | capitalize }} HPC cluster is named _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}**_ +The _**Jumphost**_ for the {{ slurm_cluster_name | capitalize }} HPC cluster is named _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}**_ ## Request an account First make sure you have an account. If you are new, please [follow these instructions to request an account](../accounts/). -## SSH config and login to UI via jumphost for users on macOS, Linux or Unix +## SSH client config and log in to UI via Jumphost -The following assumes: +Configure your SSH client with the instructions for your operating system: -* you have a ```${HOME}/.ssh``` folder with SSH keys (as generated using the instructions for requesting accounts) -* and that you received a notification that your account has been activated -* and that you are on the machine from which you want to connect to the cluster. + * Instructions for [Windows clients](../logins-windows/). + * Instructions for [macOS clients](../logins-macos/). + * Instructions for [Linux/Unix clients](../logins-linux/). -##### 1. Configure Certificate Authority's (CA) public key to verify the identity of cluster servers +## Customize your environment - * Create a ```${HOME}/.ssh/known_hosts``` file if it does not exist yet - * Append the public key from the Certificate Authority we used to sign the host keys of our machines to your ```${HOME}/.ssh/known_hosts``` file. - Open a terminal and type the following commands: +Once logged in you can customize your environment by editing your ```${HOME}/.bashrc``` file on the cluster. +The first few lines that are already present should not be changed unless you want to break your environment, +so please append your custom stuff at the bottom of this file. In case you did corrupt your ```${HOME}/.bashrc```, +you can get a fresh copy from the template located in ```/etc/skel/.bashrc```. - # - # Create new known_hosts file and append the UMCG HPC CA's public key. - # - printf '%s\n' \ - "@cert-authority {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}*,{% if public_ip_addresses is defined and public_ip_addresses[jumphost] | length %}{{ public_ip_addresses[jumphost] }},{% endif %}{% endfor %}{% for adminhost in groups['administration'] %}*{{ adminhost | regex_replace('^' + ai_jumphost + '\\+','')}},{% endfor %}*{{ stack_prefix }}-* {{ lookup('file', ssh_host_signer_ca_private_key+'.pub') }} for {{ slurm_cluster_name }}" \ - > "${HOME}/.ssh/known_hosts.new" - if [[ -e "${HOME}/.ssh/known_hosts" ]]; then - # - # When user already had a known_hosts file, then - # remove a potentially outdated CA public key for the same machines based on the slurm_cluster_name: {{ slurm_cluster_name }} - # and append all other lines to the new known_hosts file. - # - sed '/^\@cert-authority .* for {{ slurm_cluster_name }}$/d' "${HOME}/.ssh/known_hosts" \ - | sort >> "${HOME}/.ssh/known_hosts.new" - fi - # - # Make new known_hosts file the default. - # - mv "${HOME}/.ssh/known_hosts.new" "${HOME}/.ssh/known_hosts" +#### Time Zone -##### 2. Configure transparent multi-hop SSH for logins via the jumphost +The cluster runs in the Coordinated Universal Time (or UTC) time zone, which is not adjusted for daylight saving time. +The latter could confuse software when switching from winter to summer time or back resulting in newer files having older time stamps. +If you prefer to see time stamps in your local time zone, you can set your preferred time zone by configuring the TZ environment variable. +E.g. for the Netherlands: +``` +export TZ=Europe/Amsterdam +``` +If you add this command to your ```${HOME}/.bashrc``` file you can make it the default when you login. +See the [list of time zones on WikiPedia](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for other countries. - * Create a ```${HOME}/.ssh/tmp``` folder. Open a terminal and type the following command: - - mkdir -p -m 700 "${HOME}/.ssh/tmp" - - * Create a ```${HOME}/.ssh/config``` file if it does not exist. - * Add to your ```${HOME}/.ssh/config``` something like the following: - - # - # Generic stuff: only for macOS clients. - # - IgnoreUnknown UseKeychain - UseKeychain yes - IgnoreUnknown AddKeysToAgent - AddKeysToAgent yes - # - # Generic stuff: prevent timeouts - # - Host * - ServerAliveInterval 60 - ServerAliveCountMax 5 - # - # Generic stuff: connection multiplexing - # - # Multiplex connections to - # * reduce lag when logging into the same host in a second shell - # * reduce the amount of connections that are made to prevent excessive DNS lookups - # and to prevent getting blocked by a firewall, because it thinks we are executing a DoS attack. - # - # Name/location of sockets for connection multiplexing are configured using the ControlPath directive. - # In the ControlPath directive %C expands to a hashed value of %l_%h_%p_%r, where: - # %l = local hostname - # %h = remote hostname - # %p = remote port - # %r = remote username - # This makes sure that the ControlPath is - # * a unique socket that is local to machine on which the sessions are created, - # which means it works with home dirs from a shared network file system. - # (as sockets cannot be shared by servers.) - # * not getting to long as the hash has a fixed size not matter how long %l_%h_%p_%r was. - # - ControlMaster auto - ControlPath ~/.ssh/tmp/%C - ControlPersist 1m - # - ## - ### HPC cluster hosts - ## - # - # A. With DNS entry. - #{% if public_ip_addresses is defined and public_ip_addresses | length %}{% for jumphost in groups['jumphost'] %} - Host {{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} - HostName {{ public_ip_addresses[jumphost | regex_replace('^' + ai_jumphost + '\\+','')] }} - User youraccount{% endfor %}{% else %} - Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','') }} {% endfor %}{% if slurm_cluster_domain | length %}!*.{{ slurm_cluster_domain }}{% endif %} - HostName %h{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} - User youraccount{% endif %} - # - # B. Without DNS entry. - # These can only be resolved when already logged in on one of the machines with DNS entry listed above. - # - Host {% for adminhost in groups['administration'] %}*{{ adminhost | regex_replace('^' + ai_jumphost + '\\+','') }} {% endfor %}*{{ stack_prefix }}-* - User youraccount - # - ## - ### Jumphost settings for multi-hop SSH. - ## - # - # The syntax in all the ProxyCommand rules below assumes your private key is in the default location. - # The default location is: - # ${HOME}/.ssh/id_ed25519 - # for keys generated with the ed25519 algorithm. - # In case your private key file is NOT in the default location you must: - # 1. Specify the path to your private key file on the command line when logging in with SSH. - # For example: - # $> ssh -i ${HOME}/.ssh/some_other_private_key_file youraccount@jumphost_server+destination_server - # 2. Add the path to your private key file in the ProxyCommand rules below. - # For example: - # Host jumphost_server+* - # PasswordAuthentication No - # ProxyCommand ssh -X -q -i ${HOME}/.ssh/some_other_private_key_file youraccount@$(echo %h | sed 's/+[^+]*$//').some.sub.domain -W $(echo %h | sed 's/^[^+]*+//'):%p - # - # Universal jumphost settings for triple-hop SSH. - # - Host *+*+* - ProxyCommand ssh -X -q $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p - # - # Double-hop proxy settings for jumphosts{% if slurm_cluster_domain | length %} in {{ slurm_cluster_domain }} domain{% endif %}. - # - Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}+* {% endfor %}{% raw %}{% endraw %} - PasswordAuthentication No - ProxyCommand ssh -X -q youraccount@$(echo %h | sed 's/+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W $(echo %h | sed 's/^[^+]*+//'):%p - # - # Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 80 as fall-back. - # Do not use port 80 by default for SSH as it officially assigned to HTTP traffic and some firewalls will cause problems when trying to route SSH over port 80. - # - Host {% for jumphost in groups['jumphost'] %}{{ jumphost | regex_replace('^' + ai_jumphost + '\\+','')}}80+* {% endfor %}{% raw %}{% endraw %} - PasswordAuthentication No - ProxyCommand ssh -X -q youraccount@$(echo %h | sed 's/+[^+]*$//'){% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %} -W $(echo %h | sed 's/^[^+]*+//'):%p -p 80 - Replace all occurences of _**youraccount**_ with the account name you received from the helpdesk. - If you are **not on a Mac or on a very old Mac** your OpenSSH client may not understand the ```IgnoreUnknown``` configuration option and you may have to comment/disable the - ```# Generic stuff: only for macOS clients``` section listed at the top of the example ```${HOME}/.ssh/config```. - - * Make sure you are the only one who can access your ```${HOME}/.ssh``` folder. Type the following command in a terminal: - - chmod -R go-rwx "${HOME}/.ssh" - -##### 3. Login via jumphost - - * You can now login to the _UI_ named ```{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` with the account as specified in your ```${HOME}/.ssh/config```. - via the _jumphost_ named ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}{% if slurm_cluster_domain | length %}.{{ slurm_cluster_domain }}{% endif %}``` - using the alias ```{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}```. - Type the following command in a terminal: - - ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} - - * In order to override the account name specified in your ```${HOME}/.ssh/config``` you can use: - - ssh some_other_account@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} - - * If you also added the ```Host *+*+*``` code block from the example ```${HOME}/.ssh/config``` you can do tripple hops starting with a jumphost like this: - - ssh jumphost+intermediate_server+destination_server - - * In case you are on a network where the default port for _SSH_ (22) is blocked by a firewall you can try to setup _SSH_ over port 80, which is the default for HTTP and almost always allowed, using an alias like this: - - ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}80+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} - -#### Frequent Asked Questions (FAQs) and trouble shooting - -* Q: Why do I get the error ```muxserver_listen bind(): No such file or directory.```? - A: You may have failed to create the ```${HOME}/.ssh/tmp``` folder or the permissions on this folder are wrong. -* Q: Why do I get the error ```ControlPath too long```? - A: The ```ControlPath ~/.ssh/tmp/%C``` line in your ```${HOME}/.ssh/config``` file expands to a path that is too long. - Change the ```ControlPath``` line in your ```${HOME}/.ssh/config``` file to create a shorter path for the automagically created sockets. -* Q: Why do I get the error ```ssh_exchange_identification: Connection closed by remote host```? - A: Either this server does not exist (anymore). You may have a typo in the name of the server you are trying to connect to. - Check both the command you typed as well as your ```${HOME}/.ssh/config``` for typos in server names. - Or you are using the wrong private key. If your private key is not saved with the default name in the default location, - check if you specified the correct private file both for the ```ProxyCommand``` in your ```${HOME}/.ssh/config``` - as well as with the ```-i``` option for the ```ssh``` command. -* Q: Why do I get the error ```Permission denied (publickey).```? - A: This error can be caused by various configuration issues: - * Either you are using the wrong account name - * or you are using the wrong private key file - * or the permissions on your ```${HOME}/.ssh/``` dir and/or on its content are wrong - * or your account is misconfigured on our account server. - Firstly, check your account name, private key and permissions. - Secondly, check if you can login to the jumphost with a single hop - - ssh {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} - - If you can login to the jumphost, but cannot use double hop SSH to login to the UI via the jumphost, - you may have to add your private key to the SSH agent on you local machine. - To check which private key(s) are available to your SSH agent you can list them with on your local computer with: - - ssh-add -l - - If you get: - - The agent has no identities. - - then you have to add your private key with the ```ssh-add``` command, which should return output like this: - - Identity added: /path/to/your/home/dir/.ssh/id_ed25519 (key_comment) - - Your private key should now be listed when you check with ```ssh-add -l```, which should look like this: - - 256 SHA256:j/ZNnUvHYW3U$wgIapHw73SnhojjxlWkAcGZ6qDX6Lw key_comment (ED25519) - - If that did not resolve the issue, then increase the verbosity to debug connection problems (see below). -* Q: Can I increase the verbosity to debug connection problems? - A: Yes try adding ```-vvv``` like this: - ```ssh -vvv youraccount@{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` - If that does not help to figure out what is wrong please [contact the helpdesk](../contact/) and include: - 1. The command you used for your failed login attempt - 2. The output of that failed login attempt with ```-vvv``` debugging enabled - 3. A copy of your ```${HOME}/.ssh/config``` file. - **Never ever send us your private key**; It does not help to debug your connection problems, but will render the key useless as it is no longer private. - -## SSH config and login to UI via jumphost for users on Windows - -##### 1. Install PuTTY and Pageant - - * Download and install _**[PuTTY](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)**_ - If you downloaded the standalone _PuTTY_ app as opposed to the whole _PuTTY_ suite, then make sure to also download the key manager _Pageant_. - -##### 2. Load your private key in Pageant - - * Start _**Pageant**_ - * Load your private key into _**Pageant**_ - -##### 3. Configure PuTTY for transparent multi-hop SSH for logins via the jumphost - - * Start _**Putty**_ - * Go to _**Connection**_ -> _**SSH**_ -> _**Auth**_ and select _**Allow agent forwarding**_ - * Go to _**Connection**_ -> _**SSH**_ -> _**Auth**_ -> _**Private key file for authentication**_ and add your private key. - * Go to _**Connection**_ -> _**Data**_ and fill in your accountname in the _**auto-login username**_ option - -##### 4. Login via jumphost - -You can now connect to for example UI {{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} -via jumphost {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} using a double hop like this: - -{% if public_ip_addresses is defined and public_ip_addresses | length %}{% for jumphost in groups['jumphost'] %} - * In a _**Putty**_ configuration window supply the _hostname_ _**{{ public_ip_addresses[jumphost] }}**_, your **accountname** and -{% endfor %}{% else %} - * In a _**Putty**_ configuration window supply the _hostname_ _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_, your **accountname** and -{% endif %} - * Click the _**Connect**_ button... - * Once the connection is established type the following command in a terminal: - - ssh youraccount@{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} - -#### 5. Connecting to {{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} via jumphost {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} using MobaXterm. - -MobaXterm for windows is a great toolbox for remote computing. has It has a user friendly interface for supporting drag and drop file transfers directly into the virtual hpc cluster, -but also a UNIX terminal functionality to support basic commands (bash, grep, awk, sed, rsync, etc etc ) or SFTP support. -MobaXterm makes it easy to connect to _**{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ via a jumphost. - -To set up a connection to {{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }} in MobaXterm you do the following: - - * Download and install [MobaXterm](https://mobaxterm.mobatek.net/download.html) - * create a new SSH, session - * put _**{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ in the "remote host" field - * open the "advanced SSH settings" section and import your private key. - * open the "Network settings" section - * check "Connect through SSH gateway \(jump host\)" -{% if public_ip_addresses is defined and public_ip_addresses | length %}{% for jumphost in groups['jumphost'] %} -* fill-in _**{{ public_ip_addresses[jumphost] }}**_ in order to connect to _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ -{% endfor %}{% else %} - * fill-in _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ in order to connect to _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ -{% endif %} - * This will silently create an encrypted SSH tunnel to _**{{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_ and then use this tunnel in order to connect to _**{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}**_. - * Use the 'browser or sftp' tab for the more windows drag and drop interface, or the 'Shell' tab to make se of a terminal interface. - - - - -Fore more advanced information about MobaXterm please have a look at [these instructions to automate such a double hop on Windows](https://mobaxterm.mobatek.net/documentation.html#2_1_5) diff --git a/roles/online_docs/templates/mkdocs/docs/solve-rd.md b/roles/online_docs/templates/mkdocs/docs/solve-rd.md new file mode 100644 index 000000000..b3a7827d7 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/solve-rd.md @@ -0,0 +1,82 @@ +#jinja2: trim_blocks:False +{% if 'hyperchicken' in slurm_cluster_name or 'fender' in slurm_cluster_name %} +# How to find and work with Solve-RD data on {{ slurm_cluster_name | capitalize }} + +#### First: getting an account and starting a session on a User Interface (UI) server + +In order to work with Slurm and manage jobs on the cluster you'll need a personal account and start a session on a User Interface (UI) server. +If you are completely new here, please: + + * [follow these instructions to request an account](../accounts/). + * [follow these instructions to login using your account](../logins/). + +## 1. Using the EGA FUSE client. + +Solve-RD samples are read-only available on {{ slurm_cluster_name | capitalize }} via the +[EGA FUSE client](https://github.com/EGA-archive/ega-fuse-client). +This client uses the **F**ile system in **USE*rspace (FUSE) framework +to make the Solve-RD data available as if it was located on just another disk. +Hence the client will handle both transfer and on the fly decryption of the data. +The mount point (path) on ```prm``` storage where the data is located is: +``` +{{ fuse_mountpoint }} +``` +From there samples can be staged to ```/groups/${group}/tmp*/...``` for analysis. + +Staging means copying a batch of data from the original location on ```prm``` storage to ```tmp``` storage, +which is required as the ```prm``` storage is only available on the _User Interface (UI)_ +whereas ```tmp``` is also mounted on the compute nodes. See +[Keep - What is stored where on {{ slurm_cluster_name | capitalize }}](../storage/) +for details on the differences between various storage systems. +The total Solve-RD data set is huge: Don't try to stage all data simultaneously on a ```tmp``` file system: + + * It won't fit on the storage system and + * Is useless anyway as there is not a single cluster node that can possibly analyse all samples in a single job + +Select a reasonable batch size instead, stage and process those samples, use rigorous QC and when the rsults are Ok, +make a release / data freeze for the results, which can then be moved to a ```prm``` file system for longer term storage +followed by cleanup from ```tmp``` to free up space for a next batch. + +Unfortunately the _EGA FUSE client_ is not so fast, +so you may prefer to use the _EGA FUSE client_ client only to lookup which data is available from which paths +and then do the staging of large files using the _EGA Python API_ with _pyEGA3_ (see below). + +## 2. Using the EGA Python API with pyEGA3. + +To be able to use pyEGA3 you need a username and password provided by the [EGA](https://ega-archive.org). +These EGA credentials are not the same as your account for the {{ slurm_cluster_name | capitalize }} cluster +and must be requested separately. + +Add your credentials into a ```credentials.json ``` file, which is then used during data transfer. +An [example of the syntax for this credentials.json file](https://github.com/EGA-archive/ega-download-client/blob/master/pyega3/config/default_credential_file.json) +is located in the _PyEGA3_ repository at GitHub. +For more on PyEGA3 see the example code snippets below and +[the README.md in the root of the repo](https://github.com/EGA-archive/ega-download-client). + +#### Example for downloading all files corresponding to specific sample from an EGA data set accession number. + +``` +echo 'Loading module pyEGA3 ...' +module load pyEGA3 + +output_dir='/groups/solve-rd/tmp10/username/yourdir' +ega_data_set_accession='EGAD00001005352' +sample_id='E577011' + +mkdir -p -v "${outputdir}" + +echo "Looking up all sampleIDs for data set ${ega_data_set_accession} and storing them into a ${ega_data_set_accession}.tmp file ..." +pyega3 -cf credentials.json files "${ega_data_set_accession}" > "${ega_data_set_accession}.tmp" + +echo "Lookup all EGA file accession numbers corresponding to sample ID ${sample_id} ..." +grep "${sample_id}" "${ega_data_set_accession}.tmp" | awk '{print $1}' >> "${sample_id}.tmp" + +echo "Staging all files for sample ${sample} to "${output_dir}" ..." +for ega_file_accession in $(cat "${sample_id}.tmp") +do + pyega3 -cf credentials.json -c 10 fetch "${ega_file_accession}" --saveto "${output_dir}" +done +``` + +Or see more extended examples here: ```/groups/solve-rd/prm*/example_scripts``` +{% endif %} \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/specifications.md b/roles/online_docs/templates/mkdocs/docs/specifications.md index b436e4de9..0c8a042dd 100644 --- a/roles/online_docs/templates/mkdocs/docs/specifications.md +++ b/roles/online_docs/templates/mkdocs/docs/specifications.md @@ -24,9 +24,9 @@ A Logical File System (LFS) is usually a piece of a larger Physical File System In case it as a network file system you could call it a _share_. In addition to LFS-ses for _home dirs_ and the centrally deployed _software_ and _reference data_ the {{ slurm_cluster_name | capitalize }} HPC cluster has access to the following LFS-ses: - * Available _tmp_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %}/{{ mount.lfs }} {% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} - * Available _prm_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list %}/{{ mount.lfs }} {% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} - * Available _arc_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list %}/{{ mount.lfs }} {% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} + * Available _tmp_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %}{{ mount.lfs }}{% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} + * Available _prm_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list %}{{ mount.lfs }}{% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} + * Available _arc_ LFS-ses: {% if lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list | length %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list %}{{ mount.lfs }}{% if not loop.last %}, {% endif %}{% endfor %}{% else %}None{% endif %} ## Resources available to Slurm jobs diff --git a/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-mobaxterm.md b/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-mobaxterm.md new file mode 100644 index 000000000..eb9b22754 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-mobaxterm.md @@ -0,0 +1,17 @@ +# SSH agent forwarding with MobaXterm on Windows + +SSH agent forwarding can be configured for MobaXterm as follows: + +* Select the _Configuration_ menu item from the _Settings_ menu + ![MobaXterm Configuration](img/MobaXterm10.png) +* Select the _SSH_ tab + ![MobaXterm Configuration](img/MobaXterm11.png) + 1. Enable _**Use internal SSH agent "MobAgent"**_ + 2. Enable _**Forward SSH agents**_ + 3. Click the _**+**_ button to select and load your private key. + +When you now start a new session and login to a server optionally via a jumphost your private key key will be forwarded to the machine you login to. + +----- + +Back to operating system independent [instructions for data transfers](../datatransfers/) \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-openssh.md b/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-openssh.md new file mode 100644 index 000000000..2ea171495 --- /dev/null +++ b/roles/online_docs/templates/mkdocs/docs/ssh-agent-forwarding-openssh.md @@ -0,0 +1,20 @@ +# SSH agent forwarding with OpenSSH on macOS / Linux / Unix + +* Check if your private key was added to the SSH agent on your local _**client**_ by issuing the command + ```$your_client> ssh-add -l``` +* You should get a response with the key fingerprint of the private key you want to use. +* If instead you get the message ```The agent has no identities``` or ```Could not open a connection to your authentication agent```, + then you will need to add your private key: + * If your private key is located in the default path (```${HOME}/.ssh/id_ed25519```) you can use the following command: + ```$your_client> ssh-add``` + * If your key is not located in the default path, you will have to specify which private key file to add: + ```$your_client> ssh-add /path/to/my/private.key``` +* Login to {{ slurm_cluster_name | capitalize }} with SSH _agent forwarding_ enabled + can now be accomplished with the ```-A``` argument on the commandline like this: + ```$your_client> ssh -A {{ groups['jumphost'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}+{{ groups['user-interface'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}``` + +_**Note**_: You **cannot** accomplish this by configuring a ```ProxyCommand``` directive in a ```${HOME}/.ssh/conf.d/*``` config file on your local client computer. + +----- + +Back to operating system independent [instructions for data transfers](../datatransfers/) \ No newline at end of file diff --git a/roles/online_docs/templates/mkdocs/docs/storage.md b/roles/online_docs/templates/mkdocs/docs/storage.md index 843a50017..ca79f7289 100644 --- a/roles/online_docs/templates/mkdocs/docs/storage.md +++ b/roles/online_docs/templates/mkdocs/docs/storage.md @@ -38,9 +38,7 @@ Please consult the info below and make sure you know what to store where! A four year PhD project is not a single experiment! Split your work in batches / experiments that can be completed in a reasonable amount of time: weeks rather than months. Completed means the results were QC-ed and documented, the data that needs to be kept for the long term was migrated to _prm_ storage and the rest was deleted from _tmp_ or _scr_ to make room for new batches / experiments. -## Details - -### Centrally installed software +## Centrally installed software We deploy software with [EasyBuild](https://github.com/easybuilders/easybuild) in a central place on a Deploy Admin Interface (DAI) server. From there the software is synced to various storage devices that are mounted read-only on User Interface (UI) servers and compute nodes. @@ -146,7 +144,7 @@ Instead use the [Lua based module system \(Lmod\)](https://github.com/TACC/Lmod) If the software you need is not yet available, please use the following procedure: - 1. First try to install the software on a UI in a ```/groups/${group}/tmp0*/...``` folder (without EasyBuild). + 1. First try to install the software on a UI in a ```/groups/${group}/tmp*/...``` folder (without EasyBuild). 2. Test the software and evaluate if it is useful to do a proper reproducible deployment. If yes, continue and otherwise cleanup. 3. Depending on time involved in a project: @@ -157,21 +155,21 @@ If the software you need is not yet available, please use the following procedur #### Create your own personal EasyBuild environment and become member of the deploy admins group -* You can use these steps on a UI to - [create your own personal EasyBuild environment](https://gist.github.com/mmterpstra/d11ec81bf78c169ab6be5911df384496) - to deploy software with EasyBuild on a UI in a /groups/${group}/tmp0*/... folder. - Please visit [this page](https://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html) to learn how to make an EasyConfig file. -* Fork our [easybuild-easyconfigs repo on GitHub](https://github.com/molgenis/easybuild-easyconfigs) and create pull request with your newly created EasyConfig(s). -* If you are not a member of the deploy admins group yet: request membership by [sending an email to the helpdesk](../contact/). - Include in your email: - * a link to the pull request. - * the path to the module file created at the end of the deployment with EasyBuild in your own personal EasyBuild environment. - If the EasyConfig is sane and the software was deployed properly, you've passed the test and will be added to the deploy admins group. -* If you already are a member of the deploy admins group: login on a DAI server and deploy with EasyBuild in /apps/... + * You can use these steps on a UI to + [create your own personal EasyBuild environment](https://gist.github.com/mmterpstra/d11ec81bf78c169ab6be5911df384496) + to deploy software with EasyBuild on a UI in a /groups/${group}/tmp*/... folder. + Please visit [this page](https://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html) to learn how to make an EasyConfig file. + * Fork our [easybuild-easyconfigs repo on GitHub](https://github.com/molgenis/easybuild-easyconfigs) and create pull request with your newly created EasyConfig(s). + * If you are not a member of the deploy admins group yet: request membership by [sending an email to the helpdesk](../contact/). + Include in your email: + * a link to the pull request. + * the path to the module file created at the end of the deployment with EasyBuild in your own personal EasyBuild environment. + If the EasyConfig is sane and the software was deployed properly, you've passed the test and will be added to the deploy admins group. + * If you already are a member of the deploy admins group: login on a DAI server and deploy with EasyBuild in /apps/... Note: unless you really need a newer version of the ```foss``` toolchain, we suggest you use the same version as for other software already deployed in the cluster. -### Reference data +## Centrally deployed reference data We deploy reference data sets like for example the human genome in a central place, which is available on all servers: ``` @@ -180,7 +178,7 @@ We deploy reference data sets like for example the human genome in a central pla Please use them from that location as opposed to downloading yet another copy elsewhere. If your pet reference data set is missing [contact the helpdesk via email](../contact/) to have it added. -### Your personal home dir @ /home/${user} +## Your personal home dir @ /home/${user} This is were you have limited space to store your personal settings/preferences/configs like terminal colors, terminal font sizes, etc. Your home is available on all servers of a cluster, but different clusters have separate homes. @@ -192,9 +190,9 @@ A typical home dir contains << 100 Mb of data: Important: -* Your home is designed to be a **private folder**; do not try to change permissions to share data located in your home with other users. -* Your home is on _HA_ and hence not on _HP_ storage. Therefore you should try to minimize the IO load on your home to make sure everyone can enjoy a fast responsive home. -* Do not abuse your home dir, so: + * Your home is designed to be a **private folder**; do not try to change permissions to share data located in your home with other users. + * Your home is on _HA_ and hence not on _HP_ storage. Therefore you should try to minimize the IO load on your home to make sure everyone can enjoy a fast responsive home. + * Do not abuse your home dir, so: * Don't waste resources by installing in your private home dir yet another copy of the same software package that is already available centrally from the module system. * Don't run anything that causes massive random IO on your home dir. E.g. don't store job scripts submitted to cluster nodes in homes. @@ -202,9 +200,9 @@ Important: Your home is for personal preferences; not for experiments. Use a group dir for the latter (see below). -### Group dirs @ /groups/${group}/... +## Group dirs @ /groups/${group}/... -Every user is a member of at least one group. A group has access to large shared storage systems of which we have 4 types: +Every user is a member of at least one main group. A main group has access to large shared storage systems of which we have 4 types: * ```/groups/${group}/prm*/: PeRManent dirs```: large, fast dirs for rawdata and final results * ```/groups/${group}/arc*/: ARChive dirs```: large, slow dirs for archived rawdata and final results @@ -212,22 +210,27 @@ Every user is a member of at least one group. A group has access to large shared * ```/groups/${group}/scr*/: SCRatch dirs```: small, fastest dirs for _local_ temporary data Not all groups have access to all types of storage systems and not all types are available on all clusters. -The minimal requirements for a group are as follows: - -* Group leaders / PIs can request new groups. When the group is created they will be registered as the group owners. -* Group owners are responsible for - * Processing (accepting or rejecting) requests for group membership. - * Securing funding and paying the bills. - * Appointing data managers for their group. -* Data managers are responsible for the group's data on ```prm``` and ```arc``` storage systems. - * They ensure the group makes arrangements what to store how and where. E.g file naming conventions, file formats to use, etc. - * They enforce the group's policy on what to store how and where by reviewing data sets produced by other group members on ```tmp``` or ```scr``` file systems before migrating/copying them to ```prm``` and ```arc```. - * They have read-write access to all file systems including ```prm``` and ```arc```. -* Other 'regular' group members: - * Have read-only access to ```prm``` and ```arc``` file systems to check-out existing data sets. - * Have read-write access to ```tmp``` and ```scr``` file systems to produce new results. - * Can request a data manager to review and migration a newly produced data set to ```prm``` or ```arc``` file systems. -* A group has at least one owner and one data manager, but to prevent delays in processing membership request and data set reviews a group has preferably more than one owner and more than one data manager. +The minimal requirements for a main group are as follows: + + * Group leaders / PIs can request new main groups. When the main group is created they will be registered as the group owners. + * Group owners are responsible for + * Processing (accepting or rejecting) requests for group membership. + * Securing funding and paying the bills. + * Appointing data managers for their group. + * Data managers are responsible for the group's data on ```prm``` and ```arc``` storage systems. + * They ensure the group makes arrangements what to store how and where. E.g file naming conventions, file formats to use, etc. + * They enforce the group's policy on what to store how and where by reviewing data sets produced by other group members on ```tmp``` or ```scr``` file systems before migrating/copying them to ```prm``` and ```arc```. + * They have read-write access to all file systems including ```prm``` and ```arc```. + * Other _regular_ group members: + * Have read-only access to ```prm``` and ```arc``` file systems to check-out existing data sets. + * Have read-write access to ```tmp``` and ```scr``` file systems to produce new results. + * Can request a data manager to review and migrate a newly produced data set to ```prm``` or ```arc``` file systems. + * A group has at least one owner and one data manager, but to prevent delays in processing membership request and data set reviews a group has preferably more than one owner and more than one data manager. + * Optionally sub groups may be used to create more fine grained permissions to access data. + * A sub group inherits group owners, data managers and quota limits from the main group. + E.g. a sub group cannot have different data managers. + * All members of the sub group must be members of the main group. + * The members of the sub group are a subset of the members of the main group. ## Quota @@ -237,10 +240,10 @@ Home directories have _user_ quota, which means that if you run out of space, yo All other file systems use _group_ or _file set_ quota, which means that if you run out of space everybody from your group (or file set) is also out of space on that file system, but other groups are not affected. There are two limits and a timer period that determines how these interact: -* **quota (soft)**: exceed your quota and you can still write data until you've reached the (hard) limit or until the timer that kicks in expires whichever comes first. -* **limit (hard)**: exceed your (hard) limit and you are instantly prohibited from writing any data. You will need to delete something else to free up space before you can write new data. -* **timers**: after exceeding your quota the timer kicks in and if you do not reduce your data volume to less than your quota, the soft quota will temporarily become your hard limit when the timer expires. - You will need to reduce your data volume to less than your quota to reset the timer as well as the (hard) limit. + * **quota (soft)**: exceed your quota and you can still write data until you've reached the (hard) limit or until the timer that kicks in expires whichever comes first. + * **limit (hard)**: exceed your (hard) limit and you are instantly prohibited from writing any data. You will need to delete something else to free up space before you can write new data. + * **timers**: after exceeding your quota the timer kicks in and if you do not reduce your data volume to less than your quota, the soft quota will temporarily become your hard limit when the timer expires. + You will need to reduce your data volume to less than your quota to reset the timer as well as the (hard) limit. The combination of quota, larger limits and timers prevents users from permanently exceeding their quota while allowing them to temporarily consume more space to handle peak loads. Note that if you write a lot of data and fast it is possible to exceed both your quota as well as the larger limit in a time frame that is much shorter than the quota reporting interval. @@ -272,23 +275,13 @@ The report will show 11 columns: ``` ## List of storage devices / mount points used on {{ slurm_cluster_name | capitalize }} - | Path | Function | (Soft) Quota | (Hard) Limit | Backup | Cleanup | Mounted on UIs | Mounted on DAIs | Mounted on compute nodes | |:---- |:-------- | ----------:| ----------:|:------:|:-------:|:--------------:|:---------------:|:------------------------:| -{% for mount in lfs_mounts | selectattr('lfs', 'match', '^home$') | list %} -| /{{ mount.lfs }} | Home dirs from shard file system for personal settings/preferences. | 1 GB | 2 GB | Yes | No | Yes | Yes | Yes | -{% endfor %} -{% for mount in lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list %} -| /{{ mount.lfs }} | High Availability shared file system for permanent data. | Several TBs; varies per group | quota + ~10%| Yes | No | Yes | No | No | -{% endfor %} -{% for mount in lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list %} -| /{{ mount.lfs }} | High Availability shared file system for archived data. | Several TBs; varies per group | quota + ~10%| Yes | No | Yes | No | No | -{% endfor %} -{% for mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %} -| {{ mount.lfs }} | High Performance shared file system for temporary data. | Several TBs; varies per group | quota + ~10% | No | Yes, when older than 45 days | Yes | No | Yes | -{% endfor %} -{% for mount in lfs_mounts | selectattr('lfs', 'search', 'scr[0-9]+$') | list %} -| /{{ mount.lfs }} | High Performance local file system for temporary data. | Several TBs; varies per group | quota + ~10% | No | Yes, when older than 45 days | Yes | No | No | +{% for mount in lfs_mounts | selectattr('lfs', 'match', '^home$') | list %}| ```/{{ mount.lfs }}``` | Home dirs from shared file system for personal settings/preferences. | 1 GB | 2 GB | Yes | No | Yes | Yes | Yes | +{% endfor %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list %}| ```/groups/${group}/{{ mount.lfs }}``` | High Availability shared file system for permanent data. | Several TBs; varies per group | quota + ~10%| Yes | No | Yes | No | No | +{% endfor %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'arc[0-9]+$') | list %}| ```/groups/${group}/{{ mount.lfs }}``` | High Availability shared file system for archived data. | Several TBs; varies per group | quota + ~10%| Yes | No | Yes | No | No | +{% endfor %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %}| ```/groups/${group}/{{ mount.lfs }}``` | High Performance shared file system for temporary data. | Several TBs; varies per group | quota + ~10% | No | Yes, when older than 45 days | Yes | No | Yes | +{% endfor %}{% for mount in lfs_mounts | selectattr('lfs', 'search', 'scr[0-9]+$') | list %}| ```/groups/${group}/{{ mount.lfs }}``` | High Performance local file system for temporary data. | Several TBs; varies per group | quota + ~10% | No | Yes, when older than 45 days | Yes | No | No | {% endfor %} ## The life cycle of experimental data diff --git a/roles/online_docs/templates/mkdocs/mkdocs.yml b/roles/online_docs/templates/mkdocs/mkdocs.yml index 3aaf014ea..1fefc0c8a 100644 --- a/roles/online_docs/templates/mkdocs/mkdocs.yml +++ b/roles/online_docs/templates/mkdocs/mkdocs.yml @@ -5,6 +5,8 @@ theme: hljs_languages: - yaml site_dir: "/var/www/html/{{ slurm_cluster_name }}/" +extra_css: + - css/custom.css nav: - Home: 'index.md' - About: @@ -18,4 +20,8 @@ nav: - 'Data transfers - How to move data to / from {{ slurm_cluster_name | capitalize }}': 'datatransfers.md' - 'Crunch - How to manage jobs on {{ slurm_cluster_name | capitalize }}': 'analysis.md' - 'Technical specifications - Details of the {{ slurm_cluster_name | capitalize }} HPC environment': 'specifications.md' -# - 'Support - FAQs, mailinglists & courses': 'support.md' \ No newline at end of file +{% if 'hyperchicken' in slurm_cluster_name or 'fender' in slurm_cluster_name %} + - Group specific documentation: + - 'Solve-RD - How to find and work with Solve-RD data': 'solve-rd.md' +{% endif %} +# - 'Support - FAQs, mailinglists & courses': 'support.md' diff --git a/roles/openldap/tasks/main.yml b/roles/openldap/tasks/main.yml index 55a060d97..11b46ee45 100644 --- a/roles/openldap/tasks/main.yml +++ b/roles/openldap/tasks/main.yml @@ -1,9 +1,9 @@ --- -- name: install service file. +- name: Install service files. template: - src: templates/{{ item }} - dest: /etc/systemd/system/{{ item }} - mode: 644 + src: "templates/{{ item }}" + dest: "/etc/systemd/system/{{ item }}" + mode: 0600 owner: root group: root with_items: @@ -11,41 +11,33 @@ - phpldapadmin.service become: true -- name: make directory for the openldap ldif volume. +- name: Make directory for the openldap ldif volume. file: - path: /srv/openldap/custom + path: "/srv/openldap/custom" state: directory owner: root group: root become: true -- name: install ldif file for "member of" +- name: Install ldif files for overlays. copy: - src: files/01-overlay-memberof - dest: /srv/openldap/custom/01-overlay-memberof - mode: 644 + src: "files/{{ item }}" + dest: "/srv/openldap/custom/{{ item }}" + mode: 0644 owner: root group: root + with_items: + - 01-overlay-memberof become: true -- name: Daemon reload (the inplicit doesn't work) - command: bash -c "systemctl daemon-reload" - become: true - -- name: make sure service is started +- name: Make sure services are started. systemd: name: "{{ item }}" state: started + enabled: yes daemon_reload: yes with_items: - openldap.service - phpldapadmin.service become: true - -- name: start service at boot. - command: systemctl reenable "{{ item }}" - with_items: - - openldap.service - - phpldapadmin.service - become: true ... diff --git a/roles/openldap/templates/phpldapadmin.service b/roles/openldap/templates/phpldapadmin.service index a55cb0877..1075e46a9 100644 --- a/roles/openldap/templates/phpldapadmin.service +++ b/roles/openldap/templates/phpldapadmin.service @@ -10,7 +10,7 @@ ExecStartPre=-/usr/bin/docker kill %n ExecStartPre=-/usr/bin/docker rm %n ExecStartPre=/usr/bin/docker pull osixia/phpldapadmin:0.7.2 ExecStart=/usr/bin/docker run -i --name %n --network host \ - --env PHPLDAPADMIN_LDAP_HOSTS={{ uri_ldap }} \ + --env PHPLDAPADMIN_LDAP_HOSTS={{ ldap_uri | regex_replace('^ldaps?://','')}} \ osixia/phpldapadmin:0.7.2 [Install] diff --git a/roles/prom_proxy/templates/nginx.conf b/roles/prom_proxy/templates/nginx.conf deleted file mode 100644 index e30511b51..000000000 --- a/roles/prom_proxy/templates/nginx.conf +++ /dev/null @@ -1,54 +0,0 @@ -# For more information on configuration, see: -# * Official English Documentation: http://nginx.org/en/docs/ -# * Official Russian Documentation: http://nginx.org/ru/docs/ - -user nginx; -worker_processes auto; -error_log /var/log/nginx/error.log; -pid /run/nginx.pid; - -# Load dynamic modules. See /usr/share/nginx/README.dynamic. -include /usr/share/nginx/modules/*.conf; - -events { - worker_connections 1024; -} - -http { - log_format main '$remote_addr - $remote_user [$time_local] "$request" ' - '$status $body_bytes_sent "$http_referer" ' - '"$http_user_agent" "$http_x_forwarded_for"'; - - access_log /var/log/nginx/access.log main; - - sendfile on; - tcp_nopush on; - tcp_nodelay on; - keepalive_timeout 65; - types_hash_max_size 2048; - - include /etc/nginx/mime.types; - default_type application/octet-stream; - - # Load modular configuration files from the /etc/nginx/conf.d directory. - # See http://nginx.org/en/docs/ngx_core_module.html#include - # for more information. - include /etc/nginx/conf.d/*.conf; - - server { - listen 9090 default_server; - server_name {{ ansible_hostname }}; - - location / { - proxy_pass http://{{ groups['slurm-management'] | first | regex_replace('^' + ai_jumphost + '\\+','') }}:9090; - - auth_basic "Restricted Content"; - auth_basic_user_file /etc/nginx/.htpasswd; - - } - - } - - -} - diff --git a/roles/prom_server/handlers/main.yml b/roles/prom_server/handlers/main.yml index de5e2e7a2..734cbd767 100644 --- a/roles/prom_server/handlers/main.yml +++ b/roles/prom_server/handlers/main.yml @@ -2,7 +2,7 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # # Restart before reload: an reload after a restart may be redundant but should not fail, # but the other way around may fail when the impact of changes was too large for a reload. diff --git a/roles/prom_server/tasks/main.yml b/roles/prom_server/tasks/main.yml index c1d24419b..cabeeb2c6 100644 --- a/roles/prom_server/tasks/main.yml +++ b/roles/prom_server/tasks/main.yml @@ -1,62 +1,62 @@ --- -- name: Create directories for Prometheus. +- name: 'Create directories for Prometheus.' file: path: "{{ item }}" state: directory - mode: 0755 + mode: '0755' owner: '65534' with_items: - '/srv/prometheus/etc/prometheus' - '/srv/prometheus/prometheus' notify: - - restart_prometheus + - 'restart_prometheus' become: true -- name: Install settings files. +- name: 'Install settings files.' copy: src: "templates/etc/{{ item }}" dest: "/srv/prometheus/etc/prometheus/{{ item }}" - mode: 0644 + mode: '0644' owner: root group: root with_items: - - alerting.rules - - targets.json + - 'alerting.rules' + - 'targets.json' notify: - - restart_prometheus + - 'restart_prometheus' become: true -- name: Install settings files based on templates. +- name: 'Install settings files based on templates.' template: src: 'templates/etc/prometheus.yml' dest: '/srv/prometheus/etc/prometheus/prometheus.yml' - mode: 0644 + mode: '0644' owner: root group: root notify: - - restart_prometheus + - 'restart_prometheus' become: true -- name: Install service files. +- name: 'Install service files.' template: src: 'templates/prometheus.service' dest: '/etc/systemd/system/prometheus.service' - mode: 644 + mode: '0644' owner: root group: root tags: - - service-files + - 'service-files' notify: - - restart_prometheus + - 'restart_prometheus' become: true -- name: Make sure prometheus service is started and enabled on (re)boot. +- name: 'Make sure prometheus service is started and enabled on (re)boot.' systemd: - name: prometheus.service + name: 'prometheus.service' enabled: yes - state: started + state: 'started' daemon_reload: yes tags: - - start-service + - 'start-service' become: true ... \ No newline at end of file diff --git a/roles/prom_server/templates/etc/alerting.rules b/roles/prom_server/templates/etc/alerting.rules index d72d6599e..dc68c28f7 100644 --- a/roles/prom_server/templates/etc/alerting.rules +++ b/roles/prom_server/templates/etc/alerting.rules @@ -43,7 +43,7 @@ groups: description: Instance {{ $labels.instance }} will fill up within 72 hours summary: '{{ $labels.instance }} disk almost full' - alert: DiskFull - expr: node_filesystem_free{job="node",mountpoint!~"/tmp|/net|/cvmfs|/var/lib/nfs/rpc_pipefs|/cvmfs|/misc|/run/docker/netns/.+?|/cgroup.+?", fstype!~"fuse.+?"} < 5.24288e+06 + expr: node_filesystem_free{job="node",mountpoint!~"/net|/cvmfs|/var/lib/nfs/rpc_pipefs|/cvmfs|/misc|/run/docker/netns/.+?|/cgroup.+?", fstype!~"fuse.+?"} < 5.24288e+06 for: 5m labels: severity: page @@ -66,3 +66,11 @@ groups: annotations: description: Instance {{ $labels.instance }} has been rebooted. summary: '{{ $labels.instance }} rebooted' + - alert: Node drained. + expr: min_over_time(slurm_nodes_drain{instance="gearshift:9102",job="slurm_exorter"}[1h]) < slurm_nodes_drain{instance="gearshift:9102",job="slurm_exorter"} + for: 5m + labels: + severity: page + annotations: + description: 'We detected an increase in drained nodes.' + summary: 'Drained nodes increased' diff --git a/roles/prom_server/templates/etc/targets.json b/roles/prom_server/templates/etc/targets.json index c5159b8c5..78de3abdc 100644 --- a/roles/prom_server/templates/etc/targets.json +++ b/roles/prom_server/templates/etc/targets.json @@ -25,8 +25,7 @@ "gs-compute07:9100", "gs-compute08:9100", "gs-compute09:9100", - "gs-compute10:9100", - "gs-compute11:9100" + "gs-compute10:9100" ], "labels": { "env": "gearshift", diff --git a/roles/regular-users/tasks/ldap-regular-users.yml b/roles/regular-users/tasks/ldap-regular-users.yml index 5cde1b266..616794999 100644 --- a/roles/regular-users/tasks/ldap-regular-users.yml +++ b/roles/regular-users/tasks/ldap-regular-users.yml @@ -5,7 +5,7 @@ objectClass: 'organizationalUnit' bind_dn: "{{ ldap_binddn }}" bind_pw: "{{ bindpw }}" - server_uri: 'ldap:///' + server_uri: "{{ ldap_uri }}" - name: 'Make sure we have a parent entry for groups.' ldap_entry: @@ -13,25 +13,37 @@ objectClass: 'organizationalUnit' bind_dn: "{{ ldap_binddn }}" bind_pw: "{{ bindpw }}" - server_uri: 'ldap:///' + server_uri: "{{ ldap_uri }}" -- name: 'Add groups to parent entry for groups.' +- name: 'Add regular groups to parent entry for groups.' ldap_entry: dn: "cn={{ item }},ou=groups,{{ ldap_base }}" objectClass: 'posixGroup' state: present - server_uri: 'ldap:///' + server_uri: "{{ ldap_uri }}" bind_dn: "{{ ldap_binddn }}" bind_pw: "{{ bindpw }}" attributes: gidNumber: "{{ auth_groups[item].gid }}" with_items: "{{ regular_groups }}" +- name: 'Add private groups for home dirs of users to parent entry for groups.' + ldap_entry: + dn: "cn={{ item.user }},ou=groups,{{ ldap_base }}" # Use same name as user's account name for user's private group. + objectClass: 'posixGroup' + state: present + server_uri: "{{ ldap_uri }}" + bind_dn: "{{ ldap_binddn }}" + bind_pw: "{{ bindpw }}" + attributes: + gidNumber: "{{ auth_users[item.user].uid }}" # Use same GID as user's UID for user's private group. + with_items: "{{ regular_users }}" + - name: 'Add users to parent entry for users.' ldap_entry: dn: "cn={{ item.user }},ou=users,{{ ldap_base }}" state: present - server_uri: 'ldap:///' + server_uri: "{{ ldap_uri }}" objectClass: - 'inetOrgPerson' - 'posixAccount' @@ -45,17 +57,28 @@ cn: "{{ item.user }}" uid: "{{ item.user }}" uidNumber: "{{ auth_users[item.user].uid }}" - gidNumber: "{{ auth_groups['users'].gid }}" + gidNumber: "{{ auth_users[item.user].uid }}" # primary group == private group for user. homeDirectory: "/home/{{ item.user }}" loginShell: '/bin/bash' sshPublicKey: "{{ auth_users[item.user].pub_keys }}" with_items: "{{ regular_users }}" +- name: 'Update public keys.' + ldap_attr: + dn: "cn={{ item.user }},ou=users,{{ ldap_base }}" + state: exact + server_uri: "{{ ldap_uri }}" + bind_dn: "{{ ldap_binddn }}" + bind_pw: "{{ bindpw }}" + name: 'sshPublicKey' + values: "{{ auth_users[item.user].pub_keys }}" + with_items: "{{ regular_users }}" + - name: 'Add users to groups.' ldap_attr: dn: "cn={{ item.1 }},ou=groups,{{ ldap_base }}" state: present - server_uri: 'ldap:///' + server_uri: "{{ ldap_uri }}" bind_dn: "{{ ldap_binddn }}" bind_pw: "{{ bindpw }}" name: 'memberUid' diff --git a/roles/resolver/handlers/main.yml b/roles/resolver/handlers/main.yml index c7154496a..849892b06 100644 --- a/roles/resolver/handlers/main.yml +++ b/roles/resolver/handlers/main.yml @@ -2,9 +2,9 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # -- name: Restart dnsmasq service. +- name: 'Restart dnsmasq service.' service: name: dnsmasq state: restarted diff --git a/roles/resolver/tasks/main.yml b/roles/resolver/tasks/main.yml index 600578c80..38b3e2ced 100644 --- a/roles/resolver/tasks/main.yml +++ b/roles/resolver/tasks/main.yml @@ -1,39 +1,39 @@ --- -- name: Install dnsmasq +- name: 'Install dnsmasq.' yum: - state: latest + state: 'latest' update_cache: yes name: - - dnsmasq + - 'dnsmasq' become: true - notify: restart_dnsmasq + notify: 'restart_dnsmasq' -- name: Configure /etc/dnsmasq.conf to use nameservers as listed in group_vars for this cluster. +- name: 'Configure /etc/dnsmasq.conf to use nameservers as listed in group_vars for this cluster.' template: dest: '/etc/dnsmasq.conf' src: 'templates/dnsmasq.conf.j2' owner: root group: root - mode: 0644 + mode: '0644' become: true - notify: restart_dnsmasq + notify: 'restart_dnsmasq' -- name: Configure /etc/resolv.conf to use dnsmasq on localhost. +- name: 'Configure /etc/resolv.conf to use dnsmasq on localhost.' copy: dest: '/etc/resolv.conf' - content: nameserver 127.0.0.1 + content: 'nameserver 127.0.0.1' owner: root group: root - mode: 0644 + mode: '0644' become: true notify: restart_dnsmasq - -- name: Enable dnsmasq service. + +- name: 'Enable dnsmasq service.' systemd: name: 'dnsmasq.service' enabled: yes become: true - notify: restart_dnsmasq + notify: 'restart_dnsmasq' -- meta: flush_handlers +- meta: 'flush_handlers' ... \ No newline at end of file diff --git a/roles/rsyncd/meta/main.yml b/roles/rsyncd/meta/main.yml new file mode 100644 index 000000000..440138716 --- /dev/null +++ b/roles/rsyncd/meta/main.yml @@ -0,0 +1,8 @@ +# +# The rsyncd role depends on the ssh_ldap_wrapper from the ldap role +# to create "rsync-only" user accounts by prefixing public keys with a "forced command". +# +--- +dependencies: + - { role: ldap } +... \ No newline at end of file diff --git a/roles/rsyncd/tasks/main.yml b/roles/rsyncd/tasks/main.yml new file mode 100644 index 000000000..f2151b91c --- /dev/null +++ b/roles/rsyncd/tasks/main.yml @@ -0,0 +1,24 @@ +# +# Configure rsync deamon config file. +# * For use with rsync-only accounts that connect using a remote shell +# and then spwan a single-use daemon server. +# * Hence there is no systemd managed rsyncd running constantly +# and therfore no handler to (re)start a daemon. +# +--- +- name: 'Install rsync.' + yum: + state: 'latest' + update_cache: true + name: 'rsync' + become: true + +- name: 'Install rsyncd config file with "modules" for /home and /groups.' + template: + src: 'templates/rsyncd.conf' + dest: '/etc/rsyncd.conf' + owner: 'root' + group: 'root' + mode: '0644' + become: true +... \ No newline at end of file diff --git a/roles/rsyncd/templates/rsyncd.conf b/roles/rsyncd/templates/rsyncd.conf new file mode 100644 index 000000000..14c6f543c --- /dev/null +++ b/roles/rsyncd/templates/rsyncd.conf @@ -0,0 +1,39 @@ +#jinja2: trim_blocks:False + +# /etc/rsyncd: configuration file for rsync daemon mode. + +# +# Note: +# 1. This file is controlled by Ansible. +# 2. We don't use a permanently running rsyncd managed by systemd, +# but instead spawn single-use daemon servers from remote shells. +# + +# transfer logging = yes +address = localhost 127.0.0.1 +hosts allow = localhost 127.0.0.1 {% for server in groups['jumphost'] %}{{ server | regex_replace('^' + ai_jumphost + '\\+','')}}{% if not loop.last %} {% endif %}{% endfor %} +# +# chroot cannot be used with single-use deamon servers spawned on demand by unprivileged users. +# +use chroot = no +ignore nonreadable = yes +exclude = lost+found/ .DS_Store .Trash +dont compress = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2 +# +# Sane defaults for permissions: +# Strict incoming perms, which are compatible with how we work with groups on our servers. +# More lenient outgoing perms to allow clients some freedom in setting perms, +# but remove all perms for others to prevent data becoming accessible for anyone on a system after download. +# +incoming chmod = Du=rwx,Dg=rsx,Fu=rw,Fg=r,o-rwx +outgoing chmod = o-rwx + +[home] + path = /home/%RSYNC_USER_NAME%/ + read only = false + munge symlinks = no + +[groups] + path = /groups/ + read only = false + munge symlinks = no diff --git a/roles/shared_storage/README.md b/roles/shared_storage/README.md new file mode 100644 index 000000000..7c17ec489 --- /dev/null +++ b/roles/shared_storage/README.md @@ -0,0 +1,190 @@ +# Ansible Role: shared_storage + +This role mounts parts of shared storage systems (shares a.k.a. exports) on various machines of an HPC cluster depending on their purpose. + +## Terminology + + * **PFS**: **P**hysical **F**ile **S**ystem. Could also be actually virtualised nowadays, but always something only sys admins should worry about. + We never bother end users on an HPC cluster with _PFS-ses_. + * **LFS**: **L**ogical **F**ile **S**ystem. A piece of a (shared/local) storage system for a specific purpose and a specific group of users. + This is how end users see and work with storage systems: _PFS_ naming is consistent and remains constant no matter on which _PFS_ the _LFS_ is located. + When a _PFS_ is End of Life and phased out, the data can be migrated to a different _PFS_ keeping the _LFS_ and hence the path to the data for end users exactly the same. + * **home**: An _LFS_ for user's home directories. + * **tmp[0-9]{2}**: An _LFS_ for **t**e**mp**orary data. + * On a file system optimized for High Performance (HP) where possible. + * Without backups. + * with automagic cleanup of old data. + * By design mounted on both _User Interface (UI)_ and _compute nodes_ of an HPC cluster. + * **prm[0-9]{2}**: An _LFS_ for **p**e**rm**anent data: + * On a file system optimized for High Availability (HA) where possible. + * With backups. + * Without automagic cleanup of old data. + * By design only mounted on the _User Interface (UI)_ servers of an HPC cluster. + In order to crunch data from a _prm_ _LFS_ it must be staged to a _tmp_ _LFS_ first. + * **apps**: An _LFS_ for the original of the shared environment (software, modules, reference data). + * By design mounted read-write and only on the _Deploy Admin Interface (DAI)_ in **/apps**. + * Preferably on a local file system and not on shared storage as the latter is known to be horribly slow + for many ```module load/unload``` commands, which are used during dependency management, and for building software from sources. + * **env[0-9]{2}**: An _LFS_ for a copy of the shared environment (software, modules, reference data). + * By design mounted read-only on both _User Interface (UI)_ and _compute nodes_ of an HPC cluster in **/apps** + * By design mounted read-write on the _Deploy Admin Interface (DAI)_ of an HPC cluster in **/mnt/env[0-9]{2}** + The _DAI_ also contains the read-write primary copy / original of the shared environment from a local storage device and mounted in **/apps** + An rsync-based tool is used to synchronise **/apps** to all copies in **/mnt/env[0-9]{2}**, + which is read-only mounted in **/apps** on the _UI_ and _compute nodes_. + +An HPC cluster can have + * Only one **home** and one **apps** _LFS_ + * Multiple **tmp[0-9]**, **prm[0-9]** and **env[0-9]** _LFS-ses_. We keep the numbers unique over all clusters. + Hence there is only one _prm03_. If multiple machines optionally from multiple clusters have an _prm03_ _LFS_ mount, + it must contain the same data with the same permissions. + +## group_vars + +The _LFS-ses_ and _PFS-ses_ they are located on must be specified for each HPC cluster in the corresponding _group\_vars_ located at ```groups_vars/${name}-cluster/vars.yml``` +An example snippet (see below for explanation of what get's mounted where based on this example): +``` +--- +# +# Other group_vars +# +################################################################################################### +# +# Physical File Systems (PFS-ses). +# +pfs_mounts: [ + { pfs: 'isilon11', + source: 'some-storage001.stor.local:/ifs', + type: 'nfs4', + rw_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime', + ro_options: 'defaults,_netdev,vers=4.0,noatime,nodiratime,ro' }, + { pfs: 'lusty2', + source: '10.0.0.203@tcp12:10.0.0.204@tcp12:/lusty2', + type: 'lustre', + rw_options: 'defaults,lazystatfs,flock', + ro_options: 'defaults,lazystatfs,ro' }, +] +# +# Logical File Systems (LFS-ses) with: +# * specification of what needs to get mounted where or for which groups. +# * on which PFS an LFS is located. +# +lfs_mounts: [ + { lfs: 'home', + pfs: 'isilon11', + machines: "{{ groups['cluster'] }}" }, + { lfs: 'env08', + pfs: 'isilon11', + machines: "{{ groups['compute-vm'] + groups['user-interface'] }}" }, + { lfs: 'tmp08', + pfs: 'isilon11', + groups: [ + 'ateam', 'colla', 'production', 'cool-project' + ]}, + { lfs: 'prm05', + pfs: 'isilon11', + groups: [ + 'ateam', 'production' + ]}, + { lfs: 'prm03', + pfs: 'lusty2', + groups: [ + 'colla', 'production', 'cool-project' + ]}, +] +################################################################################################### +# +# Other group_vars +# +... +``` + +#### pfs_mounts + +The **pfs_mounts** variable lists all Physical File Systems and their technical specs required by the ```mount``` command. Per _PFS_: + + * **pfs**: a label for this _PFS_. + * Preferably only lowercase characters a-z. + * Used as the mountpoint in **/mnt/** on the _Sys Admin Interface (SAI)_. + * Usually the same as a (sub)folder from the file system. + Can be either the root folder if the entire file system is used only as this _PFS_ + or a subfolder when the same storage system is used by other systems / clients too. + * Optionally the label may be suffixed by a slash forward and the subfolder. + This can be used when the parent dir of the subfolder contains other subfolders + with data from other other systems / clients + that should **not** be mounted / available on the _SAI_. + * **source**: device or URL for the file system as used by the ```mount``` command. + Will always get suffixed with the ***pfs** value. + * **type**: file system type as used by the ```mount``` command. + * **rw_options**: mount options when mounting this _PFS_ **read-write** using the ```mount``` command. + * **ro_options**: mount options when mounting this _PFS_ **read-only** using the ```mount``` command. + +The example above would result in the following _PFS_ entries in ```/etc/fstab``` only on the _SAI_: +``` +some-storage001.stor.local:/ifs/isilon11 /mnt/isilon11 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 +10.0.0.203@tcp12:10.0.0.204@tcp12:/lusty2 /mnt/lusty2 lustre defaults,lazystatfs,flock 0 0 +``` + +#### lfs_mounts + +The **lfs_mounts** variable lists all Logical File Systems, which provide the user's perspective of the data. Per _LFS_: + + * **lfs**: a label for this _LFS_ + * **env[0-9]{2}** _LFS-ses_: + * Label is used as the mountpoint in **/mnt/** on the _Deploy Admin Interface (DAI)_. + * On all other machines the mountpoint is always **/apps**. + * **home** _LFS-ses_: + * Mountpoint is always **/home**. + + * **tmp[0-9]{2}** and **prm[0-9]{2}** _LFS-ses_: + * Mountpoint per groups is always **/groups/${group}/${lfs}**. + * **pfs**: a label for the _PFS_ that holds this _LFS_. + * Used to lookup the technical specs required by the ```mount``` command in the ```pfs_mounts``` variable. + * Hence for every label mentioned here there must be an entry in the list of ```pfs_mounts```. + * **machines**: Only for **env[0-9]{2}** and **home** _LFS-ses_. + * Lists the machines that should use this _LFS_ + * A cluster may use multiple _LFS-ses_ of the same type for different machines to distribute the IO load. + * A specific machine can only mount one instance of the same _LFS_ type and + hence should not be listed for multiple **env[0-9]{2}** _LFS-ses_ nor for multiple **home** _LFS-ses_. + * **groups**: Only for **tmp[0-9]{2}** and **prm[0-9]{2}** _LFS-ses_. + * Lists the groups that can use this _LFS_. + * Subfolders with correct permissions will be created automagically for the specified groups. + * Groups may have access to multiple **tmp[0-9]{2}** or **prm[0-9]{2}** _LFS-ses_. + +The example above would result in the following _LFS_ entries in ```/etc/fstab```: + * on the _SAI_: + ``` + some-storage001.stor.local:/ifs/cluster/isilon11/home /home nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + ``` + * on the _DAI_: + ``` + some-storage001.stor.local:/ifs/isilon11/home /home nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/env08 /mnt/env08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/ateam/tmp08 /groups/ateam/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/colla/tmp08 /groups/colla/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/production/tmp08 /groups/production/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/cool-project/tmp08 /groups/cool-project/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + ``` + * on the _UI_: + ``` + some-storage001.stor.local:/ifs/isilon11/home /home nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/env08 /apps nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime,ro 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/ateam/tmp08 /groups/ateam/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/colla/tmp08 /groups/colla/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/production/tmp08 /groups/production/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/cool-project/tmp08 /groups/cool-project/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/ateam/prm05 /groups/ateam/prm05 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/production/prm05 /groups/production/prm05 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + 10.0.0.203@tcp12:10.0.0.204@tcp12:/lusty2/groups/colla/prm03 /groups/colla/prm03 lustre defaults,lazystatfs,flock 0 0 + 10.0.0.203@tcp12:10.0.0.204@tcp12:/lusty2/groups/production/prm03 /groups/production/pmr03 lustre defaults,lazystatfs,flock 0 0 + 10.0.0.203@tcp12:10.0.0.204@tcp12:/lusty2/groups/cool-project/prm03 /groups/cool-project/prm03 lustre defaults,lazystatfs,flock 0 0 + ``` + * on a _compute node_: + ``` + some-storage001.stor.local:/ifs/isilon11/home /home nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/env08 /apps nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime,ro 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/ateam/tmp08 /groups/ateam/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/colla/tmp08 /groups/colla/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/production/tmp08 /groups/production/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + some-storage001.stor.local:/ifs/isilon11/groups/cool-project/tmp08 /groups/cool-project/tmp08 nfs4 defaults,_netdev,vers=4.0,noatime,nodiratime 0 0 + ``` + ``` \ No newline at end of file diff --git a/roles/shared_storage/defaults/main.yml b/roles/shared_storage/defaults/main.yml new file mode 100644 index 000000000..8824f0a5a --- /dev/null +++ b/roles/shared_storage/defaults/main.yml @@ -0,0 +1,27 @@ +--- +# +# Defaults for names of keys used to fetch quota limits for groups from an LDAP. +# +# LDAP is case insensitive, but we use lowercase only for field names, +# so we can use simple literal strings in comparisons as opposed to regexes +# to handle differences in UPPERCASE vs. lowercase. +# +# The UPPERCASE "LFS" in {{ ldap_group_quota_*_limit_template }} Ansible variables +# is a required placeholder that will get replaced with the value of the Logical File System (LFS) +# for which we will try to fetch quota limits from the LDAP. +# E.g. with ldap_group_quota_soft_limit_template='ruggroupumcgquotaLFSsoft', +# the fieldname/key to lookup the soft quota limit for the LFS prm01 is +# ruggroupumcgquotaprm01soft +# +ldap_group_object_class: 'groupofnames' +ldap_group_quota_soft_limit_template: 'ruggroupumcgquotaLFSsoft' +ldap_group_quota_hard_limit_template: 'ruggroupumcgquotaLFS' +# +# Lustre "project" quota is preferred, but older Lustre versions may not support it; +# Fall-back is "group" quota, which will work with older versions too. +# Setting is global for all Lustre file systems mounted on a cluster. +# Hence you cannot have one an older Lustre with group and a newer with project quota on the same cluster. +# +lustre_quota_type: 'project' +#lustre_quota_type: 'group' +... \ No newline at end of file diff --git a/roles/shared_storage/tasks/main.yml b/roles/shared_storage/tasks/main.yml index df79faac6..21178ac6d 100644 --- a/roles/shared_storage/tasks/main.yml +++ b/roles/shared_storage/tasks/main.yml @@ -1,18 +1,29 @@ --- -- name: 'Show the content of the group_vars/[cluster]/vars.yml lfs_mounts variable for debugging purposes.' - debug: - msg: "{{ lfs_mounts }}" +# +## +### List content of variables for debugging. +## +# +# - name: 'Show the content of the group_vars/[cluster]/vars.yml lfs_mounts variable for debugging purposes.' +# debug: +# msg: "{{ lfs_mounts }}" +# +# - name: 'Show the content of the group_vars/[cluster]/vars.yml pfs_mounts variable for debugging purposes.' +# debug: +# msg: "pfs: {{ item.pfs }} | source: {{ item.source }} | type: {{ item.type }} | rw_options: {{ item.rw_options }}." +# with_items: "{{ pfs_mounts }}" +# when: inventory_hostname in groups['sys-admin-interface'] -- name: 'Show the content of the group_vars/[cluster]/vars.yml pfs_mounts variable for debugging purposes.' - debug: - msg: "pfs: {{ item.pfs }} | source: {{ item.source }} | type: {{ item.type }} | rw_options: {{ item.rw_options }}." - with_items: "{{ pfs_mounts }}" - when: inventory_hostname in groups['sys-admin-interface'] +# +## +### Mount PFS-ses on DAI and create all required LFS-ses and subfolders on them with the correct permissions. +## +# - name: 'Mount complete Physical File Systems (PFS-ses) on SAIs.' mount: path: "/mnt/{{ item.pfs }}" - src: "{{ item.source }}" + src: "{{ item.source }}/{{ item.pfs }}" fstype: "{{ item.type }}" opts: "{{ item.rw_options }}" state: 'mounted' @@ -27,13 +38,13 @@ group: 'root' mode: '0755' state: 'directory' - with_items: "{{ lfs_mounts | selectattr('lfs', 'match', '^home$') | list }}" + with_items: "{{ lfs_mounts | selectattr('lfs', 'search', 'home') | list }}" when: inventory_hostname in groups['sys-admin-interface'] become: true - name: 'Create root groups folder for "tmp" and "prm" Logical File Systems (LFSs) on Physical File Systems (PFSs) mounted on SAIs.' file: - path: "/mnt/{{ item.pfs }}/{{ item.lfs | regex_replace('GROUP.*', '') }}" + path: "/mnt/{{ item.pfs }}/groups" owner: 'root' group: 'root' mode: '0755' @@ -45,7 +56,7 @@ - name: 'Create folder for each group on Physical File Systems (PFSs) mounted on SAIs.' file: - path: "/mnt/{{ item.0.pfs }}/{{ item.0.lfs | regex_replace('GROUP', item.1) | regex_replace('((tmp)|(prm))[0-9]+$', '') }}" + path: "/mnt/{{ item.0.pfs }}/groups/{{ item.1 }}" owner: 'root' group: "{{ item.1 }}" mode: '2750' @@ -58,7 +69,7 @@ - name: 'Create "tmp" Logical File Systems (LFSs) for each group on Physical File Systems (PFSs) mounted on SAIs.' file: - path: "/mnt/{{ item.0.pfs }}/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" + path: "/mnt/{{ item.0.pfs }}/groups/{{ item.1 }}/{{ item.0.lfs }}" owner: 'root' group: "{{ item.1 }}" mode: '2770' @@ -71,7 +82,7 @@ - name: 'Create "prm" Logical File Systems (LFSs) for each group on Physical File Systems (PFSs) mounted on SAIs.' file: - path: "/mnt/{{ item.0.pfs }}/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" + path: "/mnt/{{ item.0.pfs }}/groups/{{ item.1 }}/{{ item.0.lfs }}" owner: "{{ item.1 }}-dm" group: "{{ item.1 }}" mode: '2750' @@ -94,21 +105,27 @@ when: inventory_hostname in groups['sys-admin-interface'] become: true +# +## +### Mount all LFS-ses on cluster components/machines. +## +# + - name: 'Mount "home" Logical File System (LFS) from shared storage.' mount: - path: "/{{ item.lfs }}" - src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.lfs }}" + path: "/home" + src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.pfs }}/{{ item.lfs }}" fstype: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='type') | first }}" opts: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='rw_options') | first }}" state: 'mounted' - with_items: "{{ lfs_mounts | selectattr('lfs', 'match', '^home$') | list }}" - when: inventory_hostname in groups['cluster'] and not inventory_hostname in groups['nfs-server']|default([]) + with_items: "{{ lfs_mounts | selectattr('lfs', 'search', 'home') | list }}" + when: inventory_hostname in item.machines and not inventory_hostname in groups['nfs-server']|default([]) become: True - name: 'Mount "tmp" Logical File Systems (LFSs) per group from shared storage.' mount: - path: "/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" - src: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='source') | first }}/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" + path: "/groups/{{ item.1 }}/{{ item.0.lfs }}" + src: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='source') | first }}/{{ item.0.pfs }}/groups/{{ item.1 }}/{{ item.0.lfs }}" fstype: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='type') | first }}" opts: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='rw_options') | first }}" state: 'mounted' @@ -120,8 +137,8 @@ - name: 'Mount "prm" Logical File Systems (LFSs) per group from shared storage.' mount: - path: "/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" - src: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='source') | first }}/{{ item.0.lfs | regex_replace('GROUP', item.1) }}" + path: "/groups/{{ item.1 }}/{{ item.0.lfs }}" + src: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='source') | first }}/{{ item.0.pfs }}/groups/{{ item.1 }}/{{ item.0.lfs }}" fstype: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='type') | first }}" opts: "{{ pfs_mounts | selectattr('pfs', 'match', item.0.pfs) | map(attribute='rw_options') | first }}" state: 'mounted' @@ -134,18 +151,18 @@ - name: 'Mount "env" Logical File Systems (LFSs) from shared storage read-write on DAIs.' mount: path: "/mnt/{{ item.lfs }}" - src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.lfs }}" + src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.pfs }}/{{ item.lfs }}" fstype: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='type') | first }}" opts: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='rw_options') | first }}" state: 'mounted' - with_items: "{{ lfs_mounts | selectattr('lfs', 'search', 'env[0-9]+$') | list}}" + with_items: "{{ lfs_mounts | selectattr('lfs', 'search', 'env[0-9]+$') | list }}" when: inventory_hostname in groups['deploy-admin-interface'] become: true - name: 'Mount "apps" from one "env" Logical File System (LFS) from shared storage read-only as /apps on UIs and vcompute nodes.' mount: path: '/apps' - src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.lfs }}/apps" + src: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='source') | first }}/{{ item.pfs }}/{{ item.lfs }}/apps" fstype: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='type') | first }}" opts: "{{ pfs_mounts | selectattr('pfs', 'match', item.pfs) | map(attribute='ro_options') | first }}" state: 'mounted' @@ -153,4 +170,31 @@ - "{{ lfs_mounts | selectattr('lfs', 'search', 'env[0-9]+$') | list }}" when: inventory_hostname in item.machines become: true + +# +## +### Deploy quota management +## +# + +- name: 'Deploy script to fetch quota limits from an LDAP and set them on shared storage mounts.' + template: + src: 'templates/ldapquota.bash' + dest: '/root/ldapquota.bash' + owner: 'root' + group: 'root' + mode: '0700' + when: inventory_hostname in groups['sys-admin-interface'] + become: true + +- name: 'Create cron job for ldapquota.bash script.' + cron: + name: 'Apply quota limits from LDAP to shared storage mounts.' + minute: '15' + hour: '9,11,13,15,17,19,21' + user: 'root' + job: '/root/ldapquota.bash -a' + cron_file: 'ldapquota' + when: inventory_hostname in groups['sys-admin-interface'] + become: true ... diff --git a/roles/shared_storage/templates/ldapcache.bash b/roles/shared_storage/templates/ldapcache.bash new file mode 100755 index 000000000..6702c2ea5 --- /dev/null +++ b/roles/shared_storage/templates/ldapcache.bash @@ -0,0 +1,782 @@ +#!/bin/bash + +# +# Code Conventions: +# Indentation: TABs only +# Functions: camelCase +# Global Variables: lower_case_with_underscores +# Local Variables: _lower_case_with_underscores_and_prefixed_with_underscore +# Environment Variables: UPPER_CASE_WITH_UNDERSCORES +# + +# +## +### Environment and Bash sanity. +## +# +if [[ "${BASH_VERSINFO}" -lt 4 || "${BASH_VERSINFO[0]}" -lt 4 ]]; then + echo "Sorry, you need at least bash 4.x to use ${0}." >&2 + exit 1 +fi + +set -e # Exit if any subcommand or pipeline returns a non-zero exit status. +set -u # Raise exception if variable is unbound. Combined with set -e will halt execution when an unbound variable is encountered. +set -o pipefail # Fail when any command in series of piped commands failed as opposed to only when the last command failed. + +umask 0077 + +export TMPDIR="${TMPDIR:-/tmp}" # Default to /tmp if $TMPDIR was not defined. + +# +# Make sure dots are used as decimal separator. +# +LANG='en_US.UTF-8' +LC_NUMERIC="${LANG}" + +# +## +### Functions. +## +# +function showHelp() { + # + # Display commandline help on STDOUT. + # + cat <" "${SCRIPT_NAME}" "${_log_timestamp}" "${_log_level}" "${_problematic_line}" "${_problematic_function}") + local _log_line="${_log_line_prefix} ${_log_message}" + if [ ! -z "${mixed_stdouterr:-}" ]; then + _log_line="${_log_line} STD[OUT+ERR]: ${mixed_stdouterr}" + fi + if [ ${_status} -ne 0 ]; then + _log_line="${_log_line} (Exit status = ${_status})" + fi + + # + # Log to STDOUT (low prio <= 'WARN') or STDERR (high prio >= 'ERROR'). + # + if [[ ${_log_level_prio} -ge ${l4b_log_levels['ERROR']} || ${_status} -ne 0 ]]; then + printf '%s\n' "${_log_line}" > /dev/stderr + else + printf '%s\n' "${_log_line}" + fi + fi + + # + # Exit if this was a FATAL error. + # + if [ ${_log_level_prio} -ge ${l4b_log_levels['FATAL']} ]; then + # + # Reset trap and exit. + # + trap - EXIT + if [ ${_status} -ne 0 ]; then + exit ${_status} + else + exit 1 + fi + fi +} + +# +# Parse LDIF records and apply quota to Physical File Systems (PFSs). +# +function parseLdif () { + local _ldif_file="${1}" + local _ldap_quota_attr_key_prefix="${2,,}" # Convert to lowercase. + shift 2 + local -a _lfs_to_pfs_fake_hash=("${@}") + # + local _ldap_attr_regex='([^: ]{1,}): ([^:]{1,})' + local _fake_hash_regex='([^: ]{1,})::([^:]{1,})' + local _pos_int_regex='^[0-9]+$' + # + # Restore lfs-to-pfs mapping from "fake" hash into a real one. + # + local -A _lfs_to_pfs=() + for _mapping in ${_lfs_to_pfs_fake_hash[@]:-}; do + if [[ ${_mapping} =~ ${_fake_hash_regex} ]]; then + local _lfs_key="${BASH_REMATCH[1]}" + local _pfs_val="${BASH_REMATCH[2]}" + _lfs_to_pfs[${_lfs_key}]="${_pfs_val}" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "Added mapping for ${_lfs_key} -> ${_pfs_val} to _lfs_to_pfs hash." + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' "Skipped malformed mapping entry ${_mapping} from _lfs_to_pfs_fake_hash array." + fi + done + # + # Append the NULL character to the LDIF file, so we can detect that as EOF instead of a newline. + # + printf '\0' >> "${_ldif_file}" + # + # Substitute the blank line record separator with a # character and read records into an array. + # + IFS='#' read -r -d '' -a _ldif_records < <(sed 's/^$/#/' "${_ldif_file}") || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "Parsing LDIF file (${_ldif_file}) into records failed." + # + # Loop over records in the array and create a faked-multi-dimensional hash. + # + local _ldif_record + for _ldif_record in "${_ldif_records[@]}"; do + # + # Remove trailing white space like the new line character. + # And skip blank lines. + # + _ldif_record="${_ldif_record%%[[:space:]]}" + [[ "${_ldif_record}" == '' ]] && continue + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "LDIF record contains: ${_ldif_record}" + # + # Parse record's key:value pairs. + # + local -A _directory_record_attributes=() + local _ldif_line + while IFS=$'\n' read -r _ldif_line; do + [[ "${_ldif_line}" == '' ]] && continue # Skip blank lines. + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "LDIF key:value pair contains: ${_ldif_line}." + if [[ ${_ldif_line} =~ ${_ldap_attr_regex} ]]; then + local _key=${BASH_REMATCH[1],,} # Convert key on-the-fly to lowercase. + local _value=${BASH_REMATCH[2]} + # + # This may be a multi-valued attribute and therefore check if key already exists; + # When key already exists make sure we append instead of overwriting the existing value(s)! + # + if [[ ! -z "${_directory_record_attributes[${_key}]+isset}" ]]; then + _directory_record_attributes["${_key}"]="${_directory_record_attributes[${_key}]} ${_value}" + else + _directory_record_attributes["${_key}"]="${_value}" + fi + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " key contains: ${_key}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " value contains: ${_value}." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Failed to parse LDIF key:value pair (${_ldif_line})." + fi + done < <(printf '%s\n' "${_ldif_record}") || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "Parsing LDIF record failed." + # + # Use processed LDIF record to: + # 1. Apply quota to PFS. + # 2. Store cached meta-data in apps LFS. + # + if [[ ! -z "${_directory_record_attributes['dn']+isset}" ]]; then + # + # Parse cn from dn. + # + local _group=$(dn2cnWithEntitlementPrefix "${_directory_record_attributes['dn']}") + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' "Processing group: ${_group}..." + # + # Check if group exists on this machine: + # May be absent in case it is a private group of a user that is not entitled for this machine. + # + if [ $(getent group "${_group}") ]; then + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Group exists on this machine." + else + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Skipping group ${_group}, because it does not exist on this machine. (Most likely private group for user that is not entitled.)" + continue + fi + # + # Check if group name starts with prefix of an entitlement for which we want to process group data. + # + local _process_group='no' + local _entitlement_prefix + for _entitlement_prefix in ${entitlement_prefixes[@]}; do + local _prefix_regex="^${_entitlement_prefix}-" + if [[ ${_group} =~ ${_prefix_regex} ]]; then + _process_group='yes' + break + fi + done + if [[ ${_process_group} == 'yes' ]]; then + local _list="${entitlement_prefixes[*]}" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Group's entitlement prefix exists in list of entitlements to process: ${_list// /, }." + else + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Skipping group ${_group}, because it does not exist in an entitlement, which we want to process." + continue + fi + # + # Loop over Logical File Systems (LFSs), + # find the corresponding quota values, + # find the corresponding Physical File System (PFS) and + # apply quota settings to PFS. + # + for (( _offset = 0 ; _offset < ${#logical_file_systems[@]:-0} ; _offset++ )); do + local _lfs="${logical_file_systems[${_offset}]}" + if [[ -z ${_lfs_to_pfs[${_lfs}]+isset} ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Skipping LFS ${_lfs} without mapping to a PFS." + continue + else + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Processing quota settings for LFS ${_lfs}..." + fi + local _soft_quota_limit='' + local _hard_quota_limit='' + if [[ ! -z "${_directory_record_attributes[${_ldap_quota_attr_key_prefix}${_lfs}]+isset}" && ! -z "${_directory_record_attributes[${_ldap_quota_attr_key_prefix}${_lfs}soft]+isset}" ]]; then + _soft_quota_limit="${_directory_record_attributes[${_ldap_quota_attr_key_prefix}${_lfs}'soft']}" + _hard_quota_limit="${_directory_record_attributes[${_ldap_quota_attr_key_prefix}${_lfs}]}" + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values missing for group ${_group} on LFS ${_lfs}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Search key was ${_ldap_quota_attr_key_prefix}${_lfs}." + continue + fi + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " soft_quota_limit contains: ${_soft_quota_limit}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " hard_quota_limit contains: ${_hard_quota_limit}." + # + # Check for negative numbers and non-integers. + # + if [[ ! ${_soft_quota_limit} =~ ${_pos_int_regex} || ! ${_hard_quota_limit} =~ ${_pos_int_regex} ]]; then + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values malformed for group ${_group} on LFS ${_lfs}. Must be integers >= 0." + continue + fi + # + # Check if soft limit is smaller than or equal to the hard limit: + # lfs setquota will fail when the soft limit is larger than the hard limit. + # + if [[ ${_soft_quota_limit} -gt ${_hard_quota_limit} ]]; then + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values malformed for group ${_group} on LFS ${_lfs}. Soft limit cannot be larger than hard limit." + continue + fi + # + # Check for 0 (zero). + # When quota values are set to zero it means unlimited: not what we want. + # When zero was specified we'll interpret this as "do not allow this group to consume any space". + # Due to the technical limitations of how quota work we'll configure the lowest possible value instead: + # This is 2 * the block/stripe size on Lustre File Systems. + # + if [[ ${_soft_quota_limit} -eq 0 ]]; then + _soft_quota_limit='2M' + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Converted soft quota limit of 0 (zero) for group ${_group} on LFS ${_lfs} to lowest possible value of ${_soft_quota_limit}." + else + # Just append unit: all quota values from the IDVault are in GB. + _soft_quota_limit="${_soft_quota_limit}G" + fi + if [[ ${_hard_quota_limit} -eq 0 ]]; then + _hard_quota_limit='2M' + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Converted hard quota limit of 0 (zero) for group ${_group} on LFS ${_lfs} to lowest possible value of ${_hard_quota_limit}." + else + # Just append unit: all quota values from the IDVault are in GB. + _hard_quota_limit="${_hard_quota_limit}G" + fi + + # + # Compile and apply Lustre quota command to PFS(s). + # + local -a _pfss=($(printf '%s' "${_lfs_to_pfs[${_lfs}]}" | tr ',' ' ')) + local _pfs + for _pfs in ${_pfss[@]}; do + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Compiling quota command for LFS ${_lfs} on PFS ${_pfs}." + local _lfs_cmd="lfs setquota -g ${_group} --block-softlimit ${_soft_quota_limit} --block-hardlimit ${_hard_quota_limit} /mnt/${_pfs}" + if [ "${apply_settings}" -eq 1 ]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 " Applying quota cmd: ${_lfs_cmd}" + mixed_stdouterr="$(${_lfs_cmd})" || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "lfs setquota failed." + else + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 " Dry run quota cmd: ${_lfs_cmd}" + fi + done + # + # In case we are processing 'private groups' for individual users: + # Set quota to the smallest possible value for file systems where private groups are banned. + # + if [[ ${_ldap_quota_attr_key_prefix} =~ 'person' ]]; then + for _pfs in ${pfs_without_users[@]}; do + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Compiling quota command for private group on PFS ${_pfs}." + local _lfs_cmd="lfs setquota -g ${_group} --block-softlimit 2M --block-hardlimit 2M /mnt/${_pfs}" + if [ "${apply_settings}" -eq 1 ]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 " Applying quota cmd: ${_lfs_cmd}" + mixed_stdouterr="$(${_lfs_cmd})" || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "lfs setquota failed." + else + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 " Dry run quota cmd: ${_lfs_cmd}" + fi + done + fi + done + # + # Store cached meta-data. + # + if [[ ${_ldap_quota_attr_key_prefix} =~ 'group' ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" 0 'Processing group meta-data...' + declare -a _meta_data_keys=('ruggroupownervalue' 'ruggroupdatamanagervalue') + elif [[ ${_ldap_quota_attr_key_prefix} =~ 'person' ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" 0 'Processing private group (=user) meta-data...' + declare -a _meta_data_keys=('loginexpirationtime' 'logindisabled') + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" 0 "Found unsupported ldap_quota_attr_key_prefix ${_ldap_quota_attr_key_prefix}: skipping meta-data caching." + return + fi + declare -a _meta_data=() + local _meta_data_key + for _meta_data_key in "${_meta_data_keys[@]}"; do + local _meta_data_item="${_directory_record_attributes[${_meta_data_key}]:-NA}" + _meta_data_key="${_meta_data_key#ruggroup}" # Remove meaningless characters from key. + _meta_data_key="${_meta_data_key%value}" # Remove meaningless characters from key. + if [[ ${_meta_data_item} == 'NA' ]]; then + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " SOP violation: ${_group} lacks ${_meta_data_key}." + else + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' " Found meta-data for ${_group}: ${_meta_data_key}=${_meta_data_item}." + fi + if [[ "${#_meta_data[@]:-0}" -eq 0 ]]; then + _meta_data=("${_meta_data_key}=${_meta_data_item}") + else + _meta_data=("${_meta_data[@]}" "${_meta_data_key}=${_meta_data_item}") + fi + done + if [[ "${apply_settings}" -eq 1 && "${#meta_data_cache_dirs[@]:-0}" -ge 1 ]]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' " Storing ${_group} meta-data..." + local _cache_dir + for _cache_dir in "${meta_data_cache_dirs[@]}"; do + saveMetaDataCache "${_cache_dir}" "${_group}" "${_meta_data[@]}" + done + fi + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "dn attribute missing for ${_ldif_record}" + fi + done +} + +# +# Store meta-data in cache file. +# +function saveMetaDataCache () { + local _cache_dir="${1}" + local _object="${2}" # Either a group or a private group a.k.a. a user. + shift 2 + local -a _meta_data_items=("${@}") + local _meta_data_item + local _cache_file="${_cache_dir}/${_object}" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Writing cached meta-data to ${_cache_file}." + # + # Create cache dir if not already present and initialize cache file deleting any old content. + # Also update timestamps on cache dir and cache files. + # + if [ ! -d "${_cache_dir}" ]; then + (umask 0022; mkdir -p -m '755' "${_cache_dir}") + else + touch "${_cache_dir}" + fi + (umask 0033; touch "${_cache_file}") + printf '' > "${_cache_file}" + for _meta_data_item in "${_meta_data_items[@]:-}"; do + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Writing to cache _meta_data_item: ${_meta_data_item}" + printf '%s\n' "${_meta_data_item}" >> "${_cache_file}" + done +} + +# +# Extract a CN from a DN LDAP attribute. +# +function dn2cnWithEntitlementPrefix () { + # cn=umcg-someuser,ou=users,ou=umcg,o=rs + local _dn="$1" + local _cn='MIA' + local _regex='cn=([^, ]+)' + if [[ ${_dn} =~ ${_regex} ]]; then + _cn="${BASH_REMATCH[1]}" + fi + printf '%s' "${_cn}" +} + +# +## +### Variables. +## +# + +groups_ldif=${TMPDIR}/groups.ldif +users_ldif=${TMPDIR}/users.ldif +SCRIPT_NAME="$(basename $0 .bash)" +mixed_stdouterr='' # global variable to capture output from commands for reporting in custom log messages. + +# +# Initialise Log4Bash logging with defaults. +# +l4b_log_level="${log_level:-INFO}" +declare -A l4b_log_levels=( + ['TRACE']='0' + ['DEBUG']='1' + ['INFO']='2' + ['WARN']='3' + ['ERROR']='4' + ['FATAL']='5' +) +l4b_log_level_prio="${l4b_log_levels[${l4b_log_level}]}" + +# +## +### Main. +## +# + +# +# Get commandline arguments. +# +declare -a logical_file_systems=() +declare apply_settings=0 +while getopts "f:l:b:ah" opt; do + case $opt in + h) + showHelp + ;; + f) + logical_file_systems="${OPTARG}" + ;; + b) + backup_dir="${OPTARG}" + ;; + a) + apply_settings=1 + ;; + l) + l4b_log_level=${OPTARG^^} + l4b_log_level_prio=${l4b_log_levels[${l4b_log_level}]} + ;; + \?) + log4Bash "${LINENO}" "${FUNCNAME:-main}" '1' "Invalid option -${OPTARG}. Try $(basename $0) -h for help." + ;; + :) + log4Bash "${LINENO}" "${FUNCNAME:-main}" '1' "Option -${OPTARG} requires an argument. Try $(basename $0) -h for help." + ;; + esac +done + +# +# Initialise vars from config file. +# +declare -A lfs_to_pfs_for_groups=() +declare -A lfs_to_pfs_for_users=() +declare -a pfs_with_apps=() +declare -a pfs_without_users=() + +# +# Source config file. +# +# Config file must contain in bash syntax: +# # +# # Credentials: +# # +# LDAP_USER='some_account' +# LDAP_PASS='some_passwd' +# LDAP_SEARCH_BASE='ou=groups,ou=umcg,o=rs' +# # +# # Prefixes of IDVault entitlements for which group data must be processed. +# # +# declare -a entitlement_prefixes=('umcg' 'll') +# # +# # Mappings of Logical File Systems (LFSs) to Physical File Systems (PFSs). +# # * for groups to apply group quota. +# # * for users to apply 'private group' quota. +# # Private groups contain only a single user and have the same name + ID as that user. +# # In case an LFS is available from multiple PFSs, multiple PFSs may be listed separated by a comma. +# # +# declare -A lfs_to_pfs_for_groups=( +# ['prm02']='umcgst07' +# ['prm03']='umcgst09' +# ['tmp03']='umcgst08' +# ['tmp04']='umcgst06' +# ) +# declare -A lfs_to_pfs_for_users=( +# ['home']='umcgst07,umcgst09' +# ) +# declare -a pfs_with_apps=( +# 'umcgst07' +# 'umcgst09' +# ) +# # +# # Only allow groups to store data here and set quota for individual users to the lowest possible value (1k). +# # +# declare -a pfs_without_users=( +# 'umcgst06' +# 'umcgst08' +# ) +# Config file must be located in same location as this script. +# +config_file="$(cd -P "$( dirname "$0" )" && pwd)/${SCRIPT_NAME}.cfg" +if [[ -r "${config_file}" && -f "${config_file}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Sourcing config file ${config_file}..." + source "${config_file}" || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "Cannot source ${config_file}." +else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Config file ${config_file} missing or not accessible." +fi + +if [ "${apply_settings}" -eq 1 ]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' 'Found option -a: will fetch, list and apply settings.' +else + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' 'Option -a not specified: will only perform a "dry run" to fetch + list settings. Use -a to apply settings.' +fi + +# +# Compile list of folders where meta-data will be cached. +# +declare -a meta_data_cache_dirs=() +if [[ -d '/apps' ]]; then + meta_data_cache_dirs=('/apps/.tmp/ldap_cache/') +fi +for (( offset = 0 ; offset < ${#pfs_with_apps[@]:-0} ; offset++ )); do + if [[ -d "/mnt/${pfs_with_apps[${offset}]}/apps/" ]]; then + meta_data_cache_dir="/mnt/${pfs_with_apps[${offset}]}/apps/.tmp/ldap_cache/" + meta_data_cache_dirs=(${meta_data_cache_dirs[@]:-} "${meta_data_cache_dir}") + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Found meta-data cache dir: ${meta_data_cache_dir}." + else + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' "No apps dir found on PFS: ${pfs_with_apps[${offset}]}." + fi +done +if [[ "${#meta_data_cache_dirs[@]:-0}" -ge 1 ]]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' "List of meta_data_cache_dirs contains: $(printf "%s " "${meta_data_cache_dirs[@]}")." +else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' "List of meta_data_cache_dirs is empty: will not update/store the meta-data cache on any FS." +fi + +# +# Compile list of LDAP quota fields to retrieve. +# +ldap_group_quota_fields='cn ruggroupOwnerValue ruggroupDataManagerValue' +if [ ${#logical_file_systems[@]:-0} -ge 1 ]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Logical File Systems (LFSs) specified: ${logical_file_systems[@]}." + if [ "${logical_file_systems[0]}" == 'ALL' ]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "List of logical_file_systems contains ALL -> will fetch all known LFSs from config file ${config_file}..." + logical_file_systems=(${!lfs_to_pfs_for_groups[@]} ${!lfs_to_pfs_for_users[@]}) + fi +else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' "Not a single Logical File System (LFS) specified with -f: will not apply quota settings to corresponding Physical File Systems (PFSs)." +fi +for (( offset = 0 ; offset < ${#logical_file_systems[@]:-0} ; offset++ )); do + lfs="${logical_file_systems[${offset}]}" + # + # Check if we know on which PFS each LFS is located. + # + if [ ! -z "${lfs_to_pfs_for_groups[${lfs}]+isset}" ]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "List of lfs_to_pfs_for_groups contains mapping ${lfs} -> ${lfs_to_pfs_for_groups[${lfs}]}." + ldap_group_quota_fields="${ldap_group_quota_fields} ruggroupUMCGQuota${lfs} ruggroupUMCGQuota${lfs}Soft" + elif [ ! -z "${lfs_to_pfs_for_users[${lfs}]+isset}" ]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "List of lfs_to_pfs_for_users contains ${lfs} -> ${lfs_to_pfs_for_users[${lfs}]}." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Mapping to Physical File Sytem (PFS) missing for Logical File System (LFS) '${lfs}' in config file ${config_file}." + fi +done + +# +# Query LDAP. +# +log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Retrieving data from LDAP..." +log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "ldap_group_quota_fields to retrieve = ${ldap_group_quota_fields}." +mixed_stdouterr=$(ldapsearch -LLL -D "${LDAP_USER}" -w "${LDAP_PASS}" -b "${LDAP_SEARCH_BASE}" \ + "(ObjectClass=GroupofNames)" ${ldap_group_quota_fields} 2>&1 >"${groups_ldif}") \ + || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "ldapsearch failed." + +ldap_user_quota_fields='cn rugpersonUMCGQuotaHome rugpersonUMCGQuotaHomeSoft loginExpirationTime loginDisabled' +log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "ldap_user_quota_fields to retrieve = ${ldap_user_quota_fields}." +mixed_stdouterr=$(ldapsearch -LLL -D "${LDAP_USER}" -w "${LDAP_PASS}" -b "${LDAP_SEARCH_BASE}" \ + "(ObjectClass=person)" ${ldap_user_quota_fields} 2>&1 > "${users_ldif}") \ + || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "ldapsearch failed." +log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "ldapsearch results were saved to ${groups_ldif} and ${users_ldif}." + +# +# Emulate "fake" associative array a.k.a. hash using a normal array as we cannot pass hashes in Bash without ugly hacks. +# Then parse LDIF records. +# +declare -a lfs_to_pfs_fake_hash=() +declare key_value_combi + +for lfs_key in ${!lfs_to_pfs_for_groups[@]}; do + key_value_combi="${lfs_key}::${lfs_to_pfs_for_groups[${lfs_key}]}" + lfs_to_pfs_fake_hash=(${lfs_to_pfs_fake_hash[@]:-} ${key_value_combi}) + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "Added ${key_value_combi} from lfs_to_pfs_for_groups to lfs_to_pfs_fake_hash." +done +parseLdif "${groups_ldif}" 'ruggroupUMCGQuota' "${lfs_to_pfs_fake_hash[@]:-}" + +lfs_to_pfs_fake_hash=() +for lfs_key in ${!lfs_to_pfs_for_users[@]}; do + key_value_combi="${lfs_key}::${lfs_to_pfs_for_users[${lfs_key}]}" + lfs_to_pfs_fake_hash=(${lfs_to_pfs_fake_hash[@]:-} ${key_value_combi}) + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "Added ${key_value_combi} from lfs_to_pfs_for_users to lfs_to_pfs_fake_hash." +done +parseLdif "${users_ldif}" 'rugpersonUMCGQuota' "${lfs_to_pfs_fake_hash[@]:-}" + +# +# Cleanup tmp files. +# +if [ ${l4b_log_level_prio} -lt ${l4b_log_levels['INFO']} ]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" 0 "Debug mode: temporary files ${groups_ldif} and ${users_ldif} won't be removed." +else + rm "${groups_ldif}" + rm "${users_ldif}" +fi + +# +# Optional: Make backup of (complete) list of users and group +# Perform new LDAP search to list all fields accessible to user who performs the query. +# +if [ ! -z "${backup_dir:-}" ]; then + # + # Create directory for backups. + # + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Retrieving data from LDAP for backup..." + mixed_stdouterr=$(mkdir -m 0700 -p "${backup_dir}" 2>&1) \ + || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "Failed to create backup dir ${backup_dir}." + if [[ -d ${backup_dir} && -w ${backup_dir} ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Backup dir ${backup_dir} is Ok." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '0' "Backup dir ${backup_dir} cannot be used. Check path and permissions." + fi + # + # Create timestamp. + # + BACKUP_TS=`date "+%Y-%m-%d-T%H%M"` + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Retrieving data from LDAP for backup..." + mixed_stdouterr=$(ldapsearch -LLL -D "${LDAP_USER}" -w "${LDAP_PASS}" -b "${LDAP_SEARCH_BASE}" \ + "(ObjectClass=GroupofNames)" 2>&1 >"${backup_dir}/groups-${BACKUP_TS}.ldif") \ + || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "ldapsearch failed." + + mixed_stdouterr=$(ldapsearch -LLL -D "${LDAP_USER}" -w "${LDAP_PASS}" -b "${LDAP_SEARCH_BASE}" \ + "(ObjectClass=person)" 2>&1 > "${backup_dir}/users-${BACKUP_TS}.ldif") \ + || log4Bash 'FATAL' ${LINENO} "${FUNCNAME:-main}" $? "ldapsearch failed." + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "ldapsearch results for backup were saved to ${groups_ldif} and ${users_ldif}." +fi + +# +# Reset trap and exit. +# +log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 "Finished!" +trap - EXIT +exit 0 \ No newline at end of file diff --git a/roles/shared_storage/templates/ldapquota.bash b/roles/shared_storage/templates/ldapquota.bash new file mode 100755 index 000000000..46edac425 --- /dev/null +++ b/roles/shared_storage/templates/ldapquota.bash @@ -0,0 +1,570 @@ +#!/bin/bash + +# +# Code Conventions: +# Indentation: TABs only +# Functions: camelCase +# Global Variables: lower_case_with_underscores +# Local Variables: _lower_case_with_underscores_and_prefixed_with_underscore +# Environment Variables: UPPER_CASE_WITH_UNDERSCORES +# + +# +## +### Environment and Bash sanity. +## +# +if [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then + echo "Sorry, you need at least bash 4.x to use ${0}." >&2 + exit 1 +fi + +set -e # Exit if any subcommand or pipeline returns a non-zero exit status. +set -u # Raise exception if variable is unbound. Combined with set -e will halt execution when an unbound variable is encountered. +set -o pipefail # Fail when any command in series of piped commands failed as opposed to only when the last command failed. + +umask 0077 + +# +# LDAP is case insensitive, but we use lowercase only for field names, +# so we can use simple literal strings in comparisons as opposed to regexes +# to handle differences in UPPERCASE vs. lowercase. +# +# The UPPERCASE "LFS" in ${ldap_group_quota_*_limit_template} variables +# is a required placeholder that will get replaced with the value of the Logical File System (LFS) +# for which we will try to fetch quota limits from the LDAP. +# E.g. with ldap_group_quota_soft_limit_template='ruggroupumcgquotaLFSsoft', +# the fieldname/key to lookup the soft quota limit for the LFS prm01 is +# ruggroupumcgquotaprm01soft +# +declare ldap_group_object_class='{{ ldap_group_object_class }}' +declare ldap_group_quota_soft_limit_template='{{ ldap_group_quota_soft_limit_template }}' +declare ldap_group_quota_hard_limit_template='{{ ldap_group_quota_hard_limit_template }}' +declare -A ldap_quota_limits=() + +# +# Lustre quota type for groups: +# We prefer "project quota" for group folders, +# but we'll use "group quota" when project quota are not supported (yet). +# +declare lustre_quota_type='{{ lustre_quota_type }}' + +# +# No more Ansible variables below this point! +# +{% raw %} +# +# Global variables. +# +declare TMPDIR="${TMPDIR:-/tmp}" # Default to /tmp if ${TMPDIR} was not defined. +declare SCRIPT_NAME +SCRIPT_NAME="$(basename "${0}" '.bash')" +export TMPDIR +export SCRIPT_NAME +declare mixed_stdouterr='' # global variable to capture output from commands for reporting in custom log messages. +declare ldif_dir="${TMPDIR}/ldifs" +declare config_file='/etc/ssh/ldap.conf' + +# +# Initialise Log4Bash logging with defaults. +# +l4b_log_level="${log_level:-INFO}" +declare -A l4b_log_levels=( + ['TRACE']='0' + ['DEBUG']='1' + ['INFO']='2' + ['WARN']='3' + ['ERROR']='4' + ['FATAL']='5' +) +l4b_log_level_prio="${l4b_log_levels["${l4b_log_level}"]}" + +# +# Make sure dots are used as decimal separator. +# +LANG='en_US.UTF-8' +LC_NUMERIC="${LANG}" + +# +## +### Functions. +## +# +function showHelp() { + # + # Display commandline help on STDOUT. + # + cat <" "${SCRIPT_NAME}" "${_log_timestamp}" "${_log_level}" "${_problematic_line}" "${_problematic_function}") + local _log_line="${_log_line_prefix} ${_log_message}" + if [[ -n "${mixed_stdouterr:-}" ]]; then + _log_line="${_log_line} STD[OUT+ERR]: ${mixed_stdouterr}" + fi + if [[ "${_status}" -ne 0 ]]; then + _log_line="${_log_line} (Exit status = ${_status})" + fi + # + # Log to STDOUT (low prio <= 'WARN') or STDERR (high prio >= 'ERROR'). + # + if [[ "${_log_level_prio}" -ge "${l4b_log_levels['ERROR']}" || "${_status}" -ne 0 ]]; then + printf '%s\n' "${_log_line}" > '/dev/stderr' + else + printf '%s\n' "${_log_line}" + fi + fi + # + # Exit if this was a FATAL error. + # + if [[ "${_log_level_prio}" -ge "${l4b_log_levels['FATAL']}" ]]; then + # + # Reset trap and exit. + # + trap - EXIT + if [[ "${_status}" -ne 0 ]]; then + exit "${_status}" + else + exit 1 + fi + fi +} + +# +# Parse LDIF records and apply quota to Physical File Systems (PFSs). +# +function processFileSystems () { + local _lfs_path_regex='/mnt/([^/]+)/groups/([^/]+)/([^/]+)' + local _pos_int_regex='^[0-9]+$' + local _lfs_path + local -a _lfs_paths=("${@}") + # + # Loop over Logical File System (LFS) paths, + # find the corresponding quota values, + # and apply quota settings. + # + for _lfs_path in "${_lfs_paths[@]}"; do + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' "Processing LFS path ${_lfs_path} ..." + local _pfs_from_lfs_path + local _group_from_lfs_path + local _lfs_from_lfs_path + local _fs_type + if [[ "${_lfs_path}" =~ ${_lfs_path_regex} ]]; then + _pfs_from_lfs_path="${BASH_REMATCH[1]}" + _group_from_lfs_path="${BASH_REMATCH[2]}" + _lfs_from_lfs_path="${BASH_REMATCH[3]}" + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " found _pfs_from_lfs_path: ${_pfs_from_lfs_path}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " found _group_from_lfs_path: ${_group_from_lfs_path}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " found _lfs_from_lfs_path: ${_lfs_from_lfs_path}." + _fs_type="$(awk -v _mount_point="/mnt/${_pfs_from_lfs_path}" '{if ($2 == _mount_point) print $3}' /proc/mounts)" + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " found _fs_type: ${_fs_type}." + else + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' "Skipping malformed LFS path ${_lfs_path}." + continue + fi + # + # Reset hash and then query for quota values for this group on this LFS. + # + ldap_quota_limits=() + local _soft_quota_limit + local _hard_quota_limit + getQuotaFromLDAP "${_lfs_from_lfs_path}" "${_group_from_lfs_path}" + if [[ -n "${ldap_quota_limits['soft']+isset}" && \ + -n "${ldap_quota_limits['hard']+isset}" ]]; then + _soft_quota_limit="${ldap_quota_limits['soft']}" + _hard_quota_limit="${ldap_quota_limits['hard']}" + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values missing for group ${_group_from_lfs_path} on LFS ${_lfs_from_lfs_path}." + continue + fi + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " _soft_quota_limit contains: ${_soft_quota_limit}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " _hard_quota_limit contains: ${_hard_quota_limit}." + # + # Check for negative numbers and non-integers. + # + if [[ ! "${_soft_quota_limit}" =~ ${_pos_int_regex} || ! "${_hard_quota_limit}" =~ ${_pos_int_regex} ]]; then + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values malformed for group ${_group_from_lfs_path} on LFS ${_lfs_from_lfs_path}. Must be integers >= 0." + continue + fi + # + # Check if soft limit is larger than the hard limit as that will quota commands to fail. + # + if [[ "${_soft_quota_limit}" -gt "${_hard_quota_limit}" ]]; then + log4Bash 'ERROR' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values malformed for group ${_group_from_lfs_path} on LFS ${_lfs_from_lfs_path}. Soft limit cannot be larger than hard limit." + continue + fi + # + # Check for 0 (zero). + # When quota values are set to zero it means unlimited: not what we want. + # When zero was specified we'll interpret this as "do not allow this group to consume any space". + # Due to the technical limitations of how quota work we'll configure the lowest possible value instead: + # This is 2 * the block/stripe size on Lustre File Systems. + # + if [[ "${_soft_quota_limit}" -eq 0 ]]; then + _soft_quota_limit='2M' + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Converted soft quota limit of 0 (zero) for group ${_group_from_lfs_path} on LFS ${_lfs_from_lfs_path} to lowest possible value of ${_soft_quota_limit}." + else + # Just append unit: all quota values from the IDVault are in GB. + _soft_quota_limit="${_soft_quota_limit}G" + fi + if [[ "${_hard_quota_limit}" -eq 0 ]]; then + _hard_quota_limit='2M' + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' " Converted hard quota limit of 0 (zero) for group ${_group_from_lfs_path} on LFS ${_lfs_from_lfs_path} to lowest possible value of ${_hard_quota_limit}." + else + # Just append unit: all quota values from the IDVault are in GB. + _hard_quota_limit="${_hard_quota_limit}G" + fi + # + # Get the GID for this group, which will be used as the file set / project ID for quota accounting. + # + local _gid + _gid="$(getent group "${_group_from_lfs_path}" | awk -F ':' '{printf $3}')" + if [[ "${_fs_type}" == 'lustre' ]]; then + applyLustreQuota "${_lfs_path}" "${_gid}" "${_soft_quota_limit}" "${_hard_quota_limit}" + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " Cannot configure quota due to unsuported file system type: ${_fs_type}." + fi + done +} + +# +# Set Lustre project a.k.a. file set a.k.a folder quota limits: +# * Set project attribute on LFS path using GID as project ID. +# * Use lfs setquota to configure quota limit for project. +# +function applyLustreQuota () { + local _lfs_path="${1}" + local _gid="${2}" + local _soft_quota_limit="${3}" + local _hard_quota_limit="${4}" + local _cmd + local -a _cmds + if [[ "${apply_settings}" -eq 1 ]]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' " Executing quota commands ..." + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " Dry run: the following quota commands would have been executed with the '-a' switch ..." + fi + if [[ "${lustre_quota_type}" == 'project' ]]; then + _cmds=( + "chattr +P ${_lfs_path}" + "chattr -p ${_gid} ${_lfs_path}" + "lfs setquota -p ${_gid} --block-softlimit ${_soft_quota_limit} --block-hardlimit ${_hard_quota_limit} ${_lfs_path}" + ) + elif [[ "${lustre_quota_type}" == 'group' ]]; then + _cmds=( + "lfs setquota -g ${_gid} --block-softlimit ${_soft_quota_limit} --block-hardlimit ${_hard_quota_limit} ${_lfs_path}" + ) + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' " Unsuported Lustre quota type: ${lustre_quota_type}." + fi + for _cmd in "${_cmds[@]}"; do + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' " Applying cmd: ${_cmd}" + if [[ "${apply_settings}" -eq 1 ]]; then + mixed_stdouterr="$(${_cmd})" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Failed to execute: ${_cmd}" + fi + done +} + +function getQuotaFromLDAP () { + local _lfs="${1}" + local _group="${2}" + local _ldap_attr_regex='([^: ]{1,})(:{1,2}) ([^:]{1,})' + local _ldif_file="${ldif_dir}/${_group}.ldif" + local _ldif_record + local -a _ldif_records + local _ldap_group_quota_soft_limit_key="${ldap_group_quota_soft_limit_template/LFS/${_lfs}}" + local _ldap_group_quota_hard_limit_key="${ldap_group_quota_hard_limit_template/LFS/${_lfs}}" + # + # Query LDAP + # + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Retrieving data from LDAP..." + mixed_stdouterr=$(ldapsearch -LLL -o ldif-wrap=no \ + -H "${LDAP_HOST}" \ + -D "${LDAP_USER}" \ + -w "${LDAP_PASS}" \ + -b "${LDAP_SEARCH_BASE}" \ + "(&(ObjectClass=${ldap_group_object_class})(cn:dn:=${_group}))" \ + "${_ldap_group_quota_soft_limit_key}" \ + "${_ldap_group_quota_hard_limit_key}" \ + 2>&1 >"${_ldif_file}") \ + || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "ldapsearch failed." + # + # Parse query results. + # + while IFS= read -r -d '' _ldif_record; do + _ldif_records+=("$_ldif_record") + done < <(sed 's/^$/\x0/' "${_ldif_file}") \ + || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Parsing LDIF file (${_ldif_file}) into records failed." + # + # Loop over records in the array and create a faked-multi-dimensional hash. + # + for _ldif_record in "${_ldif_records[@]}"; do + # + # Remove trailing white space like the new line character. + # And skip blank lines. + # + _ldif_record="${_ldif_record%%[[:space:]]}" + [[ "${_ldif_record}" == '' ]] && continue + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "LDIF record contains: ${_ldif_record}" + # + # Parse record's key:value pairs. + # + local -A _directory_record_attributes=() + local _ldif_line + while IFS=$'\n' read -r _ldif_line; do + [[ "${_ldif_line}" == '' ]] && continue # Skip blank lines. + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "LDIF key:value pair contains: ${_ldif_line}." + if [[ "${_ldif_line}" =~ ${_ldap_attr_regex} ]]; then + local _key="${BASH_REMATCH[1],,}" # Convert key on-the-fly to lowercase. + local _sep="${BASH_REMATCH[2]}" + local _value="${BASH_REMATCH[3]}" + # + # Check if value was base64 encoded (double colon as separator) + # or plain text (single colon as separator) and decode if necessary. + # + if [[ "${_sep}" == '::' ]]; then + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " decoding base64 encoded value..." + _value="$(printf '%s' "${_value}" | base64 -di)" + fi + # + # This may be a multi-valued attribute and therefore check if key already exists; + # When key already exists make sure we append instead of overwriting the existing value(s)! + # + if [[ -n "${_directory_record_attributes[${_key}]+isset}" ]]; then + _directory_record_attributes["${_key}"]="${_directory_record_attributes["${_key}"]} ${_value}" + else + _directory_record_attributes["${_key}"]="${_value}" + fi + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " key contains: ${_key}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " value contains: ${_value}." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Failed to parse LDIF key:value pair (${_ldif_line})." + fi + done < <(printf '%s\n' "${_ldif_record}") || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Parsing LDIF record failed." + # + # Get Quota from processed LDIF record if this the right group. + # + local _ldap_group + if [[ -n "${_directory_record_attributes['dn']+isset}" ]]; then + # + # Parse cn from dn. + # + _ldap_group=$(dn2cn "${_directory_record_attributes['dn']}") + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Found group ${_ldap_group} in dn attribute." + else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "dn attribute missing for ${_ldif_record}" + fi + if [[ "${_ldap_group}" == "${_group}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Group from ldap record matches the group we were looking for: ${_ldap_group}." + else + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Skipping LDAP group ${_ldap_group} that does not match the LFS group ${_group} we were looking for." + continue + fi + # + # Get quota values for this group on this LFS. + # + if [[ -n "${_directory_record_attributes["${_ldap_group_quota_soft_limit_key}"]+isset}" && \ + -n "${_directory_record_attributes["${_ldap_group_quota_hard_limit_key}"]+isset}" ]]; then + ldap_quota_limits['soft']="${_directory_record_attributes["${_ldap_group_quota_soft_limit_key}"]}" + ldap_quota_limits['hard']="${_directory_record_attributes["${_ldap_group_quota_hard_limit_key}"]}" + return + else + log4Bash 'WARN' "${LINENO}" "${FUNCNAME:-main}" '0' " Quota values missing for group ${_ldap_group} on LFS ${_lfs_from_lfs_path}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' " Search keys were ${_ldap_group_quota_soft_limit_key} and ${_ldap_group_quota_hard_limit_key}." + continue + fi + done +} + + +# +# Extract a CN from a DN LDAP attribute. +# +function dn2cn () { + # cn=umcg-someuser,ou=users,ou=umcg,o=rs + local _dn="$1" + local _cn='MIA' + local _regex='cn=([^, ]+)' + if [[ "${_dn}" =~ ${_regex} ]]; then + _cn="${BASH_REMATCH[1]}" + fi + printf '%s' "${_cn}" +} + +# +## +### Main. +## +# + +# +# Get commandline arguments. +# +declare apply_settings=0 +while getopts ":l:ah" opt; do + case "${opt}" in + h) + showHelp + ;; + a) + apply_settings=1 + ;; + l) + l4b_log_level="${OPTARG^^}" + l4b_log_level_prio="${l4b_log_levels["${l4b_log_level}"]}" + ;; + \?) + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Invalid option -${OPTARG}. Try $(basename "${0}") -h for help." + ;; + :) + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Option -${OPTARG} requires an argument. Try $(basename "${0}") -h for help." + ;; + esac +done + +# +# Get credentials from config file. +# +# LDAP_USER='some_account' +# LDAP_PASS='some_passwd' +# LDAP_SEARCH_BASE='ou=groups,ou=some_org_unit,o=some_org' +# +if [[ -r "${config_file}" && -f "${config_file}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Fetching ldapsearch credentials from config file ${config_file} ..." + LDAP_HOST="$(awk '$1 == "uri" {print $2}' "${config_file}")" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Failed to parse LDAP_HOST from ${config_file}." + LDAP_USER="$(awk '$1 == "binddn" {print $2}' "${config_file}")" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Failed to parse LDAP_USER from ${config_file}." + LDAP_PASS="$(awk '$1 == "bindpw" {print $2}' "${config_file}")" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Failed to parse LDAP_PASS from ${config_file}." + LDAP_SEARCH_BASE="$(awk '$1 == "base" {print $2}' "${config_file}")" || log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" "${?}" "Failed to parse LDAP_SEARCH_BASE from ${config_file}." + log4Bash 'TRACE' "${LINENO}" "${FUNCNAME:-main}" '0' "Using LDAP credentials: LDAP_HOST=${LDAP_HOST} | LDAP_USER=${LDAP_USER} | LDAP_PASS=${LDAP_PASS} | LDAP_SEARCH_BASE=${LDAP_SEARCH_BASE}" +else + log4Bash 'FATAL' "${LINENO}" "${FUNCNAME:-main}" '1' "Config file ${config_file} missing or not accessible." +fi + +if [ "${apply_settings}" -eq 1 ]; then + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' 'Found option -a: will fetch, list and apply settings.' +else + log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" '0' 'Option -a not specified: will only perform a "dry run" to fetch + list settings. Use -a to apply settings.' +fi + +# +# Get a list of LFS paths: folders for groups, which we want to apply project quota to. +# On a SAI always in format/location: +# /mnt/${pfs}/groups/${group}/${lfs}/ +# E.g.: +# /mnt/umcgst02/groups/umcg-atd/prm08/ +# +readarray -t lfs_paths < <(find /mnt/*/groups/*/ -maxdepth 1 -mindepth 1 -type d -name "[a-z][a-z]*[0-9][0-9]*") + +# +# Create tmp dir for LDIFs with results from LDAP queries. +# +mkdir -p "${ldif_dir}" + +# +# Get quota values from LDAP and apply quota limits to file systems. +# +processFileSystems "${lfs_paths[@]:-}" + +# +# Cleanup tmp files. +# +if [[ "${l4b_log_level_prio}" -lt "${l4b_log_levels['INFO']}" ]]; then + log4Bash 'DEBUG' "${LINENO}" "${FUNCNAME:-main}" '0' "Debug mode: temporary dir ${ldif_dir} won't be removed." +else + rm -Rf "${ldif_dir}" +fi + +# +# Reset trap and exit. +# +log4Bash 'INFO' "${LINENO}" "${FUNCNAME:-main}" 0 "Finished!" +trap - EXIT +exit 0 + +{% endraw %} \ No newline at end of file diff --git a/roles/slurm-client/handlers/main.yml b/roles/slurm-client/handlers/main.yml index cf4afb71c..c1ff66494 100644 --- a/roles/slurm-client/handlers/main.yml +++ b/roles/slurm-client/handlers/main.yml @@ -2,7 +2,7 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # # Restart before reload: an reload after a restart may be redundant but should not fail, # but the other way around may fail when the impact of changes was too large for a reload. diff --git a/roles/slurm-client/tasks/main.yml b/roles/slurm-client/tasks/main.yml index 4454a2aba..87da7ce25 100644 --- a/roles/slurm-client/tasks/main.yml +++ b/roles/slurm-client/tasks/main.yml @@ -1,65 +1,65 @@ --- -- name: 'Gather facts from servers in "slurm-management" group.' +- name: 'Gather facts from servers in "sys-admin-interface" group.' setup: delegate_to: "{{ item }}" delegate_facts: true - with_items: "{{ groups['slurm-management'] }}" + with_items: "{{ groups['sys-admin-interface'] }}" -- name: Include Slurm vars. +- name: 'Include Slurm defaults from "slurm-management" role.' include_vars: - file: ../../slurm-management/defaults/main.yml - name: slurm + file: '../../slurm-management/defaults/main.yml' + name: 'slurm' -- name: Add Slurm group. +- name: 'Add Slurm group.' group: - name: slurm + name: 'slurm' gid: "{{ slurm['slurm_gid'] }}" notify: - - restart_slurmd + - 'restart_slurmd' become: true -- name: Add Munge group. +- name: 'Add Munge group.' group: - name: munge + name: 'munge' gid: "{{ slurm['munge_gid'] }}" notify: - - restart_munge - - restart_slurmd + - 'restart_munge' + - 'restart_slurmd' become: true -- name: Add Slurm user. +- name: 'Add Slurm user.' user: - name: slurm + name: 'slurm' uid: "{{ slurm['slurm_uid'] }}" - group: slurm + group: 'slurm' notify: - - restart_slurmd + - 'restart_slurmd' become: true -- name: Add Munge user. +- name: 'Add Munge user.' user: - name: munge + name: 'munge' uid: "{{ slurm['munge_uid'] }}" - group: munge + group: 'munge' notify: - - restart_munge - - restart_slurmd + - 'restart_munge' + - 'restart_slurmd' become: true -- name: Install the Slurm client with yum. +- name: 'Install the Slurm client with yum.' yum: - state: installed + state: 'installed' update_cache: yes allow_downgrade: yes name: - "slurm*{{ slurm_version }}" - "slurm-slurmd*{{ slurm_version }}" notify: - - restart_munge - - restart_slurmd + - 'restart_munge' + - 'restart_slurmd' become: true -- name: Patch slurm daemon systemd service files to use custom sub dir for PID files. +- name: 'Patch slurm daemon systemd service files to use custom sub dir for PID files.' lineinfile: path: "/usr/lib/systemd/system/{{ item }}.service" regexp: '^PIDFile=' @@ -67,58 +67,58 @@ with_items: - 'slurmd' notify: - - restart_slurmd + - 'restart_slurmd' become: true -- name: Install NHC with yum. +- name: 'Install NHC with yum.' yum: - state: latest + state: 'latest' update_cache: yes name: - 'lbnl-nhc' notify: - - restart_slurmd + - 'restart_slurmd' become: true -- name: Install munge_keyfile. +- name: 'Install munge_keyfile.' copy: - src: roles/slurm-management/files/{{ slurm_cluster_name }}_munge.key - owner: munge - group: munge - mode: 0600 - dest: /etc/munge/munge.key + src: "roles/slurm-management/files/{{ slurm_cluster_name }}_munge.key" + owner: 'munge' + group: 'munge' + mode: '0600' + dest: '/etc/munge/munge.key' notify: - - restart_munge - - restart_slurmd + - 'restart_munge' + - 'restart_slurmd' become: true -- name: Create folders for Slurm and NHC. +- name: 'Create folders for Slurm and NHC.' file: name: "{{ item.name }}" owner: "{{ item.owner }}" group: "{{ item.group }}" mode: "{{ item.mode }}" - state: directory + state: 'directory' with_items: - - name: /etc/slurm - owner: root - group: root + - name: '/etc/slurm' + owner: 'root' + group: 'root' mode: '0755' - - name: /etc/nhc - owner: root - group: root + - name: '/etc/nhc' + owner: 'root' + group: 'root' mode: '0755' - - name: /var/log/slurm - owner: slurm - group: root + - name: '/var/log/slurm' + owner: 'slurm' + group: 'root' mode: '0750' - - name: /var/spool/slurm - owner: slurm - group: root + - name: '/var/spool/slurm' + owner: 'slurm' + group: 'root' mode: '0750' - - name: /var/spool/slurmd - owner: root - group: root + - name: '/var/spool/slurmd' + owner: 'root' + group: 'root' mode: '0755' - name: '/var/run/slurm' owner: 'slurm' @@ -126,69 +126,79 @@ mode: '0775' become: true -- name: Deploy slurm prolog/epilog scripts. +- name: 'Deploy slurm prolog/epilog scripts.' copy: - src: roles/slurm-management/files/{{ item }} - dest: /etc/slurm/ - owner: root - group: root - mode: 0755 + src: "roles/slurm-management/files/{{ item }}" + dest: '/etc/slurm/' + owner: 'root' + group: 'root' + mode: '0755' with_items: - - slurm.prolog - - slurm.epilog - - slurm.taskprolog + - 'slurm.prolog' + - 'slurm.epilog' + - 'slurm.taskprolog' become: true -- name: Deploy slurm.conf. +- name: 'Deploy slurm.conf.' template: - src: roles/slurm-management/templates/slurm.conf - dest: /etc/slurm/slurm.conf - owner: root - group: root - mode: 0644 + src: 'roles/slurm-management/templates/slurm.conf' + dest: '/etc/slurm/slurm.conf' + owner: 'root' + group: 'root' + mode: '0644' notify: - - reload_slurmd + - 'restart_slurmd' become: true -- name: Configure cgroups. +- name: 'Configure cgroups.' copy: - src: roles/slurm-management/files/cgroup.conf - dest: /etc/slurm/cgroup.conf - owner: root - group: root - mode: 0644 + src: 'roles/slurm-management/files/cgroup.conf' + dest: '/etc/slurm/cgroup.conf' + owner: 'root' + group: 'root' + mode: '0644' notify: - - reload_slurmd + - 'reload_slurmd' become: true -- name: Deploy UI nhc.conf. +- name: 'Deploy UI nhc.conf.' template: - src: templates/user-interface_nhc.conf - dest: /etc/nhc/nhc.conf - owner: root - group: root - mode: 0644 + src: 'templates/user-interface_nhc.conf' + dest: '/etc/nhc/nhc.conf' + owner: 'root' + group: 'root' + mode: '0644' when: inventory_hostname in groups['user-interface'] become: true -- name: Deploy compute-vm nhc.conf. +- name: 'Deploy compute-vm nhc.conf.' template: - src: templates/compute-vm_nhc.conf - dest: /etc/nhc/nhc.conf - owner: root - group: root - mode: 0644 + src: 'templates/compute-vm_nhc.conf' + dest: '/etc/nhc/nhc.conf' + owner: 'root' + group: 'root' + mode: '0644' when: inventory_hostname in groups['compute-vm'] become: true -- name: Start slurm and munge services. +- name: 'Start slurm and munge services.' systemd: name: "{{ item }}" enabled: 'yes' state: 'started' daemon_reload: 'yes' with_items: - - munge.service - - slurmd.service + - 'munge.service' + - 'slurmd.service' + become: true + +- name: 'Allow passwordless sudo to slurm user for users in the functional_admins_group.' + template: + src: 'roles/slurm-management/templates/91-slurm' + dest: "/etc/sudoers.d/91-slurm" + owner: 'root' + group: 'root' + mode: '0440' + when: functional_admin_group is defined and functional_admin_group | length >= 1 become: true ... diff --git a/roles/slurm-client/templates/compute-vm_nhc.conf b/roles/slurm-client/templates/compute-vm_nhc.conf index 0d5339d70..fdefe3ff6 100644 --- a/roles/slurm-client/templates/compute-vm_nhc.conf +++ b/roles/slurm-client/templates/compute-vm_nhc.conf @@ -101,7 +101,7 @@ # All nodes should have their tmp filesystems mounted read/write. {% for lfs_mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %} {% for group in lfs_mount.groups %} - * || check_fs_mount_rw -f /{{ lfs_mount.lfs | regex_replace('GROUP', group) }} -t '/(gpfs|lustre|nfs)/' + * || check_fs_mount_rw -f /groups/{{ group }}/{{ lfs_mount.lfs }} -t '/(gpfs|lustre|nfs)/' {% endfor %} {% endfor %} diff --git a/roles/slurm-client/templates/user-interface_nhc.conf b/roles/slurm-client/templates/user-interface_nhc.conf index 064e8a0c2..79f88ae90 100644 --- a/roles/slurm-client/templates/user-interface_nhc.conf +++ b/roles/slurm-client/templates/user-interface_nhc.conf @@ -98,14 +98,14 @@ # All UIs should have their tmp filesystems mounted read/write. {% for lfs_mount in lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list %} {% for group in lfs_mount.groups %} - * || check_fs_mount_rw -f /{{ lfs_mount.lfs | regex_replace('GROUP', group) }} -t '/(gpfs|lustre|nfs)/' + * || check_fs_mount_rw -f /groups/{{ group }}/{{ lfs_mount.lfs }} -t '/(gpfs|lustre|nfs)/' {% endfor %} {% endfor %} # All UIs should have their prm filesystems mounted read/write. {% for lfs_mount in lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list %} {% for group in lfs_mount.groups %} - * || check_fs_mount_rw -f /{{ lfs_mount.lfs | regex_replace('GROUP', group) }} -t '/(gpfs|lustre|nfs)/' + * || check_fs_mount_rw -f /groups/{{ group }}/{{ lfs_mount.lfs }} -t '/(gpfs|lustre|nfs)/' {% endfor %} {% endfor %} diff --git a/roles/slurm-management/defaults/main.yml b/roles/slurm-management/defaults/main.yml index 1e287fe9d..975969cf2 100644 --- a/roles/slurm-management/defaults/main.yml +++ b/roles/slurm-management/defaults/main.yml @@ -4,4 +4,17 @@ slurm_uid: 497 slurm_gid: 497 munge_uid: 498 munge_gid: 498 +# +# slurm_allow_jobs_to_span_nodes +# +# We always use fast network interconnects nodes <-> large shared storage devices, +# but do not always have fast, low latency network interconnects between nodes. +# false (default) Should be used when fast, low latency network interconnects between nodes are not available. +# This will set MaxNodes = 1 for all nodes of all partitions in slurm.conf, +# which limits (MPI) jobs to max the amount of cores on a single node. +# true Should be used when fast, low latency network interconnects between nodes are present. +# Allows (MPI) jobs to use multiple nodes. +# Will set MaxNodes in slurm.conf to the total amount of compute nodes in the Slurm cluster. +# +slurm_allow_jobs_to_span_nodes: false ... diff --git a/roles/slurm-management/files/cgroup.conf b/roles/slurm-management/files/cgroup.conf index 25d761d1f..60ced80b8 100644 --- a/roles/slurm-management/files/cgroup.conf +++ b/roles/slurm-management/files/cgroup.conf @@ -1,19 +1,35 @@ -### +####################################################### # # Slurm cgroup support configuration file # # See man slurm.conf and man cgroup.conf for further # information on cgroup configuration parameters +####################################################### -#CgroupMountpoint=/etc/slurm/cgroup -CgroupReleaseAgentDir="/etc/slurm/cgroup" CgroupAutomount=yes - ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSWAPSpace=yes +# +# Lustre / GPFS / NFS clients or daemons tend to use large buffers allocated in kernel memory space. +# Those buffers count as kernel memory used by the cgroup of a job. +# Unaware users might have their jobs exceed this kernel memory limit simply by reading large files. +# The "plain RAM" limit is not affected by this option. +# Note: we limit the total amount of RAM that can be assigned to Slurm jobs to be less than the system total +# in the slurm.conf file to make sure the OS, file system daemons, etc. have enough RAM available. +# +ConstrainKmemSpace=no -# Set the allowable swap space to 100% of the requested memory -# The virtual memory space of a job should be 2 times the requested amount -TaskAffinity=yes +# +# Don't let Slurm jobs swap as swapping kills performance taking the H out of HPC. +# +AllowedSwapSpace=0 +MemorySwappiness=0 +# +# Bind job tasks to a subset of the allocated cores using sched_setaffinity +# to prevent them from swapping to other cores during job execution, +# which would decrease performance. +# Requires the Portable Hardware Locality (hwloc) library. +# +TaskAffinity=yes diff --git a/roles/slurm-management/files/logrotate_slurm b/roles/slurm-management/files/logrotate_slurm new file mode 100644 index 000000000..ba5860a08 --- /dev/null +++ b/roles/slurm-management/files/logrotate_slurm @@ -0,0 +1,20 @@ + #Slurm Logrotate Configuration + ## + /var/log/slurm/*log { + compress + missingok + nocopytruncate + nocreate + nodelaycompress + nomail + notifempty + noolddir + rotate 5 + sharedscripts + size=5M + create 640 slurm root + postrotate + systemctl reload slurmctld.service + systemctl reload slurmdbd.service + endscript + } diff --git a/roles/slurm-management/files/marvin_munge.key b/roles/slurm-management/files/marvin_munge.key new file mode 100644 index 000000000..cb39b88d0 --- /dev/null +++ b/roles/slurm-management/files/marvin_munge.key @@ -0,0 +1,57 @@ +$ANSIBLE_VAULT;1.1;AES256 +64636661666336393330633264643931376439646265633539643163326438643664356633363539 +3934376333303764623436366633383333626136303265660a383837386438623137613839643466 +37666338333534393334623562346333343838646364303637353066626663346234373536353939 +6133396632343662620a306339346138356662323863646533623162333637663831623364336535 +30393532303130633263336536336130633365353539623336303866643533346631643364336133 +61323136613162396331313336663334346634636330616437643837323837666266303662656134 +64393165373332396232643463666333353435346430346566663365623532346434316531663431 +33363238623736623933386166386262383533623435326661363066323431613934303639363439 +35646236653366346465346536626236366632613366373535353866333432653537343032353332 +61663336326430323166333333383839613038303361346331653066666432663631393463393835 +64356362373136313732316637663232646631373337643235633233636163656530303261356462 +63333438326339313732633264313035653135333965376334636332633234616234313065613132 +30326266353031656164333337333132613662663532323939336364343332643233373165613964 +66666530626639303138313535643034336437333237353432666333626235656364336533343633 +38626162353739663233636635613437656363623439353262316530653733346431623465386365 +61326130376439356262663438346231366365316265323331626432653865353738366163356164 +37313963393334366464393363663165343938626336306662373261313337363166613964653039 +62633961333131323361343365396164383934313961393332366337653939313632366239616634 +62373166633938386461376531656230653462376538626132376631656466643731383635633630 +36633233613162336364346633646434373130343961303964303064356665396566323534363837 +39313262636334663863643432626439366663643964306532363232333362653166636636356264 +36326133653232363138306336323831303538656233633266343130633964353736393264613962 +37306362653731353166663539323963336430353665376436356535373938363666666136623962 +31306435303837353365616335393165393362633937653735333962383134633163366535613538 +32643262346237663234653061666263653266653931646462353731653039396664316265323132 +66366663303936613335323038623761623237376333343062646636353438333536366431656232 +31643432316633313231653566626665663132336261363364343763386463366264333233396537 +33313836636236326339613937363966353032313937396130653866633330383566623132383430 +61626538613863313435636361663931333636383261373065316333616566343261623534343435 +63333663616234343733663864636134623735396135366331323439643737383234646234626562 +32653166633933343261383133626565666562336632633730643930363531376362653737663164 +30376437366431313064356239323435623165303830336137653963656630666163396536623830 +32656230356636366537346539623831643861396363333666636634633439333964383530666564 +37303465333238366565356130386137613234653866646636323835313634363861653834366663 +66653533376334383762623264633536373964656265326130376364643564373565626539393633 +39613262323862393331363037313335633263663732363264313266643739313930383635353933 +38323839373661383765646435643861643133366466323534353236366237306364623134323666 +37363334613638313964366330323839663837633866376364633235653038383034363166666438 +36323964626439346638663434643363653666333234356630316634613032366562303561373663 +33613438313635313733613035353636373138383132393739393065333230633066353261333561 +34343535623634393036643531613661346539363166343364643533393364623435623861623563 +39373736336236653965656465323563663634343535363532313264336564633230333235656639 +39616131643664643637346438363233393430616636613261643762336265373061623231306666 +63633933333431356434363939343262316632653535616137663939336334653061613033336162 +34313438666637363936383166313638363366633465653263396535333365666537646163323939 +39313866383635313064663832333536346433383562633162316562313761656262646237383836 +61663737363265343737633535353663313237616264656235323565393930326635353465396362 +36623961653838383632396563393337616430303464363738326439643365353138313335373962 +30333366326238636433303937366438323166366164336466356235323065333164653237393233 +32376136626336386662613463313461636466616164663832373133393264386333623036623661 +30656630303435366566613733376365343139353733393030376461353432386265613966626433 +36393935643237363563353837333034653736373466383538633432623232383265643933343337 +38393762623733396232303961626139336336643432373164316337643762636336653132663762 +32663239626334343436303064383034316338613530623434633661653666343766663133353365 +62343339316137303763643635636161353231373865373334366135633466376334383563313538 +33393664663962653132 diff --git a/roles/slurm-management/handlers/main.yml b/roles/slurm-management/handlers/main.yml index 154e4af89..2536f2d2b 100644 --- a/roles/slurm-management/handlers/main.yml +++ b/roles/slurm-management/handlers/main.yml @@ -2,55 +2,55 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # # Restart before reload: an reload after a restart may be redundant but should not fail, # but the other way around may fail when the impact of changes was too large for a reload. # -- name: Restart munge service. +- name: 'Restart munge service.' systemd: name: 'munge.service' - state: 'restarted' - daemon_reload: 'yes' + state: restarted + daemon_reload: yes become: true listen: restart_munge -- name: Restart slurmdbd service. +- name: 'Restart slurmdbd service.' systemd: name: 'slurmdbd.service' - state: 'restarted' - daemon_reload: 'yes' + state: restarted + daemon_reload: yes become: true listen: restart_slurmdbd -- name: Restart slurmctld service. +- name: 'Restart slurmctld service.' systemd: name: 'slurmctld.service' - state: 'restarted' - daemon_reload: 'yes' + state: restarted + daemon_reload: yes become: true listen: restart_slurmctld # # Service reloads after restarts. # -- name: Reload munge service. +- name: 'Reload munge service.' systemd: name: 'munge.service' - state: reloaded + state: 'reloaded' become: true listen: reload_munge -- name: Reload slurmdbd service. +- name: 'Reload slurmdbd service.' systemd: name: 'slurmdbd.service' - state: reloaded + state: 'reloaded' become: true listen: reload_slurmdbd -- name: Reload slurmctld service. +- name: 'Reload slurmctld service.' systemd: name: 'slurmctld.service' - state: reloaded + state: 'reloaded' become: true listen: reload_slurmctld ... diff --git a/roles/slurm-management/meta/main.yml b/roles/slurm-management/meta/main.yml index d4e3df4aa..1d60ad0f8 100644 --- a/roles/slurm-management/meta/main.yml +++ b/roles/slurm-management/meta/main.yml @@ -1,5 +1,4 @@ --- dependencies: - - { role: docker } - { role: mariadb } ... diff --git a/roles/slurm-management/tasks/main.yml b/roles/slurm-management/tasks/main.yml index cdfb47ed0..477f6f888 100644 --- a/roles/slurm-management/tasks/main.yml +++ b/roles/slurm-management/tasks/main.yml @@ -1,54 +1,63 @@ # Deploy Slurm control host. --- -- name: Add Slurm group. +- name: 'Add Slurm group.' group: - name: slurm + name: 'slurm' gid: "{{ slurm_gid }}" notify: - - restart_slurmdbd - - restart_slurmctld + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Add Munge group. +- name: 'Add Munge group.' group: - name: munge + name: 'munge' gid: "{{ munge_gid }}" notify: - - restart_munge - - restart_slurmdbd - - restart_slurmctld + - 'restart_munge' + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Add Slurm user. +- name: 'Add Slurm user.' user: - name: slurm + name: 'slurm' uid: "{{ slurm_uid }}" - group: slurm + group: 'slurm' notify: - - restart_slurmdbd - - restart_slurmctld + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Add Munge user. +- name: 'Add Munge user.' user: - name: munge + name: 'munge' uid: "{{ munge_uid }}" - group: munge + group: 'munge' notify: - - restart_munge - - restart_slurmdbd - - restart_slurmctld + - 'restart_munge' + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Install munge +- name: 'Install munge.' yum: - state: latest + state: 'latest' update_cache: yes name: - - munge + - 'munge' become: true -- name: Install munge.key file. +- name: 'Install a logrotate file for slurm.' + copy: + src: "files/logrotate_slurm" + owner: 'root' + group: 'root' + mode: '0644' + dest: '/etc/logrotate.d/slurm' + become: true + +- name: 'Install munge.key file.' copy: src: "files/{{ slurm_cluster_name }}_munge.key" owner: 'munge' @@ -56,27 +65,27 @@ mode: '0600' dest: '/etc/munge/munge.key' notify: - - restart_munge - - restart_slurmdbd - - restart_slurmctld + - 'restart_munge' + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Install Slurm management dependencies with yum. +- name: 'Install Slurm management dependencies with yum.' yum: - state: latest + state: 'latest' update_cache: yes name: - 'MySQL-python' - 'lua-posix' notify: - - restart_munge - - restart_slurmdbd - - restart_slurmctld + - 'restart_munge' + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Install Slurm management deamons with yum. +- name: 'Install Slurm management deamons with yum.' yum: - state: installed + state: 'installed' update_cache: yes allow_downgrade: yes name: @@ -85,12 +94,12 @@ - "slurm-slurmdbd*{{ slurm_version }}" - "slurm-perlapi*{{ slurm_version }}" notify: - - restart_munge - - restart_slurmdbd - - restart_slurmctld + - 'restart_munge' + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Patch slurm daemon systemd service files to use custom sub dir for PID files. +- name: 'Patch slurm daemon systemd service files to use custom sub dir for PID files.' lineinfile: path: "/usr/lib/systemd/system/{{ item }}.service" regexp: '^PIDFile=' @@ -99,42 +108,46 @@ - 'slurmctld' - 'slurmdbd' notify: - - restart_slurmdbd - - restart_slurmctld + - 'restart_slurmdbd' + - 'restart_slurmctld' become: true -- name: Create a database for Slurm accounting. +- name: 'Create a database for Slurm accounting.' mysql_db: - login_host: 127.0.0.1 + login_host: 'localhost' login_user: 'root' login_password: "{{ MYSQL_ROOT_PASSWORD }}" + login_unix_socket: "/var/lib/mysql/mysql.sock" name: "{{ slurm_database_name }}" state: 'present' - no_log: True + no_log: true notify: - - restart_slurmdbd + - 'restart_slurmdbd' + - 'restart_slurmctld' -- name: Make sure the slurm database user is present and grant privileges on the Slurm accounting DB. +- name: 'Make sure the slurm database user is present and grant privileges on the Slurm accounting DB.' mysql_user: - login_host: 127.0.0.1 + login_host: 'localhost' login_user: 'root' login_password: "{{ MYSQL_ROOT_PASSWORD }}" + login_unix_socket: "/var/lib/mysql/mysql.sock" name: "{{ slurm_storage_user }}" password: "{{ slurm_storage_pass }}" - host: '%' + host: 'localhost' priv: "{{ slurm_database_name }}.*:ALL" connect_timeout: 120 - no_log: True + no_log: true notify: - - restart_slurmdbd + - 'restart_slurmdbd' + - 'restart_slurmctld' -- name: Create folders for Slurm. +- name: 'Create folders for Slurm.' file: name: "{{ item.name }}" owner: "{{ item.owner }}" group: "{{ item.group }}" mode: "{{ item.mode }}" - state: directory + state: 'directory' with_items: - name: '/etc/slurm' owner: 'root' @@ -154,29 +167,28 @@ mode: '0775' become: true -- name: Install Slurm config file. +- name: 'Install Slurm config file.' template: src: 'templates/slurm.conf' dest: '/etc/slurm/slurm.conf' owner: 'root' group: 'root' mode: '0644' - notify: reload_slurmctld + notify: 'restart_slurmctld' + tags: 'slurm.conf' become: true - tags: - - slurm.conf -- name: Install Slurm DBD config file. +- name: 'Install Slurm DBD config file.' template: src: 'templates/slurmdbd.conf' dest: '/etc/slurm/slurmdbd.conf' owner: 'root' group: 'root' mode: '0644' - notify: reload_slurmdbd + notify: 'reload_slurmdbd' become: true -- name: Install Slurm scripts. +- name: 'Install Slurm scripts.' copy: src: "files/{{ item }}" dest: "/etc/slurm/{{ item }}" @@ -187,9 +199,9 @@ - 'job_submit.lua' become: true -- name: Make sure slurmdbd and munge services are enabled and started. +- name: 'Make sure slurmdbd and munge services are enabled and started.' systemd: - name: "{{item}}" + name: "{{ item }}" state: 'started' enabled: 'yes' daemon_reload: 'yes' @@ -198,39 +210,39 @@ - 'slurmdbd.service' become: true -- name: Copy Slurm DB initialization script to host running slurmdbd. +- name: 'Copy Slurm DB initialization script to host running slurmdbd.' template: src: 'templates/configure_slurm_accounting_db.bash' dest: '/etc/slurm/configure_slurm_accounting_db.bash' owner: 'root' group: 'root' mode: '0700' + tags: 'create_database' become: true # # Flush handlers to make sure Slurm DB is present and connection settings have been updated. # -- meta: flush_handlers +- meta: 'flush_handlers' -- name: Execute Slurm DB initialization script on host running slurmdbd. - shell: | - /etc/slurm/configure_slurm_accounting_db.bash - tags: - - create_database +- name: 'Execute Slurm DB initialization script on host running slurmdbd.' + command: + cmd: '/etc/slurm/configure_slurm_accounting_db.bash' register: command_result retries: 3 delay: 5 until: - - command_result.stderr == '' - - command_result.stdout != '' + - command_result.stderr|length == 0 + - command_result.stdout|length > 0 # NOTE: cannot use command_result.rc to check for the exit value of script in failed_when or changed_when # as the sacctmgr commands used in the script may return exit code 1 when trying to create/modify # something that already exists. This results in "Nothing new added." on STDOUT, but no message on STDERR. # In case something is really wrong there will be messags printed to STDERR. - failed_when: command_result.stderr != '' + failed_when: command_result.stderr|length > 0 + tags: 'create_database' become: true -- name: Make sure slurmctld service is enabled and started now that the cluster DB is present. +- name: 'Make sure slurmctld service is enabled and started now that the cluster DB is present.' systemd: name: 'slurmctld.service' state: 'started' @@ -238,40 +250,48 @@ daemon_reload: yes become: true -- name: Make backup dir. +- name: 'Make backup dir.' file: path: '/root/slurm/backup' state: 'directory' owner: 'root' group: 'root' mode: '0700' - tags: - - backup + tags: 'backup' become: true -- name: Create Slurm accounting DB backup now. +- name: 'Create Slurm accounting DB backup now.' # noqa 301 shell: | - /bin/docker run --network host --rm mariadb \ - mysqldump --all-databases -uroot -p{{ MYSQL_ROOT_PASSWORD }} -h 127.0.0.1 \ - > /root/slurm/backup/slurm.sql - tags: - - backup + mysqldump --all-databases \ + -uroot -p{{ MYSQL_ROOT_PASSWORD }} \ + -h localhost --protocol=socket \ + > /root/slurm/backup/slurm.sql + tags: 'backup' no_log: true become: true -- name: Deploy cron job to backup the Slurm accounting DB every night. Keep 7 backups. +- name: 'Deploy cron job to backup the Slurm accounting DB every night. Keep 7 backups.' cron: name: 'Slurm database backup' minute: '11' hour: '3' job: > /bin/cp --backup=numbered /root/slurm/backup/slurm.sql /root/slurm/backup/slurm_bak.sql - && /bin/docker run --network host --rm mariadb - mysqldump --all-databases -uroot -p{{ MYSQL_ROOT_PASSWORD }} -h 127.0.0.1 + && mysqldump --all-databases -uroot -p{{ MYSQL_ROOT_PASSWORD }} -h localhost --protocol=socket > /root/slurm/backup/slurm.sql && /bin/find /root/slurm/backup/slurm_bak.sql.* -mtime 7 -delete - tags: - - backup + tags: 'backup' no_log: true become: true + +- name: 'Allow passwordless sudo to slurm user for users in the functional_admins_group.' + template: + src: 'templates/91-slurm' + dest: "/etc/sudoers.d/91-slurm" + owner: 'root' + group: 'root' + mode: '0440' + when: functional_admin_group is defined and functional_admin_group | length >= 1 + become: true + ... diff --git a/roles/slurm-management/templates/91-slurm b/roles/slurm-management/templates/91-slurm new file mode 100644 index 000000000..5b58ae336 --- /dev/null +++ b/roles/slurm-management/templates/91-slurm @@ -0,0 +1,11 @@ +# +# This file is deployed with the slurm role from the Ansible playbook of the league-of-robots repo. +# DO NOT EDIT MANUALLY; update source and re-deploy instead! +# +# {{ ansible_managed }} +# +# Allow users from the {{ functional_admin_group }} group to become the slurm user. +# The {{ functional_admin_group }} group must be listed in the Ansible variable functional_admin_group in +# league-of-robots/group_vars//vars.yml +# +%{{ functional_admin_group }} ALL=(slurm) NOPASSWD:ALL diff --git a/roles/slurm-management/templates/slurm.conf b/roles/slurm-management/templates/slurm.conf index e67612a87..6b4a2a53e 100644 --- a/roles/slurm-management/templates/slurm.conf +++ b/roles/slurm-management/templates/slurm.conf @@ -1,6 +1,6 @@ ClusterName={{ slurm_cluster_name }} -ControlMachine={{ hostvars[groups['slurm-management'][0]]['ansible_hostname'] }} -ControlAddr={{ hostvars[groups['slurm-management'][0]]['ansible_hostname'] }} +ControlMachine={{ hostvars[groups['sys-admin-interface'][0]]['ansible_hostname'] }} +ControlAddr={{ hostvars[groups['sys-admin-interface'][0]]['ansible_hostname'] }} #BackupController= #BackupAddr= # @@ -120,7 +120,7 @@ AccountingStorageEnforce=limits,qos # will also enable: associations #AcctGatherProfileType=acct_gather_profile/hdf5 #JobAcctGatherFrequency=30 AccountingStorageType=accounting_storage/slurmdbd -AccountingStorageHost={{ hostvars[groups['slurm-management'][0]]['ansible_hostname'] }} +AccountingStorageHost={{ hostvars[groups['sys-admin-interface'][0]]['ansible_hostname'] }} #AccountingStorageLoc=/var/log/slurm/slurm.accounting #AccountingStoragePass= #AccountingStorageUser= @@ -133,15 +133,9 @@ HealthCheckInterval=300 # # Partitions # -# Configure maxnodes = 1 for all nodes of all partitions, -# because we hardly use MPI and when we do never between nodes, -# but only with max the amount of cores on a single node. -# Therefore we don't have fast network interconnects between nodes. -# (We use the fast network interconnects only for nodes <-> large shared storage devices) -# EnforcePartLimits=YES -PartitionName=DEFAULT State=UP DefMemPerCPU=1024 MaxNodes=1 MaxTime=7-00:00:01 -PartitionName=regular Default=YES MaxNodes=1 Nodes={{ vcompute_hostnames }} MaxCPUsPerNode={{ vcompute_max_cpus_per_node }} MaxMemPerNode={{ vcompute_max_mem_per_node }} TRESBillingWeights="CPU=1.0,Mem=0.125G" DenyQos=ds-short,ds-medium,ds-long +PartitionName=DEFAULT State=UP DefMemPerCPU=1024 MaxNodes={% if slurm_allow_jobs_to_span_nodes is defined and true %}{{ groups['compute-vm']|list|length }}{% else %}1{% endif %} MaxTime=7-00:00:01 +PartitionName=regular Default=YES MaxNodes={% if slurm_allow_jobs_to_span_nodes is defined and true %}{{ groups['compute-vm']|list|length }}{% else %}1{% endif %} Nodes={{ vcompute_hostnames }} MaxCPUsPerNode={{ vcompute_max_cpus_per_node }} MaxMemPerNode={{ vcompute_max_mem_per_node }} TRESBillingWeights="CPU=1.0,Mem=0.125G" DenyQos=ds-short,ds-medium,ds-long PartitionName=ds Default=No MaxNodes=1 Nodes={{ ui_hostnames }} MaxCPUsPerNode=1 MaxMemPerNode=1024 TRESBillingWeights="CPU=1.0,Mem=1.0G" AllowQos=ds-short,ds-medium,ds-long # # COMPUTE NODES diff --git a/roles/slurm-management/templates/slurmdbd.conf b/roles/slurm-management/templates/slurmdbd.conf index 557bdf51b..8a1f66685 100644 --- a/roles/slurm-management/templates/slurmdbd.conf +++ b/roles/slurm-management/templates/slurmdbd.conf @@ -7,6 +7,7 @@ ArchiveSuspend=no AuthInfo=/var/run/munge/munge.socket.2 AuthType=auth/munge DbdHost={{ ansible_hostname }} +DbdPort=6819 DebugLevel=info #was: 4 # Temporarily increased to find cause of crashes #DebugLevel=debug5 @@ -18,7 +19,7 @@ PurgeSuspendAfter=1month LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurm/slurmdbd.pid SlurmUser=slurm -StorageHost=127.0.0.1 +StorageHost=localhost StoragePort=3306 StoragePass={{ slurm_storage_pass }} StorageType=accounting_storage/mysql diff --git a/roles/slurm_exporter/handlers/main.yml b/roles/slurm_exporter/handlers/main.yml new file mode 100644 index 000000000..fae2b7a80 --- /dev/null +++ b/roles/slurm_exporter/handlers/main.yml @@ -0,0 +1,26 @@ +--- +# +# Important: maintain correct handler order. +# Handlers are executed in the order in which they are defined +# and not in the order in which they are listed in a "notify: handler_name" statement! +# +# Restart before reload: an reload after a restart may be redundant but should not fail, +# but the other way around may fail when the impact of changes was too large for a reload. +# +- name: 'Restart prometheus-slurm-exporter service.' + systemd: + name: 'prometheus-slurm-exporter.service' + state: restarted + daemon_reload: yes + become: true + listen: restart_prometheus-slurm-exporter +# +# Service reloads after restarts. +# +- name: 'Reload prometheus-slurm-exporter service.' + systemd: + name: 'prometheus-slurm-exporter.service' + state: reloaded + become: true + listen: reload_prometheus-slurm-exporter +... \ No newline at end of file diff --git a/roles/slurm_exporter/tasks/main.yml b/roles/slurm_exporter/tasks/main.yml index 89710afbc..01d8eb56c 100644 --- a/roles/slurm_exporter/tasks/main.yml +++ b/roles/slurm_exporter/tasks/main.yml @@ -1,49 +1,45 @@ --- -- set_fact: +- name: 'Setting service_name fact.' + set_fact: service_name: prometheus-slurm-exporter -- file: - path: /usr/local/prometheus + +- name: 'Create directory for Prometheus.' + file: + path: '/usr/local/prometheus' state: directory - mode: 0755 + mode: '0755' owner: root group: root become: true -- name: Install binary +- name: "Install Prometheus {{ service_name }} binary." copy: src: "{{ playbook_dir }}/promtools/results/{{ service_name }}" dest: "/usr/local/prometheus/{{ service_name }}" - mode: 0755 + mode: '0755' owner: root group: root + notify: "restart_{{ service_name }}" become: true -- name: Install service files. +- name: "Install {{ service_name }} service file." template: src: "templates/{{ service_name }}.service" dest: "/etc/systemd/system/{{ service_name }}.service" - mode: 644 + mode: '0644' owner: root group: root tags: - service-files + notify: "restart_{{ service_name }}" become: true -- name: install service files - command: systemctl daemon-reload - become: true - -- name: enable service at boot +- name: "Make sure {{ service_name }} service is enabled and started." systemd: name: "{{ service_name }}.service" + state: started enabled: yes - become: true - -- name: make sure servcies are started. - systemd: - name: "{{ service_name }}" - state: restarted tags: - - start-service + - start-service become: true ... \ No newline at end of file diff --git a/roles/spacewalk_client/handlers/main.yml b/roles/spacewalk_client/handlers/main.yml index 1ed86f5fe..d4544aaf5 100644 --- a/roles/spacewalk_client/handlers/main.yml +++ b/roles/spacewalk_client/handlers/main.yml @@ -2,7 +2,7 @@ # # Important: maintain correct handler order. # Handlers are executed in the order in which they are defined -# and not in the order in whch they are listed in a "notify: handler_name" statement! +# and not in the order in which they are listed in a "notify: handler_name" statement! # - name: Restart spacewalk service. service: diff --git a/roles/spacewalk_client/tasks/main.yml b/roles/spacewalk_client/tasks/main.yml index e8e475e84..ce374b7bd 100644 --- a/roles/spacewalk_client/tasks/main.yml +++ b/roles/spacewalk_client/tasks/main.yml @@ -1,74 +1,70 @@ --- -- name: Install spacewalk client repo. +- name: 'Install spacewalk client repo.' yum: - name: https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8-client/epel-7-x86_64/00742644-spacewalk-repo/spacewalk-client-repo-2.8-11.el7.centos.noarch.rpm + name: "https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/\ + spacewalk-2.8-client/epel-7-x86_64/00742644-spacewalk-repo/spacewalk-client-repo-2.8-11.el7.centos.noarch.rpm" state: present become: true -- name: Install spacewalk client packages. +- name: 'Install spacewalk client packages.' yum: name: - - rhn-client-tools - - rhn-check - - rhn-setup - - rhnsd - - m2crypto - - yum-rhn-plugin + - 'rhn-client-tools' + - 'rhn-check' + - 'rhn-setup' + - 'rhnsd' + - 'm2crypto' + - 'yum-rhn-plugin' notify: - - restart_rhnsd + - 'restart_rhnsd' become: true -- name: Enable spacewalk service. +- name: 'Enable spacewalk service.' systemd: - name: "{{ item }}" + name: 'rhnsd.service' enabled: yes state: started - with_items: - - rhnsd.service notify: - - restart_rhnsd + - 'restart_rhnsd' become: true -- meta: flush_handlers +- meta: 'flush_handlers' -- name: Register client at the spacewalk server. +- name: 'Register client at the spacewalk server.' rhn_register: state: present - activationkey: "{{activation_key}}" - server_url: "{{spacewalk_server_url}}" - channels: "{{rhn_channels}}" + activationkey: "{{ activation_key }}" + server_url: "{{ spacewalk_server_url }}" + channels: "{{ rhn_channels }}" register: result until: result is succeeded retries: 3 delay: 3 ignore_errors: yes - no_log: True + no_log: true become: true -- name: Disable gpgcheck for spacewalk repo. +- name: 'Disable gpgcheck for spacewalk repo.' lineinfile: path: '/etc/yum/pluginconf.d/rhnplugin.conf' regexp: '^gpgcheck = [0-9].*' line: 'gpgcheck = 0' become: true -- name: Remove all (non-spacewalk) repo config files from /etc/yum.repos.d/. - shell: "rm -rf /etc/yum.repos.d/*" +- name: 'Remove all (non-spacewalk) repo config files from /etc/yum.repos.d/.' + shell: 'rm -rfv /etc/yum.repos.d/*' args: warn: false + register: deleted_items + changed_when: deleted_items.stdout | length >= 1 + failed_when: deleted_items.stderr | length >= 1 become: true -- name: Clear the yum cache. - command: "yum clean all" - args: - warn: false - ignore_errors: yes - become: true - -- name: Upgrade all packages to version specified in spacewalk channel. +- name: 'Upgrade all packages to version specified in spacewalk channel.' yum: name: '*' state: latest - exclude: slurm* # Specific Slurm versions configured in slurm-* roles, because we use a custom patched version. + update_cache: yes + exclude: 'slurm*' # Specific Slurm versions configured in slurm-* roles, because we use a custom patched version. become: true ... diff --git a/roles/ssh_host_signer/handlers/main.yml b/roles/ssh_host_signer/handlers/main.yml index 42b84222a..237eb351f 100644 --- a/roles/ssh_host_signer/handlers/main.yml +++ b/roles/ssh_host_signer/handlers/main.yml @@ -1,6 +1,8 @@ --- -- name: Reload SSH configuration. - service: +- name: 'Reload SSH configuration.' + systemd: name: sshd state: reloaded become: true + listen: reload_sshd +... diff --git a/roles/ssh_host_signer/tasks/main.yml b/roles/ssh_host_signer/tasks/main.yml index f2eac5690..ae2a7fa56 100644 --- a/roles/ssh_host_signer/tasks/main.yml +++ b/roles/ssh_host_signer/tasks/main.yml @@ -1,5 +1,5 @@ --- -- name: Find SSH host keys. +- name: 'Find SSH host keys.' find: path: "{{ ssh_host_signer_key_directory }}" pattern: ssh_host_*_key @@ -7,23 +7,23 @@ failed_when: "\"matched\" not in private_keys or private_keys.matched == 0" become: true -- name: Create local temporary directory. - local_action: - module: tempfile - suffix: ssh_host_signer +- name: 'Create local temporary directory.' + tempfile: + suffix: 'ssh_host_signer' state: directory + delegate_to: localhost changed_when: false register: temporary_directory become: false -- name: Fetch public keys. +- name: 'Fetch public keys.' fetch: src: "{{ item.path }}.pub" dest: "{{ temporary_directory.path }}/public_keys/" with_items: "{{ private_keys.files }}" changed_when: false -- name: Fetch existing certificates. +- name: 'Fetch existing certificates.' fetch: src: "{{ item.path }}-cert.pub" dest: "{{ temporary_directory.path }}/existing_certificates/" @@ -31,76 +31,90 @@ with_items: "{{ private_keys.files }}" changed_when: false -- name: Check if we have a CA private key with correct permissions. +- name: 'Check if we have a CA private key with correct permissions.' file: path: "{{ ssh_host_signer_ca_private_key }}" - mode: 0600 + mode: '0600' delegate_to: localhost -- name: Sign SSH keys. - command: ssh-keygen -s {{ ssh_host_signer_ca_private_key | quote }} -P {{ ssh_host_signer_ca_private_key_pass | quote }} -I {{ ssh_host_signer_id | quote }} -h -n {{ ssh_host_signer_hostnames | quote }} "{{ temporary_directory.path }}/public_keys/{{ inventory_hostname | quote }}{{ item.path | quote }}.pub" +- name: 'Sign SSH host keys.' + command: > + ssh-keygen -h + -s {{ ssh_host_signer_ca_private_key | quote }} + -P {{ ssh_host_signer_ca_private_key_pass | quote }} + -I {{ ssh_host_signer_id | quote }} + -n {{ ssh_host_signer_hostnames | quote }} + "{{ temporary_directory.path }}/public_keys/{{ inventory_hostname | quote }}{{ item.path | quote }}.pub" with_items: "{{ private_keys.files }}" changed_when: false delegate_to: localhost - no_log: True + no_log: true -- name: Find certificates. - local_action: - module: find +- name: 'Find generated certificates (== signed host keys).' + find: path: "{{ temporary_directory.path }}/public_keys/{{ inventory_hostname }}{{ ssh_host_signer_key_directory }}" - pattern: ssh_host_*_key-cert.pub + pattern: 'ssh_host_*_key-cert.pub' + delegate_to: localhost register: certificates -- name: Compare certificates. - local_action: shell diff <(ssh-keygen -L -f {{ item.path | quote }} | tail -n +2) <(ssh-keygen -L -f {{ temporary_directory.path | quote }}/existing_certificates/{{ inventory_hostname | quote }}{{ ssh_host_signer_key_directory }}/{{ item.path | basename | quote }} | tail -n +2) +- name: 'Compare generated certificates to existing certificates from servers.' + shell: | + set -o pipefail + diff <(ssh-keygen -L -f {{ item.path | quote }} | tail -n +2) \ + <(ssh-keygen -L -f {{ existing_certificates_dir }}/{{ item.path | basename | quote }} | tail -n +2) args: - executable: /bin/bash + executable: '/bin/bash' + vars: + existing_certificates_dir: "{{ temporary_directory.path | quote }}/existing_certificates/\ + {{ inventory_hostname | quote }}{{ ssh_host_signer_key_directory }}" with_items: "{{ certificates.files }}" changed_when: false failed_when: false + delegate_to: localhost register: certificate_comparison -- name: Copy certificates back to server. +- name: 'Copy certificates back to server if the generated ones are different from the existing ones.' copy: src: "{{ item.item.path }}" dest: "{{ ssh_host_signer_key_directory }}" when: item.rc != 0 with_items: "{{ certificate_comparison.results }}" - notify: Reload SSH configuration. + notify: reload_sshd become: true -- name: Remove local temporary directory. - local_action: - module: file +- name: 'Remove local temporary directory.' + file: name: "{{ temporary_directory.path }}" state: absent + delegate_to: localhost changed_when: false -- name: Add the signed certificates to SSH configuration file. +- name: 'Add the signed certificates to SSH configuration file.' lineinfile: dest: "{{ ssh_host_signer_ssh_config }}" - line: HostCertificate {{ item.path }}-cert.pub + line: "HostCertificate {{ item.path }}-cert.pub" state: present - insertafter: HostKey {{ item.path }} + insertafter: "HostKey {{ item.path }}" backup: yes - with_items: "{{ private_keys.files | selectattr('path', 'match', ssh_host_signer_key_types) | list}}" - notify: Reload SSH configuration. + with_items: "{{ private_keys.files | selectattr('path', 'match', ssh_host_signer_key_types) | list }}" + notify: reload_sshd become: true -- name: Remove HostKey directives from the SSH configuration file for unused ssh host key types. +- name: 'Remove HostKey directives from the SSH configuration file for unused ssh host key types.' lineinfile: dest: "{{ ssh_host_signer_ssh_config }}" - line: HostCertificate {{ item.path }}-cert.pub + line: "HostCertificate {{ item.path }}-cert.pub" state: absent - with_items: "{{ private_keys.files | rejectattr('path', 'match', ssh_host_signer_key_types) | list}}" - notify: Reload SSH configuration. + with_items: "{{ private_keys.files | rejectattr('path', 'match', ssh_host_signer_key_types) | list }}" + notify: reload_sshd become: true - -- name: Remove corresponding HostCertificate directives from the SSH configuration file for unused ssh host key types. + +- name: 'Remove corresponding HostCertificate directives from the SSH configuration file for unused ssh host key types.' lineinfile: dest: "{{ ssh_host_signer_ssh_config }}" - line: HostKey {{ item.path }} + line: "HostKey {{ item.path }}" state: absent - with_items: "{{ private_keys.files | rejectattr('path', 'match', ssh_host_signer_key_types) | list}}" - notify: Reload SSH configuration. - become: true \ No newline at end of file + with_items: "{{ private_keys.files | rejectattr('path', 'match', ssh_host_signer_key_types) | list }}" + notify: reload_sshd + become: true +... \ No newline at end of file diff --git a/roles/sshd/templates/sshd_config b/roles/sshd/templates/sshd_config index 8fd1d5966..61bd7ff7d 100644 --- a/roles/sshd/templates/sshd_config +++ b/roles/sshd/templates/sshd_config @@ -1,4 +1,14 @@ +{% if inventory_hostname in groups['jumphost'] %} +# +# Extra ports where sshd will listen in addition to the default 22 +# must be listed before the default of 22, +# because the geerlingly.security role will modify the last listed Port value +# and we use the geerlingly.security role with the default of 22. +# +Port 443 +{% endif %} Port 22 + UseDNS no # diff --git a/roles/static-hostname-lookup/templates/marvin_hosts b/roles/static-hostname-lookup/templates/marvin_hosts new file mode 100644 index 000000000..337db1b28 --- /dev/null +++ b/roles/static-hostname-lookup/templates/marvin_hosts @@ -0,0 +1,43 @@ +# +## +### /etc/hosts file for EJP-RD research clusters. +## +# + +# +# localhost +# +127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 +::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 + +# +# Jumphost servers. +# +192.168.0.9 dockingport + +# +# Admin / Management machines. +# DAI = Deploy Admin Interface +# SAI = Sys Admin Interface +# +192.168.0.12 mv-sai +192.168.0.11 mv-dai + +# +# Cluster User Interfaces (UIs). +# +192.168.0.4 marvin + +# +# Shared network storage servers. +# + +# +# Cluster nodes. +# +192.168.0.7 mv-vcompute01 + +# +# Documentation webserver. +# +195.169.22.76 docs diff --git a/roles/subgroup_directories/tasks/create_subgroup_directories.yml b/roles/subgroup_directories/tasks/create_subgroup_directories.yml new file mode 100644 index 000000000..2fdc66fc7 --- /dev/null +++ b/roles/subgroup_directories/tasks/create_subgroup_directories.yml @@ -0,0 +1,83 @@ +--- +- block: + - name: "Get list of {{ group }} subgroups with version number from the LDAP." + shell: | + ldapsearch -LLL -D '{{ ldap_binddn }}' -w '{{ bindpw }}' -b '{{ ldap_base }}' -H '{{ ldap_uri }}' \ + "(ObjectClass=GroupofNames)" dn \ + | tr "=," "\n" \ + | fgrep "{{ group }}" \ + | grep -v "^{{ group }}$" \ + | grep -- "-v[0-9][0-9]*$" \ + || true + register: versioned_subgroups + - set_fact: # noqa 502 + versioned_subgroups_list: "{% if versioned_subgroups.stdout | length %}{{ versioned_subgroups.stdout.split('\n') | list }}{% endif %}" + +- block: + - name: "Get list of {{ group }} subgroups without version number and excluding *-dms groups from the LDAP." + shell: | + ldapsearch -LLL -D '{{ ldap_binddn }}' -w '{{ bindpw }}' -b '{{ ldap_base }}' -H '{{ ldap_uri }}' \ + "(ObjectClass=GroupofNames)" dn \ + | tr "=," "\n" \ + | fgrep "{{ group }}" \ + | grep -v "^{{ group }}$" \ + | grep -v -- "-v[0-9][0-9]*$\|-dms$" \ + || true + register: unversioned_subgroups + - set_fact: # noqa 502 + unversioned_subgroups_list: "{% if unversioned_subgroups.stdout | length %}{{ unversioned_subgroups.stdout.split('\n') | list }}{% endif %}" + +- name: "Create directory structure for releases with version number on {{ lfs }}." + block: + - name: "Create /groups/{{ group }}/{{ lfs }}/releases/ directory." + file: + path: "/groups/{{ group }}/{{ lfs }}/releases/" + owner: "{{ group }}-dm" + group: "{{ group }}" + mode: "{{ mode_dataset }}" + state: 'directory' + - name: "Create /groups/{{ group }}/{{ lfs }}/releases/${dataset} directory." + file: + path: "/groups/{{ group }}/{{ lfs }}/releases/{{ item | regex_replace('^' + group + '-(.*)-(v[0-9][0-9]*)$', '\\1') }}" + owner: "{{ group }}-dm" + group: "{{ group }}" + mode: "{{ mode_dataset }}" + state: 'directory' + with_items: "{{ versioned_subgroups_list }}" + - name: "Create /groups/{{ group }}/{{ lfs }}/releases/${dataset}/${version} directory." + file: + path: "/groups/{{ group }}/{{ lfs }}/releases/\ + {{ item | regex_replace('^' + group + '-(.*)-(v[0-9][0-9]*)$', '\\1') }}/\ + {{ item | regex_replace('^' + group + '-(.*)-(v[0-9][0-9]*)$', '\\2') }}" + owner: "{{ group }}-dm" + group: "{% if item | length %}{{ item }}{% else %}{{ group }}{% endif %}" + mode: "{{ mode_version }}" + state: 'directory' + with_items: "{{ versioned_subgroups_list }}" + ignore_errors: yes # Continue of this specific subgroup failed and try to create other subgroup folders. + when: versioned_subgroups_list | length > 0 + become: true + become_user: "{{ group }}-dm" + +- name: "Create directory structure for projects on {{ lfs }}." + block: + - name: "Create /groups/{{ group }}/{{ lfs }}/projects directory." + file: + path: "/groups/{{ group }}/{{ lfs }}/projects/" + owner: "{{ group }}-dm" + group: "{{ group }}" + mode: "{{ mode_project }}" + state: 'directory' + - name: "Create /groups/{{ group }}/{{ lfs }}/projects/${project} directory." + file: + path: "/groups/{{ group }}/{{ lfs }}/projects/{{ item | regex_replace('^' + group + '-(.*)$', '\\1') }}" + owner: "{{ group }}-dm" + group: "{% if item | length %}{{ item }}{% else %}{{ group }}{% endif %}" + mode: "{{ mode_project }}" + state: 'directory' + with_items: "{{ unversioned_subgroups_list }}" + ignore_errors: yes # Continue of this specific subgroup failed and try to create other subgroup folders. + when: unversioned_subgroups_list | length > 0 + become: true + become_user: "{{ group }}-dm" +... diff --git a/roles/subgroup_directories/tasks/main.yml b/roles/subgroup_directories/tasks/main.yml new file mode 100644 index 000000000..4f834c99f --- /dev/null +++ b/roles/subgroup_directories/tasks/main.yml @@ -0,0 +1,33 @@ +--- +- name: 'Loop over "prm" file systems.' + include_tasks: + file: "{{ playbook_dir }}/roles/subgroup_directories/tasks/create_subgroup_directories.yml" + vars: + lfs: "{{ lfs_item.0.lfs }}" + group: "{{ lfs_item.1 }}" + mode_project: '2750' + mode_version: '2750' + mode_dataset: '2750' + with_subelements: + - "{{ lfs_mounts | selectattr('lfs', 'search', 'prm[0-9]+$') | list }}" + - 'groups' + loop_control: + loop_var: lfs_item + when: inventory_hostname in groups['user-interface'] + +- name: 'Loop over "tmp" file systems.' + include_tasks: + file: "{{ playbook_dir }}/roles/subgroup_directories/tasks/create_subgroup_directories.yml" + vars: + lfs: "{{ lfs_item.0.lfs }}" + group: "{{ lfs_item.1 }}" + mode_project: '2770' + mode_version: '2770' + mode_dataset: '2750' + with_subelements: + - "{{ lfs_mounts | selectattr('lfs', 'search', 'tmp[0-9]+$') | list }}" + - 'groups' + loop_control: + loop_var: lfs_item + when: inventory_hostname in groups['user-interface'] +... diff --git a/roles/sudoers/tasks/main.yml b/roles/sudoers/tasks/main.yml index 953535567..2879eb12e 100644 --- a/roles/sudoers/tasks/main.yml +++ b/roles/sudoers/tasks/main.yml @@ -1,14 +1,33 @@ --- # -# Allow passwordless sudo to the datamanager users for indivual users or %groups. -# This can be specified in the groupvars regular_users. +# Allow passwordless sudo to functional accounts (e.g. for datamanager accounts) for indivual users or %groups. +# Who can become what can be specified in regular_users variable of group_vars/${clustername}/vars.yml. # -- name: 'Allow passwordless sudo to the datamanager users.' - lineinfile: - dest: '/etc/sudoers' - line: "{{ item.1 }} ALL=({{ item.0.user }}) NOPASSWD:ALL" + +- name: 'Find all ansible-managed files in /etc/sudoers.d/.' + find: + paths: '/etc/sudoers.d/' + use_regex: yes + patterns: '.*ansible-managed.*' + register: ansible_managed_sudoers + become: true + +- name: 'Remove outdated ansible-managed files from /etc/sudoers.d/.' + file: + path: "{{ item }}" + state: absent + with_items: "{{ ansible_managed_sudoers.files | map (attribute='path') | list }}" + when: item | regex_replace('.*ansible-managed-','') not in regular_users | default([]) | selectattr('sudoers', 'defined') | map (attribute='user') | list + become: true + +- name: 'Allow passwordless sudo to functional accounts.' + template: + src: 'templates/92-ansible-managed' + dest: "/etc/sudoers.d/92-ansible-managed-{{ item.user }}" + owner: 'root' + group: 'root' + mode: '0440' + with_items: + - "{{ regular_users | default([]) | selectattr('sudoers', 'defined') | list }}" become: true - with_subelements: - - "{{ regular_users | default([]) | selectattr('sudoers', 'defined') | list }}" - - 'sudoers' ... diff --git a/roles/sudoers/templates/92-ansible-managed b/roles/sudoers/templates/92-ansible-managed new file mode 100644 index 000000000..dd468398b --- /dev/null +++ b/roles/sudoers/templates/92-ansible-managed @@ -0,0 +1,9 @@ +# +# This file is deployed with the sudoers role from the Ansible playbook of the league-of-robots repo. +# DO NOT EDIT MANUALLY; update source and re-deploy instead! +# +# {{ ansible_managed }} +# +# Allow specific users or group to become the {{ item.user }} user. +# +{% if functional_admin_group is defined and functional_admin_group | length %}%{{ functional_admin_group }},{% endif %}{{ item.sudoers }} ALL=({{ item.user }}) NOPASSWD:ALL diff --git a/roles/swap/tasks/main.yml b/roles/swap/tasks/main.yml index 2625b7b1c..17bcf0601 100644 --- a/roles/swap/tasks/main.yml +++ b/roles/swap/tasks/main.yml @@ -14,34 +14,34 @@ # fallocate doesn't work on centos7, see # https://unix.stackexchange.com/questions/294600/i-cant-enable-swap-space-on-centos-7 - name: create swap file - sudo: yes + become: yes command: dd if=/dev/zero bs=1MiB count={{ swap_space * 1024 }} of={{ swap_file }} when: not swap_file_check.stat.exists - name: set permissions on swap file - sudo: yes + become: yes file: path: "{{ swap_file }}" mode: 0600 - name: format swap file - sudo: yes + become: yes command: mkswap {{ swap_file }} when: not swap_file_check.stat.exists - name: add to fstab - sudo: yes + become: yes lineinfile: dest: /etc/fstab regexp: "{{ swap_file }}" line: "{{ swap_file }} none swap sw 0 0" - name: turn on swap - sudo: yes + become: yes command: swapon -a - name: set swapiness - sudo: yes + become: yes sysctl: name: vm.swappiness value: "1" diff --git a/single_role_playbooks/cadvisor.yml b/single_role_playbooks/cadvisor.yml new file mode 100644 index 000000000..36f022524 --- /dev/null +++ b/single_role_playbooks/cadvisor.yml @@ -0,0 +1,5 @@ +--- +- hosts: sys-admin-interface + roles: + - cadvisor +... diff --git a/single_role_playbooks/figlet.yml b/single_role_playbooks/coredumps.yml similarity index 60% rename from single_role_playbooks/figlet.yml rename to single_role_playbooks/coredumps.yml index ef6c16d61..f923eff5b 100644 --- a/single_role_playbooks/figlet.yml +++ b/single_role_playbooks/coredumps.yml @@ -1,5 +1,5 @@ --- - hosts: cluster roles: - - figlet_hostname + - coredumps ... diff --git a/single_role_playbooks/figlet_motd.yml b/single_role_playbooks/figlet_motd.yml new file mode 100644 index 000000000..1dd278439 --- /dev/null +++ b/single_role_playbooks/figlet_motd.yml @@ -0,0 +1,5 @@ +--- +- hosts: cluster + roles: + - figlet_motd +... diff --git a/single_role_playbooks/fuse-layer.yml b/single_role_playbooks/fuse-layer.yml index 0038542f0..87dba9d91 100644 --- a/single_role_playbooks/fuse-layer.yml +++ b/single_role_playbooks/fuse-layer.yml @@ -1,4 +1,5 @@ --- -- hosts: deploy-admin-interface +- hosts: user-interface roles: - fuse-layer +... diff --git a/single_role_playbooks/grafana.yml b/single_role_playbooks/grafana.yml new file mode 100644 index 000000000..85bac0ed7 --- /dev/null +++ b/single_role_playbooks/grafana.yml @@ -0,0 +1,5 @@ +--- +- hosts: sys-admin-interface + roles: + - grafana +... diff --git a/single_role_playbooks/prom_proxy.yml b/single_role_playbooks/grafana_proxy.yml similarity index 62% rename from single_role_playbooks/prom_proxy.yml rename to single_role_playbooks/grafana_proxy.yml index 2485f9a50..808157069 100644 --- a/single_role_playbooks/prom_proxy.yml +++ b/single_role_playbooks/grafana_proxy.yml @@ -1,5 +1,5 @@ --- - hosts: jumphost roles: - - prom_proxy + - grafana_proxy ... diff --git a/single_role_playbooks/iptables.yml b/single_role_playbooks/iptables.yml new file mode 100644 index 000000000..f707cf52d --- /dev/null +++ b/single_role_playbooks/iptables.yml @@ -0,0 +1,5 @@ +--- +- hosts: jumphost + roles: + - 'iptables' +... \ No newline at end of file diff --git a/single_role_playbooks/logins.yml b/single_role_playbooks/logins.yml index ab2eb678f..fcc696157 100644 --- a/single_role_playbooks/logins.yml +++ b/single_role_playbooks/logins.yml @@ -1,4 +1,7 @@ --- +- hosts: jumphost + roles: + - logins - hosts: cluster roles: - logins diff --git a/single_role_playbooks/monitoring.yml b/single_role_playbooks/monitoring.yml index 5b68ce314..99d31eac9 100644 --- a/single_role_playbooks/monitoring.yml +++ b/single_role_playbooks/monitoring.yml @@ -1,29 +1,33 @@ --- -- hosts: imperator - become: true +- name: 'Install monitoring tools.' + hosts: sys-admin-interface roles: - - prom_server - - cadvisor + - prom_server + - grafana + - cadvisor -- name: Monitoring of the virtualized components - hosts: cluster - become: true +- name: 'Monitoring of the hardware components.' + hosts: metal tasks: roles: - - node_exporter + - cadvisor + - node_exporter + - ipmi_exporter -- name: Monitoring of the hardware components - hosts: metal - become: true +- name: 'Monitoring of the virtualized components.' + hosts: jumphost, cluster tasks: roles: - - cadvisor - - node_exporter - - ipmi_exporter + - node_exporter + +- name: 'Monitoring of the SLURM resource manager.' + hosts: user-interface + roles: + - slurm_exporter -- name: Airlock proxies prometheus for Grafana in the cloud - hosts: airlock - become: true +- name: 'Jumphost proxies Prometheus for Grafana in the cloud.' + hosts: jumphost tasks: roles: - - prom_proxy + - grafana_proxy +... \ No newline at end of file diff --git a/single_role_playbooks/prom_server.yml b/single_role_playbooks/prom_server.yml index 03174f789..4eee6b60d 100644 --- a/single_role_playbooks/prom_server.yml +++ b/single_role_playbooks/prom_server.yml @@ -1,5 +1,5 @@ --- -- hosts: slurm-management +- hosts: sys-admin-interface roles: - prom_server ... diff --git a/single_role_playbooks/promtools b/single_role_playbooks/promtools new file mode 120000 index 000000000..a023dc44d --- /dev/null +++ b/single_role_playbooks/promtools @@ -0,0 +1 @@ +../promtools/ \ No newline at end of file diff --git a/single_role_playbooks/rsyncd.yml b/single_role_playbooks/rsyncd.yml new file mode 100644 index 000000000..ac2f4e836 --- /dev/null +++ b/single_role_playbooks/rsyncd.yml @@ -0,0 +1,5 @@ +--- +- hosts: data-staging + roles: + - rsyncd # Note: depends on ldap role, which will be included when deploying this one. +... diff --git a/single_role_playbooks/slurm-client.yml b/single_role_playbooks/slurm-client.yml index 463f2779f..158c7d202 100644 --- a/single_role_playbooks/slurm-client.yml +++ b/single_role_playbooks/slurm-client.yml @@ -1,6 +1,6 @@ --- -- hosts: slurm-management - name: Dummy to gather facts +- name: Dummy to gather facts + hosts: sys-admin-interface tasks: [] - name: Install virtual compute nodes diff --git a/single_role_playbooks/slurm-management.yml b/single_role_playbooks/slurm-management.yml index 316cc7965..0343c4dd5 100644 --- a/single_role_playbooks/slurm-management.yml +++ b/single_role_playbooks/slurm-management.yml @@ -1,7 +1,6 @@ --- -- hosts: slurm-management +- hosts: sys-admin-interface roles: - - docker - mariadb - slurm-management vars: diff --git a/single_role_playbooks/ssh-host-ca b/single_role_playbooks/ssh-host-ca new file mode 120000 index 000000000..b858c2172 --- /dev/null +++ b/single_role_playbooks/ssh-host-ca @@ -0,0 +1 @@ +../ssh-host-ca/ \ No newline at end of file diff --git a/single_role_playbooks/subgroup_directories.yml b/single_role_playbooks/subgroup_directories.yml new file mode 100644 index 000000000..581dd58b6 --- /dev/null +++ b/single_role_playbooks/subgroup_directories.yml @@ -0,0 +1,5 @@ +--- +- hosts: user-interface + roles: + - subgroup_directories +... diff --git a/single_role_playbooks/swap.yml b/single_role_playbooks/swap.yml index 61ce56bff..499eb014e 100644 --- a/single_role_playbooks/swap.yml +++ b/single_role_playbooks/swap.yml @@ -1,4 +1,4 @@ --- -- hosts: deploy-admin-interface +- hosts: cluster roles: - swap diff --git a/static_inventories/boxy_hosts.ini b/static_inventories/boxy_hosts.ini new file mode 100644 index 000000000..e86675f98 --- /dev/null +++ b/static_inventories/boxy_hosts.ini @@ -0,0 +1,15 @@ +[jumphost] +foyer + +[sys-admin-interface] +winterpeen + +[administration:children] +sys-admin-interface + +[cluster:children] +administration + +[boxy-cluster:children] +jumphost +cluster diff --git a/fender_hosts.ini b/static_inventories/fender_hosts.ini similarity index 93% rename from fender_hosts.ini rename to static_inventories/fender_hosts.ini index ca70b9bc4..3385890eb 100644 --- a/fender_hosts.ini +++ b/static_inventories/fender_hosts.ini @@ -4,9 +4,6 @@ corridor [docs] docs -[slurm-management] -fd-sai - [sys-admin-interface] fd-sai diff --git a/gearshift_hosts.ini b/static_inventories/gearshift_hosts.ini similarity index 86% rename from gearshift_hosts.ini rename to static_inventories/gearshift_hosts.ini index 5bb261c5f..f2c981652 100644 --- a/gearshift_hosts.ini +++ b/static_inventories/gearshift_hosts.ini @@ -4,9 +4,6 @@ airlock [docs] docs -[slurm-management] -imperator - [sys-admin-interface] imperator @@ -22,7 +19,7 @@ user-interface gearshift [compute-vm] -gs-vcompute[01:11] +gs-vcompute[01:10] [cluster:children] compute-vm diff --git a/hyperchicken_hosts.ini b/static_inventories/hyperchicken_hosts.ini similarity index 93% rename from hyperchicken_hosts.ini rename to static_inventories/hyperchicken_hosts.ini index 625fa1e57..ad680b82c 100644 --- a/hyperchicken_hosts.ini +++ b/static_inventories/hyperchicken_hosts.ini @@ -4,9 +4,6 @@ portal [docs] docs -[slurm-management] -hc-sai - [sys-admin-interface] hc-sai diff --git a/static_inventories/marvin_hosts.ini b/static_inventories/marvin_hosts.ini new file mode 100644 index 000000000..09d43c732 --- /dev/null +++ b/static_inventories/marvin_hosts.ini @@ -0,0 +1,35 @@ +[jumphost] +dockingport + +[docs] +docs + +[sys-admin-interface] +mv-sai + +[user-interface] +marvin + +[deploy-admin-interface] +mv-dai + +[ldap-server] +mv-dai + +[administration:children] +sys-admin-interface +deploy-admin-interface +user-interface +ldap-server + +[compute-vm] +mv-vcompute01 + +[cluster:children] +compute-vm +administration + +[marvin-cluster:children] +cluster +jumphost +docs diff --git a/nibbler-hosts.ini b/static_inventories/nibbler-hosts.ini similarity index 92% rename from nibbler-hosts.ini rename to static_inventories/nibbler-hosts.ini index 5486fdf09..24a534c08 100644 --- a/nibbler-hosts.ini +++ b/static_inventories/nibbler-hosts.ini @@ -4,9 +4,6 @@ portal ansible_host=10.0.42.3 [nfs-server] nb-nfs ansible_host=10.0.42.16 -[slurm-management] -nb-sai ansible_host=10.0.42.12 - [sys-admin-interface] nb-sai ansible_host=10.0.42.12 diff --git a/talos_hosts.ini b/static_inventories/talos_hosts.ini similarity index 90% rename from talos_hosts.ini rename to static_inventories/talos_hosts.ini index 7d2ce4d1b..220c20005 100644 --- a/talos_hosts.ini +++ b/static_inventories/talos_hosts.ini @@ -1,12 +1,12 @@ [jumphost] reception +[data-staging] +talos + [docs] docs -[slurm-management] -tl-sai - [sys-admin-interface] tl-sai @@ -29,6 +29,7 @@ compute-vm administration [talos-cluster:children] -cluster jumphost +cluster +data-staging docs