Make alt efs sync with live #2025

kencho51 · 2024-09-05T05:51:27Z

Pull request for issue: #2000

This is a pull request for the following functionalities:

How to test?

Describe how the new functionalities can be tested by PR reviewers

In local dev

% cd /gigadb/app/tools/sync-dropbox
# execute the configure script to generate .env
% ./configure 
An .env file wasn't present, creating a new one from the default example
Sourcing .env
./.env: line 16: syntax error near unexpected token `newline'
# then fill in the gitlab private token and the gitlab repo name in the .env and execute the configure script again to generate the rclone.conf file and get the ssh private key for accessing upstream server 
% ./configure  
An .env file is present
Sourcing .env
Current environment: dev
# then check the existence of the privet key 
% ls -al ~/.ssh/id-rsa-aws-hk-gigadb.pem
-rw-r--r--@ 1 kencho  staff  1675 Sep 19 16:12 /Users/kencho/.ssh/id-rsa-aws-hk-gigadb.pem
# then execute the bats tests
 % bats tests/bats/sync_dropbox.bats 
sync_dropbox.bats
 ✓ No parameter provided
 ✓ Execute in dry run mode
 ✓ Execute in apply mode

3 tests, 0 failures
# the last test will take several minutes to complete, as it tries to copy the files from the current existing upstream staging efs to yours local dev environment

Pre-requisites

Follow the docs/SETUP_PROVISIONING.md to spin up servers

In staging as a centos user

% ssh -i path/to/staging/pem centos@$staging-bastion-ip
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Mon Sep 30 04:36:53 2024 from 3.36.204.163
[centos@ip-10-99-0-22 ~]$ ls -al /share/dropbox/
total 0
drwxrwxr-x. 2 centos centos  6 Sep 30 03:54 .
drwxr-xr-x. 4 centos centos 35 Sep 30 03:54 ..
[centos@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 4
drwxr-xr-x.  2 root gigadb   30 Sep 30 04:23 .
drwxr-xr-x. 10 root root   4096 Sep 30 04:16 ..
-rw-rw-r--.  1 root gigadb    0 Sep 30 04:23 sync_dropbox.log
[centos@ip-10-99-0-22 ~]$ /usr/local/bin/sync_dropbox --apply
2024/09/30 05:50:24 INFO : Start sync dropbox from production-staging to alt staging
2024/09/30 05:51:10 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:51:10 INFO  : Successfully sync dropbox from production-staging to alt staging
[centos@ip-10-99-0-22 ~]$ ls -al /share/dropbox/
total 12
drwxrwxr-x. 6 centos centos   82 Sep 30 05:50 .
drwxr-xr-x. 4 centos centos   35 Sep 30 03:54 ..
-rw-rw-r--. 1 centos centos  166 Apr 25 16:55 rija_test.txt
drwxrwxr-x. 3 centos centos   36 Sep 30 05:50 user0
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user109
drwxrwxr-x. 4 centos centos  167 Sep 30 05:50 user27
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user4
[centos@ip-10-99-0-22 ~]$ 
[centos@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 12
drwxr-xr-x.  2 root gigadb   30 Sep 30 04:23 .
drwxr-xr-x. 10 root root   4096 Sep 30 04:16 ..
-rw-rw-r--.  1 root gigadb 7346 Sep 30 05:51 sync_dropbox.log
[centos@ip-10-99-0-22 ~]$ cat /var/log/gigadb/sync_dropbox.log 
...
2024/09/30 05:50:35 INFO  : user27/Genome/Sample_information.csv: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/busco_full_table.csv: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/Venny_FigS3.zip: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/Binodoxys_communis_contig_level.fa: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/busco_short_summary.txt: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/SpeciesTreeAlignment.fa: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/missing_busco_list.csv: Copied (new)
2024/09/30 05:50:35 INFO  : user27/Genome/repeat.statistics.csv: Copied (new)
2024/09/30 05:50:36 INFO  : user27/Genome/repeatmasker.gff: Copied (new)
2024/09/30 05:50:36 INFO  : user27/Genome/trf.gff: Copied (new)
2024/09/30 05:50:36 INFO  : user4/brassicaceae_NCBI/Trees.tar.gz: Copied (new)
2024/09/30 05:50:37 INFO  : user27/Genome/single_copy.pep.phy: Copied (new)
2024/09/30 05:50:37 INFO  : user4/brassicaceae_NCBI/its.tar.gz: Copied (new)
2024/09/30 05:50:37 INFO  : user4/brassicaceae_NCBI/amas.tar.gz: Copied (new)
2024/09/30 05:50:37 INFO  : user4/brassicaceae_NCBI/its_id_sp.csv: Copied (new)
2024/09/30 05:50:37 INFO  : user4/brassicaceae_NCBI/matk_id_spp.csv: Copied (new)
2024/09/30 05:50:39 INFO  : user27/Genome/single_copy.cds.phy: Copied (new)
2024/09/30 05:50:39 INFO  : user4/brassicaceae_NCBI/rbcl_id_spp.csv: Copied (new)
2024/09/30 05:50:41 INFO  : user4/brassicaceae_NCBI/trn.tar.gz: Copied (new)
2024/09/30 05:50:41 INFO  : user4/brassicaceae_NCBI/trn_id_spp.csv: Copied (new)
2024/09/30 05:50:41 INFO  : user27/Metabolite/Fig7DEF_Rawdata/Sample_information1.csv: Copied (new)
2024/09/30 05:50:42 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results1.csv: Copied (new)
2024/09/30 05:50:43 INFO  : user27/Metabolite/Fig7DEF_Rawdata/TOTAL_baseCompare.csv: Copied (new)
2024/09/30 05:50:43 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results2.csv: Copied (new)
2024/09/30 05:50:43 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/Sample_information3.csv: Copied (new)
2024/09/30 05:50:44 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Integral_correction_diagram_N.png: Copied (new)
2024/09/30 05:50:44 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Integral_correction_diagram_P.png: Copied (new)
2024/09/30 05:50:45 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/MRM_detection_of_multimodal_maps_N.png: Copied (new)
2024/09/30 05:50:45 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/MRM_detection_of_multimodal_maps_P.png: Copied (new)
2024/09/30 05:50:46 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_TIC_N.png: Copied (new)
2024/09/30 05:50:46 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_TIC_P.png: Copied (new)
2024/09/30 05:50:47 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_tic_overlap_N.png: Copied (new)
2024/09/30 05:50:47 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_tic_overlap_P.png: Copied (new)
2024/09/30 05:50:47 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Raw_metabolomics_data.csv: Copied (new)
2024/09/30 05:50:48 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Sample_information2.csv: Copied (new)
2024/09/30 05:50:49 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/all_cor.csv: Copied (new)
2024/09/30 05:50:49 INFO  : user0/change.log: Copied (new)
2024/09/30 05:50:49 INFO  : user0/some/directory/foobar.ext: Copied (new)
2024/09/30 05:50:54 INFO  : user4/brassicaceae_NCBI/rbcl.tar.gz: Multi-thread Copied (new)
2024/09/30 05:51:07 INFO  : user4/brassicaceae_NCBI/outputs.tar.gz: Multi-thread Copied (new)
2024/09/30 05:51:09 INFO  : user4/brassicaceae_NCBI/matk.tar.gz: Multi-thread Copied (new)
2024/09/30 05:51:10 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:51:10 INFO  : Successfully sync dropbox from production-staging to alt staging

In staging as a lily user

% cd ops/infrastructure/envs/staging
% ansible-playbook -i ../../inventories users_playbook.yml -e "newuser=lily" -e "credentials_csv_path=~/path/to/credentials.csv" -e "gigadb_env=staging" 
% chmod 500 output/privkeys-$bastion-ip/lily
% ssh -i output/privkeys-3.36.204.163/lily lily@$bastion-ip
Activate the web console with: systemctl enable --now cockpit.socket

[lily@ip-10-99-0-22 ~]$ ls -al /share/dropbox/
total 12
drwxrwxr-x. 6 centos centos   82 Sep 30 05:50 .
drwxr-xr-x. 4 centos centos   35 Sep 30 03:54 ..
-rw-rw-r--. 1 centos centos  166 Apr 25 16:55 rija_test.txt
drwxrwxr-x. 3 centos centos   36 Sep 30 05:50 user0
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user109
drwxrwxr-x. 4 centos centos  167 Sep 30 05:50 user27
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user4
[lily@ip-10-99-0-22 ~]$ rm -r /share/dropbox/user27
[lily@ip-10-99-0-22 ~]$ ls -al /share/dropbox/
total 12
drwxrwxr-x. 5 centos centos   68 Sep 30 05:58 .
drwxr-xr-x. 4 centos centos   35 Sep 30 03:54 ..
-rw-rw-r--. 1 centos centos  166 Apr 25 16:55 rija_test.txt
drwxrwxr-x. 3 centos centos   36 Sep 30 05:50 user0
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user109
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user4
[lily@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 12
drwxr-xr-x.  2 root gigadb   30 Sep 30 04:23 .
drwxr-xr-x. 10 root root   4096 Sep 30 04:16 ..
-rw-rw-r--.  1 root gigadb 7346 Sep 30 05:51 sync_dropbox.log
[lily@ip-10-99-0-22 ~]$ /usr/local/bin/sync_dropbox --apply
2024/09/30 05:59:16 INFO : Start sync dropbox from production-staging to alt staging
2024/09/30 05:59:28 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:59:28 INFO  : Successfully sync dropbox from production-staging to alt staging
[lily@ip-10-99-0-22 ~]$ ls -al /share/dropbox/
total 12
drwxrwxr-x. 6 centos centos   82 Sep 30 05:59 .
drwxr-xr-x. 4 centos centos   35 Sep 30 03:54 ..
-rw-rw-r--. 1 centos centos  166 Apr 25 16:55 rija_test.txt
drwxrwxr-x. 3 centos centos   36 Sep 30 05:50 user0
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user109
drwxrwxr-x. 4 lily   lily    167 Sep 30 05:59 user27
drwxrwxr-x. 3 centos centos 4096 Sep 30 05:50 user4
[lily@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 20
drwxr-xr-x.  2 root gigadb    30 Sep 30 04:23 .
drwxr-xr-x. 10 root root    4096 Sep 30 04:16 ..
-rw-rw-r--.  1 root gigadb 12410 Sep 30 05:59 sync_dropbox.log
...
2024/09/30 05:59:16 INFO : Start sync dropbox from production-staging to alt staging
2024/09/30 05:59:19 INFO  : user27/102224.filesizes: Copied (new)
2024/09/30 05:59:19 INFO  : user27/.dotfiles.txt: Copied (new)
2024/09/30 05:59:19 INFO  : user27/102224.filesizes.bk: Copied (new)
2024/09/30 05:59:19 INFO  : user27/102224.md5.bk: Copied (new)
2024/09/30 05:59:19 INFO  : user27/102224.md5: Copied (new)
2024/09/30 05:59:19 INFO  : user27/readme.txt: Copied (new)
2024/09/30 05:59:20 INFO  : user27/Genome/Binodoxys_communis_chr_pep.fa: Copied (new)
2024/09/30 05:59:20 INFO  : user27/Genome/Binodoxys_communis_chr.gff: Copied (new)
2024/09/30 05:59:20 INFO  : user27/Genome/Expasion_KEGG_Fig2BC.csv: Copied (new)
2024/09/30 05:59:20 INFO  : user27/Genome/Binodoxys_communis_chr_cds.fa: Copied (new)
2024/09/30 05:59:20 INFO  : user27/Genome/Extraction_KEGG_Fig2DE.csv: Copied (new)
2024/09/30 05:59:21 INFO  : user27/Genome/Genome.md5: Copied (new)
2024/09/30 05:59:21 INFO  : user27/Genome/Gene_annotation.csv: Copied (new)
2024/09/30 05:59:21 INFO  : user27/Genome/Phylogenetic_tree_Fig2A.newick: Copied (new)
2024/09/30 05:59:21 INFO  : user27/Genome/Sample_information.csv: Copied (new)
2024/09/30 05:59:22 INFO  : user27/Genome/Venny_FigS3.zip: Copied (new)
2024/09/30 05:59:22 INFO  : user27/Genome/SpeciesTreeAlignment.fa: Copied (new)
2024/09/30 05:59:22 INFO  : user27/Genome/busco_full_table.csv: Copied (new)
2024/09/30 05:59:23 INFO  : user27/Genome/busco_short_summary.txt: Copied (new)
2024/09/30 05:59:23 INFO  : user27/Genome/missing_busco_list.csv: Copied (new)
2024/09/30 05:59:23 INFO  : user27/Genome/repeat.statistics.csv: Copied (new)
2024/09/30 05:59:23 INFO  : user27/Genome/repeatmasker.gff: Copied (new)
2024/09/30 05:59:24 INFO  : user27/Genome/Binodoxys_communis_chrom_level.fa: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Genome/single_copy.pep.phy: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Metabolite/Fig4BCDEF_Lipid_changes_in_adult_stage.csv: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Genome/Binodoxys_communis_contig_level.fa: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Genome/trf.gff: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Metabolite/Fig6BC_Lipid_changes_in_the_larval_stage.csv: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Metabolite/Fig7ABC_Biological data.csv: Copied (new)
2024/09/30 05:59:25 INFO  : user27/Genome/single_copy.cds.phy: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Fig7DEF_TAG_DAG_and_FA_of_cotton_aphids_after_parasitization.csv: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Fig9BD_Flaw rates of isotope-labeled metabolites.csv: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/FigS5_Quantitative_validation_of_metabolites.csv: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Metabolite.md5: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Integral_correction_diagram_N.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Integral_correction_diagram_P.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/MRM_detection_of_multimodal_maps_N.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/MRM_detection_of_multimodal_maps_P.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_TIC_N.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_tic_overlap_N.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_TIC_P.png: Copied (new)
2024/09/30 05:59:26 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/QC_MS_tic_overlap_P.png: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Raw_metabolomics_data.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Sample_information2.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/all_cor.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Fig7DEF_Rawdata/Sample_information1.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Fig7DEF_Rawdata/TOTAL_baseCompare.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results1.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results2.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/Sample_information3.csv: Copied (new)
2024/09/30 05:59:28 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:59:28 INFO  : Successfully sync dropbox from production-staging to alt staging
[lily@ip-10-99-0-22 ~]$

In live as a centos user

% ssh -i path/to/live/pem centos@$live-bastion-ip
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu Oct 24 05:29:34 2024 from 54.180.33.208
[centos@ip-10-99-0-185 ~]$ /usr/local/bin/sync_dropbox --dry-run
2024/10/24 05:29:54 INFO : Start sync dropbox from production-live to alt live
2024/10/24 05:30:45 INFO  : Executed: /usr/local/bin/rclone sync production-live:/share/dropbox/ /share/dropbox --dry-run --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/24 05:30:45 INFO  : Successfully sync dropbox from production-live to alt live
[centos@ip-10-99-0-185 ~]$ 
[centos@ip-10-99-0-185 ~]$ /usr/local/bin/sync_dropbox --apply
2024/10/24 05:29:54 INFO : Start sync dropbox from production-live to alt live
2024/10/24 05:30:45 INFO  : Executed: /usr/local/bin/rclone sync production-live:/share/dropbox/ /share/dropbox --dry-run --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/24 05:30:45 INFO  : Successfully sync dropbox from production-live to alt live

[centos@ip-10-99-0-185 ~]$  ls -al /var/log/gigadb/sync_dropbox.log 
total 97796
drwxrwxrwx.  2 centos centos       30 Oct 23 15:11 .
drwxr-xr-x. 11 root   root       4096 Oct 23 15:12 ..
-rw-rw-r--.  1 root   gigadb 70335364 Oct 24 05:30 sync_dropbox.log
[centos@ip-10-99-0-185 ~]$ wc -l /var/log/gigadb/sync_dropbox.log 
374089 /var/log/gigadb/sync_dropbox.log
[centos@ip-10-99-0-185 ~]$ less var/log/gigadb/sync_dropbox.log 
2024/10/24 05:22:06 INFO : Start sync dropbox from  to alt live
2024/10/24 05:22:07 NOTICE: cngb_user: Skipped copy as --dry-run is set (size 23)
2024/10/24 05:22:07 NOTICE: nohup.out: Skipped copy as --dry-run is set (size 200.488Ki)
2024/10/24 05:22:07 NOTICE: some.file: Skipped copy as --dry-run is set (size 30)
2024/10/24 05:22:07 NOTICE: uploaded_with_sftp.txt: Skipped copy as --dry-run is set (size 168)
2024/10/24 05:22:07 NOTICE: user116.orig/Function.txt: Skipped copy as --dry-run is set (size 10.790Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.addgene.addUniprot.gff: Skipped copy as --dry-run is set (size 44.357Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.cds: Skipped copy as --dry-run is set (size 34.069Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.pep: Skipped copy as --dry-run is set (size 12.116Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/RAxML_bestTree.snail.PcPm.GQ10.mind2.NoRelate.gt5maf1.biallelic.NoSCFS.pruned.min4.nwk: Skipped copy as --dry-run is set (size 7.210Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/all.repeat.gff: Skipped copy as --dry-run is set (size 150.854Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/allfile.md5sum: Skipped copy as --dry-run is set (size 1.210Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/differentStress.TMM.EXPR.matrix: Skipped copy as --dry-run is set (size 2.293Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/full_table.tsv: Skipped copy as --dry-run is set (size 14.393Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/missing_busco_list.tsv: Skipped copy as --dry-run is set (size 166)
2024/10/24 05:22:07 NOTICE: user116.orig/pcan.chr.20190806.fa: Skipped copy as --dry-run is set (size 430.149Mi)
2024/10/24 05:22:07 NOTICE: user116.orig/rRNA.gff: Skipped copy as --dry-run is set (size 11.354Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/nohup.out: Skipped copy as --dry-run is set (size 2.122Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/readme.txt: Skipped copy as --dry-run is set (size 3.288Ki)
2024/10/24 05:22:07 NOTICE: user116.orig/sample_information.txt: Skipped copy as --dry-run is set (size 39.711Ki)
........
[centos@ip-10-99-0-185 ~]$ tail /var/log/gigadb/sync_dropbox.log 
2024/10/24 05:30:44 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z98261.1.fasta: Skipped copy as --dry-run is set (size 467)
2024/10/24 05:30:44 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z98262.1.fasta: Skipped copy as --dry-run is set (size 3.021Ki)
2024/10/24 05:30:44 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z99268.2.fasta: Skipped copy as --dry-run is set (size 2.968Ki)
2024/10/24 05:30:44 NOTICE: 
Transferred:        8.789 TiB / 8.789 TiB, 100%, 81.082 GiB/s, ETA 0s
Transferred:       187034 / 187034, 100%
Elapsed time:        50.1s

2024/10/24 05:30:45 INFO  : Executed: /usr/local/bin/rclone sync production-live:/share/dropbox/ /share/dropbox --dry-run --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/24 05:30:45 INFO  : Successfully sync dropbox from production-live to alt live

In live as a lily user

% ansible-playbook -i ../../inventories users_playbook.yml -e "newuser=lily" -e "credentials_csv_path=~/path/to/credentials.csv" -e "gigadb_env=live" 
% ssh -i output/privkeys-$live-bastion-ip/lily lily@$live-bastion-ip

Activate the web console with: systemctl enable --now cockpit.socket
[lily@ip-10-99-0-185 ~]$ /usr/local/bin/sync_dropbox --dry-run
2024/10/24 05:50:47 INFO : Start sync dropbox from production-live to alt live
2024/10/24 05:51:50 INFO  : Executed: /usr/local/bin/rclone sync production-live:/share/dropbox/ /share/dropbox --dry-run --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/24 05:51:50 INFO  : Successfully sync dropbox from production-live to alt live
[lily@ip-10-99-0-185 ~]$ ls -al /var/log/gigadb/
total 166980
drwxrwxrwx.  2 centos centos        30 Oct 23 15:11 .
drwxr-xr-x. 11 root   root        4096 Oct 23 15:12 ..
-rw-rw-r--.  1 root   gigadb 105503276 Oct 24 05:51 sync_dropbox.log
[lily@ip-10-99-0-185 ~]$ cat 
2024/10/24 05:50:47 INFO : Start sync dropbox from production-live to alt live
2024/10/24 05:50:48 NOTICE: cngb_user: Skipped copy as --dry-run is set (size 23)
2024/10/24 05:50:48 NOTICE: some.file: Skipped copy as --dry-run is set (size 30)
2024/10/24 05:50:48 NOTICE: nohup.out: Skipped copy as --dry-run is set (size 200.488Ki)
2024/10/24 05:50:48 NOTICE: uploaded_with_sftp.txt: Skipped copy as --dry-run is set (size 168)
2024/10/24 05:50:48 NOTICE: user116.orig/Function.txt: Skipped copy as --dry-run is set (size 10.790Mi)
2024/10/24 05:50:48 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.addgene.addUniprot.gff: Skipped copy as --dry-run is set (size 44.357Mi)
2024/10/24 05:50:48 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.cds: Skipped copy as --dry-run is set (size 34.069Mi)
2024/10/24 05:50:48 NOTICE: user116.orig/Pomacea_canaliculata.coding.gene.V1.20190806.pep: Skipped copy as --dry-run is set (size 12.116Mi)
2024/10/24 05:50:48 NOTICE: user116.orig/RAxML_bestTree.snail.PcPm.GQ10.mind2.NoRelate.gt5maf1.biallelic.NoSCFS.pruned.min4.nwk: Skipped copy as --dry-run is set (size 7.210Ki)
....
2024/10/24 05:51:50 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z98262.1.fasta: Skipped copy as --dry-run is set (size 3.021Ki)
2024/10/24 05:51:50 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z95622.1.fasta: Skipped copy as --dry-run is set (size 2.314Ki)
2024/10/24 05:51:50 NOTICE: user78.orig/user78/MOBFinder/4.model_training_and_optimization/1.plasmid_genomes_for_different_MOB_types/non-mob/Z99268.2.fasta: Skipped copy as --dry-run is set (size 2.968Ki)
2024/10/24 05:51:50 NOTICE: 
Transferred:        8.794 TiB / 8.794 TiB, 100%, 41.572 GiB/s, ETA 0s
Transferred:       187035 / 187035, 100%
Elapsed time:       1m2.8s

2024/10/24 05:51:50 INFO  : Executed: /usr/local/bin/rclone sync production-live:/share/dropbox/ /share/dropbox --dry-run --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/24 05:51:50 INFO  : Successfully sync dropbox from production-live to alt live

How have functionalities been implemented?

Describe how the new functionalities have been implemented by the
changed code at a high level

The script sync_dropbox uses rclone sync to make sure the efs mount point /share/dropbox in alt upstream always stays as the same as the one found in the upstream. Rclone sync will only the source to the destination, changing the destination only.

The pre-defined rclone config for the sync is located at gigadb/app/tools/sync-dropbox/config/rclone.conf in the local dev encironment, and at /etc/sync_dropbox/rclone.conf in the production environments, which specifies the endpoints for production staging efs and the production live efs.

To access these endpoints through a secure connection, a ssh private key id-rsa-aws-hk-gigadb.pem needs to be pulled from the gitlab cnhk-infra variable page to the ~/.ssh/ directory, this private key is supposedly for the gigadb user to provision the upstream ec2 servers. By sharing this key, the efs in the upstream can be accessed from the alt upstream securely, otherwise the users in the alt upstream needs to copy the public key to the upstream server using this key id-rsa-aws-hk-gigadb.pem if they want to use their own ssh key pairs, which seems adding an extra layer of complication.

In order to get rid of the Permission denied error when the /usr/local/bin/sync_dropbox trying to output log in production environments, a gigadb group is created and make the /var/log/gigadb directory be r+w to the gigadb group, then add centos and other users, eg. lily to gigadb group, as a result, /var/log/gigadb can be r+w by both centos and other users.

Any issues with implementation?

None

Any changes to automated tests?

None

Any changes to documentation?

None

Any technical debt repayment?

logrotate has been implemented in this PR, the main config for the logrotate is at /etc/logrorate.d/gigadb,

/var/log/gigadb/*.log {  
            daily  # Rotate logs daily
            missingok             # Ignore missing log files
            rotate 7              # Keep 7 days of backlogs
            compress              # Compress rotated logs
            delaycompress         # Delay compression until the next rotation
            notifempty            # Do not rotate empty log files
            create 0640 root gigadb   # Create new log files with specified permissions and ownership
            dateformat -%Y%m%d  # Suffix of the rotated log file
          }

which indicates every log inside /var/log/gigadb will be rotated daily 7 times, each rotated log will be compressed with YYYYMMDD in the file name suffix, as you can see below:

% ssh -i ~/.ssh/id-rsa-aws-seoul-ken.pem [email protected]
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Fri Sep 27 03:59:58 2024 from 223.197.187.121
[centos@ip-10-99-0-226 ~]$ ls -al /var/log/gigadb/
total 40
drwxr-xr-x.  2 root   gigadb   171 Sep 29 03:13 .
drwxr-xr-x. 10 root   root    4096 Sep 29 03:13 ..
-rw-rw-r--.  1 root gigadb 10450 Sep 30 01:00 sync_dropbox.log
-rw-rw-r--.  1 root gigadb    26 Sep 26 07:43 sync_dropbox.log-20240926.gz
-rw-rw-r--.  1 root   gigadb  3134 Sep 27 03:11 sync_dropbox.log-20240927.gz
-rw-rw-r--.  1 root gigadb   612 Sep 28 03:00 sync_dropbox.log-20240928.gz
-rw-rw-r--.  1 root gigadb 11400 Sep 29 03:00 sync_dropbox.log-20240929.gz

The logrotate requires the log directory and the log file to have a secure permission, so, the log can be read and written by root user and any user in the group gigadb.

For the purpose of testing, after logging into your staging, you can force rotate the log to check if it works correctly:

% ssh -i path/to/staging/pem centos@$staging-bastion-ip
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Mon Sep 30 05:57:11 2024 from 3.36.204.163
[centos@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 20
drwxr-xr-x.  2 root gigadb    30 Sep 30 04:23 .
drwxr-xr-x. 10 root root    4096 Sep 30 04:16 ..
-rw-rw-r--.  1 root gigadb 12410 Sep 30 05:59 sync_dropbox.log
[centos@ip-10-99-0-22 ~]$ tail /var/log/gigadb/sync_dropbox.log
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Raw_metabolomics_data.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/Sample_information2.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Lipidomics_Rawdata_of_parasitic_wasps/all_cor.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Fig7DEF_Rawdata/Sample_information1.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Fig7DEF_Rawdata/TOTAL_baseCompare.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results1.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results2.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/Sample_information3.csv: Copied (new)
2024/09/30 05:59:28 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:59:28 INFO  : Successfully sync dropbox from production-staging to alt staging
[centos@ip-10-99-0-22 ~]$  /usr/local/bin/sync_dropbox --apply
[centos@ip-10-99-0-22 ~]$ tail /var/log/gigadb/sync_dropbox.log
2024/09/30 05:59:27 INFO  : user27/Metabolite/Fig7DEF_Rawdata/TOTAL_baseCompare.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results1.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/C13_flux_results2.csv: Copied (new)
2024/09/30 05:59:27 INFO  : user27/Metabolite/Rawdata_of_Isotope_test/Sample_information3.csv: Copied (new)
2024/09/30 05:59:28 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 05:59:28 INFO  : Successfully sync dropbox from production-staging to alt staging
2024/09/30 06:04:29 INFO : Start sync dropbox from production-staging to alt staging
2024/09/30 06:04:31 INFO  : There was nothing to transfer
2024/09/30 06:04:31 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/09/30 06:04:31 INFO  : Successfully sync dropbox from production-staging to alt staging
[centos@ip-10-99-0-22 ~]$ sudo /usr/sbin/logrotate -f /etc/logrotate.conf
[centos@ip-10-99-0-22 ~]$ ls /var/log/gigadb/
sync_dropbox.log  sync_dropbox.log-20240930
[centos@ip-10-99-0-22 ~]$ ls -al /var/log/gigadb/
total 20
drwxr-xr-x.  2 root gigadb    63 Sep 30 06:05 .
drwxr-xr-x. 10 root root    4096 Sep 30 06:05 ..
-rw-rw----.  1 root gigadb     0 Sep 30 06:24 sync_dropbox.log
-rw-rw-r--.  1 root gigadb 12885 Sep 30 06:04 sync_dropbox.log-20240930
# The old sync_dropbox.log has been rotated to sync_dropbox.log-20240930
# The current sync_dropbox.log is empty, all log output will be saved to here
# If the logrotation was done in daily manner, the log file with name `sync_dropbox.log-20240930.gz` will be created as because the option `delaycompress` is included in the logrorate config which dos not immediately compress the rotated log file, but only in the next rotation cycle

Additionally, I have made my staging bastion running the script for more that 7 days to test for cron job and also the logrotation, here is the results:

[centos@ip-10-99-0-208 ~]$ ls -al /var/log/gigadb/
total 20
drwxr-xr-x.  2 root gigadb  171 Oct 28 03:07 .
drwxr-xr-x. 11 root root   4096 Oct 27 03:37 ..
-rw-rw----.  1 root gigadb    0 Oct 28 03:07 sync_dropbox.log
-rw-rw-r--.  1 root gigadb  328 Oct 25 03:26 sync_dropbox.log-20241025.gz
-rw-rw----.  1 root gigadb  246 Oct 25 11:00 sync_dropbox.log-20241026.gz
-rw-rw----.  1 root gigadb  250 Oct 26 11:00 sync_dropbox.log-20241027.gz
-rw-rw----.  1 root gigadb  475 Oct 27 11:00 sync_dropbox.log-20241028
[centos@ip-10-99-0-208 ~]$ zcat /var/log/gigadb/sync_dropbox.log-20241027.gz
2024/10/26 11:00:02 INFO : Start sync dropbox from production-staging to alt staging
2024/10/26 11:00:04 INFO  : There was nothing to transfer
2024/10/26 11:00:05 INFO  : Executed: /usr/local/bin/rclone sync production-staging:/share/dropbox/ /share/dropbox --config /etc/sync_dropbox/rclone.conf --log-file /var/log/gigadb/sync_dropbox.log --log-level INFO --stats-log-level DEBUG
2024/10/26 11:00:05 INFO  : Successfully sync dropbox from production-staging to alt staging
# perform checking on 4/11/2024
[centos@ip-10-99-0-208 ~]$ ls -al /var/log/gigadb/
total 36
drwxr-xr-x.  2 root gigadb 4096 Nov  4 04:20 .
drwxr-xr-x. 11 root root   4096 Nov  3 03:47 ..
-rw-rw----.  1 root gigadb    0 Nov  4 04:20 sync_dropbox.log
-rw-rw----.  1 root gigadb  250 Oct 28 11:01 sync_dropbox.log-20241029.gz
-rw-rw----.  1 root gigadb  250 Oct 29 11:00 sync_dropbox.log-20241030.gz
-rw-rw----.  1 root gigadb  249 Oct 30 11:00 sync_dropbox.log-20241031.gz
-rw-rw----.  1 root gigadb  246 Oct 31 11:00 sync_dropbox.log-20241101.gz
-rw-rw----.  1 root gigadb  246 Nov  1 11:00 sync_dropbox.log-20241102.gz
-rw-rw----.  1 root gigadb  246 Nov  2 11:00 sync_dropbox.log-20241103.gz
-rw-rw----.  1 root gigadb  475 Nov  3 11:00 sync_dropbox.log-20241104
[centos@ip-10-99-0-208 ~]$

As a result, it is suggested the all the script outputs should be stored in the /var/log/gigadb, so the log files will be better managed automatically.

Any improvements to CI/CD pipeline?

None

This reverts commit 6fd9005.

This was referenced Sep 10, 2024

Generate rclone configuration for source of rclone sync #2005

Closed

Create a shell script wrapping rclone sync #2004

Closed

Write bats tests for correctness of rclone configuration and shell scripts #2007

Closed

kencho51 force-pushed the make-alt-efs-sync-with-live branch from 1adfa18 to b189e39 Compare September 19, 2024 01:42

kencho51 mentioned this pull request Sep 19, 2024

Update bastion playbook to add a new cronjob to execute the shell script and to create the rclone configuration #2006

Closed

kencho51 force-pushed the make-alt-efs-sync-with-live branch 2 times, most recently from 184014b to 62391b5 Compare September 27, 2024 03:01

kencho51 force-pushed the make-alt-efs-sync-with-live branch from a07939a to 6d6ddaf Compare October 23, 2024 04:39

kencho51 added 22 commits October 24, 2024 14:13

feat: create wrapper script

9d28f55

feat: create bats test

d5a577f

chore: create gitignore

87eb288

refactor: make teardown work

38729c8

feat: bring in config steps

041c707

refactor: update the log dir name to logs

70aa24c

feat: make bats test to check the log

b131322

refactor: update gitignore

6eaac03

refactor: make the script more versitle

e14356d

refactor: update the bats tests

98aaa7b

refactor: create data dir in dev

a17c771

refactor: get the upstream private key

f9110e3

refactor: read the private key from ssh dir

d8f8ea4

feat: include misc url in ansible

edc9490

feat: make sync dropbox tool available in bastion

d46e84e

refactr: make the cnhk url working

7331f87

refactor: make centos user to execute the script

e0db781

feat: make user can execute the script

68f2324

refactor: make rds skip snapshot when destroy

e13e96e

feat: create cronjob for sync dropbox

c25466e

refactor: create config dir

615d834

refactor: specify the exact path of the rclone executable in productions

e3f9ccd

kencho51 added 10 commits October 24, 2024 14:14

refactor: remove the date suffix from the log file

54c7346

refactor: create tasks for logrotate implementation

1bf8435

refactor: make the created log file rw to the gigadb group

0cf9b05

doc: update change log

f4f444b

refactor: remove extra dist file

1b7e808

refactor: add back rclone config file

79e4190

refactor: make the log dir name consistent

418572e

Revert "refactor: make rds skip snapshot when destroy"

62709ac

This reverts commit 6fd9005.

refactor: update the test after changing the log dir name

5784aa8

refactor: update the config path

b208b5d

kencho51 force-pushed the make-alt-efs-sync-with-live branch from 68ef8e1 to b208b5d Compare October 24, 2024 05:15

refactor: make sure get the rclone profile name for the upsteam live

7ca6611

kencho51 force-pushed the make-alt-efs-sync-with-live branch from 400783b to 7ca6611 Compare October 24, 2024 05:59

refactor: remove extra task for change the dir permission

9e6ffc4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make alt efs sync with live #2025

Make alt efs sync with live #2025

kencho51 commented Sep 5, 2024 •

edited

Loading

Make alt efs sync with live #2025

Are you sure you want to change the base?

Make alt efs sync with live #2025

Conversation

kencho51 commented Sep 5, 2024 • edited Loading

Pull request for issue: #2000

How to test?

In local dev

Pre-requisites

In staging as a centos user

In staging as a lily user

In live as a centos user

In live as a lily user

How have functionalities been implemented?

Any issues with implementation?

Any changes to automated tests?

Any changes to documentation?

Any technical debt repayment?

Any improvements to CI/CD pipeline?

kencho51 commented Sep 5, 2024 •

edited

Loading