Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Content onboarding scripts #26

Merged
merged 6 commits into from
Oct 22, 2019
Merged

Content onboarding scripts #26

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fc8b836
Add content_onboarding_scripts
mpoelchau Sep 25, 2019
9f75e6e
Make additional scripts executable
mpoelchau Sep 25, 2019
f550f93
Add additional scripts to PATH
mpoelchau Sep 25, 2019
0770b97
fixed rubocop validation issues
childers Oct 17, 2019
3959e00
fixing the new export command.
childers Oct 17, 2019
1f9fb4d
Merge branch 'kenttools' into content_onboarding_scripts
childers Oct 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
355 changes: 183 additions & 172 deletions Vagrantfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,177 +81,188 @@ Vagrant.configure('2') do |config|
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
# documentation for more information about their specific syntax and use.
config.vm.provision 'shell', inline: <<~SHELL
# General update of system
sudo yum update
sudo yum upgrade

su vagrant

# Use EPEL: https://fedoraproject.org/wiki/EPEL
wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -ivh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm

# Install GUI desktop
sudo yum groupinstall -y "X Window System" "Desktop"
sudo yum install -y gnome-core xfce4 xorg-x11-fonts
sudo echo "id:5:initdefault:" > /etc/inittab

# Fix fonts problem in terminal
# https://forums.anandtech.com/threads/fonts-screwed-up-in-centos-6-terminal.2186468/
sudo yum -y install terminus-fonts terminus-fonts-console

# Install newer git version
sudo yum install -y curl-devel expat-devel libcurl-devel perl-devel asciidoc \
xmlto xz zlib-devel zlib-static gettext
curl -kOL https://www.kernel.org/pub/software/scm/git/git-2.18.0.tar.xz
tar xvf git-2.18.0.tar.xz
cd git-2.18.0
make configure
./configure --prefix=/usr
make all
sudo make install install-doc install-html
cd /home/vagrant
rm -rf git-2.18.0 git-2.18.0.tar.xz

# Install python2.7
cd /home/vagrant
sudo yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel
sudo yum -y install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel
wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz
tar -xf Python-2.7.15.tar.xz
cd Python-2.7.15
./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make
sudo make altinstall
rm /home/vagrant/Python-2.7.15.tar.xz
rm -rf /home/vagrant/Python-2.7.15

# Install pip
cd /home/vagrant
export PATH="/usr/local/bin:$PATH"
wget https://bootstrap.pypa.io/ez_setup.py
sudo /usr/local/bin/python2.7 ez_setup.py
rm ez_setup.py
wget https://bootstrap.pypa.io/get-pip.py
sudo /usr/local/bin/python2.7 get-pip.py
rm get-pip.py

# Install virtualenv
sudo /usr/local/bin/pip2.7 install virtualenv

# Install RabbitMQ server (used by genomics-workspace)
sudo yum -y install erlang # prerequisite of rabbitMQ
sudo yum -y install rabbitmq-server
sudo chkconfig rabbitmq-server on # to start the daemon by default when system boots run:
sudo /sbin/service rabbitmq-server start

# Install and activate memcached (used by genomics-workspace)
sudo yum -y install memcached
sudo chkconfig memcached on # set to start at boot time

# Install the PostgreSQL (9.5)
echo 'exclude=postgresql*' | sudo tee -a /etc/yum.repos.d/CentOS-Base.repo # add line to yum repository
sudo yum -y install http://yum.postgresql.org/9.5/redhat/rhel-6-x86_64/pgdg-centos95-9.5-3.noarch.rpm # install the PostgreSQL Global Development Group (PGDG) RPM file:
sudo yum -y install postgresql95-server postgresql95-contrib postgresql95-devel
sudo service postgresql-9.5 initdb # initialize (uses default data directory: /var/lib/pgsql)
sudo chkconfig postgresql-9.5 on # make PostgreSQL startup at boot:
sudo sed '82,84s/ident/md5/g' /var/lib/pgsql/9.5/data/pg_hba.conf
export PATH=/usr/pgsql-9.5/bin:$PATH
sudo service postgresql-9.5 restart

# Install samtools
cd /home/vagrant
sudo yum install --enablerepo=epel -y ncurses-devel zlib-devel bzip2-devel xz-devel
wget https://github.com/samtools/samtools/releases/download/1.9/samtools-1.9.tar.bz2
tar xvjf samtools-1.9.tar.bz2
cd samtools-1.9
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
cd /home/vagrant
rm -rf samtools-1.9.tar.bz2 samtools-1.9

# Install glibc 2.17 (dependency of wig2BigWig)
# https://gist.github.com/harv/f86690fcad94f655906ee9e37c85b174#gistcomment-2083385
SERVER=http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/
REPO64=epel-6-x86_64
VERSION=glibc-2.17-55.fc20
SERVER64=$SERVER/$REPO64/$VERSION
sudo rpm -Uvh --force --nodeps $SERVER64/glibc-2.17-55.el6.x86_64.rpm $SERVER64/glibc-common-2.17-55.el6.x86_64.rpm $SERVER64/glibc-devel-2.17-55.el6.x86_64.rpm $SERVER64/glibc-headers-2.17-55.el6.x86_64.rpm $SERVER64/glibc-static-2.17-55.el6.x86_64.rpm

# Install zlib 1.2.7 (dependency of wig2BigWig)
sudo rpm -Uvh --force --nodeps https://rpmfind.net/linux/centos/7.5.1804/os/x86_64/Packages/zlib-1.2.7-17.el7.x86_64.rpm

# Download wig2BigWig
sudo wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/wigToBigWig -O /usr/local/bin/wigToBigWig
sudo chmod +x /usr/local/bin/wigToBigWig
wget https://github.com/GMOD/jbrowse/tree/master/bin

# Install gff3tool
sudo /usr/local/bin/pip2.7 install gff3tool

# Install coordinates conversion
sudo /usr/local/bin/pip2.7 install git+https://github.com/NAL-i5K/coordinates_conversion.git

# Install genomics-workspace
cd /home/vagrant
git clone https://github.com/NAL-i5K/genomics-workspace
cd /home/vagrant/genomics-workspace
virtualenv -p python2.7 py2.7
source py2.7/bin/activate
pip install -r requirements.txt
deactivate

# Clone other tools used
cd /home/vagrant
git clone https://github.com/NAL-i5K/wiggle-tools
git clone https://github.com/NAL-i5K/bam_to_bigwig
git clone https://gitlab.com/i5k_Workspace/apollo2_data_build_scripts.git
git clone https://github.com/GMOD/jbrowse

# Create directories used
mkdir -p /app/data/blat
mkdir -p /app/data/other-species
mkdir -p /app/data/scratch
mkdir -p /app/data/working-files
mkdir -p /app/data/tools


# install the Kent tools for data processing

#install needed libraries

sudo yum install -y mysql
sudo yum install -y mysql-devel
sudo yum install -y libpng
sudo yum install -y libpng-devel
sudo yum install -y openssl
sudo yum install -y openssl-devel
sudo yum install -y libstdc++-devel

cd /home/vagrant/app/data/tools/
wget http://hgdownload.soe.ucsc.edu/admin/exe/userApps.archive/userApps.v326.src.tgz
tar -xvzf userApps.v326.src.tgz
sudo chown -R vagrant:vagrant userApps
cd userApps

make


# cd /home/vagrant/app/data/blat
# wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit
# cd /home/vagrant


# Fix the permission of the folders
sudo chown -R vagrant /app
sudo chown -R vagrant /home/vagrant/wiggle-tools
sudo chown -R vagrant /home/vagrant/genomics-workspace
sudo chown -R vagrant /home/vagrant/bam_to_bigwig
sudo chown -R vagrant /home/vagrant/apollo2_data_build_scripts
sudo chown -R vagrant /home/vagrant/jbrowse

reboot # reboot to load GUI
# General update of system
sudo yum update
sudo yum upgrade

su vagrant

# Use EPEL: https://fedoraproject.org/wiki/EPEL
wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -ivh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm

# Install GUI desktop
sudo yum groupinstall -y "X Window System" "Desktop"
sudo yum install -y gnome-core xfce4 xorg-x11-fonts
sudo echo "id:5:initdefault:" > /etc/inittab

# Fix fonts problem in terminal
# https://forums.anandtech.com/threads/fonts-screwed-up-in-centos-6-terminal.2186468/
sudo yum -y install terminus-fonts terminus-fonts-console

# Install newer git version
sudo yum install -y curl-devel expat-devel libcurl-devel perl-devel asciidoc \
xmlto xz zlib-devel zlib-static gettext
curl -kOL https://www.kernel.org/pub/software/scm/git/git-2.18.0.tar.xz
tar xvf git-2.18.0.tar.xz
cd git-2.18.0
make configure
./configure --prefix=/usr
make all
sudo make install install-doc install-html
cd /home/vagrant
rm -rf git-2.18.0 git-2.18.0.tar.xz

# Install python2.7
cd /home/vagrant
sudo yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel
sudo yum -y install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel
wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz
tar -xf Python-2.7.15.tar.xz
cd Python-2.7.15
./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make
sudo make altinstall
rm /home/vagrant/Python-2.7.15.tar.xz
rm -rf /home/vagrant/Python-2.7.15

# Install pip
cd /home/vagrant
export PATH="/usr/local/bin:$PATH"
wget https://bootstrap.pypa.io/ez_setup.py
sudo /usr/local/bin/python2.7 ez_setup.py
rm ez_setup.py
wget https://bootstrap.pypa.io/get-pip.py
sudo /usr/local/bin/python2.7 get-pip.py
rm get-pip.py

# Install virtualenv
sudo /usr/local/bin/pip2.7 install virtualenv

# Install RabbitMQ server (used by genomics-workspace)
sudo yum -y install erlang # prerequisite of rabbitMQ
sudo yum -y install rabbitmq-server
sudo chkconfig rabbitmq-server on # to start the daemon by default when system boots run:
sudo /sbin/service rabbitmq-server start

# Install and activate memcached (used by genomics-workspace)
sudo yum -y install memcached
sudo chkconfig memcached on # set to start at boot time

# Install the PostgreSQL (9.5)
echo 'exclude=postgresql*' | sudo tee -a /etc/yum.repos.d/CentOS-Base.repo # add line to yum repository
sudo yum -y install http://yum.postgresql.org/9.5/redhat/rhel-6-x86_64/pgdg-centos95-9.5-3.noarch.rpm # install the PostgreSQL Global Development Group (PGDG) RPM file:
sudo yum -y install postgresql95-server postgresql95-contrib postgresql95-devel
sudo service postgresql-9.5 initdb # initialize (uses default data directory: /var/lib/pgsql)
sudo chkconfig postgresql-9.5 on # make PostgreSQL startup at boot:
sudo sed '82,84s/ident/md5/g' /var/lib/pgsql/9.5/data/pg_hba.conf
export PATH=/usr/pgsql-9.5/bin:$PATH
sudo service postgresql-9.5 restart

# Install samtools
cd /home/vagrant
sudo yum install --enablerepo=epel -y ncurses-devel zlib-devel bzip2-devel xz-devel
wget https://github.com/samtools/samtools/releases/download/1.9/samtools-1.9.tar.bz2
tar xvjf samtools-1.9.tar.bz2
cd samtools-1.9
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
cd /home/vagrant
rm -rf samtools-1.9.tar.bz2 samtools-1.9

# Install glibc 2.17 (dependency of wig2BigWig)
# https://gist.github.com/harv/f86690fcad94f655906ee9e37c85b174#gistcomment-2083385
SERVER=http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/
REPO64=epel-6-x86_64
VERSION=glibc-2.17-55.fc20
SERVER64=$SERVER/$REPO64/$VERSION
sudo rpm -Uvh --force --nodeps $SERVER64/glibc-2.17-55.el6.x86_64.rpm $SERVER64/glibc-common-2.17-55.el6.x86_64.rpm $SERVER64/glibc-devel-2.17-55.el6.x86_64.rpm $SERVER64/glibc-headers-2.17-55.el6.x86_64.rpm $SERVER64/glibc-static-2.17-55.el6.x86_64.rpm

# Install zlib 1.2.7 (dependency of wig2BigWig)
sudo rpm -Uvh --force --nodeps https://rpmfind.net/linux/centos/7.5.1804/os/x86_64/Packages/zlib-1.2.7-17.el7.x86_64.rpm

# Download wig2BigWig
sudo wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/wigToBigWig -O /usr/local/bin/wigToBigWig
sudo chmod +x /usr/local/bin/wigToBigWig
wget https://github.com/GMOD/jbrowse/tree/master/bin

# Install gff3tool
sudo /usr/local/bin/pip2.7 install gff3tool

# Install coordinates conversion
sudo /usr/local/bin/pip2.7 install git+https://github.com/NAL-i5K/coordinates_conversion.git

# Install genomics-workspace
cd /home/vagrant
git clone https://github.com/NAL-i5K/genomics-workspace
cd /home/vagrant/genomics-workspace
virtualenv -p python2.7 py2.7
source py2.7/bin/activate
pip install -r requirements.txt
deactivate

# Clone other tools used
cd /home/vagrant
git clone https://github.com/NAL-i5K/wiggle-tools
git clone https://github.com/NAL-i5K/bam_to_bigwig
git clone https://github.com/NAL-i5K/content_onboarding_scripts
git clone https://github.com/GMOD/jbrowse

#Make other tools executable
sudo chmod +x /home/vagrant/wiggle-tools/GCcontent2bigwig.py
sudo chmod +x /home/vagrant/wiggle-tools/gap2bigwig.py
sudo chmod +x /home/vagrant/bam_to_bigwig/bam_to_bigwig.py
sudo chmod +x /home/vagrant/content_onboarding_scripts/bin/add_metadata_to_GC_gap_bigwig_tracks.pl
sudo chmod +x /home/vagrant/content_onboarding_scripts/bin/createOrganism.py
sudo chmod -R +x /home/vagrant/jbrowse/bin

# Create directories used
mkdir -p /app/data/blat
mkdir -p /app/data/other-species
mkdir -p /app/data/scratch
mkdir -p /app/data/working-files
mkdir -p /app/data/tools

#Add tools to PATH
export PATH=$PATH:/home/vagrant/wiggle-tools:/home/vagrant/bam_to_bigwig:/home/vagrant/content_onboarding_scripts/bin:/home/vagrant/jbrowse/bin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this to:
sed 's/PATH=/PATH=/home/vagrant/wiggle-tools:/home/vagrant/bam_to_bigwig:/home/vagrant/content_onboarding_scripts/bin:/home/vagrant/jbrowse/bin:/g' .bash_profile >.bash_profile



# install the Kent tools for data processing

#install needed libraries

sudo yum install -y mysql
sudo yum install -y mysql-devel
sudo yum install -y libpng
sudo yum install -y libpng-devel
sudo yum install -y openssl
sudo yum install -y openssl-devel
sudo yum install -y libstdc++-devel

cd /home/vagrant/app/data/tools/
wget http://hgdownload.soe.ucsc.edu/admin/exe/userApps.archive/userApps.v326.src.tgz
tar -xvzf userApps.v326.src.tgz
sudo chown -R vagrant:vagrant userApps
cd userApps

make


# cd /home/vagrant/app/data/blat
# wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit
# cd /home/vagrant


# Fix the permission of the folders
sudo chown -R vagrant /app
sudo chown -R vagrant /home/vagrant/wiggle-tools
sudo chown -R vagrant /home/vagrant/genomics-workspace
sudo chown -R vagrant /home/vagrant/bam_to_bigwig
sudo chown -R vagrant /home/vagrant/apollo2_data_build_scripts
sudo chown -R vagrant /home/vagrant/jbrowse

reboot # reboot to load GUI
SHELL
end