apt-get update does not run cleanly on a brand-new cluster #622
Comments
|
Debian ‘Squeeze' end-of-life was in 2016. I think you’ll have to upgrade
the instance to ‘Jessie’. There are some third party StarCluster AMIs with
Ubuntu 16.04.4 floating around if that’s helpful for you.
On August 8, 2018 at 9:31:22 AM, kynnjo ([email protected]) wrote:
Right after creating a new cluster, I log in to it with starcluster
sshmaster <name-of-cluster>, and immediately run the command apt-get update
(as the very first command after login). The output from this command
reveals several problems:
# apt-get update
Get:1 http://archive.cloudera.com squeeze-cdh3u5 Release.gpg [198 B]
Get:2 http://archive.cloudera.com squeeze-cdh3u5 Release [2,405 B]
Ign http://us-east-1.ec2.archive.ubuntu.com raring Release.gpg
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates Release.gpg
Ign http://us-east-1.ec2.archive.ubuntu.com raring Release
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates Release
Ign http://us-east-1.ec2.archive.ubuntu.com raring/main Sources/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring/universe Sources/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring/main amd64 Packages/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring/universe amd64
Packages/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/main
Sources/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe
Sources/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/main amd64
Packages/DiffIndex
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe
amd64 Packages/DiffIndex
Get:3 http://archive.cloudera.com squeeze-cdh3u5/contrib Sources [5,001 B]
Ign http://security.ubuntu.com raring-security Release.gpg
Get:4 http://archive.cloudera.com squeeze-cdh3u5/contrib amd64
Packages [14.0 kB]
Hit http://ppa.launchpad.net raring Release.gpg
Ign http://security.ubuntu.com raring-security Release
Ign http://us-east-1.ec2.archive.ubuntu.com raring/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com raring/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com raring/universe Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com raring/universe Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/main
Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe
Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe
Translation-en
Err http://us-east-1.ec2.archive.ubuntu.com raring/main Sources
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring/universe Sources
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring/main amd64 Packages
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring/universe amd64 Packages
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring-updates/main Sources
404 Not Found [IP: 54.152.129.43 80]
Ign http://security.ubuntu.com raring-security/main Sources/DiffIndex
Err http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe Sources
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring-updates/main amd64 Packages
404 Not Found [IP: 54.152.129.43 80]
Err http://us-east-1.ec2.archive.ubuntu.com raring-updates/universe
amd64 Packages
404 Not Found [IP: 54.152.129.43 80]
Hit http://ppa.launchpad.net raring Release.gpg
Ign http://security.ubuntu.com raring-security/universe Sources/DiffIndex
Ign http://security.ubuntu.com raring-security/main amd64 Packages/DiffIndex
Get:5 http://ppa.launchpad.net raring Release.gpg [316 B]
Ign http://security.ubuntu.com raring-security/universe amd64 Packages/DiffIndex
Hit http://ppa.launchpad.net raring Release
Hit http://ppa.launchpad.net raring Release
Get:6 http://ppa.launchpad.net raring Release [9,753 B]
Ign http://archive.cloudera.com squeeze-cdh3u5/contrib Translation-en_US
Hit http://ppa.launchpad.net raring/main Sources
Ign http://archive.cloudera.com squeeze-cdh3u5/contrib Translation-en
Hit http://ppa.launchpad.net raring/main amd64 Packages
Hit http://ppa.launchpad.net raring/main Sources
Hit http://ppa.launchpad.net raring/main amd64 Packages
Get:7 http://ppa.launchpad.net raring/main Sources [860 B]
Get:8 http://ppa.launchpad.net raring/main amd64 Packages [1,072 B]
Ign http://security.ubuntu.com raring-security/main Translation-en_US
Ign http://security.ubuntu.com raring-security/main Translation-en
Ign http://security.ubuntu.com raring-security/universe Translation-en_US
Ign http://security.ubuntu.com raring-security/universe Translation-en
Err http://security.ubuntu.com raring-security/main Sources
404 Not Found [IP: 91.189.88.161 80]
Err http://security.ubuntu.com raring-security/universe Sources
404 Not Found [IP: 91.189.88.161 80]
Err http://security.ubuntu.com raring-security/main amd64 Packages
404 Not Found [IP: 91.189.88.161 80]
Err http://security.ubuntu.com raring-security/universe amd64 Packages
404 Not Found [IP: 91.189.88.161 80]
Ign http://ppa.launchpad.net raring/main Translation-en_US
Ign http://ppa.launchpad.net raring/main Translation-en
Ign http://ppa.launchpad.net raring/main Translation-en_US
Ign http://ppa.launchpad.net raring/main Translation-en
Ign http://ppa.launchpad.net raring/main Translation-en_US
Ign http://ppa.launchpad.net raring/main Translation-en
Fetched 33.7 kB in 6s (5,163 B/s)
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring/main/source/Sources
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring/universe/source/Sources
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring/main/binary-amd64/Packages
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring/universe/binary-amd64/Packages
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring-updates/main/source/Sources
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring-updates/universe/source/Sources
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring-updates/main/binary-amd64/Packages
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/raring-updates/universe/binary-amd64/Packages
404 Not Found [IP: 54.152.129.43 80]
W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/raring-security/main/source/Sources
404 Not Found [IP: 91.189.88.161 80]
W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/raring-security/universe/source/Sources
404 Not Found [IP: 91.189.88.161 80]
W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/raring-security/main/binary-amd64/Packages
404 Not Found [IP: 91.189.88.161 80]
W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/raring-security/universe/binary-amd64/Packages
404 Not Found [IP: 91.189.88.161 80]
E: Some index files failed to download. They have been ignored, or old
ones used instead.
Indeed, if I then attempt to install fastqc, apt-get complains of not being
able to authenticate some packages:
# apt-get install fastqc
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libjbzip2-java libsam-java
Suggested packages:
picard-tools
The following NEW packages will be installed:
fastqc libjbzip2-java libsam-java
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 547 kB of archives.
After this operation, 1,115 kB of additional disk space will be used.
Do you want to continue [Y/n]?
WARNING: The following packages cannot be authenticated!
libjbzip2-java libsam-java fastqc
Install these packages without verification [y/N]? n
E: Some packages could not be authenticated
Maybe StarCluster's default configuration for apt-get is out-of-date?
% starcluster --version
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [email protected]
0.95.6
% uname -ar
Linux xxx 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#622>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABq0YuQiFT7a056wD8XTIFxbT1Jo_Kaoks5uOugqgaJpZM4Vz8IB>
.
|
|
@vasisht: Sorry, I should have been clearer: I ran Sorry for the confusion. (Not sure if it is relevant to your comment, but I thought I'd clarify anyway.) |
|
It’s the same issue with the older Ubuntu version :) I use a fork of
StarCluster maintained by datacratic (
https://github.com/datacratic/StarCluster) which has some nice additions.
On August 8, 2018 at 9:49:40 AM, kynnjo ([email protected]) wrote:
@vasisht <https://github.com/vasisht>: Sorry, I should have been clearer: I
ran uname -ar on my desktop, not on the cluster. (I have not specified any
distro in my StarCluster config, so for my cluster I should be getting
whatever the default is.)
Sorry for the confusion.
(Not sure if it is relevant to your comment, but I thought I'd clarify
anyway.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#622 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABq0YkAe3Dypq5Rku4JaTtavQo-fLK92ks5uOuxzgaJpZM4Vz8IB>
.
|
|
OK, I confirmed that the default apt-get configuration for the cluster is out-of-date. Here are the non-comment lines in I am not too familiar with Ubuntu (I use mostly Debian), but I think that |
|
Yes, Raring had it’s end of life in 2014
On August 8, 2018 at 10:18:18 AM, kynnjo ([email protected]) wrote:
OK, I confirmed that the default apt-get configuration for the cluster is
out-of-date. Here are the non-comment lines in /etc/apt/sources.list:
# grep -P '^[^\#]' /etc/apt/sources.list
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring-updates main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring-updates main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring universe
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring-updates universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ raring-updates universe
deb http://security.ubuntu.com/ubuntu raring-security main
deb-src http://security.ubuntu.com/ubuntu raring-security main
deb http://security.ubuntu.com/ubuntu raring-security universe
deb-src http://security.ubuntu.com/ubuntu raring-security universe
I am not too familiar with Ubuntu (I use mostly Debian), but I think that
raring is old.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#622 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABq0YrnXRiKfJcvmkFqwdPtl3u2bfz6Lks5uOvMqgaJpZM4Vz8IB>
.
|
|
OK, I'm upgrading my cluster to xenial now, but I'm finding it to be no picnic, and having to do it greatly diminishes the utility of StarCluster. |
|
@vasisht: do you have a link for one of those third-party Ubuntu-16.04 amis? I was not able to find one with Google. Nor was I able to upgrade one of the existing amis to Ubuntu-16.04 (I had too many errors during upgrade, and I finally gave up). |
|
ami-040b6113 is a 16.04 StarCluster AMI. Let me know if that works for you.
On August 8, 2018 at 11:34:57 AM, kynnjo ([email protected]) wrote:
@vasisht <https://github.com/vasisht>: do you have a link for one of those
third-party Ubuntu-16.04 amis? I was not able to find one with Google. Nor
was I able to upgrade one of the existing amis to Ubuntu-16.04 (I had too
many errors during upgrade, and I finally gave up).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#622 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABq0YrLduZNiwTQR-L-4Go53mW6QQswHks5uOwUggaJpZM4Vz8IB>
.
|
|
@vasisht: Thank you, and sorry for the delayed reply. After I posted my previous comment I did more research on StarCluster, and came to the conclusion that he project is not being very actively maintained any more (the apparent scarcity of amis with recent versions of Ubuntu is certainly consistent with this), so I am uncertain about relying on it too much. |
|
@kynnjo: There are other options like cfncluster or elasticluster. Nextflow
is also an option if you’re porting a pipeline.
On August 19, 2018 at 10:47:38 AM, kynnjo ([email protected]) wrote:
@vasisht <https://github.com/vasisht>: Thank you, and sorry for the delayed
reply. After I posted my previous comment I did more research on
StarCluster, and came to the conclusion that he project is not being very
actively maintained any more (the apparent scarcity of amis with recent
versions of Ubuntu is certainly consistent with this), so I am uncertain
about relying on it too much.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#622 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABq0YnzXbFRHpKot6A93s5g3LO_Cgcayks5uSXqKgaJpZM4Vz8IB>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right after creating a new cluster, I log in to it with
starcluster sshmaster <name-of-cluster>, and immediately run the commandapt-get update(as the very first command after login). The output from this command reveals several problems:Indeed, if I then attempt to install
fastqc,apt-getcomplains of not being able to authenticate some packages:Maybe StarCluster's default configuration for
apt-getis out-of-date?On my desktop, my StarCluster version is as follows:
My cluster's distro is:
EDIT: FWIW, the following is the distro for my desktop (not the cluster):
The text was updated successfully, but these errors were encountered: