Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use git instead of rsync for the Gentoo overlay #106

Merged
merged 23 commits into from
Jun 21, 2021
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
1bbe56f
add step that changes gentoo overlay from using rsync to git
bedroge Jun 7, 2021
a1c3182
add step for generating post-sync hooks
bedroge Jun 9, 2021
6f2ff40
extra step for creating postsync dir
bedroge Jun 9, 2021
bf7c37c
rename step
bedroge Jun 9, 2021
66482ca
add mode for template task
bedroge Jun 9, 2021
fd52eb7
also generate metadata for custom overlays
bedroge Jun 9, 2021
d98d6ed
add postsync scripts
bedroge Jun 9, 2021
e433003
add other script, which we won't use for now
bedroge Jun 9, 2021
2f1b85e
dont run sync_gentoo_projects_xml for now
bedroge Jun 9, 2021
c88d68a
use copy instead of template for post-sync scripts
bedroge Jun 10, 2021
e2e694e
use portageq to find EPREFIX, remove .j2 extensions
bedroge Jun 10, 2021
c01cf2d
add reframe test for validating that gentoo overlay uses git
bedroge Jun 11, 2021
7a08570
fix hardcoded path
bedroge Jun 11, 2021
6fa0aff
Merge branch 'main' into gentoo_git
bedroge Jun 19, 2021
0e64d56
remove old versions of scripts in template dir
bedroge Jun 20, 2021
2c5b58f
remove commented sync_gentoo_project_xml
bedroge Jun 20, 2021
2c1a333
Add a system update step for existing Prefix
bedroge Jun 21, 2021
397ac8d
remove trailing spaces
bedroge Jun 21, 2021
9db7b4b
remove wrong 'not' operator
bedroge Jun 21, 2021
973081f
update world instead of system
bedroge Jun 21, 2021
3044dff
verbose output for ansible-playbook
bedroge Jun 21, 2021
cdeb9e7
Update of system set should go after the sync...
bedroge Jun 21, 2021
254a1b8
update world instead of system
bedroge Jun 21, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions ansible/playbooks/roles/compatibility_layer/tasks/add_overlay.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,47 @@
selectattr('eclass-overrides', 'equalto', True) | map(attribute='name') | join(' ')
}}

- name: Use git instead of rsync for the Gentoo overlay
copy:
dest: "{{ gentoo_prefix_path }}/etc/portage/repos.conf/gentoo.conf"
mode: "0644"
content: |
[DEFAULT]
main-repo = gentoo
sync-git-pull-extra-opts = --quiet

[gentoo]
priority = 1
location = {{ gentoo_prefix_path }}/var/db/repos/gentoo
sync-uri = https://github.com/gentoo/gentoo.git
bedroge marked this conversation as resolved.
Show resolved Hide resolved
sync-type = git
auto-sync = Yes
clone-depth = 1

- name: Create repo.postsync.d directory
file:
path: "{{ gentoo_prefix_path }}/etc/portage/repo.postsync.d"
state: directory
mode: '0755'

- name: Add post-sync hooks for Gentoo overlay
template:
src: "{{ item }}.j2"
dest: "{{ gentoo_prefix_path }}/etc/portage/repo.postsync.d/{{ item }}"
mode: 0755
with_items:
- sync_gentoo_cache
- sync_gentoo_dtd
- sync_gentoo_glsa
- sync_gentoo_news
- sync_overlay_cache
#- sync_gentoo_projects_xml
bedroge marked this conversation as resolved.
Show resolved Hide resolved

- name: Remove the old Gentoo overlay directory
file:
state: absent
path: "{{ gentoo_prefix_path }}/var/db/repos/gentoo"

- name: Sync the repositories
portage:
sync: 'yes'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "gentoo" ]] || exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh
bedroge marked this conversation as resolved.
Show resolved Hide resolved

# Number of jobs for egencache, default is number or processors.
parallel_jobs="$(nproc)"

if [[ -f ${repository_path}/metadata/timestamp.x ]]; then
portage_current_timestamp=$(cut -f 1 -d " " "${repository_path}/metadata/timestamp.x" )
else
portage_current_timestamp=0
fi

ebegin "Fetching metadata timestamp for ${repository_name}"
rsync -aq rsync://rsync.gentoo.org/gentoo-portage/metadata/timestamp.x "${repository_path}"/metadata/timestamp.x
eend $?
portage_new_timestamp=$(cut -f 1 -d " " "${repository_path}/metadata/timestamp.x" )

if [[ ${portage_current_timestamp} -lt ${portage_new_timestamp} ]]; then
ebegin "Fetching pre-generated metadata cache for ${repository_name}"
rsync -aq rsync://rsync.gentoo.org/gentoo-portage/metadata/md5-cache/ "${repository_path}"/metadata/md5-cache/
eend $?
else
einfo "Metadata cache for ${repository_name} already recent, no need to fetch it :-)"
fi

ebegin "Updating metadata cache for ${repository_name}"
egencache --jobs="${parallel_jobs}" --repo="${repository_name}" --update --update-use-local-desc
eend $?
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "gentoo" ]] || exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh

DTDDIR="${repository_path}"/metadata/dtd
ebegin "Updating DTDs"
if [[ -e ${DTDDIR} ]]; then
git -C "${DTDDIR}" pull -q --ff-only
else
git clone -q https://anongit.gentoo.org/git/data/dtd.git "${DTDDIR}"
fi
eend "$?"
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "gentoo" ]] || exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh

GLSADIR="${repository_path}"/metadata/glsa
ebegin "Updating GLSAs"
if [[ -e ${GLSADIR} ]]; then
git -C "${GLSADIR}" pull -q --ff-only
else
git clone -q https://anongit.gentoo.org/git/data/glsa.git "${GLSADIR}"
fi
eend "$?"
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "gentoo" ]] || exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh

NEWSDIR="${repository_path}"/metadata/news
ebegin "Updating news items"
if [[ -e ${NEWSDIR} ]]; then
git -C "${NEWSDIR}" pull -q --ff-only
else
git clone -q https://anongit.gentoo.org/git/data/gentoo-news.git "${NEWSDIR}"
fi
eend $? "Try to remove ${NEWSDIR}"
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "gentoo" ]] || exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh

ebegin "Updating projects.xml"
wget -q -P "${repository_path}"/metadata/ -N https://api.gentoo.org/metastructure/projects.xml
eend $?
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash

repository_name="${1}"

[[ ${repository_name} == "gentoo" ]] && exit 0

source {{ gentoo_prefix_path }}/lib/gentoo/functions.sh

# Number of jobs for egencache, default is number or processors.
parallel_jobs="$(nproc)"

ebegin "Updating metadata cache for ${repository_name}"
egencache --jobs="${parallel_jobs}" --repo="${repository_name}" --update --update-use-local-desc
eend $?