Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to generate setfile with multiple hosts and roles #124

Merged
merged 2 commits into from
Apr 25, 2024
Merged

Add option to generate setfile with multiple hosts and roles #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e4dd12d
Add option to generate setfile with multiple hosts and roles
h-haaks Apr 17, 2024
6ea0b1c
Rewrite to beaker-hosts option with roles
h-haaks Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 109 additions & 37 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,23 @@ To get outputs [usable in Github Actions](https://docs.github.com/en/free-pro-te

```console
$ metadata2gha
puppet_major_versions=[{"name":"Puppet 7","value":7,"collection":"puppet7"},{"name":"Puppet 6","value":6,"collection":"puppet6"}]
puppet_unit_test_matrix=[{"puppet":7,"ruby":"2.7"},{"puppet":6,"ruby":"2.5"}]
github_action_test_matrix=[{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 7","value":7,"collection":"puppet7"}},{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 6","value":6,"collection":"puppet6"}}]
puppet_major_versions=[{"name":"Puppet 8","value":8,"collection":"puppet8"},{"name":"Puppet 7","value":7,"collection":"puppet7"}]
puppet_unit_test_matrix=[{"puppet":8,"ruby":"3.2"},{"puppet":7,"ruby":"2.7"}]
puppet_beaker_test_matrix=[{"name":"Puppet 8 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet8","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet8}"}},{"name":"Puppet 7 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet7","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet7}"}}]
```

Puppet major versions formatted for readability:
```json
[
{
"name": "Puppet 8",
"value": 8,
"collection": "puppet8"
},
{
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
},
{
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
}
]
```
Expand All @@ -41,45 +41,37 @@ Puppet unit test matrix formatted for readability:
```json
[
{
"puppet": 7,
"ruby": "2.7"
"puppet": 8,
"ruby": "3.2"
},
{
"puppet": 6,
"ruby": "2.5"
"puppet": 7,
"ruby": "2.7"
}
]
```

GitHub Action test matrix formatted for readability
Beaker test matrix formatted for readability
```json
[
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
"name": "Puppet 8 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet8",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet8}"
}
},
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}"
}
}
]
```

It is also possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).
It is possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).

```console
$ metadata2gha --use-fqdn --pidfile-workaround true /path/to/metadata.json
Expand All @@ -89,21 +81,101 @@ This results in the following JSON data
```json
[
{
"name": "CentOS 7",
"value": "centos7-64{hostname=centos7-64.example.com,image=centos:7.6.1810}"
"name": "Puppet 7 - CentOS 7",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "centos7-64{hostname=centos7-64-puppet7.example.com,image=centos:7.6.1810}"
}
},
{
"name": "Debian 10",
"value": "debian10-64{hostname=debian10-64.example.com}"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7.example.com}"
}
},
{
"name": "Ubuntu 18.04",
"value": "ubuntu1804-64{hostname=ubuntu1804-64.example.com}"
"name": "Puppet 7 - Ubuntu 22.04",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "ubuntu2204-64{hostname=ubuntu2204-64-puppet7.example.com}"
}
}
]
```

It is also possible to specify a comma separated list of operating systems as used in `metadata.json` (`CentOS,Ubuntu`).
If you need custom hostname or multiple hosts in your integration tests this could be achived by using the --beaker-hosts option

Option argument is 'HOSTNAME:ROLES;HOSTNAME:..;..' where
- hosts are separated by ';'
- host number and roles are separated by ':'
- Roles follow beaker-hostgenerator syntax
If you don't need any extra roles use '1;2;..'

```console
$ metadata2gha --beaker-hosts 'foo:primary.ma;bar:secondary.a'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64primary.ma{hostname=foo-puppet7}-debian12-64secondary.a{hostname=bar-puppet7}"
}
}
]
```

If you need to Expand the matrix ie by product versions it could be achived by using the --beaker-facter option

Option argument is 'FACT:LABEL:VALUE,VALUE,..' where
- Fact, label and values are separated by ':'
- Values are separated by ','

```console
$ metadata2gha --beaker-facter 'mongodb_repo_version:MongoDB:4.4,5.0,6.0,7.0'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12 - MongoDB 4.4",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "4.4"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 5.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "5.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 6.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "6.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 7.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "7.0"
}
}
]
```

## Work with the API

Expand Down
10 changes: 10 additions & 0 deletions bin/metadata2gha
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ options = {
domain: nil,
minimum_major_puppet_version: nil,
beaker_fact: nil,
beaker_hosts: nil,
}

OptionParser.new do |opts|
Expand Down Expand Up @@ -40,6 +41,15 @@ OptionParser.new do |opts|
options[:beaker_facter] = [fact, label, values.split(',')]
end
end
opts.on('--beaker-hosts HOSTNAME:ROLES;HOSTNAME:ROLES;...', 'Expand the setfile string to create multiple hosts with custom roles. Roles string; see beaker-hostgenerator') do |opt|
options[:beaker_hosts] = {}
if opt != 'false'
opt.split(';').each do |host|
hostname, roles = host.split(':', 2)
options[:beaker_hosts][hostname] = roles
end
end
end
end.parse!

filename = ARGV[0]
Expand Down
51 changes: 43 additions & 8 deletions lib/puppet_metadata/beaker.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,34 +55,69 @@ def adjusted_os(os)
# Enforce a domain to be appended to the hostname, making it an FQDN
# @param [Optional[String]] puppet_version
# The desired puppet version. Will be appended to the hostname
# @param [Optional[Hash]] hosts
# Key: hostname, Value: roles (roles string as defined by beaker-hostgenerator )
# Override the automatically generated hostname and optionally add roles
# If more than one entry this will generate multiple hosts in the setfile
# The domain may still be set via the `domain` param.
#
# @return [nil] If no setfile is available
# @return [Array<(String, String)>] The beaker setfile description with a readable name
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil)
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil, hosts: nil)
return unless os_supported?(os)

aos = adjusted_os(os)

name = "#{aos}#{release.tr('.', '')}-64"
hostname = (puppet_version.nil? && puppet_version != 'none') ? name : "#{name}-#{puppet_version}"
human_name = "#{os} #{release}"
domain ||= 'example.com' if use_fqdn

options = {}
options[:hostname] = "#{hostname}.#{domain}" if domain
hosts_settings = []
if hosts
hosts.each do |hostname, roles|
hosts_settings << {
'name' => if roles
name + roles
elsif hosts.size > 1
hosts_settings.empty? ? "#{name}.ma" : "#{name}.a"
else
name
end,
'hostname' => ((puppet_version.nil? || puppet_version == 'none') ? hostname : "#{hostname}-#{puppet_version}") + (domain ? ".#{domain}" : ''),
}
end
else
hosts_settings << {
'name' => name,
'hostname' => if puppet_version && puppet_version != 'none'
"#{name}-#{puppet_version}" + (domain ? ".#{domain}" : '')
elsif domain
name + (domain ? ".#{domain}" : '')
else
''
end,
}
end

options = {}
# Docker messes up cgroups and some systemd versions can't deal with
# that when PIDFile is used.
image_to_use = nil
if pidfile_workaround?(pidfile_workaround, os)
return if PIDFILE_INCOMPATIBLE[os]&.include?(release)

if (image = PIDFILE_COMPATIBLE_IMAGES.dig(os, release))
options[:image] = image
image_to_use = image
end
end

human_name = "#{os} #{release}"
setfile_parts = []
hosts_settings.each do |host_settings|
options[:hostname] = host_settings['hostname'] unless host_settings['hostname'].empty?
options[:image] = image_to_use if image_to_use
setfile_parts << build_setfile(host_settings['name'], options)
end

[build_setfile(name, options), human_name]
[setfile_parts.join('-'), human_name]
end

# Return whether a Beaker setfile can be generated for the given OS
Expand Down
1 change: 1 addition & 0 deletions lib/puppet_metadata/github_actions.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ def os_release_to_beaker_setfile(os, release, puppet_collection)
pidfile_workaround: options[:beaker_pidfile_workaround],
domain: options[:domain],
puppet_version: puppet_collection,
hosts: options[:beaker_hosts],
)
end
end
Expand Down
55 changes: 54 additions & 1 deletion spec/beaker_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

it { expect(described_class.os_release_to_setfile('SLES', '11')).to be_nil }

describe 'pidfile_workaround' do
context 'with pidfile_workaround' do
describe 'true' do
[
['CentOS', '6', ['centos6-64', 'CentOS 6']],
Expand Down Expand Up @@ -57,5 +57,58 @@
it { expect(described_class.os_release_to_setfile('CentOS', '8', pidfile_workaround: ['CentOS'])).to be_nil }
end
end

context 'with domain' do
it {
expect(described_class.os_release_to_setfile('CentOS', '7', domain: 'mydomain.org')).to eq(['centos7-64{hostname=centos7-64.mydomain.org}', 'CentOS 7'])
}
end

context 'with puppet_version' do
[
['CentOS', '7', 'none', ['centos7-64', 'CentOS 7']],
['CentOS', '7', 'puppet7', ['centos7-64{hostname=centos7-64-puppet7}', 'CentOS 7']],
].each do |os, release, puppet_version, expected|
it { expect(described_class.os_release_to_setfile(os, release, puppet_version: puppet_version)).to eq(expected) }
end
end

context 'with hosts and no roles' do
[
['Debian', '12', { 'foo' => nil }, ['debian12-64{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => nil, 'bar' => nil }, ['debian12-64.ma{hostname=foo}-debian12-64.a{hostname=bar}', 'Debian 12']],
].each do |os, release, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts and roles' do
[
['Debian', '12', { 'foo' => 'myrole.ma' }, ['debian12-64myrole.ma{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => 'myrole,primary.ma' }, ['debian12-64myrole,primary.ma{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo}-debian12-64myrole,secondary.a{hostname=bar}', 'Debian 12'],],
].each do |os, release, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts, roles and domain' do
[
['Debian', '12', 'mydomain.org', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo.mydomain.org}-debian12-64myrole,secondary.a{hostname=bar.mydomain.org}', 'Debian 12'],],
].each do |os, release, domain, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, domain: domain, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts, roles, domain and puppet_version' do
[
['Debian', '12', 'mydomain.org', 'puppet7', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo-puppet7.mydomain.org}-debian12-64myrole,secondary.a{hostname=bar-puppet7.mydomain.org}', 'Debian 12'],],
].each do |os, release, domain, puppet_version, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, domain: domain, puppet_version: puppet_version, hosts: hosts)).to eq(expected) }
end
end
end
end
Loading