Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to generate setfile with multiple hosts and roles #124

Merged
merged 2 commits into from
Apr 25, 2024
Merged

Add option to generate setfile with multiple hosts and roles #124

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e4dd12d
Add option to generate setfile with multiple hosts and roles
h-haaks Apr 17, 2024
6ea0b1c
Rewrite to beaker-hosts option with roles
h-haaks Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 112 additions & 37 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,23 @@ To get outputs [usable in Github Actions](https://docs.github.com/en/free-pro-te

```console
$ metadata2gha
puppet_major_versions=[{"name":"Puppet 7","value":7,"collection":"puppet7"},{"name":"Puppet 6","value":6,"collection":"puppet6"}]
puppet_unit_test_matrix=[{"puppet":7,"ruby":"2.7"},{"puppet":6,"ruby":"2.5"}]
github_action_test_matrix=[{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 7","value":7,"collection":"puppet7"}},{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 6","value":6,"collection":"puppet6"}}]
puppet_major_versions=[{"name":"Puppet 8","value":8,"collection":"puppet8"},{"name":"Puppet 7","value":7,"collection":"puppet7"}]
puppet_unit_test_matrix=[{"puppet":8,"ruby":"3.2"},{"puppet":7,"ruby":"2.7"}]
puppet_beaker_test_matrix=[{"name":"Puppet 8 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet8","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet8}"}},{"name":"Puppet 7 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet7","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet7}"}}]
```

Puppet major versions formatted for readability:
```json
[
{
"name": "Puppet 8",
"value": 8,
"collection": "puppet8"
},
{
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
},
{
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
}
]
```
Expand All @@ -41,45 +41,37 @@ Puppet unit test matrix formatted for readability:
```json
[
{
"puppet": 7,
"ruby": "2.7"
"puppet": 8,
"ruby": "3.2"
},
{
"puppet": 6,
"ruby": "2.5"
"puppet": 7,
"ruby": "2.7"
}
]
```

GitHub Action test matrix formatted for readability
Beaker test matrix formatted for readability
```json
[
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
"name": "Puppet 8 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet8",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet8}"
}
},
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}"
}
}
]
```

It is also possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).
It is possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).

```console
$ metadata2gha --use-fqdn --pidfile-workaround true /path/to/metadata.json
Expand All @@ -89,21 +81,104 @@ This results in the following JSON data
```json
[
{
"name": "CentOS 7",
"value": "centos7-64{hostname=centos7-64.example.com,image=centos:7.6.1810}"
"name": "Puppet 7 - CentOS 7",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "centos7-64{hostname=centos7-64-puppet7.example.com,image=centos:7.6.1810}"
}
},
{
"name": "Debian 10",
"value": "debian10-64{hostname=debian10-64.example.com}"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7.example.com}"
}
},
{
"name": "Ubuntu 18.04",
"value": "ubuntu1804-64{hostname=ubuntu1804-64.example.com}"
"name": "Puppet 7 - Ubuntu 22.04",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "ubuntu2204-64{hostname=ubuntu2204-64-puppet7.example.com}"
}
}
]
```

If you need multiple hosts in your integration tests this could be achived by using the --beaker_nodes_and_roles option

Option argument is '#HOST:ROLE,ROLE,...;#HOST:..;..' where
- hosts are separated by ';'
- host number and roles are separated by ':'
- Roles are separated by ','
If you don't need any extra roles use '1;2;..'

The 'master' and 'agent' roles will be automatically added to the first host.
The 'agent' role will be automatically added to the rest.

```console
$ metadata2gha --beaker_nodes_and_roles '1:role1,other1;2:role2,other2'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64role1,other1.ma{hostname=debian12-64-puppet7-1}-debian12-64role2,other2.a{hostname=debian12-64-puppet7-2}"
}
}
]
```

It is also possible to specify a comma separated list of operating systems as used in `metadata.json` (`CentOS,Ubuntu`).
If you need to Expand the matrix ie by product versions it could be achived by using the --beaker-facter option

Option argument is 'FACT:LABEL:VALUE,VALUE,..' where
- Fact, label and values are separated by ':'
- Values are separated by ','

```console
$ metadata2gha --beaker-facter 'mongodb_repo_version:MongoDB:4.4,5.0,6.0,7.0'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12 - MongoDB 4.4",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "4.4"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 5.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "5.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 6.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "6.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 7.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "7.0"
}
}
]
```

## Work with the API

Expand Down
13 changes: 13 additions & 0 deletions bin/metadata2gha
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,19 @@ OptionParser.new do |opts|
options[:beaker_facter] = [fact, label, values.split(',')]
end
end
opts.on('--beaker_nodes_and_roles #NODE:ROLES;#NODE:ROLES;...', 'Expand the setfile string to create multiple nodes with custom roles. Separate roles using commas') do |opt|
options[:beaker_nodes_and_roles] = {}
if opt != 'false'
opt.split(';').each do |node|
node_num, roles = node.split(':', 2)
options[:beaker_nodes_and_roles][node_num] = if roles
roles.split(',')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could just keep roles as a string. no need to split it into array.

else
[]
end
end
end
end
end.parse!

filename = ARGV[0]
Expand Down
29 changes: 24 additions & 5 deletions lib/puppet_metadata/beaker.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,18 @@ def adjusted_os(os)
#
# @return [nil] If no setfile is available
# @return [Array<(String, String)>] The beaker setfile description with a readable name
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil)
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil, nodes_and_roles: nil)
return unless os_supported?(os)

aos = adjusted_os(os)

name = "#{aos}#{release.tr('.', '')}-64"
hostname = (puppet_version.nil? && puppet_version != 'none') ? name : "#{name}-#{puppet_version}"
domain ||= 'example.com' if use_fqdn

hostname = (puppet_version.nil? || puppet_version == 'none') ? name : "#{name}-#{puppet_version}"
options = {}
options[:hostname] = "#{hostname}.#{domain}" if domain
if domain || (puppet_version && puppet_version != 'none')
options[:hostname] = domain ? "#{hostname}.#{domain}" : hostname
end

# Docker messes up cgroups and some systemd versions can't deal with
# that when PIDFile is used.
Expand All @@ -82,7 +83,25 @@ def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: fals

human_name = "#{os} #{release}"

[build_setfile(name, options), human_name]
if nodes_and_roles
names = []
nodes_and_roles.each do |node, roles|
roles.map!(&:strip)
n = "#{name}#{roles.join(',')}"
n += if names.empty?
# add master role to first node
'.ma'
else
'.a'
end
options[:hostname] = hostname + "-#{node.strip}" if nodes_and_roles.size > 1
options[:hostname] = "#{options[:hostname]}.#{domain}" if options[:hostname] && domain
names << build_setfile(n, options)
end
[names.join('-'), human_name]
else
[build_setfile(name, options), human_name]
end
end

# Return whether a Beaker setfile can be generated for the given OS
Expand Down
1 change: 1 addition & 0 deletions lib/puppet_metadata/github_actions.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ def os_release_to_beaker_setfile(os, release, puppet_collection)
pidfile_workaround: options[:beaker_pidfile_workaround],
domain: options[:domain],
puppet_version: puppet_collection,
nodes_and_roles: options[:beaker_nodes_and_roles],
)
end
end
Expand Down
35 changes: 35 additions & 0 deletions spec/beaker_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,40 @@
it { expect(described_class.os_release_to_setfile('CentOS', '8', pidfile_workaround: ['CentOS'])).to be_nil }
end
end

describe 'domain' do
it {
expect(described_class.os_release_to_setfile('CentOS', '7', domain: 'mydomain.org',
use_fqdn: true)).to eq(['centos7-64{hostname=centos7-64.mydomain.org}', 'CentOS 7'])
}
end

describe 'puppet_version' do
[
['CentOS', '7', 'none', ['centos7-64', 'CentOS 7']],
['CentOS', '7', 'puppet7', ['centos7-64{hostname=centos7-64-puppet7}', 'CentOS 7']],
].each do |os, release, puppet_version, expected|
it { expect(described_class.os_release_to_setfile(os, release, puppet_version: puppet_version)).to eq(expected) }
end
end

describe 'nodes_and_roles' do
[
['CentOS', '7', { '1' => ['role1', 'role2'] }, ['centos7-64role1,role2.ma', 'CentOS 7']],
['CentOS', '7', { '1' => [], '2' => [] }, ['centos7-64.ma{hostname=centos7-64-1}-centos7-64.a{hostname=centos7-64-2}', 'CentOS 7']],
['CentOS', '7', { '1' => ['role1'], '2' => ['role2'] }, ['centos7-64role1.ma{hostname=centos7-64-1}-centos7-64role2.a{hostname=centos7-64-2}', 'CentOS 7']],
].each do |os, release, nodes, expected|
it { expect(described_class.os_release_to_setfile(os, release, nodes_and_roles: nodes)).to eq(expected) }
end
end

describe 'domain, puppet_version and nodes_and_roles' do
[
['CentOS', '7', 'mydomain.org', 'puppet7', { '1' => ['role1'], '2' => ['role2'] },
['centos7-64role1.ma{hostname=centos7-64-puppet7-1.mydomain.org}-centos7-64role2.a{hostname=centos7-64-puppet7-2.mydomain.org}', 'CentOS 7'],],
].each do |os, release, domain, puppet_version, nodes, expected|
it { expect(described_class.os_release_to_setfile(os, release, domain: domain, puppet_version: puppet_version, nodes_and_roles: nodes)).to eq(expected) }
end
end
end
end
Loading