How do I change the director_name?

[jflorian]

I have a good working setup in which I had set bacula::director_name and bacula::storage_name to the FQDN of my Director/Storage host (one and the same). This FQDN follows my convention of suffixing the OS release (e.g., scooby-f28.doubledog.org) so that when I rebuild this VM, I can have both new and old hosts existing (e.g., scooby-f30.doubledog.org). The idea here is that I should only need to change a pair of CNAMEs (e.g., bacula-dir.doubledog.org and bacula-sd.doubledog.org) in my DNS to transition to a newer Bacula server with an easy/safe rollback, if necessary.

So, while I presently have bacula::director_name set to scooby-f28.doubledog.org (and similarly for storage_name, I'd like to change this instead to bacula-dir.doubledog.org (and bacula-sd.doubledog.org), but when I do this all but one of my hosts disappear from the /etc/bacula/conf.d/client.conf file. (This one client is a very new build, scooby-f30 to be exact, but at this stage it's nothing more than I basic server with no setup yet beyond being a normal Bacula client.) I know that the exported resources are at play here, but I'm failing to determine what I need to do to rectify the problem. I've tried puppet node deactivate <FQDN> for the missing clients, but that doesn't help. I do see TODO Document the use of director_name in init.pp so I suppose I'm knocking on that door. 😉

[smortex]

Well, I guess the process is:

1. Update bacula::director_name;
2. Apply a catalog a first time on all nodes (so that correct information is updated in PuppetDB);
3. Apply a catalog a second time on the director and storage nodes (the updated data in PuppetDB will be used to produce a "fixed" catalog).

And I guess it should be correct.

Under the hood, there are a few collected resources that depend on the director name:

https://github.com/xaque208/puppet-bacula/blob/d32e0fdcf84edeea056d7e0ef009552d81a767aa/manifests/storage.pp#L84

https://github.com/xaque208/puppet-bacula/blob/d32e0fdcf84edeea056d7e0ef009552d81a767aa/manifests/director.pp#L127-L141

[jflorian]

That's exactly what I did. One of the reasons I want to make this change is because I remember the last time I built a new Bacula DIR/SD VM is that there was no easy way to try it with limited scope -- it was all or nothing because I had to follow the process you detailed. For some reason that's not working now. I haven't yet tried it, but I suspect simply trying to deploy the new canonical FQDN is going to yield the same problem. If you have any other ideas, I'm all ears.

[smortex]

These problems are hard to diagnose… maybe there are other steps and you have to run more than 2 batches of updates across your nodes to reach a consistent state.

I use puppet query to query PuppetDB and see what does puppet thinks about various exported resources. It's packaged in the puppetlabs repo (under another name, client-tools) and available in this repo: https://github.com/puppetlabs/puppetdb-cli

It may be useful to understand what is going on by querying PuppetDB:
https://puppet.com/docs/puppetdb/6.3/api/query/tutorial-pql.html

Last time I renamed a Bacula node, I also remember having a bad time not seeing what I expected, and it happened that everything was fine, I just was updating the wrong node / wrong environment causing exported resources to not be exported as I wanted…

[jflorian]

I rebuilt my PuppetDB from the PC1 repos, maybe v4 or so, but definitely not v5 or v6. Hence some of the delay. Still had no luck with any of this and I knew that my DB was rebuilt from scratch since I didn't migrate any data. I tried to push forth on my new dir/sd build and sort everything out there: new VM, no lingering cruft in case something moved elsewhere over time -- all that. Proper debugging has been challenging because I cannot get puppet query to work (https://tickets.puppetlabs.com/browse/PDB-4446?filter=-2).

However, I've made some progress by searching this model for fqdn, certname and clientcert. I'm finding that using CNAMEs with this module is pretty much impossible at present. I had two CNAMEs originally, bacula-dir.doubledog.org and bacula-sd.doubledog.org. Since these resolve to one and the same, I opted to simplify with just bacula.doubledog.org so that I needn't worry which was appropriate, because it becomes rather confusing what name is required where. This got me to a point where job.conf and client.conf had parity (no clients were referenced in job.conf where there wasn't a match in client.conf) -- so that much was good. But... now I get authentication errors where the "Director unable to authenticate with Storage daemon". I now see that bacula-dir.conf has the Director's name set to <%= $clientcert %>-dir, while bacula-sd.conf expects the CNAME. The module provides no means for overriding the name going into the former.

In summary, it seems there are way too many cases where fqdn, certname and clientcert are used where bacula::director_name or bacula::storage_name would be more correct. It doesn't matter when using canonical host names but CNAMEs reveals issues.

[zachfi]

CNAMEs are indeed somewhat troublesome as you point out @jflorian. There is some amount of desire to be able to authenticate both the client, storage, and director to each other using the Puppet CA. To this end, you will probably see some use of the trusted key words in the data to use the facts present in the node's certificate. CNAMEs sort of skirt this issue, unless they are also present as Subject Alternative Name's in the node cert.

I suspect that there could certainly be some clean up with respect to which names to use when, since ideally there are only a few places to update in order to do what you are after here.

In your last comment, are the CNAMEs that you are placing in the data variables not updating the configuration, or does bacula fail to communicate to the CNAME host?

[jflorian]

That commit resolved one of many issues. IIRC, in this particular case, it was necessary for the DIR and SD to recognize one another. I want to say the sd.conf was empty or missing something critical. My memory is a bit fuzzy. I thought I'd try to worm my way through all of the related issues but hit a bit of a stone wall once I realized the certificates were in need of SANs (as you mentioned). I wasn't sure how to do that with Puppet's CA and the task starting getting overly ambitious. Given that I'm going through a painful divorce and that I had ZERO backups (my pool was a mess so I had already wiped it to start anew), I figured it better to kick this can down the road for now. Thus I built out my latest Bacula DIR/SD using canonical hostnames (instead of the desired CNAMEs) as I've done before.

It stinks because Bacula, as designed, requires the DIR to be the same or newer version than the clients it serves. If you have bleeding edge clients (e.g., Fedora), you'll need a bleeding edge DIR too which means you need to be constantly rebuilding it. Puppet and this module make the build easy but due to the issues I started finding here, there doesn't seem to be any good/easy way to handle migrations short of "starting over" each time. Being able use CNAMEs would alleviate most of this pain.

I wish I had time to help tackle this, but my head is not in a good place this year. 🤕

[zachfi]

Not a worry @jflorian. Take all the time you need. Positive vibes are headed your way.

[jflorian]

Thanks @xaque208, divorce is awful and I don't even know if I've got to the worst part yet. 😞