Add support for finding oxide processes on a sled

[papertigers]

This PR adds support for finding all oxide processes via libcontract and relying on the fact that we deploy oxide services with the fmri prefix svc:/oxide/. This allows us to find every pid started via smf even if the smf script uses ctrun.

You can see this for yourself by running pfexec ctstat -av on a helios/illumos based system:

205     1       process owned   2492    0       -       -       
        cookie:                0
        informative event set: core
        critical event set:    hwerr empty
        fatal event set:       hwerr
        parameter set:         noorphan regent
        member processes:      2497
        inherited contracts:   none
        service fmri:          svc:/oxide/wicketd:default
        service fmri ctid:     204
        creator:               ctrun
        aux:                   
249     1       process owned   1334    0       -       -       
        cookie:                0x20
        informative event set: none
        critical event set:    hwerr empty
        fatal event set:       none
        parameter set:         inherit regent
        member processes:      2794
        inherited contracts:   none
        service fmri:          svc:/oxide/dendrite:default
        service fmri ctid:     249
        creator:               svc.startd
        aux:                   start
254     1       process owned   1334    0       -       -       
        cookie:                0x20
        informative event set: none
        critical event set:    hwerr empty
        fatal event set:       none
        parameter set:         inherit regent
        member processes:      2802
        inherited contracts:   none
        service fmri:          svc:/oxide/lldpd:default
        service fmri ctid:     254
        creator:               svc.startd
        aux:                   start

This PR is on top of:

• Pull support-bundle/queries into a standalone crate #7193

[papertigers]

@citrus-it I added you as a reviewer - if you have time do you mind looking over the libcontract bindings this is using?

[smklein]

From https://illumos.org/man/3CONTRACT/ct_pr_status_get_svc_creator#:~:text=The%20ct_pr_status_get_svc_fmri(),not%20be%20modified

  The ct_pr_status_get_svc_fmri(), ct_pr_status_get_svc_creator(), and
  ct_pr_status_get_svc_aux() functions read, respectively, the service
  FMRI, the contract's creator execname and  the creator's auxiliary field.
  The buffer pointed to by fmri, aux or creator, is freed by a call to
  ct_status_free() and should not be modified.

I'm under the impression that the ptr value here must be freed via a call to ct_status_free - is that right? or is this just saying that the lifetime of the returned result must be shorter than ContractStatus?

If it's the former (the ptr value must be freed explicitly): Are we leaking?
If it's the latter (the ptr values lives as long as ContractStatus): Is this code unsound? Would it be possible to keep using the returned CString after self has been dropped?

[papertigers]

The ptr returned is tied to the lifetime of the contract status handle ct_stathdl_t. We read that pointer as a cstr but the function returns a CString which copies the data to own it. If I understand correctly the ptr returned from the call to ct_pr_status_get_svc_fmri will not leave anything around because it will be cleaned up when ct_status_free is called via the drop method on ContractStatus.

I will verify this, but this was at least my intention upon writing this code.

[smklein]

Ah, gotcha, so:

        let cstr = unsafe { CStr::from_ptr(ptr) }

Would create a cstr that needs to have a shorter lifetime than the handle, but:

        cstr.to_owned()

Would clone it, and free us from that lifetime constraint when the function returns

[hawkw]

I wonder if it would be valuable to add an omdb command that does this and dumps the output, too? Of course, you can just run the appropriate ctstat commands for that, but it could maybe be nice to have a way to do this baked in?

[papertigers]

I wonder if it would be valuable to add an omdb command that does this and dumps the output, too? Of course, you can just run the appropriate ctstat commands for that, but it could maybe be nice to have a way to do this baked in?

Yes! The plan is to introduce a diagnostics command that lets you call the same endpoints that the support bundle sagas are using to collect all of the info.