test archive does not include split-debuginfo files

[erratic-pattern]

When building a test archive with debuginfo enabled and with split-debuginfo=unpacked (default on MacOS when using Cargo) or split-debuginfo=packed (default on Windows MVSC and default on MacOS when using rustc directly), debugging symbols are not included in the archive.

Motivation

Binaries that rely on runtime debuginfo being present will not work. This also affects debuggers and other instrumentation such as code coverage tools. Custom panic handlers and error reporters that rely on runtime debuginfo to generate backtraces or source code snippets are also affected.

On stable channel, MacOS and Windows builds do not have an option to use split-debuginfo=off (default on Linux) as a workaround, so debuginfo must always be stored in separate files on those platforms.

Minimal Example

Required Platform: MacOS or Windows

Create a new project:

mkdir missing-debug-info
cd missing-debug-info
cargo init

To simulate a use case that requires runtime debuginfo, I'm going to use the backtrace crate. Real world examples that follow this pattern are custom panic hooks for error reporting such as those installed by color_eyre and miette to print backtraces.

cargo add --dev [email protected]

Then create an integration test that uses backtrace.

tests/test_example.rs:

use backtrace::Backtrace;
use std::{env, path::PathBuf};

#[test]
fn test_debug_symbols() {
    let test_dir = env::var("CARGO_MANIFEST_DIR").map(PathBuf::from).unwrap();
    let mut found = false;
    for frame in Backtrace::new().frames().iter() {
        for symbol in frame.symbols().iter() {
            if let Some(filename) = symbol.filename() {
                if filename.starts_with(test_dir.as_path()) {
                    found = true;
                    break;
                }
            }
        }
    }
    assert!(found);
}

Validate the test works when ran normally:

cargo nextest run

Expected output:

    Finished test [unoptimized + debuginfo] target(s) in 0.03s
    Starting 1 test across 2 binaries
        PASS [   0.146s] missing-debug-info::test_example test_debug_symbols
------------
     Summary [   0.146s] 1 test run: 1 passed, 0 skipped

Create the test archive:

cargo nextest archive --archive-file archive.tar.zst

Important Step: delete the target directory to simulate a new environment.

rm -rf target/

On MacOS, dsymutil can locate the .o files (for split-debuginfo=unpacked) or .dSYM directories (for split-debuginfo=packed) even if the binary file is moved by reading the OSO stab entries in the binary. This is true even if the target directory moves, because it uses Spotlight indexer UUIDs to locate the files rather than absolute paths.

On Windows,dbghelp will also still be able to locate the .pdb files even if the binary is moved, but only if the .pdb files have not moved. I believe this is because the locations of the .pdb file are stored in the binary as absolute paths.

Now run the tests from archive:

cargo nextest run --archive-file archive.tar.zst

You should see the test fail:

  Extracting 3 binaries (including 1 non-test binaries) to /private/var/folders/0y/jf9p__mj7q37h62qwzn3mbl80000gn/T/nextest-archive-ePL2WM
   Extracted 5 files to /private/var/folders/0y/jf9p__mj7q37h62qwzn3mbl80000gn/T/nextest-archive-ePL2WM in 0.03s
    Starting 1 test across 2 binaries
        FAIL [   0.202s] missing-debug-info::test_example test_debug_symbols

--- STDOUT:              missing-debug-info::test_example test_debug_symbols ---

running 1 test
test test_debug_symbols ... FAILED

failures:

failures:
    test_debug_symbols

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s


--- STDERR:              missing-debug-info::test_example test_debug_symbols ---
thread 'test_debug_symbols' panicked at 'assertion failed: found', tests/test_example.rs:18:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

   Canceling due to test failure
------------
     Summary [   0.202s] 1 test run: 0 passed, 1 failed, 0 skipped
        FAIL [   0.202s] missing-debug-info::test_example test_debug_symbols
error: test run failed

Solution

nextest could support this by reading the compiler-artifact messages from cargo's diagnostic JSON output, checking for the presence of debugging files when relevant build options are enabled, and then including those debugging files in the output archive. This comment by luser on a similar issue provides a good overview of all the different permutations of split-debuginfo and target platforms.

Other Considerations

Other users may not want this functionality, and may prefer to keep their archive sizes small. It might be a good idea to have a new CLI option for the archive subcommand to disable this functionality.

Supporting Windows could be a challenge, because it looks like the metadata in the binary needs to be updated to point to new paths when --workspace-remap is used. I need to do more research on this.

Workaround (MacOS only)

Here's the workaround I made to get around this issue on MacOS when building with (mostly) default debug profile settings. It's a bash script to unpack the test archive, add the debug files, and then recompress. The only change you need to make to build flags is to set split-debuginfo=packed

ARCHIVE_FILE="$1"

DEBUG_FILES="$(find target -name '*.dSYM')"

if [ -n "$DEBUG_FILES" ]; then
    zstd -d "$ARCHIVE_FILE" -o tmp-archive.tar
    tar rvf tmp-archive.tar $DEBUG_FILES
    zstd --rm -f tmp-archive.tar -o "$ARCHIVE_FILE"
fi

I'm still researching a Windows workaround that is compatible with --workspace-remap.

[sunshowers]

Thanks for the report! I agree with your solution, at least on macOS. Only thing I'll add is that this can lead to an unexpected slowdown during both archive and extract, so we'll want to warn on it at least for a few months.

I don't have time to work on this soon, but would you like to take it on?

Curious what your Windows poking leads to. I found this comment which led me to this MSDN article. I think setting _NT_ALT_SYMBOL_PATH is a reasonable approach assuming it works.

[erratic-pattern]

I don't have time to work on this soon, but would you like to take it on?

Yep I'll play around with it and see if I can get it working.

Curious what your Windows poking leads to. I found rust-lang/rust#87825 (comment) which led me to this MSDN article. I think setting _NT_ALT_SYMBOL_PATH is a reasonable approach assuming it works.

Nice find. I'll run a test with that to see if it works.

Also I may have spoke too soon about MacOS working with --workspace-remap in all cases, as my CI pipeline is now failing to find the source info in some cases but in other cases it works fine. I need to dig into that as well and see what's causing it to fail. If it needs to remap headers in the binary to work with --workspace-remap that means that we'd require some dependencies on MacOS libs for this to function properly. Maybe I can find some way to remap it like the magic environment variable you mentioned for Windows.