rpm-ostree compose container-encapsulate abort with coredump

[dustymabe]

Host system details

latest COSA container

$ rpm -q rpm-ostree ostree
rpm-ostree-2023.7-1.fc38.x86_64
ostree-2023.5-2.fc38.x86_64

Expected vs actual behavior

I added export RUST_BACKTRACE=1 and export RUST_BACKTRACE=full to the top of cmd-build in coreos-assembler and then ran:

cosa build --version=39.20230918.dev.4 --force

Here's a snippet of the output:

fedora/x86_64/coreos/next-devel => 57bd45382dafb5dd6d6d92e9624f0e8a8ef4a82dbd4e8091921011152740c756
ostree diff commit from: 89217f114f3f98e58e524e5a8574e6331651caf48034ebad8026ebf55dfc04f1
ostree diff commit to:   57bd45382dafb5dd6d6d92e9624f0e8a8ef4a82dbd4e8091921011152740c756
Upgraded:
  grub2-common 1:2.06-95.fc39 -> 1:2.06-98.fc39
  grub2-efi-x64 1:2.06-95.fc39 -> 1:2.06-98.fc39
  grub2-pc 1:2.06-95.fc39 -> 1:2.06-98.fc39
  grub2-pc-modules 1:2.06-95.fc39 -> 1:2.06-98.fc39
  grub2-tools 1:2.06-95.fc39 -> 1:2.06-98.fc39
  grub2-tools-minimal 1:2.06-95.fc39 -> 1:2.06-98.fc39
  selinux-policy 38.27-1.fc39 -> 38.28-1.fc39
  selinux-policy-targeted 38.27-1.fc39 -> 38.28-1.fc39
Added:
  atheros-firmware-20230804-153.fc39.noarch
  brcmfmac-firmware-20230804-153.fc39.noarch
  mt7xxx-firmware-20230804-153.fc39.noarch
  realtek-firmware-20230804-153.fc39.noarch

New image input checksum: 2d02d92d71a1c113a48134e6c5f029eb1bb7a7466d6af4d2dc36ae5fbf611e4a
None
New build ID: 39.20230918.dev.4
info: This version of coreos-assembler is running code from outside the container.
Running:  rpm-ostree compose container-encapsulate --max-layers=50 --format-version=1 --repo=/srv/tmp/repo --label=coreos-assembler.image-config-checksum=d83c1228dbe97b9087a734e8bct
Reading packages... done
Building package mapping... done
21005 objects in 428 packages (279 source)
rpm size: 1289594980
Earliest changed package: shim-x64-15.6-2.x86_64 at 2022-07-07 19:36:06 UTC
1293 duplicates
Multiple owners:
  /usr/lib/.build-id/7a/b4825cde4b26a7f9435317457006b63fb06f23
  /usr/lib/.build-id/80/e67a24b674f7618b2232010a837849b3b28aad
  /usr/sbin/ip6tables
  /usr/sbin/ip6tables-restore
  /usr/sbin/ip6tables-save
  /usr/sbin/iptables
  /usr/sbin/iptables-restore
  /usr/sbin/iptables-save
  /usr/share/licenses/systemd/LICENSE.LGPL2.1
⠉ Generating container image...                                                                                                                                                     6
stack backtrace:
   0:     0x56355da55831 - <unknown>
   1:     0x56355da87e0f - <unknown>
   2:     0x56355da4fc19 - <unknown>
   3:     0x56355da55645 - <unknown>
   4:     0x56355da56c23 - <unknown>
   5:     0x56355da569b4 - <unknown>
   6:     0x56355da571a9 - <unknown>
   7:     0x56355da570a7 - <unknown>
   8:     0x56355da55c96 - <unknown>
   9:     0x56355da56df2 - <unknown>
  10:     0x56355d009803 - <unknown>
  11:     0x56355d0097c3 - <unknown>
  12:     0x56355d8a0006 - <unknown>
  13:     0x56355d89c437 - <unknown>
  14:     0x56355d89bf54 - <unknown>
  15:     0x56355d8fea46 - <unknown>
  16:     0x56355d4a635e - <unknown>
  17:     0x56355d5ae23c - <unknown>
  18:     0x56355d3f42f4 - <unknown>
  19:     0x56355d3f18b5 - <unknown>
  20:     0x56355d59a2bd - <unknown>
  21:     0x56355d1f018d - <unknown>
  22:     0x56355d191df8 - <unknown>
  23:     0x56355d19689e - <unknown>
  24:     0x56355d18fa14 - <unknown>
  25:     0x56355d598fcc - <unknown>
  26:     0x56355d11fb01 - <unknown>
  27:     0x56355d12c953 - <unknown>
  28:     0x56355d10d52c - <unknown>
  29:     0x56355da0bc70 - <unknown>
  30:     0x56355d9fc8aa - <unknown>
  31:     0x56355d9ebb39 - <unknown>
  32:     0x56355da5e1a5 - <unknown>
  33:     0x7f3ca2707947 - start_thread
  34:     0x7f3ca278d870 - __clone3
  35:                0x0 - <unknown>
/usr/lib/coreos-assembler/cmdlib.sh: line 22: 51766 Aborted                 (core dumped) "$@"
failed to execute cmd-build: exit status 134
[coreos-assembler]$ 

[cgwalters]

You need the debuginfo I think to get a useful stack dnf debuginfo-install rpm-ostree.

That said I bet this is fixed by ostreedev/ostree-rs-ext#531 which should be inbound soon.

[dustymabe]

Here's the trace with more info:

New build ID: 39.20230918.dev.5
info: This version of coreos-assembler is running code from outside the container.
Running:  rpm-ostree compose container-encapsulate --max-layers=50 --format-version=1 --repo=/srv/tmp/repo --label=coreos-assembler.image-config-checksum=d83c1228dbe97b9087a734e8bca0c989f78ab4a224748d68cbf7070ae28260e8 --label=coreos-at
Reading packages... done
Building package mapping... done
21005 objects in 428 packages (279 source)
rpm size: 1289594980
Earliest changed package: shim-x64-15.6-2.x86_64 at 2022-07-07 19:36:06 UTC
1293 duplicates
Multiple owners:
  /usr/lib/.build-id/7a/b4825cde4b26a7f9435317457006b63fb06f23
  /usr/lib/.build-id/80/e67a24b674f7618b2232010a837849b3b28aad
  /usr/sbin/ip6tables
  /usr/sbin/ip6tables-restore
  /usr/sbin/ip6tables-save
  /usr/sbin/iptables
  /usr/sbin/iptables-restore
  /usr/sbin/iptables-save
  /usr/share/licenses/systemd/LICENSE.LGPL2.1
⠙ Generating container image...                                                                                                                                                                                                            6
stack backtrace:
⠒ Generating container image...                                                                                                                                                                                                            4
   1:     0x562f7e8e7e0f - core::fmt::write::h782fd528ba4df57e
   2:     0x562f7e8afc19 - std::io::Write::write_fmt::hc55470322289b446
   3:     0x562f7e8b5645 - std::sys_common::backtrace::print::hea2be723c242d361
   4:     0x562f7e8b6c23 - std::panicking::default_hook::{{closure}}::h188a23319e8bb5bb
   5:     0x562f7e8b69b4 - std::panicking::default_hook::hbed06604daa300a0
   6:     0x562f7e8b71a9 - std::panicking::rust_panic_with_hook::hd80a162dfb66ecba
   7:     0x562f7e8b70a7 - std::panicking::begin_panic_handler::{{closure}}::h5c1bf0aff86cae10
   8:     0x562f7e8b5c96 - std::sys_common::backtrace::__rust_end_short_backtrace::hea91b7523222cf05
   9:     0x562f7e8b6df2 - rust_begin_unwind
  10:     0x562f7de69803 - core::panicking::panic_fmt::h1944e59e79040472
  11:     0x562f7de697c3 - core::option::expect_failed::h1a9691980f6b3519
  12:     0x562f7e700006 - core::option::Option<T>::expect::h15b1abe104be8c93
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/option.rs:898:21
  13:     0x562f7e700006 - <std::collections::hash::map::HashMap<K,V,S> as core::ops::index::Index<&Q>>::index::h17e9819318e9f017
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/collections/hash/map.rs:1339:23
  14:     0x562f7e700006 - ostree_ext::chunking::basic_packing_with_prior_build::ha6fe557868a9552a
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:606:26
  15:     0x562f7e700006 - ostree_ext::chunking::basic_packing::h4c6a523923c539a3
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:643:16
  16:     0x562f7e6fc437 - ostree_ext::chunking::Chunking::process_mapping::h52165c57f9670c5e
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:321:23
  17:     0x562f7e6fbf54 - ostree_ext::chunking::Chunking::from_mapping::h9d24b8b660238cd7
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:274:9
  18:     0x562f7e75ea46 - ostree_ext::container::encapsulate::build_oci::{{closure}}::{{closure}}::h5352237c0641f8fb
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:235:13
  19:     0x562f7e75ea46 - core::option::Option<T>::map::hf0f72aa4d0d878d6
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/option.rs:1075:29
  20:     0x562f7e75ea46 - ostree_ext::container::encapsulate::build_oci::{{closure}}::hdc3049d5b4992d3a
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:234:10
  21:     0x562f7e75ea46 - ostree_ext::container::encapsulate::build_oci::haabb2b83cc54371b
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:181:1
  22:     0x562f7e30635e - ostree_ext::container::encapsulate::build_impl::{{closure}}::{{closure}}::h9e2cbb5e5fac5ede
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:351:23
  23:     0x562f7e30635e - ostree_ext::container::encapsulate::build_impl::{{closure}}::hba9382b86d7f83c0
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:319:1
  24:     0x562f7e30635e - ostree_ext::container::encapsulate::encapsulate::{{closure}}::h530c78097c426280
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:427:6
  25:     0x562f7e30635e - rpmostree_rust::container::container_encapsulate::{{closure}}::{{closure}}::hf1c68d755bcc3b03
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/container.rs:462:14
  26:     0x562f7e30635e - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h556ca516670baecf
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/park.rs:283:63
  27:     0x562f7e30635e - tokio::runtime::coop::with_budget::h969444da29c03ef8
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/coop.rs:107:5
  28:     0x562f7e30635e - tokio::runtime::coop::budget::h85c8bcb3e72d93ad
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/coop.rs:73:5
  29:     0x562f7e30635e - tokio::runtime::park::CachedParkThread::block_on::hd9aeba08c8ab2b3d
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/park.rs:283:31
  30:     0x562f7e40e23c - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h81761356eb88fd86
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/context/blocking.rs:66:9
  31:     0x562f7e40e23c - tokio::runtime::handle::Handle::block_on::{{closure}}::h05308ce9c38f1020
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/handle.rs:308:22
  32:     0x562f7e40e23c - tokio::runtime::context::runtime::enter_runtime::hf2efc779f83c5cd9
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/context/runtime.rs:65:16
  33:     0x562f7e2542f4 - tokio::runtime::handle::Handle::block_on::h860080e4c9903c38
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/handle.rs:307:9
  34:     0x562f7e2542f4 - rpmostree_rust::container::container_encapsulate::{{closure}}::h85568d09fa351c99
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/container.rs:452:16
  35:     0x562f7e2542f4 - rpmostree_rust::progress::progress_task::h40e691924018631b
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/progress.rs:14:5
  36:     0x562f7e2518b5 - rpmostree_rust::container::container_encapsulate::hd13006244758f0b3
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/container.rs:451:18
  37:     0x562f7e3fa2bd - rpmostree_rust::ffi::_::__container_encapsulate::__container_encapsulate::he29f2d21c1e2afad
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/lib.rs:176:12
  38:     0x562f7e3fa2bd - rpmostree_rust::ffi::_::__container_encapsulate::{{closure}}::h09b25220de6440a5
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/lib.rs:47:1
  39:     0x562f7e3fa2bd - cxx::unwind::prevent_unwind::h50ad9b4fb162d8b3
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/cxx/src/unwind.rs:23:15
  40:     0x562f7e3fa2bd - rpmostreecxx$cxxbridge1$container_encapsulate
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/lib.rs:176:66
  41:     0x562f7e05018d - _ZN12rpmostreecxx21container_encapsulateEN4rust10cxxbridge13VecINS1_6StringEEE
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rpmostree-cxxrs.cxx:3629:79
  42:     0x562f7e05018d - rpmostree_compose_builtin_container_encapsulate
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/src/app/rpmostree-compose-builtin-tree.cxx:1773:3
  43:     0x562f7dff1df8 - rpmostree_handle_subcommand
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/src/app/libmain.cxx:442:33
  44:     0x562f7dff689e - _ZN12rpmostreecxx14rpmostree_mainEN4rust10cxxbridge15SliceIKNS1_3StrEEE
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/src/app/libmain.cxx:548:20
  45:     0x562f7dfefa14 - operator()
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rpmostree-cxxrs.cxx:2947:56
  46:     0x562f7dfefa14 - trycatch<rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>, int32_t*)::<lambda()>, rust::cxxbridge1::detail::Fail>
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rpmostree-cxxrs.cxx:1280:10
  47:     0x562f7dfefa14 - rpmostreecxx$cxxbridge1$rpmostree_main
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rpmostree-cxxrs.cxx:2945:32
  48:     0x562f7e3f8fcc - rpmostree_rust::ffi::rpmostree_main::hea63a89bb5defc04
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/lib.rs:800:12
  49:     0x562f7df7fb01 - rpm_ostree::inner_async_main::{{closure}}::{{closure}}::hf49ab9e1f8dc0175
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/rust/src/main.rs:44:25
  50:     0x562f7df8c953 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h343964e60de0fb76
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/blocking/task.rs:42:21
  51:     0x562f7df8c953 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h70e087711634e6a3
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/core.rs:311:17
  52:     0x562f7df8c953 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h0c9571d619ebbe20
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/loom/std/unsafe_cell.rs:14:9
  53:     0x562f7df8c953 - tokio::runtime::task::core::Core<T,S>::poll::ha4626eaece4b21f3
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/core.rs:300:30
  54:     0x562f7df6d52c - tokio::runtime::task::harness::poll_future::{{closure}}::h01ec9430916dbb73
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:476:19
  55:     0x562f7df6d52c - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdc20e16d18e0e779
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/panic/unwind_safe.rs:271:9
  56:     0x562f7df6d52c - std::panicking::try::do_call::hc174a14f2a7596b1
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:500:40
  57:     0x562f7df6d52c - std::panicking::try::h3656c34df4c0546c
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:464:19
  58:     0x562f7df6d52c - std::panic::catch_unwind::hadde4837f441f7c9
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panic.rs:142:14
  59:     0x562f7df6d52c - tokio::runtime::task::harness::poll_future::hcf3155cf9dda4d10
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:464:18
  60:     0x562f7df6d52c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h07fc23fa6c7ab3ff
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:198:27
  61:     0x562f7df6d52c - tokio::runtime::task::harness::Harness<T,S>::poll::hf15fb418432c2977
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:152:15
  62:     0x562f7e86bc70 - tokio::runtime::task::raw::RawTask::poll::hf7008478282c05d2
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/raw.rs:200:18
  63:     0x562f7e86bc70 - tokio::runtime::task::UnownedTask<S>::run::h1a0ccf56d454e12a
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/task/mod.rs:437:9
  64:     0x562f7e86bc70 - tokio::runtime::blocking::pool::Task::run::h21e698b4df40e3fb
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:159:9
  65:     0x562f7e86bc70 - tokio::runtime::blocking::pool::Inner::run::he1cc2045475ea45c
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:513:17
  66:     0x562f7e85c8aa - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h16390987f26c42d9
                               at /usr/src/debug/rpm-ostree-2023.7-1.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:471:13
  67:     0x562f7e85c8aa - std::sys_common::backtrace::__rust_begin_short_backtrace::hfd9f6ec8d3863237
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:135:18
  68:     0x562f7e84bb39 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hff7accc10ce91c85
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/thread/mod.rs:529:17
  69:     0x562f7e84bb39 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h5276f7a182657ddf
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/panic/unwind_safe.rs:271:9
  70:     0x562f7e84bb39 - std::panicking::try::do_call::h4ebf3de1efa9fc54
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:500:40
  71:     0x562f7e84bb39 - std::panicking::try::h984427f1a6c7580b
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:464:19
  72:     0x562f7e84bb39 - std::panic::catch_unwind::h3d17af7745c2aad8
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panic.rs:142:14
  73:     0x562f7e84bb39 - std::thread::Builder::spawn_unchecked_::{{closure}}::h436c092f5c135db1
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/thread/mod.rs:528:30
  74:     0x562f7e84bb39 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hcc1c370a886261bf
                               at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/ops/function.rs:250:5
  75:     0x562f7e8be1a5 - std::sys::unix::thread::Thread::new::thread_start::h3b8d48385754cce6
  76:     0x7fe14772e947 - start_thread
  77:     0x7fe1477b4870 - __clone3
  78:                0x0 - <unknown>
/usr/lib/coreos-assembler/cmdlib.sh: line 22: 69185 Aborted                 (core dumped) "$@"
failed to execute cmd-build: exit status 134

[cgwalters]

Yeah looks like the same bug

[cgwalters]

This should be fixed in git main; closing, please reopen if you can reproduce after the forthcoming release still.