Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression on x86_64-apple-darwin between 1.0.93 and 1.0.100 #1170

Open
ijackson opened this issue Aug 1, 2024 · 6 comments
Open

Regression on x86_64-apple-darwin between 1.0.93 and 1.0.100 #1170

ijackson opened this issue Aug 1, 2024 · 6 comments

Comments

@ijackson
Copy link

ijackson commented Aug 1, 2024

Hi.

Since some time a little over a month ago, we have not been able to build with --target x86_64-apple-darwin. We see this:

The following warnings were emitted during compilation:
warning: [email protected]+zstd.1.5.6: osxcross: error: cannot find libc++ headers
warning: [email protected]+zstd.1.5.6: osxcross: error: while detecting target
warning: [email protected]+zstd.1.5.6: ToolExecError: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-march=westmere" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/arti/target/x86_64-apple-darwin/release/build/zstd-sys-9cc4c92c9feeab86/out/8957a4c8b028b07b-debug.o" "-c" "zstd/lib/common/debug.c" with args o64-clang did not execute successfully (status code exit status: 1).cargo:warning=osxcross: error: cannot find libc++ headers

The breakage occurs when we try to cargo update. We know that 1.0.93 works, and that 1.0.100 doesn't.

I'm afraid I haven't been able to narrow this down or bisect it properly, because it's in an environment I don't have local access to. The failure occurs for us in CI, with the image amd64/rust:1.76.0-alpine3.18.

If it would be helpful, I could bisect it to a particular cc version.

We do have some more information: logs from a failed build, and an MR in our tree proposing to pin cc, with some discussion.
@ijackson
Copy link
Author

ijackson commented Aug 1, 2024

Our ticket https://gitlab.torproject.org/tpo/core/arti/-/issues/1507

@NobodyXu
Copy link
Collaborator

NobodyXu commented Aug 1, 2024 •
edited
Loading

The only change we introduce in 1.0.94 is for xrun:

1.0.93...1.0.94

We changed --show-sdk-platform-version to --show-sdk-version.

It might be that the xrun you use can't handle that flag?

cc @BlackHoleFox

@BlackHoleFox
Copy link
Contributor

Not sure to start here, apologies. I'm not familiar with how osxcross works, its been years since I've used it myself :(

@ijackson Does the problem go away on cc 1.0.100 if you explicitly set like MACOSX_DEPLOYMENT_TARGET=10.12 in your CI env config? That would assist in narrowing down to xcrun being the problem case since it should outright skip calling it when someone explicitly configures the target version instead.

@ijackson
Copy link
Author

Hi. Thanks for looking. Maybe we can blunder through the forest together ? :-)

I ran such a test in our CI (using the gitlab feature to set an env var). It doesn't seem to have made any significant difference, although the logs are gigantic and I'm not sure how I should interpret them.

Test with cc 1.0.11 but not the extra env var

https://gitlab.torproject.org/Diziet/arti/-/jobs/636719

create_symlink x86_64-apple-darwin16-wrapper x86_64h-apple-darwin16-xcodebuild


Given SDK does not contain libc++ headers (-stdlib=libc++ test may fail)
You may want to re-package your SDK using 'tools/gen_sdk_package.sh' on macOS

testing i386-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)
testing x86_64-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)

testing i386-apple-darwin16-clang ... works
testing i386-apple-darwin16-clang++ ... works

testing x86_64h-apple-darwin16-clang ... works
testing x86_64h-apple-darwin16-clang++ ... works

testing x86_64-apple-darwin16-clang ... works
testing x86_64-apple-darwin16-clang++ ... works

Do not forget to add

�[32m/arti/osxcross/target/bin�[0m

to your PATH variable.

All done! Now you can use o32-clang(++) and o64-clang(++) like a normal compiler.

Example usage:

Example 1: CC=o32-clang ./configure --host=i386-apple-darwin16
Example 2: CC=i386-apple-darwin16-clang ./configure --host=i386-apple-darwin16
Example 3: o64-clang -Wall test.c -o test
Example 4: x86_64-apple-darwin16-strip -x test

[... much later ...]
   Compiling libsqlite3-sys v0.28.0
   Compiling matchers v0.1.0
The following warnings were emitted during compilation:

warning: [email protected]: osxcross: error: cannot find libc++ headers
warning: [email protected]: osxcross: error: while detecting target

error: failed to run custom build command for `libsqlite3-sys v0.28.0`

Caused by:
  process didn't exit successfully: `/arti/target/release/build/libsqlite3-sys-39dc4def12e47957/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBSQLITE3_SYS_USE_PKG_CONFIG
  cargo:include=/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsqlite3-sys-0.28.0/sqlite3
  cargo:rerun-if-changed=sqlite3/sqlite3.c
  cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
  cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
  cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
  cargo:rerun-if-env-changed=SQLITE_MAX_COLUMN
  cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
  TARGET = Some(x86_64-apple-darwin)
  OPT_LEVEL = Some(s)
  HOST = Some(x86_64-unknown-linux-musl)
  cargo:rerun-if-env-changed=CC_x86_64-apple-darwin
  CC_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CC_x86_64_apple_darwin
  CC_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = Some(o64-clang)
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(false)
  cargo:rerun-if-env-changed=CFLAGS_x86_64-apple-darwin
  CFLAGS_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_apple_darwin
  CFLAGS_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-march=westmere)
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target

  --- stderr


  error occurred: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-march=westmere" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" "-DHAVE_LOCALTIME_R" "-o" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/0343851c9008f270-sqlite3.o" "-c" "sqlite3/sqlite3.c" with args o64-clang did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...
The following warnings were emitted during compilation:

warning: [email protected]: osxcross: error: cannot find libc++ headers
warning: [email protected]: osxcross: error: while detecting target
warning: [email protected]: ToolExecError: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/lzma-sys-0.1.20" "-march=westmere" "-std=c99" "-pthread" "-DHAVE_CONFIG_H=1" "-o" "/arti/target/x86_64-apple-darwin/release/build/lzma-sys-dfbbc672800f8248/out/ebd7929902bc829f-tuklib_cpucores.o" "-c" "xz-5.2/src/common/tuklib_cpucores.c" with args o64-clang did not execute successfully (status code exit status: 1).cargo:warning=osxcross: error: cannot find libc++ headers

Test with cc 1.0.11 and additional MACOSX_DEPLOYMENT_TARGET setting

https://gitlab.torproject.org/Diziet/arti/-/jobs/637107

create_symlink x86_64-apple-darwin16-wrapper x86_64h-apple-darwin16-xcodebuild


Given SDK does not contain libc++ headers (-stdlib=libc++ test may fail)
You may want to re-package your SDK using 'tools/gen_sdk_package.sh' on macOS

testing i386-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)
testing x86_64-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)

testing i386-apple-darwin16-clang ... works
testing i386-apple-darwin16-clang++ ... works

testing x86_64h-apple-darwin16-clang ... works
testing x86_64h-apple-darwin16-clang++ ... works

testing x86_64-apple-darwin16-clang ... works
testing x86_64-apple-darwin16-clang++ ... works

Do not forget to add

�[32m/arti/osxcross/target/bin�[0m

to your PATH variable.

All done! Now you can use o32-clang(++) and o64-clang(++) like a normal compiler.

Example usage:

Example 1: CC=o32-clang ./configure --host=i386-apple-darwin16
Example 2: CC=i386-apple-darwin16-clang ./configure --host=i386-apple-darwin16
Example 3: o64-clang -Wall test.c -o test
Example 4: x86_64-apple-darwin16-strip -x test

[...much later...]
   Compiling libsqlite3-sys v0.28.0
   Compiling matchers v0.1.0
   Compiling proc-macro-crate v1.3.1
   Compiling proc-macro-crate v3.1.0
The following warnings were emitted during compilation:

warning: [email protected]: Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/9613648878214335138detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
warning: [email protected]: osxcross: error: cannot find libc++ headers
warning: [email protected]: osxcross: error: while detecting target
warning: [email protected]: osxcross: error: cannot find libc++ headers
warning: [email protected]: osxcross: error: while detecting target

error: failed to run custom build command for `libsqlite3-sys v0.28.0`

Caused by:
  process didn't exit successfully: `/arti/target/release/build/libsqlite3-sys-39dc4def12e47957/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBSQLITE3_SYS_USE_PKG_CONFIG
  cargo:include=/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsqlite3-sys-0.28.0/sqlite3
  cargo:rerun-if-changed=sqlite3/sqlite3.c
  cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
  cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
  cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
  cargo:rerun-if-env-changed=SQLITE_MAX_COLUMN
  cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
  TARGET = Some(x86_64-apple-darwin)
  OPT_LEVEL = Some(s)
  HOST = Some(x86_64-unknown-linux-musl)
  cargo:rerun-if-env-changed=CC_x86_64-apple-darwin
  CC_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CC_x86_64_apple_darwin
  CC_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = Some(o64-clang)
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:warning=Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/9613648878214335138detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(false)
  cargo:rerun-if-env-changed=CFLAGS_x86_64-apple-darwin
  CFLAGS_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_apple_darwin
  CFLAGS_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-march=westmere)
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target

  --- stderr


  error occurred: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-march=westmere" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" "-DHAVE_LOCALTIME_R" "-o" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/0343851c9008f270-sqlite3.o" "-c" "sqlite3/sqlite3.c" with args o64-clang did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...
The following warnings were emitted during compilation:

warning: [email protected]+zstd.1.5.6: Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/zstd-sys-c90573e828bfe880/out/4392794065767800547detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
warning: [email protected]+zstd.1.5.6: osxcross: error: cannot find libc++ headers
warning: [email protected]+zstd.1.5.6: osxcross: error: while detecting target

@ijackson
Copy link
Author

We have discovered that cached data (target/ directories) in our CI would sometimes make this job succeed, when it would otherwise fail. Apparently not everything about the inputs is captured by cargo's cache invalidation algorithm. I think it's likely that the version range I've quoted is wrong.

@NobodyXu
Copy link
Collaborator

It might be because we are missing some re-runs for files?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ijackson @BlackHoleFox @NobodyXu