Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove package:cli_config & package:args dependencies in package:native_asset_cli #1598

Merged
merged 3 commits into from
Sep 25, 2024

Conversation

mkustermann
Copy link
Member

Originally the idea that probably led to this package:cli_config dependency was my thought that we should allow end-users to be able to override specific build config defines. For example to be able to say

# Override the cc compiler
% flutter build apk --build-config compiler.cc=<path-to-cc> ...

# Or set specific settings for a particular package
% flutter build apk --build-config package.bar.enable_asserts=true

But the idea was that the bundling tool will be responsible for changing the config.json by patching in those key/value pairs into the json tree before giving it to the hook/build.dart (the hook/build.dart isn't invoked by end users and therefore doesn't have to deal with the complexity - it can be encapsulated by the bundling tool). This would also allow an end user to make specific settings for a particular package and only that package will get it's config.json patched (not other packages)

// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

String getConfigArgument(List<String> arguments) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could actually decide that instead of making this based on arguments we set a DART_BUILD_HOOK_CONFIG environment variable containing the configuration.

@dcharkes
Copy link
Collaborator

environment variables are passed through: COMPILER__CC=... flutter build apk

Also, this is the way that the tests run on the Dart CI if I'm not mistaken. You might want to try to change the DEPS file to point to your branch on GitHub and run the pkg bots to verify if this doesn't break the tests on the Dart CI.

@mkustermann
Copy link
Member Author

I was wondering why CI tests are not running here. It was a test to see if something breaks :)

environment variables are passed through: COMPILER__CC=... flutter build apk

I'll update the package:native_assets_builder to start passing them (and for backwards compatibility - if needed - put the environment handling in for COMPILER__)

@dcharkes
Copy link
Collaborator

I was wondering why CI tests are not running here. It was a test to see if something breaks :)

We don't have infra to create a Dart SDK CL from GitHub actions here. And there are security concerns about a Dart SDK bot being triggered from GitHub. (Although now that third party contributors PRs no longer automatically run on the CI on GitHub, maybe that concern is gone.) I would have for us to have an automatic roll bot!

I'll update the package:native_assets_builder to start passing them

Are you saying you want to use package cli_config in native_assets_builder instead? We might want to avoid wanting to add handling for every environment key manually.

(and for backwards compatibility - if needed - put the environment handling in for COMPILER__)

👍

@mkustermann
Copy link
Member Author

So the original idea was the following

  • End users use bundling tools. They may want to have an ability to influence how a particular package is built (let's say I get a crash in icu4x rust code in my app - I may want to say flutter build --build-config package.icu4x.cpp_asserts=true (for package-defined flags) or flutter build --build-config package.icu4x.c_compiler.cflags+=-DEBUG (for entries in the json that the protocol already defines))
  • The bundling tool creates a config.json as it does now. Then it will patch any end-user overrides / extra defines into the config.json (in a limited fashion - e.g. we may not want to allow patching the top-level namespace in the json file).
    => This patching would be done in package:native_assets_builder (and flutter run/build, dart run/build / ... would just pass along the set of user defines)
  • The author of package:icu4x may - in the hook/build.dart check for such icu4x.cpp_assert flags. We can define where/how this is accessible from BuildConfig.
  • It would be for us to decide if we allow overriding/patching
    • anything in the json file (and users are responsible for onverriding/setting only things that make sense - e.g. not json[out_dir])
    • allow only overriding specific parts (e.g. compiler paths, ios/android/macos target api versions, ...) - basically a allow-list of keys
    • allow only setting non-protocol values (as with the icu4x.cpp_asserts example above)

Now we don't have to implement this right now. This can be added later on.

Are you saying you want to use package cli_config in native_assets_builder instead?

I don't know if it would make sense to put this json-patching mechanism into package:cli_config or somewhere else. Maybe it would fit by constructing a Config from environment variables, arguments and json and it can have a Config.generateUnifiedJson() or something like that. I'm not too familiar with the package. @dcharkes wdyt?

We might want to avoid wanting to add handling for every environment key manually.

It would be a general mechanism (as described above) but we would be in control to what extent we allow patching the config.json. This "to what extent" is basically
a) an allowlist of keys in the known protocol that we allow overriding - so that list is somewhat "manual" - but that's not a bad thing (e.g. it doesn't make sense to allow users with environment variables or --build-config flags to override json["out_dir"])
b) a section in config.json that can be completely user-defined (the user of bundling tools can pass and the hook/build.dart can use - and the code in the middle just passes it through - one could have a BuildConfig.userDefines or something else for that)

Since we don't have this general patching mechanism right now, the bundling tools don't offer those flags yet, I'd just do the patching manually for the allowlist (which we should have anyway - see a) above) of things we know about for now (which seems like COMPILER__ is the only one relevant atm).

@dcharkes
Copy link
Collaborator

package.icu4x.cpp_asserts=true

This will first require addressing #39. Currently, the only way is using environment vars. @mosuem is using this in his icu4x package for building from source or not. However, that might simply be inspecting the environment variable directly instead of using the Config class. Is that the case @mosuem?

Then it will patch any end-user overrides / extra defines into the config.json

How would we know which out of all environment variables to patch into the config.json? The idea of the Config class is that we don't have to know but that querying the config on arbitrary keys that we don't know about defaults to json and environment variables.

We can do this for keys we know, but we would mandate users using Platform.environment for variables that they define. And it would make the behavior for command-line arguments and environment variable diverge.

It would be for us to decide if we allow overriding/patching

a) an allowlist of keys in the known protocol that we allow overriding

Right, we don't want to allow overriding the output dir, and currently one can.

Config.generateUnifiedJson()

That might not work, because some config keys combine json/env-vars/cmd-args instead of override. That's currently done by passing a param to config.string(combine: true).

b) a section in config.json that can be completely user-defined (the user of bundling tools can pass and the hook/build.dart can use - and the code in the middle just passes it through - one could have a BuildConfig.userDefines or something else for that)

This is tracked in #39.

Since we don't have this general patching mechanism right now, the bundling tools don't offer those flags yet, I'd just do the patching manually for the allowlist (which we should have anyway - see a) above) of things we know about for now (which seems like COMPILER__ is the only one relevant atm).

SGTM.

As noted above this doesn't work with arbitrary environment keys that we don't know about, but we can worry about that later.

@dcharkes
Copy link
Collaborator

Note to self, requires landing #1592 first (which bumps the SDK constraint to a version that will only provide json and not yaml configs).

Base automatically changed from remove-old-serialization-format to main September 25, 2024 10:47
@mkustermann
Copy link
Member Author

How would we know which out of all environment variables to patch into the config.json? The idea of the Config class is that we don't have to know but that querying the config on arbitrary keys that we don't know about defaults to json and environment variables.

We can do this for keys we know, but we would mandate users using Platform.environment for variables that they define. And it would make the behavior for command-line arguments and environment variable diverge.

We don't want to blindly override anything in config.json (e.g. json['out_dir']) - so we want control. So there should be specific ways one can patch entries. Examples would be:

  • Explicit --build-config=<name-expr>=<value-expr> / -B<name-expr>=<value-expr> given to flutter build/run / dart build/run.
  • Explicit BUILD_CONFIG_<name-expr>=<value-expr> environment variables.
  • Explicit (and sticky - so one doesn't have to pass it every time one runs a command) .dart_tool/build_config_overrides.txt

Maybe we don't want to allow usage of environment variables at all. I find it already very weird, that our dart-lang/sdk test runner is setting C_COMPILER__OBJDUMP. This variable isn't recognized by dart/flutter build/run , neither by package:native_assets_builder nor by package:native_assets_cli - only a specific helper package for building C libraries - package:native_toolchain_c - knows about this. How can this package rely on this variable? It seems very fishy.

If we look at how flutter tool is implemented: There the flutter building/bundling tool is responsible for locating the right c compiler (e.g. from android ndk) and setting up the right config.json with the right c compiler. This is the way - with (in the future) explicit overriding capability by end users.

How would we know which out of all environment variables to patch into the config.json?

We want control, for example we may decide environment variables that start with BUILD_CONFIG should have special form that we recognize and use to patch the config.json.

Copy link

github-actions bot commented Sep 25, 2024

PR Health

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
native_assets_cli Breaking 0.9.0-wip 0.9.0-wip 0.9.0-wip ✔️
Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbols
License Headers ✔️
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/ffigen/example/libclang-example/generated_bindings.dart
pkgs/ffigen/example/shared_bindings/generate.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_shared_b_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/base_gen.dart
pkgs/ffigen/example/simple/generated_bindings.dart
pkgs/ffigen/lib/src/header_parser/clang_bindings/clang_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_decl_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_symbol_address_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_type_name_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_reserved_keyword_collision_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_comment_markup_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_dart_handle_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_enum_int_mimic_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_forward_decl_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_functions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_imported_types_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_native_func_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_opaque_dependencies_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_packed_structs_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_regress_384_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_struct_fptr_fields_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_unions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_varargs_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_cjson_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_libclang_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_sqlite_bindings.dart
pkgs/ffigen/test/native_test/_expected_native_test_bindings.dart
pkgs/jni/lib/src/third_party/generated_bindings.dart
pkgs/jni/lib/src/third_party/global_env_extensions.dart
pkgs/jni/lib/src/third_party/jni_bindings_generated.dart
pkgs/jnigen/android_test_runner/lib/main.dart
pkgs/jnigen/example/in_app_java/lib/android_utils.dart
pkgs/jnigen/example/kotlin_plugin/example/lib/main.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_bindings.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/pdfbox_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocument.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocumentInformation.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/_package.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/PDFTextStripper.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/_package.dart
pkgs/jnigen/lib/src/bindings/descriptor.dart
pkgs/jnigen/lib/src/elements/elements.g.dart
pkgs/jnigen/test/jackson_core_test/third_party/bindings/com/fasterxml/jackson/core/_package.dart
pkgs/jnigen/tool/command_runner.dart
pkgs/native_assets_builder/test_data/native_dynamic_linking/bin/native_dynamic_linking.dart
pkgs/swift2objc/lib/src/config.dart
pkgs/swift2objc/lib/src/generate_wrapper.dart
pkgs/swift2objc/lib/src/generator/_core/utils.dart
pkgs/swift2objc/lib/src/generator/generator.dart
pkgs/swift2objc/lib/src/generator/generators/class_generator.dart
pkgs/swift2objc/lib/src/parser/parsers/declaration_parsers/parse_initializer_declaration.dart
pkgs/swift2objc/lib/src/parser/parsers/declaration_parsers/parse_property_declaration.dart
pkgs/swift2objc/lib/src/transformer/_core/unique_namer.dart
pkgs/swift2objc/lib/src/transformer/_core/utils.dart
pkgs/swift2objc/lib/src/transformer/transformers/transform_property.dart
pkgs/swift2objc/test/unit/parse_initializer_param_test.dart
Package publish validation ✔️
Package Version Status
package:ffi 2.1.3 already published at pub.dev
package:ffigen 15.0.0-wip WIP (no publish necessary)
package:jni 0.12.0-wip WIP (no publish necessary)
package:jnigen 0.12.0-wip WIP (no publish necessary)
package:native_assets_cli 0.9.0-wip WIP (no publish necessary)
package:objective_c 2.1.0-wip WIP (no publish necessary)
package:swift2objc 0.0.1-wip WIP (no publish necessary)
package:swiftgen 0.0.1-wip WIP (no publish necessary)

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@dcharkes
Copy link
Collaborator

How can this package rely on this variable?

I guess in your preferred would it should have been NATIVE_TOOLCHAIN_C__C_COMPILER__OBJDUMP and be namespaced to the package. The corresponding cli arg would then be -Bnative_toolchain_c.c_compiler.objdump=....

Anyway, LGTM to this PR if it's green on a manual roll into the Dart SDK on the pkg bots.

Copy link
Collaborator

@dcharkes dcharkes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Please send a PR to roll into the Dart SDK after merging this.

@coveralls
Copy link

Coverage Status

coverage: 90.761% (-0.05%) from 90.809%
when pulling 2934f3b on remove-cli
into 54279dd on main.

@mkustermann
Copy link
Member Author

mkustermann commented Sep 25, 2024

I verified the pkg tests were failing without environment handling and added support for that specifically for recognizing C_COMPILER__{CC,LD,AR,ENV_SCRIPT,ENV_SCRIPT_ARGUMENTS}.

Though for now I'm not sure we should move this exact logic into package:native_assets_builder. I think we should instead consider getting rid of the C_COMPILER__ variable:

  • flutter doesn't set this variable
  • very unlikely external users set this for non-flutter uses native assets (or are there any?)
  • it's not obvious at all from the name that such a variable will control behavior of invocation of hook/build.dart scripts - at least we should use some prefix like DART_BUILD_HOOK_
  • we can later on have an actual general mechanism one can use to override entries in config.json.

The package:native_c_toolchain already supports discovering compilers from PATH maybe the test runner can just utilize that instead of these special variables

@mkustermann
Copy link
Member Author

Thanks, @dcharkes .

There's more refactorings I'm planning of doing, so I think we can wait with rolling this right now. Once there's meaningful changes that warrant rolling, we should publish a new version and
a) roll to dart-sdk via DEPS and
b) roll to flutter via published version

@mkustermann mkustermann merged commit 7f206e3 into main Sep 25, 2024
31 checks passed
@mkustermann mkustermann deleted the remove-cli branch September 25, 2024 12:20
@dcharkes
Copy link
Collaborator

flutter doesn't set this variable

You mean the CI? Flutter tools passes the c compiler in in the build config.

at least we should use some prefix like DART_BUILD_HOOK_

That's a good idea.

The package:native_c_toolchain already supports discovering compilers from PATH maybe the test runner can just utilize that instead of these special variables

That doesn't work for the ENV_SCRIPT and ENV_SCRIPT_ARGUMENTS. And for hermitic builds I don't believe the engprod team would be happy with adding the compilers to the path (which is what I requested initially). This is why we settled on env vars.

@dcharkes
Copy link
Collaborator

Thanks, @dcharkes .

There's more refactorings I'm planning of doing, so I think we can wait with rolling this right now. Once there's meaningful changes that warrant rolling, we should publish a new version and a) roll to dart-sdk via DEPS and b) roll to flutter via published version

I want to avoid risky changes blocking rolls of features others might work on this repo (Moritz or me), so I prefer more often rolls. Rolling into Flutter is more painful and requires publishing indeed. Rolling into the Dart SDK gives us at least 50% coverage of the embedders.

@mkustermann
Copy link
Member Author

mkustermann commented Sep 25, 2024

That doesn't work for the ENV_SCRIPT and ENV_SCRIPT_ARGUMENTS. And for hermitic builds I don't believe the engprod team would be happy with adding the compilers to the path (which is what I requested initially). This is why we settled on env vars.

These variables aren't setup in Dart CI infrastructure (neither in luci configuration, nor recipes, ...). Instead our test runner was modified to inject those variables when it runs tests (see pkg/test_runner/lib/src/configuration.dart). So we can probably convince engprod that we want extra things on PATH when running the native assets tests instead of injecting these special variables.

I want to avoid risky changes blocking rolls of features others might work on this repo (Moritz or me), so I prefer more often rolls. Rolling into Flutter is more painful and requires publishing indeed. Rolling into the Dart SDK gives us at least 50% coverage of the embedders.

Fair enough, I'll roll it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants