From aad59c6b42c57bb93a90d423b8f1ae460d3ff577 Mon Sep 17 00:00:00 2001 From: James Munns Date: Sat, 9 Feb 2019 23:27:16 +0100 Subject: [PATCH 01/14] Start working on std aware cargo PR --- text/0000-std-aware-cargo.md | 108 +++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 text/0000-std-aware-cargo.md diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md new file mode 100644 index 00000000000..886c005e4c1 --- /dev/null +++ b/text/0000-std-aware-cargo.md @@ -0,0 +1,108 @@ +- Feature Name: cargo_the_std_awakens +- Start Date: 2018-02-09 +- RFC PR: (leave this empty) +- Rust Issue: (leave this empty) + +# Summary +[summary]: #summary + +Currently, the `core` and `std` components of Rust are handled in a different way than Cargo handles other crate dependencies. This causes issues for non-mainstream targets, such as WASM, Embedded, and new not-yet-tier-1 targets. The following RFC proposes a roadmap to address these concerns in a consistent and incremental process. + +# Motivation +[motivation]: #motivation + +In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served fairly well for the bulk of users, however there are a number of less common, but not esoteric uses, that are not well served by this approach. Examples include: + +* Supporting new/arbitrary targets, such as those defined by a ".json" file +* Making modifications to `core` or `std` through use of feature flags +* Users who would like to make different optimizations to `core` or `std`, such as `opt-level = 'z'`, with `panic = "abort"` + +Previously, these needs were somewhat addressed by the external tool [xargo], which managed the recompilation of these dependencies when necessary. However, this tool has become [deprecated], and even when supported, required a nightly version of the compiler for all operation. + +This approach has [gathered support] from various [rust team members], and this RFC aims to take inspiration from tools and workflows like [xargo], and integrate them into Cargo itself. + +[xargo]: https://github.com/japaric/xargo +[deprecated]: https://github.com/japaric/xargo/issues/193 +[gathered support]: https://github.com/japaric/xargo/issues/193#issuecomment-359180429 +[rust team members]: https://www.ncameron.org/blog/cargos-next-few-years/ + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: + +- Introducing new named concepts. +- Explaining the feature largely in terms of examples. +- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. + +For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that rust sometimes intentionally diverges from common language features. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how the this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From f87ac0a0deb9897cd29107aafec839c0e681cbd5 Mon Sep 17 00:00:00 2001 From: James Munns Date: Sun, 10 Feb 2019 01:01:14 +0100 Subject: [PATCH 02/14] Fill out most of the technical content --- text/0000-std-aware-cargo.md | 204 +++++++++++++++++++++++++++++++---- 1 file changed, 184 insertions(+), 20 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 886c005e4c1..44fdfcd5f34 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -29,19 +29,24 @@ This approach has [gathered support] from various [rust team members], and this # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: +This proposal aims to make `core` and `std` feel a little bit less like a special case compared to other dependencies to the end users of Cargo. This will be achieved by using idioms used for interacting, configuring, modifying, and patching other crates in a similar method for `core` and `std`. -- Introducing new named concepts. -- Explaining the feature largely in terms of examples. -- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. +This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the Reference Level Explanation. + +1. Allow developers of root crates to recompile `core` (and `compiler-builtins`) when their desired target does not match one available as a `rustup target add` target, without the usage of a nightly compiler. This version of `core` would be built from the same source files used to build the current version of `rustc`/`cargo`. +2. Introduce the concept of "stable features" for `core`, which allow the end user to influence the behavior of their custom version of `core`, without the use of a nightly compiler. +3. Extend the new behaviors described in step 1 and 2 for `std` (and `alloc`). +4. Allow the user to provide their own custom source versions of `core` and `std`, allowing for deep customizations when necessary. This will require a nightly version of the compiler. + +As a new concept, the items above propose the existence of "stable features" for `core` and `std`. These features would be considered stable with the same degree of guarantees made for stability in the rest of the language. These features would allow configuration of certain functionalities of `core` or `std`, in a way decided at compile time. + +For example, we could propose a feature called `force-tiny-fmt`, which would use different algorithms to implement `fmt` for use on resource constrained systems. The developer of the root crate would be able to choose the default behavior, or the `force-tiny-fmt` behavior while still retaining the ability of using a stable compiler. -For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation +``` This is the technical portion of the RFC. Explain the design in sufficient detail that: - Its interaction with other features is clear. @@ -49,6 +54,176 @@ This is the technical portion of the RFC. Explain the design in sufficient detai - Corner cases are dissected by example. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +``` + +A reference-level explanation is made for each of the items enumerated above. + +## 1 - Allow developers of root crates to recompile `core` + +### Use Case + +For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of `core` for any target that can be specified as a valid target json format. + +This functionality would be possible even with the use of a stable compiler. + +Users of a nightly compiler would be able to set compile time feature flags for `core` through settings made in their `Cargo.toml`. + +### Caveats + +For users of a stable compiler, it would not be possible to modify the source code contents of `core`, or change any compile time features of `core` from the defaults used when publishing pre-compiled versions of `core`. + +The source code used to build `core` would be the same as the compiler used for building the current project. + +### User Interaction + +When compiling for a non-standard target, users may specify their target using a json file, rather than a pre-defined target. + +For example, currently a user may cross-compile by specifying a target known by Rust: + +```sh +cargo build --target thumbv7em-none-eabihf +``` + +Users would also be able to specify a json file, by providing a path to the json file to be used. + +```sh +cargo build --target thumbv7em-freertos-eabihf.json +``` + +By using a json target file, Cargo will rebuild `core` for use in the current project. When rebuilding `core`, Cargo will respect the profile settings used in the current project, including settings such as `opt-level`. + +Users of a stable compiler would not be able to customize `core` outside of these profile settings. + +For users of a nightly compiler, compile time features of `core` may be specified using the same syntax used for other crate dependencies. These specified features may include unstable features. + +```toml +[dependencies.core] +default-features = false +features = [...] +``` + +It is not necessary to explicitly mention the dependency of `core`, unless changes to features are necessary. + +### Technical Implications + +#### Stabilization of JSON target format + +As the custom target json files would become part of the stable interface of Cargo. The format used by this JSON file must become stabilized, and further changes must be made in a backwards compatible way to guarantee stability. + +#### Building of `compiler-builtins` + +Currently, `compiler-builtins` contains components implemented in the C programming language. While these dependencies have been highly optimized, the use of them would require the builder of the root crate to also have a sane compilation environment for compilation in C. + +This RFC proposes instead to use the [pure rust implementation] when compiling for a custom target, removing the need for a C compiler. + +While this may have code size or performance implications, this would allow for maximum portability. + +[pure rust implementation]: https://github.com/rust-lang-nursery/compiler-builtins + +#### `RUSTC_BOOTSTRAP` + +It is necessary to use unstable features to build `core`. In order to allow users of a stable compiler to build `core`, we would set the `RUSTC_BOOTSTRAP` environment variable **ONLY** for the compilation of `core`. + +This should be considered sound, as stable users may not change the source used to build `core`, or the features used to build `core`. + +## 2 - Introduce the concept of "stable features" for `core` + +### Use Case + +In some cases, it may be desirable to modify `core` in set of predefined manners. For example, on some targets it may be necessary to have lighter weight machinery for `fmt`. + +This step would provide a path for stabilization of compile time `core` features, which would be a subset of all total compile time features of `core`. + +### Caveats + +Initially, the list of stable compile time features for `core` would be empty, as none of the current features have had an explicit decision to be stable or not. + +### User Interaction + +Compile time features for `core` may be specified using the same Cargo.toml syntax used for other crates. + +The syntax is the same when using `unstable` and `stable` features, however the former may only be used with a nightly compiler, and use of an `unstable` feature with a stable compiler would result in a compile time error. + +The syntax for these features would look as follows: + +```toml +[dependencies.core] +default-features = false +features = [...] +``` + +It is not necessary to explicitly mentioned the dependency of `core`, unless changes to features are necessary. + +### Technical Implications + +#### Path to stabilization + +The stabilization of a `core` feature flag would require a process similar to the stabilization of a feature in the language: + +* Any new feature begins as unstable, requiring a nightly compiler +* When the feature is sufficiently explored, an RFC/PR can be made to `libcore` to promote this feature to stable +* When this has been accepted, the feature of `core` may be used with the stable compiler. + +#### Implementation of Stable Features + +There would be some mechanism of differentiating between flags used to build core, sorting them into the groups `unstable` and `stable`. This RFC does not prescribe a certain way of implementation. + +## 3 - Extend the new behaviors described for `std` (and `alloc`) + +### Use Case + +Once the design and implications of the changes have been made for `core`, it will be necessary to extend these abilities for `std`, including components like `liballoc`. + +### Caveats + +In general, the same restrictions for building `core` will apply to building `std`. These include: + +* Users of the stable compiler must use the source used to build the current rust compiler +* Only compile time features considered `stable` may be used outside of nightly. Initially the list of `stable` features would be empty, and stabilizing these features would require a PR/RFC to `libstd`. + +### User Interaction + +The building of `std` would respect the current build profile, including + +The syntax for these features would look as follows: + +```toml +[dependencies.std] +default-features = false +features = [ + "force_alloc_system", +] +``` + +It is not necessary to explicitly mention the dependency of `std`, unless changes to features are necessary. + +### Technical Implications + +None beyond the technical implications listed for `core`. + +## 4 - Allow the user to provide their own custom source versions of `core` and `std` + +### Use Case + +This will allow users of a nightly compiler to provide a custom version of `core` and `std`, without requiring the recompilation of the compiler itself. + +### Caveats + +As stability guarantees cannot be made around modified versions of `core` or `std`, a nightly compiler must always be used. + +### User Interaction + +For this interaction, the existing `patch` syntax of Cargo.toml will be used. For example: + +```toml +[patch.sysroot] +core = { path = 'my/local/core' } +std = { git = 'https://github.com/example/std' } +``` + +### Technical Implications + +The `patch.sysroot` term will be introduced for patch when referring to components such as `std` and `core`. # Drawbacks [drawbacks]: #drawbacks @@ -65,19 +240,8 @@ Why should we *not* do this? # Prior art [prior-art]: #prior-art -Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: - -- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. - -This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. -If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. - -Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that rust sometimes intentionally diverges from common language features. +* https://github.com/rust-lang/rfcs/pull/1133 +* https://github.com/japaric/xargo # Unresolved questions [unresolved-questions]: #unresolved-questions From 62b548619a33483ba295ffad89c7d420297b55a2 Mon Sep 17 00:00:00 2001 From: James Munns Date: Sun, 10 Feb 2019 01:02:08 +0100 Subject: [PATCH 03/14] Remove template lines --- text/0000-std-aware-cargo.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 44fdfcd5f34..20542aaa786 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -46,16 +46,6 @@ For example, we could propose a feature called `force-tiny-fmt`, which would use # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -``` -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. -``` - A reference-level explanation is made for each of the items enumerated above. ## 1 - Allow developers of root crates to recompile `core` From 2cb4412246d311e7959459968237003becf67a4d Mon Sep 17 00:00:00 2001 From: James Munns Date: Thu, 14 Feb 2019 19:15:07 +0100 Subject: [PATCH 04/14] Address concerns, update text to fill it all in. --- text/0000-std-aware-cargo.md | 77 ++++++++++++++++++++++++------------ 1 file changed, 52 insertions(+), 25 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 20542aaa786..9e4b6eaba9e 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -11,7 +11,7 @@ Currently, the `core` and `std` components of Rust are handled in a different wa # Motivation [motivation]: #motivation -In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served fairly well for the bulk of users, however there are a number of less common, but not esoteric uses, that are not well served by this approach. Examples include: +In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served fairly well for the bulk of users, however there are a number of less common uses of Rust, that are not well served by this approach. Examples include: * Supporting new/arbitrary targets, such as those defined by a ".json" file * Making modifications to `core` or `std` through use of feature flags @@ -29,7 +29,7 @@ This approach has [gathered support] from various [rust team members], and this # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -This proposal aims to make `core` and `std` feel a little bit less like a special case compared to other dependencies to the end users of Cargo. This will be achieved by using idioms used for interacting, configuring, modifying, and patching other crates in a similar method for `core` and `std`. +This proposal aims to make `core` and `std` feel a little bit less like a special case compared to other dependencies to the end users of Cargo. This proposal aims to minimize the number of new concepts introduced to achieve this, by interacting, configuring, modifying, and patching `core` and `std` in a similar manner to other dependent crates. This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the Reference Level Explanation. @@ -52,7 +52,9 @@ A reference-level explanation is made for each of the items enumerated above. ### Use Case -For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of `core` for any target that can be specified as a valid target json format. +For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of `core` for any target that can be specified as a valid [target json format]. + +[target json format]: https://rust-lang.github.io/rfcs/0131-target-specification.html This functionality would be possible even with the use of a stable compiler. @@ -80,7 +82,12 @@ Users would also be able to specify a json file, by providing a path to the json cargo build --target thumbv7em-freertos-eabihf.json ``` -By using a json target file, Cargo will rebuild `core` for use in the current project. When rebuilding `core`, Cargo will respect the profile settings used in the current project, including settings such as `opt-level`. +In general, any of the following would prompt Cargo to recompile `core`, rather than use a pre-compiled version: + +* A custom target json is used +* The root crate has modified the feature flags of `core` +* The root crate has set certain profile settings, such as opt-level, etc. +* The root crate has specified a `patch.sysroot` (this is defined in a later section) Users of a stable compiler would not be able to customize `core` outside of these profile settings. @@ -94,6 +101,8 @@ features = [...] It is not necessary to explicitly mention the dependency of `core`, unless changes to features are necessary. +Cargo would use the source of `core` located in the user's `SYSROOT` directory. This source code would be obtained in the same was as necessary today, through the use of `rustup component add rust-src`. If this component is missing, Cargo would exit with an error code, and would prompt the user to execute the command specified above. + ### Technical Implications #### Stabilization of JSON target format @@ -218,14 +227,26 @@ The `patch.sysroot` term will be introduced for patch when referring to componen # Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +This RFC introduces new concepts to the use of Rust and Cargo, and could be confusing for current users of Rust who have not had to consider changes to `core` or `std` previously. However, in the normal case, most users are unlikely to need these settings, while they allow users that DO need to make changes to control important steps of the build process. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? +> Why is this design the best in the space of possible designs? + +This approach borrows from existing behaviors used by Cargo to allow configuration of `core` and `std`, as if they were a regular crate dependency. + +This approach also offers an approach that can be developed and applied incrementally, allowing for time to find coner cases not considered by this RFC + +> What other designs have been considered and what is the rationale for not choosing them? + +To the author of this RFC's knowledge, there are no other open designs, other than the use tools that wrap Cargo entirely, such as [xargo]. + +[xargo]: https://github.com/japaric/xargo + +> What is the impact of not doing this? + +By not doing this, Rust will continue to be difficult to use for users and platforms "on the edge", such as new platform developers or embedded and WASM users. # Prior art [prior-art]: #prior-art @@ -236,27 +257,33 @@ Why should we *not* do this? # Unresolved questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? +## How are dependencies for `core` and `std` specified? + +For example in a `no_core` or `no_std` crate, how would we tell Cargo **not** to build the `core` and/or `std` dependencies? + +## Should `std` be rebuilt if `core` is rebuilt? + +Is it necessary to rebuild `std` using the customized `core`, even if no changes to `std` are necessary? + +## Should Cargo obtain or verify the source code for `libcore` or `libstd`? + +Right now we depend on `rustup` to obtain the correct source code for these libraries, and we rely on the user not to tamper with the contents. Are these reasonable decisions? + +## Should the custom built `libcore` and `libstd` reside locally or globally? + +e.g., should the build artifacts be placed in `target/`, only usable by this project, or in `.cargo/`, to be possibly reused by multiple projects, if they happen to have the same settings? + +## How do we handle `libcore` and `libstd`'s `Cargo.lock` file? + +Right now these are built using the global lock file in `rust-lang/rust`. Should this always be true? How should Cargo handle this gracefully? # Future possibilities [future-possibilities]: #future-possibilities -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how the this all fits into the roadmap for the project -and of the relevant sub-team. +## Unified `core` and `std` -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. +With the mechanisms specified above, it could be possible to remove the concept of `core` and `std` from the user, leaving only `core`. -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. +By using stable feature flags for `std`, we could say that `std` as a crate with `default-features = false` would essentially be `no_core`, or with `features = ["core"]`, we would be the same as `no_std`. -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. +This abstraction may not map to the actual implementation of `libcore` or `libstd`, but instead be an abstraction layer for the end developer. From 436e4cf153560b4f64e1d8b4a986e9d8c97f4eed Mon Sep 17 00:00:00 2001 From: James Munns Date: Thu, 14 Feb 2019 19:37:23 +0100 Subject: [PATCH 05/14] How to handle unexpected rebuilds of `core` and `std` --- text/0000-std-aware-cargo.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 9e4b6eaba9e..9dd598a3b76 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -277,6 +277,14 @@ e.g., should the build artifacts be placed in `target/`, only usable by this pro Right now these are built using the global lock file in `rust-lang/rust`. Should this always be true? How should Cargo handle this gracefully? +## Should profile changes always prompt a rebuild of `core`/`std`? + +For example, if a user sets their debug build to use `opt-level = 'z'`, should this rebuild `core`/`std` to use that opt level? Or should an additional flag, such as `apply-to-sysroot` be required to opt-in to this behavior, unless otherwise needed? + +This could increase compile times for users that have set profile overrides, but have not previously needed a custom `core` or `std`. + +Another option in this area is to force the use of profile overrides, as specified by [RFC2822](https://github.com/rust-lang/rfcs/blob/master/text/2282-profile-dependencies.md). + # Future possibilities [future-possibilities]: #future-possibilities From 612151092c172b25e90bc13e4d7d1a590fec91f8 Mon Sep 17 00:00:00 2001 From: Mazdak Farrokhzad Date: Thu, 14 Feb 2019 20:00:50 +0100 Subject: [PATCH 06/14] Update text/0000-std-aware-cargo.md Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 9dd598a3b76..e9bf983f369 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -11,7 +11,7 @@ Currently, the `core` and `std` components of Rust are handled in a different wa # Motivation [motivation]: #motivation -In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served fairly well for the bulk of users, however there are a number of less common uses of Rust, that are not well served by this approach. Examples include: +In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served the bulk of users fairly well. However there are a number of less common uses of Rust, that are not well served by this approach. Examples include: * Supporting new/arbitrary targets, such as those defined by a ".json" file * Making modifications to `core` or `std` through use of feature flags From d46582e4bead9be01c46efbf6b3d0da589e81c3c Mon Sep 17 00:00:00 2001 From: Mazdak Farrokhzad Date: Thu, 14 Feb 2019 20:01:07 +0100 Subject: [PATCH 07/14] Update text/0000-std-aware-cargo.md Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index e9bf983f369..03d5324b96d 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -14,7 +14,7 @@ Currently, the `core` and `std` components of Rust are handled in a different wa In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served the bulk of users fairly well. However there are a number of less common uses of Rust, that are not well served by this approach. Examples include: * Supporting new/arbitrary targets, such as those defined by a ".json" file -* Making modifications to `core` or `std` through use of feature flags +* Modifying `core` or `std` through use of feature flags * Users who would like to make different optimizations to `core` or `std`, such as `opt-level = 'z'`, with `panic = "abort"` Previously, these needs were somewhat addressed by the external tool [xargo], which managed the recompilation of these dependencies when necessary. However, this tool has become [deprecated], and even when supported, required a nightly version of the compiler for all operation. From 0b74c739bb874402c42f739624eb062eab1d063d Mon Sep 17 00:00:00 2001 From: Mazdak Farrokhzad Date: Thu, 14 Feb 2019 20:02:47 +0100 Subject: [PATCH 08/14] Apply suggestions from code review Suggestion cleanups Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 03d5324b96d..fdf6044d438 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -19,7 +19,7 @@ In today's Rust environment, `core` and `std` are shipped as precompiled objects Previously, these needs were somewhat addressed by the external tool [xargo], which managed the recompilation of these dependencies when necessary. However, this tool has become [deprecated], and even when supported, required a nightly version of the compiler for all operation. -This approach has [gathered support] from various [rust team members], and this RFC aims to take inspiration from tools and workflows like [xargo], and integrate them into Cargo itself. +This approach has [gathered support] from various [rust team members]. This RFC aims to take inspiration from tools and workflows like [xargo], integrating them into Cargo itself. [xargo]: https://github.com/japaric/xargo [deprecated]: https://github.com/japaric/xargo/issues/193 @@ -31,7 +31,7 @@ This approach has [gathered support] from various [rust team members], and this This proposal aims to make `core` and `std` feel a little bit less like a special case compared to other dependencies to the end users of Cargo. This proposal aims to minimize the number of new concepts introduced to achieve this, by interacting, configuring, modifying, and patching `core` and `std` in a similar manner to other dependent crates. -This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the Reference Level Explanation. +This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the [Reference Level Explanation][reference-level-explanation]. 1. Allow developers of root crates to recompile `core` (and `compiler-builtins`) when their desired target does not match one available as a `rustup target add` target, without the usage of a nightly compiler. This version of `core` would be built from the same source files used to build the current version of `rustc`/`cargo`. 2. Introduce the concept of "stable features" for `core`, which allow the end user to influence the behavior of their custom version of `core`, without the use of a nightly compiler. @@ -121,7 +121,7 @@ While this may have code size or performance implications, this would allow for #### `RUSTC_BOOTSTRAP` -It is necessary to use unstable features to build `core`. In order to allow users of a stable compiler to build `core`, we would set the `RUSTC_BOOTSTRAP` environment variable **ONLY** for the compilation of `core`. +It is necessary to use unstable features to build `core`. To allow users of a stable compiler to build `core`, we would set the `RUSTC_BOOTSTRAP` environment variable **ONLY** for the compilation of `core`. This should be considered sound, as stable users may not change the source used to build `core`, or the features used to build `core`. From 45d84e1268d5f72a929ceb2720f417800c2269b9 Mon Sep 17 00:00:00 2001 From: James Munns Date: Thu, 14 Feb 2019 20:30:40 +0100 Subject: [PATCH 09/14] No targets, only `master`. --- text/0000-std-aware-cargo.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index fdf6044d438..67e3684dc52 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -295,3 +295,9 @@ With the mechanisms specified above, it could be possible to remove the concept By using stable feature flags for `std`, we could say that `std` as a crate with `default-features = false` would essentially be `no_core`, or with `features = ["core"]`, we would be the same as `no_std`. This abstraction may not map to the actual implementation of `libcore` or `libstd`, but instead be an abstraction layer for the end developer. + +## Stop shipping pre-compiled `core` and `std` + +With the ability to build these crates on demand, we may want to decide not to ship `target` bundles for any users. + +This would come at a cost of increased compile times, at least for the first build, if the artifacts are cached globally. However it would remove a mental snag of having to sometimes run `rustup target add`, and confusion from some users why parts of `std` and `core` have different optimization settings (particularly for debug builds) when debugging. From 4b836a0c0d58a6ae72e51654783c0d063d03c68e Mon Sep 17 00:00:00 2001 From: James Munns Date: Sat, 16 Mar 2019 23:25:37 +0100 Subject: [PATCH 10/14] Final preparations --- text/0000-std-aware-cargo.md | 70 ++++++++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 19 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 67e3684dc52..fbd9f6ba4ca 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -6,20 +6,20 @@ # Summary [summary]: #summary -Currently, the `core` and `std` components of Rust are handled in a different way than Cargo handles other crate dependencies. This causes issues for non-mainstream targets, such as WASM, Embedded, and new not-yet-tier-1 targets. The following RFC proposes a roadmap to address these concerns in a consistent and incremental process. +As of Rust 1.33.0, the `core` and `std` components of Rust are handled in a different way than Cargo handles other crate dependencies. This causes issues for non-mainstream targets, such as WASM, Embedded, and new not-yet-tier-1 targets. The following RFC proposes a roadmap to address these concerns in a consistent and incremental process. # Motivation [motivation]: #motivation In today's Rust environment, `core` and `std` are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served the bulk of users fairly well. However there are a number of less common uses of Rust, that are not well served by this approach. Examples include: -* Supporting new/arbitrary targets, such as those defined by a ".json" file +* Supporting new/arbitrary targets, such as those defined by a custom target (".json") file * Modifying `core` or `std` through use of feature flags -* Users who would like to make different optimizations to `core` or `std`, such as `opt-level = 'z'`, with `panic = "abort"` +* Users who would like to make different optimizations to `core` or `std`, such as `opt-level = 's'`, with `panic = "abort"` Previously, these needs were somewhat addressed by the external tool [xargo], which managed the recompilation of these dependencies when necessary. However, this tool has become [deprecated], and even when supported, required a nightly version of the compiler for all operation. -This approach has [gathered support] from various [rust team members]. This RFC aims to take inspiration from tools and workflows like [xargo], integrating them into Cargo itself. +This approach has [gathered support] from various [rust team members]. This RFC aims to take inspiration from tools and workflows used by tools like [xargo], integrating them into Cargo itself. [xargo]: https://github.com/japaric/xargo [deprecated]: https://github.com/japaric/xargo/issues/193 @@ -33,8 +33,10 @@ This proposal aims to make `core` and `std` feel a little bit less like a specia This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the [Reference Level Explanation][reference-level-explanation]. +In this document, we use the term "root crate" to refer to the Rust project being built directly by Cargo. This crate contains the Cargo.toml used to guide the modifications described below. This would typically be a crate containing a binary application, or a standalone item, such as an `rlib`. + 1. Allow developers of root crates to recompile `core` (and `compiler-builtins`) when their desired target does not match one available as a `rustup target add` target, without the usage of a nightly compiler. This version of `core` would be built from the same source files used to build the current version of `rustc`/`cargo`. -2. Introduce the concept of "stable features" for `core`, which allow the end user to influence the behavior of their custom version of `core`, without the use of a nightly compiler. +2. Allow the usage of Cargo features with the `core` library, additionally introducing the concept of "stable features" for `core`, which allow the end user to influence the behavior of their custom version of `core` without the use of a nightly compiler. 3. Extend the new behaviors described in step 1 and 2 for `std` (and `alloc`). 4. Allow the user to provide their own custom source versions of `core` and `std`, allowing for deep customizations when necessary. This will require a nightly version of the compiler. @@ -52,9 +54,9 @@ A reference-level explanation is made for each of the items enumerated above. ### Use Case -For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of `core` for any target that can be specified as a valid [target json format]. +For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of `core` for any target that can be specified as a valid [custom target specification]. -[target json format]: https://rust-lang.github.io/rfcs/0131-target-specification.html +[custom target specification]: https://rust-lang.github.io/rfcs/0131-target-specification.html This functionality would be possible even with the use of a stable compiler. @@ -68,7 +70,13 @@ The source code used to build `core` would be the same as the compiler used for ### User Interaction -When compiling for a non-standard target, users may specify their target using a json file, rather than a pre-defined target. +When compiling for a non-standard target, users may specify their target using a target specification file, rather than a pre-defined target. + +> NOTE: The current target specification is described in JSON, and contains some +> implementation details regarding the use of LLVM as the compiler backend. This +> RFC does not prescribe any changes to the Target Specification format, and is +> intended to work with whatever the current/stable method of specifying a +> custom target is. For example, currently a user may cross-compile by specifying a target known by Rust: @@ -76,7 +84,7 @@ For example, currently a user may cross-compile by specifying a target known by cargo build --target thumbv7em-none-eabihf ``` -Users would also be able to specify a json file, by providing a path to the json file to be used. +Users would also be able to specify a target specification file, by providing a path to the file to be used. ```sh cargo build --target thumbv7em-freertos-eabihf.json @@ -84,7 +92,7 @@ cargo build --target thumbv7em-freertos-eabihf.json In general, any of the following would prompt Cargo to recompile `core`, rather than use a pre-compiled version: -* A custom target json is used +* A custom target specification is used * The root crate has modified the feature flags of `core` * The root crate has set certain profile settings, such as opt-level, etc. * The root crate has specified a `patch.sysroot` (this is defined in a later section) @@ -105,13 +113,13 @@ Cargo would use the source of `core` located in the user's `SYSROOT` directory. ### Technical Implications -#### Stabilization of JSON target format +#### Stabilization of a Target Specification Format -As the custom target json files would become part of the stable interface of Cargo. The format used by this JSON file must become stabilized, and further changes must be made in a backwards compatible way to guarantee stability. +As the custom target specifications (currently JSON) would become part of the stable interface of Cargo. The format used by this file must become stabilized, and further changes must be made in a backwards compatible way to guarantee stability. #### Building of `compiler-builtins` -Currently, `compiler-builtins` contains components implemented in the C programming language. While these dependencies have been highly optimized, the use of them would require the builder of the root crate to also have a sane compilation environment for compilation in C. +Currently, `compiler-builtins` contains components implemented in the C programming language. While these dependencies have been highly optimized, the use of them would require the builder of the root crate to also have a working compilation environment for compilation in C. This RFC proposes instead to use the [pure rust implementation] when compiling for a custom target, removing the need for a C compiler. @@ -151,7 +159,7 @@ default-features = false features = [...] ``` -It is not necessary to explicitly mentioned the dependency of `core`, unless changes to features are necessary. +It is not necessary to explicitly mentione the dependency of `core`, unless changes to features are necessary. ### Technical Implications @@ -182,7 +190,7 @@ In general, the same restrictions for building `core` will apply to building `st ### User Interaction -The building of `std` would respect the current build profile, including +The building of `std` would respect the current build profile, including optimization settings. The syntax for these features would look as follows: @@ -220,6 +228,10 @@ core = { path = 'my/local/core' } std = { git = 'https://github.com/example/std' } ``` +> NOTE: The use of `sysroot` as a category may be changed to a less loaded +> category name. This is likely an area for bikeshedding. `sysroot` will be +> used for the remainder of the document for consistency. + ### Technical Implications The `patch.sysroot` term will be introduced for patch when referring to components such as `std` and `core`. @@ -251,13 +263,19 @@ By not doing this, Rust will continue to be difficult to use for users and platf # Prior art [prior-art]: #prior-art -* https://github.com/rust-lang/rfcs/pull/1133 -* https://github.com/japaric/xargo +* [RFC1133] - This RFC from 2015 proposed making cargo aware of std. I still need to review in more detail to find the parts and syntax that may solve some open questions. +* [xargo] - This external tool was used to achieve a similar workflow as described above, limited to use with a nightly compiler +* [Cargo Issue 5002] - This issue proposed a syntax for explicit dependency on std +* [Cargo Issue 5003] - This issue discussed how to be backwards compatible with crates that don't explicitly mention std + +[RFC1133]: https://github.com/rust-lang/rfcs/pull/1133 +[Cargo Issue 5002]: https://github.com/rust-lang/cargo/issues/5002 +[Cargo Issue 5003]: https://github.com/rust-lang/cargo/issues/5003 # Unresolved questions [unresolved-questions]: #unresolved-questions -## How are dependencies for `core` and `std` specified? +## How are dependencies (or non-dependency) on `core` and `std` specified? For example in a `no_core` or `no_std` crate, how would we tell Cargo **not** to build the `core` and/or `std` dependencies? @@ -285,12 +303,26 @@ This could increase compile times for users that have set profile overrides, but Another option in this area is to force the use of profile overrides, as specified by [RFC2822](https://github.com/rust-lang/rfcs/blob/master/text/2282-profile-dependencies.md). +## Should providing a custom `core` or `std` require a nightly compiler? + +It is currently unknown whether it is possible to provide a custom version of `core` or `std` without unstable features, as there are some compiler intrinsics and "magic" that are necessary (the format macros and box keyword come to mind). + +I initially wrote the RFC in this manner, however I was later convinced this was not possible to do. + +I am of the opinion that if you could, then it should be allowed to use a stable compiler, but that might be too theoretical for this RFC. + +We could also move forward with the current restriction to nightly, and allow that to be lifted later by a follow-on RFC if this is possible and necessary. + +## Should we allow configurable `core` and `std` + +If we are to uphold stability guarantees for all configurations of `core` and `std`, this could require testing 2^(n+m) versions of Rust, where `n` is the number of `core` features, and `m` is the number of `std` features. This would have a negative impact on CI times. + # Future possibilities [future-possibilities]: #future-possibilities ## Unified `core` and `std` -With the mechanisms specified above, it could be possible to remove the concept of `core` and `std` from the user, leaving only `core`. +With the mechanisms specified above, it could be possible to remove the concept of `core` and `std` from the user, leaving only `std`. By using stable feature flags for `std`, we could say that `std` as a crate with `default-features = false` would essentially be `no_core`, or with `features = ["core"]`, we would be the same as `no_std`. From 5629ff15d24a4b74848ddfdca70eab2484b8c496 Mon Sep 17 00:00:00 2001 From: James Munns Date: Sat, 16 Mar 2019 23:30:41 +0100 Subject: [PATCH 11/14] update date --- text/0000-std-aware-cargo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index fbd9f6ba4ca..3eaec35e467 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -1,5 +1,5 @@ - Feature Name: cargo_the_std_awakens -- Start Date: 2018-02-09 +- Start Date: 2018-03-16 - RFC PR: (leave this empty) - Rust Issue: (leave this empty) From c368271d7c84ce1d4236490d971b9c504b070d43 Mon Sep 17 00:00:00 2001 From: Jake Goulding Date: Mon, 18 Mar 2019 15:09:03 +0100 Subject: [PATCH 12/14] Apply suggestions from code review Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 3eaec35e467..a9e9caa66e7 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -19,12 +19,12 @@ In today's Rust environment, `core` and `std` are shipped as precompiled objects Previously, these needs were somewhat addressed by the external tool [xargo], which managed the recompilation of these dependencies when necessary. However, this tool has become [deprecated], and even when supported, required a nightly version of the compiler for all operation. -This approach has [gathered support] from various [rust team members]. This RFC aims to take inspiration from tools and workflows used by tools like [xargo], integrating them into Cargo itself. +This approach has [gathered support] from various [Rust team members]. This RFC aims to take inspiration from tools and workflows used by tools like [xargo], integrating them into Cargo itself. [xargo]: https://github.com/japaric/xargo [deprecated]: https://github.com/japaric/xargo/issues/193 [gathered support]: https://github.com/japaric/xargo/issues/193#issuecomment-359180429 -[rust team members]: https://www.ncameron.org/blog/cargos-next-few-years/ +[Rust team members]: https://www.ncameron.org/blog/cargos-next-few-years/ # Guide-level explanation [guide-level-explanation]: #guide-level-explanation From c1234bfa5b31a431904442b4430fe78a0e4656d1 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Mon, 18 Mar 2019 15:09:53 +0100 Subject: [PATCH 13/14] Update text/0000-std-aware-cargo.md Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index a9e9caa66e7..1533a513672 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -109,7 +109,7 @@ features = [...] It is not necessary to explicitly mention the dependency of `core`, unless changes to features are necessary. -Cargo would use the source of `core` located in the user's `SYSROOT` directory. This source code would be obtained in the same was as necessary today, through the use of `rustup component add rust-src`. If this component is missing, Cargo would exit with an error code, and would prompt the user to execute the command specified above. +Cargo would use the source of `core` located in the user's `SYSROOT` directory. This source code would be obtained in the same way as necessary today, through the use of `rustup component add rust-src`. If this component is missing, Cargo would exit with an error code, and would prompt the user to execute the command specified above. ### Technical Implications From a9f822253f6a3f0d876a549491cf94902a70d2ba Mon Sep 17 00:00:00 2001 From: Jake Goulding Date: Mon, 18 Mar 2019 15:11:29 +0100 Subject: [PATCH 14/14] Apply suggestions from code review Co-Authored-By: jamesmunns --- text/0000-std-aware-cargo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-std-aware-cargo.md b/text/0000-std-aware-cargo.md index 1533a513672..00f3497a3f1 100644 --- a/text/0000-std-aware-cargo.md +++ b/text/0000-std-aware-cargo.md @@ -121,7 +121,7 @@ As the custom target specifications (currently JSON) would become part of the st Currently, `compiler-builtins` contains components implemented in the C programming language. While these dependencies have been highly optimized, the use of them would require the builder of the root crate to also have a working compilation environment for compilation in C. -This RFC proposes instead to use the [pure rust implementation] when compiling for a custom target, removing the need for a C compiler. +This RFC proposes instead to use the [pure Rust implementation] when compiling for a custom target, removing the need for a C compiler. While this may have code size or performance implications, this would allow for maximum portability. @@ -185,7 +185,7 @@ Once the design and implications of the changes have been made for `core`, it wi In general, the same restrictions for building `core` will apply to building `std`. These include: -* Users of the stable compiler must use the source used to build the current rust compiler +* Users of the stable compiler must use the source used to build the current Rust compiler * Only compile time features considered `stable` may be used outside of nightly. Initially the list of `stable` features would be empty, and stabilizing these features would require a PR/RFC to `libstd`. ### User Interaction