From 7f3d678c7bc85d3ac1ab2d35e1cdb63d0c3fbf3d Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 25 May 2015 20:56:03 +0000 Subject: [PATCH 01/66] Copy in template --- text/0000-cargo-libstd-awareness.md | 30 +++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 text/0000-cargo-libstd-awareness.md diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md new file mode 100644 index 00000000000..80227156359 --- /dev/null +++ b/text/0000-cargo-libstd-awareness.md @@ -0,0 +1,30 @@ +- Feature Name: cargo_libstd_awareness +- Start Date: 2015-05-26 +- RFC PR: (leave this empty) +- Rust Issue: (leave this empty) + +# Summary + +One para explanation of the feature. + +# Motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Detailed design + +This is the bulk of the RFC. Explain the design in enough detail for somebody familiar +with the language to understand, and for somebody familiar with the compiler to implement. +This should get into specifics and corner-cases, and include examples of how the feature is used. + +# Drawbacks + +Why should we *not* do this? + +# Alternatives + +What other designs have been considered? What is the impact of not doing this? + +# Unresolved questions + +What parts of the design are still TBD? From 4c0ea2a6be55a94a43eff4cbb1fe8f0e735becbb Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 26 May 2015 05:59:41 +0000 Subject: [PATCH 02/66] First version I clarified and cleaned up some things, but force-pushed as - The ideas are the same, just the wording is changed - At the time of commit, I had not yet submitted the PR --- text/0000-cargo-libstd-awareness.md | 82 ++++++++++++++++++++++++++--- 1 file changed, 74 insertions(+), 8 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 80227156359..672e5ef5d08 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -5,26 +5,92 @@ # Summary -One para explanation of the feature. +Currently, all packages implicitly depend on libstd. This makes Cargo unsuitable for packages that +need a custom-built libstd, or otherwise depend on crates with the same names as libstd and the +crates behind the facade. The proposed fixes also open the door to a future were libstd can be +Cargoized. # Motivation -Why are we doing this? What use cases does it support? What is the expected outcome? +Bare-metal work cannot use a standard build of libstd. But since any crate built with Cargo can link +with a system-installed libstd if the target matches, using Cargo for such projects can be irksome +or impossible. + +Cargoizing libstd also generally simplifies the infrastructure, and makes cross compiling much +slicker, but that is a separate discussion. + +Finally, I first raised this issue here: https://github.com/rust-lang/Cargo/issues/1096 Also, there +are some (heavily bit-rotted) projects at https://github.com/RustOS-Fork-Holding-Ground that depend +on each other in the way this RFC would make much more feasible. # Detailed design -This is the bulk of the RFC. Explain the design in enough detail for somebody familiar -with the language to understand, and for somebody familiar with the compiler to implement. -This should get into specifics and corner-cases, and include examples of how the feature is used. +The current situation seems to be more of an accident of `rustc`'s pre-Cargo history than an +explicit design decision. Cargo passes the location and name of all depended on crates to `rustc`. +This is good because it means that that no undeclared dependencies on other Cargo packages can leak +through. However, it also passes in `--sysroot /path/to/some/libdir`, the directory being were +libstd is. This means packages are free to use libstd, the crates behind the facade, or none of the +above, with Cargo being none the wiser. + +The only new interface proposed is a boolean field to the package meta telling Cargo that the +package does not depend on libstd by default. This need not imply Rust's `no_std`, as one might want +to `use` their own build of libstd by default. To disambiguate, this field is called +q`no-implicit-deps`; please, go ahead and bikeshead the name. `no-implicit-deps` is false by +default to maintain compatibility with existing packages. + +The meaning of this flag is defined in 3 phases, where each phase extends the last. The idea being +is that while earlier phases are easier to implement, later phases yield a more elegant system. + +## Phase 1 + +Add a `--no-sysroot` flag to `rustc`, and pass that to `rustc` is the case that `no-implicit-deps` +is true. + +This hotfix is enough to allow us bare-metal devs to use Cargo for our own projects, but doesn't +suffice for creating an ecosystem of packages that depend on crates behind the facade but not libstd +itself. This is because the choices are all or nothing: Either one implicitly depends on libstd or +the crates behind the facade, or they don't depend on them at all. + +## Phase 2 + +Since, passing in a directory of crates is inherently more fragile than passing in a crate +itself, make Cargo use `--no-sysroot` in all cases. + +Cargo would special case package names corresponding to the crates behind the facade, such that if +the package don't exist, it would simply pass the corresponding system crate to `rustc`. I assume +the names are blacklisted on crates.io already, so by default the packages won't exist. But users +can use config files to extend the namespace so their own modded libstds can be used instead. Even +if they don't want to change libstd but just cross-compile it, this is frankly the easiest way as +Cargo will seemliest cross compile both their project and it's transitive dependencies. + +In this way we can put packages on crates.io that depend on the crates behind the facade. Some +packages that already exist, like liblog and libbitflags, should be given features that optionally +allow them to avoid libstd and just depend directly on the crates behind the facade they really +need. + +## Phase 3 + +If/when the standard library is built with Cargo and put on crates.io, all the specially-cased +package names can be treated normally, + +The standard library is be downloaded and built from crates.io. Or equivalently, Cargo comes with a +cache of that build, as Cargo should be able cache builds between projects at this point. Just as in +phase 2, `no-implicit-deps` just prevents libstd from implicitly being appended to the list of +dependencies. + +Again, to make this as least controversial as possible, this RFC does not propose outright that the +standard library should be Cargoized. This 3rd phases just describes how this feature would work +were that to happen. # Drawbacks -Why should we *not* do this? +I really don't know of any. Development for hosted environments would hardly be very affected. # Alternatives -What other designs have been considered? What is the impact of not doing this? +Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. # Unresolved questions -What parts of the design are still TBD? +There are multiple lists of dependencies for different things (e.g. tests), Should libstd be append +to all of them in phases 2 and 3? From eee7dbab8d03e3a9c25333f2784f96a5b7800772 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 26 May 2015 16:01:57 +0000 Subject: [PATCH 03/66] Fix were/where Typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 672e5ef5d08..8450066cb03 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -7,7 +7,7 @@ Currently, all packages implicitly depend on libstd. This makes Cargo unsuitable for packages that need a custom-built libstd, or otherwise depend on crates with the same names as libstd and the -crates behind the facade. The proposed fixes also open the door to a future were libstd can be +crates behind the facade. The proposed fixes also open the door to a future where libstd can be Cargoized. # Motivation From 9da1b0285c4cc6cd7f6c7db02500312d6770aeda Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 26 May 2015 18:10:28 +0000 Subject: [PATCH 04/66] Use true defaults to avoid double negatives as suggested by @Valloric --- text/0000-cargo-libstd-awareness.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8450066cb03..4f7d3919e96 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -35,16 +35,16 @@ above, with Cargo being none the wiser. The only new interface proposed is a boolean field to the package meta telling Cargo that the package does not depend on libstd by default. This need not imply Rust's `no_std`, as one might want to `use` their own build of libstd by default. To disambiguate, this field is called -q`no-implicit-deps`; please, go ahead and bikeshead the name. `no-implicit-deps` is false by -default to maintain compatibility with existing packages. +`implicit-deps`; please, go ahead and bikeshead the name. `implicit-deps` is true by default to +maintain compatibility with existing packages. The meaning of this flag is defined in 3 phases, where each phase extends the last. The idea being is that while earlier phases are easier to implement, later phases yield a more elegant system. ## Phase 1 -Add a `--no-sysroot` flag to `rustc`, and pass that to `rustc` is the case that `no-implicit-deps` -is true. +Add a `--use-sysroot=` flag to `rustc`, where true is the default. Make Cargo pass +`--use-sysroot=false` to `rustc` is the case that `implicit-deps` is false. This hotfix is enough to allow us bare-metal devs to use Cargo for our own projects, but doesn't suffice for creating an ecosystem of packages that depend on crates behind the facade but not libstd @@ -53,8 +53,8 @@ the crates behind the facade, or they don't depend on them at all. ## Phase 2 -Since, passing in a directory of crates is inherently more fragile than passing in a crate -itself, make Cargo use `--no-sysroot` in all cases. +Since, passing in a directory of crates is inherently more fragile than passing in a crate itself, +make Cargo use `--use-sysroot=false` in all cases. Cargo would special case package names corresponding to the crates behind the facade, such that if the package don't exist, it would simply pass the corresponding system crate to `rustc`. I assume @@ -75,7 +75,7 @@ package names can be treated normally, The standard library is be downloaded and built from crates.io. Or equivalently, Cargo comes with a cache of that build, as Cargo should be able cache builds between projects at this point. Just as in -phase 2, `no-implicit-deps` just prevents libstd from implicitly being appended to the list of +phase 2, `implicit-deps = false` just prevents libstd from implicitly being appended to the list of dependencies. Again, to make this as least controversial as possible, this RFC does not propose outright that the From 630ac97a964b126931bc484dec4fbcaefbe8ea0a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 28 May 2015 20:03:47 +0000 Subject: [PATCH 05/66] Typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 4f7d3919e96..911f36da78d 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -73,7 +73,7 @@ need. If/when the standard library is built with Cargo and put on crates.io, all the specially-cased package names can be treated normally, -The standard library is be downloaded and built from crates.io. Or equivalently, Cargo comes with a +The standard library is downloaded and built from crates.io. Or equivalently, Cargo comes with a cache of that build, as Cargo should be able cache builds between projects at this point. Just as in phase 2, `implicit-deps = false` just prevents libstd from implicitly being appended to the list of dependencies. From a7425e66a9001f044fcdf85402c652a7757aac9c Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 29 May 2015 05:48:10 +0000 Subject: [PATCH 06/66] Rewrite after talking to @alexcrichton on IRC. tl;dr: - Motivation is significantly expanded - Phase 1 is cut for being too half-assed --- text/0000-cargo-libstd-awareness.md | 178 ++++++++++++++++++---------- 1 file changed, 118 insertions(+), 60 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 911f36da78d..1ad4c4bffb5 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -5,92 +5,150 @@ # Summary -Currently, all packages implicitly depend on libstd. This makes Cargo unsuitable for packages that -need a custom-built libstd, or otherwise depend on crates with the same names as libstd and the -crates behind the facade. The proposed fixes also open the door to a future where libstd can be -Cargoized. +Currently, Cargo doesn't know whether packages depend on libstd. This makes Cargo unsuitable for +packages that need a cross-compiled or custom libstd, or otherwise depend on crates with the same +names as libstd and the crates behind the facade. The proposed fixes also open the door to a future +where libstd can be Cargoized. + # Motivation -Bare-metal work cannot use a standard build of libstd. But since any crate built with Cargo can link -with a system-installed libstd if the target matches, using Cargo for such projects can be irksome -or impossible. +First some background. The current situation seems to be more of an accident of `rustc`'s pre-Cargo +history than an explicit design decision. Cargo passes the location and name of all depended-on +crates to `rustc`. This method is good for a number of reasons stemming from its fine granularity, +such as: -Cargoizing libstd also generally simplifies the infrastructure, and makes cross compiling much -slicker, but that is a separate discussion. + - No undeclared dependencies can be used -Finally, I first raised this issue here: https://github.com/rust-lang/Cargo/issues/1096 Also, there -are some (heavily bit-rotted) projects at https://github.com/RustOS-Fork-Holding-Ground that depend -on each other in the way this RFC would make much more feasible. + - Conversely, `rustc` can warn against *unused* declared dependencies -# Detailed design + - Crate/symbol names are frobbed so that packages with the overlapping names don't conflict + + +However rather than passing in libstd and its deps, Cargo lets the compiler look for them as need in +the compiler's sysroot [specifically `/lib/`]. This is quite coarse in comparison, +and we loose all the advantages of the previous method: + + - Packages may link or not link against libs in that directory as they please, with Cargo being + none the wiser. + + - Cargo-built crates with the same name as those in there will collide, as the sysroot libs don't + have their names frobbed. -The current situation seems to be more of an accident of `rustc`'s pre-Cargo history than an -explicit design decision. Cargo passes the location and name of all depended on crates to `rustc`. -This is good because it means that that no undeclared dependencies on other Cargo packages can leak -through. However, it also passes in `--sysroot /path/to/some/libdir`, the directory being were -libstd is. This means packages are free to use libstd, the crates behind the facade, or none of the -above, with Cargo being none the wiser. + - Cross compiling may fail at build-time (as opposed to the much shorter + "gather-dependencies-time") because of missing packages -The only new interface proposed is a boolean field to the package meta telling Cargo that the -package does not depend on libstd by default. This need not imply Rust's `no_std`, as one might want -to `use` their own build of libstd by default. To disambiguate, this field is called -`implicit-deps`; please, go ahead and bikeshead the name. `implicit-deps` is true by default to -maintain compatibility with existing packages. -The meaning of this flag is defined in 3 phases, where each phase extends the last. The idea being -is that while earlier phases are easier to implement, later phases yield a more elegant system. +Cargo doesn't look inside the sysroot to see what is or isn't there, but it would hardly help if it +did, because it doesn't know what any package needs. Assuming all packages need libstd, for example, +means Cargo just flat-out won't build freestanding packages that just use libcore on a platform that +doesn't support libstd. -## Phase 1 +For an anecdote: in https://github.com/RustOS-Fork-Holding-Ground I tried to rig up Cargo to cross +compile libstd for me. Since I needed to use an unstable compiler anyways, it was possible in +principle to build absolutely everything I needed with the same `rustc` version. Because of some +trouble with Cargo and target JSONs, I didn't use a custom target specification, and just used +`x86_64-gnu-linux`, meaning that depending on platform I was compiling on, I may or may have been +cross-compiling. In the case where I wasn't, I couldn't complete the build because `rustc` +complained about the libstd I was building overlapping with the libstd in the sysroot. -Add a `--use-sysroot=` flag to `rustc`, where true is the default. Make Cargo pass -`--use-sysroot=false` to `rustc` is the case that `implicit-deps` is false. +For these reasons, most freestanding projects I know of avoid Cargo altogether, and just include +submodule rust and run make in that. Cargo can still be used if one manages to get the requisite +libraries in the sysroot. But this is a tedious operation that individual projects shouldn't need to +reimplement, and one that has serious security implications if the normal libstd is modified. -This hotfix is enough to allow us bare-metal devs to use Cargo for our own projects, but doesn't -suffice for creating an ecosystem of packages that depend on crates behind the facade but not libstd -itself. This is because the choices are all or nothing: Either one implicitly depends on libstd or -the crates behind the facade, or they don't depend on them at all. +The fundamental plan proposed in this RFC is to make sure that anything Cargo builds never blindly +links against libraries in the sysroot. This is achieved by making Cargo aware of all dependencies, +including those libstd or its backing crates. That way, these problems are avoided. -## Phase 2 +For the record, I first raised this issue [here](https://github.com/rust-lang/Cargo/issues/1096). -Since, passing in a directory of crates is inherently more fragile than passing in a crate itself, -make Cargo use `--use-sysroot=false` in all cases. -Cargo would special case package names corresponding to the crates behind the facade, such that if -the package don't exist, it would simply pass the corresponding system crate to `rustc`. I assume -the names are blacklisted on crates.io already, so by default the packages won't exist. But users -can use config files to extend the namespace so their own modded libstds can be used instead. Even -if they don't want to change libstd but just cross-compile it, this is frankly the easiest way as -Cargo will seemliest cross compile both their project and it's transitive dependencies. +# Detailed design + +The only new interface proposed is a boolean field in `Cargo.toml` specifying that the package does +not depend on libstd by default. Note that this is technically orthogonal to Rust's `no_std`, as one +might want to `use` their own build of libstd by default, or implicitly depend on it but not +glob-import the prelude. To disambiguate, this field is called `implicit-deps`; please, go ahead and +bikeshead the name. `implicit-deps` is true by default to maintain compatibility with existing +packages. When true, "std" will be implicitly appended to the list of dependencies. + +When Cargo sees a package name it cannot resolve, it will query `rustc` for the default sysroot, and +look inside to see if it can find a matching rlib. [It is necessary to query `rustc` because the +`rustc` directory layout is not stabilized and `rustc` and Cargo are versioned independently. The +same version issues make giving a Cargo a whitelist of potential standard library crate-names +risky.] If a matching rlib is successful found, Cargo will copy it (or simlink it) into the +project's build directly as if it built the rlib. Each rlib in the sysroot must be paired with some +sort of manifest listing its dependencies, so Cargo can copy those too. -In this way we can put packages on crates.io that depend on the crates behind the facade. Some -packages that already exist, like liblog and libbitflags, should be given features that optionally -allow them to avoid libstd and just depend directly on the crates behind the facade they really -need. +`rustc` will have a new `--use-sysroot=` flag. When Cargo builds a package, it will +always pass `--use-sysroot=false` to `rustc`, as any rlibs it needs will have been copied to the +build directory. Cargo can and will then pass those rlibs directly just as it does with normal Cargo +deps. -## Phase 3 +If Cargo cannot find the libraries it needs in the sysroot, or a library's dependency manifest is +missing, it will complain that the standard libraries needed for the current job are missing and +give up. -If/when the standard library is built with Cargo and put on crates.io, all the specially-cased -package names can be treated normally, +## Future Compatibility -The standard library is downloaded and built from crates.io. Or equivalently, Cargo comes with a -cache of that build, as Cargo should be able cache builds between projects at this point. Just as in -phase 2, `implicit-deps = false` just prevents libstd from implicitly being appended to the list of -dependencies. +In the future, rather than giving up if libraries are missing Cargo could attempt to download them +from some build cache. In the farther future, the stdlib libraries may be Cargoized, and Cargo able +to query pre-built binaries for any arbitrary package. In that scenario, we can remove all code +relating to falling back on the sysroot to look for rlibs. + +In the meantime, developers living dangerously with an unstable compiler can package the standard +library themselves, and use their Cargo config file to get Cargo to cross compiler libstd for them. -Again, to make this as least controversial as possible, this RFC does not propose outright that the -standard library should be Cargoized. This 3rd phases just describes how this feature would work -were that to happen. # Drawbacks -I really don't know of any. Development for hosted environments would hardly be very affected. +Cargo does more work than is strictly necessary for rlibs installed in sysroot; some more metadata +must be maintained by `rustc` or its installation. + + - But in a future where Cargo can build stdlib like any other, all this cruft goes away. + # Alternatives -Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. + - Simply have `implicit-deps = false` make Cargo pass `--use-sysroot=false` to `rustc`. + + - This doesn't by-itself make a way for package to depend on only some of the crates behind the + facade. That, in turn, means Cargo is little better at cross compiling those than before. + + - While unstable compiler users can just package the standard library and depend on it as a + normal crate, it would be weird to have freestanding projects coalesce around some bootleg + libcore on crates.io. + + - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Slightly + simpler, but breaks nearly all existing packages. + + - Don't track stdlib depencies. Then, in the future when Cargo tries to obtain libs for cross + compiling, stick them in the sysroot instead. Cargo either assumes package needs all of stdlib, + or examines target to see what crates behind the facade are buildable and just goes for those. + + - Cargo does extra work if you need less of the stdlib + + - No nice migration into a world where Cargo can build stdlib without hacks. + # Unresolved questions -There are multiple lists of dependencies for different things (e.g. tests), Should libstd be append -to all of them in phases 2 and 3? + - There are multiple lists of dependencies for different things (e.g. tests), Should libstd be + append to all of them in phases 2 and 3? + + - Should rlibs in the sysroot respect Cargo name-frobbing conventions? If they don't, should Cargo + frob the name when it copies it (e.g. with `ld -i`)? + + - Just as make libstd a real dependency, we can make `rustc` a real dev dependency. The standard + library can thus be built with Cargo by depending on the associated unstable compiler. There are + some challenges to be overcome, including: + + - Teaching Cargo and its frobber an "x can build for y" relation for stable/unstable compiler + compatibility, rather than simply assuming all distinct compilers are mutually incompatible. + + - Coalescing a "virtual package" out of many different packages with disjoint dependencies. This + is needed because different `rustc` version has a different library implementation that + present the same interface. + + This almost certainly is better addressed in a later RFC. From 37e924612b3b01cee6ce14b8ec1a7e84c003d6be Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 30 May 2015 01:14:43 +0000 Subject: [PATCH 07/66] Messed up target triple in anecdote --- text/0000-cargo-libstd-awareness.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 1ad4c4bffb5..e16b7508cc3 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -48,8 +48,8 @@ For an anecdote: in https://github.com/RustOS-Fork-Holding-Ground I tried to rig compile libstd for me. Since I needed to use an unstable compiler anyways, it was possible in principle to build absolutely everything I needed with the same `rustc` version. Because of some trouble with Cargo and target JSONs, I didn't use a custom target specification, and just used -`x86_64-gnu-linux`, meaning that depending on platform I was compiling on, I may or may have been -cross-compiling. In the case where I wasn't, I couldn't complete the build because `rustc` +`x86_64-unknown-linux-gnu`, meaning that depending on platform I was compiling on, I may or may have +been cross-compiling. In the case where I wasn't, I couldn't complete the build because `rustc` complained about the libstd I was building overlapping with the libstd in the sysroot. For these reasons, most freestanding projects I know of avoid Cargo altogether, and just include From d549d150a1f64fcdb6aace59d901ef55703194b2 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 15 Jun 2015 00:20:32 -0700 Subject: [PATCH 08/66] Clarify that stable Rust won't allow linking any library in the sysroot --- text/0000-cargo-libstd-awareness.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index e16b7508cc3..4561a42eac1 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -30,7 +30,9 @@ the compiler's sysroot [specifically `/lib/`]. This is quite co and we loose all the advantages of the previous method: - Packages may link or not link against libs in that directory as they please, with Cargo being - none the wiser. + none the wiser. For the foreseeable future, libstd should be the only crate in that directory + which stable Rust code link, but unstable Rust code can also freely link std's deps, or anything + that ends up there by mistake. - Cargo-built crates with the same name as those in there will collide, as the sysroot libs don't have their names frobbed. From b3476e117770ad3478d27a8f99e150df6a96e800 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 15 Aug 2015 23:50:53 -0700 Subject: [PATCH 09/66] Motivate incrementally --- text/0000-cargo-libstd-awareness.md | 275 +++++++++++++++++----------- 1 file changed, 167 insertions(+), 108 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 4561a42eac1..9f82b7c1e0f 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -1,119 +1,194 @@ -- Feature Name: cargo_libstd_awareness +- Feature Name: Cargo_stdlib_awareness - Start Date: 2015-05-26 - RFC PR: (leave this empty) - Rust Issue: (leave this empty) # Summary -Currently, Cargo doesn't know whether packages depend on libstd. This makes Cargo unsuitable for -packages that need a cross-compiled or custom libstd, or otherwise depend on crates with the same -names as libstd and the crates behind the facade. The proposed fixes also open the door to a future -where libstd can be Cargoized. - +Currently, Cargo doesn't know what parts of the standard library packages depend on. This makes +Cargo unsuitable for packages that are typically cross compiled and only use some of the crates +behind the facade--in other words, libraries intended for freestanding use. If/when the standard +library is Cargoized in the future, the proposed fixes will also allow projects to better take +advantage of that. # Motivation -First some background. The current situation seems to be more of an accident of `rustc`'s pre-Cargo -history than an explicit design decision. Cargo passes the location and name of all depended-on -crates to `rustc`. This method is good for a number of reasons stemming from its fine granularity, -such as: - - - No undeclared dependencies can be used - - - Conversely, `rustc` can warn against *unused* declared dependencies - - - Crate/symbol names are frobbed so that packages with the overlapping names don't conflict - - -However rather than passing in libstd and its deps, Cargo lets the compiler look for them as need in -the compiler's sysroot [specifically `/lib/`]. This is quite coarse in comparison, -and we loose all the advantages of the previous method: +First, some background. `rustc` can be given crates and their location with `--extern +=`. When finding a library, `rustc` first sees if its location has been specified with +`--extern`, then looks in any directories specified with `-L`, and finally looks in the "sysroot" +[specifically `/lib/`]. Cargo passes in all dependencies it builds with `--extern`. +However Cargo does not know about the standard library, so builds of it are taken from the sysroot. + +## Short-Term Needs + +Freestanding projects need to cross-compile the standard library, as it is infeasible for anyone to +provide binaries for all possible target triples. "Vendoring" in `rustc` and cross-compiling with +make is both clunky, and unsuitable for libraries, as then one ends up with a separate copy of the +standard library for each library used. Since freestanding rust needs unstable features (and will +continue to need them for the foreseeable future), the most elegant thing to do is just get Cargo to +cross-compile the standard library. This is suitable for libraries and binaries alike, and most +easily ensures the library and it's consumers will be built to the exact same target +specification. For examples of how this is rigged up, see my simple +https://github.com/RustOS-Fork-Holding-Ground/rust/tree/rustos/Cargo, and the more flexible +https://github.com/hackndev/rust-libcore. + +The problem with this is that each and every library that is used must be forked and made to +explicitly depend on the standard library crates. This both impedes writing libraries for hosted and +freestanding use, but also for different freestanding projects, as there is currently no official +Cargoization of the standard library. We want to be able to somehow "inject" our Cargoized standard +library as an extra dependency to a library written without this injection in mind. + +Already, we have a means for overriding dependencies with Cargo, `.Cargo/config` files. Suppose that +one could specify an override for core, alloc, std, etc, and then all packages would be built with +that override as dependency instead of the binary that came with `rustc`. This precisely solves the +problem of using libraries in both contexts. + +There is one slight complication: how do we know which libraries would be linked by default? +Importantly, the crates behind the facade are not yet stabilized. Cargo versions are not currently +tied to compiler versions, so Cargo cannot be hard-coded to know the exact set of crate names. One +solution is to just wait and prioritize stabilizing at least the names of those crates. But that +would harm freestanding development in other ways. To better support the menagerie of platforms, it +would be wise to break out the implementation of std further so we can opt into e.g. networking but +not persistent storage, or vice-versa. However, since that use-case was brought up last year, the +number of crates behind the facade actually decreased---presumably to ease rapid development as 1.0 +approached. Clearly then, to keep the door open for finer-grained crates behind the facade, we +should not do anything that would motivate stabilizing the names of those crates. Instead, we can +have Cargo look in the sysroot before calling `rustc`, and only accept overrides from +`.Cargo/config` that either match a dependency, or a crate in the sysroot. + +## Long-Term Needs + +The [Rust in 2016](http://blog.rust-lang.org/2015/08/14/Next-year.html) blog post announces an +infrastructure goal of automatic cross-compilation. For the former, the plan is that Cargo will +download libstd builds for the destination platform. But note that this only works well for +platforms where libstd is buildable. For platforms where only some crates behind the facade are +buildable, the best we could do is download all the crates behind the facade that build on a +platform, and hope that is sufficient for anything the build at hand needs. For Cargo download a +bunch of binaries, and only thereafter err saying more are needed is annoying, and contrary to it's +normal behavior of determining whether all (transitive) dependencies can be built before attempting +the build itself. + +Also, https://github.com/rust-lang/rust/pull/27003#issuecomment-121677828 states that we "probably +want to eventually move in the direction of a Cargo-based build system in the future". If/when do we +do, it would be nice to ensure that, when building `rustc` or the standard library, we don't +actually link anything in the sysroot by mistake. Granted, this sanity check is not essential for a +Cargo-based build system. + +In the even longer term, we can imagine these two goals converging. Instead of merely offering a +downloaded pre-compiled standard library, we allow library maintainers to host their own binaries, +and downstream consumers to opt into the binary caches they trust. Mozilla, with the cross-compiled +libstd builds, would merely be another user of that system. Likewise, my short-term hack of using +Cargo overrides is no longer necessary. For platforms where binaries for the standard library is not +available, Cargo would just cross-compile it using the official Cargo-based build system. +Conveniently, Packages designed with the short-term hack in mind would not need to be republished. +Simply removing the suggestion of using Cargo overrides from their READMEs would be sufficient. + +## Possible solutions + +A first attempt at a solution would be to allow packages to whitelist what crates they can link from +the sysroot. The standard library would have an empty whitelist, while libraries for embedded +systems would opt into just what they actually need. This solves the problem, but is somewhat +inelegant in that it exposes the existence of the sysroot in the `Cargo.toml` schema, and thus +commits us to having the sysroot indefinitely. I don't see any advantage to making that +commitment. Consider that the cross-compiled standard library is distributed using a service +available to all library developers, the proposal I make above. We certainly don't want to allow +arbitrary Cargo libraries to be installed in a way that code that doesn't declare a dependency, and +special-casing the std lib to be installed in the sysroot while other packages aren't negates the +elegance of Mozilla dog-food-ing such a service as just another user. Of course that is all +hypothetical, but I wanted provide a concrete example of how exposing the notion of the sysroot +today could bite us down the road. + +The salient attribute of the crates in the sysroot is not that they are in the sysroot, but that +they are part of the standard library. For a second attempt, Instead of allowing packages to +whitelist which sysroot crates they can link to, we could allow them to whitelist which standard +library crates they can link to. For now, this would be implemented exactly the same way, but allows +us to step away from having a sysroot, or stop using it with Cargo. + +Finally, note that this is orthogonal to Rust's `no_std`, especially since it's recent modification +in #1184. It may seem silly to explicitly `extern crate`-ing std, but implicitly depending on it in +the Cargo file, or implicitly `extern-crate` std, but explicitly whitelist depending on it. However +both of these are naturally well-defined from the definition of `no_std` and the contents of this +RFC. - - Packages may link or not link against libs in that directory as they please, with Cargo being - none the wiser. For the foreseeable future, libstd should be the only crate in that directory - which stable Rust code link, but unstable Rust code can also freely link std's deps, or anything - that ends up there by mistake. +For the record, I first raised this issue [here](https://github.com/rust-lang/Cargo/issues/1096). - - Cargo-built crates with the same name as those in there will collide, as the sysroot libs don't - have their names frobbed. - - Cross compiling may fail at build-time (as opposed to the much shorter - "gather-dependencies-time") because of missing packages +# Detailed design +## Interface -Cargo doesn't look inside the sysroot to see what is or isn't there, but it would hardly help if it -did, because it doesn't know what any package needs. Assuming all packages need libstd, for example, -means Cargo just flat-out won't build freestanding packages that just use libcore on a platform that -doesn't support libstd. +First, we need to allow packages to opt out of implicitly depending on all standard library +crates. A new optional Boolean field is added called called `implicit-deps` to the `Cargo.toml` +schema.`implicit-deps` is true by default to maintain compatibility with existing packages. When +true, the package will implicitly depend on all crates that are part of the standard library. -For an anecdote: in https://github.com/RustOS-Fork-Holding-Ground I tried to rig up Cargo to cross -compile libstd for me. Since I needed to use an unstable compiler anyways, it was possible in -principle to build absolutely everything I needed with the same `rustc` version. Because of some -trouble with Cargo and target JSONs, I didn't use a custom target specification, and just used -`x86_64-unknown-linux-gnu`, meaning that depending on platform I was compiling on, I may or may have -been cross-compiling. In the case where I wasn't, I couldn't complete the build because `rustc` -complained about the libstd I was building overlapping with the libstd in the sysroot. +Second, we need a way for packages that have opted-out of these implicit dependencies to explicitly +depend on a subset of standard library crates. For this we add a new type of dependency, in addition +to the git, local path, and crates.io dependencies we support today. Packages will able to declare: +```toml +[dependencies.std] +stdlib = true +``` +which will declare a dependency on a crate called `std` which must be part of the standard library. -For these reasons, most freestanding projects I know of avoid Cargo altogether, and just include -submodule rust and run make in that. Cargo can still be used if one manages to get the requisite -libraries in the sysroot. But this is a tedious operation that individual projects shouldn't need to -reimplement, and one that has serious security implications if the normal libstd is modified. +It is an error to explicitly depend on a standard library crate when the entire standard library is +already depended-upon implicitly, as the explicit dependency is redundant. This can be relaxed in +the future if a need arises. -The fundamental plan proposed in this RFC is to make sure that anything Cargo builds never blindly -links against libraries in the sysroot. This is achieved by making Cargo aware of all dependencies, -including those libstd or its backing crates. That way, these problems are avoided. +## Implementation -For the record, I first raised this issue [here](https://github.com/rust-lang/Cargo/issues/1096). +Here is how we can implement this interface until future infrastructure changes. +`rustc` will gain the ability to not use a sysroot, in which case only the information specified in +`--extern` and `-L` flags will be used to find dependencies. For backwards compatibility, a sysroot +will be used by default. To control this a `--use-sysroot=` flag is added, with the +obvious meaning. It is an error to specify `--use-sysroot=false` and a custom sysroot with +`--sysroot=`. -# Detailed design - -The only new interface proposed is a boolean field in `Cargo.toml` specifying that the package does -not depend on libstd by default. Note that this is technically orthogonal to Rust's `no_std`, as one -might want to `use` their own build of libstd by default, or implicitly depend on it but not -glob-import the prelude. To disambiguate, this field is called `implicit-deps`; please, go ahead and -bikeshead the name. `implicit-deps` is true by default to maintain compatibility with existing -packages. When true, "std" will be implicitly appended to the list of dependencies. +The way in which Cargo builds packages will be modified as follows. When the standard library is +depended-on implicitly, everything happens as it does today. When the standard library isn't +depended the following two extra steps happen when Cargo is gathering all dependency metadata to see +if a build is possible. -When Cargo sees a package name it cannot resolve, it will query `rustc` for the default sysroot, and -look inside to see if it can find a matching rlib. [It is necessary to query `rustc` because the -`rustc` directory layout is not stabilized and `rustc` and Cargo are versioned independently. The -same version issues make giving a Cargo a whitelist of potential standard library crate-names -risky.] If a matching rlib is successful found, Cargo will copy it (or simlink it) into the -project's build directly as if it built the rlib. Each rlib in the sysroot must be paired with some -sort of manifest listing its dependencies, so Cargo can copy those too. + 1. Cargo queries `rustc` for the default sysroot. The means in which it does this is not specified + nor stabilized. -`rustc` will have a new `--use-sysroot=` flag. When Cargo builds a package, it will -always pass `--use-sysroot=false` to `rustc`, as any rlibs it needs will have been copied to the -build directory. Cargo can and will then pass those rlibs directly just as it does with normal Cargo -deps. + 2. Cargo computes gathers all explicit standard library dependencies of the current package and its + dependencies, and collects their paths within the sysroot. If it cannot find a build in the sysroot + for any gathered dependency, Cargo errs. -If Cargo cannot find the libraries it needs in the sysroot, or a library's dependency manifest is -missing, it will complain that the standard libraries needed for the current job are missing and -give up. +Finally, when Cargo invokes `rustc` to actually build something, it does so with +`--use-sysroot=false`, and passes in any explicitly-depended-on standard library crates (immediate, +not transitive dependencies) with `--extern`. -## Future Compatibility +Remember that Cargo current aims to work with multiple versions of `rustc`, and also that the crates +behind the facade are not stabilized. This is why Cargo needs to both query `rustc` for the sysroot +location, and use the contents of the sysroot rather than some hard-coded list to decide whether a +standard library dependency with a given (crate) name is valid. -In the future, rather than giving up if libraries are missing Cargo could attempt to download them -from some build cache. In the farther future, the stdlib libraries may be Cargoized, and Cargo able -to query pre-built binaries for any arbitrary package. In that scenario, we can remove all code -relating to falling back on the sysroot to look for rlibs. -In the meantime, developers living dangerously with an unstable compiler can package the standard -library themselves, and use their Cargo config file to get Cargo to cross compiler libstd for them. +# Drawbacks +Adds a notion of standard library dependencies that may be superfluous---see first alternative. -# Drawbacks -Cargo does more work than is strictly necessary for rlibs installed in sysroot; some more metadata -must be maintained by `rustc` or its installation. +# Alternatives - - But in a future where Cargo can build stdlib like any other, all this cruft goes away. + - It is possible we can just use wildcard dependencies for standard library crates, and don't need + to add a notion of a standard library dependency. Crucially, while there are many different + versions of the standard library, there is only one that works with any given version of `rustc`. + Specifically, we can have it so whenever Cargo comes across a wildcard crates.io dependency it + can't resolve, fallback on looking in the sysroot. In the future, the standard library could + actually be put in crates.io, released every time a compiler is released. -# Alternatives + I moved this to be an alternative because it might be a bit too magical. Also, I don't know how + much Cargo is aware of different Rust / `rustc` versions at the moment, and this RFC depends on + that awareness in some form. Finally, Semver won't be able to understand the release + trains---pre-release/beta builds are allowed in its grammar but have different semantics. - - Simply have `implicit-deps = false` make Cargo pass `--use-sysroot=false` to `rustc`. + - Simply have `implicit-deps = false` make Cargo pass `--use-sysroot=false` to `rustc`, and don't + have any way to explicitly depend on things in the sysroot. - This doesn't by-itself make a way for package to depend on only some of the crates behind the facade. That, in turn, means Cargo is little better at cross compiling those than before. @@ -122,35 +197,19 @@ must be maintained by `rustc` or its installation. normal crate, it would be weird to have freestanding projects coalesce around some bootleg libcore on crates.io. - - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Slightly - simpler, but breaks nearly all existing packages. - - - Don't track stdlib depencies. Then, in the future when Cargo tries to obtain libs for cross - compiling, stick them in the sysroot instead. Cargo either assumes package needs all of stdlib, - or examines target to see what crates behind the facade are buildable and just goes for those. + - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Simpler to + implement, but breaks nearly all existing packages. - - Cargo does extra work if you need less of the stdlib - - - No nice migration into a world where Cargo can build stdlib without hacks. + - Don't do this, and be stuck with problems detailed in the motivation section. # Unresolved questions - - There are multiple lists of dependencies for different things (e.g. tests), Should libstd be - append to all of them in phases 2 and 3? - - - Should rlibs in the sysroot respect Cargo name-frobbing conventions? If they don't, should Cargo - frob the name when it copies it (e.g. with `ld -i`)? + - There are multiple lists of dependencies for different things (e.g. tests, build-time). How + should `implicit-deps = false` affect them? - Just as make libstd a real dependency, we can make `rustc` a real dev dependency. The standard - library can thus be built with Cargo by depending on the associated unstable compiler. There are - some challenges to be overcome, including: - - - Teaching Cargo and its frobber an "x can build for y" relation for stable/unstable compiler - compatibility, rather than simply assuming all distinct compilers are mutually incompatible. - - - Coalescing a "virtual package" out of many different packages with disjoint dependencies. This - is needed because different `rustc` version has a different library implementation that - present the same interface. - - This almost certainly is better addressed in a later RFC. + library can thus be built with Cargo by depending on the associated unstable compiler. Cargo + would need to be taught an "x can build for y" relation for stable/unstable compiler + compatibility however, rather than simply assuming all distinct compilers are mutually + incompatible. This almost certainly is better addressed in a later RFC. From c22463f39bb45860176055b8590cc46b4e684555 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 30 Aug 2015 17:58:03 -0700 Subject: [PATCH 10/66] Nit: Fix Some wrongly-capitalized "Cargo"s --- text/0000-cargo-libstd-awareness.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 9f82b7c1e0f..8a87e54a4cc 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -1,4 +1,4 @@ -- Feature Name: Cargo_stdlib_awareness +- Feature Name: cargo_stdlib_awareness - Start Date: 2015-05-26 - RFC PR: (leave this empty) - Rust Issue: (leave this empty) @@ -29,7 +29,7 @@ continue to need them for the foreseeable future), the most elegant thing to do cross-compile the standard library. This is suitable for libraries and binaries alike, and most easily ensures the library and it's consumers will be built to the exact same target specification. For examples of how this is rigged up, see my simple -https://github.com/RustOS-Fork-Holding-Ground/rust/tree/rustos/Cargo, and the more flexible +https://github.com/RustOS-Fork-Holding-Ground/rust/tree/rustos/cargo, and the more flexible https://github.com/hackndev/rust-libcore. The problem with this is that each and every library that is used must be forked and made to @@ -38,7 +38,7 @@ freestanding use, but also for different freestanding projects, as there is curr Cargoization of the standard library. We want to be able to somehow "inject" our Cargoized standard library as an extra dependency to a library written without this injection in mind. -Already, we have a means for overriding dependencies with Cargo, `.Cargo/config` files. Suppose that +Already, we have a means for overriding dependencies with Cargo, `.cargo/config` files. Suppose that one could specify an override for core, alloc, std, etc, and then all packages would be built with that override as dependency instead of the binary that came with `rustc`. This precisely solves the problem of using libraries in both contexts. @@ -54,7 +54,7 @@ number of crates behind the facade actually decreased---presumably to ease rapid approached. Clearly then, to keep the door open for finer-grained crates behind the facade, we should not do anything that would motivate stabilizing the names of those crates. Instead, we can have Cargo look in the sysroot before calling `rustc`, and only accept overrides from -`.Cargo/config` that either match a dependency, or a crate in the sysroot. +`.cargo/config` that either match a dependency, or a crate in the sysroot. ## Long-Term Needs @@ -110,7 +110,7 @@ the Cargo file, or implicitly `extern-crate` std, but explicitly whitelist depen both of these are naturally well-defined from the definition of `no_std` and the contents of this RFC. -For the record, I first raised this issue [here](https://github.com/rust-lang/Cargo/issues/1096). +For the record, I first raised this issue [here](https://github.com/rust-lang/cargo/issues/1096). # Detailed design From c257f3488df608e0fad65541a26a2b6bde97e05e Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 2 Sep 2015 23:38:58 -0700 Subject: [PATCH 11/66] Detailed design for overrides --- text/0000-cargo-libstd-awareness.md | 32 +++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8a87e54a4cc..5af97e8f729 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -117,6 +117,8 @@ For the record, I first raised this issue [here](https://github.com/rust-lang/ca ## Interface +### Explicit Dependencies + First, we need to allow packages to opt out of implicitly depending on all standard library crates. A new optional Boolean field is added called called `implicit-deps` to the `Cargo.toml` schema.`implicit-deps` is true by default to maintain compatibility with existing packages. When @@ -135,6 +137,21 @@ It is an error to explicitly depend on a standard library crate when the entire already depended-upon implicitly, as the explicit dependency is redundant. This can be relaxed in the future if a need arises. +### Overrides + +Normal (non-stdlib) dependencies will be overridden the same as today. An override will apply if one +of: + + 1. The package name of the override is the name of a normally dependency. [Today] + + 2. The package in question has implicit deps, and the package name of the override is the name of a + crate in the standard library. + + 3. The package in question does not have implicit deps, and the package name matches the name of an + explicit stdlib dependency. + +Note that 1 and 3 can be combined by ignoring the type of (explicit) dependency. + ## Implementation Here is how we can implement this interface until future infrastructure changes. @@ -169,7 +186,11 @@ standard library dependency with a given (crate) name is valid. # Drawbacks -Adds a notion of standard library dependencies that may be superfluous---see first alternative. + - Adds a notion of standard library dependencies that may be superfluous---see first alternative. + + - For the time being, dependencies *within* the standard library are unspecified, so that an + override of, e.g, collections would not cause cargo to try to rebuilt std or complain it is + unable to do so. # Alternatives @@ -195,7 +216,14 @@ Adds a notion of standard library dependencies that may be superfluous---see fir - While unstable compiler users can just package the standard library and depend on it as a normal crate, it would be weird to have freestanding projects coalesce around some bootleg - libcore on crates.io. + core on crates.io. + + - Make it so that packages with implicit dependencies only depend on std. This would break + existing packages, but only those that cannot be published to crates.io or can but don't + build. This is elegant because it eliminates redundancy, while still providing a concise common + case (just one slightly more narrowly defined): packages that use stdlib crates besides std must + be explicit. If we want to do this, we have to act fast because core might be stabilized soon. + - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Simpler to implement, but breaks nearly all existing packages. From 4a8bf9b2e44be945fbf7a8e5d9a6b15df555316c Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 24 Apr 2016 22:12:15 -0700 Subject: [PATCH 12/66] Overhaul and expand on detailed design: core is stable and [reaplace] exists --- text/0000-cargo-libstd-awareness.md | 163 ++++++++++++++++------------ 1 file changed, 96 insertions(+), 67 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 5af97e8f729..3f214c7991d 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -3,13 +3,20 @@ - RFC PR: (leave this empty) - Rust Issue: (leave this empty) + # Summary Currently, Cargo doesn't know what parts of the standard library packages depend on. This makes -Cargo unsuitable for packages that are typically cross compiled and only use some of the crates -behind the facade--in other words, libraries intended for freestanding use. If/when the standard -library is Cargoized in the future, the proposed fixes will also allow projects to better take -advantage of that. +Cargo difficult to use when both the standard library (or a portion of it) and some consumer of it +need to be built together. Examples of this need are freestanding development, where the exact +compilation target is usually custom-tailed to the specific project at hand, and working on rustc or +the standard library itself. + +This new interfaces are simple, as is the the proposed implementation. However, the proposed +implementation also comes with some limitations, and past proposals of the RFC have been deemed +under-specified. I therefore suggest that this RFC be accepted an on experimental basis as an +unstable feature. + # Motivation @@ -115,82 +122,109 @@ For the record, I first raised this issue [here](https://github.com/rust-lang/ca # Detailed design + ## Interface ### Explicit Dependencies First, we need to allow packages to opt out of implicitly depending on all standard library crates. A new optional Boolean field is added called called `implicit-deps` to the `Cargo.toml` -schema.`implicit-deps` is true by default to maintain compatibility with existing packages. When -true, the package will implicitly depend on all crates that are part of the standard library. +schema. When true, the package will implicitly depend on all crates that are part of the standard +library. Second, we need a way for packages that have opted-out of these implicit dependencies to explicitly -depend on a subset of standard library crates. For this we add a new type of dependency, in addition -to the git, local path, and crates.io dependencies we support today. Packages will able to declare: +depend on a subset of standard library crates. For this we add a new "virtual version". Packages will +able to declare: +```toml +[dependencies] +std = "stdlib" +``` +or ```toml [dependencies.std] -stdlib = true +version = "stdlib" ``` which will declare a dependency on a crate called `std` which must be part of the standard library. -It is an error to explicitly depend on a standard library crate when the entire standard library is -already depended-upon implicitly, as the explicit dependency is redundant. This can be relaxed in -the future if a need arises. +When no explicit dependencies are specified, `implicit-deps` defaults to `true`. (This is necessary +for compatibility with existing packages.) When an explicit stdlib dependency is specified this +defaults to `false`. It possible to have no explicit stdlib dependencies specified, and set +`implicit-deps` to be `false` --- this would be used by `core` for example. However the reverse is +prohibited: it is not allowed to specify explicit stdlib dependencies yet also use implicit stdlib +dependencies. -### Overrides +### `[replace]` and `.cargo/config` Overrides. -Normal (non-stdlib) dependencies will be overridden the same as today. An override will apply if one -of: +In keeping with "stdlib" being a "virtual version", one can do +``` +[replace] +"some-crate:std-lib" = ... +``` +to replace a std-lib crate with one of ones choosing. This is useful when developing the standard +library and a consumer of it, like `rustc`, together. - 1. The package name of the override is the name of a normally dependency. [Today] +`.cargo/config` overrides are likewise extended as one would expect. If the Cargo package at the given +path matches a stdlib crate, the override is used instead. - 2. The package in question has implicit deps, and the package name of the override is the name of a - crate in the standard library. - 3. The package in question does not have implicit deps, and the package name matches the name of an - explicit stdlib dependency. +## Implementation -Note that 1 and 3 can be combined by ignoring the type of (explicit) dependency. +On to the gritty details! With this RFC, there are two ways to depend on std library +crates. Likewise, for the immediate future at least, we to support either building the needed parts +of the standard library from source (the main purpose of this rfc!) or using pre-built sysroot +binaries. This yields 4 combinations. -## Implementation +As a prerequisite, `rustc` will need one new flag `--sysroot-whitelist=[crates]`, a whitelist of +crates it is allowed to look for in the sysroot. Importantly this only restricts immediate `extern +crate`s in the current compilation unit; no restrictions are placed on opening transitive +dependencies. + +### Explicit dependencies and freshly-built stdlib + +Explicit stdlib dependencies, if not overridden, are simply desugared to git dependencies on +`http://github.com/rust-lang/rust` with the revision given by `rustc`. (On start-up, Cargo already +queries `rustc` for its verbose version, which includes the git revision.) Additionally for any +crate with `implicit-deps = true`, `--sysroot-whitelist=` (i.e. the empty whitelist) is passed to +`rustc` as sysroot binaries should never be used. -Here is how we can implement this interface until future infrastructure changes. +### Implicit dependencies and freshly-built stdlib -`rustc` will gain the ability to not use a sysroot, in which case only the information specified in -`--extern` and `-L` flags will be used to find dependencies. For backwards compatibility, a sysroot -will be used by default. To control this a `--use-sysroot=` flag is added, with the -obvious meaning. It is an error to specify `--use-sysroot=false` and a custom sysroot with -`--sysroot=`. +Exactly what parts of the standard library do we need to build? One can assume implicit deps on +`std` and `core`, as those are the only stable crates today. Keep in mind however that since Cargo +doesn't currently know what crates constitute the standard library for a given `rustc`, that there +is no simple way to correctly implement `.cargo/config` and `[replace]` overrides. Because this is +an experimental feature, I propose just punting and erring should overrides be given. -The way in which Cargo builds packages will be modified as follows. When the standard library is -depended-on implicitly, everything happens as it does today. When the standard library isn't -depended the following two extra steps happen when Cargo is gathering all dependency metadata to see -if a build is possible. +### Explicit dependencies and pre-built stdlib from sysroot - 1. Cargo queries `rustc` for the default sysroot. The means in which it does this is not specified - nor stabilized. +Build as today, but pass via `--sysroot-whitelist` the list of explicit stdlib dependencies whenever +`implicit-deps` is false. That way, we can be sure regardless of the built strategy that no +undeclared stdlib dependencies "leak in". - 2. Cargo computes gathers all explicit standard library dependencies of the current package and its - dependencies, and collects their paths within the sysroot. If it cannot find a build in the sysroot - for any gathered dependency, Cargo errs. +For overrides, one could filter the writelist of any crates that were overridden. On the other hand +this doesn't take into account other std-lib crates that depended on the overridden crates---they +two should be freshly built and not pulled from the sysroot so as to depend on the overridden +crates. It may be better to punt again then. -Finally, when Cargo invokes `rustc` to actually build something, it does so with -`--use-sysroot=false`, and passes in any explicitly-depended-on standard library crates (immediate, -not transitive dependencies) with `--extern`. +### Implicit dependencies and pre-built stdlib from sysroot -Remember that Cargo current aims to work with multiple versions of `rustc`, and also that the crates -behind the facade are not stabilized. This is why Cargo needs to both query `rustc` for the sysroot -location, and use the contents of the sysroot rather than some hard-coded list to decide whether a -standard library dependency with a given (crate) name is valid. +This is today, and should work as today. Again, punt on the newly-proposed types of overrides. # Drawbacks - - Adds a notion of standard library dependencies that may be superfluous---see first alternative. +Notably, all of these relate to the suggested implementation, and not the interface that is the core +of this RFC. I remain optimistic that over time these implementation issues can be resolved without +changing the proposed interface. But my optimism is no more that---speculation. - - For the time being, dependencies *within* the standard library are unspecified, so that an - override of, e.g, collections would not cause cargo to try to rebuilt std or complain it is - unable to do so. + - The override limitations already specified within the implementation section. + + - Due to the way git dependencies work, if we naively desugar explicit stdlib dependencies to them + as proposed, packages will be able to depend on *any* crate in the `rust` repo, including the + various `rustc_*` crates there is no intention of keeping around. + + - Cargo doesn't currently have any unstable features, so giving them that requires work and + planning. # Alternatives @@ -200,13 +234,12 @@ standard library dependency with a given (crate) name is valid. versions of the standard library, there is only one that works with any given version of `rustc`. Specifically, we can have it so whenever Cargo comes across a wildcard crates.io dependency it - can't resolve, fallback on looking in the sysroot. In the future, the standard library could - actually be put in crates.io, released every time a compiler is released. + can't resolve, assume it is a std lib dependency and either resolve it as such (desugar to git + dependencies) or leave it for `rustc` to find in the sysroot according to the build type. - I moved this to be an alternative because it might be a bit too magical. Also, I don't know how - much Cargo is aware of different Rust / `rustc` versions at the moment, and this RFC depends on - that awareness in some form. Finally, Semver won't be able to understand the release - trains---pre-release/beta builds are allowed in its grammar but have different semantics. + I made this to be an alternative because it might be a bit too magical. Also, Semver won't be + able to understand the release trains---pre-release/beta builds are allowed in its grammar but + have different semantics. - Simply have `implicit-deps = false` make Cargo pass `--use-sysroot=false` to `rustc`, and don't have any way to explicitly depend on things in the sysroot. @@ -218,15 +251,11 @@ standard library dependency with a given (crate) name is valid. normal crate, it would be weird to have freestanding projects coalesce around some bootleg core on crates.io. - - Make it so that packages with implicit dependencies only depend on std. This would break - existing packages, but only those that cannot be published to crates.io or can but don't - build. This is elegant because it eliminates redundancy, while still providing a concise common - case (just one slightly more narrowly defined): packages that use stdlib crates besides std must - be explicit. If we want to do this, we have to act fast because core might be stabilized soon. - + - Make it so that packages with implicit dependencies only depend on std. This would be more + elegant, but breaks packages (including ones using stable Rust) that just depend on crates.io. - - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Simpler to - implement, but breaks nearly all existing packages. + - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Simple and + elegant, but breaks all existing packages. - Don't do this, and be stuck with problems detailed in the motivation section. @@ -236,8 +265,8 @@ standard library dependency with a given (crate) name is valid. - There are multiple lists of dependencies for different things (e.g. tests, build-time). How should `implicit-deps = false` affect them? - - Just as make libstd a real dependency, we can make `rustc` a real dev dependency. The standard - library can thus be built with Cargo by depending on the associated unstable compiler. Cargo - would need to be taught an "x can build for y" relation for stable/unstable compiler - compatibility however, rather than simply assuming all distinct compilers are mutually - incompatible. This almost certainly is better addressed in a later RFC. + - Just as this makes the standard library a real dependency, we can make `rustc` a real + dev-dependency. The standard library can thus be built with Cargo by depending on the associated + unstable compiler. Cargo would need to be taught an "x can build for y" relation for + stable/unstable compiler compatibility however, rather than simply assuming all distinct + compilers are mutually incompatible. This almost certainly is better addressed in a later RFC. From 4b4d8a3c0d26f2f4a3da6e41b9ea81ac0aebc21b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 24 Apr 2016 22:22:08 -0700 Subject: [PATCH 13/66] Two more drawbacks --- text/0000-cargo-libstd-awareness.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 3f214c7991d..923fd57e9a7 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -226,6 +226,16 @@ changing the proposed interface. But my optimism is no more that---speculation. - Cargo doesn't currently have any unstable features, so giving them that requires work and planning. + - Only some crates in the rust repo (at least `core`, `alloc` and `collections`) can properly be + built just based upon their `Cargo.toml`. However it's precisely only these "foundational" crates + that will be of interest to freestanding developers. Hosted developers can get pre-built sysroots + with `rustup`, and those working on the standard library would use this in conjunction with its + build system which can deal with such things. + + - No means of compiling compiler-rt is proposed. But just as freestanding developers need to + provide `rlibc` or similar to successfully link, I think that for the time-being they can be left + to provide this themselves when linking. + # Alternatives From 1ee980f7e8160d4b807eaa110e4a6b3b7a90b25f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 10 May 2016 21:55:17 -0700 Subject: [PATCH 14/66] Fix typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 923fd57e9a7..b0582c78c89 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -158,7 +158,7 @@ dependencies. In keeping with "stdlib" being a "virtual version", one can do ``` [replace] -"some-crate:std-lib" = ... +"some-crate:stdlib" = ... ``` to replace a std-lib crate with one of ones choosing. This is useful when developing the standard library and a consumer of it, like `rustc`, together. From f0a0b3b5fbfb6968a2a33f76b1378ecea0985874 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 11 Jul 2016 23:03:30 -0700 Subject: [PATCH 15/66] Rewrite to use upcoming Cargo registries. --- text/0000-cargo-libstd-awareness.md | 323 +++++++++------------------- 1 file changed, 100 insertions(+), 223 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index b0582c78c89..8c9edbee9fa 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -6,277 +6,154 @@ # Summary -Currently, Cargo doesn't know what parts of the standard library packages depend on. This makes -Cargo difficult to use when both the standard library (or a portion of it) and some consumer of it -need to be built together. Examples of this need are freestanding development, where the exact -compilation target is usually custom-tailed to the specific project at hand, and working on rustc or -the standard library itself. - -This new interfaces are simple, as is the the proposed implementation. However, the proposed -implementation also comes with some limitations, and past proposals of the RFC have been deemed -under-specified. I therefore suggest that this RFC be accepted an on experimental basis as an -unstable feature. +Currently, Cargo doesn't know what parts of the standard library packages depend on. +By giving it this knowledge, we can make cross compilation and exotic platform development easier, simplify rustbuild, and allow anyone to easily specify bounds on the standard library for their packages. +This will allow building parts of the standard library from source, but in order to not disrupt existing workflows, the binaries that come with rustc will still be used by default. # Motivation -First, some background. `rustc` can be given crates and their location with `--extern -=`. When finding a library, `rustc` first sees if its location has been specified with -`--extern`, then looks in any directories specified with `-L`, and finally looks in the "sysroot" -[specifically `/lib/`]. Cargo passes in all dependencies it builds with `--extern`. +First, some background. +`rustc` can be given crates and their location with `--extern =`. +When finding a library, `rustc` first sees if its location has been specified with `--extern`, then looks in any directories specified with `-L`, and finally looks in the "sysroot" [specifically `/lib/rustlib//lib`]. +Cargo passes in all dependencies it builds with `--extern`. However Cargo does not know about the standard library, so builds of it are taken from the sysroot. -## Short-Term Needs - -Freestanding projects need to cross-compile the standard library, as it is infeasible for anyone to -provide binaries for all possible target triples. "Vendoring" in `rustc` and cross-compiling with -make is both clunky, and unsuitable for libraries, as then one ends up with a separate copy of the -standard library for each library used. Since freestanding rust needs unstable features (and will -continue to need them for the foreseeable future), the most elegant thing to do is just get Cargo to -cross-compile the standard library. This is suitable for libraries and binaries alike, and most -easily ensures the library and it's consumers will be built to the exact same target -specification. For examples of how this is rigged up, see my simple -https://github.com/RustOS-Fork-Holding-Ground/rust/tree/rustos/cargo, and the more flexible -https://github.com/hackndev/rust-libcore. - -The problem with this is that each and every library that is used must be forked and made to -explicitly depend on the standard library crates. This both impedes writing libraries for hosted and -freestanding use, but also for different freestanding projects, as there is currently no official -Cargoization of the standard library. We want to be able to somehow "inject" our Cargoized standard -library as an extra dependency to a library written without this injection in mind. - -Already, we have a means for overriding dependencies with Cargo, `.cargo/config` files. Suppose that -one could specify an override for core, alloc, std, etc, and then all packages would be built with -that override as dependency instead of the binary that came with `rustc`. This precisely solves the -problem of using libraries in both contexts. - -There is one slight complication: how do we know which libraries would be linked by default? -Importantly, the crates behind the facade are not yet stabilized. Cargo versions are not currently -tied to compiler versions, so Cargo cannot be hard-coded to know the exact set of crate names. One -solution is to just wait and prioritize stabilizing at least the names of those crates. But that -would harm freestanding development in other ways. To better support the menagerie of platforms, it -would be wise to break out the implementation of std further so we can opt into e.g. networking but -not persistent storage, or vice-versa. However, since that use-case was brought up last year, the -number of crates behind the facade actually decreased---presumably to ease rapid development as 1.0 -approached. Clearly then, to keep the door open for finer-grained crates behind the facade, we -should not do anything that would motivate stabilizing the names of those crates. Instead, we can -have Cargo look in the sysroot before calling `rustc`, and only accept overrides from -`.cargo/config` that either match a dependency, or a crate in the sysroot. - -## Long-Term Needs - -The [Rust in 2016](http://blog.rust-lang.org/2015/08/14/Next-year.html) blog post announces an -infrastructure goal of automatic cross-compilation. For the former, the plan is that Cargo will -download libstd builds for the destination platform. But note that this only works well for -platforms where libstd is buildable. For platforms where only some crates behind the facade are -buildable, the best we could do is download all the crates behind the facade that build on a -platform, and hope that is sufficient for anything the build at hand needs. For Cargo download a -bunch of binaries, and only thereafter err saying more are needed is annoying, and contrary to it's -normal behavior of determining whether all (transitive) dependencies can be built before attempting -the build itself. - -Also, https://github.com/rust-lang/rust/pull/27003#issuecomment-121677828 states that we "probably -want to eventually move in the direction of a Cargo-based build system in the future". If/when do we -do, it would be nice to ensure that, when building `rustc` or the standard library, we don't -actually link anything in the sysroot by mistake. Granted, this sanity check is not essential for a -Cargo-based build system. - -In the even longer term, we can imagine these two goals converging. Instead of merely offering a -downloaded pre-compiled standard library, we allow library maintainers to host their own binaries, -and downstream consumers to opt into the binary caches they trust. Mozilla, with the cross-compiled -libstd builds, would merely be another user of that system. Likewise, my short-term hack of using -Cargo overrides is no longer necessary. For platforms where binaries for the standard library is not -available, Cargo would just cross-compile it using the official Cargo-based build system. -Conveniently, Packages designed with the short-term hack in mind would not need to be republished. -Simply removing the suggestion of using Cargo overrides from their READMEs would be sufficient. - -## Possible solutions - -A first attempt at a solution would be to allow packages to whitelist what crates they can link from -the sysroot. The standard library would have an empty whitelist, while libraries for embedded -systems would opt into just what they actually need. This solves the problem, but is somewhat -inelegant in that it exposes the existence of the sysroot in the `Cargo.toml` schema, and thus -commits us to having the sysroot indefinitely. I don't see any advantage to making that -commitment. Consider that the cross-compiled standard library is distributed using a service -available to all library developers, the proposal I make above. We certainly don't want to allow -arbitrary Cargo libraries to be installed in a way that code that doesn't declare a dependency, and -special-casing the std lib to be installed in the sysroot while other packages aren't negates the -elegance of Mozilla dog-food-ing such a service as just another user. Of course that is all -hypothetical, but I wanted provide a concrete example of how exposing the notion of the sysroot -today could bite us down the road. - -The salient attribute of the crates in the sysroot is not that they are in the sysroot, but that -they are part of the standard library. For a second attempt, Instead of allowing packages to -whitelist which sysroot crates they can link to, we could allow them to whitelist which standard -library crates they can link to. For now, this would be implemented exactly the same way, but allows -us to step away from having a sysroot, or stop using it with Cargo. - -Finally, note that this is orthogonal to Rust's `no_std`, especially since it's recent modification -in #1184. It may seem silly to explicitly `extern crate`-ing std, but implicitly depending on it in -the Cargo file, or implicitly `extern-crate` std, but explicitly whitelist depending on it. However -both of these are naturally well-defined from the definition of `no_std` and the contents of this -RFC. - -For the record, I first raised this issue [here](https://github.com/rust-lang/cargo/issues/1096). +For cross-compiling, one can often download standard library binaries with rustup. +This is convenient, but one cannot expect pre-built binaries for all platforms. +In particular, embedded systems often have detailed configurations to convey as much information as possible about the hardware to the compiler. +Furthermore, not all of the stdlib is available on every platform---there is an RFC in the works to pre-build a smaller set of crates for [Cortex-M microcontrollers[(https://github.com/rust-lang/rfcs/pull/1645). +It would be nice to know if the available subset is adequate before attempting a build. +We can only do that if all packages have explicit standard library deps to cross-reference with platform requirements of each standard library crate. + +Now rustup could be augmented to build stdlib binaries in addition to downloading them, but we have extra configuration options for the standard library such as [panic strategies](https://github.com/rust-lang/rfcs/blob/master/text/1513-less-unwinding.md) along with plans to add more such as [`core` without floats](https://github.com/rust-lang/rfcs/issues/1364). +One could also cobble together a way to to tell rustup such configuration options, but we'd like to make sure that all dependencies agree with the plan before trying to execute it. +For panicking, packages can already set a `profile.dev.panic` option to require a specific strategy, and for float support we should add a (default opt-in) feature to core. +Then, if packages can explicitly depend on core they can also specify whether the float feature is needed. +Cargo would be able to infer both of these options by inspecting all crates in the dependency graph. + +Rustbuild must currently perform multiple `cargo builds`, the first to build the standard library and the rest to build things which depend on the standard library. +If rustc, rustfmt, etc, and their deps (some of which come from crates.io, and thus aren't specially tailored for building rust) declare deps on std, rustbuild wouldn't need multiple lockfiles. +Keeping multiple in sync is nuisance, and this gets us one step closer to a single `cargo build` building rustc. + +The use-cases so far mainly benefit niche corners of the Rust community, but the last should be useful for just about everyone. +Now that multiple versions of Rust have been released, it can be useful to specify the minimum version. +If and when Rust 2, a version with breaking changes, comes out, this will be all the more important. +We don't yet have a plan yet to track the language itself, but by tracking standard library dependencies we make it trivial to specify version requirements like any other package. # Detailed design +The subsections with "interface" in their title form the normitive part of this RFC. +The rest of this section just illustrate hows they would likely be implemented. -## Interface +## `Cargo.toml` Interface -### Explicit Dependencies +First and foremost, one will now be able to depend on standard library crates by version, e.g. `std = "1.10"`. +This will work just as if `std` was on crates.io---features and other modifiers are supported. -First, we need to allow packages to opt out of implicitly depending on all standard library -crates. A new optional Boolean field is added called called `implicit-deps` to the `Cargo.toml` -schema. When true, the package will implicitly depend on all crates that are part of the standard -library. +For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. +Exactly which dependencies is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary and build dependency is assured. -Second, we need a way for packages that have opted-out of these implicit dependencies to explicitly -depend on a subset of standard library crates. For this we add a new "virtual version". Packages will -able to declare: +Now, not all crates depend on `std`, so there must be a way to opt out. +For this, we introduce a new `implicit-deps` key. +It is defined by default as: ```toml -[dependencies] -std = "stdlib" +implicit-dependencies = ["primary", "build", "dev"] ``` -or -```toml -[dependencies.std] -version = "stdlib" -``` -which will declare a dependency on a crate called `std` which must be part of the standard library. +This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit elements. +A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. +The one additional rule is `"build"` must be included in the set: we have no plan for Cargoizing the default test runner (either the attribute syntax manipulation or runtime), and wish to be forward-compatible with doing so in the future. -When no explicit dependencies are specified, `implicit-deps` defaults to `true`. (This is necessary -for compatibility with existing packages.) When an explicit stdlib dependency is specified this -defaults to `false`. It possible to have no explicit stdlib dependencies specified, and set -`implicit-deps` to be `false` --- this would be used by `core` for example. However the reverse is -prohibited: it is not allowed to specify explicit stdlib dependencies yet also use implicit stdlib -dependencies. +Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. +For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. -### `[replace]` and `.cargo/config` Overrides. +## Cargo Command Line Interface -In keeping with "stdlib" being a "virtual version", one can do +A flag will be added ``` -[replace] -"some-crate:stdlib" = ... +--resolve-compiler-specific=bin,src ``` -to replace a std-lib crate with one of ones choosing. This is useful when developing the standard -library and a consumer of it, like `rustc`, together. +where either `bin` or `src` may be included in the set. +If `bin` is included, Cargo will allow the use of sysroot binaries to satisfy deps. +It is assumed that the version of all sysroot rlibs is the same as the version of Rust which the compiler implements. +If `src` is included, Cargo will fallback on a compiler-provided package registry when other registries (e.g. crates.io) fail to provide a package. +For backwards compatibility, the default is `--resolve-compiler-specific=bin`. -`.cargo/config` overrides are likewise extended as one would expect. If the Cargo package at the given -path matches a stdlib crate, the override is used instead. +## Compiler source packaging +While the standard library *interface* defined with each rustc version, the implementation, by virtue of using unstable features, is compiler-specific. +This makes the standard library unfit for crates.io. +(Additionally, the issue of dealing with nightly also makes crates.io hard to use, but that is a less clear-cut obstacle.) -## Implementation +Cargo will soon gain the ability to create and chain custom registries, as described in +https://github.com/rust-lang/cargo/pull/2361 . +Compiler's should package the source of their implementation of the standard library as a registry, which can be distributed with the compiler. +In practice, rustc will optionally contain the source in its sysroot. +Rustup may be able to put it there if the default download does not contain it already. -On to the gritty details! With this RFC, there are two ways to depend on std library -crates. Likewise, for the immediate future at least, we to support either building the needed parts -of the standard library from source (the main purpose of this rfc!) or using pre-built sysroot -binaries. This yields 4 combinations. +## Cargo implementation -As a prerequisite, `rustc` will need one new flag `--sysroot-whitelist=[crates]`, a whitelist of -crates it is allowed to look for in the sysroot. Importantly this only restricts immediate `extern -crate`s in the current compilation unit; no restrictions are placed on opening transitive -dependencies. +The one thing this RFC requires of the upcoming registry implementation is the ability to chain registries providing defaults and fallbacks when a registry is not manually specified. +This is used so crates not on crates.io are instead provided by the compiler. +[Ideally, we'd put the "interface" on crates.io to help ensure compiler-specific implementations conform, but such a mechanism is not being proposed at this time.] -### Explicit dependencies and freshly-built stdlib +The injection of implicit dependencies is completely defined by the rules described in the first subsection, so this subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. -Explicit stdlib dependencies, if not overridden, are simply desugared to git dependencies on -`http://github.com/rust-lang/rust` with the revision given by `rustc`. (On start-up, Cargo already -queries `rustc` for its verbose version, which includes the git revision.) Additionally for any -crate with `implicit-deps = true`, `--sysroot-whitelist=` (i.e. the empty whitelist) is passed to -`rustc` as sysroot binaries should never be used. +If `src` is included in the set passed with that flag, Cargo appends a local registry with path `${$CARGO_RUSTC --print sysroot}/src` to the back of the default chain. +In other words, if a package is in none of the user-specified registries contain a package, Cargo will look in the registry provided by the compiler. -### Implicit dependencies and freshly-built stdlib +If `bin` is included in the set passed with that flag (or inferred from the default), Cargo will build a mock registry by examining the contents of the sysroot. +Any binary in there will be added to the mock registry, with a version deduced the best Cargo can (e.g. from the version of the compiler). +Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep. +The mock registry will have dead last priority in the default chain, even behind the source registry. +The "building" of such a package in the mock registry will consist of copying the binary into the target directory. -Exactly what parts of the standard library do we need to build? One can assume implicit deps on -`std` and `core`, as those are the only stable crates today. Keep in mind however that since Cargo -doesn't currently know what crates constitute the standard library for a given `rustc`, that there -is no simple way to correctly implement `.cargo/config` and `[replace]` overrides. Because this is -an experimental feature, I propose just punting and erring should overrides be given. +[System packages would like to build Rust libraries 1 per system packages, and for this Cargo will need to gain some understand of prebuilt binaries. +It is hoped that when it does, the mock registry can be removed and the use of sysroot binaries will be less of a one-off hack.] -### Explicit dependencies and pre-built stdlib from sysroot +Since Cargo will copy any binaries it needs from the sysroot when packages from the mock registry are part of the build plan, it would be nice to always prevent rustc from looking in the sysroot when compiling on Cargo's behalf. +This would prevent users using the sysroot from forgetting to specify any non-default standard library dependencies. +A `--no-resolve-sysroot` flag could be added for this purpose, but this is not necessary. -Build as today, but pass via `--sysroot-whitelist` the list of explicit stdlib dependencies whenever -`implicit-deps` is false. That way, we can be sure regardless of the built strategy that no -undeclared stdlib dependencies "leak in". +## Rustbuild improvements -For overrides, one could filter the writelist of any crates that were overridden. On the other hand -this doesn't take into account other std-lib crates that depended on the overridden crates---they -two should be freshly built and not pulled from the sysroot so as to depend on the overridden -crates. It may be better to punt again then. +When building rust, binaries or source associated with the previous stage are never used, so rustbuild will always pass `--resolve-compiler-specific=` (i.e. that flag with the empty set). -### Implicit dependencies and pre-built stdlib from sysroot +In order to allow building packages that aren't specifically tailored for building rust itself (e.g. they might come from crates.io), Cargo needs to be taught to resolve standard library packages with the current workspace. Either `[[replace]]` can be used for this, or perhaps the members of the workspace would themselves act as a registry. -This is today, and should work as today. Again, punt on the newly-proposed types of overrides. +All binaries for a specific phase can be built with a single `cargo build` (barring special requirements for individual libraries). +Rather than have a multitude of build artifact directories per stage, only one is needed. +After the last compiler is build, an additional mini-stage of building just the standard library could be performed, but distributions wishing to build all deps from source in a standardized fashion (e.g. NixOS) would probably forgo this. # Drawbacks -Notably, all of these relate to the suggested implementation, and not the interface that is the core -of this RFC. I remain optimistic that over time these implementation issues can be resolved without -changing the proposed interface. But my optimism is no more that---speculation. - - - The override limitations already specified within the implementation section. - - - Due to the way git dependencies work, if we naively desugar explicit stdlib dependencies to them - as proposed, packages will be able to depend on *any* crate in the `rust` repo, including the - various `rustc_*` crates there is no intention of keeping around. - - - Cargo doesn't currently have any unstable features, so giving them that requires work and - planning. + - The mock registry for sysroot binaries is a disgusting hack. - - Only some crates in the rust repo (at least `core`, `alloc` and `collections`) can properly be - built just based upon their `Cargo.toml`. However it's precisely only these "foundational" crates - that will be of interest to freestanding developers. Hosted developers can get pre-built sysroots - with `rustup`, and those working on the standard library would use this in conjunction with its - build system which can deal with such things. + - Only some crates in the rust repo (at least `core`, `alloc` and `collections`) can properly be built just based upon their `Cargo.toml`. + However it's precisely only these "foundational" crates that will be of interest to freestanding developers. + Hosted developers can likely get pre-built binaries for the platform they need with `rustup`, just as they do today. - - No means of compiling compiler-rt is proposed. But just as freestanding developers need to - provide `rlibc` or similar to successfully link, I think that for the time-being they can be left - to provide this themselves when linking. + - No means of compiling `compiler-rt` is proposed. + But just as freestanding developers need to provide `rlibc` or similar to successfully link, I think that for the time-being they deal with this themselves. + This is no step backwards. # Alternatives - - It is possible we can just use wildcard dependencies for standard library crates, and don't need - to add a notion of a standard library dependency. Crucially, while there are many different - versions of the standard library, there is only one that works with any given version of `rustc`. + - Instead of copying binaries from the sysroot, we could just leave rustc to find them. + But then a simple `--no-resolve-sysroot` would not work, and the logic for passing `--extern` would need be more complicated. - Specifically, we can have it so whenever Cargo comes across a wildcard crates.io dependency it - can't resolve, assume it is a std lib dependency and either resolve it as such (desugar to git - dependencies) or leave it for `rustc` to find in the sysroot according to the build type. - - I made this to be an alternative because it might be a bit too magical. Also, Semver won't be - able to understand the release trains---pre-release/beta builds are allowed in its grammar but - have different semantics. - - - Simply have `implicit-deps = false` make Cargo pass `--use-sysroot=false` to `rustc`, and don't - have any way to explicitly depend on things in the sysroot. - - - This doesn't by-itself make a way for package to depend on only some of the crates behind the - facade. That, in turn, means Cargo is little better at cross compiling those than before. - - - While unstable compiler users can just package the standard library and depend on it as a - normal crate, it would be weird to have freestanding projects coalesce around some bootleg - core on crates.io. - - - Make it so that packages with implicit dependencies only depend on std. This would be more - elegant, but breaks packages (including ones using stable Rust) that just depend on crates.io. - - - Make it so all dependencies, even libstd, must be explicit. C.f. Cabal and base. Simple and - elegant, but breaks all existing packages. - - - Don't do this, and be stuck with problems detailed in the motivation section. + - Previous versions of this RFC were a simpler but more brittle. + Please refer to the git history to see them. # Unresolved questions - - There are multiple lists of dependencies for different things (e.g. tests, build-time). How - should `implicit-deps = false` affect them? + - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. + Some notion of a trusted package/registry or way to route the secret bootstrap key would be required to fix this. + + - It is unclear what should go in the lockfile when building with sysroot binaries. - - Just as this makes the standard library a real dependency, we can make `rustc` a real - dev-dependency. The standard library can thus be built with Cargo by depending on the associated - unstable compiler. Cargo would need to be taught an "x can build for y" relation for - stable/unstable compiler compatibility however, rather than simply assuming all distinct - compilers are mutually incompatible. This almost certainly is better addressed in a later RFC. + - Whether to add the `--no-resolve-sysroot` flag to rustc, as described above. From 342276306ac210a86c9f7aa6b2c0aa4ec66195ab Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 11 Jul 2016 23:11:14 -0700 Subject: [PATCH 16/66] Fix broken hyperlink --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8c9edbee9fa..54b0509e2c4 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -22,7 +22,7 @@ However Cargo does not know about the standard library, so builds of it are take For cross-compiling, one can often download standard library binaries with rustup. This is convenient, but one cannot expect pre-built binaries for all platforms. In particular, embedded systems often have detailed configurations to convey as much information as possible about the hardware to the compiler. -Furthermore, not all of the stdlib is available on every platform---there is an RFC in the works to pre-build a smaller set of crates for [Cortex-M microcontrollers[(https://github.com/rust-lang/rfcs/pull/1645). +Furthermore, not all of the stdlib is available on every platform---there is an RFC in the works to pre-build a smaller set of crates for [Cortex-M microcontrollers](https://github.com/rust-lang/rfcs/pull/1645). It would be nice to know if the available subset is adequate before attempting a build. We can only do that if all packages have explicit standard library deps to cross-reference with platform requirements of each standard library crate. From 8392c4af0ad573e1b4e68dc4fd36a35fee2e0eed Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 11 Jul 2016 23:54:54 -0700 Subject: [PATCH 17/66] Typos caught by @eternaleye, thanks! --- text/0000-cargo-libstd-awareness.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 54b0509e2c4..6aaf98a194e 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -56,14 +56,14 @@ For backwards compatibility, Cargo must inject such standard library dependencie Exactly which dependencies is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary and build dependency is assured. Now, not all crates depend on `std`, so there must be a way to opt out. -For this, we introduce a new `implicit-deps` key. +For this, we introduce a new `implicit-dependencies` key. It is defined by default as: ```toml implicit-dependencies = ["primary", "build", "dev"] ``` This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit elements. A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. -The one additional rule is `"build"` must be included in the set: we have no plan for Cargoizing the default test runner (either the attribute syntax manipulation or runtime), and wish to be forward-compatible with doing so in the future. +The one additional rule is `"dev"` must be included in the set: we have no plan for Cargoizing the default test runner (either the attribute syntax manipulation or runtime), and wish to be forward-compatible with doing so in the future. Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. @@ -109,7 +109,7 @@ Cargo likewise will have to be conservative with other metadata, e.g. both abort The mock registry will have dead last priority in the default chain, even behind the source registry. The "building" of such a package in the mock registry will consist of copying the binary into the target directory. -[System packages would like to build Rust libraries 1 per system packages, and for this Cargo will need to gain some understand of prebuilt binaries. +[System packages would like to build Rust libraries 1 per system packages, and for this Cargo will need to gain some understanding of prebuilt binaries. It is hoped that when it does, the mock registry can be removed and the use of sysroot binaries will be less of a one-off hack.] Since Cargo will copy any binaries it needs from the sysroot when packages from the mock registry are part of the build plan, it would be nice to always prevent rustc from looking in the sysroot when compiling on Cargo's behalf. From ffc442be30ceca6384ce2a427ebd048c95881578 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 10:46:37 -0700 Subject: [PATCH 18/66] Clarify what implicit deps are, including unresolved issues --- text/0000-cargo-libstd-awareness.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 6aaf98a194e..42d97af92da 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -53,9 +53,12 @@ First and foremost, one will now be able to depend on standard library crates by This will work just as if `std` was on crates.io---features and other modifiers are supported. For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. -Exactly which dependencies is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary and build dependency is assured. +These injected standard library dependencies are called "implicit dependencies" because the user does not specify them explicitly. +Exactly which dependencies will be enjected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary and build dependency is assured. +We also have to account for `core` somehow, as it is now stable so packages using it implicitly too cannot be broken. +Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. -Now, not all crates depend on `std`, so there must be a way to opt out. +Now, not all crates depend on `std`, or whatever the implicit dependencies are decided to be, so there must be a way to opt out. For this, we introduce a new `implicit-dependencies` key. It is defined by default as: ```toml @@ -154,6 +157,10 @@ After the last compiler is build, an additional mini-stage of building just the - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. Some notion of a trusted package/registry or way to route the secret bootstrap key would be required to fix this. + - It is unclear how `core` should be an implicit dependency. + `core = "^1.0"` might work but is a little weird as that core 1.0 is not stabalized. + This relies on users to not `extern crate core;` if they don't use it, to get around the stability warning on older versions of rust, which is rather sketchy. + - It is unclear what should go in the lockfile when building with sysroot binaries. - Whether to add the `--no-resolve-sysroot` flag to rustc, as described above. From 1c53ae1027ab31f5ef34c76f3731f825cd37b7fd Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 11:29:18 -0700 Subject: [PATCH 19/66] Motivate the use of registries, and move the speculative `[ideally..]` bit. --- text/0000-cargo-libstd-awareness.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 42d97af92da..66cfcfe2321 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -98,8 +98,14 @@ Rustup may be able to put it there if the default download does not contain it a ## Cargo implementation The one thing this RFC requires of the upcoming registry implementation is the ability to chain registries providing defaults and fallbacks when a registry is not manually specified. -This is used so crates not on crates.io are instead provided by the compiler. -[Ideally, we'd put the "interface" on crates.io to help ensure compiler-specific implementations conform, but such a mechanism is not being proposed at this time.] +By default, crates.io is used, but somebody may wish to prohibit that, or have Cargo first check a repository of local forks before falling back on crates.io. +This overlaps with `[[replace]]` a bit awkwardly, but is generally useful when the exact set of overrides is subject to change out of band with the current project: +the project's `Cargo.toml` would specify something like `default-registry = [ "local-forks", "crates-io" ]`, along with the defintion of "local-forks". + +This mechanism is used here used so standard library crates not on crates.io (or whatever the users' `default-registry` fallback chains happens to be) are instead provided by the compiler. +We need to avoid forcing users' packages to enumerate the contents of the compiler registry because exactly what packages the compiler provides changes from one version of Rust to the next. +For example, if a portable `collections` is written for example, Cargo should use that rather than the compiler's own package. +This allows transparently making the standard library less-compiler spefific. The injection of implicit dependencies is completely defined by the rules described in the first subsection, so this subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. @@ -142,6 +148,10 @@ After the last compiler is build, an additional mini-stage of building just the But just as freestanding developers need to provide `rlibc` or similar to successfully link, I think that for the time-being they deal with this themselves. This is no step backwards. + - Compilers could provided crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. + [Technically, this probablem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.] + Since the *interface* of the stdlib is specified, it would be neet if we could but a big crate type/interfacex on crates.io, which compiler implementations would need to match. + # Alternatives From cc8a1f2876ed738182e8592c8627a344fa6de9c5 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 17:07:19 -0700 Subject: [PATCH 20/66] Unresolved question on `cargo new` --- text/0000-cargo-libstd-awareness.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 66cfcfe2321..005b83ef0db 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -174,3 +174,6 @@ After the last compiler is build, an additional mini-stage of building just the - It is unclear what should go in the lockfile when building with sysroot binaries. - Whether to add the `--no-resolve-sysroot` flag to rustc, as described above. + + - Should `cargo new` specify `std`, or any other stdlib crates explicitly by default? + I'd hope so! From 71ba6409913ed479ea698ad9c6890ffdc547999b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 18:09:07 -0700 Subject: [PATCH 21/66] Prevent snooping binaries from the sysroot; sysroot binaries not copied --- text/0000-cargo-libstd-awareness.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 005b83ef0db..4cd8b67e58b 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -71,7 +71,7 @@ The one additional rule is `"dev"` must be included in the set: we have no plan Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. -## Cargo Command Line Interface +## Cargo command-line interface A flag will be added ``` @@ -116,14 +116,14 @@ If `bin` is included in the set passed with that flag (or inferred from the defa Any binary in there will be added to the mock registry, with a version deduced the best Cargo can (e.g. from the version of the compiler). Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep. The mock registry will have dead last priority in the default chain, even behind the source registry. -The "building" of such a package in the mock registry will consist of copying the binary into the target directory. +Packages in the mock registry are not built, and when they are (transitive) deps cargo passes them in with `--extern` and their sysroot location. +This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. -[System packages would like to build Rust libraries 1 per system packages, and for this Cargo will need to gain some understanding of prebuilt binaries. -It is hoped that when it does, the mock registry can be removed and the use of sysroot binaries will be less of a one-off hack.] +## Rustc command-line interface -Since Cargo will copy any binaries it needs from the sysroot when packages from the mock registry are part of the build plan, it would be nice to always prevent rustc from looking in the sysroot when compiling on Cargo's behalf. -This would prevent users using the sysroot from forgetting to specify any non-default standard library dependencies. -A `--no-resolve-sysroot` flag could be added for this purpose, but this is not necessary. +Since Cargo is using `--extern` for deps in all cases, rustc's sysroot lookup can and should be bypassed. +This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. +A `--no-resolve-sysroot` flag will be added for this purpose. Cargo will alway use it. ## Rustbuild improvements From 9a61baa5208530c6774cd3afb0ea2ac43b59d912 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 19:45:38 -0700 Subject: [PATCH 22/66] `test` is an implicit dev-dependency. --- text/0000-cargo-libstd-awareness.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 4cd8b67e58b..8a48cd165df 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -54,9 +54,10 @@ This will work just as if `std` was on crates.io---features and other modifiers For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. These injected standard library dependencies are called "implicit dependencies" because the user does not specify them explicitly. -Exactly which dependencies will be enjected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary and build dependency is assured. +Exactly which dependencies will be enjected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary, build, and development dependency is assured. We also have to account for `core` somehow, as it is now stable so packages using it implicitly too cannot be broken. Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. +`test`, the built-in testing framework's runtime, will also be an implicit development dependency. Now, not all crates depend on `std`, or whatever the implicit dependencies are decided to be, so there must be a way to opt out. For this, we introduce a new `implicit-dependencies` key. From 89a461aa21085bf7c28733a0ffd17d7d75ae575e Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 12 Jul 2016 19:46:29 -0700 Subject: [PATCH 23/66] Whether implicit test and dev dependencies are mandatory is unresolved --- text/0000-cargo-libstd-awareness.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8a48cd165df..d2cf33f015b 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -44,7 +44,7 @@ We don't yet have a plan yet to track the language itself, but by tracking stand # Detailed design -The subsections with "interface" in their title form the normitive part of this RFC. +The subsections with "interface" in their title form the normative part of this RFC. The rest of this section just illustrate hows they would likely be implemented. ## `Cargo.toml` Interface @@ -67,7 +67,6 @@ implicit-dependencies = ["primary", "build", "dev"] ``` This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit elements. A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. -The one additional rule is `"dev"` must be included in the set: we have no plan for Cargoizing the default test runner (either the attribute syntax manipulation or runtime), and wish to be forward-compatible with doing so in the future. Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. @@ -101,12 +100,12 @@ Rustup may be able to put it there if the default download does not contain it a The one thing this RFC requires of the upcoming registry implementation is the ability to chain registries providing defaults and fallbacks when a registry is not manually specified. By default, crates.io is used, but somebody may wish to prohibit that, or have Cargo first check a repository of local forks before falling back on crates.io. This overlaps with `[[replace]]` a bit awkwardly, but is generally useful when the exact set of overrides is subject to change out of band with the current project: -the project's `Cargo.toml` would specify something like `default-registry = [ "local-forks", "crates-io" ]`, along with the defintion of "local-forks". +the project's `Cargo.toml` would specify something like `default-registry = [ "local-forks", "crates-io" ]`, along with the definition of "local-forks". This mechanism is used here used so standard library crates not on crates.io (or whatever the users' `default-registry` fallback chains happens to be) are instead provided by the compiler. We need to avoid forcing users' packages to enumerate the contents of the compiler registry because exactly what packages the compiler provides changes from one version of Rust to the next. For example, if a portable `collections` is written for example, Cargo should use that rather than the compiler's own package. -This allows transparently making the standard library less-compiler spefific. +This allows transparently making the standard library less-compiler specific. The injection of implicit dependencies is completely defined by the rules described in the first subsection, so this subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. @@ -150,8 +149,8 @@ After the last compiler is build, an additional mini-stage of building just the This is no step backwards. - Compilers could provided crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. - [Technically, this probablem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.] - Since the *interface* of the stdlib is specified, it would be neet if we could but a big crate type/interfacex on crates.io, which compiler implementations would need to match. + [Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.] + Since the *interface* of the stdlib is specified, it would be neat if we could but a big crate type/interface on crates.io, which compiler implementations would need to match. # Alternatives @@ -162,14 +161,13 @@ After the last compiler is build, an additional mini-stage of building just the - Previous versions of this RFC were a simpler but more brittle. Please refer to the git history to see them. - # Unresolved questions - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. Some notion of a trusted package/registry or way to route the secret bootstrap key would be required to fix this. - It is unclear how `core` should be an implicit dependency. - `core = "^1.0"` might work but is a little weird as that core 1.0 is not stabalized. + `core = "^1.0"` might work but is a little weird as that core 1.0 is not stabilized. This relies on users to not `extern crate core;` if they don't use it, to get around the stability warning on older versions of rust, which is rather sketchy. - It is unclear what should go in the lockfile when building with sysroot binaries. @@ -178,3 +176,9 @@ After the last compiler is build, an additional mini-stage of building just the - Should `cargo new` specify `std`, or any other stdlib crates explicitly by default? I'd hope so! + + - Should one be able to opt-out of implicit build and development dependencies? + I'd like to create a new crate containing testing annotations as compiler plugins, but this entails creating a new sort of test-only plugin dependency (combination of development and build). + Additionally, Currently, it makes sense to always make `std` avaiable for `build.rs` since it must exist for the compiler. + But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we loose an opertunity to be able to express mandatory cross-compiling. + Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. From 4656eb9d2617c45767cb55e96b521fb0a5a2440a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 13 Jul 2016 09:59:59 -0700 Subject: [PATCH 24/66] Typos --- text/0000-cargo-libstd-awareness.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index d2cf33f015b..8805afc03ae 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -54,7 +54,7 @@ This will work just as if `std` was on crates.io---features and other modifiers For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. These injected standard library dependencies are called "implicit dependencies" because the user does not specify them explicitly. -Exactly which dependencies will be enjected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary, build, and development dependency is assured. +Exactly which dependencies will be injected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary, build, and development dependency is assured. We also have to account for `core` somehow, as it is now stable so packages using it implicitly too cannot be broken. Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. `test`, the built-in testing framework's runtime, will also be an implicit development dependency. @@ -123,7 +123,7 @@ This is different from other deps, whose binaries are placed in Cargo's output d Since Cargo is using `--extern` for deps in all cases, rustc's sysroot lookup can and should be bypassed. This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. -A `--no-resolve-sysroot` flag will be added for this purpose. Cargo will alway use it. +A `--no-resolve-sysroot` flag will be added for this purpose. Cargo will always use it. ## Rustbuild improvements @@ -179,6 +179,6 @@ After the last compiler is build, an additional mini-stage of building just the - Should one be able to opt-out of implicit build and development dependencies? I'd like to create a new crate containing testing annotations as compiler plugins, but this entails creating a new sort of test-only plugin dependency (combination of development and build). - Additionally, Currently, it makes sense to always make `std` avaiable for `build.rs` since it must exist for the compiler. - But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we loose an opertunity to be able to express mandatory cross-compiling. + Additionally, Currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. + But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we loose an opportunity to be able to express mandatory cross-compiling. Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. From 984bcb511106db494aca24ef76e06101213de31c Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 13 Jul 2016 11:53:20 -0700 Subject: [PATCH 25/66] Add `language-version` key, and alternative for compiler-rt handling --- text/0000-cargo-libstd-awareness.md | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8805afc03ae..e8c294e9810 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -83,6 +83,23 @@ It is assumed that the version of all sysroot rlibs is the same as the version o If `src` is included, Cargo will fallback on a compiler-provided package registry when other registries (e.g. crates.io) fail to provide a package. For backwards compatibility, the default is `--resolve-compiler-specific=bin`. +## Compiler command-line interface + +As will be explained, with these changes Cargo will always use `--extern` to pass dependencies, so rustc's sysroot lookup can and should be bypassed. +This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. +A `--no-resolve-sysroot` flag will be added for this purpose. +Compilers that don't have sysroot binaries should except this flag and ignore it. +Note that even if this flag is passed, rustc should still pass the sysroot to the linker for the sake of `compiler-rt`. +A `--no-link-sysroot` flag will be added to prevent that. + +Additionally, compilers besides rustc may have version numbers distinct from the version of Rust they implement. +For this purpose, the verbose version output (`$COMPILER -vV` should contain an additional line: +``` +language-version: $version +``` +For now, `$version` should be a version, not version requirement, and the patch number must be zero as patch numbers don't make sense for interfaces. +[This may be relaxed in the future for compilers which implement multiple versions.] + ## Compiler source packaging While the standard library *interface* defined with each rustc version, the implementation, by virtue of using unstable features, is compiler-specific. @@ -113,17 +130,14 @@ If `src` is included in the set passed with that flag, Cargo appends a local reg In other words, if a package is in none of the user-specified registries contain a package, Cargo will look in the registry provided by the compiler. If `bin` is included in the set passed with that flag (or inferred from the default), Cargo will build a mock registry by examining the contents of the sysroot. -Any binary in there will be added to the mock registry, with a version deduced the best Cargo can (e.g. from the version of the compiler). +Any binary in there will be added to the mock registry, with a version +taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep. The mock registry will have dead last priority in the default chain, even behind the source registry. Packages in the mock registry are not built, and when they are (transitive) deps cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. -## Rustc command-line interface - -Since Cargo is using `--extern` for deps in all cases, rustc's sysroot lookup can and should be bypassed. -This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. -A `--no-resolve-sysroot` flag will be added for this purpose. Cargo will always use it. +Since `--extern` is used even in the sysroot binary case, rustc can and will pass `--no-resolve-sysroot` to rustc in all cases. ## Rustbuild improvements @@ -161,6 +175,9 @@ After the last compiler is build, an additional mini-stage of building just the - Previous versions of this RFC were a simpler but more brittle. Please refer to the git history to see them. + - Should `--no-resolve-sysroot` influence linking after all, and `core` have a dep on some `compiler-rt-sys` crate? + This dependency could be a default feature. + # Unresolved questions - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. From af9442b4d132017a1c4d753916bd6220734e0d17 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 13 Jul 2016 12:09:09 -0700 Subject: [PATCH 26/66] Add unresolved questions for fine-grained configuration (e.g. exact CPU) --- text/0000-cargo-libstd-awareness.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index e8c294e9810..768f5e90048 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -199,3 +199,8 @@ After the last compiler is build, an additional mini-stage of building just the Additionally, Currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we loose an opportunity to be able to express mandatory cross-compiling. Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. + + - It is somewhat unclear how Cargo should deal with architecture-specific configuration that is not captured in the target spec nor Cargo feature flags (like CPU features). + [RFC #1645](https://github.com/rust-lang/rfcs/pull/1645) proposes just adding some such configuration to the target triple, whereas https://internals.rust-lang.org/t/pre-rfc-a-vision-for-platform-architecture-configuration-specific-apis/3502/26 proposes a new "scenarios" interfaces. + When building from source, this question is orthogonal to this RFC because it just reuses Cargo's existing methods of keeping binaries for different configurations separate. + When building with sysroot binaries, however, this does matter because cargo needs to deduce or assume exactly what configuration beyond the target triple applies. From d16f8c3d182b8a1c42a3ed242c924d7ffbc7dc8b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 13 Jul 2016 16:24:26 -0700 Subject: [PATCH 27/66] Either both the sysroot source and binaries can resolved, or neither --- text/0000-cargo-libstd-awareness.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 768f5e90048..2998b07e337 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -75,13 +75,12 @@ For example, if a crate explicit depends on `std` as a build-dependency, neither A flag will be added ``` ---resolve-compiler-specific=bin,src +--resolve-compiler-specific-deps= ``` -where either `bin` or `src` may be included in the set. -If `bin` is included, Cargo will allow the use of sysroot binaries to satisfy deps. +If this flag is false, Cargo will only use crates.io (or whatever registries Cargo.toml specifies to use by default). +If this flag is true, Cargo, in the case the package is not found in one of the user specified registries, fallback first on a compiler-provided package registry, and if that fails, allow the use of sysroot binaries to satisfy deps. It is assumed that the version of all sysroot rlibs is the same as the version of Rust which the compiler implements. -If `src` is included, Cargo will fallback on a compiler-provided package registry when other registries (e.g. crates.io) fail to provide a package. -For backwards compatibility, the default is `--resolve-compiler-specific=bin`. +For backwards compatibility, the default is `--resolve-compiler-specific=true`--i.e. this fallbacks will be used. ## Compiler command-line interface @@ -124,24 +123,27 @@ We need to avoid forcing users' packages to enumerate the contents of the compil For example, if a portable `collections` is written for example, Cargo should use that rather than the compiler's own package. This allows transparently making the standard library less-compiler specific. -The injection of implicit dependencies is completely defined by the rules described in the first subsection, so this subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. +Whenever Cargo encounters a `Cargo.toml`, the first thing it always does is inject any applicable implicit deps. +The idea is that by doing this so early on, most of Cargo can stay the same in only knowing or caring about explicit deps, simplifying both this RFC and its implementation. +This process is completely defined by the rules described in the first subsection, so there is really nothing to elaborate upon here. the remainder subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. -If `src` is included in the set passed with that flag, Cargo appends a local registry with path `${$CARGO_RUSTC --print sysroot}/src` to the back of the default chain. -In other words, if a package is in none of the user-specified registries contain a package, Cargo will look in the registry provided by the compiler. +If that flag is false, then everything proceeds as normally. +Dependencies specified by version only are always resolved from crates.io. -If `bin` is included in the set passed with that flag (or inferred from the default), Cargo will build a mock registry by examining the contents of the sysroot. +If that flag is true, Cargo behaves as if two registries were appended (with lowest priority), first the compiler source registry and then the compiler binary mock registry. +The first, with second-lowest priority, the compiler source registry, is located at `${$CARGO_RUSTC --print sysroot}/src`. +The second, with absolute lowest priority, the compiler binary mock registry, is generated by examining the contents of the sysroot. Any binary in there will be added to the mock registry, with a version taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep. -The mock registry will have dead last priority in the default chain, even behind the source registry. -Packages in the mock registry are not built, and when they are (transitive) deps cargo passes them in with `--extern` and their sysroot location. +Packages in the mock registry are not built, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. Since `--extern` is used even in the sysroot binary case, rustc can and will pass `--no-resolve-sysroot` to rustc in all cases. ## Rustbuild improvements -When building rust, binaries or source associated with the previous stage are never used, so rustbuild will always pass `--resolve-compiler-specific=` (i.e. that flag with the empty set). +When building rust, binaries or source associated with the previous stage are never used, so rustbuild will always pass `--resolve-compiler-specific=false`. In order to allow building packages that aren't specifically tailored for building rust itself (e.g. they might come from crates.io), Cargo needs to be taught to resolve standard library packages with the current workspace. Either `[[replace]]` can be used for this, or perhaps the members of the workspace would themselves act as a registry. From 77be419c095e6779c5e49faadbb6771b6fbb1ecf Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 13 Jul 2016 16:36:09 -0700 Subject: [PATCH 28/66] Add parenthetical. --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 2998b07e337..d545ce6970a 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -135,7 +135,7 @@ The first, with second-lowest priority, the compiler source registry, is located The second, with absolute lowest priority, the compiler binary mock registry, is generated by examining the contents of the sysroot. Any binary in there will be added to the mock registry, with a version taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). -Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep. +Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep (lest there are some implicit features it doesn't know about). Packages in the mock registry are not built, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. From afae118c49fc0eb596bd9434ea24d7bbacef60ac Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 26 Jul 2016 19:36:39 -0700 Subject: [PATCH 29/66] Use `stdlib = true|false` rather than crates.io fallback to begin with. Also fix miscellaneous things --- text/0000-cargo-libstd-awareness.md | 132 +++++++++++++++++----------- 1 file changed, 82 insertions(+), 50 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index d545ce6970a..05ec461f017 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -34,7 +34,7 @@ Cargo would be able to infer both of these options by inspecting all crates in t Rustbuild must currently perform multiple `cargo builds`, the first to build the standard library and the rest to build things which depend on the standard library. If rustc, rustfmt, etc, and their deps (some of which come from crates.io, and thus aren't specially tailored for building rust) declare deps on std, rustbuild wouldn't need multiple lockfiles. -Keeping multiple in sync is nuisance, and this gets us one step closer to a single `cargo build` building rustc. +Keeping multiple lockfiles in sync is nuisance, and this gets us one step closer to a single `cargo build` building rustc. The use-cases so far mainly benefit niche corners of the Rust community, but the last should be useful for just about everyone. Now that multiple versions of Rust have been released, it can be useful to specify the minimum version. @@ -44,22 +44,41 @@ We don't yet have a plan yet to track the language itself, but by tracking stand # Detailed design -The subsections with "interface" in their title form the normative part of this RFC. -The rest of this section just illustrate hows they would likely be implemented. +## Standard library dependencies -## `Cargo.toml` Interface +First and foremost, one will now be able to explicitly on standard library, e.g. with `std = { version = "1.10", stdlib = true }`. +From the users's perspective, `stdlib = true` simply indicates that the dependendend on crate is from the standard library. +The version for stdlib crates comes from the version of Rust their interfaces are defined in. +A version requirement must be specified. +The full beadth of options available with our existing dependencies, e.g. features and overrides, will be supported. -First and foremost, one will now be able to depend on standard library crates by version, e.g. `std = "1.10"`. -This will work just as if `std` was on crates.io---features and other modifiers are supported. +## Implicit dependencies For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. These injected standard library dependencies are called "implicit dependencies" because the user does not specify them explicitly. -Exactly which dependencies will be injected is unresolved, but a requirement at least as strong as `std = "^1.0"` as a primary, build, and development dependency is assured. -We also have to account for `core` somehow, as it is now stable so packages using it implicitly too cannot be broken. +We have an obligation to not break packages depending only on stable interfaces, so the implicit dependencies will include both `std` and `core`: +```toml +[dependencies] +core = { version = "^1.0", stdlib = true } +std = { version = "^1.0", stdlib = true } + +[dev-dependencies] +core = { version = "^1.0", stdlib = true } +std = { version = "^1.0", stdlib = true } +test = { version = "^1.0", stdlib = true } + +[build-dependencies] +core = { version = "^1.0", stdlib = true } +std = { version = "^1.0", stdlib = true } +``` +The version requirement for `core` of `^1.0` may seem odd because core was not stable at the time, but anything else would either break newer packages using core, or prevent older packages from working on versions of Rust predating cores stabilization. +Remember that rustc only complains if a unstable crate is actually imported, so the Cargo dependency on its own is harmless. + +`test` is a similar scenario. +While importing it explicitly remains unstable, is it currently injected and thus needs to be built. Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. -`test`, the built-in testing framework's runtime, will also be an implicit development dependency. -Now, not all crates depend on `std`, or whatever the implicit dependencies are decided to be, so there must be a way to opt out. +Now, not all crates depend on these crates, so there must be a way to opt out. For this, we introduce a new `implicit-dependencies` key. It is defined by default as: ```toml @@ -70,17 +89,8 @@ A manual definition may be that or almost any subset, in which case only the inc Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. - -## Cargo command-line interface - -A flag will be added -``` ---resolve-compiler-specific-deps= -``` -If this flag is false, Cargo will only use crates.io (or whatever registries Cargo.toml specifies to use by default). -If this flag is true, Cargo, in the case the package is not found in one of the user specified registries, fallback first on a compiler-provided package registry, and if that fails, allow the use of sysroot binaries to satisfy deps. -It is assumed that the version of all sysroot rlibs is the same as the version of Rust which the compiler implements. -For backwards compatibility, the default is `--resolve-compiler-specific=true`--i.e. this fallbacks will be used. +This final rule means must packages won't need to use the implicit-dependencies key, because either implicit dependencies will be used, or a `core` or `std` explicit dependency will be present. +An exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. ## Compiler command-line interface @@ -101,55 +111,81 @@ For now, `$version` should be a version, not version requirement, and the patch ## Compiler source packaging -While the standard library *interface* defined with each rustc version, the implementation, by virtue of using unstable features, is compiler-specific. +While the standard library *interface* defined with each rustc version, the implementation of many crates, by virtue of using unstable features, is compiler-specific. This makes the standard library unfit for crates.io. (Additionally, the issue of dealing with nightly also makes crates.io hard to use, but that is a less clear-cut obstacle.) -Cargo will soon gain the ability to create and chain custom registries, as described in -https://github.com/rust-lang/cargo/pull/2361 . +Cargo will soon gain the ability to create custom registries, as described in +https://github.com/rust-lang/cargo/pull/2857 . Compiler's should package the source of their implementation of the standard library as a registry, which can be distributed with the compiler. In practice, rustc will optionally contain the source in its sysroot. Rustup may be able to put it there if the default download does not contain it already. +This registry will be called the "compiler source registry". ## Cargo implementation -The one thing this RFC requires of the upcoming registry implementation is the ability to chain registries providing defaults and fallbacks when a registry is not manually specified. -By default, crates.io is used, but somebody may wish to prohibit that, or have Cargo first check a repository of local forks before falling back on crates.io. -This overlaps with `[[replace]]` a bit awkwardly, but is generally useful when the exact set of overrides is subject to change out of band with the current project: -the project's `Cargo.toml` would specify something like `default-registry = [ "local-forks", "crates-io" ]`, along with the definition of "local-forks". - -This mechanism is used here used so standard library crates not on crates.io (or whatever the users' `default-registry` fallback chains happens to be) are instead provided by the compiler. -We need to avoid forcing users' packages to enumerate the contents of the compiler registry because exactly what packages the compiler provides changes from one version of Rust to the next. -For example, if a portable `collections` is written for example, Cargo should use that rather than the compiler's own package. -This allows transparently making the standard library less-compiler specific. - Whenever Cargo encounters a `Cargo.toml`, the first thing it always does is inject any applicable implicit deps. The idea is that by doing this so early on, most of Cargo can stay the same in only knowing or caring about explicit deps, simplifying both this RFC and its implementation. -This process is completely defined by the rules described in the first subsection, so there is really nothing to elaborate upon here. the remainder subsection will focus on the meaning of the `--resolve-compiler-specific=` flag. +This process is completely defined by the rules described in the first subsection, so there is really nothing to elaborate upon here. -If that flag is false, then everything proceeds as normally. -Dependencies specified by version only are always resolved from crates.io. +Cargo has an "source" abstraction for providers of packages. +Examples of this are file-system paths, git repositories, and the upcoming registries. +For the time being, `stdlib = true` will act like another source, instructing the compiler to look at the compiler source registry, and if that is missing, then the sysroot binaries. +The remainder of this section will clarify this. -If that flag is true, Cargo behaves as if two registries were appended (with lowest priority), first the compiler source registry and then the compiler binary mock registry. -The first, with second-lowest priority, the compiler source registry, is located at `${$CARGO_RUSTC --print sysroot}/src`. -The second, with absolute lowest priority, the compiler binary mock registry, is generated by examining the contents of the sysroot. +The compiler source registry is expected to be located in `${$CARGO_RUSTC --print sysroot}/src`. +The exact format this takes will be determined when custom registries are implemented. +Based on that, it may be necessary to transform the rust repo before dumping it there. + +The "sysroot binary mock registry" is generated by examining the contents of the sysroot. Any binary in there will be added to the mock registry, with a version taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). -Cargo likewise will have to be conservative with other metadata, e.g. both aborting if any feature is requested of a dep that is resolved to this mock registry, and also aborting if `default-features = false` is specified in such a dep (lest there are some implicit features it doesn't know about). -Packages in the mock registry are not built, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. -This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. +If the compiler source registry exists, that is used, and the sysroot binary mock registry need not even be built. +If the source registry is absent, then the binary registry is used. + +When the build plan just involves the compiler source registry and/or existing types of sources, it can be executed just like today. +The awkward scenario is when packages from the sysroot binary mock registry need to be used in the build plan. +Because Cargo doesn't know much about the sysroot binaries, it must be very conservative when deciding whether or not they can be used. +For example, Cargo may assume they are built with only default features enabled but it can't know what those are. +If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan being resolved. +Cargo likewise will have to be conservative with other metadata, +Packages in the mock binaries registry are not built by Cargo, since they are prebuilt, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. +This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. Since `--extern` is used even in the sysroot binary case, rustc can and will pass `--no-resolve-sysroot` to rustc in all cases. ## Rustbuild improvements -When building rust, binaries or source associated with the previous stage are never used, so rustbuild will always pass `--resolve-compiler-specific=false`. +As advertised in the motivation section, with this RFC, rustbuild can use a single workspace to build the standard library and all executables. -In order to allow building packages that aren't specifically tailored for building rust itself (e.g. they might come from crates.io), Cargo needs to be taught to resolve standard library packages with the current workspace. Either `[[replace]]` can be used for this, or perhaps the members of the workspace would themselves act as a registry. +One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are never used; one needs to bypass the compiler source registry and sysroot binary mock registry. +To accomplish this, rustbuild's workspace will need to use `[replace]` to redirect all stdlib deps to use the workspace's packages. All binaries for a specific phase can be built with a single `cargo build` (barring special requirements for individual libraries). Rather than have a multitude of build artifact directories per stage, only one is needed. -After the last compiler is build, an additional mini-stage of building just the standard library could be performed, but distributions wishing to build all deps from source in a standardized fashion (e.g. NixOS) would probably forgo this. +After the last compiler is build, an additional mini-stage of building just the standard library could be performed, but distributions wishing to build all deps from source in a standardized fashion (e.g. probably NixOS) would forgo this. + +## Forward Compatibility + +The custom registries [PR #2857](https://github.com/rust-lang/cargo/pull/2857) starts with just mirroring existing registries. +As followup work, it expected that packages (probably just the workspace root, definitely not non-packages like cargo config) will be able to specify the "default" source, i.e. the one used when none is specified (today this is always crates.io). +Similarly, one could specify a "stdlib" source, to be used for `std = true` deps instead of the compiler source binary registry or sysroot binary mock registry. +This would simplify rustbuild as it could use that once instead of `[replace]` for each package. +This doesn't require any planning from this RFC. + +More important, it would be nice to move stdlib crates that don't use unstable features to crates.io. +`collections` and `test` almost don't use any unstable and are thus good candidates for this. +With something like what is described in the first paragraph, it could be possible for individual packages instruct Cargo to first check crates.io, and then the compiler source registry, for stdlib crates. +But this shifts the burden to individual packages, and means we'd still need to vendor source of any crate moved to crates.io in the compiler source registry for packages that didn't make the switch. + +More interesting would be to change Cargo's behavior so packages not found with the compiler src or in the sysroot are retrieved from crates.io---or vice versa (crates.io, then compiler/sysroot). +Unioning the sources in either fashion would allow standard library crates to be seamlessly moved out of the compiler registry and onto crates.io without extra work per package. +We don't want to commit either priority or even unioning at all in this RFC, however, so we instead want to keep all 3 options open (no unioning, crates.io over compiler-specific, compiler-specific over crates.io). +To achieve this, the goal is that the sysroot/compiler source registry and crates.io should contain disjoint packages. +That way the sources can be unioned without conflict. +As a guideline, all but the most transient stdlib crates should have their name reserved on crates.io. +As a policy, crates.io should only allow pushing packages whose stdlib deps are reserved. +This will keep our options open. # Drawbacks @@ -185,10 +221,6 @@ After the last compiler is build, an additional mini-stage of building just the - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. Some notion of a trusted package/registry or way to route the secret bootstrap key would be required to fix this. - - It is unclear how `core` should be an implicit dependency. - `core = "^1.0"` might work but is a little weird as that core 1.0 is not stabilized. - This relies on users to not `extern crate core;` if they don't use it, to get around the stability warning on older versions of rust, which is rather sketchy. - - It is unclear what should go in the lockfile when building with sysroot binaries. - Whether to add the `--no-resolve-sysroot` flag to rustc, as described above. From a1f8ab1397b06ee9f565219f5108e0ade733456e Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 27 Jul 2016 10:42:51 -0700 Subject: [PATCH 30/66] Clarify the forward compatibility section --- text/0000-cargo-libstd-awareness.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 05ec461f017..d76418ab83c 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -178,14 +178,19 @@ More important, it would be nice to move stdlib crates that don't use unstable f With something like what is described in the first paragraph, it could be possible for individual packages instruct Cargo to first check crates.io, and then the compiler source registry, for stdlib crates. But this shifts the burden to individual packages, and means we'd still need to vendor source of any crate moved to crates.io in the compiler source registry for packages that didn't make the switch. -More interesting would be to change Cargo's behavior so packages not found with the compiler src or in the sysroot are retrieved from crates.io---or vice versa (crates.io, then compiler/sysroot). -Unioning the sources in either fashion would allow standard library crates to be seamlessly moved out of the compiler registry and onto crates.io without extra work per package. -We don't want to commit either priority or even unioning at all in this RFC, however, so we instead want to keep all 3 options open (no unioning, crates.io over compiler-specific, compiler-specific over crates.io). -To achieve this, the goal is that the sysroot/compiler source registry and crates.io should contain disjoint packages. -That way the sources can be unioned without conflict. -As a guideline, all but the most transient stdlib crates should have their name reserved on crates.io. -As a policy, crates.io should only allow pushing packages whose stdlib deps are reserved. -This will keep our options open. +More interesting would be to change Cargo's *default* behavior to check both the compiler-specific sources (compiler source registry and sysroot binary mock registry) and crates.io. +This would allow standard crates to seamless migrate to crates.io without extra work per package. +This could be either be done where crates.io overrides the compiler-specific sources, or the compiler specific sources override crates.io. +We don't want to commit to either variant in this RFC, however, so we instead want to keep all 3 options open (no fallback, crates.io over compiler-specific, compiler-specific over crates.io). +To achieve this, we want to keep sysroot/compiler source registry and crates.io should contain disjoint packages. +That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same affect because there are no packages provided by both. + +The easiest way to achieve this is to make sure that standard library crates use names reserved on crates.io. +We don't want to bake crates.io policy into Cargo however, so instead of absolutely prohibiting stdlib deps with non-reserved names, crates.io will just lint packages being uploaded. +Also, care will be taken so that any stdlib crate that is stabalized must use a reserved name or already be published on crates.io. +That still doesn't protect unpublished packages using unstable stdlib crates without reserved names from breakage, but due to their use of unstable interfaces we have no obligation to keep them working. +Also, once we have an option to explicit provide the source for stdlib deps, they can force the behavior they want. +This seems good enough. # Drawbacks From 2334770b51b375b76d694e7e50c95f130c05ea3e Mon Sep 17 00:00:00 2001 From: jethrogb Date: Sun, 31 Jul 2016 22:46:19 -0700 Subject: [PATCH 31/66] Copy-edit 0000-cargo-libstd-awareness.md --- text/0000-cargo-libstd-awareness.md | 58 ++++++++++++++--------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index d76418ab83c..e4931f5ea13 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -34,7 +34,7 @@ Cargo would be able to infer both of these options by inspecting all crates in t Rustbuild must currently perform multiple `cargo builds`, the first to build the standard library and the rest to build things which depend on the standard library. If rustc, rustfmt, etc, and their deps (some of which come from crates.io, and thus aren't specially tailored for building rust) declare deps on std, rustbuild wouldn't need multiple lockfiles. -Keeping multiple lockfiles in sync is nuisance, and this gets us one step closer to a single `cargo build` building rustc. +Keeping multiple lockfiles in sync is a nuisance, and this gets us one step closer to a single `cargo build` building rustc. The use-cases so far mainly benefit niche corners of the Rust community, but the last should be useful for just about everyone. Now that multiple versions of Rust have been released, it can be useful to specify the minimum version. @@ -46,11 +46,11 @@ We don't yet have a plan yet to track the language itself, but by tracking stand ## Standard library dependencies -First and foremost, one will now be able to explicitly on standard library, e.g. with `std = { version = "1.10", stdlib = true }`. -From the users's perspective, `stdlib = true` simply indicates that the dependendend on crate is from the standard library. +First and foremost, one will now be able to explicitly depend on the standard library, e.g. with `std = { version = "1.10", stdlib = true }`. +From the users's perspective, `stdlib = true` simply indicates that the depended on crate is from the standard library. The version for stdlib crates comes from the version of Rust their interfaces are defined in. A version requirement must be specified. -The full beadth of options available with our existing dependencies, e.g. features and overrides, will be supported. +The full breadth of options available with our existing dependencies, e.g. features and overrides, will be supported. ## Implicit dependencies @@ -71,11 +71,11 @@ test = { version = "^1.0", stdlib = true } core = { version = "^1.0", stdlib = true } std = { version = "^1.0", stdlib = true } ``` -The version requirement for `core` of `^1.0` may seem odd because core was not stable at the time, but anything else would either break newer packages using core, or prevent older packages from working on versions of Rust predating cores stabilization. -Remember that rustc only complains if a unstable crate is actually imported, so the Cargo dependency on its own is harmless. +The version requirement for `core` of `^1.0` may seem odd because core was not stable at the time, but anything else would either break newer packages using core, or prevent older packages from working on versions of Rust predating core's stabilization. +Remember that rustc only complains if an unstable crate is actually imported, so the Cargo dependency on its own is harmless. `test` is a similar scenario. -While importing it explicitly remains unstable, is it currently injected and thus needs to be built. +While importing it explicitly remains unstable, it's currently injected and thus needs to be built. Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. Now, not all crates depend on these crates, so there must be a way to opt out. @@ -97,7 +97,7 @@ An exception is `core` itself, which must of course not depend on `core` or `std As will be explained, with these changes Cargo will always use `--extern` to pass dependencies, so rustc's sysroot lookup can and should be bypassed. This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. A `--no-resolve-sysroot` flag will be added for this purpose. -Compilers that don't have sysroot binaries should except this flag and ignore it. +Compilers that don't have sysroot binaries should accept this flag and ignore it. Note that even if this flag is passed, rustc should still pass the sysroot to the linker for the sake of `compiler-rt`. A `--no-link-sysroot` flag will be added to prevent that. @@ -106,18 +106,18 @@ For this purpose, the verbose version output (`$COMPILER -vV` should contain an ``` language-version: $version ``` -For now, `$version` should be a version, not version requirement, and the patch number must be zero as patch numbers don't make sense for interfaces. +For now, `$version` should be a version, not a version requirement, and the patch number must be zero as patch numbers don't make sense for interfaces. [This may be relaxed in the future for compilers which implement multiple versions.] ## Compiler source packaging -While the standard library *interface* defined with each rustc version, the implementation of many crates, by virtue of using unstable features, is compiler-specific. +While the standard library *interface* is defined with each rustc version, the implementation of many crates, by virtue of using unstable features, is compiler-specific. This makes the standard library unfit for crates.io. (Additionally, the issue of dealing with nightly also makes crates.io hard to use, but that is a less clear-cut obstacle.) Cargo will soon gain the ability to create custom registries, as described in https://github.com/rust-lang/cargo/pull/2857 . -Compiler's should package the source of their implementation of the standard library as a registry, which can be distributed with the compiler. +Compilers should package the source of their implementation of the standard library as a registry, which can be distributed with the compiler. In practice, rustc will optionally contain the source in its sysroot. Rustup may be able to put it there if the default download does not contain it already. This registry will be called the "compiler source registry". @@ -128,7 +128,7 @@ Whenever Cargo encounters a `Cargo.toml`, the first thing it always does is inje The idea is that by doing this so early on, most of Cargo can stay the same in only knowing or caring about explicit deps, simplifying both this RFC and its implementation. This process is completely defined by the rules described in the first subsection, so there is really nothing to elaborate upon here. -Cargo has an "source" abstraction for providers of packages. +Cargo has a "source" abstraction for providers of packages. Examples of this are file-system paths, git repositories, and the upcoming registries. For the time being, `stdlib = true` will act like another source, instructing the compiler to look at the compiler source registry, and if that is missing, then the sysroot binaries. The remainder of this section will clarify this. @@ -149,16 +149,16 @@ The awkward scenario is when packages from the sysroot binary mock registry need Because Cargo doesn't know much about the sysroot binaries, it must be very conservative when deciding whether or not they can be used. For example, Cargo may assume they are built with only default features enabled but it can't know what those are. If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan being resolved. -Cargo likewise will have to be conservative with other metadata, +Cargo likewise will have to be conservative with other metadata. Packages in the mock binaries registry are not built by Cargo, since they are prebuilt, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. -Since `--extern` is used even in the sysroot binary case, rustc can and will pass `--no-resolve-sysroot` to rustc in all cases. +Since `--extern` is used even in the sysroot binary case, cargo can and will pass `--no-resolve-sysroot` to rustc in all cases. ## Rustbuild improvements As advertised in the motivation section, with this RFC, rustbuild can use a single workspace to build the standard library and all executables. -One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are never used; one needs to bypass the compiler source registry and sysroot binary mock registry. +One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are ever used; one needs to bypass the compiler source registry and sysroot binary mock registry. To accomplish this, rustbuild's workspace will need to use `[replace]` to redirect all stdlib deps to use the workspace's packages. All binaries for a specific phase can be built with a single `cargo build` (barring special requirements for individual libraries). @@ -167,29 +167,29 @@ After the last compiler is build, an additional mini-stage of building just the ## Forward Compatibility -The custom registries [PR #2857](https://github.com/rust-lang/cargo/pull/2857) starts with just mirroring existing registries. +The custom registries [PR #2857](https://github.com/rust-lang/cargo/pull/2857) start with just mirroring existing registries. As followup work, it expected that packages (probably just the workspace root, definitely not non-packages like cargo config) will be able to specify the "default" source, i.e. the one used when none is specified (today this is always crates.io). -Similarly, one could specify a "stdlib" source, to be used for `std = true` deps instead of the compiler source binary registry or sysroot binary mock registry. +Similarly, one could specify a "stdlib" source, to be used for `stdlib = true` deps instead of the compiler source binary registry or sysroot binary mock registry. This would simplify rustbuild as it could use that once instead of `[replace]` for each package. This doesn't require any planning from this RFC. -More important, it would be nice to move stdlib crates that don't use unstable features to crates.io. +More importantly, it would be nice to move stdlib crates that don't use unstable features to crates.io. `collections` and `test` almost don't use any unstable and are thus good candidates for this. -With something like what is described in the first paragraph, it could be possible for individual packages instruct Cargo to first check crates.io, and then the compiler source registry, for stdlib crates. +With something like what is described in the first paragraph, it could be possible for individual packages to instruct Cargo to first check crates.io, and then the compiler source registry, for stdlib crates. But this shifts the burden to individual packages, and means we'd still need to vendor source of any crate moved to crates.io in the compiler source registry for packages that didn't make the switch. More interesting would be to change Cargo's *default* behavior to check both the compiler-specific sources (compiler source registry and sysroot binary mock registry) and crates.io. -This would allow standard crates to seamless migrate to crates.io without extra work per package. +This would allow standard crates to seamlessly migrate to crates.io without extra work per package. This could be either be done where crates.io overrides the compiler-specific sources, or the compiler specific sources override crates.io. We don't want to commit to either variant in this RFC, however, so we instead want to keep all 3 options open (no fallback, crates.io over compiler-specific, compiler-specific over crates.io). -To achieve this, we want to keep sysroot/compiler source registry and crates.io should contain disjoint packages. -That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same affect because there are no packages provided by both. +To achieve this, we want to keep the set of packages in sysroot/compiler source registry and crates.io disjoint. +That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same effect because there are no packages provided by both. The easiest way to achieve this is to make sure that standard library crates use names reserved on crates.io. We don't want to bake crates.io policy into Cargo however, so instead of absolutely prohibiting stdlib deps with non-reserved names, crates.io will just lint packages being uploaded. Also, care will be taken so that any stdlib crate that is stabalized must use a reserved name or already be published on crates.io. That still doesn't protect unpublished packages using unstable stdlib crates without reserved names from breakage, but due to their use of unstable interfaces we have no obligation to keep them working. -Also, once we have an option to explicit provide the source for stdlib deps, they can force the behavior they want. +Also, once we have an option to explicitly provide the source for stdlib deps, they can force the behavior they want. This seems good enough. @@ -205,14 +205,14 @@ This seems good enough. But just as freestanding developers need to provide `rlibc` or similar to successfully link, I think that for the time-being they deal with this themselves. This is no step backwards. - - Compilers could provided crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. - [Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.] - Since the *interface* of the stdlib is specified, it would be neat if we could but a big crate type/interface on crates.io, which compiler implementations would need to match. + - Compilers could provide crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. + (Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.) + Since the *interface* of the stdlib is specified, it would be neat if we could put a big crate type/interface on crates.io, which compiler implementations would need to match. # Alternatives - - Instead of copying binaries from the sysroot, we could just leave rustc to find them. + - Instead of copying binaries from the sysroot, we could just leave it to rustc to find them. But then a simple `--no-resolve-sysroot` would not work, and the logic for passing `--extern` would need be more complicated. - Previous versions of this RFC were a simpler but more brittle. @@ -235,8 +235,8 @@ This seems good enough. - Should one be able to opt-out of implicit build and development dependencies? I'd like to create a new crate containing testing annotations as compiler plugins, but this entails creating a new sort of test-only plugin dependency (combination of development and build). - Additionally, Currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. - But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we loose an opportunity to be able to express mandatory cross-compiling. + Additionally, currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. + But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we lose an opportunity to be able to express mandatory cross-compiling. Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. - It is somewhat unclear how Cargo should deal with architecture-specific configuration that is not captured in the target spec nor Cargo feature flags (like CPU features). From 61cd8ca19d60544e4df4168f1185254a3992aa88 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 1 Aug 2016 00:01:48 -0700 Subject: [PATCH 32/66] Revise a view corrections / add more of my own. --- text/0000-cargo-libstd-awareness.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index e4931f5ea13..46afafe9652 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -46,8 +46,8 @@ We don't yet have a plan yet to track the language itself, but by tracking stand ## Standard library dependencies -First and foremost, one will now be able to explicitly depend on the standard library, e.g. with `std = { version = "1.10", stdlib = true }`. -From the users's perspective, `stdlib = true` simply indicates that the depended on crate is from the standard library. +First and foremost, one will now be able to explicitly depend on standard library crates, e.g. with `std = { version = "1.10", stdlib = true }`. +From the users's perspective, `stdlib = true` simply indicates that the depended-on crate is from the standard library. The version for stdlib crates comes from the version of Rust their interfaces are defined in. A version requirement must be specified. The full breadth of options available with our existing dependencies, e.g. features and overrides, will be supported. @@ -167,7 +167,7 @@ After the last compiler is build, an additional mini-stage of building just the ## Forward Compatibility -The custom registries [PR #2857](https://github.com/rust-lang/cargo/pull/2857) start with just mirroring existing registries. +The custom registries PR https://github.com/rust-lang/cargo/pull/2857 starts with just mirroring existing registries. As followup work, it expected that packages (probably just the workspace root, definitely not non-packages like cargo config) will be able to specify the "default" source, i.e. the one used when none is specified (today this is always crates.io). Similarly, one could specify a "stdlib" source, to be used for `stdlib = true` deps instead of the compiler source binary registry or sysroot binary mock registry. This would simplify rustbuild as it could use that once instead of `[replace]` for each package. @@ -182,8 +182,8 @@ More interesting would be to change Cargo's *default* behavior to check both the This would allow standard crates to seamlessly migrate to crates.io without extra work per package. This could be either be done where crates.io overrides the compiler-specific sources, or the compiler specific sources override crates.io. We don't want to commit to either variant in this RFC, however, so we instead want to keep all 3 options open (no fallback, crates.io over compiler-specific, compiler-specific over crates.io). -To achieve this, we want to keep the set of packages in sysroot/compiler source registry and crates.io disjoint. -That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same effect because there are no packages provided by both. +To achieve this, we want to keep the sysroot/compiler source registry and crates.io disjoint: no package should be contained in both sources. +That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same effect. The easiest way to achieve this is to make sure that standard library crates use names reserved on crates.io. We don't want to bake crates.io policy into Cargo however, so instead of absolutely prohibiting stdlib deps with non-reserved names, crates.io will just lint packages being uploaded. @@ -234,8 +234,7 @@ This seems good enough. I'd hope so! - Should one be able to opt-out of implicit build and development dependencies? - I'd like to create a new crate containing testing annotations as compiler plugins, but this entails creating a new sort of test-only plugin dependency (combination of development and build). - Additionally, currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. + Currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we lose an opportunity to be able to express mandatory cross-compiling. Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. From fac0c9bf5d72adbd959f47a5a96e45ffafc7eb9c Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 4 Aug 2016 10:36:30 -0700 Subject: [PATCH 33/66] By the time this lands, Compiler-rt will be tamed. Thank you @japaric! --- text/0000-cargo-libstd-awareness.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 46afafe9652..f279e2eee85 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -98,8 +98,7 @@ As will be explained, with these changes Cargo will always use `--extern` to pas This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. A `--no-resolve-sysroot` flag will be added for this purpose. Compilers that don't have sysroot binaries should accept this flag and ignore it. -Note that even if this flag is passed, rustc should still pass the sysroot to the linker for the sake of `compiler-rt`. -A `--no-link-sysroot` flag will be added to prevent that. +Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so rustc won't use the sysroot for linking either. Additionally, compilers besides rustc may have version numbers distinct from the version of Rust they implement. For this purpose, the verbose version output (`$COMPILER -vV` should contain an additional line: @@ -201,10 +200,6 @@ This seems good enough. However it's precisely only these "foundational" crates that will be of interest to freestanding developers. Hosted developers can likely get pre-built binaries for the platform they need with `rustup`, just as they do today. - - No means of compiling `compiler-rt` is proposed. - But just as freestanding developers need to provide `rlibc` or similar to successfully link, I think that for the time-being they deal with this themselves. - This is no step backwards. - - Compilers could provide crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. (Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.) Since the *interface* of the stdlib is specified, it would be neat if we could put a big crate type/interface on crates.io, which compiler implementations would need to match. @@ -218,9 +213,6 @@ This seems good enough. - Previous versions of this RFC were a simpler but more brittle. Please refer to the git history to see them. - - Should `--no-resolve-sysroot` influence linking after all, and `core` have a dep on some `compiler-rt-sys` crate? - This dependency could be a default feature. - # Unresolved questions - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. From a062095b66b812ebafd4d4b51c8ea0df842971ed Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 5 Aug 2016 16:59:00 -0700 Subject: [PATCH 34/66] Extra 'a' typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index f279e2eee85..aede587d920 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -210,7 +210,7 @@ This seems good enough. - Instead of copying binaries from the sysroot, we could just leave it to rustc to find them. But then a simple `--no-resolve-sysroot` would not work, and the logic for passing `--extern` would need be more complicated. - - Previous versions of this RFC were a simpler but more brittle. + - Previous versions of this RFC were simpler but more brittle. Please refer to the git history to see them. # Unresolved questions From bbf1ec6a0391a380e724f91db29c25955b5dcb6f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 5 Aug 2016 17:01:56 -0700 Subject: [PATCH 35/66] "bounds" -> "semver requirements" --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index aede587d920..5bd0f0ee19a 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -7,7 +7,7 @@ # Summary Currently, Cargo doesn't know what parts of the standard library packages depend on. -By giving it this knowledge, we can make cross compilation and exotic platform development easier, simplify rustbuild, and allow anyone to easily specify bounds on the standard library for their packages. +By giving it this knowledge, we can make cross compilation and exotic platform development easier, simplify rustbuild, and allow anyone to easily specify semver requirements on the standard library for their packages. This will allow building parts of the standard library from source, but in order to not disrupt existing workflows, the binaries that come with rustc will still be used by default. From 81a923ad94f0fcaf12348a5c35513ae97ef0f21f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 5 Aug 2016 17:37:59 -0700 Subject: [PATCH 36/66] Clarify "at the time": "in Rust 1.0" --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 5bd0f0ee19a..36a6a3574e9 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -71,7 +71,7 @@ test = { version = "^1.0", stdlib = true } core = { version = "^1.0", stdlib = true } std = { version = "^1.0", stdlib = true } ``` -The version requirement for `core` of `^1.0` may seem odd because core was not stable at the time, but anything else would either break newer packages using core, or prevent older packages from working on versions of Rust predating core's stabilization. +The version requirement for `core` of `^1.0` may seem odd because core was not stable in Rust 1.0, but anything else would either break newer packages using core, or prevent older packages from working on versions of Rust predating core's stabilization. Remember that rustc only complains if an unstable crate is actually imported, so the Cargo dependency on its own is harmless. `test` is a similar scenario. From 51cfdb6ea9fa041b9831422f05b5885970247d47 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 10:59:22 -0700 Subject: [PATCH 37/66] Remove stale alternative --- text/0000-cargo-libstd-awareness.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 36a6a3574e9..5dd4fa30741 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -207,9 +207,6 @@ This seems good enough. # Alternatives - - Instead of copying binaries from the sysroot, we could just leave it to rustc to find them. - But then a simple `--no-resolve-sysroot` would not work, and the logic for passing `--extern` would need be more complicated. - - Previous versions of this RFC were simpler but more brittle. Please refer to the git history to see them. From a36a46bac16c7aca9d4feb1e407c767a4dcbeadf Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 11:12:00 -0700 Subject: [PATCH 38/66] Clarify drawback on unconstrained interfaces --- text/0000-cargo-libstd-awareness.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 5dd4fa30741..9dcdf19979e 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -203,6 +203,8 @@ This seems good enough. - Compilers could provide crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. (Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.) Since the *interface* of the stdlib is specified, it would be neat if we could put a big crate type/interface on crates.io, which compiler implementations would need to match. + [That is, the interface of the stable crates. + Unstable crates behind the std facade are a compiler-specific implementation detail, and thus it would be counter-productive, even to likewise constrain their interfaces.] # Alternatives From e6aae38718d3f80fdc7770e7d8ad0616e644020f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 11:20:55 -0700 Subject: [PATCH 39/66] Clarify drawback and fix typo --- text/0000-cargo-libstd-awareness.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 9dcdf19979e..ce6a19acfb9 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -186,7 +186,7 @@ That way unioning them together with either priority (the fallback scheme effect The easiest way to achieve this is to make sure that standard library crates use names reserved on crates.io. We don't want to bake crates.io policy into Cargo however, so instead of absolutely prohibiting stdlib deps with non-reserved names, crates.io will just lint packages being uploaded. -Also, care will be taken so that any stdlib crate that is stabalized must use a reserved name or already be published on crates.io. +Also, care will be taken so that any stdlib crate that is stabilized must use a reserved name or already be published on crates.io. That still doesn't protect unpublished packages using unstable stdlib crates without reserved names from breakage, but due to their use of unstable interfaces we have no obligation to keep them working. Also, once we have an option to explicitly provide the source for stdlib deps, they can force the behavior they want. This seems good enough. @@ -196,9 +196,8 @@ This seems good enough. - The mock registry for sysroot binaries is a disgusting hack. - - Only some crates in the rust repo (at least `core`, `alloc` and `collections`) can properly be built just based upon their `Cargo.toml`. - However it's precisely only these "foundational" crates that will be of interest to freestanding developers. - Hosted developers can likely get pre-built binaries for the platform they need with `rustup`, just as they do today. + - Even with this RFC and a nightly compiler, a single `cargo build` is incapable of building the entire standard library due to external dependencies. + But I believe we will eventually reach that goal, and furthermore this RFC will help us reach it. - Compilers could provide crates in their sysroot that don't match the Rust specification, and Cargo would be none the wiser. (Technically, this problem already exists with falling back on the sysroot binaries, but users will probably expect better when they can specify standard library dependencies explicitly.) From 7e7355706734c2474500899395915d88548853b9 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 11:26:12 -0700 Subject: [PATCH 40/66] More eloquence and details about how sysroot binaries will be used. --- text/0000-cargo-libstd-awareness.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index ce6a19acfb9..03f9669c96c 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -194,7 +194,8 @@ This seems good enough. # Drawbacks - - The mock registry for sysroot binaries is a disgusting hack. + - The mock registry for sysroot binaries is a complicated special case whose implementation will probably span many parts of Cargo. + In the near future, it is unlikely to be generalized into something more elegant. - Even with this RFC and a nightly compiler, a single `cargo build` is incapable of building the entire standard library due to external dependencies. But I believe we will eventually reach that goal, and furthermore this RFC will help us reach it. From f7fd3dfb5f597acef1b0777822f5777ad33a4e7a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:11:25 -0700 Subject: [PATCH 41/66] Fix sysroot paths --- text/0000-cargo-libstd-awareness.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 03f9669c96c..899d0512b3f 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -132,11 +132,12 @@ Examples of this are file-system paths, git repositories, and the upcoming regis For the time being, `stdlib = true` will act like another source, instructing the compiler to look at the compiler source registry, and if that is missing, then the sysroot binaries. The remainder of this section will clarify this. -The compiler source registry is expected to be located in `${$CARGO_RUSTC --print sysroot}/src`. +The compiler source registry is expected to be located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib/src`. The exact format this takes will be determined when custom registries are implemented. Based on that, it may be necessary to transform the rust repo before dumping it there. The "sysroot binary mock registry" is generated by examining the contents of the sysroot. +Just as today, binaries are located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib/$target-triple/lib`. Any binary in there will be added to the mock registry, with a version taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). From b327075fd48a3712cf5b8c8da1dd99b40739623a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:15:54 -0700 Subject: [PATCH 42/66] Add missing closing paren --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 899d0512b3f..1516702a242 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -101,7 +101,7 @@ Compilers that don't have sysroot binaries should accept this flag and ignore it Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so rustc won't use the sysroot for linking either. Additionally, compilers besides rustc may have version numbers distinct from the version of Rust they implement. -For this purpose, the verbose version output (`$COMPILER -vV` should contain an additional line: +For this purpose, the verbose version output (`$COMPILER -vV`) should contain an additional line: ``` language-version: $version ``` From d38daadee10fa9483a0c9568b5672a1389585007 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:24:32 -0700 Subject: [PATCH 43/66] Clarify how implicit dependencies are disabled --- text/0000-cargo-libstd-awareness.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 1516702a242..8f386836711 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -84,11 +84,11 @@ It is defined by default as: ```toml implicit-dependencies = ["primary", "build", "dev"] ``` -This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit elements. +This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit dependencies. A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. -Finally, if an (explicit) dependency conflicts with one of the implicit defaults, that category of implicit dependency will be skipped. -For example, if a crate explicit depends on `std` as a build-dependency, neither `std` nor any other implicit build dependency will be injected. +Finally, if an (explicit) dependency conflicts with one of the implicit defaults, implicit dependencies of the same sort will be skipped. +For example, if a crate explicit depends on `std` as a build dependency, neither `std` nor any other implicit build dependency will be injected. This final rule means must packages won't need to use the implicit-dependencies key, because either implicit dependencies will be used, or a `core` or `std` explicit dependency will be present. An exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. From f8859d54ba487c6ecf2529677c62ee7c05ae8548 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:29:36 -0700 Subject: [PATCH 44/66] Grammar for implicit dep disabling --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 8f386836711..b2c7a563506 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -84,7 +84,7 @@ It is defined by default as: ```toml implicit-dependencies = ["primary", "build", "dev"] ``` -This indicates each of `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) is augmented with implicit dependencies. +This indicates each of the `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) are augmented with implicit dependencies. A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. Finally, if an (explicit) dependency conflicts with one of the implicit defaults, implicit dependencies of the same sort will be skipped. From d208a5fab9bb31852e5ab620340b01adb3955c39 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:40:10 -0700 Subject: [PATCH 45/66] `core` is the only crate needing to use the `implicit-dependencies` key --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index b2c7a563506..fbdfb0ffe7e 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -90,7 +90,7 @@ A manual definition may be that or almost any subset, in which case only the inc Finally, if an (explicit) dependency conflicts with one of the implicit defaults, implicit dependencies of the same sort will be skipped. For example, if a crate explicit depends on `std` as a build dependency, neither `std` nor any other implicit build dependency will be injected. This final rule means must packages won't need to use the implicit-dependencies key, because either implicit dependencies will be used, or a `core` or `std` explicit dependency will be present. -An exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. +The one current exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. ## Compiler command-line interface From cb67debe8406aeb18a02010129d4a0d8fab7ddcd Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:41:43 -0700 Subject: [PATCH 46/66] Clarify stdlib deps vs rust language dep --- text/0000-cargo-libstd-awareness.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index fbdfb0ffe7e..3fb2c5e8044 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -39,7 +39,8 @@ Keeping multiple lockfiles in sync is a nuisance, and this gets us one step clos The use-cases so far mainly benefit niche corners of the Rust community, but the last should be useful for just about everyone. Now that multiple versions of Rust have been released, it can be useful to specify the minimum version. If and when Rust 2, a version with breaking changes, comes out, this will be all the more important. -We don't yet have a plan yet to track the language itself, but by tracking standard library dependencies we make it trivial to specify version requirements like any other package. +We don't yet have a plan yet to track which version of Rust is used in the current crate (in order to opt in to the use of a hitherto unstable feature). +However, because the versions of standard library crate are required to be the same as the version of the language supported by the compiler, specifying the semver requirements of a crate in the sysroot effectively specifies the semver requirements of the language itself. # Detailed design From e8b1aa23b5255d9c2955a09851f37c439cd8d3d1 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 6 Aug 2016 12:54:28 -0700 Subject: [PATCH 47/66] For now, "stdlib = true" is a source --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 3fb2c5e8044..61d202fb872 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -130,7 +130,7 @@ This process is completely defined by the rules described in the first subsectio Cargo has a "source" abstraction for providers of packages. Examples of this are file-system paths, git repositories, and the upcoming registries. -For the time being, `stdlib = true` will act like another source, instructing the compiler to look at the compiler source registry, and if that is missing, then the sysroot binaries. +For the time being, `stdlib = true` will be another source, instructing Cargo to look at the compiler source registry, and if that is missing, then the sysroot binaries. The remainder of this section will clarify this. The compiler source registry is expected to be located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib/src`. From 477d8bc145e3bb0a210b47fc93a91ee5610b6778 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 8 Aug 2016 16:50:04 -0700 Subject: [PATCH 48/66] Finally figure out how deps are passed to rustc No more `--no-resolve-sysroot` --- text/0000-cargo-libstd-awareness.md | 41 ++++++++++++++++------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 61d202fb872..649d33ca2c1 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -14,10 +14,17 @@ This will allow building parts of the standard library from source, but in order # Motivation First, some background. -`rustc` can be given crates and their location with `--extern =`. -When finding a library, `rustc` first sees if its location has been specified with `--extern`, then looks in any directories specified with `-L`, and finally looks in the "sysroot" [specifically `/lib/rustlib//lib`]. -Cargo passes in all dependencies it builds with `--extern`. -However Cargo does not know about the standard library, so builds of it are taken from the sysroot. +Rustc needs to load dependency crates in order to work. +Multiple flags exist to instruct rustc on how to find these crates. +`--extern =` tells rustc to find the crate with the given name at the given path. +This has the highest priority, overriding locations specified or inferred via other means. +`-L =` has the second highest priority, telling rustc to look for matching crates in the given directory. +The `=` part is optionally, but one variant that is crucial for Cargo's purposes is `-L dependeny=`. +The `dependeny=` part tells rustc to only look in the directory when resolving transitives deps ("deps of deps"), as opposed to `extern crate`s in the current crate. +As a last resort, rustc will look within the sysroot (specifically `/lib/rustlib//lib`). The sysroot is a hard-coded location relative to rustc, but can also be overridden with `--sysroot=`. + +Cargo passes immediate dependencies to rustc with `--extern =`, and transitive dependencies with `-L dependency=`. +However Cargo does not know about the standard library, so builds of it are taken from the sysroot, and any crate can `extern crate` any rlib that happens to be there. For cross-compiling, one can often download standard library binaries with rustup. This is convenient, but one cannot expect pre-built binaries for all platforms. @@ -93,15 +100,9 @@ For example, if a crate explicit depends on `std` as a build dependency, neither This final rule means must packages won't need to use the implicit-dependencies key, because either implicit dependencies will be used, or a `core` or `std` explicit dependency will be present. The one current exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. -## Compiler command-line interface +## Compiler language version -As will be explained, with these changes Cargo will always use `--extern` to pass dependencies, so rustc's sysroot lookup can and should be bypassed. -This will prevent users using the sysroot from depending on more crates than their cargo file declares (implicitly or explicitly).. -A `--no-resolve-sysroot` flag will be added for this purpose. -Compilers that don't have sysroot binaries should accept this flag and ignore it. -Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so rustc won't use the sysroot for linking either. - -Additionally, compilers besides rustc may have version numbers distinct from the version of Rust they implement. +Compilers besides rustc may have version numbers distinct from the version of Rust they implement. For this purpose, the verbose version output (`$COMPILER -vV`) should contain an additional line: ``` language-version: $version @@ -138,7 +139,7 @@ The exact format this takes will be determined when custom registries are implem Based on that, it may be necessary to transform the rust repo before dumping it there. The "sysroot binary mock registry" is generated by examining the contents of the sysroot. -Just as today, binaries are located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib/$target-triple/lib`. +Just as today, binaries are located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib//lib`. Any binary in there will be added to the mock registry, with a version taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). @@ -149,11 +150,14 @@ When the build plan just involves the compiler source registry and/or existing t The awkward scenario is when packages from the sysroot binary mock registry need to be used in the build plan. Because Cargo doesn't know much about the sysroot binaries, it must be very conservative when deciding whether or not they can be used. For example, Cargo may assume they are built with only default features enabled but it can't know what those are. -If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan being resolved. -Cargo likewise will have to be conservative with other metadata. -Packages in the mock binaries registry are not built by Cargo, since they are prebuilt, and when they serve as (transitive) dependencies, Cargo passes them in with `--extern` and their sysroot location. +If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan under construction. +Cargo likewise will have to be conservative inferring any other package metadata it may use. +Packages in the mock binaries registry are not built by Cargo, since they are prebuilt, and when they serve as immediate dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. -Since `--extern` is used even in the sysroot binary case, cargo can and will pass `--no-resolve-sysroot` to rustc in all cases. +Also whenever they are in any way part of the build plan, Cargo also must pass `-L dependency=/lib/rustlib//lib` so rustc can find transitive deps here. This is needed both because the mock binaries registry crates may in fact be transitive deps of the crates built from source, and also because they *themselves* may also have arbitrary mock binary registry deps. +Because of this use of `--extern` and `-L` with the mock binary registry, rustc when invoked with Cargo should never need fallback looking for binaries in the sysroot. +To prevent it from doing so with broken packages, Cargo will also pass rustc `--sysroot=` (i.e. the empty path) to prevent it from doing so. +[Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so the sysroot won't be needed for linking either.] ## Rustbuild improvements @@ -214,6 +218,7 @@ This seems good enough. - Previous versions of this RFC were simpler but more brittle. Please refer to the git history to see them. + # Unresolved questions - Users of the stable compiler should be able to build the stdlib from source, since it is trusted, but cannot because it uses unstable features. @@ -221,8 +226,6 @@ This seems good enough. - It is unclear what should go in the lockfile when building with sysroot binaries. - - Whether to add the `--no-resolve-sysroot` flag to rustc, as described above. - - Should `cargo new` specify `std`, or any other stdlib crates explicitly by default? I'd hope so! From b9aa2dad7eda73b4e1850306594a2741f9dbf9d5 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 12 Aug 2016 16:45:02 -0700 Subject: [PATCH 49/66] Add alternatives for "sysroot = true" and language version --- text/0000-cargo-libstd-awareness.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 649d33ca2c1..c7a30d4b8bb 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -218,6 +218,11 @@ This seems good enough. - Previous versions of this RFC were simpler but more brittle. Please refer to the git history to see them. + - Should the "stdlib" virtual source instead be called "sysroot" (e.g. `core = { sysroot = true, .. }`)? + This emphasizes how those dependencies are resolved as opposed to what they are for. + + - If a way to specify the language version like #1709 or #1707 is added, the version of stdlib dependencies could be pulled from that. + # Unresolved questions From 26e1b6ed4c90a34d4b6e812b72f88739124e41de Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 12 Aug 2016 18:22:22 -0700 Subject: [PATCH 50/66] Purge language about registries, misc clarifications of surrounding text --- text/0000-cargo-libstd-awareness.md | 77 +++++++++++++++-------------- 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index c7a30d4b8bb..9e8912ec5fd 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -103,59 +103,59 @@ The one current exception is `core` itself, which must of course not depend on ` ## Compiler language version Compilers besides rustc may have version numbers distinct from the version of Rust they implement. -For this purpose, the verbose version output (`$COMPILER -vV`) should contain an additional line: +For this purpose, the verbose version output (`$CARGO_RUSTC -vV`) should contain an additional line: ``` -language-version: $version +language-version: ``` -For now, `$version` should be a version, not a version requirement, and the patch number must be zero as patch numbers don't make sense for interfaces. +For now, `` should be a version, not a version requirement, and the patch number must be zero as patch numbers don't make sense for interfaces. [This may be relaxed in the future for compilers which implement multiple versions.] -## Compiler source packaging +## Compiler Source While the standard library *interface* is defined with each rustc version, the implementation of many crates, by virtue of using unstable features, is compiler-specific. This makes the standard library unfit for crates.io. (Additionally, the issue of dealing with nightly also makes crates.io hard to use, but that is a less clear-cut obstacle.) -Cargo will soon gain the ability to create custom registries, as described in -https://github.com/rust-lang/cargo/pull/2857 . -Compilers should package the source of their implementation of the standard library as a registry, which can be distributed with the compiler. -In practice, rustc will optionally contain the source in its sysroot. -Rustup may be able to put it there if the default download does not contain it already. -This registry will be called the "compiler source registry". +To get around this, Cargo will give compilers the option of distributing the source of their implementation of the standard library in a location Cargo knows of. +Conveniently, Cargo has a "source" abstraction for providers of packages. +Examples of this are file-system paths, git repositories, and the upcoming registries. +To implement this, Cargo will gain knowledge of of a new source, the "compiler source". +The compiler source, if it is present, will be located in the sysroot in `/lib/rustlib/src`. +The exact format this takes will be determined during implementation and added back to this RFC before stabilization, but that of a "local registry" is likely, now that +https://github.com/rust-lang/cargo/pull/2857 has landed. +Compilers should include the source of each crate of their implementation of the standard library in side. + +It is presumed that, Rustup may be able to put it there if the default download does not contain it already. -## Cargo implementation +## Cargo Pipeline Whenever Cargo encounters a `Cargo.toml`, the first thing it always does is inject any applicable implicit deps. The idea is that by doing this so early on, most of Cargo can stay the same in only knowing or caring about explicit deps, simplifying both this RFC and its implementation. This process is completely defined by the rules described in the first subsection, so there is really nothing to elaborate upon here. -Cargo has a "source" abstraction for providers of packages. -Examples of this are file-system paths, git repositories, and the upcoming registries. -For the time being, `stdlib = true` will be another source, instructing Cargo to look at the compiler source registry, and if that is missing, then the sysroot binaries. -The remainder of this section will clarify this. +Just as `git = ...`, and `path = ...` are parsed into a "source id", so `stdlib = true` will into a new "stdlib source id" too. +But instead of mapping to a specific source, this source id will map either to the "compiler source" as described above, or the "sysroot binary mock source", as described below. -The compiler source registry is expected to be located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib/src`. -The exact format this takes will be determined when custom registries are implemented. -Based on that, it may be necessary to transform the rust repo before dumping it there. +The "sysroot binary mock source" is generated by examining the contents of the sysroot. +Just as today, binaries are located in `/lib/rustlib//lib`. +Any binary in there will be added to the mock source, with a version +taken either from the `language-version` key described above, or the compiler version if that key is not present (as it would be with existing rustc releases). -The "sysroot binary mock registry" is generated by examining the contents of the sysroot. -Just as today, binaries are located in `${$CARGO_RUSTC --print sysroot}/lib/rustlib//lib`. -Any binary in there will be added to the mock registry, with a version -taken either from the `language-version` described above, or the compiler version if that key is not present (as it would be with existing rustc releases). +If the compiler source exists, that is used to resolve `stdlib = true` deps, and the sysroot binary mock source need not even be built. +If the compile source is absent, then the binary mock source is used. +Note that this prioritization doesn't depend on the outgoing dependencies trying to be resolved. +Once the source backing stdlib deps is picked, it is the only one used even if the other source also exists and contains the missing package---sticking arbitrarily named rlibs in the sysroot will not effect Cargo when everything is being built from source. -If the compiler source registry exists, that is used, and the sysroot binary mock registry need not even be built. -If the source registry is absent, then the binary registry is used. - -When the build plan just involves the compiler source registry and/or existing types of sources, it can be executed just like today. -The awkward scenario is when packages from the sysroot binary mock registry need to be used in the build plan. +When the build plan just involves the compiler source and/or existing types of sources, it can be executed just like today. +The awkward scenario is when packages from the sysroot binary mock source need to be used in the build plan. Because Cargo doesn't know much about the sysroot binaries, it must be very conservative when deciding whether or not they can be used. For example, Cargo may assume they are built with only default features enabled but it can't know what those are. If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan under construction. Cargo likewise will have to be conservative inferring any other package metadata it may use. -Packages in the mock binaries registry are not built by Cargo, since they are prebuilt, and when they serve as immediate dependencies, Cargo passes them in with `--extern` and their sysroot location. +Packages in the binary mock source are not built by Cargo, since they are prebuilt, and when they serve as immediate dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. -Also whenever they are in any way part of the build plan, Cargo also must pass `-L dependency=/lib/rustlib//lib` so rustc can find transitive deps here. This is needed both because the mock binaries registry crates may in fact be transitive deps of the crates built from source, and also because they *themselves* may also have arbitrary mock binary registry deps. -Because of this use of `--extern` and `-L` with the mock binary registry, rustc when invoked with Cargo should never need fallback looking for binaries in the sysroot. +Also whenever they are in any way part of the build plan, Cargo also must pass `-L dependency=/lib/rustlib//lib` so rustc can find transitive deps here. This is needed both because the binary mock source crates may in fact be transitive deps of the crates built from source, and also because they *themselves* may also have arbitrary binary mock source deps. +Because of this use of `--extern` and `-L` with the binary mock source, rustc when invoked with Cargo should never need fallback looking for binaries in the sysroot. To prevent it from doing so with broken packages, Cargo will also pass rustc `--sysroot=` (i.e. the empty path) to prevent it from doing so. [Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so the sysroot won't be needed for linking either.] @@ -163,7 +163,7 @@ To prevent it from doing so with broken packages, Cargo will also pass rustc `-- As advertised in the motivation section, with this RFC, rustbuild can use a single workspace to build the standard library and all executables. -One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are ever used; one needs to bypass the compiler source registry and sysroot binary mock registry. +One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are ever used; one needs to bypass the compiler source and sysroot binary mock source. To accomplish this, rustbuild's workspace will need to use `[replace]` to redirect all stdlib deps to use the workspace's packages. All binaries for a specific phase can be built with a single `cargo build` (barring special requirements for individual libraries). @@ -174,20 +174,20 @@ After the last compiler is build, an additional mini-stage of building just the The custom registries PR https://github.com/rust-lang/cargo/pull/2857 starts with just mirroring existing registries. As followup work, it expected that packages (probably just the workspace root, definitely not non-packages like cargo config) will be able to specify the "default" source, i.e. the one used when none is specified (today this is always crates.io). -Similarly, one could specify a "stdlib" source, to be used for `stdlib = true` deps instead of the compiler source binary registry or sysroot binary mock registry. +Similarly, one could specify a "stdlib" source, to be used for `stdlib = true` deps instead of the compiler source or sysroot binary mock source. This would simplify rustbuild as it could use that once instead of `[replace]` for each package. This doesn't require any planning from this RFC. More importantly, it would be nice to move stdlib crates that don't use unstable features to crates.io. `collections` and `test` almost don't use any unstable and are thus good candidates for this. -With something like what is described in the first paragraph, it could be possible for individual packages to instruct Cargo to first check crates.io, and then the compiler source registry, for stdlib crates. -But this shifts the burden to individual packages, and means we'd still need to vendor source of any crate moved to crates.io in the compiler source registry for packages that didn't make the switch. +With something like what is described in the first paragraph, it could be possible for individual packages to instruct Cargo to first check crates.io, and then the compiler source, for stdlib crates. +But this shifts the burden to individual packages, and means we'd still need to vendor source of any crate moved to crates.io in the compiler source for packages that didn't make the switch. -More interesting would be to change Cargo's *default* behavior to check both the compiler-specific sources (compiler source registry and sysroot binary mock registry) and crates.io. +More interesting would be to change Cargo's *default* behavior to check both the compiler-specific sources (compiler source and sysroot binary mock source) and crates.io. This would allow standard crates to seamlessly migrate to crates.io without extra work per package. This could be either be done where crates.io overrides the compiler-specific sources, or the compiler specific sources override crates.io. We don't want to commit to either variant in this RFC, however, so we instead want to keep all 3 options open (no fallback, crates.io over compiler-specific, compiler-specific over crates.io). -To achieve this, we want to keep the sysroot/compiler source registry and crates.io disjoint: no package should be contained in both sources. +To achieve this, we want to keep the sysroot/compiler source and crates.io disjoint: no package should be contained in both sources. That way unioning them together with either priority (the fallback scheme effectively crates a union source) has the same effect. The easiest way to achieve this is to make sure that standard library crates use names reserved on crates.io. @@ -200,7 +200,7 @@ This seems good enough. # Drawbacks - - The mock registry for sysroot binaries is a complicated special case whose implementation will probably span many parts of Cargo. + - The sysroot binary mock source is a complicated special case whose implementation will probably span many parts of Cargo. In the near future, it is unlikely to be generalized into something more elegant. - Even with this RFC and a nightly compiler, a single `cargo build` is incapable of building the entire standard library due to external dependencies. @@ -212,6 +212,9 @@ This seems good enough. [That is, the interface of the stable crates. Unstable crates behind the std facade are a compiler-specific implementation detail, and thus it would be counter-productive, even to likewise constrain their interfaces.] + - The name "compiler source" is unfortunate because it sounds like the source of the compiler itself. + But perhaps the best solution is to rename Cargo's "source" abstraction (and the traits that go with it). + # Alternatives From d35237fbcc5ac2b7f5d76f110d6bfa35a7278481 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 00:40:00 -0700 Subject: [PATCH 51/66] Split huge paragraph --- text/0000-cargo-libstd-awareness.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 9e8912ec5fd..99302b03333 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -152,9 +152,11 @@ Because Cargo doesn't know much about the sysroot binaries, it must be very cons For example, Cargo may assume they are built with only default features enabled but it can't know what those are. If features are explicitly requested, or the default features are disabled (by all dependent packages) then the binaries are ineligible for the build plan under construction. Cargo likewise will have to be conservative inferring any other package metadata it may use. + Packages in the binary mock source are not built by Cargo, since they are prebuilt, and when they serve as immediate dependencies, Cargo passes them in with `--extern` and their sysroot location. This is different from other deps, whose binaries are placed in Cargo's output directory, and sysroot deps today, where `--extern` isn't used as all. Also whenever they are in any way part of the build plan, Cargo also must pass `-L dependency=/lib/rustlib//lib` so rustc can find transitive deps here. This is needed both because the binary mock source crates may in fact be transitive deps of the crates built from source, and also because they *themselves* may also have arbitrary binary mock source deps. + Because of this use of `--extern` and `-L` with the binary mock source, rustc when invoked with Cargo should never need fallback looking for binaries in the sysroot. To prevent it from doing so with broken packages, Cargo will also pass rustc `--sysroot=` (i.e. the empty path) to prevent it from doing so. [Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so the sysroot won't be needed for linking either.] From 28353b2fb1a41f67f99a73481396a85d4e7cc017 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 02:02:52 -0700 Subject: [PATCH 52/66] Clamp down on dependencies; mention unstability --- text/0000-cargo-libstd-awareness.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 99302b03333..b1395b921e9 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -60,6 +60,8 @@ The version for stdlib crates comes from the version of Rust their interfaces ar A version requirement must be specified. The full breadth of options available with our existing dependencies, e.g. features and overrides, will be supported. +For the initial roll out of the feature, only normal dependencies, not build or dev dependencies, will be allowed to include explicit stdlib dependencies. + ## Implicit dependencies For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. @@ -87,18 +89,16 @@ While importing it explicitly remains unstable, it's currently injected and thus Other dependencies of `std` besides core we don't need to worry about, because they are only transitive dependencies through `std`, not direct dependencies. Now, not all crates depend on these crates, so there must be a way to opt out. -For this, we introduce a new `implicit-dependencies` key. -It is defined by default as: -```toml -implicit-dependencies = ["primary", "build", "dev"] -``` -This indicates each of the `dependencies`, `build-dependencies`, and `dev-dependencies` maps (respectively) are augmented with implicit dependencies. -A manual definition may be that or almost any subset, in which case only the included dependency maps are augmented. +The primary way is just to create a conflict. +If an (explicit) dependency has the same name as one of the implicit defaults, implicit dependencies of the same sort will be skipped. +For example, if a crate explicit depends on `std` as a regular dependency, neither `std` nor any other implicit regular dependency will be injected. +Since currently regular dependencies can be included stdlib dependencies, only regular dependencies can be opted out of. +This means we are free to change the implicit dev and build dependencies without breaking anything. -Finally, if an (explicit) dependency conflicts with one of the implicit defaults, implicit dependencies of the same sort will be skipped. -For example, if a crate explicit depends on `std` as a build dependency, neither `std` nor any other implicit build dependency will be injected. -This final rule means must packages won't need to use the implicit-dependencies key, because either implicit dependencies will be used, or a `core` or `std` explicit dependency will be present. -The one current exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly. +Opting out via conflict as described above is adequate for almost all cases. +The one exception is `core` itself, which must of course not depend on `core` or `std` implicitly or explicitly---or anything else for that matter. +For it, a key, `implicit-dependencies = `, will be introduced. +Because it doesn't generalize if we make the implicit build or dev stdlib deps optional, this key will be permanently unstable. ## Compiler language version @@ -228,6 +228,8 @@ This seems good enough. - If a way to specify the language version like #1709 or #1707 is added, the version of stdlib dependencies could be pulled from that. + - This has been deemed sufficiently complicated to warrant the introduction of unstable features to Cargo. + # Unresolved questions From 44f1e847e3363ba4925e7163d96124840e45487f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 10:15:12 -0700 Subject: [PATCH 53/66] Talk about `version = "*"` --- text/0000-cargo-libstd-awareness.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index b1395b921e9..1780a90d705 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -60,6 +60,10 @@ The version for stdlib crates comes from the version of Rust their interfaces ar A version requirement must be specified. The full breadth of options available with our existing dependencies, e.g. features and overrides, will be supported. +Just like regular dependencies, `version = "*"` is supported meaning no requirement at all, but is prohibited on `crates.io`. +For backwards compatibility, Cargo allows skipping the version field altogether for crates.io deps, in which case `version = "*"` is inferred. +This, and other such accommodations can be dropped fro stdlib dependencies since they are new. + For the initial roll out of the feature, only normal dependencies, not build or dev dependencies, will be allowed to include explicit stdlib dependencies. ## Implicit dependencies From 2ea3e83b281a404c81539c4b5b0d8b423e677a89 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 10:32:08 -0700 Subject: [PATCH 54/66] the -> a --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 1780a90d705..9a5ad1468fa 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -148,7 +148,7 @@ taken either from the `language-version` key described above, or the compiler ve If the compiler source exists, that is used to resolve `stdlib = true` deps, and the sysroot binary mock source need not even be built. If the compile source is absent, then the binary mock source is used. Note that this prioritization doesn't depend on the outgoing dependencies trying to be resolved. -Once the source backing stdlib deps is picked, it is the only one used even if the other source also exists and contains the missing package---sticking arbitrarily named rlibs in the sysroot will not effect Cargo when everything is being built from source. +Once the source backing stdlib deps is picked, it is the only one used even if the other source also exists and contains a missing package---sticking arbitrarily named rlibs in the sysroot will not effect Cargo when everything is being built from source. When the build plan just involves the compiler source and/or existing types of sources, it can be executed just like today. The awkward scenario is when packages from the sysroot binary mock source need to be used in the build plan. From 324e2257d80e8407349819b40cf4d399fc1dc203 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 12:56:14 -0700 Subject: [PATCH 55/66] Add @brson's text on sysroot binary filenames --- text/0000-cargo-libstd-awareness.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 9a5ad1468fa..e92d3ad0fa7 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -144,6 +144,10 @@ The "sysroot binary mock source" is generated by examining the contents of the s Just as today, binaries are located in `/lib/rustlib//lib`. Any binary in there will be added to the mock source, with a version taken either from the `language-version` key described above, or the compiler version if that key is not present (as it would be with existing rustc releases). +Today rustc requires crates to be named something like `lib{name}{anything}.{extension}` where name contains no dashes. +rustc's binaries follow the format `lib{name}-{hash}.{extension}`. +So to derive a crate name from the file name we can use a regex vaguely like `lib(\pXID_Startp\XID_Continue*)(-|\.).*` - the characters after "lib" but preceding a dash or dot. +This places a new restriction on file names of crates located in the sysroot. If the compiler source exists, that is used to resolve `stdlib = true` deps, and the sysroot binary mock source need not even be built. If the compile source is absent, then the binary mock source is used. From 02889b8f2be27909e562d172dda2b4480e2bb726 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 13:26:00 -0700 Subject: [PATCH 56/66] Remove obsolete unresolved question --- text/0000-cargo-libstd-awareness.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index e92d3ad0fa7..4ccbd89df17 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -249,11 +249,6 @@ This seems good enough. - Should `cargo new` specify `std`, or any other stdlib crates explicitly by default? I'd hope so! - - Should one be able to opt-out of implicit build and development dependencies? - Currently, it makes sense to always make `std` available for `build.rs` since it must exist for the compiler. - But if platform-specific parts of the `std` are exposed only with features or "scenarios" (a newly-proposed mechanism specifically for handling environment differences), then we lose an opportunity to be able to express mandatory cross-compiling. - Finally, in the far future it may be possible to build rustc on platforms where all of `std` isn't available, invalidating the reasoning that `std` is never unavailable as a build dependency. - - It is somewhat unclear how Cargo should deal with architecture-specific configuration that is not captured in the target spec nor Cargo feature flags (like CPU features). [RFC #1645](https://github.com/rust-lang/rfcs/pull/1645) proposes just adding some such configuration to the target triple, whereas https://internals.rust-lang.org/t/pre-rfc-a-vision-for-platform-architecture-configuration-specific-apis/3502/26 proposes a new "scenarios" interfaces. When building from source, this question is orthogonal to this RFC because it just reuses Cargo's existing methods of keeping binaries for different configurations separate. From bbc503b5cbca28b6e8fac08423a6c24b3c629e02 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 15:10:01 -0700 Subject: [PATCH 57/66] Talk about lockfiles. --- text/0000-cargo-libstd-awareness.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 4ccbd89df17..60144e637c0 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -169,6 +169,14 @@ Because of this use of `--extern` and `-L` with the binary mock source, rustc wh To prevent it from doing so with broken packages, Cargo will also pass rustc `--sysroot=` (i.e. the empty path) to prevent it from doing so. [Once [Rust PR #35021](https://github.com/rust-lang/rust/pull/35021/files) lands in some form, `compiler-rt` will be a Cargoized dependency so the sysroot won't be needed for linking either.] +By default, stdlib packages will be excluded from the lockfile. +Putting stdlib version numbers in lockfiles would mean you have to have an exact compiler version to build code with lockfiles, which is inconvenient. +Not having stdlib crates in lockfiles doesn't pose much +risk of failed compiles since the backwards compatibility of stdlib crates is already a high priority, and linchpin upon which the whole Rust ecosystem already depends. + +This is just a default however. +The Rust repo, for example, *should* track stdlib crates in its lockfile because that's a large part of what it builds, and it does indeed need to be booted from a precise version of rust. + ## Rustbuild improvements As advertised in the motivation section, with this RFC, rustbuild can use a single workspace to build the standard library and all executables. From b1e1b4b289000caa466d7dd0b1a3242c206d5c48 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 13 Aug 2016 15:34:24 -0700 Subject: [PATCH 58/66] Remove rustbuild ideas, and talk about implementation roadmap instead --- text/0000-cargo-libstd-awareness.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 60144e637c0..4a34edca4cd 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -177,17 +177,6 @@ risk of failed compiles since the backwards compatibility of stdlib crates is al This is just a default however. The Rust repo, for example, *should* track stdlib crates in its lockfile because that's a large part of what it builds, and it does indeed need to be booted from a precise version of rust. -## Rustbuild improvements - -As advertised in the motivation section, with this RFC, rustbuild can use a single workspace to build the standard library and all executables. - -One complication with the RFC is that that no sysroot binaries or source associated with the bootstrap compiler (or previous stage) are ever used; one needs to bypass the compiler source and sysroot binary mock source. -To accomplish this, rustbuild's workspace will need to use `[replace]` to redirect all stdlib deps to use the workspace's packages. - -All binaries for a specific phase can be built with a single `cargo build` (barring special requirements for individual libraries). -Rather than have a multitude of build artifact directories per stage, only one is needed. -After the last compiler is build, an additional mini-stage of building just the standard library could be performed, but distributions wishing to build all deps from source in a standardized fashion (e.g. probably NixOS) would forgo this. - ## Forward Compatibility The custom registries PR https://github.com/rust-lang/cargo/pull/2857 starts with just mirroring existing registries. @@ -215,6 +204,20 @@ That still doesn't protect unpublished packages using unstable stdlib crates wit Also, once we have an option to explicitly provide the source for stdlib deps, they can force the behavior they want. This seems good enough. +## Unstability and implementation road map + +This feature, and the way it affects the ecosystem, has been deemed significant enough to warrant the addition of unstable features to Crates.io. +This hasn't been designed yet, but at a minimum the version with Cargo released with stable Rust will prohibit the use of unstable features. + +The first implementation step is to get stdlib deps build with the compiler source working. +In lieu of the sysroot binary mock source, an unstable flag will simply prune stdlib deps giving us the staus quo. +This will be enough to enable porting rustbuild to the new system and experimentation by the nightly ecosystem in parallel with the rest of the PR. + +Once the remaining aspects of this are implemente---the binary mock source and lockfile filtering of stdlib deps, and rustbuild uses the new system---this feature is eligible to be stabilized. +The temporary stdlib-pruning key will be deprecated but kept around for two rust releases, however. +This will allow official Rust crates that only use `core` to express this with explicit stdlib deps, without breaking the version of Cargo the old Rust releases they must support was distributed with. +After that, the pruning key can be removed because the mock sysroot registry will offer a wholly safer way to use pre-built binaries. + # Drawbacks From 50a0862825ee31f2bd9233e12854c9e2ab8e401f Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 19 Aug 2016 11:50:30 -0700 Subject: [PATCH 59/66] Slightly update the roadmap --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 4a34edca4cd..26446187c9a 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -210,7 +210,7 @@ This feature, and the way it affects the ecosystem, has been deemed significant This hasn't been designed yet, but at a minimum the version with Cargo released with stable Rust will prohibit the use of unstable features. The first implementation step is to get stdlib deps build with the compiler source working. -In lieu of the sysroot binary mock source, an unstable flag will simply prune stdlib deps giving us the staus quo. +In lieu of the sysroot binary mock source, stdlib deps (implicit or explicit) will be pruned by default---yielding the status quo---with an unstable flag to enable them. This will be enough to enable porting rustbuild to the new system and experimentation by the nightly ecosystem in parallel with the rest of the PR. Once the remaining aspects of this are implemente---the binary mock source and lockfile filtering of stdlib deps, and rustbuild uses the new system---this feature is eligible to be stabilized. From e8f14b7118ace7d63597376e7ce85d2db1af1725 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 23 Aug 2016 09:37:10 -0700 Subject: [PATCH 60/66] Typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 26446187c9a..713bf228d9f 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -19,7 +19,7 @@ Multiple flags exist to instruct rustc on how to find these crates. `--extern =` tells rustc to find the crate with the given name at the given path. This has the highest priority, overriding locations specified or inferred via other means. `-L =` has the second highest priority, telling rustc to look for matching crates in the given directory. -The `=` part is optionally, but one variant that is crucial for Cargo's purposes is `-L dependeny=`. +The `=` part is optionally, but one variant that is crucial for Cargo's purposes is `-L dependency=`. The `dependeny=` part tells rustc to only look in the directory when resolving transitives deps ("deps of deps"), as opposed to `extern crate`s in the current crate. As a last resort, rustc will look within the sysroot (specifically `/lib/rustlib//lib`). The sysroot is a hard-coded location relative to rustc, but can also be overridden with `--sysroot=`. From c501fd9f29ab3258b5e10ef6288591c3fc635552 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 29 Aug 2016 18:13:24 -0700 Subject: [PATCH 61/66] Prevent stdlib deps as replacements --- text/0000-cargo-libstd-awareness.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 713bf228d9f..690d0a56246 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -65,6 +65,7 @@ For backwards compatibility, Cargo allows skipping the version field altogether This, and other such accommodations can be dropped fro stdlib dependencies since they are new. For the initial roll out of the feature, only normal dependencies, not build or dev dependencies, will be allowed to include explicit stdlib dependencies. +It will also be illegal to replace a crate with a stdlib dependency as that would allow the prior rule to be circumvented. ## Implicit dependencies From 2719c44c37675bb2f6f1d9829d97a069d69016e3 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 30 Aug 2016 10:53:25 -0700 Subject: [PATCH 62/66] No `stdlib = false` --- text/0000-cargo-libstd-awareness.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 690d0a56246..491c2f7cafd 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -67,6 +67,10 @@ This, and other such accommodations can be dropped fro stdlib dependencies since For the initial roll out of the feature, only normal dependencies, not build or dev dependencies, will be allowed to include explicit stdlib dependencies. It will also be illegal to replace a crate with a stdlib dependency as that would allow the prior rule to be circumvented. +Finally, an explicit `stdlib = false` will, at least initially, also be prohibited. +Firstly, a valid reason for the user to want a create that must *not* the standard library has not been demonstrated. +Secondly, exactly which crates constitute the standard library is not completely decided---perhaps someday official rust-lang crates currently on crates.io will be deemed part of the standard library. + ## Implicit dependencies For backwards compatibility, Cargo must inject such standard library dependencies for existing packages. From 54aa0019eea1e8f9ed1f1de94a14855c85b6b481 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 30 Aug 2016 10:56:02 -0700 Subject: [PATCH 63/66] Include the name of the temporary stdlib deps pruning option --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 491c2f7cafd..94ee85e3893 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -215,7 +215,7 @@ This feature, and the way it affects the ecosystem, has been deemed significant This hasn't been designed yet, but at a minimum the version with Cargo released with stable Rust will prohibit the use of unstable features. The first implementation step is to get stdlib deps build with the compiler source working. -In lieu of the sysroot binary mock source, stdlib deps (implicit or explicit) will be pruned by default---yielding the status quo---with an unstable flag to enable them. +In lieu of the sysroot binary mock source, stdlib deps (implicit or explicit) will be pruned by default---yielding the status quo---with an unstable `keep-stdlib-dependencies` configuration option to enable them. This will be enough to enable porting rustbuild to the new system and experimentation by the nightly ecosystem in parallel with the rest of the PR. Once the remaining aspects of this are implemente---the binary mock source and lockfile filtering of stdlib deps, and rustbuild uses the new system---this feature is eligible to be stabilized. From 27746685bb576c9625ea1711c719ad77ac652138 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 30 Aug 2016 23:59:52 -0700 Subject: [PATCH 64/66] Talk about `custom-implicit-stdlib-dependencies` and hack flags in general --- text/0000-cargo-libstd-awareness.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 94ee85e3893..1451fa15b9a 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -215,13 +215,20 @@ This feature, and the way it affects the ecosystem, has been deemed significant This hasn't been designed yet, but at a minimum the version with Cargo released with stable Rust will prohibit the use of unstable features. The first implementation step is to get stdlib deps build with the compiler source working. -In lieu of the sysroot binary mock source, stdlib deps (implicit or explicit) will be pruned by default---yielding the status quo---with an unstable `keep-stdlib-dependencies` configuration option to enable them. +In lieu of the sysroot binary mock source, stdlib deps (implicit or explicit) will be pruned by default---yielding the status quo---with an unstable `keep-stdlib-dependencies` configuration key to enable them. This will be enough to enable porting rustbuild to the new system and experimentation by the nightly ecosystem in parallel with the rest of the PR. -Once the remaining aspects of this are implemente---the binary mock source and lockfile filtering of stdlib deps, and rustbuild uses the new system---this feature is eligible to be stabilized. -The temporary stdlib-pruning key will be deprecated but kept around for two rust releases, however. -This will allow official Rust crates that only use `core` to express this with explicit stdlib deps, without breaking the version of Cargo the old Rust releases they must support was distributed with. -After that, the pruning key can be removed because the mock sysroot registry will offer a wholly safer way to use pre-built binaries. +Additionally, an unstable `custom-implicit-stdlib-dependencies` key will allow changing which crates are added as implicit defaults. +The tentative plan is to start disallowing unstable features on crates.io, yet that would also prevent the nightly `#![no_std]` ecosystem from broadly trying out this feature. +This workaround will allow projects to operate as if their dependencies used explicit stdlib deps to avoid depending on `std`. +It is unorthodox for `.cargo/config` to affect the build plan this much, but as a short-term workaround this is acceptable. + +It may be possible to stabilize explicit stdlib deps before the mock syroot binary is ready for wide use---the explicit stdlib deps addition to the `Cargo.toml` *language* is far simpler than the the sysroot mock binary registry *implementation*. +In that case, the `keep-stdlib-dependencies` will continue to keep the builds of users working as is, but since the prohibition of explicit stdlib deps on crates.io will disappear, `custom-implicit-stdlib-dependencies` can be removed. + +Once the remaining aspects of this are implemente---the binary mock source and lockfile filtering of stdlib deps, and rustbuild uses the new system---the `keep-stdlib-dependencies` key can also be removed: the pruning key can be removed because the mock sysroot registry will offer a wholly safer way to use pre-built binaries. +If, at that point, there is any official policy on how many prior Rust versions official crates need to work with, the removal can be delayed. +This will allow official crates to use explicit deps without breaking the old versions of Cargo those supported old Rust releases were distributed with. # Drawbacks From d85091aa53540c97265b41acf011e5f160718605 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 10 Oct 2016 11:16:18 -0700 Subject: [PATCH 65/66] Fix typo --- text/0000-cargo-libstd-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index 1451fa15b9a..f067343d210 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -68,7 +68,7 @@ For the initial roll out of the feature, only normal dependencies, not build or It will also be illegal to replace a crate with a stdlib dependency as that would allow the prior rule to be circumvented. Finally, an explicit `stdlib = false` will, at least initially, also be prohibited. -Firstly, a valid reason for the user to want a create that must *not* the standard library has not been demonstrated. +Firstly, a valid reason for the user to depend on a crate that must *not* be part of the standard library has not been demonstrated. Secondly, exactly which crates constitute the standard library is not completely decided---perhaps someday official rust-lang crates currently on crates.io will be deemed part of the standard library. ## Implicit dependencies From ee4104f130a2876ed916bcbd633ae4fd5ef81722 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 7 Jan 2017 18:55:33 -0500 Subject: [PATCH 66/66] Fix typos found by @est31 and @JinShil. Thanks! --- text/0000-cargo-libstd-awareness.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-cargo-libstd-awareness.md b/text/0000-cargo-libstd-awareness.md index f067343d210..d1a91b4f458 100644 --- a/text/0000-cargo-libstd-awareness.md +++ b/text/0000-cargo-libstd-awareness.md @@ -19,7 +19,7 @@ Multiple flags exist to instruct rustc on how to find these crates. `--extern =` tells rustc to find the crate with the given name at the given path. This has the highest priority, overriding locations specified or inferred via other means. `-L =` has the second highest priority, telling rustc to look for matching crates in the given directory. -The `=` part is optionally, but one variant that is crucial for Cargo's purposes is `-L dependency=`. +The `=` part is optional, but one variant that is crucial for Cargo's purposes is `-L dependency=`. The `dependeny=` part tells rustc to only look in the directory when resolving transitives deps ("deps of deps"), as opposed to `extern crate`s in the current crate. As a last resort, rustc will look within the sysroot (specifically `/lib/rustlib//lib`). The sysroot is a hard-coded location relative to rustc, but can also be overridden with `--sysroot=`. @@ -132,9 +132,9 @@ To implement this, Cargo will gain knowledge of of a new source, the "compiler s The compiler source, if it is present, will be located in the sysroot in `/lib/rustlib/src`. The exact format this takes will be determined during implementation and added back to this RFC before stabilization, but that of a "local registry" is likely, now that https://github.com/rust-lang/cargo/pull/2857 has landed. -Compilers should include the source of each crate of their implementation of the standard library in side. +Compilers should include the source of each crate of their implementation of the standard library inside their compiler source. -It is presumed that, Rustup may be able to put it there if the default download does not contain it already. +It is presumed that Rustup may be able to put it there if the default download does not contain it already. ## Cargo Pipeline @@ -248,6 +248,8 @@ This will allow official crates to use explicit deps without breaking the old ve - The name "compiler source" is unfortunate because it sounds like the source of the compiler itself. But perhaps the best solution is to rename Cargo's "source" abstraction (and the traits that go with it). + - This has been deemed sufficiently complicated to warrant the introduction of unstable features to Cargo. + # Alternatives @@ -259,8 +261,6 @@ This will allow official crates to use explicit deps without breaking the old ve - If a way to specify the language version like #1709 or #1707 is added, the version of stdlib dependencies could be pulled from that. - - This has been deemed sufficiently complicated to warrant the introduction of unstable features to Cargo. - # Unresolved questions