Update Rust #2761
Update Rust #2761
Conversation
This is to remove some of the potential undefined behaviors (UB). It will also remove some of the syntactic noise in #2761. Rust has alignment restrictions for types and fields beyond the hardware limitations. This means even on Wasm (which has no alignment restrictions) we have some restrictions to deal with. Recently the compiler started to check for some of the obvious sources of UB (rust-lang/rust#82523). One of these is when taking a pointer or reference to a `packed` struct. Since `packed` means no padding between fields, the fields may be unaligned, in which case getting a reference to them would be UB. To fix this, this PR removes `packed` attributes and refactors the `Bits64` type to make sure that the layout is as before. It turns out for types other than `Bits64` we don't need `packed`: all fields are word sizes so the compiler does not add any padding. For `Bits64`, we had a `u64` field which is aligned on 64-bit boundary. To avoid this we now split 64-bit payload into two 32-bit fields. A new module `static_checks` added to make sure struct sizes are as expected. Assertions in this module are checked in compile time. No extra work needed to align objects. All objects need 4 bytes alignment (checked with `core::mem::align_of`). The compiler already [aligns static objects][2], and in runtime we only allocate whole words. So both static and dynamic objects are always aligned. Debug mode assertions added in GC to check object alignment. [1]: https://doc.rust-lang.org/stable/reference/ [2]: https://github.com/dfinity/motoko/blob/59ddaa4520793b6e7038b60e3c30ff267d8415f6/src/codegen/compile.ml#L453
osa1
force-pushed
the
osa1/update_rustc
branch
from
b86b821
to
5aa4d24
Compare
|
We need to add |
|
If I use |
I wonder if this is a |
|
I'm convinced that this is a |
Can you add a link here? |
We should merge this soon and start checking object field offsets in the static assertions I've added recently. Here's an interesting bug caused by the compiler reordering fields. In this definition of object headers: type Tag = u32;
struct Obj {
tag: Tag,
}
struct Blob {
header: Obj,
len: Bytes<u32>,
}This function: unsafe extern "C" fn text_size(s: Value) -> Bytes<u32> {
(s.get_ptr() as *mut Blob).len()
}is compiled to The important part is 5: 1 to unskew the pointer, and 4 to skip the header and get the length field. This definition is fine. Now, if I refactor object header ( type Tag = u8;
type GcMetadata = u8;
pub struct Obj {
pub tag: Tag,
pub gc_metadata: GcMetadata,
// Add padding to make it 1 word
padding: u16,
}Size of this struct is the same, but that Adding godbolt: https://godbolt.org/z/Tj33Tf6c1 |
See the node added with this commit and #2761 (comment) for the problem this solves.
|
A cargo dev confirmed that the problem with |
See the node added with this commit and #2761 (comment) for the problem this solves.
|
I wonder if we could manually download the packages that URL for downloading package tarballs: |
Vendoring dependencies is causing a lot of problems in #2761 and other PRs, because `cargo vendor` currently has a bug and cannot vendor standard dependencies (deps of alloc and std, see rust-lang/wg-cargo-std-aware#23). For a long time I thought vendoring is a necessity and tried to work around this issue by vendoring dependencies manually, or using https://github.com/nix-community/naersk. However, it turns out vendoring is not necessary, and we can download dependencies in build step just fine. I also don't see any advantages of vendoring the dependencies. So in this PR we remove vendoring. Unblocks #2761.
I implemented the workaround myself, removed copied code. |
There was a problem hiding this comment.
Great work, thanks for your perseverence!
Didn't look closely at the new rust code, but have some comments about the nix code.
In nix/drun.nix we have instructions on how to use nix-update to automate updating the sha256s. Not in this PR, but maybe I’ll figure out how to do that with all the hashes we have everywhere, and maybe even write a CI check that updates them.
The new code in ./rts/cargo-vendor-tools was written by you, not copied from somewhere else? Would it be worth turning that into a separate independent projects, or better not?
Yeah it's my code. I started implementing it as a reusable package (https://github.com/osa1/cargo-vendor-tools) but then decided to include it in RTS code to give control to Motoko devs. We could still make it a package, but I think it would need to live in dfinity organization and be maintained by the Motoko team. |
|
It seems like there's a intermittent failure in some of the tests. Ubuntu job failed with this after the last commit: The error message is not helpful because it doesn't show the relevant parts. I tried running it a few times locally but it worked fine. I now restarted the job, but it would be good to reproduce the issue and see what's going wrong. |
with #2761 adding even more fixed output derivations, this may be useful. If this works, maybe I’ll extract the `run` script here to a script in the repo that works offline just as well.
Still waiting for Mic92/nix-build-uncached#40 to fix the log display issue :-( If push comes to shove, we may have to revert #2747. At least for running the tests. |
…2871) Usually, when we upgrade some dependencies, we have to manually update the hashes. This involves * changing the hash to something else (e.g. changing one character) * running the build locally * observing the error message with the actual hash * pasting that * commit and push. The `nix-update` tool can automate that. This PR makes Github do that on all pushes (even to feature branches), and if needed, push a fix to that branch. With #2761 adding even more fixed output derivations, this may be more relevant.
This PR bumps rustc version and removes xargo. std dependencies are now built
with cargo's new
-Zbuild-stdparameter.To work around rust-lang/wg-cargo-std-aware#23, we implement a tool
"vendor-rust-std-deps" that reads
Cargo.lockof a Rust toolchain std andmanually vendors all the dependencies.
These dependencies are then merged with the Rust RTS dependencies before the
building the RTS in nix.
RTS README updated with instructions to bump rustc.
New rustc will enable more const functions, new API functions, stabilizations,
and features for other RTS PRs.