Is there a migration API?

[marvin-hansen]

Hi, I have a hard time making the migration macro work. For once it's non-hermetic so I had to patch around this to make my Bazel build work. The core of the crux are file path resolution and environment variables. Then, unsurprisingly, it's not portable especially when CI worker generate isolated sandboxes dynamically. I'm still working around this limitation to make the diesel migration macro work on my BuildBuddy CI.

However, the auto migration feature is amazing in itself.

The way I understand the macro is that it basically stuffs the up & down SQL into a string that gets compiled into the binary.

From a conceptual perspective, there is no difference parsing a SQL file vs loading the same SQL from a Rust string constant as you end up with the same DDL. However, in terms of running a hermetic build, the Rust constant means zero file dependencies and that is excellent as it eliminates an entire category of errors.

For Diesel, is there a way to bypass the macro altogether and bake the auto migration programmatically?

[weiznich]

The migration API is provided by the diesel_migrations crate. Notability that does not only provide the embed_migrations! macro but also a run time based solution. It's also possible to just provide your own migration source by implementing the relevant trait. 0See the existing implementations in that crate for examples.

Then, unsurprisingly, it's not portable especially when CI worker generate isolated sandboxes dynamically.

I don't think it's possible to have an isolated macro that reads from the file system. You need to choose there: Either you use the embed_migration! macro to embed the migrations into your binary. That allows you to just ship a single binary that contains everything. Also the build "automatically" collects all the migrations. Or you choose to use the runtime based solution, which doesn't do things at compile time. That means there are no fancy path resolution, etc, but you need to ship all migration files with your binary. Or you implement a custom migration source that hard codes the relevant migrations via include_str! or similar built-in intrinsic, which in turn requires that you manually add new migrations there. (The embed_migrations! macro needs additional code for picking up new migrations behind the compilers back).

[marvin-hansen]

Thank you @weiznich

I implemented a custom migration source that hard codes the relevant migrations via include_str! and that does work
to the degree that it builds with Bazel and on my CI. That was relatively simple to do and also builds with Cargo.

https://github.com/marvin-hansen/bazel-diesel-postgres/blob/main/src/embed_migrations.rs

I don't think it's possible to have an isolated macro that reads from the file system.

This one got me thinking, and, as it turned out, in Bazel you can actually do that. It is possible because during the pre-build stage, copying files into the sandbox and setting environment variables are statically verified, similar to what Rust macros do, so that any missing file or env variable triggers a pre-build failure.

However, while exploring this avenue, I bumped into the most weirdo bug I have seen this year: The rules_rust for Bazel set CARGO_MANIFEST_DIR by using PWD, but there is a rare corner case when a crate is not in the root folder so that, somehow, CARGO_MANIFEST_DIR and PWD (and with it Bazel) point to two different folders (!). The but does not trigger for the sample repo I made, but it affects my internal mono-repo. This issue can only be mitigated by dynamically overwriting CARGO_MANIFEST_DIR in Bazel, something I still try to figure out how to do.

Long story short, yes, a custom implementation that packs the migration works for both, Cargo and Bazel;
The macro, however, currently doesn't work in Bazel due to the most weirdo bug of the year for which I have not found a solution yet.

As far as Diesel is concerned, this is solved.