Speed up builds for COOP users with a CI builder and nix cache provider

[ross-spencer]

I'm working through the Tutorial for Orcfax, running nix develop #dev-tutorial has taken a while, do we have an anticipated build time for this section? It may be worth adding an approximation to the README as there's little feedback, though if it's not expected to be too high maybe not, it could be something my end.

It has been running here, 45 mins and counting.

System:

Intel® Core™ i7-8550U CPU @ 1.80GHz × 8
32GIB Memory.

NB. Providing they're configured correctly, this is with the IOHK pre-built binary config.

[bladyjoker]

Hey @ross-spencer sorry for the late reply, no idea why I didn't get an email about this :/ That's a good idea to put an estimate for the build time!

[bladyjoker]

@ross-spencer the full CI run was ~30minutes. It's hard to tell exactly how much the nix develop github:mlabs-haskell/cardano-open-oracle-protocol#coop-env or if you're doing this withing the repo nix develop .#coop-env would take as there's couple of factors involved:

1. Your local Nix store,
2. Public Nix caches.

Our CI and my machine is configure to use a private MLabs Nix cache, so some things got reused from there. I do believe the build times should still be comparable so the first build should take 30 minutes given the availability of the public Nix caches.

Please give it a go and feel free to add the information about the build time to the README.

[ross-spencer]

Thanks @bladyjoker.

I spent a week away from working directly on this to get a bit more background knowledge in a few more Cardano components for this work.

I've started to look at this process again tonight. I've a few small things I can add to the tutorial for as far as I got. I'm looking forward to get back to it tomorrow.

I think I got unlucky tonight, the build took ~4.5 hours which is the first time on this new server which isn't too great. (I managed to get some reading done in the meantime)

I'll include the full trace, the important part seems to be:

error: file 'nar/0lqykbcdk857qw7c3pj9sw8gvbxqm42w5h1lwwnmj4as63q5i5f1.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/0ka6i4g2h7im9fd5yp7lmfz8rhl78schx5a9d5l6mw724yizhf58.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/1wsilqgjrfllrl93dvgnv1lmsn83azm9g5i9g8kb9ibcckyi3a7b.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/0j4z7pq2z31k71cwjx115pcslqr74mhvlwihv247qbj5scqjyg2b.nar.xz' does not exist in binary cache 'https://cache.iog.io'

In-full:

$ nix develop .#coop-env
path '/home/orcfax/git/mlabs/cardano-open-oracle-protocol/coop-tutorial' does not contain a 'flake.nix', searching up
trace: To make project.plan-nix for coop-pab a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for coop-pab entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: No index state specified for coop-plutus, using the latest index state that we know about (2022-06-01T00:00:00Z)!
trace: To make project.plan-nix for coop-publisher a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for coop-publisher entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: To make project.plan-nix for coop-extras-json-fact-statement-store a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for coop-extras-json-fact-statement-store entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: To make project.plan-nix for plutip a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for plutip entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
error: file 'nar/0lqykbcdk857qw7c3pj9sw8gvbxqm42w5h1lwwnmj4as63q5i5f1.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/0ka6i4g2h7im9fd5yp7lmfz8rhl78schx5a9d5l6mw724yizhf58.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/1wsilqgjrfllrl93dvgnv1lmsn83azm9g5i9g8kb9ibcckyi3a7b.nar.xz' does not exist in binary cache 'https://cache.iog.io'
error: file 'nar/0j4z7pq2z31k71cwjx115pcslqr74mhvlwihv247qbj5scqjyg2b.nar.xz' does not exist in binary cache 'https://cache.iog.io'
[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib-beam-core-0.9.2.1 (buildPhase): [20 of 32] Compiling Database.Beam.Query.Ord ( Datab[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib-beam-core-0.9.2.1 (bui[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib-beam-core-0.9.2.1 (buildPhase): [20 of 3[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib-beam-c[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-lib-beam[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-core-l[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building beam-co[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building be[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building be[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building be[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB[1/1121/1596 built, 1699 copied (12230.6 MiB)[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] b[1/1121/1596 built, 1699 copied (12230.6 MiB), 1831.[1/1121/1596 built, 1699 copied (12230.6 [1/1121/1596 built, 1699 copied (12230.6 [1/1121/1596 built, 1699 copied (12230.6 [1/11[1/1121/1596 built, 1699 copied (12230.6[1/1121/1596 built, 1699 copied [1/1121/1596 built, 1699 copied (122[1/1121/1596 built, 1699 copied (122[1/1121/1596 built, 1699 copied (122[1/1121/1596[1/1523/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building cardano-wallet-core-lib-cardano-wallet-core-2022.7.1 (buildPhase): [138 of 138] Compiling Cardano.Pool.DB.Sqlite ( sr[1/1523/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building cardano-wallet-core-lib-cardano-wallet-core-2022.7.1 (buildPhase): [138 of 138] Compiling Cardano.Pool.D[1/1523/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building cardano-wallet-core-lib-cardano-wallet-core-2022.7.1 (buildPhase): [138 of 138] Compiling Cardano.Pool.D[1/1523/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building car[1/1552/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building playground-common-lib-playground-common-1.0.0.0 (buildPhase): [ 8 of 19] Compiling Playground.Types [1/1552/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building playground-common-lib-playground-common-1.0.0.0 (buildPhase): [ 8 of 19] Compiling Playground.Types ( sr[1/1552/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building playground-common-lib-playground-common-1.0.0.0 (buildPhase): [ 9 of 19] Compiling Playground.Interprete[1/1552/1596 built, 1699 copied (12230.6 MiB), 1831.0 MiB DL] building playground-common-liWARNING: Running COOP services requires having $ export LC_CTYPE=C.UTF-8 LC_ALL=C.UTF-8 LANG=C.UTF-8

I'm not sure how best to debug this when it happens, or when it becomes a sunken cost, if that makes sense (i.e. do I quit and try again? somehow seek alternatives for the sources?). If you have any thoughts about if and how we can document debug steps, it may be helpful.

Addiitonally, I suspect I only needed a few novel components in the new environment having recently built the cardano-node:

[coop-env ~ coop-tutorial] $ ls /usr/bin/cardano/
bech32                               cardano-ping                         chain-sync-client-with-ledger-state  db-synthesizer                       scan-blocks
cardano-cli                          cardano-submit-api                   configuration/                       ledger-state                         scan-blocks-pipelined
cardano-node                         cardano-testnet                      db-analyser                          locli                                stake-credential-history
cardano-node-chairman                cardano-topology                     db-converter                         plutus-example                       trace-dispatcher-examples

But this I'm still getting up to speed a bit. I just want to keep you posted and in the loop.

I have a [coop-env ~ coop-tutorial] nix environment now so will get back to this tomorrow!

[bladyjoker]

I think I got unlucky tonight, the build took ~4.5 hours which is the first time on this new server which isn't too great. (I managed to get some reading done in the meantime)

I'll include the full trace, the important part seems to be:

Hey @ross-spencer that sounds awful :D Let me try and remove the MLabs Cache and try to build it from scratch! Usually such long build times are either due to Nix misconfig and this error could mean your Cache went down in the middle of it!

[bladyjoker]

I'm not sure how best to debug this when it happens, or when it becomes a sunken cost, if that makes sense (i.e. do I quit and try again? somehow seek alternatives for the sources?). If you have any thoughts about if and how we can document debug steps, it may be helpful.

Nix is...well...useful but comes at a cost of needing to know Nix well which can be difficult (it was for me)

[bladyjoker]

Addiitonally, I suspect I only needed a few novel components in the new environment having recently built the cardano-node:

I suggest you really only use the components that the coop-env gives you as this guarantees compatibility between the various tools used. That's the primary reason to use Nix, to have bit-level reproducible builds that compounded with tests gives you assurances that the tools are mutually compatible.

I've had situations where a 'couple of weeks' difference between cardano-cli and cardano-node was enough to warrant subtle breakages because of some uncaught incompatibilities.

[bladyjoker]

I see you built 1596 packages, that is a lot and makes sense it took so much time. I asked MLabs to provide customers with public Nix build caches, and will revisit this topic today and see where we land.

[bladyjoker]

@ross-spencer I believe the plan is to move the repo over to Plutonomicon, which has a public Nix cache, that should do the trick of making the build artifacts publicly available.

@GeorgeFlerovsky is that a correct statement?

[GeorgeFlerovsky]

Yes, correct. Hercules CI (what we use internally) can be set up to build and cache open-source repositories for free.

https://hercules-ci.com/#pricing

[ross-spencer]

Thanks Dražen, these responses are super helpful. I will keep them in mind as I work through this and see how i can contribute back. George, thanks for the clarification. I feel that will be a big help to the users here.

[GeorgeFlerovsky]

@ross-spencer @peterVG

Sorry, to clarify: Hercules CI will build open-source repositories for free and integrates with GitHub; however, to speed up the builds a binary cache of the nix dependencies for the project must be set up.

Unfortunately, Cachix—the most popular nix cache provider—no longer provides free caching for open-source projects. If you wish to speed up builds for users of COOP (including Orcfax), you will need to subscribe to Cachix or an alternative nix cache provider.

https://www.cachix.org/pricing