Use Git LFS for the videos

[fiatjaf]

The repository is getting very heavy with all these video proofs. Git LFS supposedly solves that. Is that desirable? If yes I can try to add it.

[nvk]

Yes, this is not a great solution. Wondering if it’s better to just host them on YouTube. There’s not a lot of practical game and having them here. maybe Bitcoin.tv

[fiatjaf]

I just tried this in my fork at https://github.com/fiatjaf/bitcoinbinary.alt and it seems to be working:

git lfs install
git lfs migrate import --everything --include="*/*.webm"

It rewrites the entire history to remove the actual files and replace them with links that point to the actual files which are then stored on GitHub somewhere. The external links keep working. Going forward all .webm videos will be LFSed automatically.

If rewriting history is unacceptable we can also just do LFS for files added from now going forward.

[nvk]

That’s acceptable, the only challenge it is GitHub specific. Other hosted git alternatives may not support it. But even with that in mind, shouldn’t be a big deal. Are you willing to spend time on the GitHub actions bot to support it?

[fiatjaf]

Gitea, GitLab and BitBucket all support LFS apparently, but this repository is already so dependent on the GitHub Actions bot that I don't know if that matters much.

Anyway, there are two other minor problems:

1. There is a 1GB free quota: https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-storage-and-bandwidth-usage#storage-quota
2. The videos don't play on the browser (at least for me they just load forever): https://media.githubusercontent.com/media/fiatjaf/bitcoinbinary.alt/main/poncho/poncho-v0.3.0-video.webm -- but if you download them from the same URL (by either right-clicking or using wget) they will download correctly and play.

[nvk]

Yes, would have to pay which is not a huge deal but could easily become out of control with more projects being added.

Maybe videos to to youtube and we only keep the most recent build version video here (which is what counts anyways.)

[fiatjaf]

But I really like the automated build videos!

Have you thought about asciinema? That could work well, it records just the terminal, doesn't use a lot of space and we can even embed a JS player on the website for the recordings. It's also easy to record manually from people's own computers.

I can try to get it working if you think it's a good idea.

[nvk]

I have not played with that one. Maybe the past version archive is in asciinema while the latest tweeted version is video?

[siim-m]

Just saw this discussion and thought I'd mention just in case – the Twitter video gets uploaded from the GH Actions runner's ephemeral storage, and is not reliant on any file committed to the repo. So from that point of view, everything stored could be in asciinema format.

[fiatjaf]

@siim-m I just did that on fiatjaf@2a3a108. It generates the mp4 video and uploads to Twitter, then references the tweet from the HTML and also the asciinema .cast file.

The only problem is that using the asciinema JS player on the HTML page is not working, I think it's because GitHub refuses to let us load the asciinema file from JS: https://raw.githack.com/fiatjaf/bitcoinbinary.org/main/index.html, so now I'm thinking of uploading them to asciinema.com too.

[fiatjaf]

Wait, nevermind, it will work if the repository is published as a GitHub Page. Is that the case? Then this will work.

[siim-m]

I do believe it should be hosted on GH Pages (considering the CNAME file in the repo). @nvk can you confirm? I don't have enough permissions on repo to see the Pages related stuff.

If that's the case, I think we could just reference the relative path of the .cast file rather than going via Githack.

I haven't used that JS player before and don't know how it works but if you need help I'm happy to dig in and have a go.

[nvk]

I do believe it should be hosted on GH Pages (considering the CNAME file in the repo). @nvk can you confirm? I don't have enough permissions on repo to see the Pages related stuff.

If that's the case, I think we could just reference the relative path of the .cast file rather than going via Githack.

I haven't used that JS player before and don't know how it works but if you need help I'm happy to dig in and have a go.

Yes. It’s hosted here. See your new permissions.

[nvk]

Ooops, just checked, @siim-m you were admin already.

[fiatjaf]

OK, I got this working after a million attempts.

Please take a look: https://fiatjaf.github.io/bitcoinbinary.org/

I've also changed the artifacts.sh file to be just a repository of environment variable values that can be sourced with . (and deleted two apparently useless find lines from the coldcard files because they were breaking the build).

I did that such that the <li> line for index.html could be generated automatically for all builds to include the asciinema cast and the tweet link (otherwise all artifacts.sh files would have to be changed and DRY etc.)

I've then deleted all entries from index.html on purpose just to trigger rebuilds on all projects and check if all builds would work in the new setup -- and they did (and my Twitter account got spammed with automated tweets from all projects, but I've deleted those now so the tweet links are broken). The only problem is that some of the projects are missing the SHA256 value.

I don't understand why. I've double-checked and all projects seem to be set up for taking the sha256 in the same way, it is a mystery to me why that has worked for some and not for others. Do you have any ideas, @siim-m?

(Blockstream Green, Bitcoin Core and CLN are showing the old <li> lines because in my first attempt at rebuilding everything they were taking too long, so for the second attempt I kept their old entries on index.html.)

[siim-m]

OK, I got this working after a million attempts.

Please take a look: https://fiatjaf.github.io/bitcoinbinary.org/

That looks cool! Pretty clever how that JS player works.

The only problem is that some of the projects are missing the SHA256 value.

I don't understand why. I've double-checked and all projects seem to be set up for taking the sha256 in the same way, it is a mystery to me why that has worked for some and not for others. Do you have any ideas, @siim-m?

Not very familiar with that part of the script, unfortunately. I'll have some more time tomorrow, so will take a closer look and see if I can find anything.

[siim-m]

@fiatjaf looks like it's due to various different issues and I believe they are all unrelated to what you were working on (some of them can be seen missing on the actual site as well).

Below are two examples that I looked at in more detail and I suspect the others are similar.

Example 1 - LND

In artifacts.sh the $GOPATH variable doesn't refer to anything, so it cannot find the binary. After adding it in, the SHA256 value came up fine for me.

# ...
GOPATH=~/go # Adding this line fixed the issue
SHA256=`shasum -a 256 $GOPATH/bin/lnd | cut -f 1 -d ' '`
# ...

Example 2 - Mycelium

Looking at the end of the video, looks like it didn't even build, so there would be no binary to hash.

I suspect the reason is that in repro.yml we are specifying only the major version of Java:

- name: Setup Java 8 (for Simple Bitcoin Wallet, Mycelium)
  if: ${{ matrix.project == 'simple-bitcoin-wallet' || matrix.project == 'mycelium-android' }}
  uses: actions/setup-java@v3
  with:
    distribution: 'temurin'
    java-version: '8'

But in steps.sh we have a very specific version:

export JAVA_HOME=/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.332-9/x64

On the live site there is a SHA256 value for Mycelium, so I suspect that since then there was a minor release of Java 8 and that's why it has broken.

[fiatjaf]

When are we going to deploy this?

[siim-m]

@fiatjaf, could you do a PR from your fork and I'll try to get it deployed? Note we don't want Git LFS but asciinema is fine.