Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] add script to generate and package a binary toolchain #5162

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

lrusak
Copy link
Member

@lrusak lrusak commented Feb 20, 2021

This is something I've been looking at for a while. I don't like having to use external toolchains to build our images and would like to be able to at least use our own generated toolchain if we can.

This initial step allows generating an aarch64 toolchain to be used in the arm 64/32 split builds. This will replace the arm toolchain package gcc-arm-aarch64-none-linux-gnu.

I've thought about going further to generate a toolchain that would actually replace building the toolchain when generating an image, but I'm not sure I want to go down that road yet.

Ideally this could be expanded to cover all of the external toolchains that we need but I'm not sure the extent of that yet.

This should be tested on all platforms (that use 64/32 split) before continuing (I've only build tested for now). I've also not pushed any packages to the mirrors.

If you want to test you will need to generate your own package for now:

$ ./tools/generate-toolchain 1

$ mkdir -p sources/gcc-arm-aarch64-libreelec-linux-gnueabi

$ mv gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz sources/gcc-arm-aarch64-libreelec-linux-gnueabi

$ sha256sum sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz | awk '{print $1}' > sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256

$ echo "https://sources.libreelec.tv/devel/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz" > sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.url

$ sed "s/PKG_SHA256=.*/PKG_SHA256=$(cat sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256)/" -i packages/lang/gcc-arm-aarch64-libreelec-linux-gnueabi/package.mk

Usage:

$ ./tools/generate-toolchain --help
Usage: ./tools/generate-toolchain [rev]

Output:

$ ./tools/generate-toolchain 1

<snip>
normal multithread build output
</snip

===========================================================
Toolchain was built successfully
Compressing archive: gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-1.tar.xz

 713MiB 0:00:36 [19.5MiB/s] [=============================================================================] 102%

Toolchain was packaged successfully

A Toolchain version file is generated and inserted into the packaged toolchain. This file is printed after the archive is generated

Toolchain Version File:
Toolchain name: gcc-arm-aarch64-libreelec-linux-gnueabi
Toolchain version: 10.2.0-1

Package versions:
===========================================================

autoconf
 - version: 2.69
 - sha256: 64ebcec9f8ac5b2487125a86a7760d2591ac9e1d3dbd59489633f9de62a57684
 - url: http://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz

autoconf-archive
 - version: 2019.01.06
 - sha256: 17195c833098da79de5778ee90948f4c5d90ed1a0cf8391b4ab348e2ec511e3f
 - url: http://ftpmirror.gnu.org/autoconf-archive/autoconf-archive-2019.01.06.tar.xz

automake
 - version: 1.15.1
 - sha256: af6ba39142220687c500f79b4aa2f181d9b24e4f8d8ec497cea4ba26c64bedaf
 - url: http://ftpmirror.gnu.org/automake/automake-1.15.1.tar.xz

binutils
 - version: 2.35.1
 - sha256: 3ced91db9bf01182b7e420eab68039f2083aed0a214c0424e257eae3ddee8607
 - url: http://ftp.gnu.org/gnu/binutils/binutils-2.35.1.tar.xz

bison
 - version: 3.7.4
 - sha256: a3b5813f48a11e540ef26f46e4d288c0c25c7907d9879ae50e430ec49f63c010
 - url: http://ftpmirror.gnu.org/bison/bison-3.7.4.tar.xz

bzip2
 - version: 1.0.8
 - sha256: ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269
 - url: https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz

ccache
 - version: 3.7.12
 - sha256: a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400
 - url: https://github.com/ccache/ccache/releases/download/v3.7.12/ccache-3.7.12.tar.xz

cmake
 - version: 3.19.2
 - sha256: e3e0fd3b23b7fb13e1a856581078e0776ffa2df4e9d3164039c36d3315e0c7f0
 - url: https://cmake.org/files/v3.19/cmake-3.19.2.tar.gz

configtools
 - version: c8ddc8472f8efcadafc1ef53ca1d863415fddd5f
 - sha256: 6389d62e4e55554c764c2c0deb5b42767f34d7f274728c28355fedbaa337165b
 - url: http://git.savannah.gnu.org/cgit/config.git/snapshot/config-c8ddc8472f8efcadafc1ef53ca1d863415fddd5f.tar.gz

flex
 - version: 2.6.4
 - sha256: e87aae032bf07c26f85ac0ed3250998c37621d95f8bd748b31f15b33c45ee995
 - url: https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz

gcc
 - version: 10.2.0
 - sha256: b8dd4368bb9c7f0b98188317ee0254dd8cc99d1e3a18d0ff146c855fe16c1d8c
 - url: http://ftpmirror.gnu.org/gcc/gcc-10.2.0/gcc-10.2.0.tar.xz

gettext
 - version: 0.21
 - sha256: d20fcbb537e02dcf1383197ba05bd0734ef7bf5db06bdb241eb69b7d16b73192
 - url: http://ftp.gnu.org/pub/gnu/gettext/gettext-0.21.tar.xz

glibc
 - version: 2.32
 - sha256: 1627ea54f5a1a8467032563393e0901077626dc66f37f10ee6363bb722222836
 - url: http://ftp.gnu.org/pub/gnu/glibc/glibc-2.32.tar.xz

gmp
 - version: 6.2.1
 - sha256: fd4829912cddd12f84181c3451cc752be224643e87fac497b69edddadc49b4f2
 - url: https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz

intltool
 - version: 0.51.0
 - sha256: 67c74d94196b153b774ab9f89b2fa6c6ba79352407037c8c14d5aeb334e959cd
 - url: http://launchpad.net/intltool/trunk/0.51.0/+download/intltool-0.51.0.tar.gz

libffi
 - version: 3.3
 - sha256: 72fba7922703ddfa7a028d513ac15a85c8d54c8d67f55fa5a4802885dc652056
 - url: ftp://sourceware.org/pub/libffi/libffi-3.3.tar.gz

libtool
 - version: 2.4.6
 - sha256: e3bd4d5d3d025a36c21dd6af7ea818a2afcd4dfc1ea5a17b39d7854bcd0c06e3
 - url: http://ftpmirror.gnu.org/libtool/libtool-2.4.6.tar.gz

libxml2
 - version: 2.9.10
 - sha256: aafee193ffb8fe0c82d4afef6ef91972cbaf5feea100edc2f262750611b4be1f
 - url: ftp://xmlsoft.org/libxml2/libxml2-2.9.10.tar.gz

libxslt
 - version: 1.1.34
 - sha256: 98b1bd46d6792925ad2dfe9a87452ea2adebf69dcb9919ffd55bf926a7f93f7f
 - url: ftp://xmlsoft.org/libxml2/libxslt-1.1.34.tar.gz

linux
 - version: 5.10.11
 - sha256: 02ef2b56b00fc5145701c603a5235e1265772e40d488a936b27ba65fe78e710f
 - url: https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.10.11.tar.xz

m4
 - version: 1.4.18
 - sha256: 6640d76b043bc658139c8903e293d5978309bf0f408107146505eca701e67cf6
 - url: http://ftpmirror.gnu.org/m4/m4-1.4.18.tar.bz2

make
 - version: 4.3
 - sha256: e05fdde47c5f7ca45cb697e973894ff4f5d79e13b750ed57d7b66d8defc78e19
 - url: http://ftpmirror.gnu.org/make/make-4.3.tar.gz

meson
 - version: 0.56.0
 - sha256: 291dd38ff1cd55fcfca8fc985181dd39be0d3e5826e5f0013bf867be40117213
 - url: https://github.com/mesonbuild/meson/releases/download/0.56.0/meson-0.56.0.tar.gz

mpc
 - version: 1.2.1
 - sha256: 17503d2c395dfcf106b622dc142683c1199431d095367c6aacba6eec30340459
 - url: http://ftpmirror.gnu.org/mpc/mpc-1.2.1.tar.gz

mpfr
 - version: 4.1.0
 - sha256: 0c98a3f1732ff6ca4ea690552079da9c597872d30e96ec28414ee23c95558a7f
 - url: http://ftpmirror.gnu.org/mpfr/mpfr-4.1.0.tar.xz

ninja
 - version: 1.10.2
 - sha256: ce35865411f0490368a8fc383f29071de6690cbadc27704734978221f25e2bed
 - url: https://github.com/ninja-build/ninja/archive/v1.10.2.tar.gz

openssl
 - version: 1.1.1i
 - sha256: e8be6a35fe41d10603c3cc635e93289ed00bf34b79671a3a4de64fcee00d5242
 - url: https://www.openssl.org/source/openssl-1.1.1i.tar.gz

p7zip
 - version: 16.02
 - sha256: 5eb20ac0e2944f6cb9c2d51dd6c4518941c185347d4089ea89087ffdd6e2341f
 - url: http://downloads.sourceforge.net/project/p7zip/p7zip/16.02/p7zip_16.02_src_all.tar.bz2

pathlib
 - version: 1.0.1
 - sha256: 6940718dfc3eff4258203ad5021090933e5c04707d5ca8cc9e73c94a7894ea9f
 - url: https://files.pythonhosted.org/packages/source/p/pathlib/pathlib-1.0.1.tar.gz

pigz
 - version: 2.6
 - sha256: 577673676cd5c7219f94b236075451220bae3e1ca451cf849947a2998fbf5820
 - url: https://github.com/madler/pigz/archive/v2.6.tar.gz

pkg-config
 - version: 0.29.2
 - sha256: 6fc69c01688c9458a57eb9a1664c9aba372ccda420a02bf4429fe610e7e7d591
 - url: http://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz

Python3
 - version: 3.8.7
 - sha256: ddcc1df16bb5b87aa42ec5d20a5b902f2d088caa269b28e01590f97a798ec50a
 - url: http://www.python.org/ftp/python/3.8.7/Python-3.8.7.tar.xz

rsync
 - version: 3.2.3
 - sha256: becc3c504ceea499f4167a260040ccf4d9f2ef9499ad5683c179a697146ce50e
 - url: https://download.samba.org/pub/rsync/src/rsync-3.2.3.tar.gz

sed
 - version: 4.8
 - sha256: f79b0cfea71b37a8eeec8490db6c5f7ae7719c35587f21edb0617f370eeff633
 - url: http://ftpmirror.gnu.org/sed/sed-4.8.tar.xz

setuptools
 - version: 51.1.2
 - sha256: 1f3db173c1d8f8753dce0b6c18017955863fc39a0613e5c20bfdd107f331fafb
 - url: https://github.com/pypa/setuptools/archive/v51.1.2.tar.gz

util-linux
 - version: 2.36.1
 - sha256: 09fac242172cd8ec27f0739d8d192402c69417617091d8c6e974841568f37eed
 - url: http://www.kernel.org/pub/linux/utils/util-linux/v2.36/util-linux-2.36.1.tar.xz

xmlstarlet
 - version: 1.6.1
 - sha256: 15d838c4f3375332fd95554619179b69e4ec91418a3a5296e7c631b7ed19e7ca
 - url: http://netcologne.dl.sourceforge.net/project/xmlstar/xmlstarlet/1.6.1/xmlstarlet-1.6.1.tar.gz

xz
 - version: 5.2.5
 - sha256: 5117f930900b341493827d63aa910ff5e011e0b994197c3b71c08a20228a42df
 - url: http://tukaani.org/xz/xz-5.2.5.tar.bz2

zlib
 - version: 1.2.11
 - sha256: 4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea0a2114e19bf066
 - url: http://zlib.net/zlib-1.2.11.tar.xz

zstd
 - version: 1.4.8
 - sha256: 32478297ca1500211008d596276f5367c54198495cf677e9439f4791a4c69f24
 - url: https://github.com/facebook/zstd/releases/download/v1.4.8/zstd-1.4.8.tar.gz

@heitbaum
Copy link
Contributor

Does this / Can this address the #4719 rust glibc mismatch?

@lrusak
Copy link
Member Author

lrusak commented Feb 20, 2021

Does this / Can this address the #4719 rust glibc mismatch?

No that's completely unrelated.

@heitbaum
Copy link
Contributor

Does this / Can this address the #4719 rust glibc mismatch?

No that's completely unrelated.

was <naïvely> hoping ... Guess I could always build x86 from Arch64 - lol... I'll work on getting myself a HH 2.33 glibc docker buildhost - that works.

@antonlacon
Copy link
Contributor

Before packages go to mirrors, --march=native should be removed from host cflags. config/optimize line 12.

I've thought about going further to generate a toolchain that would actually replace building the toolchain when generating an image, but I'm not sure I want to go down that road yet.

This means building everything other than the kernel with a premade toolchain too?

@SupervisedThinking
Copy link
Contributor

@lrusak
Copy link
Member Author

lrusak commented Feb 20, 2021

Shouldn't this be device specific in some way?

https://github.com/LibreELEC/LibreELEC.tv/pull/5162/files#diff-29b791fee9b13233dc1f73b6d2bd7b50d025709236b43d672511b30ba8456878R31-R35

No, the idea is to create a device agnostic toolchain

@lrusak
Copy link
Member Author

lrusak commented Feb 20, 2021

Before packages go to mirrors, --march=native should be removed from host cflags. config/optimize line 12.

This is a great point. I'll have to think about the best solution for that. In the mean time this still is considered WIP.

This means building everything other than the kernel with a premade toolchain too?

Possibly. We have talked about splitting the toolchain from the build completely but I'm not really sure if that is a good idea. The solution provided here should just be considered as an alternative to the external toolchains that we already have to provide.

@lrusak
Copy link
Member Author

lrusak commented Feb 20, 2021

I've updated the PR to allow disabling host optimizations

I've uploaded a test tarball here:
http://lrusak.libreelec.tv/public/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz
http://lrusak.libreelec.tv/public/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256

it would be good if someone can verify it would on their system:

$ ./gcc-arm-aarch64-libreelec-linux-gnueabi/bin/aarch64-libreelec-linux-gnueabi-gcc -dumpversion
10.2.0

@antonlacon
Copy link
Contributor

Two issues:

First is the gcc wrapper is hardcoding your $TOOLCHAIN path resulting in:

ian@localhost ~/le-test/gcc-arm-aarch64-libreelec-linux-gnueabi $ bin/aarch64-libreelec-linux-gnueabi-gcc -dumpversion bin/aarch64-libreelec-linux-gnueabi-gcc: line 2: /home/lukas/git/libreelec/build.LibreELEC-aarch64-toolchain.aarch64-9.80-devel/toolchain/bin/ccache: No such file or directory

Good news, I edited the wrapper to remove ccache (I don't use it), and pointing to where i extracted the tarball and 10.2.0 popped out in response.

Second issue is many symoblic links are being made with the same underlying issue with $TOOLCHAIN, resulting in broken links: https://pastebin.com/raw/WJmW5cYh

@antonlacon
Copy link
Contributor

Does PROJECT=ARM DEVICE=ARMv8 ARCH=aarch64 substitute for the dummy project? The only difference I'm seeing is the +crypto cpu flag, which a kernel won't care about because that's controlled with CONFIG_* settings.

@antonlacon
Copy link
Contributor

Some more thoughts from working with this a little:

For consideration: antonlacon@6b86f63
This changes generate-toolchain to use the ARMv8 project files. I'm not certain any of the dummy project settings are presently used in testing thus far without exporting.

The symbolic link issue can be solved at time of the tarball creation by dereferencing the links (tar -h) to duplicate the files.

I believe that building gcc:host instead of the whole toolchain will meet the needs for the kernel. It trims a couple of packages off.

Size in comparison to the arm-provided toolchain is bigger than what is built here (+40MB compressed, +150(?)MB uncompressed). I believe this is because build dependencies are being packaged in, when it should just need runtime dependencies. Packaging python3 is a significant chunk of the size difference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants