Improve kernel building automation #199

filberg · 2020-08-07T17:32:30Z

This PR refactors makefiles and build scripts to make it easier to build an updated kernel in a totally automated way.

Main changes:

allow to set "$KVER" programmatically, without needing to manually edit BuildCommon.mk
introduce the new utility function "build_latest_kernel()" to automatically check for upstream updates and launch new kernel builds whenever a new version is available.
minor reworkings and bug-fixing

This changes should make it fairly easy to put up a totally headless/automated build server via a simple cron job and easily keep up with upstream kernel updates.

Signed-off-by: Fil <fil@filberg.eu>

This fixes kernel/makefile to skip downloading the source tar and/or extracting it during kernel rebuilds, it its not needed (i.e. when $(KERNEL_EXTRACTED) already exists). The fix consists in turning the directory prerequisites from "normal" to "order-only" prerequisites, so that any timestamp change in the required folders doesn't uselessly trigger the make'ing of $(KERNEL_EXTRACTED) target. Also, this commit adds $(KERNEL_TAR_SIGNATURE) target as a prerequisite of $(KERNEL_EXTRACTED) to ensure that the corresponding signature file always exists for a given source tar. This fixes the "file not found" error when gpg-verifying the downloaded tar when building a new kernel version. Signed-off-by: Fil <fil@filberg.eu>

This commit introduces a feature that allows to programmatically override the value of "KVER" by export'ing the global PRAWNOS_KVER with the desired kernel version. This allows to build any kernel version at runtime without depending on the hard-coded value in BuildCommon.mk e.g. to build version 5.4.57: $ export PRAWNOS_KVER="5.4.57"; make kernel; Signed-off-by: Fil <fil@filberg.eu>

This introduces function build_latest_kernel() that automatically fetches the latest kernel version from the upstream URL and launches a regular kernel build for the latest version found. This should prove to be very usefull to automate kernel builds on a headless build server (e.g. via a cron job), by continuously monitoring upstream for any new release. Signed-off-by: Fil <fil@filberg.eu>

*) Minor cleanups *) Read /path/to/vmlinux.kpart from command line args instead of hard-coding it inside the script. Signed-off-by: Fil <fil@filberg.eu>

SolidHal

I would like to keep the signature in the tree

SolidHal · 2020-08-08T06:43:02Z

Thanks for the work @filberg!
it might take me some time to review this due to real life things, and my current focus being finishing up arm64 support
I'm positive when I merge in arm64 support some of this will need to be reworked, but I can take care of that :)

filberg · 2020-08-18T09:40:48Z

@SolidHal :

I would like to keep the signature in the tree

For what purpose would you like to do keep it there?
If that's for security purposes, then it's better to keep the linux-libre public key in prawnos' tree, and let the signature file be downloaded each time.
This way we maintain the flexibility of dynamically downloading the desired kernel version, while still preserving a (faint) sense of security by keeping the root of trust in-tree.

One could then add a prerequisite make target to verify that the public key is in the keyring and import it if necessary.

I can work on a PR in this direction, if you agree with the above reasoning.

Cheers!

SolidHal · 2020-08-21T21:58:11Z

as you can see in kernel/makefile we already have the linux libre signing key and use it for verification:

$(KERNEL_TAR):
	wget https://www.linux-libre.fsfla.org/pub/linux-libre/releases/$(KVER)-gnu/linux-libre-$(KVER)-gnu.tar.lz -O $(KERNEL_TAR)
	gpg --import $(KERNEL_SOURCES)/linux-libre-signing-key.gpg
	gpg --verify $(KERNEL_TAR_SIGNATURE) $(KERNEL_TAR)

I think my desire to keep the signature in the tree came from a time where I was trying to keep the entire kernel source in the tree as well, unfortunately it is too large for github.

you make a good point about the public key being the true root of trust, I guess there isn't much point to keeping the signature in the tree as well....

One last thing then, could you clear up for me the need for the "pipes/or/bars" in target dependencies like here?

$(ATH9K_EXTRACTED): | $(PRAWNOS_BUILD)

filberg · 2020-08-23T11:48:51Z

@SolidHal:

One last thing then, could you clear up for me the need for the "pipes/or/bars" in target dependencies like here?

Sure!
As I recently discovered, the pipe is used in makefiles to separate regular prerequisites from "order-only" prerequisites. Everything to the right of a pipe is an "order-only" prereq.

Basically, as explained in the corresponding commit message, changing the directory prerequisites to "order-only" ensures that dependent targets won't be uselessly rebuilt when the directory timestamp changes, for instance when building something inside the extracted kernel directory.
Otherwise, issuing a make kernel always triggers the download and extraction of the corresponding tarball, even if $KVER remains the same and even if the kernel is indeed already extracted.

For any further commenting, just ask; I'll be happy to explain if I'm able to ;-)

Cheers!

SolidHal · 2020-09-05T21:26:28Z

Ah, order only pre-reqs seem pretty great! I had actually already fixed the problem of rebuilds happening due to directory time stamps a different way, but I think I like order only pre-reqs better.
EDIT: I implemented the ordered pre-reqs and hit an interesting issue with wget and make. That portion of this PR has been cannibalized by my most recent commit so that I could spread order only pre-reqs across the code base and rip out the bad fix I had originally done (I just mass mkdir'd the build dirs).

The other parts of this PR will need some rebase work that I can handle if you don't want to. Thanks @filberg!

filberg · 2020-09-08T08:05:22Z

@SolidHal

The other parts of this PR will need some rebase work that I can handle if you don't want to. Thanks @filberg!

I can make the rebase.
I spotted a bug in my PR and I'd like to fix it while I'm at rebasing..

This fixes both the behaviour of get_build_kver() and its invocation in other parts of the script, so that it executes in the same shell as the caller's. This way, all parts of the version string can be retrieved in a single call. Before this fix, the function had to be invoked in a subshell $(foo) to capture the full version string as its output, while the export'ed values of $PRAWNOS_KVER_* could not be read by the caller in the parent shell. Signed-off-by: Fil <fil@filberg.eu>

filberg · 2020-09-15T10:57:45Z

@SolidHal Ok, I think I merged and rebased correctly with your latest work..

SolidHal · 2020-09-17T15:30:13Z

.gitignore

@@ -2,3 +2,4 @@ tmp.*
 PrawnOS-Shiba-*
 PrawnOS-initramfs.cpio.gz
 build
+kernel/sources


this can be dropped as the directory doesn't exist anymore :)

SolidHal · 2020-09-17T15:32:07Z

scripts/BuildScripts/BuildCommon.mk

+#PRAWNOS_KERNEL_IMAGE_DEBVER := 1
+#PRAWNOS_KERNEL_HEADERS_DEBVER := 1
+endif
+


I have been thinking about what the proper answer is here. I'm thinking the following should be bumped to
#PRAWNOS_KERNEL_VER := 200
#PRAWNOS_KERNEL_IMAGE_DEBVER := 200
#PRAWNOS_KERNEL_HEADERS_DEBVER := 200
where 200 was picked arbitrarily in order to prevent the local system from conflicting with future packages that could end up on the apt repo.

SolidHal · 2020-09-17T15:34:04Z

Thanks! there are a couple small adjustments that I can make after this is merged in, I added them as comments here so you can see what my intent is. I figure this PR has been around long enough (my fault), so I'll merge and make the small tweaks :)

Fil added 5 commits August 7, 2020 16:45

Remove kernel/sources from git tracking

91d1140

Signed-off-by: Fil <fil@filberg.eu>

Refactor and improve FlashKernelPart.sh

fc238dc

*) Minor cleanups *) Read /path/to/vmlinux.kpart from command line args instead of hard-coding it inside the script. Signed-off-by: Fil <fil@filberg.eu>

SolidHal reviewed Aug 8, 2020

View reviewed changes

Fil and others added 2 commits September 15, 2020 12:07

Merge branch 'master' into master

a13a956

SolidHal reviewed Sep 17, 2020

View reviewed changes

SolidHal merged commit 9a3f811 into SolidHal:master Sep 17, 2020

SolidHal added a commit that referenced this pull request Sep 17, 2020

small modifications to #199

22193bf

JeremyRand mentioned this pull request Oct 3, 2021

Document PRAWNOS_KVER #291

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve kernel building automation #199

Improve kernel building automation #199

filberg commented Aug 7, 2020

SolidHal left a comment

SolidHal commented Aug 8, 2020

filberg commented Aug 18, 2020

SolidHal commented Aug 21, 2020

filberg commented Aug 23, 2020

SolidHal commented Sep 5, 2020 •

edited

Loading

filberg commented Sep 8, 2020

filberg commented Sep 15, 2020

SolidHal Sep 17, 2020

SolidHal Sep 17, 2020

SolidHal commented Sep 17, 2020

Improve kernel building automation #199

Improve kernel building automation #199

Conversation

filberg commented Aug 7, 2020

SolidHal left a comment

Choose a reason for hiding this comment

SolidHal commented Aug 8, 2020

filberg commented Aug 18, 2020

SolidHal commented Aug 21, 2020

filberg commented Aug 23, 2020

SolidHal commented Sep 5, 2020 • edited Loading

filberg commented Sep 8, 2020

filberg commented Sep 15, 2020

SolidHal Sep 17, 2020

Choose a reason for hiding this comment

SolidHal Sep 17, 2020

Choose a reason for hiding this comment

SolidHal commented Sep 17, 2020

SolidHal commented Sep 5, 2020 •

edited

Loading