core:build draft

[DragosPopse]

This is a repost of #2874 . I moved it to a different branch rather than DragosPopse/Odin:main in order to be able to revert to the original odin version and start work on other things. Sorry for the duplicate, will try to be more careful next time.

Copy-paste of last important message

I have finalized the API. Things might still need extra testing, but so far it seems to work.

example build script:

package src_build

import "core:build"
import "core:os"
import "core:strings"
import "core:fmt"

Mode :: enum {
	Debug,
	Release,
}

Target :: struct {
	using target: build.Target,
	mode: Mode,
}

project: build.Project

target_debug: Target = {
	target = {
		name = "dbg",
		platform = {ODIN_OS, ODIN_ARCH},
	},
	mode = .Debug,
}
target_release: Target = {
	target = {
		name = "rel",
		platform = {ODIN_OS, ODIN_ARCH},
	},
	mode = .Release,
}

run_target :: proc(target: ^build.Target, mode: build.Run_Mode, args: []build.Arg, loc := #caller_location) -> bool {
	target := cast(^Target)target
	config: build.Odin_Config
	config.platform = target.platform
	config.out_file = "demo.exe" if target.platform.os == .Windows else "demo"
	
	// paths must be set with build.relpath / build.abspath in order for them to be relative to the build system root directory (build/../). build.t*path alternatives use context.temp_allocator
	config.src_path = build.trelpath(target, "src")
	config.out_dir = build.trelpath(target, fmt.tprintf("out/%s", target.name))

	switch target.mode {
	case .Debug:
		config.opt = .None
		config.flags += {.Debug}
	case .Release:
		config.opt = .Speed
	}

	switch mode {
	case .Build:
		// Pre-build stuff here
		build.odin(target, .Build, config) or_return 
		// Post-build stuff here
	case .Dev: 
		return build.generate_odin_devenv(target, config, args)
	
	case .Help:
		return false // mode not implemented
	}
	return true
}

@init
_ :: proc() {
	project.name = "Build System Demo"
	build.add_target(&project, &target_debug, run_target)
	build.add_target(&project, &target_release, run_target)
}

main :: proc() {
	info: build.Cli_Info
	info.project = &project
	info.default_target = &target_debug
	build.run_cli(info, os.args)
}

output of odin run build:

odin build "src"
        -out:"out\dbg/demo.exe"
        -subsystem:console
        -build-mode:exe
        -o:none
        -reloc-mode:default
        -debug
        -target:windows_amd64

output of odin run build -- -help:

Syntax: build.exe <flags> <target>
Available Targets:
        dbg
        rel

Builtin Flags - Only 1 [Type] group per call. Groups are incompatible
        -help
                [Help] Displays information about the build system or the target specified.

        -ols
                [Dev] Generates an ols.json for the configuration.

        -vscode
                [Dev] Generates .vscode folder for debugging.

        -build-pre-launch
                [Dev] Runs the build system before debugging. (WIP)

        -include-build-system:"<args>"
                [Dev] Include the build system as a debugging target. (WIP)

        -cwd:"<dir>"
                [Dev] Sets the CWD to the specified directory.

        -cwd-workspace
                [Dev] Sets the CWD to the root of the build system executable.

        -cwd-out
                [Dev] Sets the CWD to the output directory specified in the -out odin flag

        -launch-args:"<args>"
                [Dev] The arguments to be sent to the output executable when debugging.

        -dbg:"<debugger name>"
                [Dev] Debugger type used. Works with -vscode. Sets the ./vscode/launch.json "type" argument

output of odin run build -- -ols -vscode
ols.json:

{
	"collections": [

	],
	"enable_document_symbols": true,
	"enable_semantic_tokens": true,
	"enable_hover": true,
	"enable_snippets": true,
	"checker_args": "-out:\"out/dbg/demo.exe\" -subsystem:console -build-mode:exe -o:none -reloc-mode:default -debug -target:windows_amd64"
}

.vscode/launch.json:

{
	"version": "0.2.0",
	"configurations": [
		{
			"type": "cppvsdbg",
			"request": "launch",
			"preLaunchTask": "",
			"name": "dbg",
			"program": "C:\\dev\\Odin\\examples\\package_build\\out\\dbg\\demo.exe",
			"args": [
				""
			],
			"cwd": "${workspaceFolder}\\out\\dbg"
		}
	]
}

.vscode/tasks.json:

{
	"version": "2.0.0",
	"tasks": [

	]
}

Concept explanation

The idea of core:build would be to provide an api for running complex odin builds + a CLI to help you decide what you want to run. A Target is what it's being run by the cli or via build.run_target. This can be anything from debug/release targets to install or clean. These are setup by the user. In order to run a target, you need to create it, add it with build.add_target to a Project (a collection of targets), and either call build.run_target in code or odin run build -- <target name> via the CLI

Run_Mode is the mode the CLI runs in. It can be one of {.Build, .Dev, .Help}. The mode defaults to Build unless certain flags make it a different mode (running odin run build -- -ols sets the mode to .Dev).
The Run_Mode allows to user to determine what they want to do when the target is being ran. .Build would be the general build mode, where they can call build.odin, copy dlls, etc.
.Dev is used for configuring development enviornment (generating ols.json, information for the debugger they use. As a builtin debugger example, VSCode is implemented, but more can be added in future PRs, and the user can configure their own development environment.
.Help is for displaying information about your target to the user of the CLI. The current PR doesn't have a build.default_help proc. It's planned for later.

In order to facilitate importing build systems into eachother, the build.*path(target, path) functions make sure that your paths are relative to the build system. I have experimenting with ensuring that automatically on the library-side, but it seemed less "magical" this way (even if it still is a tiny bit). Internally, this system uses #caller_location when calling build.add_target in order to set Target.root_dir to dir(#caller_location)/../. This effectively means that the build system will always need to be built from the parent folder that it's in (odin build build as opposed to odin build .). This is the strangest part of this system, but it's going to make a lot of sense for dependency handling (being able to run imported targets like this odin run build -- project2/debug from the "main" build system.

Additional Features

• Wildcard selection for targets: odin run build -- *
• No allocations (or at least aims to be). The system is defined in a way that everything can be temp allocated. There might still be some artifacts of maps and [dynamic] arrays, but those will be removed in later PRs.

Testing for yourself

Open a command line in ODIN_ROOT/examples/package_build
run odin run build to build the project, or odin run build -- -help to display available options

Final notes

The (WIP) flags are still left to be implemented. The system of setting up custom flag is nearly in place, but i'm still thinking if it's supposed to be "general flags" or "per target" flags. There still needs to be an advanced example written where dependencies are being handled and including build systems into eachother.
If the API is good, I'd like to have this PR merged before doing other updates, as I'd like to open a clean PR for further improvements. I want to start clean on the git side, with a custom branch on my own fork where i can keep working on other things.

Future PRs (if this is accepted)

• Implement (WIP) flags
• Better user-code arg handling
• Dependency handling example

Future dream

core:build can be expanded to handle more than odin code. With the dependency handling, users could do something like import vendor:glfw/build and include the build system for glfw into their own, calling it's targets. This can be helpful for a lot of vendor libraries. vendor:glfw/build could be as simple as calling the build.bat on windows, but core:build can be expanded to be able to configure flags for C and CMake projects too.

[gingerBill]

One issue with Odin's flag argument style is that it doesn't really allow for pipe-line passing, or general shell expansion.

I chose this approach in Odin purely because it was as simple and consistent, even if it has its issues. It's also the reason why I have never written a flag library for Odin because which approach do you choose?--given that each approach has its flaws.

[DragosPopse]

I don't think you can "choose" an approach for a flag library if it needs to be in core, as every approach is so fundamentally different from each other (unix style allowing multiple single letter flags under a single - (-rf or -r -f).

My thoughts on that is that you can provide styles (.Odin .Unix .Windows), and do different things based on the style of flags the app supports. For example, core:build would use the core:argument_parser or whatever with the .Odin style.

[DragosPopse]

I'd have some ideas for this in order to make the "setup" cross-style compatible, i might give it a shot eventually when i get some extra time

[gingerBill]

Technically ols is unofficial, so having part of an official tool is kind of "weird". Same with the -vscode thing. Yes both are very useful, but I am not sure if those should be here or not.

[DragosPopse]

yeah i understand that. The flags you pass to your build system can be expanded by yourself, so the builtin flags like -ols and -vscode are things that the user would write themselves anyway, so why not provide them as "sensible defaults". Generally speaking, all .Dev flags are meant for "non-official" dev tools, like setting up your debugger, your language server, anything you need to make your development easier. Personally i think this approach is more useful than it's weird, and I'd like to keep expanding on the .Dev flags for other editors and tools that people use (and can be used in a "generic" way like configuring a json). The usefulness itself also comes from the way core:build in it's current version works, as you are able to configure your dev env specifically for a certain target

[DragosPopse]

so I've been thinking about this. The tools to generate 3rd party things is quite useful, but indeed there is a bit of "weirdness" into it. I do think the feature itself should stay, as it makes core:build more useful than a python/bat/sh script, so here are some things that I could experiment with if you end up not agreeing with 3rd party being in core:

• Make a vendor:build/dev package, or alternatively different packages for each 3rd party tool vendor:build/vscode vendor:build/ols. These can handle setting up default development environments that we support "officially" via the vendor packages. As vendor is already supposed to contain 3rd party things, this solution would make the most sense.
• Create a system to "register" dev environments, something like build.add_dev("vscode", flags_that_trigger_vscode_gen, vscode_gen_proc). This could also work well with the addition of optional vendor:build already made dev environments.

My biggest gripe with any of these solutions is that ultimately I wanted core:build to be a single self contained package, in order to allow importing build systems into eachother, a feature which is very useful when you want to build multiple projects as a single one.

[DragosPopse]

while you think about it, I'll try to handle flags a bit better, and at least separate the third party things better, and we'll see what we do with them. Right now all the .Dev flags are bound to 3rd party generation, so probably the best way to separate them for now is to make the .Dev mode triggered only by a -dev flag

[DragosPopse]

Some points discussed with @gingerBill in our talk a few weeks ago + some points that should still be discussed regarding this. If i remember correctly, the best approach currently is to leave this be for now and make it a separate project for people to play around with + maybe a poll/github-discussion to figure out whether or not people actually find something like this useful. odin build . already does plenty, and a build system has the potential to complicate the blissful simplicity of odin.

Compiler support

Compiler can help with running the build script. odin build . could look for a build.odin or a build folder-package, and run that. This would allow to run odin build . instead of odin run build, but how would that behave with flags passed to the build system? The current approach to core:build is to "build the build system", then use the generated exe multiple times to either set up your environment or build specific targets. an example of the current workflow would be

odin build build # on linux, running with -out:build so that we don't have to spell out "build.out"
./build dbg -vscode -ols # generate the vscode debug settings and ols.json for target "dbg". This command can be separated into 2 calls
./build dbg # actually build the target

How would this look with the odin build . integration? My idea would be something like this (considering the current core:build draft)

• Leave the behavior as it is, odin build . becoming a simple shortcut for odin run build or odin run-file build.odin. This should allow people to keep the simplicity of odin build ., while also allowing for all the fancy flag settings and targets to be used if required via odin build build

Furthermore, compiler support can also give us a reason to have something like odin init ., setting up all the developing environment for a new odin project.

Third party tools support is useful, but also weird.

Flags like -vscode -ols are useful, but having them in the standard is weird. Assuming the idea of how this all works remains the same, how should we proceed with these? My personal opinion is that having tooling integrated with the standard makes everything easier to setup and nice to use, but maybe there are ways to engineer around this and allow embedding third party tooling easier.

Integration with vendor and building C libraries

As it stands right now, core:build allows you to "embed" build systems from other projects into your own build system, exporting their targets and calling all of them from a single executable. This is helpful, because build packages can be written for vendor libraries to achieve something akin to this

// in your build.odin
package my_project_build

import "core:build"
import sdl2 "vendor:sdl2/build"

// in your run_target
sdl2.copy_target_output(src = sdl2.target_debug.dll_path, dest = my_target_debug_path) // or something like that

It also allows you to embed the targets of sdl2/build into your build system, so one could now run something like odin run build -- sdl2.debug
This is already a big QoL for not needing to figure out those paths yourself, or use any sort of env variables.

What is missing however and was briefly discussed was the ability to build C projects. In it's current format, all sdl2.debug can do is call another program or shell script to do that building, without much "api-available customization". So sdl2.debug could do something like execute cmake --build . or a ODIN_ROOT/vendor/sdl2/build.bat.

This effectively makes the odin-side build system of these libraries only responsible for creating a callable target via the odin run build -- target-name setup. I find this acceptable, as most C projects already have their build scripts implemented, and it's a matter of just calling those. Beside that, all C dependencies that come along with odin are third party, so nobody is going to write a build.odin for their Brand New C Project ™️ . Making this be as flexible as something like build.zig should be outside of scope, because odin doesn't aim to compile C, only to interop with it.

That being said, there are a few things that we can add to further help with configuring and building C dependencies

• Add small wrappers around common build systems. Allow setting up some common cmake, make, premake5 flags in a "typed way": so similar to how we do build.odin, we can do build.cmake or build.premake5. This has the same problem of "supporting 3rd party in 1st party", but we can get philosophical about how every compiler is technically a 3rd party thing.

Problems of a standard build system

These are some things that I didn't really discuss with Bill, but popped up in my brain as some significant problems

1. It's not entirely obvious to me how having a build system as part of the core will interact with a "multiple compilers ecosystem". Right now the flags passed to build.odin are modelled around the official odin compiler, but what if My Custom Compiler ™️ doesn't have -build-mode:dll, or it's got it under a different name, or has a new -brand-new-flag?
   This is a problem that needs to be figured both for the core standard, but also for the language specification and compiler specification as a whole. Otherwise we either risk having an incompatible-across-different-compilers core collection, or a core:build implementation that rivals the complexity of cmake, trying to generically handle every single thing under the sun.
2. Project structure: this is also related to the compiler support. As it stands right now, the feature of core:build that allows you to import build systems into eachother enforces a project structure (the build exe needs to be made right below the build package folder). This is a great big hack from me, but i also think it shows a tiny problem with the flexibility of odin build . and how people structure their work. Some people put everything in the root and call odin build ., some have a src subfolder and call odin build src, some people do the same but jump in the src folder and call odin build .. Integrating core:build with the compiler might enforce a project structure, and i'm not sure if this is a good thing or not.

These are just some of the problems that come to mind right now, but regardless they are both closely related to the fact that there probably needs to be a well thought out odin language/compiler specification laid out before proceeding with making an official build system as a core package.

Until then this is best left as a draft for sharing/experimenting ideas and discussion. It probably won't be opened again, but i would say that it's best to not close it since it already has a bunch of stuff around it and can be further used as a discussion ground