TODO

TODO
====

Tracking External Dependencies
------------------------------
Consider caching the size, location and timestamp of
any commands which are run via exec.

If any of these change, dependent commands need to be re-run.
This can be done in 'proc run', but means that we need to be able to interpret
that PATH and determine if something is really executable.

This is now supported via CheckExternalCommands on|off.
The result is stored in tmakecache under the 'exec' key.
However it isn't perfect.

- An external command may simply be a front end to another command that has actually changed.
  (e.g. ccache which calls the actual compiler later in the path)
- It only covers commands run via 'proc run'

Could allow some customisation over the method. e.g. could hash the file rather than
just look at time/size. Could use the output of a command (cc --version) instead.

Local vs non-local
------------------
It is hard to chain (e.g.) Objects, Link, Executable if the input is local but the result is non-local.

Also, Depends and Phony currently use non-local specifications for -depends and -inputs.
Is there some way to allow local names to be easily used?

I think that all names should be considered local, unless explicitly qualified as /global.
Needs some rethinking of how 'target' works'

I started implementing support for -local/-global that would affect how the following
-inputs or -depends are treated (it's in a stash). This didn't help a lot because:

- often it is the target that needs to be localised and target -local didn't quite fit
  (but perhaps needs to be made to do so).
- often it is useful to find the actual path.

Need to revisit this

Tracking Variables
------------------
It is useful to be able to determine where variables are set.
This is supported with --showvars=full to show the location
that each variables is defined.

Doesn't address bound variables in rules.

Cleaning files from the source tree
-----------------------------------

Sometimes rules will create files in the source tree.
e.g. a ctags rule

There is no easy way to clean those files

We can now do Clean/DistClean --source

Integrating foreign projects
----------------------------
autotools-based, make-based

We can just add a call to make or something similar,
but what if the external project builds libraries, etc.
which we need? We kind of want a "prebuild" dependency
which will be built before any of the given targets.
But what if we are just building clean or some local target?
What is the scope of this external dependency?

We have ExternalBuild so far

Combining tmake-based projects
------------------------------
Adding a tmake project as a subproject

automake
--------
Explain how to do automake-like things.
Also migration from automake

Installation
------------
It is useful to be able to install different classes of things.
install-docs, install-bin, install-dev, install-runtime, etc.

Installation as root
--------------------
A common sequence is:

./configure
make
sudo make install

One of the gotchas here is that if the 'make install' step performs any build operations, those
files will be created as root.

One file that will be updated as a result of make install is .makecache, but because that is
an existing file that is rewritten, it won't change owner. Should we add something to prevent
non-install rules running as root?

Yes. We now have -rootok that can be used on install rules.
We also have INSTALLDEPTARGET in rulebase.default to ensure that install deps can be built
before installing.

This isn't documented anywhere.

Propagating Defines
-------------------
While all defined variables propagate from project.spec to subdirs,
no variables defined in build.spec do so.
It would be nice to be able to propagate selected defines to subdirs.

Perhaps (although I don't like the name):

  define-propagate name1 name2 ...

In process-build-spec-file, the value of these variables
need to be captured at the end of 'Scope'. Then they can be set
for each subdir 'Scope'

Alternatively, recognise that non-local settings need to be set in project.spec
Either as a define or a proc. e.g.

  Option 1
  --------

  -- project.spec --
  define ENABLE_ABC
  -- build.spec --
  ifconfig ENABLE_ABC {
	CFlags -Dtesting=123
  }

  Option 2
  --------

  -- project.spec --
  define ABC_SPECIAL_CFLAGS -Dtesting=123
  -- build.spec --
  CFlags $ABC_SPECIAL_CFLAGS

  Option 3
  --------

  -- project.spec --
  proc EnableAbc {} {
	CFlags -Dtesting=123
  }
  -- build.spec --
  EnableAbc

Inputs changing during long compiles
------------------------------------
Previously, when using time-based comparisons, tmake would only create a virtual
timestamp for a target after the target was built. This suffered from a number of problems:

* It assumes that [file mtime] can be compared directly against [clock seconds]

* If source files change between when the 'do' started, and when it finished,
  this information is lost.

* This situation is complicated with high resolution timestamps as some tools write
  files with 1-second resolution timestamps and thus built targets can have
  timestamps before the inputs.

Note that hash-based builds don't suffer from this problem as the hash is always
compared against the previously stored hash. No ordering is required.
Therefore the solution is to use the same approach for time-based builds and
consider the mtime as a proxy hash of the source file. If the hash/mtime for any dependency
changes from the previous build, the target is out of date and needs to be rebuilt.
This simplifies the logic as the same algorithm is used in both time-based and
hash-based builds, simply with a different "hash" calculation.

This is done and simply needs documenting.

Cross Compiling vs Host Builds
------------------------------

In the future a "context"-based approach could be used. The idea is that each separate context
has it's own:

- set of define variables
- targets
- build directory tree

Consider the following (mythical) build fragment:

  Context default {
    define CPFLAGS -f
    target a -inputs b -do {
        run cp $CPFLAGS $inputs $target
    } -getvars CPFLAGS
  }

  Context host {
    define CPFLAGS -p
    target a -inputs b -do {
        run cp $CPFLAGS $inputs $target
    } -getvars CPFLAGS
  }

This keeps two distinct rules. If we use #xxx#name to represent 'name' in context 'xxx', we have:

  a: b
    var CPFLAGS=-f
      run cp $CPFLAGS $inputs $target

  #host#a: #host#b
    var CPFLAGS=-p
      run cp $CPFLAGS $inputs $target


The 'default' context omits the #context# prefix for simplicity, but can be name explicitly if required.
Sources and targets with different contexts can be used by using an explicit prefix.

All this means it is possible to do:

  Context host {
    define CC $CC_FOR_BUILD
    Executable --publish generator generator.c
  }

  Generate out <bin>#host#generator in {
    run $script $inputs $target
  }

Anything declared outside 'Context' uses the 'default' context.
There would need to be convention for mapping context to output directory.
If we simply use objdir/<context>/... there may be a naming conflict.
Could use object.<context> or could use objdir/default/ for the default
context instead of just objdir/

How does Load work? For example, $CC_FOR_BUILD may be loaded only in the default
context. Do we need to use ${#default#CC_FOR_BUILD} ?

Virtual targets
---------------

Currently we support phony targets that are always out of date and normal targets
were the target rule must generate the targets. But we may also want to have support
for virtual targets, which are like a phony target except when the rule runs we record
the dependencies and only rerun if the dependencies change even though there is no
output produced.

tmake TODO
----------
* Does it make sense to support "IncludePaths /abc" to add paths
  relative to TOPDIR? IncludePaths should never really refer to
  dirs outside the tree, so it should be OK.
* It is hard to do 'tmake clean' if settings.conf
  is required but hasn't been created yet.
  We don't want to required settings.conf to exist
  in order to clean, but things like "UseSystemLibs $LIBS" will fail
  if LIBS isn't defined.
  Can we require the use of something like: "UseSystemLibs [var LIBS]"
  when vars are used outside of rule commands? This would solve several
  problems at the cost of being a bit ugly.
  Ideally we could have a trace callback on access to an undefined variable.
  But Tcl can't do this anyway :-(
* Properly manage/document the precedence of vars:
  - command line => make these as "fixed"
  - environment => these need to be made fixed with 'import'
  - Load settings.conf => use define to override everything which is not "fixed"
  - Defaults in rulebase.* => use define? as a last resort
* I think I plan to set all variables and execute rules in a separate interpreter
  - But what about parsing of rules? Currently we rely on having access to variables
    in build.spec and rulebase.
* Alternatively, do it all in a namespace
* Consider "real" getopt processing for the tmake command line
  - actually, option handling has improved greatly
* Load should probably load values into a ::tmake(config) dictionary and
  ifconfig should look there
  - Perhaps it is possible to also import certain of these settings (by glob)
    as global variables
* archive libraries uses a pretty simple format.
  we could parse these libraries for timestamps directly in Tcl if required
* We have --warnings, but does it make sense to also capture errors and avoid
  rebuilding if nothing has changes, and just outputting the previously captured errors?
  I'm inclined not to since ccache already does a good job of this and I don't
  think it's big use case to keep recompiling with no changes.
* meson does clever things with examining the ABI of a shared library to determine
  if it needs to be relinked (i.e. an application or shared library needs to be relinked
  against it). We would do this by using a special hash generator that hashed the ABI
  rather than the contents. Would need to include some other info too such as the architecture,
  etc.
  There is some reference to this here: https://groups.google.com/forum/#!msg/bazel-discuss/8Wt3l9zfksY/E0HJNnWECwAJ
  Basically if we had an ifso tool that created an interface shared library from a shared library,
  we could link against the shared that instead of the real shared library. This would still create
  the correct NEEDED entry, but if we use hashing for the ifso library we would detect that it hadn't
  changed since last time so we don't need to relink the dependencies.
  This does seem to be what clank-ifso is supposed to do (see here: http://lists.llvm.org/pipermail/cfe-dev/2019-April/061953.html)
  There is also reference to llvm-elfabi that seems to do the same thing
  Why can't we produce a shared library with objcopy, just dropping the unused text and data sections?
  Something like: objcopy -g --remove-section=.text --remove-section=.data  libfdcallback.so.1.2.3 libfdcallback.so
  I tried this in vbox:tmake/test1/objdir/fdcallback and as far as I can tell it works.
  I will consider testing this with a patch to tmake to change publish-shared-lib.
  Will need to split into two separate rules, one which creates the symlinks, and one which
  generates the linkable so (e.g. libfdcallback.so.1.2.3 -> libfdcallback.so)
  - actually, it doesn't work. If the implmentation changes, it still changes the ifsl a bit
    since it includes the size of each function. Even though the linker doesn't strictly need
	this information. I think I would instead need to build something smarter than objcopy which can
	zero out these unneeded sizes, or else use a custom hash calculator for this target that looks
	only at the parts I want to look at.
  - I'll look at how meson implements this.
	- It has a command called 'symbolextractor'
	- On linux, this does: readelf -d $libfile | grep SONAME
	- Then: nm --dynamic --extern-only --defined-only --format=posix $libname | remove column 3 (size of text symbol)
  - Need to think about how this would work for tmake. If we supported custom hashing I could run a command to extract this
    info and then md5 the result.
	- I created shlib-hash on mac and Linux
	- Currently doesn't support cross building.
	- Need to work out how to use it and add it as a custom hasher

Known Issues
------------
Slows down with a large project

No support for non-unix platforms, e.g. msvc

Changing --build means that all orphans are forgotten since
the cached targets only include the path relative to $BUILDDIR.

The integration with autosetup works, but could be more seamless