The definition of self is inconsistent in flakes

[aanderse]

Describe the bug

When flake.nix is not located at the root of a git repository I have found a few scenarios in which the definition of self is inconsistent. Sometimes self refers to the entire repository, sometimes self refers to the directory that flake.nix resides in (and therefore excludes the rest of the repository code).

Steps To Reproduce

1. Clone this simple repository and briefly examine it.
2. Follow the README.md

$ cd subdir/
$ nix repl                                                                                                                                                                                            
Welcome to Nix 2.8.1. Type :? for help.

nix-repl> let flake = builtins.getFlake (toString ./.); in builtins.readFile ("${flake.self-as-an-output}/test.txt")
copying '/home/aaron/flakes-test/subdir'"this is a text file at the deeper inside of my git repository\n"

nix-repl> :lf .#                                                                                                     
Added 12 variables.

nix-repl> builtins.readFile ("${outputs.self-as-an-output}/test.txt")                                                
"this is a text file at the root of my git repository\n"

Why do i get different results? 🤔

Expected behavior

I would expect that self is consistently referring to the same path within my repository and includes all the same code, regardless of how I reference self.

nix-env --version output

Nix 2.8.1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/self-input-for-flakes-in-sub-directories/22774/4

[greedy]

I think I found the bug. It's a subtle operator precedence issue that causes the subdir path within the git repository to remain empty. Fix incoming soon.

[greedy]

Whelp, that didn't do it. Needs more investigating.

I thought it was this line:

subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);

and that operator precedence makes the key part interpreted as subdir.empty() ? "" : ("/" + subdir) when what's wanted is (subdir.empty() ? "" : "/") + subdir. This isn't the case though (or rather it doesn't matter).

[aanderse]

There may be inconsistencies in more places. The example I provided shows one case, but I'm sure I have had inconsistent behavior in other examples.

To be sure we're both on the same page: self is intended to point to the root of the repository, right? This is how I have been using self in a project that has flake.nix files in multiple subdirectories.

[greedy]

Hmm, I was at first misunderstanding the situation. Here's what I understand now.

builtins.getFlake (toString ./.) has to be just subdir because of the semantics of toString ./.. Evaluating toString ./. copies the directory into the store and returns the store path.

I'm actually struggling to find a good reference for what the string evaluation of a flake should be. In the current implementation, it comes from fetchTree on the git repository so that is why it gives the root of the git repository.

To me it seems more consistent that stringification of self should always be the directory with the flake.nix for self. It seems awkward for things like "${self}/some-path" to work differently depending on whether self is in a subdir of some root flake.

[roberth]

This sure is a bug, because the manual says:

The special input named self refers to the outputs and source tree of this flake.

It also contradicts the idea that the flake should produce the same result regardless of where it is placed.

[roberth]

When triaging the bugfix #7796 in the meeting today, we decided that more discussion is needed.

the flake should produce the same result regardless of where it is placed.

It seems that there's some disagreement about what constitutes a flake. It was mentioned that a subflake in a git repository is allowed to use ../.. This seems to be half of a feature that allows a subflake to use its parent flake, offering no (good) means to use the parent flake as a flake.

---

I think a solution to this subflake-or-not conundrum is to require the user to be more explicit. After all a subflake isn't really a thing, but rather an interaction between two flakes, or, as we see in the current behavior, an interaction between a flake and a repository.

What if we could turn subflake from an ontologically complicated thing into a property that's actually part of the flake? By being explicit about subflake-ness we set the right expectations. An attribute in flake.nix could achieve that.

However, I don't think we need a new mechanism. Instead, we can reuse the existing dependency mechanism to represent subflake-ness. A dependency would be sufficient.

{
  input.parent.url = "..";

  outputs = { self, parent, ... }:
    # ...
    ;
}

Without introducing any new concepts, we've made the relation between the subflake and its parent explicit, while also setting expectations about what this flake can and can't do.

In this context, what should the outPath of self and parent refer to? Arguably not the same path.

It also simplifies the rules for hermeticity. That's all about the question: which files are an input to my evaluation? With this change it should be just: the directory of the flake + the files that make up the evaluation of the inputs; no need to make an exception for flakes that exist in a git repository.

[RuRo]

@roberth in my opinion, the actual problem here is that Flakes actually provide 2 different functionalities (compared to "old" nix files):

1. "Grouping". Flakes provide a unified interface for organizing nix code and as a consequence - the ability to easily compose smaller pieces of nix (instead of requiring a huge monolith like nixpkgs).

2. "Reproducibility". Which includes dependency locking, forcing isolation during builds, disabling impure evaluation etc.

These are both great things, but unfortunately Flakes were designed from the ground up with those 2 features tied at the hip. And now this decision is biting us in the ***, because there is no way to get "Grouping" without "Reproducibility" (or vice-versa).

A "subflake" should be a flake in terms of "Grouping", but not in terms of "Reproducibility". It doesn't need it's own reproducibility, because the reproducibility is already provided by the parent flake.

The fact that Nix doesn't have any way to organize code without forcing the "Reproducibility" mechanics is (in my opinion) the root cause of a lot of issues with flakes (i.e. #3978, #5810, #5570, #5663, etc).

[roberth]

Click for off topic part of reply to @RuRo

I agree, but that's not the direction that was taken. "Grouping" is little more than a standard schema that could have been adopted without using flakes, but that's not what people did, because they wanted the "whole package", including the lock file and the CLI. There's plenty to be said about past decisions, or the architecture for that matter, but let's keep this discussion centered around "The definition of self is inconsistent in flakes" - something like the flakes we have. We already have issues and two RFCs for discussing the splitting of features.

---

A "subflake" should be a flake in terms of "Grouping", but not in terms of "Reproducibility". It doesn't need it's own reproducibility, because the reproducibility is already provided by the parent flake.

Sounds like something we can do with follows and my earlier suggestion. The parent could declare inputs foo and bar as usual, and then the subflake could look like this:

# child/flake.nix
{
  inputs.parent.url = "..";
  inputs.foo.follows = "parent/foo";
  inputs.bar.follows = "parent/bar";

  outputs = { self, parent, foo, bar, ... }:
    # ...
    ;
}

The need (yes/no) for a lock file can be inferred from the inputs. parent is a relative path input, so it doesn't need locking. foo and bar are follows, so they don't need their own lock file either. This flake has zero lockable inputs, so it does not need a lockfile.

Do you know of any other use cases or requirements that I didn't think of? I think this one is covered.

[RuRo]

let's keep this discussion centered around "The definition of self is inconsistent in flakes" - something like the flakes we have. We already have issues and two RFCs for discussing the splitting of features.

Of course, sorry, can you link me to the RFCs you are talking about?

Sounds like something we can do with follows and my earlier suggestion. The parent could declare inputs foo and bar as usual, and then the subflake could look like this:

# child/flake.nix
{
  inputs.parent.url = "..";
  inputs.foo.follows = "parent/foo";
  inputs.bar.follows = "parent/bar";

  outputs = { self, parent, foo, bar, ... }:
    # ...
    ;
}

That's a bit verbose. I vaguely recall being able to do something like parent.inputs.foo. That way, you could avoid manually following all the inputs. But I am not 100% sure, this might be a repl feature.

This flake has zero lockable inputs, so it does not need a lockfile.

Makes sense.

[roberth]

RFCs for discussing the splitting of features

• [RFC 0136] A plan to stabilize the new CLI and Flakes incrementally rfcs#136

• [RFC 0134] Carve out a store-only Nix rfcs#134

[aanderse]

Thank you so much to everyone!