[RFC 0193] TOML Flakes #193
Conversation
39e5d72 to
bba0b38
Compare
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/outlining-the-differences-between-flakes-and-nix-configs/72996/8 |
|
Couldn't this be achieved by doing something like the following? {
inherit (builtins.fromTOML (builtins.readFile ./flake.toml)) description inputs;
outputs = ...;
} |
|
@limwa Nix restricts flake |
Something I missed while editing; some very important context.
| - Potential confusion during transition period with two ways to specify inputs | ||
| - Requires tooling updates across the ecosystem | ||
|
|
||
| # Alternatives |
There was a problem hiding this comment.
One alternative worth considering: use Dhall instead of TOML.
Many of the complaints about the flake.nix ‘language’ seem to be rooted in the desire to abstract out parts of the inputs for reuse. TOML doesn't actually address this desire; it merely makes it clearer to the user that the desire can't be satisfied. The motivation to restrict how inputs are expressed seems to be to prevent arbitrary code from running, either for security reasons or so that flake input analysis can be guaranteed to terminate. Dhall addresses both concerns, while allowing many of the abstractions flake users are trying to write; it's pretty much made for exactly this use case.
There was a problem hiding this comment.
That direction is more in line with the discussion that followed in NixOS/nix#4945.
Dhall is still somewhat impractical, unless perhaps the Rust implementation could be wrapped such that Nix can use it as a library.
I believe its effect on the learning curve is also negative.
The main use case of programmatic inputs seem to be auto-follows, which could just be a feature, or getting inputs from some shared expression.
I personally wouldn't mind allowing evaluation in inputs. If someone wants to make them complicated, non-terminating, whatever, that's on them.
Regarding Dhall as a point in the configuration language design space, I might upset some folks, including my past self who was a big fan of that kind of tech before using it in conjunction with Nix.
While I appreciate the science of it, I find Turing incompleteness to be a bit of a gimmick and an unnecessary constraint for this kind of application. It has a real cost in terms of what kinds of programs can be expressed with it, while gaining very little practical value: either you get an error, or you don't.
The tradeoff is between getting 1 success and 1 mediocre error from a language like Nix vs 1 unnecessary impossibility and 1 nicer error from Dhall.
Or we could just keep it boring and use TOML.
I guess we could add a helper trampoline primop to Dhall to add some Turing completeness for the algorithmically inclined?
I'll boil this rant down into a more reasonable addition later, or I'd be happy to take a GitHub suggestion.
There was a problem hiding this comment.
Yaml could also take care of a lot of the desires here, without being fully turing complete, and without being as obscure. It has a variety of variable-like features.
That said, I think simplifying the format as much as possible, and indicating that it is deliberately limited is part of the appeal of this proposal. Making inputs templateable just leads to complexity that ends up confusing newcomers to the ecosystem.
I think most use cases for evaluation in inputs that don't just become features outright could be handled in external scripts instead, given the file would be much more programmatically editable.
There was a problem hiding this comment.
I'm not myself convinced that Dhall would be the best choice, but a lot of these objections strike me as not very strong.
Dhall is still somewhat impractical, unless perhaps the Rust implementation could be wrapped such that Nix can use it as a library.
Even if there is no convenient binding, Nix could execute dhall-to-json --file flake.dhall and use the output; I'd be shocked if the overhead of a child process when interpreting flake input files would have any human-noticeable impact on perf.
I believe its effect on the learning curve is also negative.
Simple flake.dhall file:
{
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable",
entrypoint = {
url = "github:hercules-ci/flake-parts",
inputs.nixpkgs-lib.follows = "nixpkgs",
},
}I don't think this would present more of a learning challenge to people not trying to dive into the intricacies of Dhall than the equivalent TOML would.
In other words: the strength of the argument that people don't need abstractions in their inputs is inversely proportional to the strength of the argument that it would be difficult to read and write inputs written in Dhall. If many people want it, all the more reason to use a non-data-only language. If few people want it, we won't see ‘advanced’ Dhall features (scare quotes because, you know, I'm talking about things like let bindings) very often in the wild (and users can make up their own mind whether having them is worth learning about them).
And it's worth emphasizing that this is not a hypothetical ‘what if people want it’. We know users want it. Here's this example as it might have been written in Dhall:
let dep = \(url : Text) -> { url = url, inputs.nixpkgs.follows = "nixpkgs" }
in {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable",
entrypoint = {
url = "github:hercules-ci/flake-parts",
inputs.nixpkgs-lib.follows = "nixpkgs",
},
nix-darwin = dep "github:LnL7/nix-darwin",
home-manager = dep "github:nix-community/home-manager",
zig = dep "github:Cloudef/nix-zig-stdenv",
zls = dep "github:zigtools/zls",
hyprland = dep "github:hyprwm/Hyprland",
eww = dep "github:elkowar/eww",
}This strikes me as something that said user would have been able to learn how to write, given what they wrote in Nix.
It has a real cost in terms of what kinds of programs can be expressed with it
When compared to TOML as the alternative, this isn't any sort of disadvantage; the programs TOML is able to express are a strict subset of the programs Dhall is able to express, obviously.
The tradeoff is between getting 1 success and 1 mediocre error from a language like Nix vs 1 unnecessary impossibility and 1 nicer error from Dhall.
Or we could just keep it boring and use TOML.
Using (unrestricted) Nix would be keeping it boring. In the same framework, the tradeoff is between TOML's 2 unnecessary impossibilities (not-provably-total programs, and simple deduplication of the sort we actually see people trying to write) and 1 nicer error, Dhall's 1 unnecessary impossibility (NPT programs) and 1 nicer error, or Nix's 0 unnecessary impossibilities and 1 mediocre error. On just these axes, TOML is not on the efficient frontier.
There was a problem hiding this comment.
Nix could execute
dhall-to-json --file flake.dhall
Performance is probably acceptable, but the static Nix binary would have to pull that in from somewhere for basic functionality.
simplifying the format as much as possible, and indicating that it is deliberately limited is part of the appeal of this proposal.
This. It brings clarity and better scripting.
The comparison in this thread was between Dhall and full-Nix, but the RFC does not propose either.
There was a problem hiding this comment.
TOML has been chosen by projects like Rust and Python for the exact reason that it is necessarily simple. It doesn't let you get too fancy. YAML was ruled out because it allows too much flexibility. YAML is also bad at making things clear as complexity increases. Anyone complaining about Nix as an obscure language will not find Dhall an improvement. Anyone who needs the complexity should use Nix. Anyone who doesn't need it can use TOML.
There was a problem hiding this comment.
@de11n toml is not that simple though, especially with the table vs array syntax - e.g. this is quite atrocious: https://toml.io/en/v1.0.0#array-of-tables
Let's just stick to Nix.
rfcs/0193-toml-flakes.md
Outdated
| } | ||
| ``` | ||
|
|
||
| **flake.lock** remains unchanged. |
There was a problem hiding this comment.
I guess you implicitly add locking the entrypoint flake?
There was a problem hiding this comment.
It is a regular input, so it is locked like any other input.
In an earlier iteration of this RFC I had it as a separate lock entry entrypoint.url that wasn't part of inputs, but that would just complicate the locking mechanism.
There was a problem hiding this comment.
Oh right, I see that in some place you say that entrypoint is used if specified but elsewhere it is inputs.entrypoint. (Maybe it is better to make all mentions fully qualified?)
There was a problem hiding this comment.
The possibility of specifying a different entrypoint than "entrypoint" in flake.toml's entrypoint field would make that inaccurate. Clearly this needs further simplification. Let me strip that indirection out.
rfcs/0193-toml-flakes.md
Outdated
| ## Keep current structure | ||
|
|
||
| Maintain the status quo including drawbacks described in the [motivation]. | ||
|
|
||
| ## Pure Nix with better tooling | ||
|
|
||
| Improve Nix AST manipulation tools instead of introducing a new format. | ||
| Does not solve user uncertainty. | ||
| Does not solve boilerplate. |
There was a problem hiding this comment.
I'll point out that the main complaints about this somewhat go away with taking "pure nix" tooling and also taking the admission that both user uncertainty and boilerplate in 2025 are concerns that are rapidly declining.
Bluntly: any of the major LLMs right now are perfectly fine at making decent flake.nix files, and where they fall short a different configuration language or helper probably wouldn't be much of an improvement.
There was a problem hiding this comment.
This was in reference to the outputs function more so than the inputs.
Boilerplate wasn't much of a problem even before LLMs.
Realistically how does reading 20-40% extra code which is visually similar stack up against having to evaluate code before being able to see the outcome?
Explicit is good.
rfcs/0193-toml-flakes.md
Outdated
|
|
||
| Flake output invocation is changed to go through an "entrypoint", which is the flake that provides a function to help produce the flake output attributes based on the source tree of `flake.toml`. | ||
| If `flake.toml` has an `entrypoint` field, it must name one of the `inputs` which will serve as the entrypoint. | ||
| Otherwise, if an input named `"entrypoint"` exists, it becomes the entrypoint. |
There was a problem hiding this comment.
I can imagine use cases for declaring multiple entrypoints that define different sets of outputs.
If multiple are defined, nix could basically just recursively // the outputs of the function returns; collisions should be uncommon if the ecosystem evolves around their possibility, and most use cases for this would probably be to provide two different outputs which would be completely orthogonal anyway.
E.g., having the option of specifying multiple entrypoints would make writing a flake that has both nixosConfigurations and homeManagerConfigurations possible, without the boilerplate of maintaining a specialized entrypoint flake for this use case.
There was a problem hiding this comment.
A simple flake could do that too. An entrypoint flake is just a flake with a specific function in it; anyone can call them. So you could have a flake that performs this merging task.
I'd argue that flake-parts already performs this task well, providing a decent framework for interaction between modules, as opposed to entrypoints which would typically assume ownership of the outputs, and collide at the various default attributes.
There was a problem hiding this comment.
A simple flake could do that too. ..., as opposed to entrypoints which would typically assume ownership of the outputs, ....
Actually, this begs the question if the special inputs.entrypoint magic should even be part of this proposal. It seems to me that everything that inputs.entrypoint does can already be performed by essentially doing outputs = inputs: inputs.entrypoint.makeFlake inputs ./. ...;.
Based on the title of this RFC, it feels like inputs.entrypoint might be out of scope? Maybe extra convenience features that extend the current functionality of flakes like inputs.entrypoint should be considered in separate RFCs (possibly together with stuff like NixOS/nix#5663, NixOS/nix#3843, NixOS/nix#7422 (comment), NixOS/nix#5157, NixOS/nix#5567)?
There was a problem hiding this comment.
Scope could be reduced in that way. I like the bigger picture of the combined changes, but for clarity that seems like a good idea actually.
There was a problem hiding this comment.
|
|
||
| 1. **User uncertainty**: The current structure creates confusion about what language features are available | ||
| and when evaluation restrictions apply, | ||
| because inputs and outputs are mixed in the same file, and inputs are not allowed to be arbitrary Nix expressions. |
There was a problem hiding this comment.
I feel like this RFC should not just assume that the currently enforced status-quo limitation of "Flake input metadata can't contain arbitrarily complex, possibly non-terminating computations" must be kept as is, without justification.
I'm not necessarily saying that we should remove this restriction, but the RFC should definitely explore the pros / cons of this requirement more.
For example, do we know if this restriction is actually useful / needed? I think, I read somewhere that FlakeHub is relying on the fact that flake metadata can be computed statically-ish (I've never used FlakeHub, so I am not sure if this is actually the case). Are there any more examples where this property is actually useful?
If FlakeHub is the only (or one of the few) users of this restriction / guarantee, then maybe it should be implemented as a simple evaluation "depth" limiter on the side of FlakeHub, instead of in Flakes themselves?
Or maybe it's enough to have a simple way to opt-out of this restriction? Something like
{
dynamic = true;
inputs = {
# Arbitrary nix expressions are now allowed inside inputs,
# because `dynamic` was set to true. This flake might not have
# full functionality on services like FlakeHub, but this is fine.
};
}| ## Proposed structure, simple example | ||
|
|
||
| **flake.toml**: | ||
| ```toml | ||
| description = "A simple example flake exporting GNU hello for x86_64-linux" | ||
|
|
||
| [inputs.nixpkgs] | ||
| url = "github:NixOS/nixpkgs/nixos-unstable" | ||
| ``` | ||
|
|
||
| **flake.nix**: | ||
| ```nix | ||
| { | ||
| outputs = { self, nixpkgs }: | ||
| let | ||
| system = "x86_64-linux"; | ||
| pkgs = nixpkgs.legacyPackages.${system}; | ||
| in { | ||
| packages.${system}.default = pkgs.hello; | ||
| }; | ||
| } | ||
| ``` |
There was a problem hiding this comment.
The choice of "reusing" / keeping flake.nix for the outputs part of the flake has the potential downside that in older versions of nix and other frameworks, this will look either like a malformed flake (that uses some inputs in its outputs, but doesn't declare them as inputs) or in this particular case the nixpkgs input would fall back to the global flake registry.
Maybe the split should be flake.nix -> flake.toml + outputs.nix instead of flake.toml + flake.nix?
|
I like idea, that entrypoint could be used from imported entity (like in |
|
If this is accepted, it would be nice to see a section for tool-specific metadata. unflake could use it to store its config and, generally, it would provide an extension point for userspace flake-related utilities. |
|
I've moved the entrypoint bits to #194 |
|
What would be the expected usage if one were to not declare dependencies? I imagine not placing a TOML file could instead notify the user it could not find one. For reference, flakes' |
|
@KiaraGrouwstra Simple answer is "of course" you'd put a I viewed
|
|
(for what it's worth, i believe the description field to be optional as well - maybe just |
| ## Relationship to existing files | ||
|
|
||
| - `flake.lock` remains unchanged |
There was a problem hiding this comment.
Since flake.toml defines just the inputs and the lockfile is also just metadata about inputs being pinned, could we merge flake.lock with flake.toml? Is this too radical of a change? I personally would like to not maintain 3 files for flakes;
- one for inputs
flake.toml, - one for outputs
flake.nix/outputs.nix, and - one for pinning the inputs
flake.lock.
There was a problem hiding this comment.
I also think that using three separate files is a bit excessive for flakes. And since flakes are still experimental, supporting users who prefer non-experimental workflows means you’ll need to provide a shell.nix if your development setup relies on flakes, as well as a default.nix if you export packages so they remain compatible with import clauses.
Regarding the idea of merging flake.toml and flake.lock, I don’t believe that would be advisable. For example, template repositories (such as those in NixOS/templates) depend on listing flake.lock in their .gitignore file to ensure it’s not committed to git. If flake.toml and flake.lock were combined, git wouldn’t offer a way to ignore only the lock changes made to that file. As a result, updating a template would require manual review to make sure no lock information is inadvertently added to flake.toml.
This is especially important because Nix automatically generates flake.lock (and, in turn, would update flake.toml once that file is introduced). That behavior could easily lead to unintended changes being committed without the developer noticing.
There was a problem hiding this comment.
Correct me if I'm wrong, but aren't flake.lock files usually automatically generated and left untouched by the user until an upgrade command is run?
Aside from tooling simplicity and reliability, for the use case of creating templates, wouldn't it make omitting the usually unnecessary version locks much more convenient for template creators?
As of now, you can keep version locks out of your repository with a simple .gitignore entry and keep iterating on the template directly with no additional work.
With the merge of flake.toml and flake.lock files, the process would not be as simple, requiring things like omitting generated changes from commits, using special command arguments, or just biting the bullet.
I'm not saying that it'd be a big burden with either case, but keeping the files separate just seems like the cleaner and easier to manage solution.
There was a problem hiding this comment.
@limwa Oh sorry, I didn't see your comment before starting to write mine! But I guess I'm not alone on this, since we wrote up the exact same example.
There was a problem hiding this comment.
flake.lock would no longer be a throw-away file, and the editing experience would get a bit worse.
We'd have to ensure that the most relevant bits stay at the top.
JSON doesn't support comments (or automation support may regress if you do allow it)
GitHub hides lockfile changes, and even if that's changed, you'd have to open very large diffs even if you trust the committer.
I lean towards preserving full flake.nix support without warnings, so that might be a good solution for you.
Also, fwiw, if you're using a framework, then combined with #194, you may not need a flake.nix or outputs.nix anymore.
| - Requires tooling updates across the ecosystem | ||
|
|
||
| # Alternatives | ||
| [alternatives]: #alternatives |
There was a problem hiding this comment.
One more alternative is to explicitly define "data only" subset of nix which guarantees O(input length) eval times, and introduce special name&extension&file for it.
C.f build.zig and build.zig.zon (Zig object notation) in Zig, or, for that matter, index.js and package.json.
We could have flake.nix and flake.nix.nox (Nix Object eXpression).
This of course presupposes that we do not want to "program" flake.nox.
Given that nix can be on the bootstrap path, it seems relatively valuable to stick to what we already have, rather than introduce extra dependencies on other formats.
EDIT: we could also think about using nox instead of aterms in derivations?
There was a problem hiding this comment.
Maybe it's because I have a simple mind, but I have great difficulty with nox if we could call it Nixon.
Anyway, on a more serious note, but still indulging myself, those would be highly similar formats.
I'd say step 1: allow Nix identifiers as object keys, trailing comma, remove decimal numbers => Nixon.
Step 2: rebase the Nix language on top of that, removing floats and with, maybe some other things => Jix
Material for some follow-up RFCs ;)
There was a problem hiding this comment.
Not sure if that's a follow-up RFC or a prerequisite. A simple format like this could be quite useful for many situations. flake.lock could be transitioned to a format like this, the profile manifest as well. It seems like a pretty useful idea to formalize this "data only subset" of nix.
It would also alleviate the concerns raised here: https://github.com/NixOS/rfcs/pull/193/files#r2603388837
|
|
||
| ## File structure | ||
|
|
||
| `flake.toml` becomes the leading file, |
There was a problem hiding this comment.
I think it's one of the greatest features of the nix ecosystem that everything is configured in a single language.
I doubt that adding more configuration languages makes nix more approachable, since people will still have to understand both languages.
There was a problem hiding this comment.
I do love that as well, but there is a big difference between "nix the turing-complete functional language" and "nix the language for flake inputs, flake config and flake metadata that only allows attrsets and literals".
Maybe there could be a separate file format for a subset of nix called "nix object notation" or "nix light", a language that has nix syntax but doesn't allow functions at all?
This is pretty similar in concept to zig's .zon format, which reuses zig's syntax for objects to create a json-like format.
It would just need a different file ending (though I'm not sure .nixon is a great idea :D). You could import files like that just like regular nix files, but for certain things (like the flake metadata in this RFC), only this subset of nix is allowed.
EDIT: Just noticed someone else brought that up already: https://github.com/NixOS/rfcs/pull/193/files#r2602798399
| It incorporated a "poor man's module system" into Nix where it would be of very limited functionality, | ||
| while it ossifies due the consequences in Nix of [Hyrum's law](https://www.hyrumslaw.com/) and the second order effects of reproducibility. | ||
|
|
||
| ## Flake entrypoints RFC |
There was a problem hiding this comment.
I still don't really understand why we can't have a more dynamic flake.nix entrypoint. This would make it trivial to implement something like flake.toml in nix code.
{
inputs = builtins.fromTOML (builtins.readFile ./inputs.toml);
outputs = import ./outputs.nix;
}edolstra mentioned that the flake format needs to be simpler to limit complexity while querying flake metadata, but couldn't that metadata be written to the flake.lock file? The locking process being complex shouldn't really be a problem.
There was a problem hiding this comment.
on simplifying outputs as mentioned here, a current limitation is it must be explicitly defined as a function, i.e. one cannot currently use beta reduction to delegate its parameters to the import function as you did here - that is to say, using outputs = args: import ./outputs.nix args; instead does work currently.
on inputs using a simplified language, while for reading purposes any of those ways could likely work, do note that at least one tool (flakehub cli) writes to the inputs section to update individual inputs. that usage, for one, may in fact depend on such limited complexity.
There was a problem hiding this comment.
Tools like flakehub could instruct their users to write flake inputs in specific ways. This doesn't seem like a reason to limit how people who don't use flakehub write their flakes.
There was a problem hiding this comment.
Let me wander a bit........ what happens here is that the inputs section is evaluated (1), then those values are used in another evaluation (2) with the outputs. How (1) works is different than how (2) works. We may be able to loosen the restrictions a bit on (1), but it will still need to be different in some way. Different restrictions, different access, etc. It isn't hard to remove the restrictions in Nix's source code, but while I see how it isbammoying sometimes, I dont see what large differences there would be in end-user outcomes. Perhaps you wnd up with some easier tooling to manage the inputs, but a jq-but-for-nix (eg: nix-editor) can do this already. Some enterprising individuals would embed much more complicated logic into the inputs section, create some abstractions. I've wanted something like this where an input could be just a plain nix value allowing for easier overriding or CLI options to be declared.
Putting "inputs" into toml or another format makes this distinction more clear, but doesn't change its underlying nature.
@KiaraGrouwstra how about we use this opportunity to add a The initial version of |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/ideas-for-a-data-only-subset-of-nix/73110/1 |
|
@iFreilicht flake authors that want to offer support for older versions of their flakes could do this with release branches. |
|
@truh that just puts the burden for support on the authors, though. It would be very helpful if Nix itself was smart enough to understand what version of the flake schema a certain flake is written in, and whether it is able to work with that or not. Right now the format of flakes is impossible to change because there's no control over backwards- or forwards-compatibility. Of course I understand that Nix supporting multiple versions of the flake schema at the same time is pretty much impossible because flakes are so deeply integrated into the evaluator. But even just getting a nice error message would be a big upgrade. Thankfully nix makes it quite trivial to temporarily install another version of nix. |
|
@iFreilicht Oh you mean versions of the flake format. I thought you meant semantic versioning for flakes like flakehub. We would have to be be careful to give it a name that makes it explicit that the version number refers to the flake format so that it doesn't confuse anyone. Honestly I find it hard to imagine benefits that breaking & non-interoperable changes to the flake format could bring that would outweigh the frustration of interoperability problems. Especially if stabilization remains such a difficult process. I have this concern about TOML flakes as well. It fragments the community, makes it more difficult for users to find information for their nix versions/approach and makes it more difficult for community members to offer support to people seeking help. |
As a mere community member I have a proposal for an improvement to flakes.
This can be done concurrently with the efforts to bring already developed flake related features up to scratch as part of stabilization.
Let's get out of analysis paralysis and move the project forward in a way that's positive, not just focused on tech debt and inclusive of community input.
Yes, I believe we need more flakes RFCs, precisely because they didn't fit into the original one.
Rendered