-
-
Notifications
You must be signed in to change notification settings - Fork 158
[RFC 0193] TOML Flakes #193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
bba0b38
c430f81
07a9e2e
c6d4776
1d46ee9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,253 @@ | ||
| --- | ||
| feature: toml-flakes | ||
| start-date: 2025-12-07 | ||
| author: Robert Hensing (@roberth) | ||
| co-authors: (find a buddy later to help out with the RFC) | ||
| shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||
| shepherd-leader: (name to be appointed by RFC steering committee) | ||
| related-issues: (will contain links to implementation PRs) | ||
| --- | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Make flakes easier to use, automate and learn. | ||
|
|
||
| Introduce `flake.toml` as the new leading file for flakes, | ||
| separating input specifications from output definitions. | ||
| The `flake.toml` file contains declarative metadata and input declarations, | ||
| while `flake.nix` focuses on output definitions. | ||
|
|
||
| Fixes [#4945 What language is flake.nix written in?](https://github.com/NixOS/nix/issues/4945) | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| Currently, flakes combine two distinct concerns in a single `flake.nix` file: | ||
| declaring dependencies (inputs) and implementing functionality (outputs). | ||
| This creates several problems: | ||
|
|
||
| 1. **User uncertainty**: The current structure creates confusion about what language features are available | ||
| and when evaluation restrictions apply, | ||
| because inputs and outputs are mixed in the same file, and inputs are not allowed to be arbitrary Nix expressions. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I feel like this RFC should not just assume that the currently enforced status-quo limitation of "Flake input metadata can't contain arbitrarily complex, possibly non-terminating computations" must be kept as is, without justification. I'm not necessarily saying that we should remove this restriction, but the RFC should definitely explore the pros / cons of this requirement more. For example, do we know if this restriction is actually useful / needed? I think, I read somewhere that FlakeHub is relying on the fact that flake metadata can be computed statically-ish (I've never used FlakeHub, so I am not sure if this is actually the case). Are there any more examples where this property is actually useful? If FlakeHub is the only (or one of the few) users of this restriction / guarantee, then maybe it should be implemented as a simple evaluation "depth" limiter on the side of FlakeHub, instead of in Flakes themselves? Or maybe it's enough to have a simple way to opt-out of this restriction? Something like {
dynamic = true;
inputs = {
# Arbitrary nix expressions are now allowed inside inputs,
# because `dynamic` was set to true. This flake might not have
# full functionality on services like FlakeHub, but this is fine.
};
} |
||
| See [issue #4945](https://github.com/NixOS/nix/issues/4945). | ||
| Note that while the question was originally asked in a partly rhetorical manner, | ||
| it is still a valid question, | ||
| a variation of which pops into new users' minds. | ||
| Some learning is always required, | ||
| but this is an unnecessary bump in the curve. | ||
|
|
||
| 2. **Limited automation**: Programmatically editing flake inputs requires Nix AST manipulation, | ||
| which is complex and error-prone compared to editing structured data formats. | ||
|
|
||
| By moving input specifications to a simpler format, | ||
| we enable better tooling, | ||
| reduce user confusion about evaluation restrictions, | ||
| and create a clearer separation of concerns between dependency declarations and output implementations. | ||
|
|
||
| # Detailed design | ||
| [design]: #detailed-design | ||
|
|
||
| ## File structure | ||
|
|
||
| `flake.toml` becomes the leading file, | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's one of the greatest features of the nix ecosystem that everything is configured in a single language. I doubt that adding more configuration languages makes nix more approachable, since people will still have to understand both languages.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I do love that as well, but there is a big difference between "nix the turing-complete functional language" and "nix the language for flake inputs, flake config and flake metadata that only allows attrsets and literals". Maybe there could be a separate file format for a subset of nix called "nix object notation" or "nix light", a language that has nix syntax but doesn't allow functions at all? This is pretty similar in concept to zig's It would just need a different file ending (though I'm not sure EDIT: Just noticed someone else brought that up already: https://github.com/NixOS/rfcs/pull/193/files#r2602798399 |
||
| containing input sources and follows relationships. | ||
| It complies with the [Nix JSON guideline](https://nix.dev/manual/nix/latest/development/json-guideline.html) (modulo `null`). | ||
|
|
||
| `flake.nix` remains and defines outputs. | ||
|
|
||
| ## Choice of TOML | ||
|
|
||
| See also the [alternatives] section. | ||
|
|
||
| A nice aspect of TOML is that its non-nested syntax aligns with part of a definition of _declarative systems_, having a set of _independent statements_. | ||
|
|
||
| It has a wide ecosystem of libraries for parsing, as well as good number of libraries that support round tripping edits. | ||
|
|
||
| ## Relationship to existing files | ||
|
|
||
| - `flake.lock` remains unchanged | ||
|
Comment on lines
+67
to
+69
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I also think that using three separate files is a bit excessive for flakes. And since flakes are still experimental, supporting users who prefer non-experimental workflows means you’ll need to provide a Regarding the idea of merging This is especially important because Nix automatically generates There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct me if I'm wrong, but aren't Aside from tooling simplicity and reliability, for the use case of creating templates, wouldn't it make omitting the usually unnecessary version locks much more convenient for template creators? I'm not saying that it'd be a big burden with either case, but keeping the files separate just seems like the cleaner and easier to manage solution. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @limwa Oh sorry, I didn't see your comment before starting to write mine! But I guess I'm not alone on this, since we wrote up the exact same example.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I lean towards preserving full Also, fwiw, if you're using a framework, then combined with #194, you may not need a |
||
| - `flake.nix` remains and defines outputs, and as a legacy format for the `inputs` and other metadata. | ||
|
|
||
| ## Compatibility | ||
|
|
||
| The design should support reading legacy `flake.nix` files that contain inline input specifications. | ||
|
|
||
| The following negative space is changed and may require a few projects to adapt if they already use these: | ||
| - A file named `flake.toml` will shadow `flake.nix` in file discovery | ||
|
|
||
| Flakes that do not have a `flake.toml` file remain compatible. | ||
|
|
||
| A TOML flake can not be evaluated by older implementations of Nix. | ||
|
|
||
| Other than those constraints, TOML and traditional flakes can be used and migrated back and forth without compatibility problems, as their usages in the CLI or as an `inputs` dependency do not change. | ||
|
|
||
| Validation and adoption can start in flakes with fewer users, such as those without reverse dependencies. | ||
|
|
||
| # Examples and Interactions | ||
| [examples-and-interactions]: #examples-and-interactions | ||
|
|
||
| ## Current flake.nix | ||
|
|
||
| ```nix | ||
| { | ||
| description = "A simple example flake exporting GNU hello for x86_64-linux"; | ||
|
|
||
| inputs = { | ||
| nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; | ||
| }; | ||
|
|
||
| outputs = { self, nixpkgs }: | ||
| let | ||
| system = "x86_64-linux"; | ||
| pkgs = nixpkgs.legacyPackages.${system}; | ||
| in { | ||
| packages.${system}.default = pkgs.hello; | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| ## Proposed structure, simple example | ||
|
|
||
| **flake.toml**: | ||
| ```toml | ||
| description = "A simple example flake exporting GNU hello for x86_64-linux" | ||
|
|
||
| [inputs.nixpkgs] | ||
| url = "github:NixOS/nixpkgs/nixos-unstable" | ||
| ``` | ||
|
|
||
| **flake.nix**: | ||
| ```nix | ||
| { | ||
| outputs = { self, nixpkgs }: | ||
| let | ||
| system = "x86_64-linux"; | ||
| pkgs = nixpkgs.legacyPackages.${system}; | ||
| in { | ||
| packages.${system}.default = pkgs.hello; | ||
| }; | ||
| } | ||
| ``` | ||
|
Comment on lines
+110
to
+131
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The choice of "reusing" / keeping Maybe the split should be |
||
|
|
||
| **flake.lock** remains unchanged. | ||
|
|
||
| ## Proposed structure with follows | ||
|
|
||
| **flake.toml**: | ||
| ```toml | ||
| description = "Example with follows relationships" | ||
|
|
||
| [inputs.nixpkgs] | ||
| url = "github:NixOS/nixpkgs/nixos-unstable" | ||
|
|
||
| [inputs.flake-parts] | ||
| url = "github:hercules-ci/flake-parts" | ||
|
|
||
| [inputs.flake-parts.inputs.nixpkgs-lib] | ||
| follows = "nixpkgs" | ||
| ``` | ||
|
|
||
| **flake.nix**: | ||
| ```nix | ||
| { | ||
| outputs = { self, nixpkgs, flake-parts }: | ||
| flake-parts.lib.mkFlake { inherit inputs; } { | ||
| systems = [ "x86_64-linux" ]; | ||
| perSystem = { pkgs, ... }: { | ||
| packages.default = pkgs.hello; | ||
| }; | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| **flake.lock** remains unchanged. | ||
|
|
||
| The exact schema shown above is illustrative. | ||
| The final design will follow the Nix JSON guideline for extensibility, | ||
| using records at appropriate levels and `null` for optional values. | ||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| - Introduces another file format into the Nix ecosystem | ||
| - Migration cost for existing flakes | ||
| - Potential confusion during transition period with two ways to specify inputs | ||
| - Requires tooling updates across the ecosystem | ||
|
|
||
| # Alternatives | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One alternative worth considering: use Dhall instead of TOML. Many of the complaints about the flake.nix ‘language’ seem to be rooted in the desire to abstract out parts of the inputs for reuse. TOML doesn't actually address this desire; it merely makes it clearer to the user that the desire can't be satisfied. The motivation to restrict how inputs are expressed seems to be to prevent arbitrary code from running, either for security reasons or so that flake input analysis can be guaranteed to terminate. Dhall addresses both concerns, while allowing many of the abstractions flake users are trying to write; it's pretty much made for exactly this use case.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That direction is more in line with the discussion that followed in NixOS/nix#4945. Regarding Dhall as a point in the configuration language design space, I might upset some folks, including my past self who was a big fan of that kind of tech before using it in conjunction with Nix. I guess we could add a helper trampoline primop to Dhall to add some Turing completeness for the algorithmically inclined? I'll boil this rant down into a more reasonable addition later, or I'd be happy to take a GitHub suggestion. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yaml could also take care of a lot of the desires here, without being fully turing complete, and without being as obscure. It has a variety of variable-like features. That said, I think simplifying the format as much as possible, and indicating that it is deliberately limited is part of the appeal of this proposal. Making inputs templateable just leads to complexity that ends up confusing newcomers to the ecosystem. I think most use cases for evaluation in inputs that don't just become features outright could be handled in external scripts instead, given the file would be much more programmatically editable.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not myself convinced that Dhall would be the best choice, but a lot of these objections strike me as not very strong.
Even if there is no convenient binding, Nix could execute
Simple flake.dhall file: {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable",
entrypoint = {
url = "github:hercules-ci/flake-parts",
inputs.nixpkgs-lib.follows = "nixpkgs",
},
}I don't think this would present more of a learning challenge to people not trying to dive into the intricacies of Dhall than the equivalent TOML would. In other words: the strength of the argument that people don't need abstractions in their inputs is inversely proportional to the strength of the argument that it would be difficult to read and write inputs written in Dhall. If many people want it, all the more reason to use a non-data-only language. If few people want it, we won't see ‘advanced’ Dhall features (scare quotes because, you know, I'm talking about things like let bindings) very often in the wild (and users can make up their own mind whether having them is worth learning about them). And it's worth emphasizing that this is not a hypothetical ‘what if people want it’. We know users want it. Here's this example as it might have been written in Dhall: let dep = \(url : Text) -> { url = url, inputs.nixpkgs.follows = "nixpkgs" }
in {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable",
entrypoint = {
url = "github:hercules-ci/flake-parts",
inputs.nixpkgs-lib.follows = "nixpkgs",
},
nix-darwin = dep "github:LnL7/nix-darwin",
home-manager = dep "github:nix-community/home-manager",
zig = dep "github:Cloudef/nix-zig-stdenv",
zls = dep "github:zigtools/zls",
hyprland = dep "github:hyprwm/Hyprland",
eww = dep "github:elkowar/eww",
}This strikes me as something that said user would have been able to learn how to write, given what they wrote in Nix.
When compared to TOML as the alternative, this isn't any sort of disadvantage; the programs TOML is able to express are a strict subset of the programs Dhall is able to express, obviously.
Using (unrestricted) Nix would be keeping it boring. In the same framework, the tradeoff is between TOML's 2 unnecessary impossibilities (not-provably-total programs, and simple deduplication of the sort we actually see people trying to write) and 1 nicer error, Dhall's 1 unnecessary impossibility (NPT programs) and 1 nicer error, or Nix's 0 unnecessary impossibilities and 1 mediocre error. On just these axes, TOML is not on the efficient frontier.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Performance is probably acceptable, but the static Nix binary would have to pull that in from somewhere for basic functionality.
This. It brings clarity and better scripting. The comparison in this thread was between Dhall and full-Nix, but the RFC does not propose either. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TOML has been chosen by projects like Rust and Python for the exact reason that it is necessarily simple. It doesn't let you get too fancy. YAML was ruled out because it allows too much flexibility. YAML is also bad at making things clear as complexity increases. Anyone complaining about Nix as an obscure language will not find Dhall an improvement. Anyone who needs the complexity should use Nix. Anyone who doesn't need it can use TOML. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @de11n toml is not that simple though, especially with the table vs array syntax - e.g. this is quite atrocious: https://toml.io/en/v1.0.0#array-of-tables Let's just stick to Nix. |
||
| [alternatives]: #alternatives | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One more alternative is to explicitly define "data only" subset of C.f We could have This of course presupposes that we do not want to "program" Given that EDIT: we could also think about using nox instead of aterms in derivations?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe it's because I have a simple mind, but I have great difficulty with nox if we could call it Nixon. Anyway, on a more serious note, but still indulging myself, those would be highly similar formats. Material for some follow-up RFCs ;)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure if that's a follow-up RFC or a prerequisite. A simple format like this could be quite useful for many situations. It would also alleviate the concerns raised here: https://github.com/NixOS/rfcs/pull/193/files#r2603388837 |
||
|
|
||
| ## Keep current structure | ||
|
|
||
| Maintain the status quo including drawbacks described in the [motivation]. | ||
|
|
||
| ## Pure Nix with better tooling | ||
|
|
||
| Improve Nix AST manipulation tools instead of introducing a new format. | ||
| Does not solve user uncertainty. | ||
|
|
||
| ## Use JSONC | ||
|
|
||
| JSONC is somewhat more aligned with the Nix language, including the presence of `null`, | ||
| and the approximate syntax for objects/attrsets, while still allowing comments. | ||
|
|
||
| ## Use YAML | ||
|
|
||
| YAML is widely supported but is despised among many for such things as [the Norway Problem](https://hitchdev.com/strictyaml/why/implicit-typing-removed/). | ||
|
|
||
| It is hard to parse correctly, | ||
| which poses a significant risk for an ecosystem with a major goal of supporting reproducibility, | ||
| as even an innocuous whitespace fix in parser output can cause a rift | ||
| where binary caches aren't shared between Nix versions and old expressions produce new outcomes. | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| ## Experimental `configs` commit by Eelco | ||
|
|
||
| [1dc3f53](https://github.com/NixOS/nix/commit/1dc3f5355a3786cab37a4de98ca46a859e015d89), part of stable branch [`configs`](https://github.com/NixOS/nix/compare/configs) implemented a `flake.toml` file. | ||
| It incorporated a "poor man's module system" into Nix where it would be of very limited functionality, | ||
| while it ossifies due the consequences in Nix of [Hyrum's law](https://www.hyrumslaw.com/) and the second order effects of reproducibility. | ||
|
|
||
| ## Flake entrypoints RFC | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I still don't really understand why we can't have a more dynamic flake.nix entrypoint. This would make it trivial to implement something like flake.toml in nix code. {
inputs = builtins.fromTOML (builtins.readFile ./inputs.toml);
outputs = import ./outputs.nix;
}edolstra mentioned that the flake format needs to be simpler to limit complexity while querying flake metadata, but couldn't that metadata be written to the flake.lock file? The locking process being complex shouldn't really be a problem. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. on simplifying on
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Tools like flakehub could instruct their users to write flake inputs in specific ways. This doesn't seem like a reason to limit how people who don't use flakehub write their flakes.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let me wander a bit........ what happens here is that the inputs section is evaluated (1), then those values are used in another evaluation (2) with the outputs. How (1) works is different than how (2) works. We may be able to loosen the restrictions a bit on (1), but it will still need to be different in some way. Different restrictions, different access, etc. It isn't hard to remove the restrictions in Nix's source code, but while I see how it isbammoying sometimes, I dont see what large differences there would be in end-user outcomes. Perhaps you wnd up with some easier tooling to manage the inputs, but a jq-but-for-nix (eg: nix-editor) can do this already. Some enterprising individuals would embed much more complicated logic into the inputs section, create some abstractions. I've wanted something like this where an input could be just a plain nix value allowing for easier overriding or CLI options to be declared. Putting "inputs" into toml or another format makes this distinction more clear, but doesn't change its underlying nature. |
||
|
|
||
| A companion RFC proposes a flake entrypoint mechanism that allows frameworks to handle output generation. | ||
| Combined with TOML flakes, | ||
| this provides the bigger picture of removing custom Nix code from flake metadata altogether, | ||
| allowing frameworks to handle both input processing and output generation declaratively. | ||
|
|
||
| The entrypoint mechanism can be implemented independently of TOML, | ||
| but the two features work well together. | ||
|
|
||
| ## devenv | ||
|
|
||
| The devenv tool uses a separate configuration format for specifying dependencies | ||
| and generating Nix configurations, | ||
| demonstrating that declarative input specifications can work well in practice. | ||
|
|
||
| ## Other language ecosystems | ||
|
|
||
| Most package managers separate dependency declarations from implementation code (`Cargo.toml`, `requirements.txt`, and to a fair degree `package.json`). | ||
| This proposal brings Nix flakes closer to that common pattern. | ||
|
|
||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
|
|
||
| - What outcomes result from a prototype of this feature? | ||
| - Schema validation for the inputs file | ||
| - How follows relationships are expressed in TOML | ||
|
|
||
| # Future work | ||
| [future]: #future-work | ||
|
|
||
| - Evaluate TOML round-tripping libraries | ||
|
|
||
| # Credit | ||
|
|
||
| Having been part of the Nix team, | ||
| I suspect that I've learned some of these ideas from the team, especially Eelco, | ||
| who has experimented in this direction. | ||
|
|
||
| Also another thank you to username-generic for asking the question in a recent [discourse thread](https://discourse.nixos.org/t/outlining-the-differences-between-flakes-and-nix-configs/72996/7), | ||
| and together with TLATER for making me realise I should just go ahead and write this RFC. | ||
Uh oh!
There was an error while loading. Please reload this page.