Skip to content

[RFC 0185] Redistribute redistributable software #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ekleog wants to merge 11 commits into
base: master
Choose a base branch
from
+225 −0
Open

[RFC 0185] Redistribute redistributable software #185

Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6c0e4bc
[RFC 0185] Redistribute redistributable software
Ekleog Dec 15, 2024
c48a29b
smaller titles
Ekleog Dec 15, 2024
54164cf
fix typo
Ekleog Dec 15, 2024
d218e50
handle review comments
Ekleog Dec 17, 2024
9dc23a7
add runnableOnHydra field for run restrictions
Ekleog Dec 18, 2024
27241c3
list additional drawbacks
Ekleog Dec 19, 2024
ecf9373
handle review comments and add size information
Ekleog Dec 31, 2024
bede48d
commit resolved questions and future work
Ekleog Mar 21, 2025
497f40c
explicitly exclude proprietary packages from zhf
Ekleog Mar 21, 2025
1c2b2e8
emilazy's review comments
Ekleog Mar 21, 2025
03bcdc0
Update rfcs/0185-redistribute-redistributable.md
Ekleog May 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions rfcs/0185-redistribute-redistributable.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
feature: redistribute-redistributable
start-date: 2024-12-15
author: Ekleog
co-authors: (find a buddy later to help out with the RFC)
shepherd-team: (names, to be nominated and accepted by RFC steering committee)
shepherd-leader: (name to be appointed by RFC steering committee)
related-issues: https://github.com/NixOS/nixpkgs/issues/83884
---

# Summary
[summary]: #summary

Make Hydra build and provide all redistributable software, while making sure installation methods stay as fully free as today.

# Motivation
[motivation]: #motivation

Currently, Hydra builds only free software and unfree redistributable firmware.
This means that unfree redistributable software needs to be rebuilt by all the users.
For example, using MongoDB on a Raspberry Pi 4 (aarch64, which otherwise has access to hydra's cache) takes literally days and huge amounts of swap.

Hydra could provide builds for unfree redistributable software, at minimal added costs.
This would make life much better for users of such software.
Especially when the software is still source-available even without being free software, like MongoDB.
Comment on lines +20 to +25
Copy link
Member

@emilazy emilazy Mar 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the intent isn’t to commit to providing any specific package then I think using a concrete example of MongoDB in the motivation section is misleading, as accepting this RFC does not necessarily mean that this motivation will be addressed.


# Detailed design
[design]: #detailed-design

We will add a `runnableOnHydra` field on all licenses, that will be initially set to its `free` field, and set to `true` only for well-known licenses.
Copy link
Member

@emilazy emilazy Mar 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still feel that doing this per‐package makes more sense given that I think we would want oversight to be on a package‐by‐package basis.

Copy link
Member

@Eveeifyeve Eveeifyeve Jan 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is better suited for allowHydra field in licence field, it should only be used when it's unfree we have permission to build it however it doesn't allow caching of the unfree. Where as free and and unfreeRedistributable we will keep.


Hydra will build all packages with licenses for which `redistributable && runnableOnHydra`.
It will still fail evaluation if the ISO image build or the Amazon AMIs were to contain any unfree software.

This will be done by evaluating Nixpkgs twice in `release.nix`.
Once with `allowUnfree = false` like today, plus once with `allowlistedLicenses = builtins.filter (l: l.redistributable && l.runnableOnHydra) lib.licenses`.
Then, most of the jobs will be taken from the allowlisted nixpkgs, while only the builds destined for installation will be taken from the no-unfree nixpkgs.

The list of jobs destined for installation, that cannot contain unfree software is:
- `amazonImage`
- `amazonImageAutomaticSize`
- `amazonImageZfs`
- `iso_gnome`
- `iso_minimal`
- `iso_minimal_new_kernel`
- `iso_minimal_new_kernel_no_zfs`
- `iso_plasma5`
- `iso_plasma6`
Comment on lines +43 to +48
Copy link
Member

@emilazy emilazy Mar 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The graphical ISOS were unified, and Plasma 5 is gone. But I don’t know if it’s worth listing them explicitly anyway.

- `sd_image`
- `sd_image_new_kernel`
- `sd_image_new_kernel_no_zfs`

Copy link
Member

@Eveeifyeve Eveeifyeve Jan 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add to this list apple-sdk & windows.sdk.

# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions

Copy link
Member

@alyssais alyssais Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to make sure some particular expectations are managed: While we're working on staging-next, we're looking at Hydra to identify regressions and fix them before staging-next is merged to master. With this change, there will be new Hydra jobs for non-free packages. The license terms of those packages could make it difficult or outright prevent us doing things to fix them, or even to try to reproduce locally, so it's not going to be possible in the general case to give these packages the same level of protection from regressions as we try to give free packages. So it should be understood that even though these packages are now built by Hydra and available in the binary cache, they shouldn't be expected to be any less likely to be broken by the staging process (or other PRs) than they currently are.

Copy link
Member

@alyssais alyssais Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to make sure some particular expectations are managed: While we're working on staging-next, we're looking at Hydra to identify regressions and fix them before staging-next is merged to master. With this change, there will be new Hydra jobs for non-free packages. The license terms of those packages could make it difficult or outright prevent us doing things to fix them, or even to try to reproduce locally, so it's not going to be possible in the general case to give these packages the same level of protection from regressions as we try to give free packages. So it should be understood that even though these packages are now built by Hydra and available in the binary cache, they shouldn't be expected to be any less likely to be broken by the staging process (or other PRs) than they currently are.

Copy link
Member

@7c6f434c 7c6f434c Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully (and by intention) the runnableOnHydra would exclude licenses that put obligations on Hydra just from running the tests, and thus mere reproduction would not bring onerous obligations; but indeed actually identifying the issue often comes boils down to reverse engineering a part of the program.

With these changes, here is what could happen as things currently stand, if the licenses were all to be marked `runnableOnHydra`.
This is not meant to be indicative of what should happen or not, but indicative of what could happen.
Each package's individual `license` field setup is left to its maintainers, and nixpkgs governance should conflict arise.
This RFC does not mean to indicate that it is right or wrong, and is not the right place to discuss changes to this field.
Should one have disagreements on any specific package in this list, please bring that up to that package's maintainers.

It is also suggested in this RFC that people, upon marking licenses as `runnableOnHydra`, check all the derivations that use this license.
They could then have to mark them as either `hydraPlatforms = []`, `preferLocalBuild = true` and/or `allowSubstitutes = false`.
This might be useful for packages like TPTP:
they may not yet be marked as such due to these flags having no impact on unfree packages;
but would take gigabytes on Hydra for basically no local build time improvement

With this in mind, Hydra could start building, among others:
- CUDA
- DragonflyDB
- MongoDB
- Nomad
- NVIDIA drivers
- Outline
- SurrealDB
- TeamSpeak
- Terraform
- Unrar
- Vagrant
- NixOS tests that involve such software (eg. MongoDB or Nomad)

And Hydra will keep not building, among others:
- CompCert
- DataBricks
- Elasticsearch
- GeoGebra
- Widevine CDM
Comment on lines +72 to +91
Copy link
Member

@emilazy emilazy Mar 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it’d be best to remove the list of examples entirely if this isn’t meant to spark discussion on whether the “could build”/“wouldn’t build” line drawn here is accurate?


# Drawbacks
[drawbacks]: #drawbacks

The main risk is that NixOS could end up including unfree software in an installation image if:
1. we forgot to add it to the list of no-allowed-unfree jobs, and
2. a maintainer did actually add unfree software to that build.

This seems exceedingly unlikely, making this change basically risk-free.

The only remaining drawback is that Hydra would have to evaluate Nixpkgs twice, thus adding to eval times.
However, the second eval (with no-unfree) should be reasonably small and not actually evaluate all packages, as it is only used for installation media.

# Alternatives
[alternatives]: #alternatives

### Having Hydra actually only build FOSS derivations, not even unfree redistributable firmware

This would likely break many installation scenarios, but would bring us to a consistent ethical standpoint, though it's not mine.

### Keeping the status quo

This results in very long builds for lots of software, as exhibited by the number of years people have been complaining about it.

### Having Hydra redistribute redistributable software, without verifying installation media

This would be slightly simpler to implement, but would not have the benefit of being 100% sure our installation media are free.

### Having Hydra redistribute redistributable software, with a check for the installation media

This is the current RFC.

### Building all software, including unfree non-redistributable software

This is quite obviously illegal, and thus not an option.

### Not having the `runnableOnHydra` field on licenses

This would make it impossible for Hydra to build them as things currently stand:
Hydra would then risk actually running these packages within builds for other derivations (eg. NixOS tests).

This would thus only be compatible with changes to Hydra, that would allow to tag a package as not allowed to run, but only to redistribute.
Such a change to Hydra would most likely be pretty invasive, and is thus left as future work.

# Prior art
[prior-art]: #prior-art

According to [this discussion](https://github.com/NixOS/nixpkgs/issues/83433), the current statu quo dates back to the 20.03 release meeting.
More than four years have passed, and it is likely worth rekindling this discussion, especially now that we actually have a Steering Committee.

Recent exchanges have been happening in [this issue](https://github.com/NixOS/nixpkgs/issues/83884).
Copy link
Member

@zimbatm zimbatm Dec 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, we also started building all the redistribuable+unfree packages in the nix-community sister project.

See all the unfree-redis* jobsets here: https://hydra.nix-community.org/project/nixpkgs
It's only ~400 packages. The builds are available at https://nix-community.cachix.org/

The jobset is defined in nixpkgs to make upstreaming easier:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/release-unfree-redistributable.nix

If this RFC passes it will be even better as users don't necessarily know about or want to trust a secondary cache.

Copy link
Member Author

@Ekleog Ekleog Dec 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great to know, thank you! Though we may need to do a bit more to properly handle the "cannot be run on hydra" point that was raised above.

I can already see on the hydra link you sent that eval takes <1min, so should be a negligible addition to hydra's current eval times. Build times seem to take ~half a day. AFAIU there's a single machine running the jobs. If I read correctly, hydra currently has ~5 builders, and one trunk-combined build takes ~1 day. So it means that the build times would increase by at most ~10%, and probably less considering that there is probably duplication between what the nix-community hydra builds and what nixos' hydra is already building. I'm also not taking into account machine performance, which is probably stronger on nixos' hydra than nix-community's hydra.

I think this means eval/build times are things we can reasonably live with, and if we get any surprise we can always rollback.

There's just one thing I can't find in the links you sent to properly adjust the unresolved questions: do you know how large one build closure is on nix-community's hydra? I don't know how to get it on nixos' hydra either but it'd still help confirm there's zero risk.

Copy link
Member

@zimbatm zimbatm Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this means eval/build times are things we can reasonably live with, and if we get any surprise we can always rollback.

Yes, especially since the way the unfree-redis jobset is put together is by evaluating and filtering trough all the nixpkgs derivations. So most likely the combined eval time is much smaller than the addition of both.

There's just one thing I can't find in the links you sent to properly adjust the unresolved questions: do you know how large one build closure is on nix-community's hydra?

The best I can think of is to build a script that takes all the successful store paths, pulls them from the cache, runs nix path-info -s on them and then sums up the value.

Copy link
Member Author

@Ekleog Ekleog Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your answer! I actually more or less found the answer from Hydra's UI. Here is my script:

curl https://hydra.nix-community.org/jobset/nixpkgs/cuda/channel/latest > hydra-jobs
cat hydra-jobs | grep '<td><a href="https://hydra.nix-community.org/build/' | cut -d '"' -f 2 > job-urls
for u in $(cat job-urls); curl "$u" 2>/dev/null | grep -A 1 'Output size' | tail -n 1 | cut -d '>' -f 2 >> job-sizes; wc -l < job-sizes | head -c -1; echo -n " / "; wc -l < job-urls; end
awk '{sum += $1} END {print sum}' job-sizes
# NVidia kernel packages take ~1.3GiB each and there are 334-164 = 170
# Total: 215G, so 45G without NVidia kernel packages

I got the following results:

  • For unfree-redist-full, a total of 215G, including 200G for NVidia kernel packages and 15G for the rest of the software
  • For cuda, a total of 482G

Unfortunately I cannot run the same test on NixOS' hydra, considering that it disabled the channels API.

I just updated the RFC with these numbers, it might make sense to not build all of cuda on hydra at first, considering the literally hundreds of duplicated above-1G derivations :)

Copy link
Member

@vcunat vcunat Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So with the current Hydra workflows I'd estimate that very roughly as uploading 2 TB per month to S3. (we rebuild stuff) Except that we upload compressed NARs, so it would be less.

Copy link
Member Author

@Ekleog Ekleog Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly, that it'd be reasonable to do the following?

  1. Just push everything, and
  2. if compression is not good enough rollback CUDA & NVidia kernels; and
  3. even if we need to rollback, the added <1T would not be an issue to keep "forever"

Copy link
Member

@vcunat vcunat Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. To me it doesn't even feel like a technical question. (3. is WIP so far, I think. There's no removal from cache.nixos.org yet.)


# Unresolved questions
[unresolved]: #unresolved-questions

Is the list of installation methods correct?
I took it from my personal history as well as the NixOS website, but there may be others.
Also, I may have the wrong job name, as I tried to guess the correct job name from the various links.

How large are the packages Hydra would need to additionally store?
This could be another drawback, if it is large enough to not be negligible compared to free software only.

# Future work
[future]: #future-work

Modifying Hydra to allow building and redistributing packages that it is not legally allowed to run.
This would be a follow-up project that is definitely not covered by this RFC due to its complexity, and would require a new RFC before implementation.