Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Nix Flakes #546

Open
kamadorueda opened this issue Sep 16, 2021 · 2 comments
Open

Support Nix Flakes #546

kamadorueda opened this issue Sep 16, 2021 · 2 comments
Labels
help wanted Will be solved only if someone contributes it
Milestone
Makes on Nix 3.0

Comments

@kamadorueda
Copy link
Contributor

kamadorueda commented Sep 16, 2021 •
edited
Loading

Related:

Useful links:

Examples:

Things to take into account:

  • We wan to benefit from flakes hermeticity
  • We want to keep installation and usage of non-nix-experts users as simple as it's today
@kamadorueda kamadorueda self-assigned this Sep 16, 2021
@blaggacao blaggacao mentioned this issue Sep 16, 2021
Closed
kamadorueda added a commit that referenced this issue Sep 16, 2021
@kamadorueda
feat(build): #546 packages
1e50a46 
- Expose everything as packages, pending to see
  how we can identify apps
kamadorueda added a commit that referenced this issue Sep 17, 2021
@kamadorueda
feat(build): #546 expose internals
9221aa3 
- Expose internals so the CLI can consume them
kamadorueda added a commit that referenced this issue Sep 17, 2021
@kamadorueda
feat(build): #546 flake compat cli
48776e7 
- Make the CLI compatible with Flakes via
  a feature flag
kamadorueda added a commit that referenced this issue Sep 17, 2021
@kamadorueda
refac(build): #546 simplify scope and code
f04f8ae 
- Refactor code so we know what's flake specific
- Limit scope of this experimental feature to linux x86-64
@kamadorueda
Copy link
Contributor Author

I'm just gonna drop here two good reasons not to use flakes yet:

  1. It's unstable, which means bugs in the core binary do exist and will sooner or later bite you, no matter what --option xxx yyy, ulimit, etc, you configure to try controlling file descriptor leaks:

image

for some companies like us it's not reasonable to risk reliability on production, the thing must just work, preferably for ever

on Nix stable this too-many-open-files do not happen (downgrading from Flakes was actually the fix to the problem)

  1. Evaluation cache is awesome on flakes, but runtime cache-ability is not perfect yet, for instance we cannot make use of projectPath or something similar:

image

which means that in a tree like following:

  • a
    • whatever-code
  • b
    • c.sh
    • d.nix (depends on c.sh)

if whatever-code changes, then the entire inputs.self.sourceInfo.outPath changes, and thus ./b/c.sh changes, and ./b/d.nix must be build again. This does not happen on Makes using Nix stable because using projectPath you can make d.nix depend only on the NAR serialization of b folder, and changes outside of b do not require to rebuild d.nix.

Maybe this is the reason why some nix experts are using Bazel to achieve incremental builds?

We achieved incremental builds at Fluid Attacks using nix stable and projectPath only

I'm gonna stop working towards Flakes for now

The work done so far serves as a draft on how compatibility with Flakes can be implemented

@kamadorueda kamadorueda removed their assignment Sep 17, 2021
kamadorueda added a commit that referenced this issue Sep 17, 2021
@kamadorueda
feat(build): #546 propagate inputs
572e2d3 
- This merges inputs from the flake and make them
  available to scripts under the magic makes/ folder
  as well as to the modules configuration

Relates to #540
kamadorueda added a commit that referenced this issue Sep 17, 2021
@kamadorueda
fix(build): #546 nix store not found
92592b5 
- It was missing to set the searchPaths
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 17, 2021
@blaggacao
fix(build): fluidattacks#546 infinite recursion on inputs.self
e3cf8b1 
- Assignments involving `self` are only allowed on the right hand side
  of an output. `evaluatedOutputs` violated this axiom.
- Ther reason lies within the recursiveness of these two lines in the
  builtin flake eval:

https://github.com/NixOS/nix/blob/7c90552879da4d1df99b50c85e94201981e60123/src/libexpr/flake/call-flake.nix#L44-L46

fixes: fluidattacks#573
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 17, 2021
@blaggacao
fix(build): fluidattacks#546 infinite recursion on inputs.self
be5ab07 
- Assignments involving `self` are only allowed on the right hand side
  of an output. `evaluatedOutputs` violated this axiom.
- Ther reason lies within the recursiveness of these two lines in the
  builtin flake eval:

https://github.com/NixOS/nix/blob/7c90552879da4d1df99b50c85e94201981e60123/src/libexpr/flake/call-flake.nix#L44-L46

fixes: fluidattacks#573
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 17, 2021
@blaggacao
fix(build): fluidattacks#546 infinite recursion inputs.self
94103c6 
- Assignments involving `self` are only allowed on the right hand side
  of an output. `evaluatedOutputs` violated this axiom.
- Ther reason lies within the recursiveness of these two lines in the
  builtin flake eval:
  https://git.io/Jzki9

fixes: fluidattacks#573
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
feat(build): fluidattacks#546 propagate inputs
2230484 
- Use the experimental flake support in this repo to help uncover bugs
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
feat(build): fluidattacks#546 propagate inputs
169da30 
- Use the experimental flake support in this repo to help uncover bugs
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
feat(build): fluidattacks#546 use experimental flake support
b086213 
- Use the experimental flake support in this repo to help uncover bugs
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 infinite recursion inputs.self
eb94294 
- Assignments involving `self` are only allowed on the right hand side
  of an output. `evaluatedOutputs` violated this axiom.
- Ther reason lies within the recursiveness of these two lines in the
  builtin flake eval:
  https://git.io/Jzki9

fix: fluidattacks#573
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
feat(build): fluidattacks#546 use experimental flake support
83bf22c 
- Use the experimental flake support in this repo to help uncover bugs
kamadorueda pushed a commit that referenced this issue Sep 18, 2021
@blaggacao @kamadorueda
fix(build): #546 infinite recursion inputs.self
9fc5bbe 
- Assignments involving `self` are only allowed on the right hand side
  of an output. `evaluatedOutputs` violated this axiom.
- Ther reason lies within the recursiveness of these two lines in the
  builtin flake eval:
  https://git.io/Jzki9

fix: #573
kamadorueda pushed a commit that referenced this issue Sep 18, 2021
@blaggacao @kamadorueda
feat(build): #546 use experimental flake support
07e4587 
- Use the experimental flake support in this repo to help uncover bugs
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 new string context projectSrc (flakes)
e2aa78b 
- `self.sourceInfo.outPath` brings a string context with it
- Drop it for this pattern to work
- Idempotent noop in non-flake scenario

fixes: fluidattacks#577 (has more context)
@blaggacao blaggacao mentioned this issue Sep 18, 2021
Merged
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 new string context projectSrc (flakes)
6cef42e 
- `self.sourceInfo.outPath` brings a string context with it
- Drop it for this pattern to work
- Idempotent noop in non-flake scenario

fixes: fluidattacks#577 (has more context)
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 string context on projectSrc
98944cc 
- `self.sourceInfo.outPath` brings a string context with it
- Drop it for this pattern to work
- Idempotent noop in non-flake scenario

fixes: fluidattacks#577 (has more context)
kamadorueda pushed a commit that referenced this issue Sep 18, 2021
@blaggacao @kamadorueda
fix(build): #546 string context on projectSrc
2143be2 
- `self.sourceInfo.outPath` brings a string context with it
- Drop it for this pattern to work
- Idempotent noop in non-flake scenario

fixes: #577 (has more context)
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 makes' inputs topple flakes'
22ded0d 
- Make inputs now prime over flake inputs
- Rename to reflect this special purpose (& temporary) semantics of
  this arg

fixes: fluidattacks#579
blaggacao added a commit to input-output-hk/makes that referenced this issue Sep 18, 2021
@blaggacao
fix(build): fluidattacks#546 makes' inputs topple flakes'
d54753f 
- Make inputs now prime over flake inputs
- Rename to reflect this special purpose (& temporary) semantics of
  this arg

fixes: fluidattacks#579
kamadorueda pushed a commit that referenced this issue Sep 18, 2021
@blaggacao @kamadorueda
fix(build): #546 makes' inputs topple flakes'
e668884 
- Make inputs now prime over flake inputs
- Rename to reflect this special purpose (& temporary) semantics of
  this arg

fixes: #579
@kamadorueda kamadorueda added the help wanted Will be solved only if someone contributes it label Sep 28, 2021
@kamadorueda kamadorueda moved this to 🆕 New in Makes Roadmap Sep 24, 2022
@kamadorueda kamadorueda moved this from 🆕 New to 📋 To do in Makes Roadmap Sep 24, 2022
@Ten0
Copy link

Ten0 commented Dec 24, 2023

2. runtime cache-ability is not perfect yet, for instance we cannot make use of projectPath or something similar:

Unless I'm mistaken this is fixed the next day by 2143be2, correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Will be solved only if someone contributes it
Projects
Makes Roadmap
Status: 📋 Backlog
Milestone
Makes on Nix 3.0
Development

No branches or pull requests
2 participants
@Ten0 @kamadorueda