Detect module flags by walking package.el files.
This still does not build all dependencies, because some are enabled
only if some flag or other module is disabled. But this should be close.
...not yet with all flags, but that's the next step.
This surfaced several problems not caught by the existing "full" build,
because that did not enable dependencies conditionally enabled if a
second module is enabled.
There are a small number of dependencies only enabled if a second module
is *not* enabled, which I intend to add some manual tests for.
They do not have src set, so if we pull them in without having a pin
they fail to build.
Noticed with org, so add a test for that. Probably not the only one
affected, though.
Fixes#4
This now matters for CI, as the example is still built using fetchGit
instead of the github fetcher.
The example was doing double duty as an example and a stress-test,
enabling some problematic or interesting modules. That is no longer
necessary (once the branch with dynamically generated init.el with all
modules/flags enabled lands).
Using the lockfile hash as cache key did not make much sense, since
changes to Unstraightened itself change what we fetch.
And a static restore key means we never flush our cache: that might be
ok if cache size was reasonable, but it is not (over 3 GiB).
Switch to using the date as cache key, and only restore caches from the
current month.
Add tarballs as a second cache.
I considered caching all of ~/.cache/nix, but do not want to find out
the hard way any (new) caches are not portable if I add CI for other
platforms. The tarball-cache seems to be another git tree, so it should
be ok.
This seems to be much more space-efficient: ~/.cache/nix/tarball-cache
is about 700MiB uncompressed, under 300MiB as tzst (using tar's
defaults, CI uses zstdmt but I assume will be in the same ballpark). The
gitv3 cache is multiple GiB.
CI will still build doom-example using fetchGit. I intend to shrink the
number of modules enabled in the example to keep gitv3 cache size under
control.
This may turn out to be too much (but it does at least build).
Motivation: CI's git checkouts consume an unmanageably large amount of
cache (over 3 GiB per snapshot out of 10 GiB quota), and must be cached
for acceptable build speeds. Dropping submodules should help somewhat
directly, but I want to try switching most of CI over to fetchTree's
github fetcher, which won't include submodules. This change should help
maintain parity.
I'm also seeing a submodule fetch failure in CI (for stan-mode) that I
may not need to debug if the package functions without that submodule.
After upgrading my local Nix from 2.18 to 2.22, evaluating
Unstraightened became very slow. I think this is a bug in
Nix (https://github.com/NixOS/nix/issues/10773), and it may explain
the slowness I've been seeing in CI.
Attempt to work around this by unconditionally passing `ref` to
`fetchGit`. This seems like it should not do anything, since we also
pass a `rev` and set `allRefs = true`, but it does work around the
caching issue (mostly... it looks like submodules still hit it).
Tested locally in both Nix 2.18 and Nix 2.22. I did see some odd
warnings with 2.18 (`warning: refname
'e4031935803c66eca2f076dce72b0a6a770d026c' is ambiguous`), but only for
one refname and they did not recur. Ignore that for now.
Having CI confirm allRefs false is still safe would require a network
hit for each repo, which is already problematic. But
https://github.com/NixOS/nix/issues/7120 means we'd need to drop CI's
cached content, not just its cached refs: given how much we're fetching
that seems too much.
Fetch all refs unconditionally, assuming we're typically re-fetching the
same fixed rev repeatedly, which should be cached.
CI is failing with 'Error: Invalid("no nixpkgs dependency found for
specified key: ")`, probably because of this leftover start at multiple
nixpkgs inputs.
Remove it.
The doom profile is a non-negligible fraction of the cachix binary size,
that will only get worse (any dependency change will trigger a rebuild),
and caching them is essentially useless.
Expose emacsWithPackages so we can push just that. Skip
emacsWithPackages itself and push just the -deps derivation while we're
at it: caching either emacsWithPackages or -deps is similarly useless,
but they are a bit smaller.
When I say "non-negligible", it's about 1 MiB per "full" profile: much
larger than most individual libraries but not actually large. So this is
really a micro-optimization.
Triggered by ob-ammonite (used by scala, in emacsattic) depending on
ammonite-term-repl (also in emacsattic). `eself` only contains
emacs-overlay and doomPackageSet.
All of this needs refactoring, but an explicit recursive call fixes the
immediate problem.
We need better test coverage to figure out how many more of these there
are. I don't see a way of dealing with them better, but (unless they are
revived) they should not change underneath us, so keeping these
overrides does not seem too risky.
Fix formatting for the sly-stepper pin while I'm there.
This is only a starting point. For this to be more useful, we need to
build for multiple Emacsen. But I want to get a baseline for the size of
the closure first (as well as confirming it works in the first place).
