check:quick --mode before performance investigation - devenv exec_if_modified glob traverses node_modules

research.md

check:quick --mode before Performance Investigation

Problem

devenv tasks run check:quick --mode before --no-tui takes ~623s (10+ minutes).

What check:quick runs

check:quick is a pure aggregation task (no exec). With --mode before, it runs all upstream deps:

check:quick
  ├── ts:check (depends on genie:run, pnpm:install)
  ├── megarepo:check (depends on megarepo:sync)
  └── lint:check (aggregation)
        ├── lint:check:format (depends on genie:run) ← has exec_if_modified
        ├── lint:check:oxlint (depends on genie:run, pnpm:install) ← has exec_if_modified
        ├── lint:check:genie (depends on pnpm:install) ← has exec_if_modified
        └── lint:check:genie:coverage ← has exec_if_modified

Root Cause: exec_if_modified glob traverses node_modules

The bug

The devenv-tasks binary (Rust) uses the glob crate (v0.3.3) to expand exec_if_modified patterns. This crate does not respect .gitignore and traverses into node_modules.

For this repo, the patterns match:

Scope	Files matched
Actual source files	817
Including node_modules	243,693

That's a 298x bloat. For each matched file, devenv-tasks:

1. Reads the file
2. Computes a blake3 hash
3. Stores/compares in SQLite

Worst offending patterns

Pattern	Source files	node_modules files
tests/**/*.ts	~233	~3.4M entries traversed
scripts/**/*.ts	~21	~5.3M entries traversed
scripts/**/*.js	0	~5.8M entries traversed
docs/src/**/*.ts	~171	~2.0M entries traversed

Timing Breakdown

Individual task timings (via devenv tasks --mode single)

Task	Wall time	Notes
lint:check:format	10m17s	exec_if_modified overhead
lint:check:oxlint	9m18s	exec_if_modified overhead
lint:check:genie	44.9s	exec_if_modified (fewer patterns, cached)
ts:check	48.4s	No exec_if_modified, actual tsc time
genie:run	2.5s	No exec_if_modified
megarepo:check	0.08s	Cached
pnpm:install	0.09s	Cached

Direct tool timings (bypassing devenv task runner)

Tool	Wall time	Actual work
tsc --build tsconfig.dev.json	39s	39s
/nix/store/...-lint-check-format (script directly)	25s	oxfmt: 1.7s
oxlint packages tests scripts docs .github	1.4s	1.4s
genie --check	34s (via direnv exec)	~2s
genie run	2.5s	2.4s

Overhead attribution

• devenv task runner for tasks without exec_if_modified: ~0.1s overhead
• devenv task runner for tasks with exec_if_modified: 9-10 minutes overhead
• direnv exec . environment setup: ~32s overhead per invocation

Improvement Options

Option 1: Fix exec_if_modified patterns (upstream in effect-utils)

Make patterns more specific to exclude node_modules. Examples:

# Instead of:
tests/**/*.ts
scripts/**/*.ts
packages/@livestore/*/src/**/*.ts

# Use either:
# A) Exclude node_modules in the pattern (if glob syntax supports it)
# B) Make patterns more specific:
tests/*/src/**/*.ts
tests/*/*.ts
scripts/src/**/*.ts
scripts/*.ts

Impact: Reduces file matching from 243k to 817 files. Should bring exec_if_modified from ~10min to <1s.

Option 2: Fix devenv-tasks upstream to respect .gitignore

Replace the glob crate with the ignore crate (which respects .gitignore by default, auto-skips node_modules).

Impact: Universal fix for all projects using devenv-tasks. Best long-term solution.

Option 3: Add exclude patterns to devenv-tasks config

Add an exec_if_modified_exclude field to TaskConfig that filters out matching paths.

Impact: More flexible but requires config changes in each project.

Option 4: Remove exec_if_modified for lint tasks

Since the lint tools themselves are fast (~2s), the caching benefit doesn't justify the overhead. Just always run them.

Impact: Simplest fix. lint:check:format + lint:check:oxlint would go from ~10min to ~5s total.

Option 5: Bypass devenv task runner for pre-commit

Replace the git hook:

entry = "devenv tasks run check:quick --mode before";

With a direct script that runs the tools in parallel:

tsc --build tsconfig.dev.json &
oxfmt --check packages tests scripts docs .github &
oxlint packages tests scripts docs .github &
wait

Impact: Total time ≈ max(39s tsc, 2s oxfmt, 1.4s oxlint) ≈ 40s vs current 623s (15x faster).

Projected timing after fix

With Option 1 or 2 (fixing exec_if_modified):

Task	Current	Projected
genie:run	2.5s	2.5s
pnpm:install	0.1s	0.1s
ts:check	48s	40s
lint:check:format	617s	~3s
lint:check:oxlint	558s	~3s
lint:check:genie	45s	~3s
megarepo:check	0.1s	0.1s
Total (parallel)	~623s	~45s

Secondary Issues

1. oxfmt crash: oxfmt --check panics on .github/workflows/sync-docs.yml with "internal error: entered unreachable code: Document must end with a FormatElement::Line(Hard)" - this is an oxfmt bug

2. direnv exec . overhead: ~32s per invocation (enterShell + setup tasks). Not relevant for devenv task runner path but affects direct tool invocation