Git Pack File Accumulation Issue Report for Conductor Engineering

gistfile0.md

Issue Report: Git Pack File Accumulation Causing "Too Many Open Files" Errors

Problem

When running git commit (and potentially other git operations) in repositories managed by Conductor, users may encounter:

Too many open files (os error 24)

Root Cause

The Conductor-managed repository accumulated 37 pack files in .git/objects/pack/ over time. Normal repositories have 1-3 pack files after garbage collection.

Each git operation requires opening all pack files simultaneously. Combined with:

• Multiple nested .git directories created by Conductor (observed in .conductor/*/ and .conductor/*/workdir/)
• Shell integrations like Powerlevel10k's gitstatus daemon that hold pack files open for prompt rendering
• Multiple terminal sessions, each with their own gitstatus process

This exhausts the per-process file descriptor limit (default 256 on macOS).

Technical Details

Observed directory structure:

build-agent/
├── .git/                                    # Main repo - had 37 pack files
├── workdir/.git/
├── .conductor/
│   ├── washington-v1/.git/
│   ├── washington-v1/workdir/.git/
│   ├── toronto-v2/.git/
│   ├── toronto-v2/workdir/.git/
│   └── managua-v1/.git/

Pack file accumulation cause: Git creates new pack files during fetch, clone, and certain worktree operations. Without periodic garbage collection, these accumulate. Copying .git directories or creating worktrees from existing repos can inherit or create fragmented pack files.

Recommendations for Conductor

1. Run git gc after worktree/clone operations

   git gc --auto  # Lightweight, only runs if thresholds exceeded

2. Run git repack when setting up new workdirs

   git repack -d -l  # Consolidates packs, removes redundant ones

3. Enable git maintenance in managed repos

   git maintenance start  # Sets up background maintenance scheduler

4. Set aggressive auto-gc thresholds for managed repos

   git config gc.autoPackLimit 10  # Default is 50

5. Consider using git worktree properly instead of copying .git directories, as worktrees share the parent repo's object store and don't create duplicate pack files.

6. For long-running managed environments, consider periodic maintenance:

   git maintenance run --task=gc

Resolution

Running git gc --aggressive consolidated the 37 pack files down to 2, immediately resolving the issue.