The Immutable Contract: Establishing Dependency and ABI Compatibility as a First-Class Citizen in Linux Systems.md

Title: The Immutable Contract: Establishing Dependency and ABI Compatibility as a First-Class Citizen in Linux Systems

Date: October 26, 2023 Author: Senior Platform Architect, Infrastructure & Tooling Audience: Linux Foundation, freedesktop.org, Distribution Maintainers

---

1. Problem Statement

The Linux ecosystem faces a structural crisis that is frequently misdiagnosed as user error or package management complexity. This crisis is the fragility of the shared userspace environment. While the Linux kernel maintains an exemplary standard of backward compatibility (userspace binaries from a decade ago will largely run on a modern kernel), the userspace layer—comprising the C standard library, basic utilities, language runtimes, and graphical toolkits—offers no such guarantee.

"Dependency hell" is not merely the inability to install a package; it is the systemic inability of the operating system to act as a stable platform for software over time. Currently, Linux distributions prioritize "cleanliness" (removing old libraries to save space and reduce security surface) and "freshness" (pushing the latest versions) over application longevity. This creates an environment where an application compiled on Ubuntu 20.04 is statistically unlikely to run natively on Ubuntu 24.04 without significant intervention.

Unlike Windows, where the Win32 API constitutes a rigid contract maintained for decades, or macOS, which provides versioned frameworks, Linux treats the userspace as a fluid, rolling target. This forces developers to target a specific point in time (a distribution release) rather than the platform itself.

2. Root Causes

The instability of the Linux userspace is driven by four architectural and cultural factors:

1. Identity-Based Resolution: Package managers (APT, DNF, pacman) largely resolve dependencies based on package identity and strict version numbers (e.g., Requires: libssl1.1) rather than capabilities or interfaces. When the identity changes (to libssl3), the dependency chain breaks, even if the functional capability remains.
2. Lack of Enforced ABI Stability: Apart from glibc, few library maintainers strictly adhere to forward-compatible Application Binary Interfaces (ABI). The use of ELF SONAME (Shared Object Name) allows for multiple incompatible versions to coexist theoretically, but distribution policies usually dictate the removal of older SONAMEs to prevent "repo rot," effectively orphaning binaries linked against them.
3. Language Runtime Fragility: Interpreted languages, particularly Python, have adopted a culture of breaking changes between minor versions. The deprecation cycles in runtimes are aggressive, often removing standard library modules or changing C-extension APIs, rendering applications written for Python 3.8 incompatible with Python 3.12 without code modification.
4. The "Distro Build" Monolith: Distributions operate on the assumption that they can recompile the entire world for every release. This works for open-source software within the repository but is catastrophic for proprietary software, games, or legacy in-house tools where source code or build pipelines are no longer available.

3. Observable Symptoms

The industry has reacted to this instability not by fixing the platform, but by abandoning the shared userspace model entirely.

• The Containerization Default: Developers increasingly treat Docker not just as a deployment tool, but as a compatibility crutch. We see simple scripts wrapped in multi-gigabyte container images solely to pin the specific version of glibc and Python required to run them.
• The Rise of Universal Bundles: Flatpak, Snap, and AppImage solve the problem by bundling the entire userspace runtime with the application. While effective, this creates massive duplication. A system with 50 Flatpaks might have 50 copies of libjpeg, wasting storage and preventing shared memory optimization.
• Gaming Instability: Games built for Linux in 2014 frequently fail to launch on modern distributions due to missing libraries (e.g., libpng12), forcing users to rely on community-hacked patches.
• Electron Bloat: Desktop applications are increasingly shipping their own browser engines and Node.js runtimes (Electron), effectively bypassing the system's native UI toolkits because GTK and Qt ABIs are too volatile to trust for long-term commercial support.

Paradoxically, the Windows binary (running via WINE) is often more stable on Linux than the native Linux binary, because WINE enforces the strict backward compatibility that the native Linux userspace refuses to provide.

4. Why Existing Solutions Are Incomplete

Current mitigation strategies are bypass mechanisms, not platform solutions.

• Containers (Docker/Podman): These function as "dependency prisons." They isolate the application from the host system, which breaks integration (system themes, hardware acceleration access, inter-process communication). They accept that the host environment is hostile to the application.
• Flatpak/Snap Runtimes: While these introduce the concept of "Runtimes" (SDKs), they essentially shift the problem from the host OS to the Runtime maintainer. If an app targets the GNOME 40 Runtime, it eventually becomes obsolete when the Runtime is End-of-Life (EOL), requiring the developer to port the app to a newer Runtime. It delays breakage but does not prevent it.
• Proton/Steam Runtime: Valve’s solution involves shipping a curated, fixed-point distribution inside the user’s distribution. This is a heroic engineering effort that serves as an indictment of standard Linux distributions: a third-party vendor had to build a stabilized OS layer because the actual OS vendors would not.

5. Compatibility Contract Proposal

To mature as a platform, Linux must move from a "build-target" model to a "contract-target" model. We propose the following system-level architecture changes:

A. Capability-Based Dependency Resolution

We must move beyond Package = Version logic. Dependencies should be defined by feature contracts.

• Current: Requires: libfoo.so.1
• Proposed: Requires: Capability(libfoo.feature_set_2023)

B. Forward-Compatible Loaders and Shims

The dynamic linker (ld.so) and language loaders should be enhanced to support Compatibility Shims. If an application requests an older library version that is missing, the system should be able to load a newer version that declares compatibility, or a thin translation layer (shim) that maps the old symbols to the new library. This mimics the Windows "Side-by-Side" (WinSxS) assembly strategy but adapted for ELF.

C. Mandatory Symbol Versioning

Critical system libraries (graphics, audio, input, foundational libs) must enforce strict symbol versioning. Deleting a symbol should be viewed as a breaking contract violation. If a symbol must be deprecated, it must remain present as a stub or wrapper for a minimum of 10 years (LTS+ timeline), even if it creates technical debt in the library.

D. Runtime Feature Probing

Applications should be encouraged (and tooled) to probe for features at runtime rather than assuming them based on version numbers. The OS should provide a standard "Capabilities API" that allows an application to query: "Does the current SSL library support TLS 1.3?" rather than checking if the library is version 3.0.

6. Language Runtime Example: The Python Contract

Consider a data science application certified for Python 3.12. Under the current model, running this on a system with Python 3.15 is a gamble that usually fails due to removed APIs or bytecode incompatibility.

The Proposal in Action:

1. Stable ABI Enforcement: Python exposes the abi3 Stable ABI. The ecosystem must enforce that binary extensions (NumPy, Pandas) distribute abi3 wheels by default, which are guaranteed to load on future Python versions.
2. Version Aliasing: The system Python loader should recognize that Python 3.15 is a superset of Python 3.12 capabilities. When the app requests /usr/bin/python3.12, the system acts as a hypervisor. It launches the 3.15 interpreter but activates a Compatibility Context.
3. Compatibility Context: In this mode, Python 3.15 re-injects deprecated standard library modules (via a legacy support package) and suppresses syntax warnings for older constructs. The app runs transparently.

7. Games and Legacy Software

Games are "fire-and-forget" software; developers rarely update them post-launch.

• The Problem: A game linked against libcurl.so.4 crashes when the distro upgrades to a version with different symbol versioning, or when libstdc++ changes its C++11 ABI.
• The Solution: The Core Linux Compact. A designated subset of libraries (glibc, libstdc++, libgcc, SDL, OpenAL, Vulkan loader) constitutes the "Core Compact."
• Distributions agree that these libraries will never break backward compatibility. If a breaking change is required, a new library name is chosen, and the old one remains available effectively forever.
• This allows a game compiled in 2024 to run on a distro in 2034, relying on the guarantee that the entry points it expects are immutable.

8. Impact Analysis

Implementing this contract would fundamentally change the value proposition of Linux:

• Commercial Viability: ISVs (Independent Software Vendors) can release a "Linux version" without specifying "Ubuntu 22.04 only." This reduces support costs and encourages native ports over Web/Electron wrappers.
• Reduction of Container Bloat: When the host system can be trusted, the need to containerize simple applications vanishes. This dramatically reduces disk usage and memory pressure (via shared libraries).
• Security: By allowing older applications to link against newer, backward-compatible libraries, legacy apps gain security patches (e.g., in openssl) without needing recompilation. Currently, bundled apps retain their vulnerable, bundled versions of libraries.

9. Trade-offs and Risks

This approach is not free.

• Maintenance Burden: Library maintainers must maintain old code paths. "Move fast and break things" becomes impossible for system libraries.
• Disk Space: Keeping multiple library versions or extensive compatibility shims increases the base OS footprint (though less than the duplication of Flatpaks).
• Testing Complexity: Ensuring a library is truly backward compatible requires rigorous ABI compliance testing tools (like libabigail) integrated into CI pipelines.
• Technical Debt: The OS accumulates "cruft." However, this debt is the price of a stable platform. Windows carries this debt; macOS carries this debt. Linux attempts to default on this debt, passing the cost to the user.

10. Conclusion

Linux has won the cloud and the supercomputer, but its fragmentation prevents it from being a truly durable platform for compiled software distribution. We are currently relying on external patches—containers, Flatpaks, and WINE—to provide the stability that the base system lacks.

This proposal is not a rejection of Linux's modular nature, but an evolution of it. By establishing Dependency and ABI Compatibility as a First-Class Contract, we move from a collection of loosely associated packages to a cohesive Operating System. We must stop viewing breakage as the cost of progress and start viewing compatibility as the definition of engineering excellence.

Addendum: Operational Framework and Strategic Boundaries

The following sections provide necessary structural definitions, scope boundaries, and adoption strategies to operationalize the "Dependency and ABI Compatibility" proposal. These additions are intended to clarify the proposal's intent and prevent common misconceptions regarding implementation.

11. Definitions and Terminology

To ensure cross-distribution alignment, the following terms are defined within the context of this compatibility contract:

• Platform: The aggregate set of kernel interfaces, userspace libraries, and configuration standards that an application relies upon to execute. In this context, a "Platform" is distinct from a "Distribution"; a Platform is a stable target, whereas a Distribution is a delivery mechanism.

• Contract: An explicit, versioned agreement between the Platform and the Application. A contract guarantees that a specific set of symbols, behaviors, and side effects will remain observable and valid for a defined duration (e.g., "LTS+ 10 Years"), regardless of underlying implementation changes.

• ABI (Application Binary Interface) vs. API (Application Programming Interface):

  • API refers to source-level compatibility (requires recompilation).

  • ABI refers to binary-level compatibility (existing binaries continue to function).

  • Focus: This proposal is strictly concerned with ABI stability. Source-level changes are permissible provided binary compatibility is maintained via shims or versioned symbols.

• Capability-Based Resolution: A dependency resolution logic where requirements are satisfied by the presence of a feature set or interface contract (e.g., Provides: Audio-Output-v2) rather than the specific identity of a package file (e.g., libpulseaudio.so.0).

• Compatibility Shim: A lightweight binary translation layer that bridges a requested legacy ABI to a modern implementation. Shims are transparent to the application and incur negligible performance overhead for non-critical paths.

• The Core Compact: A standardized, minimal subset of userspace libraries (glibc, libstdc++, basic graphics/audio/input stacks) designated as "Immutable." These libraries adhere to the strictest level of backward compatibility and form the universal baseline for Linux binary portability.

• Expression vs. Foundation:

  • Foundation: Components that provide hardware abstraction and basic runtime services. Must be stable (Contract-driven).

  • Expression: Components that define the user experience (Desktop Environments, specific application logic). May vary rapidly (Innovation-driven).

12. Explicit Non-Goals

• To prevent strategic misalignment, it is necessary to clearly define what this proposal does not attempt to achieve:

  • Not a Freeze on Innovation: We do not propose halting the development of new libraries or features. New ABIs can be introduced alongside old ones. The goal is accumulation of capabilities, not replacement.

  • Not a Replacement for Containers or Bundles: Docker, Flatpak, and Snap remain vital for complex isolation and specific deployment scenarios. This proposal aims to reduce the necessity of these tools for simple binaries and to make the host system a more reliable foundation for the runtimes these tools depend on.

  • Not a "One Distro" Policy: Diversity in package management (rpm, deb, apk), init systems, and desktop environments is preserved. The compatibility contract applies only to the shared library interfaces exposed to applications, not the system administration layer.

  • Not Universal Stagnation: The "Core Compact" applies only to the foundational layer. Applications and high-level frameworks are free to break compatibility, provided they do not break the underlying system contract for others.

  • Not a Windows Clone: We advocate for stability mechanisms, not Windows architectural patterns (e.g., the Registry). We utilize native Linux mechanisms (ELF, ld.so, symbol versioning) to achieve similar durability.

13. Scope and Responsibility Boundaries

Successful implementation requires a clear division of responsibility between upstream developers, distribution maintainers, and the proposed stability governance.

• In-Scope (The Core Compact):

  Standard C/C++ Runtimes: glibc, libstdc++, libgcc_s.

  System Interfaces: libsystemd (or generic equivalent), libudev.

  Media & Graphics (Loader Layer): Vulkan loader, libGL dispatch, ALSA/PipeWire client libraries, SDL2/3.

  Responsibility: Upstream projects agree to ABI stability; Distributions agree to package legacy shims if upstream breaks ABI.

• Out-of-Scope (The Application Layer):

  High-Level GUI Toolkits: GTK, Qt (except where stable LTS releases are specifically targeted).

  Language-Specific Package Managers: PyPI, npm, Cargo (internal dependency resolution is the app developer's domain).

  Desktop Environments: GNOME Shell, Plasma, Sway compositors.

• Governance:

  The Linux Userspace Standards Group (Proposed): Responsible for defining the "Core Compact" list and certifying compliance.

  Distributions: Responsible for implementing the "Compatibility Context" tooling and maintaining the repository of shims.

14. Compatibility Philosophy Statement

The central philosophy of this proposal is the separation of Infrastructure from Innovation.

• "Stability belongs to contracts; creativity belongs to expressions."

In a mature engineering discipline, the foundation must be boring so that the structure built upon it can be exciting. By forcing the foundation (glibc, loaders, core runtimes) to be constantly "fresh" and breaking, we paradoxically stifle innovation in the application layer. Developers spend significant cycles fighting the platform—constantly refactoring for new dependencies—rather than building features.

A reliable ABI contract allows the operating system to fade into the background. It transforms Linux from a "Project" that requires constant tinkering into a "Product" that serves as a reliable substrate for global computing.

15. Risk of Inaction

Failure to standardize a userspace compatibility contract will accelerate the following negative trends:

• The "Windows as Linux" Paradox: The Proton/WINE compatibility layer is already becoming a more stable target for Linux game developers than native Linux libraries. Without a native contract, the de facto standard Linux binary format will become the Windows PE format running on WINE.

• Container Sprawl: The inability to trust the host system forces every trivial utility to be wrapped in a container, leading to massive duplication of storage, memory, and security auditing overhead.

• Commercial Abandonment: Independent Software Vendors (ISVs) cannot justify the ROI of supporting "Linux" when it effectively means supporting 15 moving targets. They will retreat to Web Apps or Electron, bypassing native integration entirely.

• Erosion of Trust: Users will continue to perceive Linux as a "tinker's OS," unsuitable for long-term deployments where software must run untouched for a decade (e.g., industrial control, medical devices, archival gaming).

16. Adoption Path (Non-Disruptive)

The transition to a Contract-based model should be evolutionary, not revolutionary. We propose a three-phase opt-in model:

Phase 1: Definition and Tooling (Months 1-12)

• Establish the working group to define the "Core Compact" v1.0.

• Develop extensions for rpmbuild and dpkg-deb that automatically generate Capability-based dependencies (Provides: Capability(...)) alongside traditional versioned dependencies.

• No runtime changes; purely metadata generation.

Phase 2: The "LTS+" Repository (Months 12-24)

• Distributions introduce an optional "Compatibility" repository containing shims for the Core Compact.

• ISVs can voluntarily target the "Linux Core Compact 1.0" specification.

• Launch a compliance tool (similar to the Windows App Certification Kit) that verifies an application only links against Core Compact symbols.

Phase 3: Native Integration (Month 24+)

• Distributions begin to enable Capability-based resolution by default.

• The dynamic linker is patched to support "Shim Injection" for transparent backward compatibility.

• "Core Compact" compliance becomes a standard badge of quality for native Linux applications.

This path allows the ecosystem to gradually absorb the cost of stability without breaking existing "rolling" or "bleeding edge" workflows.