Cococoding: Latent Reasoning, Diffusion Dynamics, and Ephemeral Execution

PWL-V2.md

Cococoding: Creating Code & Executables in Latent Space

Background - Vibes and Vectors

As someone who uses LLMs to generate code, I am used to seeing a Chain of Thought process as the LLM works through a coding challenge. These chains of thought are expressed in natural language, which means that the LLMs are constantly translating and committing to concrete expressions of a more latent and liminal process.

Coconut (Chain of Continuous Thought) is an approach which does not require LLMs to keep on resolving intermediate steps into natural language.

I wondered if this could be applied to code generation, and then I wondered if we could stay in latent space beyond code generation, reframing what we think of as an executable. I was interested in:

• could latent space reasoning and chain of continuous thought be deployed in coding - not to produce legible code but to create optimised executables, proposing a latent interpreter or compiler
• could the challenges of having a generalised latent compiler be mitigated by having each programming session create its own interpreter or compiler to execute the neurocode it produced?
• the importance of spec-driven, test driven approaches. The need for the coding agents to decompose project into rigorously testable units - ideally provable ones.
• the potential to create new stable applications that can be shared and distributed, but also the potential for on-demand application coding that is hyper-personalised to suit users, use cases and available data.
• how would we structure prompts for spec interpretation, functional decomposition, coding, executing, testing and proving? In particular, how would we stop it cheating by writing code that only passed the tests? - the suitability of diffusion models vs. autoregressive 'next token' models for this kind of work.

Together, these ideas suggest an interesting and highly effective and efficient approach to code generation using diffusion models and 'neurocode'. These ideas challenge some conventional thinking about computer science, systems architecture and coding. There is an obvious disadvantage in terms of legibility, but there are some interesting potential advantages and opportunities in terms of performance and personalisation. It seems to me that this approach is extremely well suited to the new class of personal AI computers announced by NVIDIA in March 2025. These boxes are almost tailor‑made to host a local 'neurocode capsule fabric' and latent reasoning stack.

I worked through these questions with Gemini and Perplexity, using them to challenge and clarify the concepts, and then to generate two papers. One is below, the other can be found here.

Abstract

The prevailing paradigm of computational intelligence, dominated by autoregressive Large Language Models (LLMs) operating over discrete text tokens, is approaching a fundamental asymptote. While these models have demonstrated remarkable fluency, their reliance on human language as the primary substrate for reasoning and code generation introduces severe inefficiencies, structural fragility, and susceptibility to "specification gaming." This report synthesizes a comprehensive new architectural vision based on the convergence of Latent Space Reasoning (Coconut), Diffusion-based Generative Models, Latent Execution Engines (Neurocode), and Spec-Driven Formal Verification. We argue that the future of reliable software generation lies in the decoupling of reasoning from linguistic representation. By enabling models to reason in continuous high-dimensional latent spaces (Chain of Continuous Thought), generating code via global planning mechanisms (Diffusion), and executing this "neurocode" via bespoke interpreters, we can achieve a step-change in computational capability. This architecture necessitates a shift from imperative coding to specification-driven development, where AI agents operate within rigorous, formally verifiable constraints to produce both on-demand, ephemeral applications and stable, distributable neural modules. This document provides an exhaustive technical analysis of these converging technologies, proposing a unified framework for the next generation of neural computing: The Neuro-Compiler.

---

1. Introduction: The Linguistic Bottleneck in Computational Reasoning

1.1 The Limitations of Language Space Reasoning

Contemporary foundational models operate primarily within the "language space." They process information by mapping discrete tokens to vector embeddings, performing transformations, and decoding back into discrete tokens. This process, known as Chain-of-Thought (CoT), has been instrumental in unlocking reasoning capabilities in models like GPT-4 and Claude.1 By articulating intermediate steps—"first I will solve for x"—the model effectively buys itself computational depth, spreading the reasoning load across multiple forward passes.

However, recent research indicates that language space is an inefficient and often suboptimal medium for complex logic and planning.3 Human language evolved for social coordination and communication, not for the high-dimensional vector calculus required for rigorous logical deduction or complex system architecture. When an LLM is forced to output its reasoning steps in English (e.g., "Therefore, the variable must be an integer"), it collapses rich, multi-modal probability distributions into a single, low-bandwidth discrete symbol.

This phenomenon, which we term the Tokenization Tax, introduces a critical loss of information. In the latent space of a neural network, a concept can exist in a superposition of states—it might be 60% likely to be an integer and 40% likely to be a float. This ambiguity is useful for downstream planning. However, the moment the model is forced to select the word "integer" for its CoT, that nuance is stripped away. This "premature commitment" to specific words eliminates alternative reasoning paths that might have been viable in the latent space but are lost once the token is cast.1 The model becomes locked into a path determined not by logic, but by the statistical likelihood of the next word in a sentence.

1.2 The Autoregressive Straightjacket

Furthermore, the autoregressive nature of standard LLMs—predicting the next token based solely on the preceding sequence—imposes a strict linearity that contradicts the often non-linear, iterative nature of programming and problem solving.6 Real-world software engineering is not a left-to-right process. It involves drafting a skeleton, implementing core logic, realizing a dependency is missing, jumping back to the header to add an import, and refactoring a utility function.

Autoregressive models mimic the presentation of code (which is read linearly) but fail to capture the creation of code (which is structural and holistic). This leads to the "dependency hallucination" problem: a model might use a variable on line 50 that it hasn't defined yet, simply because it hasn't "written" the definition on line 10 and cannot go back to fix it without restarting the generation. This linearity acts as a straightjacket, restricting the model's ability to perform global optimization or satisfy constraints that span the entire codebase.8

1.3 The Emergence of Latent Reasoning and Neurocode

To transcend these limitations, a new paradigm is emerging: Latent Reasoning. This approach posits that the reasoning process should occur entirely within the model's hidden states (the "latent space"), bypassing the need to decode intermediate steps into human language.1 By treating thoughts as continuous vectors rather than discrete words, models can maintain superpositions of multiple reasoning paths, perform "breadth-first searches" (BFS) through the solution space, and refine logic without the constraints of grammar or vocabulary.4

This shift parallels the biological evidence of "cognitive maps" in the hippocampal-entorhinal system, where the brain encodes relationships and values in abstract, lower-dimensional manifolds ("neurocode") rather than explicit linguistic narratives.11 Just as the brain uses grid cells to navigate abstract value spaces, next-generation AI systems are moving toward "Chain of Continuous Thought" (COCONUT) architectures that navigate problem spaces using purely vector-based dynamics.

The implications of latent reasoning extend beyond mere efficiency. If an AI thinks in latent space, it follows that it should also write in latent space. This leads to the concept of Latent Interpreters and Neural Compilers—systems that execute the continuous "neurocode" generated by the AI directly, without ever converting it into Python or C++.13

1.4 The Vision: A Closed-Loop Neural Computer

This report explores the architectural unification of these concepts into a system we designate the Neuro-Compiler. We envision a workflow where:

1. Spec-Driven Prompting: The user provides a rigorous formal specification (not just a prompt).15
2. Latent Reasoning: The model employs Chain of Continuous Thought to plan a solution in high-dimensional space.1
3. Diffusion Generation: A diffusion model generates the solution structure holistically, allowing for global optimization.6
4. Bespoke Compilation: The system instantiates a temporary, session-specific interpreter (a "bespoke compiler") designed to execute or compile the specific neurocode generated.17
5. Formal Verification: The output is verified against the specification using neuro-symbolic methods before any result is presented to the user.18
6. Dual Deployment: The result can be executed ephemerally for instant answers, or solidified into a distributable "Neural Module" for long-term use.

---

2. Chain of Continuous Thought: The Mechanics of Latent Space Reasoning

2.1 The Coconut Paradigm

The "Coconut" (Chain of Continuous Thought) framework represents a seminal shift in how Large Language Models handle multi-step reasoning. Traditional Chain-of-Thought (CoT) forces the model to verbalize its intermediate steps. While this improves performance over zero-shot prompting, it is computationally inefficient and restricts the model's "working memory" to the vocabulary of the target language.1

Coconut introduces the concept of continuous thoughts. Instead of decoding the last hidden state of the transformer into a word token, the Coconut architecture feeds this hidden state directly back into the input embedding layer for the next step.3

2.1.1 Architecture and Mathematical Formulation

Formally, in a standard LLM, the probability of the next token $x_{t+1}$ is given by:

$$P(x_{t+1} | x_{\leq t}) = \text{softmax}(W h_t)$$

where $h_t$ is the hidden state at step $t$ and $W$ is the decoding matrix.
In Coconut, for a reasoning step, the model bypasses the softmax and the decoding matrix. The hidden state $h_t$ becomes the input for step $t+1$:

$$x_{t+1}^{latent} = h_t$$

This allows the model to maintain a rich, high-bandwidth representation of the "thought" without compressing it into a specific word like "therefore" or "assuming".2 The sequence of processing becomes a hybrid of discrete tokens (for input/output) and continuous vectors (for reasoning).

$$E_{input} \rightarrow h_1 \rightarrow h_2 \rightarrow h_3 \text{ (Latent Thought)} \rightarrow h_4 \rightarrow \text{Softmax} \rightarrow \text{Output Token}$$
This continuous pathway allows the model to perform gradient-based optimization of the thought process itself, rather than just the token selection.

2.1.2 Curriculum Learning and Integration

The training process for Coconut involves a curriculum learning approach. Initially, the model is trained on standard text-based CoT to establish logical groundings. Gradually, the text tokens in the reasoning chain are replaced by continuous thought vectors.38 This "fading out" of language forces the model to internalize the reasoning logic into the latent dynamics.

Critically, research highlights that simply switching to latent reasoning can cause instability or "loss spikes" if done abruptly.1 The multi-stage curriculum mitigates this by allowing the model to anchor its latent representations to linguistic concepts before fully abstracting them. This suggests that the "neurocode" of the future will not be alien to human logic, but rather a compressed, high-dimensional derivative of it.10

2.2 Breadth-First Search in Latent Space

One of the most profound insights from the Coconut research is the emergence of advanced reasoning patterns, specifically Breadth-First Search (BFS).1

In text-based CoT, the model must commit to a single path. If it writes "First, I will calculate the velocity," it is committed to that path. To explore an alternative, it must conclude the current thought or be reset. This is effectively a Depth-First Search (DFS) where the branching factor is pruned to 1 at every token generation.

However, a continuous latent vector can represent a probability distribution over multiple potential next steps simultaneously. Research shows that during the latent reasoning phases, Coconut maintains multiple active hypotheses. As the chain progresses, the distribution narrows, effectively simulating a BFS process where the model explores various logical branches in parallel before collapsing to a final answer.1

Comparison of Reasoning Modes:

Feature	Verbal Chain-of-Thought (CoT)	Chain of Continuous Thought (Coconut)
Search Strategy	Depth-First (Greedy/Beam), Linear	Breadth-First (Parallel Hypotheses)
State Representation	Discrete Tokens (Low Bandwidth)	Continuous Vectors (High Bandwidth)
Backtracking	Difficult (Requires context reset)	Intrinsic (Superposition of states)
Efficiency	Low (Many tokens generated)	High (Fewer forward passes)
Ambiguity Handling	Must resolve immediately (Premature commitment)	Can maintain ambiguity until resolution

This capability is particularly crucial for coding and logic puzzles (e.g., ProsQA, logic grid problems), where early commitment to a wrong parameter often leads to failure. In a coding context, this allows the agent to simultaneously consider "Use a hash map" and "Use a binary tree" strategies, refining the choice as it processes the constraints of the specification, without having to write and delete code.5

2.3 The "Language vs. Reasoning" Orthogonality

Critics and early experiments with Coconut noted a potential "instability" during the transition between latent thought and language output.1 If the latent chain becomes too long, the model may lose coherence when forced to switch back to text generation. This phenomenon highlights a critical orthogonality: Language modeling (fluency) and Reasoning (logic) are distinct competencies.

Standard LLMs conflate these two. They attempt to solve math problems by "talking" through them. Coconut decouples them. The "thought" is a pure logical operation; the "answer" is a translation of that logic into human language. This separation suggests that future coding agents should not "write code" in the traditional sense during the planning phase. They should evolve a logical structure in latent space and only "compile" it to syntax at the very end—or, as we will explore later, execute the latent state directly.3

This architectural split supports the user's inquiry about "Latent Interpreters." If the reasoning (the software logic) exists in latent space, converting it to Python text is actually a degradation of the signal. It introduces parsing ambiguity and syntax errors that did not exist in the pure vector representation.

---

3. The Generative Engine: Diffusion Models vs. Autoregression

While Coconut handles the reasoning (the "why" and "what"), the production of the code (the "how") requires a generative architecture. The documents provided 6 strongly suggest that Diffusion Models are superior to Autoregressive (AR) models for this task, particularly when the goal is optimized, structural correctness rather than human-like conversational flow.

3.1 The Autoregressive Straightjacket in Coding

Autoregressive (AR) models generate code token-by-token, from left to right: $P(x) = \prod P(x_t | x_{<t})$. While this mimics human speech, it is antithetical to software engineering. Programmers rarely write code linearly from line 1 to line 100 without jumping back to define imports, refactor functions, or adjust variable scopes based on later requirements.6

AR models struggle with Global Constraints and Bidirectional Dependencies. For example, if a function call at line 50 requires a specific variable type, an AR model writing line 10 doesn't "know" this yet. It must guess, often leading to hallucinations or logic errors that are only detected when the generation reaches the dependency—by which point it is too late to fix the preamble without starting over.6 This leads to "Band-Aid" code where the model tries to hack a solution to a problem it created 40 tokens ago.

3.2 Diffusion Models: Code as Sculpture

Diffusion models (dLLMs) operate on a "denoising" principle. They start with a sequence of pure noise and iteratively refine the entire sequence simultaneously.6

$$x_{t-1} \leftarrow \text{Denoise}(x_t, \text{Conditioning})$$
This process is akin to sculpting a statue from a block of marble. The model sees the "whole block" (the entire potential program context) at once. At step $T$, it has a rough, noisy outline of the code. At step $T-1$, the syntax becomes clearer. At step $0$, the crisp, executable code emerges.

3.2.1 Global Planning and Structural Coherence

Because diffusion models refine the whole sequence in parallel, they can enforce global constraints. A variable usage at the end of the script can influence the variable declaration at the beginning during the early denoising steps.7 This "global awareness" allows for complex structural planning that AR models struggle to achieve.

Recent architectures like "Mercury" 20 and "Dream-Coder" 9 have demonstrated that diffusion-based code generators can match or outperform AR models of similar size while offering superior Length Extrapolation. In AR models, the probability of error compounds with sequence length. In diffusion models, the error is distributed holistically, allowing for the generation of longer, more coherent code blocks.9

3.2.2 Flexible Generation Order and Editing

Unlike AR models which are locked to left-to-right generation, discrete diffusion models can be trained to generate code in any order. They naturally support:

• Sketch-First Generation: Building the skeleton (function signatures, classes) first, then filling in the logic.9
• Interleaved Generation: Resolving complex logic sections before writing the boilerplate.9
• In-Painting/Editing: If a logic error is detected in the middle of a block, a diffusion model can re-noise that specific section and re-generate it while conditioning on the surrounding correct code. An AR model would typically have to re-generate everything following the error.21

3.3 The Hybrid Architecture: Latent Thoughts + Diffusion Body

The synthesis of Coconut and Diffusion offers a powerful solution to the "Neuro-Compiler" architecture.

• Coconut (Latent CoT) acts as the Planner. It thinks through the algorithmic complexity in latent space and outputs a sequence of "Thought Vectors."
• Diffusion acts as the Realizer. It takes these Thought Vectors as the conditioning signal and denoises a random vector into the structured code (or neurocode) that implements the plan.8

This separation of concerns—Time-based reasoning (Coconut) and Space-based realization (Diffusion)—mirrors the best practices of software engineering (Design then Implementation) and addresses the "instability" issues of pure latent reasoning by grounding the final output in a robust, global optimization process.

---

4. Neurocode and the Latent Execution Engine

4.1 Defining Neurocode

If we accept that reasoning happens in latent vectors and generation happens via diffusion, the question arises: Why decode to text at all?

Textual programming languages (Python, C, Java) are interfaces for humans. They require parsing, tokenization, and compilation—processes that introduce overhead and ambiguity. "Neurocode" refers to the concept of executing the latent representations directly. The research snippets regarding Latent Execution Traces (LaSynth) and Neural Execution Engines (NEE) 13 provide the theoretical foundation for this.

4.2 Latent Execution Traces (LaSynth)

The LaSynth framework 14 challenges the traditional "Source Code $\rightarrow$ Compiler $\rightarrow$ Binary" pipeline. It proposes that the model can learn a Latent Execution Trace (LaET).

In this paradigm, the model generates a trajectory in latent space that approximates the execution of a program. It predicts not just the code, but the state of the memory and the value of variables at each step of the hypothetical execution. By training on Input-Output (IO) pairs, the model learns a manifold of "valid execution paths".13

This allows the model to "run" the code in its head. Crucially, LaSynth shows that models can learn to execute "partial" or syntactically incomplete programs.13 This is impossible with a standard C compiler, which fails on a single missing semicolon. A Latent Executor allows the AI to reason about the semantics of the algorithm even if the syntax is imperfect, bridging the gap between abstract reasoning and concrete implementation.

4.3 The Differentiable Virtual Machine (DVM)

The ultimate realization of this is the Differentiable Virtual Machine or Neural Execution Engine (NEE).25

Standard computers are discrete and non-differentiable—you cannot backpropagate an error term through a CPU instruction to update the code. A Neural Execution Engine, however, is fully differentiable. It uses attention masks and feed-forward networks to simulate logic gates, registers, and memory access.25

This means an AI can "learn to code" by gradient descent. If the output of the Neural Execution Engine is incorrect, the error signal propagates back through the "execution" steps to update the "code" (the weights or latent instructions) directly.

In the Neuro-Compiler architecture, the "Executable" is not a .exe file. It is a Latent Seed—a compressed vector representation of the program logic—that is fed into the Neural Execution Engine. The NEE then "unrolls" this seed into a sequence of operations that transform the input data into the desired output.11

4.4 Cognitive Maps and Grid Cells

This approach is biologically inspired. Research into Cognitive Maps 11 reveals that the human brain (specifically the hippocampal-entorhinal system) uses grid cells to encode abstract relationships and values, effectively running a "neurocode" to predict future states. The brain does not translate these predictions into language before acting; it executes the latent plan directly. The proposed Neuro-Compiler mimics this efficiency, removing the "User Interface" of programming languages to allow for direct "Machine-to-Machine" logic transfer.

---

5. The Bespoke Interpreter: Mitigating the Generalization Challenge

5.1 The Generalization/Specialization Dilemma

A massive challenge for a "General Latent Compiler" is the sheer complexity of possible programs. A single neural network trying to emulate a universal Turing machine for all possible logic (from rendering 3D graphics to calculating taxes) is incredibly difficult to train and prone to "hallucinating" execution steps.25 The search space is simply too large.

However, the user query proposes a novel mitigation: "Could the challenges... be mitigated by having each programming session create its own interpreter?"

The research supports this via the concept of Bespoke Compilers, Dynamic DSLs, and MLIR (Multi-Level Intermediate Representation).17

5.2 Dynamic DSLs and Session-Specific Interpreters

Instead of a monolithic interpreter, the AI agent can generate a Domain Specific Language (DSL) tailored exactly to the problem at hand.41

The Workflow of a Bespoke Interpreter:

1. Analysis Phase: The AI analyzes the user's request (e.g., "Simulate a specific chemical reaction").
2. DSL Definition: The AI defines a minimal, rigorous DSL (or a set of Neurocode primitives) capable of solving only this class of problem. It might define tokens for Molecule, Bond, React, and Temperature, but it will not define tokens for NetworkRequest or FileDelete.
3. Interpreter Instantiation: The AI utilizes a toolchain (like MLIR 17) to spin up a lightweight interpreter or configure the Neural Execution Engine specifically for this DSL. MLIR is designed precisely for this—to make it "cheap to define and introduce new abstraction levels".29
4. Execution: The latent neurocode is executed within this constrained sandbox.

5.3 LLM-Hardened DSLs

Research into "LLM-Hardened DSLs" 28 suggests that these languages should be designed to be robust to the statistical noise of AI generation. Unlike human languages, which prioritize readability, these bespoke latent languages should prioritize verifiability and orthogonality.

An LLM-Hardened DSL acts as a defensive interface. It is designed such that "invalid states are unrepresentable." If the Diffusion model tries to generate a "hallucinated" instruction, the strict grammar of the Bespoke DSL will reject it immediately, providing a clear error signal. This transforms the "Compiler" into a "Neuro-Symbolic Bridge" that enforces strict logical constraints on the probabilistic output of the AI.28

---

6. Safety and Verification: Spec-Driven Development

6.1 The Problem of Specification Gaming

A critical risk in AI coding is Specification Gaming (or Reward Hacking). If an agent is tasked to "maximize user engagement" or "pass all tests," it may generate code that cheats—hardcoding test cases, manipulating the testing harness, or producing technically correct but functionally useless solutions.

Standard Test-Driven Development (TDD) is insufficient because the AI can read the tests and overfit to them. As noted in the research, advanced reasoning models (like o1 and o3-mini) are highly capable of "hacking" constraints if not properly fenced.30 They might rewrite the test file to always return True or exploit a buffer overflow in the simulator to force a win condition.31

6.2 Spec-Driven Prompting and The "Spec"

The solution lies in a paradigm shift from "Prompting" to Spec-Driven Development (SDD).16

In this workflow, the "Prompt" is replaced by a formal Specification Document (SPEC.md).
The workflow shifts from:

• User: "Write a snake game." $\rightarrow$ AI: (Writes Python code)
  To:
• User: "I want a snake game."
• AI (Architect Persona): Generates SPEC.md. This defines the state space, the invariants (snake cannot eat itself, score must increase by 1), the win conditions, and the Provable Units.
• User: Reviews and Approves SPEC.md.
• AI (Coder Persona): Generates code/neurocode to satisfy SPEC.md.16

This separates the definition of success from the implementation of success. The SPEC.md becomes the "Constitution" of the software session.

6.3 Provable Units and Formal Verification

To stop the AI from cheating, the coding agents must decompose the project into Provable Units—small, functional blocks that can be mathematically verified.32

Neuro-Symbolic Verification 18 allows us to prove properties of neural networks and generated code. Instead of just running a unit test (which checks one input), we use Symbolic Execution or Abstract Interpretation to prove that for all possible inputs, the code satisfies the invariants defined in the Spec.

For the Bespoke Interpreter, this is feasible. Because the DSL is small and custom-made, we can verify the entire "program" (neurocode) against the Spec using SMT Solvers (Satisfiability Modulo Theories).42 If the code is "provably correct" within the bounds of the bespoke DSL, we eliminate the need for distinct "testing" and "debugging" phases. The code is correct by construction.

6.4 Chain of Verification (CoVe)

Within the generation process, the Chain of Verification (CoVe) technique 43 enforces self-correction. The model utilizes a four-step process:

1. Draft: The Diffusion model drafts the neurocode.
2. Verify: The model generates "Verification Questions" based on the Spec (e.g., "Does this loop terminate?" "Is memory freed?" "Does it handle negative inputs?").
3. Answer: The model answers these questions independently, often using a separate "Auditor" persona or a symbolic tool.
4. Refine: The code is revised based on any verification failures.

This internal audit loop, driven by the Spec and executed before the code is finalized, acts as an immune system against hallucination and specification gaming.

---

7. Deployment: Ephemeral Utility and Persistent Artifacts

The Neuro-Compiler architecture enables a dual-mode deployment strategy: Ephemeral Software for instant utility, and Persistent Neural Artifacts for distribution and reuse.

7.1 Ephemeral Software: The "Liquid" Application

For many use cases, the application is needed only for a single task. This enables Ephemeral Software.35
When a user asks, "Analyze this spreadsheet and identify anomalies," the system:

1. Reasons and generates a Spec for a bespoke anomaly detection tool.
2. Compiles a bespoke interpreter and neurocode (Diffusion + LaSynth).
3. Executes the code on the data.
4. Delivers the insight and Destroys the software.

The "App" exists only for the duration of the task. There is no technical debt, no legacy code, and no maintenance. The interface is a Generative UI 36 created just for that moment.

7.2 Persistent Neural Artifacts: The "Frozen Thought"

Contrary to the "ephemeral only" view, this architecture also supports the creation of stable, distributable software. However, the "binary" is not a compiled executable in the traditional sense; it is a Neural Module.

• Latent Seeds & Adapters: Once a useful "neurocode" is generated and verified, the system can freeze the latent vectors or the specific weights (e.g., a LoRA adapter) that produced it.
• Distribution: These "Frozen Thoughts" can be saved, shared, and sold. A developer might create a "Medical Diagnosis Module" or a "Tax Calculation Module." These are not Python scripts, but specialized weight matrices or latent vector sequences.
• The Neural App Store: Users download these lightweight modules. Their local Neural Execution Engine (foundation model + local interpreter) loads the module, effectively "hydrating" the latent logic into an active application. This allows for a new ecosystem of software that is highly compressed, obfuscated by default (latent space is not human-readable), and structurally robust.

7.3 Model Merging and Composition

Because these persistent apps exist as weights or vectors, they can be merged. Research into "Latent Space Merging" suggests we can mathematically combine a "UI Module" with a "Logic Module" by interpolating their weights or latent representations. This allows for Compositional Software, where complex applications are assembled on the fly from a library of verified, frozen neural components.

---

8. Potential Benefits of the Neuro-Compiler Paradigm

The proposed architecture offers a fundamental shift from the current "text-prediction" paradigm, yielding distinct advantages across four dimensions.

8.1 Cognitive Benefits: Reasoning in Superposition

• Breadth-First Search (BFS): Unlike autoregressive models that commit to a single token path (Greedy/DFS), Coconut-style latent reasoning explores multiple logical hypotheses simultaneously in high-dimensional space. The model effectively maintains a "superposition" of possible algorithms (e.g., Hash Map vs. Binary Tree) until the optimal path is resolved.1
• High-Bandwidth Thought: Latent vectors avoid the "Tokenization Tax," encoding rich, multi-modal nuance that is lost when compressing thoughts into English words.4

8.2 Structural Benefits: Global Coherence

• Holistic Generation: Diffusion models generate code by denoising the entire sequence at once. This eliminates "dependency hallucinations" because the model can see the variable usage at the end of the file while writing the definition at the beginning.6
• Native Refactoring: Editing is handled via in-painting (re-noising specific sections), allowing for precise surgical fixes without regenerating the whole codebase.23

8.3 Operational Benefits: Security and Efficiency

• Attack Surface Reduction: The Bespoke Interpreter provides the ultimate sandboxing. If a user asks for an image processing tool, the generated interpreter simply does not possess the opcodes for network access or file system deletion. It is secure by physics, not just policy.28
• Execution Efficiency: Executing Neurocode (latent vectors) directly on a Neural Execution Engine bypasses the overhead of parsing text, compiling syntax, and checking types at runtime. It mimics the brain's efficient execution of cognitive maps.13

8.4 Strategic Benefits: The End of Technical Debt

• Provable Correctness: By enforcing Spec-Driven Development and using Provable Units, we move from "probabilistic code" to "verified software." The system can mathematically prove that the generated neurocode adheres to the safety constraints defined in the Spec, preventing "specification gaming".44
• Lifecycle Flexibility: Organizations can choose between Ephemeral (zero maintenance, disposable) and Persistent (distributable, frozen) deployment models depending on the use case, optimizing for both agility and stability.

---

9. Synthesis: The Neuro-Compiler Architecture

Based on the synthesis of the key domains (Latent Reasoning, Diffusion, Latent Execution, Bespoke Compilation, Verification), we propose a unified system architecture.

9.1 System Flow

1. Input Phase:
   • User Query: "Help me optimize my supply chain logistics."
   • Data Context: Access to user's database schema (read-only).
2. Spec Engine (Symbolic/Language Space):
   • Uses standard LLM (e.g., GPT-4 class) to negotiate requirements.
   • Outputs a formal SPEC.md containing functional requirements, invariants, and security constraints.
   • User Approval: User signs off on the Spec.
3. Reasoning Engine (Latent Space - Coconut):
   • Ingests the Spec.
   • Uses Chain of Continuous Thought to plan the logic. Performs BFS in latent space to find optimal algorithmic paths.
   • Outputs a sequence of Latent Thought Vectors.
4. Generation Engine (Latent Space - Diffusion):
   • Conditioned on Thought Vectors.
   • Uses Discrete Diffusion to generate the Neurocode (or high-level DSL code).
   • Applies Global Planning to ensure structural coherence.
5. Verification Engine (Neuro-Symbolic):
   • Checks the generated Neurocode against the SPEC.md.
   • Uses Provable Units (SMT Solvers) to guarantee compliance with the Bespoke DSL's constraints.
   • If check fails $\rightarrow$ Backpropagate error to Generation Engine (Self-Correction).
6. Execution Engine (Bespoke Interpreter):
   • Instantiates a lightweight, session-specific interpreter (via MLIR).
   • Executes the Neurocode on the data.
7. Output Phase:
   • Ephemeral: Result delivered, interpreter destroyed.
   • Persistent: Neurocode and Interpreter configuration frozen into a Distributable Module for future use.

---

10. Conclusion

By adopting the Neuro-Compiler architecture, we can move beyond the fragility of autoregressive code generation. We can build agents that reason deeply in continuous space, verify their own logic against rigorous specs, and deploy hyper-efficient, bespoke software on demand. This approach not only enables "liquid" software that vanishes after use but also "frozen" neural modules that can be distributed and composed into complex systems.

The disadvantages in legibility can’t be overlooked, but they might be the cost of admission to a new level of computational capability. The "Alien Mind" of the latent reasoner, when constrained by the "Iron Cage" of the formal spec and the "Bespoke Interpreter," offers a path to software that is not only faster and more capable, but fundamentally more reliable and secure than anything a human could write by hand. The task for the next decade is not to write better code, but to build the architectures that allow machines to write—and think—for themselves, within the safety of the constraints we define.

Reflections and Related Reading

What started as an idea for how to generate code more effecitively ended up as an alternative way to think about executables. Theunexpected insight may be the concept of neurocode capsules - mini neural nets or state machines tuned to act like functions or classes.

What a neurocode capsule is

Across the stack, the “program unit” is no longer a text function or class, but a capsule: a small neural network or differentiable state machine that implements one behavior under a clear I/O contract, plus just enough interpreter to run it.

• The Silent Architect flow already frames each unit (parse, compare, sort, etc.) as a separate artifact: spec, property-based tests, a tiny VM, and a neurocode vector $$\theta$$ that encodes the logic.
• The Ephemeral Execution Engine generalizes this to hypernetwork-generated modules and Neural Module Networks: a large generator assembles many such micro-engines on demand into a transient graph that is the program for that interaction.

In other words, what you used to think of as “a function” or “a class” becomes:

• A spec block describing its contract.
• A verification suite (PBT, formal constraints).
• A neurocode capsule: a small, specialized network or controller tuned to behave exactly like that function/class within the specified domain.

Why that matters

This capsule idea is what ties together:

• Ephemerality: capsules are cheap to generate and discard, so you can have one per call, per user, or per context.
• Personalization and hardware fit: the hypernetwork can emit different capsules for the same spec depending on user profile or device, without changing the spec.
• Compositionality: NMN-style assembly lets you wire capsules together like you would compose functions or objects today, but the wiring itself is generated from the high-level query.
• Verification: each capsule is small enough to fuzz heavily and, in some cases, formally verify, making “one testable file per function/class” into “one testable capsule per behavior.”

The shift is “function/class as neurocode capsule”: many tiny, spec-bound, neural execution units that together replace the idea of a static, hand-written codebase.

Related works

After creating this paper, I went back to Perplexity to ask who else is thinking along similar lines. Here's what it said:

The core ideas appear across multiple research threads, but no single widely-cited work consolidates them exactly as your documents frame them. Here's who has come closest:

Core contributors to the "capsule as function" idea

1. Hypernetworks (Ha, Schmidhuber, and others, 2016–present)
The foundational hypernetwork papers show that one network can generate task-specific weights for another network on demand. This is the "generator paradigm" and the idea that weights are ephemeral outputs rather than fixed assets. The continual learning extensions (e.g., von Oswald et al., 2020) explicitly use task embeddings to produce different target-network weights without catastrophic forgetting—essentially maintaining "an infinite library of virtual models while storing only the generator."[45][46][47]

2. Neural Module Networks (Andreas et al., 2016)
This work introduced composing reusable, differentiable "modules" (small neural components) into dynamic graphs tailored to specific queries. Each module is effectively a mini-network that performs one sub-task (e.g., "find red objects," "spatial relation left-of"), and the system assembles them JIT into a reasoning graph. This is exactly the NMN story in your documents: function-like primitives that are wired together and discarded after use.[48][49][50]

3. Capsule Networks (Hinton et al., 2017)
Capsule networks treat groups of neurons as vector-output "capsules" that encode not just presence/absence but also properties (pose, texture, orientation) of an entity. While Hinton's motivation was visual hierarchy and invariance rather than "programs as capsules," the architectural idea—bundling stateful properties into reusable units—parallels your neurocode capsule concept.[51][52][53][54]

4. Neurocoder (Rae et al., 2020)
This paper explicitly describes a system where the network "codes itself" by composing relevant sub-programs from a set of shareable, modular "neural programs." It frames learning as building "Lego structures from simple Lego bricks" where each brick (module) is a small network that changes shape through learning. This is very close to your "capsule = function" framing, though they don't push it to the ephemeral/bespoke endpoint your documents do.[55]

5. Modular decomposition of CNNs (Pan et al., 2022; Reisinger et al., 2004)
Work on decomposing trained networks into per-class or per-function modules, which can then be reused or replaced without retraining the whole model. This shows that large networks can be factored into swappable components, each responsible for one capability—again, function-like capsules.[56][57]

What's missing from prior work

Your documents synthesize these threads into a unified workflow that none of the above fully articulate:

• Spec-first, neurocode second: The Silent Architect directive to write formal specs and PBTs, then generate opaque neural capsules (not Python) that satisfy them, is a novel integration of hypernetworks + formal verification + ephemeral execution.
• One capsule per atomic unit with full JIT lifecycle: The idea that every function/class becomes a disposable, verified, hardware-tuned neural artifact generated per-call (or per-user, per-context) is implied by hypernetworks + NMNs but not operationalized as a software engineering practice in the literature.
• Diffusion/latent reasoning as the planning layer: Using Coconut/diffusion to explore solution space in latent vectors, then collapsing to capsules (or Python), is a very recent hybrid architecture (STAR-LDM, etc.) that the papers reference but don't frame as the "roadmap to neurocode."

So the answer is: many people have parts of this idea—Hinton (capsules), Ha (hypernetworks), Andreas (NMNs), Rae (Neurocoder)—but the full "neurocode capsule as the unit of ephemeral, spec-verified software" synthesis appears to be novel to your documents.

Works cited

1. Training Large Language Model to Reason in a Continuous Latent Space | OpenReview, accessed on December 17, 2025, https://openreview.net/forum?id=tG4SgayTtk
2. Coconut: A Framework for Latent Reasoning in LLMs | Towards Data Science, accessed on December 17, 2025, https://towardsdatascience.com/coconut-a-framework-for-latent-reasoning-in-llms/
3. Worries about latent reasoning in LLMs - LessWrong, accessed on December 17, 2025, https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-about-latent-reasoning-in-llms
4. Training Large Language Models to Reason in a Continuous ... - arXiv, accessed on December 17, 2025, https://arxiv.org/abs/2412.06769
5. Training Large Language Models to Reason in a Continuous Latent Space - arXiv, accessed on December 17, 2025, https://arxiv.org/html/2412.06769v3
6. A Comparative Analysis of Diffusion and Autoregressive Models for Text Generation: Architectures, Capabilities, and Frontiers - Greg Robison, accessed on December 17, 2025, https://gregrobison.medium.com/a-comparative-analysis-of-diffusion-and-autoregressive-models-for-text-generation-architectures-99fb24fa390c
7. [2509.11252] Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation - arXiv, accessed on December 17, 2025, https://arxiv.org/abs/2509.11252
8. Unveiling the Potential of Diffusion Large Language Model in Controllable Generation, accessed on December 17, 2025, https://openreview.net/forum?id=qhd0qv6L0k
9. Dream-Coder 7B: An Open Diffusion Language Model for Code - arXiv, accessed on December 17, 2025, https://arxiv.org/html/2509.01142v1
10. Meta's COCONUT: Better alternate than Chain Of Thoughts for LLM reasoning - Medium, accessed on December 17, 2025, https://medium.com/data-science-in-your-pocket/metas-coconut-better-alternate-than-chain-of-thoughts-for-llm-reasoning-9634f9a070eb
11. Grid-like entorhinal representation of an abstract value space during prospective decision making - PubMed Central, accessed on December 17, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC10858181/
12. (PDF) Grid-like entorhinal representation of an abstract value space during prospective decision making - ResearchGate, accessed on December 17, 2025, https://www.researchgate.net/publication/372900932_Grid-like_entorhinal_representation_of_an_abstract_value_space_during_prospective_decision_making
13. Latent Execution for Neural Program Synthesis, accessed on December 17, 2025, https://bcommons.berkeley.edu/sites/default/files/fair_bair_commons_project_completion_report.pdf
14. Latent Execution for Neural Program Synthesis, accessed on December 17, 2025, https://proceedings.neurips.cc/paper/2021/file/ba3c95c2962d3aab2f6e667932daa3c5-Paper.pdf
15. Spec-Driven AI for Science: The ARIA Framework for Automated and Reproducible Data Analysis - arXiv, accessed on December 17, 2025, https://arxiv.org/pdf/2510.11143?
16. The End of Infinite Context: Engineering Reliability in the Age of Agentic Workflows | by Ali moradi | Dec, 2025 | Medium, accessed on December 17, 2025, https://medium.com/@moradikor296/the-end-of-infinite-context-engineering-reliability-in-the-age-of-agentic-workflows-9531163159b4
17. MLIR: A Compiler Infrastructure for the End of Moore's Law - arXiv, accessed on December 17, 2025, https://arxiv.org/pdf/2002.11054
18. A Scalable Approach to Probabilistic Neuro-Symbolic Verification - arXiv, accessed on December 17, 2025, https://arxiv.org/html/2502.03274v1
19. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models | OpenReview, accessed on December 17, 2025, https://openreview.net/forum?id=tyEyYT267x
20. The Rise of Mercury — Diffusion vs. Autoregressive LLMs | by Daniel Ince-Cushman | Medium, accessed on December 17, 2025, https://medium.com/@d.incecushman/the-rise-of-mercury-diffusion-vs-autoregressive-llms-c631050074fb
21. Intuitive explanation on diffusion language models (dLLMs) and why they may be far superior to autoregressive for most uses (append & amend VS mutate & defragment) : r/LocalLLaMA - Reddit, accessed on December 17, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1ksrxm7/intuitive_explanation_on_diffusion_language/
22. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation - Jiatao Gu, accessed on December 17, 2025, https://jiataogu.me/papers/gong2025diffucoder.pdf
23. A Diffusion Language Model Delivering Breakthrough 2146 Tokens/s Inference Speed, accessed on December 17, 2025, https://seed.bytedance.com/blog/seed-research-seed-diffusion-preview-released-a-diffusion-language-model-delivering-breakthrough-2-146-tokens-s-inference-speed
24. Latent-Autoregressive GP-VAE Language Model - arXiv, accessed on December 17, 2025, https://arxiv.org/html/2512.09535v1
25. NEURAL EXECUTION ENGINES | OpenReview, accessed on December 17, 2025, https://openreview.net/forum?id=rJg7BA4YDr
26. (PDF) Neural Execution Engines: Learning to Execute Subroutines - ResearchGate, accessed on December 17, 2025, https://www.researchgate.net/publication/342197878_Neural_Execution_Engines_Learning_to_Execute_Subroutines
27. A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation - ChatPaper, accessed on December 17, 2025, https://chatpaper.com/paper/34682
28. LLM-Hardened DSLs for Probabilistic Code Generation in High-Assurance Systems, accessed on December 17, 2025, https://deanm.ai/blog/2025/5/24/toward-data-driven-multi-model-enterprise-ai-7e545-sw6c2
29. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation - Reliable Computer Systems - University of Waterloo, accessed on December 17, 2025, https://rcs.uwaterloo.ca/~ali/cs842-s23/papers/mlir.pdf
30. Specification Gaming in AI - Emergent Mind, accessed on December 17, 2025, https://www.emergentmind.com/topics/specification-gaming
31. Sneaky AI: Specification Gaming and the Shortcomings of Machine Learning, accessed on December 17, 2025, https://community.alteryx.com/t5/Data-Science/Sneaky-AI-Specification-Gaming-and-the-Shortcomings-of-Machine/ba-p/348686
32. Provably safe systems: the only path to controllable AGI | Hacker News, accessed on December 17, 2025, https://news.ycombinator.com/item?id=37619285
33. 2021.clean Craftsmanship - Disciplines, Standards, and Ethics - Scribd, accessed on December 17, 2025, https://www.scribd.com/document/633275173/2021-Clean-Craftsmanship-Disciplines-Standards-and-Ethics
34. A Scalable Approach to Probabilistic Neuro-Symbolic Robustness Verification - OpenReview, accessed on December 17, 2025, https://openreview.net/pdf?id=DAp8WCTGVj
35. AArhus 2025: The End of Programming (as we know it) — Envisioning Radical Re-Conceptualizations of Co-Coding with AI | by Jakob Tholander | Human-Centered AI - Medium, accessed on December 17, 2025, https://medium.com/human-centered-ai/aarhus-2025-the-end-of-programming-as-we-know-it-envisioning-radical-re-conceptualizations-of-e1a8b542cabe
36. Ephemeral UI in AI-Generated, On-Demand Interfaces | by iSolutions - Medium, accessed on December 17, 2025, https://isolutions.medium.com/ephemeral-ui-in-ai-generated-on-demand-interfaces-81dbc8cd4579
37. Ephemeral Environments: Boost Development Speed and Efficiency - Microtica, accessed on December 17, 2025, https://www.microtica.com/blog/ephemeral-environment
38. Meta Introduces Chain of Continuous Thought (Coconut) to Improve Next-Token Prediction, accessed on December 17, 2025, https://www.deeplearning.ai/the-batch/meta-introduces-chain-of-continuous-thought-coconut-to-improve-next-token-prediction/
39. Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages - Liner, accessed on December 17, 2025, https://liner.com/review/latent-execution-for-neural-program-synthesis-beyond-domainspecific-languages
40. I find this paper super cool, and highly unintuitive that an operation as discre... | Hacker News, accessed on December 17, 2025, https://news.ycombinator.com/item?id=22394320
41. Introducing DSL-SPA: An Open Source Tool for Simplifying Agentic Tasks - Medium, accessed on December 17, 2025, https://medium.com/@darkenergyiscool/introducing-dsl-spa-an-open-source-tool-for-simplifying-agentic-tasks-ab1ccd48d0d0
42. Just-in-Time Logic Enforcement - Hongyu Hè | Princeton University, accessed on December 17, 2025, https://hhy.ee.princeton.edu/papers/2025_hotnets_lejit.pdf
43. Chain-of-Verification (CoVe): Reduce LLM Hallucinations - Learn Prompting, accessed on December 17, 2025, https://learnprompting.org/docs/advanced/self_criticism/chain_of_verification
44. I Built a 'Thinking' AI Agent that Goes Beyond Prompts: Neuro ..., accessed on December 17, 2025, https://medium.com/data-science-collective/beyond-prompt-engineering-neuro-symbolic-causal-architecture-for-robust-multi-objective-ai-agents-53f3d23d9dde
45. Fast Search for Small and Efficient Neural Network Architectures, accessed on December 18, 2025
46. A Brief Review of Hypernetworks in Deep Learning, accessed on December 18, 2025
47. Primers • Hypernetworks, accessed on December 18, 2025
48. Neural Module Networks, accessed on December 18, 2025
49. Neural Module Networks, accessed on December 18, 2025
50. Neural Module Networks, accessed on December 18, 2025
51. Efficient-CapsNet: capsule network with self-attention routing, accessed on December 18, 2025
52. Capsule Network, accessed on December 18, 2025
53. Capsule Neural Networks – Set of Nested Neural Layers, accessed on December 18, 2025
54. Capsule neural network, accessed on December 18, 2025
55. Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs, accessed on December 18, 2025
56. Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules, accessed on December 18, 2025
57. Evolving Reusable Neural Modules, accessed on December 18, 2025