Created
February 6, 2026 22:02
-
-
Save sebastiaanvisser/9ce31f61fc7885f738188596b84c8113 to your computer and use it in GitHub Desktop.
Wall Summary → ClickHouse: branch review and migration narrative
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Wall Summary → ClickHouse: Branch Review</title> | |
| <style> | |
| :root { | |
| --bg: #1a1b26; | |
| --surface: #24283b; | |
| --surface2: #2f3347; | |
| --border: #3b4261; | |
| --text: #c0caf5; | |
| --text-muted: #7982a9; | |
| --heading: #7aa2f7; | |
| --accent: #bb9af7; | |
| --green: #9ece6a; | |
| --orange: #e0af68; | |
| --red: #f7768e; | |
| --cyan: #7dcfff; | |
| --code-bg: #1e2030; | |
| } | |
| * { box-sizing: border-box; margin: 0; padding: 0; } | |
| body { | |
| font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', sans-serif; | |
| background: var(--bg); | |
| color: var(--text); | |
| line-height: 1.7; | |
| padding: 2rem; | |
| max-width: 960px; | |
| margin: 0 auto; | |
| } | |
| h1 { | |
| color: var(--heading); | |
| font-size: 2rem; | |
| margin-bottom: 0.5rem; | |
| border-bottom: 2px solid var(--border); | |
| padding-bottom: 0.75rem; | |
| } | |
| h2 { | |
| color: var(--accent); | |
| font-size: 1.5rem; | |
| margin-top: 2.5rem; | |
| margin-bottom: 1rem; | |
| border-bottom: 1px solid var(--border); | |
| padding-bottom: 0.5rem; | |
| } | |
| h3 { | |
| color: var(--cyan); | |
| font-size: 1.15rem; | |
| margin-top: 1.75rem; | |
| margin-bottom: 0.5rem; | |
| } | |
| h4 { | |
| color: var(--orange); | |
| font-size: 1rem; | |
| margin-top: 1.25rem; | |
| margin-bottom: 0.4rem; | |
| } | |
| p { margin-bottom: 0.75rem; } | |
| a { color: var(--cyan); text-decoration: none; } | |
| a:hover { text-decoration: underline; } | |
| code { | |
| font-family: 'JetBrains Mono', 'Fira Code', 'SF Mono', Menlo, monospace; | |
| background: var(--code-bg); | |
| padding: 0.15em 0.4em; | |
| border-radius: 4px; | |
| font-size: 0.9em; | |
| color: var(--green); | |
| } | |
| pre { | |
| background: var(--code-bg); | |
| border: 1px solid var(--border); | |
| border-radius: 8px; | |
| padding: 1rem 1.25rem; | |
| overflow-x: auto; | |
| margin: 0.75rem 0 1.25rem 0; | |
| font-size: 0.85rem; | |
| line-height: 1.6; | |
| } | |
| pre code { | |
| background: none; | |
| padding: 0; | |
| color: var(--text); | |
| } | |
| .diagram { | |
| background: var(--surface); | |
| border: 1px solid var(--border); | |
| border-radius: 8px; | |
| padding: 1.25rem 1.5rem; | |
| margin: 1rem 0 1.5rem 0; | |
| font-family: 'JetBrains Mono', 'Fira Code', monospace; | |
| font-size: 0.85rem; | |
| line-height: 1.8; | |
| white-space: pre; | |
| overflow-x: auto; | |
| color: var(--text-muted); | |
| } | |
| .diagram .hl { color: var(--heading); font-weight: 600; } | |
| .diagram .acc { color: var(--accent); } | |
| .diagram .grn { color: var(--green); } | |
| .diagram .org { color: var(--orange); } | |
| .diagram .red { color: var(--red); } | |
| .card { | |
| background: var(--surface); | |
| border: 1px solid var(--border); | |
| border-radius: 8px; | |
| padding: 1.25rem; | |
| margin: 0.75rem 0 1.25rem 0; | |
| } | |
| .card-title { | |
| color: var(--orange); | |
| font-weight: 600; | |
| font-size: 0.95rem; | |
| margin-bottom: 0.4rem; | |
| } | |
| ul, ol { | |
| margin: 0.5rem 0 1rem 1.5rem; | |
| } | |
| li { margin-bottom: 0.35rem; } | |
| .tag { | |
| display: inline-block; | |
| background: var(--surface2); | |
| border: 1px solid var(--border); | |
| border-radius: 4px; | |
| padding: 0.1em 0.5em; | |
| font-size: 0.8em; | |
| color: var(--accent); | |
| font-family: monospace; | |
| } | |
| .tag-green { | |
| display: inline-block; | |
| background: rgba(158, 206, 106, 0.1); | |
| border: 1px solid rgba(158, 206, 106, 0.3); | |
| border-radius: 4px; | |
| padding: 0.1em 0.5em; | |
| font-size: 0.8em; | |
| color: var(--green); | |
| font-family: monospace; | |
| } | |
| .tag-red { | |
| display: inline-block; | |
| background: rgba(247, 118, 142, 0.1); | |
| border: 1px solid rgba(247, 118, 142, 0.3); | |
| border-radius: 4px; | |
| padding: 0.1em 0.5em; | |
| font-size: 0.8em; | |
| color: var(--red); | |
| font-family: monospace; | |
| } | |
| .path { | |
| color: var(--orange); | |
| font-family: monospace; | |
| font-size: 0.9em; | |
| } | |
| table { | |
| width: 100%; | |
| border-collapse: collapse; | |
| margin: 0.75rem 0 1.25rem 0; | |
| font-size: 0.9rem; | |
| } | |
| th { | |
| background: var(--surface2); | |
| color: var(--heading); | |
| text-align: left; | |
| padding: 0.6rem 0.75rem; | |
| border: 1px solid var(--border); | |
| font-weight: 600; | |
| } | |
| td { | |
| padding: 0.5rem 0.75rem; | |
| border: 1px solid var(--border); | |
| vertical-align: top; | |
| } | |
| tr:nth-child(even) td { background: var(--surface); } | |
| .subtitle { | |
| color: var(--text-muted); | |
| font-size: 0.95rem; | |
| margin-bottom: 2rem; | |
| } | |
| .separator { | |
| border: none; | |
| border-top: 1px solid var(--border); | |
| margin: 2.5rem 0; | |
| } | |
| .highlight-box { | |
| background: linear-gradient(135deg, rgba(122,162,247,0.08), rgba(187,154,247,0.08)); | |
| border: 1px solid rgba(122,162,247,0.25); | |
| border-radius: 8px; | |
| padding: 1.25rem; | |
| margin: 1rem 0 1.5rem 0; | |
| } | |
| .warning-box { | |
| background: rgba(224,175,104,0.08); | |
| border: 1px solid rgba(224,175,104,0.25); | |
| border-radius: 8px; | |
| padding: 1rem 1.25rem; | |
| margin: 1rem 0; | |
| } | |
| .danger-box { | |
| background: rgba(247,118,142,0.08); | |
| border: 1px solid rgba(247,118,142,0.25); | |
| border-radius: 8px; | |
| padding: 1rem 1.25rem; | |
| margin: 1rem 0; | |
| } | |
| .success-box { | |
| background: rgba(158,206,106,0.08); | |
| border: 1px solid rgba(158,206,106,0.25); | |
| border-radius: 8px; | |
| padding: 1rem 1.25rem; | |
| margin: 1rem 0; | |
| } | |
| .stat-grid { | |
| display: grid; | |
| grid-template-columns: repeat(3, 1fr); | |
| gap: 1rem; | |
| margin: 1rem 0 1.5rem 0; | |
| } | |
| .stat-card { | |
| background: var(--surface); | |
| border: 1px solid var(--border); | |
| border-radius: 8px; | |
| padding: 1rem; | |
| text-align: center; | |
| } | |
| .stat-value { | |
| font-size: 1.8rem; | |
| font-weight: 700; | |
| line-height: 1.2; | |
| } | |
| .stat-label { | |
| font-size: 0.8rem; | |
| color: var(--text-muted); | |
| margin-top: 0.25rem; | |
| } | |
| .commit-header { | |
| display: flex; | |
| align-items: baseline; | |
| gap: 0.75rem; | |
| margin-bottom: 0.5rem; | |
| } | |
| .commit-hash { | |
| font-family: monospace; | |
| font-size: 0.85rem; | |
| color: var(--text-muted); | |
| background: var(--surface2); | |
| padding: 0.1em 0.4em; | |
| border-radius: 4px; | |
| } | |
| .flow-arrow { | |
| color: var(--text-muted); | |
| font-size: 1.2rem; | |
| text-align: center; | |
| margin: 0.5rem 0; | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <h1>Wall Summary → ClickHouse</h1> | |
| <p class="subtitle">Branch <code>sebas/wall-summary-clickhouse</code> — 4 commits, 27 files changed, +654 / −212 lines. Extended code review and migration narrative.</p> | |
| <div class="stat-grid"> | |
| <div class="stat-card"> | |
| <div class="stat-value" style="color: var(--green);">+654</div> | |
| <div class="stat-label">lines added</div> | |
| </div> | |
| <div class="stat-card"> | |
| <div class="stat-value" style="color: var(--red);">−212</div> | |
| <div class="stat-label">lines removed</div> | |
| </div> | |
| <div class="stat-card"> | |
| <div class="stat-value" style="color: var(--accent);">27</div> | |
| <div class="stat-label">files changed</div> | |
| </div> | |
| </div> | |
| <!-- ===================================================================== --> | |
| <h2>1. Why Are We Doing This?</h2> | |
| <p>Wall summaries are lightweight representations of wall designs. They contain per-segment data: ID, name, bond pattern, brick counts, course counts, segment bounding-box faces, and brick definitions. Multiple parts of the system depend on them: the Planner, the Tag Mapper, the Explorer (Fenestra), and the Segment Progress dashboard.</p> | |
| <h3>The old architecture was fragile</h3> | |
| <p>Before this branch, wall summaries were computed <strong>client-side</strong> in the browser. Every time a wall design was saved in Portico, the client would:</p> | |
| <ol> | |
| <li>Build the full <code>Wall</code> via WASM (<code>Wall::from_def()</code>)</li> | |
| <li>Call <code>wall.serialize_summary(key)</code> to produce a protobuf-encoded <code>WallSummary</code></li> | |
| <li>Store that summary as a <em>separate entity</em> in Rotunda (the <code>wall_summary</code> namespace)</li> | |
| <li>Rely on <code>rotunda2clickhouse</code> to replicate it to ClickHouse</li> | |
| </ol> | |
| <div class="diagram"><span class="red">OLD FLOW (removed)</span> | |
| <span class="hl">Portico (browser)</span> | |
| │ save wall design | |
| │ <span class="org">+ compute WallSummary via WASM</span> | |
| │ <span class="org">+ serialize_summary() → protobuf bytes</span> | |
| │ <span class="org">+ POST to Rotunda (wall_summary namespace)</span> | |
| ▾ | |
| <span class="hl">Rotunda</span> (stores both wall_designs <span class="red">AND wall_summary</span>) | |
| │ WAL replication | |
| ▾ | |
| <span class="hl">rotunda2clickhouse</span> | |
| │ syncs wall_summary table | |
| ▾ | |
| <span class="hl">ClickHouse</span> rotunda.wall_summary</div> | |
| <p>This had several concrete problems:</p> | |
| <ul> | |
| <li><strong>Stale data:</strong> If a wall design was edited through a code path that didn't call <code>storeSummary()</code>, the summary would be out of date. There's no guarantee every save path triggers the summary write.</li> | |
| <li><strong>Missing data:</strong> Wall designs created before the summary mechanism existed, or via backend tools, had no summary at all.</li> | |
| <li><strong>Extra network round-trip:</strong> Every wall save required an additional Rotunda POST for the summary entity — doubling write latency.</li> | |
| <li><strong>Duplicated computation:</strong> The topo-sort (segment dependency graph) was computed in TypeScript every time a <code>WallSummary</code> was instantiated, both on save and on read. Every consumer independently computed the same parent/child relationships.</li> | |
| <li><strong>Extra replication load:</strong> A dedicated <code>wall_summary</code> table was synced through the full Rotunda → ClickHouse pipeline, consuming storage and replication bandwidth for derived data.</li> | |
| </ul> | |
| <!-- ===================================================================== --> | |
| <h2>2. What Does the New Architecture Look Like?</h2> | |
| <div class="diagram"><span class="grn">NEW FLOW</span> | |
| <span class="hl">Portico (browser)</span> | |
| │ save wall design (no summary step!) | |
| ▾ | |
| <span class="hl">Rotunda</span> (stores wall_designs only, with <span class="acc">raw_value: true</span>) | |
| │ WAL replication | |
| ▾ | |
| <span class="hl">rotunda2clickhouse</span> | |
| │ syncs wall_designs (raw protobuf bytes in raw_value column) | |
| ▾ | |
| <span class="hl">ClickHouse</span> | |
| │ rotunda.wall_designs (with raw_value column) | |
| │ | |
| │ <span class="acc">SELECT wall_summary(raw_value)</span> ← Rust UDF | |
| ▾ | |
| <span class="grn">staging.wall_summary</span> (dbt view, computed on demand) | |
| │ | |
| ├── Portico: Planner, Tag Mapper | |
| ├── Explorer: Wall Segment collection, Wall Summary collection | |
| └── Explorer: Segment Progress collection</div> | |
| <p>The <code>wall_summary()</code> UDF is a Rust executable registered with ClickHouse. It takes the raw protobuf bytes of a <code>WallDesign</code>, decodes them, builds the full wall via <code>Wall::from_def()</code> (the same Rust code used everywhere else), computes the summary including segment parent dependencies, and returns a structured tuple directly in SQL.</p> | |
| <div class="highlight-box"> | |
| <strong>Key insight:</strong> The wall design's raw protobuf already contains everything needed to derive the summary. By computing it server-side in the UDF, the summary is always fresh, always complete, and requires zero client cooperation. | |
| </div> | |
| <!-- ===================================================================== --> | |
| <h2>3. Commit-by-Commit Walkthrough</h2> | |
| <!-- Commit 1 --> | |
| <h3><span class="commit-hash">e5a80c12</span> Derive wall summaries from ClickHouse UDF instead of Rotunda</h3> | |
| <p>The foundational commit. This is the bulk of the migration.</p> | |
| <h4>Rust: UDF extended with <code>brick_defs</code> and <code>faces</code></h4> | |
| <p>The existing <code>wall_summary()</code> UDF only returned <code>(id, name, bond, courses, bricks)</code>. This commit adds two new columns so downstream consumers have the complete picture:</p> | |
| <ul> | |
| <li><code>brick_defs Array(Array(String))</code> — per-segment list of brick definition names</li> | |
| <li><code>faces Array(String)</code> — per-segment JSON-encoded array of <code>OrientedBoundingBox</code> (trans3 + size)</li> | |
| </ul> | |
| <p>Faces are JSON-encoded because ClickHouse doesn't have a native "4x4 matrix" type, and the existing <code>OrientedBoundingBox</code> serialization via serde produces a clean JSON structure that's easy to parse client-side.</p> | |
| <h4>dbt: new <code>staging.wall_summary</code> view</h4> | |
| <pre><code>-- dbt/models/wall_summary.sql | |
| WITH summaries AS ( | |
| SELECT | |
| wd.key AS wall_design_key, | |
| wall_summary(wd.raw_value) AS ws | |
| FROM rotunda.wall_designs AS wd | |
| ) | |
| SELECT | |
| wall_design_key, | |
| ws.id AS segment_ids, | |
| ws.name AS segment_names, | |
| ... | |
| FROM summaries | |
| WHERE length(ws.id) > 0</code></pre> | |
| <p>This is a <code>VIEW</code> (not materialized) — it computes summaries on every query. The <code>WHERE length(ws.id) > 0</code> filter excludes wall designs where the UDF returned empty arrays (invalid or empty protobuf).</p> | |
| <h4>Portico: new <code>WallSummaryClickHouse.ts</code> fetch service</h4> | |
| <p>A clean replacement for the Rotunda-based wall summary store. Two functions:</p> | |
| <ul> | |
| <li><code>fetchWallSummary(key)</code> — single wall, used by Planner and Tag Mapper</li> | |
| <li><code>fetchAllWallSummaries()</code> — all walls, available for bulk use</li> | |
| </ul> | |
| <p>Both query <code>staging.wall_summary</code> via the ClickHouse HTTP client and map rows to the existing <code>PWallSummary</code> protobuf type, so all downstream consumers continue to work with the same <code>WallSummary</code> class.</p> | |
| <h4>Portico: consumers migrated</h4> | |
| <table> | |
| <thead><tr><th>Consumer</th><th>Before</th><th>After</th></tr></thead> | |
| <tbody> | |
| <tr> | |
| <td>Planner (<code>tabs/planner/App.tsx</code>)</td> | |
| <td>Used <code>WallSummaryStore</code> backed entity</td> | |
| <td>Loads wall design, then calls <code>fetchWallSummary(key)</code></td> | |
| </tr> | |
| <tr> | |
| <td>Tag Mapper (<code>tabs/tag-mapper/</code>)</td> | |
| <td>Instantiated <code>WallSummaryStore</code>, passed to <code>NameSceneStep</code></td> | |
| <td>Passes <code>fetchWallSummary</code> function directly</td> | |
| </tr> | |
| <tr> | |
| <td>Segment Progress (Explorer)</td> | |
| <td>Queried <code>rotunda.wall_summary</code> with <code>arrayJoin</code></td> | |
| <td>Queries <code>staging.wall_summary</code> with <code>ARRAY JOIN</code></td> | |
| </tr> | |
| <tr> | |
| <td>Wall Segment (Explorer)</td> | |
| <td>Rotunda-backed <code>WallSegmentCollection</code></td> | |
| <td>ClickHouse-backed <code>createWallSegmentCollectionDef()</code></td> | |
| </tr> | |
| <tr> | |
| <td>Wall Summary (Explorer)</td> | |
| <td>Rotunda-backed <code>WallSummaryCollection</code></td> | |
| <td>ClickHouse-backed <code>createWallSummaryCollectionDef()</code></td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <h4>Code removed</h4> | |
| <ul> | |
| <li><code>WallSummaryStore</code> class — the Rotunda-backed entity store for wall summaries <span class="tag-red">deleted</span></li> | |
| <li><code>WallDesignStore.save()</code> override — no longer calls <code>storeSummary()</code> after saving <span class="tag-red">deleted</span></li> | |
| <li><code>WallDesignStore.storeSummary()</code> method <span class="tag-red">deleted</span></li> | |
| <li><code>Wall.serialize_summary()</code> WASM method in Rust <span class="tag-red">deleted</span></li> | |
| <li><code>wall_summary</code> namespace from <code>rotunda2clickhouse.yaml</code> <span class="tag-red">deleted</span></li> | |
| <li><code>wall_summary</code> source from dbt <code>sources.yml</code> <span class="tag-red">deleted</span></li> | |
| <li><code>WallSegmentCollection</code> and <code>WallSummaryCollection</code> from Rotunda backend <span class="tag-red">deleted</span></li> | |
| </ul> | |
| <h4>CI: dbt test harness updated</h4> | |
| <p>The test script now builds <code>clickhouse-udf</code> via cargo, copies the UDF XML config into the temporary ClickHouse instance, and configures <code>user_scripts_path</code> and <code>user_defined_executable_functions_config</code> so the UDF is available during tests.</p> | |
| <!-- Commit 2 --> | |
| <hr class="separator"> | |
| <h3><span class="commit-hash">9251e3c3</span> Move segment dependency topo-sort from TypeScript to Rust UDF</h3> | |
| <p>This is the most architecturally significant simplification.</p> | |
| <h4>The problem</h4> | |
| <p>Segment dependencies (which segment physically sits on top of which) were computed client-side in TypeScript every time a <code>WallSummary</code> was instantiated. The <code>topoSort()</code> method performed O(n³) geometric comparisons — for each pair of segments, it checked if any face of segment A was geometrically "on top of" any face of segment B, then filtered for <em>direct</em> parents (no intermediate segment in between).</p> | |
| <p>This was duplicated in multiple places: the <code>WallSummary</code> constructor ran it, and <code>SiteMap.computeSegmentZeroLevels()</code> in commit 3 had its own copy.</p> | |
| <h4>The solution</h4> | |
| <p>Move the computation to Rust, inside <code>WallSummary::from(Wall)</code>, and expose the results as a <code>parents</code> field on each <code>SegmentSummary</code>.</p> | |
| <p><strong>Rust changes:</strong></p> | |
| <ul> | |
| <li>New <code>OrientedBoundingBox::on_top_of()</code> method in <code>arcade/geom/src/oriented_bbox.rs</code> — projects both OBBs to 2D, checks for corner overlap, then compares Z coordinates <span class="tag-green">+39 lines</span></li> | |
| <li>New <code>WallSummary::topo_sort()</code> method in <code>arcade/wall/src/wall_summary.rs</code> that computes the full parent graph once <span class="tag-green">+35 lines</span></li> | |
| <li>New <code>parents Array(Array(String))</code> column in the UDF output</li> | |
| <li>New <code>parents</code> field on the <code>SegmentSummary</code> protobuf message</li> | |
| </ul> | |
| <p><strong>TypeScript changes:</strong></p> | |
| <ul> | |
| <li><code>WallSummary.topoSort()</code> replaced with <code>buildFromProtoParents()</code> — simply reads parent IDs from proto and builds the graph via lookup <span class="tag-red">−40 lines</span> <span class="tag-green">+20 lines</span></li> | |
| <li><code>WallSummary.onTopOf()</code> static method removed</li> | |
| <li>Explorer <code>Wall.ts</code>: <code>mapRow</code> simplified to read <code>segment_parents</code> directly from ClickHouse row instead of constructing WallSummary objects and re-running the topo-sort</li> | |
| </ul> | |
| <div class="highlight-box"> | |
| <strong>Performance win:</strong> The topo-sort is now computed once in Rust (fast, compiled code) on the server side, and the result is cached in the dbt view. Clients read parent IDs as plain strings — no geometric computation at all. | |
| </div> | |
| <!-- Commit 3 --> | |
| <hr class="separator"> | |
| <h3><span class="commit-hash">66ff9215</span> Use precomputed segment parents in SiteMap zero-level computation</h3> | |
| <p>This is a follow-up that removes a <em>second copy</em> of the same topo-sort logic from a different part of the codebase.</p> | |
| <p><code>SiteMap.computeSegmentZeroLevels()</code> was independently computing which segments sit on top of which, using the same geometric overlap check. Now it calls a new WASM method <code>Wall.segment_parent_ids()</code> that builds a <code>WallSummary</code> from the wall and returns a <code>Map<string, string[]></code> of segment ID → parent IDs.</p> | |
| <p><strong>Removed:</strong></p> | |
| <ul> | |
| <li><code>OrientedBoundingBox.onTopOf()</code> TypeScript method (25 lines) <span class="tag-red">deleted</span></li> | |
| <li><code>isDirectlyOnTop()</code> closure in SiteMap (18 lines) <span class="tag-red">deleted</span></li> | |
| <li><code>faces</code> field from <code>SegmentNode</code> interface — no longer needed <span class="tag-red">deleted</span></li> | |
| <li><code>Rect2</code> import from <code>OrientedBoundingBox.ts</code> <span class="tag-red">deleted</span></li> | |
| </ul> | |
| <p><strong>Added:</strong></p> | |
| <ul> | |
| <li><code>Wall.segment_parent_ids()</code> WASM method in Rust (14 lines) <span class="tag-green">added</span></li> | |
| </ul> | |
| <p>Net: <strong>−53 lines of TypeScript</strong>, <strong>+14 lines of Rust</strong>. The geometric logic lives in exactly one place now.</p> | |
| <!-- Commit 4 --> | |
| <hr class="separator"> | |
| <h3><span class="commit-hash">bd9e6eba</span> Fix dbt CI: install clickhouse-udf via nix instead of cargo build</h3> | |
| <p>A pragmatic CI fix. The dbt-test GitHub Actions workflow runs in a nix shell that doesn't have cargo available. The previous commit's test script unconditionally tried to <code>cargo build -p clickhouse-udf</code>, which fails in CI.</p> | |
| <p><strong>Fix:</strong></p> | |
| <ul> | |
| <li>CI workflow: add <code>.#nixpkgs.clickhouse-udf</code> to the nix build step, putting the pre-built binary on <code>$PATH</code></li> | |
| <li>Test script: check if <code>clickhouse-udf</code> is already on <code>PATH</code> via <code>shutil.which()</code>. If yes, use it directly. If not, fall back to cargo build. This makes the script work in both CI (nix-provided binary) and local dev (cargo build).</li> | |
| </ul> | |
| <!-- ===================================================================== --> | |
| <hr class="separator"> | |
| <h2>4. What Was Simplified</h2> | |
| <h3>Data flow</h3> | |
| <div class="card"> | |
| <div class="card-title">Before: 2 Rotunda entities, client-side coupling</div> | |
| <p>Save wall design → compute summary in WASM → serialize to protobuf → POST summary to Rotunda → replicate <em>both</em> entities to ClickHouse → consumers query <code>rotunda.wall_summary</code> table → each consumer re-runs topo-sort in TypeScript.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">After: 1 Rotunda entity, server-side derivation</div> | |
| <p>Save wall design → replicate to ClickHouse → UDF derives summary on query → consumers read flat arrays with pre-computed parents.</p> | |
| </div> | |
| <h3>Code ownership</h3> | |
| <table> | |
| <thead><tr><th>Concept</th><th>Before</th><th>After</th></tr></thead> | |
| <tbody> | |
| <tr> | |
| <td>Segment parent computation</td> | |
| <td>3 places: <code>WallSummary.topoSort()</code>, <code>SiteMap.isDirectlyOnTop()</code>, conceptually in Explorer</td> | |
| <td>1 place: <code>WallSummary::topo_sort()</code> in Rust</td> | |
| </tr> | |
| <tr> | |
| <td>"On top of" geometry check</td> | |
| <td>2 places: <code>OrientedBoundingBox.onTopOf()</code> (TypeScript), implicit in SiteMap</td> | |
| <td>1 place: <code>OrientedBoundingBox::on_top_of()</code> (Rust)</td> | |
| </tr> | |
| <tr> | |
| <td>Wall summary storage</td> | |
| <td><code>WallSummaryStore</code> + Rotunda namespace + <code>rotunda2clickhouse</code> config</td> | |
| <td>None (derived on demand via UDF)</td> | |
| </tr> | |
| <tr> | |
| <td>Explorer collections for Wall</td> | |
| <td>Rotunda-backed classes (<code>WallSegmentCollection</code>, <code>WallSummaryCollection</code>)</td> | |
| <td>ClickHouse-backed collection defs (pure functions returning <code>ClickHouseCollectionDef</code>)</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <!-- ===================================================================== --> | |
| <h2>5. What Was Sped Up</h2> | |
| <h3>Wall save latency</h3> | |
| <p>Previously, saving a wall design required two sequential network calls: one for the design itself, then one for the summary. The summary call involved computing the full wall from the design in WASM, serializing it to protobuf, and POSTing it to Rotunda. The <code>save()</code> override waited for both to complete.</p> | |
| <p>Now, saving a wall design is a single Rotunda POST. The summary is never explicitly stored.</p> | |
| <h3>Client-side rendering</h3> | |
| <p>Instantiating a <code>WallSummary</code> in the old code ran <code>topoSort()</code>, which performed O(n²) geometric face comparisons plus O(n³) intermediate-segment filtering. For walls with many segments, this was noticeable. Now <code>buildFromProtoParents()</code> is O(n) — a simple map lookup.</p> | |
| <h3>SiteMap zero-level computation</h3> | |
| <p>The same O(n³) topo-sort was run again for every site map. Now it reads precomputed parents from the wall, which is O(segments × parents) — effectively O(n).</p> | |
| <h3>Explorer load time</h3> | |
| <p>Explorer collections no longer instantiate <code>WallSummary</code> objects (which triggered the topo-sort). They read flat arrays directly from ClickHouse rows and map them to <code>Val</code> objects. The wall segment collection's <code>mapRow</code> went from constructing a <code>WallSummary</code> + running topo-sort + iterating <code>sorted</code> → to directly iterating row arrays.</p> | |
| <!-- ===================================================================== --> | |
| <h2>6. Potential Regressions to Test</h2> | |
| <div class="danger-box"> | |
| <h4 style="color: var(--red); margin-bottom: 0.5rem;">Critical: Planner wall loading</h4> | |
| <p>The Planner (<code>tabs/planner/App.tsx</code>) changed from a <code>BackedEntity<WallSummary></code> to a <code>BackedEntity<C.Wall></code>. It now loads the full wall design from Rotunda, then asynchronously fetches the summary from ClickHouse. This introduces a <strong>timing dependency</strong>: the summary fetch happens in a fire-and-forget <code>async</code> method (<code>loadSummary</code>). If the ClickHouse query fails or is slow, the planning view may show no tasks.</p> | |
| <p><strong>Test:</strong> Open the Planner, select a wall design, verify that planning tasks populate correctly. Also test with ClickHouse unavailable.</p> | |
| </div> | |
| <div class="danger-box"> | |
| <h4 style="color: var(--red); margin-bottom: 0.5rem;">Critical: Tag Mapper wall summary</h4> | |
| <p>The Tag Mapper now receives a <code>fetchWallSummary</code> function instead of a store. The function throws if no summary is found (<code>if (!summary) throw new Error(...)</code>). Previously, the store would return a default empty summary.</p> | |
| <p><strong>Test:</strong> Open the Tag Mapper, select a wall design that exists in ClickHouse, verify segment names populate. Also test with a brand-new wall design that may not yet be in ClickHouse (replication delay).</p> | |
| </div> | |
| <div class="warning-box"> | |
| <h4 style="color: var(--orange); margin-bottom: 0.5rem;">High: Segment parent accuracy</h4> | |
| <p>The Rust <code>on_top_of()</code> and <code>topo_sort()</code> reimplementations should produce the same results as the old TypeScript code, but the geometric logic was ported, not shared. Subtle floating-point differences could cause different parent assignments.</p> | |
| <p><strong>Test:</strong> For a multi-segment wall with known dependencies (e.g. a wall with lintels above door openings), verify the "Depends on" column in the Explorer matches expectations. Compare with the old code on a known wall design.</p> | |
| </div> | |
| <div class="warning-box"> | |
| <h4 style="color: var(--orange); margin-bottom: 0.5rem;">High: Segment Progress query</h4> | |
| <p>The Segment Progress explorer collection's SQL was rewritten from querying <code>rotunda.wall_summary</code> to <code>staging.wall_summary</code>. The join pattern changed from <code>arrayJoin(ws.value.segments)</code> to <code>ARRAY JOIN ws.segment_ids AS id, ws.segment_bricks AS bricks</code>.</p> | |
| <p><strong>Test:</strong> Open the Segment Progress collection in the Explorer. Verify brick progress percentages match reality for active segments.</p> | |
| </div> | |
| <div class="warning-box"> | |
| <h4 style="color: var(--orange); margin-bottom: 0.5rem;">Medium: ClickHouse UDF performance under load</h4> | |
| <p>The <code>staging.wall_summary</code> view is not materialized. Every query re-runs the UDF for every wall design. The UDF decodes protobuf and builds the full wall, which includes geometric calculations. For the Explorer "Wall Summary" and "Wall Segment" collections, this means the UDF runs for <em>all</em> wall designs on every load.</p> | |
| <p><strong>Test:</strong> With production data (~hundreds of wall designs), check query latency for <code>SELECT * FROM staging.wall_summary</code>. If too slow, consider materializing the view.</p> | |
| </div> | |
| <div class="warning-box"> | |
| <h4 style="color: var(--orange); margin-bottom: 0.5rem;">Medium: Faces JSON format</h4> | |
| <p>Faces are serialized as JSON via <code>serde_json::to_string(&s.faces)</code> in Rust, then parsed in TypeScript via <code>JSON.parse()</code> and manually mapped to the proto <code>OrientedBoundingBox</code> format. The JSON keys are Rust field names (<code>trans3.matrix.mat</code>) which must match what <code>parseFacesJson()</code> expects.</p> | |
| <p><strong>Test:</strong> In the Explorer, open a Wall Segment, inspect the Faces field. Verify the 3D bounding boxes render correctly in the viewport.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Low: Old wall_summary data in Rotunda</div> | |
| <p>The <code>wall_summary</code> namespace was removed from <code>rotunda2clickhouse.yaml</code>, so the <code>rotunda.wall_summary</code> table in ClickHouse will stop receiving updates. However, the existing table and data remain. No migration or cleanup is included in this branch.</p> | |
| <p><strong>Consider:</strong> Should the old <code>rotunda.wall_summary</code> table be explicitly dropped in production? Any other consumers that query it directly?</p> | |
| </div> | |
| <!-- ===================================================================== --> | |
| <h2>7. Files Changed Summary</h2> | |
| <table> | |
| <thead><tr><th>File</th><th>Change</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>arcade/clickhouse-udf/clickhouse_function.xml</code></td><td>UDF return type: added <code>brick_defs</code>, <code>faces</code>, <code>parents</code></td></tr> | |
| <tr><td><code>arcade/clickhouse-udf/src/functions/wall_summary.rs</code></td><td>Emit <code>brick_defs</code>, <code>faces</code> (JSON), <code>parents</code> columns</td></tr> | |
| <tr><td><code>arcade/clickhouse-udf/tests/integration_test.rs</code></td><td>Test assertions for new columns</td></tr> | |
| <tr><td><code>arcade/geom/src/oriented_bbox.rs</code></td><td><span class="tag-green">new</span> <code>on_top_of()</code> method + unit test</td></tr> | |
| <tr><td><code>arcade/wall/src/wall_summary.rs</code></td><td><span class="tag-green">new</span> <code>parents</code> field, <code>topo_sort()</code> method</td></tr> | |
| <tr><td><code>arcade/wall/src/wasm/wall.rs</code></td><td><span class="tag-green">new</span> <code>segment_parent_ids()</code>, <span class="tag-red">removed</span> <code>serialize_summary()</code></td></tr> | |
| <tr><td><code>proto/terraform/wall_design/wall_design.proto</code></td><td><span class="tag-green">new</span> <code>parents</code> field on <code>SegmentSummary</code></td></tr> | |
| <tr><td><code>portico/src/proto/terraform/wall_design/wall_design.ts</code></td><td>Generated: <code>parents</code> field support</td></tr> | |
| <tr><td><code>vision/terraform-proto/terraform/wall_design/wall_design_pb2.py[i]</code></td><td>Generated: <code>parents</code> field support</td></tr> | |
| <tr><td><code>dbt/models/wall_summary.sql</code></td><td><span class="tag-green">new</span> dbt view calling UDF</td></tr> | |
| <tr><td><code>dbt/models/sources.yml</code></td><td><span class="tag-red">removed</span> <code>wall_summary</code> source</td></tr> | |
| <tr><td><code>dbt/tests/test_wall_summary.py</code></td><td><span class="tag-green">new</span> tests for the dbt view</td></tr> | |
| <tr><td><code>dbt/scripts/test</code></td><td>Build UDF for test harness, conditional cargo/nix</td></tr> | |
| <tr><td><code>nix/systems/athena/rotunda2clickhouse.yaml</code></td><td><span class="tag-red">removed</span> <code>wall_summary</code> namespace</td></tr> | |
| <tr><td><code>.github/workflows/dbt-test.yml</code></td><td>Install <code>clickhouse-udf</code> nix package in CI</td></tr> | |
| <tr><td><code>portico/src/robotics/wall-design/WallDesignStore.ts</code></td><td><span class="tag-red">removed</span> <code>WallSummaryStore</code>, <code>storeSummary()</code>, <code>save()</code> override; <code>topoSort()</code> → <code>buildFromProtoParents()</code></td></tr> | |
| <tr><td><code>portico/src/robotics/wall-design/WallSummaryClickHouse.ts</code></td><td><span class="tag-green">new</span> ClickHouse-based fetch service</td></tr> | |
| <tr><td><code>portico/src/tabs/planner/App.tsx</code></td><td>Migrated to ClickHouse summary fetch</td></tr> | |
| <tr><td><code>portico/src/tabs/tag-mapper/App.ts</code></td><td>Migrated to ClickHouse summary fetch</td></tr> | |
| <tr><td><code>portico/src/tabs/tag-mapper/NameSceneStep.ts</code></td><td>Accepts fetch function instead of store</td></tr> | |
| <tr><td><code>portico/src/geom/OrientedBoundingBox.ts</code></td><td><span class="tag-red">removed</span> <code>onTopOf()</code> method</td></tr> | |
| <tr><td><code>portico/src/robotics/site-map/SiteMap.ts</code></td><td>Uses WASM <code>segment_parent_ids()</code> instead of geometric topo-sort</td></tr> | |
| <tr><td><code>portico/src/ui/explorer/runtime/Rotunda.tsx</code></td><td><span class="tag-red">removed</span> Rotunda-backed Wall collections</td></tr> | |
| <tr><td><code>portico/src/ui/explorer/runtime/collections/ClickHouse.ts</code></td><td>Added Wall collections, support for <code>mapRow</code> returning arrays</td></tr> | |
| <tr><td><code>portico/src/ui/explorer/runtime/collections/SegmentProgress.ts</code></td><td>SQL rewritten to use <code>staging.wall_summary</code></td></tr> | |
| <tr><td><code>portico/src/ui/explorer/runtime/collections/Wall.ts</code></td><td><span class="tag-green">new</span> ClickHouse-backed collection defs (+233 lines)</td></tr> | |
| </tbody> | |
| </table> | |
| <!-- ===================================================================== --> | |
| <h2>8. Next Steps</h2> | |
| <h3>Performance</h3> | |
| <div class="card"> | |
| <div class="card-title">Materialize the wall_summary view</div> | |
| <p>Currently <code>staging.wall_summary</code> is a plain VIEW, meaning the UDF runs on every query. For the Explorer's bulk-load pattern (all wall designs at once), this could become slow as the number of designs grows. Converting to a <code>MATERIALIZED VIEW</code> with periodic refresh (like <code>metric_bricks</code>) would eliminate redundant computation.</p> | |
| <p>Alternatively, a ClickHouse <code>MATERIALIZED VIEW</code> that triggers on inserts to <code>rotunda.wall_designs</code> could keep summaries up-to-date in near-real-time without polling.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Cache UDF results in a table</div> | |
| <p>A middle ground: keep the dbt view for ad-hoc queries, but add a <code>ReplacingMergeTree</code> table that stores UDF results keyed by <code>(wall_design_key, version)</code>. Refresh only when a wall design changes.</p> | |
| </div> | |
| <h3>Simplification</h3> | |
| <div class="card"> | |
| <div class="card-title">Remove the old <code>rotunda.wall_summary</code> table</div> | |
| <p>The table still exists in ClickHouse but is no longer populated. Once confirmed no other consumers query it, drop it to avoid confusion.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Remove <code>WallSummary</code> protobuf message from Rotunda</div> | |
| <p>The protobuf type <code>terraform.wall_design.WallSummary</code> is still defined in the proto and used as the in-memory data structure. However, it's no longer stored in Rotunda as an entity. The proto could eventually be simplified or repurposed as a pure computation type rather than a storage entity.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Unify the two <code>parseFacesJson</code> functions</div> | |
| <p>There are currently two copies of <code>parseFacesJson()</code>: one in <code>WallSummaryClickHouse.ts</code> and one in <code>Wall.ts</code> (Explorer collections). They do the same thing. Extract to a shared utility.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Consider extending the UDF for other derived data</div> | |
| <p>The UDF pattern (Rust code running inside ClickHouse, decoding raw protobuf) is powerful and reusable. Other computations that are currently done client-side from wall designs could be moved server-side the same way — e.g. control point validation, brick placement predictions, or segment volume calculations.</p> | |
| </div> | |
| <h3>Testing</h3> | |
| <div class="card"> | |
| <div class="card-title">Add dbt test with real protobuf data</div> | |
| <p>The current dbt test only verifies the view schema and that invalid protos produce zero rows. Adding a JSONL fixture with a real (or realistic) wall design protobuf would test the full UDF pipeline including segment counts, parent computation, and face serialization.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="card-title">Rust integration test for multi-segment parent graph</div> | |
| <p>The existing integration test uses a single-segment wall (which has no parents). Add a test with a multi-segment wall where at least one segment sits on top of another, and verify the <code>parents</code> array is correct.</p> | |
| </div> | |
| <!-- ===================================================================== --> | |
| <hr class="separator"> | |
| <h2>9. Architecture Diagram (Final State)</h2> | |
| <div class="diagram"><span class="hl">Wall Design</span> saved in Portico | |
| │ | |
| │ single POST to Rotunda | |
| ▾ | |
| <span class="hl">Rotunda</span> → <span class="acc">rotunda2clickhouse</span> → <span class="hl">ClickHouse</span> | |
| │ | |
| │ rotunda.wall_designs (with raw_value = protobuf bytes) | |
| │ | |
| │ <span class="grn">wall_summary(raw_value)</span> Rust UDF | |
| │ │ decode protobuf | |
| │ │ Wall::from_def() | |
| │ │ WallSummary::from(wall) | |
| │ │ └ topo_sort() → parents | |
| │ │ serialize to ClickHouse tuple | |
| │ ▾ | |
| │ <span class="grn">staging.wall_summary</span> dbt view | |
| │ │ (id[], name[], bond[], courses[], bricks[], | |
| │ │ brick_defs[][], faces[], parents[][]) | |
| │ ▾ | |
| ├── <span class="org">Planner</span> fetchWallSummary(key) | |
| ├── <span class="org">Tag Mapper</span> fetchWallSummary(key) | |
| ├── <span class="org">Explorer</span> Wall Segment collection | |
| ├── <span class="org">Explorer</span> Wall Summary collection | |
| └── <span class="org">Explorer</span> Segment Progress collection | |
| <span class="hl">SiteMap</span> (live, on-robot) | |
| │ wall.segment_parent_ids() WASM method | |
| │ └ builds WallSummary in Rust, returns Map<id, parent_ids> | |
| └── computeSegmentZeroLevels() uses precomputed parents</div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment