Skip to content

Instantly share code, notes, and snippets.

@denniswon

denniswon/gist:08df11fd951fb03566b4cc67e447e026

Created February 11, 2026 00:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denniswon/08df11fd951fb03566b4cc67e447e026 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denniswon/08df11fd951fb03566b4cc67e447e026 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Plan to implement │
│ │
│ Newton Privacy Layer - Phase 1A Implementation Plan │
│ │
│ Overview │
│ │
│ Add privacy-preserving data handling to the existing Two-Phase Consensus protocol. All changes follow existing codebase patterns and │
│ maintain backward compatibility through Option<T> fields and feature flags. │
│ │
│ Dependency Layers │
│ │
│ Layer 1 (Foundation) → Layer 2 (Gateway API) → Layer 3 (Operator) → Layer 4 (Observability + Tests) │
│ NEWT-187 (HPKE) NEWT-434 (Request) NEWT-432 (Decrypt) NEWT-436 (Metrics) │
│ NEWT-185 (Envelope) NEWT-435 (Storage API) NEWT-438 (PolicyData) NEWT-437 (E2E) │
│ NEWT-186 (Ed25519) NEWT-433 (Auth) │
│ NEWT-431 (DB table) │
│ │
│ --- │
│ Layer 1: Foundation (Parallelizable) │
│ │
│ Cargo.toml Additions │
│ │
│ Workspace (/Cargo.toml): │
│ hpke = { version = "0.12", features = ["std"] } │
│ ed25519-dalek = { version = "2", features = ["rand_core", "serde"] } │
│ x25519-dalek = { version = "2", features = ["serde", "static_secrets"] } │
│ │
│ Core crate (crates/core/Cargo.toml): │
│ [dependencies] │
│ hpke = { workspace = true, optional = true } │
│ ed25519-dalek = { workspace = true, optional = true } │
│ x25519-dalek = { workspace = true, optional = true } │
│ │
│ [features] │
│ privacy = ["dep:hpke", "dep:ed25519-dalek", "dep:x25519-dalek"] │
│ default = ["rpc", "proving", "config", "signing", "eigen", "ipfs-cache", "privacy"] │
│ │
│ NEWT-187: HPKE Module │
│ │
│ New file: crates/core/src/crypto/hpke.rs │
│ │
│ Suite: X25519 KEM + HKDF-SHA256 + ChaCha20-Poly1305 (RFC 9180) │
│ │
│ Functions: │
│ - generate_keypair() -> (HpkePrivateKey, HpkePublicKey) │
│ - encrypt(recipient_pk, plaintext, aad) -> Result<(Vec<u8>, Vec<u8>), CryptoError> │
│ - decrypt(recipient_sk, encapped_key, ciphertext, aad) -> Result<Vec<u8>, CryptoError> │
│ - public_key_to_bytes/from_bytes for serialization │
│ │
│ Tests: roundtrip, wrong key, wrong AAD, empty plaintext │
│ │
│ NEWT-185: Secure Envelope │
│ │
│ New file: crates/core/src/crypto/envelope.rs │
│ │
│ pub struct SecureEnvelope { │
│ pub enc: Vec<u8>, // HPKE encapsulated key (32 bytes) │
│ pub ciphertext: Vec<u8>, // HPKE ciphertext + Poly1305 tag │
│ pub policy_client: String, // 0x-prefixed address │
│ pub chain_id: u64, // Context binding │
│ pub recipient_pubkey: String, // Operator Ed25519 pubkey (hex) │
│ } │
│ │
│ - AAD = keccak256(abi.encodePacked(policy_client, chain_id)) │
│ - seal() encrypts plaintext into envelope │
│ - open() decrypts envelope using recipient private key │
│ │
│ Tests: roundtrip, wrong key, tampered ciphertext, serde roundtrip │
│ │
│ NEWT-186: Ed25519 Signing │
│ │
│ New file: crates/core/src/crypto/ed25519.rs │
│ │
│ Key derivation from existing ECDSA keys (zero operator config changes): │
│ - derive_ed25519_from_ecdsa(ecdsa_key: &[u8; 32]) -> SigningKey (HKDF-SHA256) │
│ - ed25519_to_x25519_private/public() for HPKE keypair derivation │
│ - sign_data_ref() / verify_data_ref() for encrypted data reference signatures │
│ │
│ Tests: deterministic derivation, sign/verify roundtrip, X25519 conversion │
│ │
│ NEWT-431: DB Table │
│ │
│ New migration: crates/gateway/migrations/20260210100000_create_encrypted_data_refs.sql │
│ │
│ CREATE TABLE encrypted_data_refs ( │
│ id UUID PRIMARY KEY DEFAULT gen_random_uuid(), │
│ user_id UUID NOT NULL, │
│ policy_client_address BYTEA NOT NULL, │
│ envelope BYTEA NOT NULL, │
│ signature BYTEA NOT NULL, │
│ recipient_pubkey BYTEA NOT NULL, │
│ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), │
│ expires_at TIMESTAMPTZ, │
│ CONSTRAINT encrypted_data_refs_policy_client_len CHECK (octet_length(policy_client_address) = 20), │
│ CONSTRAINT encrypted_data_refs_recipient_pubkey_len CHECK (octet_length(recipient_pubkey) = 32), │
│ CONSTRAINT encrypted_data_refs_signature_len CHECK (octet_length(signature) = 64) │
│ ); │
│ -- Indexes on (user_id, policy_client_address), (policy_client_address), (expires_at) │
│ │
│ New repository: crates/core/src/database/encrypted_data_refs.rs (follows WasmSecretRepository pattern) │
│ │
│ Module Structure │
│ │
│ crates/core/src/crypto/ │
│ mod.rs -- re-exports, feature-gated behind "privacy" │
│ error.rs -- CryptoError enum (thiserror) │
│ hpke.rs -- HPKE encrypt/decrypt │
│ envelope.rs -- SecureEnvelope │
│ ed25519.rs -- Ed25519 derivation + signing │
│ │
│ Register pub mod crypto; in crates/core/src/lib.rs. │
│ │
│ --- │
│ Layer 2: Gateway API │
│ │
│ NEWT-434: CreateTaskRequest Extension │
│ │
│ Modify: crates/gateway/src/rpc/types/mod.rs │
│ │
│ Add 3 optional fields to CreateTaskRequest: │
│ pub encrypted_data_refs: Option<Vec<String>>, // UUIDs │
│ pub user_signature: Option<String>, // Ed25519 hex │
│ pub app_signature: Option<String>, // Ed25519 hex │
│ │
│ All Option<T> for backward compatibility. │
│ │
│ NEWT-435: Storage API │
│ │
│ New types: crates/gateway/src/rpc/types/privacy.rs │
│ - UploadEncryptedDataRequest (policy_client, envelope, signature, recipient_pubkey, ttl) │
│ - UploadEncryptedDataResponse (success, data_ref_id, error) │
│ │
│ New handler: crates/gateway/src/rpc/api/privacy.rs │
│ - upload_encrypted_data() — validates envelope, verifies Ed25519 sig, inserts to DB │
│ - Wired as "newt_uploadEncryptedData" RPC method │
│ │
│ NEWT-433: Auth Validation │
│ │
│ New processor: crates/gateway/src/processor/privacy_auth.rs │
│ - validate_privacy_signatures(&self, request: &CreateTaskRequest) -> Result<()> │
│ - Dual-signature validation: user signs (policy_client + intent_hash + refs), app signs (policy_client + intent_hash + user_sig) │
│ - Called in sync.rs:create_task() when encrypted_data_refs.is_some() │
│ │
│ --- │
│ Layer 3: Operator │
│ │
│ NEWT-432: HPKE Decryption │
│ │
│ Modify: crates/operator/src/core.rs │
│ │
│ In fetch_policy_data() (lines 450-496): │
│ 1. Add encrypted_data_refs: Option<&[EncryptedDataRef]> parameter │
│ 2. Derive Ed25519 from ECDSA key → derive X25519 for HPKE │
│ 3. Decrypt each SecureEnvelope, merge into policyTaskData │
│ │
│ Add ecdsa_key_bytes: [u8; 32] field to OperatorCore struct. │
│ │
│ Enable privacy feature: crates/operator/Cargo.toml │
│ │
│ NEWT-438: PolicyTaskData Extension │
│ │
│ Embed privacy metadata within existing PolicyData.data JSON (no Solidity changes): │
│ { "standard_data": {...}, "privacy": { "encrypted_data_ref_ids": [...], "decryption_proof": "..." } } │
│ │
│ Non-numeric fields pass through consensus median tolerance unchanged. │
│ │
│ --- │
│ Layer 4: Observability + Testing │
│ │
│ NEWT-436: Metrics │
│ │
│ Modify: crates/metrics/src/lib.rs │
│ │
│ Add privacy section using describe_*! macros with gateway_privacy_* and operator_privacy_* prefixes: │
│ - gateway_privacy_upload_total, gateway_privacy_task_total │
│ - gateway_privacy_upload_duration_seconds │
│ - operator_privacy_decryption_duration_seconds, operator_privacy_decryption_total │
│ - gateway_privacy_auth_validation_total, gateway_privacy_auth_validation_duration_seconds │
│ │
│ NEWT-437: E2E Tests │
│ │
│ New test: integration-tests/src/commands/privacy.rs │
│ │
│ Test flow: generate keypair → encrypt → upload → create task with refs + dual sigs → verify decryption → verify BLS aggregation │
│ │
│ New env var: PRIVACY_ENABLED=true │
│ New make target: make gateway_e2e two_phase_consensus=true privacy=true │
│ │
│ --- │
│ File Inventory │
│ │
│ New Files (11) │
│ ┌─────────────────────────────────────────────────────────────────────────┬──────────────────────────────┐ │
│ │ File │ Purpose │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/crypto/mod.rs │ Crypto module root │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/crypto/error.rs │ CryptoError enum │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/crypto/hpke.rs │ HPKE encrypt/decrypt │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/crypto/envelope.rs │ SecureEnvelope │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/crypto/ed25519.rs │ Ed25519 derivation + signing │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/gateway/migrations/20260210100000_create_encrypted_data_refs.sql │ DB migration │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/core/src/database/encrypted_data_refs.rs │ Repository │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/gateway/src/rpc/types/privacy.rs │ Request/response types │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/gateway/src/rpc/api/privacy.rs │ Upload handler │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ crates/gateway/src/processor/privacy_auth.rs │ Dual-signature validation │ │
│ ├─────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤ │
│ │ integration-tests/src/commands/privacy.rs │ E2E test │ │
│ └─────────────────────────────────────────────────────────────────────────┴──────────────────────────────┘ │
│ Modified Files (12) │
│ ┌───────────────────────────────────────┬───────────────────────────────────────────────┐ │
│ │ File │ Change │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ Cargo.toml (workspace) │ Add hpke, ed25519-dalek, x25519-dalek │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/core/Cargo.toml │ Add privacy feature │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/core/src/lib.rs │ Add pub mod crypto; │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/core/src/database/mod.rs │ Add encrypted_data_refs module │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/gateway/src/rpc/types/mod.rs │ Add privacy fields + module │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/gateway/src/rpc/api/mod.rs │ Add privacy module │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/gateway/src/processor/mod.rs │ Add privacy_auth module │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/operator/Cargo.toml │ Enable privacy feature │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/operator/src/core.rs │ Add ecdsa_key_bytes, modify fetch_policy_data │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ crates/metrics/src/lib.rs │ Add privacy metrics │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ integration-tests/src/commands/mod.rs │ Add privacy module │ │
│ ├───────────────────────────────────────┼───────────────────────────────────────────────┤ │
│ │ Makefile │ Add privacy E2E target │ │
│ └───────────────────────────────────────┴───────────────────────────────────────────────┘ │
│ Key Design Decisions │
│ │
│ 1. Feature-gated: #[cfg(feature = "privacy")] so zkVM builds are unaffected │
│ 2. Derive keys from ECDSA: Zero operator configuration changes required │
│ 3. No Solidity changes: Privacy metadata embedded in existing PolicyData.data JSON │
│ 4. All new fields Optional: Backward compatible — non-privacy tasks unchanged │
│ 5. Follow existing patterns: BYTEA+octet_length for DB, describe_*! for metrics, trait injection for validators
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment