PostgreSQL Extension Internals in Rust (Part 1): Crossing the C ↔ Rust FFI Boundary, MemoryContexts, varlena Layout, and Parallel Workers in pg_bio

📚 PostgreSQL & Rust (pgrx) Systems Internals Series This monograph is part of an ongoing systems engineering series exploring how we turned PostgreSQL 18 into a high-performance scientific and multidimensional computing engine using Rust:


When a database query slows down from milliseconds to minutes, application engineers usually blame SQL and reach for an external microservice, a Redis cache, or a distributed Python worker pool. Systems engineers ask a different question: how many bytes are we copying across the kernel socket just to throw 99.9% of them away?

While building pg_bio—our 7,700-line Rust extension that executes 3D structural biology, molecular graph isomorphism, and thermodynamic auditing natively inside PostgreSQL 18—and spiral, we repeatedly confronted the fundamental bottleneck of modern data-intensive architectures: the Serialization and IPC Tax.

In this first installment of the PostgreSQL Extension Internals in Rust series, we are going to open up the hood of PostgreSQL’s C kernel and trace exactly what happens when a SQL query crosses the Foreign Function Interface (FFI) boundary into compiled Rust code via pgrx. We will dissect:

  1. Why moving compute into PostgreSQL’s Shared Buffer Manager yields 36.9x to 1,513.7x speedups over client-side pipelines.
  2. How PostgreSQL’s C Function Manager (fmgr), sigsetjmp/ereport exception stack, and hierarchical MemoryContext arena allocator interact with Rust’s ownership and Drop semantics.
  3. How custom Rust structs (ResidueCoord, SparseAttentionMap, Molecule) are laid out inside PostgreSQL 8KB Heap Pages, 4-byte/1-byte varlena headers, and TOAST out-of-line tables.
  4. How #[pg_extern(immutable, parallel_safe)] teaches the PostgreSQL Cost-Based Query Planner to fan out Rust compute across parallel background workers over Dynamic Shared Memory (DSM).

1. The Architectural Bottleneck: The IPC Serialization Tax vs. Shared Buffer Compute

Consider a standard analytical workload—whether you are screening 400 multi-chain protein complexes (144 heavy atoms each), evaluating 10,000 chemical graphs, or scanning 10 million financial order-book ticks.

In a traditional client-side architecture (Python + SQLAlchemy/psycopg3 + NumPy/RDKit), executing a domain algorithm over $N$ tuples requires five sequential data movements:

\[T_{\text{total}} = \underbrace{T_{\text{buffer\_read}}}_{\text{8KB Page Pin}} + \underbrace{N \cdot (C_{\text{wire\_encode}} + C_{\text{tcp\_socket}} + C_{\text{python\_alloc}})}_{\text{The IPC & Serialization Tax (Dominates Wall-Clock Time)}} + \underbrace{N \cdot C_{\text{compute}}}_{\text{Actual Math}}\]
  1. PostgreSQL Backend: Pins the 8KB page in shared_buffers, extracts each tuple, formats it into PostgreSQL’s wire protocol (DataRow messages), and copies it into the kernel socket buffer.
  2. OS Kernel / TCP Loopback: Context-switches between the postgres backend process and the client process.
  3. Client Runtime (Python/Node/JVM): Allocates heap objects, deserializes text or binary payloads into native structs, and finally runs the algorithm.

When we move that exact same algorithm into a compiled Rust extension inside the PostgreSQL backend process, the middle term vanishes completely:

Workload Benchmark (pg_bio on PostgreSQL 18) Client-Side Pipeline (Python Pull + Deserialization) In-Database Rust (pgrx in Shared Buffers) Measured Speedup
Canonical Substructure & Hash Lookup (10,000 SMILES molecules) 1,018.64 ms 0.673 ms (B-Tree on precomputed Rust wl_hash) 1,513.7x
Multi-Chain 3D Biophysical Audit (400 complexes $\times$ 144 heavy atoms, SASA + H-Bonds + Kabsch RMSD) 7,709.11 ms 209.04 ms (0.52 ms / complex in shared memory) 36.9x
1024-Bit ECFP4 Tanimoto Similarity Scan (10,000 molecules, hardware POPCNT) 1,018.64 ms 434.46 ms (Sequential scan over varlena tuples) 2.3x

Why is the in-database execution up to three orders of magnitude faster? Because the Rust function executes inside the same OS process address space as the PostgreSQL Buffer Manager.


2. Crossing the C ↔ Rust Boundary: fmgr, Unwind Safety, and MemoryContext

PostgreSQL is written in C (C99) and has evolved since 1986 around three core invariants that are completely foreign to standard Rust programs:

  1. The Version-1 Calling Convention (FunctionCallInfo): Every SQL-callable C function receives a single pointer (PG_FUNCTION_ARGS $\rightarrow$ FunctionCallInfo fcinfo) containing an array of NullableDatum arguments and returns a 64-bit Datum.
  2. Arena-Based Memory Management (MemoryContext): PostgreSQL rarely calls raw malloc() or free(). Instead, every allocation happens via palloc() inside a hierarchical MemoryContext (TopTransactionContext $\rightarrow$ ExecutorState $\rightarrow$ ExprContext). When a tuple or query finishes, PostgreSQL wipes the entire arena in $O(1)$ time with MemoryContextReset().
  3. sigsetjmp / siglongjmp Error Handling: When a PostgreSQL C function encounters an error (ereport(ERROR, ...)), it performs a non-local C jump (siglongjmp) all the way back to the top-level transaction loop—Which would immediately trigger Undefined Behavior (UB) and leak heap allocations if it jumped across an active Rust stack frame with pending Drop destructors!

Here is how pgrx bridges these two worlds safely every time a query invokes a function in pg_bio:

sequenceDiagram autonumber participant Exec as PostgreSQL C Executor
(ExecInterpExpr) participant Guard as pgrx #[pg_guard] Wrapper
(C ABI / catch_unwind) participant Rust as pg_bio Rust Function
(#[pg_extern]) participant Mem as PostgreSQL MemoryContext
(ExprContext / palloc) Exec->>Guard: Call fn_addr(fcinfo: FunctionCallInfo) Note over Guard: Enter std::panic::catch_unwind()
Isolate C siglongjmp from Rust Drop frames Guard->>Mem: PG_DETOAST_DATUM() on varlena inputs Mem-->>Guard: Zero-copy slice or detoasted C buffer Guard->>Rust: Borrow &str / &[u8] or deserialize PostgresType Note over Rust: Execute pure Rust math
(SIMD popcount, Kabsch SVD, Z-Order) Rust-->>Guard: Return Rust value (i64, f64, or PostgresType) Guard->>Mem: palloc() output Datum in Caller's MemoryContext Note over Guard: Drop temporary Rust Vec/String allocations cleanly Guard-->>Exec: Return Datum (or convert Panic -> ereport(ERROR)) Note over Exec: Reset per-tuple ExprContext arena in O(1)

How #[pg_extern] Compiles Down to Safe C FFI

Let’s look at the very first functions in pg_bio/src/lib.rs:

use pgrx::prelude::*;
use serde::{Deserialize, Serialize};

// Emits Pg_magic_func required by PostgreSQL's dynamic loader (dfmgr.c)
// to verify PG_VERSION_NUM, block size (8192), and ABI compatibility at LOAD time.
pgrx::pg_module_magic!();

#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct ResidueCoord {
    pub x: f64,
    pub y: f64,
    pub z: f64,
    pub name: String,
}

#[inline]
fn split_by_3(x: u64) -> u64 {
    let mut res = 0;
    for i in 0..21 {
        res |= ((x >> i) & 1) << (3 * i);
    }
    res
}

#[pg_extern(immutable, parallel_safe)]
pub fn z_order_encode(x: f64, y: f64, z: f64) -> i64 {
    let map = |v: f64| -> u64 {
        let v = v.clamp(-500.0, 500.0) + 500.0;
        let scaled = (v * 1000.0) as u64;
        scaled & 0x1FFFFF
    };
    let xx = split_by_3(map(x));
    let yy = split_by_3(map(y));
    let zz = split_by_3(map(z));
    (xx | (yy << 1) | (zz << 2)) as i64
}

When cargo pgrx compiles z_order_encode(x: f64, y: f64, z: f64) -> i64, it generates two things automatically:

  1. An extern "C-unwind" wrapper annotated with #[pg_guard]:
    • It reads fcinfo.args[0], fcinfo.args[1], and fcinfo.args[2] directly as pass-by-value 64-bit Datum registers (zero heap allocations!).
    • It wraps the call in a panic/error boundary so that if Rust panics (e.g., index out of bounds), all Rust local variables are dropped before pgrx triggers PostgreSQL’s C ereport(ERROR). Conversely, if a PostgreSQL C call (pg_sys::...) raises a C elog(ERROR), pgrx intercepts the siglongjmp, converts it into a Rust panic payload to unwind Rust destructors cleanly, and then resumes the PostgreSQL C jump!
  2. The SQL DDL catalog registration:
    CREATE FUNCTION "z_order_encode"(
        "x" double precision,
        "y" double precision,
        "z" double precision
    ) RETURNS bigint
    IMMUTABLE STRICT PARALLEL SAFE
    LANGUAGE c AS 'MODULE_PATHNAME', 'z_order_encode_wrapper';
    

3. Inside the 8KB Page: Designing Custom varlena Types & Mastering TOAST

Pass-by-value scalar types like i64 (bigint) and f64 (double precision) fit inside a single 8-byte CPU register (Datum). Realistic domain objects—like a 3D atomic coordinate with an atom label (ResidueCoord), a Compressed Sparse Row attention matrix (SparseAttentionMap), or a normalized chemical graph (Molecule)—are variable-length structures.

In PostgreSQL’s storage engine, all variable-length data types (text, bytea, jsonb, vector, and every #[derive(PostgresType)] struct in pg_bio) are stored as struct varlena inside 8KB (8192-byte) Heap Pages.

flowchart LR subgraph Page["PostgreSQL 8KB (8,192-Byte) Heap Page"] direction TB PH["PageHeaderData (24B)\npd_lsn, pd_lower, pd_upper"] --> LP["Line Pointers (ItemIdData, 4B each)\nlp_off, lp_flags, lp_len"] LP --> TUP["HeapTupleHeaderData (23B + null bitmap)\nt_xmin, t_xmax, t_cid, t_ctid, t_infomask"] TUP --> COL1["Col 1: uniprot_id\n(1B short varlena)"] TUP --> COL2["Col 2: z_index (i64)\n8B Pass-by-Value Datum"] TUP --> COL3["Col 3: #[derive(PostgresType)]\n4B va_header + CBOR/Binary Payload"] end COL3 -->|"Payload <= 2,000 Bytes\n(ResidueCoord / Molecule)"| INLINE["Inline Uncompressed or\n1-Byte Short Header (va_header_1b)"] COL3 -->|"Payload > 2,000 Bytes\n(SparseAttentionMap CSR)"| TOAST["TOAST Subsystem (tuptoaster.c)\n1. LZ4 / pglz Inline Compression\n2. Out-of-Line pg_toast_oid Chunks (~2KB)"]

How #[derive(PostgresType)] Layouts Work in pg_bio

Let’s compare three custom types from pg_bio/src/lib.rs and see how their physical byte size dictates their behavior inside PostgreSQL’s TOAST (The Oversized-Attribute Storage Technique) subsystem:

// Type 1: Small 3D Coordinate (~32–48 bytes serialized)
// Always stays INLINE inside the 8KB heap tuple with a 1-byte short varlena header!
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct ResidueCoord {
    pub x: f64,
    pub y: f64,
    pub z: f64,
    pub name: String,
}

// Type 2: Medium Normalized Chemical Graph (~250–600 bytes serialized)
// Embeds precomputed 64-bit WL hash & 1024-bit ECFP4 fingerprint (16 x u64 = 128 bytes)
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Molecule {
    pub smiles: String,
    pub is_valid: bool,
    #[serde(default)]
    pub wl_hash: i64,
    #[serde(default)]
    pub ecfp_bits: Vec<u64>,
    #[serde(default)]
    pub atoms: Vec<PgBioAtom>,
    #[serde(default)]
    pub bonds: Vec<PgBioBond>,
}

// Type 3: Large Compressed Sparse Row (CSR) Matrix (1 KB – 50 KB serialized)
// Automatically compressed in-place (LZ4/pglz) or sliced into pg_toast when > 2 KB!
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct SparseAttentionMap {
    pub sequence_length: i32,
    pub entries: Vec<AttentionEntry>,
}

The Three Golden Rules of varlena Performance in Rust Extensions

When a SQL query reads a column of type Molecule or ResidueCoord, pgrx calls PG_DETOAST_DATUM and deserializes the payload into a Rust struct inside the current expression’s MemoryContext. While infinitely faster than network I/O, repeatedly detoasting and deserializing a varlena column inside an inner loop of 2.34 million rows still burns CPU cycles.

In pg_bio, we apply three storage-engine patterns to eliminate unnecessary detoasting:

  1. Extract Hot Index Keys into Pass-by-Value Generated Columns (8-Byte Datum): Never force a B-Tree index to call a varlena deserialization function during tree traversal. Instead, project the 64-bit invariant into a BIGINT GENERATED ALWAYS AS (...) STORED column:
    CREATE TABLE protein_atoms (
        atom_id SERIAL PRIMARY KEY,
        uniprot_id TEXT NOT NULL,
        coord ResidueCoord NOT NULL,
        -- Precomputed once at INSERT time; stored as a raw 8-byte pass-by-value int8!
        z_index BIGINT GENERATED ALWAYS AS (residue_z_index(coord)) STORED
    );
    
    CREATE INDEX idx_atoms_z_order ON protein_atoms USING btree (z_index);
    

    When executing a 3D bounding-box query (WHERE z_index BETWEEN lo AND hi), PostgreSQL traverses idx_atoms_z_order comparing raw 64-bit CPU integers at native memory bandwidth (~7 ms across 200,000 atoms), and only detoasts coord (ResidueCoord) for the tiny fraction of rows that survive the B-Tree range scan!

  2. Keep Compact Structs Under the TOAST_TUPLE_THRESHOLD (2,000 Bytes): PostgreSQL triggers the TOAST compressor (tuptoaster.c) when a tuple exceeds TOAST_TUPLE_THRESHOLD ($\approx 2\text{ KB}$). By encoding PgBioAtom (5 bytes) and PgBioBond (6 bytes) with compact u8/u16 indices and packing the 1024-bit Morgan ECFP4 fingerprint into sixteen u64 words (128 bytes), a drug-like Molecule fits in ~350 bytes. It never spills to the out-of-line pg_toast table, avoiding secondary index lookups on chunk_id.

  3. Replace Dense $N \times N$ Arrays with Sparse CSR Custom Types (SparseAttentionMap): A dense $1000 \times 1000$ float4[] contact map occupies 4 MB—forcing PostgreSQL to slice it into 2,000 separate out-of-line TOAST chunks across pg_toast. By storing only non-zero contacts in SparseAttentionMap (entries: Vec<AttentionEntry>), the payload shrinks by $>95\%$, staying inside inline compressed shared buffers.

4. Unlocking Parallel Query Workers: Why #[pg_extern(immutable, parallel_safe)] Matters

Look closely at the macro attributes on almost every pure compute function in pg_bio/src/lib.rs:

#[pg_extern(immutable, parallel_safe)]
pub fn smith_waterman_score(seq1: &str, seq2: &str, match_score: i32, mismatch: i32, gap: i32) -> i32 { ... }

If you write #[pg_extern] without arguments, pgrx defaults to PostgreSQL’s conservative defaults: VOLATILE and PARALLEL UNSAFE.

What happens to the PostgreSQL Query Planner when a function in your SELECT list or WHERE clause is VOLATILE and PARALLEL UNSAFE?

  1. No Index Usage or Constant Folding: If you write WHERE z_index >= z_order_encode(10.0, 10.0, 10.0) and z_order_encode is VOLATILE, PostgreSQL assumes the function might return a different number on every row (like random()). It refuses to use the B-Tree index and falls back to a full Sequential Scan, re-evaluating the constant boundary 2.34 million times! Marking it immutable tells the planner to evaluate z_order_encode(10.0, 10.0, 10.0) once during planning (constant folding) and push the resulting 64-bit integer directly into an Index Cond.
  2. Parallel Query Serialization: If any function in a query is PARALLEL UNSAFE, PostgreSQL disables parallel background workers (max_parallel_workers_per_gather). By marking our pure Rust functions parallel_safe, PostgreSQL’s executor spins up parallel worker processes that attach to the leader’s Dynamic Shared Memory (DSM) segment and divide the 8KB heap pages across all available CPU cores via an atomic Parallel Block Table Scan:
-- Parallelized In-Database Tanimoto + Smith-Waterman Screening
EXPLAIN (ANALYZE, BUFFERS)
SELECT uniprot_id,
       smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ', 2, -1, -2) AS sw_score
FROM proteins
WHERE smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ', 2, -1, -2) > 35;
                                                       QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..84210.45 rows=412 width=18) (actual time=1.842..38.419 rows=19 loops=1)
   Workers Planned: 4
   Workers Launched: 4
   Buffers: shared hit=18420
   ->  Parallel Seq Scan on proteins  (cost=0.00..83169.25 rows=103 width=18) (actual time=0.912..34.105 rows=4 loops=5)
         Filter: (smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ'::text, 2, '-1'::integer, '-2'::integer) > 35)
         Rows Removed by Filter: 468006
         Buffers: shared hit=18420
 Planning Time: 0.142 ms
 Execution Time: 38.610 ms

Because smith_waterman_score is parallel_safe and allocates only two tiny $O(M)$ stack/heap vectors inside each worker’s per-tuple ExprContext, 5 processes (1 Leader + 4 Parallel Workers) chew through 2.34 million sequences in shared buffers (shared hit=18420, zero disk reads) in 38.6 milliseconds!


5. Interactive Simulator: Inside an 8KB PostgreSQL Page (varlena, TOAST & IPC Cost)

Use the interactive simulator below to inspect how different scientific and analytical data structures are packed into an 8KB (8,192-byte) PostgreSQL Heap Page, when the TOAST threshold (2,000 bytes) triggers inline compression or out-of-line slicing, and how In-Database Rust (pgrx) compares to a client-side Python pull over 10,000 rows:

Interactive Storage Engine Inspector

8KB Heap Page, varlena Header & TOAST Layout Simulator

Select a data structure to inspect its physical tuple footprint inside PostgreSQL 18 shared buffers.

Physical Tuple Size
69 Bytes
23B HeapHdr + 8B int8 + 38B varlena
Tuples per 8KB Page
111 rows / page
99.1% Page Density (4B ItemId/row)
TOAST Storage State
INLINE (1B Short Hdr)
0 pg_toast chunks • Zero decompression
10k-Row Query Latency
0.48 ms vs 410 ms
pgrx Shared Buffers vs Python Pull
Byte 0: PageHeaderData (24B) + Line Pointers (pd_lower) 8,192 Bytes Total (1 Block in shared_buffers)
Hdr
Free
111 Inline Heap Tuples (Growing ← from byte 8192)

Storage Kernel Verdict: Because ResidueCoord is under 127 bytes, PostgreSQL uses a 1-byte short varlena header (va_header_1b) instead of a 4-byte header, packing 111 atoms per 8KB page. Combined with the pass-by-value z_index BIGINT column, B-Tree index scans never even touch the varlena payload until the final 3D distance recheck!


6. Beyond Bioinformatics: How These Patterns Apply to Any Database Domain

While pg_bio uses these storage-engine mechanics to mine 2.34 million extremophile proteins and screen molecular graphs, the exact same PostgreSQL C-Kernel + Rust (pgrx) architecture solves high-throughput bottlenecks across industry:

Database Engineering Pattern in pg_bio Bioinformatics Implementation (pg_bio) General Systems / Industry Equivalent
Pass-by-Value 64-Bit Spatial Projection (GENERATED ALWAYS AS ... STORED) z_order_encode(x, y, z) maps 3D atomic coordinates (ResidueCoord) into a 1D B-Tree integer (i64) Geospatial Fleet Routing & Game Telemetry: Bit-interleaving (lat, lon, timestamp) for sub-millisecond B-Tree bounding-box queries (see spiral)
Write-Time Graph Canonicalization + 64-Bit Hash Hückel $4n+2$ cycle normalization + molecule_wl_hash (0.67 ms lookup across 10k molecules) Fintech Fraud & Entity Resolution: Normalizing transaction subgraphs or JSON schemas at INSERT time for $O(\log N)$ exact isomorphism checks
Compact Inline varlena CSR Matrices (SparseAttentionMap) Compressing $N \times N$ ESMFold / Hi-C contact matrices to avoid out-of-line TOAST thrashing E-Commerce & AdTech Co-Occurrence Graphs: Storing sparse user-item interaction vectors inside 8KB heap pages without external graph DBs
#[pg_extern(immutable, parallel_safe)] Zero-Copy Kernels Parallel Smith-Waterman alignment (smith_waterman_score) & Kabsch 3D superposition (kabsch_rmsd) High-Frequency Financial Tick Analytics: Running custom SIMD rolling-window pricing or order-book replay across PostgreSQL parallel workers

What’s Next in Part 2

Now that we know how custom Rust structs live inside PostgreSQL 8KB pages and how #[pg_extern(immutable, parallel_safe)] crosses the C FFI boundary safely, how do we teach the PostgreSQL Cost-Based Query Planner to orchestrate custom operators like @> (VF2 Subgraph Isomorphism & RangeFinder Pocket Containment), % (1024-bit hardware POPCNT Tanimoto similarity), and <~> (Hybrid Vector + Smith-Waterman distance)?

In the next installment, we will dive into pgrx::extension_sql!, operator classes, commutator/negator planner hints, and multi-stage index recheck pipelines.


⚡ Powered by pg_bio on PostgreSQL 18 + Rust (pgrx): This discovery was executed natively inside PostgreSQL shared buffers using zero-copy Rust extensions (smith_waterman_score, 3D Kabsch SVD superposition, Z-Order B-Trees, and pgvector HNSW).

📚 Explore the Systems Architecture: PostgreSQL Query Planner & GiST R-Trees • Introducing pg_bio • Hybrid Operator <~> in Rust • The 27-Hour HNSW Index • More Bioinformatics Discoveries • Systems Research & CV

Jônatas Davi Paganini

Jônatas Davi Paganini

Jônatas Davi Paganini — Systems Researcher & Database Internals Engineer (20+ years experience). Creator of pg_bio (in-database structural biology & chemoinformatics engine for PostgreSQL in Rust) and spiral (geometry-aware multidimensional storage & Z-order indexing). Former Developer Advocate at Timescale.

1/1