📚 PostgreSQL & Rust (
pgrx) Systems Internals Series This monograph is part of an ongoing systems engineering series exploring how we turned PostgreSQL 18 into a high-performance scientific and multidimensional computing engine using Rust:
- Part 1: Crossing the C ↔ Rust FFI Boundary (
MemoryContext,varlena, TOAST & Parallel Workers) (You are here)- Companion Systems Monograph: Delegating RangeFinder Pocket Intervals, GiST R-Trees & 3D Cheminformatics to the Query Planner
- Part 2: Writing a Hybrid Database Operator in Rust (
<~>)- Part 3: Indexing 3D Space Like a Video Game (Morton Z-Order B-Trees)
- Part 4: The 27-Hour Index — HNSW Memory Spilling &
maintenance_work_mem- Storage Engine Monograph: Spiral — Geometry-Aware Multidimensional Storage & O(1) Page Seeks
When a database query slows down from milliseconds to minutes, application engineers usually blame SQL and reach for an external microservice, a Redis cache, or a distributed Python worker pool. Systems engineers ask a different question: how many bytes are we copying across the kernel socket just to throw 99.9% of them away?
While building pg_bio—our 7,700-line Rust extension that executes 3D structural biology, molecular graph isomorphism, and thermodynamic auditing natively inside PostgreSQL 18—and spiral, we repeatedly confronted the fundamental bottleneck of modern data-intensive architectures: the Serialization and IPC Tax.
In this first installment of the PostgreSQL Extension Internals in Rust series, we are going to open up the hood of PostgreSQL’s C kernel and trace exactly what happens when a SQL query crosses the Foreign Function Interface (FFI) boundary into compiled Rust code via pgrx. We will dissect:
- Why moving compute into PostgreSQL’s Shared Buffer Manager yields 36.9x to 1,513.7x speedups over client-side pipelines.
- How PostgreSQL’s C Function Manager (
fmgr),sigsetjmp/ereportexception stack, and hierarchicalMemoryContextarena allocator interact with Rust’s ownership andDropsemantics. - How custom Rust structs (
ResidueCoord,SparseAttentionMap,Molecule) are laid out inside PostgreSQL 8KB Heap Pages, 4-byte/1-bytevarlenaheaders, and TOAST out-of-line tables. - How
#[pg_extern(immutable, parallel_safe)]teaches the PostgreSQL Cost-Based Query Planner to fan out Rust compute across parallel background workers over Dynamic Shared Memory (DSM).
1. The Architectural Bottleneck: The IPC Serialization Tax vs. Shared Buffer Compute
Consider a standard analytical workload—whether you are screening 400 multi-chain protein complexes (144 heavy atoms each), evaluating 10,000 chemical graphs, or scanning 10 million financial order-book ticks.
In a traditional client-side architecture (Python + SQLAlchemy/psycopg3 + NumPy/RDKit), executing a domain algorithm over $N$ tuples requires five sequential data movements:
\[T_{\text{total}} = \underbrace{T_{\text{buffer\_read}}}_{\text{8KB Page Pin}} + \underbrace{N \cdot (C_{\text{wire\_encode}} + C_{\text{tcp\_socket}} + C_{\text{python\_alloc}})}_{\text{The IPC & Serialization Tax (Dominates Wall-Clock Time)}} + \underbrace{N \cdot C_{\text{compute}}}_{\text{Actual Math}}\]- PostgreSQL Backend: Pins the 8KB page in
shared_buffers, extracts each tuple, formats it into PostgreSQL’s wire protocol (DataRowmessages), and copies it into the kernel socket buffer. - OS Kernel / TCP Loopback: Context-switches between the
postgresbackend process and the client process. - Client Runtime (Python/Node/JVM): Allocates heap objects, deserializes text or binary payloads into native structs, and finally runs the algorithm.
When we move that exact same algorithm into a compiled Rust extension inside the PostgreSQL backend process, the middle term vanishes completely:
Workload Benchmark (pg_bio on PostgreSQL 18) |
Client-Side Pipeline (Python Pull + Deserialization) | In-Database Rust (pgrx in Shared Buffers) |
Measured Speedup |
|---|---|---|---|
Canonical Substructure & Hash Lookup (10,000 SMILES molecules) |
1,018.64 ms |
0.673 ms (B-Tree on precomputed Rust wl_hash) |
1,513.7x |
Multi-Chain 3D Biophysical Audit (400 complexes $\times$ 144 heavy atoms, SASA + H-Bonds + Kabsch RMSD) |
7,709.11 ms |
209.04 ms (0.52 ms / complex in shared memory) |
36.9x |
1024-Bit ECFP4 Tanimoto Similarity Scan (10,000 molecules, hardware POPCNT) |
1,018.64 ms |
434.46 ms (Sequential scan over varlena tuples) |
2.3x |
Why is the in-database execution up to three orders of magnitude faster? Because the Rust function executes inside the same OS process address space as the PostgreSQL Buffer Manager.
2. Crossing the C ↔ Rust Boundary: fmgr, Unwind Safety, and MemoryContext
PostgreSQL is written in C (C99) and has evolved since 1986 around three core invariants that are completely foreign to standard Rust programs:
- The Version-1 Calling Convention (
FunctionCallInfo): Every SQL-callable C function receives a single pointer (PG_FUNCTION_ARGS$\rightarrow$FunctionCallInfo fcinfo) containing an array ofNullableDatumarguments and returns a 64-bitDatum. - Arena-Based Memory Management (
MemoryContext): PostgreSQL rarely calls rawmalloc()orfree(). Instead, every allocation happens viapalloc()inside a hierarchicalMemoryContext(TopTransactionContext$\rightarrow$ExecutorState$\rightarrow$ExprContext). When a tuple or query finishes, PostgreSQL wipes the entire arena in $O(1)$ time withMemoryContextReset(). sigsetjmp/siglongjmpError Handling: When a PostgreSQL C function encounters an error (ereport(ERROR, ...)), it performs a non-local C jump (siglongjmp) all the way back to the top-level transaction loop—Which would immediately trigger Undefined Behavior (UB) and leak heap allocations if it jumped across an active Rust stack frame with pendingDropdestructors!
Here is how pgrx bridges these two worlds safely every time a query invokes a function in pg_bio:
(ExecInterpExpr) participant Guard as pgrx #[pg_guard] Wrapper
(C ABI / catch_unwind) participant Rust as pg_bio Rust Function
(#[pg_extern]) participant Mem as PostgreSQL MemoryContext
(ExprContext / palloc) Exec->>Guard: Call fn_addr(fcinfo: FunctionCallInfo) Note over Guard: Enter std::panic::catch_unwind()
Isolate C siglongjmp from Rust Drop frames Guard->>Mem: PG_DETOAST_DATUM() on varlena inputs Mem-->>Guard: Zero-copy slice or detoasted C buffer Guard->>Rust: Borrow &str / &[u8] or deserialize PostgresType Note over Rust: Execute pure Rust math
(SIMD popcount, Kabsch SVD, Z-Order) Rust-->>Guard: Return Rust value (i64, f64, or PostgresType) Guard->>Mem: palloc() output Datum in Caller's MemoryContext Note over Guard: Drop temporary Rust Vec/String allocations cleanly Guard-->>Exec: Return Datum (or convert Panic -> ereport(ERROR)) Note over Exec: Reset per-tuple ExprContext arena in O(1)
How #[pg_extern] Compiles Down to Safe C FFI
Let’s look at the very first functions in pg_bio/src/lib.rs:
use pgrx::prelude::*;
use serde::{Deserialize, Serialize};
// Emits Pg_magic_func required by PostgreSQL's dynamic loader (dfmgr.c)
// to verify PG_VERSION_NUM, block size (8192), and ABI compatibility at LOAD time.
pgrx::pg_module_magic!();
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct ResidueCoord {
pub x: f64,
pub y: f64,
pub z: f64,
pub name: String,
}
#[inline]
fn split_by_3(x: u64) -> u64 {
let mut res = 0;
for i in 0..21 {
res |= ((x >> i) & 1) << (3 * i);
}
res
}
#[pg_extern(immutable, parallel_safe)]
pub fn z_order_encode(x: f64, y: f64, z: f64) -> i64 {
let map = |v: f64| -> u64 {
let v = v.clamp(-500.0, 500.0) + 500.0;
let scaled = (v * 1000.0) as u64;
scaled & 0x1FFFFF
};
let xx = split_by_3(map(x));
let yy = split_by_3(map(y));
let zz = split_by_3(map(z));
(xx | (yy << 1) | (zz << 2)) as i64
}
When cargo pgrx compiles z_order_encode(x: f64, y: f64, z: f64) -> i64, it generates two things automatically:
- An extern
"C-unwind"wrapper annotated with#[pg_guard]:- It reads
fcinfo.args[0],fcinfo.args[1], andfcinfo.args[2]directly as pass-by-value 64-bitDatumregisters (zero heap allocations!). - It wraps the call in a panic/error boundary so that if Rust panics (e.g., index out of bounds), all Rust local variables are dropped before
pgrxtriggers PostgreSQL’s Cereport(ERROR). Conversely, if a PostgreSQL C call (pg_sys::...) raises a Celog(ERROR),pgrxintercepts thesiglongjmp, converts it into a Rust panic payload to unwind Rust destructors cleanly, and then resumes the PostgreSQL C jump!
- It reads
- The SQL DDL catalog registration:
CREATE FUNCTION "z_order_encode"( "x" double precision, "y" double precision, "z" double precision ) RETURNS bigint IMMUTABLE STRICT PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'z_order_encode_wrapper';
3. Inside the 8KB Page: Designing Custom varlena Types & Mastering TOAST
Pass-by-value scalar types like i64 (bigint) and f64 (double precision) fit inside a single 8-byte CPU register (Datum). Realistic domain objects—like a 3D atomic coordinate with an atom label (ResidueCoord), a Compressed Sparse Row attention matrix (SparseAttentionMap), or a normalized chemical graph (Molecule)—are variable-length structures.
In PostgreSQL’s storage engine, all variable-length data types (text, bytea, jsonb, vector, and every #[derive(PostgresType)] struct in pg_bio) are stored as struct varlena inside 8KB (8192-byte) Heap Pages.
How #[derive(PostgresType)] Layouts Work in pg_bio
Let’s compare three custom types from pg_bio/src/lib.rs and see how their physical byte size dictates their behavior inside PostgreSQL’s TOAST (The Oversized-Attribute Storage Technique) subsystem:
// Type 1: Small 3D Coordinate (~32–48 bytes serialized)
// Always stays INLINE inside the 8KB heap tuple with a 1-byte short varlena header!
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct ResidueCoord {
pub x: f64,
pub y: f64,
pub z: f64,
pub name: String,
}
// Type 2: Medium Normalized Chemical Graph (~250–600 bytes serialized)
// Embeds precomputed 64-bit WL hash & 1024-bit ECFP4 fingerprint (16 x u64 = 128 bytes)
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Molecule {
pub smiles: String,
pub is_valid: bool,
#[serde(default)]
pub wl_hash: i64,
#[serde(default)]
pub ecfp_bits: Vec<u64>,
#[serde(default)]
pub atoms: Vec<PgBioAtom>,
#[serde(default)]
pub bonds: Vec<PgBioBond>,
}
// Type 3: Large Compressed Sparse Row (CSR) Matrix (1 KB – 50 KB serialized)
// Automatically compressed in-place (LZ4/pglz) or sliced into pg_toast when > 2 KB!
#[derive(PostgresType, Serialize, Deserialize, Debug, Clone)]
pub struct SparseAttentionMap {
pub sequence_length: i32,
pub entries: Vec<AttentionEntry>,
}
The Three Golden Rules of varlena Performance in Rust Extensions
When a SQL query reads a column of type Molecule or ResidueCoord, pgrx calls PG_DETOAST_DATUM and deserializes the payload into a Rust struct inside the current expression’s MemoryContext. While infinitely faster than network I/O, repeatedly detoasting and deserializing a varlena column inside an inner loop of 2.34 million rows still burns CPU cycles.
In pg_bio, we apply three storage-engine patterns to eliminate unnecessary detoasting:
- Extract Hot Index Keys into Pass-by-Value Generated Columns (
8-Byte Datum): Never force a B-Tree index to call avarlenadeserialization function during tree traversal. Instead, project the 64-bit invariant into aBIGINT GENERATED ALWAYS AS (...) STOREDcolumn:CREATE TABLE protein_atoms ( atom_id SERIAL PRIMARY KEY, uniprot_id TEXT NOT NULL, coord ResidueCoord NOT NULL, -- Precomputed once at INSERT time; stored as a raw 8-byte pass-by-value int8! z_index BIGINT GENERATED ALWAYS AS (residue_z_index(coord)) STORED ); CREATE INDEX idx_atoms_z_order ON protein_atoms USING btree (z_index);When executing a 3D bounding-box query (
WHERE z_index BETWEEN lo AND hi), PostgreSQL traversesidx_atoms_z_ordercomparing raw 64-bit CPU integers at native memory bandwidth (~7 msacross 200,000 atoms), and only detoastscoord(ResidueCoord) for the tiny fraction of rows that survive the B-Tree range scan! -
Keep Compact Structs Under the
TOAST_TUPLE_THRESHOLD(2,000Bytes): PostgreSQL triggers the TOAST compressor (tuptoaster.c) when a tuple exceedsTOAST_TUPLE_THRESHOLD($\approx 2\text{ KB}$). By encodingPgBioAtom(5 bytes) andPgBioBond(6 bytes) with compactu8/u16indices and packing the 1024-bit Morgan ECFP4 fingerprint into sixteenu64words (128 bytes), a drug-likeMoleculefits in~350 bytes. It never spills to the out-of-linepg_toasttable, avoiding secondary index lookups onchunk_id. - Replace Dense $N \times N$ Arrays with Sparse CSR Custom Types (
SparseAttentionMap): A dense $1000 \times 1000$float4[]contact map occupies4 MB—forcing PostgreSQL to slice it into 2,000 separate out-of-line TOAST chunks acrosspg_toast. By storing only non-zero contacts inSparseAttentionMap(entries: Vec<AttentionEntry>), the payload shrinks by $>95\%$, staying inside inline compressed shared buffers.
4. Unlocking Parallel Query Workers: Why #[pg_extern(immutable, parallel_safe)] Matters
Look closely at the macro attributes on almost every pure compute function in pg_bio/src/lib.rs:
#[pg_extern(immutable, parallel_safe)]
pub fn smith_waterman_score(seq1: &str, seq2: &str, match_score: i32, mismatch: i32, gap: i32) -> i32 { ... }
If you write #[pg_extern] without arguments, pgrx defaults to PostgreSQL’s conservative defaults: VOLATILE and PARALLEL UNSAFE.
What happens to the PostgreSQL Query Planner when a function in your SELECT list or WHERE clause is VOLATILE and PARALLEL UNSAFE?
- No Index Usage or Constant Folding: If you write
WHERE z_index >= z_order_encode(10.0, 10.0, 10.0)andz_order_encodeisVOLATILE, PostgreSQL assumes the function might return a different number on every row (likerandom()). It refuses to use the B-Tree index and falls back to a full Sequential Scan, re-evaluating the constant boundary 2.34 million times! Marking itimmutabletells the planner to evaluatez_order_encode(10.0, 10.0, 10.0)once during planning (constant folding) and push the resulting 64-bit integer directly into anIndex Cond. - Parallel Query Serialization: If any function in a query is
PARALLEL UNSAFE, PostgreSQL disables parallel background workers (max_parallel_workers_per_gather). By marking our pure Rust functionsparallel_safe, PostgreSQL’s executor spins up parallel worker processes that attach to the leader’s Dynamic Shared Memory (DSM) segment and divide the 8KB heap pages across all available CPU cores via an atomicParallel Block Table Scan:
-- Parallelized In-Database Tanimoto + Smith-Waterman Screening
EXPLAIN (ANALYZE, BUFFERS)
SELECT uniprot_id,
smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ', 2, -1, -2) AS sw_score
FROM proteins
WHERE smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ', 2, -1, -2) > 35;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------
Gather (cost=1000.00..84210.45 rows=412 width=18) (actual time=1.842..38.419 rows=19 loops=1)
Workers Planned: 4
Workers Launched: 4
Buffers: shared hit=18420
-> Parallel Seq Scan on proteins (cost=0.00..83169.25 rows=103 width=18) (actual time=0.912..34.105 rows=4 loops=5)
Filter: (smith_waterman_score(sequence, 'MKTAYIAKQRQISFVKSHFSRQ'::text, 2, '-1'::integer, '-2'::integer) > 35)
Rows Removed by Filter: 468006
Buffers: shared hit=18420
Planning Time: 0.142 ms
Execution Time: 38.610 ms
Because smith_waterman_score is parallel_safe and allocates only two tiny $O(M)$ stack/heap vectors inside each worker’s per-tuple ExprContext, 5 processes (1 Leader + 4 Parallel Workers) chew through 2.34 million sequences in shared buffers (shared hit=18420, zero disk reads) in 38.6 milliseconds!
5. Interactive Simulator: Inside an 8KB PostgreSQL Page (varlena, TOAST & IPC Cost)
Use the interactive simulator below to inspect how different scientific and analytical data structures are packed into an 8KB (8,192-byte) PostgreSQL Heap Page, when the TOAST threshold (2,000 bytes) triggers inline compression or out-of-line slicing, and how In-Database Rust (pgrx) compares to a client-side Python pull over 10,000 rows:
8KB Heap Page, varlena Header & TOAST Layout Simulator
Select a data structure to inspect its physical tuple footprint inside PostgreSQL 18 shared buffers.
Storage Kernel Verdict: Because ResidueCoord is under 127 bytes, PostgreSQL uses a 1-byte short varlena header (va_header_1b) instead of a 4-byte header, packing 111 atoms per 8KB page. Combined with the pass-by-value z_index BIGINT column, B-Tree index scans never even touch the varlena payload until the final 3D distance recheck!
6. Beyond Bioinformatics: How These Patterns Apply to Any Database Domain
While pg_bio uses these storage-engine mechanics to mine 2.34 million extremophile proteins and screen molecular graphs, the exact same PostgreSQL C-Kernel + Rust (pgrx) architecture solves high-throughput bottlenecks across industry:
Database Engineering Pattern in pg_bio |
Bioinformatics Implementation (pg_bio) |
General Systems / Industry Equivalent |
|---|---|---|
Pass-by-Value 64-Bit Spatial Projection (GENERATED ALWAYS AS ... STORED) |
z_order_encode(x, y, z) maps 3D atomic coordinates (ResidueCoord) into a 1D B-Tree integer (i64) |
Geospatial Fleet Routing & Game Telemetry: Bit-interleaving (lat, lon, timestamp) for sub-millisecond B-Tree bounding-box queries (see spiral) |
| Write-Time Graph Canonicalization + 64-Bit Hash | Hückel $4n+2$ cycle normalization + molecule_wl_hash (0.67 ms lookup across 10k molecules) |
Fintech Fraud & Entity Resolution: Normalizing transaction subgraphs or JSON schemas at INSERT time for $O(\log N)$ exact isomorphism checks |
Compact Inline varlena CSR Matrices (SparseAttentionMap) |
Compressing $N \times N$ ESMFold / Hi-C contact matrices to avoid out-of-line TOAST thrashing | E-Commerce & AdTech Co-Occurrence Graphs: Storing sparse user-item interaction vectors inside 8KB heap pages without external graph DBs |
#[pg_extern(immutable, parallel_safe)] Zero-Copy Kernels |
Parallel Smith-Waterman alignment (smith_waterman_score) & Kabsch 3D superposition (kabsch_rmsd) |
High-Frequency Financial Tick Analytics: Running custom SIMD rolling-window pricing or order-book replay across PostgreSQL parallel workers |
What’s Next in Part 2
Now that we know how custom Rust structs live inside PostgreSQL 8KB pages and how #[pg_extern(immutable, parallel_safe)] crosses the C FFI boundary safely, how do we teach the PostgreSQL Cost-Based Query Planner to orchestrate custom operators like @> (VF2 Subgraph Isomorphism & RangeFinder Pocket Containment), % (1024-bit hardware POPCNT Tanimoto similarity), and <~> (Hybrid Vector + Smith-Waterman distance)?
In the next installment, we will dive into pgrx::extension_sql!, operator classes, commutator/negator planner hints, and multi-stage index recheck pipelines.
⚡ Powered by pg_bio on PostgreSQL 18 + Rust (pgrx): This discovery was executed natively inside PostgreSQL shared buffers using zero-copy Rust extensions (smith_waterman_score, 3D Kabsch SVD superposition, Z-Order B-Trees, and pgvector HNSW).
📚 Explore the Systems Architecture: PostgreSQL Query Planner & GiST R-Trees • Introducing pg_bio • Hybrid Operator <~> in Rust • The 27-Hour HNSW Index • More Bioinformatics Discoveries • Systems Research & CV