The HyperBEAM Book (Pre-Release)
This book teaches you to build on HyperBEAM — the OS for the verifiable internet. Start with the foundation, learn Erlang, then work through the codebase from core resolution to device implementation. By the end, you'll understand every module and be ready to build.
Contents
- HyperBEAM
- Setup
- Erlang
- Project Structure
- Arweave Utils
- HyperBEAM Core
- Devices
- Building Your Own Devices
Prologue: HyperBEAM
The foundation of the AI era: the verifiable internet. AO-Core extends HTTP with verifiability. HyperBEAM is the OS. Learn the stack (messages, devices, processes, hashpath, attestation), then what you can build on top.
Chapter 1: Setup
Get HyperBEAM running on your machine.
Chapter 2: Erlang
HyperBEAM is written in Erlang. You cannot read the code without learning the language first.
Chapter 3: Project Structure
Before reading modules, understand how the codebase is organized.
Chapter 4: Arweave Utils
Arweave blockchain primitives. HyperBEAM stores everything on Arweave—these modules are the foundation.
| Module | Description |
|---|---|
| rsa_pss | RSA-PSS signature algorithm per RFC 3447, used by ar_wallet for all cryptographic operations |
| ar_wallet | Generates RSA-4096 keypairs, loads JWK keyfiles, signs data, verifies signatures, derives addresses |
| ar_deep_hash | Computes Arweave's recursive SHA-384 hash for transaction and data item ID generation |
| ar_bundles | Creates, signs, serializes, and deserializes ANS-104 data items and bundles |
| ar_tx | Handles Layer 1 Arweave transactions including creation, signing, and serialization |
| ar_timestamp | Fetches current block height and timestamp from Arweave network for time synchronization |
| ar_rate_limiter | Token bucket rate limiter to prevent overwhelming Arweave gateways with requests |
Chapter 5: HyperBEAM Core
The runtime environment that devices execute in. Understanding these modules helps you write better devices and debug issues.
You'll start with pure utilities, then learn how data flows through the system — from configuration to storage to network to the protocol that ties everything together.
1. Utilities
Pure helper functions used throughout the codebase.
| Module | Description |
|---|---|
| hb_util | Base64url encoding, unique ID generation, type coercion, and debug printing utilities |
| hb_json | JSON encoding and decoding wrapper supporting maps, lists, binaries, and primitives |
| hb_escape | URI percent-encoding, path segment escaping, and safe string handling |
| hb_structured_fields | RFC 9651 HTTP Structured Fields parser for dictionaries, lists, and typed values |
| hb_keccak | Keccak-256 hashing via NIF, also derives Ethereum addresses from public keys |
| hb_crypto | Multi-algorithm signing helpers for RSA, ECDSA, Ed25519 with format conversion |
2. Configuration
How HyperBEAM settings flow from defaults through node config to per-message overrides.
| Module | Description |
|---|---|
| hb_opts | Hierarchical configuration resolving options from message to node config to defaults |
| hb_private | Manages private keys and signing context, keeps secrets out of cached data |
| hb_features | Feature flag system for enabling/disabling functionality and gradual rollouts |
| hb_path | Parses paths into segments, matches patterns with wildcards, handles templates |
3. Data Structures
Messages are maps. These modules define how they're created, signed, linked, and transformed.
| Module | Description |
|---|---|
| hb_maps | Enhanced map operations with automatic link resolution and nested path traversal |
| hb_message | Creates messages, computes IDs, signs with attestations, verifies signatures |
| hb_link | Content-addressed references that resolve through cache with hashpath computation |
| hb_singleton | Converts between flat path-based keys and nested hierarchical message formats |
4. Storage
Persistent data storage with pluggable backends.
| Module | Description |
|---|---|
| hb_store | Abstract storage interface defining read, write, list, delete operations |
| hb_store_opts | Parses store configuration and creates store instances with fallback chains |
| hb_store_fs | Filesystem storage backend that stores data as files in directories |
| hb_store_lmdb | LMDB backend via NIF—fast, reliable, default storage with concurrent readers |
| hb_store_rocksdb | RocksDB LSM-tree backend via NIF for write-heavy workloads |
| hb_store_lru | Wraps any store with in-memory LRU cache for improved read performance |
5. Caching
Content-addressed caching layer on top of storage.
| Module | Description |
|---|---|
| hb_cache | Stores messages by content hash, resolves links recursively, expands nested messages |
| hb_cache_control | Parses HTTP Cache-Control headers and determines cacheability of responses |
6. HTTP Client
Outbound HTTP requests to gateways and other nodes.
| Module | Description |
|---|---|
| hb_http_client | HTTP client wrapper managing Gun connections with pooling and retries |
| hb_http_client_sup | Supervisor for HTTP client connection pools handling lifecycle and restarts |
| hb_http_multi | Sends parallel requests to multiple nodes and aggregates results |
| hb_http | Convenience GET/POST helpers with codec negotiation and response parsing |
| hb_gateway_client | Arweave gateway API client for fetching transactions, data, and blocks |
| hb_client | High-level Arweave client combining gateway operations for common tasks |
| hb_router | Selects which nodes to contact based on health checks and routing preferences |
7. Remote Storage
Storage backends that fetch data over the network.
| Module | Description |
|---|---|
| hb_store_gateway | Fetches data from Arweave gateway on cache miss, read-only remote storage |
| hb_store_remote_node | Fetches data from other HyperBEAM nodes enabling distributed storage |
8. Protocol
The heart of HyperBEAM. Every request flows through hb_ao:resolve/3.
| Module | Description |
|---|---|
| hb_ao | Core protocol implementing resolve/3 that routes every request to devices |
| hb_ao_device | Loads device modules by name, caches them, dispatches calls to handlers |
9. HTTP Server
Inbound HTTP handling.
| Module | Description |
|---|---|
| hb_http_server | Cowboy handlers routing HTTP requests to hb_ao with codec negotiation |
10. WASM
WebAssembly execution environment.
| Module | Description |
|---|---|
| hb_beamr | WASM runtime interface via WAMR, manages module loading and execution |
| hb_beamr_io | Provides I/O primitives to WASM modules including stdin/stdout and files |
11. Application
OTP application structure and node lifecycle.
| Module | Description |
|---|---|
| hb_name | Registers and looks up Erlang processes by name within the node |
| hb_persistent | Stores process state durably to survive restarts with consistency guarantees |
| hb_sup | Main supervisor defining the supervision tree and restart strategies |
| hb_app | OTP application behavior with start/stop callbacks and initialization |
| hb | Top-level API for starting/stopping nodes and convenience functions |
Supplemental
Volume
Storage partitions and data routing.
| Module | Description |
|---|---|
| hb_volume | Manages storage partitions, handles mounting/unmounting and data routing |
Monitoring
Logging, tracing, debugging, and metrics.
| Module | Description |
|---|---|
| hb_format | Pretty-prints messages, stack traces, and binaries for debugging output |
| hb_event | Defines ?event macro for structured logging with levels and context |
| hb_logger | Formats log events and routes to console/file with rotation support |
| hb_tracer | Records message flow through devices with trace IDs and timing data |
| hb_debugger | Interactive step debugger for device execution with state inspection |
| hb_cache_render | Generates DOT/GraphViz graphs showing cache contents and relationships |
| hb_metrics_collector | Collects and exposes Prometheus metrics for request counts and latencies |
| hb_process_monitor | Monitors Erlang process health, detects failures, reports status |
Testing
Test utilities and conformance tests.
| Module | Description |
|---|---|
| hb_test_utils | Common test fixtures, assertion helpers, and mock setup utilities |
| hb_ao_test_vectors | Protocol conformance test vectors ensuring hb_ao compatibility |
| hb_message_test_vectors | Test vectors for message creation, signing, and ID computation |
| hb_http_benchmark_tests | Performance benchmarks measuring throughput and latency distributions |
| hb_examples | Usage examples demonstrating common patterns and best practices |
Chapter 6: Devices
Built-in devices implementing AO-Core protocol behaviors.
1. Messages & Composition
The foundational device and composition patterns.
| Module | Description |
|---|---|
| dev_message | Core device implementing get, set, keys, remove—the foundation of all devices |
| dev_stack | Chains devices into ordered execution pipelines with Fold and Map modes |
| dev_multipass | Runs message through stack multiple times until stable or limit |
| dev_apply | Dynamic path execution with base:/request: prefixes |
| dev_patch | Moving data between message paths |
| dev_dedup | Prevents duplicate message processing with configurable detection |
2. Codecs
Message encoding, decoding, and cryptographic signing.
| Module | Description |
|---|---|
| dev_codec_httpsig | HTTP Message Signatures (RFC 9421) |
| ↳ dev_codec_httpsig_conv | Conversions for signature components |
| ↳ dev_codec_httpsig_keyid | Key ID management |
| ↳ dev_codec_httpsig_proxy | Proxy support for signature forwarding |
| ↳ dev_codec_httpsig_siginfo | Signature metadata extraction |
| dev_codec_structured | RFC 9651 Structured Fields with ao-types annotations |
| dev_codec_json | JSON encoding/decoding for HTTP APIs |
| dev_codec_flat | Path-based flat map encoding for configuration |
3. Infrastructure
Node configuration, routing, and lifecycle management.
| Module | Description |
|---|---|
| dev_meta | Node configuration, initialization, and request handling |
| dev_router | Message routing with load balancing strategies |
| dev_relay | Synchronous and asynchronous message relay between nodes |
| dev_node_process | Singleton process management for node services |
| dev_hook | Lifecycle hooks for request/response processing |
| dev_manifest | Defines process configuration—device stack, scheduler, compute limits |
4. Process & Scheduling
Stateful computation units and deterministic message ordering.
| Module | Description |
|---|---|
| dev_process | Core AO process execution coordinator |
| ↳ dev_process_cache | Caches computed process states |
| ↳ dev_process_worker | Worker pools for parallel execution |
| dev_scheduler | Message ordering and slot assignment |
| ↳ dev_scheduler_cache | Caches scheduler assignments |
| ↳ dev_scheduler_formats | Format version handling |
| ↳ dev_scheduler_registry | Process registration and discovery |
| ↳ dev_scheduler_server | Long-running server process |
| dev_push | Recursive message delivery between processes |
| dev_cron | Time-based scheduled execution with cron expressions |
5. Storage & Naming
Data persistence and name resolution.
| Module | Description |
|---|---|
| dev_cache | Read/write data with access control |
| dev_lookup | Retrieve resources by ID with format negotiation |
| dev_local_name | Register human-readable names for resources |
| dev_name | Resolve names through configurable resolver chains |
6. Runtimes
Code execution environments—WASM, Lua, delegated compute.
| Module | Description |
|---|---|
| dev_wasm | Execute WebAssembly with Memory-64 support |
| ↳ dev_wasi | Virtual filesystem for WASM programs |
| ↳ dev_json_iface | JSON interface for WASM modules |
| ↳ dev_genesis_wasm | Initial WASM module configuration |
| dev_lua | Run Lua scripts in sandboxed VMs |
| ↳ dev_lua_lib | HyperBEAM API bindings for Lua |
| dev_cu | Delegate to external Compute Units |
| ↳ dev_delegated_compute | Remote computation with verification |
7. Payment
Pricing and payment in HyperBEAM.
| Module | Description |
|---|---|
| dev_p4 | Core payment ledger and pricing orchestrator |
| dev_simple_pay | Per-request pricing with balance management |
| dev_faff | Allow-list based access control |
8. Authentication
Identity and signatures.
| Module | Description |
|---|---|
| dev_auth_hook | Automatic request signing with node-hosted wallets |
| dev_codec_http_auth | HTTP Basic authentication with PBKDF2 |
| dev_codec_cookie | Cookie management and storage |
| ↳ dev_codec_cookie_auth | Cookie-based HMAC authentication |
| dev_secret | Secret key management device |
9. Arweave & Data
Permanent storage and data retrieval.
| Module | Description |
|---|---|
| dev_arweave | Arweave network interface |
| ↳ dev_arweave_block_cache | Caches Arweave blocks locally |
| dev_codec_ans104 | ANS-104 data item codec |
| ↳ dev_codec_ans104_from | Decode ANS-104 to messages |
| ↳ dev_codec_ans104_to | Encode messages to ANS-104 |
| dev_query | Cache search and discovery |
| ↳ dev_query_arweave | Arweave gateway GraphQL |
| ↳ dev_query_graphql | GraphQL query executor |
| dev_copycat | Message indexing from external sources |
| ↳ dev_copycat_arweave | Replicate from Arweave |
| ↳ dev_copycat_graphql | GraphQL-based discovery |
| dev_poda | Generates and verifies proofs that data exists on Arweave |
Supplemental Devices
Security & TEE
| Module | Description |
|---|---|
| dev_volume | Volume device for mounting, unmounting, and listing storage volumes |
| dev_snp | AMD SEV-SNP attestation for TEE |
| ↳ dev_snp_nif | Native C interface for SNP |
| dev_green_zone | Defines security boundaries for sensitive operations requiring TEE |
Utilities & Debugging
| Module | Description |
|---|---|
| dev_trie | Prefix tree implementation for efficient key lookup |
| dev_profile | Records execution timing and memory allocation |
| dev_hyperbuddy | Web-based interactive debugging UI |
| dev_cacheviz | Generates visual representation of cache contents |
| dev_whois | Looks up node information by address |
| dev_monitor | Tracks system health with alerting |
| dev_test | Test device with configurable responses |
Test Vectors
| Module | Description |
|---|---|
| ↳ dev_codec_cookie_test_vectors | Cookie parsing conformance tests |
| ↳ dev_query_test_vectors | Query device behavior tests |
| ↳ dev_lua_test | Lua runtime integration tests |
| ↳ dev_lua_test_ledgers | Lua ledger operation tests |
Chapter 7: Building Your Own Devices
Now that you understand the codebase, build your own devices.
| Tutorial | Description |
|---|---|
| Custom Devices in Erlang | Native Erlang devices with full HyperBEAM integration |
| Custom Devices in Rust | High-performance NIFs for compute-intensive operations |
| Custom Devices in C++ | Native implementations with direct memory access |