Lux Database

Overview

Lux Database (github.com/luxfi/database) is the Go storage abstraction layer for the Lux blockchain. It defines a unified key-value Database interface modeled after Ethereum's database API (enabling direct use of Geth/EVM code) and provides multiple pluggable backends: ZapDB (BadgerDB-based, default), PebbleDB (CockroachDB), LevelDB (Google), and in-memory. Also includes database wrappers for encryption, corruption detection, hooking, prefixing, versioning, metering, and linked-list storage. Used by github.com/luxfi/node for all persistent state.

Repository

Core Dependencies

Core Interface

The Database interface in database.go composes:

type Database interface {
    KeyValueReaderWriterDeleter  // Has, Get, Put, Delete
    Batcher                     // NewBatch
    Iteratee                    // NewIterator variants
    Compacter                   // Compact(start, limit)
    Syncer                      // Sync() - flush to disk
    io.Closer                   // Close
    HealthCheck(context.Context) (interface{}, error)
    Backup(w io.Writer, since uint64) (uint64, error)  // Incremental backup
    Load(r io.Reader) error                             // Restore from backup
}

Sub-Interfaces

Error Types

var ErrClosed   = errors.New("closed")
var ErrNotFound = errors.New("not found")
type ErrBackendDisabled struct { Backend string }

Memory Management

memory.go provides MemoryManager for controlling database memory usage:

• Configurable percentage of system memory
• Reserve allocation for critical operations
• Prometheus gauge metrics (usage, available, remaining)

Constants

MaxExcessCapacityFactor = 4  // Above which capacity is excessive
CapacityReductionFactor = 2  // Reduction factor for excess capacity

Storage Backends

ZapDB (Default)

The badgerdb/ package is a thin compatibility wrapper that delegates to zapdb.New(). Registered as the default backend in manager/manager.go via init().

PebbleDB

8 source files: db.go, batch.go, iterator.go, constants.go, errors.go, set.go, db_test.go, batch_test.go.

LevelDB

5 source files: db.go, constants.go, errors.go, metrics.go, db_test.go.

In-Memory (memdb)

4 source files: db.go, constants.go, iterator.go, db_test.go.

Database Wrappers

Database Infrastructure

Factory (factory/)

Pluggable database creation system:

• RegisterDatabase(name, factory) -- Register a backend
• AvailableDatabases() -- List available backends
• Default: ZapDB + memdb always available
• PebbleDB/LevelDB available via build tags

Manager (manager/)

Database lifecycle management:

• NewManager(baseDir, registerer) -- Create manager
• RegisterDatabaseType(name, creator) -- Register backend type
• Supports zapdb, pebbledb, leveldb via files manager_zapdb.go, manager_pebbledb.go, manager_leveldb.go
• Wraps databases with metering, versioning, and prefixing

Test Suite (dbtest/)

Comprehensive test harness:

• dbtest.go -- Full interface compliance tests
• benchmark.go -- Performance benchmarks
• testsuite.go -- Reusable test suite for any backend

Mocks (databasemock/)

Generated mock implementations:

• batch.go -- Mock Batch
• iterator.go -- Mock Iterator

Project Structure

~/work/lux/database/
  go.mod                 # github.com/luxfi/database, Go 1.26.1
  database.go            # Core Database/HeightIndex interfaces
  batch.go               # Batch/Batcher interfaces and BatchOps
  iterator.go            # Iterator/Iteratee interfaces
  errors.go              # ErrClosed, ErrNotFound, ErrBackendDisabled
  memory.go              # MemoryManager (system memory control)
  helpers.go             # Utility functions
  database_util.go       # Additional utilities
  Makefile               # Build/test targets

  badgerdb/              # ZapDB compatibility wrapper
  pebbledb/              # PebbleDB backend (build tag)
  leveldb/               # LevelDB backend (build tag)
  memdb/                 # In-memory backend
  encdb/                 # Encrypted database wrapper
  corruptabledb/         # Corruption testing wrapper
  hookdb/                # Event hook wrapper
  linkeddb/              # Linked-list storage
  heightindexdb/         # Height-indexed storage
  codec/                 # Serialization (with linear/ sub-pkg)
  factory/               # Pluggable backend factory
  manager/               # Database lifecycle management
  dbtest/                # Test suite and benchmarks
  databasemock/          # Generated mocks
  bin/                   # Build output

Development

cd ~/work/lux/database

# Build all packages
go build ./...

# Run all tests
go test ./...

# Run with race detector
go test -race ./...

# Build with PebbleDB support
go build -tags=pebbledb ./...

# Build with LevelDB support
go build -tags=leveldb ./...

# Run benchmarks
go test -bench=. ./dbtest/

# Run specific backend tests
go test ./pebbledb/ -tags=pebbledb -v
go test ./leveldb/ -tags=leveldb -v
go test ./memdb/ -v

Key Design Decisions

1. Ethereum-compatible interface: Matches go-ethereum's database API for seamless EVM integration
2. Build-tag backends: PebbleDB and LevelDB are compile-time optional to minimize binary size
3. ZapDB as default: BadgerDB-based ZapDB is always available, registered in init()
4. Wrapper composition: Encryption, metering, versioning are separate wrappers that can be composed
5. HeightIndex: Specialized interface for blockchain state indexed by block height using big-endian uint64 keys for efficient range queries
6. Backup/Restore: Built into the core Database interface with incremental backup support
7. Memory management: System-level memory control with configurable limits and Prometheus metrics

Related Packages