Theory Craft
When building a game data system, it helps to think in layers, from engine-agnostic core logic to designer-facing tools. DataLens is designed to support large-scale emergent simulations across engines (UE, Unity, etc.), while maintaining high memory and CPU efficiency.
Layers
1. Native DataStore (Engine-Agnostic Core)
Purpose: the true source of structured, high-performance game data, fully engine-independent.
Characteristics:
Written in pure C++ as a column-oriented, in-memory store.
Supports dynamic per-column stride, raw byte storage, and flexible typed access.
Provides both unchecked raw access for maximum performance and safe, bounds-checked operations.
Stores arbitrary types, relationships, and traits through column metadata.
Handles serialisation to and from row-major layouts efficiently.
No dependencies on Unreal, Unity, or other engines.
Acts as a low-level API for engineers to manipulate rows, columns, and values safely or directly.
Why it exists: All systems reference a dense, engine-agnostic “truth” that maximises storage efficiency. This allows other layers to build engine-specific APIs and designer-facing tools without duplicating the data.
Core API Concepts (conceptual):
DataStore store({ {4}, {8}, {16} }, 1024); // 3 columns, preallocate 1024 rows
store.SetRaw<int32_t>(row, col, value);
int32_t val = store.GetRaw<int32_t>(row, col);Serialization, column info, and safe access remain fully supported.
2. DataLens API (Engine-Native, Engineer-Facing)
Purpose: the bridge between the engine runtime and the engine-agnostic DataStore.
Characteristics:
Engine-specific (C++ in UE, C# in Unity) but engineer-facing, not designer-facing.
Handles tick updates, staged updates, threading, and asynchronous orchestration.
Manages registration of “tables” or actor types and provides idiomatic APIs for safe DataStore interaction.
Updated Note: Double-buffering for DataViews was explored but found unfeasible under Phase 1 constraints. Each DataView is now treated as a cached, independent row-major representation. DataLens orchestrates these views per tick, leaving overlapping or conflicting caches as a developer-managed concern.
3. DataView (Engine-Specific, Designer-Facing)
Purpose: Provides filtered, row-major slices of data for game logic, with read/write access to support gameplay systems.
Characteristics:
Scoped, ephemeral cache: Each DataView is a materialised row-major representation derived from one or more DataStores.
Context-sensitive: Only aggregates data when needed by game systems; not precomputed for all actors.
Orchestration: DataLens updates registered views per tick; developers manage conflicts between overlapping views.
Supports queries, joins, and calculated columns.
New Actor Knowledge Model (Experimental):
Each actor may have a scoped, imperfect “view” of the simulation state.
Actor knowledge stores reference canonical DataStores and store metadata such as confidence, belief strength, acquisition method (observed/derived/rumour), and temporal decay.
DataViews aggregate relevant actor- or topic-specific data for gameplay systems without materialising every possible state.
4. DataSystem (Engine-Specific, Designer-Facing)
Purpose: Provides column-oriented, system-level updates across subsets of the world.
Characteristics:
Engine-native and designer-friendly.
Bufferless, operating immediately after DataView updates.
Suitable for recurring computations, aggregations, or status updates that do not require per-record derived data.
Frequency-driven: Updates run at configurable intervals (frame, 10 Hz, 1 Hz).
Example Usage:
Hunger, fatigue, or energy decay.
Health or mana regeneration.
Environmental effects, automated resource production, crowd or faction adjustments.
Layer Relationships
Native DataStore: single source of truth, fully engine-independent.
DataLens API: engine-specific, manages threading, ticks, buffers, and engine integration.
DataView & DataSystem: designer-facing, flexible interface to query and update data, feeding multiple runtime systems.
Why this matters
The architecture separates engine concerns, designer access, and core data logic.
Makes it possible to reuse the same DataStore across multiple engines.
Enables complex, real-time systems such as AI, Senses, and environmental effects without engine lock-in.
Keeps designer tooling simple and intuitive while giving engineers full control under the hood.
Speculative Game System Design
This section explores how a massive-world RPG might leverage the DataLens system to solve real design challenges. It draws on Heathen’s own game designs and serves as a practical “dog food” proof of concept for DataLens. Here we demonstrate how DataLens could support both small-scale narrative experiments and large-scale, multi-system RPGs, validating the system’s ability to manage scoped knowledge, actor autonomy, and emergent events in real-time.
A small, linear narrative visual novel-style game introducing players to the world’s concepts of life, death, and cyclical existence. The player assumes the role of a fatally wounded hunter transitioning from life to death.
Gameplay Context:
The Barrow functions as a hospital, hospice, temple, school, and burial space.
As the player transitions, their perception changes: actors, events, and interactions become contextually dependent on their state.
The player chooses an “afterlife” path through interaction with necromancers, spirits, artefacts, and unfolding events.
DataLens Application:
Emergent dialogues and interactions are driven by DataLens-defined scopes of knowledge.
“World Data” is defined for each potential path:
Marvir Child (reborn as a spirit)
Lich Adherent (sacred calling)
Apprentice (linger to aid the Barrow)
Spirit Lantern (bound to the world differently)
Go Beyond (unknown fate)
Player observations (e.g., a birth, a funeral, ancestral spirits) feed into DataLens, dynamically shaping what knowledge, actors, and dialogue options are accessible.
DataLens effectively manages the scope of the player’s knowledge, informing which interactions are valid, which events can be perceived, and which conversational options are presented.
The first entry in a planned trilogy, a primarily linear narrative RPG following the player through the death of innocence via trauma, discovery of the wider world, and the selection and commitment to a life path (becoming a Heathen / Gallowglass).
Gameplay Context:
Player interactions with NPCs, events, and environments have a minor but meaningful influence on the world’s evolution.
Emergent variability is achieved while maintaining narrative structure.
DataLens Application:
DataLens defines the environment, events, and actor states.
Actors simulate semi-autonomously, allowing player influence and randomness while generally converging on the intended story beats.
DataLens supports subsystems including economy, factions, and villages, maintaining simulation state consistency and performance.
Last updated