# Theory Craft

When building a game data system, it helps to think in layers, from engine-agnostic core logic to designer-facing tools. DataLens is designed to support large-scale emergent simulations across engines (UE, Unity, etc.), while maintaining high memory and CPU efficiency.

## Layers

***

### 1. **Native DataStore (Engine-Agnostic Core)**

**Purpose:** the true source of structured, high-performance game data, fully engine-independent.

**Characteristics:**

* Written in pure C++ as a column-oriented, in-memory store.
* Supports dynamic per-column stride, raw byte storage, and flexible typed access.
* Provides both unchecked raw access for maximum performance and safe, bounds-checked operations.
* Stores arbitrary types, relationships, and traits through column metadata.
* Handles serialisation to and from row-major layouts efficiently.
* No dependencies on Unreal, Unity, or other engines.
* Acts as a low-level API for engineers to manipulate rows, columns, and values safely or directly.

**Why it exists:** All systems reference a dense, engine-agnostic “truth” that maximises storage efficiency. This allows other layers to build engine-specific APIs and designer-facing tools without duplicating the data.

**Core API Concepts (conceptual):**

```cpp
DataStore store({ {4}, {8}, {16} }, 1024); // 3 columns, preallocate 1024 rows
store.SetRaw<int32_t>(row, col, value);
int32_t val = store.GetRaw<int32_t>(row, col);
```

Serialization, column info, and safe access remain fully supported.

***

### 2. **DataLens API (Engine-Native, Engineer-Facing)**

**Purpose:** the bridge between the **engine runtime** and the **engine-agnostic DataStore**.

**Characteristics:**

* Engine-specific (C++ in UE, C# in Unity) but engineer-facing, not designer-facing.
* Handles tick updates, staged updates, threading, and asynchronous orchestration.
* Manages registration of “tables” or actor types and provides idiomatic APIs for safe DataStore interaction.

**Updated Note:** Double-buffering for DataViews was explored but found unfeasible under Phase 1 constraints. Each DataView is now treated as a cached, independent row-major representation. DataLens orchestrates these views per tick, leaving overlapping or conflicting caches as a developer-managed concern.

***

### 3. **DataView (Engine-Specific, Designer-Facing)**

**Purpose:**\
Provides filtered, row-major slices of data for game logic, with read/write access to support gameplay systems.

**Characteristics:**

* Scoped, ephemeral cache: Each DataView is a materialised row-major representation derived from one or more DataStores.
* Context-sensitive: Only aggregates data when needed by game systems; not precomputed for all actors.
* Orchestration: DataLens updates registered views per tick; developers manage conflicts between overlapping views.
* Supports queries, joins, and calculated columns.

**New Actor Knowledge Model (Experimental):**

* Each actor may have a scoped, imperfect “view” of the simulation state.
* Actor knowledge stores reference canonical DataStores and store metadata such as confidence, belief strength, acquisition method (observed/derived/rumour), and temporal decay.
* DataViews aggregate relevant actor- or topic-specific data for gameplay systems without materialising every possible state.

***

### 4. DataSystem (Engine-Specific, Designer-Facing)

**Purpose:**\
Provides column-oriented, system-level updates across subsets of the world.

**Characteristics:**

* Engine-native and designer-friendly.
* Bufferless, operating immediately after DataView updates.
* Suitable for recurring computations, aggregations, or status updates that do not require per-record derived data.
* Frequency-driven: Updates run at configurable intervals (frame, 10 Hz, 1 Hz).

**Example Usage:**

* Hunger, fatigue, or energy decay.
* Health or mana regeneration.
* Environmental effects, automated resource production, crowd or faction adjustments.

***

## **Layer Relationships**

```
[ Designer / Blueprints or ScriptableObjects ] 
        <--> 
[ DataView & DataSystem (Designer-Facing) ] 
        <--> 
[ DataLens API Layer (Engineer-Facing) ] 
        <--> 
[ Native DataStore (Engine-Agnostic) ]
```

* **Native DataStore:** single source of truth, fully engine-independent.
* **DataLens API:** engine-specific, manages threading, ticks, buffers, and engine integration.
* **DataView & DataSystem:** designer-facing, flexible interface to query and update data, feeding multiple runtime systems.

***

### **Why this matters**

* The architecture separates **engine concerns**, **designer access**, and **core data logic**.
* Makes it possible to reuse the **same DataStore** across multiple engines.
* Enables complex, real-time systems such as AI, Senses, and environmental effects without engine lock-in.
* Keeps designer tooling simple and intuitive while giving engineers full control under the hood.

## Speculative Game System Design

This section explores how a massive-world RPG might leverage the DataLens system to solve real design challenges. It draws on Heathen’s own game designs and serves as a practical “dog food” proof of concept for DataLens. Here we demonstrate **how DataLens could support both small-scale narrative experiments and large-scale, multi-system RPGs**, validating the system’s ability to manage scoped knowledge, actor autonomy, and emergent events in real-time.

***

### [Túatha Echoes: The Barrow](https://tuatha.freki.games/necromancy/)

A small, linear narrative visual novel-style game introducing players to the world’s concepts of life, death, and cyclical existence. The player assumes the role of a fatally wounded hunter transitioning from life to death.

**Gameplay Context:**

* The Barrow functions as a hospital, hospice, temple, school, and burial space.
* As the player transitions, their perception changes: actors, events, and interactions become contextually dependent on their state.
* The player chooses an “afterlife” path through interaction with necromancers, spirits, artefacts, and unfolding events.

**DataLens Application:**

* Emergent dialogues and interactions are driven by DataLens-defined scopes of knowledge.
* “World Data” is defined for each potential path:
  * Marvir Child (reborn as a spirit)
  * Lich Adherent (sacred calling)
  * Apprentice (linger to aid the Barrow)
  * Spirit Lantern (bound to the world differently)
  * Go Beyond (unknown fate)
* Player observations (e.g., a birth, a funeral, ancestral spirits) feed into DataLens, dynamically shaping what knowledge, actors, and dialogue options are accessible.
* DataLens effectively manages the scope of the player’s knowledge, informing which interactions are valid, which events can be perceived, and which conversational options are presented.

***

### [Túatha: The Oirirc Saga – Shattered](https://tuatha.freki.games/shattered/)

The first entry in a planned trilogy, a primarily linear narrative RPG following the player through the death of innocence via trauma, discovery of the wider world, and the selection and commitment to a life path (becoming a Heathen / Gallowglass).

**Gameplay Context:**

* Player interactions with NPCs, events, and environments have a minor but meaningful influence on the world’s evolution.
* Emergent variability is achieved while maintaining narrative structure.

**DataLens Application:**

* DataLens defines the environment, events, and actor states.
* Actors simulate semi-autonomously, allowing player influence and randomness while generally converging on the intended story beats.
* DataLens supports subsystems including economy, factions, and villages, maintaining simulation state consistency and performance.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://kb.heathen.group/toolkit-for-data/theory-craft.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.