20251125

Conclusion

Phase 1 work continues to refine the conceptual model of DataLens and DataView. Double-buffering for DataViews has proven unfeasible under performance and schema-evolution constraints. The current approach treats each DataView as an independently cached, row-major representation of data derived from one or more DataStores. DataLens orchestrates updates across registered DataViews per tick, leaving overlapping or conflicting views as a developer-managed concern.

Experimental work has highlighted a critical system-level challenge: to support large-scale emergent simulations, actors may require scoped, imperfect “views” of the simulation state. Storing a full copy of the simulation for each actor is impractical, so the investigation is exploring the use of dense, column-major DataStores to hold canonical data (e.g., “World Facts”) while actor-specific stores reference this data with metadata such as confidence, belief strength, acquisition method (observed, rumor, derived), and temporal decay.

DataViews are intended to provide context-sensitive, row-major representations for gameplay systems, aggregating relevant information on demand rather than precomputing every actor’s state. This approach balances performance and memory efficiency while still allowing the system to model emergent behaviour, misinformation, and incomplete knowledge.

Detailed Notes and Thought Process

1. DataView Role (Revised)

   • Acts as a lightweight, ephemeral, engine- and developer-facing cache.

   • Represents a query over one or more DataStores, materialized in row-major form.

   • Independent cache; orchestration per tick handled by DataLens.

2. Actor Knowledge Problem (Experimental)

   • Need to provide each actor with a scoped, potentially imperfect “view” of the simulation state.

   • Direct duplication of DataStores per actor is infeasible; metadata referencing canonical DataStores is being explored.

   • Metadata may include: confidence, belief strength, acquisition method (observed/derived/rumor), and decay over time.

3. DataView vs Actor Knowledge

   • DataViews remain gameplay- and context-focused: they aggregate and expose the subset of data needed for a specific scope.

   • Actor knowledge stores remain dense, memory-efficient, and only partially materialized via DataViews when needed.

4. Schema Evolution and Type Safety

   • ConvertToSchema continues to handle column additions, removals, type changes, and defaults.

   • This mechanism ensures DataViews can remain consistent with evolving DataStores.

5. Next Steps / Experimental Focus

   • Validate that dense DataStores plus context-sensitive DataViews can scale to large numbers of actors in real-time simulations.

   • Experiment with metadata-based actor knowledge representation.

   • Begin prototype tests for generating DataViews that query multiple actor and topic stores simultaneously.

   • Investigate caching strategies, priority, and tick scheduling for DataView orchestration.