20260129

Conclusion

Phase 1 of the DataLens R&D project has progressed from purely theoretical experimentation to practical, exploratory testing. The system was successfully ported from Unreal Engine 5.6 on Windows to Unreal Engine 5.7 on Linux, exposing low-level implementation and platform-specific issues. These confirm the necessity of precise coding discipline and careful memory/data layout for the DataStore and DataLens APIs.

Additionally, the DataView insert functionality was revised to support join-based updates, inspired by TSQL. Developers can now define how view cache modifications propagate to target DataStores during the DataView churn phase. The system uses precompiled intermediate expression objects, influenced by Microsoft Intermediate Language (MIL), allowing CPU-cache-aware updates and efficient CRUD-style operations. Memory reuse optimisations reduce unnecessary allocation or deletion.

Preparations for larger-scale, live validation testing are underway. While prior testing used synthetic datasets in controlled environments, game teams have identified real-world use cases for DataLens integration. This will enable evaluation of API usability and DataStore performance in live simulation scenarios. Only research-related work—system design, experimentation, test creation, and result analysis—is logged as R&D; implementation in game projects (Túatha: Shattered and Túatha: Wayfinder) is treated as production.

---

Detailed Notes and Thought Process

Exploratory R&D Focus

• This commit represents an experimental effort focused on DataLens, specifically the interaction between normalised, CPU-centric, cache-sensitive DataStores and game-specific DataViews.

• Technical uncertainty exists in efficiently mapping, filtering, and computing between underlying stores and views while preserving correctness, simplicity, and memory efficiency.

Objectives and Technical Uncertainty

• Investigate optimal structures bridging low-level DataStore (row-major, cache-aware) and high-level DataView abstractions.

• Explore approaches to pivoting, joining, and filtering rows for DataView construction with minimal allocations and high performance.

• Examine inline evaluation of calculated fields and predicates to reduce intermediate data structures and redundant memory operations.

• Address uncertainty in supporting joins, filtering, and expression evaluation in a unified pipeline given differing DataStore/DataView requirements.

Changes Implemented

• Refractor RefreshView to inline join resolution, predicate evaluation, and row materialisation. Calculated fields are evaluated during row materialisation, avoiding temporary buffers.

• Preserved memory efficiency by delaying pre-allocation of candidate row lists.

• Implemented AppendEmptyRow for correct stride and row pointer handling.

• Introduced ResolveJoinsInline and EvaluatePredicatesInline to explore per-row, inline computation strategies.

• Adjusted DataLensValueTypeUtils::Copy to reference column types correctly.

• Laid groundwork for grouping and aggregate support; full implementation deferred due to potential cache conflicts.

Current State

• RefreshView supports SELECT operations, calculated fields, joins between multiple stores, and predicate filtering (WHERE).

• Grouping and aggregate systems remain stubs; further exploration is needed for safe integration with inline evaluation and cache construction.

• Inline evaluation reduces temporary allocations and provides a clear path for optimising cache-sensitive DataStore/DataView interactions.

Next Steps / Future Exploration

• Safely integrate grouping and aggregate calculations without violating cache assumptions.

• Benchmark inline evaluation against previous candidate-row approaches to quantify performance gains.

• Explore write-back or update strategies for DataViews with inline calculated fields and aggregated columns.

R&D Context

• The work qualifies as experimental development due to technical uncertainty. Multiple approaches were tested to determine optimal methods for row mapping, join resolution, and inline evaluation—solutions are not readily deducible by a competent developer without systematic experimentation.