# 20251114

## Conclusion

DataView is now conceptualized as a thin, ephemeral, developer-facing object, similar to .NET’s `SqlCommand`:

* Holds a query definition and the results of that query as a lightweight, row-major cache.
* Can be registered with DataLens for orchestration in the future, allowing automatic query refresh and batched update commits once double-buffering is implemented.
* Registered DataViews will support optional priority for deterministic batching; currently, operations are handled manually or via callbacks.
* Non-registered DataViews or manual Query/Update/Commit operations are handled via a locking/buffering model once the orchestration layer is implemented.
* DataLens currently provides serialize/deserialize functionality for DataStores, ensuring schema-aware memory representation and type-safe conversion.

Phase 1 work so far has confirmed that:

* Column names can be used to reconcile schema changes automatically (columns added/removed or with changed types).
* Type conversions between old and new column types can be applied safely during `ConvertToSchema`.
* DataLens can deliver row-major, engine-facing views on top of column-major, cache-aware DataStore layouts.

***

## Detailed Notes and Thought Process

**1. DataView Purpose**

* Acts as a transactional, developer-facing cache.
* Contains the results of queries and staged updates (future implementation).
* Provides optional orchestration through registration with DataLens (future work).
* Does not own memory; underlying buffers and operations are managed by DataLens/DataStore.

**2. Registered vs Non-Registered Views (Planned)**

* Registered DataViews: will eventually support automatic refresh/commit each tick with optional priority.
* Non-registered / manual DataViews: operations can be executed on demand; currently the system only provides low-level DataLens hooks.

**3. Query and Update Model (Planned)**

* Queries: will return row-major results from DataStores.
* Updates: will be staged through DataView and applied via DataLens.
* Commit: will support batching updates per-store or globally once orchestration is implemented.

**4. Transactional and Cache Behavior (Planned)**

* Updates will be staged until commit.
* Queries will operate on cached results.
* DataLens will ensure thread safety and determinism during execution.

**5. Callback Mechanism**

* All DataView operations will expose callbacks for completion once orchestration is implemented.

**6. Schema and Type Conversion**

* `ConvertToSchema` reconciles DataStore dumps to match the in-memory schema, handling:
  * Column addition/removal.
  * Data type changes with conversion.
  * Default values for new columns.
* This allows developers to modify schema without breaking DataLens/DataView operations.

**7. Analogy (Planned)**

* Registered DataViews: auto-refreshing, transactional `SqlCommand` with optional prioritization.
* Non-registered DataViews: manual `SqlCommand` executed on demand, subject to DataLens scheduling.

***

## Implemented So Far

* `DataLens` serialize/deserialize for DataStores.
* Schema-aware `ConvertToSchema` in `DataStore`, automatically reconciling columns and types.
* Stubbed DataView object supporting basic query/update/commit interface (no orchestration yet).
* Verified row-major DataView access over column-major DataStore layout with minimal overhead.

***

## Next Steps

* Implement double-buffered read/write support in DataLens for registered DataViews.
* Define full C++ and C# APIs for DataView (registration, query, update, commit, callbacks).
* Implement priority and tick frequency metadata for orchestrated DataViews.
* Build sample tests for row-major queries and updates against column-major DataStore to validate performance and thread safety.