Specification of the `datta-intelligent-query-layer` feature for Apache Trino with Neo4j, OpenSearch, vLLM, Investigative Ontology, BI and Dashboards

Preview — feature under development. Behavior, screens and contracts may change without notice between releases.

How to use

This document is the reference specification for the feature. It describes a real, incremental, safe implementation that is transparent to the end user.

Official feature name

Use the name exactly as provided below in documentation, modules, services, feature flags, dashboards, metrics, endpoints and artifacts produced during implementation:

datta-intelligent-query-layer

Treat this name as the official identification of the feature. Do not rename it, do not fix the spelling, and do not introduce variations, except when necessary for derived technical names compatible with code conventions.

You are the principal architect + staff engineer + implementation lead responsible for designing and implementing a new intelligence layer for an already existing data platform.

Your role

You must act as an extremely pragmatic senior engineer, focused on implementing and operationalizing the datta-intelligent-query-layer feature, with special attention to:

evolutionary architecture;
integration with existing systems;
justified technical decisions;
incremental and safe implementation;
production-ready code;
tests, observability, rollback and documentation.

You must not propose a full rewrite of the platform nor replace the main analytical engine. Your job is to attach an intelligence layer to the current stack, preserving the end-user experience and making the most of the components already in place.

Project goal

Implement the datta-intelligent-query-layer feature, a sidecar intelligence layer for Apache Trino, which uses:

Neo4j as the graph of intelligent metadata and second-order relationships;
OpenSearch for indexing, observability, telemetry, operational search, analytics and fast retrieval of derived signals;
vLLM as the LLM infrastructure already present on the platform for assistants, explainability, diagnostics generation and possible assisted workflows;
the already existing investigative ontology as the semantic source and domain-meaning layer;
the BI layer with Datamash and dashboards as surfaces for observability, explainability and administration.

The goal is to improve Trino's planning, pruning, statistics, split selection, read prioritization and execution decisions without changing how the end user writes SQL.

Expected result in one sentence

The user keeps querying Trino normally with standard SQL; behind the scenes, the platform gains the datta-intelligent-query-layer feature, an intelligent layer of observed + learned + semantic metadata that helps the optimizer and the connector make better decisions, with a safe fallback to the default behavior.

Technical context and assumptions

Treat the following assumptions as the default, unless the repository's code or documentation shows otherwise:

Apache Trino is the platform's official relational analytical engine.
The lakehouse uses, or should primarily use, Iceberg with Parquet files, or an equivalent architecture compatible with that standard.
The end user must not notice any interface change: they keep using SQL, catalogs, schemas, tables, dashboards and normal flows.
The intelligence layer does not replace Trino/Iceberg's native metadata mechanism.
The intelligence layer cannot compromise correctness: any hard pruning may only happen when guaranteed by authoritative metadata.
The sidecar must operate as an advisor, not as the source of truth for snapshots.
The project must first be implemented in an evolutionary and observable way, with feature flags, A/B comparison, metrics and fallback.
If there are ambiguities in the repository, you must inspect the code and adapt the solution to the real stack instead of insisting on an imaginary architecture.

What needs to be implemented

You must design and implement an architecture composed of at least these blocks:

1. Trino Integration Layer

A Trino integration based on plugin/SPI, prioritizing:

a wrapper connector or an extension of the connector already used by the platform;
integration with ConnectorMetadata;
integration with ConnectorSplitManager;
support for applyFilter, statistics, split ranking, prioritization and advice;
without modifying the user's SQL;
without requiring a mental-model change for BI, investigation or engineering teams.

2. Advisor Sidecar Service

A separate service responsible for:

receiving the query context;
querying Neo4j;
querying signals in OpenSearch;
combining authoritative, observed and learned metadata;
returning a small, fast and safe PlanAdvice to the Trino connector;
exposing a short timeout, cache, fallback and clear metrics.

3. Knowledge Graph of Intelligent Metadata

A Neo4j model that represents:

catalog, schema, table, snapshot, manifest, data file, delete file, partition, column;
statistics per file/column;
query patterns, fingerprints, filters, joins, group by, workload classes;
execution history;
hotness, co-access, manifest affinity, spill risk, broadcast benefit, expected selectivity;
physical maintenance recommendations, compaction, rewrite manifests, sort/reclustering;
domain semantics derived from the investigative ontology.

4. OpenSearch Layer

Use OpenSearch to:

store operational telemetry of queries and planning;
index fingerprints, events, plans, timings, errors, bytes read, accessed columns;
allow fast operational search and analytics;
serve as support for hybrid signal retrieval and explainability;
feed operational dashboards and regression investigation;
optionally store vectors/embeddings if the platform's stack already uses that in OpenSearch.

5. Ingestion / Learning / Feedback Loop

Implement pipelines that:

consume Trino events;
update Neo4j;
update indices and aggregations in OpenSearch;
produce learned signals offline or nearline;
recompute features and scores used by the advisor;
never place heavy algorithms in the query hot path.

6. Explainability / Admin / BI

Expose for internal observability:

advisor gain metrics;
baseline vs advisor comparisons;
decision causes;
recommendation confidence;
degradations/fallbacks;
administrative and troubleshooting dashboards;
possibly LLM-assisted features to explain decisions or suggest tuning, but without allowing the LLM to change critical decisions without deterministic validation.

Mandatory implementation principles

A. Transparency to the end user

The user must keep making SQL queries normally. The integration must be transparent. Changes may only appear, optionally, in:

metrics;
explain plans;
administrative pages;
internal engineering tools.

B. Correctness first

The advisor must never introduce an incorrect result.
Heuristics may influence ranking and estimation.
Definitive read elimination may only happen based on authoritative metadata.
When in doubt, the system must fall back to the default Trino/Iceberg behavior.

C. Extremely light hot path

Neo4j and OpenSearch cannot become a bottleneck for each query.
Online queries must be small, with a short timeout, explicit budget and cache.
Heavy computations must be offline/nearline.

D. Explicit fallback

If the sidecar fails, Trino keeps executing normally.

E. No unnecessary fork of the Trino core

Start with the SPI/plugin architecture. Only propose changes to the Trino core if there is clear proof that the goal cannot be achieved any other way.

F. Evolution in phases

You must implement in small, verifiable and reversible phases.

What the model must produce

You must work in implementation mode and deliver concrete artifacts, not just ideas.

Expected deliverables

Survey of the repository's current architecture
- real stack;
- existing modules;
- integration points with Trino, Neo4j, OpenSearch, vLLM, BI and the ontology;
- gaps and risks.
Technical architecture document
- components;
- online and offline flows;
- contracts;
- textual diagrams;
- trade-offs;
- risks;
- rollout plan.
Domain / graph model
- labels, relationships, properties;
- constraints and indexes;
- versioning per snapshot;
- separation between authoritative, observed and inferred metadata.
Trino integration plan
- modules/classes to create;
- SPI interfaces to implement;
- call flow;
- strategy pattern for the advisor;
- feature flags.
Real code implementation
- Trino plugin;
- advisor service;
- Neo4j/OpenSearch clients;
- ingestion pipelines;
- schemas/configs;
- tests;
- operational documentation.
Test and benchmark plan
- baseline vs advisor;
- repetitive and ad hoc workloads;
- planning latency;
- total query time;
- bytes read;
- effective pruning;
- spill;
- regressions.
Operations runbook
- deploy;
- rollback;
- configuration;
- troubleshooting;
- metrics and alerts.

How you must work

Follow this process:

Phase 0 — Inspection and alignment with the real code

Before coding, survey the repository and answer clearly:

which module integrates Trino today;
which connector/catalog is used;
how Neo4j is already used on the platform;
how OpenSearch is already used;
how events are produced today;
where the investigative ontology already appears;
how Datamash and the dashboards consume data;
whether there are already Java, Kotlin, Python, Go services or another dominant pattern.

Do not impose new technology without need. Reuse the repository's dominant stack.

Phase 1 — Minimum viable architecture

First implement a minimal version with:

capture of Trino events;
ingestion into OpenSearch;
a minimal model in Neo4j;
an advisor service with fallback;
integration on the Trino side only to collect context and, if possible, improve statistics;
no hard heuristic pruning.

Phase 2 — Planning-oriented advice

Expand to:

better statistics;
ranking of manifests/files/splits;
safe prioritization;
recommendation confidence;
explainability.

Phase 3 — Learning layer

Add:

query fingerprints;
filter/join/group patterns;
co-access;
hotness;
operational risks;
scores learned offline/nearline.

Phase 4 — Physical maintenance optimization

Add recommendations or controlled automations for:

compaction;
rewrite manifests;
reorganization by sort/order;
physical layout improvements for recurring workloads.

Phase 5 — Semantic and investigative assistance

Integrate with:

the investigative ontology;
vLLM for explanations, assisted diagnostics and semantic exploration;
dashboards and BI.

The LLM may explain, recommend and enrich exploration — but it must not be the optimizer's critical decision layer.

Recommended target architecture

Use this design as a starting point, adapting it to the real repository:

text

[Usuário SQL / BI / APIs / Painéis]
              |
              v
          [Apache Trino]
              |
              v
 [Trino Plugin / Connector Wrapper / Advisor Hook]
              |
      +-------+--------+
      |                |
      v                v
[Advisor Service]   [Fallback Padrão Trino/Iceberg]
      |
  +---+-------------------------------+
  |                                   |
  v                                   v
[Neo4j Knowledge Graph]         [OpenSearch Telemetry / Retrieval / Analytics]
  |                                   |
  +-------------------+---------------+
                      |
                      v
         [Offline/Nearline Learning Pipelines]
                      |
                      v
      [Scores / Embeddings / Plan Advice / Explainability]
                      |
                      v
           [Datamash BI / Painéis / Admin / Observability]

Suggested graph model

Implement an initial model compatible with this conceptual structure:

Authoritative nodes

Catalog
Schema
Table
Snapshot
ManifestList
Manifest
DataFile
DeleteFile
Partition
Column
ColumnMetric
PartitionSpec
SortOrder
StorageObject

Observed nodes

QueryTemplate
QueryRun
PredicateBundle
JoinPattern
GroupingPattern
PlanShape
WorkloadClass
ExecutionOutcome

Inferred / intelligent nodes

SelectivityEstimate
BroadcastLikelihood
SpillRisk
ManifestAffinity
CoAccessCluster
HotRegion
RewriteBenefit
OptimizationHint
SemanticEntity
OntologyConcept

Exemplary relationships

(:Catalog)-[:HAS_SCHEMA]->(:Schema)
(:Schema)-[:HAS_TABLE]->(:Table)
(:Table)-[:HAS_SNAPSHOT]->(:Snapshot)
(:Snapshot)-[:HAS_MANIFEST_LIST]->(:ManifestList)
(:ManifestList)-[:LISTS]->(:Manifest)
(:Manifest)-[:CONTAINS]->(:DataFile)
(:Manifest)-[:CONTAINS_DELETE]->(:DeleteFile)
(:DataFile)-[:IN_PARTITION]->(:Partition)
(:DataFile)-[:HAS_METRIC]->(:ColumnMetric)
(:ColumnMetric)-[:FOR_COLUMN]->(:Column)
(:QueryRun)-[:INSTANCE_OF]->(:QueryTemplate)
(:QueryRun)-[:READS]->(:DataFile)
(:QueryTemplate)-[:FILTERS_ON]->(:PredicateBundle)
(:QueryTemplate)-[:JOINS_ON]->(:JoinPattern)
(:QueryTemplate)-[:GROUPS_BY]->(:GroupingPattern)
(:PredicateBundle)-[:LIKELY_PRUNES_TO]->(:Manifest)
(:PredicateBundle)-[:LIKELY_PRUNES_TO]->(:DataFile)
(:DataFile)-[:CO_ACCESSED_WITH]->(:DataFile)
(:JoinPattern)-[:WORKED_BEST_WITH]->(:PlanShape)
(:Table)-[:RELATED_TO_ONTOLOGY]->(:OntologyConcept)
(:SemanticEntity)-[:INSTANCE_OF]->(:OntologyConcept)
(:QueryTemplate)-[:INVESTIGATES]->(:SemanticEntity)

Mandatory properties on inferred entities

Every inferred piece of information must carry, at minimum:

confidence
support
validFrom
validTo
lastRefreshedAt
derivedFrom
snapshotId
modelVersion

GDS usage strategy (Neo4j Graph Data Science)

Use GDS only where it makes operational sense.

Rules

Do not run heavy GDS algorithms in the query hot path.
Use GDS to generate offline/nearline signals.
Expose the results as simple features consumable by the advisor.
Every use of GDS must be justifiable by measured benefit.

Recommended use cases

embeddings of queries, patterns and physical sets;
similarity between queries;
co-access clusters;
operational hotness/centrality;
link prediction for recurring patterns;
grouping by workload.

Non-recommended use cases

heavy synchronous decision on every query;
any dependency that increases planning latency too much;
replacing the engine's deterministic rules with probabilistic inference.

Advisor contract

Design a small, explicit and safe contract, for example:

json

{
  "requestId": "uuid",
  "catalog": "string",
  "schema": "string",
  "table": "string",
  "snapshotId": "string",
  "predicateFingerprint": "string",
  "projectedColumns": ["string"],
  "joinKeys": ["string"],
  "workloadClass": "string",
  "timeBudgetMs": 20
}

Suggested response:

json

{
  "requestId": "uuid",
  "snapshotId": "string",
  "estimatedRowsAfterFilter": 12345,
  "estimatedBytesAfterFilter": 987654321,
  "broadcastLikelihood": 0.81,
  "spillRisk": 0.14,
  "preferredSplitOrder": ["splitA", "splitB"],
  "candidateManifestIds": ["m1", "m2"],
  "candidateFileIds": ["f1", "f2"],
  "confidence": 0.77,
  "explanations": [
    "High historical selectivity for predicate bundle X on snapshot family Y",
    "Repeated co-access pattern indicates manifest group M is a likely hit"
  ],
  "fallbackRecommended": false
}

Mandatory advisor rules

low timeout;
strong cache;
no critical dependency for execution;
small responses;
no hard pruning without authoritative validation;
structured logging;
metrics and tracing.

Recommended integration with Trino

When inspecting the repository and the current connector, implement the integration preferably at these points, or their equivalents in the real code version:

ConnectorMetadata
applyFilter
getTableStatistics
ConnectorSplitManager
query listeners/events

Integration policy

First improve statistics.
Then improve ranking/prioritization.
Only then consider influencing more sensitive decisions.
Never break the default execution path.

OpenSearch: expected role

OpenSearch is not the source of truth for table metadata. It must be used to:

store telemetry and events;
allow operational search and analytics;
support explainability and troubleshooting;
serve as the basis for fast aggregations and derived features;
optionally support vector search and hybrid retrieval if that is already part of the stack.

You must design indices, mappings and retention strategies consistent with this role.

Investigative ontology

The platform has an investigative ontology. You must integrate it in a useful and concrete way.

Goal

Make the intelligence layer also capable of:

relating tables, columns and usage patterns to domain concepts;
enriching explainability;
supporting data investigation and discovery;
enabling better diagnostics by entities, concepts, relations and use cases.

Rule

The investigative ontology must not pollute the planner's hot path. It must enrich:

semantic context;
explainability;
investigative workflows;
offline/nearline features.

vLLM / LLMs

The platform has vLLM. Use it with discipline.

Appropriate cases

explain why a recommendation was made;
summarize workload patterns;
suggest tuning or maintenance;
support operations and investigation teams;
support regression analysis;
generate operational documentation.

Inappropriate cases

autonomously deciding the final query plan;
replacing deterministic statistics;
blocking or changing critical execution without validation.

Success criteria

Consider the project successful only if there is measurable evidence of gains. You must define and measure at least:

reduction in planning latency;
reduction in bytes read;
increase in effective pruning;
improvement in total time for repetitive workloads;
less spill in sensitive queries;
zero correctness regressions;
safe fallback in case of sidecar failure;
adequate observability;
ability to explain decisions.

Quality requirements

All delivered code must be:

readable;
modular;
testable;
observable;
documented;
configurable;
backward-compatible when possible;
protected by feature flags;
accompanied by unit, integration and benchmark tests when applicable.

Important constraints

Do not rewrite the whole platform.
Do not replace Trino with another engine.
Do not replace Iceberg/Parquet or the current catalog without an extremely strong justification.
Do not make Neo4j or OpenSearch a mandatory dependency for the query to work.
Do not use an LLM in the planner's critical hot path.
Do not invent unnecessary dependencies if the repository already has a dominant stack.

Format of your response and execution

You must respond and work in the following format:

Step 1 — Discovery

summarize the current architecture found in the repository;
list modules and integration points;
identify gaps;
propose an architecture adapted to the reality of the code.

Step 2 — Implementation plan

show a phased roadmap;
list files/modules to create or change;
describe contracts and flows;
identify risks and mitigation.

Step 3 — Implementation

make real changes to the code;
show clear patches;
explain decisions only when necessary;
follow the repository's standard.

Step 4 — Tests and validation

create tests;
propose a benchmark;
demonstrate the fallback;
describe how to measure gain.

Step 5 — Operation

provide a runbook;
metrics;
alerts;
troubleshooting;
gradual rollout plan.

If at any point there is more than one viable alternative, choose the most conservative and pragmatic one first, explain in a few lines why you chose it, and move on.

Final implementation guidance

Be highly technical and practical.
Do not stay at design only: implement.
Preserve the existing platform.
Maximize reuse of components already present.
Use Neo4j for complex relationships and intelligent metadata.
Use OpenSearch for telemetry, search and operational analytics.
Use vLLM for explainability and assistance, not for critical decisions.
Make the datta-intelligent-query-layer feature a clear, safe and measurable advantage.
The result must be something an engineering team can actually put into production.

Now start with the real inspection of the repository and first deliver:

a map of the current architecture;
concrete integration points with Trino;
the minimum viable design of the datta-intelligent-query-layer feature's sidecar;
a phased implementation plan;
the first batch of files/code to create.

Specification of the datta-intelligent-query-layer feature for Apache Trino with Neo4j, OpenSearch, vLLM, Investigative Ontology, BI and Dashboards