DATTA Notebook — documentation and examples

Preview — feature under development. Behavior, screens and contracts may change without notice between releases.

Serious analysis usually starts where the dashboard ends: you need code, statistics and the freedom to combine sources. The DATTA Notebook is the iterative, Jupyter-style environment where analytical SQL, graph, full-text search and PySpark live in the same notebook — with no environment to install, no credentials to hunt down and no connection string to paste before the first useful line. This page is the map: what exists, where to start and where to go next.

What you have at your disposal

• Trino 480 (analytical SQL) — massive aggregations over Parquet/ORC stored on S3/GCS.
• Neo4j (graphs) — cases, parties and legislation with their relationships.
• OpenSearch (full text) — document indexes.
• PySpark — transformations and machine learning at scale.

The previous analytical engine, Impala 4.0.0, has been discontinued: all new SQL is written for Trino, and the usage guide carries the equivalence table for anyone migrating older queries.

The documentation, and who it is for

First steps

1. Open the notebook in your browser at http://<endereco-da-plataforma>/jupyter — the environment is served by the platform itself, with no local installation.
2. Click Importar Notebook, in the top right corner, and select the notebook-exemplo-multi-fonte.dattanb file.
3. Click Iniciar Sessão Spark and wait 30 to 60 seconds — that is how long the execution environment takes to be prepared.
4. Start running cells.

Your first query

Analytical SQL straight on Trino:

SELECT programa, COUNT(*) AS total
FROM hive.assistencia_social.beneficios
GROUP BY programa;

Or the same idea in Python, with PySpark:

df = spark.sql("""
    SELECT * FROM hive.assistencia_social.cidadaos LIMIT 1000
""").toPandas()
print(df.shape)

The available data

Trino — social assistance

Total: ~23M records in Parquet — enough volume to feel the performance of real aggregations.

Neo4j — graph (13 databases)

• datta-datacatalog → Dataset, Column, LineageNode
• datta-ontology → Ontology, EntityType, DigitalTwin
• datta-graph → Processo, Parte, Empresa, Decisao

OpenSearch — full text

• processos → judicial cases
• legislacao → laws and regulations
• documentos → assorted documents

Typical use cases

1. Initial exploration — get to know the size of each table with simple SQL:

SELECT 'cidadaos' AS tabela, COUNT(*) FROM hive.assistencia_social.cidadaos
UNION ALL
SELECT 'beneficios', COUNT(*) FROM hive.assistencia_social.beneficios
-- ... and so on

2. Quality analysis — load a sample into pandas and inspect nulls and distributions:

df = spark.sql("""
    SELECT * FROM hive.assistencia_social.beneficios LIMIT 50000
""").toPandas()

print(df.isnull().sum())
print(df.describe())

3. Multi-source correlation — beneficiaries in Trino, related cases in the graph and legislation in OpenSearch, all in the same notebook. The enrichment section of the example notebook shows the full chain.

Complete example

import pyspark.sql.functions as F
import pandas as pd

# 1. Extract (Trino)
df_beneficios = spark.sql("""
    SELECT b.*, c.renda, c.municipio
    FROM hive.assistencia_social.beneficios b
    LEFT JOIN hive.assistencia_social.cidadaos c
        ON b.titular_cpf = c.cpf
    LIMIT 10000
""").toPandas()

# 2. Transform (pandas)
df_beneficios['valor_por_renda'] = \
    df_beneficios['valor'] / df_beneficios['renda'].fillna(1)

# 3. Enrich (Neo4j) — see the example notebook

# 4. Visualize
print(df_beneficios.groupby('programa')['valor'].agg(['count', 'sum', 'mean']))

Environment configuration

The connections to Trino, Neo4j and Spark arrive ready in the notebook environment: the corresponding variables are filled in by the installation and the Neo4j credentials are kept in the platform's secret vault — none of that needs (or should) appear in your code.

Changing the Trino configuration or restarting the engine are installation administration tasks, not notebook tasks. To learn which catalogs and settings exist, see the Trino configuration; to apply a change, talk to whoever administers the platform.

When something does not work

Detailed cases are in the troubleshooting section of the usage guide.

Progressive learning

Level 1 — simple SQL

• Read Pattern 1 of the guide.
• Count records per table.
• Run a simple aggregation (SUM, AVG, COUNT).

Level 2 — joins and transformations

• Read Pattern 2 of the guide.
• Run a LEFT JOIN between tables.
• Load the result into pandas and compute statistics.

Level 3 — integrated multi-source

• Read the guide's practical example.
• Combine Trino, Neo4j and OpenSearch in a single flow.
• Build an anomaly or pattern detector.

Checklist before you start

• [ ] You can open the notebook at http://<endereco-da-plataforma>/jupyter.
• [ ] Trino answers SHOW CATALOGS;.
• [ ] The tables exist (SHOW TABLES IN hive.assistencia_social;).
• [ ] The Neo4j connection works (credentials already arrive resolved in the environment; a failure here is a case for installation administration).
• [ ] You have read the Trino configuration, if Trino is new to you.
• [ ] You imported the example notebook.
• [ ] You ran the first query (SELECT COUNT(...)).
• [ ] You explored the data with LIMIT and aggregations.

Useful links

Support

• Usage guide: Guide: DATTA Multi-Source Notebook
• Technical configuration: Configuration: Trino 480 (Replaces Impala)
• Runnable example: notebook-exemplo-multi-fonte.dattanb
• Problems and suggestions: open a GitHub issue with the [notebook] tag
• Talk to the team: the #datta-data-analytics channel on Slack

Enjoy — and happy discovering.