Pipeline Copilot — User Guide

Building a complete data pipeline — connection, schema discovery, transformation, analysis and visualization — means chaining several tools and plenty of trial and error. The Pipeline Copilot removes that friction: you describe the goal in plain language and a team of specialized AI agents plans and executes each step, creating ETL pipelines, Jupyter notebooks and DATTABI dashboards while you watch the progress in real time.

1. What it does

You describe what you want in Portuguese. The copilot:

1. Plans the steps automatically using AI.
2. Connects to the data source (JDBC or native database).
3. Discovers the schema and data samples.
4. Generates the DATTAX transformation code.
5. Runs R scripts (if requested).
6. Creates the Jupyter notebook or the DATTABI dashboard.
7. Writes the results to the chosen destination (Neo4j, OpenSearch, Trino).

All of this automatically, with real-time progress on screen.

2. How to access

In the side menu, click Copilot›Pipeline. Or go directly to /copilot/pipeline.

3. Usage examples

3.1 Create a simple ETL pipeline

Objetivo: Extrair a tabela "vendas" do PostgreSQL analytics.empresa.com,
remover registros com valor nulo, e gravar no Neo4j como nos (:Venda).

The copilot:

• Creates/resolves the connection to PostgreSQL.
• Discovers the schema of the vendas table.
• Generates DATTAX transformation code (filters out nulls).
• Writes to Neo4j via MATERIALIZE AS NEO4J.

3.2 Create a notebook with R analysis

Objetivo: Conectar ao MySQL do CRM, extrair clientes com mais de 1 ano de cadastro,
aplicar clustering K-Means em R com tidyverse, e criar um notebook com os resultados.

The copilot:

• Resolves the MySQL connection.
• Discovers the customers table schema.
• Generates DATTAX to extract the data.
• Submits the R script (K-Means via cluster + tidyverse).
• Creates the .dattanb notebook with the analysis and result cells.

3.3 Create an automatic DATTABI dashboard

Objetivo: Pegar os dados de faturamento do Trino (schema finance.faturamento),
calcular receita por regiao e por mes, e criar um dashboard com graficos de barra e linha.

The copilot:

• Resolves the Trino connection.
• Generates aggregation DATTAX (region + month).
• Creates the dashboard with a bar chart (by region) and a line chart (by month).

4. Real-time progress

During execution, the interface shows the progress of each step:

5. Available agents

The copilot brings in the right agent for each need:

6. Supported R packages

The platform's managed R environment ships with the packages below pre-installed, ready for the R agent to use:

If you need an additional package, contact the administrator to include it in the platform's R environment.

7. Supported destinations

8. Limitations

• Maximum of 8 planning rounds per pipeline. Very complex goals should be split into smaller pipelines.
• R scripts can take up to 10 minutes. Larger operations should be scheduled as recurring runs.
• The copilot has no access to credentials — connections with user/password must be registered beforehand under Sistema›Conexões.
• Native sources (Neo4j, OpenSearch as a source) require the corresponding database agent to be configured.

9. Tips

• Be specific about the destination: "gravar em Neo4j como (:Venda {id, valor, data})".
• Mention the chart type for dashboards: "grafico de barra por regiao".
• For R, describe the algorithm: "clustering K-Means com k=5".
• If the connection already exists under Sistema›Conexões, the copilot reuses it automatically.
• The notebook is saved in the notebook environment (/jupyter) and can be opened and edited freely.

10. Troubleshooting

11. References

• Captain architecture — the agent that replaced the multi-agent copilot.
• Connections guide.
• DATTAX language guide.
• DATTABI.
• Notebooks.