DATTA Extract — User Guide

Extract is DATTA's ELT flow: extract data from external sources and persist it as structured datasets in the catalog, optionally materializing into Iceberg, Neo4j or OpenSearch.

Three screens coexist under DATTA Extract:

• Packages (/pipeline-pacotes.html): gallery/inventory of the extraction pipelines that already exist — search, filter, run, schedule, duplicate, archive and open each one in the right editor.
• DATTA Extract (/pipeline-designer.html): the editor, with two modes over the same package — switch on the selector in the top bar.
  • Assistente (?modo=simples): guided 3-step flow with a worktable — source, transform, target.
  • Avançado (?modo=avancado): flow canvas, with multiple sources, joins, multiple targets and undo/redo.

Changed. These were two pages, with different execution engines: the Designer submitted to Spark and Quick Extract ran inside the service JVM. They are now two modes of the same screen, on the same engine — and Spark left the ETL path. The old address (/pipeline-quick.html) still works: it redirects to the Assistente mode, preserving the package ?id=.

The gallery is the shared entry point, and both modes persist the same kind of dataset in the catalog-service.

1. When to use which

Switching modes does not change what the package can do — the capability is the same, because the engine is the same. It changes how you build. The worktable, once exclusive to the Assistente, is available in Avançado too: select a source node and use Mesa de trabalho in the configuration panel. | Find, run or edit a pipeline that already exists | Packages |

2. Packages — extraction pipeline gallery

/pipeline-pacotes.html, under the DATTA Extract › Pacotes menu. It is the inventory of everything that exists: pipelines drawn in the ETL Designer, created in the Assistente, and the external-source flows registered by the seed script (CVM and the public negative base — see External Sources).

2.1 What the screen shows

A KPI strip at the top (Pacotes, Rodando agora, Agendados, Executados hoje, Taxa de sucesso) fed by GET /api/pipelines/stats. If that endpoint fails, the strip simply does not render — the rest of the screen keeps working. The first four cards filter the list (§2.2).

The list comes from GET /api/pipelines/summary (paginated, 24 items per page) and each package shows up as a card or a table row — the Cards / Tabela switch is in the toolbar. On each item:

• Name and description and a status badge (active, paused, draft, archived). In Cards mode the package also carries the origin badge (Designer, Assistente, ETL Agent); the table has no Origin column — who created the package does not change how it is operated.
• The flow chain: source type → number of transformations → target type.
• Integration and source specification chips when the package declares that metadata (the case for the external-source flows). In the table, the integration chip sits next to the package name.
• Schedule in readable Portuguese (e.g. "Todo dia 3 de cada mês às 05:30") when a cron exists; a package scheduled by intent, with no cron expression, reads "Agendado".
• Last updated (relative) and last run with a colored status. A package that never ran shows "Nunca executado" — never a fake duration. If the service cannot determine the last run, the card reads "Última execução indisponível" with a reload button; it never turns a query failure into "Nunca executado".

2.2 Search and filter

• Search by name or description (accent- and case-insensitive), applied with a short delay while you type.
• Filters for Status and Integration. Options and counts come from facets computed by the backend over the search result — a filter with no package does not show up in the list.
• Cards that filter: clicking one of the first four indicators narrows the list; clicking the marked card again clears it.

The card number and the list it opens come from the same predicate in the backend — that is why the filtered total matches the number on the card. All four count packages, not runs: two concurrent runs of the same package count as 1 under "Rodando agora", which is what the list shows. Taxa de sucesso does not filter: it is a percentage, not a slice of the list, so it is not clickable.

• The cards are buttons: they work with Tab + Enter/Space and have a 44px touch target on small screens.
• Filtros ativos (active filters): a row below the toolbar lists what is filtering right now (search, card, status, integration), lets you remove each one and offers Limpar tudo (clear all). Filters combine with logical AND.
• Sort by most recently updated (default), name or last run.
• Atualizar (refresh) next to them reloads both list and indicators.

The Origin filter was removed from the screen: a package created in the Assistente and one created in the Designer are equally valid, so authorship provenance answers no operational question. The origin parameter is still accepted by GET /api/pipelines/summary for script consumers.

"Executados hoje" uses the timezone configured in datta.pipeline.timezone (DATTA_PIPELINE_TIMEZONE); when empty, the service JVM timezone applies — the same one the scheduler uses when a package declares no timezone of its own.

2.3 Opening a package in the right editor

Clicking the card opens the package in Avançado mode, the canvas. Each item's editor field still comes from the backend and is part of the JSON contract, but today it is always DESIGNER: the canvas draws any graph, including the linear chain the Assistente builds, and sending the user to an editor that shows less than the package holds is not justified.

When the graph is not a linear chain — more than one source, a join, more than one target, branches — the reason is shown in Portuguese on the card and in the detail panel. A linear package shows no reason at all: a line repeated identically on every card would be noise.

Previously the choice also depended on the engine — some types "only ran in the Designer" — and that no longer exists.

Navigation happens inside the shell (the side menu and the breadcrumb follow along); no link swaps the iframe from the outside.

2.4 Managing without leaving the gallery

The Ações menu on each package offers:

Every action reports the result as a Portuguese toast and reloads the list.

One honest caveat:

• Duplicating does not copy credentials. The copy gets masked passwords; the screen says so and you must re-enter them in the editor before running the copy.

2.5 Detail panel

The side panel shows origin, status, mode, version, author, schedule and creation/update dates, lets you edit name and description, and draws the package flow on a read-only canvas (the same component used in Quick Extract — §3.4). A package with no drawn steps shows the empty state instead of a blank canvas.

2.6 Empty gallery on a fresh install

On a brand-new install the gallery starts empty and explains the path: create a package in the Assistente, draw one in Avançado mode, or register the external-source flows by running scripts/seed-extract-pipelines.sh — the same script setup-datta.sh runs in Phase 7B of a fresh install (see External Sources).

2.7 Permissions

The gallery gate is ANY-OF: GET /api/pipelines/summary accepts PIPELINE_VIEW or DATTAX_EXECUTE, so someone holding only the execute permission still sees the inventory of what they can run. Only a user with neither gets the permission message — in Portuguese, no HTTP code on screen. Today the Analista role has no PIPELINE_* permission at all (but it does have DATTAX_EXECUTE, which is why it opens the gallery) and Usuário Avançado lacks PIPELINE_DELETE — see RBAC to grant them.

3. Assistente mode — Linear flow

3.1 Step 1: Connection

The picker lists connection-manager-service connections visible to the user (their PRIVATE ones + WORKSPACE + PUBLIC).

If the connection does not exist:

• The + New connection button opens the canonical dialog (compiled/dattabi/InlineConnectionDialog.js, exposed as window.DattabiInlineConnection).
• Note: in the Assistente, the + Nova conexão button redirects to /configurar/conexoes (design decision, 2026-04-25). Create the connection there and come back.
• After creation, ACOC triggers a scan in the background; the connection is already usable.

3.2 Step 2: Schema / Table / Columns

• Schema picker (when JDBC).
• Table or collection/index/label picker.
• Column checkboxes with inline preview (10 rows).

3.3 Step 3: Sink

The destination list is filtered by GET /api/pipelines/quick/capabilities: the wizard only offers what the engine knows how to write, so it never saves a package that would fail validation later. The available destinations are:

You may also pick no sink at all: the result stays as a logical Dataset in the catalog, usable directly in DATTABI.

Also executable, and available in Avançado mode:

• DATTA Search Engine (OpenSearch) — same implementation as the OpenSearch destination, with mode (index/upsert/delete) and idColumn.
• DATTA Relational Engine and HDFS — write to Iceberg through Trino. On the Spark runner these were saveAsTable into Impala/Hive and df.write to HDFS; without Spark, Iceberg is the analytical destination where the written data stays queryable from DATTABI.

Not executable yet (declared in the palette and blocked at validation, with the reason in Portuguese): REST API, MongoDB, OpenShift and S3/MinIO as targets (S3 does run as a source, just not as a target); LDAP and e-mail as sources; and the five statistical analyses, whose support is already planned. They used to not fail at all — they passed the data straight through, and the package finished "successfully" having done nothing.

3.3.1 Neo4j destination: keys, direction and relationships

• Database is mandatory. The target database is never guessed; a destination without database is rejected before any node is written.
• Label and key: Label do Nó and Propriedade Chave (MERGE). When left empty, the historical defaults apply (Entity and id), and each applied default is logged. Careful: these defaults are not a safety net — the load is only rejected when the rows carry no value for the key, so a dataset that happens to have a populated id column writes :Entity nodes without complaining. Fill both fields in.
• column:property syntax on both keys (node and target): use it when the flow column name differs from the graph property name — e.g. cnpjBasico:empresaCnpj matches column cnpjBasico against the target node property empresaCnpj. Without the colon, column and property share the name.
• Direction: SAIDA (default) writes (pipeline node)-[REL]->(target node); ENTRADA flips the arrow — (target node)-[REL]->(pipeline node). Any other value is rejected. In both directions the lookup is driven by the node the pipeline just wrote, so the cost follows the size of the load and not the size of the target label (usually the whole reference base).
• Edges belong to the run, not to the label. Each load stamps the nodes it writes with a technical dattaCargaId property holding that run's id, and the edges are created only from those nodes. Without it, a run would also re-link nodes from earlier runs and from other pipelines writing to the same label. The property stays on the node and is overwritten by the next load; it is only written when Criar Relacionamentos is on.
• The target node is never created. It is always matched; a record with no counterpart simply gets no edge, and that is intentional — the load never invents ghost nodes in the reference base.
• Writing is always idempotent (MERGE by key), so re-running does not duplicate. This destination's form has no write-mode field; it only shows up in imported or API-built packages. If the package declares append — which on the old engine meant creating brand new nodes on every run — the execution finishes normally and the run carries the Write mode ignored warning, so you can fix the package's declaration instead of the mode pretending it was honored.

3.4 Generated pipeline canvas

Still on step 3, next to the summary, the Pipeline gerado card draws the pipeline that will be created from your choices — source, each transformation in order and the target, connected by arrows.

• The drawing comes from POST /api/pipelines/quick/preview: the backend runs the same conversion it would run on save, but stores nothing and executes nothing. So the canvas shows the real pipeline, not an approximation drawn in the browser.
• It refreshes on its own as you change the steps (with a short delay), with a loading indicator.
• Ver pipeline opens the same canvas full screen.
• Clicking a node selects the matching step: an "Etapa selecionada" bar appears with a shortcut to that transformation's worktable.
• The canvas is read-only — you can zoom in, zoom out, pan and use Ajustar à tela, but nodes are neither moved nor deleted there; editing stays in the wizard steps.
• If the drawing cannot be assembled (service down, for instance), the card says so in Portuguese with Tentar novamente, and the chip-style summary remains available. No invented pipeline.

3.5 Opening an existing package in the Assistente

Today there is no way to open an existing package in the Assistente. Every package opens in Avançado mode (§2.3), and the two routes that would seem to lead to the Assistente redirect to the canvas instead:

• Direct address /pipeline-designer.html?modo=simples&id=<package-id> (and the old /pipeline-quick.html?id=…): the Assistente reads the id on boot and switches to Avançado mode instead of hydrating the wizard.
• Switching modes with the package open: the switch only rewrites ?modo= in the URL and warns that unsaved changes will be discarded — nothing of the in-memory package is handed over. With ?id= in the URL it goes back to the canvas; without it, the Assistente opens blank.

The hydration code does exist (hydrateFromDefinition/openPipeline, with the Editando pacote and Salvar alterações labels, the draft-in-progress warning, and the Abrir no Designer fallback for a non-linear graph), but it is unreachable: openPipeline only runs from requestOpen, which is never called and is not exported.

3.6 When you finish

• Executar creates and runs the package; at the end the screen offers Ver na galeria, Abrir no Designer and Acompanhar execução.
• Salvar pacote stores it without running, and offers to go straight to the gallery.
• The old "Meus Pipelines" modal is gone: the Pacotes button leads to the gallery (§2), which is the single inventory.

4. Extract Pipeline (Designer)

/pipeline-designer.html + compiled/pipeline-designer.js.

4.1 Canvas

• Components as M3 cards linked by direct connections.
• Drag-and-drop from the left sidebar onto the canvas.
• There is no auto-save: the drawing is only stored when you click Salvar. Precisely because of that, while unsaved changes exist, closing or reloading the tab raises the browser confirmation.

4.2 Available components

Neo4j destination — the Tamanho da transação (linhas) parameter

How many rows Neo4j commits per transaction. Empty keeps the historical behaviour: the whole batch becomes a single transaction.

This is not the batch size, which is how many rows travel in one request. A batch of 10,000 rows may become ten transactions of 1,000 — same request, the difference is when the database commits.

It matters on large loads. Neo4j's transaction memory ceiling (db.memory.transaction.total.max) is the sum of live transactions; without slicing, the whole load counts as one. That is what killed the CNPJ load at a 4.2 GiB ceiling and again at 8 GiB — raising the ceiling moved the failure from 78 to 131 minutes without removing it, the signature of a ceiling fighting a cause it does not govern. The versioned CNPJ package uses 10,000 on every graph destination.

Executar pacote do Extract (run another package)

Chains packages: this node runs another saved package as a step of the flow. It is how a large load is split into smaller, reusable packages — the CNPJ load, for instance, is a sequence of stages that already exist as standalone packages.

• Pacote a executar lists the saved packages. The current package is not in the list: a package that calls itself runs forever.
• Aguardar conclusão (default: on) holds the flow until the sub-package finishes. That is what makes chaining ordered — putting B after A means B should see what A wrote. Turned off, the package is fired and the flow moves on; a sub-package failure then arrives when nobody is watching.
• Tempo máximo de espera bounds the wait. Timing out does not cancel the sub-package: it keeps running, and it is this flow that stops. Empty means 6 hours.
• A failing sub-package fails the node. Going green over a load that did not happen is worse than stopping.
• Chaining is capped at 5 levels, and a package already in the chain is refused with the whole path in the message (a > b > a), so the operator sees where the loop closes.

Rows reaching this node are discarded: it orders execution, it does not carry data. Wiring it after a stage is what guarantees that stage finished first.

Columns box — the Colunas parameter

Every parameter that asks for columns (Select Columns, Drop Columns, Sort, Deduplicate, Concat, Trim, …) uses the same box, in both Extract modes. It stores a list of names — what the engine reads, and the shape you see in versioned packages: "columns": ["documento", "nome"].

• Check / uncheck picks the column. The list shows the already-picked columns first, in the stored order, then the remaining columns detected in the source preview. Marcar todas and Limpar apply to the whole list.
• The order of the checked rows is the output order for Select Columns and Concat. Drag the ⠿ handle to reorder.
• The pencil edits the name used in this parameter — to fix a spelling or type a column the preview did not return. It does not rename the column at the destination: use the Rename Columns transform for that.
• Adicionar coluna types a name by hand. That is the path when the source has not run a preview yet: with no detected columns the box is still fillable.
• Nothing checked means "all columns" in the optional parameters (Deduplicate, Trim). Where the parameter is required, the box reports Campo obrigatório while no column is checked.

A package saved by an earlier version of the screen could store objects instead of names — in that shape the executor matched no column and the node returned empty rows. Opening the node and touching the box rewrites it in the correct format.

4.3 Advanced SQL mode

In each Source or Transform component, the Advanced mode toggle shows the generated SQL / DATTAX. The user can edit it freely; the designer keeps the outputs contract (if the edit breaks the schema, it displays a pt-BR message).

4.4 Undo / Redo

• Ctrl+Z / Ctrl+Y.
• History in memory (last 50 steps).
• Closing the tab without saving discards the drawing — no draft is kept anywhere; the browser asks for confirmation first (§4.1).

4.5 Scheduling the run — the Agendar button on the toolbar

The schedule is defined on the same screen where the package is built. The Agendar button, between Salvar and Executar, opens the visual repetition editor — the same one from the Calendário tab: every N minutes/hours/days, daily, weekly, monthly or cron, with times, days, validity window and the live preview of the next runs.

The button label states what is stored:

Two rules that prevent writing half a definition:

• A package that has not been saved cannot be scheduled. The schedule lives in the stored definition; the button asks you to save first, in Portuguese.
• With unsaved changes on the drawing, the screen asks first. The editor writes the schedule onto the already saved version of the package — the drawing open on screen stays there and does not go along. The question comes before any write, and the dialog itself repeats the warning while open.

The two actions are independent in both directions: saving the schedule does not require saving the flow, and saving the flow does not erase the schedule — the Designer resends the complete definition (§4.7).

A schedule saved here is written to the package itself — it is the same definition, not a copy — and the Scheduler's Calendário tab shows the new occurrences as soon as the calendar is reloaded.

Known limitation — the "Agendamento" column of the gallery. The package gallery reads a lightweight summary that does not carry the full repetition: it shows the complete description only for older packages scheduled by a cron expression. A package scheduled through the visual editor shows up as "Agendado", without the description. And in an older package that has since been scheduled through the editor, the column keeps showing the previous cron expression, which is no longer the one that fires — the repetition that counts is always the one shown on the Designer's Agendar button and on the Scheduler's Calendário tab. Fixing this is a change to the pipeline-designer-service summary, not to the screen.

Who sees the button. Only whoever has permission to edit the package (PIPELINE_CREATE, or an administrator profile). Without it the button does not appear, and an attempt via direct access is refused in Portuguese — no HTTP code on screen. See RBAC.

The same three paths, a single editor. The component (compiled/pipeline-schedule-editor.js) is loaded by the Designer, by the Scheduler's Calendário tab and by the package gallery. There are not three scheduling screens to drift apart.

How the schedule is executed:

• Manual: runs only on a click on Executar.
• Scheduled: pipeline scheduling fires by the same computation that produces the preview — the time shown is the time that runs.
• Event: triggers when another pipeline finishes (lineage).

4.6 Watching a run on the canvas itself

While a package is running, the Designer becomes the monitoring screen: the diagram you drew stays in view and every box shows what is happening inside it. There is no longer a window on top of the drawing.

What shows up on the canvas while the load runs:

Why the "parcial" (partial) tag exists. The engine is lazy: no row crosses the flow until a destination pulls it. While a destination is still working, the count of whatever comes before it in the drawing is a mid-flight snapshot — and showing it as a final total would be misleading. When the last destination finishes, the tags disappear and the numbers on screen are the definitive ones, the same ones kept in history.

A number that rises, not a bar that fills. Each box shows how many rows have gone through it so far — never a percentage, and never a per-step progress bar. The engine reads the source as a stream, without counting the rows up front, so a per-box percentage could only be a guess. What the screen shows is a measured count: it rises as the load moves and never goes backwards. The only exact fraction on the screen is the one at the top of the step-by-step — how many steps have closed, out of the package total — and it closes by step completion or failure, not by rows processed.

When the load is split across several workers

When the source allows it, the platform splits the reading into pieces and hands each piece to a different worker, so the load finishes sooner. The canvas lights up all the same. Each box shows the flow total at that point — the sum of what every piece has reported so far — not the number of one isolated worker. That is why the parcial tag usually takes longer to disappear on a split load: it only drops when the last piece closes.

What differs from a load that runs in a single place:

• Numbers arrive in slightly wider steps. Each worker reports from time to time; the screen adds up what arrived. The number never goes backwards: each report replaces the previous one from that piece, it is not added to it.
• A box only turns green when every piece that went through it has finished. While any of them is still working, it stays blue.
• A piece that fails paints the box red, not green, and the message says which piece failed. "Finished" with a blown piece would be the worst possible information on this screen.
• A box with no measurement shows no numbers — never 0 → 0. A branch the load has not reached yet appears with its name only. Zero is an assertion; absence of measurement is not.
• If live monitoring drops mid-run, the load carries on normally and the screen says in Portuguese that it stopped receiving reports, instead of showing stale numbers as if they were current.

The per-node breakdown of that load is also kept in history: once it finishes, the Execução por Nó table in the Job Monitor shows the split per box. Split loads that ran before this version still have no such breakdown in history — the data was never written back then, and there is no way to recover it after the fact.

Why the load sometimes runs in a single place

Splitting the reading only pays off when splitting does not change the result. Some steps need to see the whole set to give the right answer — removing duplicates, sorting, limiting to N rows, joining with another table. If one of those sits after a source that was split, each piece would only see its own slice and the result would come out wrong with no error on screen. In that case the platform refuses to split and runs the whole load in one place.

What changed: the refusal now looks at where the step is, not merely at whether it exists in the flow. Deduplicating a small domain table that is read whole from end to end cannot go wrong — so it stopped vetoing the entire flow. In practice, a large package no longer goes to a single place because of one small box in the corner of the drawing: the large side goes to the workers, the small side is resolved first, and the flow finishes sooner.

The run says what happened, in Portuguese: how many branches went to the workers, how many stayed, and — when the whole load stayed in one place — which step prevented the split and which source reaches it. A source the platform cannot split (a file downloaded from an internet address, for example) also keeps the load in one place, and the warning says so in those words.

The diagram keeps responding — in read-only mode

With a load in flight the Designer is not frozen: you can click each box and read its configuration, drag the boxes around to see the drawing better, zoom in, zoom out and fit to screen. What you cannot do is change anything.

• Clicking a box opens that step's configuration in the side panel — the usual fields, with the values currently in force, only they do not accept typing. A band at the top of the panel says why, and the ‹ Passo a passo link in the header brings the monitoring back. Clicking the canvas background also returns there.
• Dragging the boxes changes only the arrangement on your screen; nothing is written while the load runs. The new arrangement is saved with the package the first time you save, after the run has finished.
• While the load is in flight these are out: save, schedule, undo and redo, pause/resume/archive, adding a step from the palette, duplicating or removing a step, the data preview and the worktable. Test connection still works — it writes nothing. Hiding the step-by-step hides the panel, it does not release editing.
• As soon as the run ends — by completion, failure or cancellation — editing comes back on its own, even with the monitoring still open on screen.

Step-by-step panel

Next to the canvas the run's step-by-step opens: one line per event, in order, coloured by outcome — green for success, amber for warning, red for failure. Each line carries the time, the step and, where it makes sense, the count at that moment.

• At the top of the panel: the run state, how many steps have closed out of the package total, and how many events the step-by-step has accumulated. The step count closes on completion or failure — it is not a percentage of rows, because the engine only knows how many are left after reading the whole source.
• Filters by step and by level, to isolate only what matters (for example, only the failures of one destination).
• Auto-scroll that follows new events — and pauses by itself when you scroll back to read something. The Acompanhar o fim button resumes following.
• Cancel run stops the load at the next batch; the step-by-step records the cancellation. See "Stopping a load", just below.
• Closing the panel hides the step-by-step without stopping the load — and, with the load still in flight, without giving editing back: the diagram is still the run's, now with the selected step's configuration beside it. While a known run exists, the Acompanhar button in the toolbar brings the step-by-step back.

Stopping a load

The request reaches the load wherever it is. It does not matter which instance of the platform answered the click, nor whether the read was split across several workers: the request applies to all of them. Chunks that have not started are abandoned; those in the middle stop at the next safe point. The run ends as cancelled, never as "failed" — cancelling is not an incident to investigate.

The stop is not instantaneous, and the reason matters. Writes to the target are done in batches. Cutting mid-batch would leave the target half written with nobody knowing where it stopped. So the load finishes the batch in progress and only then ends: on a large load that interval can run from a few seconds to a few minutes. Monitoring stays alive throughout: the step-by-step keeps receiving events until the actual end, and it is the one that says when it is over — not the click.

What stays written. Cancelling interrupts; it does not undo. Whatever was already written stays in the target. On a load into the graph the typical outcome is nodes written without the links between them, because links are created after the nodes, and without the post-load step, which runs at the very end. Treat a cancelled load as incomplete, not as partial-but-consistent: repeat it when you want the whole result. Repeating is safe on targets that write by key (graph and search), because reprocessing the same row does not duplicate it; on a target that only appends rows, check what is there before triggering again.

Temporary space. A load that reads from a URL or spills to disk uses temporary space while it runs. When it ends — by completion, failure, or cancellation — that space is released, and the release waits for the file to fall out of use: deleting before the download has let go of the file would return no space at all. Worth knowing when planning a large load of compressed files: the disk peak is twice what is being extracted at that moment — the archive and the extracted content coexist until extraction finishes.

Arriving mid-run (or after it)

Opening the Designer with a package already running goes straight to monitoring, with everything that happened before rebuilt — the screen does not start from scratch, and the diagram already answers clicks and dragging, in read-only mode (see above). A package that already finished shows the full step-by-step, with the final numbers and no partial tags — and, since no load is in flight any more, editing is released.

The step-by-step is kept for a few hours after the run ends. After that, the screen shows the summary recorded in history and says, in Portuguese, that the detail is no longer available — instead of showing an empty list with no explanation.

When the live connection drops

Monitoring arrives over a continuous channel. If it drops (flaky network, suspended tab, service restarting), the screen does not pretend to be alive: a warning appears saying the live connection dropped and the page is refreshing every 3 seconds until it returns, the screen falls back to periodic re-reads, and it goes back to live mode on its own as soon as the connection returns.

Where to see the history of every run

The Job Monitor tab (DATTA Extract › Scheduler, tab Job Monitor) lists the runs of every package, with status, duration, who triggered it, the warnings, the error with the stage where it happened, and the per-node breakdown — when the run recorded that breakdown; when it did not, the screen says so instead of filling in a plausible number. There is no log viewer on that tab: the line-by-line detail is the step-by-step described above, and it belongs to that package. See Job Monitor.

The other two tabs of the same screen answer the rest: the Calendário shows the whole month — what already ran and what will run — and the Motor, for platform administrators, shows the scheduler state and allows pausing a package.

There is no execution timeout. The package stores ExecutionPolicy.timeoutSeconds (default 300s; the versioned external-source packages declare 3600s), but nothing reads that field: neither pipeline execution engine nor GraphExecutor applies any time limit. The only timeout on the path is UrlDownloader's network timeout. A run stuck on a slow source hangs until someone steps in.

Run warnings — read them before calling the load good

A run can finish Completed and still not have done what you expected: the pass did not fail, but the data was lost along the way. When the engine notices that, it says so — the run carries a list of warnings, and each one points at where to look. The same warnings show up in amber in the step-by-step (§4.6), the moment they happen.

None of these warnings stops the execution: a legitimately empty load — the day the source published nothing — is not a defect, which is why the engine reports instead of failing.

4.7 Opening an existing pipeline in the Designer

Clicking a package whose editor = DESIGNER in the gallery (§2.3) loads the whole flow onto the canvas: nodes at their original positions, labels, each component's configuration and the edges. This is how the external-source flows (CVM and the negative base) became editable through the interface.

• While loading, the screen shows Abrindo pipeline…; if it takes too long or fails, a Portuguese message appears with Tentar novamente and Começar um pipeline em branco — never an empty canvas with no explanation.
• A chip at the top shows the name of the open pipeline, and the breadcrumb gets that name as its last link.
• The chip carries an × that closes the package and hands back a blank canvas without leaving the screen. With unsaved changes it asks what to do with them first: Continuar editando (nothing changes), Descartar alterações (closes and drops whatever was not saved) or Salvar e fechar (saves and only then closes — if the save fails, the package stays open with everything on screen). With nothing pending there is no question: the package closes straight away. Closing also drops ?id= from the address, so a reload does not resurrect the package that was just closed.
• A node whose connector type does not exist in this installation's catalog raises a warning instead of silently disappearing from the drawing.
• On save, the Designer resends the complete definition — description, schedule, execution policy, metadata and version are preserved, and the package is not unscheduled by accident.
• The same pipeline also opens by direct address: /pipeline-designer.html?id=<package-id>.

5. Inline connections

The 4 entry points (DATTABI, Avançado mode, Assistente, Catalog) use the same canonical component (InlineConnectionDialog.js) without duplication. Creation follows the Project Guidelines item 7:

1. Fingerprint sha256(type+host+port+db+user).
2. POST to /api/connections/ensure (idempotent).
3. Per-user CredentialBinding in the CredentialVault.
4. ACOC triggers an asynchronous scan.
5. The response to the frontend carries the connectionId and an SSE link to follow the scan.

6. Post-extract materialization

There is no materialization step after the run. The package's own target node does the writing, during the run, in the unified engine (pipeline execution engine → GraphExecutor) — see §3.3 for what each destination does. There is no "Materialize now" button and no run lineage in the graph.

Materialization of DATTABI datasets is a different thing, living in dattabi-service — that is where dashboards and charts with AFTER_MATERIALIZATION refresh are triggered from.

7. Troubleshooting

8. References

• Packages (gallery): /pipeline-pacotes.html — GET /api/pipelines/summary.
• Pipeline Designer: pipeline-designer-service.
• Assistente mode: reuses the same backend; canvas via POST /api/pipelines/quick/preview.
• Read-only canvas shared by the gallery and the Assistente: compiled/pipeline-canvas.js + pipeline-canvas.css (window.DattaPipelineCanvas).
• External sources (CVM, negative base): External Open-Data Sources, Public Negative Database.
• Permissions: Roles and Permissions — Access Control.
• Connection: Connection Management — User Guide.
• Catalog: catalog-service.
• DATTAX engine: DATTAX — Language Guide.
• Drivers admin: JDBC Drivers — Official Catalog and Governance.