Public Negative Database

Brazilian and international negative databases loaded into the reference graph by Datta Extract: Transparency Portal sanctions (CEIS/CNEP/CEPIM), politically exposed persons (PEP), institutions authorized by the Central Bank, and the UN Security Council consolidated list (CSNU). Screening against these databases is an inseparable part of KYB — and checking the UN list is a legal obligation (Law 13.810/2019, BCB Circular 3.978).

Like every open-data integration on the platform, each source is a DATTA Extract flow (origin=DESIGNER, files under scripts/pipelines/base-negativa/), executed by the platform's own engine, inside pipeline-designer-service, on the flow's own cron schedule. There is no dedicated service, page, or Java class per source.

Previously execution was submitted to Spark, in a dedicated container. Not any more: the engine is the service's own, and Spark left the ETL path.

Operations start at Extract › Pacotes, the gallery that lists every extraction pipeline: search the flow by name (e.g. "CEIS", "CSNU"), run, pause, duplicate or archive straight from the card, and open it for editing — in Avançado mode, because these flows are not the simple linear chains the Assistente builds (the gallery states the reason on the card). See External Open Data Sources for the general anatomy and for the permission table.

The seven flows

Per-ORIGIN destinations: each origin has its OWN Neo4j database — declared as a NEO4J dataSource of the FEBRABAN context with role=base-negativa (CEIS, CNEP, CEPIM, PEP, CSNU) or role=bcb (institutions) —, its own screening index (datta_base-negativa / datta_bcb) and its own Trino/Iceberg schema (base_negativa.* / bcb.*, overwrite mode: the sources are full snapshots). Every node carries versaoCarga (execution date) and graph reloads are MERGE-by-key.

The edge to the Federal Revenue (:Empresa) no longer exists: the Revenue base lives in the cnpj database and Neo4j has no cross-database edges. The link is by KEY — Sancao.cnpjBasico on one side, Empresa.empresaCnpj on the other — and the join is done by the KYB cross-link (sancoesDaEmpresa), never by the package (Project Guidelines §22).

URL source component capabilities

The flows use UrlSource capabilities designed for Brazilian open data — all configurable on the canvas, no code:

• Dated URL with staleness tolerance — the URL accepts the {data:pattern} placeholder (e.g. .../ceis/{data:yyyyMMdd}, @dataBase='{data:MM-yyyy}'). At run time the runner tries the current date and steps back up to N periods (Staleness tolerance field): days when the pattern has a day component, months otherwise. Portal publication lags and CEPIM publishes on irregular days — the tolerance absorbs both.
• Minimum size (bytes) — BCB's Olinda answers HTTP 200 with a header-only CSV for a not-yet-published month; a response smaller than the minimum counts as "not published" and steps the date back. HTTP status alone cannot tell. In the BCB package the floor has no effect today: its sources write the minBytes key while the component reads minimoBytes — without the expected key the 1-byte default applies, which any header-only response clears. In the Transparency Portal packages (CEIS/CNEP/CEPIM/PEP), which write minimoBytes, the floor does apply.
• Zip compression — the Transparency Portal publishes a zip with one CSV inside; the File inside the zip field picks the entry when there is more than one.
• XML format — the UN list is XML. The runner converts it to tabular records by the Record element (INDIVIDUAL/ENTITY); repeated groups (aliases) become arrays reachable from designer formulas (transform(INDIVIDUAL_ALIAS, x -> x.ALIAS_NAME)).
• Schema inference toggle — registries keyed by digit-only CPF/CNPJ turn off Infer types automatically: inference would turn "07935099120" into a number and destroy the leading zero.
• Quoting toggle (empty quote) — Brazilian open data is not always valid CSV. The CVM sanctioning and fund datasets carry a stray typographic quote inside a quoted field (alínea c"), and the RFC 4180 parser aborts the whole load, not just the row. Turning quoting off fixes it — but it is only safe when no quoted field contains the delimiter, and that condition does not hold for every source. Measured on 2026-08-05, counting fields per line with and without quoting:

Applying the toggle across the board would corrupt CEIS, CNEP and BCB without raising an error — just shifted columns. Measure the source first, with this test: if the number of fields per line changes when quoting is disabled, it is structural and the toggle is forbidden there.

• The key index is guaranteed by the platform — each batch's MERGE uses the destination's key property. With no index on it, every batch scans the whole label and the load cost grows with the square of the row count. The symptom is not slowness, it is failure: on 2026-08-05 the PEP load (133,905 rows) wrote about 90 thousand nodes into the febraban database — which had no key index at all — and blew the 60s per-batch timeout, ending with the label half written. Today graph-data-service issues CREATE INDEX ... IF NOT EXISTS on the first write of each database/label/key combination, and on both sides of the relate-by-key. You do not need to create an index by hand for a new package; if the Neo4j user lacks schema privileges the load still runs and the log records the warning.

Sources that changed on the agency side (2026-08)

Two sources stopped serving data and were replaced. Both failed the same way — HTTP 200 with an empty body, which the status code cannot tell apart from "not published yet":

• BCB: the BcBase/EntidadesSupervisionadas service began returning only the header for every competency tested (01/2025 to 08/2026), as did its sibling function Cooperativas — while the static entity sets of the same service still carry data. Replaced by the Instituicoes_em_funcionamento service, which exposes 4 sets (banks and multiples, credit unions, brokerages, consortium managers) with ~1,750 institutions. Side benefit: the new source publishes the base CNPJ, which is exactly the join key with the Receita :Empresa.
• CVM sanctioning proceedings: PAS/DADOS/pas.csv became a 404 — the directory is gone. Replaced by PROCESSO/SANCIONADOR/DADOS/processo_sancionador.zip, with two CSVs and a new schema.

The source component's Minimum size (bytes) field exists for this class of failure: it turns "200 with an empty body" into "not published", instead of letting the load write zero rows and finish as a success — with the caveat above that, in the BCB package, the key written is not the one the component reads.

Probabilistic matching — never automatic rejection

Two characteristics of the sources shape the product, and are not defects:

• The CPF comes masked in the PEP registry (only the 6 middle digits), the same format as the Federal Revenue partner files. The node stores cpfParcial + nomeNormalizado; matching against a dossier Pessoa is probabilistic, done by the KYB cross-linking with an exposed score — never by this load.
• The UN list carries no document at all. Screening is exclusively by name and aliases, on the text index (below). International homonymy is expected: the result is an input with a score for human decision, never a direct sanction edge nor automatic rejection.

Name screening index

ALL flows write a screening document to their origin's index: CEIS/CNEP/CEPIM/PEP/CSNU into datta_base-negativa and BCB institutions into datta_bcb (names configurable via the seed's BASE_NEGATIVA_INDEX and BCB_INDEX variables). The document is unified across producers — nome, nomeNormalizado, tipoRegistro (ENTIDADE/INDIVIDUO/INSTITUICAO), fonte and the origin's keys as keyword — so search does not depend on which producer wrote the record (Project Guidelines §20). The flow itself creates the index with explicit mappings (keyword for filter fields, text for name/aliases) before the first indexing — a mapping created after data is indexed would silently produce partial filter results.

This index is textual (BM25/fuzzy over names), without embedding vectors by decision: sanction screening matches spellings of proper names, not running text semantics — it is not semantic search and does not enter the platform's embedding flow (Project Guidelines §22 covers text bodies).

Registering the flows (seed)

The same seed as the other integrations — and, like them, it already runs in the installer: setup-datta.sh registers these seven flows in Phase 7B, skipping with a warning when the Revenue context does not exist yet. Details and both manual modes in External sources.

To run it standalone later:

DATTA_API_URL=http://<gateway-ip>:7070 \
DATTA_JWT=<accessToken> \
NEO4J_PASSWORD=<password> \
./scripts/seed-extract-pipelines.sh

Idempotent by name — edits made in the designer are never overwritten, and running the installer again does not duplicate a flow. BASE_NEGATIVA_INDEX is optional (default datta_base-negativa).

Air-gapped installation

Same as the other sources: download the file on a connected machine, make it available on the platform storage and, in Extract › Pacotes, duplicate the flow and open the copy in the designer, swapping the source node from URL to File (FileSource) — for the Portal zips, extract the CSV first (FileSource does not decompress). Pause the original URL flow.

Dossier certificates

The validity of the certificates extracted from the dossier is assessed offline, with no internet dependency: the expiry date is compared against the analysis date and the certificate gets a status — VALIDA, A_VENCER (inside the alert window), VENCIDA (with the days overdue) or NAO_VERIFICAVEL. The status arrives ready at the console, the compliance summary and the triage data block; the model does not compare dates.

A certificate with no readable date is NAO_VERIFICAVEL, never valid — the gap becomes an insufficient-evidence criterion, which is the correct behaviour.

The analysis date is the day in America/Sao_Paulo, not the server clock. Platform containers run in UTC; using their clock would roll the day over at 21:00 Brasília time and a certificate expiring today would come out as expired.

The alert window is platform-wide, not per service. The same window builds the badge the analyst sees and the report triage produces — different windows would make the screen say "expiring" and the report say "valid" for the same certificate. So it is set once, at datta.kyb.certidoes.diasAlerta in the chart values, which publishes it to every service. Do not set the variable on a single service.

The values substituted into {cnpj} and {numero} are encoded before the URL is assembled: the certificate number comes from the document the requester themselves submitted, and without encoding it could append parameters to the lookup the institution performs.

The online lookup is optional and ships disabled. Each verifier is registered as a web source under Configure › Connections (with automatic cataloguing) and pointed at here by certificate type; the URL goes through the anti-SSRF guard on every call. With the lookup disabled, no verifier registered, a URL refused by the guard or the authority offline, the certificate comes back with the offline assessment and an actionable message in Portuguese — never a technical error, and triage never fails because of it. A response from the authority without a recognised status preserves the offline assessment.

Known limitations (by decision)

• Federal tax clearance certificate: the public lookup has a captcha; the dossier result is "not verifiable" — a final decision, not a technical pending item. Bypassing captchas is out of the question. The debitos_federais type does not query the authority even when a verifier is configured.
• State Boards of Trade: there is no unified national open dataset; validation is documental (stamp and registration number on the minutes).
• No public verifier ships with the platform: the verifier list starts empty by decision — each institution points at the service it has a contract with.

Troubleshooting