Monitored Sources — Websites that Keep Themselves Up to Date

Legislation portals, official gazettes and institutional pages change without notice — and keeping that content current by hand means revisiting every site, comparing versions and re-importing everything just in case. With DATTA's monitored sources you register the page once, set the schedule, and the platform checks the origin on its own: it detects what changed, re-ingests only the altered pages and versions every document — the previous version is never deleted, only marked as no longer in force.

All from the Sistema›Conexões screen, with no infrastructure access whatsoever.

Architecture (delta, validity, supersede, cascade): monitored source versioning. Detailed operational cases: troubleshooting runbook.

How it works

A monitored source is a connection of type Fonte Web Monitorada (WEB_SOURCE) with a check schedule. It is registered in the same central connection registry as every other external source on the platform — nothing is stored on the side.

On each run, the platform compares the origin against what has already been ingested, always going from the cheapest signal to the most expensive: first the page's ETag header, then the hash of the extracted text and, for API sources, the delta cursor. Only what actually changed triggers the ingestion pipeline. No change, no expensive processing.

When a page changes, the document gets a new version and the previous one is marked as superseded — never deleted. The full history stays preserved for reference and audit.

Step by step: registering a web source

Registering requires the SOURCE_MONITOR_MANAGE permission.

1. In Sistema›Conexões, create a new connection and choose the Fonte Web Monitorada type.
2. Fill in the URL raiz (root URL) — the crawl's starting page (e.g. https://www.planalto.gov.br/ccivil_03/leis/). Only public http/https addresses are accepted; private or internal addresses are blocked for security, with a clear message in Portuguese.
3. Choose the collection strategy:
   • Raspar texto ("Scrape text", SCRAPE_TEXT, default) — extracts the main content of the pages;
   • Baixar arquivos ("Download files", DOWNLOAD_FILES) — downloads the linked files (PDF, Office — see the supported formats);
   • API com cursor ("API with cursor", API_CURSOR) — incremental collection through an API. There is no concrete connector available yet: the run finishes as "no changes", with a notice.
4. Select the target context (destination domain) — required: it defines where the documents will be written. The destination is never guessed.
5. Adjust the maximum depth (default 1, limit 5) and the maximum pages per sync (default 25, limit 500).
6. Save. Creation goes through the normal connection flow — with a fingerprint for deduplication and an audit record (DISCOVERY.ENSURE_SOURCE) — and the source is born with a first check already triggered.

Configuring the schedule: the Monitoring panel

Open the connection and go to the Monitoramento panel:

• Monitoramento habilitado — the switch turns scheduling on and off.
• Agenda — pick a preset: Daily at 06:00 (0 0 6 * * *), Every 6 hours (0 0 */6 * * *), Weekly (Monday 06:00) (0 0 6 * * MON) — or Custom, with a 6-field cron expression (Spring format). An invalid expression is rejected on the spot, with a message in Portuguese.
• Execution window (optional) — Start and End in HH:mm. A check that comes due outside the window waits for the next window. The window can cross midnight (e.g. 22:00–05:00). Provide both times or neither.
• Policy on detected change:
  • Reingerir ("Re-ingest", REINGERIR_AUTO, default) — the changed URLs are re-ingested automatically, with supersede versioning;
  • Apenas notificar ("Notify only", APENAS_NOTIFICAR) — the change is only recorded, for a curation decision.
• Salvar monitoramento — persists the configuration and records the SOURCE.MONITORING_UPDATED audit event.

The screen shows Last check and Next check, and the connection list flags each source with Monitorada or Sem agenda.

Every change to schedule, window or policy generates an audit event, and so does change detection itself when the policy is Notify only — in that case under the SOURCE.CHANGE_DETECTED event.

The same operations are also available for integration through the platform API — reading and writing the schedule, triggering a check and querying the history. See the API reference.

Check now

Don't want to wait for the schedule? Use the Verificar agora button on the Monitoring panel. The action requires the SOURCE_MONITOR_RUN permission.

• The check responds immediately and runs in the background: the platform returns the synchronization identifier, the RUNNING state and the path to follow it in the history.
• It uses the same lock as the scheduler: if a check is already in progress for that source, the new one is refused with a "sync in progress" message. There are never two concurrent checks of the same source.
• Limit of 10 runs per hour, per user — beyond that the call is refused for too many requests.
• Whoever triggered it is recorded both in the run and in the audit: a manual check is never confused with a scheduled one.

Synchronization history

The History panel lists paginated runs (20 per page by default; the size accepts 1 to 100). Columns: when it ran, the trigger (SCHEDULED, MANUAL or ON_CREATE), the outcome (Concluído, Sem mudanças, Falhou or Executando), pages checked, new documents, changed documents and the error message when there is one.

How to read it:

Permissions

The interface hides the controls from anyone without the permission ("Você não tem permissão para alterar/executar/visualizar o monitoramento desta fonte." — "You do not have permission to change/run/view monitoring for this source."), but the real block is always on the server: the call is refused with 403 and the event goes into the audit trail. See the roles and permissions guide.

Execution limits and fine tuning

The platform applies automatic limits to protect both the origin website and the instance. The settings live under the datta.connections.source-monitor.* configuration prefix (environment variables in parentheses):

Attention: these settings do not have their own field in the installation's declarative configuration yet — today the override is done through an environment variable on the deployment. Persist the change in the versioned configuration before closing any change to these values in production.

The schedule state and the execution lock live in the platform cache, under the keys datta:sources:monitoring:{id} (configuration, 5-minute validity) and datta:sources:sync-lock:{id} (lock, 10-minute validity).

Troubleshooting

Detailed cases — stuck lock, orphaned synchronization, false delta and validity backfill — are in the troubleshooting runbook.