DATTA BI — Refresh Scheduler (refreshing materialized datasets)

The Refresh Scheduler lets you define visually when and how each materialized DATTA BI dataset is refreshed — full or incremental — without writing a single line of technical scheduling. A good dashboard is an up-to-date dashboard, but reprocessing everything every hour wastes resources and hand-editing cron is an invitation to error. Here you pick the mode and the frequency on one screen, and the platform handles execution, history and rollback — without ever leaving a visible data "hole" in the dashboards.

Every dataset that feeds a dashboard is materialized by the platform (Neo4j, OpenSearch or Iceberg, depending on the destination chosen in the DATTAX query). The scheduler controls the life cycle of that materialization.

Open the dialog through the Atualização button on the dataset panel — it is also available inside the dashboard editor.

Full or incremental

Incremental mode requires a watermark column on the dataset: a timestamp or an increasing id. The platform keeps the watermark of the last success and, on the next run, filters only what came after it. Neo4j and OpenSearch destinations write by upsert; Iceberg produces an incremental snapshot.

The choice between full and incremental is per dataset — several dashboards reuse the same dataset and benefit together.

Configure without seeing cron

Everything in the dialog is visual. The platform converts your choice into a refresh policy and enqueues the runs in the distributed scheduler — the cron is generated internally and never exposed.

• Type — Full Refresh or Incremental Refresh.
• Frequency — on demand, hourly, daily, weekly, monthly or custom.
• Time — interpreted in the workspace's timezone.
• Execution window — an interval in which the refresh never runs (for example, "do not run between 08:00–18:00", the users' peak hours).
• Watermark — a list of the dataset's timestamp/id columns, with a preview of the last synchronized value.
• Notifications — email or Slack on success and/or failure.
• Snapshot retention — 7, 30, 90 days or unlimited.

The policy is recorded in the DATTA BI system database (datta-bi-db).

Monitor and control

Real-time status on the dataset card:

History — a timeline with duration, mode, rows processed, final watermark and a link to each run's correlated logs and traces.

Available actions:

• Atualizar agora forces a manual run, respecting the protection against overlapping executions.
• Pausar freezes the schedule without losing the configuration.
• Rollback reverts to the previous snapshot, when the configured retention still keeps it available.

Guarantees you do not have to configure

• No duplicate executions — two overlapping full refreshes of the same dataset are automatically deduplicated by a distributed lock, even with the platform running several instances.
• Bounded queue — a configurable cap on parallel refreshes per workspace; the excess waits in a first-in, first-out queue, without overwhelming the sources.
• Failure without impact — if the DATTAX pipeline fails, the previous snapshot keeps serving the dashboards; there is never a visible data "hole".
• Complete audit trail — every start, success and failure is recorded (DATTABI.REFRESH.STARTED, COMPLETED, FAILED) with author, mode, rows processed, duration and the before/after watermarks.
• Permissions — configuring and triggering refreshes requires DATTABI_DATASET_REFRESH; browsing the history requires DATTABI_VIEW.

Worked example — nightly incremental load with a protected window

1. On the panel of the case-movement dataset, click Atualização.
2. Choose Incremental Refresh and select updated_at as the Watermark column — the preview shows the last value already synchronized.
3. Set Frequency to daily at 03:00, Time in the workspace timezone and the Execution window blocking 08:00–18:00: no scheduled run intrudes on the users' peak hours.
4. Enable failure Notifications and set Snapshot retention to 30 days.
5. The next day, check the History: the nightly run processed only the new records. If a load goes wrong, Rollback returns the dashboards to the previous state immediately.

For refreshing the visuals of a dashboard (on open, manual, by interval, by cron or after materialization), see the DATTA BI guide. The platform also exposes an endpoint to trigger the immediate refresh of a materialized dataset — see the API reference.