Skip to content

Retention Policies

Retention policies control how long DIBOP keeps operational data. Proper retention settings balance your need for historical data with storage costs and compliance requirements.

What Data Is Retained

DIBOP stores several categories of operational data, each with its own retention setting:

Execution Logs

Records of every orchestration execution, including:

  • Metadata: Execution ID, orchestration name, status, start/end times, duration, trigger type
  • Step results: Per-step status, duration, and error messages
  • Payloads: Input parameters and output data for each step

Metadata and step results share one retention period. Payloads have a separate (typically shorter) retention period because they can contain large volumes of data.

API Call Logs

Records of every HTTP request DIBOP makes to external systems:

  • Metadata: Timestamp, system, method, endpoint, status code, duration
  • Payloads: Request headers, request body, response headers, response body

Like execution logs, metadata and payloads have separate retention periods.

Dead Letter Queue

Records that failed processing and were set aside for manual review:

  • Failed orchestration executions that exhausted all retry attempts
  • API calls that repeatedly failed and could not be delivered
  • Data that failed validation during canonical mapping

Alert History

Records of every alert firing, resolution, and acknowledgement.

Audit Logs

Records of administrative actions (user changes, role assignments, setting modifications, credential rotations). Audit logs typically have the longest retention because they are needed for compliance and incident investigation.

Default Retention Periods

Data Type Default Minimum Maximum
Execution metadata 90 days 30 days 365 days
Execution payloads 30 days 7 days 90 days
API call metadata 90 days 30 days 365 days
API call payloads 30 days 7 days 90 days
Dead letter queue 30 days 7 days 90 days
Alert history 180 days 30 days 365 days
Audit logs 365 days 90 days 730 days

Configuring Retention

Enterprise Admin

Enterprise Admins can adjust retention periods within the ranges set by the Platform Admin:

  1. Navigate to SETTINGS > Enterprise Settings
  2. Scroll to the Data Retention section
  3. Adjust the sliders or enter values for each data type
  4. Click Save

Reducing Retention

When you reduce a retention period, data older than the new limit will be permanently deleted during the next cleanup cycle (runs daily). This action cannot be undone.

Platform Admin

Platform Admins set the allowable ranges for retention:

  • Minimum retention: The shortest period any enterprise can set (protects against accidental data loss)
  • Maximum retention: The longest period any enterprise can set (controls storage costs)
  • Default retention: The period applied to new enterprises

How Cleanup Works

DIBOP runs a daily cleanup job (typically at 02:00 UTC) that:

  1. Identifies data older than the configured retention period
  2. Deletes the data in batches to avoid performance impact
  3. Logs the cleanup in the audit trail (e.g., "Deleted 1,234 execution logs older than 90 days")

The cleanup job processes each data type independently. If one type has a 30-day retention and another has 90 days, each is cleaned up according to its own schedule.

Compliance Considerations

Regulatory Requirements

Some industries require specific retention periods:

Regulation Typical Requirement
GDPR (EU) Personal data should not be retained longer than necessary for its purpose
SOX (US) Financial records must be retained for 7 years
HIPAA (US) Health information records must be retained for 6 years
PCI DSS Audit trail history must be retained for at least 1 year

Consult Your Compliance Team

These are general guidelines. Consult your legal and compliance team to determine the appropriate retention periods for your enterprise.

PII in Logs

Execution and API call logs may contain PII (personally identifiable information). DIBOP:

  • Automatically redacts PII-classified fields in log displays
  • Stores the full data (for replay/debugging) but restricts access to authorised users
  • Deletes all data (including PII) when the retention period expires

Shorter payload retention periods reduce the window during which PII is stored.

Export Before Expiry

If you need data beyond the retention period (for compliance, auditing, or analysis):

  1. Set up a regular export schedule
  2. Export data before it reaches the retention limit
  3. Store exports in your own archival system with appropriate access controls

See the export features in Execution Log and API Call Log.

Dead Letter Queue

The dead letter queue deserves special attention. Items in the dead letter queue represent:

  • Orchestration executions that failed after all retry attempts
  • Data that could not be delivered to a target system
  • Records that failed canonical validation

Reviewing Dead Letters

  1. Navigate to MONITOR > Dead Letter Queue (or the relevant section in Observability)
  2. Review each item to understand why it failed
  3. Choose an action:
    • Retry: Attempt to process the item again
    • Dismiss: Remove the item from the queue (it will be deleted at retention expiry)
    • Export: Download the item for manual processing

Dead Letter Retention

Dead letters are retained for the configured period (default 30 days). After that, they are permanently deleted, whether or not they have been reviewed.

Review Regularly

Check the dead letter queue at least weekly. Items that sit too long may become stale (e.g., the source data may have changed).

Storage Impact

Longer retention periods consume more storage. Approximate storage consumption:

Data Type Approximate Size per Record
Execution metadata ~2 KB
Execution payload ~10-100 KB (varies by orchestration complexity)
API call metadata ~1 KB
API call payload ~5-50 KB (varies by response size)
Dead letter item ~10-100 KB
Audit log entry ~0.5 KB

For an enterprise running 1,000 orchestration executions per day with 5 steps each (5,000 API calls), approximate daily storage:

  • Execution metadata: ~2 MB/day
  • Execution payloads: ~50 MB/day
  • API call metadata: ~5 MB/day
  • API call payloads: ~125 MB/day

At 90-day retention, this totals approximately 16 GB. Your platform administrator can advise on storage limits for your subscription tier.

Next Steps