Data Lifecycle Management
Feature Overview
We designed this feature with the understanding that data within an enterprise integration environment is not a static asset, but rather follows a complete cycle from generation, processing, consumption, to eventual retirement. Qeasy Data Integration Platform (hereinafter referred to as DataHub) abstracts the data lifecycle into four phases: Ingestion, Active, Archive, and Deletion, providing observable and controllable capabilities at each stage.
During the Ingestion phase, DataHub marks data sources and performs initial quality checks. During the Active phase, the platform ensures high-frequency access performance and change tracking. During the Archive phase, data is migrated to low-cost storage while remaining searchable. During the Deletion phase, the platform securely removes data and its derived copies according to predefined policies. The entire lifecycle is driven by a unified metadata catalog, ensuring that state changes are fully traceable.
A typical use case is when enterprise order data must be kept for three years to satisfy audit requirements, yet only the first three months require high-frequency queries. DataHub can automatically archive order data after three months while retaining query access, satisfying compliance while controlling costs.
Use Cases
We have identified three typical scenarios:
Scenario One: Compliance Retention and Cost Balance. Data in the financial or healthcare industries often has mandatory retention periods, yet long-term online storage is expensive. Through DataHub's lifecycle policies, data that exceeds the active period can be automatically transferred to cold storage, meeting retention obligations while significantly reducing costs.
Scenario Two: Development and Production Environment Isolation. Test environments often accumulate large volumes of production data copies, which increase both leak risk and storage consumption over time. DataHub supports setting differentiated deletion times based on environment tags, ensuring timely cleanup of test data.
Scenario Three: Temporary Data Governance. Temporary data generated by marketing campaigns or seasonal business loses value once the campaign ends. DataHub can automatically execute archive or deletion actions based on time windows or event triggers, without requiring manual intervention.
Configuration Guide
We recommend completing the configuration in the following steps:
-
Define Lifecycle Stages: In the DataHub console, configure lifecycle stage division rules for the target data source, using creation time or last access time as the basis, for example.
-
Set Transition Policies: Configure trigger conditions and execution actions for transitions between stages. The system supports triggering policy execution based on time thresholds, data labels, or external events.
-
Specify Storage Targets: The archive phase requires specifying cold storage storage. DataHub supports connecting to multiple low-cost storage backends, with the data migration process being transparent to query interfaces.
-
Configure Deletion Rules: The deletion phase must clearly define the deletion scope, audit log retention duration, and whether multi-level approval is required. The platform enables dual confirmation by default to minimize operational risk.
-
Enable Metadata Recording: Enable the full-lifecycle metadata recording feature. All state changes will enter the audit log for post-hoc traceability and compliance evidence.
-
Execute and Monitor: After the policy takes effect, DataHub provides a visualization panel showing execution history, current data distribution, and the next scheduled execution time.
Important Notes
We have summarized the following key points from multiple customer implementations that require special attention:
First, once lifecycle policies are enabled, they will be executed in bulk on existing data according to established rules. We recommend first validating the accuracy of transition conditions and execution actions in a test environment to avoid production data being accidentally archived or deleted.
Second, although archived data retains searchability, query performance is typically lower than that of active data. If business operations are sensitive to response time, a buffer period or fast recall channel should be reserved in the policy.
Third, deletion operations are irreversible. Even with audit logs enabled, deleted data itself cannot be recovered. For core business data, we recommend adding an approval step or delayed execution window before deletion.
Fourth, cross-region or cross-account data replication copies must be included in unified lifecycle management. DataHub supports synchronizing policies by replica tags, but the tag system needs to be planned in advance to avoid omissions.
Fifth, compliance requirements will evolve with regulatory policy changes. We recommend periodically reviewing lifecycle policies and retaining policy change records to address audit inquiries.
By properly leveraging DataHub's lifecycle management capabilities, enterprises can significantly optimize data storage costs while ensuring compliance and business continuity, thereby enhancing data governance maturity.