The Insider’s Guide to Data Archiving

Vinay Sail

Share on twitter
Share on linkedin
Share on telegram
Share on whatsapp
Share on email

Data Archiving – Definition

Data archiving is the process of retaining data for long-term storageThe data might not be in use; however, it can be brought into use and can be stored for future purposes. An archive is a collection of historical records that are kept for long-term retention and used for future reference. Typically, archives contain data that is not actively used. 

Backup Definition:

A backup is a copy of data that can be used to restore the original in the event that your data is lost or damaged. 

If a company experiences data loss due to hardware failure, human error, or natural disaster, a backup can quickly restore that data. 

Data Purging – Definition

Data purging is a term that is commonly used to describe methods that permanently erase and remove data from a storage space. 

There are many different strategies and techniques for data purging, which is often contrasted with data deletion. Deletion is often seen as a temporary preference, whereas purging removes the data permanently and opens up memory or storage space for other uses

Why is Data Archiving Important?

  • Archiving frees up resources
  • Archiving is more economical
  • Meeting legal and compliance requirements
  • Archive storage also reduces the volume of data that must be backed up
  • Improving user experience by displaying only relevant data
 

Data Archiving Strategy

  • Inventorying and determining which data must be archived
  • Based on compliance regulations, assign a retention schedule for each category
  • Choosing a data archive product
  • Proactive protection of the data archive’s integrity
  • Develop an all-inclusive archive policy
 

Question to be asked while designing archiving policy

  • What is the frequency of the data archiving and it’ effect on business operations?
  • Is there any necessity for the archive data to be used for reference or reporting etc.?
  • What are the implications of regulatory and compliance on the data archiving business requirements?
  • Is the location of your archive in the cloud or on-site?
  • What format is your data in?
 
Soft vs Hard Deletion

Soft Delete

  • Records are flagged as deleted and visible through the Recycle Bin.
  • When data is soft deleted, it still affects database performance because it’s still living in the org, and deleted records have to be excluded from any queries.
  • The data stays in the Recycle Bin for 15 days or until the Recycle Bin grows to a specific size.
  • The data is then physically deleted from the database once time or size limits are reached or when the  Recycle Bin is emptied using the UI, the API, or Apex.
 
Hard Delete
  • Bulk API supports a hard delete (physical delete) option, which allows records to bypass the Recycle Bin and immediately become available for deletion.
  • Using Bulk API’s hard delete function is recommended for deleting large data volumes to free up space sooner and keep extraneous material from affecting performance.
  • Note that the hard delete option is disabled by default and must be enabled by an administrator.

 

Field Values – Archiving Policy (1)

 
Field History
  • You can select certain fields to track and display the field history in the History related list of an object.
  • Field history data is retained for up to 18 months through your org and up to 24 months via the API.
  • Use Data Loader or the queryAll() API to retrieve field history that is from 18 to 24 months old.
 
Field Audit Trail
  • Field Audit Trail (Part of Platform Shield) lets you define a policy to retain archived field history data up to 10 years from the time the data was archived. This feature helps you comply with industry regulations related to audit capability and data retention.
  • With Field Audit Trail, you can track up to 60 fields per object. Without it, you can track only 20 fields per object.
 

Field Values – Archiving Policy (2)

What if Field Tracking needed to track more than 60 fields

Automation of archiving and purging for a large number of records

  • Data Loader CLI
  • ETL – Extract, Transform & Load

Automation of Aggregating large number of records

  • Identify records and aggregate data into a custom object
  • Once aggregation is completed, purge data from original std. or custom object

Reporting Snapshots – A way to summarised data instead of storing all the data

A reporting snapshot lets you report on historical data. Authorized users can save tabular or summary report results to fields on a custom object, then map those fields to corresponding fields on a target object. They can then schedule when to run the report to load the custom object’s fields with the report’s data. Reporting snapshots enable you to work with report data similarly to how you work with other records in Salesforce.

e.g.

  • Schedule reports daily
  • Summarized data can be loaded into a custom object
  • Archive data from the custom or std object to make way for deletion in future

Salesforce Connect : External Object

  • Salesforce Connect maps Salesforce external objects to data tables in external systems.
  • Instead of copying the data into your org, Salesforce Connect accesses the data on-demand and in real-time.
  • Use cases: Archive cases\orders can be view using Salesforce connect from external data storage, but the latest cases\orders can be view directly from Salesforce.

Using Heroku + Heroku External Object (To view archived data from Salesforce)

Using BIG Objects + Custom LWC or Aura Component to view Archived Data

Appexchange

Share on twitter
Share on linkedin
Share on telegram
Share on whatsapp
Share on email

Subscribe to our newsletter

Don't miss new updates on your email