Migration Reconciliation Processing and Validation Reporting


File Migration Reconciliation Challenges

The highest hurdle at the end of any migration is addressing any file migration reconciliation challenges. This includes identifying and handling content that has not yet successfully migrated and determining when the migration can be considered “done”. The process for reconciliation and validation needs to be rock solid to provide the migration team, project sponsors, and business users with the confidence that the migration was successful.

Defining Migration Success

What does migration success look like?

One must define migration expectations before the migration ever begins. Many organizations will say that all business data is 100% mission-critical and that only a perfect migration is acceptable. Unfortunately, this is often an unrealistic expectation. While perfect migrations do happen, they are the exception and not the rule.

Disparate Storage Platforms

There are many factors that affect the migration success rate. Platform compatibility between source and destination alone might make it impossible to achieve 100% success. For example, files that are very large on the source system might be too big for the destination system. File paths that are very long in the source system might be too long for the destination system. These differences in technical paradigms from one platform to another can sometimes cause content to be incompatible for one reason or another.

What items can’t be migrated?

It is also common for files to be “corrupt” in the source system. Zero-byte files find their way into systems all the time. Failed uploads or imports are often the culprits. There’s a written record, but there is no binary data to go with it. These items cannot be migrated. Corrupt items are easily proven as one can often trace the error in the migration back to an exception that also happens when a user tries to manually open the file on the source system.

Sufficed to say, systems are not perfect. Therefore, migrations often are not perfect. But they can be “perfected”. In the sense that everything can be accounted for, a migration can be brought to a state of completeness. This is not an effortless process, particularly when data continues to change. The key to defining success is to run a pilot migration to identify where the land mines are going to be. Then you can create a plan to either have end users manually remediate content that cannot be migrated or to leave behind content that is either corrupted or not mission-critical.

Migration Status Detail Can Be Confusing

Every migration solution worth anything includes some form of status as to the success or failure of each processed item. While migration teams come to understand these details and they are critical during the remediation process, they can sometimes be confusing to end-users. Consider the following example of migration status:

As a migration engineer, the status counts above show that the migration process did everything possible to execute a flawless migration. It shows that 883 items migrated perfectly while 6 items needed to be automatically remediated in some way to get them to migrate successfully. It also showed 72 completely ignored items.

However, as an end-user, this may be confusing. What exactly does “revised” mean? Did those items migrate successfully? What about the ignored items? As an end-user, all I know is that I have a certain number of documents in my home drive in the old system and a certain number of files in the user drive in the new system. But when those numbers do not add up, I get very confused!

Validation Reporting Makes Things Easy

Every migration should include validation reporting. It should be an accounting of content on the source system as well as content on the destination system. The goal is to identify all items in both systems and account for them, even if they are not all migrated. If done properly, there should be a “bottom line” comparison that factors in things like ignored items or items that already existed in the destination system in order to come up with matching numbers that prove the migration was successful.

Dryv includes a Validation Reporting area for every job that does exactly this. Here is an example of validation reporting from a Dryv job:

Continuing from the example above, the end-user might notice that they have 961 items in their source Box account. But after the migration, they only have 889 items in their destination OneDrive for Business account. This often causes confusion and doubt that the migration process cannot be trusted.

They then contact the migration team who provides the user with this report, showing that the filtering rule ignored 72 items. Perhaps they were video files or other unnecessary file types, for instance. Then the user remembers that they uploaded their vacation movies to their home drive years ago and it all makes sense now. The important concept in the image above is that the accounted for numbers in the example above actually match up! When we subtract the number of ignored items from the identified items in the source, the 889 item count matches the number of items in the destination!

Compliance and User Confidence

Ultimately, any successful migration will accomplish the goal of replicating content from one endpoint to another. But that is only half the battle. You must be able to prove this success through validation reporting to address any file migration reconciliation challenges. Validation reports provide the evidence needed to prove to auditors that the migration was successful while also proving to end users that their data has been kept safe. Then, and only then, can the migration be considered “done.”

Russ Houberg