Moving data from one software system to another is always challenging—but when it comes to content management data migration the fact that you’re moving tens to hundreds of millions of records across systems makes this kind of migration especially complex. With complexity comes problems—which usually cause lengthy delays in the migration process and massive headaches for you and your team.
To help avoid these problems during your own migration, we’ve identified three of the most common data migration challenges you could potentially face.
Challenge 1: Source Data Complexity
Due to the sheer volume of records you need to move, you may run into various issues with your source data. It’s never as simple as moving information from content management system A to system B, particularly when you consider you’ll need to do some analysis and configuration to work through any source data complexity. Here are a few elements that add to this complexity, and techniques to solve these challenges:
1) Data transformations: Every mainframe stores data in a certain way—and if you’re using an outdated data storage format, you may need to transform the data if you want to store it in a modern database (like Oracle or SQL Server). To learn more about three specific areas to consider regarding your data transformations, take a look at this article.
2) Codified fields: Storing 30- 40-digit claim numbers in a single field may be common practice in your current content management system, but if you’re migrating to a new system, it may be helpful to break these types of codified fields up so their easier to use. By doing so, you’ll be able to quickly see the relevant information (like the sequence number, member identifier, date, etc.) parsed out instead of having to look at one lengthy number and decode that information in your head.
3) Data normalization: If you’ve used the same content management system for many years, your organization may have the same piece of data stored in a number of different places. Going through the process of finding all the places where one piece of data is stored and ensuring it’s only stored once—in the correct place—in the new content management system is known as data normalization.
Challenge 2: Data Loss Or Corruption
Let’s face it: Any data loss or corruption can be a major problem. Your organization could be harmed if you lose even one record! Here are two strategies to help avoid any unexpected issues with your data migration that could result in data loss or data corruption:
1) Reconcile your accounts as you do migration and testing. You should know before the data migration process begins how many records are brought in and how many records you should produce in the new system (keeping in mind it’s not always 1:1, as some records in your current content management system could be duplicates). If the output doesn’t match your expected number, you’ll need to do some research to find out why.
2) Use tools to help validate the migrated data. As you move data from one content management system to the next, you need to be certain that the documents you’ve migrated into the new system match up with your expectations. For example, are states ending up in state fields? Are the right number of characters in each claim number field? There are a number of commercial utilities and services available that can assist in monitoring the veracity of each field of data.
Challenge 3: Thoroughly Testing & Validating Data
Rarely do you find an IT team that can afford testing and quality assurance, but content management migration demands it. With content management data migrations running $500,000 or greater, the cost of failure is simply too high. Here are some data validation and testing techniques we recommend:
1) Consider any data events that could have impacted data quality. For example, if you remember that your content management system experienced a glitch in March 2013, make note of it and test all the 3/13 records separately to ensure data quality.
2) Test a large volume of data for quality assurance. We recommend pulling at least 10-20% of your data to ensure you’re covering a wide breadth. If you’re moving data spanning several decades, consider pulling around 5% from each year in addition to your profile sample data.
3) Test early and test often. As soon as you’re able to start testing (even if it’s just with a subset of the configuration and code you’re going to use to migrate), do so. Continue to run tests during your entire data migration. Testing should be a continuous process, not something that begins once the last document has been migrated over to your new content management system.
Data migration is a big undertaking, so start thinking about these challenges—and the solutions you’ll use to address them—early on!
We also recommend getting technical team members involved at the outset so they can contribute to your risk analysis. If you’re able, it’s certainly worth it to partner with a team of experts who are familiar with data migration and can assist you in the process. Best of luck!
