Data Migration: Avoiding the Pitfalls
Gartner recently stated that 83% of data migrations either fail outright or exceed their allotted budgets and implementation schedules.
Transformation programmes, usually driven by an ambitious business change initiative, are a huge investment of time, resources, and money for organisations. Delay or even failure can be very expensive financially and can adversely impact the business growth strategy. Most transformation programmes experience delay and some delays are very lengthy and costly; data migration is usually at the top of the list of the reasons why and often some programme management team members are replaced as a result.
Often overlooked, data migration is a complex and specialist activity, requiring a specific skillset, in-depth experience, a best practice methodology and a proven toolset to deliver and end-to-end solution.
Common Problems & How To Avoid Them
Start Early
Don’t leave data migration until the last minute. It is a key priority for success and is very often overlooked until it’s far too late to turn around, thereby delaying the published go-live date. Aim to start the data migration activities as soon as possible, as a rule, it should be ramping-up during the early design phase and in full-flow by detailed design. Follow a structured, proven data migration methodology to form the basis of a detailed delivery plan, it will help ensure the right tasks are completed in the correct sequence and fundamental activities are not missed.
Use A Data Platform
A data migration platform is essential for a successful programme delivery. Don’t build it yourself, it is tempting and may excite some people in IT, however it will take longer, cost more money, and ultimately be thrown away after go-live and will be a huge distraction from delivering the programme itself. Select a platform that has been specially designed for data migration, rather than using a data reporting/query or system integration toolset, that maybe already licensed. A proven specialist data migration software solution will save time and, in the long run, prove to be a very cost-effective investment. Take advantage of the software-as-a-service subscription models available today, pay for what you need, only as long as you need it.
Data Quality
Don’t just assume that the data quality is okay. High quality and accurate data are fundamental to a successful implementation, bad data is unlikely to even load into the new application. Poor quality data leaves key stakeholders disappointed and underwhelmed and sometimes even leads them to question the value of the whole transformation. Data migration is the opportunity to completely transform your data into a highly accurate, consistent and fit-for-purpose data asset.
The data quality capability should enable both simple and complex business data quality rules to be defined, grouped, reported, and tracked over time. It should include dashboards to contextualise information, to enable senior management to focus key priorities and detailed error reports/listings, for operational resources to deliver data cleansing. Automated and regular reporting is required to track progress, view trends, and drive the data cleanse initiative through to a successful go-live.
The Data Team
Don’t use inexperienced resources. Data migration is complex, with many moving, interrelated parts; it can’t be delivered on time and on budget without specialist experience. A strong and multi-disciplined data team, combining technical expertise and business knowledge, is essential to successful data migration delivery. Create a dedicated data team responsible for all migration activities with an experienced data lead at the helm.
The data lead should report directly into the programme manager/director, rather than be tagged-on to a particular workstream. Data requires a high level of focus, governance and control and impacts the entire programme. Avoid having data team members embedded in other workstreams or BAU, they are usually given non data migration work and data takes a back-seat.
Clear Task Ownership
Don’t assume all of the data migration activities are being handled by an implementation partner. Ensure every activity is clearly owned and there are no gaps, especially data transformation, data quality and cleansing. Delays are common due to confusion and assumptions on the responsibilities and ownership of the activities across third parties, the business and internal technical teams.
As soon as possible ensure a full ownership matrix is agreed across all activities, it will save time and avoid gaps and subsequent delays. Commercial contracts and SoW’s should include all of these.
Some unscrupulous third parties may even prefer the migration ownership to be ambiguous to buy themselves time and have a ‘get-out-of-jail card’ when things do overrun and to avoid any commercial penalties.
Data Security
Don’t weaken the data security protocols. Moving high volumes of data off the corporate applications can weaken the sophisticated security and protection that is provided by the applications themselves. For a controlled and secure process, the data migration platform must be the central repository of all data migration activity.
A data platform must have the capability to tightly control user access; allowing specific authorisation to limited datasets and limiting authorisation to only those that require access. The flow of data must be encrypted, and enable data to segregate into clearly defined groups, usually by business area, to control who has access to what rather than a blanket authorisation to all.
Data Scope
Don’t just migrate everything. Often the data migration scope decision is that everything should be migrated. It may appear the easy option, however, the greater the amount of data that is migrated, the more data there is to cleanse, test, load, reconcile and sign-off. Aim to only migrate what is necessary from a legal, fiscal, or critical business operations perspective, the rest can be archived or placed in BI/DW.
Knowing what is being migrated and what is not must be established as early as possible, to avoid the emergence of extremely late requirements or having expensive work completed on unnecessary activities. Ensure the data migration scope is fully agreed, clearly documented and signed-off as early as possible, ideally during the design phase. Data migration is often left out of the formal programme change process, but formal change control is essential. The data team must be involved in solution design change decisions, to assess the impact of any change on data migration scope and activities.
Common Oversights
Don’t forget documents, such as PDF’ and Word documents, they are usually very high volume and complex to migrate. Some document types are closely related to the structured data and without them the business process could fail.
Don’t break the interfaces. Both internally, across the integrated application landscape, and with third parties, where they provide a service that depends on data exchange. Third party data may need to be updated and aligned with new number ranges, codes, data formats and lengths that are all compatible, so ensure you include third parties in testing.
Avoid manual activities wherever possible as these often cause single points of failure. Aim to automate the full data migration process, as the failure of a single step could derail the entire migration. Check the full data migration process can run within the go-live window, typically over a weekend, from freeze to business sign-off.
Don’t skip the testing phases for data migration. Include data migration as part of system testing and in all formal test phases. This will flush out unexpected problems and allow time for investigation and correction. It will also prove the new application actually works, across a multitude of scenarios, rather than just data manufactured to pass the test.
