Factors affecting data migration
The data migration process is an issue that causes many problems for organisations already at the project preparation stage. Our experience allows us to identify a list of critical factors influencing the proper conduct of migration.
Determining the scope of data – while indicating the areas of enterprise activity that are subject to migration seems to be a simple task, we must remember that it is only a detailed analysis of the source systems and the data contained therein that allows determining in detail which categories of data and to what extent will be migrated.
Identification of data sources – the most common problem in migration projects is lack of knowledge about all data sources. The biggest impact on this is the so-called “wild IT”, i.e. a situation where part of the data that should be in the target system is stored in local unregistered databases or datasheets.
Ignorance of data structures – a consequence of the lack of full knowledge of data sources and scope is the difficulty in technically specifying the source and target data structures. As a result, the development of a technical migration plan becomes complicated, as the processing algorithms will be subject to change every time data structures change.
Developing processing algorithms – in large projects, where data from multi-branch enterprises, from many different sources and types of systems are subject to migration, it is extremely complicated to design data processing algorithms so that they can be imported flawlessly into target structures. An additional problem is the lack of knowledge of technical conditions of the target system, i.e. import structures and system requirements as to the proper quality of imported data.
Poor data quality – we usually learn about the poor quality of data already during their processing, when it turns out that some of the data is missing, there are duplicates or it is not possible to transfer data directly (e.g. data formats, their length do not match, etc.) In such situations, migration tools must provide functionalities that will allow you to adjust the quality of the data to an acceptable level.
Impact on the organisation’s current operations – data migration projects in key systems cannot stop or significantly hinder the organisation’s operations. Therefore, it is very important that the migration process has no impact on the current operations of the organisation and that the production migration itself takes place quickly and flawlessly.
The catalogue of factors influencing data migration projects is never complete. Unforeseen situations always arise, so it is crucial to choose a migration environment and processing methodology that responds quickly and effectively to threats and errors.
Data Migration Platform
Based on the experience gained in many data migration projects to SAP systems, SI-Consulting has developed a proprietary solution to support the data migration process in the area of data cleaning, consolidation and transformation. Starting from the assumption that the biggest problem in this type of projects is the lack of full knowledge about the structure and quality of data and the need for continuous adaptation of processing algorithms, a tool has been built that allows for flexible adaptation of the processing. Each of the elements of this process, i.e. source data, control data or processing rules, can be adjusted continuously and repeatedly, thus improving the data migration algorithm in a flexible and secure way.
Data Transformation based on the Data Migration Platform is a service dedicated to organisations planning data migration in an environment where the data processing model is subject to continuous modification, and multiple adjustments of data processing algorithms are required.
Data
The migration process uses different types of control data, e.g. mapping data, validation data, in addition to the data to be processed. At the same time, each data migration project defines a unique data processing model.
A unique feature of the Data Migration Platform is the free and unrestricted access to all data at every stage of processing and from the level of each module processing this data.
This means that regardless of the type of data such as source data, control data, output data, or subsequent versions of processed data, they are freely available in the system and can be used as any type of input data and in any stream of processed data. Therefore, it is possible, e.g. to download the input data, process them in any way and use them as mapping data. Importantly, this data is not subject to time or process limitations: data downloaded or generated today in the material index data migration process can be used as validation or mapping data in the order migration process six months later.
Also, in the process of consolidation or division of records, PMD allows flexible access to data. The data version resulting from mapping the data in step three can be used as control data in the consolidation process in the next step or later. PMD also does not restrict access to data in terms of the number of data sources. If a processing rule requires simultaneous access to data from an external file, data from another processing stream and mapping data provided by the user, then this is a feasible solution.
Processing data and processing control data are available for all processing rules and processes in an unlimited way. This allows us to build any data migration model.
Data processing – iterative processing model
Processing large amounts of data often requires multiple repetitions and modifications of the same data. In each subsequent iteration, based on the experience gained, erroneous data is eliminated, the configuration is adjusted to the new requirements and the processing algorithms are corrected to obtain results with a higher data accuracy rate.
Processing operation
The data processing is based on the use of Data Packets in combination with Procedure Packages. Data packets can contain data from multiple sources simultaneously, can be modified and used repeatedly. The Procedures Packages are sets of many types of Rules. It is possible to use standard rules (validation, mapping, mass operations), which are subject to configuration, as well as non-standard rules. The non-standard rule can be based on ABAP code, which allows for programming any processing logic. An example of such a rule is the data consolidation module, where the processing is delegated to an external report, in which the algorithms of processing according to the requirements in a given business environment are embedded.
The Rule Repository allows implementing any data processing algorithm based on configurable rules and rules based on ABAP code.
The Rule Repository module is used to manage rules, where they can be created and edited. Rules whose operation requires external data (e.g. mapping data), allow keeping the rule model, i.e. multiple versions of one rule configuration. This allows for multiple data processing using different configuration data. Repeated processing is possible thanks to a mechanism of data variant reversal, i.e. reversing processing back to the indicated processing moment. After the data has been processed by the rule, it is possible to launch the Verification Procedures Package in order to control the correctness of processing on an ongoing basis.
Processing Monitor
Migration of sensitive data always carries the risk of error. During the migration of large amounts of data, it is required to provide tools to report all events and changes taking place during the migration process.
The Data Migration Platform fully supports the user in this area. The Processing Monitor is a group of processes operating in the background, which at each stage of processing collect information about the operations performed and protocols of their execution. Calling every rule and every change of data causes this fact to be recorded in the processing logs. At the same time, these processes ensure that every version of processing and control data is stored. The Processing Monitor also assists the user in the preparation of subsequent iterations of data processing.
Using the collected data, the Processing Monitor enables to:
- start tracking changes for the indicated record,
- display of consolidated and split records,
- display of processing history also for consolidated/distributed objects together with their histories,
- display the history of record attribute changes backwards to the point where the source record is found,
- display of log contents in individual modules (according to context),
- output Procedure protocols (single, collective, etc.) to screen and file.
The reporting function of the Processing Monitor can be activated at any stage of processing and for any data indicated.
Reporting
PMD provides a set of reports supporting the work with the system and being supporting functionalities for the Audit Console.
- Data Flow Report provides full information about the data flow from the data source to the export data.
- Data Preview and Comparison Report provides an opportunity to view and compare data in the processing, specifying the modified data and accessing the processing logs of the smallest data unit.
- General Log View Report – an overview of the log of general activities and events: type of deleting data, creating objects, deleting objects.
- Quality and Quantity Report generates statements for selected PMD system objects, data on the quantity and quality of processed records.
Data Migration Platform in practice
Data Migration Platform was used as part of a data migration project carried out for one of our Clients.
Migration in numbers based on statistics generated by audit functionality:
The most important benefits of PMD:
- Automated consolidation of suppliers and consolidation of customers
Approximately 400,000 suppliers and customers are processed.
After consolidation, 230,000 suppliers and customers were loaded into SAP. - The automated operation for the unification, clearing and combination of fixed assets:
Less than 500,000 fixed assets were processed.
After processing, 380,000 fixed assets were loaded. - Automated consolidation of material indices.
Over 300,000 material indices from 13 branches have been processed.
Less than 26,000 material indices were loaded after processing. - Unification of the structure of capital expenditure
Over 80,000 investment projects were processed from less than 600,000 individual items. - Migration time reduction
Productive migration time has been reduced by half, depending on the area, from 2-5 days to 1-3 days of unavailability of production systems. - Authorisation and audit log
Guarantee of proper authorisation of access to migrated data
Logging of any changes made to the data and the possibility of tracing each clearing step, including manual data corrections - particularly important for financial data - ERROR-FREE productive migration
All productive data has been loaded correctly – 100%. - Reducing the necessary involvement of employees
The number of people dedicated to 100% migration on the part of the ordering party was reduced by 40% from the planned 40 to 25 people.
Due to the availability of validation tools and reports, the possibility of parallel work and manual data correction, PMD reduced the time and workload of data analysis by the Client’s employees by further 30%.
Estimated total reduction of labour costs of the Client’s employees during migration: about 35% while ensuring quality, security and transparency of the entire migration process at a level not comparable to the traditional approach to migration based on cleaning up data in files.
Data Migration Platform provides unique functionality that is not available in other data processing support products. At the same time, as a platform open to changes, it enables easy extension of available functionalities. The Rule-based solution, as independently programmable modules, allows you to design and execute all required transformations within the Platform, without the need for additional processing in external environments. In practice, this makes it possible to fully report changes in data throughout the processing (e.g. for audit purposes), which is not possible for partial processing outside the migration environment. Full knowledge of the environment also allows to introduce changes or new functionalities in consultation with the Client. If necessary, standard solutions can be modified to fully adapt the tool to specific requirements.
PMD is a tool prepared to support data cleaning, consolidation and transformation in S/4 HANA data migration projects as part of greenfield implementations of these systems.
