Product Excellence Through Modern PLM in Fast-Paced Markets
Modern PLM systems empower businesses to achieve product excellence in fast-paced markets by enhancing collaboration, agility and innovation.
Automation in data migrations is no longer just a hype. Automation is a practical solution to speed up and optimize the entire data migration process, whilst at the same time eliminating human error and improving quality.
In previous blog posts, we compared data migrations with blood transfusions and used this analogy to illustrate the complexities of data migration and the importance of standardization. In this blog post, we will explore the pinnacle of data migration: Automation. (We promise to make it less bloody compared to the previous blog posts 😊).
In the medical world, automation is slowly being introduced. In the field of blood transfusions, likened to data migrations in the blog post, automation is still in its infancy. However, it has already brought advantages. Notably, the applied (semi) automation has promoted further standardization, increased safety, and reduced turn-around times.
Likewise, in the world of data migration, automation is not yet widely integrated into the process. However, it promises, just like with blood transfusions, to vastly reduce human errors, consistently apply best practice standards, and significantly lessen the turn-around time for building and configuring a migration engine.
Data preparation stands out as the toughest and most complex aspect of data migration. Although certain SAP objects transition smoothly from source to target with minimal alteration, many others require transformations that can be anywhere from straightforward to very difficult.
Complex migrations call for specialized tools to be used. Traditionally, the more robust data migration solutions use ETL tools (Extract-Transform-Load) such as SAP Data Services. Such tools allow a developer to create a series of repeatable steps through which the data is extracted, transformed, and loaded. Once the tools are fully prepared and the steps are extensively tested, it can be applied to load the data into the target system. Reports can be applied to ensure the quality of migration.
Although SAP Data Services and other tools are great for data migration, they’re not fully automated and need substantial preparation. Developers must create all the necessary steps, leading to potential human errors that take extra time to detect and fix.
Acknowledging the constraints of SAP Data Services and similar tools, we at Camelot have begun enhancing and fully automating our Migration Engine to address these limitations, minimize human involvement, and prevent errors. Internally referred to as Hyper Automated Migration (HAM), this approach is aimed at fully automating the entire process.
Thanks to our data migration framework we have a solid and proven approach to data migrations. Using the highly standardized migration approach in combination has enabled us to, step by step, automate parts of the migration engine’s processes and apply this at various customers with great success, vastly reducing effort and cost.
The transition began with automating data extraction, the initial stage of the data migration engine. Previously, using traditional methods, this process could take up to three weeks, depending on the number of tables and systems involved. Now, with automation, all that’s needed is a list of tables, and within seconds, the data extraction steps are programmed and ready for use.
Similarly, the pre-load validation stage ensures data readiness before loading. It includes checks for target system compatibility, mandatory field completion, data formats, tax numbers, bank details, postal codes, and custom criteria.
Creating this process in tools like SAP Data Services can take weeks to complete and is extremely labor-intensive. Once completed, it often requires many hours to execute. Automating this process reduces the initial build time to fine-tuning and adding custom rules only, significantly cutting down on traditional development time while improving performance and minimizing human error.
Just like the two examples, we at Camelot have automated various other parts of the migration engine and will continue to do so.
When discussing automation in data migration, it’s essential to recognize the various flavors or levels of automation that exist. While tools like SAP Data Services and others do automate certain aspects of the data migration process, they often require programming or configuration by developers, whether it’s through code-based or no-code approaches. However, this level of automation may not be considered “true” automation.
True automation in data migration goes beyond simple tool-based automation. It involves a solution that requires minimal human intervention, with near-zero development work needed. In this ideal scenario, the automation platform possesses the intelligence to generate migration logic autonomously, based on the functional and technical specifications provided by domain experts.
This is the cutting-edge territory where Camelot is making rapid strides.
Regardless of whether the data migration has been automated, it is imperative to uphold the utmost quality and precision throughout every phase of the migration process. This goes not only for data consistency between extracting, scoping, transformations, etc., but also for pre-load data validations, post-load comparison, and post-load reconciliation.
By implementing automated reporting on these phases and basing it on the unified data model, we guarantee that the data feeding our reports is not only based on best practices and consistency but also achieves near real-time updates. This ensures the delivery of accurate and invaluable insights into the migration processes.
However, the journey doesn’t stop there. In data migrations, multiple mock loads are necessary to verify that the data transformations yield the intended outcomes. Traditionally, this process is manual and, apart from being confusing due to complex transformation rules, it’s incredibly labor-intensive. That’s why Camelot has pioneered automated testing through Robotic Process Automation. Dive deeper into this blog post to discover how this tool not only saves costs but also minimizes errors.
Please feel free to connect with us if you are looking at an SAP transformation solution and wish to learn more about how Camelot can help you in your journey.
Data migration is one of the key competencies of the Camelot Data Tower.
Camelot’s Data Tower Concept unifies key competencies in Data Governance, Data Migration, Data Quality Management, Data Design, and Data Advisory to offer a comprehensive data management solution. Scalable for businesses of all sizes, this platform provides efficient and secure solutions. It identifies areas for improvement and potential risks, enhances data governance models, improves data quality and maturity, and prepares organizations for a seamless transformation to S/4HANA. With Camelot’s expertise, businesses can confidently navigate their data transformation journey, achieving their strategic goals more effectively.
Modern PLM systems empower businesses to achieve product excellence in fast-paced markets by enhancing collaboration, agility and innovation.
Read how the Campaign Planner & Designer (CPD) helps you to manage supply chain variability.
Explore automated production planning with our Campaign Planner & Designer.
The pharma market trends impact logistics with a shift to smaller, higher-value shipments and new temperature requirements.
© Camelot Management Consultants, Part of Accenture
