DESIGN AUTOMATED FRAMEWORK FOR SEAMLESS MIGRATION OF LARGE RELATIONAL DATABASES INTO SNOWFLAKE

Abstract

Migrating large relational databases to cloud platforms has become essential for organizations seeking enhanced scalability, cost efficiency, and performance. This paper presents an automated framework designed to seamlessly migrate large relational databases into Snowflake, addressing challenges such as migration time, data integrity, and error handling. The framework automates data extraction, transformation, and loading (ETL) processes, while integrating robust error detection and data validation mechanisms. Performance evaluation across databases ranging from 100 GB to 500 GB demonstrates consistent data transfer speeds, with an average of 72.11 GB/hr, and minimal error rates between 0.02% and 0.08%. The results confirm that the proposed framework is capable of preserving data integrity with zero data loss and offers cost efficiency, with storage costs as low as $5 per month for a 100 GB database. The automated migration framework, thus, provides a scalable, reliable, and cost-effective solution for large-scale database migration to Snowflake.