11. Data Migration and Modernization Strategies

Data Migration and Modernization Strategies¶

Section at a Glance¶

What you'll learn: - Evaluate migration strategies: Rehost (Lift & Shift), Replatform, and Refactor. - Select the appropriate Google Cloud tool for different data volumes and sources (STS, BQ DTS, Datastream, Transfer Appliance). - Implement Change Data Capture (CDC) patterns for zero-dovtime migrations. - Design "landing zone" architectures for raw data ingestion. - Assess the cost-performance trade-offs of online vs. offline migration.

Key terms: Replatform · Refactor · Change Data Capture (CDC) · Storage Transfer Service (STS) · Egress · Landing Zone

TL;DR: Data migration is the process of moving data from legacy or multi-cloud environments to GCP; modernization is the strategic upgrading of that data's architecture (e.g., moving from a relational VM to BigQuery) to unlock cloud-native scalability and analytics.

Overview¶

In the enterprise, data migration is rarely a "simple copy" operation. It is a high-stakes business maneuver driven by the need to reduce technical debt, eliminate data silos, and lower the Total Cost of Ownership (TCO). Organizations often face "gravity" issues—where massive datasets are tied to legacy on-premises hardware, making them difficult to move due to bandwidth constraints or the risk of operational downtime.

The primary challenge is not just moving bits, but moving value without breaking downstream dependencies. A "Lift and Shift" approach might be fast, but it often fails to realize the cost savings of the cloud. Conversely, a "Refactor" approach provides maximum value but requires significant engineering effort and testing.

This section provides the framework for choosing a strategy based on your business's risk tolerance, budget, and time-to-value requirements. We will bridge the gap between the technical mechanics of tools like Storage Transfer Service and the strategic necessity of modernizing data architectures for AI and advanced analytics.

Core Concepts¶

1. Migration Strategies (The 3 R's)¶

Rehost (Lift & Shift): Moving data and applications to the cloud with minimal changes (e.g., moving a SQL Server VM to Compute Engine). > ⚠️ Warning: While Rehosting is the fastest path to the cloud, it often preserves the same performance bottlenecks and high management overhead found on-premises, failing to leverage cloud-native elasticity.
Replatform (Lift and Reshape): Moving data to a managed service with minimal code changes (e.g., moving an on-premises PostgreSQL instance to Cloud SQL). This reduces operational burden without re-engineering the entire application.
Refactor (Re-architect): Completely redesigning the data architecture to use cloud-native capabilities (e.icale moving from a structured Oracle DB to BigQuery). 📌 Must Know: Refactoring is the highest-effort strategy but offers the highest ROI by enabling features like serverless scaling and decoupled storage/compute.

2. Data Movement Patterns¶

Batch Migration: Moving large blocks of historical data at scheduled intervals. Best for logs, backups, and historical archives.
Continuous/Streaming Migration (CDC): Using Change Data Capture to sync changes from a source database to a target in near real-time. This is essential for maintaining "zero-downtime" cutovers.

3. Migration Velocity vs. Volume¶

Online Migration: Utilizing existing network bandwidth (Internet or Interconnect) to stream data.
Offline Migration: Using physical hardware to transport data when network bandwidth is the bottleneck.

Architecture / How It Works¶

The following diagram illustrates a modern "Modernization" pipeline where data is moved from a multi-cloud environment (AWS S3) and an on-premises database into a unified BigQuery warehouse.

graph LR
    subgraph "Source Environments"
        AWS[AWS S3 Bucket]
        OnPrem[On-Prem Oracle DB]
    end

    subical "Ingestion Layer"
        STS[Storage Transfer Service]
        DS[Datastream - CDC]
    end

    subgraph "Google Cloud Platform"
        GCS[Cloud Storage - Landing Zone]
        DF[Dataflow - Transformation]
        BQ[BigQuery - Modernized Warehouse]
    end

    AWS --> STS
    STS --> GCS
    OnPrem --> DS
    DS --> GCS
    GCS --> DF
    DF --> BQ

AWS S3/On-Prem Oracle: The legacy data sources containing the enterprise's historical and operational data.
Storage Transfer Service (STS): An agentless service that pulls data from S3 into Google Cloud Storage.
Datastream: A serverless CDC service that captures real-time changes from the Oracle DB and streams them to GCS.
Cloud Storage (Landing Zone): Acts as a temporary, highly durable buffer for raw, un-transformed data.
Dataflow: Performs the "Refactor" step, cleaning, schema-mapping, and flattening the raw data.
BigQuery: The final destination where data is structured, partitioned, and ready for high-performance analytics.

Comparison: When to Use What¶

Option	Best For	Trade-offs	Approx. Cost Signal
Storage Transfer Service (STS)	Moving large objects from S3, Azure, or HTTP/S sources.	Great for scale; limited to object-based data.	Low (Pay for transfer/API)
BigQuery Data Transfer Service	SaaS applications (Google Ads, YouTube) and BigQuery-to-BigQuery.	Zero management; limited to supported SaaS sources.	Low to Medium
Datastream (CDC)	Database migrations with minimal downtime requirements.	Near real-time; requires network connectivity to source.	Medium (Based on data volume)
Transfer Appliance	Massive datasets (TB to PB) where network bandwidth is insufficient.	High physical logistics effort; high "time-to-first-byte."	High (Hardware rental + shipping)

How to choose: Start by assessing your Data Volume (use Transfer Appliance for PBs) and your Downtime Tolerance (use Datastream if you cannot afford a window of unavailability).

Cost Cheat Sheet¶

Scenario	Recommended Option	Key Cost Driver	Watch Out For
Bulk Historical Migration	STS	Network Egress fees from source cloud	💰 Egress fees from AWS/Azure can exceed the cost of the migration itself.
Real-time Syncing	Datastream	Volume of change events (CDC)	High "churn" in source DBs can spike costs.
One-time Massive Move	Transfer Appliance	Shipping and logistics	Data becomes "stale" while the device is in transit.
SaaS Data Ingestion	BQ DTS	Data volume processed	High frequency of transfers increases API costs.

💰 Cost Note: The single biggest "hidden" cost in migrations is Cloud Egress. Moving data into GCP is generally free, but moving it out of AWS or Azure to GCP incurs significant charges from the source provider. Always calculate your egress budget before starting a large-scale STS job.

Service & Tool Integrations¶

The "Landing Zone" Pattern:
- Use STS to move files $\rightarrow$ Store in Cloud Storage $\rightarrow$ Trigger Cloud Functions $\rightarrow$ Run Dataflow $\rightarrow$ Load to BigQuery.
The "CDC Modernization" Pattern:
- Use Datastream to capture SQL logs $\rightarrow$ Write to Cloud Storage $\rightarrow$ Use BigQuery Omni or BigLake to query the data in place without moving it further.

Security Considerations¶

Migration involves moving the organization's "crown jewels." Security must be baked into the pipeline.

Control	Default State	How to Enable / Strengthen
Encryption in Transit	TLS/SSL enabled	Use VPC Service Controls to ensure data stays within a defined perimeter.
Encryption at Rest	Google-managed keys	Implement CMEK (Customer Managed Encryption Keys) using Cloud KMS.
Access Control	IAM-based	Apply the Principle of Least Privilege; use specific Service Accounts for STS/Datastream.
Data Integrity	Checksums available	Use `gsutil hash` or built-in STS integrity checks to verify post-migration.

Performance & Cost¶

When migrating, you face a classic trade-off: Network Throughput vs. Cost.

Example Scenario: You need to migrate 100 TB of data from an on-premises data center to BigQuery. * Option A (Internet/VPN): Assuming a 1 Gbps dedicated link, the theoretical transfer time is ~10-12 days (accounting for overhead). * Cost: Low direct cost, but high "opportunity cost" of delayed analytics. * Option B (Transfer Appliance): You ship a physical appliance. * Cost: High upfront cost for the device and shipping, but the data is "in flight" even while your network is busy with production traffic.

Tuning Tip: For BigQuery migrations, always ensure your target tables are partitioned and clustered during the initial load. 💡 Tip: Loading data via BigQuery Load Jobs from GCS is free, whereas using the Streaming API incurs costs per GB.

Hands-On: Key Operations¶

1. Validating Data Integrity after an STS transfer Run this to ensure the MD5 hash of your local file matches the file in GCS.

gsutil hash -h gs://my-migration-bucket/data-file.csv

2. Triggering a BigQuery Load Job from a migrated file Once the data is in the landing zone (GCS), use this to move it into your warehouse.

bq load --source_format=CSV --autodetect my_dataset.my_table gs://my-migration-bucket/data-file.csv

💡 Tip: Using --autodetect is great for prototypes, but for production migrations, always provide a predefined Schema JSON to prevent data type mismatches.

Customer Conversation Angles¶

Q: We cannot afford any downtime for our production database. How do we move to Cloud SQL? A: We would implement a Change Data Capture (CDC) pattern using Datastream. This allows us to sync your on-prem changes to GCP in real-time, so the "cutover" is simply a matter of pointing your application to the new endpoint.

Q: We have 500TB of data in AWS S3. Is it worth moving to GCP? A: It depends on your egress costs. We should first run a cost-benefit analysis comparing the savings of GCP's BigQuery analytics against the one-time cost of AWS egress fees.

Q: If we just "Lift and Shift" our current VMs, will we actually save money? A: You will save on hardware maintenance, but you won't see the full ROI. To truly save, we should aim to "Replatform" into managed services like Cloud SQL or BigQuery to eliminate the "undifferentiated heavy lifting" of managing OS patches and backups.

Q: How do I know if the data was corrupted during the transfer? A: We use built-in checksum validation within the Storage Transfer Service and can run post-migration row-count audits using SQL to ensure the source and target are perfectly aligned.

Q: Is our data secure while it's sitting in the "Landing Zone" in GCS? A: Absolutely. We can enforce encryption using your own keys (CMEK) and use VPC Service Controls to ensure that data cannot be exfiltrated to any unauthorized internet location.

Common FAQs and Misconceptions¶

Q: Can I use Storage Transfer Service to move data from a SQL database? A: No. STS is designed for object-based data (files, images, logs). For databases, you need Datastream or BigQuery Data Transfer Service. ⚠️ Warning: Attempting to use file-based transfer tools for structured database migrations will result in loss of relational integrity.

Q: Does migrating to the cloud automatically make my data more secure? A: Not automatically. While GCP provides robust security tools, the "Shared Responsibility Model" means you are still responsible for configuring IAM and access controls correctly.

Q: Is the Transfer Appliance "plug and play"? A: No. It requires a significant workflow involving physical shipping, local network configuration, and a "copying" phase in your data center.

Q: Does BigQuery charge for the data sitting in my Landing Zone (GCS)? A: You pay standard Cloud Storage rates. This is much cheaper than BigQuery storage, so keeping raw data in GCS is a standard best practice.

Q: Can I migrate data from Azure to GCP? A: Yes, Storage Transfer Service (STS) specifically supports migrations from Azure Blob Storage.

Exam & Certification Focus¶

Design Patterns (Domain: Design Data Processing Systems): Understand when to use Batch (STS) vs. Streaming/CDC (Datastream).
Tool Selection: Be able to differentiate between Transfer Appliance (Physical/Massive) and STS (Network/Object).
Cost Optimization: Know the impact of Egress fees and the cost difference between BigQuery Load Jobs vs. Streaming Inserts. 📌 Must Know: The concept of "Data Gravity" and how it influences migration strategy.

Quick Recap¶

Rehost is fast but low-value; Refactor is high-effort but high-value.
Datastream is the go-to for zero-downtime database migrations via CDC.
Egress fees from other clouds are a primary driver of migration costs.
Cloud Storage should serve as your "Landing Zone" for all incoming migrations.
Verification (checksums/row counts) is a non-negotiable step in any migration project.