Quick Insight
Data migration is the process of transferring data between storage systems, formats, databases, or applications, typically as part of a system upgrade, consolidation, or move to the cloud. It involves extracting data from the source, transforming it to fit the target schema, loading it, and validating that it arrived complete and accurate. It differs from data integration, which keeps multiple systems connected on an ongoing basis, whereas migration is usually a one-time move with a defined end.
How Data Migration Works
- Plan and scope: Decide which data moves, which stays, and which gets archived or deleted.
- Profile the source: Check data quality before moving. Migration is the cheapest moment to fix duplicates and errors.
- Map fields: Match every source field to its target, including format changes like date styles or coordinate formats.
- Test with a sample: Run a trial migration on a subset and compare results.
- Migrate: Move the data in one go or in phases.
- Validate: Compare record counts, totals, and spot-check individual records.
- Decommission: Retire the old system only after the new one is confirmed working.
Types of Data Migration
- Storage migration: Moving data to new hardware or cloud storage.
- Database migration: Moving to a new database engine, such as from an on-premise database to a cloud warehouse.
- Application migration: Switching software, such as moving to a new CRM.
- Cloud migration: Moving data and workloads from on-premise systems to the cloud.
- Provider migration: Switching to a new third-party data source, such as a new POI data provider, which means remapping IDs, categories, and attributes.
Migration Approaches
Approach | How it works | Best for |
Big bang | Everything moves at once, usually over a short downtime window | Smaller datasets, simpler systems |
Phased (trickle) | Data moves in stages while both systems run | Large, business-critical systems |
Migration Challenges with Location Data
- ID mismatches: Old and new systems may use different identifiers for the same place.
- Geometry formats: Polygons may need converting between formats like WKT and GeoJSON, or between coordinate systems. Building footprint data delivered in a standard format reduces this work.
- Category changes: A store classified one way in the old system may sit under a different category in the new one.
A caveat: migrations almost always take longer than planned. Hidden data quality problems usually surface only once the move begins.
Frequently Asked Questions
1. How long does a data migration take?
Anywhere from days to many months, depending on data volume, complexity, and how much cleanup is needed.
2. Can data be lost during migration?
Yes, if validation is skipped. Full backups and record-level checks before decommissioning protect against permanent loss.
3. Should we clean data before or after migrating?
Before, where possible. Moving bad data just recreates the same problems in a new and more expensive system.
4. Will users face downtime?
Sometimes. Big-bang migrations usually need a short outage, while phased approaches can avoid it at the cost of running two systems for a while.
5. How do I know the migration succeeded?
Record counts match, key totals reconcile, sample records check out, and the reports people rely on produce the same results as before.
