data-migration

Topics in the source

• Data migration
• 1. Never combine schema change with data change
• 2. Use expand-then-contract for anything breaking
• 3. Know which operations take locks
• 4. Backfill in batches, outside the migration
• 5. Rehearse on real data volume
• 6. Have the plan written before you run it
• Moving between systems

Migrations are the most irreversible thing most engineers do routinely. Code can be rolled back; a dropped column cannot, and a botched backfill can corrupt data in ways no deploy fixes. Two properties make a migration safe, and both must hold: Backwards compatible: the currently deployed code works before and after it runs Reversible, or backed up: you can get back, or you have a copy you have tested restoring If a migration has neither, it needs a maintenance window and a rehearsal. Separate migrations, separate deploys: 1. Add the new structure 2. Backfill it 3. Switch the code to use it 4. Remove the old structure — later, as its own change Combining these makes rollback impossible: reverting the code leaves the data half-moved, and reverting the data loses whatever the new code wrote. Done when: each step is independently deployable and revertible. Renaming a column is not a rename in production. It is: 1. Expand — add newname, nullable 2. Dual-write — deploy code writing both; reads still use the old 3. Backfill — populate newname for existing rows, in batches 4. Switch reads — deploy code reading the new column 5. Stop writing the old 6. Contract — drop oldname, well after e

This is an excerpt. Refer to the complete source and its current requirements.