diff --git a/database/README.md b/database/README.md new file mode 100644 index 00000000..9f8624bf --- /dev/null +++ b/database/README.md @@ -0,0 +1,177 @@ +# AltZone Database Migrations + +**Why this exists:** MongoDB doesn't enforce schemas, but our backend does (via Mongoose). When a model changes, existing documents in local, staging, and production need to stay compatible. Migrations are the automated, reversible way to do that, no manual `mongosh` tweaking required. + +[`migrate-mongo`](https://github.com/seppevs/migrate-mongo) is used to track which scripts have run and in what order. + +--- + +## Quick Start + +If you just changed a Mongoose model and need to migrate data: + +```bash +# 1. Generate a new migration file +npm run migrate:create add-player-rank-field + +# 2. Open the generated file in database/migrations/ and write your up/down logic + +# 3. Run it against your local database +npm run migrate:up + +# 4. Verify the result in MongoDB Compass or mongosh +``` + +--- + +## Environment & Config + +Migrations read your root `.env` for connection details: + +| Variable | Purpose | +|---|---| +| `MONGO_USERNAME` | Database user | +| `MONGO_PASSWORD` | Database password | +| `MONGO_HOST` | Host address | +| `MONGO_PORT` | Port (usually `27017`) | +| `MONGO_DB_NAME` | Target database name | + +The tool is configured in `migrate-mongo-config.js` at project root. Key settings you should know: + +| Setting | Value | Why it matters | +|---|---|---| +| `migrationsDir` | `database/migrations` | Where your scripts live. | +| `changelogCollectionName` | `migrations_changelog` | Tracks applied migrations in MongoDB itself. | +| `moduleSystem` | `esm` | All migration files must use the `export` syntax. | + +--- + +## CLI Reference + +| Command | What it does | When to use it | +|---|---|---| +| `npm run migrate:create ` | Creates a timestamped `.js` file in `database/migrations/` | Every time you alter a model that affects existing data | +| `npm run migrate:up` | Runs all pending migrations in chronological order | After pulling new code, or after writing a new migration locally | +| `npm run migrate:down` | Reverts **only the last applied** migration | When your last migration broke something locally | +| `npm run migrate:status` | Shows which migrations are applied vs. pending | Before pushing code, confirm your local DB is in sync | + +> **Tip:** Always run `migrate:status` before switching branches. If a branch has migrations you haven't run, your local schema will be out of sync and tests will likely fail in all kinds of confusing ways. + +--- + +## Writing a Migration + +A migration file is just a plain ES module exporting `up` and `down`. Both receive the native MongoDB `Db` instance and `MongoClient`. + +Use the template at `database/migrations/migration-template.js` as your starting point. + +### Anatomy of a Migration + +```javascript +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +export const up = async (db, client) => { + // Apply your changes here: data backfills, index creation, collection renames, etc. +}; + +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +export const down = async (db, client) => { + // Reverse everything in `up`. This must leave the DB in its pre-migration state. +}; +``` + +### Common Patterns + +#### 1. Backfilling a New Required Field +If you add a required field to a Mongoose schema, existing documents will fail validation unless they have a value themselves. + +```javascript +export const up = async (db) => { + await db.collection('players').updateMany( + { rank: { $exists: false } }, + { $set: { rank: 'rookie' } } + ); +}; + +export const down = async (db) => { + await db.collection('players').updateMany( + {}, + { $unset: { rank: '' } } + ); +}; +``` + +--- + +**2. Creating an Index (When Needed)** + +If `autoIndex` is enabled in `AppModule`, Mongoose will create indexes defined in your schema automatically on startup. You generally **do not** need to create them in migrations unless: + +- You're creating an index that isn't defined in the Mongoose schema (e.g., a compound index for a specific query pattern). +- You're working on a collection that doesn't have a Mongoose model. +- You need fine-grained control over index options (e.g., partial filters, collation). + +```javascript +export const up = async (db) => { + await db.collection('matches').createIndex( + { playerId: 1, createdAt: -1 }, + { name: 'matches_playerId_createdAt' } + ); +}; + +export const down = async (db) => { + await db.collection('matches').dropIndex('matches_playerId_createdAt'); +}; +``` + +> **You should still name your indexes explicitly** if you create them manually. It makes `dropIndex` in `down` unambiguous. + +--- + +- [ ] **Indexes are explicitly named** when created manually. This prevents `dropIndex` from breaking if MongoDB's auto-naming convention changes. + +--- + +#### 3. Multi-Step Migrations +If a migration does several things, `down` must undo them in **reverse order**: + +```javascript +export const up = async (db) => { + await db.collection('users').updateMany({}, { $set: { legacy: false } }); + await db.collection('users').createIndex({ legacy: 1 }); +}; + +export const down = async (db) => { + await db.collection('users').dropIndex('legacy_1'); // Undo step 2 first + await db.collection('users').updateMany({}, { $unset: { legacy: '' } }); // Then step 1 +}; +``` + +--- + +## Safety Checklist + +Before committing a migration, verify: + +- [ ] **`down` actually works.** Run `migrate:up`, then `migrate:down`, then `migrate:up` again. If the second `up` fails, your `down` is incomplete. +- [ ] **Indexes are explicitly named.** This prevents `dropIndex` from breaking if MongoDB's auto-naming convention changes. +- [ ] **Destructive operations are gated.** Use `updateMany` with filters rather than blindly overwriting documents if/when needed. +- [ ] **The migration is idempotent.** Running `up` twice should not error or corrupt data (e.g., use `$set` instead of `$inc` for defaults). + +--- + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---|---|---| +| `migrate:up` says "no migrations to run" | Already applied, or file not in `database/migrations/` | Check `migrate:status` | +| `migrate:down` fails with "index not found" | Index was dropped manually or name mismatch | Check `db.collection.getIndexes()` in MongoDB | +| Tests fail after pulling `main` | Missing migrations on your branch | Run `migrate:up`, check `migrate:status` | +| `down` leaves orphaned data | Forgot to unset a field or drop a collection | Fix the `down` function, test locally, commit only then | + +--- \ No newline at end of file diff --git a/database/migration-template.js b/database/migration-template.js new file mode 100644 index 00000000..b9893851 --- /dev/null +++ b/database/migration-template.js @@ -0,0 +1,15 @@ +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +export const up = async (db, client) => { + // TODO: Write your migration logic here when applicable +}; + +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +export const down = async (db, client) => { + // TODO: Write your rollback logic here when applicable +}; \ No newline at end of file diff --git a/database/migrations/20260814170400-add-player-sounds.js b/database/migrations/20260814170400-add-player-sounds.js new file mode 100644 index 00000000..c6df1fb7 --- /dev/null +++ b/database/migrations/20260814170400-add-player-sounds.js @@ -0,0 +1,76 @@ +/** + * Migration: Add sounds field to player settings + * + * Use case: Player-specific sound world ("pelaajakohtainen äänimaailma") + * Per team discussion (Discord, 10-11/08/2026): + * - Sounds stored as keys, not file paths + * - Client maps keys to actual audio assets + * - Server validates that keys exist in the allowed set + */ + +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +module.exports.up = async (db, client) => { + const session = client.startSession(); + + try { + await session.withTransaction(async () => { + const players = db.collection('players'); + + const result = await players.updateMany( + { sounds: { $exists: false } }, + { + $set: { + sounds: { + default: 'player_default_soft_chime', + memberJoined: null, + memberLeft: null, + dailyTaskCompleted: null, + milestoneUnlocked: null, + votingStarted: null, + battleWon: null, + battleLost: null + }, + soundsInitializedAt: new Date() + } + }, + { session } + ); + + console.log(`[migrate-mongo] Added soulful sounds to ${result.modifiedCount} players (${result.matchedCount} matched)`); + }); + } finally { + await session.endSession(); + } +}; + +/** + * @param db {import('mongodb').Db} + * @param client {import('mongodb').MongoClient} + */ +module.exports.down = async (db, client) => { + const session = client.startSession(); + + try { + await session.withTransaction(async () => { + const players = db.collection('players'); + + const result = await players.updateMany( + { sounds: { $exists: true } }, + { + $unset: { + sounds: '', + soundsInitializedAt: '' + } + }, + { session } + ); + + console.log(`[migrate-mongo] Removed soulful sounds from ${result.modifiedCount} players`); + }); + } finally { + await session.endSession(); + } +}; \ No newline at end of file diff --git a/migrate-mongo-config.js b/migrate-mongo-config.js new file mode 100644 index 00000000..39ffab32 --- /dev/null +++ b/migrate-mongo-config.js @@ -0,0 +1,22 @@ +require('dotenv').config(); + +const { + MONGO_USERNAME, MONGO_PASSWORD, MONGO_HOST, + MONGO_PORT, MONGO_DB_NAME +} = process.env; + +const url = `mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@${MONGO_HOST}:${MONGO_PORT}/?replicaSet=rs0`; + +module.exports = { + mongodb: { + url: url, + databaseName: MONGO_DB_NAME, + options: {} + }, + migrationsDir: "database/migrations", + changelogCollectionName: "migrations_changelog", + migrationFileExtension: ".js", + useFileHash: false, + // using commonjs here because migrate-mongo doesn't support ES modules yet + moduleSystem: 'commonjs', +}; \ No newline at end of file diff --git a/package-lock.json b/package-lock.json index 467864e6..ad7af8d2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -33,6 +33,7 @@ "ioredis": "5.8.2", "jsonwebtoken": "^9.0.3", "lodash": "4.18.1", + "migrate-mongo": "^14.0.7", "mongodb": "6.20.0", "mongoose": "8.19.4", "mqtt": "5.14.1", @@ -792,7 +793,6 @@ "version": "1.5.0", "resolved": "https://registry.npmjs.org/@colors/colors/-/colors-1.5.0.tgz", "integrity": "sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==", - "dev": true, "license": "MIT", "optional": true, "engines": { @@ -4989,7 +4989,6 @@ "version": "0.6.5", "resolved": "https://registry.npmjs.org/cli-table3/-/cli-table3-0.6.5.tgz", "integrity": "sha512-+W/5efTR7y5HRD7gACw9yQjqMVvEMLBHmboM/kPWam+H+Hmyrgjh6YncVKK122YZkXrLudzTuAukUw9FnMf7IQ==", - "dev": true, "license": "MIT", "dependencies": { "string-width": "^4.2.0" @@ -5574,7 +5573,6 @@ "version": "8.0.0", "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "dev": true, "license": "MIT" }, "node_modules/encodeurl": { @@ -7119,7 +7117,6 @@ "version": "3.0.0", "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "dev": true, "license": "MIT", "engines": { "node": ">=8" @@ -8611,6 +8608,34 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, + "node_modules/migrate-mongo": { + "version": "14.0.7", + "resolved": "https://registry.npmjs.org/migrate-mongo/-/migrate-mongo-14.0.7.tgz", + "integrity": "sha512-+p7XfJDNaXPTHeo7v/ldYmVLMy8xYda0KMXSqkMUzlVndS39rMGBQHfyLrdmOKMjgciWyWpjekXzRQuj1B7HqA==", + "license": "MIT", + "dependencies": { + "cli-table3": "^0.6.5", + "commander": "^14.0.2" + }, + "bin": { + "migrate-mongo": "bin/migrate-mongo.js" + }, + "engines": { + "node": ">=20.0.0" + }, + "peerDependencies": { + "mongodb": "^4.4.1 || ^5.0.0 || ^6.0.0 || ^7.0.0" + } + }, + "node_modules/migrate-mongo/node_modules/commander": { + "version": "14.0.3", + "resolved": "https://registry.npmjs.org/commander/-/commander-14.0.3.tgz", + "integrity": "sha512-H+y0Jo/T1RZ9qPP4Eh1pkcQcLRglraJaSLoyOtHxu6AapkjWVCy2Sit1QQ4x3Dng8qDlSsZEet7g5Pq06MvTgw==", + "license": "MIT", + "engines": { + "node": ">=20" + } + }, "node_modules/mime-db": { "version": "1.54.0", "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", @@ -10653,7 +10678,6 @@ "version": "4.2.3", "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "dev": true, "license": "MIT", "dependencies": { "emoji-regex": "^8.0.0", @@ -10707,7 +10731,6 @@ "version": "5.0.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "dev": true, "license": "MIT", "engines": { "node": ">=8" @@ -10717,7 +10740,6 @@ "version": "6.0.1", "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "dev": true, "license": "MIT", "dependencies": { "ansi-regex": "^5.0.1" diff --git a/package.json b/package.json index f349a469..b2d14ac8 100644 --- a/package.json +++ b/package.json @@ -17,7 +17,11 @@ "test:watch": "jest --watch", "test:quiet": "DOTENV_CONFIG_QUIET=true jest", "sonar:analize": "docker compose -f sonar_docker-compose.yml up -d && sonar-scanner \\\n -Dsonar.projectKey=AltzoneLocal \\\n -Dsonar.sources=. \\\n -Dsonar.host.url=http://localhost:9000 \\\n -Dsonar.login=sqp_d0d523f9a2cf5e1e8f7587e0da315df0adc86568 \\\n && echo \"Remember to stop the sonar server with: \\n docker compose -f sonar_docker-compose.yml down\"\n", - "prepare": "husky" + "prepare": "husky", + "migrate:create": "migrate-mongo create", + "migrate:up": "migrate-mongo up", + "migrate:down": "migrate-mongo down", + "migrate:status": "migrate-mongo status" }, "keywords": [], "author": "", @@ -47,6 +51,7 @@ "ioredis": "5.8.2", "jsonwebtoken": "^9.0.3", "lodash": "4.18.1", + "migrate-mongo": "^14.0.7", "mongodb": "6.20.0", "mongoose": "8.19.4", "mqtt": "5.14.1", @@ -81,4 +86,4 @@ "typescript": "5.9.3", "typescript-eslint": "8.46.4" } -} \ No newline at end of file +} diff --git a/src/app.module.ts b/src/app.module.ts index 9de71f9f..a1dd9b66 100644 --- a/src/app.module.ts +++ b/src/app.module.ts @@ -72,6 +72,7 @@ const authGuardClassToUse = isTestingSession() ? BoxAuthGuard : AuthGuard; return { uri: mongoString, dbName, + autoIndex: true, }; }, }), diff --git a/src/database/migrations/.gitkeep b/src/database/migrations/.gitkeep new file mode 100644 index 00000000..e69de29b