Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 127 additions & 0 deletions migrations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# @linefusion/migrations

An opinionated database migration tool built on [Kysely](https://kysely.dev).

It turns a folder of Kysely migration files into a **self-contained Docker
image** that, on start, idempotently converges a database to exactly the
migrations baked into the image — migrating **up** when the image is ahead and
rolling **down** when it is behind.

## How it works

Each migration is a TypeScript file named `<YYYYMMDDHHMMSS>-<name>.ts` exporting
`up` (and optionally `down`) functions:

```ts
import type { Kysely } from "kysely";

export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable("users")
.addColumn("id", "integer", (c) => c.primaryKey())
.addColumn("name", "text", (c) => c.notNull())
.execute();
}

export async function down(db: Kysely<any>): Promise<void> {
await db.schema.dropTable("users").execute();
}
```

When a migration is applied, the tool records it in a
`_migrations_migrations_source` table together with its **compiled source**
(the `compiled` column). Because the down script lives in the database, a
migration can be rolled back **even after it has been removed from disk** — for
example when a container built from an older image rolls back a newer database.

Two tables are used:

- `_migrations` — Kysely's own ledger of which migrations are applied.
- `_migrations_migrations_source` — the name, path, source and compiled
script of every migration the database has ever seen (`executed` holds the
timestamp, or `NULL` once rolled back).

## CLI

```
migrate <command> [options]

Commands:
apply Idempotently converge the database to the migrations on disk
status Show whether the database is in sync with the migrations on disk
list List all migrations and their status
rollback Roll migrations back down
build Build a Docker image that runs the migrations automatically

Common options:
-d, --database <url> Database URL (env: DATABASE, default sqlite::memory:)
-m, --migrations-dir <path> Migrations folder (env: MIGRATIONS_DIR, default .)
```

Supported database URLs: `sqlite:./db.sqlite`, `sqlite::memory:`,
`postgres://…`, `mysql://…`.

### apply

`apply` is the command the container runs on start. It is safe to run any
number of times:

- **Image ahead of the database** → the missing migrations are run **up**.
- **Image behind the database** (there are applied migrations that no longer
exist on disk) → those migrations are rolled **down** using the compiled down
scripts stored in the database. Pass `--no-rollback` to fail instead.
- **In sync** → nothing happens.

```sh
migrate apply -d postgres://user:pass@localhost/app -m ./migrations
```

### rollback

```sh
migrate rollback # roll back the most recently applied migration
migrate rollback --to <name> # roll back down to (but not including) <name>
migrate rollback --all # roll everything back
```

## Docker

`migrate build` generates a Dockerfile and builds an image that carries your
migrations and the compiled migrator, with `apply` as the default command:

```sh
migrate build -t myapp-migrations:latest -m ./migrations
```

The generated image is roughly:

```dockerfile
FROM denoland/deno:2.4.0 AS build
WORKDIR /build
RUN ["deno", "compile", "--allow-all", "--no-check", "--output", "/build/migrate", "jsr:@linefusion/migrations/cli"]

FROM debian:stable-slim
WORKDIR /app
COPY --from=build /build/migrate /usr/local/bin/migrate
COPY migrations /app/migrations
ENV MIGRATIONS_DIR=/app/migrations
ENTRYPOINT ["migrate"]
CMD ["apply"]
```

Run it against any database by providing `DATABASE`:

```sh
docker run --rm -e DATABASE=postgres://user:pass@db/app myapp-migrations:latest
```

The container migrates up (or down) to match the migrations baked into the
image and exits. Drop it in as an init container / one-shot job before your app
starts and every deploy converges the schema automatically.

Useful `build` flags:

- `--dockerfile <path>` — write the Dockerfile without building.
- `--build-image` / `--runtime-image` — override the base images.
- `--platform <p>` — cross-build (e.g. `linux/arm64`).
- `--push` — push the image after building.
87 changes: 57 additions & 30 deletions migrations/cli/apply.ts
Original file line number Diff line number Diff line change
@@ -1,44 +1,71 @@
import { NO_MIGRATIONS, NoMigrations } from "kysely";
import { command } from "../commands.ts";
import { NamedMigration } from "../databases.ts";
import { $ } from "@david/dax";

import modules from "@linefusion/modules";
import type { Command } from "@cliffy/command";
import { NO_MIGRATIONS, type NoMigrations } from "kysely";

export default command("apply", "Apply migrations into the database")
.useDatabase()
.option("--allow-rollbacks", "Allows down migrations from database to be applied.", { default: false })
.action(async ({ database, allowRollbacks }) => {
const { provider, migrator } = await modules.compile(await database);
import { command, DATABASE_OPTION, MIGRATIONS_DIR_OPTION } from "../commands.ts";
import { createMigrator, displayResults, resolveDatabaseUrl, resolveMigrationsDir } from "./support.ts";

let targetMigration: string | NoMigrations = NO_MIGRATIONS;
/**
* Idempotently converges the target database to the set of migrations that
* ship on disk (i.e. baked into the container image).
*
* - If the image is *ahead* of the database (there are disk migrations that
* have not been applied yet) the missing migrations are run *up*.
* - If the image is *behind* the database (there are applied migrations that
* no longer exist on disk) those migrations are rolled *down* using the
* compiled down scripts persisted on each migration row.
*
* Running it repeatedly is safe: once the database matches the image nothing
* happens.
*/
const apply: Command = command("apply", "Idempotently converge the database to the migrations on disk")
.option(...DATABASE_OPTION)
.option(...MIGRATIONS_DIR_OPTION)
.option(
"--no-rollback",
"Fail instead of rolling back when the database is ahead of the migrations on disk.",
)
.action(async ({ database, migrationsDir, rollback }) => {
const { provider, migrator } = await createMigrator(
resolveDatabaseUrl(database),
resolveMigrationsDir(migrationsDir),
);

let latestMigration: NamedMigration | null = null;
const diskMigrations = provider.getDiskMigrations();
if (diskMigrations.length > 0) {
latestMigration = diskMigrations[diskMigrations.length - 1];
targetMigration = latestMigration.name;
}

const databaseMigrations = provider.getDatabaseOnlyMigrations().filter((m) => m.executed);
if (databaseMigrations.length > 0 && !allowRollbacks) {
throw new Error(
"There are applied migrations in the database that are not on disk. Use --allow-rollbacks to sync this build.",
);
}
const appliedAhead = provider
.getDatabaseOnlyMigrations()
.filter((migration) => migration.executed);

if (latestMigration) {
const hasSkippedMigrations = databaseMigrations.some((dm) =>
provider.getMigrationIndex(dm) < provider.getMigrationIndex(latestMigration)
);
if (hasSkippedMigrations) {
// The image is behind the database: there are applied migrations that no
// longer exist on disk. Converging means rolling them back down.
if (appliedAhead.length > 0) {
if (!rollback) {
throw new Error(
"There are applied migrations in the database that are not on disk and are older than the currently applied one.",
`The database is ahead of this image by ${appliedAhead.length} migration(s) ` +
`(${appliedAhead.map((m) => m.name).join(", ")}). ` +
`Refusing to roll back because --no-rollback was set.`,
);
}
$.logStep(
"behind",
`Database is ahead by ${appliedAhead.length} migration(s); rolling back.`,
);
}

provider.setMigrationFlags(MigrationFlags.Disk | MigrationFlags.Database);
// Target is the latest migration on disk, or a fully empty database when
// there are no migrations on disk at all.
let target: string | NoMigrations = NO_MIGRATIONS;
if (diskMigrations.length > 0) {
target = diskMigrations[diskMigrations.length - 1].name;
}

const result = await migrator.migrateTo(targetMigration);
const result = await migrator.migrateTo(target);
displayResults(result);

if (result.error) {
throw result.error;
}
});

export default apply;
125 changes: 121 additions & 4 deletions migrations/cli/build.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,123 @@
import { command } from "../commands.ts";
import { $ } from "@david/dax";

export default command("build", "Builds a migration executable.")
.action(async () => {
console.log(globalThis.location);
import type { Command } from "@cliffy/command";

import * as fs from "@std/fs";
import * as path from "@std/path";

import { command, MIGRATIONS_DIR_OPTION } from "../commands.ts";
import { resolveMigrationsDir } from "./support.ts";

const DEFAULT_BUILD_IMAGE = "denoland/deno:2.4.0";
const DEFAULT_RUNTIME_IMAGE = "debian:stable-slim";
const DEFAULT_ENTRYPOINT = "jsr:@linefusion/migrations/cli";

/**
* Renders the Dockerfile that packages the migrator together with the
* migrations found on disk.
*
* The image is built in two stages:
*
* 1. A build stage compiles the migrator CLI into a single self-contained
* binary with `deno compile`. The migration files are NOT bundled into the
* binary; they are read from disk at runtime so the same binary works for
* any set of migrations.
* 2. A minimal runtime stage carries that binary plus the migration files and
* runs `apply` on start, converging the database to the migrations baked
* into the image.
*/
export function renderDockerfile(options: {
buildImage: string;
runtimeImage: string;
entrypoint: string;
migrationsDir: string;
}): string {
return [
`# Generated by @linefusion/migrations.`,
``,
`FROM ${options.buildImage} AS build`,
`WORKDIR /build`,
`RUN ["deno", "compile", "--allow-all", "--no-check", "--output", "/build/migrate", "${options.entrypoint}"]`,
``,
`FROM ${options.runtimeImage}`,
`WORKDIR /app`,
`COPY --from=build /build/migrate /usr/local/bin/migrate`,
`COPY ${options.migrationsDir} /app/migrations`,
`ENV MIGRATIONS_DIR=/app/migrations`,
`ENTRYPOINT ["migrate"]`,
`CMD ["apply"]`,
``,
].join("\n");
}

const build: Command = command("build", "Build a Docker image that runs the migrations automatically")
.option(...MIGRATIONS_DIR_OPTION)
.option("-t, --tag <tag:string>", "Image tag to build.", { default: "migrations:latest" })
.option("--build-image <image:string>", "Base image used to compile the migrator.", {
default: DEFAULT_BUILD_IMAGE,
})
.option("--runtime-image <image:string>", "Base image the final container runs on.", {
default: DEFAULT_RUNTIME_IMAGE,
})
.option("--entrypoint <spec:string>", "Module specifier for the migrator CLI to compile.", {
default: DEFAULT_ENTRYPOINT,
})
.option("--platform <platform:string>", "Target platform passed to `docker build`.")
.option("--push", "Push the image after building.", { default: false })
.option("--dockerfile <path:string>", "Write the generated Dockerfile to this path instead of building.")
.action(async ({ migrationsDir, tag, buildImage, runtimeImage, entrypoint, platform, push, dockerfile }) => {
const migrationsPath = resolveMigrationsDir(migrationsDir);
const migrations = path.relative(Deno.cwd(), path.resolve(migrationsPath)) || ".";

if (!fs.existsSync(path.resolve(migrationsPath))) {
throw new Error(`Migrations directory "${migrationsPath}" does not exist.`);
}

const contents = renderDockerfile({
buildImage,
runtimeImage,
entrypoint,
migrationsDir: migrations,
});

// Only asked to render the Dockerfile.
if (dockerfile) {
await Deno.writeTextFile(dockerfile, contents);
$.logStep("dockerfile", `Wrote ${dockerfile}`);
return;
}

const docker = await $.which("docker");
if (!docker) {
throw new Error(
"`docker` was not found on PATH. Install Docker or use --dockerfile to only render the Dockerfile.",
);
}

// Build from a temporary Dockerfile so the whole working directory (which
// contains the migrations folder) is available as the build context.
const tempfile = await Deno.makeTempFile({ prefix: "migrations-", suffix: ".Dockerfile" });
try {
await Deno.writeTextFile(tempfile, contents);

const args = ["build", "-f", tempfile, "-t", tag];
if (platform) {
args.push("--platform", platform);
}
args.push(".");

$.logStep("build", `Building image ${tag} ...`);
await $`docker ${args}`;

if (push) {
$.logStep("push", `Pushing image ${tag} ...`);
await $`docker push ${tag}`;
}

$.logStep("success", `Image ${tag} built.`);
} finally {
await Deno.remove(tempfile).catch(() => {});
}
});

export default build;
Loading