Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c333e80
Step 1: Update Ruby version requirement and modernize gemspec file fo…
acastro2 Dec 9, 2025
1e8cea7
Step 2: Remove outdated version constraints in Gemfile Build status: …
acastro2 Dec 9, 2025
0027ee9
Step 3: Update all gem dependencies to latest stable versions Build s…
acastro2 Dec 9, 2025
fff6c6b
Step 4: Update Cucumber features and step definitions for compatibili…
acastro2 Dec 9, 2025
d582273
Step 5: Review and update Ruby 3.3 keyword argument compatibility Bui…
acastro2 Dec 9, 2025
3c05f49
Step 6: Update CI/CD configuration for Ruby 3.3 Build status: Success
acastro2 Dec 9, 2025
2164653
Step 7: Run RSpec test suite and fix failures Build status: Success
acastro2 Dec 9, 2025
e823ea0
Step 8: Run Cucumber test suite and fix failures Build status: Success
acastro2 Dec 9, 2025
60dd4fd
Step 9: Verify complete test suite passes Build status: Success
acastro2 Dec 9, 2025
436299f
Step 10: Update documentation for Ruby 3.3 requirement Build status: …
acastro2 Dec 10, 2025
1f8aef0
Update PostgreSQL to 17, Ruby to 3.3 in Docker configs, remove log fi…
acastro2 Dec 10, 2025
028b483
Add Ruby 3.2 support alongside Ruby 3.3
acastro2 Dec 10, 2025
6a075a5
Update documentation: bundler 4.0.1, Ruby 3.2-3.3 support, PostgreSQL 17
acastro2 Dec 10, 2025
0427a17
Enable Cucumber tests in CI - all implemented scenarios pass
acastro2 Dec 10, 2025
7ebc53c
Fix remaining test failures for Ruby 3.3 upgrade - Fix Git.add_all to…
acastro2 Dec 10, 2025
9b40932
Add Ruby 2.7+ support and improve README documentation - Update gemsp…
acastro2 Dec 10, 2025
ba18367
Fix CI: Extract blog.git test fixture before running Cucumber tests T…
acastro2 Dec 10, 2025
b640ea7
Fix CI: Install PostgreSQL 17 client to match server version The CI w…
acastro2 Dec 10, 2025
25c2778
Fix CI: Configure git user for Cucumber commit tests Cucumber tests w…
acastro2 Dec 10, 2025
5b7e7cc
Fix git configuration for Cucumber tests properly Set git author/comm…
acastro2 Dec 10, 2025
bf7f256
Add PostgreSQL 17 bin directory to PATH for tests Ensure pg_dump and …
acastro2 Dec 10, 2025
1196830
Fix PATH configuration to properly prepend PostgreSQL bin Use shell e…
acastro2 Dec 10, 2025
5c8486d
Fix PostgreSQL 17 non-deterministic pg_dump output PostgreSQL 17's pg…
acastro2 Dec 10, 2025
be018d9
Bump version to 1.9.0 for Ruby 3.3 and PostgreSQL 17 support
acastro2 Dec 10, 2025
08f55d2
feat: add postgres versions and ruby 3.1 to the CI
acastro2 Dec 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 27 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,11 @@ jobs:
strategy:
fail-fast: false
matrix:
ruby-version: [2.6, 2.7, 3.0]
ruby-version: [2.7, 3.0, 3.1, 3.2, 3.3]
postgres-version: [15, 16, 17]
services:
postgres:
image: postgres:13-alpine
image: postgres:${{ matrix.postgres-version }}-alpine
ports:
- 5432:5432
env:
Expand All @@ -39,8 +40,18 @@ jobs:
with:
ruby-version: ${{ matrix.ruby-version }}
bundler-cache: true
- name: Install Library Dependencies
run: sudo apt update && sudo apt install -y postgresql-client
- name: Install PostgreSQL ${{ matrix.postgres-version }} Client
run: |
# Add PostgreSQL APT repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
# Install PostgreSQL client tools
sudo apt-get install -y postgresql-client-${{ matrix.postgres-version }}
- name: Extract Test Fixtures
run: |
cd features/support
tar -xzf blog.git.tgz
- name: Setup Database
run: |
mkdir -p config
Expand All @@ -57,11 +68,15 @@ jobs:
PGPORT: 5432
PGUSER: postgres
PGPASSWORD: postgres
run: bundle exec rspec
# - name: Test with Cucumber
# env:
# PGHOST: localhost
# PGPORT: 5432
# PGUSER: postgres
# PGPASSWORD: postgres
# run: bundle exec cucumber
run: |
export PATH="/usr/lib/postgresql/${{ matrix.postgres-version }}/bin:$PATH"
bundle exec rspec
- name: Test with Cucumber
env:
PGHOST: localhost
PGPORT: 5432
PGUSER: postgres
PGPASSWORD: postgres
run: |
export PATH="/usr/lib/postgresql/${{ matrix.postgres-version }}/bin:$PATH"
bundle exec cucumber
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
Gemfile.lock
tmp
prodder-*
*.log
features/support/blog.git
1 change: 1 addition & 0 deletions .ruby-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.3.10
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ARG RUBY_VERSION=3.0
ARG RUBY_VERSION=3.3

FROM ruby:${RUBY_VERSION}

Expand Down
6 changes: 4 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
source 'https://rubygems.org'

gemspec

gem 'deject'
gem 'thor'
gem 'cocaine'
Expand All @@ -10,7 +12,7 @@ group :development, :test do
gem 'pry'
gem 'pry-remote'
gem 'rspec'
gem 'cucumber', '< 3' # FIXME
gem 'aruba', '~> 0.5.0'
gem 'cucumber'
gem 'aruba'
gem 'pg'
end
240 changes: 164 additions & 76 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,111 @@

# Prodder

[![Ruby](https://img.shields.io/badge/ruby-2.7%2B-ruby.svg)](https://www.ruby-lang.org)
[![PostgreSQL](https://img.shields.io/badge/postgresql-15%2B-blue.svg)](https://www.postgresql.org)

A tool to maintain and load your Rails application's database structure, seed
table contents, permissions and database settings based on its migration history
and the current state in production databases.

In short: `db:reset db:migrate`

1. Your project maintains `db/structure.sql`, `db/seeds.sql`, and optional
`db/quality_checks.sql` and `db/permissions.sql` files as it sees fit (ie, by using `prodder` as a script to dump
production and push it to your git repository).
2. Make sure `db/seeds.sql` includes the `schema_migrations` table.
3. Only new migrations will be run against prod's structure using its seed table contents.
4. Once a migration has been deployed, it should result in `db/structure.sql` and
`db/quality_checks.sql` files being modified, and any new seed data being added to
`db/seeds.sql` -- including the new entry in `schema_migrations`.
5. That migration never needs to be run in development again. Feel free to `rm`.
6. Any application related permission changes will result in `db/permissions.sql` being modified.

## Replacing `rake db:*`
`prodder` can be included as a railtie in your application to automatically
replace many of Rails' `db:*` tasks. The only prerequisites to its usage are
the existence of `db/structure.sql`, `db/seeds.sql` with at least the
`schema_migrations` table contents included. Optional `db/quality_checks.sql` and `db/permissions.sql`
will be loaded after seeding, which can be helpful if you wish to seed the database
prior to enforcing foreign key constraints and if you want to develop in an environment
with the same permissions setup as production.
**In short:** Synchronize your development database with production structure without re-running all migrations.

## Why Prodder?

Traditional Rails development requires running all migrations from scratch, which:

- ❌ Becomes slow as your migration history grows
- ❌ Can fail if old migrations are incompatible with current code
- ❌ Doesn't reflect actual production database state

Prodder solves this by:

- ✅ Loading production database structure directly
- ✅ Running only new migrations not yet deployed to production
- ✅ Maintaining permissions and quality checks from production
- ✅ Automatically syncing structure files from production databases

## Requirements

- **Ruby 2.7+** - This gem requires Ruby 2.7.0 or later
- **Bundler 2.0+** - For dependency management
- **PostgreSQL 15+** - Requires PostgreSQL 15.0 or later

**Note:** Support for Ruby 2.6 and PostgreSQL versions older than 15 has been removed as of the latest version. If you need to use older versions, please use a previous version of this gem.

## Overview

Prodder follows a simple workflow:

1. **Maintain structure files**: Your project keeps `db/structure.sql`, `db/seeds.sql`, and optionally `db/quality_checks.sql` and `db/permissions.sql` in version control.

2. **Include migrations table**: Ensure `db/seeds.sql` includes the `schema_migrations` table from production.

3. **Run new migrations only**: Only migrations not yet in production's `schema_migrations` table will run locally.

4. **Update structure files**: After deploying a migration to production, update your structure files by running `prodder dump` against production.

5. **Delete old migrations**: Once a migration is deployed and the structure files are updated, the migration file can be safely removed.

6. **Track permissions**: Application permission changes are captured in `db/permissions.sql` for consistent development environments.

### The Prodder Workflow

```
Production DB → prodder dump → db/*.sql files → Git → Development
db:reset + new migrations
```

## Replacing `rake db:*` Tasks

Prodder can be included as a Railtie in your Rails application to automatically
replace many of Rails' `db:*` tasks with versions that work with production-sourced
structure files.

### Prerequisites

- `db/structure.sql` - Base database structure
- `db/seeds.sql` - Seed data including `schema_migrations` table
- `db/quality_checks.sql` (optional) - Foreign keys and constraints
- `db/permissions.sql` (optional) - Database permissions for role-based access

### Installation

In your Gemfile:
Add to your Gemfile:

```ruby
gem 'prodder', require: 'prodder/railtie'
```

It doesn't really matter, but for sanity's sake, you should set your `schema_format`
to `:sql`:
Configure Rails to use SQL schema format:

```ruby
# config/application.rb
module Whatever
class Application
module YourApp
class Application < Rails::Application
config.active_record.schema_format = :sql
end
end
```

### Basic Usage

Once installed, use these commands:

```bash
# Recreate database from structure and seed files
bundle exec rake db:reset

# Run only new migrations (those not in production's schema_migrations)
bundle exec rake db:migrate

# The typical development workflow
bundle exec rake db:reset db:migrate
```

If you want to work with permissions setup like production:

```ruby
# config/database.yml

Expand Down Expand Up @@ -93,49 +151,48 @@ Things that really matter:
does not make sense to restore permissions in your environment if you're just going to run everything as a single,
most likely superuser.


### Details

This will remove the `db:*` tasks:

* `db:_dump`: an internal task used by rails to dump the schema after migrations. Obsolete.
* `db:drop:*`
* `db:create:*`
* `db:migrate`
* `db:migrate:reset`
* `db:migrate:up`
* `db:migrate:down`
* `db:fixtures:.*`
* `db:abort_if_pending_migrations`
* `db:purge:*`
* `db:charset`
* `db:collation`
* `db:rollback`
* `db:version`
* `db:forward`
* `db:reset`
* `db:schema:*`
* `db:seed`
* `db:setup`
* `db:structure:*`
* `db:test:*`
* `test:prepare`: Rails 4.1 added this task to auto-maintain the test DB schema.
- `db:_dump`: an internal task used by rails to dump the schema after migrations. Obsolete.
- `db:drop:*`
- `db:create:*`
- `db:migrate`
- `db:migrate:reset`
- `db:migrate:up`
- `db:migrate:down`
- `db:fixtures:.*`
- `db:abort_if_pending_migrations`
- `db:purge:*`
- `db:charset`
- `db:collation`
- `db:rollback`
- `db:version`
- `db:forward`
- `db:reset`
- `db:schema:*`
- `db:seed`
- `db:setup`
- `db:structure:*`
- `db:test:*`
- `test:prepare`: Rails 4.1 added this task to auto-maintain the test DB schema.

And reimplement only the following:

* `db:structure:load`: Load the contents of `db/structure.sql` into the database of your current environment.
* `db:seed`: Load `db/seeds.sql` into the database of your current environment.
* `db:quality_check`: Load `db/quality_checks.sql` into the database of your current environment, if present.
* `db:reset`: db:drop db:setup
* `db:settings`: Load the contents of `db/settings.sql` into the database of your current environment.
* `db:setup`: db:create db:structure:load db:seed db:quality_check db:settings
* `db:test:prepare`: RAILS_ENV=test db:reset db:migrate
* `db:test:clone_structure`: RAILS_ENV=test db:reset db:migrate
* `test:prepare`: db:test:prepare
* `db:drop`: Drop database as superuser
* `db:create`: Create database as `superuser` and transfer ownership to `migration_user`
* `db:migrate:*`, `db:rollback` Run migrations up/down as `migration_user`
* `db:purge:*, db:charset, db:collation, db:version, db:forward, db:rollback, db:abort_if_pending_migrations` as
- `db:structure:load`: Load the contents of `db/structure.sql` into the database of your current environment.
- `db:seed`: Load `db/seeds.sql` into the database of your current environment.
- `db:quality_check`: Load `db/quality_checks.sql` into the database of your current environment, if present.
- `db:reset`: db:drop db:setup
- `db:settings`: Load the contents of `db/settings.sql` into the database of your current environment.
- `db:setup`: db:create db:structure:load db:seed db:quality_check db:settings
- `db:test:prepare`: RAILS_ENV=test db:reset db:migrate
- `db:test:clone_structure`: RAILS_ENV=test db:reset db:migrate
- `test:prepare`: db:test:prepare
- `db:drop`: Drop database as superuser
- `db:create`: Create database as `superuser` and transfer ownership to `migration_user`
- `db:migrate:*`, `db:rollback` Run migrations up/down as `migration_user`
- `db:purge:*, db:charset, db:collation, db:version, db:forward, db:rollback, db:abort_if_pending_migrations` as
appropriate users.

See [lib/prodder/prodder.rake](lib/prodder/prodder.rake)
Expand All @@ -144,10 +201,40 @@ for more info.
This is likely to cause issues across Rails versions. No other choice really. It
has been used in anger on Rails 3.2.x and Rails 4.1.x.

Confirmed working versions of Postgres:
## Development and Testing

### Ruby Version

This project requires Ruby 2.7+ for gem usage, though development is done on Ruby 3.3+. The development Ruby version is specified in `.ruby-version` and minimum required version in the gemspec file.

### Testing Frameworks

This project uses the following testing frameworks:

- **RSpec 3.13+** for unit tests
- **Cucumber 10.x** for feature tests (upgraded from 2.x)
- **Aruba 2.x** for CLI testing (upgraded from 0.5.x)

### Running Tests

```bash
# Run RSpec tests
bundle exec rspec

# Run Cucumber features
bundle exec cucumber

# Run all tests
bundle exec rspec && bundle exec cucumber
```

### Supported PostgreSQL Versions

This gem requires PostgreSQL 15.0 or later. Tested and confirmed working on:

* 9.1.11+
* 9.2.6+
- PostgreSQL 15.x
- PostgreSQL 16.x
- PostgreSQL 17.x

## Using prodder to maintain `db/*` files

Expand Down Expand Up @@ -208,7 +295,7 @@ store:
### Quality Checks

In some cases, such as foreign key dependencies and triggers, you may wish to defer
loading constraints on your tables until _after_ your seed data has been loaded.
loading constraints on your tables until *after* your seed data has been loaded.
`prodder` treats the presence of a `quality_check_file` key in the configuration
as an indication that it should split `structure_file` into those statements which
create the base structure, and put the constraints into the `quality_check_file`.
Expand Down Expand Up @@ -257,19 +344,20 @@ $ prodder push -c prodder.yml

## TODO

* Log activity as it is performed.
* Support tracking a particular branch instead of master.
* Support specifying the options to pass to each pg_dump form.
* Select dumping only a subset of a seed table. (pg_dump won't do this ...)
- Log activity as it is performed.
- Support tracking a particular branch instead of master.
- Support specifying the options to pass to each pg_dump form.
- Select dumping only a subset of a seed table. (pg_dump won't do this ...)

## Previous Contributors

* [Kyle Hargraves](https://github.com/pd)
* [Sri Rangarajan](https://github.com/Slania)
* [Emmanuel Sambo](https://github.com/esambo)
* [Cindy Wise](https://github.com/cyyyz)
* [Robert Nubel](https://github.com/rnubel)
* [Josh Cheek](https://github.com/JoshCheek)
- [Kyle Hargraves](https://github.com/pd)
- [Sri Rangarajan](https://github.com/Slania)
- [Emmanuel Sambo](https://github.com/esambo)
- [Cindy Wise](https://github.com/cyyyz)
- [Robert Nubel](https://github.com/rnubel)
- [Josh Cheek](https://github.com/JoshCheek)
- [Alexandre Castro](https://github.com/acastro2)

## License

Expand Down
Loading