Repository navigation
fix(ci/fetch): raise MySQL buffer pool size for fetch-ubuntu - #373
Conversation
The mysql step of fetch-ubuntu fails with "Error 1206 (HY000): The total number of locks exceeds the lock table size". gost inserts all Ubuntu CVEs in one transaction, and the lock memory that transaction needs no longer fits in MySQL's default 128MB buffer pool now that the data has grown past ~75k CVEs. Double innodb_buffer_pool_size to 256MB on the service container before fetching. Only fetch-ubuntu is changed; the other fetch jobs pass with the default.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
Adjusts the Fetch CI workflow to prevent the MySQL “lock table size exceeded (Error 1206)” failure when importing Ubuntu CVEs by increasing InnoDB buffer pool memory for the fetch-ubuntu job.
Changes:
- Adds a pre-step in the MySQL fetch to resize
innodb_buffer_pool_sizeto 256MB viadocker exec. - Converts the MySQL fetch command into a multi-line script with an explanatory comment.
| File | Description |
|---|---|
| .github/workflows/fetch.yml | Resizes MySQL InnoDB buffer pool in CI before running gost fetch for Ubuntu data. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 2
Open (5)
docker exec ... mysql -e "SET GLOBAL ..."can fail transiently if the MySQL service isn’t ready… · Newdocker exec ... mysql -e "SET GLOBAL ..."can be flaky if MySQL hasn’t finished initializing yet…-ppassword) can expose it in logs/process listings.… · New268435456is a hard-to-read magic number in YAML. Using a size literal supported by MySQL (e.g.,… · New-ppassword) is typically echoed/visible to process…
268435456 is hard to read. SET GLOBAL does not accept a size suffix such as 256M (ERROR 1232), but it does accept an arithmetic expression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 3
Open (6)
SET GLOBAL innodb_buffer_pool_size=...can be applied asynchronously by MySQL; the next command… · Newdocker exec ... mysql -e "SET GLOBAL ..."can fail transiently if the MySQL service isn’t ready…docker exec ... mysql -e "SET GLOBAL ..."can be flaky if MySQL hasn’t finished initializing yet…mysqlCLI arguments and the--dbpathDSN, which can… · New-ppassword) can expose it in logs/process listings.…-ppassword) is typically echoed/visible to process…
Resolved since last review (1)
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The targeted workflow change correctly addresses the documented MySQL resource failure without affecting application behavior.
Review effort: Balanced
Findings: None
Resolved since last review (6)
SET GLOBAL innodb_buffer_pool_size=...can be applied asynchronously by MySQL; the next command…docker exec ... mysql -e "SET GLOBAL ..."can fail transiently if the MySQL service isn’t ready…docker exec ... mysql -e "SET GLOBAL ..."can be flaky if MySQL hasn’t finished initializing yet…mysqlCLI arguments and the--dbpathDSN, which can…-ppassword) can expose it in logs/process listings.…-ppassword) is typically echoed/visible to process…
What did you implement:
In the Fetch workflow, raise MySQL's
innodb_buffer_pool_sizefrom the default 128MB to 256MB before the mysql step offetch-ubuntu.That step has been failing on master and on every pull request (it was noted as a known failure in #372):
gost deletes and re-inserts all Ubuntu CVEs in a single transaction. InnoDB takes the lock memory for that transaction from the buffer pool, and the Ubuntu data has grown past what the default 128MB pool can hold:
The sqlite3, postgres and redis steps pass with the same data.
128MB is MySQL's stock default rather than a size anyone would run a production server with, so this only adjusts the CI service container and leaves gost itself unchanged. The official
mysqlimage has no environment variable for server options and aservicescontainer cannot be given command arguments, so the size is set withSET GLOBALthroughdocker exec(the buffer pool can be resized online).Only
fetch-ubuntuis changed. The other fetch jobs pass with the default buffer pool.Type of change
How Has This Been Tested?
Locally, against the
mysqlimage started with its defaults (128MB buffer pool):SET GLOBAL innodb_buffer_pool_size = 268435456throughdocker exec, confirmed the pool was resized to 256MB (the workflow writes the same value as256 * 1024 * 1024; confirmed that the expression is accepted and yields 268435456)./gost fetch --dbtype mysql ... ubuntuthen completed: 81,210 CVEs inserted, exit code 0information_schema.innodb_trx.trx_lock_memory_bytesgrew linearly with the inserted rows and peaked at about 120MB (about 55 million rows modified in the transaction)actionlint .github/workflows/*.ymlpasses.Checklist:
You don't have to satisfy all of the following.
make fmtmake testIs this ready for review?: YES