Skip to content

[Bug]: CPU and RAM usage continuously increase and reach 100% during vLEI issuance/revocation long-run test #450

Description

@qviweb

Version

0.3.0

Environment

No response

Expected behavior

Description

We are building a vLEI system using KERIA. During a 48-hour long-run load test involving vLEI issuance and revocation, we observed that both CPU and RAM usage on the KERIA server continuously increase (leak) until they hit 100%.

Even when upgrading the AWS instance specifications, the increase only slows down but the pure upward trend remains unchanged. This behavior might be related to the escrow timeout or incomplete message deletion issues mentioned in #232.

Environment / Specs

  • Cloud Provider: AWS (The issue occurs regardless of instance size, though larger specs slow down the accumulation rate)

Expected Behavior

Memory and CPU resources should be properly released after vLEI issuance/revocation operations are completed, preventing indefinite growth over long periods. Incomplete multisig escrows or processed messages should be deleted or timed out properly.

Actual behavior

  • Resource Leak: Both CPU and RAM usage increase continuously during the test.
  • CPU 100% Impact: Once the CPU usage reaches 100%, API calls and processing requests frequently fail with errors.
  • RAM 100% Impact: Once the memory usage hits 100%, the KERIA process gets terminated (likely due to OOM / Out of Memory killer).
  • After Restart: When the KERIA process automatically restarts, the CPU and RAM usage are reset back to normal levels.

Steps to reproduce

  1. Prerequisites: Passcodes for QVI and QVI vLEIs are already issued beforehand.
  2. Setup: Create 3 new LAR (Legal Accountability Representative) personas, including their passcodes.
  3. Execution (vLEI Operations):
    • Issue the following vLEIs: LE, OOR Auth, OOR, ECR Auth, and ECR.
    • Revoke all the issued vLEIs right after.
  4. Test Schedule: Run the above scenario within a 2-hour window, wait for a 2-hour idle interval, and then repeat it sequentially.
  5. Duration: Conduct this loop as a long-run test for 48 hours.

Activity

  1. iFergal commented on Jul 24, 2026

    @iFergal
    Collaborator

    @qviweb Have you configured the tocks, particularly for the escrower?

    Seems like something is getting stuck in escrow though, do debug logs show what's still in escrow after each scenario interval?

  2. kentbull commented on Oct 7, 2026

    @kentbull
    Collaborator

    To Fergal's point, the recent release 0.4.1 includes a comprehensive tock configuration feature where 100% of long running tasks (Doers/DoDoers/doified functions) now support configuring their scheduler recurrence via a custom tock setting, as detailed in #465.

    While a flame graph will likely reveal where the CPU is going, you can run experiments with different tock settings to see what brings your CPU or RAM usage down.

    Sometimes OOBI re-resolution loops can occupy a lot of CPU time. There are likely OOBI resolution or other escrow timeouts that are not properly expiring work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions