Proposal: Custom Project Roles

[maxgraustenzel-create]

Summary

Proposal to add custom project roles to Harbor, enabling system administrators to create roles with flexible permission combinations.

Related Issues

• #18124 (main issue)
• #18143, #21306, #12062, #8632, #1486 (related)

Implementation

• Repository: https://github.com/maxgraustenzel-create/harbor
• Branch: origin/18124-custom-role-feature
• Status: ~60% complete

Discussion

This proposal is ready for community review and discussion at the next community meeting.

Community Meeting

If wished, I could present this proposal and a demonstration of the role functionality at the next community meeting for feedback and discussion.

/kind proposal

[wy65701436]

Built-in roles must remain completely immutable to serve as a secure baseline.

[maxgraustenzel-create]

Ok for me

[chlins]

@maxgraustenzel-create Please update this part to align the conclusion(built-in role is immutable).

[wy65701436]

Migrating rbac_role.go's static policies into role_permission is cleaner, but we must protect against:

• High DB query volume during login/CLI handshakes.
• Missing indexes on role_permission combined with role_type and associated keys.

[chlins]

Migrating built-in role permissions from rbac_role.go into role_permission creates two sources of truth for the same roles. Please outline the workflow we should follow in the future. For example, if we need to add a new built-in role, should we add it to both the code and the database?

[maxgraustenzel-create]

My proposal is that there will be no more hardcoded role permissions in the rbac_role.go. The roles and permissions are defined exclusively inside the database. The corresponding datasets are inserted to the tables as part of the version update workflow.

The role table is extended with some attributes, and one of them is used to identify build-in roles that can't be modified (checked on frontend and backend). If future versions introduce new build-in roles, the corresponding data needs to be added to the DB as part of version update.

[chlins]

Please incorporate into the design how we will insert new built-in types into the database in future versions, whether through migration SQL or through code.

[wy65701436]

Sorry for the back and forth.

The proposal suggests session-scoped caching at login, with updates applying on the next login. While this is simple, it poses a severe security risk in enterprise, high-availability, or multi-replica Harbor setups:

• If a custom role's permission is revoked (e.g., a "Security Auditor" role has access to artifacts revoked), the user can continue executing actions because their session-scoped memory cache on their specific pod is stale.
• Logouts or token expirations are not immediate enough for compliance-critical revocations.

[maxgraustenzel-create]

For my understanding the same caching mechanism is used for the role assignement itself. The use case that somebody gets a critical role revoked while logged in is probably more likely and is managed in the same way.

[wy65701436]

Yeah, I get where you're coming from about role assignments having some cache lag today, but there’s a big difference here.

Right now, standard roles are hardcoded as compile-time constants in rbac_role.go. Since they’re locked in Go memory, they can never change while the server is running. But by moving these to the database for custom roles, we're introducing a fully dynamic state. We can’t really apply the same static caching assumptions to dynamic DB records, especially when a single role change can instantly impact thousands of users across multiple projects.

Since we want to ship this safely in v2.16.0, we should make sure changes to custom roles propagate across all Harbor instances without a massive lag.Instead of building a super complex sync or tracking lock system, maybe we can go with a lightweight compromise:

• Whenever an admin edits or deletes a custom role, we write a simple version tag or timestamp in Redis (like custom_roles:last_update).
• When evaluating permissions during a session, the evaluator does a quick check against that Redis timestamp. If the local cache is out of date, it lazily reloads the role definitions.

How does that sound to you?

[Vad1mo]

I would not use Redis here. This add another complexity dimension. A KISS approach would be if a custom role changes, we can invalidate all sessions, forcing users to log in again. Custom role change is something that will happen once or twice during the entire lifecycle of a custom_role. Once a role is defined, it is unlikely to ever change again. People (admins) who are capable of changing roles will not do it since they are unable to predict consequences for the users since they are disconnected from the user base.

[wy65701436]

maybe not invalidate all sessions, but the sessions that associate with the updated role.

[wy65701436]

cc @Vad1mo — based on the discussion in the last community meeting, @maxgraustenzel-create is leaning toward my Redis-based suggestion.

Since you had a different opinion, I asked @maxgraustenzel-create to sync with you as well. Let’s discuss together and decide which approach would be best to move forward with.

[Vad1mo]

Why I do not want this on Redis:

Putting authorization behind a per-request Redis round-trip (a version-key check is still a round-trip, or a local cache of the version key, which just moves the staleness question up one level) puts the most security-critical path behind an availability and latency dependency. We just got that exact failure mode:
goharbor/harbor#23335. The config cache's FetchOrSave Redis path, hit per request, piled up on a keyMutex, exhausted the DB connection pool, and hung core permanently. Authz per request is precisely that hot path. There have also been repeated attempts to move config caching off Redis to memory for the same traffic
reason (goharbor/harbor#19156, goharbor/harbor#19163, goharbor/harbor#19164). Adding a brand new Redis dependency for RBAC walks straight back into the problem those PRs were trying to escape.

[Vad1mo]

@maxgraustenzel-create Hi, could you please clarify the Harbor scenario you mentioned earlier? I just tested a specific use case with two users: admin and test. First, admin granted test permissions for Project A and Project B. Once test logged in, both projects were visible. Then, admin revoked these permissions and removed test from both projects. As a result, test immediately lost access to Projects A and B without needing to re-log in.

@chlins is right that role assignment is immediate.

What's frozen at login (stale until session refresh)

The thing chlins test never varied: user.GroupIDs. The session middleware rebuilds the user entirely from the session blob, with no DB re-read:

// session.go:Generate
userInterface := store.Get(req.Context(), "user")
user, ok := userInterface.(models.User) // GroupIDs comes straight from here
return local.NewSecurityContext(&user)

GroupIDs is set once at login (oidc/helper.go:494 for OIDC, ldap/ldap.go:194 for LDAP). So for a session-based login (UI, and OIDC web), if you remove a user from an LDAP/OIDC group, ListRoles's entity_id in (GroupIDs) clause still uses the old groups until the session is rewritten. Same applies to SysAdminFlag/AdminRoleInAuth, which IsSysAdmin() reads off the frozen
session user.

[chlins]

@maxgraustenzel-create Any updates or conclusion for this?

[maxgraustenzel-create]

the discussion below presents the several approaches regarding the caching. The proposed choice caches permissions on the node level for 1 second which allows to reduce the performance impact.

The proposed caching solution is configurable through environment variables, allowing to balance between performance and session coherence. The use case where 1 second of cache propagation between nodes is considered to long is covered through the cache environment variables. Those variables further allow to balance the work between redis and the DB.

[wy65701436]

do we consider?

• If a custom role includes ResourceRobot:ActionCreate, how do we prevent a user with this role from creating a robot account with permissions that exceed the user's custom role scope?
• A user with role-management permissions within a project must not be allowed to assign custom roles that contain permissions higher than their own assigned role.

[maxgraustenzel-create]

I have implemented mechanisms that prevent privilege escalation (on frontend and backend).

• Users can only create robots with less or equal permissions than they have. Where necessary i implemented a mapping between robot and role permissions.
• Users can only assign roles with less or equal permissions than they have.

[maxgraustenzel-create]

Options for the cache

• 0: -- no cache
• 1.X — session-scope: permissions cached per user session.
• 2.X — system-scope: permissions cached once per node, shared
  across all sessions and users.

Criteria

• Request time — added per-request latency on role-evaluating endpoints
  (measured vs upstream where available).
• Stale window — how long a role change can go unseen (cross-node propagation /
  revocation timeliness).
• Redis impact — does the approach put Redis on the authz hot path?

#	Approach (proposer)	Request time	Stale window	Redis impact	Verdict
0	DB every request (baseline)	all auth +14–56 %	none (always fresh)	none	Correct but too slow as default
1	session-scope cache
1.1	At login (initial, Max)	session ~0 · basic +14–56 %	until session ends	none	Withdrawn — long stale window
1.2	L1 session + opt-in L2 Redis (review)	session ~0 · basic +14–56 %	= L1 TTL; must be large (N sessions/node reload on expiry)	none by default; opt-in L2, but N reads/node on expiry (1/session)	Deferred — Harbor-wide perf follow-up
1.3	Login + session invalidation (Vad1mo)	session ~0 · basic +14–56 %	none (fresh or kickout)	(relogin spike)	Set aside — write-back race + reverse lookup
1.4	Session cache + Redis version-key (wy65701436)	session ~0 · basic +14–56 %	~1 s (on version bump)	read-path: version-key per request (+ reload spike)	Rejected — Redis on authz path (#23335)
2	system-scope cache
2.1	Redis-only shared cache (considered)	all auth: +1 Redis hop/request	~0 (delete on write)	read-path: GET every request	Not pursued — Redis on every request (cf. #19156/63/64)
2.2	L1 node + opt-in L2 Redis (chosen)	all auth ~0 (parity; 0 hops)	= L1 TTL (1s)	none by default; opt-in L2, 1 read/node on expiry	Implemented

[wy65701436]

We need to run a proper stress test with a sustained load of 500 requests, not just a single benchmark. We need accurate cost metrics, and our commercial deployments handle massive loads. Also, can you grab the CPU/Memory trends for the DB pod? I'm worried the current resource limits won't be enough after we bump Harbor to v2.16.

This data is essential for my evaluation of the solution.

[maxgraustenzel-create]

Can you please provide additional information about the required stresstest ? Is there already a sort of scenario to evaluate login performance that I shall reuse?

[wy65701436]

We can use the https://github.com/goharbor/perf to do the stress test. cc @chlins can you help?

[chlins]

@maxgraustenzel-create Yes, you can use https://github.com/goharbor/perf to benchmark some harbor API, please refer to the README for usage, and feel free to reach out me if you have any question.

[maxgraustenzel-create]

Hi Chlins and Wang, this are the comparative test results of the performance tests of the role feature.

Custom roles — performance

Builds compared: upstream · fork, permissions from DB query (cache off) · fork, permissions via cache (L1 in-proc + L2 Redis).

Cache behavior

• Warm-up: on the first permission check a background pass eagerly loads all roles + permissions into L1 (in-proc sync.Map) and L2 (Redis) — no cold-miss stampede.
• Invalidation on change: any role create/update/delete bumps a Redis version key → every node drops its L1 within ~1 s (cross-node), and the changed role's L2 entry is deleted.
• Invalidation on time: 30-min TTL on both L1 and L2 → self-heals out-of-band DB changes (e.g. direct SQL).

A cached entry is refreshed by (a) a role change anywhere (≤1 s) or (b) its 30-min TTL, whichever comes first.

Test setup

Bare-metal (32c / 31 GB), all services + Postgres + Redis as containers. Postgres tuned (max_connections=1024, shared_buffers=4 GB); DB snapshot restored and ANALYZEd before each scenario. Load: k6 (xk6-harbor), 500 VUs, 5000 iters/endpoint, session-cookie auth, warm-up pass per endpoint, 3 passes (mean reported). Metric: end-to-end request latency (ms); (%) = change vs upstream for the same endpoint. Upstream baseline is the exact commit the fork is based on, so the delta isolates the feature.

Admin workload — feature is free

A sysadmin skips the project role evaluator, so there is no role lookup to cache. All existing endpoints land within measurement noise of upstream, and cache vs no-cache is indistinguishable. (list-roles is a new endpoint, ~230 ms, with no upstream equivalent.)

Non-admin (project member) — the cache earns its keep

Every request evaluates the member's project role → a role_permission lookup. Here the cache matters. Pooled over built-in and custom roles (same result either way):

Role-evaluating endpoints (project-scoped)

Endpoint	upstream	from DB query	via cache
get-project	275	356 (+30 %)	275 (0 %)
list-project-members	213	299 (+40 %)	217 (+2 %)
list-repositories	323	398 (+23 %)	312 (−4 %)
get-repository	174	272 (+56 %)	185 (+7 %)
list-artifacts	641	732 (+14 %)	660 (+3 %)
list-artifact-tags	243	334 (+38 %)	246 (+1 %)

DB lookup: +14–56 % vs upstream; cache: upstream parity.

Endpoints with no per-request role lookup — unaffected

Endpoint	upstream	from DB query	via cache
login	275	270 (−2 %)	269 (−2 %)
get-v2	48	48 (+1 %)	47 (−2 %)
list-projects	941	907 (−4 %)	944 (0 %)
list-audit-logs	12852	10919 (−15 %)	12410 (−3 %)

Flat across all three — auth (login), registry token (get-v2), and the two list endpoints whose cost is the query/membership filter, not per-request role evaluation.

New endpoint

Endpoint	upstream	from DB query	via cache
list-roles	101 (404)	286 (+182 %)	283 (+179 %)

list-roles has no upstream equivalent (fast 404). On the fork it is ~+180 % with or without the cache, because the list path (ctl.List → populate) reads each role's permissions directly from the DB and bypasses the cache — new-endpoint cost, and a candidate for a future optimization (it is a management endpoint, not a hot read path).

Takeaway

The custom-roles feature is upstream-equivalent with the cache on. The cache is not a micro-optimization: for real project members it removes a 10–55 % per-request penalty on common read endpoints by eliminating the role-permission DB query. Admins never hit that path, so for them the feature is free regardless.

[wy65701436]

Based on my understanding, the upstream response/latency should equal the DB query. Why, then, are we seeing poor performance on the role-evaluation endpoints?

[maxgraustenzel-create]

Upstream has only the buildin roles that are hardcoded. The DB implementation queries the permissions on every request from the DB which leads to poor performance. The cache implementation uses a cache (two layers) with a ttl of 1 second in order to mediate this. The performance of the cached version are similar to the upstream.

[maxgraustenzel-create]

Based on the comment from Vadim I did a new implementation ... now the redis cache is disabled by default and only the in memory cach is working by default. However redis cache can be enabled through environment variables.

The results are similar for the two caching solutions (cache off = db query on every request):

Endpoint	upstream	cache off	L1 (default)	L1+L2
get-project	213	246 (+15.9%)	216 (+1.6%)	210 (−1.2%)
list-project-members	174	203 (+17.1%)	172 (−0.7%)	169 (−2.8%)
list-repositories	288	350 (+21.3%)	285 (−0.9%)	292 (+1.1%)
get-repository	149	184 (+23.2%)	147 (−1.5%)	146 (−2.3%)
list-artifacts	611	649 (+6.3%)	599 (−1.9%)	599 (−2.0%)
list-artifact-tags	193	246 (+27.2%)	194 (+0.5%)	195 (+0.8%)
overall mean	271	313 (+15.4%)	269 (−0.8%)	268 (−1.1%)

[chlins]

@maxgraustenzel-create I would like to know what happens in a high-availability scenario where Harbor is deployed with multiple core instances (pods). If we only enable memory caching, what happens when requests are routed to different pods with inconsistent memory caches?

[maxgraustenzel-create]

The memory cache has by default a TTL of 1 second. In a cluster setup, on the instances where the change happened the cache is updated directly, On the other instances the requests that arrive for permissions update the cache if it is older than 1 second. The cache is updated with the new value from the DB.

Two environment variables Permission_cache_L1_TTL and Permission_cache_L2_TTL allow to further fine tune this behavior. Negative values in the environment variables disable the cache.

• L1=1s, L2=-1 --> default config described above --> 1s stale, no redis
• L1=-1, L2=-1 --> no cache, db query for permissions on every request --> no stale, heavy on DB
• L1=-1, L2=30m --> use redis on every request, redis is invalidated on changes and aligned every 30min with database as autohealing --> no stale, heavy on Redis
• L1=1s, L2=30m --> two levels of cache --> 1s stale, light on redis and on db

[chlins]

I think we can remove this as there's an unanimous conclusion from above.

[chlins]

One more question here: if Harbor has previously issued a v2 token to a Docker client to pull an image, and this token has a 30-minute expiration, but the user's role changes during these 30 minutes, is the old token still valid? Is there a potential risk of permission leakage here?

[maxgraustenzel-create]

From my understanding:

The v2 JWT tokens encode the access rights (repository:name:actions) for specific registry actions.

Harbor's authorization is evaluated only at token-issue time and frozen into the signed token's access claim. After issue, the token is validated statelessly by the registry (distribution).

As a result, changes made in Harbor afterward — removing a user from a project, downgrading their role, or editing the permissions of a (custom or built-in) role — are not reflected in tokens already issued.

They take effect on the next token negotiation, i.e. after the current token expires (token_expiration, default 30 min). If tighter bounds are needed for a sensitive deployment, lowering token_expiration shrinks the window.

Docker Registry v2 behavior is based on a standard and is identical across project removal, built-in role downgrades, and custom roles — custom roles neither introduce nor worsen it.

[chlins]

Understood. I know this isn't specific to the new feature, just wanted to confirm if the underlying behavior had changed at all. Thanks for clarifying!

[chlins]

@maxgraustenzel-create Please update the proposal to align with our recent discussions and resolve the comment threads that have reached a consensus, so we can move this forward.

[Vad1mo]

My proposal:

1. Land the implementation with the same per-request DB evaluation Harbor already uses. No Redis, no caching.
2. Benchmark it with goharbor/perf (the login/CLI handshake path wy65701436 flagged on line 195 about query volume and role_permission indexing is the right thing to measure).
3. If the numbers miss the requirement, then we add a targeted performance improvement, caching or otherwise, backed by the benchmark that justifies it.

Caching should be a measured response to a measured problem, not a starting assumption. Let's get the correct, simple version in, measure it, and optimize only what the data tells us to.