Skip to content

AGENGS.md

Latest commit

 

History

History
623 lines (517 loc) · 17.6 KB

AGENGS.md

File metadata and controls

623 lines (517 loc) · 17.6 KB
Raw

AGENGS.md - File Management System Implementation Guide

🎯 Project Overview

High-Performance, Secure File Management System

  • Framework: Next.js 16 (App Router + Turbopack)
  • Auth: ZITADEL (OIDC) via Auth.js v5
  • Storage: MinIO (S3-compatible) - Multi-Bucket Architecture
  • Caching: Redis (Upstash/Local)
  • Permission System: YAML-based RBAC/ACL with prefix matching
  • Runtime: Node.js 22+

📐 Architecture Decisions

Storage Strategy: Multi-Bucket

user-files/          → Personal home directories (home/${username}/*)
shared-files/        → Shared folders, projects (shared/*, projects/*)
temp-uploads/        → Staging area for uploads before final placement

Metadata Strategy: S3 + Redis (NO DATABASE)

  • S3 Object Metadata: Immutable file properties (owner, created_at, hash)
  • Redis Cache: Dynamic data (permissions, listings, sessions, audit logs)
  • Permission Config: YAML file in Git → Synced to Redis on deploy/schedule

Permission Model

  • 3 Levels: OWNER (RWD), EDITOR (RW), VIEWER (R)
  • Prefix-based: /projects/alpha/* matches all sub-paths
  • Inheritance: Child paths inherit parent permissions
  • Priority System: Higher priority wins (admin=100, role=50, user=80)
  • Home Guarantee: Every user has OWNER rights to home/${username}/*

⚠️ Next.js 16 Migration Notes

Breaking Changes from Next.js 15

Feature Next.js 15 Next.js 16
Request Interceptor middleware.ts proxy.ts
Auth Wrapper auth() in middleware auth() in proxy
searchParams searchParams: { } searchParams: Promise<{ }>
params params: { } params: Promise<{ }>
Default Bundler Webpack Turbopack
Node.js Version 18.17+ 20.9+ (recommended 22+)

Auth.js v5 Configuration (CSRF Fix)

// src/lib/auth.ts - Required for Next.js 16
export const authConfig: NextAuthConfig = {
  // ... providers

  // CRITICAL: Required for localhost development
  trustHost: true,

  // Cookie settings for proper CSRF handling
  cookies: {
    sessionToken: {
      name: `authjs.session-token`,
      options: {
        httpOnly: true,
        sameSite: "lax",
        path: "/",
        secure: process.env.NODE_ENV === "production",
      },
    },
    csrfToken: {
      name: `authjs.csrf-token`,
      options: {
        httpOnly: true,
        sameSite: "lax",
        path: "/",
        secure: process.env.NODE_ENV === "production",
      },
    },
  },
};

Server Actions for Auth (Recommended)

// src/app/auth/signin/actions.ts
"use server";

import { signIn } from "@/lib/auth";

export async function signInWithZitadel() {
  await signIn("zitadel", { redirectTo: "/explorer" });
}

Proxy Pattern (replaces middleware)

// src/proxy.ts - Next.js 16+
import { auth } from "@/lib/auth";
import { NextResponse } from "next/server";

export default auth((req) => {
  const isLoggedIn = !!req.auth?.user;
  // ... protection logic
});

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico|api/auth).*)"],
};

🗂️ Project Structure

file-management-system/
├── .env.local                          # Environment variables
├── permissions.yaml                    # Central permission configuration
│
├── src/
│   ├── types/
│   │   ├── index.ts                   # Core TypeScript types
│   │   ├── s3.types.ts                # S3-specific types
│   │   └── permissions.types.ts       # Permission types
│   │
│   ├── lib/
│   │   ├── redis.ts                   # Redis client singleton
│   │   ├── s3-client.ts               # MinIO/S3 client singleton
│   │   └── auth.ts                    # NextAuth configuration
│   │
│   ├── services/
│   │   ├── s3Service.ts               # Low-level S3 operations (multi-bucket)
│   │   ├── permissionService.ts       # RBAC/ACL engine (Redis + YAML)
│   │   ├── cacheService.ts            # Redis caching helpers
│   │   ├── auditService.ts            # Audit logging to Redis
│   │   └── syncService.ts             # YAML → Redis sync logic
│   │
│   ├── app/
│   │   ├── api/
│   │   │   ├── files/
│   │   │   │   ├── list/route.ts      # GET: List files/folders
│   │   │   │   ├── upload-url/route.ts # POST: Generate presigned upload URL
│   │   │   │   ├── download-url/route.ts # POST: Generate presigned download URL
│   │   │   │   ├── preview-url/route.ts # POST: Generate presigned preview URL (images/PDF)
│   │   │   │   ├── create-folder/route.ts # POST: Create folder
│   │   │   │   ├── rename/route.ts     # POST: Rename file/folder
│   │   │   │   └── delete/route.ts     # DELETE: Delete file/folder
│   │   │   ├── permissions/
│   │   │   │   └── sync/route.ts       # POST: Sync YAML to Redis (webhook)
│   │   │   └── auth/
│   │   │       └── [...nextauth]/route.ts # NextAuth handlers
│   │   │
│   │   ├── (dashboard)/
│   │   │   ├── layout.tsx              # Protected layout with auth
│   │   │   └── explorer/
│   │   │       └── [[...path]]/page.tsx # File explorer dynamic route
│   │   │
│   │   └── layout.tsx                  # Root layout
│   │
│   ├── components/
│   │   ├── explorer/
│   │   │   ├── FileExplorer.tsx        # Main explorer component
│   │   │   ├── Breadcrumb.tsx          # Path navigation
│   │   │   ├── FileList.tsx            # File/folder listing
│   │   │   ├── FilePreview.tsx         # Image/PDF preview modal
│   │   │   ├── FileItem.tsx            # Individual file/folder item
│   │   │   ├── UploadZone.tsx          # Drag-and-drop upload
│   │   │   ├── PermissionBadge.tsx     # Visual permission indicators
│   │   │   └── ContextMenu.tsx         # Right-click menu
│   │   │
│   │   └── ui/                         # Shadcn UI components
│   │       ├── button.tsx
│   │       ├── badge.tsx
│   │       ├── dialog.tsx
│   │       └── ... (other shadcn components)
│   │
│   └── proxy.ts                        # Auth proxy for protected routes (Next.js 16+)
│
├── Dockerfile                          # Multi-stage production build
├── .dockerignore                       # Docker build exclusions
├── .nvmrc                              # Node.js version (22)
│
├── scripts/
│   └── sync-permissions.ts             # CLI script to sync YAML → Redis
│
├── .github/
│   └── workflows/
│       └── sync-permissions.yml        # Auto-sync on permissions.yaml changes
│
└── package.json

🔐 Environment Variables

# .env.local

# NextAuth / Auth.js v5
NEXTAUTH_URL=http://localhost:3000
NEXTAUTH_SECRET=your-super-secret-key-min-32-chars
AUTH_TRUST_HOST=true

# ZITADEL OIDC
ZITADEL_ISSUER=https://your-instance.zitadel.cloud
ZITADEL_CLIENT_ID=your-client-id
ZITADEL_CLIENT_SECRET=your-client-secret

# MinIO (S3)
S3_ENDPOINT=http://localhost:9000
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET_USER_FILES=user-files
S3_BUCKET_SHARED_FILES=shared-files
S3_BUCKET_TEMP_UPLOADS=temp-uploads

# Redis
REDIS_URL=redis://localhost:6379
# OR for Upstash:
# UPSTASH_REDIS_REST_URL=https://your-instance.upstash.io
# UPSTASH_REDIS_REST_TOKEN=your-token

# App Config
MAX_FILE_SIZE_MB=100
ALLOWED_FILE_TYPES=.pdf,.jpg,.jpeg,.png,.gif,.doc,.docx,.xls,.xlsx,.zip
PRESIGNED_URL_EXPIRY_SECONDS=3600

📋 permissions.yaml Structure

# permissions.yaml
# This file is the source of truth for all access control
# Changes to this file trigger automatic sync to Redis

version: "1.0"
updated_at: "2024-02-02T10:00:00Z"

# Role-based permissions
roles:
  admin:
    priority: 100
    permissions:
      - path: "*"
        level: OWNER
        description: "Full system access"

  engineering:
    priority: 50
    permissions:
      - path: "shared/engineering/*"
        level: EDITOR
      - path: "projects/*"
        level: EDITOR
      - path: "shared/public/*"
        level: VIEWER

  marketing:
    priority: 50
    permissions:
      - path: "shared/marketing/*"
        level: EDITOR
      - path: "projects/campaigns/*"
        level: EDITOR
      - path: "shared/public/*"
        level: VIEWER

  employee:
    priority: 10
    permissions:
      - path: "shared/public/*"
        level: VIEWER

# User-specific overrides (higher priority than roles)
users:
  john@company.com:
    priority: 80
    permissions:
      - path: "projects/secret-alpha/*"
        level: OWNER
      - path: "shared/marketing/*"
        level: OWNER  # Override role permission

  jane@company.com:
    priority: 80
    permissions:
      - path: "projects/secret-alpha/*"
        level: EDITOR

# Default permissions for authenticated users
defaults:
  authenticated:
    permissions:
      - path: "shared/public/*"
        level: VIEWER

# Home directory is auto-granted (don't need to specify)
# Every user automatically gets OWNER rights to home/${username}/*

🔄 YAML Sync Workflow

Manual Sync (CLI)

npm run sync-permissions

Auto-sync on Git Push (GitHub Actions)

# .github/workflows/sync-permissions.yml
name: Sync Permissions to Redis

on:
  push:
    paths:
      - 'permissions.yaml'
    branches:
      - main

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - run: npm install
      - run: npm run sync-permissions
        env:
          REDIS_URL: ${{ secrets.REDIS_URL }}

Scheduled Sync (Cron)

// Can be triggered via Vercel Cron or external scheduler
// POST /api/permissions/sync
// Authenticates via API key in header

🏗️ Implementation Phases

Phase 1: Foundation (Priority) ✅ COMPLETED

  • Setup Next.js 16 project with TypeScript
  • Install dependencies (see below)
  • Create .env.local with all variables
  • Setup MinIO (Docker)
  • Setup Redis (Docker)
  • Create permissions.yaml with initial config
  • Define all TypeScript types
  • Configure Node.js 22 via .nvmrc

Phase 2: Core Services

  • lib/redis.ts - Redis connection
  • lib/s3-client.ts - MinIO client
  • services/s3Service.ts - Multi-bucket wrapper
  • services/cacheService.ts - Redis helpers
  • services/syncService.ts - YAML parser + Redis sync
  • services/permissionService.ts - RBAC engine
  • services/auditService.ts - Audit logging

Phase 3: Authentication ✅ COMPLETED

  • lib/auth.ts - Auth.js v5 + ZITADEL OIDC config
  • Extend session with roles and username
  • proxy.ts - Protected route guards (Next.js 16 proxy pattern)
  • Server Actions for signin (app/auth/signin/actions.ts)
  • Test auth flow end-to-end with ZITADEL

Phase 4: API Routes

  • /api/files/list - List files with permission check
  • /api/files/upload-url - Generate presigned upload URL
  • /api/files/download-url - Generate presigned download URL
  • /api/files/create-folder - Create folder
  • /api/files/rename - Rename (copy + delete)
  • /api/files/delete - Delete with permission check
  • /api/permissions/sync - Webhook for YAML sync

Phase 5: UI Components

  • Setup Shadcn UI
  • FileExplorer.tsx - Main container
  • Breadcrumb.tsx - Path navigation
  • FileList.tsx - File/folder grid/list view
  • FileItem.tsx - Individual item with context menu
  • UploadZone.tsx - Drag-and-drop
  • PermissionBadge.tsx - Visual indicators
  • Error boundaries and loading states

Phase 6: Advanced Features

  • File preview (images, PDFs) - Internal preview without download
  • Multipart upload for large files (>5MB)
  • Search functionality (Redis FT.SEARCH)
  • Batch operations (multi-select)
  • Folder download (zip)
  • Share links with expiry (presigned URLs)

Phase 7: Monitoring & Operations

  • Rate limiting (Upstash Rate Limit)
  • Audit log viewer UI
  • Redis cache invalidation strategy
  • Error tracking (Sentry)
  • Performance monitoring
  • Docker Compose for local dev

📦 Dependencies

{
  "dependencies": {
    "next": "^16.1.0",
    "react": "^19.2.0",
    "react-dom": "^19.2.0",
    "next-auth": "^5.0.0-beta.30",
    "@auth/core": "^0.41.0",
    
    "@aws-sdk/client-s3": "^3.967.0",
    "@aws-sdk/s3-request-presigner": "^3.967.0",
    
    "ioredis": "^5.9.2",
    
    "zod": "^4.3.6",
    "yaml": "^2.8.2",
    
    "@radix-ui/react-context-menu": "^2.2.16",
    "@radix-ui/react-dialog": "^1.1.15",
    "@radix-ui/react-dropdown-menu": "^2.1.16",
    "@radix-ui/react-slot": "^1.2.4",
    "@radix-ui/react-tooltip": "^1.2.8",
    "class-variance-authority": "^0.7.1",
    "clsx": "^2.1.1",
    "tailwind-merge": "^3.4.0",
    "lucide-react": "^0.563.0"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "typescript": "^5",
    "tailwindcss": "^4",
    "@tailwindcss/postcss": "^4",
    "eslint": "^9",
    "eslint-config-next": "16.1.6"
  },
  "engines": {
    "node": ">=22.0.0"
  }
}

🔑 Key Design Patterns

Permission Resolution Algorithm

function resolvePermission(user, path) {
  // 1. Check home directory (auto OWNER)
  if (path.startsWith(`home/${user.username}/`)) {
    return 'OWNER';
  }
  
  // 2. Collect all matching rules (user + roles)
  const matches = [];
  
  // User-specific rules
  matches.push(...user.permissions.filter(p => matchesPrefix(path, p.path)));
  
  // Role-based rules
  user.roles.forEach(role => {
    matches.push(...role.permissions.filter(p => matchesPrefix(path, p.path)));
  });
  
  // 3. Sort by priority (highest first)
  matches.sort((a, b) => b.priority - a.priority);
  
  // 4. Return highest priority match
  return matches[0]?.level || null;
}

Redis Caching Strategy

// Cache Key Patterns
'perm:{userId}:{path}'          // TTL: 5 min
'listing:{bucket}:{prefix}'      // TTL: 30 sec
'session:{sessionId}'            // TTL: 1 hour
'config:permissions'             // TTL: none (manual invalidate)
'audit:{userId}'                 // Sorted Set, 30 days

Presigned URL Gatekeeper

async function generatePresignedUrl(user, path, action) {
  // 1. Check permission
  const permission = await permissionService.check(user, path);
  
  if (action === 'download' && !['VIEWER', 'EDITOR', 'OWNER'].includes(permission)) {
    throw new Error('FORBIDDEN');
  }
  
  if (action === 'upload' && !['EDITOR', 'OWNER'].includes(permission)) {
    throw new Error('FORBIDDEN');
  }
  
  // 2. Generate presigned URL
  const url = await s3Service.getPresignedUrl(path, action, 3600);
  
  // 3. Audit log
  await auditService.log(user, action, path);
  
  return url;
}

🚀 Quick Start Commands

# 1. Clone and install
git clone <repo-url>
cd file-management-system
npm install

# 2. Setup environment
cp .env.example .env.local
# Edit .env.local with your values

# 3. Start dependencies (Docker)
docker compose up -d  # MinIO + Redis

# 4. Initialize buckets
npm run init-buckets

# 5. Sync permissions to Redis
npm run sync-permissions

# 6. Run development server
npm run dev

# 7. Open browser
open http://localhost:3000

🧪 Testing Checklist

  • User can login via ZITADEL
  • Home folder automatically accessible
  • Shared folder access based on role
  • Upload file to allowed path
  • Upload blocked to restricted path
  • Download file with VIEWER permission
  • Rename/delete blocked for VIEWER
  • Permission badge shows correct level
  • Breadcrumb navigation works
  • Cache invalidates on file operations
  • Audit log records all actions
  • YAML sync updates permissions live
  • Presigned URLs expire correctly
  • Multi-bucket routing works

📝 Notes for Developers

Security Checklist

  • ✅ Never expose S3 credentials to client
  • ✅ All S3 operations server-side only
  • ✅ Validate file types before presigned URL
  • ✅ Check permissions before every operation
  • ✅ Audit log all file access
  • ✅ Rate limit API endpoints
  • ✅ Sanitize file names (no ../, no absolute paths)
  • ✅ Validate path prefixes match user permissions

Performance Tips

  • Cache permission checks (5 min TTL)
  • Cache folder listings (30 sec TTL)
  • Use Redis pipelining for bulk ops
  • Implement lazy loading for large folders
  • Use React Server Components for listings
  • Debounce search queries

Common Pitfalls

  • ❌ Don't check permissions client-side only
  • ❌ Don't trust client-provided paths
  • ❌ Don't skip audit logging
  • ❌ Don't cache presigned URLs too long
  • ❌ Don't forget to invalidate cache on mutations

🎯 Current Status

Phase: Phase 3 Complete (Authentication) Completed:

  • ✅ Next.js 16 + Turbopack setup
  • ✅ Auth.js v5 + ZITADEL OIDC integration
  • ✅ Server Actions for authentication
  • ✅ Docker + Docker Compose configuration
  • ✅ MinIO + Redis local development
  • ✅ TypeScript types defined
  • ✅ Core services implemented

Next Steps:

  1. Complete API routes (Phase 4)
  2. Build UI components (Phase 5)
  3. Add file preview & search (Phase 6)

Development: npm run dev (requires Node.js 22+) 🚀