NeptuneSearch 🪐📽️

NeptuneSearch is an AI-powered semantic video search application. It allows users to upload videos, automatically processes and extracts frames, and then enables natural language semantic search over the video content.

The project is divided into a dynamic frontend built with Svelte and Tailwind CSS, and a backend powered by FastAPI, PyTorch, Transformers, and FAISS.

Features ✨

• Video Upload & Processing: Upload videos which are automatically decoded and processed frame-by-frame using OpenCV.
• AI-Powered Semantic Search: Uses vision-language models to understand video content.
• Fast Similarity Search: Utilizes FAISS to rapidly find the most relevant video segments based on your natural language queries.
• Smart Segment Merging: Intelligently groups matching consecutive frames into cohesive video segments for playback.

Project Structure 📁

NeptuneSearch/
├── backend/                  # Python FastAPI Application
│   ├── api/                  # API routes and database/response models
│   ├── core/                 # AI & Video Processing logic
│   │   ├── feature_extractor.py # AI model feature extraction (e.g. CLIP)
│   │   ├── search_engine.py  # FAISS similarity search implementation
│   │   ├── segment_merger.py # Groups continuous semantic frames
│   │   └── video_processor.py# OpenCV video frame extraction
│   ├── config/               # Application settings and configuration
│   ├── utils/                # Helper functions & system loggers
│   ├── app.py                # FastAPI server entry point
│   └── requirements.txt      # Python backend dependencies
└── frontend/                 # Svelte + Vite Application
    ├── src/                  # Svelte components and assets
    ├── index.html            # Main HTML template
    ├── package.json          # Node dependencies
    └── vite.config.js        # Vite bundler configuration

Setup & Installation 🚀

Backend Setup

1. Navigate to the backend directory:

   cd backend

2. Create and activate a Python virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows use: venv\Scripts\activate

3. Install the required dependencies:

   pip install -r requirements.txt

   (Note: The project requires PyTorch and installs the CUDA 12.8 version by default for GPU acceleration)

4. Start the FastAPI server:

   python app.py

   The backend server will start on the configured host and port, providing API endpoints and serving extracted static frames.

Frontend Setup

1. Navigate to the frontend directory:

   cd frontend

2. Install the Node.js dependencies:

   npm install

3. Start the Vite development server:

   npm run dev

   The frontend application will be accessible in your browser (typically at http://localhost:5173).