Log Retrieval and Analysis System Architecture

Team: NullCrew

• Gaurav Chaurasia
• Shruti Jain
• Balraj Ashwath

High Level Design

System Overview

The Log Retrieval and Analysis System is a comprehensive enterprise solution designed to collect, process, analyze, and visualize log data from various sources within the ServiceNow instance. The architecture follows a multi-layered approach with specialized agents for log collection, centralized processing, and AI-powered analysis.

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                 User Interface Layer                                    │
│                                                                                         │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐    ┌──────────────┐  │
│  │  Analysis       │    │  Log Retrieval  │    │  Real-time      │    │  Admin       │  │
│  │  Dashboard      │    │  Interface      │    │  Monitoring     │    │  Console     │  │
│  └─────────────────┘    └─────────────────┘    └─────────────────┘    └──────────────┘  │
└───────────────────────────────────┬─────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                   API Layer                                             │
│                                                                                         │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐    ┌──────────────┐  │
│  │  Log Retrieval  │    │  Analysis       │    │  Reporting      │    │  Admin       │  │
│  │  API            │    │  API            │    │  API            │    │  API         │  │
│  └─────────────────┘    └─────────────────┘    └─────────────────┘    └──────────────┘  │
└───────────────────────────────────┬─────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                 Service Layer                                           │
│                                                                                         │
│  ┌─────────────────────────────────────────────────────────────────────────────────┐    │
│  │                         Orchestration Service                                   │    │
│  └─────────────────────────────────────────────────────────────────────────────────┘    │
│                                                                                         │
│  ┌──────────────┐   ┌───────────────┐   ┌──────────────────┐    ┌───────────────────┐   │
│  │ Log          │   │ Log           │   │ Analysis         │    │ Notification      │   │
│  │ Collection   │   │ Processing    │   │ Engine           │    │ Service           │   │
│  │ Service      │   │ Service       │   │                  │    │                   │   │
│  └──────┬───────┘   └───────┬───────┘   └────────┬─────────┘    └───────────────────┘   │
└─────────┼─────────────────────────────────┬──────┼──────────────────────────────────────┘
          │                                 │      │
          ▼                                 ▼      ▼
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────────┐
│  Log Collection Layer   │  │   Processing Layer      │  │    Analysis Layer           │
│                         │  │                         │  │                             │
│  ┌─────────────────┐    │  │  ┌─────────────────┐    │  │  ┌─────────────────────┐    │
│  │ MID Log         │    │  │  │ LogAnalyzer     │    │  │  │ Pattern             │    │
│  │ Retriever Agent │    │  │  │                 │    │  │  │ Recognition Engine  │    │
│  └─────────────────┘    │  │  │ ┌─────────────┐ │    │  │  └─────────────────────┘    │
│                         │  │  │ │ExtractZip   │ │    │  │                             │
│  ┌─────────────────┐    │  │  │ │Content      │ │    │  │  ┌─────────────────────┐    │
│  │ Node Log        │    │  │  │ └─────────────┘ │    │  │  │ AI-based Error      │    │
│  │ Retriever Agent │    │  │  │ ┌─────────────┐ │    │  │  │ Analysis Engine     │    │
│  └─────────────────┘    │  │  │ │ProcessFiles │ │    │  │  └─────────────────────┘    │
│                         │  │  │ └─────────────┘ │    │  │                             │
│  ┌─────────────────┐    │  │  └─────────────────┘    │  │  ┌─────────────────────┐    │
│  │ System Log      │    │  │                         │  │  │ Correlation         │    │
│  │ Retriever Agent │    │  │  ┌─────────────────┐    │  │  │ Engine              │    │
│  └─────────────────┘    │  │  │ File            │    │  │  └─────────────────────┘    │
│                         │  │  │ Chunking        │    │  │                             │
│  ┌─────────────────┐    │  │  │ Service         │    │  │  ┌─────────────────────┐    │
│  │ Custom Log      │    │  │  └─────────────────┘    │  │  │ Recommendation      │    │
│  │ Retriever Agent │    │  │                         │  │  │ Engine              │    │
│  └─────────────────┘    │  │                         │  │  └─────────────────────┘    │
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────────┘
          │                            │                             │
          └────────────────┬───────────┴─────────────────┬───────────┘
                           ▼                             ▼
             ┌─────────────────────────┐  ┌─────────────────────────────┐
             │  Data Storage Layer     │  │  Caching Layer              │
             │                         │  │                             │
             │  ┌─────────────────┐    │  │  ┌─────────────────────┐    │
             │  │ Log Bundle      │    │  │  │ Processed Logs      │    │
             │  │ Storage         │    │  │  │ Cache               │    │
             │  └─────────────────┘    │  │  └─────────────────────┘    │
             │                         │  │                             │
             │  ┌─────────────────┐    │  │  ┌─────────────────────┐    │
             │  │ Log File        │    │  │  │ Analysis Results    │    │
             │  │ Details Storage │    │  │  │ Cache               │    │
             │  └─────────────────┘    │  │  └─────────────────────┘    │
             │                         │  │                             │
             │  ┌─────────────────┐    │  │                             │
             │  │ Analysis        │    │  │                             │
             │  │ Results Storage │    │  │                             │
             │  └─────────────────┘    │  │                             │
             └─────────────────────────┘  └─────────────────────────────┘

Component Details

1. User Interface Layer

• Analysis Dashboard: Visualizes log analysis results, provides insights, and identifies patterns or anomalies.
• Log Retrieval Interface: UI for users to request logs from specific sources, time periods, or based on transactions.
• Real-time Monitoring: Live view of log processing and analysis activities.
• Admin Console: For system configuration, agent management, and performance monitoring.

2. API Layer

• Log Retrieval API: Endpoints for requesting log collection from various sources.
• Analysis API: Interfaces for requesting and receiving log analysis results.
• Reporting API: Generates standardized or custom reports from analysis results.
• Admin API: Management and configuration endpoints.

3. Service Layer

• Orchestration Service: Coordinates the overall workflow between components.
• Log Collection Service: Manages the various retriever agents and aggregates logs.
• Log Processing Service: Handles extraction, parsing, and preparation of logs for analysis.
• Analysis Engine: Core service that routes logs to appropriate analysis components.
• Notification Service: Alerts users about completed analyses or detected critical issues.

4. Log Collection Layer

• MID Log Retriever Agent: Collects logs from MID servers for specific transactions.
• Node Log Retriever Agent: Retrieves logs from Node.js applications or searches for patterns.
• System Log Retriever Agent: Collects system-level logs.
• Custom Log Retriever Agent: Extensible framework for additional log sources.

5. Processing Layer

• LogAnalyzer: Core component that extracts and processes log files from ZIP attachments.
  • ExtractZipContent: Extracts files from ZIP archives.
  • ProcessFiles: Processes extracted files and creates records.
• File Chunking Service: Splits large log files into manageable chunks for processing.

6. Analysis Layer

• Pattern Recognition Engine: Identifies known patterns in logs.
• AI-based Error Analysis Engine: Uses machine learning to detect and classify errors.
• Correlation Engine: Connects related events across different log sources.
• Recommendation Engine: Suggests solutions based on detected issues.

7. Data Storage Layer

• Log Bundle Storage: Stores the original ZIP files containing logs.
• Log File Details Storage: Stores individual log files or chunks.
• Analysis Results Storage: Persists analysis outcomes, patterns, and recommendations.

8. Caching Layer

• Processed Logs Cache: Temporarily stores frequently accessed logs.
• Analysis Results Cache: Caches analysis results for improved performance.

Data Flow

1. User initiates a log retrieval request through the UI or API.
2. Orchestration Service dispatches the request to the appropriate Log Retriever Agents.
3. Agents collect logs and bundle them into ZIP archives.
4. LogAnalyzer extracts files from the ZIP and processes them.
5. Large files are chunked by the File Chunking Service.
6. Analysis Layer components process the logs to identify issues and patterns.
7. Results are stored and presented to the user via the Dashboard.

Scalability Considerations

• Asynchronous processing with message queues for log extraction and analysis tasks
• Horizontal scaling of retriever agents and analysis engines
• Database sharding for high-volume log storage
• Configurable retention policies for logs and analysis results
• Resource limits for CPU and memory-intensive operations

Security Measures

• Role-based access control for log viewing and analysis
• Encryption of sensitive log data at rest and in transit
• Audit trail for all log retrieval and analysis operations
• Data masking for PII and sensitive information in logs
• Token-based API authentication

Monitoring and Maintenance

• Health metrics for all components
• Performance monitoring for processing times
• Queue depth monitoring
• Error rate tracking
• Automated scaling triggers
• Log rotation and archiving