[gerashegalov]

Summary

• detect and stream-decompress .zstd Spark event-log segments from local paths, S3 materialization, and tar bundles
• preserve plain and LZ4 event-log handling while replacing the LZ4-only compression flag with an explicit codec
• ignore decoded JSON values that are not event objects instead of raising AttributeError
• preserve UTF-8 characters across decompressor chunks and reject event records larger than 64 MiB
• load zstandard only for .zstd logs so the source bundle still handles help, plain, and LZ4 paths without site packages
• add the zstandard runtime dependency for installed package usage
• isolate unreadable segments to their owning application and return classified incomplete API results instead of propagating decoder failures

Fixes #271

Testing

• python -m py_compile yarn-resource-cost/*.py
• python -m unittest discover -s yarn-resource-cost -p 'test*.py' on Python 3.10, 3.11, and 3.12 (60 tests each)
• minimum dependency validation with zstandard==0.23.0
• python yarn-resource-cost/package_yarn_job_cost.py --output-dir /tmp/yarn-resource-cost-dist
• standalone plain-log CLI and --help checks with site packages disabled
• mixed healthy/corrupt application isolation and retryability classification checks
• installed console error-handling check
• git diff --check

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported dependency-loading, UTF-8 boundary, and unbounded-record issues are fixed, and no new actionable failures remain.

Summary

This PR adds streaming support for Zstandard-compressed Spark event logs while preserving plain and LZ4 handling.

• Detects codecs for local files, S3-materialized segments, and tar members.
• Lazily loads zstandard so standalone plain/LZ4 usage remains dependency-independent.
• Preserves split UTF-8 characters and bounds decompressed event records at 64 MiB.
• Isolates compressed-stream errors by application and propagates retryability through the application API.
• Expands tests and packaging metadata for installed and standalone execution.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Event-log path] --> B{Input layout}
    B -->|Local file or directory| C[Open event segments]
    B -->|S3| D[Materialize segments locally]
    B -->|Tar bundle| E[Extract member streams]
    C --> F{Filename codec}
    D --> F
    E --> F
    F -->|Plain| G[Read byte chunks]
    F -->|.lz4| H[Decode LZ4Block chunks]
    F -->|.zstd| I[Stream Zstandard decompression]
    G --> J[Assemble bounded UTF-8 records]
    H --> J
    I --> J
    J --> K[Parse JSON event objects]
    K --> L[Application metadata]
    H -->|Read error| M[Record per-application error]
    I -->|Read error| M
    J -->|Oversized record| M
    M --> N[Propagate retryability and incomplete metrics]

Loading

_{Reviews (3) · Last reviewed commit: "Isolate event log read failures"}

[tgravescs]

Medium — unguarded errors abort the whole batch
yarn_job_cost_api.py:221 — read_event_log_metadata is called without the error guard that parse_yarn_logs below it has, but the new codec paths add several raise sites. A .zstd segment with a damaged frame header (partial S3 object, or a non-zstd file with a .zstd name) propagates zstd.ZstdError: Unknown frame descriptor out of calculate_emr_application_usage, contradicting its docstring promise that incomplete logs come back as complete=False, retryable=True. Same for the new ValueErrors (missing zstandard package, line-length limit). One bad segment takes down metadata for every app in the directory.

Can we add test for that as well?
tests for corrupt zstd file would be good as well.

[gerashegalov]

@tgravescs Addressed in e032112. Event-log reader failures now use a domain error with retryability: corrupt/truncated compressed data is retryable, while missing decoder support and the record-size limit are not. Metadata discovery catches failures per segment, attributes them to the application inferred from the event-log path, and continues processing healthy applications. calculate_emr_application_usage returns complete=False with the affected application’s warning and retryability instead of propagating the decoder exception. Added mixed healthy/corrupt directory coverage, corrupt S3 Zstandard API coverage, permanent-error classification coverage, and LZ4/Zstandard reader classification tests.