THESIS_CONTEXT.md

Thesis Context

Working project identity

System name: SignBridge

Institutional context: Camarines Sur Polytechnic Colleges, College of Computer Studies.

Application domain: Two-way communication at LGU service counters involving Deaf or hard-of-hearing citizens and hearing personnel.

Prototype objective: Recognize dynamic FSL gestures as text and provide a reverse path from typed/spoken-language content to FSL video output.

This document separates facts verified from the repository from proposed thesis statements. It is not a substitute for the approved manuscript, ethics documents, participant records, or final experimental report.

Specific objectives represented in the working methodology

The working chapter3_revision.md proposes that the study:

1. Train a deep-learning model using MediaPipe landmarks and LSTM networks to recognize dynamic FSL gestures used in LGU transactions.
2. Develop a prototype that converts spoken Tagalog into FSL video output for two-way interaction.
3. Evaluate classification performance, operational robustness, and user-centered usability/readiness.

The current code implements landmark-based classification, typed text/gloss clip assembly, on-demand text-to-speech, and local Faster Whisper Large-v3-Turbo microphone transcription. It preserves the original Filipino, English, or Taglish transcript and performs a separate speech-to-English pass for the editable sign-generation text when needed. This implementation fact is not evidence of transcription accuracy; that still requires formal evaluation.

Verified technical facts

As observed on 2026-09-19:

• The recognition input contract is 30 frames × 162 features.
• The 162 features represent 21 landmarks per hand plus 12 selected upper-body pose landmarks, each with x/y/z coordinates.
• MediaPipe Task API hand and pose models are used rather than the legacy Holistic API.
• The classifier contains LSTM layers of 64, 128, and 256 units, followed by Dense(128), Dropout(0.5), and a softmax head.
• Stage 2 freezes the first two LSTM layers and fine-tunes the third LSTM, Dense(128), and FSL output head.
• The ASL metadata has 94,477 rows, 250 signs, and 21 participant IDs.
• The current FSL train.csv has 2,360 rows, 159 unique labels, and 12 categories.
• The current FSL labels.csv has 159 rows and 159 unique labels.
• The legacy cached FSL augmented feature array observed during the 2026-09-22 audit has shape (11,790, 30, 162) and its saved encoder contains 159 classes. At the former 5× expansion, this represents 2,358 successfully processed source sequences, two fewer than the CSV row count. This legacy cache predates the leakage-safe source split and must not be used to report the final model.
• The cached ASL feature array has shape (94,477, 30, 162) and its saved encoder contains 250 classes.
• Cached ASL and FSL arrays, trained ASL/FSL Keras models, MediaPipe task models, a clip index, and generated videos are present locally.
• The application is a PyQt5 desktop GUI with a shared camera thread and separate workspace and Deaf-user windows.
• Incoming speech transcription uses a local multilingual Faster Whisper Large-v3-Turbo model with runtime downloads disabled, NVIDIA FP16 preference, and CPU float32 recovery.
• The repository contains classification helpers and an offline speech WER/CER/latency evaluator, but no automated test suite or consolidated experimental-results report.

Dataset reconciliation required

The 2026-09-29 curated-recording preparation is a new proposed experiment, not a correction of the earlier reported model score. Its current folder audit found 1,407 physical videos in the 70 retained class folders, 1,404 distinct byte contents, 235 original-test contents reserved, and 1,169 unique training-pool videos. Of those, 240 were not listed in either original CSV and need FSL label/quality review. See the curated run guide and its generated config/lgu_dataset/curated_report.json. These figures are video/manifest counts, not extracted landmark samples, augmented samples, or model results. This changes the source pool and the way original-test content is reserved; new evaluation must identify the exact manifest and new checkpoint. Signer independence remains unverified.

The 2026-09-28 SHA-256 audit of the current 929-row LGU profile found 15 duplicate-content groups (36 rows), with eight groups (21 rows) crossing its saved train/validation/test split. The previously reported 92.86% held-out source-path score has not been recomputed after content deduplication and may be inflated. The original train/test manifests also contain exact copies across them and some conflicting labels. See docs/LIVE_RECOGNITION_AUDIT.md and run python evaluation/audit_fsl_sources.py to reproduce the counts. No raw video, manifest, model, or split was changed by this audit.

chapter3_revision.md currently describes a combined FSL corpus of 155 classes and 2,704 raw sequences. That statement does not match the current CSV metadata of 159 unique labels and 2,360 rows.

Do not silently replace one set of figures with the other. Before updating the thesis manuscript, determine:

• Which CSV version corresponds to the trained models/fsl_finetuned.keras artifact.
• Whether every CSV row points to an existing, readable video.
• Which labels are Mendeley-source, custom LGU, and fingerspelling classes.
• Whether duplicate recordings or aliases exist.
• How many sequences successfully yield landmarks; this may be lower than CSV row count.
• The actual cached sample and class counts used for training.
• Whether augmentation produced exactly five samples per successfully processed video.
• Whether splits were performed by clip or by signer; clip-level random splitting may leak signer-specific information.

Dataset figures in the final paper should be generated from a versioned manifest and linked to the exact model run.

The legacy cache demonstrates why these distinctions matter: 2,360 FSL manifest rows became 11,790 augmented samples, consistent with 2,358 successfully processed source sequences at 5× augmentation. The two skipped or failed rows should be identified during the leakage-safe cache rebuild before final reporting.

Methodology-to-code alignment issues

• The working chapter says Stage 2 uses balanced class weights, but src/model/train_fsl.py does not currently pass class_weight to model.fit.
• The working chapter cites 155 FSL output classes, but the model factory default is 105 and runtime training derives the actual head size from the loader. The saved encoder/model pair, not the default argument alone, determines the deployed class count.
• The working chapter describes spoken Tagalog-to-FSL. The runtime now provides an offline multilingual speech-to-English review step followed by the existing English gloss/clip pipeline, but its Tagalog transcription, translation, and resulting sign accuracy still require formal evaluation.
• ASL training now assigns complete participants to separate train/validation/test sets, and model selection uses validation rather than test data. Results from models trained before this correction remain legacy results and should not be presented as participant-independent.
• FSL training now splits source-video paths before augmentation and augments training only. The saved split manifest proves path separation, but byte-identical copies under different paths crossed the split. Content grouping and verified signer identifiers are needed before independent performance can be claimed.
• ASL, FSL, and live inference now share preprocessing version landmarks_v2_missing_mask. Missing landmark blocks remain zero after normalization. Models trained from the legacy caches are not directly comparable to models retrained with this corrected preprocessing.
• Model sidecars bind the preprocessing version and SHA-256 checksum to each newly trained Keras artifact. Existing models without sidecars retain the legacy real-time normalization path until they are deliberately retrained.
• Live inference now receives original-orientation camera frames and resamples a hand-bounded gesture to the existing (30, 162) input. The model, training split, and saved evaluation metrics are unchanged. Source-video replay is diagnostic only; Chapter 4 live accuracy requires new trials.
• Environmental robustness, minimum-hardware validation, SPED teacher validation, SUS, and scenario success testing are described as evaluation goals, but completed study data was not found in the repository.
• evaluation/metrics.py and evaluation/evaluate_speech.py provide calculations, not proof that a representative evaluation was executed.

Evaluation framework

2026-09-25 LGU vocabulary and context revision

The current original FSL training CSV was recomputed as 2,360 unique source paths, 159 classes and 12 categories. The scenario-focused manifest derives its vocabulary from the Deaf-side performed sequences of the 44 authored test cases and retains 929 existing paths, 70 classes and nine represented categories. It excludes all 89 unused classes and two unavailable rows from retained classes; raw recordings remain untouched. These are manifest/path counts, not newly extracted or augmented sample counts. See the cleaning audit and rationale.

This is a change in vocabulary scope requiring a new FSL training run and separate evaluation; existing full-vocabulary scores cannot represent the reduced model. The architecture, landmark contract, normalization, confidence threshold and split ratios are unchanged. Manifest-specific caches and separate artifact paths prevent mixing the reduced and original experiments.

The hidden tree is currently an authored prior, not a learned model of actual LGU conversations. An importer supports verified training-only sequences when collected. The current isolated corpus has no conversation order labels. Context adds one stability observation for unexpected visual winners without changing their class or permitting low-confidence output. This changes runtime acceptance behavior and may increase latency; neither accuracy improvement nor computational efficiency has been established. A-Z assembly, repeated-sign release gating and the 1.2-second settling delay also require end-to-end testing.

The 44 conversation scripts are authored development/scenario instruments with no participant results. They now define the classifier vocabulary, so success on those same scripts measures scoped functional performance and must not be represented as independent evidence of generalization. Keep actual Chapter 4 trial responses out of sequence-prior training; collect signer-separated, held-out variations and compare context on/off using the same model and preprocessing. Measure false acceptance, out-of-branch signs, spelling/repeated-letter accuracy, task success and latency including the configured delay. Focused code checks and a GUI smoke test do not establish linguistic correctness or deployment readiness.

2026-09-26 ASL and LGU FSL training run

The corrected ASL model used 94,477 stored landmark sequences from 250 classes, split by participant into 63,549 train, 12,815 validation and 18,113 test sequences. Its held-out ASL test accuracy was 42.30%. The 70-class LGU FSL run extracted all 929 manifest videos, split them by source into 650 train, 139 validation and 140 test videos, and made four additional training-only variants per train video. The saved FSL checkpoint achieved 130/140 correct source-path test classifications (92.86%). The later SHA-256 audit found exact-content duplicates across the FSL split, so that score is not a clean independent estimate. Its metadata records signer_disjoint: false; signer-independent and live-conversation accuracy remain unmeasured. These numbers come from the saved ASL sidecar, FSL split manifest, FSL training log, and FSL test metrics, not from the Chapter 4 scripts. See the model-learning explanation.

When results are produced, preserve the following with each run:

• Git commit and working-tree state.
• Dataset manifest/version and label map.
• Train/validation/test split files and random seeds.
• Participant/signer separation strategy.
• Model architecture, hyperparameters, and saved artifact checksum.
• Hardware, OS, Python, TensorFlow, and MediaPipe versions.
• Accuracy, macro and weighted precision/recall/F1, and confusion matrix.
• End-to-end latency distribution, not only a single average.
• Environmental-condition protocol and results.
• SUS responses and aggregation procedure.
• Human sign-accuracy validation protocol and annotator qualifications.
• Scenario definitions and task-success criteria.

Ethical and accessibility considerations

• Obtain the required consent and institutional approval before collecting or using participant video.
• Treat sign videos, faces, names, and participant metadata as potentially identifiable research data.
• Document retention, access, anonymization, and deletion procedures outside the source repository.
• Do not imply that ASL and FSL are interchangeable; ASL is used here only as a transfer-learning source.
• Include qualified FSL users or validators when judging sign correctness and comprehensibility.
• Store pseudonymous signer IDs rather than participant names in the training manifest. Keep consent records and the identity-to-pseudonym key outside the source repository and restrict access according to the approved protocol.
• Describe the prototype's vocabulary and non-manual-marker limitations clearly.
• Do not characterize the system as an interpreter replacement or production-ready public service without appropriate validation.

Source-of-truth order

For future work, use this precedence:

1. Approved thesis/ethics requirements for research intent and participant rules.
2. Versioned dataset manifest and experiment record for empirical counts/results.
3. Executed source code and saved configuration for implemented behavior.
4. docs/ARCHITECTURE.md and this document for maintained summaries.
5. chapter3_revision.md as working revision notes that must be reconciled before final use.