SignBridge

SignBridge is a Python desktop thesis prototype for two-way communication between Deaf or hard-of-hearing citizens and hearing personnel at local government unit (LGU) service counters. The application combines real-time Filipino Sign Language (FSL) recognition with text display, optional text-to-speech, and text/gloss-to-sign video assembly.

The project is associated with Camarines Sur Polytechnic Colleges, College of Computer Studies.

Current capabilities

• Capture one shared webcam feed for the workspace and Deaf-user windows.
• Extract 21 landmarks from each hand and 12 upper-body pose landmarks with the MediaPipe Task API.
• Classify 30-frame landmark sequences with a three-layer LSTM model.
• Accumulate accepted FSL predictions into a displayed sentence.
• Send typed or recognized text to the Deaf-user display.
• Select a microphone, monitor its live level, and transcribe Filipino, English, or Taglish with a local Faster Whisper Large-v3-Turbo model.
• Preserve the original speech transcript and separately produce editable English text before optionally generating an FSL video.
• Speak text on demand using Filipino gTTS with an offline pyttsx3 fallback.
• Browse the local FSL clip library and stitch selected signs into an MP4 with FFmpeg.
• Match complete known phrases and fingerspell unfamiliar words with A-Z clips.
• Use a hidden LGU sequence context aid without completing unperformed signs.
• Calculate classification metrics, confusion matrices, latency summaries, and SUS scores through evaluation utilities.

These are implemented prototype features, not evidence by themselves of model accuracy, usability, or deployment readiness. See Thesis Context for the distinction between verified repository facts and research claims that still require validation.

Repository layout

.
├── main.py                    # Application and training entry point
├── setup_clip_library.py      # Creates directories and clip index
├── requirements.txt           # Python dependencies
├── chapter3_revision.md       # Working methodology revision notes
├── src/
│   ├── data/                  # Landmark extraction, loaders, augmentation
│   ├── model/                 # LSTM architecture and training scripts
│   ├── inference/             # Real-time recognition and clip assembly
│   ├── gui/                   # PyQt5 windows, widgets, threads, bridge
│   └── tts.py                 # Legacy offline TTS helper
├── evaluation/                # Metrics and evaluation utilities
├── dataset/                   # Local raw ASL/FSL datasets (ignored by Git)
├── data/                      # Processed arrays and generated clips (ignored)
├── models/                    # Trained and MediaPipe models (ignored)
├── UI Guide/                  # Interface reference images
└── docs/                      # Architecture, research context, changelog

Requirements

• Windows is the primary desktop target represented by the current code.
• Python 3 with the packages in requirements.txt.
• A webcam for live capture.
• FFmpeg and FFprobe available on PATH for clip generation and verified stitching.
• Internet access for Filipino gTTS; pyttsx3 is the offline fallback.
• A one-time download of the approximately 1.55 GB local Faster Whisper model; speech recognition and translation are offline afterward.
• The local datasets and trained model artifacts for training or inference.

The existing local virtual environment was observed using Python 3.13. Recreate dependencies from requirements.txt rather than relying on that machine-specific environment.

TensorFlow 2.11 and later do not support CUDA GPU training on native Windows. Use WSL2 or another TensorFlow-supported GPU environment for accelerated training.

Setup

From the project root in PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python setup_speech_model.py
python setup_clip_library.py

setup_clip_library.py expects the FSL metadata and videos under dataset/FSL/ and writes data/clip_library/index.json.

setup_speech_model.py downloads the multilingual Faster Whisper Large-v3-Turbo model to models/faster-whisper-large-v3-turbo/. The application loads this exact directory with runtime downloads disabled. It prefers NVIDIA CUDA FP16 and recovers on CPU float32 if CUDA inference is not available. Speech recognition and speech-to-English translation work without internet access after setup.

Run

# Launch both connected windows
python main.py

# Launch one window only
python main.py --workspace
python main.py --deaf

# Rebuild the clip-library index
python main.py --setup

# Training stages
python main.py --train-asl
python main.py --train-fsl

The default dual-window mode creates one camera thread and shares its frames between the workspace and Deaf-user views.

Expected local artifacts

Live inference expects:

• models/fsl_finetuned.keras
• models/fsl_finetuned.metadata.json for models trained with corrected v2 preprocessing; legacy models without a sidecar remain supported
• data/fsl_processed/fsl_encoder_classes.npy
• models/mediapipe/hand_landmarker.task
• models/mediapipe/pose_landmarker_lite.task
• models/faster-whisper-large-v3-turbo/model.bin

Clip generation expects:

• data/clip_library/index.json
• the indexed FSL video files
• an FFmpeg executable on PATH

Most datasets, trained models, processed arrays, and generated videos are intentionally ignored by Git because they are large or reproducible local artifacts. Model files and their metadata sidecars therefore need to be copied together when training and deployment use different machines or checkouts; a Git push does not transfer ignored model artifacts.

Training and evaluation

LGU vocabulary and conversation support

The FSL training command now defaults to a scenario-focused manifest derived automatically from the Deaf-side performed sequences in the 44 generated test cases: 929 available source entries and 70 classes, including A-Z. It excludes every label unused by those scenarios plus two missing paths, preserving all original files. See dataset cleaning decisions and training steps. For duplicate video filenames and replacement counts, see the LGU re-recording checklist. After replacing recordings, use the curated LGU retraining and signer guide to build a content-unique manifest, preserve the baseline model, and run the new FSL experiment. The exact 70 retained model labels and their recording counts are in config/lgu_dataset/labels.csv.

python tools/prepare_lgu_dataset.py
# After the compatible ASL base is available:
python main.py --train-fsl
# Select the reduced model only after training its matching encoder:
$env:SIGNBRIDGE_FSL_PROFILE = "lgu"
python main.py

The reduced run writes models/fsl_lgu.keras, data/fsl_lgu_processed/, and evaluation/lgu/. Without the runtime environment variable, the GUI still loads the existing full-vocabulary model. Set SIGNBRIDGE_FSL_PROFILE=full for the original training profile.

Recognition retains the visual model's top choice; hidden sequence branches only add a stability check for unexpected signs. The current branches are authored priors. Learned counts require independently verified training conversation sequences; isolated recordings do not provide those sequences. The interface shows “Please wait a moment…” for a 1.2-second settling interval, capped at three seconds during continuous accepted input. No tree is displayed. Use Finish word between fingerspelled words and lower hands between repeated identical letters. Unknown gestures still require deliberate spelling.

44 Chapter 4 conversation test scripts include expected keyword outputs and blank observation fields; they are not completed evaluation results. Run the focused functional checks with python -m unittest discover -s tests -v and the offscreen GUI smoke check with python tests/smoke_transaction_gui.py.

How the ASL-to-FSL model learns and produces text explains the landmark inputs, frozen layers, learned parameters, and the limits of keyword recognition.

How the live detector, cooldown, and translation-preparation indicators work traces the on-screen states to the inference worker and explains why still fingerspelling pictures are not direct inputs to the current video loader.

Live recognition audit and recording plan documents the train/live frame-selection mismatch, source-content duplicates, and a proposed capture protocol before the next FSL retraining run.

Stage 1 pre-trains the LSTM on the Google ASL landmark dataset. Stage 2 replaces the output head and fine-tunes the upper temporal/feature layers on FSL data. The fixed input contract is 30 frames by 162 values per frame.

The current training pipeline uses canonical preprocessing version landmarks_v2_missing_mask. It keeps missing landmarks at zero, uses the same normalization for ASL, FSL, and live inference, and writes new versioned caches without overwriting the legacy arrays. Existing models without a metadata sidecar continue using legacy normalization; a newly trained model activates v2 only when its matching .metadata.json sidecar is present and its checksum is valid. ASL is split by complete participant. FSL is split by original source video before augmentation, and only training sources are augmented.

For a stronger signer-independent FSL evaluation, create a signer manifest:

python evaluation\fsl_signer_manifest.py
# Fill dataset\FSL\signer_manifest.csv with pseudonymous IDs such as S001.
python evaluation\fsl_signer_manifest.py --check

Use the same pseudonymous ID for every video made by the same person. Do not put participant names in the file. Every class needs recordings from at least three identified signers. When a complete manifest is present, FSL training automatically uses signer-disjoint splits; otherwise it records a source-path split. Current byte-identical files cross that path-based split; see the live recognition audit.

Retraining order after these methodology changes:

python main.py --train-asl
python main.py --train-fsl

Do not skip ASL retraining: FSL training now refuses to use an ASL base model from an incompatible preprocessing version. Keep each .keras file together with its matching .metadata.json sidecar when copying models between the Windows and Antigravity/WSL environments.

The new FSL run writes its exact split under data/fsl_processed/ and writes the test predictions and aggregate metrics under evaluation/. Do not report the legacy model's 81% result as the outcome of this corrected protocol.

For a separate unseen-webcam test, copy and edit the example manifest, record complete clips, and run:

Copy-Item evaluation\fsl_video_manifest.example.csv evaluation\fsl_video_manifest.csv
python evaluation\evaluate_fsl_videos.py evaluation\fsl_video_manifest.csv

Use exact encoder labels for known signs and __REJECT__ for resting, transitions, unrelated movement, and unknown signs. This evaluator reports known-sign accuracy and macro F1, acceptance coverage, false acceptance on reject clips, per-video latency, and individual predictions. Because it scores complete recorded clips, it does not by itself measure continuous sign segmentation or GUI latency.

Classification helpers are in evaluation/metrics.py. Offline speech recognition can be measured from labeled WAV files with:

Copy-Item evaluation\speech_manifest.example.csv evaluation\speech_manifest.csv
# Edit speech_manifest.csv so every row points to a labeled local WAV file.
python evaluation\evaluate_speech.py evaluation\speech_manifest.csv `
  --output evaluation\speech_results.csv

The FSL training script now writes aggregate metrics, per-class precision/ recall/F1, a labeled confusion-matrix CSV, a row-normalized confusion-matrix image, and training curves under evaluation/.

Chapter 3's study-level instruments can be aggregated after data collection:

Copy-Item evaluation\sped_validation.example.csv evaluation\sped_validation.csv
Copy-Item evaluation\sus_responses.example.csv evaluation\sus_responses.csv
Copy-Item evaluation\task_results.example.csv evaluation\task_results.csv
Copy-Item evaluation\latency_trials.example.csv evaluation\latency_trials.csv
# The robustness input is normally produced by evaluate_fsl_videos.py.
# Replace every example row with actual observations before running this.
python evaluation\evaluate_study.py `
  --sped evaluation\sped_validation.csv `
  --sus evaluation\sus_responses.csv `
  --tasks evaluation\task_results.csv `
  --robustness evaluation\fsl_video_results.csv `
  --latency evaluation\latency_trials.csv

This produces evaluation/study_summary.json plus scored SPED and SUS detail files. Example CSV values are schema demonstrations only and must never be reported as thesis findings.

The evaluator reports detected language, hypothesis, language probability, latency, word error rate (WER), and character error rate (CER). The repository has focused transaction-support tests but no completed real-user evaluation report. Any reported result should include the exact model artifact, audio manifest, speaker protocol, environment, and command used.

Project documentation

• Architecture
• Thesis Context
• Changelog
• Ubuntu WSL2 Retraining Guide
• Agent Instructions