DeepShield: An Explainable AI Framework for Multimodal Deepfake Detection
Abstract
Deepfakes increasingly combine manipulated facial appearance, synthetic or replaced
speech, temporal inconsistencies, audio-visual synchronization artifacts, and context-level
anomalies. A detector that relies on only one cue can therefore miss manipulations that
preserve that cue while altering another. This thesis presents DeepShield, an explainable,
multimodal, reliability-aware deepfake forensic framework that integrates Face, Voice,
Temporal, Lip-Sync, and Background evidence through mask-aware learned attention
fusion. The Face branch combines Swin Transformer, official GenConViT-ED,
GenConViT-VAE, and frequency-domain evidence; the Voice branch uses MFCC and
Log-Mel features with a CNN-LSTM; the Temporal branch models ordered face
sequences using BiLSTM and temporal attention; the Lip-Sync branch analyzes aligned
mouth-audio evidence; and the Background branch uses CLIP and DINOv2
representations over original and face-suppressed frames. Validation-only temperature
scaling and a locked decision threshold are used before a single frozen internal TEST
evaluation. The project-wide eligible primary registry contained 254,228 media items.
Earlier leakage-safe and broad visual stages used 177,584 selected samples and 180,206
visual sequences respectively, while the final tightly aligned multimodal canonical
experiment used 21,544 A/V cases split into 17,234 TRAIN, 2,159 validation, and 2,151
frozen TEST cases. The final frozen internal TEST achieved ROC-AUC 0.998277,
accuracy 0.992562, precision 1.000000, recall 0.992385, F1-score 0.996178, expected
calibration error 0.007261, and Brier score 0.005121. The results demonstrate strong
performance on the locked aligned evaluation cohort while the thesis deliberately limits
claims to this internal experimental setting. DeepShield is positioned as an automated
forensic screening framework rather than legal proof, with explicit modality masks,
uncertainty-aware decisions, and explainability outputs designed to support human
review.
Collections
- 2026-2030 [51]