Models are short-span, editors are continuous
The classifier reasons over individual utterances. An editor works with a continuous timeline, so predictions must be windowed, overlapped, and re-anchored to clip positions.
Capstone · AI media systems
DeployedSignalsFrame
A browser-based video editor connected to transcription, multimodal emotion and sentiment analysis, cloud GPU inference, and an intelligence-aware timeline.
- Product
- Web video editor
- System
- 2 repositories
- Model
- 3 input modalities
- Output
- 10 prediction classes
01 · Problem framing
Why this is genuinely hard.
A video editor shows frames and waveforms. It does not show meaning. Finding the moment a speaker became excited, frustrated, or disengaged is manual scrubbing. Closing that gap requires four constraints to hold at once, and each one pushes the architecture in a different direction.
The model stack cannot run in a browser tab
A BERT encoder, a 3D convolutional video network, and an audio CNN exceed what is reasonable to ship to a client. The compute boundary is forced, not stylistic.
Inference takes seconds to minutes
Editing interfaces are built around immediate feedback. Long-running analysis needs progress, cancellation semantics, and partial results rather than a frozen spinner.
Predictions must be inspectable to be usable
An editor will not cut footage based on an unexplained label. Each segment carries ranked classes and evidence so the human keeps the final decision.
02 · Product surface
The software, not a mockup.
The deployed application starts locally in the browser and exposes cloud sync and AI features after sign-in. Its editor combines asset management, playback, a resizable timeline, AI analysis, transcription, notes, and export.
Editing
Tracks, layers, trim and split tools, playback, snapping, zoom, and export.
Intelligence
Emotion and sentiment distributions, AI markers, quick actions, and explainability.
Workflow
Import, process, inspect, edit, and export inside one product surface.
Boundaries
Local editing stays available while cloud sync and AI require an authenticated project.
03 · Ownership and scope
Mohammad's documented contribution
- Frontend product surface and reusable React interface work within a multi-contributor monorepo
- Professional editing interface and timeline-oriented workflows
- Transcription workflow, progress states, and failure feedback in the product
- Integration of asynchronous AI behavior into the editing experience
- Primary authorship of the separate Video-Intelligence-Engine ML repository
The team's complete system
SignalsFrame is team-built. The product repository includes WebGPU rendering, Convex schema design, S3 upload flows, Inngest timeline analysis, Clerk auth, and browser-side media tooling contributed across multiple engineers. Mohammad's role is strongest on the product interface and the ML pipeline he documented separately.
A hiring review should treat this as collaborative systems work with clear individual scope, not sole authorship of the entire platform.
04 · Cloud architecture
Clear service boundaries.
Interactive editing stays in the browser. Auth, persistence, and job state live in Convex. Media lands in S3. Inngest coordinates chunking and Modal calls. The ML repository keeps preprocessing, checkpoint loading, inference, and API contracts in one installable package.
Cloud topology
Editor, orchestration, and GPU inference
Verified from the public SignalsFrame and Video-Intelligence-Engine repositories: browser editor, Clerk auth, Convex persistence, S3 media, Inngest job orchestration, and Modal GPU services.
01 · Client
Browser editor
React/TypeScript editor with WebGPU rendering path, timeline, assets, and AI analysis workspace.
02 · App plane
Clerk + Convex + S3
Authentication, project/asset state, signed upload URLs, and timeline analysis job records.
03 · Orchestration
Inngest workflow
Prepare media, chunk segments, call Modal inference, persist timeline segments back to Convex.
Storage
AWS S3 presigned uploads for project media
Chunking
6 s max segments with 500 ms overlap for timeline analysis
Transcription
Separate Modal Whisper service for speech-to-text
Inference
Modal A10G multimodal affect endpoint
Primary inference path
Modal serverless GPU
Containerized inference app with model weights on a Modal Volume, HTTP contract tested against Convex expectations.
Alternate cloud target
Amazon SageMaker + S3
Training and real-time inference scripts package the shared ML library for SageMaker endpoints. Deploy scripts exist; live production routing should be verified separately.
01
Segment
faster-whisper
02
Cut
FFmpeg utterances
03
Tensorize
video · audio · text
04
Infer
late-fusion model
05
Return
typed timeline
Max upload size
2 GB
per file
Project storage ceiling
10 GB
per project
Signed URL lifetime
15 min
S3 presigned
Inference timeout
180 s
default per call
Analysis window
6 s
max segment
Window overlap
500 ms
boundary stability
Inference GPU
A10G
Modal container
Transcription GPU
T4
Whisper base
05 · Request lifecycle
Asynchronous work, honest states.
Most of the engineering difficulty is not the model call. It is representing a long-running distributed job inside an interface where the user keeps editing while the job is still in flight.
Job lifecycle
Every state the interface has to represent
Analysis is asynchronous, so the product cannot assume a single success path. The editor renders progress for the happy path and distinct, recoverable outcomes for everything else.
- 01Queued
- 02Preparing media
- 03Chunking
- 04Inferring
- 05Persisting segments
- 06Complete
Non-happy outcomes
Partial
One or more utterances fail during inference
The response reports a failed utterance count and the timeline keeps every segment that did succeed.
Failed
Auth rejection, upstream timeout, or unreadable media
The job ends in a terminal error state with a user-safe message instead of a raw stack trace.
Stale
The editor timeline changes while a job is still running
Returned segments are remapped against current clip positions, or discarded when they no longer apply.
Segmentation strategy
6s windows, 500ms overlap
The model reasons over short spans, but editors care about continuous footage. Overlapping windows keep predictions stable across boundaries so a single emotion shift is not split into two contradictory segments.
win 01
0.0–6.0s
win 02
5.5–11.5s
win 03
11.0–17.0s
win 04
16.5–20.0s
0s5s10s15s20s
A 20s selection produces 4 inference windows at a 5.5s stride. Chunk count drives cost and latency, so window size is a product decision as much as a model one.
06 · Data model
Job state is a first-class record.
Analysis is not a fire-and-forget request. Jobs, chunks, and segments are stored so the UI can subscribe, resume, and reconcile results against an edited timeline.
usersIdentity records linked to Clerk JWT issuer configuration.
projectsEditor project ownership, format, and cloud-sync membership.
assetsUploaded media metadata pointing at S3 objects rather than blobs.
uploadIntentsShort-lived records that authorize and track presigned uploads.
timelineSnapshotsSerialized editor state so a session can be restored or replayed.
timelineAnalysisJobsOne row per analysis request with status, progress, and error fields.
analysisChunksThe per-window unit of work sent to inference, keyed to a job.
analysisSegmentsReturned emotion and sentiment predictions anchored to timeline positions.
07 · Intelligence engine
Shared training and inference semantics.
The ML repository documents a late-fusion classifier over text, video, and audio. Training and inference share tensorization expectations, which is what makes a checkpoint trained offline usable behind a live endpoint.
Each utterance returns ranked emotion and sentiment predictions mapped back onto the editing timeline. No public benchmark numbers are committed in the repository, so macro-F1 appears here as a model selection metric during training rather than an advertised production accuracy.
Multimodal model graph
Documented late-fusion design
Architecture below matches the public ML repository README and inference engine. The neural network module itself is referenced by training and inference code but was not present in the public tree at review time—so this diagram reflects documented design, not a fully open-sourced model implementation.
Text
BERT encoder
128-d features
Video
R3D-18
128-d features
Audio
1D CNN · log-mel
128-d features
384-d concatenated representation
Linear · BatchNorm · ReLU · Dropout → 256-d
7-way emotion head
3-way sentiment head
Tensor contract
Identical on both sides
- Video
- 30 RGB frames at 224×224
- Audio
- 16 kHz mono, 64-bin log-mel, hop 512, padded to 300
- Text
- BERT tokenization over the transcript segment
Training configuration
Reproducibility over one-off runs
- Framework
- PyTorch 2.5.1
- Training instance
- ml.g5.xlarge
- Epochs
- 25
- Optimizer
- AdamW with parameter groups
- Schedule
- Cosine decay with linear warmup, or plateau
- Precision
- Mixed precision with gradient scaling
- Imbalance
- Focal loss, class weights, weighted sampler
- Regularization
- EMA weights, dropout, label smoothing
- Selection metric
- emotion_macro_f1
- Tracking
- TensorBoard, optional Weights & Biases
- Reported metrics
- Accuracy, balanced accuracy, macro and weighted F1
- Diagnostics
- Per-class precision/recall and confusion matrix
08 · Decisions and trade-offs
Every choice cost something.
A system this shape has no free decisions. Each row below names the alternative that was rejected and the price paid for the option that shipped.
Run inference on serverless GPU
Instead ofIn-browser inference via WebGPU or WASM
The three-tower model is far too heavy for a client. Serverless GPU keeps the bundle small and cost proportional to use. The accepted cost is cold-start latency and a hard network dependency.
Overlapping 6s analysis windows
Instead ofOne inference pass over the whole clip
Fixed windows bound memory and keep predictions stable across boundaries. The accepted cost is duplicated compute in the overlap region and more calls per selection.
Convex job rows plus Inngest steps
Instead ofA bespoke queue with client polling
Durable step execution gives retries and observability, while job rows are directly subscribable by the editor. The accepted cost is another service in the critical path.
Presigned S3 uploads direct from the client
Instead ofProxy media bytes through application compute
Two-gigabyte files should never traverse app functions. The accepted cost is client-side management of URL expiry and retry.
Late fusion over three independent towers
Instead ofEarly fusion or cross-modal attention
Each tower trains, debugs, and degrades independently, which matters when one modality is missing or corrupt. The accepted cost is weaker modelling of fine-grained cross-modal interaction.
Local-first editing, cloud and AI behind sign-in
Instead ofRequire authentication for all functionality
A first-time visitor can edit immediately, which materially changes activation. The accepted cost is maintaining two state paths and communicating the boundary in the UI.
09 · Production-shaped engineering
The repository anticipates operational failure.
Checkpoint compatibility
Loads raw state dictionaries, trainer exports, legacy wrappers, and known BERT key-prefix drift.
Per-utterance isolation
The timeline contract reports failed utterances instead of collapsing the whole analysis into one opaque error.
Reproducible training
Seed control, mixed precision, gradient accumulation, EMA, configurable loss, and structured metrics utilities.
Imbalance handling
Focal loss, class weighting, weighted sampling, label smoothing, and macro-F1 checkpoint selection.
Deployment parity
Shared preprocessing, inference engine, and HTTP contract logic target both Modal and SageMaker paths.
Typed product states
The UI represents extraction, upload, transcription, processing, completion, and failure explicitly.
10 · Testing strategy
Tested at the seams.
The inference boundary is where a product bug and a model bug would otherwise blame each other, so it has its own contract tests.
01
Unit and integration
Vitest suites across the web app, shared core package, and Convex function layer.
02
End-to-end
Playwright specs exercising real editor flows in a browser rather than mocked components.
03
Service contract
Dedicated tests for the inference boundary: authorization handling, chunk and batch response mapping, and user-safe error messages.
11 · Known limitations
What is still wrong, and the fix.
This section exists because a reviewer will find these anyway. Naming them with a concrete next step is more useful than a case study that claims everything worked.
Preprocessing skew between training and serving
The training dataset normalizes video with Kinetics mean and standard deviation, while the inference tensorizer only scales pixels into a unit range.
Next stepMove normalization into the shared preprocessing package so both paths cannot drift, then re-evaluate.
No committed validation results
The experiment matrix defines ablations for focal loss, class weighting, sampling, and text unfreezing, but no scored output is checked in.
Next stepCommit per-run metrics and confusion matrices so claims about quality are reproducible.
Model module missing from the public tree
Training and inference both import the network definition, so the public repository is not independently runnable end to end.
Next stepPublish the module or document exactly where the definition lives.
Domain shift from the training data
MELD is scripted television dialogue. Real uploads are podcasts, interviews, and vlogs with different pacing and audio conditions.
Next stepBuild a small in-domain calibration set and tune per-class thresholds before trusting absolute labels.
Unmeasured cold-start latency
Serverless GPU containers add first-call delay that directly shapes how the progress UI should behave.
Next stepInstrument p50 and p95 end-to-end latency, then decide whether to keep containers warm during active sessions.
Ambiguous canonical deployment target
Repository metadata points at one host while the deploy script targets another, which makes the production path unclear to a reviewer.
Next stepDocument a single canonical deploy target and remove the unused path.
12 · Evidence boundaries
What the public repos support, and what they do not.
Supported by repository evidence
- End-to-end product and ML integration across two public repositories
- Convex schema, Inngest workflow, S3 uploads, and Modal client contracts in SignalsFrame
- MELD-style training and inference pipeline, SageMaker scripts, and Modal deployment in the ML repo
- Contract tests aligning inference responses with product expectations
- Live deployed editor at signalsframe.com with an AI analysis workspace
Not claimed without extra proof
- Published macro-F1, accuracy, or benchmark leaderboard results
- Sole authorship of the full SignalsFrame monorepo or the WebGPU renderer
- Proof that a SageMaker endpoint currently serves production traffic
- Full open-source release of the neural network module in the public ML tree
13 · Engineering review
What this work demonstrates.
The strongest signal is not the number of technologies. It is the ability to connect a dense product interface to asynchronous AI services while preserving understandable states, honest system boundaries, and a path from model output back to user action.