One-Shot Lip-Based Biometric Authentication

Lip-based biometric authentication verifies a person from the way their lips move while they speak, using nothing more than an ordinary RGB camera. The one-shot systems that existed before this work had a security hole: they modeled how a person speaks but not what you say, so a recording of the enrolled person saying anything at all was enough to get in. This work closes that hole by requiring the right person saying the right phrase, and still reaches 3.23% false acceptance and 3.86% false rejection on an open-set test.

Published in Image and Vision Computing (2024), co-authored with Ratko Grbić at FERIT Osijek. This was my master’s thesis work.

Authenticating a person by how they say a phrase

Lip-based biometric authentication (LBBA) is a method of verifying someone’s identity from video of their mouth region while they speak a phrase. Biometrics are conventionally split into physical features, such as fingerprint, face, palm and iris, and behavioral features, such as voice, handwriting and posture. What makes the mouth region during speech unusual is that it carries both at once, and this work is about exploiting that combination rather than either half alone.

The first is physical: the shape, imprint and texture of the lips themselves. These are unique in the way fingerprints are. A 1974 forensic study examined 1,364 people aged 3 to 60 and found no two subjects with the same lip print, then re-examined those prints every month for three years and found no significant change in any of them.

The second is behavioral: the visual style of speech. This covers the whole region around the mouth, including the lips, the position and visibility of teeth, the oral cavity and the tongue. Two people pronouncing the same phrase do not produce identical visual sequences.

The practical appeal is that both signals come from a plain camera. No fingerprint reader, no depth sensor, no infrared. That is also what makes the attack surface interesting.

Why the previous one-shot systems could be defeated with a recording

Because they modeled behavior as style-of-speech only, with no connection to the words being spoken.

The prior work arrived at this in two steps. The first generation discriminated on appearance alone, training on a single fixed phrase, which left it open to both the print attack and the video replay attack. The second generation added style of speech by training across multiple phrases, which closed the print attack but not the replay: a positive pair still meant two utterances by the same speaker and a negative pair meant utterances by different speakers, so the phrase never entered the decision at all. The consequence is a video replay attack: hold up a recording of the enrolled person saying literally anything, and the system authenticates.

Face recognition, the incumbent camera-based biometric, is defeated by three classes of presentation attack: the print attack, the video replay attack, and the mask attack. Video replay is the one that earlier liveness methods, built on blink detection and gaze tracking, struggled most to catch, and it is the one that a system modeling only style of speech remains fully exposed to.

Two further problems followed from the same design. The authentication key is an embedding derived from a person’s face, so unlike a password it cannot be reissued once compromised. And because negatives only ever differed by speaker, there was no way to tell whether the network had learned anything behavioral at all, or had simply overfit to physical appearance.

How tying the phrase to the identity closes the hole

By redefining what counts as a negative pair.

Only Type 1 is positive. Type 2 is the addition that matters: the same person saying a different phrase is now a negative, which forces the embedding to encode what is being said rather than only who is saying it.

The phrase is encoded into the template at the enrollment stage, so the practical effect is that spoofing now requires a video of the enrolled person saying that person’s exact enrolled phrase, instead of any video of them at all. To my knowledge this was the first time phrase information had been used this way in one-shot deep learning LBBA.

Why I had to rebuild the dataset before I could train anything

Because no existing LBBA dataset had enough phrase diversity to learn from.

XM2VTS, the dataset the field standardised on, contains three unique phrases spoken eight times per speaker, and those phrases are relatively long. Three phrases cannot support a model whose whole job is discriminating between phrases, and longer phrases are the wrong shape for authentication, where shorter ones are preferred.

So I rebuilt the GRID dataset instead. GRID has 33 speakers, 15 female and 18 male, and 33,000 recordings made on a Canon XM2 camcorder at 25 FPS and 720x576 resolution. Its phrases follow a strict grammar:

command(4)  color(4)  preposition(4)  letter(25)  digit(10)  adverb(4)

which produces sentences like “place blue at F 9 now”. Because GRID ships word alignments, I could cut each recording down to a sub-pattern and treat that fragment as an authentication phrase. I compared four candidate sub-patterns on speaker count, phrase count, utterances per phrase and the number of positive pairs each would yield, then chose command, color, preposition:

Worth being explicit that this makes the problem harder than XM2VTS rather than easier. GRID phrases frequently differ by a single word, so the model has to separate “bin blue at” from “bin blue by”.

How the model turns a video of a mouth into a comparable embedding

Preprocessing runs in two stages. MediaPipe Face Mesh, a pretrained MobileNetV2-based model, infers 468 facial landmarks on every frame. The mouth region is then cropped as the rectangle defined by landmark indices 57, 287, 164 and 18, resized to 100x50 pixels and converted to grayscale. Any recording containing a frame where landmark detection confidence fell below 50% was discarded, which removed roughly 4% of the data.

The network is siamese: two branches with shared weights, each producing an embedding, built on a LipNet-inspired backbone with a custom head.

The result is a 256-dimensional L2-normalized embedding compared with cosine similarity. Because the network learns an embedding function rather than a classifier over known users, enrolling someone new takes a single example. That is the one-shot property, and it is what makes the approach deployable at all.

The loss function, and the part of this I would reuse anywhere

The most reusable idea here is a triplet loss that accepts only positive pairs and manufactures its own hard negatives inside each batch.

Two things make siamese training difficult. Negatives vastly outnumber positives, which biases the network toward rejecting everything. And easy negatives teach it nothing, so the negatives need to be hard for learning to happen at all.

The trick is a batch constraint: every positive pair in a batch is sampled from a different utterance set. That guarantees any two different positive pairs also form a valid negative pair, so a batch built this way carries its own negatives for free.

Passing the batch through both branches gives two embedding matrices, and multiplying them yields a similarity matrix where the positive scores land on the main diagonal and every off-diagonal entry is a negative.

The loss then has two terms, one built on the hardest negative in each row and one on the mean of the negatives, weighted equally with a margin of 0.5.

Keeping both allows tuning how much the single hardest negative drives learning relative to the negatives as a whole. Mining across the batch rather than the whole dataset was a deliberate speed tradeoff.

Accuracy on held-out speakers

3.23% false acceptance and 3.86% false rejection on the open-set test set.

The protocol was open-set, meaning no speaker appears in more than one split: 25 speakers for training, 4 for validation and 4 for test, chosen for balanced gender representation. I sampled 100,000 positive pairs for training and 10,000 each for validation and test, with batch sizes of 80, 40 and 40. Training ran for 15 epochs with Adam at a learning rate of 1e-3 and no dropout. Augmentation covered per-frame deletion and duplication at probability 0.1, per-video rotation of up to 20 degrees at probability 0.15, and per-video horizontal flip at probability 0.2.

One methodological point worth stating plainly: the decision threshold was fixed at 0.45, the equal error rate point measured on the training set, and then applied unchanged to validation and test. Nothing was tuned on the test set.

Because the loss creates negatives implicitly, those 10,000 test positives expand to 400,000 evaluated pairs, of which 2.5% are Type 1, 24.15% Type 2, 1.16% Type 3 and 72.20% Type 4.

Earlier one-shot LBBA work reports lower raw error, 0.93% equal error rate in a closed-set setting on a single 20-digit XM2VTS phrase, and 1.65% in an open-set setting. Those systems only have to decide whether the speaker is the right person. This one also has to decide whether the phrase is the right phrase, across 64 phrases that often differ by one word. Comparable error rates on a materially harder task is the actual result.

What the errors say about choosing authentication phrases

Pick phrases that differ by more than one short word.

The error analysis was the part I found most useful, because it turns into a deployment rule rather than a number.

The model separates behavioral differences more confidently than physical ones, which is to say it is better at catching the same person saying the wrong phrase than a different person saying the right one. That finding needs a caveat I want to keep attached to it: the test set is only 1.16% Type 3 against 24.15% Type 2, so class imbalance explains some of the gap.

Most misclassifications happen when two phrases differ by exactly one word, and the preposition category is the worst offender. Prepositions are the shortest words in the chosen sub-pattern, so they produce the shortest lip movements and the least signal. Error counts fall steadily as more words differ between phrases. And for different-speaker pairs, most errors occur when the phrase is identical, which is exactly the case where no behavioral signal is left and the model has to fall back on physical features alone.

The practical consequence for anyone deploying this: assign strictly different authentication phrases across users, and never let two users be distinguished by a single short word.

Limitations and what I would do next

The class imbalance above is a real limitation on how strongly the behavioral versus physical conclusion can be stated. GRID is also a controlled studio dataset, with consistent lighting, frontal pose and a fixed camera, so these numbers are an upper bound rather than a field result.

The most interesting direction is phrase engineering at the viseme level. A viseme is the visual equivalent of a phoneme, the smallest distinguishable unit of lip movement, and several phonemes can map to the same viseme. Choosing authentication phrases whose visemes are maximally distinct, rather than whose letters are, should raise the floor on the single-word confusions above. Beyond that, evaluating under harder conditions than a studio setup: varied lighting, off-axis pose and mobile cameras.

Koch, B. and Grbić, R. “One-shot lip-based biometric authentication: Extending behavioral features with authentication phrase information.” Image and Vision Computing, 2024.

Read the paper →