How facial recognition works in video surveillance, step by step: finding, aligning and matching faces, why the threshold matters most, and privacy done right.
Video surveillance performs one to many identification: searching one face against many enrolled people. That is far harder than the one to one check your phone does when it unlocks, and vendor figures often quote the easier task.
There is no single accuracy number. A system has a threshold, and choosing it is a trade off based on what happens after a match.
Anything that acts automatically needs a much tighter threshold than an alert a person reviews first.
Camera placement and enrolment quality decide results more than the AI does. A well placed camera at face height on a doorway beats a better model badly mounted, every time.
Keep faces on your own servers, keep a human in the loop, and keep an audit trail.
Facial recognition identifies or confirms a person by measuring their face and comparing it with stored records.
Three different tasks are often blurred together.
Face detection: finding that a face is in the frame, and where. No identity at all. Counting people, redacting faces for privacy and triggering a recording are detection tasks.
Face verification, or one to one matching: confirming that a face matches one claimed identity. Your phone does this when it unlocks. It already knows who you claim to be and only has to agree or disagree.
Face identification, or one to many matching: searching a face against everyone enrolled to find who it is, or to decide confidently that it is nobody on the list. Video surveillance does this, and it is much harder, because every extra enrolled person is another chance of a false match.
Keep this in mind whenever you read a vendor figure. One to one numbers look far better, and they are often quoted where one to many is what is being sold.
Every serious system, ours included, follows the same five steps.
Find. AI scans each frame and marks every face, along with a few key points such as the centres of the eyes, the tip of the nose and the corners of the mouth.
Straighten. Using those points, each face is rotated and scaled into a standard position, so eyes and mouth sit in the same place on every face the system compares. Without this, a tilted head changes the measurement more than a different person would. It is unglamorous, and it does more for real world reliability than most AI upgrades.
Describe. The straightened face is converted into a faceprint: a compact string of numbers that captures what makes the face distinctive. It is not a photograph and the photo cannot be rebuilt from it, but it is still biometric personal data and must be handled as such.
Compare. The faceprint is compared with every enrolled faceprint, producing a similarity score for each.
Decide. The best scoring person counts as a match only if the score clears the threshold you have set. If nobody clears it, the right answer is "no match", and a system that cannot confidently say "no match" is not fit for one to many use.
A face recognition system has no single accuracy figure, because it has no single operating point. It has a dial.
Turn the threshold down and the system matches more readily: you catch more of the people you are looking for, and you also get more false matches where it names the wrong person. Turn it up and false matches become rare, but the system misses more people it should know. You cannot reduce both at once. You choose where to sit, and the right choice depends on what happens next.
The two mistakes have different consequences. For a watchlist alert that an operator reviews before anyone acts, a somewhat looser setting can be reasonable, because the operator is the safeguard. For anything that acts automatically, such as opening a door, the setting must be much tighter, because a false match means an unauthorised entry with nobody checking.
In Visylix the threshold is a setting you control, with a conservative default, and we calibrate it with you on your own cameras during commissioning, with a tighter setting for higher security uses.
This is also why we do not publish a headline accuracy percentage for face recognition. A percentage only means something alongside the threshold, the number of people enrolled, the camera angles, the lighting and the population it was measured on. Change any of those and the number changes.
Ask a vendor five questions instead of "how accurate is it?" At what threshold was that figure measured, and what was the false match rate at that setting? Was it one to one or one to many, and against how many enrolled people? What were the capture conditions, and how do they compare with a camera above head height in mixed light? Who was in the test population? Can the threshold be set per camera? A vendor who answers all five clearly deserves more attention than one quoting a bigger number.
Most failures come from the camera, not the AI. In rough order of the damage they do:
Head angle. A face turned away loses the key points the straightening step needs. This is the biggest cause of missed recognition on CCTV, and it is a mounting problem: cameras placed high for wide coverage look down at heads.
Pixels on the face. What matters is not the camera's megapixels but how many pixels land across the face. A 4K camera covering a wide lobby can capture fewer usable face pixels than a 2MP camera aimed at a doorway.
Motion blur and low light. Slow exposures in dim light smear detail, and infrared at night changes how skin looks.
Then masks, heavy glasses, helmets, hair and scarves, which each hide part of the face; ageing and changes in appearance since enrolment, which build up over months and years; and twins and close relatives, which remain hard for every system.
The lesson: camera placement and enrolment quality decide your results. A camera at face height on a natural choke point, with good enrolment photos, wins every time.
Accept every face the camera finds and you will enrol poor records and raise matches on faces too blurred or turned to justify one. Serious systems judge how recognisable each face is before acting on it. Visylix does this in two complementary ways.
The obvious checks: how confident the detection is, how large the face is, how sharp it is and which way the head is turned.
A deeper check: how distinctive the face really is. A face can be sharp, frontal and well lit and still carry too little detail to match safely. Visylix scores this instantly, with no extra processing.
Those scores act as gates, with different strictness for different jobs. Enrolment has a quality floor, because a weak enrolment photo becomes a permanent bad record. Automatic updates to a stored face have an even stricter floor, because small errors add up over time.
Liveness is checked at enrolment too. If someone could enrol a printed photo or a face on a screen, that record would be poisoned for every future match, so Visylix blocks it at the door.
Live video offers something a photo never can: many looks at the same person.
Instead of picking one frame and matching on that alone, Visylix combines every look at a face as the person crosses the scene, giving more weight to the clear, frontal moments and little to the blurred or turned ones.
The result is far steadier than any single frame, which matters on CCTV, where any one frame may be poor.
"Self learning" is used loosely in this market, so here is exactly what Visylix does.
Each camera finds its own threshold. A camera facing a bright window at four in the afternoon behaves differently from one in a windowless corridor, so one global setting is wrong for most cameras. Visylix studies the scores each camera produces and fine tunes that camera's threshold, only ever tightening it, never below the limit the operator has set.
Stored faces stay current. When a confident, high quality match occurs, the stored record moves gently toward the person's current appearance, so gradual changes such as a new hairstyle or a few years of ageing do not erode recognition. The change is slow and the quality bar is strict.
What never happens is the AI quietly retraining itself on your people. Learning faces automatically would mean keeping face images for training, a decision with legal and consent consequences that belongs to you and your counsel, not to a background job. And the failure is severe: a system that wrongly learns two people as one damages recognition for both, permanently.
If a vendor says their face recognition "learns your people automatically and gets better every day", ask what happens when it learns two people as one.
A match only helps if the system knows what that person means to you. In Visylix, enrolled faces sit in watchlists with a category, such as blocked, watched, allowed or VIP, so a recognised face raises the right level of alert. Unknown person alerts flag faces that are not on any list. Every match is raised for an operator to confirm.
That difference drives real operations. A VIP arriving at a hotel entrance and a barred individual at the same door are the same technical event and completely different responses.
In Visylix a face match feeds the whole automation layer. Sudarshan, the live intelligence layer, shows recognition on the focused feed with severity rings and an alert rail, and lets operators arm a watch just by describing it in plain words. Sudarshan Rules can combine a face match with other conditions, in time sequences and with branching actions, including access control events, so "a recognised face at a door with no badge swipe" becomes an automatic rule rather than something an operator has to notice. Actions include notifications, alarm incidents in the operator console, talkdown through the camera speaker, signed webhooks, MQTT, SIEM and PSIM.
Radha, the Visylix AI copilot, runs on an on premise language model with 111 tools. Operators can ask about recognition events in plain English, Hindi or Hinglish, and enrol faces or mark false alarms through Radha. Its 13 high impact actions always ask for confirmation, it has no delete tool, and every action is audit logged.
Face recognition is one of 22 AI analytics built in house, available per camera per month on the cameras you choose, on top of Starter and Pro plans that include unlimited cameras on the VMS licence from ₹4,999 or $49 per month.
Facial recognition draws more regulatory attention than any other video analytic, rightly. Design choices have direct legal consequences.
Where the data lives. Faceprints are biometric personal data. Under India's Digital Personal Data Protection Act, the EU General Data Protection Regulation (which treats biometrics as a special category) and similar laws, sending them to a third party cloud needs justification, a lawful basis and usually a detailed contract. Visylix runs entirely on your own infrastructure, recognition and copilot included, with no cloud dependency and full support for air gapped sites, so faces enrolled at a site stay at that site. That does not make a deployment lawful by itself, but it removes the cross border transfer question, often the hardest part of an assessment. Visylix also supports consent records and erasure of enrolled people.
Bias. Independent evaluation, most systematically by the US National Institute of Standards and Technology in its ongoing face recognition programme, has repeatedly found that error rates vary by demographic group, and that the variation differs a lot between algorithms. Two things follow: a single global accuracy figure hides it, and any deployment making consequential decisions about people needs human review and an appeal route.
Proportionality. Regulators and works councils usually ask "why do you need it here?" before "is it accurate?" Recognition on a server room door is easy to justify. Across a general office floor, it rarely is. Use it where it is defensible.
A short governance checklist. Document the lawful basis before enrolment. Set a retention period for faceprints and enforce it. Restrict who can enrol, view and export records, and audit those actions; Visylix offers 46 permissions across 4 roles and a tamper evident audit trail for exactly this. Post notices where recognition operates. Keep a human in the loop for consequential decisions, with a route to challenge a match. Record your thresholds per camera. And review watchlists regularly, removing people who no longer belong there.
A camera captures a face. Software finds it and straightens it using points such as the eyes and nose. AI turns the straightened face into a compact numeric description, a "faceprint", and compares it with the faceprints of known people. If the closest one is similar enough to pass the threshold you set, the system reports a match.
It depends on the threshold you choose, the number of people enrolled and the conditions at the camera, so no single number answers it honestly. A system set to rarely miss anyone produces more false matches; one set to rarely misidentify misses more people. Ask what the false match rate is at the setting you plan to use, and test on your own cameras.
Detection finds that a face is present and where it is, with no identity involved. Recognition compares that face against enrolled people to work out who it is. Detection alone is enough for people counting, privacy redaction and occupancy, and it carries far lighter legal obligations because no biometric identity is processed.
Often, but placement decides it. You need enough pixels across the face at a reasonable angle. Cameras mounted high for wide coverage usually look down at the tops of heads. Most recognition projects add a few well placed cameras at entrances rather than relying on the whole existing estate.
Only with good lighting. Infrared lets cameras see at night but changes how faces look, which makes recognition less reliable than in daylight. Entrances used for recognition should have proper visible lighting.
Without a liveness check, it can. This matters most at enrolment: a photo or screen enrolled in place of a real face corrupts that record for every future match. Visylix checks liveness at enrolment, blocking printed and on screen faces at the point where they would do the most damage.
No. Visylix runs face recognition entirely on your own servers, including on fully air gapped sites, so enrolled faces and their faceprints never leave your premises.
Matching needs only the faceprint, and the original photo cannot be rebuilt from it. Faceprints are still biometric personal data under most laws and carry the same obligations, so this reduces exposure rather than removing the legal question.
Visylix uses face recognition for security: watchlists, unknown person alerts, VIP arrivals and investigations. Time and attendance is a separate category with its own consent, payroll and labour law rules, so it stays with dedicated HR systems.