As a landmark facial recognition biometric study at Carnegie Mellon University established two years ago, retailers using existing security cameras can grab real-time images of shoppers and identify them within seconds using public databases. (The CMU study at the time limited itself to public Facebook images.)
There are three primary ways this can be used in retail. First option: it can be used to try and positively identify that customer by name, like the CMU study. It can then search for that name in its customer database.
Second option: it ignores a name for the time being, instead merely capturing the facial data points and noting what purchases the person attached to that face makes. Then, when the cameras catch that same face again (say, perhaps four days later), it will remember the prior purchases. It can either use that to send digital ads or coupons for that shopper or can merely note shopping patterns. (Note: 47 percent of people that we noticed buying Lime-flavored Diet Coke returned a few days later to buy Kleenex, red-colored hammers, Liquid Plumber and loaves of French bread. Please don't ask us to explain it: we're just closed-circuit cameras.)
The third option is the least interesting and it's the approach that the Russian chain is trying: Using the images to guess gender and age-range and use that solely to send ads and promotions.
But such activities need not end with the same channel where they began. Once a shopper is identified in-store and is matched with a CRM profile—or they are identified anonymously in-store and a purchase profile of this unknown-person-with-this-specific-face is slowly built—that information can theoretically be married to data from that person's desktop-shopping E-Commerce efforts or their tablet/smartphone's M-Commerce efforts.
The question, then, is whether it has to start in-store. What if this hypothetical chain pushes some attractive incentives to get lots of customers and prospects to download its free mobile app? And buried in the terms & conditions is the right for the app to monitor images?
The next selfie or Snapchat that the shopper sends is captured and the facial data points are noted. The app itself may already have a name of the shopper (it probably does), but if not, the phone provides plenty of clues. And geolocation knows where the phone goes and certainly when it walks into one of the chain's stores.
Here's where it gets even freakier. Once the mobile app has identified the face of the shopper—and has linked it to whatever mobile shopper that customer has done—it can tell the in-store camera databases what to look for. When that shopper walks in, it can connect the mobile activity with any observed in-store activity.
And if the desktop device has a camera enabled for any purpose, there's more potential. Invasive and creepy to the extreme, but potential nonetheless.
With that in mind, let's look at how Ulybka Radugi is using the technology today. This is a live shopper-interacting experiment.
According to the marketing head of the vendor that the Russian chain is working with, the system uses the video surveillance to supplement or replace a loyalty card. "If the customer has no loyalty card or doesn't want to identify himself with a loyalty card, then the system recognizes his general mood, gender and age in order to use this data for targeting of the content," said Ekaterina Savchenko, head of international marketing for vendor Synqera.
The process then gets more complicated. "If the customer identified himself with a loyalty card, the system double checks the customer age and gender with data sourced through facial recognition," Savchenko said. "If the Synqera system sees that the loyalty card data differs from the camera data, then it evaluates the correctness of the camera data (probability defined for the particular user's gender and age) and, if it is high, gives it priority."
The reason this process is both complicated and useful is that CRM cards, despite their intent, are sometimes shared among family members and even friends. The biometrics help make sure that message or promotion being displayed is the right one.
One "trick is to accumulate the user statistics linked to the each unique card number for future sessions. It is based on the logic that the whole family may use one card and, for example, both husband and wife will come to the store," Savchenko said. "The system will then learn that these few users are linked to one card and base the analysis and relevant content on the facial recognition data."
A few years back, global food giant Unilever experimented with a vending machine that literally accepted a smile as acceptable tender for ice cream. The Russian trial takes that same idea and brings it into the convenience store aisle.
Ulybka Radugi is using the smile-detecting capability to try and guess whether the shopper is pleased by the message being displayed. (Note: A smile may not mean approval. "Yo, check out this really stupid ad. Who would be so dumb to buy this overpriced thing? It's so bad, it's funny." But for the moment, let's assume that a smile is a good thing.)
"Users' smiles are used for the evaluation of the content effectiveness, similar to the way that digital cameras will activate once the lens sees a person’s smile," Savchenko said, who added that it's also used in those stores as a way to get a prize of some kind. In those cases, the smile is prompted and rewarded, similar to giving a poppy a dog treat after he performs a trick. "If the user smiles, he gets a virtual achievement badge or extra loyalty bonuses to his card," she said.
Clearly, this technology has lots of potential to be used in various marketing and even payment means, above and beyond the usual security methods, such as having a way to enforce when shoplifters are "banned for life" for visiting a chain's stores. But it also has many risks, especially legal risks to the chain.
The privacy—and associated shopper backlash—risks are obvious, but shoppers (especially younger shoppers) seem to have developed an almost infinite capacity for tolerating such efforts. Make the incentive strong enough—and use the data in subtle enough ways so that you're not forcing the customer to know how far you've gone—and privacy will be a trivial concern. Not saying that it should be a trivial concern, but merely our belief that it will be.
There's also the practical side, which is that facial recognition today is still not that accurate. More to the point, it can be fooled very easily, even by shoppers who are not trying to fool the system.
As one facial recognition trial participant said about a retail trial held earlier this year, that system was often confused by someone choosing to wear sunglasses and not shave.
Like voice recognition, look for this technology to get an order of magnitude more accurate over the next couple of years. In the meantime, though, using it to do anything beyond choosing which ad or discount to display is probably asking for trouble. Or in facial recognition terminology, your next selfie would likely be a big frownie.