How AR and AI are Driving Innovation Amid Social Distancing

Theresa Novicky knows makeup.

Based in Raleigh, North Carolina, Novicky, a beautician, loves to help her clients, many of them brides-to-be, wear the right looks for their big day. While Novicky takes care of the rest of the makeup, she leaves the selection of the lipstick to the brides themselves. Typically, they visit many stores looking for just the right shade—it’s a big day, after all, and everything has to be just so.

Before the COVID-19 pandemic, Novicky’s clients could simply test lipstick shades in-person at beauty retailers. Touchless restrictions and rolling lockdowns, however, have thrown that routine for a loop. Today, instead of shopping in-person, a bride can use a makeup app to test-drive a variety of shades, leveraging the wonders of augmented reality (AR) to find just the right dazzle for the special occasion.

StartupNation exclusive discounts and savings on Dell products and accessories: Learn more here

A new makeup routine

Makeup apps, including ones from L’Oreal, Estée Lauder, Neutrogena, Madison Reed, Sally Beauty, among others, use Perfect Corp’s technology as the underlying engine. The company’s proprietary YouCam works using mostly a combination of AR and machine learning (ML) technologies.

Using a phone or desktop camera, YouCam recognizes the customer’s facial features via ML programs. It locates 106 different points on the face to create and display a 3D, synthetic image. Once that YouCam has a model of the face, it can color relevant portions. The selected lipstick color is virtually overlaid on the model’s lips, enabling the customer to visualize what the product will look like on them. YouCam also adjusts according to customer skin tone and lighting, making sure the virtual look is as close to the real-world one as possible.

YouCam leverages ML in two key ways. ML employs pattern recognition to help identify facial features, and ML algorithms suggest shades to customers based on products they’ve already chosen so far. The app can also upsell and suggest skincare products based on observed conditions. If YouCam identifies a dark spot, for example, a concealer pops up as a recommendation.

Related: Take a Deeper Look into the Dell XPS Family of Solutions

Touchless is timely and trendy

Facial recognition technology is not new. It first emerged in the 1960s with funding from the military.

“However, as technology and camera lenses improved, the utility of facial recognition (became more apparent),” Alice Chang, founder and CEO of Perfect Corp, said.

Today it is combined with AR and artificial intelligence (AI) effects for consumer use. Companies like Warby Parker for eyeglasses and Allbirds shoes are using similar technologies so customers can take their products for a test drive.

“Given consumers’ demands to try before they buy, it was natural for virtual try-ons (in beauty to take off first) before paving the way for others,” Chang said.

Neil Saunders, managing director of GlobalData’s retail division, said that AR- and AI-driven beauty technology was slowly gaining momentum even before the pandemic.

“Very few people actually want to go to a store and try on makeup. They are often already made up and don’t want to apply different ones,” Saunders said. “For retailers, there’s the problem of wastage of samples. Plus there’s the issue of hygiene, with customers sharing products.”

The hygiene part of the virtual try-on equation has become front and center during COVID-19. To adjust to the new pandemic landscape, Perfect Corp. has added new features so customers can go contactless when in the store. They can simply use voice and gesture controls to try on makeup virtually. The company’s YouCam technology also recognizes face masks of an in-store customer when it recreates the 3D mesh model of the face.

Virtual makeup try-ons are cleaner with less waste. Retailers spend less on samples they can’t sell and customers access a more hygienic way of trying on products.

Sign Up: Receive the StartupNation newsletter!

A wide palette of benefits

Apart from the immediate benefits of hygiene during the pandemic, AR- and AI-driven beauty tech helps retailers meet their customers where they are: on their devices.

“Consumers will continue to rely on technology as a means of connecting the dots between their physical and digital worlds,” Chang said. “As we spend more time in front of screens than ever before, beauty tech invites an interactive, hyper-engaged experience that rivals that of a physical one.”

Saunders doesn’t expect in-person experiences to go away entirely. Rather, he envisions AR apps becoming a part of the beauty landscape after the pandemic.

“Most people want a mix of digital and in-person experiences, they swing back and forth,” he explained.

AR-driven apps also drive customer engagement, Saunders said, which is valuable currency in the omnichannel e-commerce world. That customer engagement might even lead to bolder purchases: “These AR apps can be entertaining for the customer and very few people would want to try on a radical shade of lipstick or eyeshadow. The virtual experience, however, makes that easier.”

A subsequent bonus for retailers? Increased sales. For instance, Madison Reed, which sells hair color, found that 38 percent of those who use the company’s virtual try-on tool go on to buy products. In 2017, Perfect Corp. also found that users of the YouCam app are 1.6 times more likely to buy cosmetics compared to non-users.

Saunders also sees retailers using these digital footprints to their advantage: “There’s a lot of very rich data you can get from these types of apps to help with marketing, personalization and product development efforts.”

Retailers can analyze app usage patterns, track every click, and understand which products are proving to be more popular, and direct marketing and production resources accordingly.

Given that the global market for cosmetics is poised to reach almost $430 billion by 2022, AR and AI technologies also give retailers a chance at a greater slice of that pie. Makeup made easy, with just the right dab of drama, is here to stay. And that is the foundation of good news for retailers and customers alike.