Erasing glasses in real time: 12 FPS on an iPhone 12

Virtual try-on has a problem nobody puts in the marketing video: it works beautifully for people who don’t already wear glasses. For everyone else (which, at an eyewear company, is most of your customers), the experience is broken. You point the camera at your face, the app renders a gorgeous new frame, and it sits on top of the glasses you’re already wearing. Two pairs of glasses. It looks ridiculous, and worse, it doesn’t answer the only question the customer has: do these look good on me? So we built the thing that sounds impossible when you say it out loud: remove the glasses the user is wearing, live, from the camera feed, before rendering the new ones. On the phone. Fast enough that it feels like video, not a slideshow. ...

Aug 7, 2024 · 6 min · Ashish Saini