It has been eight months since the U.S. issued novel coronavirus-related stay-at-home orders. While restaurants and some social activities have since resumed, many offices remain closed and non-essential workers continue to work remotely from their houses. With the relocation of so many people from offices equipped with enterprise-level bandwidth, to measly residential wi-fi connections, it is not surprising that networks are overloaded and zoom call quality can be low.
Researchers at NVIDIA have developed a solution to the problem affecting millions of workers across the world. They are using neural networks – a technique utilized in AI — to replace outdated video transmission practices. Not only is this technology improving the quality of calls for existing video users, but it is also broadening accessibility for those who did not have sufficient bandwidth to make video calls previously. “We want to provide a better experience for video communications with AI so even people who only have access to extremely low bandwidth can still upgrade from voice to video calls,” said Arun Mallya, an AI researcher at NVIDIA.
Mallya and his colleague Ming-Yu Liu began working on decreasing the amount of bandwidth required to make video calls, after experiencing video call dropouts themselves. The researchers soon realized they could replace the video codec that is usually used to compress and decompress video during a call, with a generative adversarial network (GAN.) Implementing these AI techniques takes up just one-tenth of the network bandwidth that was required previously. “Instead of streaming the entire screen of pixels, the AI software analyzes the key facial points of each person on a call and then intelligently re-animates the face in the video on the other side,” says NVIDIA. “This makes it possible to stream video with far less data flowing back and forth across the internet.”
The work NVIDIA is doing is a great example of innovation brought about by COVID-19, which will now become a part of the fabric of our lives. “The pandemic motivated us because everyone is doing video conferencing now, so we explored how we can ease the bandwidth bottlenecks so providers can serve more people at the same time,” said Liu. The director of advanced products at NVIDIA agrees. “Video conferencing is going through a renaissance,” Andrew Page said. “Through the pandemic, we’ve all lived through its warts, but video is here to stay now as a part of our lives going forward because we are visual creatures.”
The team behind minimizing video bandwidth has also applied GANs to improve other dimensions of video calls. It is now possible for users to face away from the camera, but appear on the video call as if they are looking directly into the camera, facilitating a more direct eye-to-eye style of communication. “With computer vision techniques, we can locate a person’s head over a wide range of angles, and we think this will help people have more natural conversations,” says NVIDIA’s Ting-Chun Wang, a researcher involved in the innovation.
NVIDIA is combining these changes, as well as face re-lighting, and real-time language translation and transcription, into a new product it calls ‘Maxine.’ While the use case NVIDIA is proposing provides solutions for many users struggling to stay on video calls, there are potential downsides to the use of this technology. The GAN technique is commonly used in the creation of ‘deep fakes,’ or media-generated for disinformation purposes. As with so many advances in technology, the re-animation of people’s faces can be created for benevolent reasons such as in this scenario. But the flip side of that coin is that GANs can also be misused by bad actors with nefarious intent in other use cases. To the user, it is not immediately clear that a video doctored by GAN technology is fake and being used to deceive and misrepresent the truth. As the use of AI and GANs in video becomes more ubiquitous, a new problem around transparency arises, giving technology companies yet another problem to research and solve.
Artificial Intelligence is being implemented in industries all over the world and is a central theme of the research undertaken at UCIPT. Our work in the HOPE study is using data to assess and shift behavioral outcomes among HIV and other populations.