Introduction

Low resolution imaging/vision uses less energy per frame. High resolution imaging/vision enables precise visual understanding. Mobile vision systems would benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary.

Unfortunately, today, many systems have a substantial pause in frame delivery whenever there seems to be a change in sensor resolution – as much as 280 ms. Switching to a new resolution takes time since the system has to go through a lot of reconfiguration and manage memory before it can start showing frames again. This delay makes it harder to take advantage of smart features that could save energy by changing resolution.

Key Features

• Seamless camera resolution switching: Designed to eliminate the delay (up to 280 ms) that usually occurs when changing a camera's resolution.
• Energy efficiency: Allow applications to lower resolution when full detail isn't needed, allowing for lower system energy use.
• Shorter reconfiguration time: Reusing memory buffers and reconfiguring the camera with the app execution can shorten Banner reconfiguration time to make mobile vision apps faster and more efficient.

Solution

We created Banner, a media framework that makes changing camera resolution much faster by improving existing systems like V4L2. Banner skips most of the usual slow steps by: (1) reusing memory instead of allocating it again, and (2) setting the sensor format at the same time the app is running. The figure below compares how resolution changes work before and after adding Banner to the system.

Process

Banner uses two main tricks to speed things up: parallel reconfiguration and format-oblivious memory management. With parallel reconfiguration, the camera starts switching resolutions while the app is still using the old one—so there's no waiting around. And with format-oblivious memory, Banner keeps using the same memory buffers no matter the resolution, which avoids the slow process of reallocating memory every time.

The figure below illustrates the parallel reconfiguration strategy and format-oblivious memory management.

Results

As shown in the figure below, Banner cuts the resolution switch time from 226 ms to just 33 ms – completely removing the delay between frames. It also reduces the total reconfiguration time by more than half – 226 ms to 105 ms. This allows vision apps to save over 49% of system power by switching the camera to 480p instead of downsampling from 1080p. These results, tested on a Jetson TX2 with a cloud-based workload, show that Banner makes it possible for mobile vision apps to quickly change resolution and find a better balance between energy use and accuracy.