LensCap

LensCap is a split-process application development framework that protects user's visual privacy against malicious data collection in cloud-based AR applications. With LensCap, an AR application is split into a visual process with full access to operate on camera frames (but with network permission revoked) and a network process to maintain Internet communications (but with camera permission revoked), with the data transactions between split processes monitored and shown to users for approval at a fine granularity. With LensCap, users can decide what forms of visual data can be transmitted to the network, while still allowing visual data to be used for AR purposes on devices. We prototype LensCap as an Android library and demonstrate its usability as a plugin in Unreal Engine. Performance evaluation results on five AR apps confirm that visual privacy can be preserved with an insignificant latency penalty (< 1.3 ms) at 60 FPS.

Running AR apps on today’s mobile devices poses serious privacy concerns, potentially revealing private user information in a user’s visual environment to third party entities without the user’s knowledge. Under the current permission enforcement model, an AR app is given Internet permission at installation time and granted camera permission and external storage write permission at runtime by users. Developers are required to prompt users with contextual information about why certain permissions are required, but such permissions are seemingly justified for proper AR operation; camera frames are necessary to visually integrate virtual content with a user’s physical environment and Internet connectivity is needed for cloud-powered services or multiplayer networking. But once enabled, malicious developers of AR apps could silently collect camera frames and the information derived from them for malicious intent, including sending visual data to a private server, unbeknownst to the user. Without granular control over what kind of visual data is accessible for local storage or cloud storage, those collected camera frames could contain very private data at any given time, ranging from credit cards left on the table, text recognized from business documents on laptop monitors, to critical facial identities. How do we protect users from surreptitious collection of visual data while maintaining usable visual computing for AR applications?

LensCap adopts split-process access control in the application layer, integrated into the app development flow. An AR application is split into a visual process with full access to operate on camera frames (but with network permission revoked) and a network process to maintain Internet communications (but with camera permission revoked), enforced by extending the legacy Android permission enforcement. We enable both processes to present user interfaces through screen-based overlay composition. Then, data related to camera frames that need to be used in the network process can only be transmitted out of the visual process boundary through our trusted LensCap communication services, wrapped around trusted AR frameworks, and subject to the user’s monitoring and approval through LensCap data usage notifications at a fine granularity. If users wish to allow network access to entire camera frames, e.g., for social media sharing or cloud-powered vision, they can enable such permission. On the other hand, if a user wants to limit network access to only the camera pose, e.g., for multiplayer purposes, the user will be able to do so while still enjoying a full AR overlay on the device.

LensCap split-process app development framework enables: (i) AR apps and vision libraries to have expressive access of camera frames, their processing, and their rendering; (ii) fine-grained user control of the potential transmission of visual data; (iii) detailed context provided to users, regarding what data is sent to the cloud and at what times.

LensCap is integrated into the AR app development flow in Unreal Engine. At a high level, AR developers relying on the UE platform can download LensCap as a plugin. Please refer to our GitHub repo and UE documentations for detailed instructions, e.g., understanding how to add a Plugin to an existing UE project and how to make LensCap-integrated UE apps deployable to Android devices. Currently, LensCap only supports the collection of visual data exposed by the Google ARCore API. As detailed in the paper, LensCap wraps around the Google ARCore API and monitors the usage of those APIs at the LensCap split-process boundary. To use those APIs, simply search the type of visual data you need to collect from the user in UE Blueprint and place the function into your visual coding flow just as the legacy UE development. LensCap automatically handles the data communication across UE and Android. The figure belows shows the usage of one LensCap API in UE Blueprint to collect camera pose from the visual process to the network process, generated from the Google ARCore API.

We evaluated LensCap in five AR apps with different types of visual data (camera pose, lighting estimation, point clouds, face tracking, and full frame) required to be shared between devices. These apps are developed in UE (v4.24) and then deployed to Android devices (Google Pixel 4 XL). A local desktop server serves as a cloud server, storing and passing data among mobile devices. Uploading and downloading data uses an OkHttp client implementation with a local WiFi connection.

LensCap is able to maintain the same application performance, as demonstrated in the figure below. In other words, the adoption of split-process access control does not appear to influence app performance.

LensCap split-process access control only introduces a negligible latency in its inter-process communications, as (i) the 0.2 ms to 0.3 ms overhead in T1 (touch to visual process) is all but invisible, compared with the latency needs for gaming and other interactive touch-based applications; and (ii) the 0.3 ms to 1.3 ms latency of T2 (visual process to network process) and T4 (network process to visual process) caused by the inter-process communication is negligible (without impairing the app performance), even for transmitting the entire camera frame, and even for round-trip operations and interactions across the two processes

Apart from the previous quantitative evaluation, we perform a user study to observe users’ hands-on experience of LensCap-integrated AR apps. We found out that all participants felt that LensCap would improve their confidence while using untrusted AR apps. They already feel safe when the LensCap functioning logo is displayed and even safer when notification banners are prompted for asking their permissions.

Jinhan Hu, Andrei Iosifescu, Robert LiKamWa. 2021. LensCap: Split-Process Framework for Fine-Grained Visual Privacy Control for Augmented Reality.
MobiSys '21.
[ Full Paper (PDF)]