Talks

Smart IoT Speakers, while connected over a network, currently only produce sounds that come directly from the individual devices. We envision a future where smart speakers collaboratively produce a fabric of spatial audio, capable of perceptually placing sound in a range of locations in physical space. This could provide audio cues in homes, offices and public spaces that are flexibly linked to various positions. The perception of spatialized audio relies on binaural cues, especially the time difference and the level difference of incident sound at a user’s left and right ears. Traditional stereo speakers cannot create the spatialization perception for a user when playing binaural audio due to auditory crosstalk, as each ear hears a combination of both speaker outputs. We present Xblock, a novel time-domain pose-adaptive crosstalk cancellation technique that creates a spatial audio perception over a pair of speakers using knowledge of the user’s head pose and speaker positions. We build a prototype smart speaker IoT system empowered by Xblock, explore the effectiveness of Xblock through signal analysis, and discuss future perceptual user studies and future work.

What we feel from handling liquids in vessels produces unmistakably fluid tactile sensations. These stimulate essential perceptions in home, laboratory, or industrial contexts. Feeling fluid interactions from virtual fluids would similarly enrich experiences in virtual reality. We introduce Geppetteau, a novel string-driven weight shifting mechanism capable of providing perceivable tactile sensations of handling virtual liquids within a variety of vessel shapes. These mechanisms widen the range of augmentable shapes beyond the state-of-the-art of existing mechanical systems. In this work, Geppetteau is integrated into conical, spherical, cylindrical, and cuboid shaped vessels. Variations of these shapes are often used for fluid containers in our day-to-day. We studied the effectiveness of Geppetteau in simulating fine and coarse-grained tactile sensations of virtual liquids across three user studies. Participants found Geppetteau successful in providing congruent physical sensations of handling virtual liquids in a variety of physical vessel shapes and virtual liquid volumes and viscosities

The JMARS XR Platform is a virtual reality platform that allows multiple users to collaborate on the analysis of planetary terrain data. Based on ASU’s Mars Space Flight Facility JMARS GIS tool, it is designed to support a wide range of scientific research tasks, including geologic mapping, feature analysis, and mission planning.

NASA Science Theater at AGU 2022

LensCap is a split-process application development framework that protects user's visual privacy against malicious data collection in cloud-based AR applications. With LensCap, an AR application is split into a visual process with full access to operate on camera frames (but with network permission revoked) and a network process to maintain Internet communications (but with camera permission revoked), with the data transactions between split processes monitored and shown to users for approval at a fine granularity. With LensCap, users can decide what forms of visual data can be transmitted to the network, while still allowing visual data to be used for AR purposes on devices. We prototype LensCap as an Android library and demonstrate its usability as a plugin in Unreal Engine. Performance evaluation results on five AR apps confirm that visual privacy can be preserved with an insignificant latency penalty (< 1.3 ms) at 60 FPS.

High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates. To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different “rhythmic pixel regions” in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms. We integrate our encoding and decoding hardware modules into existing video pipelines. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. Evaluating our system on a Xilinx FPGA platform over three vision workloads shows 43 − 64% reduction in interface traffic and memory footprint, while providing controllable task accuracy.

High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates. To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different “rhythmic pixel regions” in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms. We integrate our encoding and decoding hardware modules into existing video pipelines. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. Evaluating our system on a Xilinx FPGA platform over three vision workloads shows 43 − 64% reduction in interface traffic and memory footprint, while providing controllable task accuracy.

The physical world, in all its complexity, has visual information that has varying spatial and temporal quality needs. Mobile systems, including smartphones, tablets, and headsets, could benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary for computer vision and augmented reality tasks. This talk will discuss challenges and opportunities of mobile operating systems and vision sensing pipeline architectures to flexibly support dynamic multi-resolution workloads with rapid reconfigurability at low latencies, and expressiveness to meet computational needs with minimal developer burden.

Current VR/AR systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, an ungrounded mixed-reality interface, capable of affording the users a realistic haptic sensation of fluid behaviors in vessels. The chief mechanism behind SWISH is in the use of virtual reality tracking and motor actuation to actively relocate the center of gravity of a handheld vessel, emulating the moving center of gravity of a handheld vessel that contains fluid. In addition to solving challenges related to reliable and efficient motor actuation, our SWISH designs place an emphasis on reproducibility, scalability, and availability to the maker culture. Our virtual-to-physical coupling uses Nvidia Flex's Unity integration for virtual fluid dynamics with a 3D printed augmented vessel containing a motorized mechanical actuation system. This opens the potential for multi-modal interactions with programmable fluids in virtual environments for chemistry education, worker training, and immersive entertainment.

Augmented reality provides a precise overlay of digital material over the physical world, enabling use cases in data visualization, instructional guidance, immersive education and entertainment. But beyond simple overlays, what opportunities can we pursue to further integrate our digital and physical worlds? In this talk, we will showcase some recent research projects of augmented reality, including SWISH devices that make AR fluids feel fluid-like and the GLEAM framework for illuminating virtual AR materials with lighting estimates from the physical environment. With a fuller integration between the digital and the physical, we’ll be able to create opportunities for future immersivity.

As the amount of Augmented Reality usage is drastically increasing, individuals are unknowingly using these devices to welcome strangers into their private space. The goal of this project is to protect AR users from exposing their private information, like credit card information, to malicious Application Developers.

Low resolution imaging/vision uses less energy per frame. High resolution imaging/vision enables precise visual understanding. We propose a mobile vision systems that benefits from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary.

...

The uitlity of computing is derived from the kinds of input the computing system can sense. With the ability to interpret subjects, environments, peripheral objects, and other information, our computers will eventually realize ubiquitous computing, a future where computing invisibly enhances our world. This will include technology that locates missing keys, tracks dietary habits, stimulates globally-connected education, and helps Alzheimer's patients remember their loved ones. Unfortunately, many barriers stand in the way of ubiquitous computing. The Mobile Systems Research Studio aims to tackle the systems implications of ubiquitous computing. Among many other projects, we envision designs for (1) ultra-low power consumption so devices do not need to charge from a wall; (2) privacy mechanisms to give users awareness and control over their sensed data; and (3) rich multimedia interaction for mixed-reality visualization. Whatever the future of computing may look like, the Mobile Systems Research Studio plans to address fundamental systems challenges in getting there.

This talk will introduce Robert LiKamWa's research philosophy in designing the future of computing systems, an application of this philosophy to his prior work, and his strategy to move ubiquitous computing forward.

Starfish retrofits vision libraries for split-process execution. This allows multiple background applications to transparently share computation results through library function caching.

We teardown and analyze the Google Glass power consumption to motivate and inspire system efficiency research directions.

We characterize the energy consumption of image sensors to reveal opportunities for energy proportionality to tradeoff quality for power consumption. We describe simple techniques to enable such proportionality.

The MoodSense project studies the use of supervised machine learning to mine inferences from smartphone usage patterns. We analyze text, call, email, location, app usage, and website browsing patterns against mood activeness and valence.