Through-Screen Computing

The advancement in transparent screen technology has promoted adoption of full-screen design on mobile devices, reducing the area occupied by optical sensors to maximize the devices' screen-to-body ratio. In modern smartphones, front-facing optical sensors, such as ambient light sensor and camera, now must be placed under the transparent screen to capture ambient light and visual information. Motivated by this trend, we propose Through-Screen Computing in this dissertation. It is a new concept that refers to the processing of light signals for various computing purposes such as communication, sensing, and imaging, where the light comes from the physical world and passes through a special medium -- the transparent screen -- before reaching the under-screen optical sensors. This concept opens up new challenges and opportunities in connectivity, privacy, and security of future devices equipped with transparent screens. In this dissertation, we outline a vision for through-screen computing and address the challenges of transparent screens acting as both passive blockers and active interferers of input light signals.

This dissertation focuses on two subsystems in the context of through-screen computing: Through-Screen Visible Light Communication (VLC) and Screen Perturbation for Visual Privacy Protection. In the context of VLC, the full-screen trend challenges the deployment of this technology. We propose Through-Screen VLC with under-screen optical sensors as receivers. To address the attenuation of the light by the transparent screen, we develop SpiderWeb, a system exploiting the color domain to mitigate the color-pulling effect introduced by the transparent screen. We also leverage the Under-Screen Camera (USC) for VLC and design novel demodulation algorithms to reduce multi-pixel screen interference and improve performance. Experimental results show significant improvements in both data rate and transmission range, using  a prototype we build with two commercial smartphones. For privacy protection, we propose Screen Perturbations, modifying pixels displayed on the transparent screen to embed speckled color blocks and color shifts in the final image captured by the USC. Screen perturbations can be exploited to disrupt advanced deep neural networks used on image classification and face recognition tasks. We first design two image-specific methods to add screen perturbations to the images captured by USC. Next, we develop Unicorn, a universal screen perturbation method optimized for robustness in various conditions. All these designed perturbations work successfully against various deep neural network-based image classification services with high success rates.

Through these two subsystems, as well as the proposed theoretical and experimental approaches and results, we demonstrate the transformative potentials of through-screen computing, setting the stage for future research and development on various computing purposes in the era of transparent screen and full-screen devices.