Bo Dong

Email bo {dot} dong [at] princeton {dot} edu

“I have no special talent. I am only passionately curious.” –Albert Einstein

Short Biography: I am currently a visiting researcher at Princeton University and working closely with Prof. Felix Heide. My research lies at the intersection of machine learning, computer vision, computer graphics, and neuroscience, and it seeks solutions to enhance the adaptation capability, explainability, and computational & energy efficiency of computational imaging systems on various energy-limited platforms, such as edge devices, drones, and mobile robots, under different realistic degraded conditions (e.g., external environmental changes, malfunctioning sensors, and weight uncertainty of neural networks). The overarching vision of my research is to leverage bio-plausible neuroscientific observations and theories. This brings the tantalizing promise of human-like homeostatic computational imaging systems, which can spontaneously maintain a relatively constant internal environment. The endowed homeostasis capability can decouple physiology from immediate external conditions and thereby significantly extend the range of available application scenarios. I received a Ph.D. in Computer Science from the College of William and Mary in 2015.

News

Mar 1, 2022	Two papers accepted at CVPR 2022
Oct 26, 2021	One paper accepted at Applied AI Letters
Oct 1, 2021	I joined Princeton University as a Visiting Researcher! Very exciting!
Aug 26, 2021	One paper accepted at CGF 2021
Jul 22, 2021	One paper accepted at ICCV 2021

Selected Publications

CVPR’22

Glass Segmentation using Intensity and Spectral Polarization Cues

Mei, Haiyang, Dong, Bo, Dong, Wen, Yang, Jiaxi, Baek, Seung-Hwan, Heide, Felix, Peers, Pieter, Wei, Xiaopeng, and Yang, Xin

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Abs

Transparent and semi-transparent materials pose significant challenges for existing scene understanding and segmentation algorithms due to their lack of RGB texture which impedes the extraction of meaningful features. In this work, we exploit that the light-matter interactions on glass materials provide unique intensity-polarization cues for each observed wavelength of light. We present a novel learning-based glass segmentation network that leverages both trichromatic (RGB) intensities as well as trichromatic linear polarization cues from a single photograph captured without making any assumption on the polarization state of the illumination. Our novel network architecture dynamically fuses and weights both the trichromatic color and polarization cues using a novel global-guidance and multi-scale self-attention module, and leverages global cross-domain contextual information to achieve robust segmentation. We train and extensively validate our segmentation method on a new large-scale RGB-Polarization dataset (RGBP-Glass), and demonstrate that our method outperforms state-of-the-art segmentation approaches by a significant margin.
CVPR’22

Spiking Transformers for Event-based Single Object Tracking

Zhang, Jiqing, Dong, Bo, Zhang, Haiwei, Ding, Jianchuan, Heide, Felix, Yin, Baocai, and Yang, Xin

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Abs

Event-based cameras bring a unique capability to tracking, being able to function in challenging real-world conditions as a direct result of their high temporal resolution and high dynamic range. These imagers capture events asynchronously that encode rich temporal and spatial information. However, effectively extracting this information from events remains an open challenge. In this work, we propose a spiking transformer network, STNet, for single object tracking. STNet dynamically extracts and fuses information from both temporal and spatial domains. In particular, the proposed architecture features a transformer module to provide global spatial information and a spiking neural network (SNN) module for extracting temporal cues. The spiking threshold of the SNN module is dynamically adjusted based on the statistical cues of the spatial information, which we find essential in providing robust SNN features. We fuse both feature branches dynamically with a novel cross-domain attention fusion algorithm. Extensive experiments on two event-based datasets, FE240hz and EED, validate that the proposed STNet outperforms existing state-of-the-art methods in both tracking accuracy and speed with a significant margin.
ICCV’21

Object Tracking by Jointly Exploiting Frame and Event Domain

Zhang, Jiqing, Yang, Xin, Fu, Yingkai, Wei, Xiaopeng, Yin, Baocai, and Dong, Bo

In IEEE International Conference on Computer Vision (ICCV) 2021

Abs HTML PDF

Inspired by the complementarity between conventional frame-based and bio-inspired event-based cameras, we propose a multi-modal based approach to fuse visual cues from the frame- and event-domain to enhance the single object tracking performance, especially in degraded conditions (e.g., scenes with high dynamic range, low light, and fast-motion objects). The proposed approach can effectively and adaptively combine meaningful information from both domains. Our approach’s effectiveness is enforced by a novel designed cross-domain attention schemes, which can effectively enhance features based on self- and cross-domain attention schemes; The adaptiveness is guarded by a specially designed weighting scheme, which can adaptively balance the contribution of the two domains. To exploit event-based visual cues in single-object tracking, we construct a large-scale frame-event-based dataset, which we subsequently employ to train a novel frame-event fusion based model. Extensive experiments show that the proposed approach outperforms state-of-the-art frame-based tracking methods by at least 10.4% and 11.9i% in terms of representative success rate and precision rate, respectively. Besides, the effectiveness of each key component of our approach is evidenced by our thorough ablation study.
CVPR’21 Oral

Depth-Aware Mirror Segmentation

Mei, Haiyang, Dong, Bo, Dong, Wen, Peers, Pieter, Yang, Xin, Zhang, Qiang, and Wei, Xiaopeng

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Oral 2021

Abs HTML PDF

We present a novel mirror segmentation method that leverages depth estimates from ToF-based cameras as an additional cue to disambiguate challenging cases where the contrast or relation in RGB colors between the mirror reflection and the surrounding scene is subtle. A key observation is that ToF depth estimates do not report the true depth of the mirror surface, but instead return the total length of the reflected light paths, thereby creating obvious depth discontinuities at the mirror boundaries. To exploit depth information in mirror segmentation, we first construct a large-scale RGB-D mirror segmentation dataset, which we subsequently employ to train a novel depth-aware mirror segmentation framework. Our mirror segmentation framework first locates the mirrors based on color and depth discontinuities and correlations. Next, our model further refines the mirror boundaries through contextual contrast taking into accountboth color and depth information. We extensively validate our depth-aware mirror segmentation method and demonstrate that our model outperforms state-of-the-art RGB and RGB-D based methods for mirror segmentation. Experimental results also show that depth is a powerful cue for mirror segmentation.
MobiSys’20

EMO: Real-Time Emotion Recognition from Single-Eye Images for Resource-Constrained Eyewear Devices

Wu, Hao, Feng, Jinghao, Tian, Xuejin, Sun, Edward, Liu, Yunxin, Dong, Bo, Xu, Fengyuan, and Zhong, Sheng

In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services 2020

Abs HTML PDF

Real-time user emotion recognition is highly desirable for many applications on eyewear devices like smart glasses. However, it is very challenging to enable this capability on such devices due to tightly constrained image contents (only eye-area images available from the on-device eye-tracking camera) and computing resources of the embedded system. In this paper, we propose and develop a novel system called EMO that can recognize, on top of a resource-limited eyewear device, real-time emotions of the user who wears it. Unlike most existing solutions that require whole-face images to recognize emotions, EMO only utilizes the single-eye-area images captured by the eye-tracking camera of the eyewear. To achieve this, we design a customized deep-learning network to effectively extract emotional features from input single-eye images and a personalized feature classifier to accurately identify a user’s emotions. EMO also exploits the temporal locality and feature similarity among consecutive video frames of the eye-tracking camera to further reduce the recognition latency and system resource usage. We implement EMO on two hardware platforms and conduct comprehensive experimental evaluations. Our results demonstrate that EMO can continuously recognize seven-type emotions at 12.8 frames per second with a mean accuracy of 72.2%, significantly outperforming the state-of-the-art approach, and consume much fewer system resources.
SIGGRAPH’15

Measurement-Based Editing of Diffuse Albedo with Consistent Interreflections

Dong, Bo, Dong, Yue, Tong, Xin, and Peers, Pieter

ACM Trans. Graph. 2015

Abs PDF Video

We present a novel measurement-based method for editing the albedo of diffuse surfaces with consistent interreflections in a photograph of a scene under natural lighting. Key to our method is a novel technique for decomposing a photograph of a scene in several images that encode how much of the observed radiance has interacted a specified number of times with the target diffuse surface. Altering the albedo of the target area is then simply a weighted sum of the decomposed components. We estimate the interaction components by recursively applying the light transport operator and formulate the resulting radiance in each recursion as a linear expression in terms of the relevant interaction components. Our method only requires a camera-projector pair, and the number of required measurements per scene is linearly proportional to the decomposition degree for a single target area. Our method does not impose restrictions on the lighting or on the material properties in the unaltered part of the scene. Furthermore, we extend our method to accommodate editing of the albedo in multiple target areas with consistent interreflections and we introduce a prediction model for reducing the acquisition cost. We demonstrate our method on a variety of scenes and validate the accuracy on both synthetic and real examples.
CVPR’14

Scattering Parameters and Surface Normals from Homogeneous Translucent Materials using Photometric Stereo

Dong, Bo, Moore, Kathleen D., Zhang, Weiyi, and Peers, Pieter

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2014

Abs HTML PDF Video

This paper proposes a novel photometric stereo solution to jointly estimate surface normals and scattering parameters from a globally planar, homogeneous, translucent object. Similar to classic photometric stereo, our method only requires as few as three observations of the translucent object under directional lighting. Naively applying classic photometric stereo results in blurred photometric normals. We develop a novel blind deconvolution algorithm based on inverse rendering for recovering the sharp surface normals and the material properties. We demonstrate our method on a variety of translucent objects.