The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable. To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics.
People
- PI Zhisheng Yan
- External Collaborator Klara Nahrstedt, UIUC
- Students Taslim Murad, Bo Chen
Publications
- Context-aware Image Compression Optimization for Visual Analytics Offloading Bo Chen, Zhisheng Yan, and Klara Nahrstedt ACM MMSys, (ACM International Conference on Multimedia Systems), June, 2022. [PDF]
- DAO: Dynamic Adaptive Offloading for Video Analytics Taslim Murad, Anh Nguyen, and Zhisheng Yan ACM MM (ACM International Conference on Multimedia), October 2022. [PDF]
Code and Data
Funding
This project is sponsored by the National Science Foundation CRII Program under award number 2144764.