The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable.
To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics.

People

• PI
  Zhisheng Yan
• External Collaborator
  Klara Nahrstedt (UIUC), Bo Chen (UIUC)
• Graduate Students
  Anh Phan Nguyen, Taslim Murad, Jingwei Liao
• Undergraduate Students
  Thai-hoa Nguyen

Publications

• ST-360: Spatial–Temporal Filtering-Based Low-Latency 360-Degree Video Analytics Framework
  Jiaxi Li, Jingwei Liao, Bo Chen, Anh Nguyen, Aditi Tiwari, Qian Zhou, Zhisheng Yan, and Klara Nahrstedt
  ACM TOMM (ACM Transactions on Multimedia Computing, Communications, and Applications), September 2024. [PDF]
• ImmerScope: Multi-view Video Aggregation at Edge towards Immersive Content Services
  Bo Chen, Hongpeng Guo, Mingyuan Wu, Zhe Yang, Zhisheng Yan, and Klara Nahrstedt
  ACM SenSys (ACM Conference on Embedded Networked Sensor Systems), November 2024. [PDF]
• NeRFHub: A Context-Aware NeRF Serving Framework for Mobile Immersive Applications
  Bo Chen, Zhisheng Yan, Bo Han, and Klara Nahrstedt
  ACM MobiSys (ACM International Conference on Mobile Systems, Applications and Services), June 2024. [PDF]
• LiFteR: Unleash Learned Codecs in Video Streaming with Loose Frame Referencing
  Bo Chen, Zhisheng Yan, Yinjie Zhang, Zhe Yang, and Klara Nahrstedt
  USENIX NSDI (USENIX Symposium on Networked Systems Design and Implementation), April 2024. [PDF]
• Vesper: Learning to Manage Uncertainty in Video Streaming
  Bo Chen, Mingyuan Wu, Hongpeng Guo, Zhisheng Yan, and Klara Nahrstedt
  ACM MMSys (ACM International Conference on Multimedia Systems), April 2024. [PDF]
• Context-aware Optimization for Bandwidth-Efficient Image Analytics Offloading
  Bo Chen, Zhisheng Yan, and Klara Nahrstedt
  ACM TOMM (ACM Transactions on Multimedia Computing, Communications, and Applications), December 2023. [PDF]
• Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays
  Anh Nguyen and Zhisheng Yan
  Sensors, April 2023. [PDF]
• Context-aware Image Compression Optimization for Visual Analytics Offloading
  Bo Chen, Zhisheng Yan, and Klara Nahrstedt
  ACM MMSys (ACM International Conference on Multimedia Systems), June 2022. [PDF]
• DAO: Dynamic Adaptive Offloading for Video Analytics
  Taslim Murad, Anh Nguyen, and Zhisheng Yan
  ACM MM (ACM International Conference on Multimedia), October 2022. [PDF]

Funding

This project is sponsored by the National Science Foundation CAREER Program under award number 2144764.