Remote commanders have traditionally visualized emergency sites and led the operation using videos captured by helmet cameras of firefighters. Unfortunately, these video systems suffer from a fundamental problem, i.e., commanders are only able to see a single view of the emergency site at a time. This limitation restricts the situation awareness of commanders and leads to productivity and safety issues such as miscommunication of locations and failure to identify dangerous events. This project combines 360-degree videos and augmented reality to enable remote commanders to achieve 360-degree situation awareness of the entire emergency site in all viewing directions and enhance firefighting productivity and safety. The 360-degree video viewing benefits various emergency response communities in planning and training. To this end, we propose an augmented 360 video viewing system. We first understand commanders' needs to identify and categorize important objects and events in firefighter response. We then design an object detection model to enable automatic machine detection of important objects in 360 firefighting videos. Finally, we validate the augmented 360 video viewing through system and user evaluation.

People

• PI at George Mason
  Zhisheng Yan
• PI at UIUC
  Klara Nahrstedt
• Postdocs
  Qian Zhou, Bo Chen
• Graduate Students
  Jiaxi Li, Jingwei Liao, Aditi Tiwari, Anh Nguyen, Ayush Sarkar
• Undergraduate Students
  David Joy, Emmanuel Hernandez

Publications

• Latency-Aware 360-Degree Video Analytics Framework for First Responders Situational Awareness
  Jiaxi Li, Jingwei Liao, Bo Chen, Anh Nguyen, Aditi Tiwari, Qian Zhou, Zhisheng Yan, and Klara Nahrstedt
  NOSSDAV (ACM 33rd Workshop on Network and Operating System Support for Digital Audio and Video), June, 2023. [PDF]
• A 360-Degree Video Analytics Service for In-Classroom Firefighter Training
  Ayush Sarkar, Ahn Nguyen, Zhisheng Yan, and Klara Nahrstedt
  CPS-ER (CPS-IOT Week Workshop on Cyber Physical Systems for Emergency Response), June, 2022. [PDF]

Talks

• Illinois Fire Service Institute Retreat
  360-Degree Video Analytics Framework for First Responders Situational Awareness, presented by Jiaxi Li, Jingwei Liao, and Aditi Tiwari.

  [Slides]

• ACM NOSSDAV'23 Presentation
  Latency-Aware 360-Degree Video Analytics Framework for First Responders Situational Awareness, presented by Zhisheng Yan.

  [Slides]

Deliverables

• 360ObjectAnnotator
  360ObjectAnnotator is the first open object detection annotation tool for 360 videos. It allows human annotators to label objects by visualizing a normal field of view of a 360 video, enabling distortion-free annotations.

  [Link]

Funding

This collaborative project is sponsored by the National Science Foundation EAGER Program under award number 2140620 and 2140645.