Augmented 360 Video for Situation Awareness in Firefighting

Remote commanders have traditionally visualized emergency sites and led the operation using videos captured by helmet cameras of firefighters. Unfortunately, these video systems suffer from a fundamental problem, i.e., commanders are only able to see a single view of the emergency site at a time. This limitation restricts the situation awareness of commanders and leads to productivity and safety issues such as miscommunication of locations and failure to identify dangerous events. This project combines 360-degree videos and augmented reality to enable remote commanders to achieve 360-degree situation awareness of the entire emergency site in all viewing directions and enhance firefighting productivity and safety. The 360-degree video viewing benefits various emergency response communities in planning and training. To this end, we propose an augmented 360 video viewing system. We first understand commanders' needs to identify and categorize important objects and events in firefighter response. We then design an object detection model to enable automatic machine detection of important objects in 360 firefighting videos. Finally, we validate the augmented 360 video viewing through system and user evaluation.

People

  • PI at George Mason
    Zhisheng Yan
  • PI at UIUC
    Klara Nahrstedt
  • Postdocs
    Qian Zhou, Bo Chen
  • Graduate Students
    Jiaxi Li, Jingwei Liao, Aditi Tiwari, Anh Nguyen, Ayush Sarkar
  • Undergraduate Students
    David Joy, Emmanuel Hernandez

Publications

  • Latency-Aware 360-Degree Video Analytics Framework for First Responders Situational Awareness
    Jiaxi Li, Jingwei Liao, Bo Chen, Anh Nguyen, Aditi Tiwari, Qian Zhou, Zhisheng Yan, and Klara Nahrstedt
    NOSSDAV (ACM 33rd Workshop on Network and Operating System Support for Digital Audio and Video), June, 2023. [PDF]
  • A 360-Degree Video Analytics Service for In-Classroom Firefighter Training
    Ayush Sarkar, Ahn Nguyen, Zhisheng Yan, and Klara Nahrstedt
    CPS-ER (CPS-IOT Week Workshop on Cyber Physical Systems for Emergency Response), June, 2022. [PDF]

Talks

  • Illinois Fire Service Institute Retreat
    360-Degree Video Analytics Framework for First Responders Situational Awareness, presented by Jiaxi Li, Jingwei Liao, and Aditi Tiwari.

    [Slides]

  • ACM NOSSDAV'23 Presentation
    Latency-Aware 360-Degree Video Analytics Framework for First Responders Situational Awareness, presented by Zhisheng Yan.

    [Slides]

Deliverables

  • 360ObjectAnnotator
    360ObjectAnnotator is the first open object detection annotation tool for 360 videos. It allows human annotators to label objects by visualizing a normal field of view of a 360 video, enabling distortion-free annotations.

    [Link]

Funding

This collaborative project is sponsored by the National Science Foundation EAGER Program under award number 2140620 and 2140645.