CIRC: Dev: SMILE: Scan to Multi-sensorial Interactive Learning Environment

VR (Virtual Reality)-based immersive and interactive environments can be a great resource for learning and training, especially for concepts that involve safety aspects by interacting with inflammable or breakable objects. The tedious nature of developing these VR experiences continues to be a limiting factor for VR becoming mainstream. The main goal of this project is the design and development of an innovative infrastructure, SMILE (Scan to Multi-sensorial Interactive Learning Environment), focusing on: (1) nearly automated construction of VR environments that mimic real-world indoor scenes; (2) interactions with virtual objects involving multiple senses such as touch, visual, aural, and smell. The SMILE infrastructure will undergo rigorous software testing to ensure safe interactions before being deployed to support Internet-scale collaboration among users.

In the exploratory phase of this project, the team will address the following: (1) Democratize the creation of VR environments for subject-matter experts that are realistic in reflecting the ambiance of a real-world scene while providing the needed interactions with objects in the scene. (2) Apply sensorial information directly to 3D objects, enabling multiple sensations during interactions with the virtual objects. (3) Internet-scale collaborations in SMILE through multi-modal inter-sender and inter-receiver skew synchronization, based on the temporal requirements of multi-sensorial interactions. This will be tested by deploying SMILE for use in the sites of the project partners: the University of Texas at Dallas, the University of Illinois at Urbana-Champaign, the New Jersey Institute of Technology, and the Dallas College. (4) A coverage-based fuzz testing module where numerous inputs can be generated and executed efficiently to test corner cases in the SMILE components, and configurable static analysis where algorithms can be flexibly tuned to check the most critical safety properties in SMILE.

Virtual chemistry laboratory experiments for assessing the performance of SMILE will be designed and developed through research collaboration with the Dallas College. Through this collaboration, the SMILE project will have a significant impact on the broad undergraduate student community. SMILE could be used for different learning environments, apart from the Chemistry lab case study, promoting education involving K-16. The project will involve doctoral students, undergraduate students, and K-12 students.

Open-source resources produced through the SMILE project include a multi-sensory database of material properties for sensorial displays, Unity-based VR authoring tools, and automated software testing tools for VR-based applications. These will be made available for a period of five years through the project website at: https://labs.utdallas.edu/multimedialab/.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Award Number: 2603236
Principal Investigator: Balakrishnan Prabhakaran
Funds Obligated: $514,915
State: NY