CAREER: Vision Systems for an Evolving World

When transitioning into a dark room most people initially struggle to see, but are quickly able to adjust and see in the new setting. In fact, people continue to see and understand the world around them even as the appearance of the world changes in many different ways. In contrast, our computer vision systems have limited ability to understand the world if it changes. Imagine if the first time one drove at twilight or in the snow one could no longer recognize the road. No doubt, driver’s training would require many hours of sitting in the passenger seat at all times of day, within different weather conditions, across different cities, and so on before allowing a new driver behind the wheel. This project aims to study and build new models, algorithms, and measures of success enabling the next generation of visual recognition systems to be resilient to an evolving visual world. The project will integrate research with education and outreach to K-12 students.

This project advocates for resilient vision systems through a new integrated approach which iterates between generalizing across available visual domains and rapidly adapting given new domain data. Prior approaches optimize for independent criteria, either generalization across multiple domains, or adaptation to a new target domain, which limits advancement towards the larger goal of creating vision systems that can operate in more domains over time. Further, existing solutions are slow to adapt, relying on substantial new observations before updates can be made. The project will work on: 1) Model design and learning approaches for multi-domain generalization that facilitates future adaptation. 2) Transformative visual domain adaptation algorithms that are capable of rapidly adapting to a target domain using limited target observations and without accessing a large auxiliary source of data, reducing compute demands. 3) Algorithms that enable vision systems to expand the set of domains they can successfully operate in over time. Finally, this project will introduce a benchmark and new evaluation protocols to measure the resilience of visual recognition models to changing domains.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Award Number: 2622839
Principal Investigator: Judy Hoffman
Funds Obligated: $412,579
State: CA