TSVP Talk: "Revisiting Scalarization in Multi-Task Learning" by Han Zhao

Title: Revisiting Scalarization in Multi-Task Learning

Abstract: Linear scalarization, i.e., combining all loss functions by a weighted sum, has been the default choice in the literature of multi-task learning (MTL) since its inception. In recent years, there has been a surge of interest in developing Specialized Multi-Task Optimizers (SMTOs) that treat MTL as a multi-objective optimization problem. However, it remains open whether there is a fundamental advantage of SMTOs over scalarization. In fact, heated debates exist in the community comparing these two types of algorithms, mostly from an empirical perspective. In this talk, I will revisit scalarization from a theoretical perspective. I will be focusing on linear MTL models and studying whether scalarization is capable of fully exploring the Pareto front. Our findings reveal that, in contrast to recent works that claimed empirical advantages of scalarization, scalarization is inherently incapable of full exploration, especially for those Pareto optimal solutions that strike the balanced trade-offs between multiple tasks. More concretely, when the model is under-parametrized, we reveal a multi-surface structure of the feasible region and identify necessary and sufficient conditions for full exploration. This leads to the conclusion that scalarization is in general incapable of tracing out the Pareto front. Our theoretical results provide a more intuitive explanation of why scalarization fails beyond non-convexity. I will conclude the talk by briefly discussing the extension of our results to general nonlinear neural networks.

Profile: Dr. Han Zhao is an Assistant Professor of Computer Science and, by courtesy, of Electric and Computer Engineering at the University of Illinois Urbana-Champaign (UIUC). He is also an Amazon Visiting Academic at Amazon AI. Dr. Zhao earned his Ph.D. degree in machine learning from Carnegie Mellon University. His research interest is centered around trustworthy machine learning, with a focus on algorithmic fairness, robust generalization under distribution shifts and model interpretability. He has been named a Kavli Fellow of the National Academy of Sciences and has been selected for the AAAI New Faculty Highlights program. His research has been recognized through a Google Research Scholar Award, an Amazon Research Award, and a Meta Research Award.

Language: English, no interpretation.

Target audience: General audience / everyone at OIST and beyond.
Freely accessible to all OIST members and guests without registration.

This talk will also be broadcast online via Zoom:

※ Please note that this event may be recorded and the videos uploaded. In addition, photos may be taken during the event. These are intended for publication online (the OIST website, social media, etc.)※