Date/Time : 22 Apr 2022 14:00
Venue : via Zoom
Speaker : Zhou Pan, Sea Limited

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract:  To train networks, lookahead algorithm updates its fast weights k times via an inner-loop optimizer before updating its slow weights once by using the latest fast weights. Any optimizer, e.g. SGD, can serve as the inner-loop optimizer, and the derived lookahead generally enjoys remarkable test performance improvement over the vanilla optimizer. But theoretical understandings on the test performance improvement of lookahead remain absent yet. To solve this issue, we theoretically justify the advantages of lookahead in terms of the excess risk error which measures the test performance. Specifically, we prove that lookahead using SGD as its inner-loop optimizer can better balance the optimization error and generalization error to achieve a smaller excess risk error than vanilla SGD on (strongly) convex problems and nonconvex problems with Polyak-Łojasiewicz condition which has been observed/proved in neural networks. Moreover, we show the stagewise optimization strategy which decays the learning rate several times during training can also benefit lookahead in improving its optimization and generalization errors on strongly convex problems. Finally, we propose a stagewise locally-regularized lookahead (SLRLA) algorithm which sums up the vanilla objective and a local regularizer to minimize at each stage and provably enjoys optimization and generalization improvement over the conventional (stagewise) lookahead. Experimental results on CIFAR10/100 and ImageNet testify its advantages.

Add to calendar :

Date/Time : 08 Apr 2022 10:00
Venue : via Zoom
Speaker : Gianmarco Mengaldo, National University of Singapore

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract: We provide a perspective on the challenges and opportunities in computational modeling for industry-relevant applications towards the Exascale Era. As an example, we present the successful deployment of high-fidelity Large-Eddy Simulation (LES) technologies based on spectral/hp element methods to simulate the aerodynamics of a real automotive car, namely the Elemental Rp1 model. The simulation presents the common challenges of an industry-relevant problem, namely high Reynolds number and complex geometry. To the best of the authors’ knowledge, this simulation represents the first fifth-order accurate transient LES of an entire real car geometry using spectral element methods. Moreover, this constitutes a key milestone towards considerably expanding the computational design envelope currently allowed in industry, where steady-state modelling remains the standard. We also draw some parallels with other fields, namely numerical weather prediction, where large scale simulations play a central role and have been used since the 1950s.

Add to calendar :

Date/Time : 01 Apr 2022 10:00
Venue : via Zoom
Speaker : Rafael Peres Da Silva, National University of Singapore

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Large-scale cancer omics studies have highlighted the diversity of patient molecular profiles and the importance of leveraging this information to deliver the right drug to the right patient at the right time. Key challenges in learning predictive models for this include the high-dimensionality of omics data and heterogeneity in biological and clinical factors affecting patient response. The use of multi-task learning techniques has been widely explored to address dataset limitations for in vitro drug response models, while domain adaptation (DA) has been employed to extend them to predict in vivo responses. In both of these transfer learning settings, noisy data for some tasks (or domains) can substantially reduce the performance for others compared to single-task (domain) learners, i.e. lead to negative transfer (NT).

This talk introduces TUGDA (ISMB 2021), a novel multi-task unsupervised DA method, that addresses these limitations in a unified framework by quantifying uncertainty in predictors and weighting their influence on shared feature representations. TUGDA’s ability to rely more on predictors with low-uncertainty allowed it to notably reduce cases of NT for in vitro models (94% overall) compared to state-of-the-art methods. For DA to in vivo settings, TUGDA improved over previous methods for patient-derived xenografts (9 out of 14 drugs) as well as patient datasets (significant associations in 9 out of 22 drugs). TUGDA’s ability to avoid NT thus provides a key capability as we try to integrate diverse drug-response datasets to build consistent predictive models with in-vivo utility.

Add to calendar :

Date/Time : 24 Mar 2022 15:00
Venue : via Zoom
Speaker : Patrick Rebeschini, Oxford University

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory.

In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the “offset condition”) instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory. (Based on joint work with V. Kanade and T. Vaškevičius)

Add to calendar :

Date/Time : 18 Mar 2022 10:00
Venue : via Zoom
Speaker : Yian Ma, University of California at San Diego

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract:  I will introduce some recent progress towards understanding the scalability of Markov chain Monte Carlo (MCMC) methods and their comparative advantage with respect to variational inference. I will discuss an optimization perspective on the infinite dimensional probability space, where MCMC leverages stochastic sample paths while variational inference projects the probabilities onto a finite dimensional parameter space. Three ingredients will be the focus of this discussion: non-convexity, dimension dependence, and stochasticity. This line of work is motivated by epidemic prediction, where we need uncertainty quantification for credible predictions and informed decision making with complex models and evolving data.

Add to calendar :

Date/Time : 11 Mar 2022 10:00
Venue : via Zoom
Speaker : Jean Honorio, Purdue University

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Several problems in machine learning are combinatorial (e.g., clustering, learning sparse representations.) Convex relaxations have been used in the past, in order to circumvent the NP-hardness of the combinatorial problems, albeit being approximate solutions. My long-term research goal is to uncover large classes of continuous relaxations (beyond convexity) of worst-case NP-hard problems, that admit exact solutions.

In this talk, I will introduce a unifying framework, which uses the power of continuous relaxations (beyond convexity), Karush-Kuhn-Tucker conditions, primal-dual certificates and concentration inequalities. This framework has produced novel algorithms not only for classical worst-case NP-hard problems, e.g., Bayes nets, structured prediction, clustering, but also for aspects of fairness and meta learning.

As a first example, we study the problem of fair sparse regression on a biased dataset where bias depends upon a hidden binary attribute. The presence of a hidden attribute adds an extra layer of complexity to the problem by combining sparse regression and clustering with unknown binary labels. The corresponding optimization problem is combinatorial but we propose an invex relaxation, a subclass of nonconvex relaxations. We then provide exact recovery guarantees.

As a second example, we study the problem of exact partitioning of high-order models. Exact partitioning can be formulated as a tensor optimization problem. We relax this high-order combinatorial problem to a convex conic form problem. To this end, we carefully define the Caratheodory symmetric tensor cone, and show its convexity, and the convexity of its dual cone. We also provide exact recovery guarantees.

Add to calendar :

Date/Time : 04 Mar 2022 10:00
Venue : via Zoom
Speaker : Yunlong Hou, National University of Singapore

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract: We design and analyze VA-LUCB, a parameter-free algorithm, for identifying the best arm under the fixed-confidence setup and under a stringent constraint that the variance of the chosen arm is strictly smaller than a given threshold. An upper bound on VA-LUCB’s sample complexity is shown to be characterized by a fundamental variance-aware hardness quantity H_VA. By proving a lower bound, we show that sample complexity of VA-LUCB is optimal up to a factor logarithmic in H_VA. Extensive experiments corroborate the dependence of the sample complexity on the various terms in H_VA. By comparing VA-LUCB’s empirical performance to a close competitor RiskAverse-UCB-BAI by David et al. (2018), our experiments suggest that VA-LUCB has the lowest sample complexity for this class of risk-constrained best arm identification problems, especially for the riskiest instances.

Add to calendar :

Date/Time : 18 Feb 2022 10:00
Venue : via Zoom
Speaker : Yang Haizhao, Purdue University

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract: Learning operators between infinitely dimensional spaces is an important learning task arising in wide applications in machine learning, imaging science, mathematical modeling and simulations, etc. This talk introduces the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

Add to calendar :

Date/Time : 11 Feb 2022 10:00
Venue : via Zoom
Speaker : Soufiane Hayou, National University of Singapore

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Abstract: The theoretical analysis of infinite-width neural networks has led to many interesting practical results (choice of initialization schemes, choice of Bayesian priors etc.). However, the traditional infinite-width framework focuses on fixed depth networks and omits the large depth behavior of these models. In this talk, I will cover different topics on the infinite-width-then-infinite-depth networks and show how we can leverage some theoretical results to obtain principled algorithms. The following papers will be covered in this talk:

• On the Impact of the activation function on deep neural networks training. ICML 2019. (Joint work with A. Doucet and J. Rousseau)
• Stable ResNet. AISTATS 2021. (Joint work with E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau)

Add to calendar :