Date/Time : 07 Sep 2023 10:00 - 12:00
Venue : I4-01-03 (Innovation 4.0 Building)
Speaker : Jean Bernard Lasserre, Laboratory for Analysis and Architecture of Systems

Add to calendar :

Date/Time : 23 Mar 2023 16:00
Venue : S17 #04-06 (SR1)
Speaker : Tan Nguyen, University of California, Los Angeles

Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision. Like neural network layers, these attention mechanisms are often developed by heuristics and experience. In this talk, I present two novel mathematically principled frameworks to facilitate the process of designing efficient and robust transformer models with better expressivity.

In my first framework, I interpret self-attention in transformers as the Nadaraya-Watson estimator in a nonparametric regression setting. In the context of transformers, this Nadaraya-Watson estimator is constructed from the kernel density estimators (KDE) of the keys and queries in self-attention. However, such KDEs are not robust to contaminated samples. By viewing KDE as the solution to the kernel regression problem in a reproducing kernel Hilbert space, I adopt the state-of-the-art robust KDE methods based on, e.g., robust kernel regression and median-of-mean estimator, to design new robust self-attention mechanisms. The resulting family of robust self-attentions can be tailored to various transformer architectures.

In my second framework, I show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer. Thus, I prove the primal-dual relationship between a self-attention layer and a neural network layer. Using my framework, I derive popular attention layers used in practice and propose two new attention mechanisms: 1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head (Attention-SH) derived from using less training data to fit the model. I empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model’s accuracy, and improving the model’s efficiency in a variety of practical applications.

Add to calendar :

Date/Time : 13 Jan 2022 09:00
Venue : via Zoom
Speaker : Holden Lee, Duke University

Join Zoom Meeting
https://nus-sg.zoom.us/j/81518116179?pwd=bGkvanY2aTlra1pLeEpKQzBSaVdBdz09

Meeting ID: : 815 1811 6179
Passcode: 461200

Abstract: Probabilistic methods in machine learning have made impressive progress in generating realistic images and videos, as well as building interpretable models with uncertainty quantification. In this talk, I will provide algorithms for two fundamental problems in probabilistic machine learning: sampling from multimodal distributions and estimating normalizing constants. I will show how to combine Langevin dynamics–an algorithm based on gradient descent with Gaussian noise–with tools from statistical physics and numerical analysis to design faster algorithms for these problems. Finally, I will touch on an application to understanding the representational power of normalizing flows, a deep generative model that provides explicit likelihood computation and produces high-resolution natural images.

Add to calendar :

Date/Time : 10 Jan 2022 13:00
Venue : via Zoom
Speaker : Hadi Daneshmand, Princeton University

Join Zoom Meeting
https://nus-sg.zoom.us/j/83955852990?pwd=VUpkT1o2M2pzN01YWWxVTTJ1QmpLQT09

Meeting ID: : 839 5585 2990
Passcode: 158210

Abstract:  Deep neural networks have achieved incredible performance in various domains by automatizing feature extraction with less need for conventional feature engineering. When neurons are well-connected and optimized, their performance goes beyond the human level for image processing. Yet, wiring and optimizing many neurons is a cumbersome engineering task. Over 70 years of research and optimization breakthroughs have advanced the training of deep networks. In this talk, we will cover the main outcomes of these breakthroughs. The talk presents recent studies on batch normalization, the fundamental building of deep neural networks. Leveraging tools from Markov chain theory, we will shed light on the inner workings of batch normalization for deep learning. Then, we will review escaping challenging attractors for neural learning with first-order optimization methods.

Add to calendar :

Date/Time : 05 Jan 2022 09:00
Venue : via Zoom
Speaker : Pan Wenyu, University of Chicago

Join Zoom Meeting
https://nus-sg.zoom.us/j/84440678768?pwd=RUZ3Y2RCTTB3SStuQWtuOWE0YlNDQT09

Meeting ID: 844 4067 8768
Passcode: 224158

Abstract:  Let $\Gamma$ be a Zariski dense Kleinian Schottky subgroup of $\operatorname{PSL}_2(\mathbb{C})$. Let $\Lambda_{\Gamma}\subset \mathbb{C}$ be its limit set, endowed with a Patterson-Sullivan measure $\mu$. I will present the joint work with Jialun Li and Frédéric Naud in which we showed that the Fourier transform of $\mu$ enjoys polynomial decay.  

For $\operatorname{PSL}_2(\mathbb{R})$, this was proved by Bourgain-Dyatlov.  As a corollary, all limit sets of Zariski dense Kleinian groups have a positive Fourier dimension. Another application is uniform “essential” spectral gap for any finite cover of $\Gamma \backslash \mathbb{H}^3$.  I will outline the main ingredients of the proof and talk about some future problems.

Add to calendar :

Date/Time : 13 Dec 2021 09:00
Venue : via Zoom
Speaker : Reza Gheissari, University of California, Berkeley

Join Zoom Meeting
https://nus-sg.zoom.us/j/85754837628?pwd=ZytjTUI2TVBtU3l2Y01PM0lpazBNUT09

Meeting ID: 857 5483 7628
Passcode: 359207

Abstract: Since its introduction in 1920, the Ising model has been one of the most studied models of phase transitions in statistical physics. In its low-temperature regime, the model has two thermodynamically stable phases, which, when in contact with each other, form an interface: a random curve in 2D and a random surface in 3D. In this talk, I will survey the rich phenomenology of this interface in 2D and 3D, and describe recent progress in understanding its geometry in various parameter regimes where different surface phenomena and universality classes emerge.

Add to calendar :

Date/Time : 09 Dec 2021 09:00
Venue : via Zoom
Speaker : Zhong Yiqiao, Joe, Stanford University

Join Zoom Meeting
https://nus-sg.zoom.us/j/82977751761?pwd=ZXowNm93bXE3RTE5NVd0bVBPSEhPUT09

Meeting ID: 829 7775 1761
Password: 717960

Abstract:  A mystery of modern neural networks is their surprising generalization power in overparametrized regime: they comprise so many parameters that they can interpolate the training set, even if actual labels are replaced by purely random ones; despite this, they achieve good prediction error on unseen data.

To demystify the above phenomena, we focus on two-layer neural networks in the neural tangent (NT) regime. Under a simple data model where n inputs are d-dimensional isotropic vectors and there are N hidden neurons, we show that as soon as Nd >> n, the minimum eigenvalue of the empirical NT kernel is bounded away from zero, and therefore the network can exactly interpolate arbitrary labels.

Next, we study the generalization error of NT ridge regression (including min-norm interpolation). We show that in the same overparametrization regime Nd >> n, in terms of generalization errors, NT ridge regression is well approximated by kernel ridge regression (infinite-width kernel), which is further approximated by polynomial ridge regression. A surprising phenomenon is the “self-induced” regularization due to the high-degree components of the activation function.

Add to calendar :

Date/Time : 06 Dec 2021 09:00
Venue : via Zoom
Speaker : Yuhua Zhu, Stanford University

Join Zoom Meeting
https://nus-sg.zoom.us/j/87958362587?pwd=OThycThnUXNUZXcxQzBndlUxaFNVdz09

Meeting ID: 879 5836 2587
Passcode: 628767

Abstract: As the continuous limit of many discretized algorithms, PDEs can provide a qualitative description of algorithm’s behavior and give principled theoretical insight into many mysteries in machine learning. In this talk, I will give a theoretical interpretation of three machine learning algorithms using Fokker-Planck (FP) equations. In the first one, we provide a mathematically rigorous explanation of why resampling outperforms reweighting in correcting biased data when stochastic gradient-type algorithms are used in training. In the second one, we propose a new method to alleviate the double sampling problem in model-free reinforcement learning, where the FP equation is used to do error analysis for the algorithm. In the last one, inspired by an interactive particle system whose mean-field limit is a non-linear FP equation, we develop an efficient gradient-free method that finds the global minimum exponentially fast.

Add to calendar :

Date/Time : 06 Dec 2021 14:30
Venue : via Zoom
Speaker : David Hansen, Max Planck Institute for Mathematics

Join Zoom Meeting
https://nus-sg.zoom.us/j/87960829555?pwd=c0ZsRGJXajhDVmI1bnA1WU4vNXBVUT09

Meeting ID: 879 6082 9555
Passcode: 145729

Abstract: Local Shimura varieties are non-archimedean analytic spaces analogous to Shimura varieties, whose cohomology is expected to realize (in a precise sense) both the local Langlands correspondence and the local Jacquet-Langlands correspondence. I’ll give a gentle introduction to the theory of local Shimura varieties, and explain some results that can be proven about their cohomology using current technology. Some of this material is joint work with Tasho Kaletha and Jared Weinstein.

Add to calendar :