Date/Time : 09 Jan 2025 16:00
Venue : S17 #04-06 (SR1)
Speaker : Govind Menon, Brown University

We study the training dynamics of the Deep Linear Network (DLN) from the perspective of the geometric theory of dynamical systems. Rigorous results by several authors are unified into a thermodynamic framework for deep learning. The analysis begins with a characterization of the invariant manifolds and Riemannian geometry in the DLN. This is followed by exact formulas for a Boltzmann entropy, as well as stochastic gradient descent of free energy using a Riemannian Langevin Equation. Several links between the DLN and other areas of mathematics are discussed, along with some open questions.

Add to calendar :

Date/Time : 08 Jan 2025 16:30
Venue : S17 #04-04 (SR3)
Speaker : Govind Menon, Brown University

We study the training dynamics of the Deep Linear Network (DLN) from the perspective of the geometric

theory of dynamical systems. Rigorous results by several authors are unified into a thermodynamic framework for deep learning. The analysis begins with a characterization of the invariant manifolds and Riemannian geometry in the DLN. This is followed by exact formulas for a Boltzmann entropy, as well as stochastic gradient descent of free energy using a Riemannian Langevin Equation. Several links between the DLN and other areas of mathematics are discussed, along with some open questions.

Add to calendar :

Date/Time : 18 Dec 2024 15:00
Venue : S17 #06-11 (SR6)
Speaker : Ding Lijun, University of California, San Diego

We revisit a formulation technique for inequality-constrained optimization problems that has been known for decades: the substitution of squared variables for nonnegative variables. Using this technique, inequality constraints are converted to equality constraints via the introduction of a squared-slack variable. Such formulations have the superficial advantage that inequality constraints can be dispensed with altogether. But there are clear disadvantages, not least being that first-order optimal points for the squared-variable reformulation may not correspond to first-order optimal points for the original problem, because the Lagrange multipliers may have the wrong sign. Extending previous results, this paper shows that points satisfying approximate second-order optimality conditions for the squared-variable reformulation also, under certain conditions, satisfy approximate second-order optimality conditions for the original formulation, and vice versa. Such results allow us to adapt complexity analysis of methods for equality-constrained optimization to account for inequality constraints. On the algorithmic side, we examine squared-variable formulations for several interesting problem classes, including bound-constrained quadratic programming, linear programming, and nonnegative matrix factorization. We show that algorithms built on these formulations are surprisingly competitive with standard methods.

Add to calendar :

Date/Time : 23 Oct 2024 16:00
Venue : S17 #04-04 (SR3)
Speaker : Zhong Zixin, The Hong Kong University of Science and Technology (Guangzhou)

We propose a novel piecewise stationary linear bandit (PSLB) model, where the environment randomly samples a context from an unknown probability distribution at each changepoint, and the quality of an arm is measured by its return averaged over all contexts. The contexts and their distribution, as well as the changepoints are unknown to the agent. We design Piecewise-Stationary ε-Best Arm Identification+ (PSεBAI+), an algorithm that is guaranteed to identify an ε-optimal arm with probability ≥1−δ and with a minimal number of samples. PSεBAI+ consists of two subroutines, PSεBAI and Naïve ε-BAI (NεBAI), which are executed in parallel. PSεBAI actively detects changepoints and aligns contexts to facilitate the arm identification process. When PSεBAI and NεBAI are utilized judiciously in parallel, PSεBAI+ is shown to have a finite expected sample complexity. By proving a lower bound, we show the expected sample complexity of PSεBAI+ is optimal up to a logarithmic factor. We compare PSεBAI+ to baseline algorithms using numerical experiments which demonstrate its efficiency. Both our analytical and numerical results corroborate that the efficacy of PSεBAI+ is due to the delicate change detection and context alignment procedures embedded in PSεBAI.

Add to calendar :

Date/Time : 25 Sep 2024 16:00
Venue : S17 #05-12 (SR4)
Speaker : Hsuan-Tien Lin, National Taiwan University

Complementary-Label Learning (CLL) is a weakly supervised learning paradigm that is claimed to be useful in situations where collecting true labels is expensive. This talk begins by walking the audience through two advances in CLL. The first work discovered that the CLL loss design from Unbiased Risk Estimator (URE) suffers from high variance in gradient estimation. To address this, a novel surrogate complementary loss (SCL) framework is proposed, which reduces variance and improves gradient alignment, mitigating the overfitting issue. The second work introduces a new approach to CLL by reducing the problem to estimating the probability of complementary classes. This framework sidesteps the limitations of traditional methods and improves robustness in noisy environments, offering a broader perspective for both deep and non-deep models. Empirical results demonstrate the efficacy of both approaches in improving CLL performance. Finally, the speaker will share some of the ongoing attempts in making CLL more realistic, including some first-hand experience in collecting real-world datasets and releasing an open-source library.

Add to calendar :

Date/Time : 09 Sep 2024 16:00
Venue : S17 #05-11 (SR5)
Speaker : Jean-Bernard Lasserre, LAAS-CNRS

We briefly describe the Christoffel-Darboux (CD) kernel and the Christoffel function (CF). While well-known tools in orthogonal polynomials and approximation theory, it is only recently that CF’s properties have been shown to be very useful and powerful (as well as easy to implement) to help solve some problems

— in data analysis and datra mining where the underlying (empirical) measure is usually discrete, e.g. for outlier detection, support inference, density approximation, and

— in approximation of discontinuous functions.

Finally we will also briefly describe some (in the author’s opinion surprising) links of the CF with seemingly less disconnected areas (Pell equation, equilibrium measure, convex optimization and positivity certificates).

Add to calendar :

Date/Time : 23 Aug 2024 10:00
Venue : S17 #06-11 (SR6)
Speaker : Yixuan Wang, Caltech

We drew inspiration from the Kolmogorov–Arnold representation theory, using a composition of sum of 1D functions to represent higher dimensional functions, and proposed KANs, which has learnable activation functions parameterized by splines. Compared with MLPs, it will have much better scaling laws and interpretability since we use a much smaller network with 1D functions, provided that the underlying function has such a smooth representation, and in practice we can make KANs arbitrarily wide and deep. We demonstrate the wide applicability of this paradigm by examples of function pitting, PDE solving (PINNs), and applications to scientific problems. I will also discuss the most up-to-date developments of KANs and present some theoretical understandings.

Add to calendar :

Date/Time : 30 Jul 2024 15:00
Venue : S17 #04-05 (SR2)
Speaker : Min XU, Carnegie Mellon University

Cryo-electron tomography (cryo-ET) 3D imaging technology captures a 3D image of a single cell’s subcellular structures at sub-molecular resolution and in a near-native state. It provides unprecedented opportunities for systematically studying the native spatial organization of subcellular structures, especially macromolecules. However, cryo-ET has high structural complexity and imaging limits, such as high structural diversity and crowding, low signal-to-noise ratio, and missing values. These have made the automated systematic analysis of such images extremely difficult. I will present our recent efforts to develop methods for automated analysis of cryo-ET data.

Add to calendar :

Date/Time : 19 Jun 2024 14:00
Venue : S17 #04-04 (SR3)
Speaker : Yuan Yancheng, Hong Kong Polytechnic University

Convex clustering model has attracted growing attention in the machine learning community. In this talk, we will introduce some recent progress of convex clustering model on model performance and numerical computation. Moreover, we will introduce a newly developed solver for solving the convex clustering model with GPU acceleration.  

Add to calendar :