JULAIN-Journal-Club.md

@@ -55,6 +55,26 @@ Passcode: 092935
 **Abstract:**<br>
 While sensory representations in the brain depend on context, it remains unclear how such modulations are implemented at the biophysical level, and how processing layers further in the hierarchy can extract useful features for each possible contextual state. Here, we first demonstrate that thin dendritic branches are well suited to implementing contextual modulation of feedforward processing. Such neuron-specific modulations exploit prior knowledge, encoded in stable feedforward weights, to achieve transfer learning across contexts. In a network of biophysically realistic neuron models with context-independent feedforward weights, we show that modulatory inputs to thin dendrites can solve linearly non-separable learning problems with a Hebbian, error-modulated learning rule. Finally, we demonstrate that local prediction of whether representations originate either from different inputs, or from different contextual modulations of the same input, results in representation learning of hierarchical feedforward weights across processing layers that accommodate a multitude of contexts.
 
+### 15 May 2023: Foundation Models for Robustness to Distribution Shifts<br>
+author and presenter: Ananya Kumar (Stanford)
+
+Zoom link: https://fz-juelich-de.zoom.us/j/94480107394?pwd=dFdFZTRwaytONzJrcEp3WkJ0bmNqUT09<br>
+Meeting ID: 944 8010 7394<br>
+Passcode: 092935
+
+paper:
+[Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution.](https://arxiv.org/abs/2202.10054) <br>
+Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, Percy Liang. International Conference on Learning Representations, (ICLR Oral) 2022. 
+
+[Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation.](https://arxiv.org/abs/2204.00570) <br>
+Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. HaoChen, Tengyu Ma, Percy Liang. International Conference on Machine Learning (ICML Long Talk) 2022
+
+**Abstract:**<br>
+When ML systems are deployed, they often face test examples that are different from training---this leads to a large drop in accuracy. The foundation model paradigm (pretraining general-purpose representations from broad unlabeled data, and then adapting to a variety of tasks we care about) has emerged as one of the most effective ways to improve robustness to novel test examples. But how should we adapt good foundation models robustly, and how should we pretrain good models? (1, Adaptation) In the first part of the talk, we will explain why the standard approach of fine-tuning all model parameters can distort good pretrained representations and underperform out-of-distribution. The theory leads to practical insights and better methods for fine-tuning. Our methods have led to state-of-the-art accuracies on ImageNet and in applications such as satellite remote sensing, wildlife conservation, and radiology. (2, Pretraining) Next, we will examine how foundation models can learn good representations. We show that contrastive pretraining on unlabeled data from many domains, and then transferring to labeled data from one domain, improves accuracy even on the domains where we had no labels. We explain why pretraining can work differently from some classical domain adaptation intuitions. Our theory predicts phenomena on real datasets, and leads to improved algorithms. (3, Future Work) Finally, we discuss some exciting future research directions on foundation models.
+ 
+**Bio:**<br>
+Ananya is a final year PhD student at Stanford University advised by Percy Liang and Tengyu Ma. His PhD work focuses on representation learning, foundation models, and reliable machine learning.
+
 ## Past Meetings
 
 ### 20 March 2023: Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels<br>
@@ -516,4 +536,4 @@ A training schedule using filter pruning and orthogonal reinitialization
 
 
 ---
-last change: 22.03.2023 sw
\ No newline at end of file
+last change: 31.03.2023 sw
\ No newline at end of file