Machine Learning Stories

We all know what GANs are. The stories of the tough game between the Generator and Discriminator which ends in neither of them winning - but instead reaching a mutual agreement or a compromise(the Nash Equilibrium) have been going around since 2014. Along with the story, the disadvantages of GAN have also gained popularity. Let's address one of them for time being: Mode Collapse. Mode Collapse:  Let's consider a 2D example - as it is easier to visualize a 2D plane. Suppose the following is your data: There is a 2D plane of size 400x100. However, data of your interest is restricted to 3 small clusters compared to the whole plane. Basically you want the GAN to generate only points that belong in the red clusters. Real Data You design an ordinary GAN to mimic your data given the 2D space. However, your GAN is able to capture only partial data. It is not able to generate the data present at the top left corner. Generated Data This is referred to as Mode Col...

Identifying the most influential parameters in a representation learned for one task plays an important role in continual learning. This paper addresses two learning directions:  1) completely eliminating catastrophic forgetting using weight pruning(dropping neurons/weights?), where a hard selection mask freezes the most certain parameters (BLLL-PRN)   2) reducing catastrophic forgetting by adaptively modifying the learning rates based on how uncertain the parameter values are (BLLL-REG)( This method has the added advantage that there's no need have the information of task boundaries ) Since we have considered a Bayesian approach, we have to look at Bayesian Neural networks. The posterior over the weights of the neural network is learned in a BNN. One of the popular ways to perform inference in a BNN is using Variational Inference. This paper exactly follows BayesByBackProp algorithm by [Blundell, 2015]. Each weight parameter of the network is represente...