Learning Convolutional Neural Networks with Deep Part Embeddings

We propose a novel concept of Deep Part Embeddings (DPEs), which can be used to learn new Convolutional Neural Networks (CNNs) for different classes. We define DPE as a neuron of a trained CNN along with its network of filter activations that is interpretable as a part of a class that the neuron contributes to. Given a new class mathcal{C}, we explore the idea of combining different DPEs that intuitively constitute mathcal{C}, from trained CNNs (not on mathcal{C}), into a network that learns the class mathcal{C} with few training samples. An important application of our proposed framework is the ability to modify a CNN trained on n classes to learn a new class with limited training data without significantly affecting its performance on the n classes. We visually illustrate the different network architectures and extensively evaluate their performance against the baselines.