The paper presents an approach for learning the filters of a convolutional NN, for an image classification task, without making use of target labels. The algorithm proceeds in two steps: learning a transformation of the original image and then learning a classifier using this new representation. For the first step, patches are sampled from an image collection, each patch will then correspond to a surrogate class and a classifier will be trained to associate transformed versions of the patches to the corresponding class labels using a convolutional net. In a second step, this net is replicated on whole images leading to a transformed representation of the original image. A linear classifier is then trained using this representation as input and the target labels relative to the image collection. Experiments are performed on different image collections and a comparison with several baselines is provided.