Methods and systems for performing transfer learning with basis scaling and pruning. One method includes obtaining a pre-trained deep convolutional neural network (DCNN), decomposing each weight matrix of the DCNN, and decomposing each convolutional layer by applying the respective decomposed weight matrix to the convolution layer to form a first layer which comprises the left matrix for convolution, and a second layer which comprises the right matrix for convolution. The method also includes providing a basis-scaling convolutional layer having a weight matrix that is derived by a function of singular values and the right singular vectors and training the basis scaling factors of the basis-scaling convolutional layers.