In this paper, we propose a Memory-Based Neural Network(MBNN) for Radar Automatic Target Recognition (RATR) based on High Resolution Range Profile (HRRP) in imbalanced case to learn how to find out the discriminative representations and generalize the ability to barely appeared target samples of some categories. Specifically, we utilize a Convolutional Neural Network (CNN) to explore discriminative features among HRRP samples and employ a memory module to record misclassified samples or samples that are correctly classified with low confidence into a external storage, we called it buffer. Then we leverage a Long Short Term Memory (LSTM) to merge the classified samples with some of the most similar ones in the buffer to make the final decision. It is worth noting that MBNN can be inserted as a plug-and-play module into any discriminative methods. Effectiveness and efficiency are evaluated on the measured data.