Currently, I am trying to build a classifier that will identify writer using CNN. The dataset contains 71 classes, each class contains English capital letters. When I am using VGG16, VGG19, ResNet50, or even any simple convolutional neural network, I am facing overfitting problem. The validation accuracy can keep up with the training accuracy upto 20% or slightly greater than that, but when the training accuracy increases beyond 30%, the validation accuracy still remains around 20%.
I have tried to use data augmentation, dropout layers, even increased numbers of fully connected layers. But still I am facing the same overfitting problem. I can't get what is the cause behind this overfitting and trying to find solution.