I took over some code from someone and my task was to reproduce the same model and performance in pytorch. I was given best hyper-parameters for that model as well. After playing around with it for quite sometime, I see that if I set drop out rate to zero, then my test performance is the best. it stops earlier, the smaller dropout is. In fact it outperforms previous model a bit. While the previous model had significant dropout rates.
The data size is about 33 million rows. And the neural net is like 4-5 layers. Total input embedding is ~1000.
Though I am happy to see the performance, I am wondering if it is some kind of red flag. Because without dropout i don't really have any other regularization. And it performing good with this may mean there might be some data leakage or something?
Looking for some wisdom around dropout in this context.
Neural net: no dropout gives the best test score. Is that bad?
527 views Asked by obi wan At
1
There are 1 answers
Related Questions in MACHINE-LEARNING
- Trained ML model with the camera module is not giving predictions
- Keras similarity calculation. Enumerating distance between two tensors, which indicates as lists
- How to get content of BLOCK types LAYOUT_TITLE, LAYOUT_SECTION_HEADER and LAYOUT_xx in Textract
- How to predict input parameters from target parameter in a machine learning model?
- The training accuracy and the validation accuracy curves are almost parallel to each other. Is the model overfitting?
- ImportError: cannot import name 'HuggingFaceInferenceAPI' from 'llama_index.llms' (unknown location)
- Which library can replace causal_conv1d in machine learning programming?
- Fine-Tuning Large Language Model on PDFs containing Text and Images
- Sketch Guided Text to Image Generation
- My ICNN doesn't seem to work for any n_hidden
- Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
- How can I resolve this error and work smoothly in deep learning?
- ModuleNotFoundError: No module named 'llama_index.node_parser'
- Difference between model.evaluate and metrics.accuracy_score
- Give Bert an input and ask him to predict. In this input, can Bert apply the first word prediction result to all subsequent predictions?
Related Questions in NEURAL-NETWORK
- Influence of Unused FFN on Model Accuracy in PyTorch
- How to train a model with CSV files of multiple patients?
- Does tensorflow have a way of calculating input importance for simple neural networks
- My ICNN doesn't seem to work for any n_hidden
- a problem for save and load a pytorch model
- config QConfig in pytorch QAT
- How can I convert a flax.linen.Module to a torch.nn.Module?
- Spiking neural network on FPGA
- Error while loading .keras model: Layer node index out of bounds
- Matrix multiplication issue in a Bidirectional LSTM Model
- Recommended way to use Gymnasium with neural networks to avoid overheads in model.fit and model.predict
- Loss is not changing. Its remaining constant
- Relationship Between Neural Network Distances and Performance
- Mapping a higher dimension tensor into a lower one: (B, F, D) -> (B, F-n, D) in PyTorch
- jax: How do we solve the error: pmap was requested to map its argument along axis 0, which implies that its rank should be at least 1, but is only 0?
Related Questions in DROPOUT
- model.eval() disables only dropout layers defined in outer class in Pytorch
- why nn.Dropout change the elements values of a tensor?
- Cannot generate random ints inside TensorFlow2.0 "call" function
- dropout mask in back propagation
- where to add dropout layer in the model?
- How to fix random seed in pytorch, while keeping the dropout randomness?
- Dropout with permutation in Pytorch
- An example usage for tf.keras.layers.GaussianDropout in TensorFlow2 for deep GRU network
- Bayesian Neural Network without dropout in the inference
- Activating the dropout in model eval mode in Pytorch
- Keras produce different index per training step
- Cannot reduce overfitting on BERT+BiGRU+Softmax network for named entity recognition
- Add Dropout to Model U-Net Tensorflow
- How to add dropout layers automatically to a neural network in pytorch
- How does Spatial Dropout work in the inference compared to Dropout?
Related Questions in REGULARIZED
- Using narxnet with trainbr: the predicted value at time t seems to depend in part on the target value at time t. But we don't know that yet
- Error with relaxed poisson regression in tidymodels
- Understanding Lasso and Ridge Regression
- What is the difference between using weight decay in an optimizer and weight decay in neural network layers?
- How remove numbers from url using RewriteRule in .htaccess file?
- SGD and Adam with Weight Decay are the Same as L2 Regularization in PyTorch?
- Prediction of new data with fitted regularization models
- Applying regularization in a custom layer in Keras/Tensorflow
- Why do the coefficents from orginal regularized regression (ordinalNet) have the wrong sign?
- How to implement regularization
- Convergence warning: how to do action after it?
- R problem with regularize.values(x, y, ties, missing(ties)
- Is passing activity_regularizer as argument to Conv2D() the same as passing it seperately right after Conv2D()? (Tensorflow)
- How to add L1 Regularization in MLPClaccifier?
- Keras: regularizing loss for an output based on the other outputs
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Sometimes, these things happen. Once my neural-net was not working and so I was advised to add Batch Normalization layers in it, and then it worked so well. But then in another problem, Batch Normalization made my neural-net worse. This is all because of backpropagation. Sometimes, adding some layer makes the neural-net get stuck in a local-minimum, while sometimes it helps to get out of it. I am not quite sure why is this, but I think it is because of BackPropagation.
The answer is no. Its just because of backpropagation.
NOTE - If you feel I am wrong anywhere in this post, then please comment it.