What is overfitting in machine learning?
1 Answer
Overfitting is a common problem in machine learning where a model learns its training data too closely, including noise and random details, so it performs very well on that data but poorly on new data it has not seen.
A simple analogy: imagine a student preparing for an exam by memorising answers to last year's question paper word for word. If the exam repeats the same questions, they score full marks. If the questions are different, even slightly, they struggle, because they memorised answers rather than understanding concepts. That is overfitting.
The opposite problem is underfitting: the model is too simple or undertrained to capture important patterns, so it performs poorly even on training data. This is like a student who barely studied.
The goal is generalisation: learning patterns that apply to new, unseen data.
How do you spot overfitting? Data scientists split data into separate sets:
1. Training data, used to teach the model.
2. Validation data, used to tune and check the model during development.
3. Test data, used for a final, unbiased evaluation.
If the model scores very high on training data but much lower on validation or test data, it is likely overfitting.
Why does overfitting happen?
The model is too complex for the amount of data.
The training data is too small or not representative.
Training continues for too long.
The data contains noise or errors the model learns as if they were real patterns.
How do you prevent it?
1. More data, and more diverse data.
2. Simpler models, when appropriate.
3. Regularisation techniques that penalise excessive complexity.
4. Dropout in neural networks, which randomly turns off some neurons during training so the network does not rely on specific pathways.
5. Early stopping, ending training when validation performance stops improving.
6. Cross-validation, testing on multiple splits of data.
7. Data augmentation, creating variations of training examples, such as rotated or cropped images.
Why does this matter beyond data science? Overfitting explains why AI systems sometimes fail in real-world conditions that differ from their training environment, such as a model trained on hospital images from one machine performing worse on images from another. It is also related to concerns about AI benchmarks, since models tuned heavily to specific tests may not generalise well.
Even for business users, the lesson applies: when evaluating any AI system, test it on fresh, realistic examples, not just the examples used to build it.