The purpose of models is not to fit the data but to
sharpen the questions.
Samuel Karlin
Even for the industrial engineer trying to control a production process, the design of a model is just a step in the continuous process of understanding what happens in the plant. One can compare this quotation with another famous one:
The purpose of computing is insight, not numbers.
Richard Hamming, opening quote of Numerical Methods for Scientists and Engineers, McGraw-Hill, 1962
A familiar concept in adaptative models is overlearning. If your model is (temporarily) fitting the data really very well, it will probably not be robust : it depends too much on transient components and properties of the data. It has stopped learning and started recording, memorizing, compressing the data, not abstracting it. That can easily happen if you don’t have enough data with sufficient dimension and sufficient volatility in regard to the complexity of the underlying mechanism and the precision you require in one learning step or one hierarchical level. Feeding more data is not the solution, especially if you are not prepared to redesign your model or improve and filter your inputs.
That is the same in educating humans: in many cases, confronted with a series of exercices, it will be easier for the pupil to learn a false and chaotic series of local rules and exceptions about what to do than to reshape and extend his understanding. The next series of exercices will not be a motivation to simplify the previous system of beliefs, it will just mainly be erased to leave room to the new urgent version about tomorrow’s homework, and so on. What will remain is certainly not sorted according to certainty : all rules remembered are more or less equal in status, all knowledge will a series of unconnected and fading islands with a few if any lighthouses.
When one asks students struggling with some material, to explain aloud their reasoning when solving a problem, a large part of their troubles are enshrined coincidences taken as infaillible shortcuts. They project all feelings of confusion or complexity on the subject matter and on the teacher, not on the current state of their learning process, exactly as they have been prepared to do by student folklore.