🖍️Antoine Cornuéjols - Collected Quotes

"Hence, has machine learning uncovered truths that escaped the notice of philosophy, psychology, and biology? On one hand, it can be argued that machine learning has at least provided grounds for some of the claims of philosophy regarding the nature of knowledge and its acquisition. Against pure empiricism, induction requires prior knowledge, if only in the form of a constrained hypothesis space. In addition, there is a kind of conservation law at play in induction. The more a priori knowledge there is, the easier learning is and the fewer data are needed, and vice versa. The statistical study of machine learning allows quantifying this trade-off." (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)

"In effect, machine learning research has already brought us several interesting concepts. Most prominently, it has stressed the benefit of distinguishing between the properties of the hypothesis space - its richness and the valuation scheme associated with it - and the characteristics of the actual search procedure in this space, guided by the training data. This in turn suggests two important factors related to sequencing effects, namely forgetting and the nonoptimality of the search procedure. Both are key parameters than need to be thoroughly understood if one is to master sequencing effects." (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)

"On the other hand, the algorithms produced in machine learning during the last few decades seem quite remote from what can be expected to account for natural cognition. For one thing, there is virtually no notion of knowledge organization in these methods. Learning is supposed to arise on a blank slate, albeit a constrained one, and its output is not supposed to be used for subsequent learning episodes. Neither is there any hierarchy in the 'knowledge' produced. Learning is not conceived as an ongoing activity but rather as a one-shot process more akin to data analysis than to a gradual discovery development or even to an adaptive process. " (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)

"[...] the theory that establishes a link between the empirical fit of the candidate hypothesis with respect to the data and its expected value on unseen events becomes essentially inoperative if the data are not supposed to be independent of each other. This requirement is obviously at odds with most natural learning settings, where either the learner is actively searching for data or where learning occurs under the guidance of a teacher who is carefully choosing the data and their order of presentation." (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)

"There are many control parameters to a learning system. The question is to identify, at a sufficiently high level, the ones that can play a key role in sequencing effects. Because learning can be seen as the search for an optimal hypothesis in a given space under an inductive criteria defined over the training set, three means to control learning readily appear. The first one corresponds to a change of the hypothesis space. The second consists in modifying the optimization landscape. This can be done by changing either the training set (for instance, by a forgetting mechanism) or the inductive criteria. Finally, one can also fiddle with the exploration process. For instance, in the case of a gradient search, slowing down the search process can prevent the system from having time to find the local optimum, which, in turn, can introduce sequencing effects." (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)

"While it has been always considered that a piece of information could at worst be useless, it should now be acknowledged that it can have a negative impact. There is simply no theory of information at the moment offering a framework ready to account for this in general." (Antoine Cornuéjol, "The Necessity of Order in Machine Learning: Is Order in Order?", 2007)