TEMA: The Evolutionary Meta-learning Agent
Smart Internet Technology Research Group

Aims

- To design and develop a mechanism to facilitate machine learning among an arbitrary set of tasks.

- The system developed should allow various other applications to be “plugged in” to provide a solution to a range of arbitrary machine learning tasks.

Introduction

- The norm in current applications of machine learning requires expert knowledge of machine learning in order to select and implement algorithms that will support a task satisfactorily.

- Despite the availability of tools such as WEKA, C5 and MATLAB’s Neural Network Toolkit, the amount of duplicate work done over different machine learning tasks is still quite significant. This is primarily due to the tweaking of the algorithm implementation to improve results. The large number of algorithms available also magnifies this problem.

- To limit the knowledge a user requires and to reduce the redundancy involved in utilisating machine learning, an evolutionary meta-learning agent is proposed

Meta-learning

- The automation of the algorithm selection process requires an algorithm, called a meta-learning algorithm, that is capable of selecting an appropriate machine learning algorithm given the problem at hand. The figure below depicts the typical mechanics of such a meta-learning algorithm.

*click on image to enlarge

- Once trained, the meta-learner given a dataset (pertaining to the problems at hand) will predict the appropriate base-learner to employ. This base learner is then given the same dataset and produces an appropriate classifier.

Issues

- Aside from the selection of the meta-learner (and the pool of base-learners), several other issues must also be addressed in the development of the framework.

- How a machine learning problem is defined. To provide a plug-in that supplies machine learning solutions to a generic field of problems, the problem definition is paramount. Aside from the data, this includes information on application restrictions and other background information. The problem definition also has ramifications for the meta-learner as it must distinguish between both similar and varying problems in order to select an appropriate classifier.

- To make the agent evolutionary, not only must it be made incremental, but it must also be able to evolve with the development of new algorithms (as well as meta-algorithms). Essentially, careful planning is required when designing the agent so that it has all the flexibility required to facilitate fundamental change.

- To refine the accuracy of the meta-learner (in this case, the training and testing error on various data sets relative to the predicted classifier), the meta-data should be updated whenever possible. This means that the testing of the new datasets must be conducted on the various algorithms and their variations. This will lead to an enormous amount of computation. This means that the agent should be streamlined to be extremely efficient. It should also conduct such testing offline.

- A knowledge base must be accumulated so that the agent can at least act, albeit primitively. An alternative is to have a default procedure for algorithm selection.