New Framework for High Speed DM

Data stream mining has operated under the same framework ever since its inception around two decades ago.  This framework assumes that change is constantly taking place in the stream and requires constant update of the data model each with the arrival of each and every data instance.

However, in real world systems change dynamics are not uniform over time. Typically, phases of high level of change (volatility) is punctuated by periods of low volatility. We propose a new framework that senses the level of volatility in the stream and adjusts the learning paradigm to the level of volatility. In periods of high volatility, instances will need to be monitored on a per instance basis and ensemble methods will need to be employed to cope with rapid changes in the stream. On the other hand in periods of low volatility, minimal changes need to be made to models already developed, thus enabling much higher stream processing speeds to be attained.

Recent research, reported in IEEE IJCNN 2016 shows that the new framework enables significant gains in speed to be made with compromising on model accuracy.