A fast and adaptive threat detection and prevention architecture

Abstract

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. We propose a fast and adaptive Threat Detection and Prevention Architecture based on stream processing and machine learning algorithms. The proposed architecture combines the adaptability of online trained machine learning algorithms with the efficiency of batch trained methods. We create a dataset by capturing both legitimate and malicious traffic and compare two ways of combining packets into flows, one gathering all packets in a time window and the other analyzing only the first few packets of each flow. Besides our created dataset, we also analyze our proposal on real data composed of fixed-broadband Internet user traffic from one of the major Brazilian network operators. In order to evaluate our detection architecture, we develop five classification algorithms and two anomaly detection methods. In fact, the proposed architecture provides adaptability to new traffic behavior and achieves a high accuracy rate and a good trade-off between attack detection and false positive rate in anomaly detection. We further propose an improved scheme, based on Software Defined Networks, that automatically prevents threats by only analyzing the first few packets of a flow. The proposal promptly and efficiently blocks threats, is robust, and can scale up, even on scenarios in which the attacker employs spoofed IP address. Moreover, we evaluate the scalability, by increasing the number of stream processing cores and allocating more resources to sensor elements. The results shows an accuracy higher than 90% and threat detection time of four microseconds, which promptly enables counter measures.