Probabilistic Transition-Based Approach for Detecting Application-Layer DDoS Attacks in Encrypted Software-Defined Networks

With the emergence of cloud computing, many attacks, including Distributed Denial-of-Service (DDoS) attacks, have changed their direction towards cloud environment. In particular, DDoS attacks have changed in scale, methods, and targets and become more complex by using advantages provided by cloud computing. Modern cloud computing environments can benefit from moving towards Software-Defined Networking (SDN) technology, which allows network engineers and administrators to respond quickly to the changing business requirements. In this paper, we propose an approach for detecting application-layer DDoS attacks in cloud environment with SDN. The algorithm is applied to statistics extracted from network flows and, therefore, is suitable for detecting attacks that utilize encrypted protocols. The proposed detection approach is comprised of the extraction of normal user behavior patterns and detection of anomalies that significantly deviate from these patterns. The algorithm is evaluated using DDoS detection system prototype. Simulation results show that intermediate application-layer DDoS attacks can be properly detected, while the number of false alarms remains low.