Deep Learning-Based Unsupervised Anomaly Detection in Network Traffic Using RNN and CNN Autoencoders
Neha Tripathi Tripathi
Paper Contents
Abstract
In an era of rapidly expanding network infrastructures, the ability to detect anomalies in network traffic is critical for maintaining cybersecurity. Traditional rule-based systems are increasingly insufficient for identifying sophisticated or novel attack patterns embedded within high volumes of legitimate traffic. This study explores the application of deep learning techniques, specifically Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN) based autoencoders, for unsupervised anomaly detection in network traffic captured using Wireshark. The research begins with the collection of real-time network traffic data through Wireshark, followed by preprocessing steps including feature selection, label encoding, normalization, and sequence generation through a sliding window approach. Two deep learning models an RNN-based autoencoder and a CNN-based autoencoder were then trained to reconstruct normal traffic patterns. Deviations in reconstruction, measured using Mean Squared Error (MSE), were used to identify potential anomalies based on a statistically defined threshold. Model evaluation was conducted using classification metrics such as accuracy, recall, and F1 score, with a particular focus on the minority class representing anomalies. While both models achieved high accuracy above 95%, the recall and F1 scores remained low, revealing challenges associated with the heavily imbalanced nature of network traffic data. Comparative analysis indicated that the CNN model performed slightly better than the RNN model in detecting anomalous patterns, likely due to its ability to capture local feature correlations. To complement quantitative analysis, extensive exploratory data analysis (EDA) was performed. Visualizations such as confusion matrices, IP communication heatmaps, packet length distributions, and time-series plots provided deeper insights into traffic behavior and model performance. These graphical analyses highlighted structural traffic patterns and helped interpret anomalies detected by the models. Overall, the findings demonstrate that deep learning autoencoders can effectively learn normal network behavior; however, their performance in detecting subtle anomalies is limited by class imbalance and the lack of explicit anomaly labels. This research lays the groundwork for enhancing network anomaly detection systems by integrating advanced feature engineering, synthetic data augmentation, and hybrid detection architectures combining deep learning with rule-based approaches.
Copyright
Copyright © 2025 Neha Tripathi. This is an open access article distributed under the Creative Commons Attribution License.