distributed-network-event-detection-system/graph_based_intrusion_detection/report/report.tex

%! Author = alex
%! Date = 2/26/24

% Preamble
\documentclass[12pt]{article}

% Packages
\usepackage{time}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage[backend=biber, style=authoryear, citestyle=authoryear-comp]{biblatex}
\addbibresource{references.bib}


\title{A Distributed System for Deep Packet Inspection and Event Detection}
\author{Alexander Blank \\ Leipzig University - Chair of Mathematics and Computer Science}
\date{\today}

% Document
\begin{document}

    \maketitle

    \begin{abstract}
        This paper proposes a conceptual design for a distributed system aimed at enhancing deep packet
        inspection and event detection within large-scale network environments, a cornerstone of network security.
        It explores the integration of scalable, fault-tolerant architectures with modern technologies such as
        Apache Kafka for message brokering and Redis for state management, highlighting the potential for real-time
        analysis and anomaly detection.
        Through theoretical examination and design proposals, this work seeks to underscore the challenges and
        solutions associated with protecting expansive network infrastructures.
        While acknowledging the existence of similar systems, this project emphasizes the educational process of
        designing a network security solution, offering insights into system scalability, fault tolerance,
        and the practical application of theoretical knowledge.
        This endeavor not only contributes to the academic exploration of network security challenges but also
        enhances understanding of distributed system architectures' role in mitigating these challenges.
    \end{abstract}


    \section{Introduction}\label{sec:introduction}
    The escalating complexity and scale of modern networks pose significant challenges to traditional
    network security measures, necessitating the development of innovative, scalable solutions capable of real-time
    threat detection and analysis.
    This paper introduces a practical distributed system designed to meet these demands through deep packet inspection
    (DPI) and event detection.
    Utilizing a scalable, fault-tolerant architecture integrated with Apache Kafka for efficient message brokering and
    Redis for effective state management, the system offers a robust solution for monitoring and analyzing
    network traffic across large-scale environments.

    Detailing the development and implementation phases, this work highlights the system's design principles,
    key functionalities, and its ability to adaptively scale, ensuring comprehensive network security coverage.
    By showcasing the system's deployment and its proficiency in detecting diverse network anomalies, the paper aims to
    contribute to the enhancement of network security mechanisms, addressing the critical need for systems that can
    evolve in tandem with the networks they protect.

    Due to a length limitation, the paper focuses on the conceptual design and the system's key components, omitting
    detailed implementation and deployment aspects.
    More detailed information on the system's implementation and deployment can be found in the
    project's repository or provided by the author upon request.


    \section{Method}\label{sec:method}
    The distributed system is composed of at least 4 components, each with a specific role in the overall
    functionality of the system.
    Each component can be scaled horizontally without having to change the configuration of the system.
    The work will be distributed automatically to the available components, and the system will be able to
    handle the increased load without any manual intervention.

    To actually use the results of the system, additional components can be added, that react to, or interact
    with changes in the network state or the individual nodes' analysis results.

    Figure~\ref{fig:system_diagram} shows a diagram of the system's components and their interactions.
    % show a diagram of the system
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.9\textwidth]{resources/system_diagram}
        \caption{The systems components and their interactions.}
        \label{fig:system_diagram}
    \end{figure}

    \subsection{Packet Processor}\label{subsec:packet-processor}
    This component is responsible for analyzing batches of incoming packets,
    based on processing steps defined in the configuration of the system.

    The system accepts network packets from a Kafka topic, which are then processed by the packet processors.
    Each processor takes in a predefined number of packets and processes them
    based on a list of processing steps, which are defined in the configuration of the system.
    Processing steps are defined as a list of functions, which are applied to each packet in the batch.
    The resulting information is written to a Kafka topic, which is then consumed by the state merger component.

    \subsection{State Merger}\label{subsec:state-merger}
    This component is responsible for merging the information produced by the packet processor into a global state.
    Merging behaviour is defined in the configuration of the system.
    Each processing step also needs a merging function, that defines how the information produced by the processing
    step is merged into the global state.

    A state merger will take in batches of processed packets, apply the respective merging functions to them and
    save the resulting global network state to a Redis database.
    A locking mechanism is used to ensure that no information is lost on concurrent writes to the database.

    \subsection{Work Dispatcher}\label{subsec:work-dispatcher}
    This component is responsible for distributing the work resulting from the network state to a work queue.
    Work packages contain the node identity of the node that should be analyzed and the creation time to ensure that
    timeouts can clean up old packages that might otherwise clog up the work queue.
    An interval can be set, that defines how often each node should be analyzed.

    Additionally to dispatching work packages, the work dispatcher also takes care of the results of said work packages
    by processing result messages from the result queue and updating the node registry accordingly.
    If a result does not come in after a given timeout, the work dispatcher will dispatch the work package again,
    taking into consideration any changes in network state that happened until then.

    \subsection{Node Analysis}\label{subsec:node-analysis}
    This component is responsible for analyzing the network state of a given node and producing results based on
    configured analysis steps.
    The analysis steps are defined in the configuration of the system and contain a dataset function, that defines
    how the data for the analysis is retrieved from the global network state, and a model and scaler, that define
    the actual analysis model and, if needed, a scaling model, that gets applied to the extracted dataset before
    it is fed into the model.
    Depending on the model used, a model function needs to be given, that defines how to interact with it.
    For each analysis step, a interpretation function can be given, that will be applied to the models result to
    produce a usable interpretation or human-readable result.
    The set of results of each analysis step will then get written to the resul queue.

    \subsection{Node Registry}\label{subsec:node-regsitry}
    The node registry contains all information analyzed by the system, such as the last time a node was analyzed
    and the result of each of the analysis steps.
    The registry can then be used by other components to react to these results, such as a recognition of an attack,
    that might trigger a countermeasure.

    \subsection{Processing Step}\label{subsec:processing-step}
    As a proof of concept, a simple processing step is implemented,  that converts the current network state based
    on the packets it receives into a graph representation of the network.
    This graph representation then serves as a base for further processing steps and the event detection component.
    The graph encompasses the nodes of the network as vertices and the connections between the nodes as edges.
    Each edge gets the data associated to the connection on a per connection and per second basis as edge attributes,
    based on the configuration of the source fields.
    Since this has the disadvantage of being limited to a resolution of one second and having a rather complex merging
    function, it should be noted that this is only a proof of concept and can be replaced by more sophisticated
    processing steps, which are tailored to the specific use case of the system.

    \subsection{Analysis Models}
    There are two implemented models as a proof of concept for the event detection component.

    The first model is a simple random forest classifier, which is trained on the dataset of network traffic
    developed in the lecture.
    Its purpose is to detect an incoming attack for any given node in the network.
    The attack is defined bye the attack column given by the dataset.
    On unseen test data, the model achieves a precision of 99.9\% and a recall of 99.9\% being trained on appropriaetly
    weighted data to reflect the huge class imbalance in the dataset.


    The second model is a neural network classifier aimed at detecting nodes that are behaving anomalously, i.e.\ nodes
    that are sources of attacks.
    The neural network is also trained on the dataset of network traffic developed in the lecture.
    This model achieves a precision of 99.9\% and a recall of 95.2\% on unseen test data.
    The lower recall is due to the fact that, as stated above, the dataset is highly imbalanced, and as a rather
    complex model architecture, the neural network is prone to overfitting on the majority class.
    Extending the dataset with more data from the minority class would likely improve the recall of the model.

    However, the models are not the focus of this paper, and therefore, they are not discussed in detail and can
    be considered a proof of concept for the event detection component.
    They can easily be replaced by more sophisticated models, which are trained on more extensive datasets.


    \section{Future Work}
    Since the current implementation is only a proof of concept, there are many areas that can be improved and
    extended in the future.

    The current implementation is based on a single network state, that needs to be written to and read from the
    Redis database on each update, limiting the scalability of the system.
    A shard-based approach can be used to distribute the network state across multiple distinct network states, which
    enables the global state to be distributed across multiple Redis instances, thus improving the scalability of the
    system.

    The current processing approach is based on a network graph representation, which is updated in its entirety on each
    update of the network state, limiting the scalability of the merging step.
    To improve the performance of the system, switching to partial graph updates, which only update the parts of the
    graph that have changed, would be beneficial.
    Additionally, a binary merge method can be used to further distribute the work load on the merging step.
    Switching to different database architectures, such as a graph database for the network state and a time-series
    database for the network traffic, can also improve the performance of the system.

    Interacting with Kafka and Redis require serialization and deserialization of the data, which can be a bottleneck
    and security risk.
    At the moment, python's pickle module is used for serialization and deserialization, which is not secure and
    needs to be replaced by a more secure serialization method such as Apache Avro upon deployment.

    However, the most important future work is the extension of the system's analysis capabilities.
    At the moment, the models are heavily limited by the dataset used to train them.
    Extending the dataset with more data from the minority class would likely improve the recall, and thus the
    overall performance of the models.
    Different approaches to the analysis of the network state can be implemented, such as unsupervised learning on the
    graph representation for clustering or anomaly detection or transformer based approaches for capturing long-range dependencies
    in the traffic patterns of the network could prove beneficial.
    Additional representations of the network state can be implemented, that allow for a different perspective on
    the network state.


    \section{Conclusion}
    In summary, this paper has presented a conceptual design for a distributed system to enhance deep packet
    inspection and event detection within large-scale network environments.
    Despite its theoretical nature and the limitations posed by the current implementation,
    the proposed system demonstrates significant potential for real-time analysis, anomaly detection,
    and network security enhancement. Future work will focus on improving scalability, performance,
    and analysis capabilities, with the aim of developing a more sophisticated and efficient system.
    Through continuous improvement and adaptation, this project contributes to the ongoing evolution of network
    security mechanisms, ensuring they remain effective against the challenges of modern network infrastructures.



    \printbibliography


\end{document}