distributed-network-event-detection-system/README.md

# Distributed Network Event Detection System

An alpha, graph-based Python network-intrusion-detection prototype. Its real-time components consume packet messages from Kafka, build and merge network graph state in Redis, dispatch node-analysis work, run model-based analysis, and print node results.

## Status and non-reproducible limitations

> **Not an end-to-end runnable deployment as checked out.** No packet producer/input-capture process or tested worker orchestration is provided. The included workers wait for messages on Kafka topics; starting them alone does not capture traffic.

- The package metadata is alpha and only classifies Python 3.11. Its `readme = "README.md"` refers to a README missing from `graph_based_intrusion_detection/`, which may prevent standard package build metadata generation.
- Node analysis loads models and scalers from hard-coded absolute paths outside this repository. Although similarly named model files exist under `resources/models/`, the configured paths do not point there. Update the configuration for an environment before starting the node analyzer.
- Kafka and Redis addresses are hard-coded as `localhost:9092` and `localhost:6379` in the workers.
- The local Compose stack uses unpinned `latest` container images and is not a reproducible production deployment.
- No test suite, project test command, console entry point, or application deployment definition is present.

Do not use this prototype as a production intrusion-detection control without supplying and validating the missing capture, model-path, orchestration, retention, and security configuration.

## Components and data flow

The real-time code uses these Kafka topics and Redis state:

1. `packet_processor` consumes `packets`, builds processed graph state, and publishes `processed_states`.
2. `state_merger` consumes `processed_states` and stores merged network state in Redis.
3. `work_dispatcher` reads network state, tracks nodes in Redis, sends work to `node_analysis`, and consumes `node_analysis_results`.
4. `node_analyzer` consumes node-analysis work, loads configured models/scalers, and publishes results.
5. `network_state_visualizer` reads the node registry from Redis and prints it to the terminal.

This describes the implemented connections, not a validated deployment order or a complete ingestion pipeline.

## Prerequisites and setup status

The nested package at `graph_based_intrusion_detection/` declares dependencies including Keras, TensorFlow, Kafka Python, Redis, NetworkX, NumPy, pandas, SciPy, scikit-learn, tqdm, and python-dotenv. Its package metadata identifies Python 3.11.

A standard package installation cannot be documented as reproducible until the missing nested README referenced by `pyproject.toml` is resolved. After package dependencies and metadata have been made installable, run the worker modules from `graph_based_intrusion_detection/` as shown below. No dependency lockfile is provided.

Before starting `node_analyzer`, make the configured model and scaler paths in `src/graph_based_intrusion_detection/utils/config.py` valid for the runtime. Set `LOG_FILE_PATH` and `LOG_LEVEL` before starting workers; for example: `export LOG_FILE_PATH=/tmp/network-event-detection.log` and `export LOG_LEVEL=INFO`. No environment-based broker or Redis configuration is implemented.

## Local infrastructure

From the nested package directory, start the provided local Kafka/Redis dependencies:

```bash
cd graph_based_intrusion_detection/deployment/kafka
docker compose up
```

This starts ZooKeeper, Kafka, and Redis and publishes Kafka on host port `9092` and Redis on host port `6379`. It is local development infrastructure only; the Compose file has no authentication, persistent-volume, image-pinning, or application-worker service configuration. Keep Kafka, Redis, and every message producer isolated to a trusted network: workers unpickle message content and must never consume messages from untrusted publishers.

## Worker commands

After dependencies are installed, models are configured, and Kafka/Redis are running locally, each worker has a direct Python module entry point. Run these from `graph_based_intrusion_detection/`; an optional trailing worker ID is accepted.

```bash
python -m graph_based_intrusion_detection.realtime_event_detection.packet_processor [worker-id]
python -m graph_based_intrusion_detection.realtime_event_detection.state_merger [worker-id]
python -m graph_based_intrusion_detection.realtime_event_detection.work_dispatcher [worker-id]
python -m graph_based_intrusion_detection.realtime_event_detection.node_analyzer [worker-id]
python -m graph_based_intrusion_detection.realtime_event_detection.network_state_visualizer [worker-id]
```

These are long-running workers. The repository does not establish a safe startup order, supervision model, shutdown procedure, health check, packet producer, or end-to-end validation command.

## Development and testing

No tests or test runner are present. The notebooks under `notebooks/` are exploratory artifacts rather than a documented operational workflow. There are no lint, format, build, or deployment scripts in the package metadata.

## Repository layout

- `graph_based_intrusion_detection/src/graph_based_intrusion_detection/` — package source
  - `realtime_event_detection/` — Kafka/Redis workers
  - `packet_processing/`, `graph_processing/`, `analysis/` — graph construction and model analysis
  - `utils/config.py` and `utils/constants.py` — hard-coded model paths, topics, ports, and runtime constants
- `graph_based_intrusion_detection/deployment/kafka/docker-compose.yml` — local ZooKeeper, Kafka, and Redis stack
- `graph_based_intrusion_detection/resources/models/` — checked-in model/scaler artifacts (not the paths currently configured for node analysis)
- `graph_based_intrusion_detection/notebooks/` — exploratory notebooks
- `infosheet.md` — capture/device notes, not operating documentation