AUTONOMOUS CYBER DEFENSE AGENTS UTILIZING REINFORCEMENT LEARNING FOR SELF-HEALING NETWORK SECURITY

Abstract

In a world of ever-advancing cyberattacks, static defenses are powerless against dynamic attacks. Self-healing, autonomous cyber-defense agents with deep reinforcement learning (DRL) as their feature offer robust and adaptive security by identifying, containing, and healing from attacks in the absence of extensive human intervention. This work poses the question: Can DRL-based agents automatically defend and self- heal multiple networks in realistic adversary environments? We introduce a hierarchical DRL approach and apply it in CybORG++ scenarios, breaking down defense activities detection, isolation, and recovery into sub-policies of experts controlled by a master policy. Our experiments with various adversary scenarios, including APT-style stealthy attacks, demonstrate that our agents outperform flat policies by 15–25% better recovery times, 30% better false positives, and better clean host ratios maintenance. Moreover, transformer network-based entity-based DRL possesses stronger zero-shot generalization across unseen network topologies than MLP-based agents. Simulations show agents recovered around 90% of the crashed nodes in specified recovery windows, validating system-level robustness. Nevertheless, there are still certain limitations: simulated environments lag behind real-world complexity, and DRL agents represent high-training-cost entities in terms of heavy logging infrastructure. These work counters in three respects: (1) an empirically validated self-healing agent for supplying complete-spectrum cyber protection across a hierarchical topology; (2) experimentation with network scenario generalizability; and (3) an end-to-implement autonomous defense system for high-value systems and enterprise networks. This is a landmark step in cyber defense and an economical, smart, and feasible vision for future self-healing security infrastructure.