Development of an n8n-Based Automated Framework for Penetration Testing Using Large Language Models and Structured Evidence

Abstract

This research aims to develop and evaluate an evidence-driven automated framework for penetration testing by integrating large language models with hallucination-aware verification. The proposed framework addresses key limitations of traditional penetration testing, including heavy reliance on expert analysts, fragmented use of multiple security tools, heterogeneous tool outputs, and inconsistent results across repeated testing cycles. The framework employs n8n as the central workflow orchestration platform within a Docker container stack. It classifies assessment targets into four categories: network service systems, modern web applications, API security systems, and systems involving secret leakage or insecure configuration. Based on the target category, the framework selects an appropriate testing strategy, collects outputs from relevant security assessment modules, transforms them into structured evidence, and stores them in a PostgreSQL database. GPT-4o is then used through prompt engineering and structured output, without model fine-tuning, to support vulnerability interpretation, evidence correlation, and risk prioritization. To reduce the risk of unsupported or hallucinated findings, the framework incorporates a hallucination-aware verification mechanism requiring every reported finding to be supported by tool-generated evidence or execution logs before inclusion in the final report. The experiment was conducted using four deliberately vulnerable systems: Metasploitable3, OWASP Juice Shop, OWASP crAPI, and OWASP WrongSecrets. Each target was tested six times, resulting in 24 campaign runs. The experimental results indicate that the proposed framework consistently collected structured evidence, achieving structured-evidence consistency above 97% across all targets. The verification pass rate of LLM-generated findings ranged from 83.02% to 90.32%, while the average traceability score ranged from 89.83% to 96.83%. These results suggest that combining workflow automation, structured evidence management, and evidence-based verification of large language model outputs can improve the reliability, consistency, and traceability of automated penetration testing.