Hi folks,
today I want to share some notes on IDS testing frameworks. I've been always using Ftester to test my snort installation, in my experimental security laboratory, in public agencies and even in private companies without changing almost anything. So I am quite a "fan" of ftester framework. It's easy to use, it has a clear infrastructure and it works pretty fine especially when you decide to stop to write hand written rules and you want to automate testing by reversing snort's rule files... it just works perfectly.
Few days ago I came across Pytbull: an Intrusion Detection/Prevention System (IDS/IPS) Testing Framework for Snort and Suricata. It can be used to test the detection and blocking capabilities of an IDS/IPS, to compare IDS/IPS, to compare configuration modifications and to check/validate configurations.
The framework is shipped with about 300 tests grouped in 8 testing modules:
clientSideAttacks: this module uses a reverse shell to provide the server with instructions to download remote malicious files. This module tests the ability of the IDS/IPS to protect against client-side attacks.
testRules: basic rules testing. These attacks are supposed to be detected by the rules sets shipped with the IDS/IPS.
badTraffic: Non RFC compliant packets are sent to the server to test how packets are processed.
fragmentedPackets: various fragmented payloads are sent to server to test its ability to recompose them and detect the attacks.
multipleFailedLogins: tests the ability of the server to track multiple failed logins (e.g. FTP). Makes use of custom rules on Snort and Suricata.
evasionTechniques: various evasion techniques are used to check if the IDS/IPS can detect them.
shellCodes: send various shellcodes to the server on port 21/tcp to test the ability of the server to detect/reject shellcodes.
denialOfService: tests the ability of the IDS/IPS to protect against DoS attempts
It is easily configurable and could integrate new modules in the future.
Everything is great, but unfortunately in its current version it does not work at all :(.
After you installed the dependencies, if you try to run the "first stable release" (0.3) you get errors. Just a fast look-into the code to see that a little mistake (this is a classic mistake that students do using 'cut and paste' ;) "f.close()" is made. Object F is not declared ... So just remove it, maintaining ftp.quit() and it will pass most of the controls. But still after a while you will get errors on import tests.
Concluding... I know pytbull will be my new favorite snort testing tool, it has a lot of new features, automatisms and it's a new project (which means that it will be more upgraded if compared to ftester). But, even if it has been released in the "first stable release" I was not able to run it. It still has some issues ( I call them: youth issues .... issues due to youth ... a very new project might comes with few experience... etc..). Summing up. During my last experimental session (I'll discuss more on this in future ) I still used ftester. But I am confident that pytbull folks will fix their great project just in time for my next experimental session :). I will definitely love pytbull framework.
