Presented at SANS @ Night, Wednesday, June 27, 2018 (7:15-8:15 PM), by Matthew J. Harmon (GSEC, GCIH, GCIA, CISSP, @mjharmon). Matthew is Principal Consultant at IT Risk Limited specializing in GRC, Technology Risk Assessments, Remediation, and Interim CISO services. He is a SANS Community & Mentor Instructor celebrating his 10-year anniversary, teaching SEC 401 (Security Essentials), SEC 504 (Hacker Tools, Techniques, Exploits & Incident Handling), and SEC 464 (Hacker Guard, IT Operations Baselining). He is also a Computer Science Course Author & Instructor at St. Paul College, teaching:
- 2461 70 & 71 Computer Networking 3 - Linux
- 2480 40 Network Security & Penetration Prevention
- 2482 40 Security Incident Handling, Response and Disaster Recovery
- 2484 40 Ethical Hacking & Countermeasures
Matthew is also a Cyber Security Summit Cyber Range Committee Member (building team competition and hack-a-thon events) and works with the NorSec Foundation on cyber range research, development, and malware analysis.
Presentation repository: https://github.com/itriskltd/Information-Security-and-Risk-Public-Presentations
Agenda
- Cyber Ranges: what they are and why we need more of them
- Offensive and defensive practice
- Design and product validation proving grounds
- Once interconnected, they become more than the sum of their parts
- Examples of current large-scale cyber ranges
- Considerations of cyber range design
- How to safely build your own cyber range
What Is a Cyber Range?
In this context, a cyber range is any contained environment representative of a realistic enterprise network, exclusively used for detonating malware, testing new Metasploit modules, or running some random code found on a PasteBin or uploaded to your not-a-honeypot-backup-DNS-server. It is disconnected from the Internet and your internal network, able to be quickly refreshed after testing.
Is My Dev Environment a Cyber Range?
No, but integrating a cyber range with your DevOps process brings red and blue teams together into a nice shade of purple. Training offensively allows developers to think offensively and build around application or framework weaknesses.
Examples of Cyber Ranges
NATO War Games with JYVSECTEC
JYVSECTEC out of Finland has built several full cyber ranges:
- NorthernBank: A financial organization simulation including payment processors, users, cash dispensary, and end-user payment systems
- Funnel: A road tunnel provider with all SCADA components for controlling traffic flow through a tunnel
- Watti: An electricity company simulation
- RNA: An Internet Service Provider simulation
Source: https://jyvsectec.fi/wp-content/uploads/2017/02/JYVSECTEC-cyber-range.pdf
SANS NetWars
CyberCity is a 1:87 scale miniaturized physical city featuring SCADA-controlled electrical power distribution, water, transit, hospital, bank, retail, and residential infrastructures. https://www.sans.org/netwars/cybercity
DFIR Tournament provides Digital Forensics, Incident Response, and Threat Hunting scenarios. https://www.sans.org/netwars/dfir-tournament
Other Examples
- Capture the Flag events
- Follow-the-Maze style challenges (e.g., the SANS Holiday Special)
- Scenario Simulation, Execution, and Observation systems: MITRE’s CALDERA, Uber’s Metta
- Defensive exercises such as the CCDC (Collegiate Cyber Defense Competition)
- Offensive exercises such as the OSCP Lab
MERIT’s Michigan Cyber Range
MERIT’s Cyber Range started in 2012. “Powered by Merit Network, the nation’s longest-running research and education network, the Michigan Cyber Range is the nation’s largest unclassified, network-accessible cybersecurity training platform.”
- YouTube Channel: https://www.youtube.com/user/michigancyberrange/videos
- Website: https://www.merit.edu/cyberrange/
- Alphaville: A showcase of what is possible when gaming and cyber range eSports competitions combine. https://www.youtube.com/watch?v=9E08xSGviRI
The National Cyber Range
A DARPA project from 2009-2012, later managed by the DoD Test Resources Management Center, providing “mission-tailored, hi-fidelity cyber environments that enable independent and objective testing and evaluation of advanced cyberspace capabilities.”
- https://www.acq.osd.mil/dte-trmc/ncr.html
- Overview PDF: https://www.acq.osd.mil/dte-trmc/docs/20150224_NCR%20Overview_DistA.pdf (Distribution Statement A, SB Case Number 15-S-0994)
Architecture
The National Cyber Range consists of multiple facilities: the Range Operations Center (the “Dungeon Masters”), Reconfigurable Test Suites (competing teams), the Range Support Center (NPCs that keep everything working), the Security Office (where “character sheets” are stored), and the High Security Data Center.
Operational Procedures
Hardware and software definition, expected results, resource allocation, tool provisioning and configuration, range configuration loaded into the tool being tested, test execution including event-specific simulations, and sanitization of the environment and device being tested.
Compliance Integration
For organizations without DARPA’s budget, the key question is: “Does Product A close a requirements gap?” – meaning, “How does adding a technology to my existing environment reduce my threat surface?” Cyber ranges serve as a proving ground for vendor evaluation, third-party risk assessment, and product validation.
In a representative scenario, left and right enclaves simulate a full Microsoft enterprise (Domain Controller, Exchange Server, Print Server, 600 users with firewall) and a remote facility (400 Windows 7 users), connected through spanning and corporate switches to a simulated internet. Data enters via SneakerNet and results are walked out through a clean system.
Each step of the acquisition, development, engineering, or testing process fits into a cyber range – from setting requirements, through vendor evaluation and security architecture review, to baselining, vulnerability assessment, aggressive resiliency testing, and large-scale environment simulation.
Cyber Range Design Principles
- Contained: Isolated from production networks
- Auto Scaling: Grows as you add resources
- Routing between separate networks: Support for multiple enclaves
- Encrypted, routed, peer tunnels: Secure inter-range communication
- Explicit and Validated Authorization: No implicit trust
- Rapid Restoration: Quick reset after detonation
- Portable: For competitions and field exercises
Proof of Concept
Hardware
- Raspberry Pi cluster (seven Pi’s connected through a Blackbox powered hub)
- PC Engines apu2c4 running OPNsense (built on HardenedBSD)
- USB Armory running custom Debian Linux as a Hardware Security Module
- YubiKey for key storage
Software Components
- OPNsense on HardenedBSD (chosen over pfSense for proactive security feature integration)
- BIRD & FRR for routing
- tinc (GRE/BGP) & WireGuard (GRE/OSPF) for tunneling
- iPXE / netboot.xyz for network booting
- QEMU for architecture emulation and shim
- YubiKey for authentication
One Raspberry Pi is dedicated to serving images over iPXE and key management via the certificate authority on the USB Armory. Another is the management route-server running BIRD and WireGuard. The apu2c4 runs OPNsense with tinc. The remaining five devices are available for attack and defense scenarios.
For future builds: the Ground Electronics Circumference Raspberry Pi cluster board from CrowdSupply. https://www.crowdsupply.com/ground-electronics/circumference/
Secure Deployment
Hostile Authentication Terminal (H.A.T.)
An init RAM disk-based tool that assumes everything is compromised. It dumps and analyzes RAM, pulls BIOS/EFI, validates known firmware, and once state is known, begins bootstrap via iPXE over HTTPS (experimental, work by P. Danek, netboot.xyz, et al.) or via rsync’d encrypted ZFS snapshots.
Multi-Tier Authentication System (M.T.A.S.)
An init RAM disk-based authentication system using:
- YubiKey-based GPG and x509 certificates
- fwknop with GPG for single-packet authorization
- Encrypted credential and routing packages
- GPG-unlocked routing packages
- Bootstrap server connection
- ZFS snapshot decryption key
- Peer configuration and public keys
YubiKey Configuration
The YubiKey operates as a Hardware Security Module supporting OTP + CCID + GPG mode with touch-before-authentication. References:
- https://github.com/drduh/YubiKey-Guide#4.7-requiring-touch-to-authenticate
- https://github.com/Yubico/libu2f-host
- https://developers.yubico.com/libu2f-host/Mode_switch_YubiKey.html
echo -e '\x06\x00\x00\x00' | u2f-host -d -a sendrecv -c c0Adversary Simulation Tools
- MITRE CALDERA (Windows): https://github.com/mitre/caldera
- Uber Metta (Windows, macOS, Linux): https://github.com/uber-common/metta
- Netflix Simian Army: https://github.com/Netflix/ – Chaos Monkey randomly takes down systems, Janitor Monkey cleans up temp files and remnants, Conformity Monkey checks configurations and ensures systems are as expected
- Dumpster Fire: https://github.com/TryCatchHCF/DumpsterFire
- Malware detonation capabilities
dn42: Security Research Network
The design is modeled after dn42, a decentralized security research network associated with the Chaos Computer Congress and Freinet Germany, using AS424242. https://nixnodes.net/dn42/graph/
Routing Architecture
BGP through tinc with FRR at the network edge: tinc was chosen for its solid history and peer recommendations. It sends frequent PING/PONG packets discovering new routes, making it perfect for discovering and adding new BGP peers.
OSPF through WireGuard with BIRD internally: used as the internal protocol because it is quiet and doesn’t send any data when idle. WireGuard is formally verified.
Calming the Route Chaos: BIRD Communities
BGP communities are used to automatically test for latency and bandwidth, setting appropriate values to optimize traffic flow (sourced from https://dn42.net/howto/Bird-communities):
Latency communities (AS 64511):
| Community | Latency |
|---|---|
| (64511, 1) | 0 to 2.7ms |
| (64511, 2) | 2.7ms to 7.3ms |
| (64511, 3) | 7.3ms to 20ms |
| (64511, 4) | 20ms to 55ms |
| (64511, 5) | 55ms to 148ms |
| (64511, 6) | 148ms to 403ms |
| (64511, 7) | 403ms to 1097ms |
| (64511, 8) | 1097ms to 2981ms |
| (64511, 9) | > 2981ms |
Formula: latency in [exp(x-1), exp(x)] ms for x < 10
Bandwidth communities:
| Community | Bandwidth |
|---|---|
| (64511, 21) | >= 0.1 Mbit |
| (64511, 22) | >= 1 Mbit |
| (64511, 23) | >= 10 Mbit |
| (64511, 24) | >= 100 Mbit |
| (64511, 25) | >= 1000 Mbit |
Formula: bw >= 10^(x-2) Mbit, where bw = min(up, down) for asymmetric connections
Encryption communities:
| Community | Encryption Level |
|---|---|
| (64511, 31) | Not encrypted |
| (64511, 32) | Encrypted with unsafe VPN solution |
| (64511, 33) | Encrypted with safe VPN (no PFS – typical OpenVPN p2p falls here) |
| (64511, 34) | Encrypted with safe VPN with PFS (Perfect Forward Secrecy) |
Example: a lab data center connection to the closest peer would be 1, 25, 34 – latency under 2.7ms (both have MICE no more than two hops away), 1 Gig fiber uplink, and encryption with tinc.
With a port of WireGuard for HardenedBSD, exploration into using it for external pipes (not just internal) was underway.
Certification & Accreditation
It doesn’t have to be boring:
- Device testing through crowdsourced penetration tests over the cyber range network
- Ongoing bug bounties with explicit permission built in
- Hack-a-thons anywhere via an interconnected range
Why Build a Cyber Range?
- Proper security research
- Cyber eSport competitions
- Corp-to-Corp war games
- Device testing & claim validation (medical, ICS, toy bears, smart TVs)
- Students writing malware (in a safe environment)
- Connect your range to your peers – long-distance LAN parties
Shoot for the stars.
Sources, References & Credits
- Colonel Stephanie Horvath, US Army MN National Guard
- JYVSECTEC
- US Dept of Defense
- SANS NetWars: https://www.sans.org/netwars/cybercity
- dn42:
- BIRD Communities: https://dn42.net/howto/Bird-communities
- BIRD Community Latency Script: https://github.com/Mic92/bird-dn42/blob/master/bgp-community.rb
- SpaceX Night Launch (original videographer unknown)
- CrowdSupply Circumference: https://www.crowdsupply.com/ground-electronics/circumference/
- YubiKey Configuration:
- Lou Ann Jensen
- David La Belle
- Minneapolis Starry Night (author unknown, found on r/minneapolis)
- Uber Metta: https://github.com/uber-common/metta
- MITRE CALDERA: https://github.com/mitre/caldera
- BSD RP BGP examples: https://bsdrp.net/documentation/examples/bgp_route_reflector_and_confederation_using_quagga_and_bird
- @HyperionGray Dark Web Map: https://blog.hyperiongray.com/dark-web-map-introduction/
Post-Script: For Network Engineers
If the previous slides made you think about routing loops and other insanity, BIRD BGP communities with automatic calculations help calm the chaos:
- https://bsdrp.net/documentation/examples/bgp_route_reflector_and_confederation_using_quagga_and_bird
- https://github.com/Mic92/bird-dn42/blob/master/bgp-community.rb
Presentation Repository: https://bit.ly/ITRiskPres
Contact: matthew@itriskltd.com