How to Build Complex, Large-Scale
Emulated Networks
T-Labs 2010
Simon Knight, Hung Nguyen, Matthew Roughan, Nick
Falkner, Olaf Maennel, Randy Bush
[email protected]
1 / 44
About Me
• Simon Knight
• PhD Student at The University of Adelaide
2 / 44
Adelaide
3 / 44
Data Networks
• Routers, Links, Autonomous Systems
Router A
Router B
AS2
Router C
Autonomous
System
AS1
AS3
Internetwork
4 / 44
Small Internet
5 / 44
Research Platforms
Scale
Simulation
Emulation
Hardware
Realism
6 / 44
Emulation
• Virtual routers in software emulate hardware routers
• Realism, but need configuration
Router 1
Router 2
Router 3
Routing
Process
Routing
Process
Routing
Process
Virtual Machine 1
Virtual Machine 2
Virtual Machine 3
7 / 44
Configuration Problem
• Need to configure large number of devices
• Many lines of code
• Low-level configuration language
• and allocate resources (IPs, AS numbers)
• Repeated trials difficult
• Limits results
• Time consuming, tedious, doesn’t scale to large
networks
• Error-prone
8 / 44
Configuration Errors
• Valid syntax, wrong semantics
• E.g. link IP addresses
192.168.0.1
129.168.0.2
A
192.168.3.1
B
192.168.1.1
192.168.3.2
192.168.1.2
D
192.168.2.1
192.168.2.2
C
9 / 44
Configuration Errors
• Link from A to B has address wrong
• Link will not carry traffic
192.168.0.1
129.168.0.2
A
B
D
C
10 / 44
Configuration Errors
• Network automatically routes around problem
• User may not realise, but results impacted
A
B
D
C
11 / 44
Small Internet
Link failure here much harder to spot
12 / 44
Netkit
• Creates and connects Virtual Machines (VMs)
• Use User-mode Linux (UML)
• Quagga routing, used on real routers
• Requires configurations
Quagga
Routing
Quagga
Routing
Quagga
Routing
UML VM1
UML VM2
UML VM3
Linux Host Machine
13 / 44
User-mode Linux
• Runs Linux instances inside Linux host machine
• Can run standard Linux software
• We use Quagga to allow UML machine to act as
router
• Can also run DNS (using standard BIND) package
• Netkit labs for setting up email, DNS
• Can run large-scale realistic network
• test network email implementation
14 / 44
User-mode Linux
• Standard analysis tools - ping, traceroute,
nmap,tcpdump
• Easy to add new software packages
• Full scripting - bash perl, python, etc
• Runs on standard hardware
• Have run up to 200 VMs on a standard desktop Linux
machine
• Can also interconnect machines using a ssh tunnel
quickly build up a big lab
15 / 44
Netkit Lab Structure
Configurations placed in a folder
LAB
lab.conf r1A r1A.startup r1B
r1B.startup r1C r1C.startup
LAB/r1A
etc
root
LAB/r1A/etc
resolv.conf shadow
snmp
LAB/r1A/etc/zebra
bgpd.conf daemons
ospfd.conf zebra.conf
ssh
zebra
16 / 44
Using Netkit
lab.conf specifies collision domains
(point-to-point links here)
r1A[0]=10.0.0.12
r1A[1]=10.0.0.4
r1B[0]=10.0.0.12
r1B[1]=10.0.0.8
r1C[0]=10.0.0.8
r1C[1]=10.0.0.4
17 / 44
Using Netkit
Starting lab
lstart
Stopping lab
lhalt
Check running VMs
vlist
18 / 44
Quagga
• Cisco-like syntax
• Can be used on real routers
• run on a Linux machine and communicate with Cisco,
Juniper, etc hardware routers
• Supports OSPF, RIP, IS-IS, BGP
• Allows more realistic studies than simulation
• Realism requires configurations!
19 / 44
Script based solution
• Partial solution
• Limited checking
• Manual or semi-automated resource allocation
Resource
Allocations
Automated Script
Router
Configuration
Files
Configuration
Templates
20 / 44
Abstraction
• Better solution
• Operators work with networks not config files
• Aim: provide abstraction
• automated tools to convert high-level descriptions into
low-level configurations
• Verification now possible
21 / 44
AutoNetkit
• Generates Netkit configurations
Physical & Logical
Topologies
Network
Description
Policy
Fragments
Resource
Allocation
Plugins
Network Model
Compiler
Netkit
Configuration
Files
Deployment
Verification
Netkit Host
22 / 44
AutoNetkit
• Template based
• Not limited to Netkit/Quagga
• Automated Resource allocation
• Automatically assign IP addresses
• DNS configuration
• Automatically setup based on allocated IPs
23 / 44
AutoNetkit: Creating Networks
myAS
myInternet
myInternet = Internetwork()
myAS = myInternet.addAS(1234)
24 / 44
AutoNetkit: Adding Routers
rA
rB
rC
myAS
myInternet
rA = myAS.addRouter("Router_A")
rB = myAS.addRouter("Router_B")
rC = myAS.addRouter("Router_C")
25 / 44
AutoNetkit: Adding Links
rA
rB
rC
myAS
myInternet
myAS.addLink(rA, rB)
myAS.addLink(rA, rC)
myAS.addLink(rB, rC)
26 / 44
AutoNetkit: Configuring and Deploying
rA
rB
Configs
rC
myAS
myInternet
Netkit Host
myHost
myInternet.compile()
myInternet.addRouting()
myNk =Netkit("myHost", "sknight")
myNk.deploy(myInternet)
27 / 44
Extending: Plugins
• Network model allows us to study network
• Powerful for research
• Use plugins which interact with network model
Physical & Logical
Topologies
Network
Description
Policy
Fragments
Resource
Allocation
Plugins
Network Model
Compiler
Netkit
Configuration
Files
Deployment
Verification
Netkit Host
28 / 44
Extending: Graphs
• Use networkx
• http://networkx.lanl.gov/
• Python graph package
• Convert network into a graph
• Routers become nodes, links become edges
• Allows powerful analysis
• Use math functions for network analysis
• Networkx includes many graph functions
• shortest paths, node degree, tests for fully connected
29 / 44
Graphs: Plotting
Uses Networkx and Graphviz
Plot of AS 1234
Router_A
.10 .6
eth0 eth1
10.0.0.4 /30
.5
eth1
10.0.0.8/30
Router_B
.13
eth0
10.0.0.12/30
.9
eth1 .14
eth0
Router_C
30 / 44
Graphs: Simple Example
Place DNS server in middle of network
# High-level user command
myInternet.addDNS()
# Low-level plugins called:
G = getGraphStructure(network)
if nx.is_strongly_connected(G):
# Get list of all central routers
centerList = nx.center(G)
# Return first router in list
centralRouter = centerList[0]
myAS.setDNSServer(centralRouter)
31 / 44
Graphs: Advanced Example
• Optimisation for traffic engineering
• Based on Internet traffic engineering by optimizing
OSPF weights
• Overview:
• Traffic matrix provided which gives end-to-end traffic
•
•
•
•
•
loads
Calculate load on each link
A link is high cost as load approaches capacity
Vary IGP weights, calculate new costs
Calculate total cost for weights
Use genetic algorithm to choose best weights
• Automated in AutoNetkit
• Carries out optimisation, applies to network, deploys,
verifies
32 / 44
Verification
• Measurement: automated traceroute tests
• Especially important for large networks
• Run shortest-path algorithm over networkx graph
• Compare to traceroutes run across network
33 / 44
Performance
• 200 node test network
• BGP, OSPF, DNS
• Manual
• 2 days to configure
• thousands of lines of low-level configuration code
• 1 day to test (traceroutes)
• AutoNetkit
• 1 hour to configure
• hundreds of lines of high-level code
• automatic testing
34 / 44
BGP Policy: Fragment approach
• Text based fragments
• Code-snippets
• straightforward to implement
• Limitations
• Locked to vendor (eg Cisco IOS, Juniper JunOS)
• Low-level, per-device
• Limited verification or interaction modelling
35 / 44
BGP Policy: n-strata
• Formally provable by following mathematical
properties
• Work of Tim Griffin - builds on metarouting
• Policy broken into set of strata
• eg 3-level could represent peering relationship
• Customer: 0
• Peer: 1
• Provider: 2
36 / 44
BGP Policy: n-strata
• These values are passed around the network
• Policy placed on links, describes actions to take
based on values
• eg don’t advertise provider links to peers
• eg prefer peer over provider link
• Can model these actions mathematically
• Prove properties such as convergence
37 / 44
BGP Policy: n-strata Implementation
# eBGP links
myInternet.addLink(r1B, r2A, route_map="h")
myInternet.addLink(r2A, r3B, route_map="d")
myInternet.addLink(r1C, r3A, route_map="g")
• Implemented in AutoNetkit
• using community attributes and local-preference
values
• Work in progress
38 / 44
BGP Policy: Where next
• Want to describe routing at high level in terms of
business goals
• n-strata may be formal, but is detailed and technical
• Integrate other research into BGP policy
• visualisation, simplification, verification
• Formal building blocks for policy
39 / 44
Future Work
Add GUI,
Network Design
Physical & Logical
Topologies
Network
Description
Policy
Fragments
Support more
Routing Protocols
(RIP, IS-IS)
Resource
Allocation
Plugins
New Plugins
(Optimisation,
Analysis)
Network Model
Compiler
Netkit
Configuration
Files
Deployment
Verification
Formal Methods
for Verification
Netkit Host
Deploy to Cisco
and Juniper routers
40 / 44
Help Out
• Design patterns used by ISPs
• Network design
• PoP layout, redundancy requirements, etc
• Bgp policy examples, high level goals
• Info on how operators build and manage networks
41 / 44
Conclusion
• AutoNetkit
• Tool to simplify configure of complex, large-scale
emulated networks
• Enables low-cost research of ISP size networks
• Available at:
http://bandicoot.maths.adelaide.edu.au/AutoNetKit/
42 / 44
Questions and Discussion
43 / 44