Design av pålitelighet i Telenors IP nett
Redundans og reserveleggingsmekanismer
Teleforum 2015 - Ove Tøien
1
Telenor IP Network (BRUT 2.0)
Services
Business
Residential /Business
Internet Services
L3-VPN Services
L2–VPN Services
Wholesale
Packet Voice
E-Line service
Vula service
Packet Voice
Gateway
IP/MPLS
Core
Mobile
base station backhole
PS Core
CS Core
Broadcasting
BRUT 2.0 – A NON STOP NETWORK
BRUT 2.0 is designed to be a Non Stop Network by implementing
a hole range of measures to reduce Service downtime caused
by nodal HW / SW faults, infrastructure faults or security
attacks.
Highlights
• Infrastructure redundancy
• Nodal redundancy
• Thorough Hardware testing and Inspection
• Thorough SW testing
• Network Scaling testing (signaling and performance)
• Non Stop Routing functionality
• In Service Software Upgrade
• Security defense mechanism
3
00 Month 0000
IP NETWORKS DEPENDS ON LAYER1
L1 AND IP NETWORK IS DESIGNED TOGETHER
Core DWDM network
•
2 independent networks
•
40 x 100Gbit/s channels
•
Connectivity between
larger city's
•
Connected to Nordic
DWDM-network
Bodø
HONNINGSVÅG
ØKSFJORDNES
TANA
LAKSELV
Tromsø
TROMSØ
TROMSØ
FINNSNES
FINNSNES
HARSTAD
HARSTAD
SORTLAND
SORTLAND
•
Fauske
Core IP/MPLS
nodes collocated
with Core DWDM
Edge IP/MPLS
nodes collocated
with Metro
DWDM
MOLDE
FAUSKE
ØRNES
TONNES
MO I RANA
HEMNESBERGET
MO I RANA
HEMNESBERGET
MOSJØEN
NESNA
TONNES
SANDNESSJØEN
NESNA
MOSJØEN
NAMSKOGAN
NAMSOS
LERKENDAL
STEINKJER
STØREN
SURNADAL
LEVANGER
LERKENDAL
ÅROLIA
BERKÅK
SUNNDALSØRA
ULSTEINVIK
OPPDAL
RØROS
ØRSTA
VOLDA
MÅLØY
TYNSET
NORDFJOREID
ALVDAL
DOMBÅS
FLORØ
SKEI
OTTA
FØRDE
KYRKJEBØ
HØYANGER
TRYSIL
SOGNDAL
RUTLEDAL
LEIKANGER
LILLEHAMMER
FAGERNES
LINDÅS
DALEKVAM
VOSS
HAMAR
GOL
NORHEIMSUND
BERGEN
GJØVIK
GEILO
JESSHEIM
ODDA
HUSNES
STORD
EDLAND
Bergen
ØLEN
SAND
HAUGESUND
VINJE
KOPERVIK
HØNEFOSS
RJUKAN
BRYNE
TØNSBERG
SKIEN
PORSGRUNN
DRANGEDAL
EVJE
FLEKKEFJORD
FARSUND
MANDAL
ASKIM
BLAKSTAD
KRISTIANSAND
MOSS
SARPSBORG
FREDRIKSTAD
SANDEFJORD
LARVIK
KRAGERØ
RISØR
TVEDESTRAND
ARENDAL
GRIMSTAD
LILLESAND
EGERSUND
LILLESTRØM
OSLO
HOLMESTRAND
SELJORD
FORUS VEST
SANDNES
KONGSVINGER
HOKKSUND
KONGSBERG
NOTODDEN
DRAMMEN
HJELMELAND
STAVANGER
ELVERUM
HOV
BRANDBU
KINSARVIK
Kristiansand
FAUSKE
BODØ
BODØ
INNDYR
BRØNNØYSUND
ÅLESUND
00 Month 0000
NARVIK
HAMARØY
ØRNES
INNDYR
SANDNESSJØEN
Trondheim
4
STORSTEINNES
MOEN
MOEN
KAUTOKEINO
NARVIK
SVOLVÆR
LEVANGER
STEINKJER
TRONDHEIM
Tønsberg
KIRKENES
SVOLVÆR
LEKNES
HAMSUND HAMARØY
LEKNES
HAMSUND
TRONDHEIM
AURE KYRKSÆTERØRA
KRISTIANSUND
ORKANGER
RENSVIK MELDAL
TRONDHEIM
Stavanger
ALTA
KARASJOK
NAMSOS
Skien
VADSØ
BRENNA
BRØNNØYSUND
Oslo
VARDØ
HAMMERFEST
•
Ålesund
KJØLLEFJORD
HAVØYSUND
HALDEN
Metro Core DWDM network
•
Ring structure between
core node pair
•
40 x 100 Gbit/s channels
•
Connectivity between core
and
•
Gives 2 independent
routes to core network
•
24 subnet / rings
Brut 2.0 logical topologya “ladder design”
IP/MPLS
Access
IP/MPLS
Edge
• Redundant design: “blue” and “red” side
• Follow DWDM redundant infrastructure
•
•Between Edge Routers connected to
core sites: a ladder design!
• Between these routers the network will
always look the same.
• Number of steps will vary: 1-9
• There is always link-, node- and site
redundancy.
“Ladder”
Max
9 steps
IP/MPLS
Core
• Delay and jitter is controllable
• Access Routers: Ring topology
• No MPLS FR, rerouting rely on protocol
convergence
IP/MPLS
Edge
IP/MPLS
Access
Structure Brut 2.0 (Norway)
24 Core, 150 Unified, 34 Mobile, 26 Voice, 2 Borders, 10 RR
Router Reflectors
Internet
Border
IP/MPLS Core
Voice
Cisco
DialUp
(IP)
IP/MPLS Edge
(PE routers)
Unified
BNG
Leid linje
TRIP
NGV Core
CS core
Mobile
CE
Voice
WiMAX BS
IP/MPLS Access
(PE routers)
xDSL
CSS
L2 Access
CSS
Fixed
Mobile
Customer Connections
Routing and MPLS Transport
- Design Principles
• All customer routes must be announced by BGP
• IP unicast traffic:
• All customer traffic must be MPLS switched
• Customer routes must not be installed on core routers
• All MPLS switched traffic must be protected by a fast reroute
mechanism (LFA / RSVP-TE FRR) to minimize the impacts from
network failures.
8
BGP Topologies
Why several BGP topologies
– Reduce mutual negative influence of “BGP poisoned” routes
between services
There are 4 separate peering topologies, completely
independent of each other:
1 for Internet Services routes
1 for Non-Internet Services routes
1 for Mobile Services routes
1 for Voice Services routes
•Only routes related to services covered by a specific
topology are announced in that topology
BRUT 2.0 – Security design
Brut 2.0
Several layers of infrastructure security.
Route filtering
Controlling routes
internal and
announced from and
to our Autonomous
system
Packet filtering
Protect infrastructure
by control source and
destination of
packets
 Separation of Plane
 Control
 Management
 Forwarding




Protocol security
Node protection
Network protection
Protect routing and
switching protocols
from interception and
unauthorized
connections.
Protect nodes from
unauthorized access
with centralized AAA,
firewall rules,
control/management
/forwarding plane
separation
Hide infrastructure
addresses, rate
limiting control
traffic, discard
segmented control
traffic
DOS/DDOS protection
Scrubber
Multicast security
No Hairpin routing
 Layer 2 segmentation
 Flow monitoring
These guidelines are the basis of Brut 2.0 security design
Security Shell Design
Access / Border
Shell 2
Core / Access router
Shell 3
Internet
•Box security (Control plane)
•Routing security
•Switcing security
•DOS attack Protection
Shell 1
•Box security(Control Plane)
•Box security(Control plane)
•Layer 2 security
•Routing security
•Hairpin routing
•Switching security
•Multicast security
DSLAM
•Layer 2 security
•Layer 3 security
•Multicast security
•DOS attack ”protection”
•Box security (control plane)
IP Core
IP Edge
Access
Customer network
•Layer 2 security
VPN
•Box security (Control plane)
•DOS attack Protection ?
•Layer 3 security
•Layer 2 security
•Multicast security
•Service security
•DOS attack protection
Packet Direcion
CE
Thanks!
[email protected]