IPQ806x Hardware
acceleration
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
NSS acceleration model
• Features
• Designed for Home Gateways (CPE)
• Flow detection based “All-or-nothing”
offload
• Acceleration supports:
•
IPv4, IPv6, NAT, PPPoE, L2TP, VLAN, Qdisc
• Performance gain
• Linux: 640k pps (bridged) – 220kpps (routed)
• NSS: 7200k pps (bridged) – 7200kpps (routed)
• 11x (bridged) – 32x (routed)
• Functional behavior
• 0% cpu load seen in Linux
• Keep Linux counters up to date
• Does not require functional changes at an
upper level (user space)
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
ECM Front End Inspect Packets and Events
• Inspect all outgoing packets at POSTROUTING chain by registering post routing hooks.
• Inspect conntrack and device events to destroy and regenerate connection.
• Inspect NSS status and stats to update the connection state and statistics info in Linux and ECM DB.
IPCT_DESTROY
/ IPCT_MARK
INTERFACE
NOTIFIER
BOND NOTIFIER
CONNTRACK NOTIFIER
LINUX CONNTRACK
MTU / UP/ DOWN /
Bond Link/
Release/
Enslave
L3 PKTs
L2 PKTs
NETFILTER
IPvX
POST ROUTING
L3 PKTs
FRONT END
L2 PKTs
NETFILTER
PF_BRIDGE
POST ROUTING
NSS Status
and Stats Sync
NSS Driver
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
NSS Firmware
Example of IPv4 TCP rule creation
Linux
Linux net stack
ECM
Connection Established
Routing or
Bridging
Netfiter
Postrouting
Netfiter
Prerouting
Check TCP
Connection can be
accel?
NSS Driver
Send the Packet
to Linux
No
TCP: ACK
Or Syn+ACK
TCP Data
Match Any
Accelerated
Rule?
Create
IPv4 Rule
Transmit
the packet
Add the rule & reply the establish cmd
Packet
The Accelerated TCP flow
NSS Firmware
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Example of IPv4 TCP rule destroy
Linux Kernel
ECM
Linux net stack
Remove the connection
Conntrack destroy event
Conntrack
Connection
Destroy ?
Routing /
Bridging
Netfiter
Postrouting
Netfiter
Prerouting
NSS Driver
Send the Packet
to Linux
Destroy
IPv4 Rule
IPv4 Rule
Sync
Transmit
The packet
No
TCP: FIN
TCP Data
Is it a
connected TCP
Data?
Add the rule & reply the establish cmd
The Accelerated TCP flow
NSS Firmware
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Packet
Example of IPv4 rule API
• Common messaging interfaces
•
•
•
Protocol type: IPv4, IPv6, PPP, LAG…
Rule type: CREATE, DESTROY, CONN_STAT_SYNC , NODE_STAT_SYNC…
Callback/args pointers: will be passed back in the FW ACK/NACK reply
• IPv4 rule create message structure example
struct nss_ipv4_rule_create_msg {
/* Request */
uint16_t valid_flags;
struct nss_ipv4_protocol_tcp_rule tcp_rule;
struct nss_ipv4_pppoe_rule pppoe_rule;
struct nss_ipv4_qos_rule qos_rule;
struct nss_ipv4_dscp_rule dscp_rule;
struct nss_ipv4_vlan_rule vlan_primary_rule;
struct nss_ipv4_vlan_rule vlan_secondary_rule;
/* Indicate which of the parameters below is filled-in
Indirectly says which operation to be done on the flow */
/* Bit flags associated with the rule */
/* src_ip, dst_ip, src_port, dst_port, proto */
/* src_mac, dst_mac, src_iface, dst_iface, src_mtu, dst_mtu,
nat_src_ip, nat_dst_ip, nat_src_port, nat_dst_port */
/* TCP related accleration parameters */
/* flow_session_id, flow_remote_mac, ret_session_id, ret_remote_mac */
/* flow_qos_tag, ret_qos_tag, */
/* flow_dscp, ret_dstp */
/* ingress_vlan_tag, egress_vlan_tag */
/* ingress_vlan_tag, egress_vlan_tag – for QinQ */
/* Response */
uint32_t index;
/*Slot ID for cache stats to host OS */
uint16_t rule_flags;
struct nss_ipv4_5tuple tuple;
struct nss_ipv4_connection_rule conn_rule;
};
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Interfaces & Connections statistics update
• Stats updates sent
periodically from the
Firmware
• Per-interfaces stats
(update net_devices)
• Per-connections stats
(update conntracks)
• Minor modifications to
ppp/l2tp/ipsec… layers
for iface look-up and
stats update
conntracks
stats++
Linux
ECM
Look-up
conntrack
ethN
ppp0
br0
tun0
Look-up
interface
stats++
NSS Driver
NODE_STATS_SYNC
message
NSS Firmware
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
CONN_STATS_SYNC
message
Qdisc acceleration
tbf
nsstbf
1:
10:
10:
prio
10:1
100:
tbf
f
i
p
f
o
r
e
d
1000:
nssprio
10:3
10:1
10:2
tbf
1:
100:
200:
300:
2000:
r
e
d
10:2
nsstbl
nsstbl
n f
s i
s f
p o
n r
s e
s d
1000:
10:3
200:
300:
2000:
n r
s e
s d
# tc qdisc add dev eth0 root handle 1: tbf rate 1000Mbit burst 100k limit 100
# tc qdisc add dev eth0 root handle 1: nsstbl rate 1000Mbit burst 100k
# tc qdisc add dev eth0 parent 1: handle 10: prio bands 3
# tc qdisc add dev eth0 parent 1: handle 10: nssprio bands 3
# tc qdisc add dev eth0 parent 10:1 handle 100: tbf rate 2Mbit burst 10k limit 100
# tc qdisc add dev eth0 parent 10:1 handle 100: nsstbl rate 2Mbit burst 10k
# tc qdisc add dev eth0 parent 100: handle 1000: pfifo limit 100
# tc qdisc add dev eth0 parent 100: handle 1000: nsspfifo limit 100
# tc qdisc add dev eth0 parent 10:2 handle 200: tbf rate 40Mbit burst 30k limit 100
# tc qdisc add dev eth0 parent 10:2 handle 200: nsstbl rate 40Mbit burst 30k
# tc qdisc add dev eth0 parent 200: handle 2000: red limit 100k min 30k max 80k \
# tc qdisc add dev eth0 parent 200: handle 2000: nssred limit 100k min 30k max 80k \
avpkt 1k burst 55 probability 0.20
avpkt 1k burst 55 probability 0.50
# tc qdisc add dev eth0 parent 10:3 handle 300: red limit 100k min 30k
maxof80k
# tcOttawa,
qdiscOn,add
dev eth0 parent 10:3 handle 300: nssred limit 100k min 30k max 80k \
Proceedings
netdev\0.1, Feb 14-17, 2015,
Canada
avpkt 1k burst 55 probability 0.30
avpkt 1k burst 55 probability 0.50 set_default