SCHED_DEADLINE
How to use it
Juri Lelli
Retis Lab – SSSUP
Pisa (Italy), June 26th 2014
TeCIP Insitute, Scuola Superiore Sant'Anna
Area della Ricerca CNR, Via G. Moruzzi 1
56127 Pisa, Italy
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
2/39
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
3/39
OS kernels and schedulers
●
Linux is an OS kernel (not a full OS as it doesn't
include FS utilities, WM, GUIs, compilers, etc.)
●
Composed by several sub-systems
●
One of which is the Scheduler
●
●
Decides which process to run where and when,
achieving an apparent simultaneous execution of
multiple processes on the same shared HW
Traditionally General Purpose: didn't include
proper Real-Time scheduling features
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
4/39
A new scheduling policy/class
SCHED_DEADLINE
SCHED_DEADLINE
SCHED_FIFO
SCHED_FIFO
SCHED_RR
SCHED_RR
SCHED_BATCH
SCHED_BATCH
SCHED_NORMAL
SCHED_NORMAL
SCHED_IDLE
SCHED_IDLE
deadline.c
rt.c
fair.c
Linux scheduler
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
5/39
An history of LKML posts
●
●
started within the ACTORS EU project (2008)
originally developed and maintained by Dario
Faggioli et al.
●
then I switched in (v4 April 6th, 2012)
●
9 versions posted
●
Merged by Linus on Sun, 2 Feb 2014 17:12:22
●
WE ARE IN MAINLINE!!! YAY!!!
●
Current stable is Linux 3.14.2
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
6/39
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
7/39
Predictability and Isolation
4 CPU-hog processes (i.e. while(1)) on 4 CPUs
SCHED_NORMAL
(default Linux
scheduling policy)
Jun 26, 2014
SCHED_DEADLINE
● finer-grained control over tasks
scheduling
● tasks don't interfere with each other
Juri Lelli - SCHED_DEADLINE @ Retis Lab
8/39
Sporadic task model
activation/wakeup
absolute deadline
start time
computation time
relative deadline
period
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
9/39
Real-time scheduling
algorithms
●
●
●
tasks are executed in priority order
Rate Monotonic (RM) → tasks with
shorter periods have higher priorities
Earliest Deadline First (EDF) → tasks
with earlier deadlines have higher
priorities
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
10/39
Static vs. Dynamic (UP)
τ1 → 2 every 5
2 4
+ ≃0.97
5 7
τ2 → 4 every 7
97% overall load
τ1
2
4
6
8
10
12
14
16
18
t
2
4
6
8
10
12
14
16
18
t
2
4
6
8
10
12
14
16
18
t
2
4
6
8
10
12
14
16
18
t
τ2
RM → not more than 69%
overall load (when n is
big)
τ1
EDF → can achieve 100%
τ2
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
11/39
Deadline scheduling (EDF)
τ1 → 5 every 9
τ2 → 2 every 6
~88.9%
overall load
τ1
t
τ2
t
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
12/39
Deadline scheduling (EDF)
τ1 → blocks just after the second activation
τ1 → resumes with the third instance of τ2
τ1
t
τ2
t
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
13/39
Deadline scheduling (EDF)
τ1 → second job behaves bad
τ1
t
τ2
t
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
14/39
Reservation Based scheduling
●
Concurrent real-time tasks compete for
resources (CPU time)
●
Resource Reservation mechanism
●
A task is allowed to execute for:
●
–
Q time units (runtime)
–
in every interval of length P (period)
Task utilization is U=Q/P
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
15/39
Reservation Based scheduling
●
●
●
●
●
Each reservation has its own dynamic
deadline (Constant Bandwidth Server)
Reservations are scheduled by EDF
EDF gives higher priority to more
urgent tasks
CBS slows down (or throttles)
misbehaving tasks
EDF + CBS ensures temporal
isolation
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
16/39
EDF + CBS
plain EDF
(block/unblock) →
τ1
t
τ2
t
τ1 → blocks just after the second activation
τ1 → resumes with the third instance of τ2
(CBS “unblock rule” applied)
τ1
t
τ2
t
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
17/39
EDF + CBS
plain EDF (bad
task) →
τ1
t
τ2
t
τ1 → second job behaves bad
→ once budget exausted, delay until
next activation period
τ1
t
τ2
t
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
18/39
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
19/39
Adds
●
●
●
●
new syscalls: sched_setattr(), sched_getattr()
system wide deadline bandwidth management
coupled with RT
–
/proc/sys/kernel/sched_rt_runtime_us
–
/proc/sys/kernel/sched_rt_period_us
rt_bw is enforced at runtime (rt throttling)
dl_bw is reserved with AC, and enforced at
runtime
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
20/39
Symmetric Multi-Processors
●
●
DL tasks are kept into CPU-specific
runqueues
implements Global EDF (DL tasks are
migrated among runqueues to achieve):
–
on an M-CPU system the M earliest DL
ready tasks are always running;
–
affinity/cpusets settings of all the DL tasks
is always respected
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
21/39
SMP ready!
4 CPU-hog processes (i.e. while(1)) on 4 CPUs
+ 1 more after a while...
new cpuhog5 arrival causes
a migration for cpuhog2
Jun 26, 2014
system stabilizes into a new
condition, similar to
partitioned scheduling
Juri Lelli - SCHED_DEADLINE @ Retis Lab
22/39
Interface
struct sched_attr {
u32 size;
u32 sched_policy;
u64 sched_flags;
/* SCHED_NORMAL, SCHED_BATCH */
s32 sched_nice;
/* SCHED_FIFO, SCHED_RR */
u32 sched_priority;
/* SCHED_DEADLINE */
u64 sched_runtime;
u64 sched_deadline;
u64 sched_period;
};
int sched_setattr(pid_t pid, const struct sched_attr *attr, unsigned int flags);
int sched_getattr(pid_t pid, const struct sched_attr *attr, unsigned int size,
unsigned int flags);
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
23/39
Example
#include <sched.h>
...
struct sched_attr attr;
attr.size = sizeof(struct attr);
attr.sched_policy = SCHED_DEADLINE;
attr.sched_runtime = 30000000;
attr.sched_period = 100000000;
attr.sched_deadline = attr.sched_period;
...
if (sched_setattr(gettid(), &attr, 0))
perror("sched_setattr()");
...
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
24/39
RR over sporadic task model
simple rule of thumb, see literature for
optimal settings...
average
runtime [ns]
deadline [ns]
period [ns]
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
25/39
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
26/39
Clone and Compile
●
Clone Linus' repo
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
●
Configure, compile and install
cp /boot/config`uname r` .config
yes "" | make oldconfig
make j `cat /proc/cpuinfo | grep processor | wc l | awk '{print $1 + 1}'`
sudo make modules_install install
updateinitramfs c k VERSION && updategrub
or wait for next Ubuntu, Fedora, etc.
release...
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
27/39
Test
●
rt-app
git clone [email protected]:gbagnoli/rtapp.git
cd rtapp
./autogen.sh (requires autotools)
./configure –withdeadline –withjson
make && make install
rtapp t 100000:10000:d D10 or rtapp taskset.json
●
schedtool
git clone [email protected]:jlelli/schedtooldl.git
cd schedtooldl
make && make install
schedtool E t 10000000:100000000 e yes
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
28/39
encapsulate a “yes” process
inside a 10 ms, 100ms reservation
schedtool E t 10000000:100000000 e yes
see execution trace with
kernelshark
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
29/39
encapsulate a “yes” process
inside a 10 ms, 100ms reservation
schedtool E t 10000000:100000000 e yes
10 ms
Jun 26, 2014
100 ms
Juri Lelli - SCHED_DEADLINE @ Retis Lab
30/39
Experiments with random tasks
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
31/39
Static vs. Dynamic (UP)
τ1 → 2 every 5
2 4
+ ≃0.97
5 7
τ2 → 4 every 7
97% overall load
τ1
2
4
6
2
4
6
2
4
2
4
τ2
8
10
12
14
16
18
t
RM → not more than 69%
overall load (when n is
big)
WITH rt-app!
8
10
12
14
16
18
t
6
8
10
12
14
16
18
t
6
8
10
12
14
16
18
t
τ1
EDF → can achieve 100%
τ2
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
32/39
Provide QoS to VMs
●
●
KVM creates one thread per VPCU and
one thread per VM doing I/O
encapsulate VPCU threads inside
reservations to provide QoS guarantees
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
33/39
Provide QoS to VMs
●
●
a single-core VM runs a CPU-bound activity: sysbench, calculation
of prime numbers up to a certain value
we vary the reservation budget of its VCPU
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
34/39
Provide QoS to VMs
●
●
a single-core VM runs a network activity: iperf server with external
requests from a client running on different HW
we vary the reservation budget of its VCPU
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
35/39
Provide QoS to VMs
●
●
vm1 (visualized) runs iperf server, vm2 sysbench
load of vm2 doesn't affect vm1 performance
vm2 reservation
vm1 reservation
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
36/39
Provide QoS to VMs
●
●
both vm1 (visualized) and vm2 run iperf servers
we still have a “certain degree” of isolation (probably better
with dedicated HW or different configurations)
vm2 reservation
vm1 reservation
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
37/39
Outline
●
●
Basics and status
What SCHED_DEADLINE is
implementing
●
API and usage
●
Demos
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
38/39
Thanks!
Questions ?
[email protected]
Jun 26, 2014
Juri Lelli - SCHED_DEADLINE @ Retis Lab
39/39