ioco Theory for Probabilistic Automata
M. Gerhold, M.I.A. Stoelinga
18/04/15
10th MBT Workshop, London 2015
1
OUTLINE
1. ioco Theory
2. pQTS and pioco
3. Testing with pQTS
2
Marcus Gerhold
18/04/15
Ideally: Impl. conforms to Spec iff Impl. passes test suite
Implementation
≡?
Implementation
passes tests
3
Marcus Gerhold
18/04/15
3/60
ioco Theory
ioco = Input / Output Conformance
Tretmans (1996) Test Generations with Inputs, Outputs,
and Repetitive Quiescence
Tretmans (2008) Model Based Testing with labeled transition
Systems
4
Marcus Gerhold
18/04/15
ioco Theory
●
●
●
●
●
●
●
hioco:
Osch (2006); Hybrid input-output conformance and test generation
mioco:
Heerink (1998); Ins and Outs in Refusal Testing
(r)tioco:
Brandan Briones, Brinksma (2005); A test generation framework for
quiescent real-time systems
Krichen, Tripakis (2004); Black Box Conformance Testing for real-time
systems
Hessel, Larsen, Mikucionis, Nielsen, Petersson, Skou (2008);
Online Testing of real-time systems using Uppaal
sioco:
Tretmans, Frantzen, Willemse (2004); A symbolic Framework for MBT
uioco:
van der Bijl, Rensink (2003); Compositional Testing with ioco
Marcus Gerhold
5
18/04/15
ioco Theory
●
●
●
Stokkink, Timmer, Stoelinga (2012); Talking Quiescence a rigorous theory
that supports parallel composition, action hiding and determinisation
Stokkink, Timmer, Stoelinga (2013); Divergent Quiescent Transition
Systems
Brinksma, Timmer, Stoelinga (2014); An Introduction to Model-Based
Testing
6
Marcus Gerhold
18/04/15
ioco Theory
Definition (QTS): A quiescent transition system is a five tuple
δ
δ
O
A=⟨ S , s 0 , L I , L ,→⟩
with
S
1eur?
LI
a finite set of states
the starting state
the input alphabet
→
the output alphabet + quiescence
the transition relation
s0
δ
LO
δ
coffee!
tea!
req_tea? req_coffee?
Important: Quiescent output action δ
7
Marcus Gerhold
18/04/15
ioco Theory
Definition (out): For a given QTS A and trace
σ
we write
δ
out A (σ )=after (σ)∩LO
to denote the set of all output actions as well as quiescence after
trace σ.
2eur?, δ
●
out( ε ) = {δ}
●
out( 1eur? ) = {coffee!}
●
out( 1eur? 2eur? ) = {coffee!}
●
out( 1eur? coffee! ) = {δ}
●
out( 2eur? ) = {δ}
●
out( δ ) = {δ}
●
out( coffee ) = ∅
1eur?
1eur?, 2eur?
coffee!
1eur?, 2eur?, δ
8
Marcus Gerhold
18/04/15
ioco Theory
Definition (ioco): Let Imp be an implementation and Spec
a specification. Furthermore let Imp be input-enabled, then
Imp ⊆ioco Spec ⇔ ∀ σ ∈traces (Spec):out Imp (σ)⊆out Spec (σ)
Intuition: Imp ioco Spec iff for all traces of Spec:
If Imp gives output x! After trace σ, then so can Spec
Some examples
9
Marcus Gerhold
18/04/15
ioco Theory
i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ)
Impl:
Spec:
δ
δ
1eur?
1eur?
ioco
1eur?
coffee!
1eur?,
δ
tea!
δ
coffee!
δ
out Impl (1eur?)={coffee!}⊆{tea! , coffee!}=out Spec (1eur?)
...
10
Marcus Gerhold
18/04/15
ioco Theory
i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ)
Spec:
Impl:
δ
1eur?
δ
not ioco
1eur?
1eur?
tea!
coffee!
coffee!
δ
1eur?, δ
1eur?, δ
out Impl (1eur? )={tea! ,coffee!}⊈{coffee!}=out Spec (1eur? )
11
Marcus Gerhold
18/04/15
ioco Theory
i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ)
Impl:
1eur?
Spec:
1eur?
1eur?
coffee!
1eur?,
δ
δ
δ
1eur?
tea!
1eur?,
δ
1eur?
ioco
tea!
δ
coffee!
δ
out Impl (1eur?)={coffee!}⊆{tea! , coffee!}=out Spec (1eur?)
...
12
Marcus Gerhold
18/04/15
Why probabilities?
done!
shuffle?
song_1!
●
Wanted
...
shuffle?
δ
song_n!
shuffle?
●
Existing MBT testing tools
(e.g. JtorX..) already choose
next steps probabilistically
13
Marcus Gerhold
18/04/15
pQTS and pioco
Definition (pQTS): A probabilistic quiescent transition system is a five tuple
A=⟨ S , s 0 , L I , LδO , Δ⟩
with
S
s0
LI
δ
LO
a finite set of states
the starting state
the input alphabet
the output alphabet + quiescence
Δ⊆{(s ,μ )∈S×Distr (L×S)}
Input reactive/output generative probabilistic transition relation
Different to: Segala (1995); Modeling and Verification of Randomized
Distributed Real-Time Systems
There:
Δ⊆{(s , a , μ)∈S ×L×Distr (S)}
14
Marcus Gerhold
18/04/15
pQTS and pioco
Δ⊆{(s ,μ )∈S×Distr (L×S)}
Such that for ( s ,μ )∈ Δ , we either have a distribution only containing the same
Input or a distribution containing only outputs and possibly quiescence
(=input reactive / output generative).
Distribution:
∑x∈ X μ (x )=1
μ : X →[0,1]
Examples:
1
3
a? a?
1
3
1
3
a?
b!
1
3
c!
1
3
1
3
δ
a?
1
2
1
2
c!
b!
15
Marcus Gerhold
18/04/15
pQTS and pioco
Δ⊆{(s ,μ )∈S×Distr (L×S)}
Such that for ( s ,μ )∈ Δ , we either have a distribution only containing the same
Input or a distribution containing only outputs and possibly quiescence
(=input reactive / output generative).
Distribution:
∑x∈ X μ (x )=1
μ : X →[0,1]
Examples:
1
3
a? d?
1
3
1
3
a?
b!
1
3
a?
1
3
1
3
1
2
δ
a?
1
2
b!
16
Marcus Gerhold
18/04/15
pQTS and pioco
reset
Example:
b!
δ
1
2
a?
1
2
1
a? 2
1
2
b!
a?
c!
1
2
c!
a?
1
2 a?
b!
1
7
c!
d!
6
7
But what is the probability to get the trace a?c! ?
17
Marcus Gerhold
18/04/15
pQTS and pioco
How do we resolve the remaining non-determinisms?
Definition (Adversary): An adversary E is a function
E : Path ( A)→Distr (Distr ( L×S )∪Halting)
such that for each finite path π, if E( π)(μ)> 0 then (last (π) ,μ )∈ Δ .
Definition (Path Probability): We define the function
Q E : Path →[0,1]
Inductively as follows:
E
Q (s0 )=1
Q E (πμ a s)=Q E (π) E (π)(μ )μ (a , s)
Instead of using adversaries (functions), we use probability spaces
associated to adversaries to get a probability measure.
18
Marcus Gerhold
18/04/15
pQTS and pioco
Example: Let E be the adversary
reset
a?
b!
δ
1
2
a?
1
1
a? 2
1
2
b!
c!
1
2
0
1
2
a?
1
2 a?
c!
b!
1
7
c!
d!
6
7
Q E (πμ a s)=Q E (π) E (π)(μ )μ (a , s)
3
4
Then the probability for trace a?c! equals .
Instead of using probability spaces associated to adversaries we use
probability spaces associated with trace distributions.
−1
P H (σ)=P E (trace (σ))
19
Marcus Gerhold
18/04/15
pQTS and pioco
Paths & Traces vs Adversaries & Trace Distributions
s
Actions after path s a?p ?
Actions after trace a? ?
a?
a?
p
t
b!
c!
s
p
b!
Actions with positive prob. after Adversary E
Actions with positive prob. after Trace Distribution H?
a?
0
a?
1
1
t
c!
20
Marcus Gerhold
18/04/15
pQTS and pioco
δ
1eur?
Ioco based on:
0
δ
out A (σ )=after (σ)∩LO
1
1eur?
1 coffee!
Pioco based on: k ∈ℕ∧ H ∈trd ( A , k)
k
outcont (H , A)={H ' ∈trd ( A , k +1) : H ⊆k H ' ∧∀ σ∈L L I : P H ' (σ)=0 }
Intuition:
Actions with positive prob. after Trace distribution H?
●
It prolongs a trace distribution by length 1
●
Only schedules output after length k
21
Marcus Gerhold
18/04/15
pQTS and pioco Theory
Definition (pioco): Let Imp be an implementation and
Spec be a specification. Furthermore let Imp be input
enabled, then
Imp ⊆pioco Spec ⇔ ∀ k ∈ℕ ∀ H ∈ trd (Spec , k ):
outcont (H , Imp)⊆outcont ( H , Spec)
Intuition: Imp pioco to Spec iff for all trace distributions H
of Spec:
If every behaviour of Impl can be imitated by Spec
Some examples
22
Marcus Gerhold
18/04/15
pQTS and pioco
Example: Take an adversary E of length 1 and a corresponding trace
distribution H
δ
1eur?
1
1eur?
1 coffee!
δ
1
pioco
tea!
0
1eur?
1 coffee!
∀ k ∈ℕ ∀ H ∈ trd(Spec , k ):
outcont (H , Imp)⊆outcont ( H , Spec)
23
Marcus Gerhold
18/04/15
pQTS and pioco
Example: Take an adversary E of length 1 and a corresponding trace
distribution H
δ
1eur?
1
tea! 2
1
1eur?
1
2
δ
Not pioco
coffee!
tea! ?
1
1eur?
1
2
coffee!
Not:
∀ k ∈ℕ ∀ H ∈ trd(Spec , k ):
outcont (H , Imp)⊆outcont ( H , Spec)
24
Marcus Gerhold
18/04/15
pQTS and pioco
So far we have only considered non-probabilistic QTS
Theorem: Let Imp and Spec be two QTS and Imp be inputenabled, then
Imp ⊆ioco Spec ⇔ Imp ⊆pioco Spec
pQTS
pioco
ioco
QTS
25
Marcus Gerhold
18/04/15
pQTS and pioco
Example: Take an adversary E of length 1 and a corresponding trace
distribution H
δ
1
1eur?
tea! 1
2
1eur?
δ
1
pioco
1
1
2
coffee!
tea!
1
2
1eur?
1
2
coffee!
∀ k ∈ℕ ∀ H ∈ trd(Spec , k ):
outcont (H , Imp)⊆outcont ( H , Spec)
26
Marcus Gerhold
18/04/15
pQTS and pioco
Example: Take an adversary E of length 1 and a corresponding trace
distribution H
δ
1
1eur?
tea!
1eur?
1
2
1
8
1
2
δ
1
Not pioco
coffee!
7
8
tea!
1
2
1eur?
1
1
2
coffee!
Not:
∀ k ∈ℕ ∀ H ∈ trd(Spec , k ):
outcont (H , Imp)⊆outcont ( H , Spec)
27
Marcus Gerhold
18/04/15
pQTS and pioco
Imp is always input enabled. What if Spec is too?
Theorem (known): Let Imp and Spec be two input enabled QTS, then
Imp ⊆ioco spec ⇔ Imp ⊆traces Spec
Theorem: Let Imp and Spec be two input enabled pQTS, then
Imp ⊆pioco spec ⇔ Imp ⊆TD Spec
28
Marcus Gerhold
18/04/15
pQTS and pioco
Theorem (Transitivity): Let A, B and C be three QTS and let A and B
be input-enabled, then
A ⊆ioco B and B ⊆ioco C ⇒ A ⊆ioco C
Theorem (Transitivity): Let A, B and C be three pQTS and let A and
B be input-enabled, then
A ⊆pioco B and B ⊆pioco C ⇒ A ⊆pioco C
29
Marcus Gerhold
18/04/15
Testing with pQTS
Verdict 1: System should not give any
erroneous output (Classical ioco testing).
Verdict 2: Output frequencies should be
matched as specified (Statistical testing).
30
Marcus Gerhold
18/04/15
Testing with pQTS
Verdict 1: Classical ioco testing
1. Given Imp and Spec
2. Derive test cases from Spec
3. Run test case and Imp in parallel
composition
4. If erroneous output is detected, fail
31
Marcus Gerhold
18/04/15
Testing with pQTS
Definition (test): A test is a pQTS of the form
t=⟨ S , s 0 , LO , L I ∪{δ}, Δ ⟩
such that t is acyclic, conntected and internally deterministic .
Example:
δ
shuffle!
song_n?
song_1?
…
pass
fail
δ
fail
song_n?
song_1?
…
fail
pass
pass
+ annotation
Summarize in annotated Test suite.
32
Marcus Gerhold
18/04/15
Testing with pQTS
Definition (Verdict Function 1): Given a test t, we define the verdict
function 1 v t : pQTS →{pass , fail} as
v 1 ( A )=pass if ∀ σ∈ traces( A||t)∩ctraces (t ):a (σ )=pass
v 1 ( A )=fail otherwise
Example:
t:
Impl:
song_1!
δ
δ
shuffle!
song_n?
song_1?
…
pass
shuffle?
fail
δ
fail
song_n?
song_1?
…
fail
pass
pass
Marcus Gerhold
33
18/04/15
Testing with pQTS
done!
shuffle?
p1
done!
song_1!
shuffle?
1
n
...
...
shuffle?
shuffle?
δ
song_1!
pn
song_n!
shuffle?
δ
1
n
song_n!
shuffle?
Marcus Gerhold
34
18/04/15
Testing with pQTS
Verdict 2: (Statistical Testing)
1. Given Imp and Spec
2. Derive (reuse) test cases from Spec
3. We run test case and Imp in parallel
composition
4. Hypothesis Testing and Sampling
35
Marcus Gerhold
18/04/15
Testing with pQTS
Statistical Testing
k m
O∈(L )
●
We call
●
Therefore assume m different trace distributions
●
Treat sample as Bernoulli experiment, with success in i if σ=σi
●
Define expected value of σ as E H⃗ (σ )= 1 ∑i=1 P H (σ)
●
●
●
a sample of depth k and width m
⃗ = H 1 ,…, H m
H
m
⃗
Then E H becomes a distribution
m
i
For given level of significance, are frequencies of O in some
acceptable distance of expected frequency?
Unify over all those balls, yields
Obs( A , α)
...
H⃗ 1
H⃗ d
H⃗ 2
36
Marcus Gerhold
18/04/15
pQTS and pioco
Theorem: Given level of significance α , we know that for all pQTS A
and B
Obs( A , α)⊆Obs (B , α)⇔ A ⊆TD B
M. Stoelinga, F. Vaandrager (2003); A Testing Scenario for probabilistic Automata
Definition (Verdict Function 2): Given a test t, a level of
m
significance α ϵ [0,1] and O∈(Traces(Impl , k )) we define the
α
verdict function 2 v t :pQTS →{pass , fail} as
α
v 2 ( A)=pass if O∈Obs (Spec )
α
v 2 ( A)=fail otherwise
37
Marcus Gerhold
18/04/15
Testing with pQTS
Definition (Overall Verdict): Given a test t, a level of
significance α ϵ [0,1] and O∈(Traces(Impl , k )) m, we define the
verdict function v : pQTS→{pass , fail } as
v ( A )=pass if v 1 =pass∧v α2 =pass
v ( A )=fail otherwise
done!
shuffle?
done!
p1 song_1!
shuffle?
1
n
...
...
shuffle?
δ
song_1!
shuffle?
pn
song_n!
shuffle?
δ
1
n
song_n!
shuffle?
38
Marcus Gerhold
18/04/15
Conclusion
●
Definition pQTS
●
Definition pioco
Future Work
●
Enable internal actions (pDQTS)
●
Prove soundness and (theoretical) completeness
●
Use more advanced statistical methods
●
Case studies
Thank you for your attention!
39
Marcus Gerhold
18/04/15