ioco Theory for Probabilistic Automata M. Gerhold, M.I.A. Stoelinga 18/04/15 10th MBT Workshop, London 2015 1 OUTLINE 1. ioco Theory 2. pQTS and pioco 3. Testing with pQTS 2 Marcus Gerhold 18/04/15 Ideally: Impl. conforms to Spec iff Impl. passes test suite Implementation ≡? Implementation passes tests 3 Marcus Gerhold 18/04/15 3/60 ioco Theory ioco = Input / Output Conformance Tretmans (1996) Test Generations with Inputs, Outputs, and Repetitive Quiescence Tretmans (2008) Model Based Testing with labeled transition Systems 4 Marcus Gerhold 18/04/15 ioco Theory ● ● ● ● ● ● ● hioco: Osch (2006); Hybrid input-output conformance and test generation mioco: Heerink (1998); Ins and Outs in Refusal Testing (r)tioco: Brandan Briones, Brinksma (2005); A test generation framework for quiescent real-time systems Krichen, Tripakis (2004); Black Box Conformance Testing for real-time systems Hessel, Larsen, Mikucionis, Nielsen, Petersson, Skou (2008); Online Testing of real-time systems using Uppaal sioco: Tretmans, Frantzen, Willemse (2004); A symbolic Framework for MBT uioco: van der Bijl, Rensink (2003); Compositional Testing with ioco Marcus Gerhold 5 18/04/15 ioco Theory ● ● ● Stokkink, Timmer, Stoelinga (2012); Talking Quiescence a rigorous theory that supports parallel composition, action hiding and determinisation Stokkink, Timmer, Stoelinga (2013); Divergent Quiescent Transition Systems Brinksma, Timmer, Stoelinga (2014); An Introduction to Model-Based Testing 6 Marcus Gerhold 18/04/15 ioco Theory Definition (QTS): A quiescent transition system is a five tuple δ δ O A=⟨ S , s 0 , L I , L ,→⟩ with S 1eur? LI a finite set of states the starting state the input alphabet → the output alphabet + quiescence the transition relation s0 δ LO δ coffee! tea! req_tea? req_coffee? Important: Quiescent output action δ 7 Marcus Gerhold 18/04/15 ioco Theory Definition (out): For a given QTS A and trace σ we write δ out A (σ )=after (σ)∩LO to denote the set of all output actions as well as quiescence after trace σ. 2eur?, δ ● out( ε ) = {δ} ● out( 1eur? ) = {coffee!} ● out( 1eur? 2eur? ) = {coffee!} ● out( 1eur? coffee! ) = {δ} ● out( 2eur? ) = {δ} ● out( δ ) = {δ} ● out( coffee ) = ∅ 1eur? 1eur?, 2eur? coffee! 1eur?, 2eur?, δ 8 Marcus Gerhold 18/04/15 ioco Theory Definition (ioco): Let Imp be an implementation and Spec a specification. Furthermore let Imp be input-enabled, then Imp ⊆ioco Spec ⇔ ∀ σ ∈traces (Spec):out Imp (σ)⊆out Spec (σ) Intuition: Imp ioco Spec iff for all traces of Spec: If Imp gives output x! After trace σ, then so can Spec Some examples 9 Marcus Gerhold 18/04/15 ioco Theory i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ) Impl: Spec: δ δ 1eur? 1eur? ioco 1eur? coffee! 1eur?, δ tea! δ coffee! δ out Impl (1eur?)={coffee!}⊆{tea! , coffee!}=out Spec (1eur?) ... 10 Marcus Gerhold 18/04/15 ioco Theory i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ) Spec: Impl: δ 1eur? δ not ioco 1eur? 1eur? tea! coffee! coffee! δ 1eur?, δ 1eur?, δ out Impl (1eur? )={tea! ,coffee!}⊈{coffee!}=out Spec (1eur? ) 11 Marcus Gerhold 18/04/15 ioco Theory i ioco s =def ∀ σ ∈traces (Spec): out Imp (σ)⊆out Spec (σ) Impl: 1eur? Spec: 1eur? 1eur? coffee! 1eur?, δ δ δ 1eur? tea! 1eur?, δ 1eur? ioco tea! δ coffee! δ out Impl (1eur?)={coffee!}⊆{tea! , coffee!}=out Spec (1eur?) ... 12 Marcus Gerhold 18/04/15 Why probabilities? done! shuffle? song_1! ● Wanted ... shuffle? δ song_n! shuffle? ● Existing MBT testing tools (e.g. JtorX..) already choose next steps probabilistically 13 Marcus Gerhold 18/04/15 pQTS and pioco Definition (pQTS): A probabilistic quiescent transition system is a five tuple A=⟨ S , s 0 , L I , LδO , Δ⟩ with S s0 LI δ LO a finite set of states the starting state the input alphabet the output alphabet + quiescence Δ⊆{(s ,μ )∈S×Distr (L×S)} Input reactive/output generative probabilistic transition relation Different to: Segala (1995); Modeling and Verification of Randomized Distributed Real-Time Systems There: Δ⊆{(s , a , μ)∈S ×L×Distr (S)} 14 Marcus Gerhold 18/04/15 pQTS and pioco Δ⊆{(s ,μ )∈S×Distr (L×S)} Such that for ( s ,μ )∈ Δ , we either have a distribution only containing the same Input or a distribution containing only outputs and possibly quiescence (=input reactive / output generative). Distribution: ∑x∈ X μ (x )=1 μ : X →[0,1] Examples: 1 3 a? a? 1 3 1 3 a? b! 1 3 c! 1 3 1 3 δ a? 1 2 1 2 c! b! 15 Marcus Gerhold 18/04/15 pQTS and pioco Δ⊆{(s ,μ )∈S×Distr (L×S)} Such that for ( s ,μ )∈ Δ , we either have a distribution only containing the same Input or a distribution containing only outputs and possibly quiescence (=input reactive / output generative). Distribution: ∑x∈ X μ (x )=1 μ : X →[0,1] Examples: 1 3 a? d? 1 3 1 3 a? b! 1 3 a? 1 3 1 3 1 2 δ a? 1 2 b! 16 Marcus Gerhold 18/04/15 pQTS and pioco reset Example: b! δ 1 2 a? 1 2 1 a? 2 1 2 b! a? c! 1 2 c! a? 1 2 a? b! 1 7 c! d! 6 7 But what is the probability to get the trace a?c! ? 17 Marcus Gerhold 18/04/15 pQTS and pioco How do we resolve the remaining non-determinisms? Definition (Adversary): An adversary E is a function E : Path ( A)→Distr (Distr ( L×S )∪Halting) such that for each finite path π, if E( π)(μ)> 0 then (last (π) ,μ )∈ Δ . Definition (Path Probability): We define the function Q E : Path →[0,1] Inductively as follows: E Q (s0 )=1 Q E (πμ a s)=Q E (π) E (π)(μ )μ (a , s) Instead of using adversaries (functions), we use probability spaces associated to adversaries to get a probability measure. 18 Marcus Gerhold 18/04/15 pQTS and pioco Example: Let E be the adversary reset a? b! δ 1 2 a? 1 1 a? 2 1 2 b! c! 1 2 0 1 2 a? 1 2 a? c! b! 1 7 c! d! 6 7 Q E (πμ a s)=Q E (π) E (π)(μ )μ (a , s) 3 4 Then the probability for trace a?c! equals . Instead of using probability spaces associated to adversaries we use probability spaces associated with trace distributions. −1 P H (σ)=P E (trace (σ)) 19 Marcus Gerhold 18/04/15 pQTS and pioco Paths & Traces vs Adversaries & Trace Distributions s Actions after path s a?p ? Actions after trace a? ? a? a? p t b! c! s p b! Actions with positive prob. after Adversary E Actions with positive prob. after Trace Distribution H? a? 0 a? 1 1 t c! 20 Marcus Gerhold 18/04/15 pQTS and pioco δ 1eur? Ioco based on: 0 δ out A (σ )=after (σ)∩LO 1 1eur? 1 coffee! Pioco based on: k ∈ℕ∧ H ∈trd ( A , k) k outcont (H , A)={H ' ∈trd ( A , k +1) : H ⊆k H ' ∧∀ σ∈L L I : P H ' (σ)=0 } Intuition: Actions with positive prob. after Trace distribution H? ● It prolongs a trace distribution by length 1 ● Only schedules output after length k 21 Marcus Gerhold 18/04/15 pQTS and pioco Theory Definition (pioco): Let Imp be an implementation and Spec be a specification. Furthermore let Imp be input enabled, then Imp ⊆pioco Spec ⇔ ∀ k ∈ℕ ∀ H ∈ trd (Spec , k ): outcont (H , Imp)⊆outcont ( H , Spec) Intuition: Imp pioco to Spec iff for all trace distributions H of Spec: If every behaviour of Impl can be imitated by Spec Some examples 22 Marcus Gerhold 18/04/15 pQTS and pioco Example: Take an adversary E of length 1 and a corresponding trace distribution H δ 1eur? 1 1eur? 1 coffee! δ 1 pioco tea! 0 1eur? 1 coffee! ∀ k ∈ℕ ∀ H ∈ trd(Spec , k ): outcont (H , Imp)⊆outcont ( H , Spec) 23 Marcus Gerhold 18/04/15 pQTS and pioco Example: Take an adversary E of length 1 and a corresponding trace distribution H δ 1eur? 1 tea! 2 1 1eur? 1 2 δ Not pioco coffee! tea! ? 1 1eur? 1 2 coffee! Not: ∀ k ∈ℕ ∀ H ∈ trd(Spec , k ): outcont (H , Imp)⊆outcont ( H , Spec) 24 Marcus Gerhold 18/04/15 pQTS and pioco So far we have only considered non-probabilistic QTS Theorem: Let Imp and Spec be two QTS and Imp be inputenabled, then Imp ⊆ioco Spec ⇔ Imp ⊆pioco Spec pQTS pioco ioco QTS 25 Marcus Gerhold 18/04/15 pQTS and pioco Example: Take an adversary E of length 1 and a corresponding trace distribution H δ 1 1eur? tea! 1 2 1eur? δ 1 pioco 1 1 2 coffee! tea! 1 2 1eur? 1 2 coffee! ∀ k ∈ℕ ∀ H ∈ trd(Spec , k ): outcont (H , Imp)⊆outcont ( H , Spec) 26 Marcus Gerhold 18/04/15 pQTS and pioco Example: Take an adversary E of length 1 and a corresponding trace distribution H δ 1 1eur? tea! 1eur? 1 2 1 8 1 2 δ 1 Not pioco coffee! 7 8 tea! 1 2 1eur? 1 1 2 coffee! Not: ∀ k ∈ℕ ∀ H ∈ trd(Spec , k ): outcont (H , Imp)⊆outcont ( H , Spec) 27 Marcus Gerhold 18/04/15 pQTS and pioco Imp is always input enabled. What if Spec is too? Theorem (known): Let Imp and Spec be two input enabled QTS, then Imp ⊆ioco spec ⇔ Imp ⊆traces Spec Theorem: Let Imp and Spec be two input enabled pQTS, then Imp ⊆pioco spec ⇔ Imp ⊆TD Spec 28 Marcus Gerhold 18/04/15 pQTS and pioco Theorem (Transitivity): Let A, B and C be three QTS and let A and B be input-enabled, then A ⊆ioco B and B ⊆ioco C ⇒ A ⊆ioco C Theorem (Transitivity): Let A, B and C be three pQTS and let A and B be input-enabled, then A ⊆pioco B and B ⊆pioco C ⇒ A ⊆pioco C 29 Marcus Gerhold 18/04/15 Testing with pQTS Verdict 1: System should not give any erroneous output (Classical ioco testing). Verdict 2: Output frequencies should be matched as specified (Statistical testing). 30 Marcus Gerhold 18/04/15 Testing with pQTS Verdict 1: Classical ioco testing 1. Given Imp and Spec 2. Derive test cases from Spec 3. Run test case and Imp in parallel composition 4. If erroneous output is detected, fail 31 Marcus Gerhold 18/04/15 Testing with pQTS Definition (test): A test is a pQTS of the form t=⟨ S , s 0 , LO , L I ∪{δ}, Δ ⟩ such that t is acyclic, conntected and internally deterministic . Example: δ shuffle! song_n? song_1? … pass fail δ fail song_n? song_1? … fail pass pass + annotation Summarize in annotated Test suite. 32 Marcus Gerhold 18/04/15 Testing with pQTS Definition (Verdict Function 1): Given a test t, we define the verdict function 1 v t : pQTS →{pass , fail} as v 1 ( A )=pass if ∀ σ∈ traces( A||t)∩ctraces (t ):a (σ )=pass v 1 ( A )=fail otherwise Example: t: Impl: song_1! δ δ shuffle! song_n? song_1? … pass shuffle? fail δ fail song_n? song_1? … fail pass pass Marcus Gerhold 33 18/04/15 Testing with pQTS done! shuffle? p1 done! song_1! shuffle? 1 n ... ... shuffle? shuffle? δ song_1! pn song_n! shuffle? δ 1 n song_n! shuffle? Marcus Gerhold 34 18/04/15 Testing with pQTS Verdict 2: (Statistical Testing) 1. Given Imp and Spec 2. Derive (reuse) test cases from Spec 3. We run test case and Imp in parallel composition 4. Hypothesis Testing and Sampling 35 Marcus Gerhold 18/04/15 Testing with pQTS Statistical Testing k m O∈(L ) ● We call ● Therefore assume m different trace distributions ● Treat sample as Bernoulli experiment, with success in i if σ=σi ● Define expected value of σ as E H⃗ (σ )= 1 ∑i=1 P H (σ) ● ● ● a sample of depth k and width m ⃗ = H 1 ,…, H m H m ⃗ Then E H becomes a distribution m i For given level of significance, are frequencies of O in some acceptable distance of expected frequency? Unify over all those balls, yields Obs( A , α) ... H⃗ 1 H⃗ d H⃗ 2 36 Marcus Gerhold 18/04/15 pQTS and pioco Theorem: Given level of significance α , we know that for all pQTS A and B Obs( A , α)⊆Obs (B , α)⇔ A ⊆TD B M. Stoelinga, F. Vaandrager (2003); A Testing Scenario for probabilistic Automata Definition (Verdict Function 2): Given a test t, a level of m significance α ϵ [0,1] and O∈(Traces(Impl , k )) we define the α verdict function 2 v t :pQTS →{pass , fail} as α v 2 ( A)=pass if O∈Obs (Spec ) α v 2 ( A)=fail otherwise 37 Marcus Gerhold 18/04/15 Testing with pQTS Definition (Overall Verdict): Given a test t, a level of significance α ϵ [0,1] and O∈(Traces(Impl , k )) m, we define the verdict function v : pQTS→{pass , fail } as v ( A )=pass if v 1 =pass∧v α2 =pass v ( A )=fail otherwise done! shuffle? done! p1 song_1! shuffle? 1 n ... ... shuffle? δ song_1! shuffle? pn song_n! shuffle? δ 1 n song_n! shuffle? 38 Marcus Gerhold 18/04/15 Conclusion ● Definition pQTS ● Definition pioco Future Work ● Enable internal actions (pDQTS) ● Prove soundness and (theoretical) completeness ● Use more advanced statistical methods ● Case studies Thank you for your attention! 39 Marcus Gerhold 18/04/15
© Copyright 2024