1. No category

Linköping Studies in Science and Technology
Licentiate Thesis No. 1582
Mission Experience: How to Model and Capture it to
Enable Vicarious Learning
by
Dennis Andersson
Department of Computer and Information Science
Linköpings universitet
SE-581 83 Linköping, Sweden
Linköping 2013
This is a Swedish Licentiate´s Thesis
Swedish postgraduate education leads to a Doctor´s degree and/or a Licentiate´s degree.
A Doctor´s degree comprises 240 ECTS credits (4 years of full-time studies).
A Licentiate´s degree comprises 120 ECTS credits.
Copyright © 2013 Dennis Andersson
ISBN 978-91-7519-660-2
ISSN 0280-7971
Printed by LiU-Tryck 2013
URL: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-90727
Mission Experience: How to Model and Capture it to
Enable Vicarious Learning
by
Dennis Andersson
May 2013
ISBN 978-91-7519-660-2
Linköping Studies in Science and Technology
Licentiate Thesis No. 1582
ISSN 0280-7971
LiU-Tek-Lic-2013:16
ABSTRACT
Organizations for humanitarian assistance, disaster response and military activities are
characterized by their special role in society to resolve time-constrained and potentially lifethreatening situations. The tactical missions that these organizations conduct regularly are
significantly dynamic in character, and sometimes impossible to fully comprehend and
predict. In these situations, when control becomes opportunistic, the organizations are forced
to rely on the collective experience of their personnel to respond effectively to the unfolding
threats. Generating such experience through traditional means of training, exercising and
apprenticeship, is expensive, time-consuming, and difficult to manage.
This thesis explores how and why mission experience should be utilized in emergency
management and military organizations to improve performance. A multimedia approach for
capturing mission experience has further been tested in two case studies to determine how the
commanders’ experiences can be externalized to enable vicarious learning. These studies
propose a set of technical, methodological, and ethical issues that need to be considered when
externalizing mission experience, based on two aforementioned case studies complemented
by a literature review. The presented outcomes are (1) a model aligning abilities that tactical
organizations need when responding to dynamic situations of different familiarity, (2) a
review of the usefulness of several different data sources for externalization of commanders’
experiences from tactical operations, and (3) a review of methodological, technical, and
ethical issues to consider when externalizing tactical military and emergency management
operations. The results presented in this thesis indicate that multimedia approaches for
capturing mission histories can indeed complement training and exercising as a method for
generating valuable experience from tactical missions.
This work has been supported by the Swedish Defense Research Agency, the Swedish Armed
Forces, the Swedish Civil Contingencies Agency (formerly the Swedish Rescue Services
Agency and the Swedish Emergency Management Agency) and Forum Securitatis.
Department of Computer and Information Science
Linköpings universitet
SE-581 83 Linköping, Sweden
ACKNOWLEDGMENTS
This thesis is meant to summarize several years of research, a journey that has given me
many encounters with great and intelligent people, generating invaluable insights to assist
my research. To all those people, some of whom are mentioned below, I would like to
send my deepest thanks, and a message that I hope for the future to hold many more
encounters of this kind, helping me become a better researcher and person.
This work would never have been conducted if it were not for my great supervisor, Prof.
Niklas Hallberg, at the Swedish Defense Research Agency, who has always fought hard
to make way for me as a researcher. His coaching has been beyond comparison and I
cannot enough thank him for these efforts.
On the coaching side, my secondary advisor Associate Professor Sofie Pilemalm at the
Department of Management and Engineering at Linköping University has always given
me great support and fantastic advice when I needed it the most, quickly sweeping away
my doubts and concerns.
Advice has also poured in from Professor Henrik Eriksson at the Department of
Computer and Information Science at Linköping University, who has been a great tutor in
writing and a wizard of flexible solutions. Without his help I would never have been able
to navigate through the academic jungle.
My fourth advisor, Associate Professor Alexander Bordetsky at the Department of
Information Science at the Naval Postgraduate School, took me in for six months in
Monterey, California, to learn the trades of American IS research. His mentoring has
broadened my view on research and rewarded me with many pieces to the puzzle that I
am trying to solve.
In addition to my advisors mentioned above, I have received a lot of help and support
from colleagues and friends worldwide. I would like to dedicate a special thank you to
my colleagues Johan Allgurén and Mirko Thorstensson who took me in to join the MIND
research group as a newly graduated student, and introduced me to the beautiful world of
research. They have always believed in me, and fed me with interesting challenges and
tasks that motivate my research. Equally motivating and stimulating has been the
recurring discussions with Pär-Anders Albinsson, whose scientific mindset and constant
thoroughness serves as a role-model to me. Their advice has often helped me to reduce
the risk of cutting scientific corners too fast.
For the particular studies presented in this thesis, I would like to dedicate an extra thank
you to my hard-working co-authors, aforementioned Sofie Pilemalm and Niklas Hallberg,
as well as PhD student Amy Rankin at the Department of Computer and Information
Science at Linköping University. An extra thank you also to our dedicated librarian
Barbro Löwemo who has assisted my literature reviews with superb professionalism.
An early draft of this thesis was reviewed by Dr. Jiri Trnka, who gave me a lot of advice
on how to finalize my work. A later draft was then carefully reviewed by my colleague
Teodor Sommestad who contributed tremendously to this thesis with his constructive
advice and recommendations, improving both quality and readability. I would also like to
[i]
direct my thanks to Pär Hjalmewik who has helped eliminate a set of grammatical errors
in this thesis.
A significant amount of support has been given to my research efforts by the Forum
Securitatis graduate school, initiated by Professor Martin Holmberg and now supervised
by Professor Peter Stenumgaard. It is a privilege to participate in this research collective
and reap the benefits of collaboration and organization. Further support to this work has
been given by the Swedish Defense Research Agency, the Swedish Rescue Services
Agency1, the Swedish Emergency Management Agency2 and the Swedish Armed Forces.
In the last stages of putting this work together, Anne Moe, Coordinator of Research
Studies Administration at the Department of Computer and Information Science at
Linköping University, made this thesis become reality with valuable advice and guidance
that greatly reduced the administrative workload of being a PhD student.
Finally, it is with great pleasure that I dedicate this thesis to my beloved Magdalena and
our emerging family. I have enjoyed limitless support from Magdalena on both personal
and professional level. I truly look forward to spending a life-time building a better future
with you, Emanuel and Ossian.
To all of you mentioned above, to my family and friends who always support me in
everything I do, and to everyone contributing to my professional or personal life, thank
you!
Slycke, April 2013
Dennis Andersson
1
Statens Räddningsverk. Seized to exist in 2009, replaced by the Swedish Civil Contingencies Agency
(Myndigheten för samhällsskydd och beredskap).
2
Krisberedskapsmyndigheten. Seized to exist in 2009, replaced by the Swedish Civil Contingencies
Agency (Myndigheten för samhällsskydd och beredskap).
[ii]
LIST OF THESIS PUBLICATIONS
This thesis is primarily based on the following publications, which are appended to the
end of the thesis:
Paper I: Andersson, D., & Rankin, A. (2012). Sharing Mission Experience in Tactical
Organisations. Proceedings of the 9th International ISCRAM Conference. Vancouver,
Canada: April 22-25.
Paper II: Andersson D., Pilemalm S., & Hallberg N. (2008) Evaluation of Crisis
Management Operations using Reconstruction and Exploration. Proceedings of the 5th
International ISCRAM Conference, Washington, DC: May 4-7 (pp. 118-125).
Paper III: Pilemalm, S., Andersson, D., & Hallberg, N. (2008). Reconstruction and
Exploration of Large-scale Distributed Operations – Multimedia tools for Evaluation of
Emergency Management Response. Journal of Emergency Management, 6(4), pp. 31-47.
Published by Weston Medical Publishing, LLC.
Paper IV: Andersson, D. (2011). Privacy and Distributed Tactical Operations Evaluation.
Proceedings of the 4th International Conference on Advances in Human-oriented and
Personalized Mechanisms, Technologies, and Services (CENTRIC 2011). Barcelona,
Spain: October 23-29.
For a complete list of references to academic publications by the author to date, see pages
71-72.
[iii]
[This page is intentionally left blank]
[iv]
CONTENTS
Chapter 1: Introduction ..................................................................................................... 1
1.1
Objective and Research questions ........................................................................ 2
1.2
Contributions ........................................................................................................ 2
1.3
Assumptions and Limitations ............................................................................... 3
1.4
Outline .................................................................................................................. 3
1.5
Reading directions ................................................................................................ 3
Chapter 2: Background ..................................................................................................... 5
2.1
Tactical Organizations.......................................................................................... 5
2.2
Command and control theories ............................................................................ 5
2.3
Decision-making models ...................................................................................... 9
2.4
Mission Experience ............................................................................................ 12
2.5
Reconstruction & Exploration ............................................................................ 13
2.6
Knowledge Transfer ........................................................................................... 19
2.7
Workplace ethics ................................................................................................ 22
2.8
Research gap ...................................................................................................... 23
Chapter 3: Methods ......................................................................................................... 25
3.1
Study design ....................................................................................................... 25
3.2
Case studies ........................................................................................................ 25
3.3
Literature review ................................................................................................ 27
Chapter 4: Summary of appended papers ....................................................................... 31
4.1
Sharing mission experience in tactical organisations ......................................... 31
4.2 Evaluation of crisis management operations using Reconstruction and
Exploration.................................................................................................................... 34
4.3
Reconstruction and Exploration of large-scale distributed tactical operations .. 37
4.4
Privacy and Distributed Tactical Operations Evaluation ................................... 40
[v]
Chapter 5: Results ........................................................................................................... 43
5.1 What abilities are important for commanders, to cope with dynamic decisionmaking situations that arise in tactical military and emergency management
operations? .................................................................................................................... 43
5.2 How should commanders’ mission experience be externalized in order to enable
vicarious learning from tactical military and emergency management operations? ..... 45
5.3 What technical, methodological, and ethical issues need to be considered when
externalizing mission experience from tactical military and emergency management
commanders? ................................................................................................................ 47
Chapter 6: Discussion ..................................................................................................... 49
6.1
Tactical command and control in dynamic situations ........................................ 49
6.2
Transferal of mission experience ....................................................................... 49
6.3
Reconstruction and Exploration ......................................................................... 50
6.4
Privacy................................................................................................................ 51
6.5
Summary ............................................................................................................ 52
6.6
Future work ........................................................................................................ 52
6.7
Conclusions ........................................................................................................ 52
References ......................................................................................................................... 55
List of Figures ................................................................................................................... 67
List of Tables .................................................................................................................... 69
Author’s publications ........................................................................................................ 71
Paper I: Sharing Mission Experience in Tactical Organisations ..................................... 73
Paper II: Evaluation of crisis management operations using Reconstruction and
Exploration........................................................................................................................ 87
Paper III: Reconstruction & Exploration of Large-Scale Distributed Tactical Operations
– Multimedia tools for evaluation of emergency management response ......................... 99
Paper IV: Privacy and Distributed Tactical Operations Evaluation .............................. 123
[vi]
Chapter 1
INTRODUCTION
Crisis management and military organizations are regularly exposed to complex, risky
and sometime dangerous situations where difficult decisions can be the difference
between life and death. With such extreme decision-making contexts, society benefits
from any improvement possible in the decision-making process. This research deals with
the acquisition and transfer of mission experience that may benefit the commanders’
ability of making decisions in time-constrained situations.
Humanitarian aid, disaster relief, and military organizations rely heavily on their
personnel to conduct and observe missions and exercises to acquire the necessary
knowledge for their work. In other words, these organizations need experienced
personnel (King, 2005). This statement is consistent with the more general idea of
naturalistic decision making which states that decisions are often based on recognition
and mapping of current situations to prior knowledge rather than through careful analysis
of current alternatives as the rationalistic decision making models suggest (Klein, 1993;
Howard, 1966).
Tactical command and control (C2) of operations involve many decision points for
commanders (Alberts & Hayes, 2006, p.8). These decisions can sometimes be urgent and
require immediate actions (ibid., p. 82) limiting the commander’s ability to carefully
analyze each option and therefore to rely on recognition primed decision making or
experience to respond in the best way (Klein, 1993).
As experience seems to be an important factor in tactical decision-making, understanding
how to acquire and utilize it plays a central role in development of tactical organizations.
Experience can, from an organization management point of view, be defined as “what
transpires in an organization as it performs its tasks” (Argote & Miron-Spektor, 2011).
Making mission experience explicit, i.e., externalization, is a major challenge for tactical
organizations (Nonaka, 1991; King, 2005). Synthesis is also needed along with
compilation, storage and organization of accumulated data, information and knowledge,
to enable retrieval and dissemination using effective methods and systems (Nonaka,
1991).
Composite knowledge, such as mission experience, has both tacit and explicit
components. Although Polanyi (1966) claimed that they are not meaningful to separate in
practice, conceptually this separation allows theorization about the interaction between
the two to stimulate knowledge creation (Nonaka, 1991). Explicit knowledge can be
stored and shared within and between organizations, as it can be articulated, while tacit
knowledge requires socialization or internalization processes (ibid.). Today, most tactical
organizations rely on within-organization socialization and internalization of
standardized routines, while knowledge transfer between organizations is rare (King,
2005). There is thus room and potential for new and improved methods of knowledge
sharing, both within and between organizations.
[1]
This research is motivated by the challenges proposed above, and addresses methods of
synthesizing, compiling and storing data and information pertaining to the experiences
acquired from conducting tactical military and emergency management operations and
exercises.
1.1 OBJECTIVE AND RESEARCH QUESTIONS
The objective of this thesis is to explore approaches of sharing mission experience from
the perspective of C2. This by addressing the following research questions (RQ):
RQ 1. What abilities are important for commanders, to cope
with dynamic decision-making situations that arise in tactical
military and emergency management operations?
RQ 2. How should commanders’ mission experience be
externalized in order to enable vicarious learning from
tactical military and emergency management operations?
RQ 3. What technical, methodological, and ethical issues need
to be considered when externalizing mission experience from
tactical military and emergency management commanders?
This thesis is based on four scholarly research articles, appended as Papers I-IV. Each of
these articles represents an individual study, which contribute to answering the three
RQs.
1.2 CONTRIBUTIONS
By answering the RQs defined above, this thesis contributes to research on knowledge
transfer for tactical teams. The following contributions are yielded from the process of
answering the research questions above:
 A model to describe, and separate, abilities needed to cope with dynamic
situations such as tactical military and emergency management operations.
 An approach for collecting and externalizing mission histories by recording C2
activities through reconstruction.
 A review of usefulness of different data sources for the externalization of mission
experience.
 A review of technical, methodological, and ethical issues that need to be
considered when externalizing tactical military and emergency management
operations.
In addition, mission histories, as externalized through the reconstruction & exploration
approach presented in this thesis, can serve an immediate purpose to support after-action
reviews, hot wash, debriefings and disseminations after missions, as well to act as a
knowledge base for (1) future decision making, (2) training and education of
commanders in military and civil service organizations, and (3) sharing knowledge within
and between such organizations.
[2]
1.3 ASSUMPTIONS AND LIMITATIONS
Although an organization can have a wide scope of tasks, this study is limited to tactical
missions conducted by emergency response and military organizations. The approach
proposed in this research was initially developed for post-mission analysis and afteraction reviews, a purpose for which mission history reconstruction has been applied since
the 1990’s (Worm, Jenvald & Morin, 1998). The proposed model adds another dimension
to mission reconstruction by focusing explicitly on experience transfer. Verification of
the proposed methodology, however, is out of scope of this thesis.
The technical, methodological and ethical issues that are explored in this thesis were
originally studied from an evaluation point of view, a somewhat different purpose than
that of knowledge transfer. Whether the same issues apply in both contexts or not has not
been investigated. It should also be noted that these issues are fetched from two case
studies performed at Swedish organizations and a literature review primarily based on
ethics literature from western culture. Cultural differences, e.g., in organization
management, laws and ethics, may affect the validity of this study if applied elsewhere as
cultural aspects have not been considered while performing the studies presented herein.
1.4 OUTLINE
This chapter, Chapter 1, describes the motivation and scoping of this thesis by defining
research questions, scientific contribution and limitations of the research. Chapter 2
describes the related background and theory that the thesis is based upon.
The third chapter briefly describes the methods used in each of the four studies, while
Chapter 4 summarizes the studies in relation to the RQs. Chapter 5 presents the results,
i.e., the outcomes of the four studies, which are then elaborated upon, discussed, and
concluded in Chapter 6.
1.5 READING DIRECTIONS
The suggested reading directions differ depending on the reader’s interest and familiarity
of the topics discussed in this thesis. Novice readers who want a brief introduction to
research on adjacent and backing theories that the rest of the work relies upon are advised
to start by reading Chapter 2. For better understanding of the work that the thesis is based
upon, Chapters 3-4 describe the research methods and summarizes each underlying study,
while the appended Papers I-IV provide the full story that should then be followed by
Chapters 5-6. Anyone interested in just a brief summary of the work presented in this
thesis should skip Chapters 3-4, and concentrate on Chapters 5-6, possibly adding parts
of Chapter 4 for a motivation of the presented ideas where appropriate.
[3]
[This page is intentionally left blank]
[4]
Chapter 2
BACKGROUND
This chapter, Background, presents research on (1) tactical organizations such as military
and emergency management organizations, (2) tactical command and control theories on
how teams operate in tactical situations and thus give hints on how to study C2, (3)
decision-making models providing different viewpoints on the cognition behind decision
making, (4) mission experience on defining what mission experience really means, (5)
reconstruction and exploration as a methodology for evaluation of operations of the kind
that tactical military and emergency management units typically conduct, (6) knowledge
transfer on theories from the knowledge management field on how knowledge can be
transferred, and (7) workplace ethics including definitions relevant to studying the
concept of privacy in a workplace setting. Finally, a summary of the identified research
gap is outlined.
2.1 TACTICAL ORGANIZATIONS
The term tactical organization is used throughout this document to denominate all
organizations, civil and military alike, designed to perform field missions requiring
tactical command and control. Tactical organizations typically operate in complex and
dynamic environments and include both military organizations, humanitarian aid/disaster
relief organizations (HA/DR) and emergency services; with examples ranging from
national armed forces, the International Red Cross/Crescent and UNESCO to local fire
brigades, police forces and health-care departments.
Tactical organizations can sometimes be described as adhocracies, ad-hoc operative
organizations that rely on specialized teams to perform their respective tasks (Mintzberg,
1979). An adhocracy is further described as a contemporary collection of teams that
“operate in an environment that is both dynamic and complex, demanding innovation of a
fairly sophisticated nature” (Mintzberg & McHugh, 1985, p.160). A tactical organization
can thus be seen as a special case of an adhocracy, one that specifically deals with tactical
operations in complex and dynamic environments. The adhocracy analogy yields access
to a body of knowledge on how these organizations function, what their operational
constraints and requirements are, and how they affect what experience needs to be
modeled and captured to support dissemination and knowledge transfer.
2.2 COMMAND AND CONTROL THEORIES
Command and control (C2) is “the exercise of authority and direction by a properly
designated commander over assigned forces in the accomplishment of the mission. C2
functions are performed through an arrangement of personnel, equipment,
communications, facilities, and procedures which are employed by commanders in
planning, directing, coordinating, and controlling forces and operations in the
accomplishment of the mission.” (Builder, Bankes & Nordin, 1999, p. 11). Different
models have been proposed in various attempts to better understand and improve the C2
capability, such as the C2 systems functional model (Finley, Muckler, Gainer &
[5]
Obermayer, 1975), Lawson’s process model (Lawson, 1981), SHOR (Wohl, 1981),
OODA (Boyd, 1987), Joint Cognitive Systems (Hollnagel, 1993), Dynamic OODA
(Brehmer, 2005), M-OODA, T-OODA, and C-OODA (Breton & Rousseau, 2008). More
information on the history of, and relationships between, different C2 models and theories
can be found in Builder et al.’s (1999) extensive review.
2.2.1 THE OODA LOOP
Many C2 doctrines today are inspired by the OODA-loop (Boyd, 1987). The OODA loop
has four components: Observe, Orient, Decide and Act, which form a cycle that the
commander must iterate again and again. Boyd’s original claim was that the ability to
complete an iteration of the loop quicker than an opponent gives an edge in battle
(Brehmer, 2005). Attempts have since been made to generalize the model and apply it to
new domains, as exemplified by Boyd himself in the more general, although no longer
loop, version of OODA, see Figure 2.1.
Figure 2.1. The extended OODA loop (Boyd, 1987). Reproduced from
http://dnipogo.org/john-r-boyd.
The popularity of the OODA loop has led to several spin-offs that attempt to model C2
from specific viewpoints, such as teamwork, cognition, and control, with the belief that
models based on OODA will stand a better chance of reaching acceptance in the military
community due to its familiarity within the community (Breton & Rousseau, 2008, p.
28). The M-OODA loop is a modular version of OODA that allows extensions such as
for cognition or teamwork, see Figure 2.2 (ibid.).
[6]
Dynamic and complex
situations
OODA Loop
Team DM
T-OODA Loop
M-OODA Loop
Cognitition of C2
decision cycle
C-OODA Loop
Figure 2.2. M-OODA enables further extensions such as T-OODA or C-OODA
depending on the modeling needs (after Breton and Rousseau, 2008).
2.2.2 DYNAMIC OODA
Brehmer (2005) stated that the OODA loop should not be seen as a model of C2, but
rather as an explanation of why American fighter pilots were performing better than their
opponents during the Korean War, i.e., by iterating the loop quicker. To facilitate the
contextual elements such as mission, goal, command concept and other activities,
Brehmer took a different path than Breton and Rousseau to introduce the Dynamic
OODA loop (DOODA), see Figure 2.3. The DOODA loop introduces cybernetic inputs
and outputs to the system as well as an entry and exit condition which, according to
Brehmer, was lacking in Boyd’s OODA loop (ibid.).
Figure 2.3. Brehmer’s Dynamic OODA loop (adapted by Oosthuizen & Roodt, 2012).
Copyright © 2012 CSIR South Africa. Reprinted with permission.
Brehmer’s intention with the Dynamic OODA loop was to create a unified model of
military command that is applicable on all command levels, albeit still with a single
commander in focus just as Boyd had. On the tactical level, this model highlights the
[7]
importance of adapting to the changing environment, i.e., being flexible. The reliance on
flexibility and adaptability is nothing new, but can be dated back at least to the 19th
century when the Prussian army recognized a need for modernization after being defeated
by the Napoleonic forces. They developed mission command (auftragstaktik) allowing
their subordinates to be more flexible and gain initiative and momentum over opponents
who employed traditional authorative command (befehlstaktik) (Widder, 2002). This
command method has been doctrine in the German forces ever since, and is now widely
spread over the world, it is for example still used in the latest version of the Swedish
Military Strategic Doctrine (Försvarsmakten, 2011).
2.2.3 JOINT COGNITIVE SYSTEMS
A tactical unit, comprised of technological systems, C2 systems and support systems, can
be described as a Joint Cognitive System (JCS) (Hollnagel, 1993). In tactical HA/DR
operations, multiple organizations are working toward the same goal under different
command, sometimes leading to uncertainties and ambiguities regarding responsibilities
and authorities (Morin, 2001). JCS models frequently and deliberately focus on the
control component, whereas commanding is more central in the models of C2 discussed
above.
The military command concept is in cognitive systems engineering sometimes referred to
as direct, e.g., in the Contextual Control Model (COCOM) (Hollnagel, 1993) and the
subsequent Extended Control Model (ECOM) (Hollnagel & Woods, 2005). Both these
models are based on the premise that no procedural patterns exist and that actions are
instead determined by the context and the competence of the JCS (ibid.). This reasoning
is similar to the ideas behind military mission command and Brehmer’s DOODA loop, as
they too highlight the importance of adaptivity and flexibility.
The minimalistic COCOM model is intended to be applicable on both the individual
level, for tactical organizations and in the strategic domain (ibid.). It has only three
constituents: (1) competence which represents the set of possible actions and responses
that a JCS can apply to a situation, (2) control refers to the way in which the competence
is applied, and (3) constructs are the (internal) descriptions of the situation that the JCS
employ to evaluate an event and select appropriate actions (ibid.).
The control in COCOM can take one of four different modes: (1) scrambled, which is
basically random selection without reflection, (2) opportunistic, where the next action is
determined by salient features of the current state with limited planning, (3) tactical,
where short-term control more or less follows known procedures and rules, and (4)
strategic, which has a longer time horizon than tactical and may correspond to higherlevel goals (ibid.).
A tactical unit can work in all four control modes, of which the strategic is preferable
(ibid.). Typical for a high-risk dynamic situation is that time does not allow the unit to
operate in strategic mode. Instead, the unit turns to tactical mode, or even opportunistic if
competence is inadequate for tactical (Svensson, 2010). The scrambled mode is the least
preferred and subject to deficient situation assessment.
[8]
2.3 DECISION-MAKING MODELS
Sensing, understanding, decision making and action performance are all important in
most models of tactical C2 (including the JCS COCOM model). Hence, there is an
apparent need to understand these four concepts in order to understand tactical missions.
2.3.1 SENSING
Boyd (1987) used the word observe to describe sensing, Brehmer (2005) instead chose
the term information collection. Although the terminology can be seen as nuances, they
do have one important difference namely that information collection has an active tense
whereas observation is rather passive. Brehmer thus included active searching for
information in his sensing concept. Regardless of whether it is called observation or
information collection, the products of these processes are put together to create a mental
model of the current situation, i.e., situation understanding.
2.3.2 SITUATION UNDERSTANDING
Situation understanding, or sensemaking, is an art of finding contextual rationality from
vague or chaotic pieces of information (Weick, 1993). Decision-making research has
seen a plethora of definitions of sensemaking, e.g.:
 “the process of placing stimuli into some kind of framework” (Waterman, 1990,
p.41).
 “a motivated, continuous effort to understand connections (which can be among
people, places, and events) in order to anticipate their trajectories and act
effectively” (Klein, Moor & Hoffman, 2006a, p.71).
Waterman’s definition is close to the Orient part of Boyd’s OODA loop, i.e., interpreting
the environment and framing it by personal context such as previous knowledge. Weick
(1995) stated similarly that sensemaking is simply about “making sense, and about
structuring the unknown”.
Klein and colleagues created a model of sensemaking based on the ground that it is not
adequately understood, and that there is a need for a framework to discuss it. Their main
critique was that it is often incorrectly understood as a single step in a linear sequential
decision making process, i.e., that sensemaking leads directly to decisions. Instead they
argued that sensemaking is a continuous process without a clearly defined beginning and
end that may influence decision and actions (Klein, Moor & Hoffman, 2006b). They
proposed a framework using a data/frame theory as shown in Figure 2.4. The theory says
that data, or stimuli, is interpreted by the sensemaker, and then recognized and put into
some kind of reference frame. These frames are then elaborated as more and more data
becomes available. Continuous conscious or subconscious questioning of said frames
help reframing and making sense by coming to better and more refined conclusions
(ibid.). A frame is thus a mental model, or construction, of a sensemaker, i.e., a construct
in a JCS.
[9]
Figure 2.4. A frames-based model of sensemaking (Klein et al., 2006b). Copyright ©
2006 IEEE. Reprinted with permission.
Endsley (1995) proposed a model of situation awareness (SA) that models three levels of
awareness: (1) perception, (2) understanding, and (3) projection, see Figure 2.5.
Perception, or level 1 SA, is the state in which a JCS stands when passively observing, or
not being able to comprehend, an ongoing situation. The only control modes available for
a JCS in this level are scrambled or opportunistic. In order for a JCS to open up the
tactical control mode, the situation must be understood (level 2 SA) so that the proper
routines can be implemented. Level 3 SA, or projection, is the desired state, which can be
reached through a rigorous process of sensemaking (Starbuck & Milliken, 1988). It is
only through level 3 SA that the strategic control mode can be utilized.
Sensemaking can thus be summarized as a retrospective process of interpreting stimuli
that can improve a decision maker’s SA (Endsley, 2004). In strategic operations the time
span may be long enough to allow careful analysis and rational reasoning to deduce the
best route forward. However, in dynamic tactical environments the situation may require
quick and reactive response.
[10]
Figure 2.5. Situation awareness (Endsley 1995). Copyright (1995) Human Factors and
Ergonomics Society. Reprinted with permission.
2.3.3 DECISION MAKING
Rational decision making has seen several different angles and is by many considered to
be a superior decision-making strategy. The main idea is that decisions are made by
applying knowledge and biases to select what is perceived to be the best among multiple
alternatives (e.g., Tversky & Kahneman, 1981). In dynamic situations like military
operations or emergency response, optimal or rational decisions cannot always be made,
e.g., when the problem space grows too complex to explore fully, or the time horizon
narrows too much (Fredholm & Åström, 2010). Instead, decision making then turns into a
function for gaining control (Brehmer, 1992) and the JCS falls into opportunistic control
mode because it is lacking in time or competence to select an appropriate plan (Hollnagel
& Woods, 2005). In these cases, commanders tend to resort to experience, habits or
association instead of rational analytics (ibid.). Klein (1993) labeled this recognitionprimed decision making (RPDM). The recognition-primed decision making model, see
Figure 2.6, is highly compatible with Klein et al.’s sensemaking framework (Klein et al.,
2006b) as they both build on the idea that the target interpret cues and assess a situation
based on personal experience.
[11]
Experience the situation
in a changing context
•
•
Reassess situation
Seek more information
No
Is the situation
familiar?
Yes
Yes
Are expectancies
violated?
Activation of information
from memory
No
•
•
•
•
Plausible goals
Expectancies
Relevant cues
Actions 1..N
Mental simulation
of action(n)
Yes, but
Modify
Will it work?
No
Yes
Implement
Figure 2.6. Recognition-primed decision making in complex, and changing contexts
(after Klein, 1993).
2.4 MISSION EXPERIENCE
In RPDM, experience is a central factor. In fact, Klein (1995) claimed that lack of
experience may lead to poor sense of typicality, poor ability to see patterns and make
discriminations, no basis for expectancies and missing causal knowledge. This in turn
may lead to failure to recognize anomalies, misjudging urgencies, misinterpreted
situations and unawareness of weaknesses in a selected course of action (ibid.). These
circumstances are what Hollnagel and Woods (2005) referred to as lack of competence,
which forces the tactical unit to employ opportunistic control at best.
Mission experience is what transpires in organizations as they perform missions (Argote
& Miron-Spektor, 2011). It can be acquired primarily through hands-on practice (King,
2005). Events that are particularly infrequent or unforeseen are extra difficult to gain
expertise in controlling. Organizations that are required to deal with these must therefore
rely on finding ways to do so without the assistance of standard operating procedures,
i.e., improve their ability to perform opportunistic control, e.g., through adaptation of
known procedures or improvisation (Mendonça, Beroggi & Wallace, 2001; Zumel,
Franco & Beutler, 2008; Trnka, Rankin, Jungert, Lundberg, Granlund, Granlund &
Johansson, 2009; Rankin, Dahlbäck & Lundberg, 2013).
[12]
2.5 RECONSTRUCTION & EXPLORATION
Evaluation of operations is a key enabler for tactical organizations to better understand
their operations and evolve. Both real and exercised operations are candidates for
evaluation, and characterized by the multiple sources of complexity involved. With
multiple ongoing tasks at different sites, extensive dynamism and complex cause-andeffect relationships; maintaining situation awareness of the operations is difficult. Extra
complexity is added with the unexpected events and actions taken over time (Brehmer
1992; Albinsson, Wong, Pilemalm & Morin, 2005; Alberts & Hayes, 2007; Andersson,
2009).
In large-scale complex multi-agency operations, keeping track of cause and effects may
be cumbersome for an analyst. To support the analyst, Reconstruction and Exploration
(R&E) has been suggested as an evaluation approach, using synchronized playback of
heterogeneous data such as audio, video, GPS tracks, system logs, and other sensory data
(Morin, 2001, 2002). R&E can also be used to assist training instructors in their afteraction reviews (AAR), as it helps clarifying what actually happened at different sites.
Such AARs can be conducted within respective team; or on a larger scale, highlighting
intra-team coordination and the complexity of inter-organizational command (U.S. Army
Combined Arms Center, 2011).
The Reconstruction phase involves several processes for collecting and synchronizing a
rich dataset, a mission history, modeling the course of events of a tactical operation. The
data types used are dependent on the nature of that operation, and of the required
analyses, as such R&E is a flexible approach allowing the researcher to customize the
model to current needs. The resulting mission history needs to convey both contextual
data such as the organization model, and data samples, as exemplified in Figure 2.7.
Mission history
Organization model
(participating units)
Data collection
model (links events
to units)
Events
(chronological list)
Data
(list of data samples)
Figure 2.7. A representation of mission histories (from Paper II).
The Exploration phase involves playback and analysis of the mission history. Exploration
can be thought of as an iterative process consisting of three steps in each cycle:
visualization, analysis and integration (Andersson, 2009); allowing the analyst to
[13]
feedback analysis products into the model. Iterating this methodology enables raising the
analysis to higher abstraction levels, until reaching the questions in focus (Albinsson et
al., 2005).
The R&E approach has been used extensively to evaluate training events, simulations and
experiments (e.g., Morin, Jenvald & Worm, 1998; Thorstensson, Morin & Jenvald, 1999;
Thorstensson, Axelsson, Morin & Jenvald, 2001; Lantz, Andersson, Jungert & Levin,
2009; Andersson, Mullins, Bourakov, Bordetsky & Forsgren, 2011). The main
application of R&E in these cases has been mission efficiency analysis (MEA), i.e., a
hybrid between teamwork and task work analysis (Worm et al., 1998). A survey has
indicated that these reconstructed mission histories may be applicable for knowledge
transfer by combining it with traditional textbooks and hypermedia information into a set
of digital courseware (Jenvald, Rejnus, Morin & Thorstensson, 1998; Jenvald, Morin &
Rejnus, 2000).
2.5.1 SOFTWARE FOR R&E
The R&E approach was developed together with a set of tools that support the data
compilation and visualization steps. The first of those tools was the MIND Studio, a piece
of software with the capability of retrospectively visualizing a time-synchronized
multimedia model of the reconstructed course of events (Jenvald, 1999; Morin, 2001),
see Figure 2.8. The MIND Studio provides several customizable viewports into one
dataset, allowing the explorers to review the mission history from different angles. The
main navigation control is the timeline, which allows all data to be visualized in causal
order as they happened.
Figure 2.8. A screenshot from MIND Studio, exploring an Airborne Battalion’s final
exercise, ASÖ’03.
Compiling a dataset for the MIND Studio requires implementation of data adapters that
can interpret the data that was captured through the reconstruction phase of R&E. These
implementations require understanding of the different data formats which may be
[14]
cumbersome, but it also enables MIND to stay data agnostic, ensuring that data from any
data source can be visualized, once interpreted.
The F-REX Studio is the Swedish Defense Research Agency’s successor of MIND,
developed to better support massive datasets and collaborative analysis of C2 exercises
and operations (Andersson, 2009). The need for this redevelopment was identified with
the observation that data collection methods and tools were improving quicker than the
ability to manage voluminous data. An early version of the framework, Figure 2.9,
provides roughly the same features as the original MIND framework albeit perceived as
more flexible and modular by the developers, allowing for more rapid module
development. Although the visual differences between MIND and F-REX are minor, the
technology shift meant lower costs for the development and maintenance of the
framework.
Figure 2.9. F-REX is the successor of MIND, here used to explore the interactions of a
commander during a training session at the Sandö facility (from Paper II).
Figure 2.10 shows a more recent version capable of handling massive datasets from
tactical operations also in the virtual realm such as Cyber Defense Exercises. These
operations typically generate comparatively huge datasets when capturing also virtual
stimuli such as IP traffic and hack attempts, calling for careful design and implementation
of the data processing units and the visualization tools (Andersson, Granåsen, Sundmark,
Holm & Hallberg, 2011).
The R&E approach supports a wide system perspective gathering factual, simultaneous,
real-time data on multiple organizational levels on multiple geographical locations and
from multiple actors. Use of exploration tools, such as MIND and F-REX, enables the
analyst to quickly switch focus between different viewpoints in any substantial dataset to
[15]
get a birds-eye view of the DTO (Andersson, 2009). In summary, the R&E approach can
be seen as a method of capturing and disseminating experience in the form of
multimedia-rich stories (Rao & Goldman-Segall, 1995; Liu, Liao & Pratt, 2009).
Figure 2.10. A recent version of F-REX Studio used to explore the interactions between
team members along with stimuli captured from both the physical and the virtual
realm at a national cyber defense training session (Andersson, 2012).
2.5.2 MIXED METHODS DATA ANALYSIS
R&E is a mixed methods approach that tries to overcome the shortcomings of both
qualitative and quantitative methods by combining them and letting them complement
each other. Mixed methods research is comparatively new, and sometimes criticized, but
it has been increasingly popular in social and behavioral research (Lieber, 2009), and
medical research (Schifferdecker & Reed, 2009). As the research approaches are similar,
it can be rewarding to follow the methodological developments in these areas and see if
they can be applied to mixed methods C2 research as well.
As the size of the mixed methods research project scales up, the need for proper tools
increases rapidly. Logically, software frameworks have emerged to assist researchers in
managing data, results, and findings, such as EthnoNotes (Lieber, Weisner & Presley,
2003) and its successor Dedoose3. QDA Miner4 is another popular alternative which
focuses specifically on the coding and analysis processes, while also assisting the
researchers with semi-automatic report generation (Lewis & Maas, 2007). While these
3
http://www.dedoose.com, retrieved 2013-03-05.
4
http://provalisresearch.com/products/qualitative-data-analysis-software, retrieved 2013-03-05.
[16]
commercial products are powerful, they do not fully support the C2 use case in which
cues from the environment as well as other contextual data has to be integrated with the
decision-making cues to analyze the thought processes and understand why a certain
decision was made. For this purpose, F-REX is commonly used as well as the CITE
toolset5, which also spun off from MIND. CITE, with less on big data management, takes
a more holistic approach compared to F-REX, supporting exercise controllers by
including planning and management tools in addition to the evaluation component.
Venkatesh, Brown and Bala (2013) studied the applicability of mixed methods in the
Information systems (IS) research field. They accurately describe a helpful set of
guidelines for when and how to work with mixed methods instead of traditional
qualitative or quantitative methods, and illustrate the guidelines to two earlier IS research
papers. These guidelines may be applicable to determine whether R&E can and should be
used to externalize mission experience.
2.5.3 MISSION ANALYSIS
There are many alternative approaches for assessing tactical operations, some taking the
route of focusing on team performance, including the widely spread Situation Awareness
Global Assessment Technique (SAGAT) (Endsley, 2000) and Command Team
Effectiveness (CTEF) (Essens, Vogelaar, Mylle, Blendell, Paris, Halpin & Baranski,
2005). R&E on the other hand, was primarily developed to analyze mission efficiency
from a cybernetics perspective (Ashby, 1956; Worm et al., 1998), see Figure 2.11. Using
this approach, the output can be seen as a consequence of the system, its inputs and any
disturbances. The accuracy of any efficiency metric will be determined by the level of
detail in which the input, output and disturbances have been modeled and captured.
Figure 2.11. A control theory based model of tactical missions (Worm et al., 1998).
Copyright (1998) Elsevier. Reprinted with permission.
5
Visuell Systemteknik i Linköping AB, http://www.vsl.se/Page.aspx?content=CiteBase, retrieved 2013-0305.
[17]
Mission efficiency can be qualitatively determined through R&E, by reconstructing the
initial state (the reference), the system itself (including the control element) and the
course of events (control, disturbances, and output). There is, however, an inherent
complexity in the dynamicity of system properties rendering the system difficult to
measure (Brehmer, 1992). To overcome this, a wide system boundary is used, extending
the unit of analysis to tactical units including technological systems, C2 systems and
support systems (Worm et al., 1998; Worm, 1999). By explicitly avoiding assessment of
individual thought processes, the problem of measuring mission efficiency is somewhat
simplified. Worm and colleagues (1998) utilized this by measuring the system through
information processing and resource utilization, thus defining mission efficiency as the
impact and outcome of a mission related to the resources utilized (ibid.), see Figure 2.12.
Mission Efficiency:
The impact and outcome of an emergency
response mission, related to the resources
utilized.
Initial mission state
Mission course of
events
Information
processing during
the mission
Decision making
and actions during
the mission
Utilization of
available resources
during the mission
Figure 2.12. Model employed to measure the efficiency of emergency response missions
(after Worm et al., 1998).
The mission efficiency analysis method uses subjective analysis of the mission outcome
and course of events as a basis for analyzing team performance, topped with objective
measures of resource utilization and possibly information processing (Worm et al., 1998).
Similar methodologies are commonly adopted for mission debriefings such as through
AARs (U.S. Army Combined Arms Center, 2011). More objective alternatives have been
proposed, based on a set of specific task performance metrics, such as exposure to risk in
urban warfare (Rowe, 2009; Sadagic, Welch, Basu, Darken, Kumar, Fuchs, Cheng,
Frahm, Kolsch, Rowe, Towles, Wachs & Lastra, 2009) or communication (LaVoie, Foltz,
Rosenstein, Oberbreckling, Chatham & Psotka, 2008).
[18]
Situation awareness is another popular indicator of team performance, measured with,
e.g., SAGAT (Endsley, 2000). The technique uses interruptions in task performance to
objectively assess situation understanding of the team members. The method is intrusive
as it requires interruptions of task performance to measure the SA at certain times, unless
post-tests are used, in which case the accuracy of the metric is subject to the team
members’ ability to retroactively simulate prior situation understanding. Non-intrusive,
on-the-fly, assessment of SA can be acquired subjectively by observers, although the
accuracy is then subject to the observers’ ability to convert observable actions into a
measure of SA.
The CTEF model was proposed by NATO as a comprehensive approach to assessing
command teams, including both task outcomes and team outcomes (Essens et al., 2005;
Hof, de Koning & Essens, 2010). The instrument is an important component in the
assessment of tactical units modeling team effectiveness as a function of processes
applied to preconditions. Decision making and plan execution are two of the many
processes that CTEF covers, making the model compatible with OODA and other models
of C2. CTEF however, shows an explicit focus on motivating and other non-task oriented
behaviors, thereby implying that domain knowledge is just one of many components that
make a tactical team function (ibid.).
2.6 KNOWLEDGE TRANSFER
Experience is a relativistic form of knowledge always contingent and contextual, i.e., it
cannot be made objective and universal (Cilliers, 2000). Yet, learning from experience is
possible not only from one’s own experience, but also from others’, i.e., vicarious or
observational learning (Argote & Miron-Spektor, 2011, p.1126). It is thus relevant to
explore different methods of transferring knowledge in the pursuit for alternative,
possibly cheaper, ways of acquiring mission experience than through hands-on practice.
2.6.1 KNOWLEDGE TRANSFER MODELS
Research on knowledge transfer is plentiful, and many attempts have been made to
explore the concept by stimulating certain behaviors such as improvisation and
communication through innovative training and role-playing (e.g., Mendonça & Fiedrich,
2006; Trnka & Jenvald, 2006). Other studies of particular interest include a virtual reality
approach by trying to simulate realistic operations to induce learning effects similar to
those gained from complex operations (Lafond & DuCharme, 2011).
One of the most well-known, although sometimes criticized, knowledge transfer models
is the SECI model (Nonaka, 1991; Nonaka & Takeuchi, 1995; Gourlay, 2003). According
to the model, there are interaction processes between tacit and explicit knowledge: (1)
socialization (empathizing, acquiring tacit knowledge from others’ tacit knowledge), (2)
externalization (articulating, creating explicit knowledge from tacit), (3) combination
(cross-referencing multiple bodies of explicit knowledge, connecting the dots), and (4)
internalization (embodying, or applying externalized knowledge in practice). These four
interaction processes form a spiral, see Figure 2.13, which organizations and individuals
can iterate over and over again to generate new knowledge (Nonaka, 1991).
[19]
Socialization
Externalization
Internalization
Combination
Explicit knowledge
Tacit knowledge
Tacit knowledge
Explicit knowledge
Figure 2.13. The SECI model of knowledge creation (after Nonaka & Takeuchi, 1995).
A third dimension, Ba, has been added to the SECI model to represent the shared context
without which shared knowledge cannot be created, i.e., a common baseline to share
knowledge or experience (Nonaka, Toyama & Konno, 2000). Generation and
regeneration of Ba provides energy, quality and a place to perform the individual
conversion between tacit and explicit, allowing the individual to move along the
knowledge-creating spiral (ibid.). To successfully attain experience through vicarious
learning, it is thus essential to establish this common understanding of the context in
which a mission occurs.
2.6.2 TRAINING AS A MEANS OF KNOWLEDGE TRANSFER
Training has a role in any tactical organization as the main method of acquiring and
refining skills, both physical skills and more cognitive skills such as decision making,
leadership and C2. It is through this training that tactical units learn the basics and
generate a library of procedures and tricks to resolve standard tasks. It is through training
that trainees learn how to apply their means to reach a well-defined end.
The decision-making process that occur in a dynamic situation is typically complex in
nature and not considered an end by itself, but rather a means through which the, often
ill-defined, objectives can be reached. As the events unfold and the situation evolves, the
commander will need to adapt and reassess, so the decision-making process is continuous
and interwoven with the performance phases (Means, Salas, Crandall & Jacobs, 1993).
Training commanders for such events through traditional rationalistic decision theory
have not been shown very successful, neither has bias reduction training (ibid.). Instead,
decision-making training must capture the characteristics of the tasks and the context
within which they are performed (ibid.).
2.6.3 DIGITAL COURSEWARE FOR MISSION EXPERIENCE TRANSFER
Properly organized training promotes rapid learning by giving trainees the opportunity to
internalize knowledge that they acquired through other means, and receive constructive
feedback. Full-scale training yields an implicit opportunity for all participants to put their
own actions into a larger scope, to see their roles from a new perspective. However, there
are at least two major obstacles that hinder such learning: (1) the training scene is
complex with several units operating in parallel at multiple locations and under time
[20]
pressure, and (2) there is a lack of methods, tools, and resources to support the
documentation and visualization of complex rescue operations for retrospective learning
(Jenvald et al., 2000).
The feedback is typically generated by letting human evaluators monitor the trainees and
report what they find in debriefings, critiques, and written reports. R&E has been
proposed, and tried at a Swedish emergency response training event in Alvesta, as an
alternative solution to create a digital set of courseware that may help trainees
retrospectively review and reflect upon their actions and roles within a larger context
(ibid.). This courseware consists of a reconstructed mission history combined with
textbook information and hypermedia information accessible through Internet, i.e., a
combined set of externalized knowledge, with more contextual information than
traditional procedure manuals. Results from a case study at Umeå University indicate that
such courseware can be used to complement training even for trainees that did not
participate in the reconstructed event (ibid.).
2.6.4 ALTERNATIVE METHODS FOR KNOWLEDGE TRANSFER
Using multimedia tools as a means of retaining organizational memory was proposed
already in 1995 with the release of the video-annotating tool Constellations (Rao &
Goldman-Segall, 1995). Further, Abecker and Decker (1999) proposed an architecture for
composing an organizational memory system (OMS) using stored multimedia stories,
such as annotated videos, images, and text. They list several issues and challenges in
creating such systems, most notably the importance of enabling connections in the
system, i.e., connecting people to people, people to knowledge, knowledge to knowledge
(ibid.). Many tools and frameworks have been developed since based on the assumption
that multimedia stimulates learning, as well as cognitive theories to explain multimedia
learning, and its unknowns (e.g., Mayer, 2005).
Another theoretical framework for comprehension of physical systems through
multimedia presented by Hegarty (2005) integrates encoded auditory and visual features
into an internal representation closely resembling a reconstructed multimedia model such
as the mission history. Her work informs that comprehension of multimedia models of
physical phenomenon requires the student to have high domain knowledge, meaning that
if these conclusions are applicable for knowledge transfer of mission experience it may
bear impact on who the target group for such learning systems are (ibid.).
2.6.5 MEDIA RICHNESS AND KNOWLEDGE TRANSFER
The media richness theory was proposed by Daft and Lengel (1986), to categorize
communications mediums based on their capacity to process rich information, i.e., a
phone call cannot reproduce visual cues and is therefore less rich than video
conferencing, but more rich than textual e-mails and letters, which cannot represent
anything but text. They argued that higher richness is more effective, a viewpoint that has
since been debated and criticized since it does not seem compatible with how people
choose media for their communication (e.g., Ngwenyama & Lee, 1997). Other studies
have shown that lean media can be as effective or even better compared to rich media for
presenting analyzable tasks, while high media richness mediums are beneficial primarily
[21]
for non-analyzable tasks such as subjective interpretations (e.g., Lim & Benbasat, 2000;
Otondo, van Scotter, Allen & Palvia, 2008).
Liu et al. (2009) showed that media richness theory is insufficient, but can be combined
with flow theory and technology acceptance models to explain the role of multimedia in
e-learning. The combination of all three of audio, video and text outperformed any single,
or combination of the other two mediums for perceived usefulness and concentration
among learners. This is contradictory to earlier research (e.g., Lee, Cheung & Chen,
2005), for which they offer the explanation of technology acceptance. Their findings thus
imply that adapting learning material to the learner’s technology acceptance level or other
preferences may increase the generated effect.
2.7 WORKPLACE ETHICS
The view of war scenes and emergency operations as a workplace may be somewhat
disturbing for the non-initiated. However, for those who operate in tactical organizations,
the workplace can very well be a scene for combat or accidents. When the tactical units
arrive at the scene, they are expected to operate in the best interest of the society (Miller,
2005). When R&E has been employed at these organizations, albeit at training events and
not real operations, sometimes a certain resistance has been encountered from trainees
who were feeling uneasy about the monitoring of themselves, even though the unit of
evaluation in R&E typically is a tactical team as opposed to individuals (see Paper III).
This subsection looks into three important topics for ethics research that may bear impact
on analyzing R&E from an ethical point of view.
2.7.1 PRIVACY
A powerful and distinct definition of privacy was given by Warren and Brandeis (1890):
“the right to be let alone”. This definition is still in use but it does not come without
complication if applied to workplace ethics since there is normally an agreement, a
contract, between the employer and the employees. A prima facie conflict immediately
arises when weighing privacy versus contract under these premises. A more applicable
definition was given by Aiello and Klob (1996) as “the ability for an individual to control
the use of their own personal data, wherever it might be recorded”. This definition is both
practical and seems acceptable in work life. The immediate consequence of the definition
is that all non-controlled monitoring of controllers in a tactical operation can be regarded
as an infringement on privacy, and therefore problematic.
Privacy infringements can have many negative consequences such as lack of trust,
anxiety and alienation (Kizza & Ssanyu, 2005). Miller (2005) argued that privacy
infringements can be accepted to a higher degree on public service employees such as the
police force. This does not, however, imply that the negative consequences of privacy
infringements do not apply to them.
2.7.2 R&E AND W ORKPLACE SURVEILLANCE
It has already been established that performance of actions is a vital component of the
controlling system in any tactical operation. Reconstructing a mission history thus
requires capturing of these actions and the cues in the environment that lead to decisions
[22]
activating them. These interactions can be captured either by sensors, e.g., video cameras,
or by human observers; however both methods can be paralleled with monitoring of the
work scene. Thus, reconstructing a mission history may subject the employees to some
level of monitoring, and therefore possibly privacy infringements.
2.7.3 INFORMED CONSENT
One of the primary measures to reduce the negative effects of privacy infringements is
consent. However, it is not an uncomplicated issue, as consent is hardly valuable unless it
is truly informed (Malek, 2005). To qualify as informed consent, five important
conditions have to be fulfilled (ibid.):
 the subject is given all information,
 the subject understands all information,
 the subject is able and allowed to make a choice,
 the subject makes the choice independently, and
 the subject gives consent.
Obtaining such consent can be hard, as all information may be a fuzzy concept when not
even the explorers know on beforehand what results the R&E will yield. The subjects can
thus not be certain of what the explorers will see and what results the data can generate. It
is therefore questionable whether the consent qualifies as informed or not.
2.8 RESEARCH GAP
A plethora of research has been dedicated to the understanding of C2 from many different
angles over the last decades (Alberts & Hayes, 2006; Brehmer, 1992; 2000; 2009;
Kahneman & Tversky, 1979; Essens et al., 2005; Pigeau & McCann, 2002; Weick &
Sutcliffe, 2001). The emergence of JCS and cognitive engineering highlights the same
type of problems from a different viewpoint (Hollnagel, 1993; Hollnagel & Woods,
2005). The fields of knowledge creation and transfer have also received much attention
(Hijikata, Takenaka, Kusumura & Nishida, 2007; Nonaka, 1991; Novak, 2007;
Pettersson, 2009; Rice & Rice, 2000).
The relationship between C2 and experience has been established by Klein’s (1993)
widely recognized model of RPDM as well as Brehmer’s (1992; 2000) model of dynamic
decision making in C2. Yet, little attention has been paid to the complexity of acquiring
and sharing experience targeted at resolving tactical events that have not been
encountered before by the performing organizations and adhocracies. A rare exception is
the limited study on R&E to create digital courseware from the Alvesta field trial
(Jenvald et al., 2000). This thesis fills the gap by investigating a new methodology for
externalizing and sharing mission experience, through the lens of decision making in
dynamic C2 situations.
[23]
[This page is intentionally left blank]
[24]
Chapter 3
METHODS
The research questions in this thesis are addressed through four separate studies on
different aspects of applying R&E in mission training. The studies, Papers I-IV, employ
various qualitative methods ranging from case studies with participatory observation to
literature reviews. All four studies are summarized in Chapter 4, whereas the results are
detailed in light of the provided background theory to answer RQ1-3 in Chapter 5, with
follow-up discussions in Chapter 6.
3.1 STUDY DESIGN
This thesis presents four studies of using R&E for transferal and dissemination of mission
experience. The research questions are approached using theoretical propositions
grounded in the evidence collected from the four attached studies, i.e., Papers I-IV. The
study design is thus comprised of four parts, as listed in Table 3.1. Each method is
explained briefly in the following sections of this chapter.
Table 3.1. Study design
Study
Main research approach
Data collection methods
I
Literature review (narrative)
Literature search
II
Case study (exploratory)
Participatory observation, after-action
review, group interview
III
Case study (exploratory)
Direct observation, participatory
observation, after-action reviews
IV
Literature review (narrative)
Literature search
3.2 CASE STUDIES
A case study is typically applied to study a real phenomenon, e.g. individuals,
configurations, events, missions, and organizations. They are suitable when the
researchers cannot control the unfolding events, but want to answer questions such as
when and how (Yin, 1994). There are mainly three types of case studies: (1) exploratory
where the researcher investigates a phenomenon to identify patterns and characteristics,
often without a priori knowledge, (2) descriptive where the researcher investigates certain
features based on a founding theory, and (3) explanatory where the researcher tries to
explain a phenomenon through detailed analysis, requiring a baseline and guided data
collection (ibid.).
[25]
The studies presented in Paper II and Paper III were conducted as exploratory case
studies, with evidence collected through direct observation, participatory observation,
after-action reviews, and group interviews.
3.2.1 DIRECT OBSERVATION
Direct observation is a data collection method that aims to be unobtrusive, conducted by
outside observers in an attempt to not affect the studied phenomenon, i.e., to avoid bias.
The researchers thus want to observe sampled situations from a spectator perspective, i.e.,
nonreactive (Bernard, 2011). However, direct observation often becomes blatant and
reactive if the studied persons are aware of the spectators and put on a show for the
audience (ibid.). This raises an ethical issue of whether to use reactive observation, by
allowing the subjects to be aware of the monitoring, or to use nonreactive observation,
i.e., hiding the observation from the subjects (ibid.).
The case study in Paper III employed reactive observations to study other research
groups in their exercise analysis, in an attempt to get evidence for comparison against the
self-experienced R&E method. Additionally, in both Paper II and Paper III, reactive
observation was employed in the reconstruction itself to collect information on the
exercise events as they unfolded.
3.2.2 PARTICIPATORY OBSERVATION
Participatory observation is a common, but demanding, method for qualitative data
collection. The researchers assume a role in the studied phenomenon in real-life settings
by participating in the normal activities. Participant observation allows understanding of
the phenomenon from the inside as opposed to the spectator perspective given by direct
observations (ibid.).
Participatory observation was used as a means of data collection in the case studies
presented in Paper II and Paper III, where the researchers themselves utilized R&E, in
conjunction with the exercise management staff, to analyze and disseminate the course of
events from the exercise. In the Swedish Armed Forces case study (Paper III), the
difference between direct and participatory observation is exemplified by the researcher
conducting the analysis (participatory observation) versus observing other analysts
conducting it (direct observation).
3.2.3 AFTER-ACTION REVIEWS
After-action review (AAR) is a process designed to conduct structured debriefings after
military training sessions (U.S. Army Combined Arms Center, 2011). The process has
been adopted worldwide, both by military and civilian organizations, e.g., the HA/DR
domain (Sexton & McConnan, 2003). A well-executed AAR reviews what was supposed
to occur, what actually happened, what was right and wrong with what happened, and
how the task should be done differently the next time (U.S. Army Combined Arms
Center, 2011). An AAR can be formal, with external observers and controllers, complex
training aids and scheduled, or informal with an internal chain of command and simple
training aids and conducted when a need has been identified (ibid.). For maximum effect,
[26]
an AAR should be scheduled during, or shortly after, the completion of an activity and
based on objective data (ibid.).
The study in Paper II used a formal AAR in emergency response training to increase
learning of the training session. The study in Paper III used daily formal AARs in a
military setting, to conclude the activities after each exercise session. R&E was used in
both studies to provide objective data for the AAR sessions.
3.2.4 GROUP INTERVIEW
Interviews are often used to gain understanding of peoples’ experiences and can be
conducted in many forms, from unstructured and ad-hoc, to formal and controlled,
interactions with the respondent (Seidman, 1998; Bernard, 2011). Interviews can target
one respondent, or a group of respondents. The latter is often called group interview, or
focus group, and serves the purpose of extracting a collective understanding, including
the group dynamics and individual perspectives that may differ among the group (Stewart
& Shamdasani, 1990).
A group interview can have several purposes, including (1) exploratory, which are often
used to satisfy the researchers’ curiosity, e.g., to test the feasibility of a more complex
study, (2) pretest, which serve as a pilot to a larger study, (3) triangulation, in which the
focus groups are used to add methodological rigor, and (4) phenomenological, when the
interview is a regular data source to be used for exploratory analysis (Frey & Fontana,
1991).
While subjectivity is not eliminated by moving from one to many respondents, it can be
reduced – leading to higher validity of the collected data, however it comes with the risk
of suppressing individual thoughts and opinions due to group dynamics mechanisms
(ibid.).
The study in Paper II includes a group interview session from August 2007, eight months
after the AAR. The seminar included some 30 respondents divided into groups of 3-4
who discussed nine topics broadly among themselves before sharing their thoughts to the
researchers in an open floor format. The seminar had an exploratory purpose, trying to
identify how R&E can be used, and what potential issues there may be when using it for
exercise assessment. The discussion topics, or questions, are listed in Paper II.
3.2.5 CASE STUDY EVIDENCE ANALYSIS
In case study research, the analysts often needs to resort to experience and literature to
interpret and present evidence, since statistical analysis cannot always be used (Tellis,
1997). As the study presented in this thesis is theory-building and exploratory, with
exclusively qualitative evidence, data interpretation is a matter of “examining,
categorizing, tabulating or otherwise recombining the evidence” (Yin, 1994). The
evidence used for this analysis consists primarily of findings from Papers I-IV.
3.3 LITERATURE REVIEW
A literature review is a “systematic search of published work to find out what is already
known about the intended research topic” (Robinson & Reed, 1998, p.58). There can be
[27]
many reasons for conducting literature reviews, including avoidance of duplication,
sharpening and deepening a theoretical framework, identify knowledge gaps and to study
definitions and characteristics of other previous work with the aid of adopting them in a
new research project (ibid.). A literature review can take on various forms, such as (1)
narrative, where the review critiques and summarizes a body of literature to draw
conclusions on a specific topic, often without obvious selection criteria, (2) systematic,
where a rigorous and well-defined selection criteria is used to define a comprehensive set
of literature for the purpose of answering well-focused questions, (3) meta-analysis,
where statistical analysis is employed on a large body of quantitative findings to integrate
them and generate increased understanding, and (4) meta-synthesis, where findings from
a body of qualitative literature are assessed and integrated to find common core elements
and themes (Cronin, Ryan & Coughlan, 2008).
The literature reviews in Papers I and IV are both narrative in character, and aim to build
founding theories for the reasoning presented in the respective study. In the study
presented in Paper I, the synthesis led to design of an explanatory model which was
discussed and refined through ad-hoc interviews before being finalized. The results from
case study in Paper IV were refined after open floor discussions following a presentation
of the initial findings.
3.3.1 LITERATURE REVIEW DATA COLLECTION INSTRUMENTS
The literature reviews in Paper I and Paper IV both used a combination of Internet-based
instruments to collect data for the review:
 EBSCOhost6 is a meta-search engine by EBSCO Industries Inc., cross-searching
five indexed databases of periodicals, academic journals and more. The included
databases are PsycARTICLES, PsycINFO, Military & Government Collection,
Academic Search Premier, and International Security & Counter Terrorism
Reference Center.
 ScienceDirect7 is a scientific database, with journal articles and book chapters
published by Elsevier B.V.
 IEEE Xplore8 is a digital library holding scientific and technical content published
by the Institute of Electrical and Electronic Engineers (IEEE). The main contents
are journal articles, conference proceedings, technical standards and eBooks.
 Web of Science9 is a multidisciplinary data citation database by Thomson Reuters,
indexing a comprehensive set of journal articles and conference proceedings,
making them searchable through their own search engine, and enabling selection
based on citation count.
6
http://search.epnet.com
7
http://www.sciencedirect.com
8
http://ieeexplore.ieee.org
9
http://thomsonreuters.com/products_services/science/science_products/a-z/web_of_science/
[28]
 Google Scholar10 is a search service from Google Inc. covering a broad set of
scholarly literature, including articles, theses, books, abstracts, court opinions, etc.
In addition to the above, the Swedish Defense Research Agency (FOI) library hosts a
service for its employees through which they can submit keywords to the librarians, who
then return a set of relevant academic publications and technical reports matching the
keywords. This service was used extensively in both literature reviews to complement
searches using the five Internet based engines.
10
http://scholar.google.com
[29]
[This page is intentionally left blank]
[30]
Chapter 4
SUMMARY OF APPENDED PAPERS
This thesis is founded on four research articles: (1) Sharing mission experience in tactical
organisations, (2) Evaluation of crisis management operations using reconstruction and
exploration, (3) Reconstruction & exploration of large-scale distributed tactical
operations – multimedia tools for evaluation of emergency management response, and (4)
Privacy and distributed tactical operations evaluation.
Paper I: The study Sharing mission experience in tactical organisations explores the
relationship between flexibility, adaptability, improvisation, resilience, innovation and
creativity with the objectives of increasing understanding of what mission experience is,
why it is important, and how it can be shared.
Paper II: The study Evaluation of crisis management operations using reconstruction
and exploration presents a case study on R&E conducted at the Swedish Rescue Services
training facility in Sandö 2006 with the objective of finding out how and if R&E and can
used to support emergency response training.
Paper III: The study Reconstruction & exploration of large-scale distributed tactical
operations – multimedia tools for evaluation of emergency management response
presents a case study from the first application of R&E in a large-scale environment with
the Swedish Armed Forces at their main C2 training facility in 2005. The objective of this
study is to explore how R&E compares to other research methods for evaluation of largescale military tactical exercises.
Paper IV: The study Privacy and distributed tactical operations evaluation takes a
somewhat different approach compared to the above three with the objective to explore
the highly interesting topic of ethical effects that R&E may have on personnel exposed to
recording and evaluation, and how it can be morally justified.
This chapter presents major findings from the four appended articles, organized per
article, with the findings followed by a short summary of theory, reasoning and
underlying data motivating the results that were originally presented in the corresponding
appended paper.
4.1 SHARING MISSION EXPERIENCE IN TACTICAL ORGANISATIONS
The main outcome of the study is the FAIRIC model, see Figure 4.1. The model outlines
a framework to distinguish between six commonly used, but seldom explained, concepts
in emergency management, and tactical mission control. The model provides a common
vocabulary to discuss knowledge gained from mission experience; and thereby
understand related literature. In the context of modeling tactical operations for knowledge
transfer, the model provides a mechanism to start analyzing what mission experience
consists of and which integral components thereof that need to be externalized in order to
enable sharing and transferring of mission experience over boundaries of time and space.
[31]
Unfamiliarity of new situation
Improvisation
Innovation
Resilience
Creativity
Adaptability
Flexibility
Proactiveness
Figure 4.1. The FAIRIC model (from Paper I).
4.1.1 OBJECTIVES
The objective of this study is to identify important outcomes of mission experience,
define how they relate to each other, and lay out a theory of how mission experience can
be shared within and between organizations through other methods than hands-on
practice.
4.1.2 BACKGROUND
It stands clear that sharing mission experience is a complex task, requiring first and
foremost a definition of what mission experience really is; which by itself is non-trivial.
Cillier (2005) stated that “knowledge cannot be fixed in a representational way but is
always contingent and contextual”. Weick (1969, p.206) similarly concluded that “if an
organization is to learn anything, then the distribution of its memory, the accuracy of that
memory, and the conditions under which the memory is treated become crucial
characteristics.” It seems thus, that mission experience is highly dependent on context
and circumstantial events and conditions. As such, it becomes clear that it is not primarily
the particular solution of a problem that constitutes mission experience, but rather the
underlying decision-making process. According to the well-known RPDM model (Klein,
1993) some of the most valuable knowledge gained from a mission is an increased
dictionary of goals, cues, expectancies and plausible actions, i.e., an improved bank of
situations to match and recognize future events against.
Nonaka’s (1991) model of knowledge creation, i.e., the SECI model, outlines
socialization, externalization, combination, and internalization, as the four main
components of knowledge generation. By regarding these four steps as a spiral as in
Figure 4.2, Nonaka’s model describes a method for knowledge transfer that explains how
[32]
mission experience can be transferred. Socialization, in this context, can for example
refer to observation of, and participation in, in real missions. Externalization on the other
hand, is often done through debriefings, and reporting. Commonly, mission experience is
externalized as written reports or entered into a lessons learned database. In the best of
worlds, the products of the externalization process can easily be distributed within and
between organizations. Relating findings and results from these externalization products
to an existing body of knowledge can yield increased insight. Nonaka refers to this
phenomenon as combination. Finally, the internalization process occurs when
individuals, or organizations, operationalize their newly added knowledge by applying it
in the field. The spiral analogy implies that new experience from using the acquired
knowledge in the field can be socialized or externalization again, followed by a
combination or internalization process, etc.
Socialization
Externalization
Internalization
Combination
Explicit knowledge
Tacit knowledge
Tacit knowledge
Explicit knowledge
Figure 4.2. The SECI model (after Nonaka & Takeuchi, 1995).
The SECI model thus gives four integral components of sharing mission experience,
externalization, combination, internalization, and socialization. Turning back to King’s
(2005) statements that humanitarian aid (HA) organizations rely heavily on their
personnel to conduct or observe missions and exercises to acquire necessary knowledge,
it becomes clear that socialization is the most important knowledge sharing component
used by many HA organizations. King’s further findings indicate that externalization
processes are not effectively used since knowledge is shared mainly through
apprenticeships and occasional seminars. His critique indicates that there is room for
improvement in this area. While there are many actively used systems in the world today,
e.g., lessons learned systems; few solutions seem to address the need for representing
contextual knowledge as identified above. Further, many of these systems seem to be
designed for externalizing and collecting knowledge, not for disseminating or allowing
internalization thereof.
4.1.3 RESULTS
To fully understand what features of a dynamic control situation to model and capture,
one must comprehend what experience the operators possess and what cues they
recognize against when they make recognition-primed decisions. This work reviews
literature on dynamic decision-making, emergency management, and organizational
[33]
memory to find six abilities that can be identified as central for tactical command in
dynamic environments: flexibility, adaptability, improvisation, resilience, innovation, and
creativity (FAIRIC). The review reveals that while these abilities are commonly
described as important, they are rarely distinguished between, or even explicitly defined.
As an example, improvisation alone is found to have at least 42 co-existing definitions in
one referenced study (Moorman & Miner, 1998) – some of which refer to one or several
of the other FAIRIC concepts. In an attempt to improve understanding of how to model
and capture dynamic decision making situations, the FAIRIC model is proposed,
highlighting nuances and differences between these commonly referred abilities.
The proposed FAIRIC model is derived from existing literature and outlines how they
relate to each other from two aspects: unfamiliarity of the situation and proactiveness as
shown in Figure 4.1. A flexible organization, according to this model, can be expected to
have taken proactive measures to cope with situations that they can foresee; whereas an
adaptive organization is expected to react on changes in the environment as they occur,
and cope with the new situation from these premises. An improvising organization is one
that has the ability to react with timeliness to respond to situations which could not be
foreseen, while the innovative organization is more proactive, e.g., by inventing new
procedures to respond to unforeseen events before they occur.
4.2 EVALUATION OF CRISIS MANAGEMENT OPERATIONS USING
RECONSTRUCTION AND EXPLORATION
This case study shows that multimedia supported mixed method research can indeed
improve the effectiveness of emergency response training, by allowing the trainees to get
more out of the training events and to some extent even learn from scenarios where they
were not able to participate, such as when there are too many trainees to allow everyone
to train in their operational role.
4.2.1 OBJECTIVES
The objective of this study is to explore how R&E can be used to support emergency
management C2 training, and to determine what kinds of objective data best support AAR
in this context.
4.2.2 BACKGROUND
Disaster relief is a growth-market where money and resources are continuously being
pushed into the system in an effort to save human lives. Meanwhile, in the field, the relief
agencies struggle to manage excess inventory and a logistical nightmare. The real need
often lies in logistics and effectively managing supply chains (Thomas & Kopczak,
2005). This gap in need versus effort is one example of where efficiency in disaster relief
operations can be improved. Complex inter-agency operations such as disaster relief,
terrorism response and emergency management require commanders from several
independent organizations to control dynamic situations that are often impossible to fully
understand and predict. Continuous assessment and team performance evaluation, from
both live operations and training, can help the organizations improve and be better
prepared for the disasters of tomorrow.
[34]
As operations grow bigger and more complex, the effort to collect and analyze data
increases rapidly. Without careful planning and management the assessment task thus
become overwhelming (Albinsson et al., 2005). While a common approach to evaluate
large and complex operations (exercises in particular) is to rely on quantitative measure
and self-reported questionnaires, such assessment tend to focus on the respondents’
perception of the operations, and correlate strongly with the level of accomplishment of
known pre-defined objectives. These methods, however, are less fit to explore
breakdowns and determine the need for improvement in organizations that do not match
the predefined objectives.
Cognitive analysis approaches and many other qualitative analysis methods are
comparatively demanding in terms of time and resource (Blandford & Wong, 2004).
Such methods have the ability to post-hoc capture insights on particular events such as
unforeseen breakdowns, and may enable understanding of causality relationships.
However, because of the workload and effort needed with these methods, the analysis
tend to focus on a selected few incidents and based on the perception of a handful
individuals rather than taking the whole operation into account. Therefore, traditional
qualitative research methods may also be insufficient when an overall picture based all
actors, activities and collaborations involved is desired (Morin & Albinsson, 2005).
This study introduces R&E as a mixed methods approach to assess distributed tactical
operations, overcoming some of the short-comings of traditional qualitative and
quantitative methods. Further, the F-REX framework is introduced as a toolset to conduct
R&E of crisis management operations. A case study is presented where R&E and F-REX
were used to conduct multimedia supported after-action reviews at the Swedish Rescue
Services Agency’s (SRSA) training facility in Sandö, 2006.
The R&E approach enables analysts to objectively post-hoc explore a course of events
and determine causality relationships through careful visualization and analysis of
collected data (Morin, 2001). The approach can be separated into two parts, the
Reconstruction process and the Exploration process as outlined in Figure 4.3. The end
state of the reconstruction process can be thought of as a dataset complete enough to
answer predetermined questions, hypotheses and problems. One of the shortcomings of
quantitative methods as described above was the non-ability to cope with unforeseen
events such as breakdowns. Using reconstruction to capture these requires foresight and
thorough planning, and may generate lots of data that might never be useful. In many
cases the ability to do so is restricted by resource availability, thus there is a tradeoff
between completeness of the dataset and cost/complexity to capture it.
The exploration process is conducted post-hoc by analysts using proper tools, such as FREX, to combine the collected data into a time-synchronized model for presentation, or
visual analysis. The process is often iterative as the analysts have the ability to feed data
back into the model during the analysis process thus creating higher levels of data
abstraction until a level is reached where the requested causal relationships can be
determined, or questions answered.
[35]
Reconstruction
Exploration
Metadata
Analysis products
Domain
analysis
Topics
Problems
Priorities
Hypotheses
Modeling
Conceptual model
Instrumental plan
Analysis
Presentation
Data collection
Instrumentation
Data integration
Procedures
Equipment
Software
Feedback
Log files
Data repository
(Revised)
Mission history
Figure 4.3. Reconstruction and Exploration (after Andersson, 2009).
Mission histories, the underlying data model used in F-REX, are split into two parts: one
organizational model which represents participating units, objects and contextual a priori
information in the operations and one data collection model which consists of a
chronological list of captured data samples. The mission history then maintains links
between elements in the two models to allow tracking for instance data that relates to
particular entities or stages in the operation.
4.2.3 CASE STUDY SCENARIO
The event in focus for this case study was a training session of evacuating a burning
house with several injuries and casualties. This scenario is fairly standard to the
organization, yet it is complex enough to highlight the centrality of C2. The primary data
collection nodes were concentrated around the commander trainees, their communication
devices and notable events in the context that affect the ongoing situation.
A dataset was compiled and presented using F-REX to the participating trainees to
discuss the training event in an AAR session. The session was hosted by F-REX
operators together with SRSA instructors. The presentation was used to provide a
baseline around which the trainees could reflect and view the events from a different
perspective. The trainees were encouraged to openly share their reflections on the
training event as well as this debriefing method.
4.2.4 RESULTS
The principal findings of the AAR were (1) the F-REX system is helping trainees recall
details of events in a chaotic environment, (2) R&E is useful for post-hoc clarification of
what orders were actually given during the event, (3) multimedia recordings helped
trainees that were NOT in the role of commander to understand the dilemmas that the
trainee in the role of commander was challenged with, and (4) recorded radio
communication was perceived as the most useful data source to understand and analyze
the course of events.
The participants were generally overwhelmed by the capabilities of this framework and
found it hard to decide what could or should be included or excluded in a debriefing
[36]
method like the one they were subjected to. However, a common perception was that it
was often hard to grasp the massive amount of data that was presented to them. Another
much missed feature was the face-to-face communication between commanders and
subordinates which was captured, but not included in the presentation.
4.3 RECONSTRUCTION AND EXPLORATION OF LARGE-SCALE DISTRIBUTED
TACTICAL OPERATIONS
This case study showed that R&E can indeed be used for large-scale operations, perhaps
even further emphasizing the strengths of the approach as the operation scales up in size
and complexity. However, quite a few notes to consider regarding both technology and
methodology were detected, such as the importance of key indicators or initiated subject
matter experts that can initiate the exploration process. On the technology side, a lesson
to stress is the importance of a priori verification of the data collection infrastructure and
data processing capability to support the magnitude of the dataset being generated. As
size and complexity grows beyond current data mangling capabilities, a trade-off emerges
between data set completeness and time to process data. Another major lesson to report is
the difficulty of getting participants to cooperate in data collection. Recording the
interactions of an individual over a long period of time clearly imposes a privacy
violation, which some participants find so intrusive that they disobey direct instructions.
A priori knowledge of the exercise may guide the researchers to limit the data collection,
possibly increasing acceptance while at the same time making the datasets more
manageable. However, the tradeoff here lies in that creating a data collection boundary
increases the risk of not collecting data of the unforeseen breakdowns which was one of
the main benefits of using the approach in the first place. Careful consideration is thus
needed before reducing the data collection.
4.3.1 OBJECTIVES
The objective of this case study is to determine the applicability of R&E in large-scale
military contexts. In addition, identification of methodological and technical issues of the
application thereof is a high priority.
4.3.2 BACKGROUND
This study introduces R&E to the context of large-scale military C2 exercises. The study
explores benefits, drawbacks, limitations and new requirements for the R&E approach
and supporting tools in the context of large-scale distributed military C2 exercises. R&E
in this context is then compared to some traditional qualitative and quantitative research
methods.
Quantitative research is often based on the belief that is possible to identify general truths
based on rigorous scientific research and controlled experiments. They rely heavily on
objective measurements, statistical procedures, hypothesis testing and validity (Byrne,
2002). Commonly, these methods are implemented as questionnaires or interviews using
pre-defined alternatives or post-hoc grouping of related answers (classification or coding)
to enable statistical analysis. A common approach for the analyst is to look for causality
[37]
by searching for correlation or tendencies between quantified variables to reject or
confirm hypotheses.
In the context of military operations, quantitative research is often less resource
demanding than qualitative or mixed methods research. Attitude surveys can be used to
assess respondents’ situational awareness (Endsley, 1995) of the operation or to identify
general tendencies and attitudes. Some researchers have also tried to use quantitative
methods to measure human performance (e.g., Baron, Kruser & Huey, 1990), albeit not
without controversy. In large-scale contexts attitude questionnaires may indeed be useful
to provide the respondents’ view of the course of events, but they can rarely be used to
explain system breakdowns. In other words, they cannot be used to tell what happened
during an exercise or why, rendering any result de-contextualized and difficult to validate.
Thus quantitative methods are insufficient unless the whats and the whys are already
known.
Qualitative research methods on the other hand provide the ability to explore individual,
social, organizational, and cultural phenomena. The emphasis is put on subjectivity,
sensemaking and interpretation as opposed to a strict set of rules as for their quantitative
counter-parts (Denzin & Lincoln, 2000). The qualitative group of research methods
includes e.g., interview techniques, free-form questionnaires, participatory observation,
and contextual inquiry. When humans and artifacts are the subject of research, cognitive
analysis methods are often used, such as Critical Incident Technique (CIT), Critical
Decision Method (CDM), and Contextual Inquiry (CI).
CIT relies on self-report of incidents relating to satisfaction and efficiency or
dissatisfaction and inefficiency (Flanagan, 1954; Dean, 1998). The technique is used,
e.g., to improve work routines. CDM is a retrospective interview technique with the aim
of getting deeper understanding of incidents than through CIT (Hoffman, Crandall &
Shadbolt, 1998). Finally, CI was developed specifically with the aim of recognizing the
importance of context when interpreting data (Beyer & Holtzblatt, 1999). It is often
implemented as participatory observation of standard work flows and processes followed
by process tracing, e.g., through interviews to make the flows and processes explicit and
comprehensible.
Consequently, quantitative and qualitative methods have their own strengths and
shortcomings. A flexible mixed method approach such as R&E allows the researcher to
overcome some of the shortcomings and tailor the method and make it fit with the
research question at hand. During large-scale military C2 exercises, the workload of a
single operator may shift tremendously over the course of time. As the system is large
and complex with many participants, some at remote and inaccessible locations, pausing
the exercise for probing interviews may not be an option. Similarly, questionnaire
research can be hard to conduct because of the multitude of events that may occur in a
short period of time, reducing the probability that the operators are able to remember and
separate details of each event until the time when they are available to complete a
questionnaire. Hence, although questionnaires can be distributed and answered, the
contextual circumstances may render validity of the collected data questionable.
Participatory observation and contextual inquiry are two methods that do seem to come
close in applicability for large-scale military exercises, with observers being able to trace
[38]
multiple locations and processes over time, to find causality. The main drawback of such
methods is the need for many observers performing in-depth investigations which is both
expensive and hard to coordinate. Even so, the methods still do not solve the problem of
integrating multiple perspectives, i.e., to connect the traces and find inter-trace causality.
The R&E approach takes a wide system perspective by gathering factual, simultaneous
real-time data on multiple organizational levels at multiple geographical locations.
Multimedia tools such as F-REX enable the analyst to follow a timeline over multiple
processes and switch focus post-hoc. Similarly to participatory observation, this method
allows the researchers to trace processes, with the main difference that the researchers
can do this retrospectively and shift focus between processes to determine inter-trace
causality that limits the aforementioned methods. A major critique against R&E
compared to qualitative methods, is that it does not outline an analysis process to guide
the analyst through the process. In many respects, reconstruction often leaves the analyst
with a huge dataset, but no guidance in where to start exploring it.
Retrospection and the ability to quickly get an overview of causality in a complex C2
situation are found to be two of the most important features of R&E which separate the
approach from traditional methods. Together, these abilities enable the analyst to spend
less time figuring out what happened, instead focusing on the more important why.
4.3.3 CASE STUDY SCENARIO
R&E and F-REX were used in case study a large-scale military command exercise with
the Swedish Armed Forces in 2005 to determine its applicability in a real setting. The
exercise involved approximately 300 participants in a networked environment running a
tabletop exercise including a multitude of technical artifacts such as battle simulators and
common operational picture providers. The setting provided an opportunity to test R&E
and F-REX in an environment much larger than anything they had ever been tested in
before. The exercise consisted of four scenarios that were being played over a period of
two weeks. R&E was used during the entire exercise to capture and reconstruct the series
of events, allowing researchers and exercise managers to review the mission history from
different perspectives using the F-REX tool. Concurrently, the exercise was continuously
being evaluated by parallel research groups using traditional qualitative and quantitative
methods. With these parallel groups assessing the exercise, the F-REX/R&E work could
focus on perceived benefits and experienced problems of this approach in the large-scale
setting compared to other methods.
4.3.4 RESULTS
Early on analysis induced a great deal of technical issues that highlighted the technical
difficulty of recording, managing and compiling large datasets. The issues ranged from
data mangling performance to audio transmission breakdowns. To continue the case
study data collection was scaled down after the first day, to include only a selected range
of workstations and operators, as opposed to the previous ambition of capturing all.
Following that, the technical part of the study went smoothly.
Using the R&E approach in such a large-scale operation also introduced methodological
issues, including the fact that many participants disobeyed their instructions to wear
[39]
recording equipment. Over time, the number of participants who blatantly ignored the
instructions kept growing leading to a noticeable amount of data being lost. Arguably a
more severe methodological issue was the perceived difficulty in navigating and
exploring the massive mission histories. Analysts without proper domain knowledge or
exercise insight had a hard time trying to explore the mission history and produce any
valuable findings. This was partly solved later on by using observer notes from 20
experienced subject matter experts who reported on team performance at chosen
locations in the exercise.
R&E was successfully able to provide the root cause of several mistakes and confusing
situations. As these findings were reported during the AARs, military personnel and
exercise management started to pay more and more attention to the R&E approach. At
the last couple of days, exercise managers where directly using the mission history to
search for root causes shortly after they experienced a problem. With the assistance of
experienced F-REX operators, they were able to clarify obscurities and find explanations
to exercise-related problems that they encountered.
4.4 PRIVACY AND DISTRIBUTED TACTICAL OPERATIONS EVALUATION
Evaluation of distributed tactical operation using after-action reviews, or mixed methods
such as R&E, may undoubtedly cause privacy infringements for the evaluated. The
magnitude of these infringements cannot always be determined a priori, which makes
them difficult to defend against. Evaluations, however, are necessary for organization to
evolve and improve, which in the case of emergency management may imply that our
society becomes better protected.
In an ideal world the evaluators always strive to get continuous informed consent from
participants in any operation that is being evaluated in an attempt to minimize the
negative effects. In reality, however, this is difficult to achieve because the evaluators
may not always be able to predict what information will be collected and how it can and
will be used. Accepting minor privacy infringements may thus be required for the greater
good, and should be expected by employees working in these organizations. Care must be
taken though so that the negative effects do not become overwhelming and render the
evaluation counter-productive, e.g., by reducing the teams’ level of creativity and
autonomy. Finding methods to minimize the magnitude of privacy infringements and
their effects is subject to future work.
4.4.1 OBJECTIVE
This study was performed as a literature review with the objective of identifying why
some C2 trainees feel uneasy about data collection on their actions, and what ethical
issues to consider before employing too comprehensive information gathering in the
context of R&E.
4.4.2 BACKGROUND
Aiello and Klob (1996) proposed a definition of privacy as “the ability for an individual
to control the use of their own personal data, wherever it might be recorded”. From a
strict interpretation of that definition, a privacy infringement occurs whenever data is
[40]
collected about persons without their knowledge, as well as when the persons are unable
to stop collection or erase earlier collected data. Evaluation of distributed tactical
operations through methods such as R&E, involves a large amount of data collection on
individuals and their actions. Although the intent is seldom or never to collect any
personal data or assess individual performance, recording a teams’ actions without
recording its’ members, and thereby collecting personal data, is virtually impossible.
4.4.3 RESULTS
The most important result of this study is the awareness that R&E of distributed tactical
operations may cause privacy infringements. In many respects the data collection for
reconstruction is similar to workplace surveillance, which has been shown to impose
negative effects on the workers (Kizza & Ssanyu, 2005), such as:
 lack of trust between workers, supervisors, and management,
 stress and anxiety,
 repetitive strain injuries because of refraining from taking breaks,
 lack of individual creativity,
 reduced or no peer social support,
 lack of self-esteem,
 worker alienation,
 lack of communication, and
 psychological effects.
It can certainly be argued that the purpose of evaluating tactical operations is different
from that of workplace monitoring, and that operators agreeing with the motives may
invalidate some of the above effects, i.e., the lack of trust issue might not be as imminent.
However, it is reasonable to suspect that stress and anxiety may be an outcome from
knowing that every action and decision being made is recorded and evaluated. The same
holds true for at least lack of creativity, lack of self-esteem and lack of communication.
The latter of which has been clearly shown by operators removing or turning off
recording equipment to enable themselves to speak freely with co-workers (see Paper
III).
Palm (2007) noticed that technology enables employers to shift performance monitoring
towards individuals as opposed to teams which used to be the case. The same hold true
for evaluation of tactical operations. Gadgets, such as voice recorders, GPS trackers, and
video cameras, become smaller, cheaper, and more accessible – allowing evaluators to
collect more data so easily that they may not always reflect upon whether the data is
actually needed or not. This can be problematic since excessive data collection may
negatively affect the unit of observation, and doing so without reason should thus be
strongly discouraged. The main risks of continuous and systematic collection of personal
data are unavoidability, continuity, dependency, and identifiability (ibid.). The first of
these four points can be remedied by giving all the participants in the operation subject to
evaluation the right to decline monitoring. Such a remedy, however, does not come
[41]
without complication; a consequence may for instance be that the declining person is
temporarily relieved from the team. Will opting out increase the risk of system
breakdowns at this, or a future, operation? In the end, is there a risk that the consequence
of tactical operations’ evaluation is decreased performance?
From a contractarian point of view, the dependency relationship between employers and
employees might be seen as a contract where the employees give up some fundamental
rights to privacy by accepting the job offer, especially so in the public sector
organizations where tax payers have a reasonable right to demand that their tax money is
used for greater good. One may argue then that monitoring is motivated for the greater
good and that the employee has consented. Such reasoning however, leads to the
immediate question, was the consent informed?
Informed consent can be defined as “voluntary consent based on adequate understanding
of relevant facts” (Malek, 2005). For consent to classify as informed the subject must be
able and allowed to make a choice while understanding all information there is to know
about the consent. In the above scenario, consent would not be qualified as informed if,
for instance, the subject is economically dependent on the job. Another complication is
the fact that the subjects may not be able to comprehend what information can be
retrieved from the collected data when combined. It may not even be apparent to the
evaluators on beforehand, making it difficult to assure that the consent is informed.
Mechanisms are therefore needed to enable revocation of said consent.
As often is the case in ethical questions, this is a case of Ross’ pluralism with a trade-off
between the duty of beneficence and duty of non-maleficence (Ross, 1930). As an
example, how can the society demand that a human being enters a nuclear reactor to stop
a meltdown? The risks for that person are obviously tremendously high, but if nothing is
done a calamity is bound to happen. What is the moral thing to do?
To some extent it may be reasonable to put forward a utilitarian argument that the society
has the right to demand privacy infringements from public-service employees and
continuous assessment of their readiness level, but care must be taken so that systematic
evaluation does not become counterproductive and decrease their performance. It is a
disturbing and contradictive thought that the price the society pays for a higher perceived
level of safety may in fact be a decrease of the same.
[42]
Chapter 5
RESULTS
Chapter 5 presents the results of this thesis related to research questions RQ1-3:
RQ 1. What abilities are important for commanders, to cope
with dynamic decision-making situations that arise in tactical
military and emergency management operations?
RQ 2. How should commanders’ mission experience be
externalized in order to enable vicarious learning from
tactical military and emergency management operations?
RQ 3. What technical, methodological, and ethical issues need
to be considered when externalizing mission experience from
tactical military and emergency management commanders?
The first research question, RQ1, is answered in section 5.1 by summarizing a literature
review (Paper I) of tactical missions, explicitly focusing on abilities needed to cope with
dynamic situations.
RQ2 is addressed in section 5.2, with results building extracted from the two case studies
found in Paper II and III, to illustrate how mission experience should be externalized.
Finally, a set of issues have been identified in the case studies (Paper II and III) and the
privacy study (Paper IV) to address RQ3 based on self-experienced problems from R&E
of C2 as reported in section 5.3.
5.1 WHAT ABILITIES ARE IMPORTANT FOR COMMANDERS, TO COPE WITH
DYNAMIC DECISION-MAKING SITUATIONS THAT ARISE IN TACTICAL
MILITARY AND EMERGENCY MANAGEMENT OPERATIONS?
Investigating what mission experience is requires understanding of how the subject
perceives a mission, and what knowledge is acquired from it, both tacit and explicit. The
study in Paper I approaches this task through a narrative literature review on tactical C2,
investigating literature on C2 in dynamic situations. Studying this topic is complex, partly
because it is multi-disciplinary, and different disciplines employ different vocabularies
and frames of reference. To get a broad overview of the topic, the literature review
includes samples from the diverse fields such as C2, cognitive engineering, resilience
engineering, knowledge management, and decision making.
The missions that emergency management and military teams conduct, span over a broad
spectrum of operation types. Normally, organizations have routines to practice standard
and familiar scenarios. Often, however, these organizations operate in environments that
are dynamic and complex, where infrequent and unforeseen situations occur (Mendonça
& Hu, 2009).
[43]
Teams that operate under the above conditions can be classified as adhocracies
(Mintzberg, 1979). The infrequent and unforeseen events are critical to organizations, and
society may depend on them being resolved. The nature of such events makes them
difficult to exercise and manage with standard operating procedures. Under these
circumstances, time-pressure is often a factor, ruling out slow analytical decision-making
processes. The RPDM model shows a thought process where decision makers rely on
prior knowledge, or experience, to make decisions (Klein, 1993). The experience,
according to this model, guides the decision maker by allowing recognition of familiar
situations. From a commander perspective, the recognition process thus motivates the
need for experience, as it improves their ability of making rapid decisions in dynamic
time-pressured, and potentially life-threatening, situations.
The literature review in Paper I takes off from the claim that one of the most important
abilities to successfully control an unforeseen event is innovation (Mintzberg & McHugh,
1985). An investigation into referenced literature and dictionaries shows that innovation
is closely linked to creativity, but also that other researchers suggest instead adaptability
and flexibility as key enablers for successful performance in dynamic environments.
Further reading introduces two more concepts that seem to dominate literature on control
of dynamic situations: improvisation and resilience. The review has thus revealed six
abilities that organizations employ to cope with dynamic situations, although a dictionary
lookup show that they are not easy to separate, see Table 5.1. While there are nuances
between these concepts, the reviewed articles do not use these nuances to separate
between them. Some authors even use terms such as flexibility and adaptability
interchangeably whereas some differentiate between them, making it hard to relate two
separate pieces of work to each other.
Table 5.1. Selected terms related to managing changing environments and conditions,
from (a) The American Heritage Dictionary of the English Language, (b) Collins
English Dictionary – Complete and Unabridged, or (c) WordNet 3.0 (from Paper
I).
Concept
Explanation
Synonyms
Adaptability
ability of becoming suitable to a particular
situationc
variableb, versatileb, changeableb
Flexibility
responsiveness to changea
adaptablea, resilientb, elastica
Resilience
the ability to recover quickly from illness,
change, or misfortunea
buoyancya, flexibilityb,
adaptabilityb
Improvisation
to invent, perform or compose something
extemporaneouslya
extemporizeb, ad-libb, inventb, get
byc, dealc
Innovation
act of introducing something newa
inventionc, conceptionc, creativityc
Creativity
the ability to create
a
productivityb, innovationc
The study in Paper I carefully investigates the use of each of the six terms (FAIRIC) to
determine what the nuances really mean, how each of the abilities should be interpreted,
and in what situations they are needed. The FAIRIC model lays them out on two
[44]
dimensions: unfamiliarity of the situation, and proactiveness. The idea behind the model
is that these two dimensions represent the two most important differences that separate
the FAIRIC abilities (consider e.g. the dictionary explanations of responsiveness to
change vs. ability of becoming suitable to a particular situation). The six abilities can be
mapped onto the two dimensions as shown earlier in Figure 4.1. The abilities on the lefthand side represent a lower degree of proactiveness compared to those on the right, e.g.,
innovation can be employed in advance to prepare for new situations, whereas
improvisation has an element of time pressure in it, meaning it is primarily employed
reactively in dynamic situations as a response to the unfolding events. The unfamiliarity
dimension shows that improvisation and innovation is employed primarily in unfamiliar
situations, whereas more familiar situations can be responded to by having a high degree
of adaptability (e.g., adapting the organization or procedure in response to change, i.e.,
reactive) or by implementing flexibility (e.g., having a flexible organization or process
that allows solving of multiple events, i.e., proactive). The six FAIRIC abilities should be
considered separate, yet related, and usable in different scenarios. The reviewed literature
has identified all these six abilities as important, although the magnitude of their
importance has not been compared.
To summarize, mission experience increases the ability of creating better recognitionprimed decisions through an increased knowledge base of situations. The study in Paper I
showed that this increased ability is linked to the six FAIRIC abilities, which are an
important, yet non-comprehensive, set of experience-guided abilities that tactical military
and emergency management organizations employ in order to cope with dynamic
situations. Increased mission experience does thus improve the commanders’ ability of
performing opportunistic control through the reactive abilities of adaptability, resilience
and improvisation. These abilities are guided by recognition, a process which seems
dependent on mission experience.
In dynamic situations, tactical control is preferred over opportunistic, requiring more
proactive measures, such as preparation and projection. Preparing for the entire spectra of
scenarios that military and emergency management organizations may face is impossible.
However, knowledge and experience increase the scope of tactical scenarios these
organizations can manage by stimulating creativity, guiding innovation, and increasing
flexibility.
5.2 HOW SHOULD COMMANDERS’ MISSION EXPERIENCE BE EXTERNALIZED
IN ORDER TO ENABLE VICARIOUS LEARNING FROM TACTICAL MILITARY
AND EMERGENCY MANAGEMENT OPERATIONS?
In the case studies of both Paper II and Paper III, reconstruction is used to compile
mission histories of the studied exercises. Both studies showed that partial reconstruction
can be used for dissemination to help participants recall what happened during each
exercise session. As such, R&E has been shown valuable to stimulate recollection during
after-action review by presenting an undisputable ground truth to focus discussions. The
Sandö case study (Paper II) additionally showed that non-participating trainees can use
R&E to internalize knowledge similar to that experience acquired by those who
participated in the training, i.e., vicarious learning.
[45]
The compiled mission histories enclosed a massive amount of data concerning the
decision making processes. In the Sandö case study, most data capturing was focused on
the commander, whereas in the Armed Forces case study (Paper III) a representative
collection of key individuals was selected for inclusion in the data collection phase of the
reconstruction. The captured data in both cases constitute a significant set of decisionmaking cues, such as what the trainees see and hear, and how they interact with the
environment and the team. Arguably the most important cues during tactical operations
are communication; in fact C2 depends on the continuity of communication across
echelon (Alberts & Hayes, 2003). This is also supported by findings from both case
studies, as communication was claimed to be the most useful for understanding the
ongoing situation. Two other aspects that were highlighted by trainees in the Sandö case
study are time synchronization and objectivity, i.e., the respondents particularly
appreciated the ability to get an ordered list of factual data that could not be disputed, as
provided by the model presented in Figure 2.7. Such a list helped the trainees recall the
events in order of occurrence and retrospectively reflect upon cause and effect
relationships.
Other data sources, such as GPS tracks, observer notes, photos, and video, were
considered secondary, but useful for navigation. For the visual cues, the reason may be
because most of the respondents participated in the exercise and already had a clear
picture of the context that these data sources could provide. From the case studies, it is
hard to generalize from this data whether or not such data sources have any value for
vicarious learning. However, it can be assumed from the findings of e.g. Liu and
colleagues (2009) that a combination of high media richness data such as video and photo
are useful in combination with other data if the technology acceptance is high enough
among the students. The Swedish Armed Forces case study highlights the importance of
observer notes, as they can guide the exploration phase of an otherwise potentially
overwhelming mission history. To summarize, the following externalization findings
were generated from the reconstruction of mission histories during the Sandö and the
Swedish Armed Forces case studies11:
 Recorded voice communication, both face-to-face and through radio, is
considered the most important to analyze and understand the decision-making
process of commanders, and is therefore vital for the transferal of mission
experience (Paper II).
 Triaging logs are useful for mission efficiency analysis, but provide none, or
little, additional value for transferal of mission experience (Paper II).
 Photos and videos help trainees remember the chaotic events that occurred during
a mission (Paper II). It is expected that they provide additional value for the
purpose of mission experience transfer.
 Screen captures were not shown useful for evaluation of virtual training sessions
(Paper III). Their value for mission experience transfer is unknown.
11
It should be noted that these findings may be highly contextual, and not generalizable to other tactical
scenarios, e.g., with sparse communication.
[46]
 GPS tracks and observer notes are useful to help understand the exploration of
mission histories, and may therefore be valuable to guide the mission experience
transfer process (Paper II).
Other important lessons from the case studies include the stressing of time
synchronization and objectivity as necessary to enable causal analysis. Maintaining
proper time stamping of all data is thus a necessity to ensure intact ordering of the events
during exploration. It is also advisory to refrain from inputting subjective data into the
dataset, as the objectivity may then be violated.
5.3 WHAT TECHNICAL, METHODOLOGICAL, AND ETHICAL ISSUES NEED TO
BE CONSIDERED WHEN EXTERNALIZING MISSION EXPERIENCE FROM
TACTICAL MILITARY AND EMERGENCY MANAGEMENT COMMANDERS?
The case studies from Sandö (Paper II) and the Swedish Armed Forces (Paper III) show
that there are indeed a number of technical and methodological issues that must be
considered when capturing information in the operative environment of a tactical team.
One of the biggest issues identified from the case studies is the difficulty in managing the
massive datasets that were generated during the exercises, leaving the analysts with large
amounts of data but without the ability to explore and analyze it. While there are certainly
methods to deal with larger datasets, e.g., through more computational power; visual
exploration and human processing is often necessary to make sense of reconstructed
mission histories. A set of indicators of what data to process may help analysts reduce the
dataset to explore by filtering the streams as shown in Figure 5.1, where indicators I1 and
I2 are used with a time window (yellow) to reduce the synchronized data set consisting of
two video streams and one audio stream in the top left image (a) to the much smaller set
in the bottom right (e), leaving the analyst with less data to explore, thereby reducing the
workload. Such indicators can be generated, e.g., by observer notes as in the Sandö study
(Paper III) and by using data fusion techniques through which computers preprocess the
dataset to find anomalies, outliers and interesting patterns to guide the analysts.
a)
b)
d)
c)
e)
Figure 5.1. Indicators help analysts find interesting data points in massive data sets,
thereby greatly reducing the data set, exemplified by moving through the outlined
sequence of reducing the dataset in (a) to that in (e) through the intermediate
steps (b), (c), and (d).
[47]
The Sandö study (Paper II) showed that another methodological problem associated with
reconstruction of mission experience is the fact that the exploration process may
overwhelm the audience. Creating methods and tools to support internalization of mission
experience is a difficult task which may possibly draw from the fields of data
visualization and interactive learning.
Assuming that decision-making cues are made up of stimuli to the five human senses,
capturing them is a matter of recording and synthesizing these stimuli. Sensor systems
can do a great job of synthesizing these signals in almost any environment. However, it is
always a challenge to setup the sensor systems in such a way that they indeed capture the
inputs that the team members are exposed to, and to separate them from stimuli that the
team members do not acknowledge. The internal sensemaking and decision-making
processes that occur within each individual is also difficult to capture without invasive
measurements, making it hard to realize exactly which stimuli that really led to each
particular decision. For practical reasons, these processes are omitted from the model and
the synthesis.
The proposed approach for synthesizing mission experience into something that can be
transferred through visual and analytical exploration is based on massive data collection
from the team, operation and the environment. Conducting such data collection without
violating any individuals’ privacy is another major challenge as discussed in Paper IV.
The privacy issue does not only concern the mental health of the team members, but also
the society as a whole when there is an imminent risk that the organization’s ability to
perform decreases from team members loosing self-esteem and autonomy as a
consequence of privacy infringements. The privacy issue is connected to the question of
informed consent, which has been identified as problematic because of the complexity of
making consent truly informed as well as the complexity of maintaining informed
consent when circumstances change, such as personal data is accidently captured,
mission objective changes, and unexpected findings are revealed during exploration.
In summary, the main technical (T), methodological (M), and ethical (E) issues that have
been identified for capturing and modeling of tactical mission experience are:
 difficulty in managing massive datasets [T, M],
 difficulty in navigating/exploring massive datasets without indicators [M],
 information overload overwhelming explorers [T, M],
 replaying decision making cues other than auditory and visual [T],
 coupling stimuli to decision making [M],
 avoiding privacy infringements [E], and
 maintaining continuous informed consent [E].
[48]
Chapter 6
DISCUSSION
This section discusses how mission experience can be transferred, and what role R&E
plays for the externalization/internalization process. The summarized discussion is
followed by a brief introduction on proposed future work, building on the findings from
the studies presented in this thesis. Finally, the conclusions related to the objective, as
stated in the introduction of this thesis, are presented.
6.1 TACTICAL COMMAND AND CONTROL IN DYNAMIC SITUATIONS
Existing models of C2, decision making and sensemaking all provide support to
understanding what mission experience really is, how it can be modeled, and how it can
be captured. They show that controllers in a tactical operation need an objective, shared
understanding and the ability to observe and make sense of an ongoing situation.
However, in reality objectives are often ill-defined and sometimes conflicting (Klein,
Ross, Moon, Klein, Hoffman & Hollnagel, 2003), and time and resources may be lacking
to properly analyze the situation to make sense of it. Tactical control mode can still be
applied if the situations are somewhat familiar and standard operating procedures (SOPs)
have been defined. However, a tactical organization may be poorly equipped to deal with
infrequent events for which they have not been trained (Harbom, 2010). What is left then
is the reliance upon experience and prior knowledge, i.e., going into opportunistic control
mode. In these situations, decision making can be regarded as recognition-primed, which
is highly dependent on the prior existence of a knowledge base.
The models reviewed in this thesis highlight that controlling tactical missions are above
all a matter of sensing the dynamic environment, adapting the JCS, and responding as the
situation requires, often as a result of employing at least one of the six FAIRIC abilities.
The experience gained from the mission is just that, i.e., the ability to sense and make
sense of a scenario such as the one that just unfolded, the ability to assess and understand
the phenomena that occurred, and the ability to counter it with appropriate measures.
Having such experience improves the likelihood of a tactical unit being able to stay in
tactical control mode during future events, thus enabling them to perform better.
6.2 TRANSFERAL OF MISSION EXPERIENCE
“The importance of context and history means that there is no substitute for experience”
(Cilliers, 2000, p.12). Certainly this thesis does not dispute Cilliers’ argument. The main
point is that experience of others can be used to stimulate creation of new tacit knowledge
on the recipient side. This knowledge can never substitute mission experience, but it can
complement, and it is a much better substitute than the alternative of having no
knowledge at all about a certain type of mission or operation.
The SECI model for knowledge transfer seems applicable in a mission-experience
context, telling us that knowledge transfer is not about moving or copying knowledge,
but rather about the generation of new knowledge. Internalizing a dataset captured from a
[49]
tactical operation will thus stimulate creation of new knowledge as opposed to replicating
the knowledge that the studied commander or team member gained. If this knowledge
can be used for recognition and to make better future decisions, then this approach opens
up new ways for organizations to manage their memory from infrequent events, possibly
addressing the challenge put forward by Means et al. (1993): to provide a classroomversion of the mission history, complete with characteristic cues and tasks to study for the
trainee.
Consider for instance the Fukushima incident in 2011 where an earthquake triggered both
a nuclear accident and a tsunami. This chain of events had not been anticipated and no
organization was fully prepared for it. It is reasonable to believe that disaster response
organizations worldwide will go through several iterations of staff turnover before an
incident like this happens again, if ever. It is also reasonable to believe that the managers
that actually did handle this very incident learned something from the events, i.e., they
acquired tacit knowledge, and experience. Making this knowledge explicit is, according
to the SECI model, the only method through which it can be made persistent and shared
over boundaries of time and space. A common approach to externalize such knowledge in
tactical organizations is the manual approach, i.e., to design a procedure or create a
manual of how to handle incidents, based on one or several stereotypical situations. Often
these manuals are written by someone outside the team with the experience, and thus the
accuracy of the manual is subject to the quality of an interview and the ability of the
interviewer to extract information from the interviewee.
Mixed methods approaches such as R&E are superior for externalization of knowledge in
the sense that they can extract information that subjects are not conscious of, while also
being able to include traditional data sources such as interviews as integrated components
of an externalized mission history. The interpretation of it all, however, is left to the
explorer, leaving a larger cognitive burden for anyone internalizing a multimedia-heavy
mission history compared to studying a written manual. Hence, the method cannot
replace existing methods, but should rather be seen as a complement to them.
With proper experience, a tactical unit can improve flexibility and adaptability, thus
reducing the amount of unfamiliar situations, however, mission experience may also be
valuable to improve improvisation and innovation, making the organizations more
capable of handling those events that cannot be foreseen. Ultimately, an increased arsenal
of internalized mission histories will improve decision making, rendering the tactical unit
more efficient. To summarize, knowledge transfer through externalization and reinternalization adds a new method, unrestricted by boundaries of time and space, through
which an organization may boost its’ ability to perform.
6.3 RECONSTRUCTION AND EXPLORATION
The two presented case studies show that R&E as a mixed methods approach is capable
of capturing and propagating auditory and visual cues in decision making scenarios. The
studies employ the approach in one emergency response and one large-scale military
setting. R&E as an approach for modeling exercises and experiments for C2 analysis and
AAR has been conceptually verified in several field trials (e.g., Thorstensson et al., 1999;
Thorstensson, Johansson, Andersson & Albinsson, 2007; Andersson et al., 2011). The
[50]
only data that point to its applicability for transferring mission experience are the Sandö
case study, in which several trainees commented that they saw value in internalizing
mission history to sense some of what the subject commander was exposed to (see Paper
II) and a survey from the Alvesta field trial (Jenvald et al., 2000). Still, these data points
encourage future attempts to investigate the concept of mission experience transferal
through R&E.
A key to successfully being able to share the experience is that as much as possible of the
operation is synthesized in such a way that the learners can relive the experience and
understand the cues and dilemmas that the team members were facing during the actual
operation. In most tactical scenarios, the information that needs to be synthesized is
composed of visual and auditory cues, decisions, and actions, plus the context under
which the operation is executed. Capturing the cues, decisions, and actions in a holistic
sense is quite straight-forward using sensor systems in the operative environment and on
the team. This works well until the moment comes when the captured artifacts must be
coupled, i.e., deciding which cue led to one particular decision and set of actions, and
vice versa.
Quantification is useful to generate indices and indicators that can assist exploration and
learning. However, the decision making cues and context must be propagated to the
recipients for them to truly understand the reasons and motives for decisions and actions,
and to get a sensation of experiencing the reconstructed scenarios. Qualitative data seems
more appropriate to propagate such cues, since the underlying mental processes cannot be
captured and are not fully understood. Thus, quantitative and qualitative methods
complement each other in this setting, allowing better completeness (Venkatesh et al.,
2013) which might be an essential property if a reconstructed mission history is to be
embraced and internalized.
6.4 PRIVACY
Capturing datasets to reconstruct mission experience introduce a privacy infringement
concern, which must always be kept in mind to reduce the risk of unwanted mental
effects. At a minimum, informed consent is needed from all participants subject to
monitoring and data capture. This consent should also be kept continuously informed,
i.e., any individual must have the right to know if the conditions under which the consent
was given are going to be violated, and always have the ability to withdraw the consent.
Taking such precautionary measures is not as easy as it may sound, since it can be
difficult to predict what data can and will be used for at the time consent is requested. If it
would have been easy to determine when an update of the consent is needed, then privacy
would probably not be a complicated issue; unfortunately that is not always the case.
In pursuit of progress it seems inevitable that tactical operations are evaluated and that
organizations try to learn and remember from both live operations and exercised. The
monitoring and continuous assessment processes can create a sensation of a “big brother”
environment (consider Orwell’s classic novel 1984). This thesis concludes that exercise
and assessment managers must acknowledge the problem and become aware of the
potential privacy infringements that assessment introduce. Doing so enables them to have
open discussions and creates a climate where everyone is aware of the consequences for
[51]
themselves, making it possible for them to give consent that can be classified as truly
informed.
6.5 SUMMARY
R&E is a mixed methods approach for evaluation of tactical operations. As showcased in
this thesis it can be applied to create explicit models of the experience that teams acquire
through tactical operations. The primary purpose of such explicit models is to serve as a
baseline for immediate after-action review. However, the Sandö case study showed that
the models can also be seen as externalized experience which other trainees can study and
internalize through interactive exploration. While this result has not been validated, it is
interesting enough to promote so that future research may find alternative and cheaper
methods of sharing rare mission experience.
Internalizing such a mission history can complement regular training and exercising, and
yield an alternative for organizations that wish to maintain knowledge from rare events;
while at the same time opening up the ability to share these explicit products, to enable
transferal of mission experience within and between organizations. A final word of
warning is that R&E should never replace training and exercising, but that it may
beneficially complement it.
6.6 FUTURE WORK
The most obvious need for future work is that of verifying the R&E’s usefulness as a
mediator for experience transfer through empirical studies. Ideally, such studies should
be longitudinal and experimental to explore a teams’ level of experience as it changes
over time and compare that to teams who train the same scenarios in a more traditional
manner, i.e., measuring the dependent variable derivative of a teams’ effectiveness as a
function of the independent variable internalization method.
Furthermore, the technical and methodological issues that have been identified in this
thesis all need to be resolved in order for the proposed approach to become truly usable
by the tactical organizations. Fortunately, the areas of big data and visual exploration
receive plenty of attention in the research community today, yielding reason to hope that
the future holds solutions that may be adoptable for R&E, possibly countering the
identified data management, navigation, and information overload issues. Research is
also being conducted on reproduction of tactile and olfactory stimuli, although it is not
clear what effects this will have on learning, nor what it will do to technology acceptance
among the intended trainees.
Finally, further investigations, e.g., through interviews and experiments, need to be
conducted in order to resolve the ethical matters that have been raised in this thesis, i.e.,
primarily how to avoid privacy infringements and maintain continuous informed consent.
6.7 CONCLUSIONS
This work presents a new approach to understand how experience affects performance in
tactical teams. The FAIRIC model shows six abilities which are all guided by experience
and can improve the team’s ability to stay in tactical or at least opportunistic control
mode in a dynamic situation. The model may be used as a unified vocabulary to guide
[52]
future research on dynamic decision making in tactical military and emergency
management operations.
Data from the Sandö case study indicates that externalizing mission experience through
reconstruction allows vicarious learning through exploration. To successfully accomplish
this, the product of the externalization must be time synchronized and objective. A
mission history implementing this also need to include the major cues for decision
making, in the Sandö case study identified as communication.
Both case studies lack a proper investigation into what data sources are most needed to
enable internalization if the exploring subject has not participated in the exercise. The
findings are thus limited, but other research has shown that a combination of high and
low media richness outperforms any single data source for learning (Liu et al., 2009),
leading to the conclusion that media such as GPS tracks and system logs provide a useful
baseline together with more high media richness data such as video and photos to provide
a context and understanding of the unfolding events.
Furthermore, several issues have been identified, methodological, technical and ethical.
The issues range from difficulties in managing and navigating big data, to avoiding
information overload among explorers of the mission history. Arguably, the most severe
issue of all is to overcome the challenges of maintaining continuous informed consent to
avoid privacy infringement. The identification of these issues raises awareness so that
precautionary measures can be initiated to successfully avoid running into these problems
when externalizing mission experience from tactical military and emergency
management operations.
[53]
[This page is intentionally left blank]
[54]
REFERENCES
Abecker, A., & Decker, S. (1999). Organizational Memory: Knowledge Acquisition,
Integration, and Retrieval Issues. Lecture Notes in Computer Science, 1570/1999,
113–124.
Aiello, J. R., & Klob, K. J. (1996). Electronic Performance Monitoring: A Risk Factor for
Workplace Monitoring. In S. L. Slater & L. R. Murphy (Eds.), Organizational Risk
Factors and Job Stress (pp. 163–179). American Psychological Association.
Alberts, D. S., & Hayes, R. E. (Eds.). (2003). Power to the Edge. Washington, DC: DoD
Command and Control Research Program.
Alberts, D. S., & Hayes, R. E. (Eds.). (2006). Understanding Command and Control.
Washington, DC: DoD Command and Control Research Program.
Alberts, D. S., & Hayes, R. E. (Eds.). (2007). Planning: Complex Endeavors.
Washington, DC: DoD Command and Control Research Program.
Albinsson, P.-A., Wong, W., Pilemalm, S., & Morin, M. (2005). Using a ReconstructionExploration approach to Empower Cognitive Analysis Methods: Possibilities and
Challenges. Proceedings of the 49th Annual Meeting of the Human Factors and
Ergonomics Society (pp. 224–228). Orlando, FL.
Andersson, D. (2009). F-REX: Event-Driven Synchronized Multimedia Model
Visualization. Proceedings of the 15th International Conference on Distributed
Multimedia Systems (pp. 140–145). Redwood City, CA: Knowledge Systems
Institute.
Andersson, D. (2012). F-REX anpassad för IT-försvarsövningar i NCS3 (FOI Memo
4336) [eng: F-REX configured for IT defense exercises]. Linköping, Sweden:
Swedish Defense Research Agency.
Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011). Analysis
of a Cyber Defense Exercise using Exploratory Sequential Data Analysis.
Proceedings of the 16th International Command and Control Research and
Technology Symposium. Québec City, Canada: DoD Command and Control
Research Program.
Andersson, D., Mullins, S., Bordetsky, A., Bourakov, E., & Forsgren, R. (2011). MIO
Experiment Data Capture. Proceedings of the 10th International Conference on
Telecommunication Systems: Modeling and Analysis. Prague, Czech Republic.
Argote, L., & Miron-Spektor, E. (2011). Organizational Learning: From Experience to
Knowledge. Organization Science, 22(5), 1123–1137.
[55]
Ashby, W. R. (1956). An Introduction to Cybernetics. London, UK: Chapman and Hall.
Baron, S., Kruser, D. S., & Huey, B. M. (Eds.). (1990). Quantitative Modeling of Human
Performance in Complex, Dynamic Systems. Washington, DC: National Academy.
Bernard, H. R. (2011). Research methods in antropology: qualitative and quantitative
approaches (5th ed.). Plymouth, UK: AltaMira.
Beyer, H., & Holtzblatt, K. (1999). Contextual Design. Interactions, 6(1), 32–42.
Blandford, A., & William Wong, B. L. (2004). Situation awareness in emergency
medical dispatch. International Journal of Human-Computer Studies, 61(4), 421–
452.
Boyd, J. R. (1987). A discourse on winning and loosing. Air University Library
Document No. M-U 43947. Retrieved March 5, 2013, from http://dnipogo.org/johnr-boyd/
Brehmer, B. (1992). Dynamic decision making: Human control of complex systems. Acta
Psychologica, 81(3), 211–241.
Brehmer, B. (2000). Dynamic Decision Making in Command and Control. In C. McCann
& R. Pigeau (Eds.), The Human in Command: Exploring the Modern Military
Experience (pp. 233–248). New York, NY: Kluwer Academic/Plenum Publishers.
Retrieved December 3, 2012, from
http://www.nada.kth.se/kurser/kth/2D1416/litteratur/Artman.pdf
Brehmer, B. (2005). The Dynamic OODA Loop: Amalgamating Boyd’s OODA Loop
and the Cybernetic Approach to Command and Control. Proceedings of the 10th
International Command and Control Research and Technology Symposium.
McLean, VA: DoD Command and Control Research Program.
Brehmer, B. (2009). Command without Commanders. Proceedings of the 14th
International Command and Control Research and Technology Symposium.
Washington, DC: DoD Command and Control Research Program.
Breton, R., & Rousseau, R. (2008). The M-OODA: Modelling of the OODA loop as a
modular functional system (TM 2008-130). Valcartier, Canada: Defense Research
and Development Canada.
Builder, C. H., Bankes, S. C., & Nordin, R. (1999). Command Concepts: A theory
derived from the practice of Command and Control (MR-775-OSD). Santa Monica,
CA: RAND.
Cilliers, P. (2000). Knowledge, Complexity, and Understanding. Emergence, 2(4), 7–13.
[56]
Cronin, P., Ryan, F., & Coughlan, M. (2008). Undertaking a literature review: a step-bystep approach. British Journal of Nursing, 17(1), 38–43. Retrieved March 19, 2013,
from http://www.ncbi.nlm.nih.gov/pubmed/18399395
Daft, R. L., & Lengel, R. H. (1986). Organizational Information Requirements, Media
Richness and Structural Design. Management Science, 32(5), 554–571.
Dean, P. J. (1998). A qualitative method of assessment and analysis for changing the
organizational culture. Performance improvement, 37(2), 14–23.
Denzin, N. K., & Lincoln, Y. S. (Eds.). (2000). Handbook of Qualitative Research (2nd
ed.). Thousand Oaks, CA: SAGE.
Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems.
Human Factors, 37(1), 32–64.
Endsley, M. R. (2000). Direct Measurement of Situation Awareness: Validity and Use of
SAGAT. In M. R. Endsley & D. J. Garland (Eds.), Situation Awareness Analysis
and Measurement. Mahwah, NJ: Lawrence Erlbaum.
Endsley, M. R. (2004). Situation awareness: Progress and Directions. In S. P. Banbury &
S. Tremblay (Eds.), A Cognitive Approach to Situation Awareness: Theory and
Application (pp. 317–241). Aldershot, UK: Ashgate.
Essens, P., Vogelaar, A., Mylle, J., Blendell, C., Paris, C., Halpin, S., & Baranski, J.
(2005). Military Command Team Effectiveness: Model and Instrument for
Assessment and Improvement (RTO-TR-HFM-087). Neuilly-sur-Seine Cedex,
France: NATO Research and Technology Organisation.
Finley, D. L., Muckler, F. A., Gainer, C. A., & Obermayer, R. W. (1975). An analysis
and evaluation methodology for Command and Control: Final technical report (p.
96). Northridge, CA: Manned Systems Science.
Flanagan, J. C. (1954). The critical incident technique. Psychological bulletin, 51(4),
327–358. Retrieved March 21, 2013, from
http://www.ncbi.nlm.nih.gov/pubmed/13177800
Fredholm, L., & Åström, M. (2010). Incident command and decision making. In L.
Fredholm & A.-L. Göransson (Eds.), Emergency Response Management in Today’s
Complex Society (pp. 109–122). Karlstad, Sweden: Swedish Civil Contingencies
Agency.
Frey, J. H., & Fontana, A. (1991). The group interview in social research. Social Science
Journal, 28(2), 175–188.
[57]
Försvarsmakten. (2011). Militärstrategisk doktrin 2012 - med doktrinära grunder (MSD
12) (M7739-354023). Stockholm, Sweden: Försvarsmakten.
Gourlay, S. (2003). The SECI model of knowledge creation: some empirical
shortcomings. Proceedings of the 4th European Conference on Knowledge
Management (pp. 377–385). Oxford, UK.
Harbom, S. (2010). Management of various types of emergency. In L. Fredholm & A.-L.
Göransson (Eds.), Emergency Response Management in Today’s Complex Society
(pp. 73–108). Karlstad, Sweden: Swedish Civil Contingencies Agency.
Hegarty, M. (2005). Multimedia Learning about Physical Systems. In R. E. Mayer (Ed.),
The Cambridge Handbook of Multimedia Learning (pp. 447–465). Cambridge, UK:
Cambridge University.
Hijikata, Y., Takenaka, T., Kusumura, Y., & Nishida, S. (2007). Interactive Knowledge
Externalization and Combination for SECI. K-CAP 07 (pp. 151–158). Whistler,
Canada.
Hof, T., De Koning, L., & Essens, P. (2010). Measuring Effectiveness of Teams and
Multi-team Systems in Operation. Proceedings of the 15th International Command
and Control Research and Technology Symposium. Santa Monica, CA: DoD
Command and Control Research Program.
Hollnagel, E. (1993). Human reliability analysis: Context and control. London, UK:
Academic Press.
Hollnagel, E., & Woods, D. D. (2005). Control and Cognition. In E. Hollnagel & D. D.
Woods (Eds.), Joint Cognitive Systems: Foundations of Cognitive Systems
Engineering. Boca Raton, FL: Taylor & Francis.
Howard, R. A. (1966). Decision Analysis: Applied Decision Theory. Proceedings of the
4th International Conference on Operational Research (pp. 55–77). Boston, MA:
Wiley-Interscience.
Jenvald, J. (1999). Methods and Tools in Computer-Supported Taskforce Training.
Linköping Studies in Science and Technology, dissertation no. 598. Linköping,
Sweden: Linköping University.
Jenvald, J., Morin, M., & Rejnus, L. (2000). Developing Digital Courseware from
Multimedia Documentation of Full-Scale Chemical Exercises. Proceedings of the
NBC 2000 Symposium on Nuclear, Biological and Chemical Threats in the 21st
Century. Espoo, Finland.
[58]
Jenvald, J., Rejnus, L., Morin, M., & Thorstensson, M. (1998). Computer-Supported
Assessment of Emergency Planning for Rescue Operations (FOA-R--98-00910-505).
Linköping, Sweden: Swedish Defense Research Establishment.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under
risk. Econometrica, 47(2), 263–292.
King, D. J. (2005). Humanitarian Knowledge Management. Proceedings of the 2nd
International ISCRAM Conference, Brussels, Belgium.
Kizza, J. M., & Ssanyu, J. (2005). Workplace Surveillance. In J. Weckert (Ed.),
Electronic Monitoring in the Workplace: Controversies and Solutions. Hershey, PA:
Idea Group.
Klein, G. A. (1993). A Recognition-Primed Decision (RPD) Model of Rapid Decision
Making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.),
Decision making in action: Models and Methods (pp. 138–147). Norwood, NJ:
Ablex.
Klein, G. A. (1995). Naturalistic Decision Making and Wildland Firefighting. Findings
From the Wildland Firefighters Human Factors Workshop. Retrieved March 6,
2013, from http://www.fs.fed.us/t-d/pubs/htmlpubs/htm95512855/page13.htm
Klein, G. A., Moon, B., & Hoffman, R. R. (2006a). Making Sense of Sensemaking 1:
Alternative Perspectives. IEEE Intelligent Systems, 21(4), 70–73.
Klein, G. A., Moon, B., & Hoffman, R. R. (2006b). Making Sense of Sensemaking 2: A
Macrocognitive Model. IEEE Intelligent Systems, 21(5), 88–92.
Klein, G. A., Ross, K. G., Moon, B. M., Klein, D. E., Hoffman, R. R., & Hollnagel, E.
(2003). Macrocognition. IEEE Intelligent Systems, 18(3), 81–85.
Lafond, D., & DuCharme, M. B. (2011). Complex Decision Making Experimental
Platform (CODEM): A Counter-Insurgency Scenario. IEEE Symposium on
Computational Intelligence for Security and Defence Applications (pp. 72–79).
Paris, France.
Lantz, F., Andersson, D., Jungert, E., & Levin, B. (2009). Decision Support for
Monitoring the Status of Individuals. Proceedings of the 15th International
Conference on Distributed Multimedia Systems (pp. 123–129). Redwood City, CA:
Knowledge Systems Institute.
LaVoie, N., Foltz, P. W., Rosenstein, M., Oberbreckling, R., Chatham, R., & Psotka, J.
(2008). Automated Support for AARs: Exploiting Communication to Assess Team
Performance. Proceedings of the 2008 Interservice/Industry Training, Simulation,
and Education Conference. Orlando, FL.
[59]
Lawson, Jr., J. S. (1981). Command Control as a Process. IEEE Control Systems
Magazine, 1(1), 5–11.
Lee, M. K. O., Cheung, C. M. K., & Chen, Z. (2005). Acceptance of Internet-based
learning medium: the role of extrinsic and intrinsic motivation. Information &
Management, 42(8), 1095–1104.
Lewis, R. B., & Maas, S. M. (2007). QDA Miner 2.0: Mixed-model qualitative data
analysis software. Field Methods, 19(1), 87–108.
Lieber, E. (2009). Mixing Qualitative and Quantitative Methods: insights into design and
analysis issues. Journal of Ethnographic & Qualitative Research, 3(4), 218–227.
Lieber, E., Weisner, T. S., & Presley, M. (2003). EthnoNotes: An Internet-Based Field
Note Management Tool. Field Methods, 15(4), 405–425.
Lim, K. H., & Benbasat, I. (2000). The Effect of Multimedia on Perceived Equivocality
and Perceived Usefulness of Information Systems. MIS Quarterly, 24(3), 449–471.
Liu, S.-H., Liao, H.-L., & Pratt, J. A. (2009). Impact of Media Richness and Flow on elearning Technology Acceptance. Computers & Education, 52(3), 599–607.
Malek, J. (2005). Informed Consent. In C. Mitcham (Ed.), Encyclopedia of Science,
Technology and Ethics, vol. 2 (pp. 1016–1019). Detroit, MI: Macmillan Reference
USA.
Mayer, R. E. (2005). Cognitive Theory of Multimedia Learning. In R. E. Mayer (Ed.),
The Cambridge Handbook of Multimedia Learning (pp. 31–48). Cambridge, UK:
Cambridge University.
Means, B., Salas, E., Crandall, B., & Jacobs, T. O. (1993). Training decision makers for
the real world. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.),
Decision making in action: Models and Methods (pp. 306–326). Norwood, NJ:
Ablex.
Mendonça, D., Beroggi, G. E. G., & Wallace, W. A. (2001). Decision support for
improvisation during emergency response operations. International Journal of
Emergency Management (IJEM), 1(1), 30–39.
Mendonça, D., & Fiedrich, F. (2006). Training for improvisation in emergency
management: opportunities and limits for information technology. International
Journal of Emergency Management (IJEM), 3(4), 348–363.
Mendonça, D., & Hu, Y. (2009). Cognitive Underpinnings of Resilience: A case study of
group decision in Emergency Response. In P. Nemeth, E. Hollnagel, & S. Dekker
[60]
(Eds.), Resilience Engineering Perspectives, vol. 2: Preparation & Restoration (pp.
179–204). Surrey, UK: Ashgate.
Miller, S. (2005). Guarding the Guards: The Right to Privacy, and Workplace
Surveillance and Monitoring in Policing. In J. Weckert (Ed.), Electronic Monitoring
in the Workplace: Controversies and Solutions. Hershey, PA: Idea Group.
Mintzberg, H. (1979). The Structuring of Organizations: A Synthesis of the Research.
Englewood Cliffs, NJ: Prentice-Hall.
Mintzberg, H., & McHugh, A. (1985). Strategy Formation in an Adhocracy.
Administrative Science Quarterly, 30(2), 160–197.
Moorman, C., & Miner, A. S. (1998). Organizational Improvisation and Organizational
Memory. The Academy of Management Review, 23(4), 698–723.
Morin, M. (2001). MIND – Methods and Tools for Visualization of Rescue Operations.
Proceedings of the International Emergency Management Society’s Eight Annual
Conference (TIEMS 2001). Oslo, Norway.
Morin, M. (2002). Multimedia Representation of Distributed Tactical Operations.
Linköping Studies in Science and Technology, dissertation no. 771. Linköping,
Sweden: Linköping University.
Morin, M., & Albinsson, P.-A. (2005). Exploration and Context in Communication
Analysis. In C. A. Bowers, E. Salas, & F. Jentsch (Eds.), Creating High-Tech
Teams: Practical Guidance on Work Performance and Technology. Washington,
DC: American Psychological Association.
Morin, M., Jenvald, J., & Worm, A. (1998). Training Emergency Management and
Response: Creating the Big Picture. Proceedings of the International Emergency
Management Society’s Fifth Annual Conference (TIEMS 98) (pp. 553–561).
Washington, DC.
Ngwenyama, O. K., & Lee, A. S. (1997). Communication Richness in Electronic Mail:
Critical Social Theory and the Contextuality of Meaning. MIS Quarterly, 21(2),
145–167.
Nonaka, I. (1991). The Knowledge-Creating Company. Harvard Business Review, 69(6),
96–104.
Nonaka, I., & Takeuchi, H. (1995). The Knowledge-Creating Company: How Japanese
Companies Create the Dynamics of Innovation (p. 304). Oxford, UK: Oxford
University.
[61]
Nonaka, I., Toyama, R., & Konno, N. (2000). SECI , Ba and Leadership: a Unified
Model of Dynamic Knowledge Creation. Long Range Planning, 33, 5–34.
Novak, J. (2007). Helping Knowledge Cross Boundaries: Using Knowledge Visualization
to Support Cross-Community Sensemaking. Proceedings of the 40th Annual Hawaii
International Conference on System Sciences (pp. 1–10). Big Island, HI.
Oosthuizen, R., & Roodt, J. H. S. (2012). Coping with Complexity in Command and
Control. Proceedings of the 17th International Command and Control Research and
Technology Symposium. Fairfax, VA: DoD Command and Control Research
Program.
Otondo, R. F., Van Scotter, J. R., Allen, D. G., & Palvia, P. (2008). The complexity of
richness: Media, message, and communication outcomes. Information &
Management, 40, 21–30.
Palm, E. (2007). The Ethics of Workplace Surveillance. Royal Institute of Technology,
Stockholm, Sweden.
Pettersson, U. (2009). Sharing Knowledge: How to Highlight Proven Experience in the
Swedish Armed Forces. Proceedings of the 6th International ISCRAM Conference.
Gothenburg, Sweden.
Pigeau, R., & McCann, C. (2002). Re-conceptualizing command and control. Canadian
Military Journal, 3(1), 53–64.
Polanyi, M. (1966). The Tacit Dimension. Garden City, NY: Doubleday & Company.
Rankin, A., Dahlbäck, N., & Lundberg, J. (2013). A case study of factor influencing role
improvisation in crisis response teams. Cognition, Technology & Work, 15(1), 79–
93.
Rao, V. S., & Goldman-Segall, R. (1995). Capturing Stories in Organizational Memory:
The Role of Multimedia Systems. Proceedings of the 28th Annual Hawaii
International Conference on System Sciences (pp. 333–341). Kihei, HI: IEEE
Computer Society.
Rice, J. L., & Rice, B. S. (2005). The Applicability of the SECI model to MultiOrganizational Endeavours: an Integrative Review. International Journal of
Organizational Behaviour, 9(8), 671–682.
Robinson, D., & Reed, V. (Eds.). (1998). The A-Z of social research jargon. Aldershot,
UK: Ashgate.
Ross, W. D. (1930). The Right and the Good. Oxford, UK: Oxford University.
[62]
Rowe, N. C. (2009). Automated Instantaneous Performance Assessment for MarineSquad Urban-Terrain Training. Proceedings of the 14th International Command and
Control Research and Technology Symposium. Washington, DC: DoD Command
and Control Research Program.
Sadagic, A., Welch, G., Basu, C., Darken, C., Kumar, R., Fuchs, H., Cheng, H., et al.
(2009). New Generation of Instrumented Ranges: Enabling Automated Performance
Analysis. Proceedings of the 2009 Interservice/Industry Training, Simulation, and
Education Conference (I/ITSEC). Orlando, FL.
Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical
education: basic guidelines for researchers. Medical Education, 43(7), 637–44.
Seidman, I. (1998). Interviewing as Qualitative Research: A Guide for Researchers in
Education and the Social Sciences. New York, NY: Teachers College.
Sexton, R., & McConnan, I. (2003). A comparative study of after action review (AAR) in
the context of southern Africa Crisis. Retrieved March 20, 2013, from
http://www.alnap.org/pool/files/aar_case_study.pdf
Starbuck, W. H., & Milliken, F. J. (1988). Executives’ perceptual filters: What they
notice and how they make sense. In D. Hambrick (Ed.), The Executive Effect:
Concepts and Methods for Studying Top Managers (pp. 35–65). Greenwich, CT: JAI
Press. Retrieved March 5, 2013, from
http://pages.stern.nyu.edu/~wstarbuc/Hambrick.htm
Stewart, D. W., & Shamdasani, P. N. (1990). Focus Groups: Theory and Practice.
Thousand Oaks, CA: SAGE.
Svensson, S. (2010). The theory of fundamental tactics. Emergency Response
Management in Today’s Complex Society (pp. 185–196). Karlstad, Sweden:
Swedish Civil Contingencies Agency.
Tellis, W. (1997). Application of a Case Study Methodology. The Qualitative Report,
3(3). Retrieved March 18, 2013, from http://www.nova.edu/ssss/QR/QR33/tellis2.html
Thomas, A. S., & Kopczak, L. R. (2005). From Logistics to Supply Chain Management:
The path forward in the humanitarian sector (p. 15). San Francisco, CA: Fritz
Institute.
Thorstensson, M., Axelsson, M., Morin, M., & Jenvald, J. (2001). Monitoring and
analysis of command post communication in rescue operations. Safety Science,
39(1-2), 51–60.
[63]
Thorstensson, M., Johansson, M., Andersson, D., & Albinsson, P.-A. (2007). Ökat utbyte
av övningar - Metoder och teknik för uppföljning och utvärdering av
övningsverksamhet vid SRV:s skola i Sandö (FOI-R--2305) [eng: Improved outcome
of exercises - Methods and tools for training and evaluation at the Swedish Rescue
Services school at Sandö]. Linköping, Sweden: Swedish Defense Research Agency.
Thorstensson, M., Morin, M., & Jenvald, J. (1999). Monitoring and Visualisation Support
for Management of Medical Resources in Mass-Casualty Incidents. Proceedings of
the International Emergency Management Society’s Sixth Annual Conference
(TIEMS 99) (pp. 179–188). Delft, The Netherlands.
Trnka, J., & Jenvald, J. (2006). Role-Playing Exercise: A Real-Time Approach to Study
Collaborative Command and Control. International Journal of Intelligent Control
and Systems, 11(4), 218–228.
Trnka, J., Rankin, A., Jungert, E., Lundberg, J., Granlund, R., Granlund, H., &
Johansson, B. J. E. (2009). Information Support in Modern Crisis and Disaster
Response Operations. Linköping, Sweden: Linköping University.
Tversky, A., & Kahneman, D. (1981). The Framing of Decisions and the Psychology of
Choice. Science, 211(4481), 453–458.
U.S. Army Combined Arms Center. (2011). Leader’s Guide to After-Action Reviews
(AAR). Fort Leavenworth, Kansas: Headquarters Department of the Army.
Warren, S. D., & Brandeis, L. D. (1890). The Right to Privacy. Harvard Law Review,
4(5), 193–220.
Waterman Jr., R. H. (1990). Adhocracy: The power to change. Memphis, TN: Whittle
Direct Books.
Weick, K. E. (1969). The Social Psychology of Organizing. Reading, MA: AddisonWesley.
Weick, K. E. (1993). The Collapse of Sensemaking in Organizations: The Mann Gulch
Disaster. Administrative Science Quarterly, 38(4), 628–652.
Weick, K. E. (1995). Sensemaking in Organizations. Thousand Oaks, CA: SAGE.
Weick, K. E., & Sutcliffe, K. M. (2001). Managing the unexpected: Assuring high
performance in an age of uncertainty. San Francisco, CA: Jossey-Bass.
Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the Qualitative-Quantitative
Divide: Guidelines for Conducting Mixed Methods Research in Information
Systems. MIS Quarterly, 37(1), 21–54.
[64]
Widder, W. (2002). Auftragstaktik and Innere Führung: Trademarks of German
Leadership. Military Review, 82(5), 3–9.
Wohl, J. G. (1981). Force Management Decision Requirements for Air Force Tactical
Command and Control. IEEE Transactions on Systems, Man and Cybernetics, 11(9),
618–639.
Worm, A. (1999). Mission Efficiency Analysis of Tactical Joint Systems. The Symposium
of the RTO Studies, Analysis and Simulation (SAS) Panel. Issy les Moulineaux,
France: NATO Research and Technology Organisation.
Worm, A., Jenvald, J., & Morin, M. (1998). Mission efficiency analysis: evaluating and
improving tactical mission performance in high-risk, time-critical operations. Safety
Science, 30, 79–98.
Yin, R. K. (1994). Case Study Research: Design and Methods. Thousand Oaks, CA:
SAGE.
Zumel, N. B., Franco, Z. E., & Beutler, L. E. (2008). Improvisation as a Training
Framework for Emergency Managers. Proceedings of the 13th International
Command and Control Research and Technology Symposium. Seattle, WA: DoD
Command and Control Research Program.
[65]
[This page is intentionally left blank]
[66]
LIST OF FIGURES
Figure 2.1. The extended OODA loop (Boyd, 1987) ......................................................... 6
Figure 2.2. M-OODA (Breton and Rousseau, 2008) .......................................................... 7
Figure 2.3. The Dynamic OODA loop (Oosthuizen & Roodt, 2012) ................................. 7
Figure 2.4. A frames-based model of sensemaking (Klein et al., 2006b)......................... 10
Figure 2.5. Situation awareness (Endsley 1995) ............................................................... 11
Figure 2.6. Recognition-primed decision making (Klein, 1993) ...................................... 12
Figure 2.7. A representation of mission histories (Paper II) ............................................. 13
Figure 2.8. MIND Studio .................................................................................................. 14
Figure 2.9. F-REX Studio (Paper II) ................................................................................. 15
Figure 2.10. A recent version of F-REX Studio (Andersson, 2012)................................. 16
Figure 2.11. A control theory based model of tactical missions (Worm et al., 1998) ...... 17
Figure 2.12. Mission efficiency analysis model (Worm et al., 1998). .............................. 18
Figure 2.13. The SECI model (Nonaka & Takeuchi, 1995) ............................................. 20
Figure 4.1. The FAIRIC model (Paper I). ......................................................................... 32
Figure 4.2. The SECI model (Nonaka & Takeuchi, 1995) ............................................... 33
Figure 4.3. Reconstruction and Exploration (Andersson, 2009) ...................................... 36
Figure 5.1. Indicators for navigation................................................................................. 47
[67]
[This page is intentionally left blank]
[68]
LIST OF TABLES
Table 3.1. Study design ..................................................................................................... 25
Table 5.1. The FAIRIC concepts (from Paper I) .............................................................. 44
[69]
[This page is intentionally left blank]
[70]
AUTHOR’S PUBLICATIONS
Albinsson, P.-A., & Andersson, D. (2006). Computer-Aided Football Training:
Exploiting Advances in Distributed Tactical Operations Research. Proceedings of
the 6th International Sports Engineering Conference (pp. 185–190). Munich,
Germany: Springer.
Albinsson, P.-A., & Andersson, D. (2008). Extending the attribute explorer to support
professional team-sport analysis. Information Visualization, 7(2), 163–169.
Andersson, D. (2009). F-REX: Event-Driven Synchronized Multimedia Model
Visualization. Proceedings of the 15th International Conference on Distributed
Multimedia Systems (pp. 140–145). Redwood City, CA: Knowledge Systems
Institute.
Andersson, D. (2011). Privacy and Distributed Tactical Operations Evaluation.
Proceedings of the 4th International Conference on Advances in Human-oriented
and Personalized Mechanisms, Technologies, and Services. Barcelona, Spain.
Andersson, D. (2013). A Knowledge Base for Capturing Comprehensive Mission
Experience. Proceedings of the 46th Hawaii International Conference on System
Science. Wailea, HI: IEEE Computer Society.
Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011a).
Exploratory sequential data analysis of a cyber defence exercise. Proceedings of the
First International Defense and Homeland Security Simulation Workshop. Rome,
Italy.
Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011b). Analysis
of a Cyber Defense Exercise using Exploratory Sequential Data Analysis.
Proceedings of the 16th International Command and Control Research and
Technology Symposium. Québec City, Canada: DoD Command and Control
Research Program.
Andersson, D., Mullins, S., Bordetsky, A., Bourakov, E., & Forsgren, R. (2011). MIO
Experiment Data Capture. Proceedings of the 10th International Conference on
Telecommunication Systems: Modeling and Analysis. Prague, Czech Republic.
Andersson, D., Pilemalm, S., & Hallberg, N. (2008). Evaluation of crisis management
operations using Reconstruction and Exploration. Proceedings of the 5th
International ISCRAM Conference (pp. 118–125). Washington, DC.
Andersson, D., & Rankin, A. (2012). Sharing Mission Experience in Tactical
Organisations. In L. Rothkrantz, J. Ristvej, & Z. E. Franco (Eds.), Proceedings of
the 9th International ISCRAM Conference (pp. 1–10). Vancouver, Canada.
[71]
Andersson, D., & Skagert, C. (2004). Managing Massive Datasets from Distributed
Tactical Operations. Master thesis. Linköping, Sweden: Linköping University.
Gustafsson, O., Amiri, K., Andersson, D., Blad, A., Bonnet, C., Cavallaro, J. R.,
Declerck, J., et al. (2010). Architectures for Cognitive Radio Testbeds and
Demonstrators – An Overview. The 5th International Conference on Cognitive
Radio Oriented Wireless Networks and Communications. Cannes, France.
Hammervik, M., Andersson, D., & Hallberg, J. (2010). Capturing a Cyber Defence
Exercise. The first national symposium on Technology and Methodology for Security
and Crisis Management (p. 30). Linköping, Sweden.
Holm, H., Ekstedt, M., & Andersson, D. (2012). Empirical Analysis of System-Level
Vulnerability Metrics through Actual Attacks. IEEE Transactions on Dependable
and Secure Computing, 9(6), 825–837.
Lantz, F., Andersson, D., Jungert, E., & Levin, B. (2009). Decision Support for
Monitoring the Status of Individuals. Proceedings of the 15th International
Conference on Distributed Multimedia Systems (pp. 123–129). Redwood City, CA:
Knowledge Systems Institute.
Pilemalm, S., Andersson, D., & Hallberg, N. (2008). Reconstruction and Exploration of
Large-scale Distributed Operations: Multimedia tools for Evaluation of Emergency
Management Response. Journal of Emergency Management, 6(4), 31–47.
Sigholm, J., & Andersson, D. (2011). Privacy on the Battlefield? Ethical Issues of
Emerging Military ICTs. Proceedings of the 9th Computer Ethics Philosophy
Enquiry. Milwaukee, WI.
Svenmarck, P., Andersson, D., Lindahl, B., Hedström, J., & Lif, P. (2009). Tactical
Reconnaissance using Groups of Partly Autonomous UGVs. Proceedings of the 8th
International Conference on Engineering Psychology and Cognitive Ergonomics:
HCII 2009 (pp. 326–335). San Diego, CA: Springer.
[72]
Department of Computer and Information Science
Linköpings universitet
Licentiate Theses
Linköpings Studies in Science and Technology
Faculty of Arts and Sciences
No 17
No 28
No 29
No 48
No 52
No 60
No 71
No 72
No 73
No 74
No 104
No 108
No 111
No 113
No 118
No 126
No 127
No 139
No 140
No 146
No 150
No 165
No 166
No 174
No 177
No 181
No 184
No 187
No 189
No 196
No 197
No 203
No 212
No 230
No 237
No 250
No 253
No 260
No 283
No 298
No 318
No 319
No 326
No 328
No 333
No 335
No 348
No 352
No 371
No 378
Vojin Plavsic: Interleaved Processing of Non-Numerical Data Stored on a Cyclic Memory. (Available at: FOA,
Box 1165, S-581 11 Linköping, Sweden. FOA Report B30062E)
Arne Jönsson, Mikael Patel: An Interactive Flowcharting Technique for Communicating and Realizing Algorithms, 1984.
Johnny Eckerland: Retargeting of an Incremental Code Generator, 1984.
Henrik Nordin: On the Use of Typical Cases for Knowledge-Based Consultation and Teaching, 1985.
Zebo Peng: Steps Towards the Formalization of Designing VLSI Systems, 1985.
Johan Fagerström: Simulation and Evaluation of Architecture based on Asynchronous Processes, 1985.
Jalal Maleki: ICONStraint, A Dependency Directed Constraint Maintenance System, 1987.
Tony Larsson: On the Specification and Verification of VLSI Systems, 1986.
Ola Strömfors: A Structure Editor for Documents and Programs, 1986.
Christos Levcopoulos: New Results about the Approximation Behavior of the Greedy Triangulation, 1986.
Shamsul I. Chowdhury: Statistical Expert Systems - a Special Application Area for Knowledge-Based Computer
Methodology, 1987.
Rober Bilos: Incremental Scanning and Token-Based Editing, 1987.
Hans Block: SPORT-SORT Sorting Algorithms and Sport Tournaments, 1987.
Ralph Rönnquist: Network and Lattice Based Approaches to the Representation of Knowledge, 1987.
Mariam Kamkar, Nahid Shahmehri: Affect-Chaining in Program Flow Analysis Applied to Queries of Programs, 1987.
Dan Strömberg: Transfer and Distribution of Application Programs, 1987.
Kristian Sandahl: Case Studies in Knowledge Acquisition, Migration and User Acceptance of Expert Systems,
1987.
Christer Bäckström: Reasoning about Interdependent Actions, 1988.
Mats Wirén: On Control Strategies and Incrementality in Unification-Based Chart Parsing, 1988.
Johan Hultman: A Software System for Defining and Controlling Actions in a Mechanical System, 1988.
Tim Hansen: Diagnosing Faults using Knowledge about Malfunctioning Behavior, 1988.
Jonas Löwgren: Supporting Design and Management of Expert System User Interfaces, 1989.
Ola Petersson: On Adaptive Sorting in Sequential and Parallel Models, 1989.
Yngve Larsson: Dynamic Configuration in a Distributed Environment, 1989.
Peter Åberg: Design of a Multiple View Presentation and Interaction Manager, 1989.
Henrik Eriksson: A Study in Domain-Oriented Tool Support for Knowledge Acquisition, 1989.
Ivan Rankin: The Deep Generation of Text in Expert Critiquing Systems, 1989.
Simin Nadjm-Tehrani: Contributions to the Declarative Approach to Debugging Prolog Programs, 1989.
Magnus Merkel: Temporal Information in Natural Language, 1989.
Ulf Nilsson: A Systematic Approach to Abstract Interpretation of Logic Programs, 1989.
Staffan Bonnier: Horn Clause Logic with External Procedures: Towards a Theoretical Framework, 1989.
Christer Hansson: A Prototype System for Logical Reasoning about Time and Action, 1990.
Björn Fjellborg: An Approach to Extraction of Pipeline Structures for VLSI High-Level Synthesis, 1990.
Patrick Doherty: A Three-Valued Approach to Non-Monotonic Reasoning, 1990.
Tomas Sokolnicki: Coaching Partial Plans: An Approach to Knowledge-Based Tutoring, 1990.
Lars Strömberg: Postmortem Debugging of Distributed Systems, 1990.
Torbjörn Näslund: SLDFA-Resolution - Computing Answers for Negative Queries, 1990.
Peter D. Holmes: Using Connectivity Graphs to Support Map-Related Reasoning, 1991.
Olof Johansson: Improving Implementation of Graphical User Interfaces for Object-Oriented Knowledge- Bases,
1991.
Rolf G Larsson: Aktivitetsbaserad kalkylering i ett nytt ekonomisystem, 1991.
Lena Srömbäck: Studies in Extended Unification-Based Formalism for Linguistic Description: An Algorithm for
Feature Structures with Disjunction and a Proposal for Flexible Systems, 1992.
Mikael Pettersson: DML-A Language and System for the Generation of Efficient Compilers from Denotational
Specification, 1992.
Andreas Kågedal: Logic Programming with External Procedures: an Implementation, 1992.
Patrick Lambrix: Aspects of Version Management of Composite Objects, 1992.
Xinli Gu: Testability Analysis and Improvement in High-Level Synthesis Systems, 1992.
Torbjörn Näslund: On the Role of Evaluations in Iterative Development of Managerial Support Systems, 1992.
Ulf Cederling: Industrial Software Development - a Case Study, 1992.
Magnus Morin: Predictable Cyclic Computations in Autonomous Systems: A Computational Model and Implementation, 1992.
Mehran Noghabai: Evaluation of Strategic Investments in Information Technology, 1993.
Mats Larsson: A Transformational Approach to Formal Digital System Design, 1993.
No 380
No 381
No 383
No 386
No 398
No 402
No 406
No 414
No 417
No 436
No 437
No 440
FHS 3/94
FHS 4/94
No 441
No 446
No 450
No 451
No 452
No 455
FHS 5/94
No 462
No 463
No 464
No 469
No 473
No 475
No 476
No 478
FHS 7/95
No 482
No 488
No 489
No 497
No 498
No 503
FHS 8/95
FHS 9/95
No 513
No 517
No 518
No 522
No 538
No 545
No 546
FiF-a 1/96
No 549
No 550
No 557
No 558
No 561
No 563
Johan Ringström: Compiler Generation for Parallel Languages from Denotational Specifications, 1993.
Michael Jansson: Propagation of Change in an Intelligent Information System, 1993.
Jonni Harrius: An Architecture and a Knowledge Representation Model for Expert Critiquing Systems, 1993.
Per Österling: Symbolic Modelling of the Dynamic Environments of Autonomous Agents, 1993.
Johan Boye: Dependency-based Groudness Analysis of Functional Logic Programs, 1993.
Lars Degerstedt: Tabulated Resolution for Well Founded Semantics, 1993.
Anna Moberg: Satellitkontor - en studie av kommunikationsmönster vid arbete på distans, 1993.
Peter Carlsson: Separation av företagsledning och finansiering - fallstudier av företagsledarutköp ur ett agentteoretiskt perspektiv, 1994.
Camilla Sjöström: Revision och lagreglering - ett historiskt perspektiv, 1994.
Cecilia Sjöberg: Voices in Design: Argumentation in Participatory Development, 1994.
Lars Viklund: Contributions to a High-level Programming Environment for a Scientific Computing, 1994.
Peter Loborg: Error Recovery Support in Manufacturing Control Systems, 1994.
Owen Eriksson: Informationssystem med verksamhetskvalitet - utvärdering baserat på ett verksamhetsinriktat och
samskapande perspektiv, 1994.
Karin Pettersson: Informationssystemstrukturering, ansvarsfördelning och användarinflytande - En komparativ
studie med utgångspunkt i två informationssystemstrategier, 1994.
Lars Poignant: Informationsteknologi och företagsetablering - Effekter på produktivitet och region, 1994.
Gustav Fahl: Object Views of Relational Data in Multidatabase Systems, 1994.
Henrik Nilsson: A Declarative Approach to Debugging for Lazy Functional Languages, 1994.
Jonas Lind: Creditor - Firm Relations: an Interdisciplinary Analysis, 1994.
Martin Sköld: Active Rules based on Object Relational Queries - Efficient Change Monitoring Techniques, 1994.
Pär Carlshamre: A Collaborative Approach to Usability Engineering: Technical Communicators and System
Developers in Usability-Oriented Systems Development, 1994.
Stefan Cronholm: Varför CASE-verktyg i systemutveckling? - En motiv- och konsekvensstudie avseende
arbetssätt och arbetsformer, 1994.
Mikael Lindvall: A Study of Traceability in Object-Oriented Systems Development, 1994.
Fredrik Nilsson: Strategi och ekonomisk styrning - En studie av Sandviks förvärv av Bahco Verktyg, 1994.
Hans Olsén: Collage Induction: Proving Properties of Logic Programs by Program Synthesis, 1994.
Lars Karlsson: Specification and Synthesis of Plans Using the Features and Fluents Framework, 1995.
Ulf Söderman: On Conceptual Modelling of Mode Switching Systems, 1995.
Choong-ho Yi: Reasoning about Concurrent Actions in the Trajectory Semantics, 1995.
Bo Lagerström: Successiv resultatavräkning av pågående arbeten. - Fallstudier i tre byggföretag, 1995.
Peter Jonsson: Complexity of State-Variable Planning under Structural Restrictions, 1995.
Anders Avdic: Arbetsintegrerad systemutveckling med kalkylprogram, 1995.
Eva L Ragnemalm: Towards Student Modelling through Collaborative Dialogue with a Learning Companion,
1995.
Eva Toller: Contributions to Parallel Multiparadigm Languages: Combining Object-Oriented and Rule-Based
Programming, 1995.
Erik Stoy: A Petri Net Based Unified Representation for Hardware/Software Co-Design, 1995.
Johan Herber: Environment Support for Building Structured Mathematical Models, 1995.
Stefan Svenberg: Structure-Driven Derivation of Inter-Lingual Functor-Argument Trees for Multi-Lingual
Generation, 1995.
Hee-Cheol Kim: Prediction and Postdiction under Uncertainty, 1995.
Dan Fristedt: Metoder i användning - mot förbättring av systemutveckling genom situationell metodkunskap och
metodanalys, 1995.
Malin Bergvall: Systemförvaltning i praktiken - en kvalitativ studie avseende centrala begrepp, aktiviteter och
ansvarsroller, 1995.
Joachim Karlsson: Towards a Strategy for Software Requirements Selection, 1995.
Jakob Axelsson: Schedulability-Driven Partitioning of Heterogeneous Real-Time Systems, 1995.
Göran Forslund: Toward Cooperative Advice-Giving Systems: The Expert Systems Experience, 1995.
Jörgen Andersson: Bilder av småföretagares ekonomistyrning, 1995.
Staffan Flodin: Efficient Management of Object-Oriented Queries with Late Binding, 1996.
Vadim Engelson: An Approach to Automatic Construction of Graphical User Interfaces for Applications in
Scientific Computing, 1996.
Magnus Werner : Multidatabase Integration using Polymorphic Queries and Views, 1996.
Mikael Lind: Affärsprocessinriktad förändringsanalys - utveckling och tillämpning av synsätt och metod, 1996.
Jonas Hallberg: High-Level Synthesis under Local Timing Constraints, 1996.
Kristina Larsen: Förutsättningar och begränsningar för arbete på distans - erfarenheter från fyra svenska företag.
1996.
Mikael Johansson: Quality Functions for Requirements Engineering Methods, 1996.
Patrik Nordling: The Simulation of Rolling Bearing Dynamics on Parallel Computers, 1996.
Anders Ekman: Exploration of Polygonal Environments, 1996.
Niclas Andersson: Compilation of Mathematical Models to Parallel Code, 1996.
No 567
No 575
No 576
No 587
No 589
No 591
No 595
No 597
No 598
No 599
No 607
No 609
FiF-a 4
FiF-a 6
No 615
No 623
No 626
No 627
No 629
No 631
No 639
No 640
No 643
No 653
FiF-a 13
No 674
No 676
No 668
No 675
FiF-a 14
No 695
No 700
FiF-a 16
No 712
No 719
No 723
No 725
No 730
No 731
No 733
No 734
FiF-a 21
FiF-a 22
No 737
No 738
FiF-a 25
No 742
No 748
No 751
No 752
No 753
Johan Jenvald: Simulation and Data Collection in Battle Training, 1996.
Niclas Ohlsson: Software Quality Engineering by Early Identification of Fault-Prone Modules, 1996.
Mikael Ericsson: Commenting Systems as Design Support—A Wizard-of-Oz Study, 1996.
Jörgen Lindström: Chefers användning av kommunikationsteknik, 1996.
Esa Falkenroth: Data Management in Control Applications - A Proposal Based on Active Database Systems,
1996.
Niclas Wahllöf: A Default Extension to Description Logics and its Applications, 1996.
Annika Larsson: Ekonomisk Styrning och Organisatorisk Passion - ett interaktivt perspektiv, 1997.
Ling Lin: A Value-based Indexing Technique for Time Sequences, 1997.
Rego Granlund: C3Fire - A Microworld Supporting Emergency Management Training, 1997.
Peter Ingels: A Robust Text Processing Technique Applied to Lexical Error Recovery, 1997.
Per-Arne Persson: Toward a Grounded Theory for Support of Command and Control in Military Coalitions, 1997.
Jonas S Karlsson: A Scalable Data Structure for a Parallel Data Server, 1997.
Carita Åbom: Videomötesteknik i olika affärssituationer - möjligheter och hinder, 1997.
Tommy Wedlund: Att skapa en företagsanpassad systemutvecklingsmodell - genom rekonstruktion, värdering och
vidareutveckling i T50-bolag inom ABB, 1997.
Silvia Coradeschi: A Decision-Mechanism for Reactive and Coordinated Agents, 1997.
Jan Ollinen: Det flexibla kontorets utveckling på Digital - Ett stöd för multiflex? 1997.
David Byers: Towards Estimating Software Testability Using Static Analysis, 1997.
Fredrik Eklund: Declarative Error Diagnosis of GAPLog Programs, 1997.
Gunilla Ivefors: Krigsspel och Informationsteknik inför en oförutsägbar framtid, 1997.
Jens-Olof Lindh: Analysing Traffic Safety from a Case-Based Reasoning Perspective, 1997
Jukka Mäki-Turja:. Smalltalk - a suitable Real-Time Language, 1997.
Juha Takkinen: CAFE: Towards a Conceptual Model for Information Management in Electronic Mail, 1997.
Man Lin: Formal Analysis of Reactive Rule-based Programs, 1997.
Mats Gustafsson: Bringing Role-Based Access Control to Distributed Systems, 1997.
Boris Karlsson: Metodanalys för förståelse och utveckling av systemutvecklingsverksamhet. Analys och värdering
av systemutvecklingsmodeller och dess användning, 1997.
Marcus Bjäreland: Two Aspects of Automating Logics of Action and Change - Regression and Tractability,
1998.
Jan Håkegård: Hierarchical Test Architecture and Board-Level Test Controller Synthesis, 1998.
Per-Ove Zetterlund: Normering av svensk redovisning - En studie av tillkomsten av Redovisningsrådets rekommendation om koncernredovisning (RR01:91), 1998.
Jimmy Tjäder: Projektledaren & planen - en studie av projektledning i tre installations- och systemutvecklingsprojekt, 1998.
Ulf Melin: Informationssystem vid ökad affärs- och processorientering - egenskaper, strategier och utveckling,
1998.
Tim Heyer: COMPASS: Introduction of Formal Methods in Code Development and Inspection, 1998.
Patrik Hägglund: Programming Languages for Computer Algebra, 1998.
Marie-Therese Christiansson: Inter-organisatorisk verksamhetsutveckling - metoder som stöd vid utveckling av
partnerskap och informationssystem, 1998.
Christina Wennestam: Information om immateriella resurser. Investeringar i forskning och utveckling samt i
personal inom skogsindustrin, 1998.
Joakim Gustafsson: Extending Temporal Action Logic for Ramification and Concurrency, 1998.
Henrik André-Jönsson: Indexing time-series data using text indexing methods, 1999.
Erik Larsson: High-Level Testability Analysis and Enhancement Techniques, 1998.
Carl-Johan Westin: Informationsförsörjning: en fråga om ansvar - aktiviteter och uppdrag i fem stora svenska
organisationers operativa informationsförsörjning, 1998.
Åse Jansson: Miljöhänsyn - en del i företags styrning, 1998.
Thomas Padron-McCarthy: Performance-Polymorphic Declarative Queries, 1998.
Anders Bäckström: Värdeskapande kreditgivning - Kreditriskhantering ur ett agentteoretiskt perspektiv, 1998.
Ulf Seigerroth: Integration av förändringsmetoder - en modell för välgrundad metodintegration, 1999.
Fredrik Öberg: Object-Oriented Frameworks - A New Strategy for Case Tool Development, 1998.
Jonas Mellin: Predictable Event Monitoring, 1998.
Joakim Eriksson: Specifying and Managing Rules in an Active Real-Time Database System, 1998.
Bengt E W Andersson: Samverkande informationssystem mellan aktörer i offentliga åtaganden - En teori om
aktörsarenor i samverkan om utbyte av information, 1998.
Pawel Pietrzak: Static Incorrectness Diagnosis of CLP (FD), 1999.
Tobias Ritzau: Real-Time Reference Counting in RT-Java, 1999.
Anders Ferntoft: Elektronisk affärskommunikation - kontaktkostnader och kontaktprocesser mellan kunder och
leverantörer på producentmarknader, 1999.
Jo Skåmedal: Arbete på distans och arbetsformens påverkan på resor och resmönster, 1999.
Johan Alvehus: Mötets metaforer. En studie av berättelser om möten, 1999.
No 754
No 766
No 769
No 775
FiF-a 30
No 787
No 788
No 790
No 791
No 800
No 807
No 809
FiF-a 32
No 808
No 820
No 823
No 832
FiF-a 34
No 842
No 844
FiF-a 37
FiF-a 40
FiF-a 41
No. 854
No 863
No 881
No 882
No 890
FiF-a 47
No 894
No 906
No 917
No 916
FiF-a-49
FiF-a-51
No 919
No 915
No 931
No 933
No 938
No 942
No 956
FiF-a 58
No 964
No 973
No 958
FiF-a 61
No 985
No 982
No 989
No 990
No 991
Magnus Lindahl: Bankens villkor i låneavtal vid kreditgivning till högt belånade företagsförvärv: En studie ur ett
agentteoretiskt perspektiv, 2000.
Martin V. Howard: Designing dynamic visualizations of temporal data, 1999.
Jesper Andersson: Towards Reactive Software Architectures, 1999.
Anders Henriksson: Unique kernel diagnosis, 1999.
Pär J. Ågerfalk: Pragmatization of Information Systems - A Theoretical and Methodological Outline, 1999.
Charlotte Björkegren: Learning for the next project - Bearers and barriers in knowledge transfer within an
organisation, 1999.
Håkan Nilsson: Informationsteknik som drivkraft i granskningsprocessen - En studie av fyra revisionsbyråer,
2000.
Erik Berglund: Use-Oriented Documentation in Software Development, 1999.
Klas Gäre: Verksamhetsförändringar i samband med IS-införande, 1999.
Anders Subotic: Software Quality Inspection, 1999.
Svein Bergum: Managerial communication in telework, 2000.
Flavius Gruian: Energy-Aware Design of Digital Systems, 2000.
Karin Hedström: Kunskapsanvändning och kunskapsutveckling hos verksamhetskonsulter - Erfarenheter från ett
FOU-samarbete, 2000.
Linda Askenäs: Affärssystemet - En studie om teknikens aktiva och passiva roll i en organisation, 2000.
Jean Paul Meynard: Control of industrial robots through high-level task programming, 2000.
Lars Hult: Publika Gränsytor - ett designexempel, 2000.
Paul Pop: Scheduling and Communication Synthesis for Distributed Real-Time Systems, 2000.
Göran Hultgren: Nätverksinriktad Förändringsanalys - perspektiv och metoder som stöd för förståelse och
utveckling av affärsrelationer och informationssystem, 2000.
Magnus Kald: The role of management control systems in strategic business units, 2000.
Mikael Cäker: Vad kostar kunden? Modeller för intern redovisning, 2000.
Ewa Braf: Organisationers kunskapsverksamheter - en kritisk studie av ”knowledge management”, 2000.
Henrik Lindberg: Webbaserade affärsprocesser - Möjligheter och begränsningar, 2000.
Benneth Christiansson: Att komponentbasera informationssystem - Vad säger teori och praktik?, 2000.
Ola Pettersson: Deliberation in a Mobile Robot, 2000.
Dan Lawesson: Towards Behavioral Model Fault Isolation for Object Oriented Control Systems, 2000.
Johan Moe: Execution Tracing of Large Distributed Systems, 2001.
Yuxiao Zhao: XML-based Frameworks for Internet Commerce and an Implementation of B2B
e-procurement,
2001.
Annika Flycht-Eriksson: Domain Knowledge Management in Information-providing Dialogue systems, 2001.
Per-Arne Segerkvist: Webbaserade imaginära organisationers samverkansformer: Informationssystemarkitektur
och aktörssamverkan som förutsättningar för affärsprocesser, 2001.
Stefan Svarén: Styrning av investeringar i divisionaliserade företag - Ett koncernperspektiv, 2001.
Lin Han: Secure and Scalable E-Service Software Delivery, 2001.
Emma Hansson: Optionsprogram för anställda - en studie av svenska börsföretag, 2001.
Susanne Odar: IT som stöd för strategiska beslut, en studie av datorimplementerade modeller av verksamhet som
stöd för beslut om anskaffning av JAS 1982, 2002.
Stefan Holgersson: IT-system och filtrering av verksamhetskunskap - kvalitetsproblem vid analyser och beslutsfattande som bygger på uppgifter hämtade från polisens IT-system, 2001.
Per Oscarsson: Informationssäkerhet i verksamheter - begrepp och modeller som stöd för förståelse av informationssäkerhet och dess hantering, 2001.
Luis Alejandro Cortes: A Petri Net Based Modeling and Verification Technique for Real-Time Embedded
Systems, 2001.
Niklas Sandell: Redovisning i skuggan av en bankkris - Värdering av fastigheter. 2001.
Fredrik Elg: Ett dynamiskt perspektiv på individuella skillnader av heuristisk kompetens, intelligens, mentala
modeller, mål och konfidens i kontroll av mikrovärlden Moro, 2002.
Peter Aronsson: Automatic Parallelization of Simulation Code from Equation Based Simulation Languages, 2002.
Bourhane Kadmiry: Fuzzy Control of Unmanned Helicopter, 2002.
Patrik Haslum: Prediction as a Knowledge Representation Problem: A Case Study in Model Design, 2002.
Robert Sevenius: On the instruments of governance - A law & economics study of capital instruments in limited
liability companies, 2002.
Johan Petersson: Lokala elektroniska marknadsplatser - informationssystem för platsbundna affärer, 2002.
Peter Bunus: Debugging and Structural Analysis of Declarative Equation-Based Languages, 2002.
Gert Jervan: High-Level Test Generation and Built-In Self-Test Techniques for Digital Systems, 2002.
Fredrika Berglund: Management Control and Strategy - a Case Study of Pharmaceutical Drug Development,
2002.
Fredrik Karlsson: Meta-Method for Method Configuration - A Rational Unified Process Case, 2002.
Sorin Manolache: Schedulability Analysis of Real-Time Systems with Stochastic Task Execution Times, 2002.
Diana Szentiványi: Performance and Availability Trade-offs in Fault-Tolerant Middleware, 2002.
Iakov Nakhimovski: Modeling and Simulation of Contacting Flexible Bodies in Multibody Systems, 2002.
Levon Saldamli: PDEModelica - Towards a High-Level Language for Modeling with Partial Differential
Equations, 2002.
Almut Herzog: Secure Execution Environment for Java Electronic Services, 2002.
No 999
No 1000
No 1001
No 988
FiF-a 62
No 1003
No 1005
No 1008
No 1010
No 1015
No 1018
No 1022
FiF-a 65
No 1024
No 1034
No 1033
FiF-a 69
No 1049
No 1052
No 1054
FiF-a 71
No 1055
No 1058
FiF-a 73
No 1079
No 1084
FiF-a 74
No 1094
No 1095
No 1099
No 1110
No 1116
FiF-a 77
No 1126
No 1127
No 1132
No 1130
No 1138
No 1149
No 1156
No 1162
No 1165
FiF-a 84
No 1166
No 1167
No 1168
FiF-a 85
No 1171
FiF-a 86
No 1172
No 1183
No 1184
No 1185
No 1190
Jon Edvardsson: Contributions to Program- and Specification-based Test Data Generation, 2002.
Anders Arpteg: Adaptive Semi-structured Information Extraction, 2002.
Andrzej Bednarski: A Dynamic Programming Approach to Optimal Retargetable Code Generation for Irregular
Architectures, 2002.
Mattias Arvola: Good to use! : Use quality of multi-user applications in the home, 2003.
Lennart Ljung: Utveckling av en projektivitetsmodell - om organisationers förmåga att tillämpa
projektarbetsformen, 2003.
Pernilla Qvarfordt: User experience of spoken feedback in multimodal interaction, 2003.
Alexander Siemers: Visualization of Dynamic Multibody Simulation With Special Reference to Contacts, 2003.
Jens Gustavsson: Towards Unanticipated Runtime Software Evolution, 2003.
Calin Curescu: Adaptive QoS-aware Resource Allocation for Wireless Networks, 2003.
Anna Andersson: Management Information Systems in Process-oriented Healthcare Organisations, 2003.
Björn Johansson: Feedforward Control in Dynamic Situations, 2003.
Traian Pop: Scheduling and Optimisation of Heterogeneous Time/Event-Triggered Distributed Embedded
Systems, 2003.
Britt-Marie Johansson: Kundkommunikation på distans - en studie om kommunikationsmediets betydelse i
affärstransaktioner, 2003.
Aleksandra Tešanovic: Towards Aspectual Component-Based Real-Time System Development, 2003.
Arja Vainio-Larsson: Designing for Use in a Future Context - Five Case Studies in Retrospect, 2003.
Peter Nilsson: Svenska bankers redovisningsval vid reservering för befarade kreditförluster - En studie vid
införandet av nya redovisningsregler, 2003.
Fredrik Ericsson: Information Technology for Learning and Acquiring of Work Knowledge, 2003.
Marcus Comstedt: Towards Fine-Grained Binary Composition through Link Time Weaving, 2003.
Åsa Hedenskog: Increasing the Automation of Radio Network Control, 2003.
Claudiu Duma: Security and Efficiency Tradeoffs in Multicast Group Key Management, 2003.
Emma Eliason: Effektanalys av IT-systems handlingsutrymme, 2003.
Carl Cederberg: Experiments in Indirect Fault Injection with Open Source and Industrial Software, 2003.
Daniel Karlsson: Towards Formal Verification in a Component-based Reuse Methodology, 2003.
Anders Hjalmarsson: Att etablera och vidmakthålla förbättringsverksamhet - behovet av koordination och
interaktion vid förändring av systemutvecklingsverksamheter, 2004.
Pontus Johansson: Design and Development of Recommender Dialogue Systems, 2004.
Charlotte Stoltz: Calling for Call Centres - A Study of Call Centre Locations in a Swedish Rural Region, 2004.
Björn Johansson: Deciding on Using Application Service Provision in SMEs, 2004.
Genevieve Gorrell: Language Modelling and Error Handling in Spoken Dialogue Systems, 2004.
Ulf Johansson: Rule Extraction - the Key to Accurate and Comprehensible Data Mining Models, 2004.
Sonia Sangari: Computational Models of Some Communicative Head Movements, 2004.
Hans Nässla: Intra-Family Information Flow and Prospects for Communication Systems, 2004.
Henrik Sällberg: On the value of customer loyalty programs - A study of point programs and switching costs,
2004.
Ulf Larsson: Designarbete i dialog - karaktärisering av interaktionen mellan användare och utvecklare i en
systemutvecklingsprocess, 2004.
Andreas Borg: Contribution to Management and Validation of Non-Functional Requirements, 2004.
Per-Ola Kristensson: Large Vocabulary Shorthand Writing on Stylus Keyboard, 2004.
Pär-Anders Albinsson: Interacting with Command and Control Systems: Tools for Operators and Designers,
2004.
Ioan Chisalita: Safety-Oriented Communication in Mobile Networks for Vehicles, 2004.
Thomas Gustafsson: Maintaining Data Consistency in Embedded Databases for Vehicular Systems, 2004.
Vaida Jakoniené: A Study in Integrating Multiple Biological Data Sources, 2005.
Abdil Rashid Mohamed: High-Level Techniques for Built-In Self-Test Resources Optimization, 2005.
Adrian Pop: Contributions to Meta-Modeling Tools and Methods, 2005.
Fidel Vascós Palacios: On the information exchange between physicians and social insurance officers in the sick
leave process: an Activity Theoretical perspective, 2005.
Jenny Lagsten: Verksamhetsutvecklande utvärdering i informationssystemprojekt, 2005.
Emma Larsdotter Nilsson: Modeling, Simulation, and Visualization of Metabolic Pathways Using Modelica,
2005.
Christina Keller: Virtual Learning Environments in higher education. A study of students’ acceptance of educational technology, 2005.
Cécile Åberg: Integration of organizational workflows and the Semantic Web, 2005.
Anders Forsman: Standardisering som grund för informationssamverkan och IT-tjänster - En fallstudie baserad på
trafikinformationstjänsten RDS-TMC, 2005.
Yu-Hsing Huang: A systemic traffic accident model, 2005.
Jan Olausson: Att modellera uppdrag - grunder för förståelse av processinriktade informationssystem i
transaktionsintensiva verksamheter, 2005.
Petter Ahlström: Affärsstrategier för seniorbostadsmarknaden, 2005.
Mathias Cöster: Beyond IT and Productivity - How Digitization Transformed the Graphic Industry, 2005.
Åsa Horzella: Beyond IT and Productivity - Effects of Digitized Information Flows in Grocery Distribution, 2005.
Maria Kollberg: Beyond IT and Productivity - Effects of Digitized Information Flows in the Logging Industry,
2005.
David Dinka: Role and Identity - Experience of technology in professional settings, 2005.
No 1191
No 1192
No 1194
No 1204
No 1206
No 1207
No 1209
No 1225
No 1228
No 1229
No 1231
No 1233
No 1244
No 1248
No 1263
FiF-a 90
No 1272
No 1277
No 1283
FiF-a 91
No 1286
No 1293
No 1302
No 1303
No 1305
No 1306
No 1307
No 1309
No 1312
No 1313
No 1317
No 1320
No 1323
No 1329
No 1331
No 1332
No 1333
No 1337
No 1339
No 1351
No 1353
No 1356
No 1359
No 1361
No 1363
No 1371
No 1373
No 1381
No 1386
No 1387
No 1392
No 1393
No 1401
No 1410
No 1421
No 1427
No 1450
No 1459
No 1466
Andreas Hansson: Increasing the Storage Capacity of Recursive Auto-associative Memory by Segmenting Data,
2005.
Nicklas Bergfeldt: Towards Detached Communication for Robot Cooperation, 2005.
Dennis Maciuszek: Towards Dependable Virtual Companions for Later Life, 2005.
Beatrice Alenljung: Decision-making in the Requirements Engineering Process: A Human-centered Approach,
2005.
Anders Larsson: System-on-Chip Test Scheduling and Test Infrastructure Design, 2005.
John Wilander: Policy and Implementation Assurance for Software Security, 2005.
Andreas Käll: Översättningar av en managementmodell - En studie av införandet av Balanced Scorecard i ett
landsting, 2005.
He Tan: Aligning and Merging Biomedical Ontologies, 2006.
Artur Wilk: Descriptive Types for XML Query Language Xcerpt, 2006.
Per Olof Pettersson: Sampling-based Path Planning for an Autonomous Helicopter, 2006.
Kalle Burbeck: Adaptive Real-time Anomaly Detection for Safeguarding Critical Networks, 2006.
Daniela Mihailescu: Implementation Methodology in Action: A Study of an Enterprise Systems Implementation
Methodology, 2006.
Jörgen Skågeby: Public and Non-public gifting on the Internet, 2006.
Karolina Eliasson: The Use of Case-Based Reasoning in a Human-Robot Dialog System, 2006.
Misook Park-Westman: Managing Competence Development Programs in a Cross-Cultural Organisation - What
are the Barriers and Enablers, 2006.
Amra Halilovic: Ett praktikperspektiv på hantering av mjukvarukomponenter, 2006.
Raquel Flodström: A Framework for the Strategic Management of Information Technology, 2006.
Viacheslav Izosimov: Scheduling and Optimization of Fault-Tolerant Embedded Systems, 2006.
Håkan Hasewinkel: A Blueprint for Using Commercial Games off the Shelf in Defence Training, Education and
Research Simulations, 2006.
Hanna Broberg: Verksamhetsanpassade IT-stöd - Designteori och metod, 2006.
Robert Kaminski: Towards an XML Document Restructuring Framework, 2006.
Jiri Trnka: Prerequisites for data sharing in emergency management, 2007.
Björn Hägglund: A Framework for Designing Constraint Stores, 2007.
Daniel Andreasson: Slack-Time Aware Dynamic Routing Schemes for On-Chip Networks, 2007.
Magnus Ingmarsson: Modelling User Tasks and Intentions for Service Discovery in Ubiquitous Computing,
2007.
Gustaf Svedjemo: Ontology as Conceptual Schema when Modelling Historical Maps for Database Storage, 2007.
Gianpaolo Conte: Navigation Functionalities for an Autonomous UAV Helicopter, 2007.
Ola Leifler: User-Centric Critiquing in Command and Control: The DKExpert and ComPlan Approaches, 2007.
Henrik Svensson: Embodied simulation as off-line representation, 2007.
Zhiyuan He: System-on-Chip Test Scheduling with Defect-Probability and Temperature Considerations, 2007.
Jonas Elmqvist: Components, Safety Interfaces and Compositional Analysis, 2007.
Håkan Sundblad: Question Classification in Question Answering Systems, 2007.
Magnus Lundqvist: Information Demand and Use: Improving Information Flow within Small-scale Business
Contexts, 2007.
Martin Magnusson: Deductive Planning and Composite Actions in Temporal Action Logic, 2007.
Mikael Asplund: Restoring Consistency after Network Partitions, 2007.
Martin Fransson: Towards Individualized Drug Dosage - General Methods and Case Studies, 2007.
Karin Camara: A Visual Query Language Served by a Multi-sensor Environment, 2007.
David Broman: Safety, Security, and Semantic Aspects of Equation-Based Object-Oriented Languages and
Environments, 2007.
Mikhail Chalabine: Invasive Interactive Parallelization, 2007.
Susanna Nilsson: A Holistic Approach to Usability Evaluations of Mixed Reality Systems, 2008.
Shanai Ardi: A Model and Implementation of a Security Plug-in for the Software Life Cycle, 2008.
Erik Kuiper: Mobility and Routing in a Delay-tolerant Network of Unmanned Aerial Vehicles, 2008.
Jana Rambusch: Situated Play, 2008.
Martin Karresand: Completing the Picture - Fragments and Back Again, 2008.
Per Nyblom: Dynamic Abstraction for Interleaved Task Planning and Execution, 2008.
Fredrik Lantz: Terrain Object Recognition and Context Fusion for Decision Support, 2008.
Martin Östlund: Assistance Plus: 3D-mediated Advice-giving on Pharmaceutical Products, 2008.
Håkan Lundvall: Automatic Parallelization using Pipelining for Equation-Based Simulation Languages, 2008.
Mirko Thorstensson: Using Observers for Model Based Data Collection in Distributed Tactical Operations, 2008.
Bahlol Rahimi: Implementation of Health Information Systems, 2008.
Maria Holmqvist: Word Alignment by Re-using Parallel Phrases, 2008.
Mattias Eriksson: Integrated Software Pipelining, 2009.
Annika Öhgren: Towards an Ontology Development Methodology for Small and Medium-sized Enterprises,
2009.
Rickard Holsmark: Deadlock Free Routing in Mesh Networks on Chip with Regions, 2009.
Sara Stymne: Compound Processing for Phrase-Based Statistical Machine Translation, 2009.
Tommy Ellqvist: Supporting Scientific Collaboration through Workflows and Provenance, 2009.
Fabian Segelström: Visualisations in Service Design, 2010.
Min Bao: System Level Techniques for Temperature-Aware Energy Optimization, 2010.
Mohammad Saifullah: Exploring Biologically Inspired Interactive Networks for Object Recognition, 2011
No 1468
No 1469
No 1476
No 1481
No 1485
FiF-a 101
No 1490
No 1503
No 1504
No 1506
No 1507
No 1509
No 1510
No 1513
No 1523
No 1550
No 1558
No 1582
No 1586
No 1588
No 1592
Qiang Liu: Dealing with Missing Mappings and Structure in a Network of Ontologies, 2011.
Ruxandra Pop: Mapping Concurrent Applications to Multiprocessor Systems with Multithreaded Processors and
Network on Chip-Based Interconnections, 2011.
Per-Magnus Olsson: Positioning Algorithms for Surveillance Using Unmanned Aerial Vehicles, 2011.
Anna Vapen: Contributions to Web Authentication for Untrusted Computers, 2011.
Loove Broms: Sustainable Interactions: Studies in the Design of Energy Awareness Artefacts, 2011.
Johan Blomkvist: Conceptualising Prototypes in Service Design, 2011.
Håkan Warnquist: Computer-Assisted Troubleshooting for Efficient Off-board Diagnosis, 2011.
Jakob Rosén: Predictable Real-Time Applications on Multiprocessor Systems-on-Chip, 2011.
Usman Dastgeer: Skeleton Programming for Heterogeneous GPU-based Systems, 2011.
David Landén: Complex Task Allocation for Delegation: From Theory to Practice, 2011.
Kristian Stavåker: Contributions to Parallel Simulation of Equation-Based Models on
Graphics Processing Units, 2011.
Mariusz Wzorek: Selected Aspects of Navigation and Path Planning in Unmanned Aircraft Systems, 2011.
Piotr Rudol: Increasing Autonomy of Unmanned Aircraft Systems Through the Use of Imaging Sensors, 2011.
Anders Carstensen: The Evolution of the Connector View Concept: Enterprise Models for Interoperability
Solutions in the Extended Enterprise, 2011.
Jody Foo: Computational Terminology: Exploring Bilingual and Monolingual Term Extraction, 2012.
Anders Fröberg: Models and Tools for Distributed User Interface Development, 2012.
Dimitar Nikolov: Optimizing Fault Tolerance for Real-Time Systems, 2012.
Dennis Andersson: Mission Experience: How to Model and Capture it to Enable Vicarious Learning, 2013.
Massimiliano Raciti: Anomaly Detection and its Adaptation: Studies on Cyber-physical Systems, 2013.
Banafsheh Khademhosseinieh: Towards an Approach for Efficiency Evaluation of
Enterprise Modeling Methods, 2013.
Martin Sjölund: Tools for Understanding, Debugging, and Simulation Performance Improvement of EquationBased Models, 2013.