Linköping Studies in Science and Technology Licentiate Thesis No. 1582 Mission Experience: How to Model and Capture it to Enable Vicarious Learning by Dennis Andersson Department of Computer and Information Science Linköpings universitet SE-581 83 Linköping, Sweden Linköping 2013 This is a Swedish Licentiate´s Thesis Swedish postgraduate education leads to a Doctor´s degree and/or a Licentiate´s degree. A Doctor´s degree comprises 240 ECTS credits (4 years of full-time studies). A Licentiate´s degree comprises 120 ECTS credits. Copyright © 2013 Dennis Andersson ISBN 978-91-7519-660-2 ISSN 0280-7971 Printed by LiU-Tryck 2013 URL: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-90727 Mission Experience: How to Model and Capture it to Enable Vicarious Learning by Dennis Andersson May 2013 ISBN 978-91-7519-660-2 Linköping Studies in Science and Technology Licentiate Thesis No. 1582 ISSN 0280-7971 LiU-Tek-Lic-2013:16 ABSTRACT Organizations for humanitarian assistance, disaster response and military activities are characterized by their special role in society to resolve time-constrained and potentially lifethreatening situations. The tactical missions that these organizations conduct regularly are significantly dynamic in character, and sometimes impossible to fully comprehend and predict. In these situations, when control becomes opportunistic, the organizations are forced to rely on the collective experience of their personnel to respond effectively to the unfolding threats. Generating such experience through traditional means of training, exercising and apprenticeship, is expensive, time-consuming, and difficult to manage. This thesis explores how and why mission experience should be utilized in emergency management and military organizations to improve performance. A multimedia approach for capturing mission experience has further been tested in two case studies to determine how the commanders’ experiences can be externalized to enable vicarious learning. These studies propose a set of technical, methodological, and ethical issues that need to be considered when externalizing mission experience, based on two aforementioned case studies complemented by a literature review. The presented outcomes are (1) a model aligning abilities that tactical organizations need when responding to dynamic situations of different familiarity, (2) a review of the usefulness of several different data sources for externalization of commanders’ experiences from tactical operations, and (3) a review of methodological, technical, and ethical issues to consider when externalizing tactical military and emergency management operations. The results presented in this thesis indicate that multimedia approaches for capturing mission histories can indeed complement training and exercising as a method for generating valuable experience from tactical missions. This work has been supported by the Swedish Defense Research Agency, the Swedish Armed Forces, the Swedish Civil Contingencies Agency (formerly the Swedish Rescue Services Agency and the Swedish Emergency Management Agency) and Forum Securitatis. Department of Computer and Information Science Linköpings universitet SE-581 83 Linköping, Sweden ACKNOWLEDGMENTS This thesis is meant to summarize several years of research, a journey that has given me many encounters with great and intelligent people, generating invaluable insights to assist my research. To all those people, some of whom are mentioned below, I would like to send my deepest thanks, and a message that I hope for the future to hold many more encounters of this kind, helping me become a better researcher and person. This work would never have been conducted if it were not for my great supervisor, Prof. Niklas Hallberg, at the Swedish Defense Research Agency, who has always fought hard to make way for me as a researcher. His coaching has been beyond comparison and I cannot enough thank him for these efforts. On the coaching side, my secondary advisor Associate Professor Sofie Pilemalm at the Department of Management and Engineering at Linköping University has always given me great support and fantastic advice when I needed it the most, quickly sweeping away my doubts and concerns. Advice has also poured in from Professor Henrik Eriksson at the Department of Computer and Information Science at Linköping University, who has been a great tutor in writing and a wizard of flexible solutions. Without his help I would never have been able to navigate through the academic jungle. My fourth advisor, Associate Professor Alexander Bordetsky at the Department of Information Science at the Naval Postgraduate School, took me in for six months in Monterey, California, to learn the trades of American IS research. His mentoring has broadened my view on research and rewarded me with many pieces to the puzzle that I am trying to solve. In addition to my advisors mentioned above, I have received a lot of help and support from colleagues and friends worldwide. I would like to dedicate a special thank you to my colleagues Johan Allgurén and Mirko Thorstensson who took me in to join the MIND research group as a newly graduated student, and introduced me to the beautiful world of research. They have always believed in me, and fed me with interesting challenges and tasks that motivate my research. Equally motivating and stimulating has been the recurring discussions with Pär-Anders Albinsson, whose scientific mindset and constant thoroughness serves as a role-model to me. Their advice has often helped me to reduce the risk of cutting scientific corners too fast. For the particular studies presented in this thesis, I would like to dedicate an extra thank you to my hard-working co-authors, aforementioned Sofie Pilemalm and Niklas Hallberg, as well as PhD student Amy Rankin at the Department of Computer and Information Science at Linköping University. An extra thank you also to our dedicated librarian Barbro Löwemo who has assisted my literature reviews with superb professionalism. An early draft of this thesis was reviewed by Dr. Jiri Trnka, who gave me a lot of advice on how to finalize my work. A later draft was then carefully reviewed by my colleague Teodor Sommestad who contributed tremendously to this thesis with his constructive advice and recommendations, improving both quality and readability. I would also like to [i] direct my thanks to Pär Hjalmewik who has helped eliminate a set of grammatical errors in this thesis. A significant amount of support has been given to my research efforts by the Forum Securitatis graduate school, initiated by Professor Martin Holmberg and now supervised by Professor Peter Stenumgaard. It is a privilege to participate in this research collective and reap the benefits of collaboration and organization. Further support to this work has been given by the Swedish Defense Research Agency, the Swedish Rescue Services Agency1, the Swedish Emergency Management Agency2 and the Swedish Armed Forces. In the last stages of putting this work together, Anne Moe, Coordinator of Research Studies Administration at the Department of Computer and Information Science at Linköping University, made this thesis become reality with valuable advice and guidance that greatly reduced the administrative workload of being a PhD student. Finally, it is with great pleasure that I dedicate this thesis to my beloved Magdalena and our emerging family. I have enjoyed limitless support from Magdalena on both personal and professional level. I truly look forward to spending a life-time building a better future with you, Emanuel and Ossian. To all of you mentioned above, to my family and friends who always support me in everything I do, and to everyone contributing to my professional or personal life, thank you! Slycke, April 2013 Dennis Andersson 1 Statens Räddningsverk. Seized to exist in 2009, replaced by the Swedish Civil Contingencies Agency (Myndigheten för samhällsskydd och beredskap). 2 Krisberedskapsmyndigheten. Seized to exist in 2009, replaced by the Swedish Civil Contingencies Agency (Myndigheten för samhällsskydd och beredskap). [ii] LIST OF THESIS PUBLICATIONS This thesis is primarily based on the following publications, which are appended to the end of the thesis: Paper I: Andersson, D., & Rankin, A. (2012). Sharing Mission Experience in Tactical Organisations. Proceedings of the 9th International ISCRAM Conference. Vancouver, Canada: April 22-25. Paper II: Andersson D., Pilemalm S., & Hallberg N. (2008) Evaluation of Crisis Management Operations using Reconstruction and Exploration. Proceedings of the 5th International ISCRAM Conference, Washington, DC: May 4-7 (pp. 118-125). Paper III: Pilemalm, S., Andersson, D., & Hallberg, N. (2008). Reconstruction and Exploration of Large-scale Distributed Operations – Multimedia tools for Evaluation of Emergency Management Response. Journal of Emergency Management, 6(4), pp. 31-47. Published by Weston Medical Publishing, LLC. Paper IV: Andersson, D. (2011). Privacy and Distributed Tactical Operations Evaluation. Proceedings of the 4th International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services (CENTRIC 2011). Barcelona, Spain: October 23-29. For a complete list of references to academic publications by the author to date, see pages 71-72. [iii] [This page is intentionally left blank] [iv] CONTENTS Chapter 1: Introduction ..................................................................................................... 1 1.1 Objective and Research questions ........................................................................ 2 1.2 Contributions ........................................................................................................ 2 1.3 Assumptions and Limitations ............................................................................... 3 1.4 Outline .................................................................................................................. 3 1.5 Reading directions ................................................................................................ 3 Chapter 2: Background ..................................................................................................... 5 2.1 Tactical Organizations.......................................................................................... 5 2.2 Command and control theories ............................................................................ 5 2.3 Decision-making models ...................................................................................... 9 2.4 Mission Experience ............................................................................................ 12 2.5 Reconstruction & Exploration ............................................................................ 13 2.6 Knowledge Transfer ........................................................................................... 19 2.7 Workplace ethics ................................................................................................ 22 2.8 Research gap ...................................................................................................... 23 Chapter 3: Methods ......................................................................................................... 25 3.1 Study design ....................................................................................................... 25 3.2 Case studies ........................................................................................................ 25 3.3 Literature review ................................................................................................ 27 Chapter 4: Summary of appended papers ....................................................................... 31 4.1 Sharing mission experience in tactical organisations ......................................... 31 4.2 Evaluation of crisis management operations using Reconstruction and Exploration.................................................................................................................... 34 4.3 Reconstruction and Exploration of large-scale distributed tactical operations .. 37 4.4 Privacy and Distributed Tactical Operations Evaluation ................................... 40 [v] Chapter 5: Results ........................................................................................................... 43 5.1 What abilities are important for commanders, to cope with dynamic decisionmaking situations that arise in tactical military and emergency management operations? .................................................................................................................... 43 5.2 How should commanders’ mission experience be externalized in order to enable vicarious learning from tactical military and emergency management operations? ..... 45 5.3 What technical, methodological, and ethical issues need to be considered when externalizing mission experience from tactical military and emergency management commanders? ................................................................................................................ 47 Chapter 6: Discussion ..................................................................................................... 49 6.1 Tactical command and control in dynamic situations ........................................ 49 6.2 Transferal of mission experience ....................................................................... 49 6.3 Reconstruction and Exploration ......................................................................... 50 6.4 Privacy................................................................................................................ 51 6.5 Summary ............................................................................................................ 52 6.6 Future work ........................................................................................................ 52 6.7 Conclusions ........................................................................................................ 52 References ......................................................................................................................... 55 List of Figures ................................................................................................................... 67 List of Tables .................................................................................................................... 69 Author’s publications ........................................................................................................ 71 Paper I: Sharing Mission Experience in Tactical Organisations ..................................... 73 Paper II: Evaluation of crisis management operations using Reconstruction and Exploration........................................................................................................................ 87 Paper III: Reconstruction & Exploration of Large-Scale Distributed Tactical Operations – Multimedia tools for evaluation of emergency management response ......................... 99 Paper IV: Privacy and Distributed Tactical Operations Evaluation .............................. 123 [vi] Chapter 1 INTRODUCTION Crisis management and military organizations are regularly exposed to complex, risky and sometime dangerous situations where difficult decisions can be the difference between life and death. With such extreme decision-making contexts, society benefits from any improvement possible in the decision-making process. This research deals with the acquisition and transfer of mission experience that may benefit the commanders’ ability of making decisions in time-constrained situations. Humanitarian aid, disaster relief, and military organizations rely heavily on their personnel to conduct and observe missions and exercises to acquire the necessary knowledge for their work. In other words, these organizations need experienced personnel (King, 2005). This statement is consistent with the more general idea of naturalistic decision making which states that decisions are often based on recognition and mapping of current situations to prior knowledge rather than through careful analysis of current alternatives as the rationalistic decision making models suggest (Klein, 1993; Howard, 1966). Tactical command and control (C2) of operations involve many decision points for commanders (Alberts & Hayes, 2006, p.8). These decisions can sometimes be urgent and require immediate actions (ibid., p. 82) limiting the commander’s ability to carefully analyze each option and therefore to rely on recognition primed decision making or experience to respond in the best way (Klein, 1993). As experience seems to be an important factor in tactical decision-making, understanding how to acquire and utilize it plays a central role in development of tactical organizations. Experience can, from an organization management point of view, be defined as “what transpires in an organization as it performs its tasks” (Argote & Miron-Spektor, 2011). Making mission experience explicit, i.e., externalization, is a major challenge for tactical organizations (Nonaka, 1991; King, 2005). Synthesis is also needed along with compilation, storage and organization of accumulated data, information and knowledge, to enable retrieval and dissemination using effective methods and systems (Nonaka, 1991). Composite knowledge, such as mission experience, has both tacit and explicit components. Although Polanyi (1966) claimed that they are not meaningful to separate in practice, conceptually this separation allows theorization about the interaction between the two to stimulate knowledge creation (Nonaka, 1991). Explicit knowledge can be stored and shared within and between organizations, as it can be articulated, while tacit knowledge requires socialization or internalization processes (ibid.). Today, most tactical organizations rely on within-organization socialization and internalization of standardized routines, while knowledge transfer between organizations is rare (King, 2005). There is thus room and potential for new and improved methods of knowledge sharing, both within and between organizations. [1] This research is motivated by the challenges proposed above, and addresses methods of synthesizing, compiling and storing data and information pertaining to the experiences acquired from conducting tactical military and emergency management operations and exercises. 1.1 OBJECTIVE AND RESEARCH QUESTIONS The objective of this thesis is to explore approaches of sharing mission experience from the perspective of C2. This by addressing the following research questions (RQ): RQ 1. What abilities are important for commanders, to cope with dynamic decision-making situations that arise in tactical military and emergency management operations? RQ 2. How should commanders’ mission experience be externalized in order to enable vicarious learning from tactical military and emergency management operations? RQ 3. What technical, methodological, and ethical issues need to be considered when externalizing mission experience from tactical military and emergency management commanders? This thesis is based on four scholarly research articles, appended as Papers I-IV. Each of these articles represents an individual study, which contribute to answering the three RQs. 1.2 CONTRIBUTIONS By answering the RQs defined above, this thesis contributes to research on knowledge transfer for tactical teams. The following contributions are yielded from the process of answering the research questions above: A model to describe, and separate, abilities needed to cope with dynamic situations such as tactical military and emergency management operations. An approach for collecting and externalizing mission histories by recording C2 activities through reconstruction. A review of usefulness of different data sources for the externalization of mission experience. A review of technical, methodological, and ethical issues that need to be considered when externalizing tactical military and emergency management operations. In addition, mission histories, as externalized through the reconstruction & exploration approach presented in this thesis, can serve an immediate purpose to support after-action reviews, hot wash, debriefings and disseminations after missions, as well to act as a knowledge base for (1) future decision making, (2) training and education of commanders in military and civil service organizations, and (3) sharing knowledge within and between such organizations. [2] 1.3 ASSUMPTIONS AND LIMITATIONS Although an organization can have a wide scope of tasks, this study is limited to tactical missions conducted by emergency response and military organizations. The approach proposed in this research was initially developed for post-mission analysis and afteraction reviews, a purpose for which mission history reconstruction has been applied since the 1990’s (Worm, Jenvald & Morin, 1998). The proposed model adds another dimension to mission reconstruction by focusing explicitly on experience transfer. Verification of the proposed methodology, however, is out of scope of this thesis. The technical, methodological and ethical issues that are explored in this thesis were originally studied from an evaluation point of view, a somewhat different purpose than that of knowledge transfer. Whether the same issues apply in both contexts or not has not been investigated. It should also be noted that these issues are fetched from two case studies performed at Swedish organizations and a literature review primarily based on ethics literature from western culture. Cultural differences, e.g., in organization management, laws and ethics, may affect the validity of this study if applied elsewhere as cultural aspects have not been considered while performing the studies presented herein. 1.4 OUTLINE This chapter, Chapter 1, describes the motivation and scoping of this thesis by defining research questions, scientific contribution and limitations of the research. Chapter 2 describes the related background and theory that the thesis is based upon. The third chapter briefly describes the methods used in each of the four studies, while Chapter 4 summarizes the studies in relation to the RQs. Chapter 5 presents the results, i.e., the outcomes of the four studies, which are then elaborated upon, discussed, and concluded in Chapter 6. 1.5 READING DIRECTIONS The suggested reading directions differ depending on the reader’s interest and familiarity of the topics discussed in this thesis. Novice readers who want a brief introduction to research on adjacent and backing theories that the rest of the work relies upon are advised to start by reading Chapter 2. For better understanding of the work that the thesis is based upon, Chapters 3-4 describe the research methods and summarizes each underlying study, while the appended Papers I-IV provide the full story that should then be followed by Chapters 5-6. Anyone interested in just a brief summary of the work presented in this thesis should skip Chapters 3-4, and concentrate on Chapters 5-6, possibly adding parts of Chapter 4 for a motivation of the presented ideas where appropriate. [3] [This page is intentionally left blank] [4] Chapter 2 BACKGROUND This chapter, Background, presents research on (1) tactical organizations such as military and emergency management organizations, (2) tactical command and control theories on how teams operate in tactical situations and thus give hints on how to study C2, (3) decision-making models providing different viewpoints on the cognition behind decision making, (4) mission experience on defining what mission experience really means, (5) reconstruction and exploration as a methodology for evaluation of operations of the kind that tactical military and emergency management units typically conduct, (6) knowledge transfer on theories from the knowledge management field on how knowledge can be transferred, and (7) workplace ethics including definitions relevant to studying the concept of privacy in a workplace setting. Finally, a summary of the identified research gap is outlined. 2.1 TACTICAL ORGANIZATIONS The term tactical organization is used throughout this document to denominate all organizations, civil and military alike, designed to perform field missions requiring tactical command and control. Tactical organizations typically operate in complex and dynamic environments and include both military organizations, humanitarian aid/disaster relief organizations (HA/DR) and emergency services; with examples ranging from national armed forces, the International Red Cross/Crescent and UNESCO to local fire brigades, police forces and health-care departments. Tactical organizations can sometimes be described as adhocracies, ad-hoc operative organizations that rely on specialized teams to perform their respective tasks (Mintzberg, 1979). An adhocracy is further described as a contemporary collection of teams that “operate in an environment that is both dynamic and complex, demanding innovation of a fairly sophisticated nature” (Mintzberg & McHugh, 1985, p.160). A tactical organization can thus be seen as a special case of an adhocracy, one that specifically deals with tactical operations in complex and dynamic environments. The adhocracy analogy yields access to a body of knowledge on how these organizations function, what their operational constraints and requirements are, and how they affect what experience needs to be modeled and captured to support dissemination and knowledge transfer. 2.2 COMMAND AND CONTROL THEORIES Command and control (C2) is “the exercise of authority and direction by a properly designated commander over assigned forces in the accomplishment of the mission. C2 functions are performed through an arrangement of personnel, equipment, communications, facilities, and procedures which are employed by commanders in planning, directing, coordinating, and controlling forces and operations in the accomplishment of the mission.” (Builder, Bankes & Nordin, 1999, p. 11). Different models have been proposed in various attempts to better understand and improve the C2 capability, such as the C2 systems functional model (Finley, Muckler, Gainer & [5] Obermayer, 1975), Lawson’s process model (Lawson, 1981), SHOR (Wohl, 1981), OODA (Boyd, 1987), Joint Cognitive Systems (Hollnagel, 1993), Dynamic OODA (Brehmer, 2005), M-OODA, T-OODA, and C-OODA (Breton & Rousseau, 2008). More information on the history of, and relationships between, different C2 models and theories can be found in Builder et al.’s (1999) extensive review. 2.2.1 THE OODA LOOP Many C2 doctrines today are inspired by the OODA-loop (Boyd, 1987). The OODA loop has four components: Observe, Orient, Decide and Act, which form a cycle that the commander must iterate again and again. Boyd’s original claim was that the ability to complete an iteration of the loop quicker than an opponent gives an edge in battle (Brehmer, 2005). Attempts have since been made to generalize the model and apply it to new domains, as exemplified by Boyd himself in the more general, although no longer loop, version of OODA, see Figure 2.1. Figure 2.1. The extended OODA loop (Boyd, 1987). Reproduced from http://dnipogo.org/john-r-boyd. The popularity of the OODA loop has led to several spin-offs that attempt to model C2 from specific viewpoints, such as teamwork, cognition, and control, with the belief that models based on OODA will stand a better chance of reaching acceptance in the military community due to its familiarity within the community (Breton & Rousseau, 2008, p. 28). The M-OODA loop is a modular version of OODA that allows extensions such as for cognition or teamwork, see Figure 2.2 (ibid.). [6] Dynamic and complex situations OODA Loop Team DM T-OODA Loop M-OODA Loop Cognitition of C2 decision cycle C-OODA Loop Figure 2.2. M-OODA enables further extensions such as T-OODA or C-OODA depending on the modeling needs (after Breton and Rousseau, 2008). 2.2.2 DYNAMIC OODA Brehmer (2005) stated that the OODA loop should not be seen as a model of C2, but rather as an explanation of why American fighter pilots were performing better than their opponents during the Korean War, i.e., by iterating the loop quicker. To facilitate the contextual elements such as mission, goal, command concept and other activities, Brehmer took a different path than Breton and Rousseau to introduce the Dynamic OODA loop (DOODA), see Figure 2.3. The DOODA loop introduces cybernetic inputs and outputs to the system as well as an entry and exit condition which, according to Brehmer, was lacking in Boyd’s OODA loop (ibid.). Figure 2.3. Brehmer’s Dynamic OODA loop (adapted by Oosthuizen & Roodt, 2012). Copyright © 2012 CSIR South Africa. Reprinted with permission. Brehmer’s intention with the Dynamic OODA loop was to create a unified model of military command that is applicable on all command levels, albeit still with a single commander in focus just as Boyd had. On the tactical level, this model highlights the [7] importance of adapting to the changing environment, i.e., being flexible. The reliance on flexibility and adaptability is nothing new, but can be dated back at least to the 19th century when the Prussian army recognized a need for modernization after being defeated by the Napoleonic forces. They developed mission command (auftragstaktik) allowing their subordinates to be more flexible and gain initiative and momentum over opponents who employed traditional authorative command (befehlstaktik) (Widder, 2002). This command method has been doctrine in the German forces ever since, and is now widely spread over the world, it is for example still used in the latest version of the Swedish Military Strategic Doctrine (Försvarsmakten, 2011). 2.2.3 JOINT COGNITIVE SYSTEMS A tactical unit, comprised of technological systems, C2 systems and support systems, can be described as a Joint Cognitive System (JCS) (Hollnagel, 1993). In tactical HA/DR operations, multiple organizations are working toward the same goal under different command, sometimes leading to uncertainties and ambiguities regarding responsibilities and authorities (Morin, 2001). JCS models frequently and deliberately focus on the control component, whereas commanding is more central in the models of C2 discussed above. The military command concept is in cognitive systems engineering sometimes referred to as direct, e.g., in the Contextual Control Model (COCOM) (Hollnagel, 1993) and the subsequent Extended Control Model (ECOM) (Hollnagel & Woods, 2005). Both these models are based on the premise that no procedural patterns exist and that actions are instead determined by the context and the competence of the JCS (ibid.). This reasoning is similar to the ideas behind military mission command and Brehmer’s DOODA loop, as they too highlight the importance of adaptivity and flexibility. The minimalistic COCOM model is intended to be applicable on both the individual level, for tactical organizations and in the strategic domain (ibid.). It has only three constituents: (1) competence which represents the set of possible actions and responses that a JCS can apply to a situation, (2) control refers to the way in which the competence is applied, and (3) constructs are the (internal) descriptions of the situation that the JCS employ to evaluate an event and select appropriate actions (ibid.). The control in COCOM can take one of four different modes: (1) scrambled, which is basically random selection without reflection, (2) opportunistic, where the next action is determined by salient features of the current state with limited planning, (3) tactical, where short-term control more or less follows known procedures and rules, and (4) strategic, which has a longer time horizon than tactical and may correspond to higherlevel goals (ibid.). A tactical unit can work in all four control modes, of which the strategic is preferable (ibid.). Typical for a high-risk dynamic situation is that time does not allow the unit to operate in strategic mode. Instead, the unit turns to tactical mode, or even opportunistic if competence is inadequate for tactical (Svensson, 2010). The scrambled mode is the least preferred and subject to deficient situation assessment. [8] 2.3 DECISION-MAKING MODELS Sensing, understanding, decision making and action performance are all important in most models of tactical C2 (including the JCS COCOM model). Hence, there is an apparent need to understand these four concepts in order to understand tactical missions. 2.3.1 SENSING Boyd (1987) used the word observe to describe sensing, Brehmer (2005) instead chose the term information collection. Although the terminology can be seen as nuances, they do have one important difference namely that information collection has an active tense whereas observation is rather passive. Brehmer thus included active searching for information in his sensing concept. Regardless of whether it is called observation or information collection, the products of these processes are put together to create a mental model of the current situation, i.e., situation understanding. 2.3.2 SITUATION UNDERSTANDING Situation understanding, or sensemaking, is an art of finding contextual rationality from vague or chaotic pieces of information (Weick, 1993). Decision-making research has seen a plethora of definitions of sensemaking, e.g.: “the process of placing stimuli into some kind of framework” (Waterman, 1990, p.41). “a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively” (Klein, Moor & Hoffman, 2006a, p.71). Waterman’s definition is close to the Orient part of Boyd’s OODA loop, i.e., interpreting the environment and framing it by personal context such as previous knowledge. Weick (1995) stated similarly that sensemaking is simply about “making sense, and about structuring the unknown”. Klein and colleagues created a model of sensemaking based on the ground that it is not adequately understood, and that there is a need for a framework to discuss it. Their main critique was that it is often incorrectly understood as a single step in a linear sequential decision making process, i.e., that sensemaking leads directly to decisions. Instead they argued that sensemaking is a continuous process without a clearly defined beginning and end that may influence decision and actions (Klein, Moor & Hoffman, 2006b). They proposed a framework using a data/frame theory as shown in Figure 2.4. The theory says that data, or stimuli, is interpreted by the sensemaker, and then recognized and put into some kind of reference frame. These frames are then elaborated as more and more data becomes available. Continuous conscious or subconscious questioning of said frames help reframing and making sense by coming to better and more refined conclusions (ibid.). A frame is thus a mental model, or construction, of a sensemaker, i.e., a construct in a JCS. [9] Figure 2.4. A frames-based model of sensemaking (Klein et al., 2006b). Copyright © 2006 IEEE. Reprinted with permission. Endsley (1995) proposed a model of situation awareness (SA) that models three levels of awareness: (1) perception, (2) understanding, and (3) projection, see Figure 2.5. Perception, or level 1 SA, is the state in which a JCS stands when passively observing, or not being able to comprehend, an ongoing situation. The only control modes available for a JCS in this level are scrambled or opportunistic. In order for a JCS to open up the tactical control mode, the situation must be understood (level 2 SA) so that the proper routines can be implemented. Level 3 SA, or projection, is the desired state, which can be reached through a rigorous process of sensemaking (Starbuck & Milliken, 1988). It is only through level 3 SA that the strategic control mode can be utilized. Sensemaking can thus be summarized as a retrospective process of interpreting stimuli that can improve a decision maker’s SA (Endsley, 2004). In strategic operations the time span may be long enough to allow careful analysis and rational reasoning to deduce the best route forward. However, in dynamic tactical environments the situation may require quick and reactive response. [10] Figure 2.5. Situation awareness (Endsley 1995). Copyright (1995) Human Factors and Ergonomics Society. Reprinted with permission. 2.3.3 DECISION MAKING Rational decision making has seen several different angles and is by many considered to be a superior decision-making strategy. The main idea is that decisions are made by applying knowledge and biases to select what is perceived to be the best among multiple alternatives (e.g., Tversky & Kahneman, 1981). In dynamic situations like military operations or emergency response, optimal or rational decisions cannot always be made, e.g., when the problem space grows too complex to explore fully, or the time horizon narrows too much (Fredholm & Åström, 2010). Instead, decision making then turns into a function for gaining control (Brehmer, 1992) and the JCS falls into opportunistic control mode because it is lacking in time or competence to select an appropriate plan (Hollnagel & Woods, 2005). In these cases, commanders tend to resort to experience, habits or association instead of rational analytics (ibid.). Klein (1993) labeled this recognitionprimed decision making (RPDM). The recognition-primed decision making model, see Figure 2.6, is highly compatible with Klein et al.’s sensemaking framework (Klein et al., 2006b) as they both build on the idea that the target interpret cues and assess a situation based on personal experience. [11] Experience the situation in a changing context • • Reassess situation Seek more information No Is the situation familiar? Yes Yes Are expectancies violated? Activation of information from memory No • • • • Plausible goals Expectancies Relevant cues Actions 1..N Mental simulation of action(n) Yes, but Modify Will it work? No Yes Implement Figure 2.6. Recognition-primed decision making in complex, and changing contexts (after Klein, 1993). 2.4 MISSION EXPERIENCE In RPDM, experience is a central factor. In fact, Klein (1995) claimed that lack of experience may lead to poor sense of typicality, poor ability to see patterns and make discriminations, no basis for expectancies and missing causal knowledge. This in turn may lead to failure to recognize anomalies, misjudging urgencies, misinterpreted situations and unawareness of weaknesses in a selected course of action (ibid.). These circumstances are what Hollnagel and Woods (2005) referred to as lack of competence, which forces the tactical unit to employ opportunistic control at best. Mission experience is what transpires in organizations as they perform missions (Argote & Miron-Spektor, 2011). It can be acquired primarily through hands-on practice (King, 2005). Events that are particularly infrequent or unforeseen are extra difficult to gain expertise in controlling. Organizations that are required to deal with these must therefore rely on finding ways to do so without the assistance of standard operating procedures, i.e., improve their ability to perform opportunistic control, e.g., through adaptation of known procedures or improvisation (Mendonça, Beroggi & Wallace, 2001; Zumel, Franco & Beutler, 2008; Trnka, Rankin, Jungert, Lundberg, Granlund, Granlund & Johansson, 2009; Rankin, Dahlbäck & Lundberg, 2013). [12] 2.5 RECONSTRUCTION & EXPLORATION Evaluation of operations is a key enabler for tactical organizations to better understand their operations and evolve. Both real and exercised operations are candidates for evaluation, and characterized by the multiple sources of complexity involved. With multiple ongoing tasks at different sites, extensive dynamism and complex cause-andeffect relationships; maintaining situation awareness of the operations is difficult. Extra complexity is added with the unexpected events and actions taken over time (Brehmer 1992; Albinsson, Wong, Pilemalm & Morin, 2005; Alberts & Hayes, 2007; Andersson, 2009). In large-scale complex multi-agency operations, keeping track of cause and effects may be cumbersome for an analyst. To support the analyst, Reconstruction and Exploration (R&E) has been suggested as an evaluation approach, using synchronized playback of heterogeneous data such as audio, video, GPS tracks, system logs, and other sensory data (Morin, 2001, 2002). R&E can also be used to assist training instructors in their afteraction reviews (AAR), as it helps clarifying what actually happened at different sites. Such AARs can be conducted within respective team; or on a larger scale, highlighting intra-team coordination and the complexity of inter-organizational command (U.S. Army Combined Arms Center, 2011). The Reconstruction phase involves several processes for collecting and synchronizing a rich dataset, a mission history, modeling the course of events of a tactical operation. The data types used are dependent on the nature of that operation, and of the required analyses, as such R&E is a flexible approach allowing the researcher to customize the model to current needs. The resulting mission history needs to convey both contextual data such as the organization model, and data samples, as exemplified in Figure 2.7. Mission history Organization model (participating units) Data collection model (links events to units) Events (chronological list) Data (list of data samples) Figure 2.7. A representation of mission histories (from Paper II). The Exploration phase involves playback and analysis of the mission history. Exploration can be thought of as an iterative process consisting of three steps in each cycle: visualization, analysis and integration (Andersson, 2009); allowing the analyst to [13] feedback analysis products into the model. Iterating this methodology enables raising the analysis to higher abstraction levels, until reaching the questions in focus (Albinsson et al., 2005). The R&E approach has been used extensively to evaluate training events, simulations and experiments (e.g., Morin, Jenvald & Worm, 1998; Thorstensson, Morin & Jenvald, 1999; Thorstensson, Axelsson, Morin & Jenvald, 2001; Lantz, Andersson, Jungert & Levin, 2009; Andersson, Mullins, Bourakov, Bordetsky & Forsgren, 2011). The main application of R&E in these cases has been mission efficiency analysis (MEA), i.e., a hybrid between teamwork and task work analysis (Worm et al., 1998). A survey has indicated that these reconstructed mission histories may be applicable for knowledge transfer by combining it with traditional textbooks and hypermedia information into a set of digital courseware (Jenvald, Rejnus, Morin & Thorstensson, 1998; Jenvald, Morin & Rejnus, 2000). 2.5.1 SOFTWARE FOR R&E The R&E approach was developed together with a set of tools that support the data compilation and visualization steps. The first of those tools was the MIND Studio, a piece of software with the capability of retrospectively visualizing a time-synchronized multimedia model of the reconstructed course of events (Jenvald, 1999; Morin, 2001), see Figure 2.8. The MIND Studio provides several customizable viewports into one dataset, allowing the explorers to review the mission history from different angles. The main navigation control is the timeline, which allows all data to be visualized in causal order as they happened. Figure 2.8. A screenshot from MIND Studio, exploring an Airborne Battalion’s final exercise, ASÖ’03. Compiling a dataset for the MIND Studio requires implementation of data adapters that can interpret the data that was captured through the reconstruction phase of R&E. These implementations require understanding of the different data formats which may be [14] cumbersome, but it also enables MIND to stay data agnostic, ensuring that data from any data source can be visualized, once interpreted. The F-REX Studio is the Swedish Defense Research Agency’s successor of MIND, developed to better support massive datasets and collaborative analysis of C2 exercises and operations (Andersson, 2009). The need for this redevelopment was identified with the observation that data collection methods and tools were improving quicker than the ability to manage voluminous data. An early version of the framework, Figure 2.9, provides roughly the same features as the original MIND framework albeit perceived as more flexible and modular by the developers, allowing for more rapid module development. Although the visual differences between MIND and F-REX are minor, the technology shift meant lower costs for the development and maintenance of the framework. Figure 2.9. F-REX is the successor of MIND, here used to explore the interactions of a commander during a training session at the Sandö facility (from Paper II). Figure 2.10 shows a more recent version capable of handling massive datasets from tactical operations also in the virtual realm such as Cyber Defense Exercises. These operations typically generate comparatively huge datasets when capturing also virtual stimuli such as IP traffic and hack attempts, calling for careful design and implementation of the data processing units and the visualization tools (Andersson, Granåsen, Sundmark, Holm & Hallberg, 2011). The R&E approach supports a wide system perspective gathering factual, simultaneous, real-time data on multiple organizational levels on multiple geographical locations and from multiple actors. Use of exploration tools, such as MIND and F-REX, enables the analyst to quickly switch focus between different viewpoints in any substantial dataset to [15] get a birds-eye view of the DTO (Andersson, 2009). In summary, the R&E approach can be seen as a method of capturing and disseminating experience in the form of multimedia-rich stories (Rao & Goldman-Segall, 1995; Liu, Liao & Pratt, 2009). Figure 2.10. A recent version of F-REX Studio used to explore the interactions between team members along with stimuli captured from both the physical and the virtual realm at a national cyber defense training session (Andersson, 2012). 2.5.2 MIXED METHODS DATA ANALYSIS R&E is a mixed methods approach that tries to overcome the shortcomings of both qualitative and quantitative methods by combining them and letting them complement each other. Mixed methods research is comparatively new, and sometimes criticized, but it has been increasingly popular in social and behavioral research (Lieber, 2009), and medical research (Schifferdecker & Reed, 2009). As the research approaches are similar, it can be rewarding to follow the methodological developments in these areas and see if they can be applied to mixed methods C2 research as well. As the size of the mixed methods research project scales up, the need for proper tools increases rapidly. Logically, software frameworks have emerged to assist researchers in managing data, results, and findings, such as EthnoNotes (Lieber, Weisner & Presley, 2003) and its successor Dedoose3. QDA Miner4 is another popular alternative which focuses specifically on the coding and analysis processes, while also assisting the researchers with semi-automatic report generation (Lewis & Maas, 2007). While these 3 http://www.dedoose.com, retrieved 2013-03-05. 4 http://provalisresearch.com/products/qualitative-data-analysis-software, retrieved 2013-03-05. [16] commercial products are powerful, they do not fully support the C2 use case in which cues from the environment as well as other contextual data has to be integrated with the decision-making cues to analyze the thought processes and understand why a certain decision was made. For this purpose, F-REX is commonly used as well as the CITE toolset5, which also spun off from MIND. CITE, with less on big data management, takes a more holistic approach compared to F-REX, supporting exercise controllers by including planning and management tools in addition to the evaluation component. Venkatesh, Brown and Bala (2013) studied the applicability of mixed methods in the Information systems (IS) research field. They accurately describe a helpful set of guidelines for when and how to work with mixed methods instead of traditional qualitative or quantitative methods, and illustrate the guidelines to two earlier IS research papers. These guidelines may be applicable to determine whether R&E can and should be used to externalize mission experience. 2.5.3 MISSION ANALYSIS There are many alternative approaches for assessing tactical operations, some taking the route of focusing on team performance, including the widely spread Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 2000) and Command Team Effectiveness (CTEF) (Essens, Vogelaar, Mylle, Blendell, Paris, Halpin & Baranski, 2005). R&E on the other hand, was primarily developed to analyze mission efficiency from a cybernetics perspective (Ashby, 1956; Worm et al., 1998), see Figure 2.11. Using this approach, the output can be seen as a consequence of the system, its inputs and any disturbances. The accuracy of any efficiency metric will be determined by the level of detail in which the input, output and disturbances have been modeled and captured. Figure 2.11. A control theory based model of tactical missions (Worm et al., 1998). Copyright (1998) Elsevier. Reprinted with permission. 5 Visuell Systemteknik i Linköping AB, http://www.vsl.se/Page.aspx?content=CiteBase, retrieved 2013-0305. [17] Mission efficiency can be qualitatively determined through R&E, by reconstructing the initial state (the reference), the system itself (including the control element) and the course of events (control, disturbances, and output). There is, however, an inherent complexity in the dynamicity of system properties rendering the system difficult to measure (Brehmer, 1992). To overcome this, a wide system boundary is used, extending the unit of analysis to tactical units including technological systems, C2 systems and support systems (Worm et al., 1998; Worm, 1999). By explicitly avoiding assessment of individual thought processes, the problem of measuring mission efficiency is somewhat simplified. Worm and colleagues (1998) utilized this by measuring the system through information processing and resource utilization, thus defining mission efficiency as the impact and outcome of a mission related to the resources utilized (ibid.), see Figure 2.12. Mission Efficiency: The impact and outcome of an emergency response mission, related to the resources utilized. Initial mission state Mission course of events Information processing during the mission Decision making and actions during the mission Utilization of available resources during the mission Figure 2.12. Model employed to measure the efficiency of emergency response missions (after Worm et al., 1998). The mission efficiency analysis method uses subjective analysis of the mission outcome and course of events as a basis for analyzing team performance, topped with objective measures of resource utilization and possibly information processing (Worm et al., 1998). Similar methodologies are commonly adopted for mission debriefings such as through AARs (U.S. Army Combined Arms Center, 2011). More objective alternatives have been proposed, based on a set of specific task performance metrics, such as exposure to risk in urban warfare (Rowe, 2009; Sadagic, Welch, Basu, Darken, Kumar, Fuchs, Cheng, Frahm, Kolsch, Rowe, Towles, Wachs & Lastra, 2009) or communication (LaVoie, Foltz, Rosenstein, Oberbreckling, Chatham & Psotka, 2008). [18] Situation awareness is another popular indicator of team performance, measured with, e.g., SAGAT (Endsley, 2000). The technique uses interruptions in task performance to objectively assess situation understanding of the team members. The method is intrusive as it requires interruptions of task performance to measure the SA at certain times, unless post-tests are used, in which case the accuracy of the metric is subject to the team members’ ability to retroactively simulate prior situation understanding. Non-intrusive, on-the-fly, assessment of SA can be acquired subjectively by observers, although the accuracy is then subject to the observers’ ability to convert observable actions into a measure of SA. The CTEF model was proposed by NATO as a comprehensive approach to assessing command teams, including both task outcomes and team outcomes (Essens et al., 2005; Hof, de Koning & Essens, 2010). The instrument is an important component in the assessment of tactical units modeling team effectiveness as a function of processes applied to preconditions. Decision making and plan execution are two of the many processes that CTEF covers, making the model compatible with OODA and other models of C2. CTEF however, shows an explicit focus on motivating and other non-task oriented behaviors, thereby implying that domain knowledge is just one of many components that make a tactical team function (ibid.). 2.6 KNOWLEDGE TRANSFER Experience is a relativistic form of knowledge always contingent and contextual, i.e., it cannot be made objective and universal (Cilliers, 2000). Yet, learning from experience is possible not only from one’s own experience, but also from others’, i.e., vicarious or observational learning (Argote & Miron-Spektor, 2011, p.1126). It is thus relevant to explore different methods of transferring knowledge in the pursuit for alternative, possibly cheaper, ways of acquiring mission experience than through hands-on practice. 2.6.1 KNOWLEDGE TRANSFER MODELS Research on knowledge transfer is plentiful, and many attempts have been made to explore the concept by stimulating certain behaviors such as improvisation and communication through innovative training and role-playing (e.g., Mendonça & Fiedrich, 2006; Trnka & Jenvald, 2006). Other studies of particular interest include a virtual reality approach by trying to simulate realistic operations to induce learning effects similar to those gained from complex operations (Lafond & DuCharme, 2011). One of the most well-known, although sometimes criticized, knowledge transfer models is the SECI model (Nonaka, 1991; Nonaka & Takeuchi, 1995; Gourlay, 2003). According to the model, there are interaction processes between tacit and explicit knowledge: (1) socialization (empathizing, acquiring tacit knowledge from others’ tacit knowledge), (2) externalization (articulating, creating explicit knowledge from tacit), (3) combination (cross-referencing multiple bodies of explicit knowledge, connecting the dots), and (4) internalization (embodying, or applying externalized knowledge in practice). These four interaction processes form a spiral, see Figure 2.13, which organizations and individuals can iterate over and over again to generate new knowledge (Nonaka, 1991). [19] Socialization Externalization Internalization Combination Explicit knowledge Tacit knowledge Tacit knowledge Explicit knowledge Figure 2.13. The SECI model of knowledge creation (after Nonaka & Takeuchi, 1995). A third dimension, Ba, has been added to the SECI model to represent the shared context without which shared knowledge cannot be created, i.e., a common baseline to share knowledge or experience (Nonaka, Toyama & Konno, 2000). Generation and regeneration of Ba provides energy, quality and a place to perform the individual conversion between tacit and explicit, allowing the individual to move along the knowledge-creating spiral (ibid.). To successfully attain experience through vicarious learning, it is thus essential to establish this common understanding of the context in which a mission occurs. 2.6.2 TRAINING AS A MEANS OF KNOWLEDGE TRANSFER Training has a role in any tactical organization as the main method of acquiring and refining skills, both physical skills and more cognitive skills such as decision making, leadership and C2. It is through this training that tactical units learn the basics and generate a library of procedures and tricks to resolve standard tasks. It is through training that trainees learn how to apply their means to reach a well-defined end. The decision-making process that occur in a dynamic situation is typically complex in nature and not considered an end by itself, but rather a means through which the, often ill-defined, objectives can be reached. As the events unfold and the situation evolves, the commander will need to adapt and reassess, so the decision-making process is continuous and interwoven with the performance phases (Means, Salas, Crandall & Jacobs, 1993). Training commanders for such events through traditional rationalistic decision theory have not been shown very successful, neither has bias reduction training (ibid.). Instead, decision-making training must capture the characteristics of the tasks and the context within which they are performed (ibid.). 2.6.3 DIGITAL COURSEWARE FOR MISSION EXPERIENCE TRANSFER Properly organized training promotes rapid learning by giving trainees the opportunity to internalize knowledge that they acquired through other means, and receive constructive feedback. Full-scale training yields an implicit opportunity for all participants to put their own actions into a larger scope, to see their roles from a new perspective. However, there are at least two major obstacles that hinder such learning: (1) the training scene is complex with several units operating in parallel at multiple locations and under time [20] pressure, and (2) there is a lack of methods, tools, and resources to support the documentation and visualization of complex rescue operations for retrospective learning (Jenvald et al., 2000). The feedback is typically generated by letting human evaluators monitor the trainees and report what they find in debriefings, critiques, and written reports. R&E has been proposed, and tried at a Swedish emergency response training event in Alvesta, as an alternative solution to create a digital set of courseware that may help trainees retrospectively review and reflect upon their actions and roles within a larger context (ibid.). This courseware consists of a reconstructed mission history combined with textbook information and hypermedia information accessible through Internet, i.e., a combined set of externalized knowledge, with more contextual information than traditional procedure manuals. Results from a case study at Umeå University indicate that such courseware can be used to complement training even for trainees that did not participate in the reconstructed event (ibid.). 2.6.4 ALTERNATIVE METHODS FOR KNOWLEDGE TRANSFER Using multimedia tools as a means of retaining organizational memory was proposed already in 1995 with the release of the video-annotating tool Constellations (Rao & Goldman-Segall, 1995). Further, Abecker and Decker (1999) proposed an architecture for composing an organizational memory system (OMS) using stored multimedia stories, such as annotated videos, images, and text. They list several issues and challenges in creating such systems, most notably the importance of enabling connections in the system, i.e., connecting people to people, people to knowledge, knowledge to knowledge (ibid.). Many tools and frameworks have been developed since based on the assumption that multimedia stimulates learning, as well as cognitive theories to explain multimedia learning, and its unknowns (e.g., Mayer, 2005). Another theoretical framework for comprehension of physical systems through multimedia presented by Hegarty (2005) integrates encoded auditory and visual features into an internal representation closely resembling a reconstructed multimedia model such as the mission history. Her work informs that comprehension of multimedia models of physical phenomenon requires the student to have high domain knowledge, meaning that if these conclusions are applicable for knowledge transfer of mission experience it may bear impact on who the target group for such learning systems are (ibid.). 2.6.5 MEDIA RICHNESS AND KNOWLEDGE TRANSFER The media richness theory was proposed by Daft and Lengel (1986), to categorize communications mediums based on their capacity to process rich information, i.e., a phone call cannot reproduce visual cues and is therefore less rich than video conferencing, but more rich than textual e-mails and letters, which cannot represent anything but text. They argued that higher richness is more effective, a viewpoint that has since been debated and criticized since it does not seem compatible with how people choose media for their communication (e.g., Ngwenyama & Lee, 1997). Other studies have shown that lean media can be as effective or even better compared to rich media for presenting analyzable tasks, while high media richness mediums are beneficial primarily [21] for non-analyzable tasks such as subjective interpretations (e.g., Lim & Benbasat, 2000; Otondo, van Scotter, Allen & Palvia, 2008). Liu et al. (2009) showed that media richness theory is insufficient, but can be combined with flow theory and technology acceptance models to explain the role of multimedia in e-learning. The combination of all three of audio, video and text outperformed any single, or combination of the other two mediums for perceived usefulness and concentration among learners. This is contradictory to earlier research (e.g., Lee, Cheung & Chen, 2005), for which they offer the explanation of technology acceptance. Their findings thus imply that adapting learning material to the learner’s technology acceptance level or other preferences may increase the generated effect. 2.7 WORKPLACE ETHICS The view of war scenes and emergency operations as a workplace may be somewhat disturbing for the non-initiated. However, for those who operate in tactical organizations, the workplace can very well be a scene for combat or accidents. When the tactical units arrive at the scene, they are expected to operate in the best interest of the society (Miller, 2005). When R&E has been employed at these organizations, albeit at training events and not real operations, sometimes a certain resistance has been encountered from trainees who were feeling uneasy about the monitoring of themselves, even though the unit of evaluation in R&E typically is a tactical team as opposed to individuals (see Paper III). This subsection looks into three important topics for ethics research that may bear impact on analyzing R&E from an ethical point of view. 2.7.1 PRIVACY A powerful and distinct definition of privacy was given by Warren and Brandeis (1890): “the right to be let alone”. This definition is still in use but it does not come without complication if applied to workplace ethics since there is normally an agreement, a contract, between the employer and the employees. A prima facie conflict immediately arises when weighing privacy versus contract under these premises. A more applicable definition was given by Aiello and Klob (1996) as “the ability for an individual to control the use of their own personal data, wherever it might be recorded”. This definition is both practical and seems acceptable in work life. The immediate consequence of the definition is that all non-controlled monitoring of controllers in a tactical operation can be regarded as an infringement on privacy, and therefore problematic. Privacy infringements can have many negative consequences such as lack of trust, anxiety and alienation (Kizza & Ssanyu, 2005). Miller (2005) argued that privacy infringements can be accepted to a higher degree on public service employees such as the police force. This does not, however, imply that the negative consequences of privacy infringements do not apply to them. 2.7.2 R&E AND W ORKPLACE SURVEILLANCE It has already been established that performance of actions is a vital component of the controlling system in any tactical operation. Reconstructing a mission history thus requires capturing of these actions and the cues in the environment that lead to decisions [22] activating them. These interactions can be captured either by sensors, e.g., video cameras, or by human observers; however both methods can be paralleled with monitoring of the work scene. Thus, reconstructing a mission history may subject the employees to some level of monitoring, and therefore possibly privacy infringements. 2.7.3 INFORMED CONSENT One of the primary measures to reduce the negative effects of privacy infringements is consent. However, it is not an uncomplicated issue, as consent is hardly valuable unless it is truly informed (Malek, 2005). To qualify as informed consent, five important conditions have to be fulfilled (ibid.): the subject is given all information, the subject understands all information, the subject is able and allowed to make a choice, the subject makes the choice independently, and the subject gives consent. Obtaining such consent can be hard, as all information may be a fuzzy concept when not even the explorers know on beforehand what results the R&E will yield. The subjects can thus not be certain of what the explorers will see and what results the data can generate. It is therefore questionable whether the consent qualifies as informed or not. 2.8 RESEARCH GAP A plethora of research has been dedicated to the understanding of C2 from many different angles over the last decades (Alberts & Hayes, 2006; Brehmer, 1992; 2000; 2009; Kahneman & Tversky, 1979; Essens et al., 2005; Pigeau & McCann, 2002; Weick & Sutcliffe, 2001). The emergence of JCS and cognitive engineering highlights the same type of problems from a different viewpoint (Hollnagel, 1993; Hollnagel & Woods, 2005). The fields of knowledge creation and transfer have also received much attention (Hijikata, Takenaka, Kusumura & Nishida, 2007; Nonaka, 1991; Novak, 2007; Pettersson, 2009; Rice & Rice, 2000). The relationship between C2 and experience has been established by Klein’s (1993) widely recognized model of RPDM as well as Brehmer’s (1992; 2000) model of dynamic decision making in C2. Yet, little attention has been paid to the complexity of acquiring and sharing experience targeted at resolving tactical events that have not been encountered before by the performing organizations and adhocracies. A rare exception is the limited study on R&E to create digital courseware from the Alvesta field trial (Jenvald et al., 2000). This thesis fills the gap by investigating a new methodology for externalizing and sharing mission experience, through the lens of decision making in dynamic C2 situations. [23] [This page is intentionally left blank] [24] Chapter 3 METHODS The research questions in this thesis are addressed through four separate studies on different aspects of applying R&E in mission training. The studies, Papers I-IV, employ various qualitative methods ranging from case studies with participatory observation to literature reviews. All four studies are summarized in Chapter 4, whereas the results are detailed in light of the provided background theory to answer RQ1-3 in Chapter 5, with follow-up discussions in Chapter 6. 3.1 STUDY DESIGN This thesis presents four studies of using R&E for transferal and dissemination of mission experience. The research questions are approached using theoretical propositions grounded in the evidence collected from the four attached studies, i.e., Papers I-IV. The study design is thus comprised of four parts, as listed in Table 3.1. Each method is explained briefly in the following sections of this chapter. Table 3.1. Study design Study Main research approach Data collection methods I Literature review (narrative) Literature search II Case study (exploratory) Participatory observation, after-action review, group interview III Case study (exploratory) Direct observation, participatory observation, after-action reviews IV Literature review (narrative) Literature search 3.2 CASE STUDIES A case study is typically applied to study a real phenomenon, e.g. individuals, configurations, events, missions, and organizations. They are suitable when the researchers cannot control the unfolding events, but want to answer questions such as when and how (Yin, 1994). There are mainly three types of case studies: (1) exploratory where the researcher investigates a phenomenon to identify patterns and characteristics, often without a priori knowledge, (2) descriptive where the researcher investigates certain features based on a founding theory, and (3) explanatory where the researcher tries to explain a phenomenon through detailed analysis, requiring a baseline and guided data collection (ibid.). [25] The studies presented in Paper II and Paper III were conducted as exploratory case studies, with evidence collected through direct observation, participatory observation, after-action reviews, and group interviews. 3.2.1 DIRECT OBSERVATION Direct observation is a data collection method that aims to be unobtrusive, conducted by outside observers in an attempt to not affect the studied phenomenon, i.e., to avoid bias. The researchers thus want to observe sampled situations from a spectator perspective, i.e., nonreactive (Bernard, 2011). However, direct observation often becomes blatant and reactive if the studied persons are aware of the spectators and put on a show for the audience (ibid.). This raises an ethical issue of whether to use reactive observation, by allowing the subjects to be aware of the monitoring, or to use nonreactive observation, i.e., hiding the observation from the subjects (ibid.). The case study in Paper III employed reactive observations to study other research groups in their exercise analysis, in an attempt to get evidence for comparison against the self-experienced R&E method. Additionally, in both Paper II and Paper III, reactive observation was employed in the reconstruction itself to collect information on the exercise events as they unfolded. 3.2.2 PARTICIPATORY OBSERVATION Participatory observation is a common, but demanding, method for qualitative data collection. The researchers assume a role in the studied phenomenon in real-life settings by participating in the normal activities. Participant observation allows understanding of the phenomenon from the inside as opposed to the spectator perspective given by direct observations (ibid.). Participatory observation was used as a means of data collection in the case studies presented in Paper II and Paper III, where the researchers themselves utilized R&E, in conjunction with the exercise management staff, to analyze and disseminate the course of events from the exercise. In the Swedish Armed Forces case study (Paper III), the difference between direct and participatory observation is exemplified by the researcher conducting the analysis (participatory observation) versus observing other analysts conducting it (direct observation). 3.2.3 AFTER-ACTION REVIEWS After-action review (AAR) is a process designed to conduct structured debriefings after military training sessions (U.S. Army Combined Arms Center, 2011). The process has been adopted worldwide, both by military and civilian organizations, e.g., the HA/DR domain (Sexton & McConnan, 2003). A well-executed AAR reviews what was supposed to occur, what actually happened, what was right and wrong with what happened, and how the task should be done differently the next time (U.S. Army Combined Arms Center, 2011). An AAR can be formal, with external observers and controllers, complex training aids and scheduled, or informal with an internal chain of command and simple training aids and conducted when a need has been identified (ibid.). For maximum effect, [26] an AAR should be scheduled during, or shortly after, the completion of an activity and based on objective data (ibid.). The study in Paper II used a formal AAR in emergency response training to increase learning of the training session. The study in Paper III used daily formal AARs in a military setting, to conclude the activities after each exercise session. R&E was used in both studies to provide objective data for the AAR sessions. 3.2.4 GROUP INTERVIEW Interviews are often used to gain understanding of peoples’ experiences and can be conducted in many forms, from unstructured and ad-hoc, to formal and controlled, interactions with the respondent (Seidman, 1998; Bernard, 2011). Interviews can target one respondent, or a group of respondents. The latter is often called group interview, or focus group, and serves the purpose of extracting a collective understanding, including the group dynamics and individual perspectives that may differ among the group (Stewart & Shamdasani, 1990). A group interview can have several purposes, including (1) exploratory, which are often used to satisfy the researchers’ curiosity, e.g., to test the feasibility of a more complex study, (2) pretest, which serve as a pilot to a larger study, (3) triangulation, in which the focus groups are used to add methodological rigor, and (4) phenomenological, when the interview is a regular data source to be used for exploratory analysis (Frey & Fontana, 1991). While subjectivity is not eliminated by moving from one to many respondents, it can be reduced – leading to higher validity of the collected data, however it comes with the risk of suppressing individual thoughts and opinions due to group dynamics mechanisms (ibid.). The study in Paper II includes a group interview session from August 2007, eight months after the AAR. The seminar included some 30 respondents divided into groups of 3-4 who discussed nine topics broadly among themselves before sharing their thoughts to the researchers in an open floor format. The seminar had an exploratory purpose, trying to identify how R&E can be used, and what potential issues there may be when using it for exercise assessment. The discussion topics, or questions, are listed in Paper II. 3.2.5 CASE STUDY EVIDENCE ANALYSIS In case study research, the analysts often needs to resort to experience and literature to interpret and present evidence, since statistical analysis cannot always be used (Tellis, 1997). As the study presented in this thesis is theory-building and exploratory, with exclusively qualitative evidence, data interpretation is a matter of “examining, categorizing, tabulating or otherwise recombining the evidence” (Yin, 1994). The evidence used for this analysis consists primarily of findings from Papers I-IV. 3.3 LITERATURE REVIEW A literature review is a “systematic search of published work to find out what is already known about the intended research topic” (Robinson & Reed, 1998, p.58). There can be [27] many reasons for conducting literature reviews, including avoidance of duplication, sharpening and deepening a theoretical framework, identify knowledge gaps and to study definitions and characteristics of other previous work with the aid of adopting them in a new research project (ibid.). A literature review can take on various forms, such as (1) narrative, where the review critiques and summarizes a body of literature to draw conclusions on a specific topic, often without obvious selection criteria, (2) systematic, where a rigorous and well-defined selection criteria is used to define a comprehensive set of literature for the purpose of answering well-focused questions, (3) meta-analysis, where statistical analysis is employed on a large body of quantitative findings to integrate them and generate increased understanding, and (4) meta-synthesis, where findings from a body of qualitative literature are assessed and integrated to find common core elements and themes (Cronin, Ryan & Coughlan, 2008). The literature reviews in Papers I and IV are both narrative in character, and aim to build founding theories for the reasoning presented in the respective study. In the study presented in Paper I, the synthesis led to design of an explanatory model which was discussed and refined through ad-hoc interviews before being finalized. The results from case study in Paper IV were refined after open floor discussions following a presentation of the initial findings. 3.3.1 LITERATURE REVIEW DATA COLLECTION INSTRUMENTS The literature reviews in Paper I and Paper IV both used a combination of Internet-based instruments to collect data for the review: EBSCOhost6 is a meta-search engine by EBSCO Industries Inc., cross-searching five indexed databases of periodicals, academic journals and more. The included databases are PsycARTICLES, PsycINFO, Military & Government Collection, Academic Search Premier, and International Security & Counter Terrorism Reference Center. ScienceDirect7 is a scientific database, with journal articles and book chapters published by Elsevier B.V. IEEE Xplore8 is a digital library holding scientific and technical content published by the Institute of Electrical and Electronic Engineers (IEEE). The main contents are journal articles, conference proceedings, technical standards and eBooks. Web of Science9 is a multidisciplinary data citation database by Thomson Reuters, indexing a comprehensive set of journal articles and conference proceedings, making them searchable through their own search engine, and enabling selection based on citation count. 6 http://search.epnet.com 7 http://www.sciencedirect.com 8 http://ieeexplore.ieee.org 9 http://thomsonreuters.com/products_services/science/science_products/a-z/web_of_science/ [28] Google Scholar10 is a search service from Google Inc. covering a broad set of scholarly literature, including articles, theses, books, abstracts, court opinions, etc. In addition to the above, the Swedish Defense Research Agency (FOI) library hosts a service for its employees through which they can submit keywords to the librarians, who then return a set of relevant academic publications and technical reports matching the keywords. This service was used extensively in both literature reviews to complement searches using the five Internet based engines. 10 http://scholar.google.com [29] [This page is intentionally left blank] [30] Chapter 4 SUMMARY OF APPENDED PAPERS This thesis is founded on four research articles: (1) Sharing mission experience in tactical organisations, (2) Evaluation of crisis management operations using reconstruction and exploration, (3) Reconstruction & exploration of large-scale distributed tactical operations – multimedia tools for evaluation of emergency management response, and (4) Privacy and distributed tactical operations evaluation. Paper I: The study Sharing mission experience in tactical organisations explores the relationship between flexibility, adaptability, improvisation, resilience, innovation and creativity with the objectives of increasing understanding of what mission experience is, why it is important, and how it can be shared. Paper II: The study Evaluation of crisis management operations using reconstruction and exploration presents a case study on R&E conducted at the Swedish Rescue Services training facility in Sandö 2006 with the objective of finding out how and if R&E and can used to support emergency response training. Paper III: The study Reconstruction & exploration of large-scale distributed tactical operations – multimedia tools for evaluation of emergency management response presents a case study from the first application of R&E in a large-scale environment with the Swedish Armed Forces at their main C2 training facility in 2005. The objective of this study is to explore how R&E compares to other research methods for evaluation of largescale military tactical exercises. Paper IV: The study Privacy and distributed tactical operations evaluation takes a somewhat different approach compared to the above three with the objective to explore the highly interesting topic of ethical effects that R&E may have on personnel exposed to recording and evaluation, and how it can be morally justified. This chapter presents major findings from the four appended articles, organized per article, with the findings followed by a short summary of theory, reasoning and underlying data motivating the results that were originally presented in the corresponding appended paper. 4.1 SHARING MISSION EXPERIENCE IN TACTICAL ORGANISATIONS The main outcome of the study is the FAIRIC model, see Figure 4.1. The model outlines a framework to distinguish between six commonly used, but seldom explained, concepts in emergency management, and tactical mission control. The model provides a common vocabulary to discuss knowledge gained from mission experience; and thereby understand related literature. In the context of modeling tactical operations for knowledge transfer, the model provides a mechanism to start analyzing what mission experience consists of and which integral components thereof that need to be externalized in order to enable sharing and transferring of mission experience over boundaries of time and space. [31] Unfamiliarity of new situation Improvisation Innovation Resilience Creativity Adaptability Flexibility Proactiveness Figure 4.1. The FAIRIC model (from Paper I). 4.1.1 OBJECTIVES The objective of this study is to identify important outcomes of mission experience, define how they relate to each other, and lay out a theory of how mission experience can be shared within and between organizations through other methods than hands-on practice. 4.1.2 BACKGROUND It stands clear that sharing mission experience is a complex task, requiring first and foremost a definition of what mission experience really is; which by itself is non-trivial. Cillier (2005) stated that “knowledge cannot be fixed in a representational way but is always contingent and contextual”. Weick (1969, p.206) similarly concluded that “if an organization is to learn anything, then the distribution of its memory, the accuracy of that memory, and the conditions under which the memory is treated become crucial characteristics.” It seems thus, that mission experience is highly dependent on context and circumstantial events and conditions. As such, it becomes clear that it is not primarily the particular solution of a problem that constitutes mission experience, but rather the underlying decision-making process. According to the well-known RPDM model (Klein, 1993) some of the most valuable knowledge gained from a mission is an increased dictionary of goals, cues, expectancies and plausible actions, i.e., an improved bank of situations to match and recognize future events against. Nonaka’s (1991) model of knowledge creation, i.e., the SECI model, outlines socialization, externalization, combination, and internalization, as the four main components of knowledge generation. By regarding these four steps as a spiral as in Figure 4.2, Nonaka’s model describes a method for knowledge transfer that explains how [32] mission experience can be transferred. Socialization, in this context, can for example refer to observation of, and participation in, in real missions. Externalization on the other hand, is often done through debriefings, and reporting. Commonly, mission experience is externalized as written reports or entered into a lessons learned database. In the best of worlds, the products of the externalization process can easily be distributed within and between organizations. Relating findings and results from these externalization products to an existing body of knowledge can yield increased insight. Nonaka refers to this phenomenon as combination. Finally, the internalization process occurs when individuals, or organizations, operationalize their newly added knowledge by applying it in the field. The spiral analogy implies that new experience from using the acquired knowledge in the field can be socialized or externalization again, followed by a combination or internalization process, etc. Socialization Externalization Internalization Combination Explicit knowledge Tacit knowledge Tacit knowledge Explicit knowledge Figure 4.2. The SECI model (after Nonaka & Takeuchi, 1995). The SECI model thus gives four integral components of sharing mission experience, externalization, combination, internalization, and socialization. Turning back to King’s (2005) statements that humanitarian aid (HA) organizations rely heavily on their personnel to conduct or observe missions and exercises to acquire necessary knowledge, it becomes clear that socialization is the most important knowledge sharing component used by many HA organizations. King’s further findings indicate that externalization processes are not effectively used since knowledge is shared mainly through apprenticeships and occasional seminars. His critique indicates that there is room for improvement in this area. While there are many actively used systems in the world today, e.g., lessons learned systems; few solutions seem to address the need for representing contextual knowledge as identified above. Further, many of these systems seem to be designed for externalizing and collecting knowledge, not for disseminating or allowing internalization thereof. 4.1.3 RESULTS To fully understand what features of a dynamic control situation to model and capture, one must comprehend what experience the operators possess and what cues they recognize against when they make recognition-primed decisions. This work reviews literature on dynamic decision-making, emergency management, and organizational [33] memory to find six abilities that can be identified as central for tactical command in dynamic environments: flexibility, adaptability, improvisation, resilience, innovation, and creativity (FAIRIC). The review reveals that while these abilities are commonly described as important, they are rarely distinguished between, or even explicitly defined. As an example, improvisation alone is found to have at least 42 co-existing definitions in one referenced study (Moorman & Miner, 1998) – some of which refer to one or several of the other FAIRIC concepts. In an attempt to improve understanding of how to model and capture dynamic decision making situations, the FAIRIC model is proposed, highlighting nuances and differences between these commonly referred abilities. The proposed FAIRIC model is derived from existing literature and outlines how they relate to each other from two aspects: unfamiliarity of the situation and proactiveness as shown in Figure 4.1. A flexible organization, according to this model, can be expected to have taken proactive measures to cope with situations that they can foresee; whereas an adaptive organization is expected to react on changes in the environment as they occur, and cope with the new situation from these premises. An improvising organization is one that has the ability to react with timeliness to respond to situations which could not be foreseen, while the innovative organization is more proactive, e.g., by inventing new procedures to respond to unforeseen events before they occur. 4.2 EVALUATION OF CRISIS MANAGEMENT OPERATIONS USING RECONSTRUCTION AND EXPLORATION This case study shows that multimedia supported mixed method research can indeed improve the effectiveness of emergency response training, by allowing the trainees to get more out of the training events and to some extent even learn from scenarios where they were not able to participate, such as when there are too many trainees to allow everyone to train in their operational role. 4.2.1 OBJECTIVES The objective of this study is to explore how R&E can be used to support emergency management C2 training, and to determine what kinds of objective data best support AAR in this context. 4.2.2 BACKGROUND Disaster relief is a growth-market where money and resources are continuously being pushed into the system in an effort to save human lives. Meanwhile, in the field, the relief agencies struggle to manage excess inventory and a logistical nightmare. The real need often lies in logistics and effectively managing supply chains (Thomas & Kopczak, 2005). This gap in need versus effort is one example of where efficiency in disaster relief operations can be improved. Complex inter-agency operations such as disaster relief, terrorism response and emergency management require commanders from several independent organizations to control dynamic situations that are often impossible to fully understand and predict. Continuous assessment and team performance evaluation, from both live operations and training, can help the organizations improve and be better prepared for the disasters of tomorrow. [34] As operations grow bigger and more complex, the effort to collect and analyze data increases rapidly. Without careful planning and management the assessment task thus become overwhelming (Albinsson et al., 2005). While a common approach to evaluate large and complex operations (exercises in particular) is to rely on quantitative measure and self-reported questionnaires, such assessment tend to focus on the respondents’ perception of the operations, and correlate strongly with the level of accomplishment of known pre-defined objectives. These methods, however, are less fit to explore breakdowns and determine the need for improvement in organizations that do not match the predefined objectives. Cognitive analysis approaches and many other qualitative analysis methods are comparatively demanding in terms of time and resource (Blandford & Wong, 2004). Such methods have the ability to post-hoc capture insights on particular events such as unforeseen breakdowns, and may enable understanding of causality relationships. However, because of the workload and effort needed with these methods, the analysis tend to focus on a selected few incidents and based on the perception of a handful individuals rather than taking the whole operation into account. Therefore, traditional qualitative research methods may also be insufficient when an overall picture based all actors, activities and collaborations involved is desired (Morin & Albinsson, 2005). This study introduces R&E as a mixed methods approach to assess distributed tactical operations, overcoming some of the short-comings of traditional qualitative and quantitative methods. Further, the F-REX framework is introduced as a toolset to conduct R&E of crisis management operations. A case study is presented where R&E and F-REX were used to conduct multimedia supported after-action reviews at the Swedish Rescue Services Agency’s (SRSA) training facility in Sandö, 2006. The R&E approach enables analysts to objectively post-hoc explore a course of events and determine causality relationships through careful visualization and analysis of collected data (Morin, 2001). The approach can be separated into two parts, the Reconstruction process and the Exploration process as outlined in Figure 4.3. The end state of the reconstruction process can be thought of as a dataset complete enough to answer predetermined questions, hypotheses and problems. One of the shortcomings of quantitative methods as described above was the non-ability to cope with unforeseen events such as breakdowns. Using reconstruction to capture these requires foresight and thorough planning, and may generate lots of data that might never be useful. In many cases the ability to do so is restricted by resource availability, thus there is a tradeoff between completeness of the dataset and cost/complexity to capture it. The exploration process is conducted post-hoc by analysts using proper tools, such as FREX, to combine the collected data into a time-synchronized model for presentation, or visual analysis. The process is often iterative as the analysts have the ability to feed data back into the model during the analysis process thus creating higher levels of data abstraction until a level is reached where the requested causal relationships can be determined, or questions answered. [35] Reconstruction Exploration Metadata Analysis products Domain analysis Topics Problems Priorities Hypotheses Modeling Conceptual model Instrumental plan Analysis Presentation Data collection Instrumentation Data integration Procedures Equipment Software Feedback Log files Data repository (Revised) Mission history Figure 4.3. Reconstruction and Exploration (after Andersson, 2009). Mission histories, the underlying data model used in F-REX, are split into two parts: one organizational model which represents participating units, objects and contextual a priori information in the operations and one data collection model which consists of a chronological list of captured data samples. The mission history then maintains links between elements in the two models to allow tracking for instance data that relates to particular entities or stages in the operation. 4.2.3 CASE STUDY SCENARIO The event in focus for this case study was a training session of evacuating a burning house with several injuries and casualties. This scenario is fairly standard to the organization, yet it is complex enough to highlight the centrality of C2. The primary data collection nodes were concentrated around the commander trainees, their communication devices and notable events in the context that affect the ongoing situation. A dataset was compiled and presented using F-REX to the participating trainees to discuss the training event in an AAR session. The session was hosted by F-REX operators together with SRSA instructors. The presentation was used to provide a baseline around which the trainees could reflect and view the events from a different perspective. The trainees were encouraged to openly share their reflections on the training event as well as this debriefing method. 4.2.4 RESULTS The principal findings of the AAR were (1) the F-REX system is helping trainees recall details of events in a chaotic environment, (2) R&E is useful for post-hoc clarification of what orders were actually given during the event, (3) multimedia recordings helped trainees that were NOT in the role of commander to understand the dilemmas that the trainee in the role of commander was challenged with, and (4) recorded radio communication was perceived as the most useful data source to understand and analyze the course of events. The participants were generally overwhelmed by the capabilities of this framework and found it hard to decide what could or should be included or excluded in a debriefing [36] method like the one they were subjected to. However, a common perception was that it was often hard to grasp the massive amount of data that was presented to them. Another much missed feature was the face-to-face communication between commanders and subordinates which was captured, but not included in the presentation. 4.3 RECONSTRUCTION AND EXPLORATION OF LARGE-SCALE DISTRIBUTED TACTICAL OPERATIONS This case study showed that R&E can indeed be used for large-scale operations, perhaps even further emphasizing the strengths of the approach as the operation scales up in size and complexity. However, quite a few notes to consider regarding both technology and methodology were detected, such as the importance of key indicators or initiated subject matter experts that can initiate the exploration process. On the technology side, a lesson to stress is the importance of a priori verification of the data collection infrastructure and data processing capability to support the magnitude of the dataset being generated. As size and complexity grows beyond current data mangling capabilities, a trade-off emerges between data set completeness and time to process data. Another major lesson to report is the difficulty of getting participants to cooperate in data collection. Recording the interactions of an individual over a long period of time clearly imposes a privacy violation, which some participants find so intrusive that they disobey direct instructions. A priori knowledge of the exercise may guide the researchers to limit the data collection, possibly increasing acceptance while at the same time making the datasets more manageable. However, the tradeoff here lies in that creating a data collection boundary increases the risk of not collecting data of the unforeseen breakdowns which was one of the main benefits of using the approach in the first place. Careful consideration is thus needed before reducing the data collection. 4.3.1 OBJECTIVES The objective of this case study is to determine the applicability of R&E in large-scale military contexts. In addition, identification of methodological and technical issues of the application thereof is a high priority. 4.3.2 BACKGROUND This study introduces R&E to the context of large-scale military C2 exercises. The study explores benefits, drawbacks, limitations and new requirements for the R&E approach and supporting tools in the context of large-scale distributed military C2 exercises. R&E in this context is then compared to some traditional qualitative and quantitative research methods. Quantitative research is often based on the belief that is possible to identify general truths based on rigorous scientific research and controlled experiments. They rely heavily on objective measurements, statistical procedures, hypothesis testing and validity (Byrne, 2002). Commonly, these methods are implemented as questionnaires or interviews using pre-defined alternatives or post-hoc grouping of related answers (classification or coding) to enable statistical analysis. A common approach for the analyst is to look for causality [37] by searching for correlation or tendencies between quantified variables to reject or confirm hypotheses. In the context of military operations, quantitative research is often less resource demanding than qualitative or mixed methods research. Attitude surveys can be used to assess respondents’ situational awareness (Endsley, 1995) of the operation or to identify general tendencies and attitudes. Some researchers have also tried to use quantitative methods to measure human performance (e.g., Baron, Kruser & Huey, 1990), albeit not without controversy. In large-scale contexts attitude questionnaires may indeed be useful to provide the respondents’ view of the course of events, but they can rarely be used to explain system breakdowns. In other words, they cannot be used to tell what happened during an exercise or why, rendering any result de-contextualized and difficult to validate. Thus quantitative methods are insufficient unless the whats and the whys are already known. Qualitative research methods on the other hand provide the ability to explore individual, social, organizational, and cultural phenomena. The emphasis is put on subjectivity, sensemaking and interpretation as opposed to a strict set of rules as for their quantitative counter-parts (Denzin & Lincoln, 2000). The qualitative group of research methods includes e.g., interview techniques, free-form questionnaires, participatory observation, and contextual inquiry. When humans and artifacts are the subject of research, cognitive analysis methods are often used, such as Critical Incident Technique (CIT), Critical Decision Method (CDM), and Contextual Inquiry (CI). CIT relies on self-report of incidents relating to satisfaction and efficiency or dissatisfaction and inefficiency (Flanagan, 1954; Dean, 1998). The technique is used, e.g., to improve work routines. CDM is a retrospective interview technique with the aim of getting deeper understanding of incidents than through CIT (Hoffman, Crandall & Shadbolt, 1998). Finally, CI was developed specifically with the aim of recognizing the importance of context when interpreting data (Beyer & Holtzblatt, 1999). It is often implemented as participatory observation of standard work flows and processes followed by process tracing, e.g., through interviews to make the flows and processes explicit and comprehensible. Consequently, quantitative and qualitative methods have their own strengths and shortcomings. A flexible mixed method approach such as R&E allows the researcher to overcome some of the shortcomings and tailor the method and make it fit with the research question at hand. During large-scale military C2 exercises, the workload of a single operator may shift tremendously over the course of time. As the system is large and complex with many participants, some at remote and inaccessible locations, pausing the exercise for probing interviews may not be an option. Similarly, questionnaire research can be hard to conduct because of the multitude of events that may occur in a short period of time, reducing the probability that the operators are able to remember and separate details of each event until the time when they are available to complete a questionnaire. Hence, although questionnaires can be distributed and answered, the contextual circumstances may render validity of the collected data questionable. Participatory observation and contextual inquiry are two methods that do seem to come close in applicability for large-scale military exercises, with observers being able to trace [38] multiple locations and processes over time, to find causality. The main drawback of such methods is the need for many observers performing in-depth investigations which is both expensive and hard to coordinate. Even so, the methods still do not solve the problem of integrating multiple perspectives, i.e., to connect the traces and find inter-trace causality. The R&E approach takes a wide system perspective by gathering factual, simultaneous real-time data on multiple organizational levels at multiple geographical locations. Multimedia tools such as F-REX enable the analyst to follow a timeline over multiple processes and switch focus post-hoc. Similarly to participatory observation, this method allows the researchers to trace processes, with the main difference that the researchers can do this retrospectively and shift focus between processes to determine inter-trace causality that limits the aforementioned methods. A major critique against R&E compared to qualitative methods, is that it does not outline an analysis process to guide the analyst through the process. In many respects, reconstruction often leaves the analyst with a huge dataset, but no guidance in where to start exploring it. Retrospection and the ability to quickly get an overview of causality in a complex C2 situation are found to be two of the most important features of R&E which separate the approach from traditional methods. Together, these abilities enable the analyst to spend less time figuring out what happened, instead focusing on the more important why. 4.3.3 CASE STUDY SCENARIO R&E and F-REX were used in case study a large-scale military command exercise with the Swedish Armed Forces in 2005 to determine its applicability in a real setting. The exercise involved approximately 300 participants in a networked environment running a tabletop exercise including a multitude of technical artifacts such as battle simulators and common operational picture providers. The setting provided an opportunity to test R&E and F-REX in an environment much larger than anything they had ever been tested in before. The exercise consisted of four scenarios that were being played over a period of two weeks. R&E was used during the entire exercise to capture and reconstruct the series of events, allowing researchers and exercise managers to review the mission history from different perspectives using the F-REX tool. Concurrently, the exercise was continuously being evaluated by parallel research groups using traditional qualitative and quantitative methods. With these parallel groups assessing the exercise, the F-REX/R&E work could focus on perceived benefits and experienced problems of this approach in the large-scale setting compared to other methods. 4.3.4 RESULTS Early on analysis induced a great deal of technical issues that highlighted the technical difficulty of recording, managing and compiling large datasets. The issues ranged from data mangling performance to audio transmission breakdowns. To continue the case study data collection was scaled down after the first day, to include only a selected range of workstations and operators, as opposed to the previous ambition of capturing all. Following that, the technical part of the study went smoothly. Using the R&E approach in such a large-scale operation also introduced methodological issues, including the fact that many participants disobeyed their instructions to wear [39] recording equipment. Over time, the number of participants who blatantly ignored the instructions kept growing leading to a noticeable amount of data being lost. Arguably a more severe methodological issue was the perceived difficulty in navigating and exploring the massive mission histories. Analysts without proper domain knowledge or exercise insight had a hard time trying to explore the mission history and produce any valuable findings. This was partly solved later on by using observer notes from 20 experienced subject matter experts who reported on team performance at chosen locations in the exercise. R&E was successfully able to provide the root cause of several mistakes and confusing situations. As these findings were reported during the AARs, military personnel and exercise management started to pay more and more attention to the R&E approach. At the last couple of days, exercise managers where directly using the mission history to search for root causes shortly after they experienced a problem. With the assistance of experienced F-REX operators, they were able to clarify obscurities and find explanations to exercise-related problems that they encountered. 4.4 PRIVACY AND DISTRIBUTED TACTICAL OPERATIONS EVALUATION Evaluation of distributed tactical operation using after-action reviews, or mixed methods such as R&E, may undoubtedly cause privacy infringements for the evaluated. The magnitude of these infringements cannot always be determined a priori, which makes them difficult to defend against. Evaluations, however, are necessary for organization to evolve and improve, which in the case of emergency management may imply that our society becomes better protected. In an ideal world the evaluators always strive to get continuous informed consent from participants in any operation that is being evaluated in an attempt to minimize the negative effects. In reality, however, this is difficult to achieve because the evaluators may not always be able to predict what information will be collected and how it can and will be used. Accepting minor privacy infringements may thus be required for the greater good, and should be expected by employees working in these organizations. Care must be taken though so that the negative effects do not become overwhelming and render the evaluation counter-productive, e.g., by reducing the teams’ level of creativity and autonomy. Finding methods to minimize the magnitude of privacy infringements and their effects is subject to future work. 4.4.1 OBJECTIVE This study was performed as a literature review with the objective of identifying why some C2 trainees feel uneasy about data collection on their actions, and what ethical issues to consider before employing too comprehensive information gathering in the context of R&E. 4.4.2 BACKGROUND Aiello and Klob (1996) proposed a definition of privacy as “the ability for an individual to control the use of their own personal data, wherever it might be recorded”. From a strict interpretation of that definition, a privacy infringement occurs whenever data is [40] collected about persons without their knowledge, as well as when the persons are unable to stop collection or erase earlier collected data. Evaluation of distributed tactical operations through methods such as R&E, involves a large amount of data collection on individuals and their actions. Although the intent is seldom or never to collect any personal data or assess individual performance, recording a teams’ actions without recording its’ members, and thereby collecting personal data, is virtually impossible. 4.4.3 RESULTS The most important result of this study is the awareness that R&E of distributed tactical operations may cause privacy infringements. In many respects the data collection for reconstruction is similar to workplace surveillance, which has been shown to impose negative effects on the workers (Kizza & Ssanyu, 2005), such as: lack of trust between workers, supervisors, and management, stress and anxiety, repetitive strain injuries because of refraining from taking breaks, lack of individual creativity, reduced or no peer social support, lack of self-esteem, worker alienation, lack of communication, and psychological effects. It can certainly be argued that the purpose of evaluating tactical operations is different from that of workplace monitoring, and that operators agreeing with the motives may invalidate some of the above effects, i.e., the lack of trust issue might not be as imminent. However, it is reasonable to suspect that stress and anxiety may be an outcome from knowing that every action and decision being made is recorded and evaluated. The same holds true for at least lack of creativity, lack of self-esteem and lack of communication. The latter of which has been clearly shown by operators removing or turning off recording equipment to enable themselves to speak freely with co-workers (see Paper III). Palm (2007) noticed that technology enables employers to shift performance monitoring towards individuals as opposed to teams which used to be the case. The same hold true for evaluation of tactical operations. Gadgets, such as voice recorders, GPS trackers, and video cameras, become smaller, cheaper, and more accessible – allowing evaluators to collect more data so easily that they may not always reflect upon whether the data is actually needed or not. This can be problematic since excessive data collection may negatively affect the unit of observation, and doing so without reason should thus be strongly discouraged. The main risks of continuous and systematic collection of personal data are unavoidability, continuity, dependency, and identifiability (ibid.). The first of these four points can be remedied by giving all the participants in the operation subject to evaluation the right to decline monitoring. Such a remedy, however, does not come [41] without complication; a consequence may for instance be that the declining person is temporarily relieved from the team. Will opting out increase the risk of system breakdowns at this, or a future, operation? In the end, is there a risk that the consequence of tactical operations’ evaluation is decreased performance? From a contractarian point of view, the dependency relationship between employers and employees might be seen as a contract where the employees give up some fundamental rights to privacy by accepting the job offer, especially so in the public sector organizations where tax payers have a reasonable right to demand that their tax money is used for greater good. One may argue then that monitoring is motivated for the greater good and that the employee has consented. Such reasoning however, leads to the immediate question, was the consent informed? Informed consent can be defined as “voluntary consent based on adequate understanding of relevant facts” (Malek, 2005). For consent to classify as informed the subject must be able and allowed to make a choice while understanding all information there is to know about the consent. In the above scenario, consent would not be qualified as informed if, for instance, the subject is economically dependent on the job. Another complication is the fact that the subjects may not be able to comprehend what information can be retrieved from the collected data when combined. It may not even be apparent to the evaluators on beforehand, making it difficult to assure that the consent is informed. Mechanisms are therefore needed to enable revocation of said consent. As often is the case in ethical questions, this is a case of Ross’ pluralism with a trade-off between the duty of beneficence and duty of non-maleficence (Ross, 1930). As an example, how can the society demand that a human being enters a nuclear reactor to stop a meltdown? The risks for that person are obviously tremendously high, but if nothing is done a calamity is bound to happen. What is the moral thing to do? To some extent it may be reasonable to put forward a utilitarian argument that the society has the right to demand privacy infringements from public-service employees and continuous assessment of their readiness level, but care must be taken so that systematic evaluation does not become counterproductive and decrease their performance. It is a disturbing and contradictive thought that the price the society pays for a higher perceived level of safety may in fact be a decrease of the same. [42] Chapter 5 RESULTS Chapter 5 presents the results of this thesis related to research questions RQ1-3: RQ 1. What abilities are important for commanders, to cope with dynamic decision-making situations that arise in tactical military and emergency management operations? RQ 2. How should commanders’ mission experience be externalized in order to enable vicarious learning from tactical military and emergency management operations? RQ 3. What technical, methodological, and ethical issues need to be considered when externalizing mission experience from tactical military and emergency management commanders? The first research question, RQ1, is answered in section 5.1 by summarizing a literature review (Paper I) of tactical missions, explicitly focusing on abilities needed to cope with dynamic situations. RQ2 is addressed in section 5.2, with results building extracted from the two case studies found in Paper II and III, to illustrate how mission experience should be externalized. Finally, a set of issues have been identified in the case studies (Paper II and III) and the privacy study (Paper IV) to address RQ3 based on self-experienced problems from R&E of C2 as reported in section 5.3. 5.1 WHAT ABILITIES ARE IMPORTANT FOR COMMANDERS, TO COPE WITH DYNAMIC DECISION-MAKING SITUATIONS THAT ARISE IN TACTICAL MILITARY AND EMERGENCY MANAGEMENT OPERATIONS? Investigating what mission experience is requires understanding of how the subject perceives a mission, and what knowledge is acquired from it, both tacit and explicit. The study in Paper I approaches this task through a narrative literature review on tactical C2, investigating literature on C2 in dynamic situations. Studying this topic is complex, partly because it is multi-disciplinary, and different disciplines employ different vocabularies and frames of reference. To get a broad overview of the topic, the literature review includes samples from the diverse fields such as C2, cognitive engineering, resilience engineering, knowledge management, and decision making. The missions that emergency management and military teams conduct, span over a broad spectrum of operation types. Normally, organizations have routines to practice standard and familiar scenarios. Often, however, these organizations operate in environments that are dynamic and complex, where infrequent and unforeseen situations occur (Mendonça & Hu, 2009). [43] Teams that operate under the above conditions can be classified as adhocracies (Mintzberg, 1979). The infrequent and unforeseen events are critical to organizations, and society may depend on them being resolved. The nature of such events makes them difficult to exercise and manage with standard operating procedures. Under these circumstances, time-pressure is often a factor, ruling out slow analytical decision-making processes. The RPDM model shows a thought process where decision makers rely on prior knowledge, or experience, to make decisions (Klein, 1993). The experience, according to this model, guides the decision maker by allowing recognition of familiar situations. From a commander perspective, the recognition process thus motivates the need for experience, as it improves their ability of making rapid decisions in dynamic time-pressured, and potentially life-threatening, situations. The literature review in Paper I takes off from the claim that one of the most important abilities to successfully control an unforeseen event is innovation (Mintzberg & McHugh, 1985). An investigation into referenced literature and dictionaries shows that innovation is closely linked to creativity, but also that other researchers suggest instead adaptability and flexibility as key enablers for successful performance in dynamic environments. Further reading introduces two more concepts that seem to dominate literature on control of dynamic situations: improvisation and resilience. The review has thus revealed six abilities that organizations employ to cope with dynamic situations, although a dictionary lookup show that they are not easy to separate, see Table 5.1. While there are nuances between these concepts, the reviewed articles do not use these nuances to separate between them. Some authors even use terms such as flexibility and adaptability interchangeably whereas some differentiate between them, making it hard to relate two separate pieces of work to each other. Table 5.1. Selected terms related to managing changing environments and conditions, from (a) The American Heritage Dictionary of the English Language, (b) Collins English Dictionary – Complete and Unabridged, or (c) WordNet 3.0 (from Paper I). Concept Explanation Synonyms Adaptability ability of becoming suitable to a particular situationc variableb, versatileb, changeableb Flexibility responsiveness to changea adaptablea, resilientb, elastica Resilience the ability to recover quickly from illness, change, or misfortunea buoyancya, flexibilityb, adaptabilityb Improvisation to invent, perform or compose something extemporaneouslya extemporizeb, ad-libb, inventb, get byc, dealc Innovation act of introducing something newa inventionc, conceptionc, creativityc Creativity the ability to create a productivityb, innovationc The study in Paper I carefully investigates the use of each of the six terms (FAIRIC) to determine what the nuances really mean, how each of the abilities should be interpreted, and in what situations they are needed. The FAIRIC model lays them out on two [44] dimensions: unfamiliarity of the situation, and proactiveness. The idea behind the model is that these two dimensions represent the two most important differences that separate the FAIRIC abilities (consider e.g. the dictionary explanations of responsiveness to change vs. ability of becoming suitable to a particular situation). The six abilities can be mapped onto the two dimensions as shown earlier in Figure 4.1. The abilities on the lefthand side represent a lower degree of proactiveness compared to those on the right, e.g., innovation can be employed in advance to prepare for new situations, whereas improvisation has an element of time pressure in it, meaning it is primarily employed reactively in dynamic situations as a response to the unfolding events. The unfamiliarity dimension shows that improvisation and innovation is employed primarily in unfamiliar situations, whereas more familiar situations can be responded to by having a high degree of adaptability (e.g., adapting the organization or procedure in response to change, i.e., reactive) or by implementing flexibility (e.g., having a flexible organization or process that allows solving of multiple events, i.e., proactive). The six FAIRIC abilities should be considered separate, yet related, and usable in different scenarios. The reviewed literature has identified all these six abilities as important, although the magnitude of their importance has not been compared. To summarize, mission experience increases the ability of creating better recognitionprimed decisions through an increased knowledge base of situations. The study in Paper I showed that this increased ability is linked to the six FAIRIC abilities, which are an important, yet non-comprehensive, set of experience-guided abilities that tactical military and emergency management organizations employ in order to cope with dynamic situations. Increased mission experience does thus improve the commanders’ ability of performing opportunistic control through the reactive abilities of adaptability, resilience and improvisation. These abilities are guided by recognition, a process which seems dependent on mission experience. In dynamic situations, tactical control is preferred over opportunistic, requiring more proactive measures, such as preparation and projection. Preparing for the entire spectra of scenarios that military and emergency management organizations may face is impossible. However, knowledge and experience increase the scope of tactical scenarios these organizations can manage by stimulating creativity, guiding innovation, and increasing flexibility. 5.2 HOW SHOULD COMMANDERS’ MISSION EXPERIENCE BE EXTERNALIZED IN ORDER TO ENABLE VICARIOUS LEARNING FROM TACTICAL MILITARY AND EMERGENCY MANAGEMENT OPERATIONS? In the case studies of both Paper II and Paper III, reconstruction is used to compile mission histories of the studied exercises. Both studies showed that partial reconstruction can be used for dissemination to help participants recall what happened during each exercise session. As such, R&E has been shown valuable to stimulate recollection during after-action review by presenting an undisputable ground truth to focus discussions. The Sandö case study (Paper II) additionally showed that non-participating trainees can use R&E to internalize knowledge similar to that experience acquired by those who participated in the training, i.e., vicarious learning. [45] The compiled mission histories enclosed a massive amount of data concerning the decision making processes. In the Sandö case study, most data capturing was focused on the commander, whereas in the Armed Forces case study (Paper III) a representative collection of key individuals was selected for inclusion in the data collection phase of the reconstruction. The captured data in both cases constitute a significant set of decisionmaking cues, such as what the trainees see and hear, and how they interact with the environment and the team. Arguably the most important cues during tactical operations are communication; in fact C2 depends on the continuity of communication across echelon (Alberts & Hayes, 2003). This is also supported by findings from both case studies, as communication was claimed to be the most useful for understanding the ongoing situation. Two other aspects that were highlighted by trainees in the Sandö case study are time synchronization and objectivity, i.e., the respondents particularly appreciated the ability to get an ordered list of factual data that could not be disputed, as provided by the model presented in Figure 2.7. Such a list helped the trainees recall the events in order of occurrence and retrospectively reflect upon cause and effect relationships. Other data sources, such as GPS tracks, observer notes, photos, and video, were considered secondary, but useful for navigation. For the visual cues, the reason may be because most of the respondents participated in the exercise and already had a clear picture of the context that these data sources could provide. From the case studies, it is hard to generalize from this data whether or not such data sources have any value for vicarious learning. However, it can be assumed from the findings of e.g. Liu and colleagues (2009) that a combination of high media richness data such as video and photo are useful in combination with other data if the technology acceptance is high enough among the students. The Swedish Armed Forces case study highlights the importance of observer notes, as they can guide the exploration phase of an otherwise potentially overwhelming mission history. To summarize, the following externalization findings were generated from the reconstruction of mission histories during the Sandö and the Swedish Armed Forces case studies11: Recorded voice communication, both face-to-face and through radio, is considered the most important to analyze and understand the decision-making process of commanders, and is therefore vital for the transferal of mission experience (Paper II). Triaging logs are useful for mission efficiency analysis, but provide none, or little, additional value for transferal of mission experience (Paper II). Photos and videos help trainees remember the chaotic events that occurred during a mission (Paper II). It is expected that they provide additional value for the purpose of mission experience transfer. Screen captures were not shown useful for evaluation of virtual training sessions (Paper III). Their value for mission experience transfer is unknown. 11 It should be noted that these findings may be highly contextual, and not generalizable to other tactical scenarios, e.g., with sparse communication. [46] GPS tracks and observer notes are useful to help understand the exploration of mission histories, and may therefore be valuable to guide the mission experience transfer process (Paper II). Other important lessons from the case studies include the stressing of time synchronization and objectivity as necessary to enable causal analysis. Maintaining proper time stamping of all data is thus a necessity to ensure intact ordering of the events during exploration. It is also advisory to refrain from inputting subjective data into the dataset, as the objectivity may then be violated. 5.3 WHAT TECHNICAL, METHODOLOGICAL, AND ETHICAL ISSUES NEED TO BE CONSIDERED WHEN EXTERNALIZING MISSION EXPERIENCE FROM TACTICAL MILITARY AND EMERGENCY MANAGEMENT COMMANDERS? The case studies from Sandö (Paper II) and the Swedish Armed Forces (Paper III) show that there are indeed a number of technical and methodological issues that must be considered when capturing information in the operative environment of a tactical team. One of the biggest issues identified from the case studies is the difficulty in managing the massive datasets that were generated during the exercises, leaving the analysts with large amounts of data but without the ability to explore and analyze it. While there are certainly methods to deal with larger datasets, e.g., through more computational power; visual exploration and human processing is often necessary to make sense of reconstructed mission histories. A set of indicators of what data to process may help analysts reduce the dataset to explore by filtering the streams as shown in Figure 5.1, where indicators I1 and I2 are used with a time window (yellow) to reduce the synchronized data set consisting of two video streams and one audio stream in the top left image (a) to the much smaller set in the bottom right (e), leaving the analyst with less data to explore, thereby reducing the workload. Such indicators can be generated, e.g., by observer notes as in the Sandö study (Paper III) and by using data fusion techniques through which computers preprocess the dataset to find anomalies, outliers and interesting patterns to guide the analysts. a) b) d) c) e) Figure 5.1. Indicators help analysts find interesting data points in massive data sets, thereby greatly reducing the data set, exemplified by moving through the outlined sequence of reducing the dataset in (a) to that in (e) through the intermediate steps (b), (c), and (d). [47] The Sandö study (Paper II) showed that another methodological problem associated with reconstruction of mission experience is the fact that the exploration process may overwhelm the audience. Creating methods and tools to support internalization of mission experience is a difficult task which may possibly draw from the fields of data visualization and interactive learning. Assuming that decision-making cues are made up of stimuli to the five human senses, capturing them is a matter of recording and synthesizing these stimuli. Sensor systems can do a great job of synthesizing these signals in almost any environment. However, it is always a challenge to setup the sensor systems in such a way that they indeed capture the inputs that the team members are exposed to, and to separate them from stimuli that the team members do not acknowledge. The internal sensemaking and decision-making processes that occur within each individual is also difficult to capture without invasive measurements, making it hard to realize exactly which stimuli that really led to each particular decision. For practical reasons, these processes are omitted from the model and the synthesis. The proposed approach for synthesizing mission experience into something that can be transferred through visual and analytical exploration is based on massive data collection from the team, operation and the environment. Conducting such data collection without violating any individuals’ privacy is another major challenge as discussed in Paper IV. The privacy issue does not only concern the mental health of the team members, but also the society as a whole when there is an imminent risk that the organization’s ability to perform decreases from team members loosing self-esteem and autonomy as a consequence of privacy infringements. The privacy issue is connected to the question of informed consent, which has been identified as problematic because of the complexity of making consent truly informed as well as the complexity of maintaining informed consent when circumstances change, such as personal data is accidently captured, mission objective changes, and unexpected findings are revealed during exploration. In summary, the main technical (T), methodological (M), and ethical (E) issues that have been identified for capturing and modeling of tactical mission experience are: difficulty in managing massive datasets [T, M], difficulty in navigating/exploring massive datasets without indicators [M], information overload overwhelming explorers [T, M], replaying decision making cues other than auditory and visual [T], coupling stimuli to decision making [M], avoiding privacy infringements [E], and maintaining continuous informed consent [E]. [48] Chapter 6 DISCUSSION This section discusses how mission experience can be transferred, and what role R&E plays for the externalization/internalization process. The summarized discussion is followed by a brief introduction on proposed future work, building on the findings from the studies presented in this thesis. Finally, the conclusions related to the objective, as stated in the introduction of this thesis, are presented. 6.1 TACTICAL COMMAND AND CONTROL IN DYNAMIC SITUATIONS Existing models of C2, decision making and sensemaking all provide support to understanding what mission experience really is, how it can be modeled, and how it can be captured. They show that controllers in a tactical operation need an objective, shared understanding and the ability to observe and make sense of an ongoing situation. However, in reality objectives are often ill-defined and sometimes conflicting (Klein, Ross, Moon, Klein, Hoffman & Hollnagel, 2003), and time and resources may be lacking to properly analyze the situation to make sense of it. Tactical control mode can still be applied if the situations are somewhat familiar and standard operating procedures (SOPs) have been defined. However, a tactical organization may be poorly equipped to deal with infrequent events for which they have not been trained (Harbom, 2010). What is left then is the reliance upon experience and prior knowledge, i.e., going into opportunistic control mode. In these situations, decision making can be regarded as recognition-primed, which is highly dependent on the prior existence of a knowledge base. The models reviewed in this thesis highlight that controlling tactical missions are above all a matter of sensing the dynamic environment, adapting the JCS, and responding as the situation requires, often as a result of employing at least one of the six FAIRIC abilities. The experience gained from the mission is just that, i.e., the ability to sense and make sense of a scenario such as the one that just unfolded, the ability to assess and understand the phenomena that occurred, and the ability to counter it with appropriate measures. Having such experience improves the likelihood of a tactical unit being able to stay in tactical control mode during future events, thus enabling them to perform better. 6.2 TRANSFERAL OF MISSION EXPERIENCE “The importance of context and history means that there is no substitute for experience” (Cilliers, 2000, p.12). Certainly this thesis does not dispute Cilliers’ argument. The main point is that experience of others can be used to stimulate creation of new tacit knowledge on the recipient side. This knowledge can never substitute mission experience, but it can complement, and it is a much better substitute than the alternative of having no knowledge at all about a certain type of mission or operation. The SECI model for knowledge transfer seems applicable in a mission-experience context, telling us that knowledge transfer is not about moving or copying knowledge, but rather about the generation of new knowledge. Internalizing a dataset captured from a [49] tactical operation will thus stimulate creation of new knowledge as opposed to replicating the knowledge that the studied commander or team member gained. If this knowledge can be used for recognition and to make better future decisions, then this approach opens up new ways for organizations to manage their memory from infrequent events, possibly addressing the challenge put forward by Means et al. (1993): to provide a classroomversion of the mission history, complete with characteristic cues and tasks to study for the trainee. Consider for instance the Fukushima incident in 2011 where an earthquake triggered both a nuclear accident and a tsunami. This chain of events had not been anticipated and no organization was fully prepared for it. It is reasonable to believe that disaster response organizations worldwide will go through several iterations of staff turnover before an incident like this happens again, if ever. It is also reasonable to believe that the managers that actually did handle this very incident learned something from the events, i.e., they acquired tacit knowledge, and experience. Making this knowledge explicit is, according to the SECI model, the only method through which it can be made persistent and shared over boundaries of time and space. A common approach to externalize such knowledge in tactical organizations is the manual approach, i.e., to design a procedure or create a manual of how to handle incidents, based on one or several stereotypical situations. Often these manuals are written by someone outside the team with the experience, and thus the accuracy of the manual is subject to the quality of an interview and the ability of the interviewer to extract information from the interviewee. Mixed methods approaches such as R&E are superior for externalization of knowledge in the sense that they can extract information that subjects are not conscious of, while also being able to include traditional data sources such as interviews as integrated components of an externalized mission history. The interpretation of it all, however, is left to the explorer, leaving a larger cognitive burden for anyone internalizing a multimedia-heavy mission history compared to studying a written manual. Hence, the method cannot replace existing methods, but should rather be seen as a complement to them. With proper experience, a tactical unit can improve flexibility and adaptability, thus reducing the amount of unfamiliar situations, however, mission experience may also be valuable to improve improvisation and innovation, making the organizations more capable of handling those events that cannot be foreseen. Ultimately, an increased arsenal of internalized mission histories will improve decision making, rendering the tactical unit more efficient. To summarize, knowledge transfer through externalization and reinternalization adds a new method, unrestricted by boundaries of time and space, through which an organization may boost its’ ability to perform. 6.3 RECONSTRUCTION AND EXPLORATION The two presented case studies show that R&E as a mixed methods approach is capable of capturing and propagating auditory and visual cues in decision making scenarios. The studies employ the approach in one emergency response and one large-scale military setting. R&E as an approach for modeling exercises and experiments for C2 analysis and AAR has been conceptually verified in several field trials (e.g., Thorstensson et al., 1999; Thorstensson, Johansson, Andersson & Albinsson, 2007; Andersson et al., 2011). The [50] only data that point to its applicability for transferring mission experience are the Sandö case study, in which several trainees commented that they saw value in internalizing mission history to sense some of what the subject commander was exposed to (see Paper II) and a survey from the Alvesta field trial (Jenvald et al., 2000). Still, these data points encourage future attempts to investigate the concept of mission experience transferal through R&E. A key to successfully being able to share the experience is that as much as possible of the operation is synthesized in such a way that the learners can relive the experience and understand the cues and dilemmas that the team members were facing during the actual operation. In most tactical scenarios, the information that needs to be synthesized is composed of visual and auditory cues, decisions, and actions, plus the context under which the operation is executed. Capturing the cues, decisions, and actions in a holistic sense is quite straight-forward using sensor systems in the operative environment and on the team. This works well until the moment comes when the captured artifacts must be coupled, i.e., deciding which cue led to one particular decision and set of actions, and vice versa. Quantification is useful to generate indices and indicators that can assist exploration and learning. However, the decision making cues and context must be propagated to the recipients for them to truly understand the reasons and motives for decisions and actions, and to get a sensation of experiencing the reconstructed scenarios. Qualitative data seems more appropriate to propagate such cues, since the underlying mental processes cannot be captured and are not fully understood. Thus, quantitative and qualitative methods complement each other in this setting, allowing better completeness (Venkatesh et al., 2013) which might be an essential property if a reconstructed mission history is to be embraced and internalized. 6.4 PRIVACY Capturing datasets to reconstruct mission experience introduce a privacy infringement concern, which must always be kept in mind to reduce the risk of unwanted mental effects. At a minimum, informed consent is needed from all participants subject to monitoring and data capture. This consent should also be kept continuously informed, i.e., any individual must have the right to know if the conditions under which the consent was given are going to be violated, and always have the ability to withdraw the consent. Taking such precautionary measures is not as easy as it may sound, since it can be difficult to predict what data can and will be used for at the time consent is requested. If it would have been easy to determine when an update of the consent is needed, then privacy would probably not be a complicated issue; unfortunately that is not always the case. In pursuit of progress it seems inevitable that tactical operations are evaluated and that organizations try to learn and remember from both live operations and exercised. The monitoring and continuous assessment processes can create a sensation of a “big brother” environment (consider Orwell’s classic novel 1984). This thesis concludes that exercise and assessment managers must acknowledge the problem and become aware of the potential privacy infringements that assessment introduce. Doing so enables them to have open discussions and creates a climate where everyone is aware of the consequences for [51] themselves, making it possible for them to give consent that can be classified as truly informed. 6.5 SUMMARY R&E is a mixed methods approach for evaluation of tactical operations. As showcased in this thesis it can be applied to create explicit models of the experience that teams acquire through tactical operations. The primary purpose of such explicit models is to serve as a baseline for immediate after-action review. However, the Sandö case study showed that the models can also be seen as externalized experience which other trainees can study and internalize through interactive exploration. While this result has not been validated, it is interesting enough to promote so that future research may find alternative and cheaper methods of sharing rare mission experience. Internalizing such a mission history can complement regular training and exercising, and yield an alternative for organizations that wish to maintain knowledge from rare events; while at the same time opening up the ability to share these explicit products, to enable transferal of mission experience within and between organizations. A final word of warning is that R&E should never replace training and exercising, but that it may beneficially complement it. 6.6 FUTURE WORK The most obvious need for future work is that of verifying the R&E’s usefulness as a mediator for experience transfer through empirical studies. Ideally, such studies should be longitudinal and experimental to explore a teams’ level of experience as it changes over time and compare that to teams who train the same scenarios in a more traditional manner, i.e., measuring the dependent variable derivative of a teams’ effectiveness as a function of the independent variable internalization method. Furthermore, the technical and methodological issues that have been identified in this thesis all need to be resolved in order for the proposed approach to become truly usable by the tactical organizations. Fortunately, the areas of big data and visual exploration receive plenty of attention in the research community today, yielding reason to hope that the future holds solutions that may be adoptable for R&E, possibly countering the identified data management, navigation, and information overload issues. Research is also being conducted on reproduction of tactile and olfactory stimuli, although it is not clear what effects this will have on learning, nor what it will do to technology acceptance among the intended trainees. Finally, further investigations, e.g., through interviews and experiments, need to be conducted in order to resolve the ethical matters that have been raised in this thesis, i.e., primarily how to avoid privacy infringements and maintain continuous informed consent. 6.7 CONCLUSIONS This work presents a new approach to understand how experience affects performance in tactical teams. The FAIRIC model shows six abilities which are all guided by experience and can improve the team’s ability to stay in tactical or at least opportunistic control mode in a dynamic situation. The model may be used as a unified vocabulary to guide [52] future research on dynamic decision making in tactical military and emergency management operations. Data from the Sandö case study indicates that externalizing mission experience through reconstruction allows vicarious learning through exploration. To successfully accomplish this, the product of the externalization must be time synchronized and objective. A mission history implementing this also need to include the major cues for decision making, in the Sandö case study identified as communication. Both case studies lack a proper investigation into what data sources are most needed to enable internalization if the exploring subject has not participated in the exercise. The findings are thus limited, but other research has shown that a combination of high and low media richness outperforms any single data source for learning (Liu et al., 2009), leading to the conclusion that media such as GPS tracks and system logs provide a useful baseline together with more high media richness data such as video and photos to provide a context and understanding of the unfolding events. Furthermore, several issues have been identified, methodological, technical and ethical. The issues range from difficulties in managing and navigating big data, to avoiding information overload among explorers of the mission history. Arguably, the most severe issue of all is to overcome the challenges of maintaining continuous informed consent to avoid privacy infringement. The identification of these issues raises awareness so that precautionary measures can be initiated to successfully avoid running into these problems when externalizing mission experience from tactical military and emergency management operations. [53] [This page is intentionally left blank] [54] REFERENCES Abecker, A., & Decker, S. (1999). Organizational Memory: Knowledge Acquisition, Integration, and Retrieval Issues. Lecture Notes in Computer Science, 1570/1999, 113–124. Aiello, J. R., & Klob, K. J. (1996). Electronic Performance Monitoring: A Risk Factor for Workplace Monitoring. In S. L. Slater & L. R. Murphy (Eds.), Organizational Risk Factors and Job Stress (pp. 163–179). American Psychological Association. Alberts, D. S., & Hayes, R. E. (Eds.). (2003). Power to the Edge. Washington, DC: DoD Command and Control Research Program. Alberts, D. S., & Hayes, R. E. (Eds.). (2006). Understanding Command and Control. Washington, DC: DoD Command and Control Research Program. Alberts, D. S., & Hayes, R. E. (Eds.). (2007). Planning: Complex Endeavors. Washington, DC: DoD Command and Control Research Program. Albinsson, P.-A., Wong, W., Pilemalm, S., & Morin, M. (2005). Using a ReconstructionExploration approach to Empower Cognitive Analysis Methods: Possibilities and Challenges. Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society (pp. 224–228). Orlando, FL. Andersson, D. (2009). F-REX: Event-Driven Synchronized Multimedia Model Visualization. Proceedings of the 15th International Conference on Distributed Multimedia Systems (pp. 140–145). Redwood City, CA: Knowledge Systems Institute. Andersson, D. (2012). F-REX anpassad för IT-försvarsövningar i NCS3 (FOI Memo 4336) [eng: F-REX configured for IT defense exercises]. Linköping, Sweden: Swedish Defense Research Agency. Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011). Analysis of a Cyber Defense Exercise using Exploratory Sequential Data Analysis. Proceedings of the 16th International Command and Control Research and Technology Symposium. Québec City, Canada: DoD Command and Control Research Program. Andersson, D., Mullins, S., Bordetsky, A., Bourakov, E., & Forsgren, R. (2011). MIO Experiment Data Capture. Proceedings of the 10th International Conference on Telecommunication Systems: Modeling and Analysis. Prague, Czech Republic. Argote, L., & Miron-Spektor, E. (2011). Organizational Learning: From Experience to Knowledge. Organization Science, 22(5), 1123–1137. [55] Ashby, W. R. (1956). An Introduction to Cybernetics. London, UK: Chapman and Hall. Baron, S., Kruser, D. S., & Huey, B. M. (Eds.). (1990). Quantitative Modeling of Human Performance in Complex, Dynamic Systems. Washington, DC: National Academy. Bernard, H. R. (2011). Research methods in antropology: qualitative and quantitative approaches (5th ed.). Plymouth, UK: AltaMira. Beyer, H., & Holtzblatt, K. (1999). Contextual Design. Interactions, 6(1), 32–42. Blandford, A., & William Wong, B. L. (2004). Situation awareness in emergency medical dispatch. International Journal of Human-Computer Studies, 61(4), 421– 452. Boyd, J. R. (1987). A discourse on winning and loosing. Air University Library Document No. M-U 43947. Retrieved March 5, 2013, from http://dnipogo.org/johnr-boyd/ Brehmer, B. (1992). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81(3), 211–241. Brehmer, B. (2000). Dynamic Decision Making in Command and Control. In C. McCann & R. Pigeau (Eds.), The Human in Command: Exploring the Modern Military Experience (pp. 233–248). New York, NY: Kluwer Academic/Plenum Publishers. Retrieved December 3, 2012, from http://www.nada.kth.se/kurser/kth/2D1416/litteratur/Artman.pdf Brehmer, B. (2005). The Dynamic OODA Loop: Amalgamating Boyd’s OODA Loop and the Cybernetic Approach to Command and Control. Proceedings of the 10th International Command and Control Research and Technology Symposium. McLean, VA: DoD Command and Control Research Program. Brehmer, B. (2009). Command without Commanders. Proceedings of the 14th International Command and Control Research and Technology Symposium. Washington, DC: DoD Command and Control Research Program. Breton, R., & Rousseau, R. (2008). The M-OODA: Modelling of the OODA loop as a modular functional system (TM 2008-130). Valcartier, Canada: Defense Research and Development Canada. Builder, C. H., Bankes, S. C., & Nordin, R. (1999). Command Concepts: A theory derived from the practice of Command and Control (MR-775-OSD). Santa Monica, CA: RAND. Cilliers, P. (2000). Knowledge, Complexity, and Understanding. Emergence, 2(4), 7–13. [56] Cronin, P., Ryan, F., & Coughlan, M. (2008). Undertaking a literature review: a step-bystep approach. British Journal of Nursing, 17(1), 38–43. Retrieved March 19, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/18399395 Daft, R. L., & Lengel, R. H. (1986). Organizational Information Requirements, Media Richness and Structural Design. Management Science, 32(5), 554–571. Dean, P. J. (1998). A qualitative method of assessment and analysis for changing the organizational culture. Performance improvement, 37(2), 14–23. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2000). Handbook of Qualitative Research (2nd ed.). Thousand Oaks, CA: SAGE. Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 37(1), 32–64. Endsley, M. R. (2000). Direct Measurement of Situation Awareness: Validity and Use of SAGAT. In M. R. Endsley & D. J. Garland (Eds.), Situation Awareness Analysis and Measurement. Mahwah, NJ: Lawrence Erlbaum. Endsley, M. R. (2004). Situation awareness: Progress and Directions. In S. P. Banbury & S. Tremblay (Eds.), A Cognitive Approach to Situation Awareness: Theory and Application (pp. 317–241). Aldershot, UK: Ashgate. Essens, P., Vogelaar, A., Mylle, J., Blendell, C., Paris, C., Halpin, S., & Baranski, J. (2005). Military Command Team Effectiveness: Model and Instrument for Assessment and Improvement (RTO-TR-HFM-087). Neuilly-sur-Seine Cedex, France: NATO Research and Technology Organisation. Finley, D. L., Muckler, F. A., Gainer, C. A., & Obermayer, R. W. (1975). An analysis and evaluation methodology for Command and Control: Final technical report (p. 96). Northridge, CA: Manned Systems Science. Flanagan, J. C. (1954). The critical incident technique. Psychological bulletin, 51(4), 327–358. Retrieved March 21, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/13177800 Fredholm, L., & Åström, M. (2010). Incident command and decision making. In L. Fredholm & A.-L. Göransson (Eds.), Emergency Response Management in Today’s Complex Society (pp. 109–122). Karlstad, Sweden: Swedish Civil Contingencies Agency. Frey, J. H., & Fontana, A. (1991). The group interview in social research. Social Science Journal, 28(2), 175–188. [57] Försvarsmakten. (2011). Militärstrategisk doktrin 2012 - med doktrinära grunder (MSD 12) (M7739-354023). Stockholm, Sweden: Försvarsmakten. Gourlay, S. (2003). The SECI model of knowledge creation: some empirical shortcomings. Proceedings of the 4th European Conference on Knowledge Management (pp. 377–385). Oxford, UK. Harbom, S. (2010). Management of various types of emergency. In L. Fredholm & A.-L. Göransson (Eds.), Emergency Response Management in Today’s Complex Society (pp. 73–108). Karlstad, Sweden: Swedish Civil Contingencies Agency. Hegarty, M. (2005). Multimedia Learning about Physical Systems. In R. E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 447–465). Cambridge, UK: Cambridge University. Hijikata, Y., Takenaka, T., Kusumura, Y., & Nishida, S. (2007). Interactive Knowledge Externalization and Combination for SECI. K-CAP 07 (pp. 151–158). Whistler, Canada. Hof, T., De Koning, L., & Essens, P. (2010). Measuring Effectiveness of Teams and Multi-team Systems in Operation. Proceedings of the 15th International Command and Control Research and Technology Symposium. Santa Monica, CA: DoD Command and Control Research Program. Hollnagel, E. (1993). Human reliability analysis: Context and control. London, UK: Academic Press. Hollnagel, E., & Woods, D. D. (2005). Control and Cognition. In E. Hollnagel & D. D. Woods (Eds.), Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. Boca Raton, FL: Taylor & Francis. Howard, R. A. (1966). Decision Analysis: Applied Decision Theory. Proceedings of the 4th International Conference on Operational Research (pp. 55–77). Boston, MA: Wiley-Interscience. Jenvald, J. (1999). Methods and Tools in Computer-Supported Taskforce Training. Linköping Studies in Science and Technology, dissertation no. 598. Linköping, Sweden: Linköping University. Jenvald, J., Morin, M., & Rejnus, L. (2000). Developing Digital Courseware from Multimedia Documentation of Full-Scale Chemical Exercises. Proceedings of the NBC 2000 Symposium on Nuclear, Biological and Chemical Threats in the 21st Century. Espoo, Finland. [58] Jenvald, J., Rejnus, L., Morin, M., & Thorstensson, M. (1998). Computer-Supported Assessment of Emergency Planning for Rescue Operations (FOA-R--98-00910-505). Linköping, Sweden: Swedish Defense Research Establishment. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263–292. King, D. J. (2005). Humanitarian Knowledge Management. Proceedings of the 2nd International ISCRAM Conference, Brussels, Belgium. Kizza, J. M., & Ssanyu, J. (2005). Workplace Surveillance. In J. Weckert (Ed.), Electronic Monitoring in the Workplace: Controversies and Solutions. Hershey, PA: Idea Group. Klein, G. A. (1993). A Recognition-Primed Decision (RPD) Model of Rapid Decision Making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and Methods (pp. 138–147). Norwood, NJ: Ablex. Klein, G. A. (1995). Naturalistic Decision Making and Wildland Firefighting. Findings From the Wildland Firefighters Human Factors Workshop. Retrieved March 6, 2013, from http://www.fs.fed.us/t-d/pubs/htmlpubs/htm95512855/page13.htm Klein, G. A., Moon, B., & Hoffman, R. R. (2006a). Making Sense of Sensemaking 1: Alternative Perspectives. IEEE Intelligent Systems, 21(4), 70–73. Klein, G. A., Moon, B., & Hoffman, R. R. (2006b). Making Sense of Sensemaking 2: A Macrocognitive Model. IEEE Intelligent Systems, 21(5), 88–92. Klein, G. A., Ross, K. G., Moon, B. M., Klein, D. E., Hoffman, R. R., & Hollnagel, E. (2003). Macrocognition. IEEE Intelligent Systems, 18(3), 81–85. Lafond, D., & DuCharme, M. B. (2011). Complex Decision Making Experimental Platform (CODEM): A Counter-Insurgency Scenario. IEEE Symposium on Computational Intelligence for Security and Defence Applications (pp. 72–79). Paris, France. Lantz, F., Andersson, D., Jungert, E., & Levin, B. (2009). Decision Support for Monitoring the Status of Individuals. Proceedings of the 15th International Conference on Distributed Multimedia Systems (pp. 123–129). Redwood City, CA: Knowledge Systems Institute. LaVoie, N., Foltz, P. W., Rosenstein, M., Oberbreckling, R., Chatham, R., & Psotka, J. (2008). Automated Support for AARs: Exploiting Communication to Assess Team Performance. Proceedings of the 2008 Interservice/Industry Training, Simulation, and Education Conference. Orlando, FL. [59] Lawson, Jr., J. S. (1981). Command Control as a Process. IEEE Control Systems Magazine, 1(1), 5–11. Lee, M. K. O., Cheung, C. M. K., & Chen, Z. (2005). Acceptance of Internet-based learning medium: the role of extrinsic and intrinsic motivation. Information & Management, 42(8), 1095–1104. Lewis, R. B., & Maas, S. M. (2007). QDA Miner 2.0: Mixed-model qualitative data analysis software. Field Methods, 19(1), 87–108. Lieber, E. (2009). Mixing Qualitative and Quantitative Methods: insights into design and analysis issues. Journal of Ethnographic & Qualitative Research, 3(4), 218–227. Lieber, E., Weisner, T. S., & Presley, M. (2003). EthnoNotes: An Internet-Based Field Note Management Tool. Field Methods, 15(4), 405–425. Lim, K. H., & Benbasat, I. (2000). The Effect of Multimedia on Perceived Equivocality and Perceived Usefulness of Information Systems. MIS Quarterly, 24(3), 449–471. Liu, S.-H., Liao, H.-L., & Pratt, J. A. (2009). Impact of Media Richness and Flow on elearning Technology Acceptance. Computers & Education, 52(3), 599–607. Malek, J. (2005). Informed Consent. In C. Mitcham (Ed.), Encyclopedia of Science, Technology and Ethics, vol. 2 (pp. 1016–1019). Detroit, MI: Macmillan Reference USA. Mayer, R. E. (2005). Cognitive Theory of Multimedia Learning. In R. E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 31–48). Cambridge, UK: Cambridge University. Means, B., Salas, E., Crandall, B., & Jacobs, T. O. (1993). Training decision makers for the real world. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and Methods (pp. 306–326). Norwood, NJ: Ablex. Mendonça, D., Beroggi, G. E. G., & Wallace, W. A. (2001). Decision support for improvisation during emergency response operations. International Journal of Emergency Management (IJEM), 1(1), 30–39. Mendonça, D., & Fiedrich, F. (2006). Training for improvisation in emergency management: opportunities and limits for information technology. International Journal of Emergency Management (IJEM), 3(4), 348–363. Mendonça, D., & Hu, Y. (2009). Cognitive Underpinnings of Resilience: A case study of group decision in Emergency Response. In P. Nemeth, E. Hollnagel, & S. Dekker [60] (Eds.), Resilience Engineering Perspectives, vol. 2: Preparation & Restoration (pp. 179–204). Surrey, UK: Ashgate. Miller, S. (2005). Guarding the Guards: The Right to Privacy, and Workplace Surveillance and Monitoring in Policing. In J. Weckert (Ed.), Electronic Monitoring in the Workplace: Controversies and Solutions. Hershey, PA: Idea Group. Mintzberg, H. (1979). The Structuring of Organizations: A Synthesis of the Research. Englewood Cliffs, NJ: Prentice-Hall. Mintzberg, H., & McHugh, A. (1985). Strategy Formation in an Adhocracy. Administrative Science Quarterly, 30(2), 160–197. Moorman, C., & Miner, A. S. (1998). Organizational Improvisation and Organizational Memory. The Academy of Management Review, 23(4), 698–723. Morin, M. (2001). MIND – Methods and Tools for Visualization of Rescue Operations. Proceedings of the International Emergency Management Society’s Eight Annual Conference (TIEMS 2001). Oslo, Norway. Morin, M. (2002). Multimedia Representation of Distributed Tactical Operations. Linköping Studies in Science and Technology, dissertation no. 771. Linköping, Sweden: Linköping University. Morin, M., & Albinsson, P.-A. (2005). Exploration and Context in Communication Analysis. In C. A. Bowers, E. Salas, & F. Jentsch (Eds.), Creating High-Tech Teams: Practical Guidance on Work Performance and Technology. Washington, DC: American Psychological Association. Morin, M., Jenvald, J., & Worm, A. (1998). Training Emergency Management and Response: Creating the Big Picture. Proceedings of the International Emergency Management Society’s Fifth Annual Conference (TIEMS 98) (pp. 553–561). Washington, DC. Ngwenyama, O. K., & Lee, A. S. (1997). Communication Richness in Electronic Mail: Critical Social Theory and the Contextuality of Meaning. MIS Quarterly, 21(2), 145–167. Nonaka, I. (1991). The Knowledge-Creating Company. Harvard Business Review, 69(6), 96–104. Nonaka, I., & Takeuchi, H. (1995). The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation (p. 304). Oxford, UK: Oxford University. [61] Nonaka, I., Toyama, R., & Konno, N. (2000). SECI , Ba and Leadership: a Unified Model of Dynamic Knowledge Creation. Long Range Planning, 33, 5–34. Novak, J. (2007). Helping Knowledge Cross Boundaries: Using Knowledge Visualization to Support Cross-Community Sensemaking. Proceedings of the 40th Annual Hawaii International Conference on System Sciences (pp. 1–10). Big Island, HI. Oosthuizen, R., & Roodt, J. H. S. (2012). Coping with Complexity in Command and Control. Proceedings of the 17th International Command and Control Research and Technology Symposium. Fairfax, VA: DoD Command and Control Research Program. Otondo, R. F., Van Scotter, J. R., Allen, D. G., & Palvia, P. (2008). The complexity of richness: Media, message, and communication outcomes. Information & Management, 40, 21–30. Palm, E. (2007). The Ethics of Workplace Surveillance. Royal Institute of Technology, Stockholm, Sweden. Pettersson, U. (2009). Sharing Knowledge: How to Highlight Proven Experience in the Swedish Armed Forces. Proceedings of the 6th International ISCRAM Conference. Gothenburg, Sweden. Pigeau, R., & McCann, C. (2002). Re-conceptualizing command and control. Canadian Military Journal, 3(1), 53–64. Polanyi, M. (1966). The Tacit Dimension. Garden City, NY: Doubleday & Company. Rankin, A., Dahlbäck, N., & Lundberg, J. (2013). A case study of factor influencing role improvisation in crisis response teams. Cognition, Technology & Work, 15(1), 79– 93. Rao, V. S., & Goldman-Segall, R. (1995). Capturing Stories in Organizational Memory: The Role of Multimedia Systems. Proceedings of the 28th Annual Hawaii International Conference on System Sciences (pp. 333–341). Kihei, HI: IEEE Computer Society. Rice, J. L., & Rice, B. S. (2005). The Applicability of the SECI model to MultiOrganizational Endeavours: an Integrative Review. International Journal of Organizational Behaviour, 9(8), 671–682. Robinson, D., & Reed, V. (Eds.). (1998). The A-Z of social research jargon. Aldershot, UK: Ashgate. Ross, W. D. (1930). The Right and the Good. Oxford, UK: Oxford University. [62] Rowe, N. C. (2009). Automated Instantaneous Performance Assessment for MarineSquad Urban-Terrain Training. Proceedings of the 14th International Command and Control Research and Technology Symposium. Washington, DC: DoD Command and Control Research Program. Sadagic, A., Welch, G., Basu, C., Darken, C., Kumar, R., Fuchs, H., Cheng, H., et al. (2009). New Generation of Instrumented Ranges: Enabling Automated Performance Analysis. Proceedings of the 2009 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC). Orlando, FL. Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical education: basic guidelines for researchers. Medical Education, 43(7), 637–44. Seidman, I. (1998). Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. New York, NY: Teachers College. Sexton, R., & McConnan, I. (2003). A comparative study of after action review (AAR) in the context of southern Africa Crisis. Retrieved March 20, 2013, from http://www.alnap.org/pool/files/aar_case_study.pdf Starbuck, W. H., & Milliken, F. J. (1988). Executives’ perceptual filters: What they notice and how they make sense. In D. Hambrick (Ed.), The Executive Effect: Concepts and Methods for Studying Top Managers (pp. 35–65). Greenwich, CT: JAI Press. Retrieved March 5, 2013, from http://pages.stern.nyu.edu/~wstarbuc/Hambrick.htm Stewart, D. W., & Shamdasani, P. N. (1990). Focus Groups: Theory and Practice. Thousand Oaks, CA: SAGE. Svensson, S. (2010). The theory of fundamental tactics. Emergency Response Management in Today’s Complex Society (pp. 185–196). Karlstad, Sweden: Swedish Civil Contingencies Agency. Tellis, W. (1997). Application of a Case Study Methodology. The Qualitative Report, 3(3). Retrieved March 18, 2013, from http://www.nova.edu/ssss/QR/QR33/tellis2.html Thomas, A. S., & Kopczak, L. R. (2005). From Logistics to Supply Chain Management: The path forward in the humanitarian sector (p. 15). San Francisco, CA: Fritz Institute. Thorstensson, M., Axelsson, M., Morin, M., & Jenvald, J. (2001). Monitoring and analysis of command post communication in rescue operations. Safety Science, 39(1-2), 51–60. [63] Thorstensson, M., Johansson, M., Andersson, D., & Albinsson, P.-A. (2007). Ökat utbyte av övningar - Metoder och teknik för uppföljning och utvärdering av övningsverksamhet vid SRV:s skola i Sandö (FOI-R--2305) [eng: Improved outcome of exercises - Methods and tools for training and evaluation at the Swedish Rescue Services school at Sandö]. Linköping, Sweden: Swedish Defense Research Agency. Thorstensson, M., Morin, M., & Jenvald, J. (1999). Monitoring and Visualisation Support for Management of Medical Resources in Mass-Casualty Incidents. Proceedings of the International Emergency Management Society’s Sixth Annual Conference (TIEMS 99) (pp. 179–188). Delft, The Netherlands. Trnka, J., & Jenvald, J. (2006). Role-Playing Exercise: A Real-Time Approach to Study Collaborative Command and Control. International Journal of Intelligent Control and Systems, 11(4), 218–228. Trnka, J., Rankin, A., Jungert, E., Lundberg, J., Granlund, R., Granlund, H., & Johansson, B. J. E. (2009). Information Support in Modern Crisis and Disaster Response Operations. Linköping, Sweden: Linköping University. Tversky, A., & Kahneman, D. (1981). The Framing of Decisions and the Psychology of Choice. Science, 211(4481), 453–458. U.S. Army Combined Arms Center. (2011). Leader’s Guide to After-Action Reviews (AAR). Fort Leavenworth, Kansas: Headquarters Department of the Army. Warren, S. D., & Brandeis, L. D. (1890). The Right to Privacy. Harvard Law Review, 4(5), 193–220. Waterman Jr., R. H. (1990). Adhocracy: The power to change. Memphis, TN: Whittle Direct Books. Weick, K. E. (1969). The Social Psychology of Organizing. Reading, MA: AddisonWesley. Weick, K. E. (1993). The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster. Administrative Science Quarterly, 38(4), 628–652. Weick, K. E. (1995). Sensemaking in Organizations. Thousand Oaks, CA: SAGE. Weick, K. E., & Sutcliffe, K. M. (2001). Managing the unexpected: Assuring high performance in an age of uncertainty. San Francisco, CA: Jossey-Bass. Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems. MIS Quarterly, 37(1), 21–54. [64] Widder, W. (2002). Auftragstaktik and Innere Führung: Trademarks of German Leadership. Military Review, 82(5), 3–9. Wohl, J. G. (1981). Force Management Decision Requirements for Air Force Tactical Command and Control. IEEE Transactions on Systems, Man and Cybernetics, 11(9), 618–639. Worm, A. (1999). Mission Efficiency Analysis of Tactical Joint Systems. The Symposium of the RTO Studies, Analysis and Simulation (SAS) Panel. Issy les Moulineaux, France: NATO Research and Technology Organisation. Worm, A., Jenvald, J., & Morin, M. (1998). Mission efficiency analysis: evaluating and improving tactical mission performance in high-risk, time-critical operations. Safety Science, 30, 79–98. Yin, R. K. (1994). Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE. Zumel, N. B., Franco, Z. E., & Beutler, L. E. (2008). Improvisation as a Training Framework for Emergency Managers. Proceedings of the 13th International Command and Control Research and Technology Symposium. Seattle, WA: DoD Command and Control Research Program. [65] [This page is intentionally left blank] [66] LIST OF FIGURES Figure 2.1. The extended OODA loop (Boyd, 1987) ......................................................... 6 Figure 2.2. M-OODA (Breton and Rousseau, 2008) .......................................................... 7 Figure 2.3. The Dynamic OODA loop (Oosthuizen & Roodt, 2012) ................................. 7 Figure 2.4. A frames-based model of sensemaking (Klein et al., 2006b)......................... 10 Figure 2.5. Situation awareness (Endsley 1995) ............................................................... 11 Figure 2.6. Recognition-primed decision making (Klein, 1993) ...................................... 12 Figure 2.7. A representation of mission histories (Paper II) ............................................. 13 Figure 2.8. MIND Studio .................................................................................................. 14 Figure 2.9. F-REX Studio (Paper II) ................................................................................. 15 Figure 2.10. A recent version of F-REX Studio (Andersson, 2012)................................. 16 Figure 2.11. A control theory based model of tactical missions (Worm et al., 1998) ...... 17 Figure 2.12. Mission efficiency analysis model (Worm et al., 1998). .............................. 18 Figure 2.13. The SECI model (Nonaka & Takeuchi, 1995) ............................................. 20 Figure 4.1. The FAIRIC model (Paper I). ......................................................................... 32 Figure 4.2. The SECI model (Nonaka & Takeuchi, 1995) ............................................... 33 Figure 4.3. Reconstruction and Exploration (Andersson, 2009) ...................................... 36 Figure 5.1. Indicators for navigation................................................................................. 47 [67] [This page is intentionally left blank] [68] LIST OF TABLES Table 3.1. Study design ..................................................................................................... 25 Table 5.1. The FAIRIC concepts (from Paper I) .............................................................. 44 [69] [This page is intentionally left blank] [70] AUTHOR’S PUBLICATIONS Albinsson, P.-A., & Andersson, D. (2006). Computer-Aided Football Training: Exploiting Advances in Distributed Tactical Operations Research. Proceedings of the 6th International Sports Engineering Conference (pp. 185–190). Munich, Germany: Springer. Albinsson, P.-A., & Andersson, D. (2008). Extending the attribute explorer to support professional team-sport analysis. Information Visualization, 7(2), 163–169. Andersson, D. (2009). F-REX: Event-Driven Synchronized Multimedia Model Visualization. Proceedings of the 15th International Conference on Distributed Multimedia Systems (pp. 140–145). Redwood City, CA: Knowledge Systems Institute. Andersson, D. (2011). Privacy and Distributed Tactical Operations Evaluation. Proceedings of the 4th International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services. Barcelona, Spain. Andersson, D. (2013). A Knowledge Base for Capturing Comprehensive Mission Experience. Proceedings of the 46th Hawaii International Conference on System Science. Wailea, HI: IEEE Computer Society. Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011a). Exploratory sequential data analysis of a cyber defence exercise. Proceedings of the First International Defense and Homeland Security Simulation Workshop. Rome, Italy. Andersson, D., Granåsen, M., Sundmark, T., Holm, H., & Hallberg, J. (2011b). Analysis of a Cyber Defense Exercise using Exploratory Sequential Data Analysis. Proceedings of the 16th International Command and Control Research and Technology Symposium. Québec City, Canada: DoD Command and Control Research Program. Andersson, D., Mullins, S., Bordetsky, A., Bourakov, E., & Forsgren, R. (2011). MIO Experiment Data Capture. Proceedings of the 10th International Conference on Telecommunication Systems: Modeling and Analysis. Prague, Czech Republic. Andersson, D., Pilemalm, S., & Hallberg, N. (2008). Evaluation of crisis management operations using Reconstruction and Exploration. Proceedings of the 5th International ISCRAM Conference (pp. 118–125). Washington, DC. Andersson, D., & Rankin, A. (2012). Sharing Mission Experience in Tactical Organisations. In L. Rothkrantz, J. Ristvej, & Z. E. Franco (Eds.), Proceedings of the 9th International ISCRAM Conference (pp. 1–10). Vancouver, Canada. [71] Andersson, D., & Skagert, C. (2004). Managing Massive Datasets from Distributed Tactical Operations. Master thesis. Linköping, Sweden: Linköping University. Gustafsson, O., Amiri, K., Andersson, D., Blad, A., Bonnet, C., Cavallaro, J. R., Declerck, J., et al. (2010). Architectures for Cognitive Radio Testbeds and Demonstrators – An Overview. The 5th International Conference on Cognitive Radio Oriented Wireless Networks and Communications. Cannes, France. Hammervik, M., Andersson, D., & Hallberg, J. (2010). Capturing a Cyber Defence Exercise. The first national symposium on Technology and Methodology for Security and Crisis Management (p. 30). Linköping, Sweden. Holm, H., Ekstedt, M., & Andersson, D. (2012). Empirical Analysis of System-Level Vulnerability Metrics through Actual Attacks. IEEE Transactions on Dependable and Secure Computing, 9(6), 825–837. Lantz, F., Andersson, D., Jungert, E., & Levin, B. (2009). Decision Support for Monitoring the Status of Individuals. Proceedings of the 15th International Conference on Distributed Multimedia Systems (pp. 123–129). Redwood City, CA: Knowledge Systems Institute. Pilemalm, S., Andersson, D., & Hallberg, N. (2008). Reconstruction and Exploration of Large-scale Distributed Operations: Multimedia tools for Evaluation of Emergency Management Response. Journal of Emergency Management, 6(4), 31–47. Sigholm, J., & Andersson, D. (2011). Privacy on the Battlefield? Ethical Issues of Emerging Military ICTs. Proceedings of the 9th Computer Ethics Philosophy Enquiry. Milwaukee, WI. Svenmarck, P., Andersson, D., Lindahl, B., Hedström, J., & Lif, P. (2009). Tactical Reconnaissance using Groups of Partly Autonomous UGVs. Proceedings of the 8th International Conference on Engineering Psychology and Cognitive Ergonomics: HCII 2009 (pp. 326–335). San Diego, CA: Springer. [72] Department of Computer and Information Science Linköpings universitet Licentiate Theses Linköpings Studies in Science and Technology Faculty of Arts and Sciences No 17 No 28 No 29 No 48 No 52 No 60 No 71 No 72 No 73 No 74 No 104 No 108 No 111 No 113 No 118 No 126 No 127 No 139 No 140 No 146 No 150 No 165 No 166 No 174 No 177 No 181 No 184 No 187 No 189 No 196 No 197 No 203 No 212 No 230 No 237 No 250 No 253 No 260 No 283 No 298 No 318 No 319 No 326 No 328 No 333 No 335 No 348 No 352 No 371 No 378 Vojin Plavsic: Interleaved Processing of Non-Numerical Data Stored on a Cyclic Memory. (Available at: FOA, Box 1165, S-581 11 Linköping, Sweden. FOA Report B30062E) Arne Jönsson, Mikael Patel: An Interactive Flowcharting Technique for Communicating and Realizing Algorithms, 1984. Johnny Eckerland: Retargeting of an Incremental Code Generator, 1984. Henrik Nordin: On the Use of Typical Cases for Knowledge-Based Consultation and Teaching, 1985. Zebo Peng: Steps Towards the Formalization of Designing VLSI Systems, 1985. Johan Fagerström: Simulation and Evaluation of Architecture based on Asynchronous Processes, 1985. Jalal Maleki: ICONStraint, A Dependency Directed Constraint Maintenance System, 1987. Tony Larsson: On the Specification and Verification of VLSI Systems, 1986. Ola Strömfors: A Structure Editor for Documents and Programs, 1986. Christos Levcopoulos: New Results about the Approximation Behavior of the Greedy Triangulation, 1986. Shamsul I. Chowdhury: Statistical Expert Systems - a Special Application Area for Knowledge-Based Computer Methodology, 1987. Rober Bilos: Incremental Scanning and Token-Based Editing, 1987. Hans Block: SPORT-SORT Sorting Algorithms and Sport Tournaments, 1987. Ralph Rönnquist: Network and Lattice Based Approaches to the Representation of Knowledge, 1987. Mariam Kamkar, Nahid Shahmehri: Affect-Chaining in Program Flow Analysis Applied to Queries of Programs, 1987. Dan Strömberg: Transfer and Distribution of Application Programs, 1987. Kristian Sandahl: Case Studies in Knowledge Acquisition, Migration and User Acceptance of Expert Systems, 1987. Christer Bäckström: Reasoning about Interdependent Actions, 1988. Mats Wirén: On Control Strategies and Incrementality in Unification-Based Chart Parsing, 1988. Johan Hultman: A Software System for Defining and Controlling Actions in a Mechanical System, 1988. Tim Hansen: Diagnosing Faults using Knowledge about Malfunctioning Behavior, 1988. Jonas Löwgren: Supporting Design and Management of Expert System User Interfaces, 1989. Ola Petersson: On Adaptive Sorting in Sequential and Parallel Models, 1989. Yngve Larsson: Dynamic Configuration in a Distributed Environment, 1989. Peter Åberg: Design of a Multiple View Presentation and Interaction Manager, 1989. Henrik Eriksson: A Study in Domain-Oriented Tool Support for Knowledge Acquisition, 1989. Ivan Rankin: The Deep Generation of Text in Expert Critiquing Systems, 1989. Simin Nadjm-Tehrani: Contributions to the Declarative Approach to Debugging Prolog Programs, 1989. Magnus Merkel: Temporal Information in Natural Language, 1989. Ulf Nilsson: A Systematic Approach to Abstract Interpretation of Logic Programs, 1989. Staffan Bonnier: Horn Clause Logic with External Procedures: Towards a Theoretical Framework, 1989. Christer Hansson: A Prototype System for Logical Reasoning about Time and Action, 1990. Björn Fjellborg: An Approach to Extraction of Pipeline Structures for VLSI High-Level Synthesis, 1990. Patrick Doherty: A Three-Valued Approach to Non-Monotonic Reasoning, 1990. Tomas Sokolnicki: Coaching Partial Plans: An Approach to Knowledge-Based Tutoring, 1990. Lars Strömberg: Postmortem Debugging of Distributed Systems, 1990. Torbjörn Näslund: SLDFA-Resolution - Computing Answers for Negative Queries, 1990. Peter D. Holmes: Using Connectivity Graphs to Support Map-Related Reasoning, 1991. Olof Johansson: Improving Implementation of Graphical User Interfaces for Object-Oriented Knowledge- Bases, 1991. Rolf G Larsson: Aktivitetsbaserad kalkylering i ett nytt ekonomisystem, 1991. Lena Srömbäck: Studies in Extended Unification-Based Formalism for Linguistic Description: An Algorithm for Feature Structures with Disjunction and a Proposal for Flexible Systems, 1992. Mikael Pettersson: DML-A Language and System for the Generation of Efficient Compilers from Denotational Specification, 1992. Andreas Kågedal: Logic Programming with External Procedures: an Implementation, 1992. Patrick Lambrix: Aspects of Version Management of Composite Objects, 1992. Xinli Gu: Testability Analysis and Improvement in High-Level Synthesis Systems, 1992. Torbjörn Näslund: On the Role of Evaluations in Iterative Development of Managerial Support Systems, 1992. Ulf Cederling: Industrial Software Development - a Case Study, 1992. Magnus Morin: Predictable Cyclic Computations in Autonomous Systems: A Computational Model and Implementation, 1992. Mehran Noghabai: Evaluation of Strategic Investments in Information Technology, 1993. Mats Larsson: A Transformational Approach to Formal Digital System Design, 1993. No 380 No 381 No 383 No 386 No 398 No 402 No 406 No 414 No 417 No 436 No 437 No 440 FHS 3/94 FHS 4/94 No 441 No 446 No 450 No 451 No 452 No 455 FHS 5/94 No 462 No 463 No 464 No 469 No 473 No 475 No 476 No 478 FHS 7/95 No 482 No 488 No 489 No 497 No 498 No 503 FHS 8/95 FHS 9/95 No 513 No 517 No 518 No 522 No 538 No 545 No 546 FiF-a 1/96 No 549 No 550 No 557 No 558 No 561 No 563 Johan Ringström: Compiler Generation for Parallel Languages from Denotational Specifications, 1993. Michael Jansson: Propagation of Change in an Intelligent Information System, 1993. Jonni Harrius: An Architecture and a Knowledge Representation Model for Expert Critiquing Systems, 1993. Per Österling: Symbolic Modelling of the Dynamic Environments of Autonomous Agents, 1993. Johan Boye: Dependency-based Groudness Analysis of Functional Logic Programs, 1993. Lars Degerstedt: Tabulated Resolution for Well Founded Semantics, 1993. Anna Moberg: Satellitkontor - en studie av kommunikationsmönster vid arbete på distans, 1993. Peter Carlsson: Separation av företagsledning och finansiering - fallstudier av företagsledarutköp ur ett agentteoretiskt perspektiv, 1994. Camilla Sjöström: Revision och lagreglering - ett historiskt perspektiv, 1994. Cecilia Sjöberg: Voices in Design: Argumentation in Participatory Development, 1994. Lars Viklund: Contributions to a High-level Programming Environment for a Scientific Computing, 1994. Peter Loborg: Error Recovery Support in Manufacturing Control Systems, 1994. Owen Eriksson: Informationssystem med verksamhetskvalitet - utvärdering baserat på ett verksamhetsinriktat och samskapande perspektiv, 1994. Karin Pettersson: Informationssystemstrukturering, ansvarsfördelning och användarinflytande - En komparativ studie med utgångspunkt i två informationssystemstrategier, 1994. Lars Poignant: Informationsteknologi och företagsetablering - Effekter på produktivitet och region, 1994. Gustav Fahl: Object Views of Relational Data in Multidatabase Systems, 1994. Henrik Nilsson: A Declarative Approach to Debugging for Lazy Functional Languages, 1994. Jonas Lind: Creditor - Firm Relations: an Interdisciplinary Analysis, 1994. Martin Sköld: Active Rules based on Object Relational Queries - Efficient Change Monitoring Techniques, 1994. Pär Carlshamre: A Collaborative Approach to Usability Engineering: Technical Communicators and System Developers in Usability-Oriented Systems Development, 1994. Stefan Cronholm: Varför CASE-verktyg i systemutveckling? - En motiv- och konsekvensstudie avseende arbetssätt och arbetsformer, 1994. Mikael Lindvall: A Study of Traceability in Object-Oriented Systems Development, 1994. Fredrik Nilsson: Strategi och ekonomisk styrning - En studie av Sandviks förvärv av Bahco Verktyg, 1994. Hans Olsén: Collage Induction: Proving Properties of Logic Programs by Program Synthesis, 1994. Lars Karlsson: Specification and Synthesis of Plans Using the Features and Fluents Framework, 1995. Ulf Söderman: On Conceptual Modelling of Mode Switching Systems, 1995. Choong-ho Yi: Reasoning about Concurrent Actions in the Trajectory Semantics, 1995. Bo Lagerström: Successiv resultatavräkning av pågående arbeten. - Fallstudier i tre byggföretag, 1995. Peter Jonsson: Complexity of State-Variable Planning under Structural Restrictions, 1995. Anders Avdic: Arbetsintegrerad systemutveckling med kalkylprogram, 1995. Eva L Ragnemalm: Towards Student Modelling through Collaborative Dialogue with a Learning Companion, 1995. Eva Toller: Contributions to Parallel Multiparadigm Languages: Combining Object-Oriented and Rule-Based Programming, 1995. Erik Stoy: A Petri Net Based Unified Representation for Hardware/Software Co-Design, 1995. Johan Herber: Environment Support for Building Structured Mathematical Models, 1995. Stefan Svenberg: Structure-Driven Derivation of Inter-Lingual Functor-Argument Trees for Multi-Lingual Generation, 1995. Hee-Cheol Kim: Prediction and Postdiction under Uncertainty, 1995. Dan Fristedt: Metoder i användning - mot förbättring av systemutveckling genom situationell metodkunskap och metodanalys, 1995. Malin Bergvall: Systemförvaltning i praktiken - en kvalitativ studie avseende centrala begrepp, aktiviteter och ansvarsroller, 1995. Joachim Karlsson: Towards a Strategy for Software Requirements Selection, 1995. Jakob Axelsson: Schedulability-Driven Partitioning of Heterogeneous Real-Time Systems, 1995. Göran Forslund: Toward Cooperative Advice-Giving Systems: The Expert Systems Experience, 1995. Jörgen Andersson: Bilder av småföretagares ekonomistyrning, 1995. Staffan Flodin: Efficient Management of Object-Oriented Queries with Late Binding, 1996. Vadim Engelson: An Approach to Automatic Construction of Graphical User Interfaces for Applications in Scientific Computing, 1996. Magnus Werner : Multidatabase Integration using Polymorphic Queries and Views, 1996. Mikael Lind: Affärsprocessinriktad förändringsanalys - utveckling och tillämpning av synsätt och metod, 1996. Jonas Hallberg: High-Level Synthesis under Local Timing Constraints, 1996. Kristina Larsen: Förutsättningar och begränsningar för arbete på distans - erfarenheter från fyra svenska företag. 1996. Mikael Johansson: Quality Functions for Requirements Engineering Methods, 1996. Patrik Nordling: The Simulation of Rolling Bearing Dynamics on Parallel Computers, 1996. Anders Ekman: Exploration of Polygonal Environments, 1996. Niclas Andersson: Compilation of Mathematical Models to Parallel Code, 1996. No 567 No 575 No 576 No 587 No 589 No 591 No 595 No 597 No 598 No 599 No 607 No 609 FiF-a 4 FiF-a 6 No 615 No 623 No 626 No 627 No 629 No 631 No 639 No 640 No 643 No 653 FiF-a 13 No 674 No 676 No 668 No 675 FiF-a 14 No 695 No 700 FiF-a 16 No 712 No 719 No 723 No 725 No 730 No 731 No 733 No 734 FiF-a 21 FiF-a 22 No 737 No 738 FiF-a 25 No 742 No 748 No 751 No 752 No 753 Johan Jenvald: Simulation and Data Collection in Battle Training, 1996. Niclas Ohlsson: Software Quality Engineering by Early Identification of Fault-Prone Modules, 1996. Mikael Ericsson: Commenting Systems as Design Support—A Wizard-of-Oz Study, 1996. Jörgen Lindström: Chefers användning av kommunikationsteknik, 1996. Esa Falkenroth: Data Management in Control Applications - A Proposal Based on Active Database Systems, 1996. Niclas Wahllöf: A Default Extension to Description Logics and its Applications, 1996. Annika Larsson: Ekonomisk Styrning och Organisatorisk Passion - ett interaktivt perspektiv, 1997. Ling Lin: A Value-based Indexing Technique for Time Sequences, 1997. Rego Granlund: C3Fire - A Microworld Supporting Emergency Management Training, 1997. Peter Ingels: A Robust Text Processing Technique Applied to Lexical Error Recovery, 1997. Per-Arne Persson: Toward a Grounded Theory for Support of Command and Control in Military Coalitions, 1997. Jonas S Karlsson: A Scalable Data Structure for a Parallel Data Server, 1997. Carita Åbom: Videomötesteknik i olika affärssituationer - möjligheter och hinder, 1997. Tommy Wedlund: Att skapa en företagsanpassad systemutvecklingsmodell - genom rekonstruktion, värdering och vidareutveckling i T50-bolag inom ABB, 1997. Silvia Coradeschi: A Decision-Mechanism for Reactive and Coordinated Agents, 1997. Jan Ollinen: Det flexibla kontorets utveckling på Digital - Ett stöd för multiflex? 1997. David Byers: Towards Estimating Software Testability Using Static Analysis, 1997. Fredrik Eklund: Declarative Error Diagnosis of GAPLog Programs, 1997. Gunilla Ivefors: Krigsspel och Informationsteknik inför en oförutsägbar framtid, 1997. Jens-Olof Lindh: Analysing Traffic Safety from a Case-Based Reasoning Perspective, 1997 Jukka Mäki-Turja:. Smalltalk - a suitable Real-Time Language, 1997. Juha Takkinen: CAFE: Towards a Conceptual Model for Information Management in Electronic Mail, 1997. Man Lin: Formal Analysis of Reactive Rule-based Programs, 1997. Mats Gustafsson: Bringing Role-Based Access Control to Distributed Systems, 1997. Boris Karlsson: Metodanalys för förståelse och utveckling av systemutvecklingsverksamhet. Analys och värdering av systemutvecklingsmodeller och dess användning, 1997. Marcus Bjäreland: Two Aspects of Automating Logics of Action and Change - Regression and Tractability, 1998. Jan Håkegård: Hierarchical Test Architecture and Board-Level Test Controller Synthesis, 1998. Per-Ove Zetterlund: Normering av svensk redovisning - En studie av tillkomsten av Redovisningsrådets rekommendation om koncernredovisning (RR01:91), 1998. Jimmy Tjäder: Projektledaren & planen - en studie av projektledning i tre installations- och systemutvecklingsprojekt, 1998. Ulf Melin: Informationssystem vid ökad affärs- och processorientering - egenskaper, strategier och utveckling, 1998. Tim Heyer: COMPASS: Introduction of Formal Methods in Code Development and Inspection, 1998. Patrik Hägglund: Programming Languages for Computer Algebra, 1998. Marie-Therese Christiansson: Inter-organisatorisk verksamhetsutveckling - metoder som stöd vid utveckling av partnerskap och informationssystem, 1998. Christina Wennestam: Information om immateriella resurser. Investeringar i forskning och utveckling samt i personal inom skogsindustrin, 1998. Joakim Gustafsson: Extending Temporal Action Logic for Ramification and Concurrency, 1998. Henrik André-Jönsson: Indexing time-series data using text indexing methods, 1999. Erik Larsson: High-Level Testability Analysis and Enhancement Techniques, 1998. Carl-Johan Westin: Informationsförsörjning: en fråga om ansvar - aktiviteter och uppdrag i fem stora svenska organisationers operativa informationsförsörjning, 1998. Åse Jansson: Miljöhänsyn - en del i företags styrning, 1998. Thomas Padron-McCarthy: Performance-Polymorphic Declarative Queries, 1998. Anders Bäckström: Värdeskapande kreditgivning - Kreditriskhantering ur ett agentteoretiskt perspektiv, 1998. Ulf Seigerroth: Integration av förändringsmetoder - en modell för välgrundad metodintegration, 1999. Fredrik Öberg: Object-Oriented Frameworks - A New Strategy for Case Tool Development, 1998. Jonas Mellin: Predictable Event Monitoring, 1998. Joakim Eriksson: Specifying and Managing Rules in an Active Real-Time Database System, 1998. Bengt E W Andersson: Samverkande informationssystem mellan aktörer i offentliga åtaganden - En teori om aktörsarenor i samverkan om utbyte av information, 1998. Pawel Pietrzak: Static Incorrectness Diagnosis of CLP (FD), 1999. Tobias Ritzau: Real-Time Reference Counting in RT-Java, 1999. Anders Ferntoft: Elektronisk affärskommunikation - kontaktkostnader och kontaktprocesser mellan kunder och leverantörer på producentmarknader, 1999. Jo Skåmedal: Arbete på distans och arbetsformens påverkan på resor och resmönster, 1999. Johan Alvehus: Mötets metaforer. En studie av berättelser om möten, 1999. No 754 No 766 No 769 No 775 FiF-a 30 No 787 No 788 No 790 No 791 No 800 No 807 No 809 FiF-a 32 No 808 No 820 No 823 No 832 FiF-a 34 No 842 No 844 FiF-a 37 FiF-a 40 FiF-a 41 No. 854 No 863 No 881 No 882 No 890 FiF-a 47 No 894 No 906 No 917 No 916 FiF-a-49 FiF-a-51 No 919 No 915 No 931 No 933 No 938 No 942 No 956 FiF-a 58 No 964 No 973 No 958 FiF-a 61 No 985 No 982 No 989 No 990 No 991 Magnus Lindahl: Bankens villkor i låneavtal vid kreditgivning till högt belånade företagsförvärv: En studie ur ett agentteoretiskt perspektiv, 2000. Martin V. Howard: Designing dynamic visualizations of temporal data, 1999. Jesper Andersson: Towards Reactive Software Architectures, 1999. Anders Henriksson: Unique kernel diagnosis, 1999. Pär J. Ågerfalk: Pragmatization of Information Systems - A Theoretical and Methodological Outline, 1999. Charlotte Björkegren: Learning for the next project - Bearers and barriers in knowledge transfer within an organisation, 1999. Håkan Nilsson: Informationsteknik som drivkraft i granskningsprocessen - En studie av fyra revisionsbyråer, 2000. Erik Berglund: Use-Oriented Documentation in Software Development, 1999. Klas Gäre: Verksamhetsförändringar i samband med IS-införande, 1999. Anders Subotic: Software Quality Inspection, 1999. Svein Bergum: Managerial communication in telework, 2000. Flavius Gruian: Energy-Aware Design of Digital Systems, 2000. Karin Hedström: Kunskapsanvändning och kunskapsutveckling hos verksamhetskonsulter - Erfarenheter från ett FOU-samarbete, 2000. Linda Askenäs: Affärssystemet - En studie om teknikens aktiva och passiva roll i en organisation, 2000. Jean Paul Meynard: Control of industrial robots through high-level task programming, 2000. Lars Hult: Publika Gränsytor - ett designexempel, 2000. Paul Pop: Scheduling and Communication Synthesis for Distributed Real-Time Systems, 2000. Göran Hultgren: Nätverksinriktad Förändringsanalys - perspektiv och metoder som stöd för förståelse och utveckling av affärsrelationer och informationssystem, 2000. Magnus Kald: The role of management control systems in strategic business units, 2000. Mikael Cäker: Vad kostar kunden? Modeller för intern redovisning, 2000. Ewa Braf: Organisationers kunskapsverksamheter - en kritisk studie av ”knowledge management”, 2000. Henrik Lindberg: Webbaserade affärsprocesser - Möjligheter och begränsningar, 2000. Benneth Christiansson: Att komponentbasera informationssystem - Vad säger teori och praktik?, 2000. Ola Pettersson: Deliberation in a Mobile Robot, 2000. Dan Lawesson: Towards Behavioral Model Fault Isolation for Object Oriented Control Systems, 2000. Johan Moe: Execution Tracing of Large Distributed Systems, 2001. Yuxiao Zhao: XML-based Frameworks for Internet Commerce and an Implementation of B2B e-procurement, 2001. Annika Flycht-Eriksson: Domain Knowledge Management in Information-providing Dialogue systems, 2001. Per-Arne Segerkvist: Webbaserade imaginära organisationers samverkansformer: Informationssystemarkitektur och aktörssamverkan som förutsättningar för affärsprocesser, 2001. Stefan Svarén: Styrning av investeringar i divisionaliserade företag - Ett koncernperspektiv, 2001. Lin Han: Secure and Scalable E-Service Software Delivery, 2001. Emma Hansson: Optionsprogram för anställda - en studie av svenska börsföretag, 2001. Susanne Odar: IT som stöd för strategiska beslut, en studie av datorimplementerade modeller av verksamhet som stöd för beslut om anskaffning av JAS 1982, 2002. Stefan Holgersson: IT-system och filtrering av verksamhetskunskap - kvalitetsproblem vid analyser och beslutsfattande som bygger på uppgifter hämtade från polisens IT-system, 2001. Per Oscarsson: Informationssäkerhet i verksamheter - begrepp och modeller som stöd för förståelse av informationssäkerhet och dess hantering, 2001. Luis Alejandro Cortes: A Petri Net Based Modeling and Verification Technique for Real-Time Embedded Systems, 2001. Niklas Sandell: Redovisning i skuggan av en bankkris - Värdering av fastigheter. 2001. Fredrik Elg: Ett dynamiskt perspektiv på individuella skillnader av heuristisk kompetens, intelligens, mentala modeller, mål och konfidens i kontroll av mikrovärlden Moro, 2002. Peter Aronsson: Automatic Parallelization of Simulation Code from Equation Based Simulation Languages, 2002. Bourhane Kadmiry: Fuzzy Control of Unmanned Helicopter, 2002. Patrik Haslum: Prediction as a Knowledge Representation Problem: A Case Study in Model Design, 2002. Robert Sevenius: On the instruments of governance - A law & economics study of capital instruments in limited liability companies, 2002. Johan Petersson: Lokala elektroniska marknadsplatser - informationssystem för platsbundna affärer, 2002. Peter Bunus: Debugging and Structural Analysis of Declarative Equation-Based Languages, 2002. Gert Jervan: High-Level Test Generation and Built-In Self-Test Techniques for Digital Systems, 2002. Fredrika Berglund: Management Control and Strategy - a Case Study of Pharmaceutical Drug Development, 2002. Fredrik Karlsson: Meta-Method for Method Configuration - A Rational Unified Process Case, 2002. Sorin Manolache: Schedulability Analysis of Real-Time Systems with Stochastic Task Execution Times, 2002. Diana Szentiványi: Performance and Availability Trade-offs in Fault-Tolerant Middleware, 2002. Iakov Nakhimovski: Modeling and Simulation of Contacting Flexible Bodies in Multibody Systems, 2002. Levon Saldamli: PDEModelica - Towards a High-Level Language for Modeling with Partial Differential Equations, 2002. Almut Herzog: Secure Execution Environment for Java Electronic Services, 2002. No 999 No 1000 No 1001 No 988 FiF-a 62 No 1003 No 1005 No 1008 No 1010 No 1015 No 1018 No 1022 FiF-a 65 No 1024 No 1034 No 1033 FiF-a 69 No 1049 No 1052 No 1054 FiF-a 71 No 1055 No 1058 FiF-a 73 No 1079 No 1084 FiF-a 74 No 1094 No 1095 No 1099 No 1110 No 1116 FiF-a 77 No 1126 No 1127 No 1132 No 1130 No 1138 No 1149 No 1156 No 1162 No 1165 FiF-a 84 No 1166 No 1167 No 1168 FiF-a 85 No 1171 FiF-a 86 No 1172 No 1183 No 1184 No 1185 No 1190 Jon Edvardsson: Contributions to Program- and Specification-based Test Data Generation, 2002. Anders Arpteg: Adaptive Semi-structured Information Extraction, 2002. Andrzej Bednarski: A Dynamic Programming Approach to Optimal Retargetable Code Generation for Irregular Architectures, 2002. Mattias Arvola: Good to use! : Use quality of multi-user applications in the home, 2003. Lennart Ljung: Utveckling av en projektivitetsmodell - om organisationers förmåga att tillämpa projektarbetsformen, 2003. Pernilla Qvarfordt: User experience of spoken feedback in multimodal interaction, 2003. Alexander Siemers: Visualization of Dynamic Multibody Simulation With Special Reference to Contacts, 2003. Jens Gustavsson: Towards Unanticipated Runtime Software Evolution, 2003. Calin Curescu: Adaptive QoS-aware Resource Allocation for Wireless Networks, 2003. Anna Andersson: Management Information Systems in Process-oriented Healthcare Organisations, 2003. Björn Johansson: Feedforward Control in Dynamic Situations, 2003. Traian Pop: Scheduling and Optimisation of Heterogeneous Time/Event-Triggered Distributed Embedded Systems, 2003. Britt-Marie Johansson: Kundkommunikation på distans - en studie om kommunikationsmediets betydelse i affärstransaktioner, 2003. Aleksandra Tešanovic: Towards Aspectual Component-Based Real-Time System Development, 2003. Arja Vainio-Larsson: Designing for Use in a Future Context - Five Case Studies in Retrospect, 2003. Peter Nilsson: Svenska bankers redovisningsval vid reservering för befarade kreditförluster - En studie vid införandet av nya redovisningsregler, 2003. Fredrik Ericsson: Information Technology for Learning and Acquiring of Work Knowledge, 2003. Marcus Comstedt: Towards Fine-Grained Binary Composition through Link Time Weaving, 2003. Åsa Hedenskog: Increasing the Automation of Radio Network Control, 2003. Claudiu Duma: Security and Efficiency Tradeoffs in Multicast Group Key Management, 2003. Emma Eliason: Effektanalys av IT-systems handlingsutrymme, 2003. Carl Cederberg: Experiments in Indirect Fault Injection with Open Source and Industrial Software, 2003. Daniel Karlsson: Towards Formal Verification in a Component-based Reuse Methodology, 2003. Anders Hjalmarsson: Att etablera och vidmakthålla förbättringsverksamhet - behovet av koordination och interaktion vid förändring av systemutvecklingsverksamheter, 2004. Pontus Johansson: Design and Development of Recommender Dialogue Systems, 2004. Charlotte Stoltz: Calling for Call Centres - A Study of Call Centre Locations in a Swedish Rural Region, 2004. Björn Johansson: Deciding on Using Application Service Provision in SMEs, 2004. Genevieve Gorrell: Language Modelling and Error Handling in Spoken Dialogue Systems, 2004. Ulf Johansson: Rule Extraction - the Key to Accurate and Comprehensible Data Mining Models, 2004. Sonia Sangari: Computational Models of Some Communicative Head Movements, 2004. Hans Nässla: Intra-Family Information Flow and Prospects for Communication Systems, 2004. Henrik Sällberg: On the value of customer loyalty programs - A study of point programs and switching costs, 2004. Ulf Larsson: Designarbete i dialog - karaktärisering av interaktionen mellan användare och utvecklare i en systemutvecklingsprocess, 2004. Andreas Borg: Contribution to Management and Validation of Non-Functional Requirements, 2004. Per-Ola Kristensson: Large Vocabulary Shorthand Writing on Stylus Keyboard, 2004. Pär-Anders Albinsson: Interacting with Command and Control Systems: Tools for Operators and Designers, 2004. Ioan Chisalita: Safety-Oriented Communication in Mobile Networks for Vehicles, 2004. Thomas Gustafsson: Maintaining Data Consistency in Embedded Databases for Vehicular Systems, 2004. Vaida Jakoniené: A Study in Integrating Multiple Biological Data Sources, 2005. Abdil Rashid Mohamed: High-Level Techniques for Built-In Self-Test Resources Optimization, 2005. Adrian Pop: Contributions to Meta-Modeling Tools and Methods, 2005. Fidel Vascós Palacios: On the information exchange between physicians and social insurance officers in the sick leave process: an Activity Theoretical perspective, 2005. Jenny Lagsten: Verksamhetsutvecklande utvärdering i informationssystemprojekt, 2005. Emma Larsdotter Nilsson: Modeling, Simulation, and Visualization of Metabolic Pathways Using Modelica, 2005. Christina Keller: Virtual Learning Environments in higher education. A study of students’ acceptance of educational technology, 2005. Cécile Åberg: Integration of organizational workflows and the Semantic Web, 2005. Anders Forsman: Standardisering som grund för informationssamverkan och IT-tjänster - En fallstudie baserad på trafikinformationstjänsten RDS-TMC, 2005. Yu-Hsing Huang: A systemic traffic accident model, 2005. Jan Olausson: Att modellera uppdrag - grunder för förståelse av processinriktade informationssystem i transaktionsintensiva verksamheter, 2005. Petter Ahlström: Affärsstrategier för seniorbostadsmarknaden, 2005. Mathias Cöster: Beyond IT and Productivity - How Digitization Transformed the Graphic Industry, 2005. Åsa Horzella: Beyond IT and Productivity - Effects of Digitized Information Flows in Grocery Distribution, 2005. Maria Kollberg: Beyond IT and Productivity - Effects of Digitized Information Flows in the Logging Industry, 2005. David Dinka: Role and Identity - Experience of technology in professional settings, 2005. No 1191 No 1192 No 1194 No 1204 No 1206 No 1207 No 1209 No 1225 No 1228 No 1229 No 1231 No 1233 No 1244 No 1248 No 1263 FiF-a 90 No 1272 No 1277 No 1283 FiF-a 91 No 1286 No 1293 No 1302 No 1303 No 1305 No 1306 No 1307 No 1309 No 1312 No 1313 No 1317 No 1320 No 1323 No 1329 No 1331 No 1332 No 1333 No 1337 No 1339 No 1351 No 1353 No 1356 No 1359 No 1361 No 1363 No 1371 No 1373 No 1381 No 1386 No 1387 No 1392 No 1393 No 1401 No 1410 No 1421 No 1427 No 1450 No 1459 No 1466 Andreas Hansson: Increasing the Storage Capacity of Recursive Auto-associative Memory by Segmenting Data, 2005. Nicklas Bergfeldt: Towards Detached Communication for Robot Cooperation, 2005. Dennis Maciuszek: Towards Dependable Virtual Companions for Later Life, 2005. Beatrice Alenljung: Decision-making in the Requirements Engineering Process: A Human-centered Approach, 2005. Anders Larsson: System-on-Chip Test Scheduling and Test Infrastructure Design, 2005. John Wilander: Policy and Implementation Assurance for Software Security, 2005. Andreas Käll: Översättningar av en managementmodell - En studie av införandet av Balanced Scorecard i ett landsting, 2005. He Tan: Aligning and Merging Biomedical Ontologies, 2006. Artur Wilk: Descriptive Types for XML Query Language Xcerpt, 2006. Per Olof Pettersson: Sampling-based Path Planning for an Autonomous Helicopter, 2006. Kalle Burbeck: Adaptive Real-time Anomaly Detection for Safeguarding Critical Networks, 2006. Daniela Mihailescu: Implementation Methodology in Action: A Study of an Enterprise Systems Implementation Methodology, 2006. Jörgen Skågeby: Public and Non-public gifting on the Internet, 2006. Karolina Eliasson: The Use of Case-Based Reasoning in a Human-Robot Dialog System, 2006. Misook Park-Westman: Managing Competence Development Programs in a Cross-Cultural Organisation - What are the Barriers and Enablers, 2006. Amra Halilovic: Ett praktikperspektiv på hantering av mjukvarukomponenter, 2006. Raquel Flodström: A Framework for the Strategic Management of Information Technology, 2006. Viacheslav Izosimov: Scheduling and Optimization of Fault-Tolerant Embedded Systems, 2006. Håkan Hasewinkel: A Blueprint for Using Commercial Games off the Shelf in Defence Training, Education and Research Simulations, 2006. Hanna Broberg: Verksamhetsanpassade IT-stöd - Designteori och metod, 2006. Robert Kaminski: Towards an XML Document Restructuring Framework, 2006. Jiri Trnka: Prerequisites for data sharing in emergency management, 2007. Björn Hägglund: A Framework for Designing Constraint Stores, 2007. Daniel Andreasson: Slack-Time Aware Dynamic Routing Schemes for On-Chip Networks, 2007. Magnus Ingmarsson: Modelling User Tasks and Intentions for Service Discovery in Ubiquitous Computing, 2007. Gustaf Svedjemo: Ontology as Conceptual Schema when Modelling Historical Maps for Database Storage, 2007. Gianpaolo Conte: Navigation Functionalities for an Autonomous UAV Helicopter, 2007. Ola Leifler: User-Centric Critiquing in Command and Control: The DKExpert and ComPlan Approaches, 2007. Henrik Svensson: Embodied simulation as off-line representation, 2007. Zhiyuan He: System-on-Chip Test Scheduling with Defect-Probability and Temperature Considerations, 2007. Jonas Elmqvist: Components, Safety Interfaces and Compositional Analysis, 2007. Håkan Sundblad: Question Classification in Question Answering Systems, 2007. Magnus Lundqvist: Information Demand and Use: Improving Information Flow within Small-scale Business Contexts, 2007. Martin Magnusson: Deductive Planning and Composite Actions in Temporal Action Logic, 2007. Mikael Asplund: Restoring Consistency after Network Partitions, 2007. Martin Fransson: Towards Individualized Drug Dosage - General Methods and Case Studies, 2007. Karin Camara: A Visual Query Language Served by a Multi-sensor Environment, 2007. David Broman: Safety, Security, and Semantic Aspects of Equation-Based Object-Oriented Languages and Environments, 2007. Mikhail Chalabine: Invasive Interactive Parallelization, 2007. Susanna Nilsson: A Holistic Approach to Usability Evaluations of Mixed Reality Systems, 2008. Shanai Ardi: A Model and Implementation of a Security Plug-in for the Software Life Cycle, 2008. Erik Kuiper: Mobility and Routing in a Delay-tolerant Network of Unmanned Aerial Vehicles, 2008. Jana Rambusch: Situated Play, 2008. Martin Karresand: Completing the Picture - Fragments and Back Again, 2008. Per Nyblom: Dynamic Abstraction for Interleaved Task Planning and Execution, 2008. Fredrik Lantz: Terrain Object Recognition and Context Fusion for Decision Support, 2008. Martin Östlund: Assistance Plus: 3D-mediated Advice-giving on Pharmaceutical Products, 2008. Håkan Lundvall: Automatic Parallelization using Pipelining for Equation-Based Simulation Languages, 2008. Mirko Thorstensson: Using Observers for Model Based Data Collection in Distributed Tactical Operations, 2008. Bahlol Rahimi: Implementation of Health Information Systems, 2008. Maria Holmqvist: Word Alignment by Re-using Parallel Phrases, 2008. Mattias Eriksson: Integrated Software Pipelining, 2009. Annika Öhgren: Towards an Ontology Development Methodology for Small and Medium-sized Enterprises, 2009. Rickard Holsmark: Deadlock Free Routing in Mesh Networks on Chip with Regions, 2009. Sara Stymne: Compound Processing for Phrase-Based Statistical Machine Translation, 2009. Tommy Ellqvist: Supporting Scientific Collaboration through Workflows and Provenance, 2009. Fabian Segelström: Visualisations in Service Design, 2010. Min Bao: System Level Techniques for Temperature-Aware Energy Optimization, 2010. Mohammad Saifullah: Exploring Biologically Inspired Interactive Networks for Object Recognition, 2011 No 1468 No 1469 No 1476 No 1481 No 1485 FiF-a 101 No 1490 No 1503 No 1504 No 1506 No 1507 No 1509 No 1510 No 1513 No 1523 No 1550 No 1558 No 1582 No 1586 No 1588 No 1592 Qiang Liu: Dealing with Missing Mappings and Structure in a Network of Ontologies, 2011. Ruxandra Pop: Mapping Concurrent Applications to Multiprocessor Systems with Multithreaded Processors and Network on Chip-Based Interconnections, 2011. Per-Magnus Olsson: Positioning Algorithms for Surveillance Using Unmanned Aerial Vehicles, 2011. Anna Vapen: Contributions to Web Authentication for Untrusted Computers, 2011. Loove Broms: Sustainable Interactions: Studies in the Design of Energy Awareness Artefacts, 2011. Johan Blomkvist: Conceptualising Prototypes in Service Design, 2011. Håkan Warnquist: Computer-Assisted Troubleshooting for Efficient Off-board Diagnosis, 2011. Jakob Rosén: Predictable Real-Time Applications on Multiprocessor Systems-on-Chip, 2011. Usman Dastgeer: Skeleton Programming for Heterogeneous GPU-based Systems, 2011. David Landén: Complex Task Allocation for Delegation: From Theory to Practice, 2011. Kristian Stavåker: Contributions to Parallel Simulation of Equation-Based Models on Graphics Processing Units, 2011. Mariusz Wzorek: Selected Aspects of Navigation and Path Planning in Unmanned Aircraft Systems, 2011. Piotr Rudol: Increasing Autonomy of Unmanned Aircraft Systems Through the Use of Imaging Sensors, 2011. Anders Carstensen: The Evolution of the Connector View Concept: Enterprise Models for Interoperability Solutions in the Extended Enterprise, 2011. Jody Foo: Computational Terminology: Exploring Bilingual and Monolingual Term Extraction, 2012. Anders Fröberg: Models and Tools for Distributed User Interface Development, 2012. Dimitar Nikolov: Optimizing Fault Tolerance for Real-Time Systems, 2012. Dennis Andersson: Mission Experience: How to Model and Capture it to Enable Vicarious Learning, 2013. Massimiliano Raciti: Anomaly Detection and its Adaptation: Studies on Cyber-physical Systems, 2013. Banafsheh Khademhosseinieh: Towards an Approach for Efficiency Evaluation of Enterprise Modeling Methods, 2013. Martin Sjölund: Tools for Understanding, Debugging, and Simulation Performance Improvement of EquationBased Models, 2013.
© Copyright 2024