The paper initiates a comprehensive conceptual framework for an integrated simulation environment for unmanned autonomous systems (UAS) that is capable of supporting the design, analysis, testing, and evaluation from a “system of systems” perspective. The paper also investigates the current state of the art of modeling and performance assessment of UAS and their components and identifies directions for future developments. All the components of a comprehensive simulation environment focused on the testing and evaluation of UAS are identified and defined through detailed analysis of current and future required capabilities and performance. The generality and completeness of the simulation environment is ensured by including all operational domains, types of agents, external systems, missions, and interactions between components. The conceptual framework for the simulation environment is formulated with flexibility, modularity, generality, and portability as key objectives. The development of the conceptual framework for the UAS simulation reveals important aspects related to the mechanisms and interactions that determine specific UAS characteristics including complexity, adaptability, synergy, and high impact of artificial and human intelligence on system performance and effectiveness.
1. Introduction
The increasing need of avoiding the exposure of humans to “dull, dirty, or dangerous” [1] missions in conjunction with the sustained multidisciplinary technological progress in the past two decades is the main reasons behind the exponential growth rate in the development, deployment, and operation of unmanned autonomous systems (UASs). More than twenty countries have invested substantial resources towards the development and the manufacturing UASs for a wide range of applications, both in the military and civilian domains [2]. Although human personnel are part of the overall system, the UAS includes as primary component one or several unmanned vehicles (UVs) accomplishing their role within the mission with different levels of autonomy, ranging from remote control to fully automated mission completion [3, 4] including adaptation and decision making in response to changing operational conditions. Most UAS applications rely on aerial vehicles (UAVs); however, land, maritime, and space vehicles are also commonly used fulfilling primary or secondary roles within the system. The US military has currently more than 3 dozens of UAV platforms in service [1] ranging from small size to full size aircraft with different propulsion systems, including fixed wing UVs, rotary wing UVs, and airship. The Predator and the Global Hawk (with the Air Force), the Hunter (with the Army), and the Pioneer (with the Marine Corps) are just a few of the best known. More than 20 systems are based on unmanned ground vehicles (UGVs) and more than 15 on maritime unmanned vehicles (UMVs).
The requirement for increased complexity missions and high levels of autonomy determines the current and future UAS designs to become more and more systemic and intelligent. The individual agents composing UAS are numerous and sophisticated and feature different capabilities and characteristics. Additionally, UASs are called to operate within extended, uncertain, and, at times, extreme environmental conditions, possibly in complicated socio-political contexts for fulfilling missions and tasks requiring high levels of intelligence. The novelty and complexity of individual agent characteristics and interactions within and outside the UAS pose new challenges at all stages—that is, research, design, testing, and operation—of the product lifecycle. To fully benefit from the UAS systemic characteristics and synergistic capabilities, methodologies at each of the stages are necessary and need to be applied across platforms, subsystems, operational domains, environments, missions, and applications. These methodologies must be based on a “system of systems” perspective addressing issues related to the complexity, integration, communicability, and interoperability within and outside the UAS. In this context, the Department of Defense (DOD) Information Technology Standards Registry (DISR) (formerly DOD Joint Technical Architecture) [5] formulates a general framework for information technology across all DOD systems. The 4D/RCS architecture [6] establishes a reference model for the structure and organization of unmanned vehicles software to ensure system integration, effectiveness, and interoperability. In conjunction with such standards and architectures, the Joint Architecture for Unmanned Systems (JAUSs) establishes “a common language consisting of well-defined messages, enabling internal and external communication between unmanned systems” [7] providing a framework that is independent with respect to vehicle platform, mission, computer resource, technology, and operator use. This strategy adopted for the design and operation of UAS must be extended to the development of adequate simulation tools [8] that are called to support the UAS developer in determining early to which extent the desired capabilities of the entire complex system within complex operational environments are reached. This goal can be achieved if the mechanisms through which characteristics of the subsystems—from within the UAS and outside—impact the system performance are correctly identified, thoroughly investigated, and well understood. Key aspects to be addressed are interdependencies, robustness, synergy, as well as the potential for developing dedicated strategies, methodologies, metrics, and software for modeling and testing through simulation. To face all these challenges and find solutions to all these diverse, complex, and systemic problems, simulation tools that are themselves different in nature, complex, flexible, maintainable [9], and designed from a “system of systems” perspective are needed.
Typically, UAS modeling and simulation efforts have been directed towards supporting specific applications and were focused primarily on vehicle dynamics and control [10–15]. Commercially available flight simulation packages can be customized and used for limited-scope UAS analysis [16]. Larger-scale efforts focused on generality, integration, and interoperability include U.S. Army’s Multiple Unified Simulation Environment (MUSE) [17], the Naval Air Systems Command (NAVAIR) UAV Unmanned Combat Aerial Vehicle (UCAV) Distributed Simulation Infrastructure [18], the United States and United Kingdom joint distributed simulation environment, Project Churchill [1], Boeing’s Man-In-the-Loop Air-to-Air System Performance Evaluation Model (MILAASPEM) [19], the MissionLab for the design and testing of robotic configurations [20], and the Modular Semi-Automated Forces (ModSAF) to evolve into OneSAF [21, 22].
The simulation tools currently available focus on limited/isolated design issues and/or on personnel training. In general they are lacking the capabilities of addressing the high complexity and systemic nature of UAS across all types of agents, missions, operational, and environmental conditions. They also fail to provide adequate means for comprehensive testing and evaluation. In this effort, an attempt has been made for laying the foundation towards the formulation of a conceptual frame for an integrated UAS simulation environment (UAS-SE) that will provide flexible and comprehensive simulation tools for the development, testing, verification, and validation of UAS. The main components and their interactions are analyzed from a “system of systems” perspective aimed for testing and evaluation (T&E) of UAS across all operational domains to include all types of vehicles, on-board and external mission equipment, individual agent intelligence, human operators and managers, static and dynamic environment, communications and flow of information, and different mission scenarios and objectives. The availability of models and performance assessment tools through simulation is investigated and directions for future developments are identified.
In the next section, the main strategic objectives in defining the conceptual framework are outlined. The top level architecture of the simulation environment is shown in Section 3. In Sections 4 and 5, the input and output modules are described. The central components of the UAS-SE grouped in the Simulation Nucleus are described in Section 6 followed by conclusions and a list of references.
2. General Strategy for the Design of the Conceptual Framework for the UAS Simulation Environment
UASs typically consist of sets of sophisticated and different entities including several categories of human personnel. A comprehensive UAS-SE must model all these components and include specific characteristics related to system intelligence, complexity, autonomous operation, and collaborative operation. A hierarchical modular structure must be defined that is highly flexible such that it can be organized ad hoc depending on specific simulation modes and combinations of modes such as
testing and evaluation,
system behavior analysis and prediction,
personnel training,
artificial intelligence components training,
hardware in the loop simulation,
software in the loop simulation,
human in the loop simulation.
Specific simulation scenarios are associated with each of these simulation modes. The capabilities of the UAS-SE must allow the selection and the customized and flexible formulation of these scenarios to include a variety of
levels of agent and system autonomy,
levels of component models sophistication,
types of subsystem interaction,
types of missions,
levels of risk and event occurrence probability,
levels of external agents and systems intelligence,
testing and evaluation objectives,
auxiliary and connected actions for system analysis.
The primary strategic requirements for the conceptual frameworks can be summarized as follows.
Address directly the complexity and systemic character of UAS.
Ensure generality and completeness by including all operational domains, types of agents, external systems, missions, and interactions between components.
Ensure flexibility and expandability through a modular architecture and standard internal interfaces between components that allow further development, additions, and modifications.
Accommodate testing and evaluation methodologies pertinent to all domains relevant to the operation of UAS.
Manage uncertainties and dynamics of the environment and assessment of their effects on overall system performance and operation.
Manage the evaluation of system synergy, effectiveness, autonomy, and intelligence.
3. Top-Level Architecture of the UAS Simulation Environment
The top-level block diagram of a comprehensive UAS-SE is shown in Figure 1. The UAS-SE consists of a Simulation Nucleus, an Input Module, and an Output Module operated by two categories of personnel, the Simulation Human Manager and the Simulation Human Operator. For specific modes of operation of the UAS-SE, interactions with external hardware and/or human operators are possible.
UAS-SE—top-level system block diagram.
Prior to the simulation, the Simulation Human Manager (SHM) provides all the necessary data for a complete characterization of all systems and subsystems as well as the complete menu of compatible simulation options according to objective requirements and availability of data and/or routines. Note that the SHM should not be confused with other automatic/virtual “managers”, which are components of the UAS or the external systems.
The Simulation Human Operator (SHO)—also referred to as the “user”—interactively defines the simulation process by providing specific information regarding the type and number of system components, general simulation mode, simulation sophistication level, specific objectives or missions, general mission conditions, mission details, and testing and evaluation scenario. Note that the SHO is a real person using the simulation and must not be confused with several other categories of “human operators” who are part of the UAS and/or the external systems and are modeled within the UAS-SE.
4. Input Module
The Input Module (see Figure 2) includes the following four submodules:
Operator Interface,
General Systems Data Files,
Simulation Manager,
Simulation Modes.
UAS-SE—input module.
The Operator Interface consists of all the software elements allowing user-program interaction for simulation definition and initialization. User-friendly graphical user interfaces are necessary to allow the setup of the simulation scenario by selecting all the relevant parameters such as simulation mode, number and type of system components, mission type and details, level of complexity of individual agent operation, and system integration.
The General Systems Data Files contain all the preloaded information necessary for performing correctly the entire set of eligible simulation configurations for all systems and components. They are provided by the simulation human manager.
The Simulation Manager submodule prepares the basic simulation frame according to operator’s options. It identifies the simulation configuration, monitors the compatibility between operator’s interactive options and available data, processes the data, and distributes them to the modules of the Simulation Nucleus.
The Simulation Modes sub-module groups the simulation scenarios in several categories—as listed in Section 2—and manages the specific parameters and simulation configurations accordingly.
5. Output Module
The Output Module (see Figure 3) includes the following four sub-modules:
Output Manager,
Output Data,
Output Interface,
Evaluation Metrics.
UAS-SE—output module.
The Output Manager organizes the selection, processing, storage, and displaying of the data according to operator’s options.
The content of the Output Data is the result of the numerical simulation and their structure can be either predetermined or specifically requested by the operator.
The Output Interface consists of all the software elements allowing the program to present, display, and store the numerical results of the simulation according to the user’s needs and options.
The Evaluation Metrics sub-module processes the output data to determine parameters relevant to performance and effectiveness assessment based on user input and testing and evaluation scenario. The metrics and mission scenarios must be capable of capturing the systemic, adaptive, and intelligent nature of the UAS. The UAS-SE must provide the capability to assess component performance but also subsystem and system synergy, in other words, evaluate the impact of individual agent interactions and cooperation on overall performance and capabilities.
With the assessment of overall operational effectiveness as the ultimate objective, the T&E of UAS must address the following issues assumed at all architectural levels, that is, component, sub-system, and system:
performance,
availability,
operational efficiency,
operational versatility.
Performance assessment implies the evaluation of the level of agent/system significant capabilities in fulfilling its designed role. Some of these capabilities are easily expressed as physical quantities and form basic design parameters such as maximum altitude and speed for an aircraft or measurement accuracy for a sensor. Many such capabilities of the individual components at the lowest hierarchical level are embedded into the simulation through adequate modeling. It is important to determine the level of component performance throughout the operational envelope and in the presence of extreme, adverse conditions as well as after the occurrence of failures or malfunctions. The more challenging task of defining metrics, algorithms, and testing scenarios for quantitative evaluation of complex systems is currently under investigation [23, 24]. First steps for a general framework for multiagent performance evaluation have been performed [25–27]. However, the formulation of more appropriate general metrics and procedures needs to be addressed. The assessment of the level of intelligent behavior of the UAS and its impact on overall performance is a key aspect. Both human and artificial intelligence must be considered for this task. Formulating metrics for this purpose is very challenging considering that the mechanisms of cognitive processes and intelligent behavior are not well understood and adequate models are not yet available. In recent years, significant efforts have been performed towards the evaluation of human operator and UAS capabilities such as situational awareness [28, 29], perception [30], and human-robot team interaction [31, 32]. Although the social implications of general robotic applications are occasionally acknowledged [24], the social and political impact of UAS has only been discussed in the context of acceptance of airborne UAS into the unrestricted airspace [33]. However, it is expected that UAS will affect more and more people in more different ways. It should also be recognized that the human decision process is very often influenced by social, cultural, and political aspects. It is reasonable to anticipate that artificial intelligence will have to incorporate such features too. Therefore, it is necessary to consider them in future comprehensive UAS-SE for proper modeling of the decision process and for generalized performance assessment.
Availability of UAS and its components is determined by reliability, maintainability, and supportability. Reliability can be defined as the expectancy for a system to accomplish assigned tasks at a certain level of performance under a specific set of conditions [34]. This characteristic can be measured using the statistics of failure occurrence such as mean time between failure or mean number of failure per hour of operation [15, 35]. The failures are assumed to prevent the completion of the task or reduce significantly the level of performance. They can be classified as hardware failures or the result of human or artificial intelligence failures due to lack of situational awareness, wrong or risky decisions, fatigue, and so forth. Maintainability is the characteristic of a system to sustain continuous full capability operation under normal corrective, predictive, or preventive maintenance. Limited information is available in this area. Metrics for software maintainability have been developed [36]. Analytical models to assess the impact of reliability and maintainability on service quality of mass-transit systems were developed based on Markov chains [37]. The Supportability of a system can be essentially considered as its ability of sustaining continuous full capability operation with reduced logistical resources [38, 39].
Operational efficiency is determined by the amount and complexity of maintenance, logistic support, and system operations required to fulfill the UAS mission. Cost of ownership and operation is also an important aspect to be considered. Other evaluation metrics have been formulated with respect to system operation only [31].
Operational versatility incorporates system characteristics such as flexibility, adaptability, and composability. Flexibility represents the capability of a system to perform the same or very similar tasks with different components and/or architectures. Metrics for flexibility evaluation were proposed by Robby et al. [40]. Adaptability is the ability of a system to modify, redistribute, or incorporate resources to maintain a certain level of performance in the presence of modified operational requirements, changes of internal parameters (such as failures), or different environmental conditions. Metrics have been proposed for software architecture only [41, 42]. Finally, composability can be defined as the capability of a system to use building blocks for reconfiguration to achieve specific objectives [43]. This concept has recently received substantial attention in relation to the development of composable software and simulation environments. However, assessment of composability level for UAS has yet to be addressed.
It should be noted that the direct evaluation of UAS availability, operational efficiency, and operational versatility must be accompanied by an assessment of their impact on the general performance capabilities.
6. Simulation Nucleus
The Simulation Nucleus includes six major components. The first four modules form the UAS:
Individual Agent Module,
Auxiliary Systems Module,
Mission Equipment Module,
UAS Management and Decision Module,
Environment Module,
External Systems Module.
6.1. Individual Agent Module
All the components of the Individual Agent Module are assumed to be parts of the UAS. They are grouped in 5 sub-modules as presented in Figure 4.
UAS-SE—individual agent module.
(i) The Vehicle Submodule includes manned and unmanned (tele-operated, automatic, autonomous) vehicles, operating with different levels of autonomy and on-board intelligence, in various environments, in the air [2, 44, 45], on the ground [46], on sea surface [47], underwater [48], or in space [49]. An example of a simplified Individual Agent Module for an unmanned aerial vehicle is shown in Figure 5. The sub-module includes models of the aircraft general dynamics [50, 51] and subsystems such as on-board sensors [52], actuators [53], landing gear [54], engine [55], and control system [56, 57].
The individual agent module for an unmanned aerial vehicle.
(ii) The Fixed Entities Submodule includes—even if mobile—field ground stations [58] possibly with operator interfaces [59], command centers, and communication relays. Implementations of comprehensive models of these elements within UAS simulations have not yet been performed.
(iii) The Humans Submodule includes all human personnel with active and passive attributions within the UAS. Most efforts to model intelligent behavior are based on the framework provide by Rasmussen [60]. In this sub-module, aspects such as operator attention, information perception and processing [61], human-machine interface issues [62], and operator workload [63] are addressed.
(iv) The Communication Networks Submodule includes node interactions, material pathways, information flow, and processing. These aspects have been addressed mostly in social-political contexts but not related to UAS. Analyses for technical characteristics such as latencies and errors are available for specific systems such as in Kongezos and Allen [64] for wireless communications. Aspects of UAV communications in urban environment are analyzed by Poh [65].
(v) The World Model Submodule includes the representation of the external world of each individual agent as a result of sensor fusion, intelligent data processing, and information transfer with other individual agents within and outside the UAS [66, 67]. The situational awareness of the system and its components has its source in this sub-module.
6.2. Auxiliary Systems Module
The Auxiliary Systems Module includes sub-modules representing systems whose action may be considered passive in nature, or secondary, temporary, limited in authority, limited in time, replaceable/redundant, and so forth, from the point of view of the UAS operation and/or the simulation environment. This module includes the following sub-modules as presented in Figure 6.
UAS-SE—auxiliary systems module.
(i) Trajectory Planning Submodule. This sub-module hosts algorithms to compute the desired trajectory subject to an optimization process unless the trajectory is imposed due to other requirements. To determine that trajectory is an important feature of system autonomy and has received careful consideration as part of the navigation and control system development [68] resulting in a large variety of methodologies and algorithms [69, 70].
(ii) Transportation Network and Subsystems. The deployment of UAS may include an important phase requiring delivery of the UAS or some of its components to the operational area.
(iii) Maintenance and Support. General frameworks for the design, operation, and impact on performance of logistics are developed [71]. However, modeling and simulation for UAS performance assessment is not yet available.
(iv) Human operators. Modeling the human operator performing auxiliary system tasks will address similar issues as modeling the human personnel part of the UAS, possibly at a lower level of sophistication. This distinction is considered necessary to allow more detailed analysis of performance and component contributions to the complex system.
(v) Objective Mission Risk and Threats. One objective of this sub-module is to manage the risk models associated with the components of the UAS-SE and trigger events such as failures or environmental conditions changes. These models rely mostly on statistics of event occurrence given specific conditions [72]. Another objective is to process data for UAS mission risk assessment [73, 74].
(vi) Component failure, malfunction, and abnormal conditions typically have a direct important effect on system performance, capabilities, and mission success. Models for certain classes of actuator and sensor failures affecting the unmanned vehicle and on-board equipment are available [75–77]. Malfunctions of human operators, at the level of subsystem interaction, and man-machine interaction require further analysis and modeling.
6.3. Mission Equipment Module
The Mission Equipment Module has the following two main components as presented in Figure 7:
Mission Equipment Control Submodule,
Equipment Components Submodule.
UAS-SE—mission equipment module.
The equipment that serves the purpose of the individual agents or the UAS as a whole in conjunction with its mission may consist of sensors and actuators, data acquisition system, data processing system, weapon systems, robotic arms, auxiliary robots, on-board laboratory, delivery systems, or other payload. It should be noted that only those elements and characteristics directly related to the UAS operation need to be modeled and not necessarily the specific functioning of the mission equipment systems.
6.4. UAS Management and Decision Module
The UAS Management and Decision Module incorporates models of the most important features of human and artificial intelligence required for the operation of the UAS. It hosts the following processes as shown in Figure 8:
UAS Management Submodule,
Individual Agent Intelligence and Decision Making,
Human Intelligence and Decision Making,
Intelligent Adaptive Control of Agents, Groups, and Systems,
Agent Status Self-Assessment.
UAS-SE—UAS management and decision module.
The models in this sub-module focus primarily on cognitive processes [78] leading to decision making [79–81], situational awareness [29, 82], agent status self-assessment [83], risk assessment, mission evaluation, system status assessment, mission redefinition according to system failure, environment changes, and/or operator intervention [84]. The decision algorithms incorporate a certain level of authority and multicriterial strategies. Artificial intelligence techniques such as fuzzy reasoning, machine learning, swarm intelligence, and artificial neural networks can concur to perform these tasks.
An important problem of UAS operation is the air space management [1] (ASM). The main characteristics and capabilities of the real UAS system in this respect must be adequately present in the simulation environment. Using faster than real-time parallel computation, the future states of any dynamic sub-system within the simulation environment can be determined and used to model the “perfect” situational evaluation and prediction tool. Known statistics-based performance metrics of the real ASM system—such as detection rates, evaluation error, accuracy—can be used to alter the “perfect” system for a realistic simulation. However, for ASM analysis and design, more accurate simulation tools are needed. Advanced ASM development typically relies on machine learning [85, 86] and a four-dimensional-trajectory representation (position and time) [87, 88] to support a variety of conflict detection and conflict resolution algorithms [89–93].
6.5. Environment Module
The Environment Module (shown in Figure 9) is expected to capture the complexity and diversity of the operation domains that the UAS are/will be called to cover. The models required for a comprehensive implementation can be grouped in the following categories:
Atmospheric Submodule,
Terrain and Underground Submodule,
Sea Surface Submodule,
Undersea Submodule,
Extra-Atmospheric Submodule,
Urban and Indoors Submodule,
Social and Political Environment.
UAS-SE—environment module.
The first six sub-modules must include models of physical characteristics and natural and artificial obstacles. Autonomous operation implies high situational awareness and adaptability with respect to the environment including obstacle detection, identification, and evaluation followed by avoidance [94]. Designing such a system is an extremely challenging task. Simulating it may be less complex if its main characteristics can be well estimated such as: computational delay, accuracy, correct identification probability, and operational range.
Modeling the social and political environment for UAS evaluation has not yet been attempted. However, there is a growing concern with respect to issues such as impact on the environment, impact on human general activity (e.g., traffic), acceptability of “intelligent” robots, artificial intelligence decisions with moral implications, collateral damage, social perception, and acceptability of risks [95, 96].
Modeling environmental uncertainty, variability, extreme, and unusual situations is critical for a comprehensive and detailed testing and evaluation.
6.6. External Systems Module
The External Systems Module includes entities that are not part of the UAS or the environment but interact with the UAS in a significant manner for the purpose of the mission (shown in Figure 10). They are categorized as
Targets,
Hostile External Systems,
Friendly and Neutral External Systems.
UAS-SE—external systems module.
The target models must correspond to the on-board detection algorithms and provide pertinent information such as infrared or radar signature to allow the performance evaluation of the detection algorithms.
7. Conclusions
A comprehensive conceptual framework has been initiated with the purpose of developing an integrated simulation environment focused on the testing and evaluation of UAS.
The high-level structure and building blocks of the simulation environment have been formulated based on current and future required capabilities and performance.
The conceptual framework addresses directly the complexity and systemic character of UAS. It empowers the development of tools for performance and effectiveness evaluation of highly adaptive intelligent systems.
The comprehensive simulation environment features a high level of generality and covers a diversity of unmanned autonomous systems and their missions over all operational domains.
The current state of the art of modeling and performance assessment of UAS and their components has been assessed and needs and gaps identified for future investigation.
Anonymous2007Washington, DC, USAOffice of the Secretary of DefenseAnonymous2005Washington, DC, USAOffice of the Secretary of DefenseHuangH. M.2004Gaithersburg, Md, USANational Institute of Standards and TechnologyNIST Special Publication 1011HuangH.-M.chui-min.huang@nist.govPavekK.Kerry.Pavek@knox.army.milNovakB.NovakB@tacom.army.milAlbusJ.james.albus@nist.govMessinaE.elene.messina@nist.govA framework for autonomy levels for unmanned systems (ALFUS)Proceedings of the AUVSI's Unmanned Systems North America2005Baltimore, Md, USA849863Anonymous2007U.S. Department of DefenseAlbusJ.2002Gaithersburg, Md, USAU.S. Department of Commerce, National Institute of Standards and Technology, Technology AdministrationAnonymous2005Office of the Under Secretary of Defense for Acquisition, Technology and LogisticsPageE. H.SmithR.Introduction to military training simulation: a guide for discrete event simulationists1Proceedings of the Winter Simulation Conference1998Washington, DC, USA5360VerbraeckA.ValentinE. C.The use of building blocks to enhance flexibility and maintainability of simulation models and simulation librariesProceedings of the 13th European Simulation Symposium (ESS '01)2001Gent, BelgiumJodehN. M.BlueP. A.WaldronA. A.Development of small unmanned aerial vehicle research platform: modeling and simulating with flight test validation1Proceedings of AIAA Modeling and Simulation Technologies Conference2006Keystone, Colo, USA369391MuellerE. R.Hardware-in-the-loop simulation design for evaluation of unmanned aerial vehicle control systems1Proceedings of AIAA Modeling and Simulation Technologies Conference2007Hilton Head, SC, USA5305432-s2.0-19644396592PerhinschiM. G.NapolitanoM. R.CampaG.FravoliniM. L.A simulation environment for testing and research of neurally augmented fault tolerant control laws based on non-linear dynamic inversion1Proceedings of the AIAA Modeling and Simulation Technologies Conference20041471572-s2.0-0142226215RasmussenS. J.steven.rasmussen@afrl.af.milChandlerP. R.phillip.chandler@afrl.af.milMultiUAV: a multiple UAV simulation for investigation of cooperative control1Proceedings of the Winter Simulation Conference2002San Diego, Calif, USA869877YasarM.BridgesD. O.MallapragadaG.HornJ. F.A simulation test bed for coordination of unmanned rotorcraft and ground vehicles1Procedings of AIAA Modeling and Simulation Technologies Conference2006Keystone, Colo, USA4004122-s2.0-7544222948PatchettC.charles.patchett@baesystems.comSastryV.v.v.s.s.sastry@cranfield.ac.ukA preliminary model of accident causality for uninhabited autonomous air systems and its implications for their decision architecturesProceedings of the UKSim 10th International Conference on Computer Modelling and Simulation (EUROSIM/UKSim '08)2008Cambridge, UK4874922-s2.0-5014909710410.1109/UKSIM.2008.136CraigheadJ.craighea@cse.usf.eduMurphyR.murphy@cse.usf.eduBurkeJ.jlburke4@cse.usf.eduGoldiezB.bgoldiez@ist.ucf.eduA survey of commercial & open source unmanned vehicle simulatorsProceedings of IEEE International Conference on Robotics and Automation2007Rome, Italy8528572-s2.0-3634901806310.1109/ROBOT.2007.363092ClarkI.MiksaR.ChildsD.MeadK.The benefits of simulation to support training1999321TwesmeJ.CorzineA.Naval air systems command (NAVAIR) unmanned aerial vehicle (UAV) unmanned combat aerial vehicle (UCAV) distributed simulation infrastructureProceedings of the 2nd AIAA “Unmanned Unlimited” Systems, Technologies, and Operations—Aerospace, Land, and Sea Conference and Workshop and Exhibit2003San Diego, Calif, USALawsonK. E.ButlerC. T.Overview of Man-in-the-Loop Air-to-Air System Performance Evaluation Model (MIL-AASPEM) II, D658-10485-1The Boeing Company, 1995MackenzieD. C.doug@cc.gatech.eduArkinR. C.arkin@cc.gatech.eduCameronJ. M.jonathan.m.cameron@jpl.nasa.govMultiagent Mission Specification and Execution19974129522-s2.0-0039474434SardellaJ. M.HighD. L.Integration of fielded army aviation simulators with ModSAF: the eighth army training solutionProceedings of Interservice/Industry Training, Simulation and Education ConferenceNovember 2000Orlando, Fla, USADrewesP.Simulation based approach for unmanned system command and controlProceedings of Florida Conference on Recent Advances in Robotics2006Miami, Fla, USAO'DayS.SteinbergM.YglesiasC.Metrics for intelligent autonomyProceedings of Performance Metrics for Intelligent Systems Workshop (PerMIS '04)2004Gaithersburg, Md, USASteinfeldA.steinfeld@cmu.eduFongT.KaberD.Common metrics for human-robot interaction2006Proceedings of the 1st ACM Conference on Human-Robot Interaction2006Salt Lake City, Utah, USA33402-s2.0-3142750705AlbusJ.Metrics and performance measures for intelligent unmanned ground vehiclesProceedongs of Performance Metrics for Intelligent Systems Workshop (PerMIS '02)2002Gaithersburg, Md, USAHarbourJ. L.BruemmerD. J.FewD. A.Measuring unmanned vehicle system performance: challenges and opportunitiesAUVSI Unmanned Systems North America2006Orlando, Fla, USADimouC.cdimou@issel.ee.auth.grSymeonidisA. L.asymeon@ee.auth.grMitkasP. A.mitkas@eng.auth.grTowards a generic methodology for evaluating MAS performanceProceedings of the International Conference on Integration of Knowledge Intensive Multi-Agent Systems (KIMAS '07)2007Waltham, Mass, USA1741792-s2.0-3454879080110.1109/KIMAS.2007.369805MulgandS. S.HarperK. A.ZachariasG. L.MenkeT.SAMPLE: situation awareness model for pilot-in-the-loop evaluationProceedings of the 9th Conference on Computer Generated Forces and Behavioral Representation2000Orlando, Fla, USAAdamsJ. A.Unmanned vehicle situation awareness: a path forwardProceedings of the Human Systems Integration Symposium2007Annapolis, Md, USATouchtonB.btouch@ufl.eduGalluzzoT.galluzzt@ufl.eduKentD.kentd@ufl.eduCraneC.ccrane@ufl.eduPerception and planning architecture for autonomous ground vehicles2006391240472-s2.0-1044424176110.1109/MC.2006.436FreedyA.FreedyE.DeVisserJ.A complete simulation environment for measuring and assessing human-robot team performanceProceedings of IEEE Safety, Security, and Rescue Robotics Conference, and Performance Metrics for Intelligent Systems Workshop2006Gaithersburg, Md, USAPinaP. E.CummingsM. L.CrandallJ. W.Della PennaM.Identifying generalizable metric classes to evaluate human-robot teamsProceedings of the Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction2008Amsterdam, The NederlandsAnonymous2008London, UKCivil Aviation Authority, Directorate of Airspace Policy CAP-722Anonymous2003Washington, DC, USAU.S. Department of Defense, Office of the Secretary of DefenseAnonymous1982Washington, DC, USADepartment of DefensePeercyD. E.A software maintainability evaluation methodology1981743433512-s2.0-0019595379DersinP.DurandJ.Mass-transit system service quality: tradeoff analysis on reliability, maintainability and logisticsProceedings of the Annual Reliability and Maintainability Symposium1995Washington, DC, USA515528MortinD. E.Analysis of logistic supportability for complex systemsProceedings of the Annual Reliability and Maintainability Symposium1990Los Angeles, Calif, USA250253JohnC. G.SchultzR. E.Simulation modeling for military logistics and supportability studiesProceedings of the Winter Simulation ConferenceDecember 1991Phoenix, Ariz, USA485490Robbyrobby@ksu.eduDeLoachS. A.sdeloach@ksu.eduKolesnikovV. A.valkov@ksu.eduUsing design metrics for predicting system flexibility3922Proceedings of the 9th International Conference on Fundamental Approaches to Software Engineering (FASE '06)2006Vienna, Austria184198Lecture Notes in Computer Science2-s2.0-003631926710.1007/11693017_15ChungL.SubramanianN.Process-oriented metrics for software architecture adaptabilityProceedings of the 5th IEEE International Conference on Requirements Engineering2001Toronto, Canada3103112-s2.0-0001551129RotaruO. P.Octavian.Rotaru@Amdocs.comDobreM.Marian.Dobre@Amdocs.comReusability metrics for software componentsProceedings of the 3rd ACS/IEEE International Conference on Computer Systems and Applications2005Cairo, Egypt81882-s2.0-002844607810.1109/AICCSA.2005.1387023PettyM. D.Two aspects of composability: lexicon and theoryProceedings of Defense Modeling and Simulation Office Workshop on Composable Modeling and Simulation2002Monterey, Calif, USAShimH.KooT. J.HoffmannF.SastryS.A comprehensive study of control design for an autonomous helicopter4Proceedings of the IEEE Conference on Decision and Control (CDC '98)1998Tampa, Fla, USA36533658SukhatmeG. S.MontgomeryJ. F.Heterogeneous robot group control and applicationsProceedings of the International Conference of the Association for Unmanned Vehicle Systems1999Baltimore, Md, USADurrant-WhyteH.A critical review of the state-of-the-art in autonomous land vehicle systems and technology2001SAND2001-3685Sandia National LaboratoriesLeonessaA.aleo@seatech.fau.eduMandelloJ.jmandell@seatech.fau.eduMorelY.ymorel@seatech.fau.eduVidalM.mvidal@seatech.fau.eduDesign of a small, multi-purpose, autonomous surface vessel1Proceedings of IEEE Oceans Conference2003San Diego, Calif, USA544550Anonymous2004Washington, DC, USAU.S. Department of the NavyStroupeA. W.SinghS.SimmonsR.Technology for autonomous space systems2002CMU-RI-TR-00-02Pittsburgh, Pa, USAThe Robotics Institute, Carnegie Mellon UniversityZipfelP. H.2000Reston, Va, USAAIAAAIAA Education SeriesDreierM. E.2007Reston, Va, USAAIAAAIAA Education SeriesGargD.KumarM.Sensor modeling and multi-sensor data fusion2005W911NF-01-10434Army Research OfficeCaponeP.Actuator modeling and aircraft related performance: selection of the appropriate actuation systemProceedings of the AIAA Modeling and Simulation Technologies Conference2000Denver, Colo, USAPritchardJ.Overview of landing gear dynamics20013811301372-s2.0-0031211959LichtsinderM.LevyY.levy@aerodyne.technion.ac.ilJet engine model for control and real-time simulations200612847457532-s2.0-3384570178910.1115/1.1915391SteinbergM. L.A comparison of intelligent, adaptive, and nonlinear flight control lawsProceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit1999Portland Ore, USAAIAA 99-4044WiseK. A.Guidance and control for military systems: future challenges5Proceedings of the AIAA Guidance, Navigation and Control Conference and Exhibit2007Hilton Head, SC, USA53825389De CrescenzioF.francesca.decrescenzio@unibo.itMirandaG.giovanni.miranda@unibo.itPersianiF.franco.persiani@unibo.itBombardiT.tiziano.bombardi@unibo.itAdvanced interface for UAV (Unmanned Aerial Vehicle) ground control station1Proceedings of the AIAA Modeling and Simulation Technologies Conference and Exhibit2007Hilton Head, SC, USA4864942-s2.0-33646396233CummingsM. L.MeyersK.ScottS. D.Modified cooper harper evaluation tool for unmanned vehicle displaysProceedings of the Unmanned Vehicle Systems Canada Conference2006Montebello, CanadaRasmussenJ.RasmussenJ.RouseW. B.Models of mental strategies in process plant diagnosis1981New York, NY, USAPlenum PressMetzgerU.ParasuramanR.Effects of automated conflict cueing and traffic density on air traffic controller performance and visual attention in a datalink environment200616343362BarbatoG.FeitshansG.WilliamsR.HughesT.Operator vehicle interface laboratory: unmanned combat air vehicle controls & displays for suppression of enemy air defencesProceedings of the 12th International Symposium on Aviation Psychology2003Dayton, Ohio, USAKellerJ.Human performance modeling for discrete-event simulation: workload1Proceedings of the Winter Simulation Conference2002San Diego, Calif, USA157162KongezosV. K.AllenC. R.Wireless communication between A.G.V.'s (Autonomous Guided Vehicle) and the industrial network C.A.N. (Controller Area Network)1Proceedings of IEEE International Conference on Robotics and Automation2002Washington, DC, USA434437PohS. C. T.2004Monterey, Calif, USANaval Postgraduate SchoolEvansC. P.III2004University of FloridaTouchtonR.KentD.GalluzzoT.Planning and modeling extensions to the joint architecture for unmanned systems (JAUS) for application to unmanned ground vehicles5804Unmanned Ground Vehicle Technology VIIMarch 2005Orlando Fla, USA146155Proceedings of SPIE2-s2.0-2754448569010.1117/12.603631VachtsevanosG.giv@ee.gatech.eduTangL.DrozeskiG.GutierrezL.From mission planning to flight control of unmanned aerial vehicles: strategies and implementation tools20052911011152-s2.0-003536996510.1016/j.arcontrol.2004.11.002PachterM.ChandlerP.Challengers of autonomous control1998184BortoffS. A.Path planning for UAVs1Proceedings of the American Control ConferenceJune 2000Chicago, Ill, USA364368GanslerJ. S.LucyshynW.2006University of Maryland and Naval Postgraduate SchoolUMD-LM-06-040HaimesY.20042ndNew York, NY, USAJohn Wiley & SonsWeibelR. E.HansmanR. J.Jr.An integrated approach to evaluating risk mitigation measures for UAV operational concepts in the NAS1Proceedings of InfoTech at Aerospace: Advancing Contemporary Aerospace Technologies and Their IntegrationSeptember 2005Arlington, Va, USA5095192-s2.0-2442472242ReeceC.WalkerR.FultonN.CampbellD.A casualty risk analysis for unmanned aerial system (UAS) operations over inhabited areasProceedings of the 12th Australian International Aerospace Congress, and the 2nd Australasian Unmanned Air Vehicles Conference2007Melbourne, AustraliaBoškovićJ. D.LiS.-M.MehraR. K.On-line failure detection and identification (FDI) and adaptive reconfigurable control (ARC) in aerospace applications4Proceedings of the American Control Conference2001262526262-s2.0-0031075855PerhinschiM. G.CampaG.NapolitanoM. R.LandoM.MassottiL.FravoliniM. L.Modeling and simulation of a fault tolerant control system2006261110BaconB. J.GregoryI. M.General equations of motion for a damaged asymmetric aircraft1Proceedings of the AIAA Atmospheric Flight Mechanics Conference and Exhibit2007Hilton Head, SC, USA6375JhaK.WrayR.LebiereC.Towards a complete, multi-level cognitive architectureProceedings of the International Conference for Cognitive Modeling2007Ann Arbor, Mich, USAHansonM. L.HarperK. A.An intelligent agent for supervisory control of teams of unihabited combat air vehicles (UCAVs)Proceedings of the Unmanned Systems Conference2000Orlando, Fla, USATouchtonR.2006University of FloridaNarayanP.WuP.CampbellD.WalkerR.An intelligent control architecture for unmanned aerial systems (Uas) in the national airspace system (Nas)Proceedings of the 2nd Australasian Unmanned Air Vehicle Systems Conference2007Melbourne, AustraliaNehmeC. E.CrandallJ. W.CummingM. L.Using discrete-event simulation to model situational awareness of unmanned-vehicle operatorsProceedings of the Virginia Modeling, Analysis and Simulation Center Capstone Conference2008Norfolk, Va, USAReichardK. M.kmr5@psu.eduCrowE. C.Intelligent self-situational awareness for unmanned and robotic platformsProceedings of AUVSI Unmanned Systems North America2005Baltimore, Md, USA122112352-s2.0-0030173862PedersonK.LethinR.SpringerJ.ManoharR.MelhemR.Enabling cognitive architectures for UAV mission planningProceedings of the 10th Annual Workshop High Performance Embedded Computing2006Burlington, Mass, USAZhangX.YoonS.DiBonaP.An ensemble learning and problem solving architecture for airspace managementProceedings of the 21st Innovative Applications of Artificial Intelligence Conference20092032102-s2.0-24944537843YoonS.Sungwook.Yoon@asu.eduKambhampatiS.rao@asu.eduHierarchical strategy learning with hybrid representationsProceedings of the AAAI Workshop on Acquiring Planning Knowledge via Demonstration200752562-s2.0-0027574520MazzucchelliL.MonteleoneA.A 4D immersive virtual reality system for air traffic controlProceedings of the 7th EUROCONTROL Innovative Research Workshop & ExhibitionDecember 2008EUROCONTROL Experimental CentrePorrettaM.m.porretta@imperial.ac.ukDupuyM.-D.SchusterW.MajumdarA.OchiengW.Performance evaluation of a novel 4D trajectory prediction model for civil aircraft20086133934202-s2.0-003110313210.1017/S0373463308004761KoseckaJ.TomlinC.PappasG.SastryS.Generation of conflict resolution maneuvers for air traffic management3Proceedings of IEEE International Conference on Intelligent Robots and Systems1997Grenoble, France159816032-s2.0-0031362628KucharJ. K.jkkuchar@mit.eduYangL. C.A review of conflict detection and resolution modeling methods2000141791892-s2.0-0347571380DowekG.MuñozC.Conflict detection and resolution for 1,2,... N aircraft1Proceedings of the 7th AIAA Aviation Technology, Integration, and Operations Conference2007Belfast, Northern Ireland4034152-s2.0-37249028071FarleyT. C.ErzbergerH.Fast-time simulation evaluation of a conflict resolution algorithm under high air traffic demandProceedings of the 7th USA/Europe ATM R&D SeminarJuly 2007Barcelona, SpainSharmaR.rajiet10@rediffmail.comGhoseD.dghose@aero.iisc.ernet.inSwarm intelligence based collision avoidance between realistically modelled UAV clustersProceedings of the American Control ConferenceJuly 2007New York, NY, USA389238972-s2.0-864424899310.1109/ACC.2007.4282177KellyA.alonzo@ri.cmu.eduStentzA.AmidiO.Toward reliable off road autonomous vehicles operating in challenging environments2006255-64494832-s2.0-003431582710.1177/0278364906065543DeGarmoM. T.2004McLean, Va, USAFederal Aviation Administration, MITRE Center for Advanced Aviation System DevelopmentSleemanR.CoxA.ColledgeJ.Systems approach to unmanned air vehicle development and certificationImprovements in System Safety: Proceedings of the 16th Safety-Critical Systems Symposium2008Bristol, UK