Artificial Intelligence Performance Evaluation Tool

The objective of this research effort is to design, develop, and demonstrate an Artificial Intelligence (AI) Performance Evaluation Tool that streamlines, standardizes, and enhances the evaluation of AI-enabled systems across various operational conditions. This Phase I STTR effort will focus on exploring innovative methodologies to quantify the performance, reliability, and robustness of AI models, particularly in applications related to sensors and information systems within the test and evaluation missions of the Air Force Test Center (AFTC). To demonstrate early-stage feasibility, the proposed tool must be capable of evaluating AI-enabled sensor suites (e.g., EO/IR, Radar, and Radio Frequency systems) and align with AFTC’s primary simulation and digital engineering environments, such as the Advanced Framework for Simulation, Integration, and Modeling (AFSIM), Ansys Test and Evaluation Tool Kit (TETK), or the Joint Simulation Environment (JSE). By addressing critical gaps in assessing AI robustness under dynamic and contested conditions—including real-time inference latency, adversarial resilience, and data drift detection— the effort aims to establish early-stage feasibility for an adaptive framework capable of identifying key performance indicators (KPIs), assessing trustworthiness through explainability metrics, and ensuring compliance with evolving Department of Defense (DoD) AI standards, such as those delineated in the DoD Responsible AI Strategy and Implementation Pathway. This tool will be designed to integrate seamlessly with the current test environment of the 412th Test Wing and support Operationally Focused Advanced Battle Management System (ABMS) initiatives by providing actionable insights that enable faster and more informed decision-making processes. Phase I will emphasize technical innovation, feasibility validation, and initial use-case alignment to ensure the framework’s scalability and adaptability for Phase II development and field-level implementation. The alignment of the initial use case must specifically demonstrate how the proposed evaluation tool will ingest AFTC-representative sensor data to ensure the framework’s scalability and adaptability for Phase II development. The overarching goal is to accelerate the operational utility and trusted deployment of autonomous and AI-driven systems in support of Air Force modernization priorities while improving mission readiness and capability sustainment.

The full objective, description, Phase I, II and III requirements and evaluation criteria are in the official topic on DSIP.