Initial State of Gap:

We do not currently have a quantitative or otherwise objective set of measures to evaluate operator proficiency (assessment of an individual) and training effectiveness (assessment of the effectiveness of the design and conduct of the training) for future long-duration, exploration-class missions (LDEM). Current operator training relies largely on informal subjective assessment and feedback, which has not been documented outside of the operations community. While operator proficiency may be achieved, it is unclear what the relationship is to good training effectiveness. Furthermore, it is not clear if current evaluation practices will be sufficient to meet the needs of new types of tasks, different mission operations, and changing training practices emerging from future missions.

Interim Steps:

1. Formalize current state of guidelines, ground rules, and processes, used for determining, assessing and monitoring operator proficiency during training and training effectiveness, including informal, subjective methods of feedback.
   • Determine and distinguish metrics for operator proficiency versus training effectiveness.
   • Determine areas of exploration-type training (e.g., asteroid tethering) for which no or limited guidelines exist.
   • Determine the suitability of available measures to the projected needs of long-duration, exploration-class missions.
2. Solicit and conduct research to identify and validate needed objective measures of operator proficiency and training effectiveness appropriate for the projected needs of long-duration long-distance space missions.
   • Consider feasibility of existing objective measures of operator proficiency and training effectiveness that are established in other domains and their applicability to the human spaceflight domain.
   • Determine the appropriateness of different measures to different training goals (e.g., declarative vs. procedural knowledge; task vs. skill; retention vs. transfer), and to different training settings (e.g., pre-flight vs. on-board; initial vs. refresher; just-in-time vs. performance support)
3. Document recommended validated objective measures of operator proficiency and of training effectiveness for future missions. Report in appropriate publications, standards, stakeholder panels, and/or risk Evidence Reports.

Approach: 

• Conduct observations, interviews, workshops and Technical Interchange Meetings (TIMs) as appropriate to gather subject matter expert and stakeholder knowledge, arrive at consensus on important issues, and vet proposed objective measures of operator proficiency and of training effectiveness.
• Use literature reviews to refine critical inputs for objective measures of operator proficiency and of training effectiveness.
• Utilize solicited or directed projects to conduct experimental research to identify and validate objective measures of operator proficiency and of training effectiveness for the needs of long-duration, exploration-class missions.
• Emphasize cross-risk collaborations that interact with technology for exploration missions (e.g., in-flight medical technologies, human-robotic interactions, control of spacecraft). These technologies will necessarily require crew training.
• Validate in an operational context and refine objective measures produced through solicited and directed research and use of in-flight or analog data.