Building an experiment pipeline

Individual differences among humans can significantly affect their teaming with AI agents. CREW supports a set of cognitive tests to quantify these differences. On the other hand, human-AI teaming experiments involves many steps and stages, including playing instruction videos, cognitive testing and organizing the order of human-AI experiments. In CREW, we provide a pipeline for automating this process. The framework integrates various media files (e.g., instruction videos or pictures), inter-trial intervals, executable Python scripts, and Unity builds, ensuring a smooth and effective workflow. The pipeline requires minimal effort from researchers during proctoring, as a single click initiates the sequential execution of experiments. The pipeline allows restart from interruption points.

1) To get started, navigate into CREW/Experiment_Pipeline/, add your instruction videos and images to CREW/Experiment_Pipeline/Image_and_video.

2) Add the cognitive test game builds to CREW/Experiment_Pipeline/game_build. We provide a set of pre-built tests, summarized below.

3) Modify CREW/Experiment_Pipeline/Script/Human_subject_test.py to organize the experiment session order and timing. To launch the experiment session, simply run this python script.