The last experiment session of PsyIntEC is completed on ten participants. In the experiment three reference tasks (Towers of Hanoi) were used. In two of the tasks a robot arm was working alone in parallell with the human co-worker and the other robot arm. When the human co-worker has made a move on the third task (collaboration task), one of the robot arms made a move on the collaboration task. Which arm to use was selected as the first arm that has completed a move on their own task, and therefore different robot arms collaborate with the human co-worker on each move. Each partitipant completed two conditions of the experiment; one with adaptive robot behavior and one without. Half of the participants started without adaption, and the other half with adaption. 

The adaptive robot behavior reads sensor values from the ECG, EMG, GSR and EEG sensors every 15:th second. A time window of the last 15 seconds is then analyzed. If the participant experiences negative valence (increased EMG corrugator activity) and is aroused (increased ECG, GSR and EEG activity) the robot behavior is adapted by lowering the speed of both robot arms. If the participant experiences positive valence (increased EMG zygomatic activity) the speed of the robot arms is increased. The speed change is discrete with three steps; low, medium and high. The activity of each sensor is compared to a baseline activity. The baseline is continously updated to cover for human co-workers adapting to the task and thus their baseline is changed.

The idea is that human co-workers will feel less negative stress if robot behavior is adapted to their psychphysiological responses. The data from the recordings will be analyzed during the last two weeks of the PsyIntEC experiment.

The initial results from ten participants show that adaption can have effect on emotional response. We did however see both an increase in positive and negative valence for the adaptive compared to the static version, with small or no changes in arousal.

In the PsyIntEC project experiments on 70 participants have been conducted and data from ECG, eight EEG channels, EMG zygomatic, EMG corrugator and GSR sensors have been analyzed. The main purpose of the analysis was to detect any differences in psychophysiological measurements between completing the reference task with or without robots. The reference task used in the experiments is playing the Towers of Hanoi (ToH) puzzle game alone or turn taking with a human or robot.

The psychophysiological data was used to estimate the emotional states of each participant according to the two- dimensional valence and arousal scale. Valence, describing if the emotion is negative, positive, or neutral, and arousal, describing the physiological activation state of the body ranging from low to high.

Each participant data set was checked for errors. Errors were usually caused by problems like participants that did not completely understand the rules of the ToH game, or failure of one or more sensors. This caused whole or parts of the data to be invalid. The signals were cut into one segment for each game (three games per setting and participant) and one segment for the baseline period. The first minute of the baseline period was skipped to remove initial non-relaxed activation in subjects. For the rest of the baseline period, a baseline value was calculated for each sensor. The ECG data was an exception and was further processed to extract the heart rate before the baseline was calculated.

For each data segment the mean, minimum and maximum for each sensor were calculated and the baseline (average activation of the baseline period) was subtracted. The mean, min and max values for all eight EEG sensors were then averaged to get single values for EEG activation. After this the values were averaged over all 70 subjects. The average values over all subjects were then compared between the four different game types.

The two EMG sensors showed clear differences between playing alone or with another human compared to the unpredictable robot. This suggest that playing with the unpredictable robot can produce both positive and negative feelings in subjects, the differences are however not statistically significant. It is reasonable to assume that the difference is through spikes of activity, rather than slow changes. This would mean that if the EMG sensors registered more active during a setting, much of that would be lost when averaging over the whole setting due to resting periods between the spikes. A more detailed analysis on shorter time intervals is needed to see short-term changes in emotional states.

The EEG sensors showed increased attention and cognitive load for all collaborative tasks, but no differences were shown between collaborating with a human or robots. As for the EMG sensors, the differences in EEG activity are not statistically significant.

The heart rate activity showed small but significant differences between the game types, with a decrease in activity for playing with an unpredictable robot compared to playing without robots. This is an indication of increased arousal when playing without robots, but does not say if positive or negative emotions were produced.

The GSR showed a significant increase in activity when playing with a human experimenter. This is a sign of increased arousal, maybe due to fear of performing badly. It does however not say if the feelings experienced were positive or negative.

The PsyIntEC project is currently in the last phase of a two months experiment session covering the following tasks:

• T2 Baseline human factors study, single human worker
• T3 Human factors study: pair of cooperating workers
• T5 Human-robot factors study of turn taking

After the experiment session the data analysis part will begin.

The signals from the biosensors will be used to estimate the emotional state of each participant in each task according to the two-dimensional valence and arousal scale. Valence means the type of emotion ranging from negative (e.g. disgust), neutral and positive (e.g. joy). Arousal means the strength of an emotion ranging from low to high. Each signal is usually a measure of either valence or arousal as discussed below.

If a data set is valid (it may contain errors for various of reasons, for example the participant not understanding the rules of the puzzle task) the first step is to remove the baseline. The baseline is the individual activity in sensors during a relaxed state. The effect of baseline is that increased activity with the same amplitude in two individuals can mean very different strengths of activation depending on the amplitude of the baseline in each individual. After baseline removal internal and external artifacts needs to be removed from the raw signal (for example eye blink, electro-magnetic field) and, in the case of EEG, band pass filtered to extract the relevant frequency bands (mostly alpha and beta bands).

After pre-processing we have the following signals:

• GSR is a good indicator of arousal but also on other cognitive aspects such as alertness and effort. Higher GSR levels have been correlated with high involvement in a task, most likely due to high motivation. There is also some evidence of GSR being a relatively strong indicator of negative emotions (negative valence) but no such correlations has been made for positive emotions.
• Zygomatic EMG increases with positive emotions (valence) and decreases with negative valence. It is a stronger indicator of positive than negative valence.
• Corrugator EMG increases with negative valence and decreases with positive valance. It is a stronger indicator of negative than positive valence (i.e. the opposite of zygomatic EMG).
• ECG. The heart rate will be extracted from the ECG signals by locating and counting the high positive R-waves. Increased heart rate is a good indicator of arousal, especially for negative emotions.
• Alpha band from EEG is increased during relaxing states (neutral, low arousal) and is reduced during activity. There is evidence that for negative emotions there are a relatively stronger alpha activity in the right frontal hemisphere than the left, and vice versa for positive emotions.
• Beta band from EEG is a strong indicator of alertness and concentration in a task and of mental or physical activity. It can be seen as a measurement of arousal since for example high activity and alertness in a task that produces a negative emotion can boost the strength of that emotion.
• Raw EEG can be used to see the mental load of a participant during a task.

The different signals will be averaged in a participant over each task. A task is a Towers of Hanoi game either in single human, human-human, human-robot predictable or human-robot unpredictable setting. Each participant has repeated each setting three times. The average values of the signals will be used to calculate the relative difference in emotional stress in a participant between the different settings, and the average relative differences over all participants will be used to see if there are any noticeable trends. We are especially interested in differences in emotional stress between playing with and without robots.

Experiments in the PsyIntEC project are undergoing for single human performing the reference task, human-human collaboration and human-robot collaboration.vThe Tower of Hanoi (ToH) puzzle was selected as the reference task for the study. ToH is a mathematical puzzle game consisting of three rods, and a number of disks of different sizes that can slide onto any rod. The goal of the puzzle is to start from a given configuration of the disks on the leftmost peg and to arrive in a minimal number of moves at the same configuration on the rightmost peg. Tower of Hanoi is originally a single player game. In the collaborative experiments the human-human or human-robot take turns to complete the game.

Upon arrival, participants are given general information about the experiment and a description explaining the Tower of Hanoi puzzle. Before starting the experiment, three practice moves of ToH is presented by the experimenter in order to acquaint subjects with the game. Their written consent explaining psychophysiological measurements and providing anonymity is obtained.

When entering the lab room each participant is seated in a fixed chair at the table where the game task is set up. The psychophysiological sensors measuring heart rate (ECG), electromyography (EMG), galvanic skin response (GSR) and electroencephalography (EEG) are attached, and the eye-tracker is equipped. The subjects are seated in a closed laboratory with controlled lighting and with little distraction from the outside. They are seated in a chair with fixed height and predefined position, this height and position is to be constant during all experiments in order to be able to compare the data between different experiments. One experimenter is always present in the laboratory room to monitor the experiments, but is completely hidden behind a screen and is instructed to be as quiet as possible. The presence of an experimenter is for safety reasons when using the robot arms and for monitoring that all data are recorded correctly. Surveillance of the robot arms and subject is done using live feed from a video camera. In addition the robot control software has an emergency stopping sequence implemented if it detects any defects in the program execution.

The GSR, EMG and ECG signals are recorded using active electrodes attached using conductive gel. The EEG signals are recording using a Headcap with active electrodes. We will use EEG signals from left and right frontal, central, anterior temporal and parietal regions (F3, F4, C3, C4, T3, T4, P3, P4 positions according to the 10-20 system and referenced to Cz), in total eight channels.

A Microsoft Kinect camera is used to monitor the moves made by humans and, in experiments 3 and 4, the robot arms. From the camera software a logfile is stored with move made, timestamp of the move, and if the move was optimal or not. The Kinect camera also measures the distance to the subject’s face in regular timestamps and stores the data in a logfile.

The participants are also equipped with a Jazz Novo head-mounted eye-tracker. It records the position (x and y coordinates) of both eyes and the rotation and tilt of the head (by using an accelerometer). All data from the eye-tracker are recorded and stored on the main computer.

All experiments are recorded using the digital camcorder placed in front and to the right of the participant and elevated to get a better overview of the experiment. The camcorder is equipped with a microphone that provide the opportunity to record utterances from the subject during the experiment(s) in the attempt to model the oral communication between subjects or vocalizations when the subject was alone.

Each participant will perform the four experiments. In each experiment the participant will play the tower of Hanoi game three times in total. He/she is asked to make one move at a time with only one hand operating at a given time, and that a move cannot be undone. When all four experiments are finished and questionnaires are filled in the experimenter removes the sensors and thanks the subject.

During the experiment session the operator instructs the participant about what task to perform next. Next to the participant is a laptop controlled by the operator using remote desktop. The operator shows signs on the laptop telling the participant what to do next, or which questionnaire to fill in. Each participant is awarded a cinema ticket for participating in the experiment.

The hardware/software platform in the PsyIntEC project has been undergoing some changes. Previously we have used two Viper Adept S650 robot arms using custom made electro magnetic grippers to pick up the Towers of Hanoi disks. This solution proved to be very sensitive to correct placement of the disks by the human co-worker. To solve this the grippers has been replaced by Robotiq 2-Finger Adaptive Grippers (http://robotiq.com/en/products/two-finger-robot-gripper) This solution is not suffering from the placement sensivity since we can use pins to place the disks on as in the original Towers of Hanoi game. A demonstration video of the new platform can be found at  http://www.youtube.com/watch?v=6S9QO9cJGac. 

We are currently starting up the experiment sessions. The participants will play the ToH game alone, collaboration with another human, and collaboration with a robot while biosensors are attached. The biosensors will record biophysiological data from the participant regarding emotions like stress, enjoyment, boredom, excitement, etc.