Relevant variables

The columns used in this example are:

• UserID, which identifies participants
• block_type, in order to select only assessment blocks
• trial_type, in order to compare approach and avoid trials
• cat, in order to compare test and control stimuli
• attempt, in order to select only the first attempt at each trial
• response, in order to select only correct first responses
• rt, in order to calculate medians for avoid_test, approach_test, avoid_control, and approach_control

Preprocessing

Only select trials from assessment blocks, using the first response of the first attempt of each trial.

ds_aat <- subset(ds_aat, block_type %in% c("assess1", "assess2"))
ds_aat <- subset(ds_aat, attempt == 0)

Defining the sets function

The sets function receives data from a single participant and returns a list of datasets for each condition. In this case, we will generate four data frames, containing the trials from: avoid test, approach test, avoid control, and approach control.

aat_fn_sets <- function (ds) {
  return (list(
    avoid_test = subset(ds, trial_type == "avoid" & cat == "test"),
    approach_test = subset(ds, trial_type == "approach" & cat == "test"),
    avoid_control = subset(ds, trial_type == "avoid" & cat == "control"),
    approach_control = subset(ds, trial_type == "approach" & cat == "control")
  ))
}

Defining the score function

The score function receives these four data frames from a single participant and for each:

1. selects only correct responses
2. calculates the median RT of remaining responses

Finally, it returns the double difference between the four median RTs.

aat_fn_score <- function (sets) {
  median_avoid_test <- median(subset(sets$avoid_test, response == 1)$rt)
  median_approach_test <- median(subset(sets$approach_test, response == 1)$rt)
  median_avoid_control <- median(subset(sets$avoid_control, response == 1)$rt)
  median_approach_control <- median(subset(sets$approach_control, response == 1)$rt)
  return ((median_avoid_test - median_approach_test) - (median_avoid_control - median_approach_control))
}

Calculating a score without the splithalfr

By combining the sets and score functions, a score for a single participant can be calculated. For instance, the score of UserID 1 can be calculated via the statement below.

aat_fn_score(aat_fn_sets(subset(ds_aat, UserID == 1)))

Calculating scores with the spltihalfr

To calculate scores for each participant, call sh_apply with four arguments:

1. the dataset
2. the column that identifies participants in the dataset
3. the sets function
4. the score function

The sh_apply function will return a data frame with one row per participant, and two columns: one that identifies participants (“UserID” in this example) and a column “score”, that contains the output of the score function.

aat_scores <- sh_apply(ds_aat, "UserID", aat_fn_sets, aat_fn_score)

Checking scores

It is recommended to check your scoring method by calculating the score of a representative participant via a different approach. For splithalfr tests, the author has done so via Excel.

Calculating split scores

To calculate split-half scores for each participant, call sh_apply with an additional split_count argument, which specifies how many splits should be calculated. For each participant and split, the splithalfr will randomly divide the dataset of each element of sets into two halves that differ at most by one in size. When called with a split_count argument that is higher than zero, sh_apply returns a data frame with the following columns:

• UserID, which identifies participants
• split, which counts splits
• score_1 and score_2, which are the scores calculated for each of the split datasets

Note that in this example UserID 294 had a relatively low number of correct responses; it was sufficient to yield an AAT score but may yield missing data in some of the random splits.

aat_splits <- sh_apply(ds_aat, "UserID", aat_fn_sets, aat_fn_score, 1000)

Estimating reliability averaged over splits

Next, the output of sh_apply can be analyzed in order to estimate reliability. By default, functions are provided that automatically calculate mean Spearman-Brown (mean_sb_by_split) and Flanagan-Rulon (mean_fr_by_split) coefficients. If any missing values were encountered in the data provided to these functions, they give a warning, and then pairwise remove the missing data before calculating reliability.

# Spearman-Brown
mean_sb_by_split(aat_splits)