binaryRL (version 0.9.0)

func_epsilon: Function: Exploration Strategy

Description

The exploration strategy parameters are threshold, epsilon, and lambda.

• Epsilon-first strategy: Used when only threshold is set. Subjects choose randomly for trials less than threshold and by value for trials greater than `threshold.

• Epsilon-greedy strategy: Used if threshold is default (1) and epsilon is set. Subjects explore with probability epsilon throughout the experiment.

• Epsilon-decreasing strategy: Used if threshold is default (1), and lambda is set. In this strategy, the probability of random choice (exploration) decreases as trials increase. The parameter lambda controls the rate at which this probability declines with each trial.

Usage

func_epsilon(
  i,
  L_freq,
  R_freq,
  L_pick,
  R_pick,
  L_value,
  R_value,
  var1 = NA,
  var2 = NA,
  threshold = 1,
  epsilon = NA,
  lambda = NA,
  alpha,
  beta
)

Value

A numeric value, either 0 or 1. 0 indicates no exploration (choice based on value), and 1 indicates exploration (random choice) for that trial.

Arguments

i

The current row number.

L_freq

The frequency of left option appearance

R_freq

The frequency of right option appearance

L_pick

The number of times left option was picked

R_pick

The number of times left option was picked

L_value

The value of the left option

R_value

The value of the right option

var1

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

var2

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

threshold

[integer] Controls the initial exploration phase in the epsilon-first strategy. This is the number of early trials where the subject makes purely random choices, as they haven't yet learned the options' values. For example, threshold = 20 means random choices for the first 20 trials. For epsilon-greedy or epsilon-decreasing strategies, `threshold` should be kept at its default value.

$$ P(x) = \begin{cases} \text{trial} \le \text{threshold}, & x=1 \text{ (random choosing)} \\ \text{trial} > \text{threshold}, & x=0 \text{ (value-based choosing)} \end{cases} $$

default: threshold = 1

epsilon-first: threshold = 20, epsilon = NA, lambda = NA

epsilon

[numeric] A parameter used in the epsilon-greedy exploration strategy. It defines the probability of making a completely random choice, as opposed to choosing based on the relative values of the left and right options. For example, if `epsilon = 0.1`, the subject has a 10 choice and a 90 relevant when `threshold` is at its default value (1) and `lambda` is not set.

$$P(x) = \begin{cases} \epsilon, & x=1 \text{ (random choosing)} \\ 1-\epsilon, & x=0 \text{ (value-based choosing)} \end{cases}$$

epsilon-greedy: threshold = 1, epsilon = 0.1, lambda = NA

lambda

[vector] A numeric value that controls the decay rate of exploration probability in the epsilon-decreasing strategy. A higher `lambda` value means the probability of random choice will decrease more rapidly as the number of trials increases.

$$ P(x) = \begin{cases} \frac{1}{1+\lambda \cdot trial}, & x=1 \text{ (random choosing)} \\ \frac{\lambda \cdot trial}{1+\lambda \cdot trial}, & x=0 \text{ (value-based choosing)} \end{cases} $$

epsilon-decreasing threshold = 1, epsilon = NA, lambda = 0.5

alpha

[vector] Extra parameters that may be used in functions.

beta

[vector] Extra parameters that may be used in functions.

Examples

if (FALSE) {
func_epsilon <- function(
  # Trial number
  i,
  # Number of times this option has appeared
  L_freq,
  R_freq,
  # Number of times this option has been chosen
  L_pick,
  R_pick,
  # Current value of this option
  L_value,
  R_value,
  # Extra variables
  var1 = NA,
  var2 = NA,
  
  # Free Parameters
  threshold = 1,
  epsilon = NA,
  lambda = NA,
  # Extra parameters
  alpha,
  beta
){
  # Epsilon-First: random choosing before a certain trial number
  if (i <= threshold) {
    try <- 1
  } else if (i > threshold & is.na(epsilon) & is.na(lambda)) {
    try <- 0
  # Epsilon-Greedy: random choosing throughout the experiment with probability epsilon
  } else if (i > threshold & !(is.na(epsilon)) & is.na(lambda)){
    try <- sample(
      c(1, 0),
      prob = c(epsilon, 1 - epsilon),
      size = 1
    )
  # Epsilon-Decreasing: probability of random choosing decreases as trials increase
  } else if (i > threshold & is.na(epsilon) & !(is.na(lambda))) {
    try <- sample(
      c(1, 0),
      prob = c(
        1 / (1 + lambda * i),
        lambda * i / (1 + lambda * i)
      ),
      size = 1
    )
  }
  else {
    try <- "ERROR"
  }

  return(try)
}
}