expected_r() is especially useful at the design stage of a
study, where sample size planning typically proceeds from an assumed
value of the population correlation \(\rho\). Because the value of
\(r\) a researcher should expect to observe on average is
smaller in absolute value than the assumed \(\rho\), plugging
\(\rho\) directly into sampling-distribution machinery (precision of
\(r\), width of a confidence interval, power of a test of
\(H_0\!: \rho = 0\)) systematically over-promises on the realized
precision or power. Substituting expected_r(rho, n) for the bare
\(\rho\) in such planning calculations corrects the leading-order
over-promise. See Examples for a design-stage walk-through.
The exact formula. Under bivariate normality, if
\(r\) is the sample Pearson correlation in a sample of size \(n\),
$$\mathrm{E}[r \mid \rho, n] \;=\;
\rho \,\cdot\,
{}_2F_1\!\left(\tfrac{1}{2},\, \tfrac{1}{2};\, \tfrac{n+1}{2};\, \rho^2 \right)
\,\cdot\,
\frac{\Gamma\!\left(\tfrac{n}{2}\right)^2}
{\Gamma\!\left(\tfrac{n-1}{2}\right)\,\Gamma\!\left(\tfrac{n+1}{2}\right)}.$$
Here \({}_2F_1(a, b; c; z) = \sum_{k=0}^{\infty} \frac{(a)_k (b)_k}{(c)_k\, k!}\, z^k\)
is the Gauss hypergeometric function with Pochhammer symbols
\((x)_k = x(x+1)\cdots(x+k-1)\) (Hotelling, 1953, Section 7 for the
moments of \(r\), Section 3, equation 25, for the exact density;
Olkin & Pratt, 1958, equation 3.2, for this closed form).
For \(\rho^2 < 1\) the series converges absolutely; this implementation
sums by a numerically stable forward recurrence and stops when the
relative contribution of the next term falls below a tolerance.
Tuning the series convergence (rarely needed). Two internal
tuning constants control the \({}_2F_1\) series summation:
DMAR.expected_r.tol, relative tolerance for stopping
the series (default 1e-15).
DMAR.expected_r.max_iter, maximum number of series
terms before issuing a non-convergence warning (default
5000L).
Both have sensible defaults; advanced users who need different values
(e.g., for very near-boundary \(|\rho|\) where the series converges
slowly) can set them via options(), for example
options(DMAR.expected_r.tol = 1e-12). They are deliberately
hidden from the function signature so as not to clutter the everyday
user's view of the call.
Why the bias is present even though \(s^2\) is unbiased for
\(\sigma^2\). The downward bias arises because \(r\) is a nonlinear
function of unbiased sample moments. By Jensen's inequality and the
concavity of the square root in the denominator of \(r\), the expectation
of the ratio is not the ratio of the expectations. The bias is largest
when \(n\) is small or \(|\rho|\) is moderate; as \(n \to \infty\),
\(\mathrm{E}[r] \to \rho\).
Sign and magnitude. The bias \(\rho - \mathrm{E}[r]\) has the
same sign as \(\rho\) and is approximately
\(\rho(1 - \rho^2)/[2(n - 1)]\) to leading order (Fisher, 1915;
Hotelling, 1953; Ghosh, 1966); the exact formula above incorporates all
higher-order corrections. For \(\rho = 0.5\), \(n = 10\), the bias
is about \(+0.021\); for \(\rho = 0.5\), \(n = 30\), about
\(+0.0065\).
Olkin-Pratt (1958) unbiased estimator of \(\rho\). The
companion to this expected-value calculation is the Olkin-Pratt
unbiased estimator of \(\rho\) given an observed \(r\):
$$\tilde{\rho}_{\text{OP}}(r, n) \;=\;
r \,\cdot\, {}_2F_1\!\left(\tfrac{1}{2},\, \tfrac{1}{2};\, \tfrac{n-2}{2};\, 1 - r^2 \right).$$
Olkin & Pratt (1958) prove
\(\mathrm{E}[\tilde{\rho}_{\text{OP}}(r, n) \mid \rho, n] = \rho\)
exactly, for every \(\rho \in (-1, 1)\) and \(n \ge 4\). The
commonly quoted first-order approximation
\(\tilde{\rho} \approx r\,[1 + (1 - r^2)/(2(n - 3))]\) (Olkin, 1967)
is the truncation of the OP series at \(k = 1\); the full series is
what makes the estimator exactly unbiased.
Although the unbiased estimator is straightforward to apply, it is
rarely used because in most downstream uses (significance testing,
Fisher's \(Z\) confidence intervals, structural-equation models) the
bias is small relative to other sources of uncertainty. Where it does
matter, meta-analyses with many small samples, reliability /
validity coefficients estimated from short calibration samples,
design-stage estimates feeding into AIPE sample size machinery
the correction is well worth applying.
Connection to Fisher's Z transform. The variance-stabilizing
transform \(z = \tanh^{-1}(r)\), proposed in passing in Fisher (1915,
p. 521) and developed in Fisher (1921), has approximate variance
\(1/(n-3)\) regardless of \(\rho\), but \(\mathrm{E}[z]\) also
carries a small-sample bias of order \(1/n\) (Hotelling, 1953,
Section 8). Hotelling (1953, Sections 9--10) gives bias-adjusted and
variance-stabilized refinements of \(Z\), e.g.
\(z - (3z + r)/(4n)\).