# Reliability

The term

*reliability* generally refers to the consistency of a measure. The statistical approach to estimating reliability varies depending upon the purpose of the measure. In order to measure the reliability of a household scale, for example, you might weigh and re-weigh objects at multiple points in time. A reliable scale will provide the same results (within some acceptable margin of error) at each point of measurement. This is referred to as a test re-test approach to estimating reliability (using a

Pearson product-moment correlation coefficient). The test re-test approach is also commonly used to evaluate the reliability of questionnaires -- especially when the construct being measured is assumed to be stable over time (e.g., IQ scores).

The construct being measured by a treatment outcome questionnaire, however, is expected to change over time (e.g.,

global distress). For this reason, an alternative statistical approach is applied. The most common method of estimating the reliability of treatment outcome questionnaires is to use

Chronbach's coefficient alpha. Chronbach's coefficient alpha estimates the consistency of items included in a questionnaire. A high coefficient indicates that the items are consistently measuring the same underlying construct. Because all items in the adult outcome questionnaire inventory correlate with the

global distress factor, reliability of the questionnaires tends to increase with the number of items, but with rapidly diminishing returns. In other words, questionnaires with 15 items may be considerably more reliable than questionnaires containing only 5 items, but they may have comparable reliability to questionnaires of 30 or more items.

The

standard error of measurement (SEM) is a function of the both the reliability and standard deviation of the measure. Test scores for individuals are commonly reported as plus/minus the standard error of measurement in order to convey information regarding the measurement error.

The

Reliable Change Index is also a function of the standard error of measurement. If change on the questionnaire exceeds the Reliable Change Index, then the patient can be considered "significantly improved", meaning that the observed change would not be expected by chance alone at a probability of less than 5%.

To estimate the SEM and RCI for ACORN questionnaires with a known standard deviation and coefficent alpha, use the Excel based SEM Calculation Utility.

The coefficient alphas for specific questionnaires will be posted and updated as providers submit data. Preliminary testing indicates that a coefficient alpha of .94 for both a 20 items and 15 item version of Adult Outcome Questionnaire. An 11 eleven items depression/anxiety scale has a coefficient alpha of .92. See

Adult Outcome Questionnaire normative data.

For estimates of reliability for questionnaires of varying length see:

Item count and reliability.

### ACORN Manual Table of Contents

Click on the following to view content