Evaluating the Reliability of Measures of Organizational Characteristics: Issues and Recommendations

Alex Sox-Harris, PhD
Stanford University

Kenneth Nieser, PhD
Stanford University

ABSTRACT:
Measures of organizational characteristics, such as implementation climate, organizational climate, and organizational culture play diverse roles in implementation studies. These characteristics can be conceptualized as contextual determinants of quality, moderators or mediators of implementation strategy effectiveness, or even implementation outcomes. It is therefore essential that the reliability of these measures be evaluated before use. Reliability is a broad term that refers to several distinct aspects of psychometric soundness. In this presentation, we focus on the aspects of reliability that refer to a measure’s ability to precisely characterize and differentiate between organizations. Are measured differences between organizations trustworthy? Do the measured differences reflect true differences or measurement error? More quantitatively, organization-level reliability refers to the proportion of variance that is between, rather than within, organizations. Using measures with unknown or low reliability risks making incorrect judgements about organizational characteristics and their relationships to other implementation science constructs. In this presentation, we highlight how the multilevel nature of data on organizational characteristics complicates evaluation of measure reliability. We also make recommendations regarding which methods should and should not be used to calculate the reliability of these measures.