Introduction
Thomas Gilbert states that prompt reinforcement in the form of feedback will inevitably lead to improvements in the motivation cell of his behavior engineering model. But, how is worker motivation sustained when the feedback is delayed for years, as it often is for those involved in very long-term work projects? This research explores the issue of long-term work goals. However, in this synopsis, instead of discussing the study results, we will focus on how the researchers designed their qualitative research methods to identify factors related to long-term motivation and describe how they conducted inter-rater reliability analysis to validate their findings.
Article
Bateman, T. S., & Barry, B. (2012). Masters of the long haul: Pursuing long-term work goals. Journal of Organizational Behavior, 33, 984-1006. doi: http://dx.doi.org/10.1002/job.1778
Background
Theories on worker motivation have typically focused on the short-term. Immediate considerations are the driving forces that keep most adults focused and motivated in their respective workplaces. Workers involved in projects dealing with discovery, innovation, or the development of enduring institutions, however, work under a time horizon that can extend many years, even decades, into the future. In such cases, feedback on the success or failure of their efforts is not forthcoming and, depending on circumstances, is often delayed. Researchers sought to determine how current theories on motivation might apply to workers involved in long-term work and, if the theories did not apply, to identify factors that could serve to expand existing theories that would address workers in it “for the long haul.”
They began by considering two strands in the current literature: long-term thinking and time horizons, and long-term goals and goal pursuit, including the role of self-regulation. They compared proximal (near) goals with distal (far) goals, and examined self-regulatory processes of workers involved in very long-term projects.
Qualitative Research Methods
The researchers used a qualitative research strategy of theory elaboration to expand the framework of factors influencing motivation. Similar to a grounded theory approach, they collected data and from those data created hierarchical structures. Rather than proposing new theory, they added these hierarchies to the existing theory framework.
PURPOSIVE SAMPLING: The researchers conducted semi-structured interviews with purposively selected 25 individuals whose professional work had very lengthy time-horizons. Their work was marked by three characteristics:
- eventual success of their efforts could be years, decades or generations in the future
- progress was made at a very slow rate
- there was a significant chance for failure
All participants worked in knowledge-based organizations in the United States. They worked in fields that included securities, nanotechnology, biomedical science, biodiversity, astronomy, and the creation of a cultural institution dedicated to long-term cultural change. Participant ages ranged from 24-60 years, with a median age of 50. Tenure in their current organizational affiliations ranged from 1 to 27 years, with a median tenure of 6 years.
DATA COLLECTION AND ANALYSIS: The researchers recorded and transcribed data elicited from the following question categories: goals, motivation, perspective on the future, persistence, work context, and bio-descriptive. From the data, the researchers identified 24 first order categories, then 8 second-order themes. The researchers constantly compared data categories with each other and the fit of participant responses within each category.
INTER-RATER RELIABILITY TESTING: This research involved multiple ‘raters’ or ‘coders’ for analyzing qualitative data (two researchers and two doctoral students). To assess the reliability of their data associations, the researchers selected 80 responses whose characterizations they agreed on. The researchers then selected two doctoral students unfamiliar with the project and trained them in the coding categories and methods employed. The researchers then asked the students to independently code the same 80 responses according to first-order categories and second-order themes.
The researchers then conducted two reliability tests using Cohen’s Kappa, the preferred method of determining inter-rater reliability of nominal (categorical) data. The first test compared the two raters’ codes. The coders agreed 72 percent of the time, with a kappa value of 0.70 (p < .0001), indicating substantial agreement. Where coders agreed on themes, they also agreed on first-order categories 86 percent of the time.
The second reliability test assessed agreement between the coders and the researchers’ judgments. Coders agreed with the authors in 85 percent of the cases, yielding a Kappa value of 0.83 (p < .0001), indicating nearly perfect agreement. At least one of the two coders agreed with the authors’ judgments 90 percent of the time. The inter-rater reliability analyses indicate high reliability in coding interview comments according to motivational themes.
Questions for OPWL-N Members
What tips do you have for using unstructured, semi-structured, or structured interviews to collect qualitative data? In what situations should you use each and why? What methods are used to assess the reliability of data obtained from multiple raters/coders?
Workplace Oriented Research Central (WORC)
Prepared by OPWL Graduate Assistant, Susan Virgilio
Directed by OPWL Professor, Yonnie Chyung
Posted on October 24, 2013