Thought Leadership on Strategic Human Resources Solutions presented by Sibson Consulting.
Many organizations use regression data analysis1 to help make major decisions about how to select, motivate, compensate and reward their employees. Unfortunately, some users of regression embark on misguided expeditions by simply entering data from a group of variables into statistical software and accepting the results as factual. This expeditious approach often yields chance associations without bases in reality. Furthermore, regression analyses can go astray in many other ways that yield false conclusions even in the hands of adept practitioners.
Regression is not a magic elixir. Good results depend on careful development of a model and regard for the variables that are included — and excluded — from the analysis. Heads of organization development and research, human resources and division and function heads that depend on statistical techniques to make decisions should understand which results make sense and are worthy to act upon, and which do not. In this article, we explore one of the ways regression analysis can trip up both producers and consumers of data, leading to false inferences about what the data really says. The article includes two examples, one from the world of corporate sales and another from higher education to illustrate that contextual analysis applies across all types of institutions.
What Can Go Wrong?
Take the example of a CEO who learns that a multi-company regression shows that organizations achieve higher sales when their sales forces are more engaged. As the company’s leader, he declares war on disengagement in order to increase sales. Is he right to do so?
Interestingly, the correct answer is, “No.” This is because what is true for groups may not be true for individuals. If a group’s engagement level were to increase, the group’s sales would be expected to increase by a certain amount. However, that same relationship is not necessarily true for individuals. In fact, the opposite might be true. Individuals who are less engaged may have greater sales for a number of reasons: They may be more willing to take risks, more activated solely by personal gain and less concerned about the success of others, or harder negotiators for the best territories.
Differences in patterns of results at different levels of analyses (groups versus individuals) are relatively common in applied research. Confusing the two occurs commonly enough to deserve a label: the ecological fallacy. That term refers to inferences falsely made about individuals based on results found at the group level, when in fact, predictions made at each level of analysis can be quite distinct.
Inferences drawn from individual-level analyses also have their limitations. For example, one organization could relate employee engagement with sales and conclude that a positive relationship indicates that increases in employee engagement are associated with increases in an individual salesperson’s sales. They would be right. Nevertheless, it would be impossible to conclude that this relationship holds in other organizations because organizations tend to have designed, differentiated cultures that exert unique influence over their employees. Further, the employees themselves may be dissimilar from one organization to another because they have self-selected into a chosen organization for their personal reasons. It cannot be assumed, then, that the relationships that exist in one place will be found elsewhere under a new context with different kinds of people. Therefore, in order to generalize from one institution to another, it would be necessary to replicate results within several organizations.
Looking at Organizations and People Together
A form of regression known as contextual regression2 can be used to avoid limited or false conclusions in surveys and other forms of applied research. Contextual regression permits simultaneous examination of group- and individual-level data. It allows users to determine if a given relationship holds true for every organizational group.
Figure 1 below illustrates a simplified scenario. Part A shows the relationship between engagement and sales at the group level (for each organization), whereas Part B shows the relationship at the individual level across organizations. The results of contextual regression in which organizations and people are analyzed together are illustrated in Part C. In this case, it would be correct to claim that engagement positively influences sales for “your organization,” but not as much as it could, gauging by the patterns observed in other organizations.
Practically speaking, the findings in Part C suggest that there is something about “your organization” that dampens the relationship between engagement and sales outcomes. It may be that the sales plan is poorly designed — salespeople do not believe that extra effort will be rewarded through the plan. Perhaps a team selling structure that is unique to the organization diffuses personal motivation among the sales personnel. Alternatively, the company might sell exclusively in markets that are highly contested, and where selling and gaining market share is particularly difficult. Many possibilities exist, but the results make it clear that “your organization” is different.
The fact that behavior within organizations may be idiosyncratic challenges the idea of a survey norm. Figure 2 shows three organizations with identical means for engagement and turnover intentions. Any one organization could serve as the normative contrast for the other two, but it is unclear what the benefit of knowing this would be since the norms arise through various means and have very different implications regarding influence on actual behavior. That is, even though the norms or standards of comparisons are alike, they do not have the same meanings in their contexts. Do equivalent norms really imply that the three organizations in Figure 2 are the same, or are they happenstance convergences of three diverse stories?
About the authors:
Michael O’Malley, a PhD in social psychology and quantitative methods, is a senior vice president and human capital consultant for Sibson Consulting. He has more than 25 years of experience in compensation, performance management, quantitative methods and organizational research, design and effectiveness. He can be reached at 212.251.5444 or email@example.com.
Demi Farina is an associate human capital consultant for Sibson Consulting with experience in compensation assessment and design, career structures and organizational effectiveness. She can be reached at 919.233.6652 or firstname.lastname@example.org.