In most organisations, engagement research has become a synonym for “the voice of the employee”. Results are presented as an objective picture of organisational reality, and the response rate is often taken as implicit confirmation that the findings are valid. The reality is that organisations have never had more data about their people, and have never been less sure what that data actually means.
The higher the response rate, the greater the confidence in the conclusions - and here sits one of the most persistent methodological fallacies in contemporary HR practice: the assumption that having “enough answers” is enough to have an accurate picture of the organisation. But the question is not only how many people answered. It is who stayed outside that answer, and what their absence systematically means for the interpretation.
If your response rate is 57%, did you measure employee engagement, or the engagement of those who still want to talk to the organisation?
Convenience samples, methodological bias, and the illusion of objectivity
In recent years organisations have become extremely sophisticated in how they measure employee experience. Engagement surveys, pulse surveys, wellbeing indices, organisational-climate dashboards and various people-analytics tools are now standard practice. Their results go into board presentations, strategic plans, and decisions about leadership, organisational design and investment in people.
At first glance it looks as though we have never had more data about people. At the same time, a much more important question rarely gets asked: how representative is that data of the organisation we are talking about?
In a great many companies the response rate in engagement research rarely exceeds 50–60%. Sometimes it is considerably lower. Yet the results are frequently interpreted as though they represented an objective picture of every employee's experience, reliable enough to base cultural and organisational decisions on.
That is where the problem starts - not because HR research is meaningless, quite the opposite. The problem starts when we begin treating the methodological limits of our research as unimportant technical detail rather than as something that directly affects the quality of organisational decisions.
What a response rate is, and why it matters more than it looks
The response rate is the percentage of people who took part in a study relative to the total number it was sent to.
response rate = number of responses / number invited × 100
If an organisation has 2,000 employees and 800 complete the survey, the response rate is 40%.
At first glance this looks like an ordinary administrative figure. In research methodology it is a good deal more: an indicator of the potential risk of sample bias and of the limits on generalising the results.
It is important to understand that a low response rate does not automatically mean the data is “bad”, just as a high one does not guarantee the quality of the conclusions. The key question is not only how many people answered, but:
- who did not answer, and
- whether those who did not participate differ systematically from those who did.
In research methodology this is called nonresponse bias.
Groves and Peytcheva (2008), in one of the most-cited papers on the subject, showed that the response rate is not a sufficient indicator of data quality unless we understand the structure of the nonresponse. In other words, the problem is not simply a small number of answers. The problem arises when the absence of answers is not random - and in organisations it very often is not random.
People do not take part in surveys at random
In many companies the response rate is still treated as an operational or communications problem:
- send more reminders,
- get managers to “push people”,
- make the survey shorter,
- or make it more visually attractive.
But it is not merely a process KPI. Contemporary literature in survey methodology and organisational behaviour shows that participation in research is not neutral behaviour. Employees participate, and fail to participate, for different reasons.
Employees who more often take part in engagement research tend to:
- have more trust in the organisation,
- feel psychologically safe,
- believe their voice means something,
- or already be relatively connected to the system.
Those who very often do not participate at all tend to:
- not believe in the anonymity,
- believe nothing will change,
- feel professionally distanced,
- or have already left the organisation psychologically.
And here we reach the heart of it: the organisation is then not measuring the experience of all its employees, but the experience of those still connected enough to want to be part of the conversation.
An example: when “good results” hide an organisational problem
A large manufacturing organisation ran its annual engagement survey:
- the response rate was around 57%,
- the engagement score was above the industry benchmark,
- leadership results were stable,
- wellbeing indicators were “within expected bounds”.
The conclusion was that the organisation was passing through a period of change relatively steadily, and that no serious intervention was required.
Several months later there appeared:
- rising turnover in operations,
- an increased number of sick days,
- falling performance in certain plants,
- and a serious retention problem.
Subsequent analysis showed that employees from exactly those units had barely participated in the survey. The most critical segment of the system had not been present in the data - and the organisation had been making decisions on the basis of that data.
That is perhaps the greatest danger in contemporary HR analytics. Data can look sophisticated enough to create an illusion of objectivity even when the methodological foundation is very fragile.
The convenience sample: when an organisation hears only its most available voices
Organisations frequently use the response rate as implicit proof of the quality of the research. The more employees answer, the more reliable the results are assumed to be. Contemporary literature is increasingly cautious about that assumption. A high response rate does not by itself guarantee the absence of bias, just as a low one does not automatically make the data unusable. The far more important questions are:
- whether the sample adequately reflects the structure of the population, and
- whether there are systematic differences between the responding and non-responding groups.
A meta-analytic study (Wu et al., 2022) problematises reliance on the response rate as the primary indicator of data quality still further. The authors point out that the characteristics of non-participating groups often carry more methodological weight than the response percentage itself. In an organisational context that matters particularly, because those who do not participate are often the employees with lower trust, higher burnout, weaker identification with the organisation, or greater psychological distance from it.
So an engagement score sometimes does not measure the state of the organisation, but the state of its most available and most willing part.
Most engagement research uses what methodology calls a convenience sample. The survey is sent to everyone, and then whoever wants to answer, answers. At first glance that looks inclusive. Methodologically it has a serious limitation: it is not controlled for the structure of the population. Certain parts of the organisation are more likely to be represented in the data than others.
The answer: a representative sample is a methodological minimum, not an academic luxury
If we cannot include the entire employee population - and in practice we often cannot - the question is no longer “how do we get more answers?” but “how do we ensure the sample genuinely represents the organisation?”
A representative sample means the structure of respondents proportionally reflects the structure of the population along the relevant dimensions:
- function,
- organisational level,
- location,
- type of work,
- seniority,
- or other factors that matter to organisational experience.
The goal is not for everyone to answer. The goal is for every relevant part of the system to be adequately represented.
In practice that means:
- defining the population,
- identifying the key strata,
- sampling proportionally,
- selecting randomly within groups,
- and monitoring the response rate by segment of the organisation.
That approach does not produce “perfect” data. It produces something considerably more valuable: control over bias, and a more responsible picture of the organisation.
The biases organisations tend to ignore
Unrepresentative samples rarely arrive alone. They usually come with a set of methodological and psychological biases that complicate interpretation further.
Nonresponse bias. As above, this arises when those who did not participate differ systematically from those who did. In an organisational context it often means the data excludes employees with lower trust, higher burnout, or weaker identification with the organisation. The results then look healthier than the organisation actually is.
Survivorship bias. Organisations often analyse only those still in the system. No longer visible are the employees who resigned, withdrew psychologically, or stopped participating in organisational processes. The system then measures the experience of the survivors rather than the experience of the organisation.
Social desirability bias. Even when employees do participate, an open question remains: are they answering honestly, or safely? In organisations with weak trust in anonymity or a high degree of control, employees frequently give socially desirable answers. The results look tidy, stable and professional, while the real problems retreat into informal conversation, passive distance and quiet turnover.
Leadership confirmation bias. One of the least-discussed problems is how leadership interprets results. When the data confirms the existing narrative - “the culture is stable”, “the change is going well”, “people are satisfied” - organisations rarely raise methodological limitations. When the data shows a problem, questions about the validity of the research appear far more often. Organisations tend to treat data as objective precisely when it confirms what they already wanted to hear.
Benchmarks as psychological safety rather than analytical safety. An industry benchmark reinforces the impression of validity further. “We are above the industry average” sounds like serious evidence of organisational success. But a benchmark only means something if we understand how the data was collected, what the response rates of the reference organisations were, how engagement was defined, and how methodologically comparable those organisations are. Without that, organisations are comparing differently selected samples rather than different organisations. A benchmark often functions more as psychological reassurance than as a serious analytical tool.
Practical implications: when data becomes a selective reality
In practice, the problem of unrepresentative samples is not merely methodological. It bears directly on the quality of organisational decision-making. When leaders make decisions on the basis of data, they are not deciding on the basis of “the truth about the organisation”, but on the basis of the version of the organisation that made it into the sample.
That has three important consequences.
First, decisions about culture and leadership rest on a stabilised picture of the system. If the sample predominantly contains employees with higher trust and greater engagement, the organisation may wrongly conclude that its problems are peripheral rather than systemic.
Second, interventions get aimed at symptoms rather than at the structure of the problem. If the data does not “see” the disengaged population, the organisation will invest in additional engagement programmes for the already-engaged, instead of addressing the causes of withdrawal.
Third, an illusion of stability appears at precisely the moment the system is becoming polarised: one part of the organisation remains visible through the data while another becomes statistically invisible.
In that sense, sampling is not an academic problem. It directly determines the quality of strategic decisions.
The difference between measuring and understanding
This may be the greatest challenge facing contemporary HR. We have more data than ever before, and the quantity of data by itself guarantees no understanding of the system.
Organisations can have handsome dashboards, stable engagement scores, benchmark results and precise multi-year trends, and at the same time systematically fail to see the parts of the organisation most distanced from the system. Which is why the question worth asking is: who did not participate, and what would their answer have changed?
Until we understand who did not participate and why, there is a real risk that we are not making decisions on the basis of data about the organisation, but on the basis of data about its most available version. The difference between those two pictures is not a statistical nuance. It is the difference between managing a system and managing a projection of it.
