I show data on two trends undermining surveys: the collapse of human response rates and the increase of AI agents. I'll also discuss downstream implications and propose some possible solutions.
In terms of response rates, having run some of the largest online panels for multi-national corporations from 1997 through 2016, without some active panel management (removing non-responders from the database) response rates declined to under 1% in many of the online respondent sources that I evaluated for acquisition. If you notice, the industry never talks about response rates any more. In addition to BOTs this industry gets along by allowing "respondents" to complete many, many surveys in one session. Not a pretty picture.
That’s an interesting insight and shockingly low! This also highlights that a lot of work across different sectors need to be done, what would your opinion be on how the industry can improve the survey quality?
Three areas that are important to promote within the online sample research industry is improving transparency in sourcing-where is my sample really coming from, establishing some basic operating and research standards-today there none, and educating the users of online sample on what is actually going on behind the curtains to put together their sample and how that can impact their results. But I'm not optimistic that even this can be achieved as the industry has been well aware of the current sampling limitations for many years and it has not been able to resolve the decline in quality.
Having worked in the Market Research world side of this, I can tell you that bots are probably 20% of traffic? Research panel providers try and do a lot to screen bots out at the early recruitment phase and have limits on the amount of surveys individual accounts can take. But there isn't a lot of coordination across the industry for this. My suggestion is always check you source. Some panels are literally selling you junk, while others are making much more effort to clean their sample pools.
At some point I think the entire online world will need some sort of 'online human proof' that goes well beyond captcha!
Thanks! I didn’t know about that Odd Lots episode but that sounds very interesting, I will definitely give it a listen!I completely agree to always check your sources and that there should be some industry standards out there. There are some great start-ups that can offer bot detection solutions but using these is certainly not the norm yet.
This is a very insightful article with well considered recommendations, thank you Lauren. Giving participants a better experience is something we've been presenting about at every market research conference myself and the Nexxt Intelligence | inca team has been to this year. The market research industry is, very correctly, focused on efforts to minimize AI bots entering surveys but the participant experience remains shockingly overlooked. Most surveys don't look any different today to how they did 20 years ago. What other form of online communication has stood still for that time?! It seems crazy, and self-defeating, to me to put a lot of focus on efforts to get real, verified participants to take surveys and then give them a bad experience.
Related -- from March onwards I've seen a surge in the number of merchants who follow any electronic transaction or interaction w/ a request to rate, and even write about, the customer experience. (Have not yet seen this, tho, so far, for the toilet paper I've bought on-line). I'm sick of it, and now usually ignore the feedback-seekers. Sellers on Ebay and similar sites are particularly insistent, one seller sent me 2 emails beseeching a favorable rating, and informed me that she had already rated me favorably, implying that I, the customer, needed to return the favor.
Completely agree that this survey culture is starting to overdo it - especially since this also reduces the appetite for surveys with higher stakes for people
(Disclaimer: I work a national official statistics department, but am a cyber security professional, not a methodologist or social scientist; I also of course do not speak for anyone here except me, as a concerned "supporter".) I came here via B. Scheiner's referencing of the work. This trend is ominous and I can only think that it may create a positive (as in reinforcing, not as in "good") cycle with the garbage produced by other brands of "bot".
Lauren - do you have a point of view on the BLS' Household survey, which delivers the unemployment rate, and the Establishment survey, which presents data on employment additions or subtractions in various industries?
I know the BLS is reporting lower response rates, but still in the 70% range.
Agreed. But I do wonder how they achieve that. I run a large community of job seekers (30,000+) and I constantly see anecdotal evidence of more unemployment than the official figures suggest. Having said that, I do NOT want the number to be higher (some have agendas around that, I don't). I just want to have confidence in its accuracy.
Re: paying more, as a market researcher, we're happy to pay more for survey responses, but that just increases the incentive for survey fraud. Higher incentives alone aren't a solution - have to be combined with other strategies
One way to reduce fraud right off the bat is to conduct an identity validation check on the prospective respondent. Currently there is no validation conducted in any of the popular respondent sources that fuel the online survey business. Some claim that the double opt-in is some sort of validation but it is not.
Raising incentives does increase the number of people willing to participate in surveys, it does not solve the fraud problem, but it does increase the number of available quality survey takers. There are plenty of good tools available to prevent against survey fraud such as Efrain mentions, but only higher incentives will actually improve the quality and attentiveness of respondents.
I think we should build better AI agents and solve entropy issue synthtetically. Please leave the idea of getting answers from real people and stop calling real consumers. Welcome in the digital twins era...
This is a thoughtful analysis - thank you for writing this. And you've demonstrated how easy it is to build an AI agent to answer surveys. I appreciate the fact that your solutions are a starting point. I think the industry needs to consider more solutions, 2 of which I feel , are unexplored but could be interesting. What if survey participants had to be verified using blockchain backed identity tokens? This could be a secure way to verify real participants and can be set up to prevent bots from accessing. Or, have a third party review system that verifies research panels, like McAfee does for data integrity. would be curious to hear your thoughts on these ideas.
In terms of response rates, having run some of the largest online panels for multi-national corporations from 1997 through 2016, without some active panel management (removing non-responders from the database) response rates declined to under 1% in many of the online respondent sources that I evaluated for acquisition. If you notice, the industry never talks about response rates any more. In addition to BOTs this industry gets along by allowing "respondents" to complete many, many surveys in one session. Not a pretty picture.
That’s an interesting insight and shockingly low! This also highlights that a lot of work across different sectors need to be done, what would your opinion be on how the industry can improve the survey quality?
Three areas that are important to promote within the online sample research industry is improving transparency in sourcing-where is my sample really coming from, establishing some basic operating and research standards-today there none, and educating the users of online sample on what is actually going on behind the curtains to put together their sample and how that can impact their results. But I'm not optimistic that even this can be achieved as the industry has been well aware of the current sampling limitations for many years and it has not been able to resolve the decline in quality.
Nice post. Definitely something many people don't think about, but a really important topic. There was a great Odd Lots podcast episode recently about the BLS in the US and the decline of survey participation and what this might be doing to official economic metrics (https://www.bloomberg.com/news/articles/2025-04-30/why-us-labor-statistics-and-other-data-are-deteriorating-jobs-report?utm_source=website&utm_medium=share&utm_campaign=copy) scary stuff!
Having worked in the Market Research world side of this, I can tell you that bots are probably 20% of traffic? Research panel providers try and do a lot to screen bots out at the early recruitment phase and have limits on the amount of surveys individual accounts can take. But there isn't a lot of coordination across the industry for this. My suggestion is always check you source. Some panels are literally selling you junk, while others are making much more effort to clean their sample pools.
At some point I think the entire online world will need some sort of 'online human proof' that goes well beyond captcha!
Thanks! I didn’t know about that Odd Lots episode but that sounds very interesting, I will definitely give it a listen!I completely agree to always check your sources and that there should be some industry standards out there. There are some great start-ups that can offer bot detection solutions but using these is certainly not the norm yet.
This is a very insightful article with well considered recommendations, thank you Lauren. Giving participants a better experience is something we've been presenting about at every market research conference myself and the Nexxt Intelligence | inca team has been to this year. The market research industry is, very correctly, focused on efforts to minimize AI bots entering surveys but the participant experience remains shockingly overlooked. Most surveys don't look any different today to how they did 20 years ago. What other form of online communication has stood still for that time?! It seems crazy, and self-defeating, to me to put a lot of focus on efforts to get real, verified participants to take surveys and then give them a bad experience.
Related -- from March onwards I've seen a surge in the number of merchants who follow any electronic transaction or interaction w/ a request to rate, and even write about, the customer experience. (Have not yet seen this, tho, so far, for the toilet paper I've bought on-line). I'm sick of it, and now usually ignore the feedback-seekers. Sellers on Ebay and similar sites are particularly insistent, one seller sent me 2 emails beseeching a favorable rating, and informed me that she had already rated me favorably, implying that I, the customer, needed to return the favor.
Merchant, leave your customer alone !
Completely agree that this survey culture is starting to overdo it - especially since this also reduces the appetite for surveys with higher stakes for people
(Disclaimer: I work a national official statistics department, but am a cyber security professional, not a methodologist or social scientist; I also of course do not speak for anyone here except me, as a concerned "supporter".) I came here via B. Scheiner's referencing of the work. This trend is ominous and I can only think that it may create a positive (as in reinforcing, not as in "good") cycle with the garbage produced by other brands of "bot".
I agree! That’s why it’s extra important to draw attention to these issues publicly too!
Lauren - do you have a point of view on the BLS' Household survey, which delivers the unemployment rate, and the Establishment survey, which presents data on employment additions or subtractions in various industries?
I know the BLS is reporting lower response rates, but still in the 70% range.
Do you have a point of view on this?
I don’t have specific insights in the BLS, but 70 percent would be very good!
Agreed. But I do wonder how they achieve that. I run a large community of job seekers (30,000+) and I constantly see anecdotal evidence of more unemployment than the official figures suggest. Having said that, I do NOT want the number to be higher (some have agendas around that, I don't). I just want to have confidence in its accuracy.
The best that can be done is to get full transparency into the sample used and the methodology used to process the data.
Re: paying more, as a market researcher, we're happy to pay more for survey responses, but that just increases the incentive for survey fraud. Higher incentives alone aren't a solution - have to be combined with other strategies
One way to reduce fraud right off the bat is to conduct an identity validation check on the prospective respondent. Currently there is no validation conducted in any of the popular respondent sources that fuel the online survey business. Some claim that the double opt-in is some sort of validation but it is not.
Completely agree! - it needs to be combined with better detection strategies to work
Raising incentives does increase the number of people willing to participate in surveys, it does not solve the fraud problem, but it does increase the number of available quality survey takers. There are plenty of good tools available to prevent against survey fraud such as Efrain mentions, but only higher incentives will actually improve the quality and attentiveness of respondents.
I think we should build better AI agents and solve entropy issue synthtetically. Please leave the idea of getting answers from real people and stop calling real consumers. Welcome in the digital twins era...
This is a thoughtful analysis - thank you for writing this. And you've demonstrated how easy it is to build an AI agent to answer surveys. I appreciate the fact that your solutions are a starting point. I think the industry needs to consider more solutions, 2 of which I feel , are unexplored but could be interesting. What if survey participants had to be verified using blockchain backed identity tokens? This could be a secure way to verify real participants and can be set up to prevent bots from accessing. Or, have a third party review system that verifies research panels, like McAfee does for data integrity. would be curious to hear your thoughts on these ideas.