6 steps to screening success (i.e. "How do I screen participants for my study?")

2018-04-10 07:04 AM By Luke Freeman

“Always remember that you are absolutely unique. Just like everyone else.” – Margaret Mead

Aren’t people incredible? Every day I’m reminded about how unique people are... but our uniqueness is always through the lense of things that are similar. If there was only one eccentric person in the world we wouldn’t have the word “eccentric” after all.

Screening participants is about finding participants with the right combination of attributes that make them distinct and suitable for your study, but also representative of a group of people. Finding just the right “kind of person” for research is tough – it can often be one of the hardest parts of research.

In this article we’ll take you through the steps we recommend when screening participants for research.

The wrong “Guy”

Back in 2006, Guy Goma became an overnight celebrity after a case of mistaken identity. He was interviewed live on BBC News TV as an internet expert when in fact the currently unemployed computer technician had been at the BBC for a job interview.

His two attributes of being named “Guy” and being at the BBC reception at the right time led him to be “not the Guy they were looking for” and the poor guy didn’t realise until he was live on television (at which point he bravely powered on and answered the questions anyway).

While I can certain admire his guts on national television, the last thing you want as a researcher is to pick the wrong guy to base your research on.

The difficulty in finding the right participants is often quoted by researchers as the reason for having underpowered studies and the reason that businesses often don’t even conduct much-needed product or market research. The process can seem difficult, confusing and time-consuming – but it can actually be straightforward and easy.

Here at Positly we help researchers find the right participants all the time and would love to share what we’ve learned. Following the steps below will lead you to have more significant research, reduce risk, save you money and save you time.

Six steps to screening success:

Step 1. Define your criteria 

The attributes of the participants you want in your research could vary from incredibly specific to very broad; however, most research will fall somewhere between the two extremes.

Sometimes researchers are interested in broad general populations, in these cases they’re less interested in specific demographic of behavioural attributes and more interested in having a representative sample of high quality participants or balancing the sample by an attribute (e.g. political affiliation) to compare in an experiment. Other times researchers have incredibly niche criteria, these can be hard to recruit for, but certainly not impossible.

Why are you screening?

The reason for screening will determine how you screen. Are you screening for reflective feedback? What about being a representative user? Or just that they are available? Is it mostly about quality?

Typically the main two categories are relating to the attributes of the participant (e.g. demographics, psychographics, behaviours etc) and the quality of the participant (e.g. providing insightful answers, answering honestly, reading carefully).

Fortunately at Positly we have a number of existing quality measures that we use to pre-screen quality participants for you so you don’t have to worry about low quality participants. We’ll be writing another post about how you can add in quality measures yourself if you choose, but for now, let’s look at the participant attributes.

Participant attributes

Get together with your team and list out the attributes of your target participants. It's like the classic game Guess Who, the attributes make the person. Once you have the attributes you can narrow down to the types of criteria you can use to determine those attributes.

For example, an “active email user” could be defined as “sends at least 5 emails per day” and then they could be asked how many emails they send per day.

Just as important as what you want to see in a participant is what you don’t want to see in a participant. You could be excluding people who fall far outside the average user profile (e.g. participants who are too technically literate or not technically literate enough), people who work for competitors, or people who would be considered naive  for an intervention or assessment (e.g. you are testing CBT therapy and you want people who haven’t previously used it).

Cost of screening

You’ll need to balance the costs of screening-out inappropriate participants from the costs including participants who aren’t necessarily entirely ideal. It can be more expensive to get a perfectly screened group of participants than is worthwhile – you’ll need to make a pragmatic choice here.

A common pragmatic choice is to simply collect all the data and then split the analysis into at subgroups of "most ideal participant" and "less ideal" to examine if there are even differences – this works only if your sample size is large enough.

Sometimes the cost of inappropriate participants is very low (e.g. just excluding them from the data or analysing separately), and sometimes having a few can even be advantageous. You may be planning to exclude the more extreme participant (e.g. on technical literacy) in your final study but it could be worth including them in your pilot your studies to see how questions or tasks may be difficult for the average participant (if they’re more likely to struggle and tell you about it then you can design better research.

However, sometimes the cost of an inappropriate participant can be very high in effort or money (e.g. longitudinal studies, high incentives, travel costs, editing costs) and other times it could be costs in reputation or failure (e.g. embarrassment in the boardroom, failure to publish).

The tradeoff between how much screening costs and how much benefit you’ll get is something you’ll need to keep in mind throughout this process.

Number of attributes

If you are screening for a single study, ideally your screener will be less than 10 questions or take less than 2 minutes to complete (including demographics).

Take the time to think about which of the attributes are really important to you. Also consider what you can gain from using pre-screened participants initially to narrow the funnel. At Positly we already pre-screen on common demographic attributes so that you can use them for targeting.

If someone is doing the screening task on the expectation that they might then take part in the main study they will be more frustrated if they get disqualified at the end of a long screener than if they got disqualified earlier on (or didn’t need to be disqualified). Be respectful of participants, they’re helping you and they are essential to research – treat them with the respect they deserve.

When it comes to the number of attributes that you collect, keep in mind the privacy of potential participants and don’t collect data points that you don’t plan on using or that could be easily de-identified without their consent.

Balanced, representative and weighted samples

Often you may need to do more than simply excluding participants entirely based off an attribute, you could be trying to have a certain distributions of attributes in your sample.

If you're going to be conducting quantitative analysis you'll think ahead of time if you need to weight the responses, balance the participants or match the representativeness of a sample. These are all ways of ensuring that your data will be more meaningful and statistically powerful. It’s going to require that you screen for relevant attributes about participants up front.

The act of balancing your sample is to pick even numbers of different groups so that you can compare them all evenly without having to do any complicated math. For example, I may want 50% PC users and 50% Mac users, or 33% Republicans and 33% Democrats and 33% Other.

Participants may also be selected on the basis that they are representative of a target population, this is often done by stratifying the participants that you recruit into different sub groups with quotas (e.g. Men and Women each by age <50 and 50+) or statistical matching (e.g. ensuring the average age of a group is the same as the target group).

Weighting involves multiplying each participant’s responses by how much their attributes are over-represented or under-represented in the sample. For example, if I wanted 50% male and female comparison and I only had 20 men and 10 women I would weight the men’s responses each as 0.5. If you are using weighting alone you can ask these questions at the end of the study as they do not determine who is allowed to participate.

Step 2. Pick your pool 

The quality of your research is going to be influenced heavily by the quality and relevance of your participant pool.

In some cases getting feedback from your friends and family may be a good start and is better than nothing, but in most cases it’s not going to be very helpful. Friends and family are unlikely to be your ideal participant. Not only are they unlikely to have the right attributes nor be representative, they’ll also be overly kind to you (e.g. you’ll miss out on the more brutal feedback you need) or sensitive about their privacy (e.g. your friends may not want to share information with you about their lingerie preferences or the state of their mental health).

There are a lot of sites that exist for “exchanging surveys” where people do each other favours by filling in each others surveys (some sites provide “credits” so that your surveys rank higher if you take more surveys yourself). As I am sure many of you can imagine, in most cases a pool of fellow researchers is not going to a representative sample – unless you think this looks like a normal person.

Universities and college often have participant pools available for their researchers to use. These are primarily student participants who are often participating for course credit or as a requirement of their course. While this makes in-person studies easier and can sometimes reduce the cost of paying participants (if it’s a course credit requirement), the student participants are rarely a representative population and are often in high demand which results in low numbers of participants for researchers.

If you are a company with existing customers they may sometimes make for good participants. If you pick the right people they could be representative of your customer base (although, not necessarily your potential new customers). You’d need to go to lengths to ensure that you get quality responses. For example, ensuring that you provide anonymity for their responses and that you don’t provide an incentive so large that you get non-representative customers (just give everyone a fair payment/gift certificate/credit). Furthermore, if you are looking to study how you may expend your customer base (e.g. new products, marketing strategy) then your existing customers may not be representative of non-customers – after all, they already use your product or service.

In addition to customer bases there may be other places where you can find a near-perfect audience. For example, if you are studying soccer players you could partner with a soccer club or soccer association.

These considerations are why we find it’s often best to (a) pay someone a clear and fair amount that’s enough to appreciate their time but not too much to incentivise bad behaviour; (b) recruit from a neutral and broad audience (e.g. micro-tasking, research panels, classifieds, advertising, relevant partners); (c) pre-screen for quality and basic demographics; and finally (d) adjust for representativeness.

We built Positly to help find high quality research participants that don’t have the common drawbacks mentioned above in some of these methods.

Step 3. Describe your screener or study 

When building and publishing your screener you’ll need to have an appropriate description of your screening study and a description of your main study where it makes sense.

In some cases you only need to describe the screener, it could be because you don’t want to give away too much until you know who the participants are, or because you don’t want to influence the answers of the participants. Whereas in other cases it makes more sense to describe the main study so that you only get participants who are likely to continue onto the main study.

In both cases the screening activity and main study descriptions should include a title and description. Here’s some tips for what you could include (not all are always necessary and sometimes you will intentionally exclude some details to prevent influencing the sample).

  • Activity title

    • Clear and concise

    • Differentiated from other activities

    • Relevant to your target participants

    • Unlikely to bias the sample

  • Activity description

    • Description of what’s involved

    • Who it is aimed at (where it doesn’t influence answers)

    • Participant consent and other ethics requirements (e.g. withdrawal)

    • The reason you are doing it

    • How many questions there are or how long it will take.

The central purpose of these is to find the right audience, avoid influencing that audience in ways that may compromise your study while managing expectations and ensuring that the participant is properly informed to decide if they want to continue.

The title and description will be required when posting to Positly (and the same would be required on most other platforms). In many cases it helps to include it again on the activity itself and asking the participant to confirm before continuing.

In this description it’s important to be careful about what information you provide and what you withhold. You want the participant to be sufficiently informed but you also want to avoid influencing the results of the study or provide people the opportunity to self-select by giving things away too early.

However, in some cases where the prevalence of a certain population is low you may find that you need to be more transparent up front about what you are after and then use the screener only to confirm that the person is suitable (see this example) – this is called a “transparent screener” and is similar to how recruiting participants in newspaper classifieds used to work.

Step 4. Write and order your questions 

Now that you’ve taken the time to think through your criteria, pool and written your screener description, here’s the part most of you have probably been waiting for.

Writing your screening questions

While defining and refining your criteria you would have a list of attributes you need to find questions for.

The three main categories of attributes are:

  • Demographics
    e.g. age, gender, ethnicity, income

  • Behaviours
    i.e. things people do, ways people act

  • Psychographics
    i.e. personality, values, opinions, attitudes, assessments

The way you ask a question can often change the answers you get. Typically demographic questions are mutually exclusive and collectively exhaustive (e.g. a single person cannot be both aged 25-34 and age 85-94). Behavioural questions are matters of fact and often asked using the Five W’s and H (when, what, where, who, why, how), specific numbers (e.g. how many times this month have you eaten an apple?) or selected from lists (e.g. which of these movies do you intend to watch?). Psychographics are often asked using scales, comparisons, and assessments. When asking psychographic questions it is preferable to use standardised scale formats (such as the Likert scale) or standardised assessments (such as the PHQ-2, IQ or cognitive reflection test).

Key principles for writing questions

Image via BioWrite.co.uk

1. KISS: Keep it simple, short, specific
Use simple words and clear sentences. Avoid, jargon, technical language, ambiguous words, idioms and acronyms. Remember participants are humans, talk to them like people (and by “talk” I mean as if you were having a conversation using simple colloquial language).
Accuracy vs Precision

Image via Keydifferences.com

2. Be precise and accurate
The words you use will matter, they will impact the way people interpret them and therefore how they answer them. While you must keep it simple, you must also remain both precise and accurate in your questions and answer options.
Leading Questions
3. Don't lead them astray
Ask questions in ways that don’t encourage a specific answer (otherwise you’ll bias your results from the beginning) – these are called “leading questions” and they are the devil (in the details) that you must be vigilant to avoid. It shouldn’t be obvious that there is a “right” answer.
Mutually Exclusive
4. Be exclusive
Wherever possible your questions should include all possible options and have no overlap so that people always fit into a single category. In some instances this may involve including an “other”, “other (specify)” or “none of the above” option.
Don't be too binary
5. Don’t be so binary
Often binary screener questions will tend give away the “right” answer due to their higher level of suggestibility. Instead offer multiple options that people can pick from – and if you’re using a multiple select list (like a checkbox) it’s best to include a decoy (an answer that could not possibly be right) or a limit so that people don’t just select all the options. If you are after someone that has recently watched the Iron Man movie I’d suggest asking “Which of the following movies have you seen recently?” and providing at least one non-existent movie instead of simply asking them yes or no.
Tip Responsibly
6. Tip Responsibly
Sometimes it can be hard to make a question text sufficiently clear without becoming verbose (and thus backfiring). This is where a tip text on the question itself or explanatory text above the question can help you to provide additional clarity without making the question difficult to parse. I still recommend keeping this short and simple – the adage “everything must be made as simple as possible, but not one bit simpler” seems to fit here.
British vs American English

Image via teacherdiane.com

7. Think global, act local
Remember that we all have different contexts and some questions simply do not make sense to ask everyone. The internet has been around for a long time and I still occasionally find a store that ships internationally but only accepts 5 digit ZIP codes (US-centric) – they probably wonder why no-one buys their products from abroad. Take appropriate steps to customise the questions to be relevant to all participants, this could involve special question logic that changes how you ask a question. However, if you ask the same question in different ways with different people, you’ll need to “think globally” about how you’ll combine the data again at the end.
British vs American English

8. Be entirely random or neatly ordered
The order of your questions will almost certainly have an effect (small or large) on your data. Therefore it’s important to either (a) follow a “natural ordering” the options (if one exists) or (b)  properly randomise the options for every single participant. For example, options for income or education make sense to have in ascending order whereas gender and favourite movies can almost always be randomised.
These principles are important to keep in mind when writing any questions, but they’re essential when writing screening questions.

Order your questions

Now you have a list of attributes and you know what questions you want to ask people, you’ll need to decide in what order.

You can easily start off by removing anything that you can already determine from the source of recruitment (e.g. the standard demographics offered by a platform, or if someone has a Facebook account and you’ve recruited them from Facebook). Then move anything that is purely for weighting or profiling to the end of the main study.

At this point you’ll be left with your core qualifying and disqualifying questions. The best way to approach this is to think of it as a funnel. Here are your main goals

  • Identify people who wouldn’t qualify as quickly as possible

  • Disqualify early based on obvious blockers (e.g technical requirements)

  • Start broad and then narrow (e.g. “which of these sports do you play?” and only if they select “soccer” ask “how often do you play soccer?”)

  • Protect the integrity of all the questions  (i.e. don’t give away the “right” answer to a later question in the text of an earlier question)

If you want to nerd out further on question ordering I can recommend this article.

Don't waste data

Some researchers conduct multiple studies over time and in many cases a participant may not qualify for a single study but they may qualify for a future study. If you or your team plan to conduct a lot of research you may consider doing omnibus screeners that could qualify people for multiple different studies

Step 5. Build your screener 

By this stage you should have decided on precisely what you’re screening for, how you plan to ask it and in what order you plan to ask the questions. Now you’re ready to build the screener!

You can use your favourite survey software for this, but I strongly recommend GuidedTrack

We have built a library of screening questions for you so that all you need to do to get started is simply include them into your screener by referencing the relevant program in GuidedTrack.

To demonstrate how to use the library questions I’ve made this example screener for you which is screening for non-heterosexual men aged 25-34 who regularly use dating apps and use Facebook for 2 or more hours a week.

GuidedTrack Screening Program Example

As you can see, all I need to do is introduce the study and then ask a few prebuilt screening questions with a little logic every step of the way. In this case I am combining this screener with the targeting options available in Positly. Because I can already target age and gender I do not need to ask these questions in the screener.

As an added benefit, the participant attributes will be automatically passed through to Positly for use in targeting on my main study. This is because I’m using the prebuilt screening question programs and the EndOfActivityButton program in GuidedTrack (if you plan to use Positly and you aren’t using the prebuilt programs you’ll need to ensure that you include your custom attributes in your end of activity link so that you can use it for targeting later).

Step 6. Testing and signoff 

At this point all that’s left to do is to test your screener to see if it behaves exactly how you expect it to. Try different combinations and check the process to see that it works.

Once you are confident it works, make sure to check with anyone who may need to sign it off. In academia this could be an ethics board (IRB) and in a company this may be a supervisor. The last thing you want to do is to find yourself in hot water because you didn’t follow the correct processes or to find your results are unusable because you didn’t screen correctly.

Working alone? It can’t hurt to ask someone else to have a look, especially if it’s your first time.

Ready to launch… and take off!

Give yourself a pat on the back and start adding your screening activity details to your platform of choice (of course, we recommend you use Positly) and launch it. You can then sit back and relax as you watch the data come in and suitable participants qualifying for your main study!