Federally funded teen pregnancy prevention programs aim to improve sexual and reproductive health outcomes for youth across the United States. To determine these programs’ effectiveness, evaluations often assess a common set of behavioral outcomes, including measures of sexual activity and unprotected sex. For a recent evaluation of the Pulse mobile health app focusing on Black and Latinx women ages 18 to 20, Child Trends incorporated measures of unprotected sex that were recommended by the program funder, the Office of Population Affairs (OPA). These measures included the following: 1) sex without using any method of contraception, 2) sex without using a condom, and 3) sex without using a hormonal or long-acting (LARC) method of contraception.[1] Child Trends researchers found inconsistent responses across these three measures of unprotected sex on the online baseline survey.
Before fielding the longer-term follow-up survey, Child Trends conducted cognitive interviews with evaluation participants to gain insight into how their understanding of question wording may have led to these inconsistent responses. We then revised the survey measures and field tested the new measures on a six-month follow-up survey to assess whether the revisions reduced inconsistencies in participants’ responses. This research brief presents findings from these cognitive interviews, along with recommendations for updating the survey questions that OPA grantees use to assess unprotected sex. Our findings provide valuable insight into the ways that survey measures can be adjusted to most accurately reflect teen and young adult behaviors.
The original unprotected sex questions may have led to inaccurate responses.
Implementing the following recommendations improved the accuracy of unprotected sex measures and was associated with more consistent responses in our six-month follow-up survey.
Among Black and Latinx women ages 18 to 20 who participated in the Pulse evaluation, almost 10 percent of sexually active participants responded inconsistently to the three measures of unprotected sex on the baseline survey (listed below). The original questions were derived from the Office of Population Affairs’ survey items for teen pregnancy prevention program evaluations (all had three response categories: yes, no, or don’t know). In all cases, the inconsistent responders answered “yes” to having sexual intercourse without any method of contraception (question 1), but “no” to ever having vaginal intercourse without using a hormonal or LARC method (question 3) and/or “no” to ever having vaginal intercourse without a condom (question 2). In other words, these respondents answered that they had had sex at least once without using any method—but also that they always used a condom or a hormonal or LARC method of contraception when they’d had sex.
1. In the past 3 months, have you had vaginal intercourse without you or your partner using any method of birth control? That means you had sex and didn’t use a condom or use birth control pills, the shot, the patch, the ring, IUD, or implant.
2. In the past 3 months, have you had vaginal intercourse without using a condom?
3.In the past 3 months, have you had vaginal intercourse without using any of the following methods of birth control?
Birth control pills
The shot (for example, Depo Provera)
The patch (for example, Ortho Evra)
The ring (for example, NuvaRing)
IUD (for example, Mirena, Skyla, or Paragard)
Implant (for example, Implanon or Nexplanon)
a. Yes
b. No
c. Don’t know
We conducted cognitive interviews to better understand the inconsistency in responses across measures of unprotected sex. Cognitive interviews were conducted with 13 of the 47 participants who had provided inconsistent responses. Interviews were conducted via phone and lasted 30 to 60 minutes. Respondents received a $25 Amazon gift card as a thank you for participating. We obtained IRB approval for all study activities.
Rooted in cognitive psychology, cognitive interviews are a method for identifying potential survey problems—especially in the response process—by tapping into the thought processes of respondents as they interpret and answer questions. During the semi-structured interviews, we asked participants to think aloud as they answered each question and probed for common challenges that people have when answering survey questions.
For the first seven interviews, we probed on potential issues with the original survey questions. Based on participant responses in this first round of cognitive interviews, we revised the survey questions and tested them in a second round of six cognitive interviews. We kept the original questions in our six-month follow-up survey and added the new questions to the end of survey. We ran cross-tabulations of 488 sexually active women to analyze the internal consistency of the new survey questions compared to the original questions.
Based on the cognitive interviews, we identified three primary challenges with the survey questions that could lead to inconsistent responses.
The original survey questions asked participants if they have had vaginal intercourse without using any method of birth control, a condom, or a hormonal/LARC method. Participants found this wording confusing: It is counterintuitive to answer “yes” if one did not use a method each time they had vaginal intercourse, and “no” if they did use a method each time. For example, one participant [incorrectly] reworded Q3 (sex without hormonal/LARC method) as, “Am I taking any method of birth control?” and reported that someone would answer, “Yes … I take birth control.” Another participant noted “The ‘without’ is confusing … ‘without’ means you didn’t … but you have to read it and then re‐read it to understand.”
Participants said they had to go back and reread text-heavy questions like Q1 and Q3 multiple times. One participant said of Q1, “[the question] was medium difficulty to understand because you have to reread it … you have to keep going back to the question to understand it.” Another participant, after responding to Q1, noted the importance of keeping questions simple and to the point: “[This question] is easy if you just stop reading where the question mark is.”
We would expect participants to answer “yes” to having sex without a birth control method if they had had a single instance of unprotected sex in the past three months. However, several participants said that they, or other young women, might not answer this way if they had used a method on some occasions but not on others. They said some people might be embarrassed to say they had had unprotected sex, and felt that it wasn’t accurate or fair to answer “yes” to sex without a method in the past three months if they had used a method most of the time. For instance, one participant said that if someone used a method every time they had sex except once, they would likely answer “no” to the measure of sex without any method of birth control (Q1) because “the majority of time they were using birth control except for that one time.” Another participant said, “if it was me, I wouldn’t really count that one time, because it was ONE time.”
Previous research has found that social desirability bias—the tendency of research participants to exaggerate or inaccurately report their behavior to make it seem more socially acceptable—can lead to overreporting contraceptive use. Our findings suggest that this bias, in combination with the binary nature of response options, may have led some participants to inaccurately report that they did not have sex without a method.
Based on our findings from the cognitive interviews, we have the following recommendations for updating survey questions that measure unprotected sex.
We recommend asking participants if they did use contraception during vaginal intercourse, rather than asking if they had vaginal intercourse without This will reduce confusion and misinterpretation about what a “yes” or a “no” answer to these questions conveys.
Providing participants with more options than a binary “yes” or “no” to indicate whether they had vaginal intercourse without a birth control method allows them to select options that more closely match their experiences. For coding purposes, evaluators could then code anyone who does not indicate that they used contraception “all of the time” as having had unprotected sex. A “most of the time” option can potentially reduce reporting bias by allowing participants to report that they are generally consistent contraceptive users.
This will reduce participants’ mental fatigue while taking the survey.
Because questions about sex with a hormonal or LARC method were especially confusing to participants not using these methods, we suggest inserting an initial question that can be used to identify anyone who has used hormonal or LARC methods in the past three months. Any participant who has not used hormonal or LARC methods in the past three months can be skipped out of subsequent questions about the consistency of hormonal or LARC method use. A suggestion for wording and formatting the skip question and response options is shown below.
Based on our findings, and in keeping with our recommendations, we reworded the survey questions to be in the affirmative and incorporated additional response options to address discomfort in responding honestly on binary measures. Instead of asking whether participants had vaginal intercourse without the specified method(s) in the past three months, we asked how often they had used the method(s) in the past three months: “all of the time,” “most of the time,” “some of the time,” or “none of the time.” Any sexually active respondent who used contraception less than “all of the time” was coded as having had unprotected sex.
We developed the following recommended survey questions:
1. In the past 3 months, have you used any of these methods of birth control? (yes/no responses)
a. Condom
b. Birth control pills
c. The shot (for example, Depo Provera)
d. The patch (for example, Ortho Evra)
e. The ring (for example, NuvaRing)
f. IUD (for example, Mirena, Skyla, or Paragard)
g. Implant (for example, Implanon or Nexplanon)
h. Other
If respondent answers “yes” to any of 1a-1g:
2. In the past 3 months, how often have you used any method of birth control, including condoms, during vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)
If respondent answers “yes” to 1a:
3. In the past 3 months, how often have you used condoms during vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)
If respondent answers “yes” to any of 1b-1g:
4. In the past 3 months, how often did you use at least one of the following methods when you had vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)
a. Birth control pills
b. The shot (for example, Depo Provera)
c. The patch (for example, Ortho Evra)
d. The ring (for example, NuvaRing)
e. IUD (for example, Mirena, Skyla, or Paragard)
f. Implant (for example, Implanon or Nexplanon)
A separate document with these questions, along with coding directions to create binary measures of unprotected sex, can be found here.
Our findings from the second round of cognitive interviews suggest that the revisions improved participants’ comprehension of questions and their willingness to report instances of unprotected sex. Almost all participants understood the new questions and answered the way we would expect them to, based on their descriptions of their method use in the past three months. Participants also said that someone who used a condom or birth control method every time except once would likely answer “most of the time” or “some of the time” to the new measure of contraceptive use, instead of reporting (inaccurately) that they did not have sex without a method in the past three months on the previous measure.
Our analyses of the six-month follow-up survey data also found lower rates of inconsistency across the new survey questions. Ten percent of respondents had inconsistent responses across the original questions, while only 4 percent had inconsistent responses across the new questions. Additionally, we found higher reports of unprotected sex based on the new survey questions, as compared to the original survey questions, possibly because the option to report using a method “most of the time” or “some of the time” reduced the social desirability bias that may make young women reluctant to report having had unprotected sex.
Our work highlights that cognitive interviews are an effective way to assess participants’ understanding of survey questions and provide insight into reporting inconsistencies. In this study, cognitive interviews allowed program participants to directly inform the wording of questions, thus reducing their discomfort when taking the survey and increasing the accuracy of responses. Improved participation and response quality, in turn, can improve the reliability of evaluation findings.
Our research expands on existing research on the measurement of unprotected sex among teens. In line with findings from other studies, we found that young women had difficulty responding to a dichotomous measure of unprotected sex. Teens, especially those enrolled in a teen pregnancy prevention program, recognize that only one answer in a dichotomous yes/no scenario would be socially perceived as “desirable.” Teens who generally feel that their behavior aligns with what is considered socially desirable (i.e., using contraception)—including, perhaps, those teens who have only had sex one time without contraception but are otherwise consistent users of contraception—are likely to choose the socially desirable response because it fits with their self-perception. Providing more response options allows participants to identify a response that aligns with their behavior (e.g., using contraceptive most of the time or some of the time) and yields more accurate responses. The new questions allow evaluators to assess binary measures of unprotected sex, but also to examine nuances in consistency of method use. This aligns with calls from other researchers to move away from dichotomous measures of perfect contraceptive use. For example, a program can still be considered successful if participants increase the consistency of their contraceptive use, even if they do not achieve 100 percent contraceptive consistency.
The use of a broader range of response options in questions about contraceptive consistency may also help program implementers identify interim changes in behavior and help them tailor their message to the specific progress of program participants. For example, if an implementer notices at an interim survey that many participants have gone from using contraception none of the time to using it most of the time, they can reinforce the importance of contraceptive consistency. If they see that their participants are reporting using contraception most of the time or all of the time (on either a baseline or interim survey), they may choose to dispel common myths that lead teens to discontinue contraceptive use.
We would like to thank Mónica Caudillo from the University of Maryland and Elizabeth Cook from Child Trends for their thoughtful review and feedback on an initial draft of this research, and Kate Welti from Child Trends for her careful review of the research brief. We also thank Genevieve Martínez-García and Milagros Garrido of Healthy Teen Network for the development of Pulse.
This publication was made possible by Grant Number TP2AH000038 from the Office of Population Affairs (OPA), U.S. Department of Health and Human Services (HHS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the OPA or HHS.
[1] Evaluators of teen pregnancy prevention programs often measure unprotected sex, rather than the percentage of teens using contraception, in order to include teens who are not sexually active in the analysis (these respondents can be coded as 0 rather than dropped from the analysis).
© Copyright 2024 ChildTrendsPrivacy Statement
Newsletter SignupLinkedInThreadsYouTube