research study retention strategies

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 4, Issue 2
• Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• V C Brueton 1 ,
• J F Tierney 1 ,
• S Stenning 1 ,
• S Meredith 1 ,
• S Harding 2 ,
• I Nazareth 3 ,
• 1 MRC Clinical Trials Unit at UCL, London , UK
• 2 Ethnicity and Health, MRC Social and Public Health Sciences Unit , 4 Lilybank Gardens, Glasgow , UK
• 3 PRIMENT Clinical Trials Unit, Research Department of Primary Care and Population Health , UCL Medical School, London , UK
• Correspondence to Brueton VC; v.brueton{at}ucl.ac.uk

Objective To quantify the effect of strategies to improve retention in randomised trials.

Design Systematic review and meta-analysis.

Data sources Sources searched: MEDLINE, EMBASE, PsycINFO, DARE, CENTRAL, CINAHL, C2-SPECTR, ERIC, PreMEDLINE, Cochrane Methodology Register, Current Controlled Trials metaRegister, WHO trials platform, Society for Clinical Trials (SCT) conference proceedings and a survey of all UK clinical trial research units.

Review methods Included trials were randomised evaluations of strategies to improve retention embedded within host randomised trials. The primary outcome was retention of trial participants. Data from trials were pooled using the fixed-effect model. Subgroup analyses were used to explore the heterogeneity and to determine whether there were any differences in effect by the type of strategy.

Results 38 retention trials were identified. Six broad types of strategies were evaluated. Strategies that increased postal questionnaire responses were: adding, that is, giving a monetary incentive (RR 1.18; 95% CI 1.09 to 1.28) and higher valued incentives (RR 1.12; 95% CI 1.04 to 1.22). Offering a monetary incentive, that is, an incentive given on receipt of a completed questionnaire, also increased electronic questionnaire response (RR 1.25; 95% CI 1.14 to 1.38). The evidence for shorter questionnaires (RR 1.04; 95% CI 1.00 to 1.08) and questionnaires relevant to the disease/condition (RR 1.07; 95% CI 1.01 to 1.14) is less clear. On the basis of the results of single trials, the following strategies appeared effective at increasing questionnaire response: recorded delivery of questionnaires (RR 2.08; 95% CI 1.11 to 3.87); a ‘package’ of postal communication strategies (RR 1.43; 95% CI 1.22 to 1.67) and an open trial design (RR 1.37; 95% CI 1.16 to 1.63). There is no good evidence that the following strategies impact on trial response/retention: adding a non-monetary incentive (RR=1.00; 95% CI 0.98 to 1.02); offering a non-monetary incentive (RR=0.99; 95% CI 0.95 to 1.03); ‘enhanced’ letters (RR=1.01; 95% CI 0.97 to 1.05); monetary incentives compared with offering prize draw entry (RR=1.04; 95% CI 0.91 to 1.19); priority postal delivery (RR=1.02; 95% CI 0.95 to 1.09); behavioural motivational strategies (RR=1.08; 95% CI 0.93 to 1.24); additional reminders to participants (RR=1.03; 95% CI 0.99 to 1.06) and questionnaire question order (RR=1.00, 0.97 to 1.02). Also based on single trials, these strategies do not appear effective: a telephone survey compared with a monetary incentive plus questionnaire (RR=1.08; 95% CI 0.94 to 1.24); offering a charity donation (RR=1.02, 95% CI 0.78 to 1.32); sending sites reminders (RR=0.96; 95% CI 0.83 to 1.11); sending questionnaires early (RR=1.10; 95% CI 0.96 to 1.26); longer and clearer questionnaires (RR=1.01, 0.95 to 1.07) and participant case management by trial assistants (RR=1.00; 95% CI 0.97 to 1.04).

Conclusions Most of the trials evaluated questionnaire response rather than ways to improve participants return to site for follow-up. Monetary incentives and offers of monetary incentives increase postal and electronic questionnaire response. Some strategies need further evaluation. Application of these results would depend on trial context and follow-up procedures.

• Randomised trials

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/

https://doi.org/10.1136/bmjopen-2013-003821

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

This is the most comprehensive review of strategies specifically designed to improve retention in randomised trials, including many unpublished trials and data.

Although our searches were extensive, some less well reported, ongoing, or unpublished trials, or trials conducted outside the UK might have been missed.

Most of the evidence relates to increasing questionnaire response rather than ways to increase return of participants to sites.

Introduction

Loss of participants during trial follow-up can introduce bias and reduce power affecting the generalisability, validity and reliability of results. 1 , 2 If losses are fewer than 5% they may lead to minimum bias, while 20% loss can threaten trial validity. 2 Missing data from losses to follow-up can be dealt with statistically; however, the risk of bias can remain. 3

Trialists adopt various strategies to try to improve retention and generate maximum data return or compliance to follow-up procedures. These strategies are designed to motivate and keep participants or site clinicians engaged in a trial, but many are untested. 4 , 5 A systematic review of strategies to retain participants in cohort studies suggests that providing incentives can improve retention. 6 Edwards’ systematic review on methods to increase response rates to postal and electronic questionnaires across a range of study types found that including monetary incentives, keeping the questionnaire short and contacting people before questionnaires were sent were ways to increase response rates. 7 However, heterogeneity of effects was an issue and it is unclear which strategies are applicable to randomised trials. Moreover, reasons for loss to follow-up in cohort studies and surveys may differ from randomised trials. In trials, participants may be randomised to a study arm that is not their preferred choice and so strategies that improve retention in other study types cannot necessarily be extrapolated to randomised trials.

As loss to follow-up can compromise the validity of findings from randomised trials, delay results and potentially increase trial costs, we conducted a systematic review to assess the effect of strategies to improve retention in randomised trials.

The methods were prespecified in the Cochrane review protocol. 8

Trials included

We included randomised trials that compared strategies to increase participant retention embedded in ‘host’ randomised trials across disease areas and settings. These strategies should have been designed for use after participants were recruited and randomised. Retention trials embedded in cohort studies and surveys were excluded.

Identification of retention trials

We searched MEDLINE (1950 to May 2012), EMBASE (1980 to May 2012), PsycINFO (1806 to May 2012), DARE (to May 2012), Cochrane CENTRAL and CINAHL (1981 to May 2012) using randomised controlled trial filters, where possible, and free text terms for retention (see online supplementary appendix 1 MEDLINE search). C2-SPECTR (to May 2009) and ERIC (1966 to May 2009) were only searched to May 2009 because of difficulties encountered with database and search platform changes. PreMEDLINE was searched to May 2009 but not subsequently because the free text records ultimately appear in MEDLINE. For search updates we also included the Cochrane Methodology Register, Current Controlled Trials metaRegister of Controlled Trials and WHO trials registry. Reference lists of relevant publications, reviews, included studies and abstracts of Society for Clinical Trials meetings from 1980 to 2012 were also reviewed. No language restrictions were applied. All UK clinical trial units were surveyed to identify further eligible trials and the review was advertised at the Society for Clinical Trials Meeting in 2010.

Trial selection

Two reviewers (VCB and GR) independently screened potentially eligible trials with disagreements resolved by a third author (SS). Information was sought from investigators to clarify eligibility where this was unclear.

Data extraction

Data were extracted for each retention and host trial by one author (VCB) and checked by another (JFT). For retention trials, data were extracted on start time in relation to the host trial, aim, primary outcome, follow-up type, strategy to improve retention and comparator/s, including the frequency and time the strategy was administered, and numbers randomised, included and retained at the primary analysis. Data on sequence generation, allocation concealment, blinding and outcome reporting were extracted for each retention trial to assess risk of bias. 9 Data extracted for each host trial were: aim, comparators, primary outcome, disease area and setting. In addition, information on the sequence generation and allocation concealment was extracted to confirm that host trials were randomised. Missing or ambiguous data were queried or obtained through contact with trial authors.

Statistical analysis

Retention was the primary outcome. Most of the retention strategies were applied during follow-up for the host trial. For three host trials, the retention strategy was applied in further follow-up of trial participants after completion. For four host trials, the strategy was applied during the pilot phase, and, for one other host trial, the retention strategy was applied before the host trial started. Where retention trials specified the primary outcome as the retention rate at a particular time point, this was used in the analysis. Where trials reported retention at multiple time points, without specifying which one was the primary outcome, we used the earliest time point in the analysis to see the initial impact on retention or response of introducing the strategy. Where trials reported time to retention, without specifying the primary time point, we used the final time point in the analysis, taking account of any censoring if data were available.

Retention trials with insufficient data could not be included in meta-analyses and were described qualitatively. Otherwise, risk ratios and their 95% CIs for retention were used to determine the effect of strategies on this outcome. The participant was the unit of analysis. Where clustering was ignored in the analysis of cluster randomised trials, we inflated the SEs using the intraclass correlation coefficients from appropriate external sources. 10–12

For factorial trials 13 , 14 that investigated different categories of strategies to improve retention, we included all trial comparisons in the relevant analyses and labelled these accordingly. For one factorial trial, 15 where the data were not available to perform this, only the broad trial comparisons (main effects) were included in the analyses. Where there were multiple comparisons in a single trial 16 within the same category of strategy, to avoid double counting, the intervention arms were combined and compared with the control arm. Similarly, for three-armed trials 17 , 18 that compared two similar intervention arms with one control arm, the intervention arms were combined and compared with the control arm. For these trials, we also compared each intervention arm with the control arm, as separate trial comparisons, in exploratory analyses. Note that these approaches resulted in more trial comparisons than trials.

Heterogeneity was examined by the χ 2 test, at 0.10 level of significance, and the I 2 statistic, 19 and explored through subgroup analyses. If there was no substantial heterogeneity, risk ratios were pooled using the fixed-effect model, but if heterogeneity was detected and was not explained by subgroup or sensitivity analyses, we did not pool the results. If heterogeneity could not be explained, we used the random effects model to assess the robustness of the results to the choice of model. To assess the robustness of the results, sensitivity analyses were conducted that excluded quasi-randomised trials.

The diversity of trials and interventions identified meant that not all of our prespecified subgroup analyses were appropriate or possible. Therefore, different types of strategies were analysed separately and new subgroups were defined within these strategies prior to analysis. These new analyses are listed in tables 1 ⇓ ⇓ – 4 .

• View inline

Characteristics of included incentive trials

Characteristics of included communication trials

Characteristics of included trials evaluating new questionnaire strategies

Characteristics of other trials

The absolute benefits of effective retention strategies were based on applying meta-analysis risk ratios to representative control arm retention rates. 20 All statistical analyses were conducted using RevMan5.

We identified 38 eligible randomised retention trials from 24 304 records (see online supplementary figure S4). Retention trials were embedded in host trials, details of which are provided in the full Cochrane review. 12 Twenty-eight retention trials were published in full, 13–18 21–40 two in the grey literature 14 , 34 and eight are unpublished ( unpublished trials by Edwards, Svobodva, Letley, Maclennan, Land, Bailey 1, Bailey 2, Marson ). Unpublished trials were identified by word of mouth, reference lists of relevant literature and a survey of UK clinical trials units. Four retention trial publications contained two trials each. 18 , 32 , 33 , 35

Participants and settings

Eligible retention trials were from different geographical areas and clinical settings. Clinical areas ranged from exercise and alcohol dependency to treatment and screening for cancer ( tables 1 ⇑ ⇑ – 4 ). 12

Outcomes for strategies to improve retention were measured by: return of postal or electronic questionnaires 13–15 , 18 , 21 , 22 , 24 , 25 , 27 , 29–34 36–40 ( unpublished trials by Edwards, Svobodva, Letley, Maclennan, Land, Bailey 1, Bailey 2 Marson ) or biomedical data 17 ( Bailey 1, unpublished ) , a combination of postal, telephone and email follow-up 35 or face to face follow-up/retention. 16 , 28

Design of included retention trials

One retention trial was cluster randomised ( Land unpublished ), four were factorial trials 13–16 and there was one three-armed 17 and three four-armed trials. 18 , 32 Five trials were quasi-randomised, 16 , 28 , 29 , 33 allocating participants by either their identification numbers, 28 , 29 day of clinic visit 16 or by random selection of half the sample for the intervention and half for the control group. 33 All strategies targeted individual trial participants except one which targeted sites ( Land unpublished ).

Twenty-nine retention trials started during follow-up of the host trial 13 , 15 , 16 , 18 , 21 , 22 , 24–36 , 38 , 41 ( Edwards, Land, Maclennan, Bailey, Svoboda, unpublished ). One trial followed children of mothers who participated in the MRC ORACLE trial. 39 Two trials followed up participants in smoking cessation trials after the host trial finished. 17 , 40 Another retention trial randomised participants before the host trial started. 23 Four trials started during the pilot phase of the host trial 18 , 32 , 37 ( Letley unpublished ). For one trial, it is unclear when the retention trial started in relation to the host trial. 14

Incentive strategies

There were 14 retention trials of incentives and 19 trial comparisons. Thirteen trials investigating incentive strategies targeted questionnaire response, with only one targeting participant retention. 16 Incentive strategies aimed at improving questionnaire response were: vouchers, 18 , 29 , 39 cash, 25 a charity donation, 18 entry into a prize draw, 14 , 18 , 30 cheque 17 offers of study results 24 , 40 and a certificate of appreciation. 15 , 16 Incentive strategies aimed at participant retention were: lapel pins and a certificate of appreciation. 16 The UK incentives ranged in value from £5 to £20 18 , 29 , 39 ( Bailey 1, Baily 2, unpublished ) and from $2 to $10 for US-based trials, and were provided as either cash or voucher. Offers of entry into prize draws ranged from £25 to £250 for UK 18 , 30 and US$50 for US-based trials 14 ( table 1 ); there was no information available on the chance of winning a prize. One trial evaluated giving a monetary incentive with a promise of a further incentive for return of trial data ( Bailey 2 unpublished ).

Communication strategies

There were 14 retention trials of communication strategies and 20 trial comparisons. Most of the communication strategies targeted questionnaire response, with only one targeted at the return of biomedical test kits. 35 Strategies evaluated were: enhanced letters, that is, those with additional information about trial processes or with an extra feature, for example, signed by a principal investigator 15 ( Marson unpublished ), use of additional telephone reminders 35 ( Maclennan unpublished ), a calendar including reminders of when to return a questionnaire 34 , text and/or email reminders 21 , 31 , 35 and reminders to sites of upcoming assessments versus no additional reminder ( Land unpublished ). One trial used a package of postal communication strategies called the Total Design Method (TDM) 37 and another used recorded delivery of questionnaires 38 ( table 2 ).

Five trials evaluated communication and incentive strategies 13–15 , 25 , 35 ( tables 1 and 2 ). The incentives were: certificates of appreciation for study involvement, 15 study-branded pens, 13 a US$2 coin 14 and a US$5 bill 25 or fridge magnets. 35 The communication strategies were: first or second class outward post, 13–15 stamped and business reply envelopes, 13 letters signed by different study personnel, 15 letters posted at different times, 15 telephone survey 25 and text messages. 35

New questionnaire formats

The effect of a change in questionnaire format on response to questionnaires was evaluated in eight trials. The 10 comparison formats evaluated were ( table 3 ): questionnaire length 27 , 32 , 36 ( Edwards unpublished Svoboda unpublished ), order of questions ( Letley unpublished 33 ) and relevance of questionnaires in the context of research in alcohol dependence 32 .

Behavioural strategies

There were two retention trials of motivational behavioural strategies, one in an exercise trial 26 and another in a parenting trial 23 ( table 4 ). A behavioural strategy was defined as giving participants information about goal setting and time management to facilitate successful trial completion. One retention trial was run prior to the host trial, 23 where only participants who completed the orientation/retention trial were included in the subsequent parenting trial.

Case management

Case management defined as outreach, service planning linkage, monitoring and advocacy was compared with usual follow-up in a cancer screening trial 28 ( table 4 ). This strategy involved trial assistants managing participant follow-up by arranging services to enable participants to keep trial follow-up appointments.

Methodology strategies

One trial included an open trial versus blind trial design to evaluate the impact on questionnaire response 22 ( table 4 ).

Trials not included in the meta-analyses

Two included trials could not be included in the meta-analysis 30 ( Letley unpublished ). For one trial, the host trial participants included randomised and non-randomised participants 30 and the author confirmed that the participants in the retention trial were from both cohorts and these data could not be separated. For the other trial, retention trial outcome data were not available ( Letley unpublished ).

Risk of bias in included trials

Twenty-four trials describe adequate sequence generation 15 , 16 , 18 , 22–24 , 26 , 30–32 , 34 , 35 , 37 , 39 , 40 ( unpublished trials Bailey 1 , Bailey 2, Letley, Land, Maclennan, Marson ). There was insufficient information about the sequence generation for 10 trials, but they were all described as randomised 13 , 14 , 17 , 21 , 25 , 27 , 36 , 38 ( Edwards, Svoboda unpublished ). Five trials used quasi-randomisation. 16 , 28 , 29 , 33 Fifteen trials reported adequate sequence generation and allocation concealment 18 , 22 , 24 , 26 , 31 , 32 , 34 , 39 , 40 ( Letley, Maclennan, Bailey 1 , Bailey 2, unpublished ).

Blinding of participants to the intervention was not possible for incentive strategies, offers of incentives, behavioural or case management strategies and different types of communication and questionnaire format strategies. For one trial that evaluated the effect of a blind versus open design on retention this was not applicable. 22 For some trials, participants were aware of the intervention but unaware of the evaluation 14 , 16 , 23 , 30 , 33 , 39 ( Maclennan, Marson unpublished ). For another trial, 26 exercise sessions were not separated according to the behavioural intervention, that is, walking and swimming, and potential contamination between groups could have led to bias. For other trials, blinding of participants or trial personnel to the outcome or intervention was not reported. The primary outcome measure for this review was retention, and this was well reported. Authors were contacted for clarification of any exclusions after randomisation if this was unclear from retention trial reports. Although retention trial protocols were not available for included trials, the published and unpublished reports included reported all expected outcomes for retention.

The effects of strategies

There were 14 retention trials of incentives, 19 trial comparisons with 16 253 comparisons. Across incentive subgroups, there was considerable heterogeneity (p<0.00001; figure 1 A). So we did not pool the results for incentives. Unless otherwise stated, results from the random effects model were similar. Three trials (3166 participants) that evaluated the effect of giving monetary incentives to participants showed that the addition of monetary incentives is more effective than no incentive at increasing response to postal questionnaires (RR=1.18; 95% CI 1.09 to 1.28; p<0.0001, heterogeneity p=0.21; figure 1 A). A sensitivity analysis excluding the quasi-randomised trial by Gates et al 29 shows a similar effect (RR=1.31; 95% CI 1.11 to 1.55; p=0.002). Also, based on two web-based trials (3613 participants, figure 1 A), an offer of a monetary incentive promotes greater return of electronic questionnaires than no offer (RR=1.25; 95% CI 1.14 to 1.38, p<0.00001, heterogeneity p=0.14). However, a single trial comparison suggests that an offer of a monetary donation to charity does not increase response to electronic questionnaires (RR=1.02, 95% CI 0.78 to 1.32; p=0.90; figure 1 A).

• Download figure
• Open in new tab
• Download powerpoint

(A) Incentive strategies: main analysis addition of incentive versus no incentive, (B) incentives: addition of £20 vs £10 incentive, (C) incentives addition of: monetary incentive versus offer of entry into prize draw.

On the basis of three trials (6322 participants), there is no clear evidence that the addition of non-monetary incentives improved questionnaire response (RR=1.00; 95% CI 0.98 to 1.02; p=0.91) but there is some heterogeneity (p=0.02; figure 1 A). A sensitivity analysis excluding the quasi-randomised trial by Bowen et al 16 showed a similar effect (RR=1.00; 95% CI 0.93 to 1.08; p=0.99, heterogeneity p=0.01). Two trials (1138 participants) evaluating offers of non-monetary incentives suggest that an offer of a non-monetary incentive is neither more nor less effective than no offer (RR=0.99; 95% CI 0.95 to 1.03; p=0.60; heterogeneity p=0.52) at improving questionnaire response ( figure 1 A).

In exploratory analyses, the different incentive arms that were combined for the main analysis do not appear to show differential effects (see online supplementary figure S5).

Two trials (902 participants) show that higher value incentives are better at increasing response to postal questionnaires than lower value incentives (RR 1.12; 95% CI 1.04 to 1.22; p=0.005; heterogeneity p=0.39) irrespective of how they are given ( figure 1 B).

Two trial comparisons (297 participants) provide no clear evidence that giving a monetary incentive is better than an offer of entry into a prize draw for improving response to postal questionnaires (RR=1.04; 95% CI 0.91 to 1.19; p=0.56, heterogeneity p=0.18; figure 1 C).

One trial could not be included in the analysis, 30 but showed a higher response in the group offered entry into a prize draw (70.5%) compared with the group not offered entry into the draw (65.8%).

There were 14 trials of communication strategies and 20 comparisons with 9822 participants. The communication strategies were so diverse that these were analysed separately.

Results from two trials (2479 participants) show that an enhanced letter is neither more nor less effective than a standard letter for increasing response to postal questionnaires (RR=1.01; 95% CI 0.97 to 1.05; p=0.70; heterogeneity p=0.80; figure 2 A). Although based on a single trial (226 participants), the TDM package seems much more effective than a customary postal communication method at increasing questionnaire return (RR=1.43, 95% CI 1.22 to 1.67; p<0.0001; figure 2 B). Based on the relevant arms of three trials (1888 participants), there is no clear evidence that priority post is either more or less effective than regular post at increasing trial questionnaire return (RR=1.02; 95% CI 0.95 to 1.09; p=0.55; heterogeneity p=0.53; figure 2 C).

Communication strategies: (A) enhanced versus standard letter, (B) total design versus customary post, (C) priority versus regular post, (D) additional reminders to participants versus usual follow-up, (E) telephone survey versus monetary incentive and questionnaire.

Six trials (3401 participants) evaluated the effect of different types of reminders to participants on questionnaire response. There is no clear evidence that a reminder is either more or less effective than no reminder (RR=1.03; 95% CI 0.99 to 1.06; p=0.13; heterogeneity p=0.73) at improving trial questionnaire response ( figure 2 D). One trial (700 participants) showed no clear evidence that a telephone survey is either more or less effective than a monetary incentive and a questionnaire for improving questionnaire response (RR=1.08; 95% CI 0.94 to 1.24; p=0.27; figure 2 E). Based on one cluster randomised trial (272 participants), a monthly reminder to sites of upcoming assessment was neither more nor less effective than the usual follow-up (RR=0.96; 95% CI 0.83 to 1.11; p=0.57). However, one small trial (192 participants) suggested that recorded delivery is more effective than a telephone reminder (RR=2.08; 95% CI 1.11 to 3.87; p=0.02). Based on one other trial (664 participants), there is no clear evidence that sending questionnaires early increased or decreased response (RR=1.10; 95% CI 0.96 to 1.26; p=0.19).

New questionnaire strategies

Eight trials with 10 comparisons (21 505 participants) evaluated the effect of a new questionnaire format on questionnaire response. Although there is only some heterogeneity between the questionnaire subgroups (p=0.11; figure 3 ), it did not seem reasonable to pool the results based on such different interventions.

Questionnaires: new format versus standard format.

Five trials (7277 participants) compared the effect of short versus long questionnaires on postal questionnaire response. There is only a suggestion that short questionnaires may be better (RR=1.04; 95% CI 1.00 to 1.08; p=0.07, heterogeneity p=0.14; figure 3 ). Based on one trial (900 participants), there is no clear evidence that long and clear questionnaires are more or less effective than shorter condensed questionnaires for increasing questionnaire response (RR=1.01, 0.95–1.07; p=0.86; figure 3 ). Two quasi-randomised trials (9435 participants) also show no good evidence that placing disease/condition questions before generic questions is either more or less effective than vice versa at increasing questionnaire response (RR=1.00, 0.97–1.02; p=0.75, heterogeneity p=0.44; figure 3 ). One trial by Letley ( unpublished ), not included in this analysis, provided no estimate of effect.

In the context of research on reducing alcohol consumption, there is also evidence that more relevant questionnaires, that is, those relating to alcohol use, increase response rates (RR 1.07; 95% CI 1.01 to 1.14; p=0.03, figure 3 ).

Behavioural/motivational strategies

Two community-based trials (273 participants) show no clear evidence that the behavioural/motivational strategies used are either more or less effective than standard information for retaining participants (RR=1.08; 95% CI 0.93 to 1.24; p=0.31, heterogeneity p=0.93).

Case management strategies

One trial (703 participants) evaluated the effect of intensive case management procedures on retention. There is no evidence that intensive case management is either more or less effective than usual follow-up in the population examined (RR=1.00; 95% CI 0.97 to 1.04; p=0.99).

One fracture prevention trial (538 participants) evaluated the effect of participants knowing their treatment allocation (open trial) compared with participants blind/unaware of their allocation on questionnaire response. The open design led to higher response rates (RR=1.37; 95% CI 1.16 to 1.63; p=0.0003).

Absolute benefits of strategies to improve retention

The absolute benefits of effective strategies on typical questionnaire response are illustrated in table 5 . Based on a 40% baseline response rate for postal questionnaires, the addition of a monetary incentive is estimated to increase response by 92 questionnaires/1000 sent (95% CI 50 to 131). With a baseline response rate of 30%, as seen in the included online trial, the addition of an offer of a monetary incentive is estimated to increase response by 140 questionnaires/1000 sent (95% CI 86 to 193).

Absolute benefit of effective strategies to improve retention

Thirty-eight randomised retention trials were included in this review, evaluating six broad types of strategies to increase questionnaire response and retention in randomised trials. Trials were conducted across a spectrum of disease areas, countries, healthcare and community settings ( tables 1 ⇑ ⇑ – 4 ). Strategies with the clearest impact on questionnaire response were: addition of monetary incentives compared with no incentive for return of postal questionnaires, addition of an offer of a monetary incentive when compared with none for return of electronic questionnaires and an offer of £20 vouchers when compared with £10 for return of postal questionnaires and biomedical test kits. The evidence was less clear about the effect of shorter questionnaires rather than longer questionnaires and for questionnaires of greater relevance to the questions being studied. Recorded delivery of questionnaires, the TDM, a ‘package’ of postal communication strategies with reminder letters and an open trial design appear more effective than standard procedures. These strategies were tested in single trials and may need further evaluation. The addition of a non-monetary incentive or an offer of a non-monetary incentive compared with no incentive did not increase or decrease trial questionnaire response. ‘Enhanced’ letters, letters delivered by priority post or additional reminders were also no more effective than standard communication. Altering questionnaire structure does not seem to increase response. No strategy had a clear impact on increasing the number of participants returning to sites for follow-up.

Strengths and weaknesses

This is the most comprehensive review of strategies specifically designed to improve retention in randomised trials, including many unpublished trials and data. Although our searches were extensive, some less well-reported, ongoing, or unpublished trials, or trials conducted outside the UK might have been missed.

Most of the trials used appropriate methods for randomisation or at least stated that they were randomised. For trials that did not describe their methods well or provide further information, there remains a potential risk of selection bias. Sensitivity analyses excluding quasi-randomised trials did not affect the results. In this context, where motivating participants to provide data or attend clinics is often the target of the interventions and so appropriately influences the outcome; lack of blinding is less of a concern. Retention is the outcome and was obtained for all but two trials therefore, attrition and selective outcome reporting bias are probably unimportant. Although the retention trials were fairly well conducted, this could be improved, and they were often poorly reported. This may be because they were designed when loss to follow-up became a problem in a trial, rather than pre-planned prior to the start of the host trial.

Few trials are available for behavioural, case management and methodological strategies (only 1 or 2 each) and this affects the power of the result for these strategies. The use of open trials to increase questionnaire response can only be applied to trials where blinding is not required; based on our result, this strategy would need to be evaluated in different trial contexts if it were to be applied in other areas. All included trials were conducted in higher income countries. Therefore, the effective strategies may not be socially, culturally or economically appropriate to trials conducted in low-resource settings. The diversity of strategies and the low number of trials meant that we could not examine the impact of, for example, trial setting and disease area as planned. Moreover, most of the evidence relates to increasing questionnaire response rather than participant retention in follow-up. Many trials require participants to return to sites for follow-up and monitoring; however, barriers to follow-up do exist and are trial and participant specific depending on the disease area, treatment and population group. Return for follow-up at sites depends on participant preferences and the demands of the trial. 42 Barriers to follow-up at site could be alleviated by using tailored strategies to encourage participants to return to sites for follow-up and monitoring. Studies that evaluate such strategies are particularly needed.

Edwards’ extensive review of methods to increase response to postal and electronic questionnaires found that monetary incentives and recorded delivery of questionnaires improved response. 7 However, unlike our review, they also found that non-monetary incentives, shorter questionnaires, use of handwritten addresses, stamped return envelopes (as opposed to franked return envelopes) and first class outward mailing were effective. We did, however, find that a ‘package’ including an enhanced letter with several reminders was effective. The trials included in the Edwards’ review were embedded in surveys, cohort studies and trials, and there was substantial heterogeneity in the results, which was not a particular problem in this review. 7 Moreover, we included 8 unpublished trials and 18 other trials not included by Edwards. 12

Nakash et al ’s 43 small systematic review of ways to increase response to postal questionnaires in healthcare was not exclusive to randomised trials. They found reminder letters, telephone contact and short questionnaires increased response to postal questionnaires. There was no evidence that incentives were effective. A systematic review of methods to increase retention in population-based cohort studies had no meta-analysis, but suggested that incentives were associated with increased retention. 6

Prior to our review, it was not clear which, if any, of these strategies could be extrapolated to randomised trials. We also identified additional strategies that may improve trial questionnaire response or retention, for example, methodological strategies.

Implications

Although giving monetary incentives upfront seems effective, offering and giving these after receipt of data could be a cost-effective strategy, because those not returning questionnaires would not receive an incentive. The addition of non-monetary incentives, for example, lapel pins and certificates of appreciation, or offers of these, did not increase response or retention, perhaps because these items are not valued by participants. Offers of monetary incentives were also an effective strategy in the context of an online electronic questionnaire, thus, it would be beneficial for trialists to know which is more effective: an offer of a monetary incentive or an upfront monetary incentive in a head-to-head trial comparison.

The value of incentives used in the UK evaluations ranged from GBP5 to GBP20 and for US-based studies was US$2 to US$10. For offers of entries into prize draws, the values were higher, ranging from GBP25 to GBP250 for the UK prize draws and US$50 for US-based prize draws. The value of monetary incentive should not be so high as to be perceived as payment or coercion for data but more as an appreciation for efforts made by participants. A cost-effectiveness analysis for additional responses gained after incentive strategies were introduced was reported for only some incentive trials. 18 , 25 , 29 , 30 , 39 As costs increase the cost-benefit associated with incentive strategies would need to be updated if incentives were to be used to improve retention in future trials.

Priority post, enhanced letters (eg, signed by the principal investigator) and different types of additional reminders are used by trialists in current research practice, but these were not found to be effective. The former may not be considered important and too many reminders, over and above standard procedures, could be counterproductive.

Although appearing very effective, the TDM for postal questionnaires could be labour-intensive to implement, expensive and may no longer be applicable to some participant groups, for example, young people used to other modes of communication, or in trials using email, text or online data collection. Recorded delivery could be useful to ensure trial follow-up supplies to reach their intended destination, but careful planning to avoid inconvenience for the participant might be necessary. Open trials to increase questionnaire response can only be used where blinding is not required. This could be counterproductive, however, as unblinded trials can cause biased outcome assessment or loss to follow-up if a participant or clinician has a treatment preference.

Questionnaire length and relevance may need further evaluation as there is only a suggestion that these are effective in the context of randomised trials. Also, telephone follow-up compared with a monetary incentive sent with a questionnaire needs further evaluation possibly with a cost-benefit analysis as both could be expensive in time and human resources. Evaluations of strategies that encourage participants to return to sites for follow-up visits and monitoring are particularly needed because many trials collect outcome data in this way.

Trialists should consider including well thought out and adequately powered evaluations of strategies to increase retention in randomised trials with clear definitions of retention strategies and retention measures. Trialists could incorporate evaluations of strategies to improve retention at the design stage so that power, sample size and funding are taken into account. Retention trials were often poorly reported and trialists should adhere to the consort guidelines for trial reporting to facilitate the synthesis of results in future methodology reviews.

There is less research on ways to increase return of participants to trial sites for follow-up and on the effectiveness of strategies to retain trial sites in cluster and individual randomised trials. Research in both areas would be very beneficial to trialists. Application of the results of this review would depend on trial setting, population, disease area, budget allowance and follow-up procedures.

Conclusions

Trialists should consider using monetary incentives and offers of monetary incentives to increase postal and electronic questionnaire response, depending on trial setting, population, disease area, budget and usual follow-up procedures.

Future evaluations of retention strategies in randomised trials should be carefully planned and adequately powered, and the retention strategies and measures of retention should be clearly defined. More research on ways to increase return of participants to sites for follow-up and on ways to retain sites in cluster and individual randomised trials are also needed.

Acknowledgments

The authors would like to thank the following: all authors of included published trials for providing extra unreported data; principal investigators for data on trials in progress or completed and unpublished (Julia Bailey UCL for data for Bailey 1 and Bailey 2; Graeme MacLennan for data for MacLennan; Stephanie Land data for Land) and the coordinators of the UK Clinical Trials Units who responded to our survey with information about ongoing and/or unpublished completed trials. The authors also thank Cara Booker, SPHRU, for search strategy information; Angela Young, Librarian UCL, for assistance with searching databases and Ian White, MRC Bio Statistics Unit Cambridge, for helpful comments on the analysis of multiarm trials. They also acknowledge Shaun Treweek, Mike Clarke, Andy Oxman, Karen Robinson and Anne Eisinga for comments on the review protocol; and Phil Edwards, Elie Akl, Lisa Maguire, Jean Suvan, Shaun Treweek, Mike Clarke and Karen Robinson for comments on the review.

• Fewtrell MS ,
• Kennedy K ,
• Singhal A ,
• Schulz KF ,
• Robinson KA ,
• Dennison CR ,
• Wayman DM ,
• Harding S ,
• Edwards PJ ,
• Roberts IG ,
• Clarke MJ ,
• Brueton VC ,
• Tierney J ,
• Higgins J ,
• ↵ University of Aberdeen. Aberdeen ICCs . 2013 . Ref Type: Online Source. http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/ .
• Stenning S ,
• Cochran C ,
• Cotton SC ,
• Dennis CL ,
• Renfroe EG ,
• Heywood G ,
• Foreman L ,
• Thornquist M ,
• Goodman G ,
• Rezaishiraz H ,
• Khadjesari Z ,
• Kalaitzaki E ,
• Higgins JPT ,
• Thompson SG ,
• Schunemann H ,
• Avenell A ,
• Chaffin M ,
• Funderburk B ,
• Cockayne S ,
• Torgerson D
• Couper PM ,
• Peytchev A ,
• Strecher JV ,
• Beilin LJ ,
• Slattery J ,
• Farrell B ,
• Havstad S ,
• Vernon SW ,
• Williams M ,
• Withers E ,
• Leigh Brown AP ,
• Kennedy A ,
• Tilbrook HE ,
• Jayakody S ,
• McCambridge J ,
• McColl EM ,
• Eccles MPM ,
• Rousseau NSB ,
• Ziegler RG ,
• Thompson FEJ ,
• Sutherland HJ ,
• Nazareth I ,
• ↵ Hughes JFR. Free reprints to increase the return of follow-up questionnaires. Abstracts from the Tenth Meeting of the Society of Clinical Trials, 15–18 May 1989; Minneapolis. 1989;10:316–54 .
• Appleton R ,
• Counsell C ,
• Jorstad-Stein E ,

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Files in this Data Supplement:

• Data supplement 1 - Online supplement

This article is based on a Cochrane Review published in the Cochrane Database of Systematic Reviews (CDSR) 2013, Issue 12, doi:10.1002/14651858.MR 000032.pub2 (see www.thecochranelibrary.com for information)

Contributors VCB wrote the protocol for the review with comments from JFT, GR, SS, SM, IN and SH. JFT and VCB designed the searches with comments from SH. VCB conducted the searches, screened all abstracts, and full articles of potentially eligible trials. VCB and GR screened potentially eligible trial articles. SS acted as a third reviewer. Data extraction was conducted by VCB and checked by JT. JT designed the analysis plan with VCB. VCB conducted the analysis with advice on interpretation of results from JFT, SS, IN and GR. VCB wrote the first draft of the review with critical comments from all authors.

Funding This project was funded by the Medical Research Council Population Health Sciences Research Network grant number PHSRN 30.

Competing interests None.

Provenance and peer review Not commissioned; externally peer reviewed.

Data sharing statement No additional data are available.

Read the full text or download the PDF:

Customize Your Path

Filters Applied

Customize Your Experience.

Utilize the "Customize Your Path" feature to refine the information displayed in myRESEARCHpath based on your role, project inclusions, sponsor or funding, and management center.

Participant and patient engagement, recruitment, and retention

Need assistance with participant engagement, recruitment, or retention?

Get help with participant engagement, recruitment, or retention.

Contact the Recruitment Innovation Center (RIC):

• Submit a help request

Successful participant engagement, recruitment, and retention in research studies  can be one of the most challenging aspects of conducting research. Effective and efficient engagement, recruitment, and retention strategies are critical to successfully achieving enrollment targets while also prioritizing participant safety, well-being and trust.

This page provides resources, best practices and tools for developing and implementing robust engagement, recruitment, and retention strategies. However, it’s important to note that there is no one-size-fits-all approach to good planning. In fact, effective engagement, recruitment and retention planning should:

• Be proactive
• Start upstream at study design
• Anticipate and account for downstream barriers
• Be data-driven and evidence-based
• Be thoughtful and realistic
• Include input from all relevant stakeholders
• Be participant-centered
• Be resourced appropriately
• Be adjusted as needed

Request a free consultation with the Recruitment Innovation Center or consider the Consultations and Help resources available in the “Related Resources” section.

Population or cohort discovery

Participant or patient engagement and recruitment strategies.

Once the cohort or population has been identified, recruitment plans should consider which strategies and messaging are most appropriate to engage the intended audience, while also adhering to Federal Regulations and institutional policies and guidance.

Before identified potential participants can be recruited engagement must happen. Engagement, in this sense, is about delivering content to the right person, at the right time, with the right frequency, through the most effective channel.

True stakeholder engagement should happen as early in the study design process as possible. However, that’s not always an option when Duke is a participating site in “someone else’s research study” – when this is the case, we think of engagement as the strategies used to share information about a study with potential participants. This may include one or more of the following channels:

• Duke Health Research Volunteer Registry
• ResearchMatch
• Social media (e.g., Facebook ads)
• Print media (e.g., flyers, brochures, etc.)
• Online/digital content (e.g., a study website or search engine ads)
• Radio or TV spots
• Letters or emails
• Phone scripts for personal phone calls
• MyChart messages

Advertising, regardless of the chosen channel(s), must follow federal regulations, Duke policies, and Duke branding requirements. Links below provide information about allowable typography, colors, use of logos, and additional tools.

• DUHS Brand Center
• School of Medicine Brand Guidelines
• Duke University Brand Guide
• DUHS Policy and Guidelines for Advertising Research
• DOCR Social Media Recruitment SOP

Discover Duke Research 

The Recruitment Innovation Center manages the Discover Duke Research Facebook and Instagram pages and offers free social media marketing consults to all Duke research teams. During a social media marketing consult, the RIC team and investigator will work together to begin drafting a social media marketing plan to submit to the IRB for approval. Ads can be launched by the RIC from the Discover Duke Research accounts (Facebook/Instagram, Reddit, and Google) or the investigator may launch them from another approved social media channel per the Duke Social Media Recruitment SOP . RIC services (e.g., assisting in the development of a social media marketing plan, acquiring high quality images for advertising, managing the campaign, etc.,) are free of charge but there is a fee for the ads themselves.

• Request a  social media marketing consult .
• Review other available Policies, Procedures and Guidance in the "Related Resources" section of this page.

Duke Health Clinical Trials Directory 

The Recruitment Innovation Center manages content on the Duke Health Clinical Trials Directory . Reviewing the SIP Console Training in LMS (course #00129901) is recommended before reaching out to the RIC for assistance ( [email protected] ) in having a study posted on the Directory.

Engagement, recruitment and retention planning should take into consideration best practices for literacy, numeracy, and health literacy as well as principles of good readability. Principles of readability and tailoring to the health literacy needs of a population include adherence to the following principles:

• Responsibility : Clear research communication is the responsibility of all stakeholders in the research enterprise
• Life Cycle Adherence : All research communication should be clear and easy to comprehend throughout the research life cycle
• Partnership : Research communication should be developed in partnership with the intended audience(s)
• Cultural Sensitivity and Respect : are integral to clear communication about research
• Tailoring : Research materials for participants should integrate literacy and health literacy practices, including plain language, numeracy, visualization and design techniques, and cultural considerations
• Evaluation : Participant research materials should be evaluated to ensure the intended audience can understand the information
• Confirmation : In-person communication should encourage dialogue and confirm understanding
• Institutional Support : All stakeholders in the institutional research enterprise should support the development and implementation of organization policies that integrate literacy, numeracy and principles of good readability

Engagement, Recruitment, and Retention Certificate Program

The Engagement, Recruitment, and Retention Certificate Program is a certificate and skills-building program designed for Clinical Research study teams. The intention of the program is to help staff develop and expand competency in participant engagement, recruitment, and retention.

Participant Retention Strategies

Good retention starts with a strong, feasible, well-considered engagement and recruitment plan and a participant-centered research study. Many a study has been derailed by inadequate attention to recruitment and retention barriers and lack of effective strategies to overcome them.

Approaching recruitment and retention by ensuring a protocol is participant-centered will enhance the chances of a successfully completed study that finishes enrollment on time with a strong retention rate. Being participant-centered requires an investigator to consider every element of the project from the perspective of a participant and a variety of other characteristics (e.g., relevance of the study question to them, motivation - altruism, compensation, access to novel therapy, etc. - disease state, therapeutic options and opportunity costs, etc.). Adopting both a participant-focus and a quality-by-design framework can help examine study objectives and identify the factors that are critical to achieving them, while minimizing the burden of participation for participants. In terms of recruitment and retention, this should (at minimum) include attention to the following:

• Eligibility criteria – each criterion should be reviewed for its importance to achieving the study aims, its effect on the availability of the population and its acceptability to providers, participants and (if applicable) patient advocacy organizations.
• Accrual feasibility – does the study population as described actually exist? Is there one (or more) particular criteria that will weed out a large number of potential participants? Are there adjustments to the eligibility criteria that should be made?
• Adequate compensation – not all studies have the option of compensating participants. However, it’s important to consider the time, effort and hassle participants are enduring to take part in a study. Compensation in the form of money, expense reimbursement (e.g., travel, parking, meals), meaningful/useful tchotchkes or give-aways, etc., are all appropriate methods of compensating participants.
• Study procedures and event schedules – are the study procedures, including their invasiveness and risks, length and frequency of visits and location of participation particularly burdensome for the target population?
• Feasibility in the clinic – examine the study from the perspective of providers and clinic staff and their daily clinic operations. Will the study impact their clinic workflow? Engage them early in study design to mitigate against this risk; otherwise, ensure flexibility to minimize disruption.
• Reducing the burden of participation – research participants are often busy people, deeply embedded in dynamic personal lives. A research study is not part of their job. Providing flexible appointment times, short visits, and convenient locations for in-person study visits can ensure that it’s easy for participants to remain in a study. Consolidate visits when possible and provide “remote” options when feasible.
• Clear, frequent communication – Well-timed communication, including study visit reminders, via participant-preferred methods (SMS, email, phone call), an updated website, newsletters about study progress and milestones achieved, etc. are all good ways to keep participants engaged and interested in a study. Lay summaries of findings can be useful, especially if an abstract or a poster about the project is published.
• An attitude of humble gratitude – People are not required to participate in research. Their participation is a gift to be appreciated. Frequent communication of gratitude and recognition of the time and effort participants are sharing is often cited by participants as important to their continued participation. A commitment to providing a lay-summary of the study results at the end of the study is a great way to demonstrate appreciation and turn research participants into research evangelists.

Through Innovation, Connections, Collaborations and Education, the Recruitment Innovation Center is available to help with recruitment needs. For assistance and support across the enterprise, reach out to the RIC at [email protected] or request a consult today.

• Conference Coverage
• CRO/Sponsor
• 2023 Salary Survey

• Publications
• Conferences

Retention Strategies for Keeping Participants Engaged

Applied Clinical Trials

A case study of the Parkinson’s Progression Markers Initiative.

Engaging Emerging Clinical Investigators Yields Unique Benefits for All

New investigators practicing in community-based settings bring enthusiasm and a fresh approach to clinical research, while also boosting collaboration and merging personal connection and patient-centric insights.

The Site Council Value Calculator: A New Tool for Optimizing Site Payments

Giving sites and sponsors an effective tool for principled negotiations.

Accelerating Clinical Trial Activation

The activation of each clinical trial involves hundreds of tasks, along with many dependencies and handoffs, some of which may be automated, but many require manual processes.

Clinical Trials in Indonesia: Challenges and Opportunities for Industry Sponsors

Improved diversity in clinical trials can be achieved by exploring opportunities for research in developing countries that have not historically participated in large-scale, industry-sponsored clinical trials.

Patient and Site Personnel Perceptions of Retail Pharmacy Involvement in Clinical Research

Despite industry-wide excitement over the involvement of retail pharmacies in clinical research, there is little information currently available on how retail pharmacies are perceived by investigative sites and patients.

How to Improve Quality of Study Delivery? Work with Technology-enabled Sites

Practical ways CROs and sponsors can support sites and improve quality.

2 Commerce Drive Cranbury, NJ 08512

609-716-7777

• Sign Up Now
• -- Navigate To -- CR Dashboard Connect for Researchers Connect for Participants
• Log In Log Out Log In
• Recent Press
• Papers Citing Connect
• Connect for Participants
• Connect for Researchers
• Connect AI Training
• Managed Research
• Prime Panels
• MTurk Toolkit
• Health & Medicine
• Enterprise Accounts
• Conferences
• Knowledge Base

4 Strategies to Improve Participant Retention In Online Longitudinal Research Studies

Whenever researchers conduct a longitudinal research study, one concern takes precedence over almost all others: retaining participants. This is especially true for many studies conducted in the era of COVID-19, as researchers are interested in understanding how people adapt and cope to rapidly changing circumstances.

Fortunately, retaining participants in longitudinal research studies is easier with technology. In this article, we offer four sampling strategies to maximize retention in longitudinal studies conducted online. We also report data on the effectiveness of these sampling strategies in a large and demanding longitudinal study we conducted. 

The tips in this article are drawn from a chapter in the newest book in SAGE’s Innovation in Methods Series, Conducting Online Research on Amazon Mechanical Turk and Beyond . This book, written by Leib Litman and Jonathan Robinson , provides both students and experienced researchers with essential information about the online platforms most often used for social science research.

How to Maximize Longitudinal Study Retention

1. select the appropriate participants for your study.

Convenience sampling is the most economical and uncomplicated way to source research participants. Having fewer barriers to participation can help a research project reach completion quicker. This explains the popularity of sites like Mechanical Turk (MTurk). 

Yet, some participants spend more time on MTurk and complete more studies than others. The experience of these superworkers may pose a problem for some studies, but in other cases, their experience may be an advantage. Experienced survey takers are more likely to complete demanding tasks and more willing to participate in multiple waves of a longitudinal research study.

2. Be Transparent about the Study and Payment

When recruiting participants for research, clear communication is imperative. Participants need to know exactly what they are agreeing to do at the start of the longitudinal study. If you want people to participate in five surveys over five months tell them this at the outset. Also, be clear about compensation. We recommend making details about what will be expected from participants and how they will be compensated as clear and transparent as possible before participants have officially enrolled in the study.

3. Increasing Payments or Awarding a Bonus

Participant recruitment is only one part of the success. When structuring the study compensating participants well is crucial to retaining participants in longitudinal research studies. There are multiple ways to incentivize participants across multiple waves of a longitudinal study. First, you could devise a payment schedule of increasing incentives to sustain participant motivation. If participants receive $1 for the first survey, they may receive $2 for the second survey and $3 for the third.

Another strategy is to keep absolute compensation the same, but shorten the length of each follow-up survey, thereby increasing the hourly rate.

Finally, a third strategy is to provide a large bonus for completing all waves of the study. Often, combining an increasing payment schedule with a bonus is the most effective strategy.

When running a longitudinal study, it is important to realize the demand you are making on people’s time. People are more likely to enroll in the study and to stick with it when the compensation for the initial survey is competitive and the payment increases for each subsequent survey . These sampling strategies aim to increase that retention rate.

4. Give Participants Reminders and Make Accessing the Study Easy

People on MTurk have busy lives and can understandably forget to complete a follow-up survey. Therefore, we recommend (a) making follow-up surveys incredibly easy to access and (b) providing people with friendly reminders about the need to participate.

To further ensure maximum retention, consider contacting participants before the launch of a follow-up survey, upon the launch of that follow-up survey, and once more on the final day of each survey’s availability to remind them of the survey’s impending expiration. Within the CloudResearch MTurk Toolkit you can schedule these email notifications in advance and automate the process of sending them.

In addition to the factors above, some sampling strategies originally identified for maximizing retention in offline longitudinal research studies may be effective for online studies. For example, a project logo and project identity can help participants feel they are part of the project and increase retention (Ribisl et al., 1996). Regardless of the sampling methods in research, thoughtful planning to minimize participation difficulty will always increase the likelihood of study success.

What is a Good Retention Rate for a Longitudinal Research Study?

More than a year ago, we managed a large longitudinal research study for a client . The longitudinal study required more than 2,500 people to participate in a two to three-hour forecasting task each week for 16 weeks. Using all the sampling strategies described above, we had a total of 2,785 people complete at least one session. More importantly, 63% of people completed 15 of 16 weeks and 60% of people completed all sixteen weeks of the study!

This level of retention would be nearly unimaginable for a face-to-face study given the demands of the study schedule. When studies are not this demanding, we regularly see retention rates near 70 to 80%.

For more information on running longitudinal research studies online and sampling strategies, including case studies of successful projects, see Chapter 9 on longitudinal studies in the book Conducting Online Research on Amazon Mechanical Turk and Beyond . 

Hall, M. P., Lewis, Jr., N. A., Chandler, J., & Litman, L. (2020). Conducting longitudinal research on Amazon Mechanical Turk. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon Mechanical Turk and beyond (234-263) . Sage Academic Publishing. Thousand Oaks: CA

Ribisl, K. M., Walton, M. A., Mowbray, C. T., Luke, D. A., Davidson, W. S., & Bootsmiller, B. J. (1996). Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations. Evaluation and Program Planning, 19(1), 1–25. doi:10.1016/0149- 7189(95)00037-2

Related Articles

Best practices that can affect data quality on mturk.

Of late, researchers have reported a decrease in data quality on Mechanical Turk (MTurk). To combat the issue, we recently developed some data quality solutions, which are described in detail...

Pay Justice in a Newly Developing Economy

In 2020, the Covid-19 pandemic initiated an unprecedented shift in the American workforce. In a matter of weeks a significant portion of the labor force shifted to working online, a...

SUBSCRIBE TO RECEIVE UPDATES

2024 grant application form, personal and institutional information.

• Full Name * First Last
• Position/Title *
• Affiliated Academic Institution or Research Organization *

Detailed Research Proposal Questions

• Project Title *
• Research Category * - Antisemitism Islamophobia Both
• Objectives *
• Methodology (including who the targeted participants are) *
• Expected Outcomes *
• Significance of the Study *

Budget and Grant Tier Request

• Requested Grant Tier * - $200 $500 $1000 Applicants requesting larger grants may still be eligible for smaller awards if the full amount requested is not granted.
• Budget Justification *

Research Timeline

• Projected Start Date * MM slash DD slash YYYY Preference will be given to projects that can commence soon, preferably before September 2024.
• Estimated Completion Date * MM slash DD slash YYYY Preference will be given to projects that aim to complete within a year.
• Project Timeline *
• Name This field is for validation purposes and should be left unchanged.

• Name * First Name Last Name
• I would like to request a demo of the Sentry platform
• Name * First name Last name
• Email This field is for validation purposes and should be left unchanged.

• Name * First Last
• Phone This field is for validation purposes and should be left unchanged.
• Name * First and Last
• Please select the best time to discuss your project goals/details to claim your free Sentry pilot for the next 60 days or to receive 10% off your first Managed Research study with Sentry.

• Email * Enter Email Confirm Email
• Organization
• Job Title *

Advarra Clinical Research Network

Onsemble Community

Diversity, Equity, & Inclusion

Participants & Advocacy

Advarra Partner Network

Community Advocacy

Institutional Review Board (IRB)

Institutional Biosafety Committee (IBC)

Data Monitoring Committee (DMC)

Endpoint Adjudication (EAC)

GxP Services

Research Compliance & Site Operations

Professional Services

Coverage Analysis​

Budget Negotiation​

Calendar Build 

Technology Staffing

Clinical Research Site Training

Research-Ready Training

Custom eLearning

Services for

Sponsors & CROs

Sites & Institutions

BioPharma & Medical Tech

Oncology Research

Decentralized Clinical Trials

Early Phase Research

Research Staffing

Cell and Gene Therapy

Ready to Increase Your Research Productivity?

Solutions for need.

Clinical Trial Management

Clinical Data Management

Research Administration

Study Startup

Site Support and Engagement

Secure Document Exchange

Research Site Cloud

Solutions for Sites

Enterprise Institution CTMS

Health System/Network/Site CTMS

Electronic Consenting System

eSource and Electronic Data Capture

eRegulatory Management System

Research ROI Reporting

Automated Participant Payments

Life Sciences Cloud

Solutions for Sponsors/CROs

Clinical Research Experience Technology

Center for IRB Intelligence

Not Sure Where To Start?

Resource library.

White Papers

Case Studies

Ask the Experts

Frequently Asked Questions

COVID-19 Support

About Advarra

Consulting Opportunities

Leadership Team

Our Experts

Accreditation & Compliance

Join Advarra

Learn more about our company team, careers, and values. Join Advarra’s Talented team to take on engaging work in a dynamic environment.

Retention in Clinical Trials: Keeping Patients on Protocols

While patient recruitment is key to starting a clinical trial, patient retention may be more critical in ensuring the trial moves through the phases. Keeping participants in a trial ultimately helps keep a study on track, saving the site time, money and resources in the process. Any study delay can be harmful to the study and the site as a whole.

Since going through a clinical trial is a voluntary process, participants have the right to exit the study at any given time, without any given reason. Participants may drop out of a study for an unavoidable reason, however, many of the reasons participants leave a study are preventable.

This infographic provides insights to average dropout rates, reasons dropouts occur and solutions for better patient retention.

Download a larger version of the infographic

It’s clear there’s a need for more effective ways to keep patients on protocols. While some dropouts will always be inevitable, it’s crucial to prevent both study delays and missing study data for regulatory submissions. When possible, understanding why a participant chooses to discontinue a study is valuable information to obtain. Their feedback may provide insight into what changes can be implemented to improve the patient experience.

How should we plan for dropouts?

Seven of 100 known patients from the top of the funnel complete the trial. 18% of patients randomized end up dropping out.

The “Leaky Pipe” Analysis¹

The “typical” funnel shows the process of patient participation and is used to identify gaps and find where to “fill the funnel” or “manage the leaks.” Industry benchmark data suggests that on average across all protocol phases and therapeutic areas, you need to identify ~10 patients to randomize 1. (The Leaky Pipe Analysis was contributed by Beth Harper of Clinical Performance Partners, Inc.)

Here’s an example of the funnel:

• 100 patients identified or available move to the pre-screen step.
• 31 patients are pre-screen qualified and move to consent.
• 13 patients are consented and move to screening.
• 9 patients are randomized.
• 7 patients completed the trial.

What are some common reasons dropouts occur?  

Some common reasons are:

• Inconvenient location
• Schedule conflicts
• Personal/family matters
• Physically unable
• Financial constraints
• Lack of appreciation
• Forgetting visits
• Fear and anxiety
• Condition not improving
• Side effects
• Refusal to comply
• Misunderstood expectations

Or, the patient simply changed their mind (it is their right to do so). Multiple factors can play a role in one’s decision to drop out, and each patient has their own circumstances and motivations. Some of these factors are external, but others can be prevented.

How do patients who have withdrawn from a study compare to those who have completed one?²

Based on a survey asking study participants about their experiences:

• 35% of patients who dropped out of a study early thought it was difficult to understand the Informed Consent Form compared to just 16% who completed the trial.
• 64% of patients who dropped out of a study early were satisfied that their questions were answered during the Informed Consent Form discussion compared to 89% of patients who completed the trial.
• 38% of patients who dropped out of a study early thought the site visits were stressful compared to 16% who completed the trial.
• 47% of patients who dropped out of a study early said they were motivated by “myself” to stay enrolled in the study compared to 78% who completed the trial.

How well did the study meet those participants’ expectations?²

For patients who dropped out early, 6% said the study fell short of their expectations, 45% said it met their expectations and 21% said it exceeded their expectations.

For patients who completed the trial, 8% said the study fell short of their expectations, 37% said it met their expectations and 34% said it exceeded their expectations.

More than double the percentage of patients who dropped out of a study reported that it didn’t meet their expectations compared to those who completed it. More than half of patients who completed a study said it exceeded their expectations.

What are reasons to participate in a clinical research study?³

Based on a survey asking study participants about their experiences, the following were reasons to continue the study:

• 43% said to help advance science
• 34% said to obtain better treatment for their condition
• 34% said to receive monetary compensation
• 29% said to obtain education about treatment
• 19% said for free medication and/or treatment
• 16% said their primary care physician and/or specialist recommended the study

What motivates patients to continue participation?

• Minimize burdens during protocol design
• Set expectations up-front during ICF discussion
• Explain importance of their participation
• Promptly respond to inquiries
• Send reminders for upcoming visits
• Provide a comfortable and patient-friendly environment
• Show appreciation and recognition
• Accommodate their schedule as much as possible

When possible, try to understand why participants withdraw, and use that feedback to improve the patient experience.

• Beth Harper, Benchmark data from Clinical Performance Partners, Inc. and PhESi – 1998-2012
• http://www.medavante-prophase.com/wp-content/uploads/2018/09/2013_ciscrp_study_ineligible_participants_and_those_who_drop_out.pdf
• https://www.ciscrp.org/wp-content/uploads/2019/12/Participation-Experiences-04DEC-1.pdf

Back to Resources

Ready to Advance Your Clinical Research Capabilities?

Subscribe to our monthly email.

Receive updates monthly about webinars for CEUs, white papers, podcasts, and more.

Advancing Clinical Research: Safer, Smarter, Faster ™

Copyright © 2024 Advarra. All Rights Reserved.

• Privacy Policy
• Terms of Use
• Cookie Policy

• Get new issue alerts Get alerts
• Submit a Manuscript

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Strategies for participant retention in long term clinical trials

A participant –centric approaches.

Poongothai, Subramani; Anjana, Ranjit Mohan 1 ; Aarthy, Ramasamy; Unnikrishnan, Ranjit 1 ; Venkat Narayan, K. M. 2 ; Ali, Mohammed K. 2 ; Karkuzhali, Kulasegaran; Mohan, Viswanathan 1

Department of Clinical Trials, Madras Diabetes Research Foundation, Chennai, Tamil Nadu, India

1 Department of Diabetology, Dr. Mohan's Diabetes Specialities Centre and Madras Diabetes Research Foundation, Chennai, Tamil Nadu, India

2 Department of Epidemiology, Emory University, Atlanta, Georgia

Address for correspondence: Dr. Subramani Poongothai, Department of Clinical Trials, Madras Diabetes Research Foundation, No 4. Conran Smith Road, Gopalapuram, Chennai - 600 086, Tamil Nadu, India. E-mail: [email protected]

Received July 23, 2021

Accepted December 21, 2021

A clinical trial is the most foolproof method to evaluate the efficacy of a new intervention. Successful completion of clinical trials depends on the retention of the participants enrolled. Poor participant retention can lead to significant time and cost burden and have potentially adverse biases on the results. A high retention rate of participants is an important criterion for the validity and credibility of randomized controlled clinical trials. Many long-term trials fail due to low retention of study participants. Efforts at participant retention should start even before the first participant is recruited into the study. Retention is not only the responsibility of the investigators but also all other stakeholders in a clinical trial. In recent years, retention materials, participant camps, and introduction of national study coordinators have helped in improving retention. Quality of the relationship developed between the research staff and the study participant is a key factor for success of any trial. In our experience, in the context of resource-challenged low- and middle-income countries, we have found that it is possible to achieve high retention rates, 95%–100%. The rapport built between the investigating team and the participant plays a vital role in retention. In addition, personalized care, including listening to the participant's problems and enabling to contact investigators or study team at any time of the day, has shown benefits in retention.

INTRODUCTION

The randomized clinical trial is the most scientific method to access the effectiveness of treatment.[ 1 ] The success of a clinical trial depends on the ability to recruit participants in a systematic manner, to ensure adherence to and integrity of intervention, and to retain the participants through the entire study duration and follow-up. Power of the study will be affected if the retention rate is poor, and selection bias is introduced.[ 2 ] The reliability of results and validity of the study are affected by poor retention.[ 3 ] Successful recruitment and retention of participants is the most challenging aspect in the conduct of clinical trials.[ 4 ] Participant selection is crucial for retention. Retention is a continuous process, and plans for retention strategies should start during protocol development and from the onset of recruitment. Therefore, it is essential to understand effective retention strategies, as participant retention is a major practical challenge – often the “Achilles Heel” of trials – especially in resource-constrained situations, as in low- and middle-income countries.

There are many challenges in achieving high retention, which include lack of trust, socioeconomic barriers, inconveniences, real or perceived adverse events from intervention, migration of the subject from the study site, personal reasons, and influence of media.[ 5 ] Complexity of the study, nature of disease studied, need to take time off from work, education and occupation level, and interference from physicians not involved in the study and family members are other factors to be considered.[ 6 ] At times, retention remains a challenge even if recruitment is successful. Recent data between the years 2000 and 2006 examining trends in recruitment and retention showed that recruitment dropped by 16% whereas retention dropped by 21%.[ 7 ] Studies have shown an 88% dropout due to the following reasons – lost to follow-up, nonadherence to protocol, and withdrawal of consent.[ 8 ]

On the other hand, multiple strategies which include the presence of a study team that is respectful and supportive of participants during the entire study period, developing a good relationship with the participant, access to the investigators at any time of the day, and meticulous monitoring and corrective action on an ongoing basis, when all these are well employed, can help enhance retention.[ 9 ] In some cases, provision of additional investigations related to clinical needs other than the trial investigations may also help in retention.[ 10 ]

CHALLENGES IN RETENTION

Retention and recruitment are interlinked. When the duration of the study becomes long and the number of study visits increase, it becomes an added burden to the participants and to the study team.[ 11 ] This factor needs to be considered during the design of the study, and strategies put in place to ensure unbiased recruitment and high retention.

On average, 25%–26% of the participants drop out after expressing their consent.[ 12 13 ] More than 90% of the studies are delayed due to failed enrollment or challenges to participant retention, which includes loss of participants to follow-up.[ 14 ] In a survey assessing the challenges in participant retention, it was found that 44% of the participants feared side effects. Fear of study procedures and change of residence were reported by 47% as the reasons for dropping. Lack of support from the family physician and family members was reported by 27%, while 55% of the participants stated that the lack of dedicated approach by an investigator's team and 9% each reported negative media publicity and perceived lack of efficacy of the treatment as reasons for dropping out.[ 4 ] Other retention challenges include inconvenient location, schedule conflicts, perception that the disease condition is not improving, lack of appreciation for their participation, and unrealistic expectations from the treatment.

Retaining recruited participants is also a major challenge. Regardless of the objective of research projects, the required data collection is only possible if adequate and enough of the selected subjects continue to participate throughout the trial.[ 15 ] There are, however, several reasons beyond design, experimental drug, or type of study that prevent high retention of the study population. In recent years, the rate of retention in major global trials has increased, on average, significantly, as shown in Table 1 , and this is largely due to the involvement of national study coordinators in long-term global clinical trials.[ 4 ]

HOW TO OVERCOME RETENTION CHALLENGES

It is important to recognize the symptoms and signs of nonadherence of participants during the study, which include missed study visits, difficulty in reaching by phone or failure to return calls, and impatience during clinic visits. To address these challenges, there are retention strategies and the role of respective stakeholders plays a major role, as shown in Figure 1 .

ROLE OF VARIOUS STAKEHOLDERS

Drug discovery and development are possible only with the participation of key stakeholders within the industry. These stakeholders include the participants, research team, sponsors, regulators, monitors, government agencies, and contract research organizations Contract Research Organization (CROs).

The primary stakeholders who discover the drug and initiate the research process are the sponsors. In recent years, sponsors increasingly utilize the assistance of clinical research organizations (CROs) for conducting a clinical trial. The CROs conduct the trial as directed by the sponsor and they act as one of the stakeholders.

Researchers or the principal investigators (PIs) are the stakeholders, hired by the sponsor to conduct the clinical trial. In a study team led by the PIs, the study coordinators play a major role in ensuring high retention. However, the most important stakeholders are the participants who contribute by providing clinical data to evaluate the efficacy of an investigational product.

ROLE OF THE PRINCIPAL INVESTIGATOR

The PI is a key responsible person and accountable for conducting the clinical trial. The PI ensures that the study is ethically conducted in compliance with the protocol and ensuring access to medical care of the study subjects. Recruitment, retention, training, appraisal, and supervision of team members is also the responsibility of the PI.

ROLE OF THE STUDY COORDINATOR

In most of the instances, the participant dropout is well managed by the study team except for those due to adverse events which makes it difficult to achieve a 100% retention rate, even as a best-case scenario.[ 12 13 14 ] The key person for the successful retention in the study team is the study coordinator. The Diabetes Control and Complications Trial/Epidemiology of Diabetes Interventions and Complications and Outcome Reduction with Initial Glargine Intervention studies have served as a model for the role of the study coordinator in a clinical trial.[ 22 23 ] In addition, study coordinators have made remarkable contributions to the overall research endeavor for high participant retention and data completion rates.[ 24 ] In recent years, some sponsors have introduced the concept of a national study coordinator, who guides all the study coordinators at their respective sites. The national coordinators work closely with all the sites with the help of the sponsors, a development that has led to very high retention rates.[ 25 ]

STUDY PARTICIPANTS

Participants are volunteers who have the liberty to withdraw from the study at any point of time. It is very important to ensure that the participants are taken care of in terms of the following –answering their concerns and attending to their medical management and investigations. Challenges for the study participants to continue in the study include adverse events, migration, and physician and family interference.

Providing a comfortable waiting room and also spending more time with the participants will ensure a lot of trust and confidence. The goal of the study coordinator is to build such a rapport with the participants and also to inform the participants of their value in the study.[ 26 ]

Table 2 shows the various retention methods which have been used for many long-term studies. Listening to the participant problems and spending more time with the person is essential and helps in achieving high retention.

Methods used to improve retention

Appointment reminders.

This is very important, as one of the most common mistakes many of research sites make is to rely on participants to remember their next appointment. Although some participants will remember when they are supposed to come back, it is still the task of the team to remind them. Phone calls, emails, and reminder cards will jog their memory and remind them to come back for their scheduled appointment.

Reimbursement/incentives

Travel reimbursement to the participants at each visit and meal vouchers help in retention. Participants in trials are offered rewards such as monetary payments, free medical care, and food. Such rewards are not considered benefits of study participation but rather incentives and motivation for participation. Receiving a monetary payment or receiving something free that would normally have to be paid, is considered as an inducement. Any inducement may be coercive or an undue influence on a potential study participant, and therefore, incentives are to be planned accordingly with approval of the Ethics Committee. As incentives for participation are potentially coercive, the amount and conditions of such incentives must be reviewed and approved by the Institutional Ethics Committee.

Participant newsletter

Newsletters that highlight the importance of research or offer tips on daily living with their condition, would really help them. Participants can be engaged with E-mail newsletters to keep them connected throughout the trial. Surveys showed that participants are eager to receive ongoing education about their disease state, which could be established through the distribution of participant newsletters, brochures, etc.

Self-monitoring tools

In many diabetes trials, capillary glucose meters are provided to measure the blood glucose, and in some cardiology studies, blood pressure monitors are provided to the participants.

Making clinic visit pleasant

Minimize waiting time.

Participants in trials always need special attention. It is important that the study procedures are done quickly in order to reduce the waiting time.

Waiting room facilities

Separate room for the study participants and availability of newspapers, magazines to read, and television to watch at the waiting room would make them more comfortable. Availability of free WiFi may be an additional benefit.

Spending more time

Talking to the participants and spending some time by listening to their problems gives a lot of comfort to the participants.

SOCIAL INTERACTION

• Increased, enjoyable contact with the study team
• Provide opportunity for participants to ask questions
• Provide a “listening ear.”

INFORMED CONSENT FORMS

Strategies for retention start from day 1 when the informed consent form has been signed. As it is important to provide participants with all the details regarding the study at the beginning itself, the handouts, especially the informed consent forms, should provide a good sense of what to expect and what the trial would involve. It is also important to make sure that the informed consent is easy to understand, so that participants know exactly what to expect from the trial.

INVOLVING FAMILIES

A common reason for participants to withdraw from clinical trials prematurely is due to the influence of friends or family who are uncomfortable or unfamiliar with clinical trials. Participants may be enthusiastic at the start of the trials, but later, the enthusiasm declines. This change can be prevented by providing participants with educational materials and involving family members.

The most common retention techniques involve phone calls, visit reminders, birthday or anniversary cards, reimbursement of study participants for their time, and expenses incurred because of trial participation.

Figure 2 shows other retention strategies followed at our centre.

Table 3 shows the various retention challenges and the solutions adapted at the Madras Diabetes Research Foundation (MDRF). MDRF is a 100% nonprofit organization, located in Chennai, India, which conducts full-fledged research on diabetes and related complications. MDRF is designated as the Indian Council of Medical Research Centre for Advanced Research on Diabetes and the International Diabetes Federation Centre of Excellence in Diabetes Care. It has an established track record of conducting high-quality surveillance and clinical trials.

RETENTION TOOLS

In recent years, many retention materials have been developed, which have helped to retain the participants in long-term studies. It is the duty of the sponsor as well as the investigator to ensure that these materials are culturally appropriate.

Participant education's camps have taken off in a big way retain participants for long-term studies. These camps are mostly on participant education, cookery demonstrations, and simple exercise programs, which are conducted at the study site. During these camps, the participant's family members are invited and this in turn gives confidence and faith to the participant's family. Honoring the research participants and arranging annual luncheons is one of the most effective retention strategies. All these activities require approval from the Ethics Committee.

Figure 3 shows the overall retention strategies that can be used for a long- term follow-up.

CONCLUSION AND RECOMMENDATIONS

Retention of patients in clinical trials is a continuous process and remains a key factor for the success of the clinical trial. Participant retention remains a significant problem in many trials. Adopting a more participant-centric approach by listening to participants and by implementing their feedback is essential. Motivation for participation in clinical trials is multifactorial but can effectively attack the problem of participant dropouts. Respect for participants, helps in establishing trust and rapport, which in turn leads to better retention. Effective informed consent, showing appreciation to make their participation personal, and meaningful and building good relationship with participants by sending greeting cards for their birthday and for festivals between visits encourage retention. Creating a social community for the participants gives them an opportunity to feel like they are an integral part of the study, which would help them to share their experience. Maintaining a health record for each participant enhances retention.

Apart from the different retention strategies discussed for both clinical trials as well as for mental health studies, quality of the relationship developed between the research staff and the study participant is a key factor to overcome the challenges faced in retention.

It is important to put the patient first at every stage of the study journey. Hence, there is a need to value and support the participants throughout the study, which has shown evidence for less dropouts from the trial. Therefore, our emphasis is to develop patient-focused protocols by using the patient feedback. This would help us resolve the most common issues around patient retention, which in turn influence the success of a clinical trial.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Acknowledgement

The authors would like to acknowledge and thank Ms. Mokkapati Lalasa, of Madras Diabetes Research Foundation for her assistance in the research paper.

Clinical trials; retention; stakeholders

• + Favorites
• View in Gallery

Readers Of this Article Also Read

The changing perspective of clinical trial designs, advice to authors for avoiding flaws in preparation of original research..., research studies on screening tests, introduction to qualitative research methods – part i, a cross-sectional study evaluating the knowledge, attitude, and practice of....

Strategies for Getting Involved

Identify your interests Take advantage of undergraduate research initiatives Identify faculty members who share your interests Stop! Prepare before contacting faculty members Contact a faculty member Prepare for your initial meeting with a faculty member Ensure a successful first meeting with a faculty member Keep in mind. . .

Identify your interests.

• What topics interest you?
• What relevant coursework have you taken or do you plan to take?
• Is there something you are passionate about and would like to explore?
• What volunteer or work experience do you have?
• What organizations, sports, or activities do you participate in?
• What are your favorite courses?

Take advantage of undergraduate research initiatives. A variety of programs exist to support undergraduate research on campus.

• The Office for Undergraduate Research in the College of Natural Sciences provides access to a range of resources, from the Freshman Research Initiative to student groups to a database of summer research opportunities.
• Various undergraduate research opportunities exist within the Cockrell School of Engineering , including paid and volunteer research positions and for-credit research programs.
• The College of Liberal Arts provides comprehensive information on events and resources in the College. In addition, there are many undergraduate research scholarship opportunities through the College of Liberal Arts.
• Students in any college can participate in the Bridging Disciplines Programs . BDP s allow you to develop an interdisciplinary specialization that complements your major and combines coursework, internships, and research. The BDP Research Coordinators can help you identify research experiences and connect you with a faculty research mentor.
• Organizations around campus provide opportunities for students with similar interests to discuss their work and promote involvement in undergraduate research. The Science Undergraduate Research Group ( SURG e) and Liberal Arts Undergraduate Research Chapter ( LAUNCH ) include students who work to foster a cohesive undergraduate research community.

Identify faculty members who share your interests.

• Search EUREKA for potential research projects and faculty who work on your preferred topics.
• Read faculty profiles on departmental web pages.
• Browse organized research units on campus.

Stop! Prepare before contacting faculty members.

• Approach faculty with respect. Before you approach a professor, do your homework. Read her or his web site, find out about recent publications, and have a good understanding of that person’s research.
• Make sure you can clearly articulate your interests. An Advisor at the Office of Undergraduate Research can help.

Contact a faculty member.

• Be professional. Write an email in a formal, respectful tone. Address the faculty member as ‘Prof. X’ or ‘Dr. Y.’
• Include a sentence or two stating your interests and topics you would like to explore (primarily those that the faculty member shares with you).
• Show your enthusiasm and commitment.

Prepare for your initial meeting with a faculty member.

• Review what made you interested in this topic (courses, college experiences, family/background, media).
• Look over the faculty member’s curriculum vitae and/or publications.
• Think about skills and qualities you can bring to the project.

Ensure a successful first meeting with a faculty member.

• Be confident, excited, and relaxed.
• Introduce yourself and address the faculty member as ‘Prof. X’ or ‘Dr. Y.’
• Look presentable. This isn’t a job interview, but you should dress neatly.
• Ask the professor what will be expected of you.

Keep in mind . . .

• Be willing to start “at the bottom.” Just remember, eventually you will reach the top.
• Be yourself. The faculty member will want to get to know your interests and ideas.
• Think about your availability. How much time do you have to commit to this project? (Remember that your workload shifts during midterms and finals. Don’t overestimate how much time you can contribute.)
• Be reliable. Faculty members are looking for students who are not just interested in their work, but who are reliable and can be counted on.
• Don’t get discouraged. If this faculty member does not have an undergraduate position available, she or he might be able to recommend another project or faculty member for you.

Study reveals increasing influence of non-cognitive skills on academic achievement from childhood to adolescence

• Download PDF Copy

In a recent study published in Nature Human Behavior , researchers assessed the associations between cognitive and non-cognitive (NCS) skills and academic achievement from ages 7 to 16.

Children who can regulate their impulses and attention and are motivated and emotionally stable perform better in school, independent of their cognitive abilities.

These socioemotional characteristics have been described as NCS. NCS predicting better educational outcomes can be classified into three overlapping domains – personality traits, motivational factors, and self-regulatory strategies.

Research on twins has revealed that genetic differences between individuals contribute to differences in NCS. Most NCS are moderately heritable.

Besides, studies have observed genetic correlations between NCS and academic achievement. These genetic links have been confirmed using deoxyribonucleic acid (DNA)-based methods.

Genome-wide association studies (GWAS) have identified genetic variants correlated with the completion of formal education.

A polygenic score (PGS) based on GWAS results predicts greater academic motivation, increased self-control, and more adaptive personality traits. Further, NCS genetics are related to health-risk behaviors, gratification delay, openness to experience, and conscientiousness.

About the study

In the present study, researchers examined associations of NCS and cognitive skills with academic achievement over development. Participants were twins born in England and Wales during 1994-96 and were part of the Twins Early Development Study.

The current study analyzed data collected when twins were aged 4, 7, 9, 12, and 16. The team focused on two broad NCS dimensions modeled as latent factors – education-specific NCS and domain-general self-regulation skills.

Related Stories

• Expanded use of common antibiotic cuts child mortality by 14% in Sub-Saharan Africa
• Elderly with type 2 diabetes struggle with self-care and medication adherence
• DNA scavengers break down antibiotic-resistant genes

At age 9, information on education-specific NCS was obtained from teachers, parents, and twins’ self-reports.

Measures included the classroom environment questionnaire, self-perceived academic ability, and academic interest. At age 12, data were similarly collected on the following measures: self-perceived academic ability, literacy and mathematics environment questionnaires, and academic interest.

At age 16, education-specific NCS were assessed via twins’ self-reports on an academic self-concept scale, school engagement, grit, academic ambition, mathematics self- efficacy , time spent studying math, mathematics interest, attitudes toward school, and curiosity. The strengths and difficulties questionnaire examined behavioral and emotional self-regulation at all ages.

Four tests were administered to measure cognitive ability at ages 7, 9, and 12. At age 16, a composite non-verbal and verbal test was administered for measuring cognitive ability.

Academic achievement at ages 7, 9, and 12 was assessed using teacher reports. At age 16, the General Certificate of Secondary Education (GCSE) exam scores were used to evaluate academic achievement.

Latent factors of domain-general self-regulation skills and education-specific NCS positively correlated with academic achievement at developmental stages.

The effect sizes increased with age. Further, both non-cognitive factors were significantly associated with academic achievement beyond cognitive skills. Notably, the heritability of NCS was significantly different across developmental stages and raters.

Further, the researchers found that genetic factors associated with cognitive skills accounted for 21% to 36% of the total variance in academic achievement. NCS-associated genetic effects accounted for up to 32.5% of the variance, independent of cognitive skills. Moreover, 5% to 37% of the variance in academic achievement was independent of genetic effects associated with NCS and cognitive skills.

Next, the researchers calculated PGSs for non-cognitive and cognitive factors and assessed their associations with academic, cognitive, and non-cognitive phenotypes.

Cognitive PGS significantly predicted variations in NCS over development. Likewise, non-cognitive PGS predicted variations in NCS, independent of the cognitive PGS. While the association was smaller at younger ages for parent-reported education-specific NCS, it increased over developmental stages.

Non-cognitive and cognitive PGSs predicted variations in verbal, non-verbal, and general cognitive abilities at all stages of development.

Associations between cognitive PGS and academic achievement were evident at seven years, which remained consistent over time. By contrast, effects were weaker for non-cognitive PGS in early ages but increased over time, reaching the same levels as cognitive PGS at 16 years.

Conclusions

In sum, the study investigated associations of non-cognitive and cognitive genetics with educational achievement during compulsory education.

The findings indicate that NCS predicted academic achievement over time, which was substantial even when cognitive skills were accounted for. These associations were largely due to shared genetic factors.

Further, the effects of non-cognitive genetics were sustained even after family-fixed effects were accounted for.

Together, these results underscore the crucial role of NCS in primary and secondary education. As such, fostering NCS could be an avenue for successful academic interventions and strategies.

Malanchini M, Allegrini AG, Nivard MG, et al. (2024) Genetic associations between non-cognitive skills and academic achievement over development. Nature Human Behavior. doi : 10.1038/s41562-024-01967-9 . https://www.nature.com/articles/s41562-024-01967-9

Posted in: Child Health News | Medical Science News | Medical Research News | Medical Condition News

Tags: Children , DNA , Education , Efficacy , Genetic , Genetics , Genome , Research , Twins

Tarun Sai Lomte

Tarun is a writer based in Hyderabad, India. He has a Master’s degree in Biotechnology from the University of Hyderabad and is enthusiastic about scientific research. He enjoys reading research papers and literature reviews and is passionate about writing.

Please use one of the following formats to cite this article in your essay, paper or report:

Sai Lomte, Tarun. (2024, August 28). Study reveals increasing influence of non-cognitive skills on academic achievement from childhood to adolescence. News-Medical. Retrieved on August 28, 2024 from https://www.news-medical.net/news/20240828/Study-reveals-increasing-influence-of-non-cognitive-skills-on-academic-achievement-from-childhood-to-adolescence.aspx.

Sai Lomte, Tarun. "Study reveals increasing influence of non-cognitive skills on academic achievement from childhood to adolescence". News-Medical . 28 August 2024. <https://www.news-medical.net/news/20240828/Study-reveals-increasing-influence-of-non-cognitive-skills-on-academic-achievement-from-childhood-to-adolescence.aspx>.

Sai Lomte, Tarun. "Study reveals increasing influence of non-cognitive skills on academic achievement from childhood to adolescence". News-Medical. https://www.news-medical.net/news/20240828/Study-reveals-increasing-influence-of-non-cognitive-skills-on-academic-achievement-from-childhood-to-adolescence.aspx. (accessed August 28, 2024).

Sai Lomte, Tarun. 2024. Study reveals increasing influence of non-cognitive skills on academic achievement from childhood to adolescence . News-Medical, viewed 28 August 2024, https://www.news-medical.net/news/20240828/Study-reveals-increasing-influence-of-non-cognitive-skills-on-academic-achievement-from-childhood-to-adolescence.aspx.

Suggested Reading

Cancel reply to comment

• Trending Stories
• Latest Interviews
• Top Health Articles

Global and Local Efforts to Take Action Against Hepatitis

Lindsey Hiebert and James Amugsi

In this interview, we explore global and local efforts to combat viral hepatitis with Lindsey Hiebert, Deputy Director of the Coalition for Global Hepatitis Elimination (CGHE), and James Amugsi, a Mandela Washington Fellow and Physician Assistant at Sandema Hospital in Ghana. Together, they provide valuable insights into the challenges, successes, and the importance of partnerships in the fight against hepatitis.

Addressing Important Cardiac Biology Questions with Shotgun Top-Down Proteomics

In this interview conducted at Pittcon 2024, we spoke to Professor John Yates about capturing cardiomyocyte cell-to-cell heterogeneity via shotgun top-down proteomics.

A Discussion with Hologic’s Tim Simpson on the Future of Cervical Cancer Screening

Tim Simpson

Hologic’s Tim Simpson Discusses the Future of Cervical Cancer Screening.

Latest News

Newsletters you may be interested in

Your AI Powered Scientific Assistant

Hi, I'm Azthena, you can trust me to find commercial scientific answers from News-Medical.net.

A few things you need to know before we start. Please read and accept to continue.

• Use of “Azthena” is subject to the terms and conditions of use as set out by OpenAI .
• Content provided on any AZoNetwork sites are subject to the site Terms & Conditions and Privacy Policy .
• Large Language Models can make mistakes. Consider checking important information.

Great. Ask your question.

Azthena may occasionally provide inaccurate responses. Read the full terms .

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions .

Provide Feedback

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Identifying research priorities for effective retention strategies in clinical trials

Affiliations.

• 1 North West Hub for Trials Methodology Research/Clinical Trial Research Centre, Biostatistics, University of Liverpool, Institute of Child Health, Alder Hey NHS Trust, Liverpool, L12 2AP, UK. [email protected].
• 2 ConDuCT-II Hub for Trials Methodology Research, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK.
• 3 Centre for Public Health, Queen's University of Belfast, University Road, Belfast, BT7 1NN, UK.
• 4 North West Hub for Trials Methodology Research/Clinical Trial Research Centre, Biostatistics, University of Liverpool, Block F Waterhouse Building, 1-5 Brownlow Street, Liverpool, L69 3GL, UK.
• 5 North West Hub for Trials Methodology Research/Clinical Trial Research Centre, Biostatistics, University of Liverpool, Institute of Child Health, Alder Hey NHS Trust, Liverpool, L12 2AP, UK.
• PMID: 28859674
• PMCID: PMC5580283
• DOI: 10.1186/s13063-017-2132-z

Background: The failure to retain patients or collect primary-outcome data is a common challenge for trials and reduces the statistical power and potentially introduces bias into the analysis. Identifying strategies to minimise missing data was the second highest methodological research priority in a Delphi survey of the Directors of UK Clinical Trial Units (CTUs) and is important to minimise waste in research. Our aim was to assess the current retention practices within the UK and priorities for future research to evaluate the effectiveness of strategies to reduce attrition.

Methods: Seventy-five chief investigators of NIHR Health Technology Assessment (HTA)-funded trials starting between 2009 and 2012 were surveyed to elicit their awareness about causes of missing data within their trial and recommended practices for improving retention. Forty-seven CTUs registered within the UKCRC network were surveyed separately to identify approaches and strategies being used to mitigate missing data across trials. Responses from the current practice surveys were used to inform a subsequent two-round Delphi survey with registered CTUs. A consensus list of retention research strategies was produced and ranked by priority.

Results: Fifty out of seventy-five (67%) chief investigators and 33/47 (70%) registered CTUs completed the current practice surveys. Seventy-eight percent of trialists were aware of retention challenges and implemented strategies at trial design. Patient-initiated withdrawal was the most common cause of missing data. Registered CTUs routinely used newsletters, timeline of participant visits, and telephone reminders to mitigate missing data. Whilst 36 out of 59 strategies presented had been formally or informally evaluated, some frequently used strategies, such as site initiation training, have had no research to inform practice. Thirty-five registered CTUs (74%) participated in the Delphi survey. Research into the effectiveness of site initiation training, frequency of patient contact during a trial, the use of routinely collected data, the frequency and timing of reminders, triggered site training and the time needed to complete questionnaires was deemed critical. Research into the effectiveness of Christmas cards for site staff was not of critical importance.

Conclusion: The surveys of current practices demonstrates that a variety of strategies are being used to mitigate missing data but with little evidence to support their use. Six retention strategies were deemed critically important within the Delphi survey and should be a primary focus of future retention research.

Keywords: Attrition; Clinical trials; Missing data; Missing data strategies; Retention; Study design.

PubMed Disclaimer

Conflict of interest statement

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

An example of the scoring…

An example of the scoring software used in round 2 of the Delphi…

Cohort of National Institute of…

Cohort of National Institute of Health Research (NIHR) Health Technology Assessment Programme (HTA)-funded…

Similar articles

• The trials methodological research agenda: results from a priority setting exercise. Tudur Smith C, Hickey H, Clarke M, Blazeby J, Williamson P. Tudur Smith C, et al. Trials. 2014 Jan 23;15:32. doi: 10.1186/1745-6215-15-32. Trials. 2014. PMID: 24456928 Free PMC article.
• Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK Clinical Trial Units and a qualitative study. Blatch-Jones A, Nuttall J, Bull A, Worswick L, Mullee M, Peveler R, Falk S, Tape N, Hinks J, Lane AJ, Wyatt JC, Griffiths G. Blatch-Jones A, et al. Trials. 2020 Apr 3;21(1):304. doi: 10.1186/s13063-020-04234-0. Trials. 2020. PMID: 32245506 Free PMC article.
• What are the main inefficiencies in trial conduct: a survey of UKCRC registered clinical trials units in the UK. Duley L, Gillman A, Duggan M, Belson S, Knox J, McDonald A, Rawcliffe C, Simon J, Sprosen T, Watson J, Wood W. Duley L, et al. Trials. 2018 Jan 8;19(1):15. doi: 10.1186/s13063-017-2378-5. Trials. 2018. PMID: 29310685 Free PMC article.
• Strategies to improve retention in randomised trials. Brueton VC, Tierney J, Stenning S, Harding S, Meredith S, Nazareth I, Rait G. Brueton VC, et al. Cochrane Database Syst Rev. 2013 Dec 3;(12):MR000032. doi: 10.1002/14651858.MR000032.pub2. Cochrane Database Syst Rev. 2013. PMID: 24297482 Free PMC article. Review.
• Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Bower P, Brueton V, Gamble C, Treweek S, Smith CT, Young B, Williamson P. Bower P, et al. Trials. 2014 Oct 16;15:399. doi: 10.1186/1745-6215-15-399. Trials. 2014. PMID: 25322807 Free PMC article. Review.
• Collaborative care model versus usual care for the management of musculoskeletal and co-existing mental health conditions: a randomised feasibility mixed-methods study. Teixeira MJC, Ahmed R, Tehrany R, Jaggi A, Ramanuj P. Teixeira MJC, et al. BMJ Open. 2024 Feb 21;14(2):e079707. doi: 10.1136/bmjopen-2023-079707. BMJ Open. 2024. PMID: 38387980 Free PMC article. Clinical Trial.
• How do trial teams plan for retention during the design stage of the trial? A scoping review. Murphy E, Gillies K, Shiely F. Murphy E, et al. Trials. 2023 Dec 4;24(1):784. doi: 10.1186/s13063-023-07775-2. Trials. 2023. PMID: 38049833 Free PMC article. Review.
• Establishing Requirements for Technology to Support Clinical Trial Retention: Systematic Scoping Review and Analysis Using Self-determination Theory. Gamble E, Linehan C, Heavin C. Gamble E, et al. J Med Internet Res. 2023 Apr 13;25:e38159. doi: 10.2196/38159. J Med Internet Res. 2023. PMID: 37052985 Free PMC article. Review.
• How do trial teams plan for retention during the design stage of the trial? A scoping review protocol. Murphy E, Gillies K, Shiely F. Murphy E, et al. Trials. 2022 Nov 17;23(1):944. doi: 10.1186/s13063-022-06866-w. Trials. 2022. PMID: 36397085 Free PMC article.
• Electronic Health Record-Based Recruitment and Retention and Mobile Health App Usage: Multisite Cohort Study. Coughlin JW, Martin LM, Zhao D, Goheer A, Woolf TB, Holzhauer K, Lehmann HP, Lent MR, McTigue KM, Clark JM, Bennett WL. Coughlin JW, et al. J Med Internet Res. 2022 Jun 10;24(6):e34191. doi: 10.2196/34191. J Med Internet Res. 2022. PMID: 35687400 Free PMC article.
• Toerien M, Brookes ST, Metcalfe C, de Salis I, Tomlin Z, Peters TJ, et al. A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials. 2009;10:52. doi: 10.1186/1745-6215-10-52. - DOI - PMC - PubMed
• Marcellus L. Are we missing anything? Pursuing research on attrition. CJNR (Can J Nurs Res) 2004;36(3):82–98. - PubMed
• Flick SN. Managing attrition in clinical research. Clin Psychol Rev. 1988;8(5):499–515. doi: 10.1016/0272-7358(88)90076-1. - DOI
• Gul RB, Ali PA. Clinical trials: the challenge of recruitment and retention of participants. J Clin Nurs. 2010;19(1/2):227–33. doi: 10.1111/j.1365-2702.2009.03041.x. - DOI - PubMed
• Tudur Smith C, Hickey H, Clarke M, Blazeby J, Williamson P. The trials methodological research agenda: results from a priority setting exercise. Trials. 2014;15(1):32. doi: 10.1186/1745-6215-15-32. - DOI - PMC - PubMed

Publication types

• Search in MeSH

Related information

Grants and funding.

• G0901530/MRC_/Medical Research Council/United Kingdom
• MR/K024310/1/MRC_/Medical Research Council/United Kingdom
• MR/K025643/1/MRC_/Medical Research Council/United Kingdom

LinkOut - more resources

Full text sources.

• BioMed Central
• Europe PubMed Central
• PubMed Central

Other Literature Sources

• scite Smart Citations
• MedlinePlus Health Information

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Med Res Methodol

Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis

Samantha teague.

1 Centre for Social and Early Emotional Development, School of Psychology, Faculty of Health, Deakin University, Burwood, Geelong, Victoria 3125 Australia

George J. Youssef

2 Murdoch Children’s Research Institute, Centre for Adolescent Health, Royal Children’s Hospital, Melbourne, Australia

Jacqui A. Macdonald

3 Department of Paediatrics, Royal Children’s Hospital, University of Melbourne, Melbourne, Australia

Emma Sciberras

Adrian shatte.

6 School of Engineering & Information Technology, Faculty of Science & Technology, Federation University, Melbourne, Australia

Matthew Fuller-Tyszkiewicz

Chris greenwood, jennifer mcintosh, craig a. olsson.

4 Melbourne School of Psychological Sciences, University of Melbourne, Parkville, Australia

Delyse Hutchinson

5 National Drug and Alcohol Research Centre, University of New South Wales, Sydney, Australia

Associated Data

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Participant retention strategies that minimise attrition in longitudinal cohort studies have evolved considerably in recent years. This study aimed to assess, via systematic review and meta-analysis, the effectiveness of both traditional strategies and contemporary innovations for retention adopted by longitudinal cohort studies in the past decade.

Health research databases were searched for retention strategies used within longitudinal cohort studies published in the 10-years prior, with 143 eligible longitudinal cohort studies identified (141 articles; sample size range: 30 to 61,895). Details on retention strategies and rates, research designs, and participant demographics were extracted. Meta-analyses of retained proportions were performed to examine the association between cohort retention rate and individual and thematically grouped retention strategies.

Results identified 95 retention strategies, broadly classed as either: barrier-reduction, community-building, follow-up/reminder, or tracing strategies. Forty-four of these strategies had not been identified in previous reviews. Meta-regressions indicated that studies using barrier-reduction strategies retained 10% more of their sample (95%CI [0.13 to 1.08]; p  = .01); however, studies using follow-up/reminder strategies lost an additional 10% of their sample (95%CI [− 1.19 to − 0.21]; p  = .02). The overall number of strategies employed was not associated with retention.

Conclusions

Employing a larger number of retention strategies may not be associated with improved retention in longitudinal cohort studies, contrary to earlier narrative reviews. Results suggest that strategies that aim to reduce participant burden (e.g., flexibility in data collection methods) might be most effective in maximising cohort retention.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0586-7) contains supplementary material, which is available to authorized users.

Longitudinal cohort studies play a central role in advancing understanding of the onset and progression of physical and mental health problems. Cohort studies assess, and often compare, the incidence of a condition within a group of people who share common characteristics (e.g., being born in the same year) [ 1 ]. A key advantage of longitudinal cohort studies over other research designs is that repeated measures data temporally orders exposures and outcomes to facilitate causal inference [ 2 ]. However, significant and systematic attrition can reduce the generalisability of outcomes and the statistical power to detect effects of interest [ 3 ]. Systematic attrition in longitudinal research occurs most often in older, non-white male participants with limited education and/or multiple health problems [ 4 ]. Long duration and repeated assessments can also increase attrition due to the significant burden on participants [ 4 ]. Given the expense of longitudinal cohort studies, effective strategies that engage and retain cohort participants are critical to the integrity of research outcomes [ 5 , 6 ].

In the last decade, longitudinal data collection methods and cohort retention strategies have evolved considerably. So too have participant expectations of organisations (research and otherwise) that seek information from individuals [ 7 , 8 ]. Established retention strategies within longitudinal cohort studies include: cash or gift incentives, sending reminder letters to participants, re-sending surveys, and offering alternative methods of data collection (for a review, see [ 6 ]). Booker et al. [ 6 ] demonstrated that these strategies were effective in longitudinal cohort studies that used the traditional data collection methods of postal surveys, face-to-face visits (home or on-site), and telephone interviews or surveys. However, these cohort retention strategies may not be as well suited to contemporary methods of collecting longitudinal data, such as web and mobile surveys [ 9 ], wearable sensors (e.g., FitBits) [ 10 ], short message services (SMS) [ 11 ], and groupware systems (e.g., video conferencing) [ 12 ]. Novel methods of engaging participants such as web advertising [ 13 ], social media [ 14 ], and electronic reminders [ 15 ], are also now being employed in cohort studies using both traditional and modern longitudinal data collection methods.

A systematic review on the effectiveness of established and emerging cohort retention strategies in longitudinal cohort studies would provide guidance to researchers and funders on maximising cohort maintenance within these high investment programs of research. Previous reviews of retention strategies in health research include [ 4 , 6 , 16 , 17 ]; only one of these reviews focused specifically on longitudinal cohort research designs [ 6 ]. Booker et al. [ 6 ] conducted a narrative review of retention strategies in longitudinal cohort studies, including incentives, reminders, repeat visits/questionnaires, and alternative methods of data collection, finding that incentives and reminder strategies improved cohort retention. However, this review was limited by the small number of studies identified for each retention strategy, which resulted in the identification of a restricted breadth of retention strategies and the inability to synthesise findings empirically.

Further, Booker et al. [ 6 ] did not include research completed after 2006 and thus were unable to investigate emerging cohort retention strategies. Brueton et al. [ 16 ] completed a more recent review of retention strategies that included both established and emerging digital data collection retention strategies. However, the authors specifically excluded longitudinal cohort studies and instead focused on participant retention in intervention trials. Differences between intervention and longitudinal cohort studies, such as research design factors (e.g., study duration) and the motivations of the participants in joining or withdrawing from studies, may impact the usefulness of retention strategies across both study designs [ 4 , 6 ].

A review of retention strategies reported in modern longitudinal cohort studies is pertinent and timely, given the emergence of digital retention strategies alongside established retention methods. Maximising cohort retention in longitudinal research can reduce the administration costs of conducting research, improve the efficiency of research processes, and reduce outcome biases for studies by adopting an evidence-based cohort retention framework. In this review, we aimed to: (i) identify retention strategies used in recent longitudinal cohort studies; (ii) examine whether retention rate was moderated by different study or participant characteristics (i.e., number of waves, study duration, sample size, population type, gender, age, country); (iii) estimate the retention rate in studies that use specific retention strategies, and contrast this retention rate with studies that do not use specific retention strategies; (iv) examine whether retention rate is associated with the number of retention strategies used; (v) examine which retention strategies were the strongest independent predictors of retention rate; and (vi) contrast the retention rate based on whether studies utilised emerging or established strategies. Moreover, to ensure that recent innovations in retention strategies were identified, this review focused on literature published within the past 10 years.

Search strategy

A systematic review was performed as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 18 ]. Two search strategies were implemented. First, the electronic databases Medline, PsycINFO, Embase, CINAHL, AMED, and the Cochrane Library, were searched in July, 2016 using search terms relevant to three themes: (i) attrition, (ii) retention, and (iii) study design (Additional file 1 : Table S1). The electronic search was limited to articles published from 2006 onwards in English, to avoid duplication of literature with the previous review on this topic [ 6 ]. The search was adapted to suit each database. Second, the reference lists of all articles selected for review were manually searched.

Inclusion and exclusion criteria

The inclusion and exclusion criteria were determined prior to implementing the search strategy. Articles were included in the review if: (i) the article described a cohort study, which was defined as a representative sample of a group or population who share a common experience or condition [ 2 ], (ii) the article reported at least one wave of follow-up data collection with a participant/proxy, (iii) participant retention data were reported, and (iv) retention strategies were reported. Articles were excluded if: (i) the article was not available in English; (ii) the article was not published in a peer-reviewed publication (e.g., conference abstracts or dissertations); and, (iii) the article’s research design was cross-sectional, involved data linkage only, or the article was a clinical or non-clinical trial evaluating the effectiveness of treatment regimens or intervention/prevention programmes (for an existing review of retention in intervention studies, see (12)).

Study selection, data extraction, and quality assessment

The search strategy resulted in 9225 articles after removing duplicates. In total, 141 articles were identified, screened, and determined to be eligible for inclusion (see Fig.  1 ). Data were extracted and summarised for each of the 141 articles on: (i) the research design, including baseline sample population and sample size, the number of data collection waves reported, and the duration in years between the first and last waves of data collection; (ii) the cohort demographics, including mean sample age at baseline (or age range if mean age was not reported), proportion of male participants, country of cohort participants, and whether the cohort was clinical or non-clinical; and (iii) retention data, including the retention rate between baseline and the final data collection wave reported, and the specific retention strategies. Finally, we examined the suitability of each article in addressing the current study’s research question. Articles that listed cohort attrition or retention as a research question or objective were categorised as “retention-focused”, and conversely articles that did not focus on attrition or retention were categorised as “non-retention-focused”. No articles were excluded on the basis of this quality assessment.

PRISMA procedural flow chart of the search and identification process

Statistical method

We used meta-analysis (and meta-regression) to address the aims of the study. Meta-analyses were conducted using the Metafor package v1.9.8 [ 19 ] in R software v3.3.1 [ 20 ]. The retention rate, defined as the number of individuals who remained in the study at the last wave of data collection as a proportion of the total number of participants recruited at the baseline assessment, was the primary effect size measure of interest. All meta-analyses were conducted using inverse variance weighting, with random effects specified to account for between study heterogeneity. A binomial-normal model (with logit link) was used as the basis for analysis, which is appropriate when the effect size of interest is measured as a proportion. Where appropriate, meta-analytic effects were back-transformed to represent the median meta-analytic retention rate. We also report the I 2 statistic as a measure of study heterogeneity, interpreted using the guidelines of Higgins et al. [ 21 ]. Meta-analyses were conducted when at least two independent studies contributed to the meta-analysis.

To examine the effect of gender on retention rate, we created a binary variable to denote studies as comprising a higher proportion of either male or female samples (Proportion of male participants in “male” grouping: M(SD)  = 73.6%(0.20); Proportion of female participants in “female” grouping: M(SD)  = 75.0%(0.21)). To examine the effect of country development level on retention rate, each study country was categorised as either high or low development level by using a mean-split of each nation’s Human Development Index – a measure of relative opportunity for longevity, education, and income, with a score range of 0 (low) to 1 (high) (Low HDI group M(SD)  = 0.66(0.10); High HDI group M(SD)  = 0.92(0.02)) [ 22 ]. Retention strategies were coded as either established or emerging , depending on their presence or absence in any of the earlier systematic reviews on participant retention strategies [ 4 , 6 , 16 , 17 ]. Finally, all meta-regressions adjusted for study duration and number of waves (except when these were specifically examined as predictor variables), given these were deemed to be likely confounding variables in analyses.

Cohort, participant, and article characteristics

The 141 articles identified for review described 143 cohorts (41 clinical and 102 non-clinical). Cohorts are summarised in Table ​ Table1. 1 . Overall the mean sample size reported in the first wave of each article was 3585 participants (range = 30 to 61,895). Articles reported a mean retention rate of 73.5% (SD = 20.1%), with 4.6 waves (SD = 8.0), over 4.3 years (SD = 5.0). The average baseline participant age was 30.0 years (SD = 22.0), and the average baseline proportion of male participants across samples was 40% (SD = 0.30). Studies were conducted in 28 different countries with a mean Human Development Index of 0.79 (SD = 0.15), indicating that studies were more likely to be conducted in countries with high-levels human development. Cohort attrition/retention was identified as a specific research question or objective of interest in 55 of the 141 articles, indicating that most articles were not focused on participant retention. Retention-focused articles reported significantly more retention strategies than non-retention-focused articles (non-retention-focused: M(SD)  = 3.3(3.1); retention-focused: M(SD)  = 11.0(7.02); t (141)  = − 9.00, p  < .001); however, no differences were found for the study sample size, number of waves, study duration, or retention rate. High heterogeneity was identified in all results, as expected given the diversity of research questions, methodologies, and cohorts across articles [ 21 ].

Description of cohorts reported in the included articles

Relationship between retention rate and study or participant characteristics

To examine whether retention rate was moderated by study characteristics (i.e., number of waves, study duration, sample size, study focus on retention strategies or not) or by participant characteristics (i.e., population type, gender, age, country development level), a series of meta-regressions was performed, one for each characteristic under examination. Retention rate was not moderated by: number of waves (b < 0.001; 95%CI [− 0.02 to 0.03], p  = .77); study duration (b = − 0.02; 95%CI [− 0.06 to 0.02]; p  = .34); sample size (b < − 0.001; 95%CI [− 0.00 to 0.00]; p  = 0.48); or articles’ focus on retention strategies (b = − 0.12; 95%CI [− 0.54 to 0.30]; p  = .57). Additionally, retention rate was not associated with the sample characteristics of: cohort type (clinical or non-clinical) (b = 0.04; 95%CI [− 0.42 to – 0.51]; p  = .86); mean age (b = 0.02; 95%CI [− 0.01 to 0.01]; p  = .74); or country development level (b = 0.11; 95%CI [− 0.46 to 0.68]; p  = .71). However, gender was a significant moderator of retention rate (b = − 0.67; 95%CI [− 1.14 to − 0.20]; p  < .01)). Namely, cohorts with more female participants (median retention = 81.5%, 95%CI [77.6% to 84.9%]) reported higher retention rates than articles with more male participants (median retention = 70.1%; 95%CI [60.1% to 78.5%]), after controlling for study duration and number of waves.

Relationship between retention rate and retention strategy types

A total of 95 retention strategies was identified, with an average of 6.2 strategies per article (SD = 6.2). The most common retention strategies were: cash/voucher incentives to complete a follow-up assessment ( n  = 59), sending a postcard or letter reminder to complete a follow-up assessment ( n  = 43), and offering participants alternative methods of data collection, such as completing an interview face-to-face or over the phone ( n  = 36).

Retention strategies were grouped into four main retention strategy domains: (i) barrier-reduction strategies , such as offering childcare services, assistance with parking and transport, and engaging a participant sub-sample to evaluate data collection methods for the next wave; (ii) community-building strategies , such as creating a recognisable study brand via logos and colour schemes, giving away study merchandise to create a sense of project community (e.g., t-shirts with study logo), and sharing study results, news and events with participants via newsletters, social media, and feedback reports; (iii) strategies to improve follow-up rates within each wave , including cash or voucher incentives for varying levels of assessment completion, and use of phone calls, SMS, house visits, mail and email reminders to participants to complete assessments; and (iv) tracing strategies , such as collecting the details of alternative contact persons for each participant at baseline, using public or non-public records to find updated contact information for participants, and collecting detailed participant contact information via a locator document (e.g. full name, address, social security number, phone numbers, email addresses, etc.). The most commonly reported category was strategies to improve follow-up rates within waves, identified 306 times within the 143 cohorts, followed by barrier-reduction strategies (adopted 268 times), community-building strategies (adopted 181 times), and tracing strategies (adopted 138 times).

Table  2 presents the retention strategies used, grouped by retention strategy domain. It compares the retention rate for those studies that did, or did not utilise a specific retention strategy type or domain. Of the 95 individual retention strategies examined, three demonstrated moderation of the retention rate. First, improved retention was associated with offering participants alternative methods of data collection (e.g., completing an interview face-to-face or over the phone) (median retention using strategy = 86.1%; median retention not using strategy = 76.3%; b = 0.24, p  = .01), and having participants complete a locator document at baseline (median retention using strategy = 90.9%; median retention not using strategy = 78.1%; b = 0.49, p  = 0.02). Finally, lower retention was associated with use of phone call reminders to participants to complete a follow-up wave (median retention using strategy = 72.7%; median retention not using strategy = 80.6%; b = 0.25, p =  .05). There was weak evidence against the null hypothesis of no moderation effect for a further three strategies. This included having consistent research team members (median retention using strategy = 87.3%; median retention not using strategy = 78.1%; b = 0.67, p  = .09); offering site and home visits for data collection (median retention using strategy = 83.9%; median retention not using strategy = 77.4%; b = 0.46, p  = .07); and sending participants thank you, birthday or holiday cards (median retention using strategy = 84.9%; median retention not using strategy = 77.5%; b = 0.50, p  = .07). There was no evidence to support a moderated retention rate by any other specific retention strategy type.

Median meta-analytic retention rates for each retention strategy

All inferential analyses adjusted for study duration and number of waves

na insufficient studies to perform meta-analysis

N No. effect in analysis

* p  < .05

** p  < .01

To examine whether the specific strategy domains of barrier-reduction, community-building, follow-up/reminder, and tracing retention strategies were associated with retention rate, a binary variable was created for each domain that denoted whether a study did or did not utilise one or more specific strategy types within that domain. As shown in Table ​ Table2, 2 , after controlling for study duration and number of waves, studies that utilised any barrier-reduction strategy had higher retention rates than those that did not use a barrier strategy (median retention using barrier strategies = 81.1%; median retention not using barrier strategies = 70.7%; b = 0.61, p =  .01). Again after controlling for the study duration and number of waves, surprisingly, articles that reported use of at least one follow-up/reminder strategy had lower retention rates when compared to studies that did not utilise any follow-up/reminder (median retention using follow-up/reminder strategies = 76.4%; median retention not using follow-up/reminder strategies = 86.1%; b = − 0.32, p  < .01). No relationships were found between retention rate and the use of any community-building or tracing retention strategies.

Relationship between retention rate and number of strategies used

To examine whether the cumulative number of retention strategies was associated with retention rate, we meta-regressed retention rate on to continuous variables representing the cumulative number of strategies used across strategy domains, and then within each domain separately. Greater number of retention strategies used (across all domains) was not associated with higher retention rate (b = 0.02; 95%CI [− 0.12 to 0.05], p  = .21). When examined within each domain, controlling for study duration and number of waves, we found accumulation of barrier-reduction strategies was associated with higher retention (b = 0.12; 95%CI [0.02 to 0.22]; p =  .02). In separate meta-regressions, no relationships with retention were identified between number of community-building strategies (b = − 0.03; 95%CI [− 0.18 to 0.11]; p =  0.63), follow-up strategies (b = − 0.03; 95%CI [− 0.14 to 0.09]; p  = 0.65), or tracing strategies (b = 0.10; 95%CI [− 0.07 to 0.28]; p =  .25).

Identifying strongest independent predictors of retention rate

Three separate meta-regression models were estimated to examine strongest predictors of retention rate within strategy domains and types. Table  3 -Model 1 shows that when examining retention strategy types as cumulative variables for each domain, barrier-reduction was independently associated with higher retention (b = 0.17; 95%CI [0.03 to 0.31]; p  = .02) and follow-up strategies was independently associated with lower retention (b = − 0.15; 95%CI [− 0.29 to − 0.01]; p  = .04) beyond the effects of other retention strategy types. By contrast, Table ​ Table3-Model 3 -Model 2 demonstrates that when the retention rate was regressed on to all the binary indicator variables denoting whether the study did or did not utilise at least one strategy within that domain, only the use of follow-up/reminder strategies was independently associated with reduced retention rate (b = − 0.83; 95%CI [− 1.4 to − 0.27]; p  < .01).

Meta-analytic regression results between retention strategy themes and retention rate

* p < .05

** p < .01

Finally, we investigated whether the associations between individual strategies and retention rate remained after controlling for other effective individual strategies in a single model (see Table ​ Table3 3 Model 3). A meta-regression model was created by entering only individual retention strategies that were associated with a retention rate at the p  < .10 level (as discussed in [ 23 , 24 ]). Six individual strategies were eligible: (i) offering alternative methods of data collection; (ii) consistency in the research staff; (iii) offering site and home visits; (iv) thank you and birthday cards; (v) phone call reminders; and (vi) the use of a locator form (i.e., alternate contacts). Offering participants alternative methods of data collection was associated with improved retention, whilst the use of phone call reminders was associated with reduced retention (b = 0.59; 95%CI [0.13 to 1.05]; p =  0.01; b = − 0.72; 95%CI [− 1.18 to − 0.25]; p  < .01, respectively). No associations were found between retention rates and the remaining four individual strategies.

Relationship between retention rate and emerging strategies

The final group-level analysis investigated the association between emerging retention strategies and retention rates. Within these 95 retention strategies, 44 emerging strategies were identified, including the application of social media and SMS to assist in tracing participants lost to follow-up, and the application of study websites and social media profiles for keeping participants up-to-date with the study’s news and events. Meta-regressions demonstrated that articles reporting a higher frequency of emerging retention strategies had higher retention, after controlling for study duration and number of waves (b = 0.08; 95%CI [0.01 to 0.16]; p =  .03). Despite this, there was no difference in overall retention rates between those articles that did and did not report the use of emerging retention strategies (median retention using emerging strategies = 80.1%; median retention not using emerging strategies = 75.0%; b = 0.27, p  = .27).

This study aimed to identify retention strategies employed in longitudinal cohort studies during the past decade, and to examine their effectiveness. We identified 143 longitudinal cohort studies that described retention strategies and outcomes, resulting in 95 different retention strategies. We then investigated whether study or participant characteristics moderated retention, the relationship between retention rate and retention strategy type, and whether new cohort retention strategies have emerged since previous reviews. In so doing, this study is the first meta-analysis of retention strategies conducted in longitudinal cohort studies. This research particularly complements the previous narrative review that investigated cohort retention strategies in longitudinal research [ 6 ], and the wider literature investigating participant retention strategies across health research designs (e.g., 4,16,17). Such research has important implications for maximising cohort retention and reducing research administration costs, which will subsequently improve the efficacy and quality of health research.

We first investigated how study or participant characteristics may influence cohort retention. Study characteristics included sample size, study duration, number of waves, and country development level - none of which were associated with retention rate. Participant characteristics included mean age at baseline, cohort type (clinical or non-clinical), and gender. We found that cohort studies with a higher proportion of male participants had lower retention rates than studies with a higher proportion of female participants; no associations were found for participants’ age or cohort type. While difficulties in retaining male participants are well-documented in previous research (e.g., 4,20,21), our study noted that cohorts with a higher proportion of male participants were also more likely to be clinical samples than cohorts with a higher proportion of female participants. In addition, cohorts with a higher proportion of male participants were also disproportionately focused on high-risk groups, such as substance use and men who have sex with men (e.g., the Bangkok Men who have Sex with Men Cohort Study (BMCS) [ 25 ] and the International Multicenter ADHD Genetics (IMAGE) study [ 26 ]). Thus, the difficulties in retention reported in this study and the wider literature could potentially be attributed to the differential impact of these clinical issues that affect men more than women. Researchers working with hard-to-retain populations, such as men in particular clinical groupings, may benefit from investigating what retention strategies work within their specific populations and settings beyond the core retention strategies identified in this review.

Second, we investigated the relationship between retention rate and retention strategies. We identified 95 different retention strategies, grouped thematically into four classes: barrier-reduction, community-building, follow-up, and tracing . Specific strategies associated with improved retention rates included the barrier-reduction strategy of offering alternative methods of data collection to participants (e.g., completing an interview over the phone or in person); and the tracing strategy of collecting detailed contact information from participants at baseline via a locator document. Further, weak evidence was found for one community-building and two further barrier-reduction strategies: (i) sending participants thank you, birthday or holiday cards; (ii) having consistent research team members, and; (iii) offering site and home visits for data collection.

Overall, barrier-reduction strategies emerged as the strongest predictor of improved retention. Barrier-reduction strategies may be particularly useful in longitudinal research given participants are likely to experience significant changes in their capacity to remain involved across the study’s duration (typically years). Follow-up/reminder strategies, such as incentives and reminders, were associated with significantly poorer retention. This result was surprising, given that the previous review investigating retention strategies in longitudinal cohort studies found the opposite, that use of these follow-up/reminder strategies resulted in improved retention rates [ 6 ]. The lack of support for follow-up/reminder strategies found in the current review could be due to a number of extraneous variables including: (i) timing: studies may have implemented this strategy after other retention efforts proved ineffective; (ii) participant burden: the studies using follow-up/reminder strategies may have involved a high data collection burden (e.g., long surveys); (iii) sampling: studies using follow-up/reminder strategies may be over-represented in studies of difficult-to-retain populations, such as men. However, these explanations are unlikely, given that follow-up/reminder strategies were identified in most of the cohorts included in this review (111 out of the 143 cohorts), and the cohorts employing follow-up/reminder strategies did not differ by research design (sample size: t (141)  = .67, p  = .50; no. waves: t (141)  = −.43, p  = .67) or participant characteristics (age: t (141)  = −.11, p  = .91; gender: χ 2 (2, n  = 143)  = .37, p  = .85; HDI: χ 2 (2, n = 143)  = .01, p  = .97). Differences were observed only for study duration (any M (SD) = 3.9(4.4); none (SD) = 5.8(6.4); t (141)  = 2.00, p  = .05). Alternatively, participants may perhaps view follow-up/reminder strategies as the research team “badgering” them to complete assessments, thereby damaging rapport. This negative perspective of follow-up/reminder strategies may be further exacerbated if the research team has not implemented sufficient barrier-reduction strategies to help make it easier for participants to remain involved in the study. Future research could consider investigating participants’ perspectives of retention strategies in longitudinal cohort studies, ensuring that both active and inactive participants are included, to better understand the costs and benefits of different approaches.

Interestingly, the current study found that simply adding more cohort retention strategies did not result in higher retention rates. These results contradict the findings of Robinson et al. [ 17 ] and Davis et al. [ 4 ], who both found that the use of more retention strategies across multiple classes was associated with improved retention rates. However, neither study specifically examined participant retention in longitudinal cohort studies, and both synthesised their retention results using a narrative rather than meta-analytic approach. Given that the implementation of retention strategies can be costly in terms of both time and money, the overall number of strategies employed is important to evaluate. The interaction of quantity of retention strategies used and provision of flexibility needs to be better understood, given research protocols that accommodate the changing lives of participants should remain a key focus of retention efforts.

Finally, we examined whether studies utilising new or emerging retention strategies had improved retention compared with studies using established strategies. Of the 95 retention strategies described in the included articles, 44 were identified as an emerging retention strategy that had not yet been described in extant systematic reviews examining participant retention [ 4 , 6 , 16 , 17 ]. Emerging strategies included using social media and SMS to assist in tracing participants lost to follow-up, and the use of study websites and social media profiles for keeping participants up-to-date with study news and events. Emerging retention strategies were endorsed by only a handful of studies, and the use of a single emerging strategy was not significantly associated with retention rate. However, we found that studies that employed more emerging retention strategies were associated with improved retention rates. Importantly, emerging strategies were identified across all four retention strategy domains (barrier-reduction, community-building, follow-up/reminder, and tracing), demonstrating that the association between emerging strategies and improvements in retention are due to the use of modern technology to help achieve core cohort engagement goals. Thus, we recommend that researchers continue to innovate their retention efforts, particularly where such strategies may reduce participant burden.

The current study has a number of limitations. First, the number of articles that focused on reporting retention strategies in detail was proportionally low compared to the number of articles that did not focus on reporting retention strategies. Although retention strategies were identified within 143 longitudinal cohorts, only 55 included cohort retention as a key focus area. Very few articles ( n  = 12) were identified that reported strategy-specific retention rates within the longitudinal cohort studies. The number of retention strategies reported by articles ranged from one to 32, with 35 of the 141 articles describing only one retention strategy. Longitudinal cohort studies should aim to publish protocol papers that delineate their cohort retention strategies, and ensure that the protocol is updated as retention efforts evolve.

Second, net retention rates were calculated by the difference between the first and last wave of data collection reported in the article. Where specified, ineligible participants (e.g., participants recruited after the first wave, or deceased participants) were excluded from the retention rate calculation. However, some articles did not provide detailed information on the eligibility of the sample at the final wave, and thus it is possible that the retention rates calculated for some studies may be slightly inaccurate. This limitation could be addressed by researchers providing details on the eligibility of their samples at each wave.

Third, high levels of heterogeneity were reported for most analyses in this study. This may best be explained by two factors. First, we expected to identify high heterogeneity given the diversity of research questions, methodologies, and cohorts reported across articles. Second, only a small number of studies were eligible for most meta-regressions in this paper, which reduces the precision of heterogeneity estimates [ 27 ]. This limitation could be addressed in future work, which could aim to investigate the effectiveness of different retention strategies within different subgroups.

Finally, by nature of synthesising retention results across different samples and settings, the current study is unable to disaggregate nuanced effects of various retention strategies across specific contexts and populations, given results are pooled across multiple studies. The current study did address this broadly by investigating the effects of study and sample characteristics on retention.

A final point to note is that available to researchers are a range of statistical or methodological approaches that can minimise potential biases introduced with attrition. Whilst beyond the scope of this paper, these approaches include formal statistical methods for addressing missingness due to attrition such as multiple imputation or full information maximum likelihood methods [ 28 , 29 ]. Moreover, researchers may address attrition methodologically by using replacement sampling approaches that recruit new participants into a study to replace those who have dropped out, based on shared characteristics measured in the original sampling frame [ 30 , 31 ]. All these methods provide useful avenues to address attrition once any employed retention strategies have been used to retain the largest proportion of the original sample as possible.

Overall, this study has important implications for the retention efforts of longitudinal cohort studies. Combined, these results suggest that researchers need to be strategic in choosing how to invest their resources to better target participant retention, rather than simply increasing the number of strategies applied. Projects should invest both time and funding into matching retention strategies to the sample prior to implementation, including careful consideration of unintended burden for participants. Finally, given the high number of emerging retention strategies identified, longitudinal research methods clearly continue to evolve. Longitudinal cohort studies may benefit from open and regular protocol revision to incorporate new strategies, particularly where these strategies may offer greater flexibility to participants.

Additional file

Table S1. Terms used in the electronic search strategy, adjusted as required for each database. (DOCX 12 kb)

Acknowledgements

This paper represents a collaborative effort on behalf of the Centre for Social and Early Emotional Development (SEED) Lifecourse Sciences Theme. The contribution from authors to this paper was part of a broader collaborative effort from SEED Lifecourse Sciences Theme members. We request that investigators who are members of the theme be listed at the end of the manuscript. Contributions made by other individuals have been acknowledged in the usual way.

^SEED Lifecourse Sciences Theme : Including the primary investigators already listed and: Sharyn Bant, Sophie Barker, Anna Booth, Tanja Capic, Laura Di Manno, Alisha Gulenc, Genevieve Le Bas, Primrose Letcher, Claire Ann Lubotzky, Gypsy O’Dea, Jessica Opie, Melissa O’Shea, Evelyn Tan, and Jo Williams.

This work was supported by the Centre for Social and Early Emotional Development (SEED), Faculty of Health, Deakin University Geelong. Open-access publishing costs for this work were supported by a Deakin University Faculty of Health HDR Open-Access Publishing Grant. DH is financially supported by an Australian Unity Senior Research Fellowship; and CO is financially supported by an Australian Research Council Principal Research Fellowship.

Availability of data and materials

Abbreviations.

Authors’ contributions

ST conceptualised and designed the study, completed the data collection, carried out the analyses, contributed to the interpretation of the data, and wrote the manuscript. GY carried out the analyses, and contributed to the interpretation of the data, the write-up of the results and preparation of tables, and revision of the manuscript. JAM contributed to the data analysis, interpretation of the results, and the write-up and revision of the manuscript. ES conceptualised and designed the study, supervised data collection, and contributed to the interpretation of the data and revision of the manuscript. AS contributed to the data collection, interpretation of the results, and revision of the manuscript. MF contributed to the data analysis, interpretation of the results and revision of the manuscript. CG, JM, and CO critically reviewed and revised the manuscript for important intellectual content. DH conceptualised and designed the study, coordinated and supervised data collection and the preparation of data for analysis, and contributed to the interpretation of the data and revision of the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Samantha Teague, Email: [email protected] .

George J. Youssef, Email: [email protected] .

Jacqui A. Macdonald, Email: [email protected] .

Emma Sciberras, Email: [email protected] .

Adrian Shatte, Email: [email protected] .

Matthew Fuller-Tyszkiewicz, Email: [email protected] .

Chris Greenwood, Email: [email protected] .

Jennifer McIntosh, Email: [email protected] .

Craig A. Olsson, Email: [email protected] .

Delyse Hutchinson, Phone: 0425 218 628, Email: [email protected] .

the SEED Lifecourse Sciences Theme: Sharyn Bant , Sophie Barker , Anna Booth , Tanja Capic , Laura Di Manno , Alisha Gulenc , Genevieve Le Bas , Primrose Letcher , Claire Ann Lubotzky , Jessica Opie , Melissa O’Shea , Evelyn Tan , and Jo Williams

• Open access
• Published: 22 August 2024

Patterns of health workforce turnover and retention in Aboriginal Community Controlled Health Services in remote communities of the Northern Territory and Western Australia, 2017–2019

• Prabhakar Veginadu   ORCID: orcid.org/0000-0002-5723-0996 1 ,
• Deborah J. Russell 1 ,
• Yuejen Zhao 2 ,
• Steven Guthridge 3 ,
• Mark Ramjan 4 ,
• Michael P. Jones 5 ,
• Supriya Mathew 1 ,
• Michelle S. Fitts 6 ,
• Lorna Murakami-Gold 7 ,
• Narelle Campbell 8 ,
• Annie Tangey 9 ,
• John Boffa 10 ,
• Bronwyn Rossingh 11 ,
• Rosalie Schultz 9 ,
• John Humphreys 12 &
• John Wakerman 1  

Human Resources for Health volume  22 , Article number:  58 ( 2024 ) Cite this article

134 Accesses

Metrics details

Aboriginal Community Controlled Health Services (ACCHSs) in Australia aim to optimise access to comprehensive and culturally safe primary health care (PHC) for Aboriginal populations. Central to quality service provision is the retention of staff. However, there is lack of published research reporting patterns of staff turnover and retention specific to ACCHSs. This study quantified staff turnover and retention in regional and remote ACCHSs in the Northern Territory (NT) and Western Australia (WA), and examined correlations between turnover and retention metrics, and ACCHSs’ geographical and demographic characteristics.

The study used 2017–2019 payroll data for health workers in 22 regional and remote PHC clinics managed by 11 ACCHSs. Primary outcome measures included annual turnover and 12-month stability rates, calculated at both clinic and organisation levels.

There was a median of five client-facing (Aboriginal health practitioners, allied health professionals, doctors, nurses/midwives, and ‘other health workers’ combined) and two non-client-facing (administrative and physical) staff per remote clinic, at any timepoint. Mean annual turnover rates for staff were very high, with 151% turnover rates at the clinic level and 81% turnover rates at the organisation level. Mean annual turnover rates for client-facing staff were 164% and 75%, compared to 120% and 98% for non-client-facing staff, at clinic and organisational levels, respectively. Mean 12-month stability rates were low, with clinic-level stability rates of only 49% and organisation-level stability rates of 58%. Mean annual clinic-level turnover rates were 162% for non-Aboriginal staff and 81% for Aboriginal staff. Both workforce metrics were moderately to highly correlated with the relative remoteness of clinics, size of regular clients serviced, and average annual headcount of employees in each clinic ( p values < 0.01).

Conclusions

Participating ACCHSs in remote NT and WA have very high turnover and low retention of healthcare staff. Overall, clinic-level turnover rates increase as distance from regional centres increases and are lower for Aboriginal staff, suggesting that greater employment of Aboriginal staff could help stabilise staffing. Improved retention could reduce burden on ACCHSs’ resources and may also support quality of service delivery due to improved cultural safety and continuity of care.

Peer Review reports

Introduction

Timely and continued access to appropriate primary health care (PHC) is critical for the overall well-being of individuals [ 1 ]. Access to PHC decreases with increasing geographical remoteness [ 2 ], and a key contributor to poor access in the remote context is the limited availability of a stable health workforce [ 3 ]. The quality of care and patient outcomes are improved when care is provided by the same PHC professional over time [ 4 ]. In rural and remote contexts, continuity of patient care and the development of trusting relationships between patients and their PHC providers may be significantly impeded by the constant movement of health workers into and out of these communities [ 5 , 6 ]. An unstable workforce has a negative impact on patient engagement with local PHC services [ 7 ]. Decreased access to PHC is associated with suboptimal health outcomes, such as increased hospitalisations [ 8 ].

In Australia, Aboriginal Community Controlled Health Services (ACCHSs) and state or territory governments operate PHC clinics which provide services to remote Aboriginal communities. ACCHSs aim to improve access to comprehensive PHC for Aboriginal populations in Australia, and as the name suggests, are governed by boards comprising, mainly, local community members, which reflects community ownership of local health services [ 9 ]. ACCHSs are underpinned by a model of PHC tailored to meet the needs and expectations of local communities [ 10 ], thereby reducing the most common barriers to healthcare access for Aboriginal people relating to cultural appropriateness and acceptability [ 11 ] and tackling the adverse effects of disempowerment and racial discrimination on their health [ 12 ]. ACCHSs seek to employ staff from local Aboriginal communities. In 2021–22, among the PHC services funded by the Commonwealth Indigenous Australians’ Health Program, 52% of total full-time equivalent positions in ACCHSs were held by Aboriginal people, compared to 38% in non-ACCHSs [ 13 ]. Previous research shows that in remote clinics operated by the Northern Territory (NT) Government Department of Health (DoH), the likelihood of staff turnover is lower in Aboriginal staff compared to non-Aboriginal staff; however, there are no comparable data for remote ACCHSs [ 14 ].

Qualitative research notes that increasing health workforce stability (and reducing turnover) are essential aspects of strengthening the ACCHS workforce [ 15 ], and staff turnover and retention are amongst the most important service delivery issues experienced by ACCHSs [ 16 ]. Two key measures of turnover and retention relevant to the Australian rural and remote workforce context are the annual turnover rate and 12-month stability rate [ 17 ]. In NT DoH remote clinics the mean annual turnover rates of nurses and Aboriginal health practitioners (AHPs) were 148% and 79%, respectively, while mean 12-month stability rates were 48% and 76%, respectively [ 14 ].

The objectives of this study, therefore, were to: a) quantify the most recent pre-COVID-19 pandemic (2017–2019) PHC staff turnover and retention rates in 11 ACCHSs in regional and remote NT and Western Australia (WA); and b) examine the correlations between workforce turnover and retention metrics and ACCHS clinic characteristics, such as geographical remoteness, regular client service population, and clinic workforce size.

Study setting

The study is set in two Australian jurisdictions, NT and WA, which cover over 50% of Australia’s landmass (4 million km 2 ). There are a total of 39 ACCHSs providing services in these two jurisdictions [ 18 ], of which 11 participated in this study. Participating ACCHSs provided PHC services to 30 communities (via local clinics or outreach services), and collectively serviced around 63,500 Aboriginal people between 2017 and 2019. The ACCHSs serviced regional, remote or very remote communities.

The study used payroll system data extracted from the administrative databases of the participating ACCHSs, from 1 January 2017 to 31 December 2019. The data comprised de-identified individual-level information on all employees who received payments directly from the ACCHSs’ payrolls. The participating ACCHSs also recruit some employees through employment agencies, for example, remote area nurses and midwives (hereafter referred to as RANs) and general practitioner (GPs) locums. Those RANs sourced via agencies generally receive payments directly from their recruiting agency rather than through the ACCHSs’ payrolls. While the nature of agency employment model innately contributes to turnover in health services, data about agency-employed RANs were not included in the analyses as they were not consistently available in the payroll data. The measures reported in this study, therefore, relate only to directly employ permanent and casual (non-agency and non-locum) staff.

The structure of the payroll system and corresponding software differed between the ACCHSs. As a result, there were considerable variations in the type and nature of data elements in the payroll data provided for this study. The data cleaning procedure included ensuring key payroll variables were consistently prepared for the analyses (see Supplementary file for details). Staff were assigned to one of seven employment categories based on their role description: (1) administrative (e.g., reception staff, finance officers, human resource staff, policy officers, managers, etc.); (2) AHPs (i.e., healthcare practitioners registered with the Aboriginal and Torres Strait Islander Health Practice Board of Australia); (3) allied health professionals; (4) doctors; (5) RANs; (6) other health workers such as community liaison officers, health promotion officers, counsellors, etc.; and (7) physical grades such as drivers, cleaners, gardeners, tradespeople, maintenance staff, etc. For analyses, AHPs, allied health professionals, doctors, nurses, and other health workers were further grouped as ‘ client-facing ’ staff; administrative and physical categories were grouped as ‘ non-client-facing ’.

Turnover and retention metrics were calculated at two levels: (a) community or clinic level and (b) service-wide or organisation level . Organisation-level analysis included 11 participating ACCHSs, while analysis at the clinic level was restricted to 22 clinics serviced by the ACCHSs. The decision to exclude eight clinics for the clinic-level analysis was to ensure that only clinics that were actively running and staffed for the entire study period, were included. For example, there were some clinics in the participating ACCHSs that had transitioned from NT DoH to community-controlled administration during the study period, and therefore, only had partial data available for analysis (see Supplementary file for details). Excluding such clinics enabled reasonable comparison among the ACCHSs when grouped into categories for further analyses.

Employee exits were defined at the clinic and organisation levels, following the approach by Russell et al . [ 14 ]. Exits were measured at the clinic level (clinic-level turnover) if an employee left one of the 22 clinics in which PHC services were being delivered for a period of more than 12 weeks. Employee movement between different clinics, even while remaining employed by a given ACCHS, was thus considered an exit, according to this definition. Second, an exit was measured at the organisation level if an employee ceased (for more than 12 weeks) employment with the ACCHS. In this instance, movements between clinics while remaining employed by a single ACCHS were not measured as an exit. Defining exits at these two levels facilitates better understanding of the direct impact of staff turnover on interpersonal continuity of care for patients attending individual clinics (clinic level) and overall disruption to the functioning of the ACCHS (organisation level).

Health workforce supply was measured using two metrics at each of clinic and ACCHS levels—total number of unique employees and average headcount, defined as follows:

Total number of unique employees = number of individuals employed during 3 years

Average headcount = average number of individuals employed at any given point in time during 3 years

Two key workforce turnover and stability metrics were calculated:

Summary statistics for each key workforce metric were analysed at the organisation and clinic level for all staff, and according to their employment category (i.e., client-facing and non-client-facing staff) and Aboriginal status (i.e., Aboriginal and non-Aboriginal staff). Analyses were conducted by Aboriginal regular client service populations (hereafter referred to as ‘service population’) and by distances between the remote clinic and the nearest regional centre. The service population were defined as those Aboriginal clients who attended an ACCHS PHC clinic three or more times in the 24 months immediately preceding each study year (2017–2019) [ 19 ] and were estimated using ACCHSs’ Communicare electronic medical records of PHC utilisation during the period 2015–2018. Service population size of ACCHSs was classified into three categories (≤ 2000, 2001–10000, > 10,000) and for clinics, into six categories (≤ 500, 501–1000, 1001–1500, 1501–2000, 2001–10000, > 10,000). ‘Regional centres’ were defined as those with acute care hospital facilities. Distances to the nearest regional centre were measured in kilometres using straight-line distances in Google Maps. At the organisation level, ACCHSs were described as ‘regional’ or ‘remote’ based on the proximity to the nearest regional centre. An ACCHS was categorised as ‘remote’ if > 50% of the communities it serviced were further than 50 km from the nearest regional centre. For the clinic-level summary, distance to the nearest regional centre was distributed into four categories (≤ 50, 51–200, 201–500, > 500).

Summary statistics were reported as means with standard deviations (SD) or medians with interquartile ranges (IQR). Mann–Whitney U test was used to compare key workforce metrics between the two staff groups analysed—client-facing vs. non-client-facing and Aboriginal vs. non-Aboriginal. Spearman rank correlation coefficients (ρ) were used with a statistical significance test to explore associations between clinic-level key workforce metrics and community characteristics. Absolute value of ρ (|ρ|) indicated the strength of the correlation. A p value of < 0.05 was considered statistically significant and has not been adjusted for multiple inferences as this is primarily a descriptive study.

The median service population of the 22 clinics over the 3-year study period was 1110 (IQR 620, 1758). The median straight-line distances from the clinics to the nearest regional centre was 185 km (IQR 9, 467).

Clinic-level turnover and retention

At any point in time in a given year, the median number of client and non-client-facing staff working in a remote clinic (i.e., clinics > 50 km from the nearest regional centre, as previously defined) was 5 (IQR 4, 8) and 2 (IQR 1, 6), respectively.

Overall, 1690 staff ceased providing services in the 22 clinics during the study period. The mean annual turnover rate was estimated as 151% (± 124.2) for all staff (Table  1 ); for client-facing staff, this was 164%, compared to 120% for non-client-facing staff (Table  1 ). The 12-month stability rates averaged 49% (± 17.9) overall, with similar means for client-facing (50%) and non-client-facing (49%) staff (Table  1 ). There was no statistically significant difference in annual turnover and 12-month stability rates between the two staff groups.

Clinics in the category representing those that are furthest from the nearest regional centre had the highest mean annual turnover rates among all staff at 355% (± 85) and the lowest 12-month stability rates at 26% (± 11.1), compared to clinics in other categories (Table  2 ). The trends in the metrics were similar for categories based on number of service clients, where annual turnover rates and 12-month stability rates for clinics representing the smallest category averaged at 376% (± 81.4) and 27% (± 9.9), respectively (Table  1 ).

The median average headcount of Aboriginal and non-Aboriginal staff working in a clinic was 6 (IQR 4, 18) and 5 (IQR 4, 10), respectively. Noting that five of the 22 clinics did not have any Aboriginal staff employed during the study period, the mean annual turnover rates among Aboriginal staff in the remaining 17 clinics was 81% which was significantly lower than that of non-Aboriginal staff at 162% ( p  = 0.025) (Fig.  1 A). The 12-month stability rates were also significantly different between the two groups ( p  = 0.019), with higher stability among Aboriginal staff at 61%, compared to 49% among non-Aboriginal staff (Fig.  1 B and Supplementary file Table S1).

Summary of clinic-level ( A ) turnover and ( B ) stability metrics, by staff employment category and Aboriginal status

Organisation-level turnover and retention

There were 1630 exits for all staff from the 11 ACCHSs over the study period. This was an overall annual turnover rate of 81% (± 42.9), while the mean 12-month stability rate for 11 ACCHSs was estimated as 58% (± 13.3) (Table  3 ). Remote ACCHSs had higher mean annual turnover rates (97%) and lower 12-month stability rates (53%), than regional ACCHSs (Table  4 ).

On average, the median number of Aboriginal and non-Aboriginal staff working in a remote ACCHS per year was 6 (IQR 3, 24) and 5 (IQR 4, 78), respectively. The mean annual turnover rates were 88% and 75% for Aboriginal and non-Aboriginal staff, respectively, with comparable 12-month stability rates (Supplementary file Table S2). Neither turnover nor stability rates significantly differed between Aboriginal and non-Aboriginal staff groups at the organisation level.

Correlation of turnover and retention metrics with clinic characteristics

Correlations between the clinic-level key workforce metrics for staff by employment category and distance to the nearest regional centre, service population size, and average annual headcount were moderate to high (|ρ|= 0.5 − 0.9, p values < 0.01). For Aboriginal staff, these correlations were lower in magnitude than for non-Aboriginal staff (|ρ|= 0.2 − 0.7) and statistically significant only for distance to the nearest regional centre. (Table  5 ). Annual turnover rates were positively correlated with remoteness, while 12-month stability rates were negatively correlated. The direction of association was opposite for service population size and average annual headcount, that is, annual turnover rates were negatively correlated with these indicators and 12-month stability rates were positively correlated. Correlations between the workforce metrics and total unique employees working at each clinic were mostly low or negligible and non-significant.

Importantly, this study describes the pre-pandemic patterns of turnover and retention of PHC staff in ACCHSs in remote and regional areas of NT and WA. The findings indicate very high clinic-level turnover (mean 151%) and low retention (mean 49%) rates, with similar retention rates among client-facing and non-client-facing staff. While not directly comparable, these rates are consistent with those reported in a previous study of remote health care staff in NT DoH clinics [ 14 ]. The rates were considerably higher than what were considered acceptable turnover rates among health workforce, elsewhere, where rates over 20% are considered high [ 20 ]. PHC workforce turnover has been shown to be negatively associated with PHC service utilisation, especially in the first 12 months after a usual provider of care leaves, and have positive associations with utilisation of urgent and emergency care [ 21 ]. An unstable clinical workforce also compromises the continuity and quality of care generally [ 22 , 23 ].

There is a substantial difference in the mean turnover and stability rates at clinic and organisation levels, with higher turnover and lower retention seen at the clinic level. This is not unexpected, because ACCHS staff may be moved between clinics within the organisation. Workforce and service needs of remote clinics fluctuate for multiple reasons, and ACCHSs may redeploy staff from a different clinic to temporarily fill a vacancy or to supplement the existing PHC service capacity in another. Staffing patterns comprising considerable workforce movements between clinics may impede the development of rapport between residents and health providers. The impact on interpersonal continuity of care, however, may be mitigated if returning short-term staff and longer-term staff who are relocated within the organisation work in the same clinic, and thus have local knowledge and established relationships with residents [ 7 ].

One key finding, which differs from previously reported findings, relates to the correlation between clinic-level workforce metrics and remoteness (as measured by distance to nearest acute care hospital facility) and service population size. In this study annual turnover rates significantly increased with increasing remoteness, ranging from 64% (regional centres) to 355% (the most remote clinics), whereas a recent study amongst NT DoH remote clinics found no significant correlation [ 14 ]. This difference in association could be because of greater variation in remoteness of ACCHS clinics included in this study—regionally based to very remote clinics—whereas the NT DoH clinics had less variation in remoteness [ 14 ]. Similar gradients associated with the size of service populations, were found. The turnover rate, for example, ranged from 376% for the smallest communities to 49% for the largest services, a greater than sevenfold difference. Given the high costs of staff turnover, particularly in the remote context [ 16 ]—e.g., associated with frequent staff recruitment, on-boarding and training, and agency and locums salary to temporarily fill the vacancy—and the current shortfall of funding for these remote clinics, estimated at $80 million per year for the NT [ 24 ], the study findings have profound implications for funding remote PHC. It suggests that further research is needed to ensure that an equitable, needs-based funding formula is developed which takes into account not only the burden of disease in these communities but also the higher costs of delivering frontline services, compared to urban or peri-urban populations. Such high turnover rates in the most remote locations also have implications for the re-design of appropriate service delivery models to optimise interpersonal continuity of care, including offering flexible employment arrangements to enhance staff retention.

Another notable finding pertains to clinic-level staff turnover stratified by Aboriginal status. While the analysis did not factor in the proportion of local and non-local Aboriginal staff, relatively lower clinic-level turnover rates among Aboriginal staff (81%), compared to non-Aboriginal counterparts (162%), might reflect favourably on the employment of local Aboriginal community members who are likely to have a greater connection and sense of responsibility and commitment to their community. Employment of local Aboriginal people is critical for ensuring culturally sensitive and appropriate service delivery in remote Aboriginal communities, as they are more familiar with the local cultures and contexts. Aboriginal staff in client-facing roles, in particular, have been shown to have a positive impact on service access, care acceptance and overall experience among Aboriginal patients [ 11 , 25 ]. Nevertheless, annual turnover rates averaging 81% among remote Aboriginal staff are still considerably higher than average turnover rates in other healthcare settings [ 20 ]. However, these high turnover rates highlight the need for future research to better understand and address context-specific factors impacting Aboriginal staff retention, particularly that of local Aboriginal staff. This may include providing increased personal and professional support for Aboriginal health professionals, providing pathways to career advancement, resources to enhance skills and capability, ensuring culturally safe and respectful workplaces, remuneration appropriately reflecting expertise, and equitable provision of non-financial incentives such as subsidised housing [ 26 ].

It is important to note that the turnover and retention rates presented in this study are likely to underestimate the actual turnover rates and overestimate stability rates. This was confirmed in a sensitivity analysis with and without agency-employed RANs for one of the participating ACCHS, which included agency RANs paid via their payroll system (specific findings not reported to maintain confidentiality). In that ACCHS agency RANs provided a substantial proportion of PHC services. NT DoH had previously estimated 42% of its RAN workforce were agency-employed [ 14 ]. These findings reflect the extent of the reliance on locum and agency staff for PHC services in remote clinics. High use of agency RANs is cost-ineffective, mainly owing to the higher costs associated with their recruitment to fill existing vacancies, such as agency fees, travel, housing, and orientation and induction, which further compounds the already high costs of providing PHC in remote health services [ 27 ].

Participating ACCHSs have initiated a range of retention strategies, such as flexible employment conditions, cash retention bonuses and non-financial incentives (such as subsidised housing). Notwithstanding these initiatives, stabilisation of the remote health workforce has remained elusive. Workforce stabilisation could lessen the financial burden on ACCHSs, improve the cultural and clinical competence of staff [ 28 ], and reduce the workload by decreasing the need to continually orient and support new colleagues [ 29 ]. From patients’ perspectives, stabilisation of the workforce could result in improved interpersonal continuity of care and service quality [ 23 ], thereby fewer preventable hospital admissions [ 30 ].

The study findings must be interpreted with some caution because of limitations arising from data availability and quality. First, comprehensive individual-level data on ACCHSs’ use of GP locums and agency-employed RANs were unavailable and could not be integrated with the payroll data. This precludes obtaining a complete picture of the actual extent of staff movements on the ground. Limited data and anecdotal information from participating ACCHSs indicate high use of GP locums and agency-employed RANs in remote communities. As such, the presented turnover and retention metrics may have been underestimated and overestimated, respectively. Second, the analyses did not account for variations among staff based on the type of employment contracts, i.e., casuals, fixed-term, or continuing. This means that casual staff who continue to be on the ACCHSs’ payroll but did not have any work hours to log for more than 12 weeks and departures of staff on temporary contracts during the study period would have been counted as an exit, thus contributing to the turnover measure. Thirdly, the small size of some participating ACCHS clinics, where some clinics had as few as one staff on average at any given point, meant that there was statistical instability in these turnover rates because of the small denominator. Finally, the collection, recording and configuration of payroll data varied among the ACCHSs, which required extensive data manipulation to enable comparability, for example, grouping staff into the seven pre-defined employment categories and resolving inconsistent staff employment information. There might have been some errors during data interpretation and manipulation that potentially impacted the findings. However, the research was conducted closely with relevant ACCHS staff to ensure maximum accuracy of information and minimal errors in the process and to validate the findings at each stage of progress.

This landmark study provides a comprehensive description of the pattern of turnover and retention of healthcare staff in regional and remote ACCHSs in the NT and WA. Turnover is very high in regional centres and extraordinarily high in remote clinics. Overall, clinic-level turnover rates are also higher for non-Aboriginal staff than for Aboriginal staff and increases as distances to regional centres increase. These staffing patterns not only are likely to impose an additional financial and workload burden on remote ACCHSs in terms of increased resources required in relation to frequent staff recruitment and orientation but may also compromise the quality of care and health outcomes for remote Aboriginal community members. These findings have important implications for workforce policy and ensuring the equitable funding of ACCHSs.

Availability of data and materials

The data sets generated and analysed during the current study are not publicly available due to identifiability of ACCHSs staff and the need to protect their privacy.

Abbreviations

• Aboriginal Community Controlled Health Services

Aboriginal health practitioners

Department of Health

General practitioner

Interquartile range

Northern Territory

• Primary health care

Primary Health Network

Remote area nurses and midwives

Standard deviation

Western Australia

World Health Organization. Continuity and coordination of care: a practice brief to support implementation of the WHO framework on integrated people-centred health services. Geneva: WHO; 2018.

Google Scholar  

Australian Institute of Health and Welfare. Spatial variation in Aboriginal and Torres Strait Islander people’s access to primary health care. Canberra: AIHW; 2015.

Wakerman J, Humphreys J, Russell D, Guthridge S, Bourke L, Dunbar T, et al. Remote health workforce turnover and retention: what are the policy and practice priorities? Hum Resour Health. 2019;17(1):99.

Article   PubMed   PubMed Central   Google Scholar  

van Walraven C, Oake N, Jennings A, Forster AJ. The association between continuity of care and outcomes: a systematic and critical review. J Eval Clin Pract. 2010;16(5):947–56.

Article   PubMed   Google Scholar  

Busbridge MJ, Smith A. Fly in/fly out health workers: a barrier to quality in health care. Rural Remote Health. 2015;15(2):3339.

CAS   PubMed   Google Scholar  

Tarlier DS, Browne AJ, Johnson J. The influence of geographical and social distance on nursing practice and continuity of care in a remote First Nations community. Can J Nurs Res. 2007;39(3):126–48.

PubMed   Google Scholar  

Nolan-Isles D, Macniven R, Hunter K, Gwynn J, Lincoln M, Moir R, et al. Enablers and barriers to accessing healthcare services for Aboriginal people in New South Wales, Australia. Int J Environ Res Public Health. 2021. https://doi.org/10.3390/ijerph18063014 .

Zhao Y, Thomas SL, Guthridge SL, Wakerman J. Better health outcomes at lower costs: the benefits of primary care utilisation for chronic disease management in remote Indigenous communities in Australia’s Northern Territory. BMC Health Serv Res. 2014;14(1):463.

Pearson O, Schwartzkopff K, Dawson A, Hagger C, Karagi A, Davy C, et al. Aboriginal community controlled health organisations address health equity through action on the social determinants of health of Aboriginal and Torres Strait Islander peoples in Australia. BMC Publ Health. 2020;20(1):1859.

Article   CAS   Google Scholar  

Tilton E, Thomas D. Core functions of primary health care: a framework for the Northern Territory. Darwin: Northern Territory Aboriginal Health Forum; 2011.

Gomersall JS, Gibson O, Dwyer J, O’Donnell K, Stephenson M, Carter D, et al. What Indigenous Australian clients value about primary health care: a systematic review of qualitative evidence. Aust N Z J Public Health. 2017;41(4):417–23.

Article   Google Scholar  

Baba JT, Brolan CE, Hill PS. Aboriginal medical services cure more than illness: a qualitative study of how Indigenous services address the health impacts of discrimination in Brisbane communities. Int J Equity Health. 2014;13(1):56.

Australian Institute of Health and Welfare. Aboriginal and Torres Strait Islander health performance framework: 2020 summary report. Canberra: AIHW; 2020.

Russell DJ, Zhao Y, Guthridge S, Ramjan M, Jones MP, Humphreys JS, et al. Patterns of resident health workforce turnover and retention in remote communities of the Northern Territory of Australia, 2013–2015. Hum Resour Health. 2017;15(1):52.

Jongen C, McCalman J, Campbell S, Fagan R. Working well: strategies to strengthen the workforce of the Indigenous primary healthcare sector. BMC Health Serv Res. 2019;19(1):910.

Australian Health Ministers’ Advisory Council. National Aboriginal and Torres Strait Islander health workforce strategic framework 2016–2023. Canberra: Commonwealth of Australia; 2017.

Russell DJ, Humphreys JS, Wakerman J. How best to measure health workforce turnover and retention: five key metrics. Aust Health Rev. 2012;36(3):290–5.

National Aboriginal Community Controlled Health Organisation. NAACHO members n.d., https://www.naccho.org.au/naccho-members/ ]

Australian Institute of Health Welfare. Aboriginal and Torres Strait Islander specific primary health care: results from the nKPI and OSR collections. Canberra: AIHW; 2023.

Kosel KC, Olivo T. The business case for workforce stability. Irving: VHA Inc; 2002.

Sabety AH, Jena AB, Barnett ML. Changes in health care use and outcomes after turnover in primary care. JAMA Intern Med. 2021;181(2):186–94.

Reddy A, Pollack CE, Asch DA, Canamucio A, Werner RM. The effect of primary care provider turnover on patient experience of care and ambulatory quality of care. JAMA Intern Med. 2015;175(7):1157–62.

Wong ST, Regan S. Patient perspectives on primary health care in rural communities: effects of geography on access, continuity and efficiency. Rural Remote Health. 2009;9(1):1142.

Zhao Y, Wakerman J, Zhang X, Wright J, VanBruggen M, Nasir R, et al. Remoteness, models of primary care and inequity: medicare under-expenditure in the Northern Territory. Aust Health Rev. 2022;46(3):302–8.

Gwynne K, Lincoln M. Developing the rural health workforce to improve Australian Aboriginal and Torres Strait Islander health outcomes: a systematic review. Aust Health Rev. 2017;41(2):234–8.

Meyer L, Joseph TL, Anderson-Smith B, et al. Career pathways for the Aboriginal and Torres Strait Islander health workforce: literature review report. Melbourne: The Lowitja Institute; 2020.

Zhao Y, Russell DJ, Guthridge S, Ramjan M, Jones MP, Humphreys JS, et al. Costs and effects of higher turnover of nurses and Aboriginal health practitioners and higher use of short-term nurses in remote Australian primary care services: an observational cohort study. BMJ Open. 2019;9(2): e023906.

McCullough K, Whitehead L, Bayes S, Williams A, Cope V. The delivery of primary health care in remote communities: a grounded theory study of the perspective of nurses. Int J Nurs Stud. 2020;102: 103474.

Argent J, Lenthall S, Hines S, Rissel C. Perceptions of Australian remote area nurses about why they stay or leave: a qualitative study. J Nurs Manag. 2022;30(5):1243–51.

Barker I, Steventon A, Deeny S. Continuity of care in general practice is associated with fewer ambulatory care sensitive hospital admissions: a cross-sectional study of routinely collected, person-level data. Clin Med. 2017;17(Suppl 3): s16.

Download references

Acknowledgements

We wish to acknowledge and thank the Boards of Directors and all staff working in the 11 participating ACCHSs. Following a previous research project on the NT Government sector remote PHC clinics, the executives from the participating ACCHSs have approached our research team to undertake the current research, and have since engaged with our team in every step of research design and development. We acknowledge their contribution to the research.

The study received funding from the Australian Research Council’s Discovery funding scheme (project number DP190100328) and the Medical Research Future Fund through the National Health and Medical Research Council and Central Australian Academic Health Science Network. The information and opinions contained in it do not necessarily reflect the views or policy of the Commonwealth of Australia (or the Department of Health).

Author information

Authors and affiliations.

Menzies School of Health Research, Charles Darwin University, PO Box 795, Alice Springs, Northern Territory, 0871, Australia

Prabhakar Veginadu, Deborah J. Russell, Supriya Mathew & John Wakerman

Northern Territory Department of Health, Darwin, Northern Territory, Australia

Yuejen Zhao

Menzies School of Health Research, Charles Darwin University, Darwin, Northern Territory, Australia

Steven Guthridge

Top End Population and Primary Health Care, Northern Territory Government, Darwin, Northern Territory, Australia

Mark Ramjan

Psychology Department, Macquarie University, North Ryde, New South Wales, Australia

Michael P. Jones

Institute for Culture and Society, Western Sydney University, Parramatta, New South Wales, Australia

Michelle S. Fitts

Poche Centre for Indigenous Health and Well-Being, Flinders University, Alice Springs, Northern Territory, Australia

Lorna Murakami-Gold

Flinders Rural and Remote Health Northern Territory, College of Medicine and Public Health, Flinders University, Darwin, Northern Territory, Australia

Narelle Campbell

Ngaanyatjarra Health Service, Alice Springs, Northern Territory, Australia

Annie Tangey & Rosalie Schultz

Central Australian Aboriginal Congress, Alice Springs, Northern Territory, Australia

Miwatj Health Aboriginal Corporation, Nhulunbuy, Northern Territory, Australia

Bronwyn Rossingh

Monash University School of Rural Health, Bendigo, Victoria, Australia

John Humphreys

You can also search for this author in PubMed   Google Scholar

Contributions

JW, JH, SG, YZ, and MPJ conceived the research. MPJ, SG, YZ, DJR and PV designed and coordinated the study. PV, DJR, SG, MPJ, YZ, MR and JW contributed to the analysis and interpretation of the data. PV drafted the original manuscript and DJR contributed to the drafting. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Prabhakar Veginadu .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was received from the Human Research Ethics Committee of the Northern Territory Department of Health and Menzies School of Health Research (reference number 2019–3428), the Central Australian Human Research Ethics Committee (CA-19–3493) and the Western Australian Aboriginal Health Ethics Committee (WAAHEC-938).

Consent for publication

Not applicable.

Competing interests

None to report.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Veginadu, P., Russell, D.J., Zhao, Y. et al. Patterns of health workforce turnover and retention in Aboriginal Community Controlled Health Services in remote communities of the Northern Territory and Western Australia, 2017–2019. Hum Resour Health 22 , 58 (2024). https://doi.org/10.1186/s12960-024-00942-9

Download citation

Received : 11 November 2023

Accepted : 08 August 2024

Published : 22 August 2024

DOI : https://doi.org/10.1186/s12960-024-00942-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health workforce
• Indigenous health services
• Remote health
• Aboriginal and Torres Strait Islander peoples

Human Resources for Health

ISSN: 1478-4491

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 23 August 2024

The role of emotions in academic performance of undergraduate medical students: a narrative review

• Nora Alshareef 1 , 2 ,
• Ian Fletcher 2 &
• Sabir Giga 2  

BMC Medical Education volume  24 , Article number:  907 ( 2024 ) Cite this article

180 Accesses

Metrics details

This paper is devoted to a narrative review of the literature on emotions and academic performance in medicine. The review aims to examine the role emotions play in the academic performance of undergraduate medical students.

Eight electronic databases were used to search the literature from 2013 to 2023, including Academic Search Ultimate, British Education Index, CINAHL, Education Abstract, ERIC, Medline, APA Psych Articles and APA Psych Info. Using specific keywords and terms in the databases, 3,285,208 articles were found. After applying the predefined exclusion and inclusion criteria to include only medical students and academic performance as an outcome, 45 articles remained, and two reviewers assessed the quality of the retrieved literature; 17 articles were selected for the narrative synthesis.

The findings indicate that depression and anxiety are the most frequently reported variables in the reviewed literature, and they have negative and positive impacts on the academic performance of medical students. The included literature also reported that a high number of medical students experienced test anxiety during their study, which affected their academic performance. Positive emotions lead to positive academic outcomes and vice versa. However, Feelings of shame did not have any effect on the academic performance of medical students.

The review suggests a significant relationship between emotions and academic performance among undergraduate medical students. While the evidence may not establish causation, it underscores the importance of considering emotional factors in understanding student performance. However, reliance on cross-sectional studies and self-reported data may introduce recall bias. Future research should concentrate on developing anxiety reduction strategies and enhancing mental well-being to improve academic performance.

Peer Review reports

Introduction

Studying medicine is a multi-dimensional process involving acquiring medical knowledge, clinical skills, and professional attitudes. Previous research has found that emotions play a significant role in this process [ 1 , 2 ]. Different types of emotions are important in an academic context, influencing performance on assessments and evaluations, reception of feedback, exam scores, and overall satisfaction with the learning experience [ 3 ]. In particular, medical students experience a wide range of emotions due to many emotionally challenging situations, such as experiencing a heavy academic workload, being in the highly competitive field of medicine, retaining a large amount of information, keeping track of a busy schedule, taking difficult exams, and dealing with a fear of failure [ 4 , 5 , 6 ].Especially during their clinical years, medical students may experience anxiety when interacting with patients who are suffering, ill, or dying, and they must work with other healthcare professionals. Therefore, it is necessary to understand the impact of emotions on medical students to improve their academic outcomes [ 7 ].

To distinguish the emotions frequently experienced by medical students, it is essential to define them. Depression is defined by enduring emotions of sadness, despair, and a diminished capacity for enjoyment or engagement in almost all activities [ 4 ]. Negative emotions encompass unpleasant feelings such as anger, fear, sadness, and anxiety, and they frequently cause distress [ 8 ]. Anxiety is a general term that refers to a state of heightened nervousness or worry, which can be triggered by various factors. Test anxiety, on the other hand, is a specific type of anxiety that arises in the context of taking exams or assessments. Test anxiety is characterised by physiological arousal, negative self-perception, and a fear of failure, which can significantly impair a student’s ability to perform well academically [ 9 , 10 ]. Shame is a self-conscious emotion that arises from the perception of having failed to meet personal or societal standards. It can lead to feelings of worthlessness and inadequacy, severely impacting a student’s motivation and academic performance [ 11 , 12 ]. In contrast, positive emotions indicate a state of enjoyable involvement with the surroundings, encompassing feelings of happiness, appreciation, satisfaction, and love [ 8 ].

Academic performance generally refers to the outcomes of a student’s learning activities, often measured through grades, scores, and other formal assessments. Academic achievement encompasses a broader range of accomplishments, including mastery of skills, attainment of knowledge, and the application of learning in practical contexts. While academic performance is often quantifiable, academic achievement includes qualitative aspects of a student’s educational journey [ 13 ].

According to the literature, 11–40% of medical students suffer from stress, depression, and anxiety due to the intensity of medical school, and these negative emotions impact their academic achievement [ 14 , 15 ]. Severe anxiety may impair memory function, decrease concentration, lead to a state of hypervigilance, and interfere with judgment and cognitive function, further affecting academic performance [ 16 ]. However, some studies have suggested that experiencing some level of anxiety has a positive effect and serves as motivation that can improve academic performance [ 16 , 17 ].

Despite the importance of medical students’ emotions and their relation to academic performance, few studies have been conducted in this area. Most of these studies have focused on the prevalence of specific emotions without correlating with medical students’ academic performance. Few systematic reviews have addressed the emotional challenges medical students face. However, there is a lack of comprehensive reviews that discuss the role of emotions and academic outcomes. Therefore, this review aims to fill this gap by exploring the relationship between emotions and the academic performance of medical students.

Aim of the study

This review aims to examine the role emotions play in the academic performance of undergraduate medical students.

A systematic literature search examined the role of emotions in medical students’ academic performance. The search adhered to the concepts of a systematic review, following the criteria of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 18 ]. Then, narrative synthesise was done to analyse the retrieved literature and synthesise the results. A systematic literature search and narrative review provide complete coverage and flexibility to explore and understand findings. Systematic search assures rigour and reduces bias, while narrative synthesis allows for flexible integration and interpretation. This balance improves review quality and utility.

Eligibility criteria

Inclusion criteria.

The study’s scope was confined to January 2013 to December 2023, focusing exclusively on undergraduate medical students. The research encompassed articles originating within medical schools worldwide, accepting content from all countries. The criteria included only full-text articles in English published in peer-reviewed journals. Primary research was considered, embracing quantitative and mixed-method research. The selected studies had to explicitly reference academic performance, test results, or GPA as key outcomes to address the research question.

Exclusion criteria

The study excluded individuals beyond the undergraduate medical student demographic, such as students in other health fields and junior doctors. There was no imposed age limit for the student participants. The research specifically focused on articles within medical schools, excluding those from alternative settings. It solely considered full-text articles in English-language peer-reviewed journals. Letters or commentary articles were excluded, and the study did not limit itself to a particular type of research. Qualitative studies were excluded from the review because they did not have the quantitative measures required to answer the review’s aim. This review excluded articles on factors impacting academic performance, those analysing nursing students, and gender differences. The reasons and numbers for excluding articles are shown in Table  1 .

Information sources

Eight electronic databases were used to search the literature. These were the following: Academic Search Ultimate, British Education Index, CINAHL, Education Abstract, ERIC, Medline, APA Psych Articles and APA Psych Info. The databases were chosen from several fields based on relevant topics, including education, academic evaluation and assessment, medical education, psychology, mental health, and medical research. Initially, with the help of a subject librarian, the researcher used all the above databases; the databases were searched with specific keywords and terms, and the terms were divided into the following concepts emotions, academic performance and medical students. Google Scholar, EBSCOhost, and the reference list of the retrieved articles were also used to identify other relevant articles.

Search strategy

This review started with a search of the databases. Eight electronic databases were used to search the literature from 2013 to 2023. Specific keywords and terms were used to search the databases, resulting in 3,285,208 articles. After removing duplicates, letters and commentary, this number was reduced to 1,637 articles. Exclusion and inclusion criteria were then applied, resulting in 45 articles. After two assessors assessed the literature, 17 articles were selected for the review. The search terms are as follows:

Keywords: Emotion, anxiety, stress, empathy, test anxiety, exam anxiety, test stress, exam stress, depression, emotional regulation, test scores, academic performance, grades, GPA, academic achievement, academic success, test result, assessment, undergraduate medical students and undergraduate medical education.

Emotions: TI (Emotion* OR Anxiety OR Stress OR empathy) OR emotion* OR (test anxiety or exam anxiety or test stress or exam stress) OR (depression) OR AB ((Emotion* OR Anxiety OR Stress OR empathy) OR emotion* OR (test anxiety or exam anxiety or test stress or exam stress)) (MH “Emotions”) OR (MH “Emotional Regulation”) DE “EMOTIONS”.

Academic performance: TI (test scores or academic performance or grades or GPA) OR (academic achievement or academic performance or academic success) OR (test result* OR assessment*) OR AB (test scores or academic performance or grades or GPA) OR (academic achievement or academic performance or academic success) OR test result* OR assessment*.

Medical Students: TI (undergraduate medical students OR undergraduate medical education) OR AB (undergraduate medical students OR undergraduate medical education), TI “medical students” OR AB “medical students” DE “Medical Students”.

Selection process

This literature review attempts to gather only peer-reviewed journal articles published in English on undergraduate medical students’ negative and positive emotions and academic performance from January 2013 to December 2023. Their emotions, including depression, anxiety, physiological distress, shame, happiness, joy, and all emotions related to academic performance, were examined in quantitative research and mixed methods.

Moreover, to focus the search, the author specified and defined each keyword using advanced search tools, such as subject headings in the case of the Medline database. The author used ‘MeSH 2023’ as the subject heading, then entered the term ‘Emotion’ and chose all the relevant meanings. This method was applied to most of the keywords.

Studies were included based on predefined criteria related to study design, participants, exposure, outcomes, and study types. Two independent reviewers screened each record, and the report was retrieved. In the screening process, reviewers independently assessed each article against the inclusion criteria, and discrepancies were resolved through consensus during regular team meetings. In cases of persistent disagreement, a third reviewer was consulted. Endnote library program was used for the initial screening phase. This tool was used to identify duplicates, facilitated the independent screening of titles and abstracts and helped to retrieve the full-text articles. The reasons for excluding the articles are presented in Table  1 .

Data collection process

Two independent reviewers extracted data from the eligible studies, with any discrepancies resolved through discussion and consensus. If the two primary reviewers could not agree, a third reviewer served as an arbitrator. For each included study, the following information was extracted and recorded in a standardised database: first author name, publication year, study design, sample characteristics, details of the emotions exposed, outcome measures, and results.

Academic performance as an outcome for medical students was defined to include the following: Exam scores (e.g., midterm, final exams), Clinical assessments (e.g., practical exams, clinical rotations), Overall grade point average (GPA) or any other relevant indicators of academic achievement.

Data were sought for all outcomes, including all measures, time points, and analyses within each outcome domain. In cases where studies reported multiple measures or time points, all relevant data were extracted to provide a comprehensive overview of academic performance. If a study reported outcomes beyond the predefined domains, inclusion criteria were established to determine whether these additional outcomes would be included in the review. This involved assessing relevance to the primary research question and alignment with the predefined outcome domains.

Quality assessment

The quality and risk of bias in included studies were assessed using the National Institute of Health’s (NIH) critical appraisal tool. The tool evaluates studies based on the following domains: selection bias, performance bias, detection bias, attrition bias, reporting bias, and other biases. Two independent reviewers assessed the risk of bias in each included study. Reviewers worked collaboratively to reach a consensus on assessments. Discrepancies were resolved through discussion and consensus. In cases of persistent disagreement, a third reviewer was consulted.

To determine the validity of eligible articles, all the included articles were critically appraised, and all reviewers assessed bias. The validity and reliability of the results were assessed by using objective measurement. Each article was scored out of 14, with 14 indicating high-quality research and 1 indicating low-quality research. High-quality research, according to the NIH (2013), includes a clear and focused research question, defines the study population, features a high participation rate, mentions inclusion and exclusion criteria, uses clear and specific measurements, reports results in detail, lists the confounding factors and lists the implications for the local community. Therefore, an article was scored 14 if it met all criteria of the critical appraisal tool. Based on scoring, each study was classified into one of three quality categories: good, fair or poor. The poorly rated articles mean their findings were unreliable, and they will not be considered, including two articles [ 16 , 19 ]. Seventeen articles were chosen after critical appraisal using the NIH appraisal tool, as shown in Table  2 .

Effect measures

For each outcome examined in the included studies, various effect measures were utilised to quantify the relationship between emotions and academic performance among undergraduate medical students. The effect measures commonly reported across the studies included prevalence rat, correlation coefficients, and mean differences. The reviewer calculated the effect size for the studies that did not report the effect. The choice of effect measure depended on the nature of the outcome variable and the statistical analysis conducted in each study. These measures were used to assess the strength and direction of the association between emotional factors and academic performance.

The synthesis method

The findings of individual studies were summarised to highlight crucial characteristics. Due to the predicted heterogeneity, the synthesis involved pooling effect estimates and using a narrative method. A narrative synthesis approach was employed in the synthesis of this review to assess and interpret the findings from the included studies qualitatively. The narrative synthesis involved a qualitative examination of the content of each study, focusing on identifying common themes. This synthesis was employed to categorise and interpret data, allowing for a nuanced understanding of the synthesis. Themes related to emotions were identified and extracted for synthesis. Control-value theory [ 20 ] was used as an overarching theory, providing a qualitative synthesis of the evidence and contributing to a deeper understanding of the research question. If the retrieved articles include populations other than medical, such as dental students or non-medical students, the synthesis will distinguish between them and summarise the findings of the medical students only, highlighting any differences or similarities.

The Control-Value Theory, formulated by Pekrun (2006), is a conceptual framework that illustrates the relationship between emotions and academic achievement through two fundamental assessments: control and value. Control pertains to the perceived ability of a learner to exert influence over their learning activities and the results they achieve. Value relates to a student’s significance to these actions and results. The theory suggests that students are prone to experiencing good feelings, such as satisfaction and pride when they possess a strong sense of control and importance towards their academic assignments. On the other hand, individuals are prone to encountering adverse emotions (such as fear and embarrassment) when they perceive a lack of control or worth in these particular occupations. These emotions subsequently impact students’ motivation, learning strategies, and, eventually, their academic achievement. The relevance of control-value theory in reviewing medical student emotions and their influence on academic performance is evident for various reasons. This theory offers a complete framework that facilitates comprehending the intricate connection between emotions and academic achievement. It considers positive and negative emotions, providing a comprehensive viewpoint on how emotions might influence learning and performance. The relevance of control and value notions is particularly significant for medical students due to their frequent exposure to high-stakes tests and difficult courses. Gaining insight into the students’ perception of their power over academic assignments and the importance they attach to their medical education might aid in identifying emotional stimuli and devising remedies. Multiple research has confirmed the theory’s assertions, showing the critical influence of control and value evaluations on students’ emotional experiences and academic achievements [ 21 , 22 ].

Data extraction

For this step, a data extraction sheet was developed using the data extraction template provided by the Cochrane Handbook. To ensure the review is evidence-based and bias-free, the Cochrane Handbook strongly suggests that more than one reviewer review the data. Therefore, the main researcher extracted the data from the included studies, and another reviewer checked the included, excluded and extracted data. Any disagreements were resolved via discussion by a third reviewer. The data extraction Table  2 identified all study features, including the author’s name, the year of publication, the method used the aim of the study, the number and description of participants, data collection tools, and study findings.

Finalisation of references and study characteristics

Prisma sheet and the summary of final studies that have been used for the review.

When the keywords and search terms related to emotions, as mentioned above, in the eight databases listed, 3,285,208 articles were retrieved. After using advanced search and subject headings, the number of articles increased to 3,352,371. Similarly, searching for the second keyword, ‘academic performance,’ using all the advanced search tools yielded 8,119,908 articles. Searching for the third keyword, ‘medical students’, yielded 145,757 articles. All terms were searched in article titles and abstracts. After that, the author combined all search terms by using ‘AND’ and applied the time limit from 2013 to 2023; the search narrowed to 2,570 articles. After duplicates, letters and commentary were excluded, the number was reduced to 1,637 articles. After reading the title and abstract to determine relevance to the topic and applying the exclusion and inclusion criteria mentioned above, 45 articles remained; after the quality of the retrieved literature was assessed by two reviewers, 17 articles were selected for the review. The PRISMA flow diagram summarising the same is presented in Fig.  1 . Additionally, One article by Ansari et al. (2018) was selected for the review; it met most inclusion and exclusion criteria except that the outcome measure is cognitive function and not academic performance. Therefore, it was excluded from the review. Figure  1 shows the Prisma flow diagram (2020) of studies identified from the databases.

Prisma flow diagram (2020)

Study characteristics

Table  2 , summarising the characteristics of the included studies, is presented below.

Findings of the study

Country of the study.

Many of the studies were conducted in developing countries, with the majority being conducted in Europe ( n  = 4), followed by Pakistan ( n  = 2), then Saudi Arabia ( n  = 2), and the United States ( n  = 2). The rest of the studies were conducted in South America ( n  = 1), Morocco ( n  = 1), Brazil ( n  = 1), Australia ( n  = 1), Iran ( n  = 1), South Korea ( n  = 1) and Bosnia and Herzegovina ( n  = 1). No included studies were conducted in the United Kingdom.

Study design

Regarding study design, most of the included articles used a quantitative methodology, including 12 cross-sectional studies. There were two randomised controlled trials, one descriptive correlation study, one cohort study, and only one mixed-method study.

Population and study setting

Regarding population and setting, most of the studies focused on all medical students studying in a medical school setting, from first-year medical students to those in their final year. One study compared medical students with non-medical students; another combined medical students with dental students.

The study aims varied across the included studies. Seven studies examined the prevalence of depression and anxiety among medical students and their relation to academic performance. Four studies examined the relationship between test anxiety and academic performance in medical education. Four studies examined the relationship between medical students’ emotions and academic achievements. One study explored the influence of shame on medical students’ learning.

Study quality

The studies were assessed for quality using tools created by the NIH (2013) and then divided into good, fair, and poor based on these results. Nine studies had a high-quality methodology, seven achieved fair ratings, and only three achieved poor ratings. The studies that were assigned the poor rating were mainly cross-sectional studies, and the areas of weakness were due to the study design, low response rate, inadequate reporting of the methodology and statistics, invalid tools, and unclear research goals.

Outcome measures

Most of the outcome measures were heterogenous and self-administered questionnaires; one study used focus groups and observation ward assessment [ 23 ]. All the studies used the medical students’ academic grades.

Results of the study

The prevalence rate of psychological distress in the retrieved articles.

Depression and anxiety are the most common forms of psychological distress examined concerning academic outcomes among medical students. Studies consistently show concerningly high rates, with prevalence estimates ranging from 7.3 to 66.4% for anxiety and 3.7–69% for depression. These findings indicate psychological distress levels characterised as moderate to high based on common cut-off thresholds have a clear detrimental impact on academic achievement [ 16 , 24 , 25 , 26 ].

The studies collectively examine the impact of psychological factors on academic performance in medical education contexts, using a range of effect sizes to quantify their findings. Aboalshamat et al. (2015) identified a small effect size ( η 2 = 0.018) for depression’s impact on academic performance, suggesting a modest influence. Mihailescu (2016) found a significant negative correlation between levels of depression/anxiety (rho=-0.14, rho=-0.19), academic performance and GPA among medical students. Burr and Beck Dallaghan (2019) reported professional efficacy explaining 31.3% of the variance in academic performance, indicating a significant effect size. However, Del-Ben (2013) et al. did not provide the significant impact of affective changes on academic achievement, suggesting trivial effect sizes for these factors.

In conclusion, anxiety and depression, both indicators of psychological discomfort, are common among medical students. There is a link between distress and poor academic performance results, implying that this relationship merits consideration. Table  3 below shows the specific value of depression and anxiety in retrieved articles.

Test anxiety

In this review, four studies examined the relationship between test anxiety and academic performance in medical education [ 27 , 28 , 29 , 30 ]. The studies found high rates of test anxiety among medical students, ranging from 52% [ 27 ] to as high as 81.1% [ 29 ]. Final-year students tend to experience the highest test anxiety [ 29 ].

Test anxiety has a significant negative correlation with academic performance measures and grade point average (GPA) [ 27 , 28 , 29 ]. Green et al. (2016) found that test anxiety was moderately negatively correlated with USMLE score ( r = − 0.24, p  = 0.00); high test anxiety was associated with low USMLE scores in the control group, further suggesting that anxiety can adversely affect performance. The findings that a test-taking strategy course reduced anxiety without improving test scores highlight the complex nature of anxiety’s impact on performance.

Nazir et al. (2021) found that excellent female medical students reported significantly lower test anxiety than those with low academic grades, with an odds ratio of 1.47, indicating that students with higher test anxiety are more likely to have lower academic grades. Kim’s (2016) research shows moderate correlations between test anxiety and negative achievement emotions such as anxiety and boredom, but interestingly, this anxiety does not significantly affect practical exam scores (OSCE) or GPAs. However, one study found that examination stress enhanced academic performance with a large effect size (W = 0.78), with stress levels at 47.4% among their sample, suggesting that a certain stress level before exams may be beneficial [ 30 ].

Three papers explored shame’s effect on medical students’ academic achievement [ 24 , 31 , 32 ]. Hayat et al. (2018) reported that academic feelings, like shame, significantly depend on the academic year. shame was found to have a slight negative and significant correlation with the academic achievement of learners ( r =-0.15). One study found that some medical students felt shame during simulations-based education examinations because they had made incorrect decisions, which decreased their self-esteem and motivation to learn. However, others who felt shame were motivated to study harder to avoid repeating the same mistakes [ 23 ].

Hautz (2017) study examined how shame affects medical students’ learning using a randomised controlled trial where researchers divided the students into two groups: one group performed a breast examination on mannequins and the other group on actual patients. The results showed that students who performed the clinical examination on actual patients experienced significantly higher levels of shame but performed better in examinations than in the mannequin group. In the final assessments on standardised patients, both groups performed equally well. Therefore, shame decreased with more clinical practice, but shame did not have significant statistics related to learning or performance. Similarly, Burr and Dallaghan (2019) reported that the shame level of medical students was (40%) but had no association with academic performance.

Academic performance, emotions and medical students

Three articles discussed medical students’ emotions and academic performance [ 23 , 24 , 32 ]. Burr and Dallaghan (2019) examine the relationship between academic success and emotions in medical students, such as pride, hope, worry, and shame. It emphasises the links between academic accomplishment and professional efficacy, as well as hope, pride, worry, and shame. Professional efficacy was the most significant factor linked to academic performance, explaining 31.3% of the variance. The importance of emotions on understanding, processing of data, recall of memories, and cognitive burden is emphasised throughout the research. To improve academic achievement, efforts should be made to increase student self-efficacy.

Hayat et al. (2018) found that positive emotions and intrinsic motivation are highly connected with academic achievement, although emotions fluctuate between educational levels but not between genders. The correlations between negative emotions and academic achievement, ranging from − 0.15 to -0.24 for different emotions, suggest small but statistically significant adverse effects.

Behren et al.‘s (2019) mixed-method study found that students felt various emotions during the simulation, focusing on positive emotions and moderate anxiety. However, no significant relationships were found between positive emotions and the student’s performance during the simulation [ 23 ].

This review aims to investigate the role of emotions in the academic performance of undergraduate medical students. Meta-analysis cannot be used because of the heterogeneity of the data collection tools and different research designs [ 33 ]. Therefore, narrative synthesis was adopted in this paper. The studies are grouped into four categories as follows: (1) The effect of depression and anxiety on academic performance, (2) Test anxiety and academic achievement, (3) Shame and academic performance, and (4) Academic performance, emotions and medical students. The control-value theory [ 20 ], will be used to interpret the findings.

The effect of depression and anxiety on academic performance

According to the retrieved research, depression and anxiety can have both a negative and a positive impact on the academic performance of medical students. Severe anxiety may impair memory function, decrease concentration, lead to a state of hypervigilance, interfere with judgment and cognitive function, and further affect academic performance [ 4 ]. Most of the good-quality retrieved articles found that anxiety and depression were associated with low academic performance [ 16 , 24 , 25 , 26 ]. Moreira (2018) and Mihailescu (2016) found that higher depression levels were associated with more failed courses and a lower GPA. However, they did not find any association between anxiety level and academic performance.

By contrast, some studies have suggested that experiencing some level of anxiety reinforces students’ motivation to improve their academic performance [ 16 , 34 ]. Zalihic et al. (2017) conducted a study to investigate anxiety sensitivity about academic success and noticed a positive relationship between anxiety level and high academic scores; they justified this because when medical students feel anxious, they tend to prepare and study more, and they desire to achieve better scores and fulfil social expectations. Similarly, another study found anxiety has a negative impact on academic performance when excessive and a positive effect when manageable, in which case it encourages medical students and motivates them to achieve higher scores [ 35 ].

In the broader literature, the impact of anxiety on academic performance has contradictory research findings. While some studies suggest that having some level of anxiety can boost students’ motivation to improve their academic performance, other research has shown that anxiety has a negative impact on their academic success [ 36 , 37 ]. In the cultural context, education and anxiety attitudes differ widely across cultures. High academic pressure and societal expectations might worsen anxiety in many East Asian societies. Education is highly valued in these societies, frequently leading to significant academic stress. This pressure encompasses attaining high academic marks and outperformance in competitive examinations. The academic demands exerted on students can result in heightened levels of anxiety. The apprehension of not meeting expectations can lead to considerable psychological distress and anxiety, which can appear in their physical and mental health and academic achievement [ 38 , 39 ].

Test anxiety and academic achievement

The majority of the studies reviewed confirm that test anxiety negatively affects academic performance [ 27 , 28 , 29 ]. Several studies have found a significant correlation between test anxiety and academic achievement, indicating that higher levels of test anxiety are associated with lower exam scores and lower academic performance [ 40 , 41 ]. For example, Green et al. (2016) RCT study found that test anxiety has a moderately significant negative correlation with the USMLE score. They found that medical students who took the test-taking strategy course had lower levels of test anxiety than the control group, and their test anxiety scores after the exam had improved from the baseline. Although their test anxiety improved after taking the course, there was no significant difference in the exam scores between students who had and had not taken the course. Therefore, the intervention they used was not effective. According to the control-value theory, this intervention can be improved if they design an emotionally effective learning environment, have a straightforward instructional design, foster self-regulation of negative emotions, and teach students emotion-oriented regulation [ 22 ].

Additionally, according to this theory, students who perceive exams as difficult are more likely to experience test anxiety because test anxiety results from a student’s negative appraisal of the task and outcome values, leading to a reduction in their performance. This aligns with Kim’s (2016) study, which found that students who believed that the OSCE was a problematic exam experienced test anxiety more than other students [ 9 , 22 , 42 ].

In the wider literature, a meta-analysis review by von der Embse (2018) found a medium significant negative correlation ( r =-0.24) between test anxiety and test performance in undergraduate educational settings [ 43 ] . Also, they found a small significant negative correlation ( r =-0.17) between test anxiety and GPA. This indicates that higher levels of test anxiety are associated with lower test performance. Moreover, Song et al. (2021) experimental study examined the effects of test anxiety on working memory capacity and found that test anxiety negatively correlated with academic performance [ 44 ]. Therefore, the evidence from Song’s study suggests a small but significant effect of anxiety on working memory capacity. However, another cross-sectional study revealed that test anxiety in medical students had no significant effect on exam performance [ 45 ]. The complexities of this relationship necessitate additional investigation. Since the retrieved articles are from different countries, it is critical to recognise the possible impact of cultural differences on the impact of test anxiety. Cultural factors such as different educational systems, assessment tools and societal expectations may lead to variances in test anxiety experience and expression across diverse communities [ 46 , 47 ]. Culture has a substantial impact on how test anxiety is expressed and evaluated. Research suggests that the degree and manifestations of test anxiety differ among different cultural settings, emphasising the importance of using culturally validated methods to evaluate test anxiety accurately. A study conducted by Lowe (2019) with Canadian and U.S. college students demonstrated cultural variations in the factors contributing to test anxiety. Canadian students exhibited elevated levels of physiological hyperarousal, but U.S. students had more pronounced cognitive interference. These variations indicate that the cultural environment has an influence on how students perceive and respond to test anxiety, resulting in differing effects on their academic performance in different cultures. Furthermore, scholars highlight the significance of carrying out meticulous instruments to assess test anxiety, which are comparable among diverse cultural cohorts. This technique guarantees that the explanations of test scores are reliable and can be compared across different populations. Hence, it is imperative to comprehend and tackle cultural disparities in order to create efficient interventions and assistance for students who encounter test anxiety in diverse cultural environments. Therefore, there is a need for further studies to examine the level of test anxiety and cultural context.

Shame and academic performance

The review examined three studies that discuss the impact of feelings of shame on academic performance [ 23 , 24 , 48 ]. Generally, shame is considered a negative emotion which involves self-reflection and self-evaluation, and it leads to rumination and self-condemnation [ 49 ]. Intimate examinations conducted by medical students can induce feelings of shame, affecting their ability to communicate with patients and their clinical decisions. Shame can increase the avoidance of intimate physical examinations and also encourage clinical practice [ 23 , 24 , 48 ].

One study found that some medical students felt shame during simulations-based education examinations because they had made incorrect decisions, which decreased their self-esteem and motivation to learn. However, others who felt shame were motivated to study harder to avoid repeating the same mistakes [ 23 ]. Shame decreased with more clinical practice, but shame did not affect their learning or performance [ 48 ]. The literature on how shame affects medical students’ learning is inconclusive [ 31 ].

In the broader literature, shame is considered maladaptive, leading to dysfunctional behaviour, encouraging withdrawal and avoidance of events and inhibiting social interaction. However, few studies have been conducted on shame in the medical field. Therefore, more research is needed to investigate the role of shame in medical students’ academic performance [ 49 ]. In the literature, there are several solutions that can be used to tackle the problem of shame in medical education; it is necessary to establish nurturing learning settings that encourage students to openly discuss their problems and mistakes without the worry of facing severe criticism. This can be accomplished by encouraging medical students to participate in reflective practice, facilitating the processing of their emotions, and enabling them to derive valuable insights from their experiences, all while avoiding excessive self-blame [ 50 ]. Offering robust mentorship and support mechanisms can assist students in effectively managing the difficulties associated with intimate examinations. Teaching staff have the ability to demonstrate proper behaviours and provide valuable feedback and effective mentoring [ 51 ]. Training and workshops that specifically target communication skills and the handling of sensitive situations can effectively equip students to handle intimate tests, hence decreasing the chances of them avoiding such examinations due to feelings of shame [ 52 ].

The literature review focused on three studies that examined the relationship between emotions and the academic achievements of medical students [ 23 , 24 , 32 ].

Behren et al. (2019) mixed-method study on the achievement emotions of medical students during simulations found that placing students in challenging clinical cases that they can handle raises positive emotions. Students perceived these challenges as a positive drive for learning and mild anxiety was considered beneficial. However, the study also found non-significant correlations between emotions and performance during the simulation, indicating a complex relationship between emotions and academic performance. The results revealed that feelings of frustration were perceived to reduce students’ interest and motivation for studying, hampered their decision-making process, and negatively affected their self-esteem, which is consistent with the academic achievement emotions literature where negative emotions are associated with poor intrinsic motivation and reduced the ability to learn [ 3 ].

The study also emphasises that mild anxiety can have positive effects, corroborated by Gregor (2005), which posits that moderate degrees of anxiety can improve performance. The author suggests that an ideal state of arousal (which may be experienced as anxiety) enhances performance. Mild anxiety is commonly seen as a type of psychological stimulation that readies the body for upcoming challenges, frequently referred to as a “fight or flight” response. Within the realm of academic performance, this state of heightened arousal can enhance concentration and optimise cognitive functions such as memory, problem-solving skills, and overall performance. However, once the ideal point is surpassed, any additional increase in arousal can result in a decline in performance [ 53 ]. This is additionally supported by Cassady and Johnson (2002), who discovered that a specific level of anxiety can motivate students to engage in more comprehensive preparation, hence enhancing their performance.

The reviewed research reveals a positive correlation between positive emotions and academic performance and a negative correlation between negative emotions and academic performance. These findings align with the control–value theory [ 8 , 22 ], which suggests that positive emotions facilitate learning through mediating factors, including cognitive learning strategies such as strategic thinking, critical thinking and problem-solving and metacognitive learning strategies such as monitoring, regulating, and planning students’ intrinsic and extrinsic motivation. Additionally, several studies found that extrinsic motivation from the educational environment and the application of cognitive and emotional strategies improve students’ ability to learn and, consequently, their academic performance [ 23 , 24 , 32 ]. By contrast, negative emotions negatively affect academic performance. This is because negative emotions reduce students’ motivation, concentration, and ability to process information [ 23 , 24 , 32 ].

Limitations of the study

This review aims to thoroughly investigate the relationship between emotions and academic performance in undergraduate medical students, but it has inherent limitations. Overall, the methodological quality of the retrieved studies is primarily good and fair. Poor-quality research was excluded from the synthesis. The good-quality papers demonstrated strengths in sampling techniques, data analysis, collection and reporting. However, most of the retrieved articles used cross-section studies, and the drawback of this is a need for a more causal relationship, which is a limitation in the design of cross-sectional studies. Furthermore, given the reliance on self-reported data, there were concerns about potential recall bias. These methodological difficulties were noted in most of the examined research. When contemplating the implications for practice and future study, the impact of these limitations on the validity of the data should be acknowledged.

The limitation of the review process and the inclusion criteria restricted the study to articles published from January 2013 to December 2023, potentially overlooking relevant research conducted beyond this timeframe. Additionally, the exclusive focus on undergraduate medical students may constrain the applicability of findings to other health fields or educational levels.

Moreover, excluding articles in non-English language and those not published in peer-reviewed journals introduces potential language and publication biases. Reliance on electronic databases and specific keywords may inadvertently omit studies using different terms or indexing. While the search strategy is meticulous, it might not cover every relevant study due to indexing and database coverage variations. However, the two assessors’ involvement in study screening, selection, data extraction, and quality assessment improved the robustness of the review and ensured that it included all the relevant research.

In conclusion, these limitations highlight the need for careful interpretation of the study’s findings and stress the importance of future research addressing these constraints to offer a more comprehensive understanding of the nuanced relationship between emotions and academic performance in undergraduate medical education.

Conclusion and future research

The review exposes the widespread prevalence of depression, anxiety and test anxiety within the medical student population. The impact on academic performance is intricate, showcasing evidence of adverse and favourable relationships. Addressing the mental health challenges of medical students necessitates tailored interventions for enhancing mental well-being in medical education. Furthermore, it is crucial to create practical strategies considering the complex elements of overcoming test anxiety. Future research should prioritise the advancement of anxiety reduction strategies to enhance academic performance, focusing on the control-value theory’s emphasis on creating an emotionally supportive learning environment. Additionally, Test anxiety is very common among medical students, but the literature has not conclusively determined its actual effect on academic performance. Therefore, there is a clear need for a study that examines the relationship between test anxiety and academic performance. Moreover, the retrieved literature did not provide effective solutions for managing test anxiety. This gap highlights the need for practical solutions informed by Pekrun’s Control-Value Theory. Ideally, a longitudinal study measuring test anxiety and exam scores over time would be the most appropriate approach. it is also necessary to explore cultural differences to develop more effective solutions and support systems tailored to specific cultural contexts.

The impact of shame on academic performance in medical students was inconclusive. Shame is a negative emotion that has an intricate influence on learning outcomes. The inadequacy of current literature emphasises the imperative for additional research to unravel the nuanced role of shame in the academic journeys of medical students.

Overall, emotions play a crucial role in shaping students’ academic performance, and research has attempted to find solutions to improve medical students’ learning experiences; thus, it is recommended that medical schools revise their curricula and consider using simulation-based learning in their instructional designs to enhance learning and improve students’ emotions. Also, studies have suggested using academic coaching to help students achieve their goals, change their learning styles, and apply self-testing and simple rehearsal of the material. Moreover, the study recommended to improve medical students’ critical thinking and autonomy and changing teaching styles to support students better.

Data availability

all included articles are mentioned in the manuscript, The quality assessment of included articles are located in the supplementary materials file no. 1.

Weurlander M, Lonn A, Seeberger A, Hult H, Thornberg R, Wernerson A. Emotional challenges of medical students generate feelings of uncertainty. Med Educ. 2019;53(10):1037–48.

Article   Google Scholar  

Boekaerts M, Pekrun R. Emotions and emotion regulation in academic settings. Handbook of educational psychology: Routledge; 2015. pp. 90–104.

Google Scholar  

Camacho-Morles J, Slemp GR, Pekrun R, Loderer K, Hou H, Oades LG. Activity achievement emotions and academic performance: a meta-analysis. Educational Psychol Rev. 2021;33(3):1051–95.

Aboalshamat K, Hou X-Y, Strodl E. Psychological well-being status among medical and dental students in Makkah, Saudi Arabia: a cross-sectional study. Med Teach. 2015;37(Suppl 1):S75–81.

Mirghni HO, Ahmed Elnour MA. The perceived stress and approach to learning effects on academic performance among Sudanese medical students. Electron Physician. 2017;9(4):4072–6.

Baessler F, Zafar A, Schweizer S, Ciprianidis A, Sander A, Preussler S, et al. Are we preparing future doctors to deal with emotionally challenging situations? Analysis of a medical curriculum. Patient Educ Couns. 2019;102(7):1304–12.

Rowe AD, Fitness J. Understanding the role of negative emotions in Adult Learning and Achievement: a Social Functional Perspective. Behav Sci (Basel). 2018;8(2).

Pekrun R, Frenzel AC, Goetz T, Perry RP. The control-value theory of achievement emotions: An integrative approach to emotions in education. Emotion in education: Elsevier; 2007. pp. 13–36.

Zeidner M. Test anxiety: The state of the art. 1998.

Cassady JC, Johnson RE. Cognitive test anxiety and academic performance. Contemp Educ Psychol. 2002;27(2):270–95.

Tangney JP, Dearing RL. Shame and guilt: Guilford Press; 2003.

Fang J, Brown GT, Hamilton R. Changes in Chinese students’ academic emotions after examinations: pride in success, shame in failure, and self-loathing in comparison. Br J Educ Psychol. 2023;93(1):245–61.

York TT, Gibson C, Rankin S. Defining and measuring academic success. Practical Assess Res Evaluation. 2019;20(1):5.

Abdulghani HM, Irshad M, Al Zunitan MA, Al Sulihem AA, Al Dehaim MA, Al Esefir WA, et al. Prevalence of stress in junior doctors during their internship training: a cross-sectional study of three Saudi medical colleges’ hospitals. Neuropsychiatr Dis Treat. 2014;10:1879–86.

Moreira de Sousa J, Moreira CA, Telles-Correia D, Anxiety. Depression and academic performance: a Study Amongst Portuguese Medical Students Versus non-medical students. Acta Med Port. 2018;31(9):454–62.

Junaid MA, Auf AI, Shaikh K, Khan N, Abdelrahim SA. Correlation between academic performance and anxiety in Medical students of Majmaah University - KSA. JPMA J Pakistan Med Association. 2020;70(5):865–8.

MihĂIlescu AI, Diaconescu LV, Donisan T, Ciobanu AM, THE INFLUENCE OF EMOTIONAL, DISTRESS ON THE ACADEMIC PERFORMANCE IN UNDERGRADUATE MEDICAL STUDENTS. Romanian J Child Adolesc Psychiatry. 2016;4(1/2):27–40.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372.

Hahn H, Kropp P, Kirschstein T, Rücker G, Müller-Hilke B. Test anxiety in medical school is unrelated to academic performance but correlates with an effort/reward imbalance. PLoS ONE. 2017;12(2):1–13.

Pekrun R. The control-value theory of achievement emotions: assumptions, corollaries, and Implications for Educational Research and Practice. Educational Psychol Rev. 2006;18(4):315–41.

Graesser AC. Emotions are the experiential glue of learning environments in the 21st century. Learn Instruction. 2019.

Pekrun R, Perry RP. Control-value theory of achievement emotions. International handbook of emotions in education: Routledge; 2014. pp. 120 – 41.

Behrens CC, Dolmans DH, Gormley GJ, Driessen EW. Exploring undergraduate students achievement emotions during ward round simulation: a mixed-method study. BMC Med Educ. 2019;19(1):316.

Burr J, Beck-Dallaghan GL. The relationship of emotions and Burnout to Medical Students’ academic performance. Teach Learn Med. 2019;31(5):479–86.

Zalihić A, Mešukić S, Sušac B, Knezović K, Martinac M. Anxiety sensitivity as a predictor of academic success of medical students at the University of Mostar. Psychiatria Danubina. 2017;29(Suppl 4):851–4.

Del-Ben CM, Machado VF, Madisson MM, Resende TL, Valério FP, Troncon LEDA. Relationship between academic performance and affective changes during the first year at medical school. Med Teach. 2013;35(5):404–10.

Nazir MA, Izhar F, Talal A, Sohail ZB, Majeed A, Almas K. A quantitative study of test anxiety and its influencing factors among medical and dental students. J Taibah Univ Med Sci. 2021;16(2):253–9.

Green M, Angoff N, Encandela J. Test anxiety and United States Medical Licensing Examination scores. Clin Teacher. 2016;13(2):142–6.

Ben Loubir D, Serhier Z, Diouny S, Battas O, Agoub M, Bennani Othmani M. Prevalence of stress in Casablanca medical students: a cross-sectional study. Pan Afr Med J. 2014;19:149.

Kausar U, Haider SI, Mughal IA, Noor MSA, Stress levels; stress levels of final year mbbs students and its effect on their academic performance. Prof Med J. 2018;25(6):932–6.

Hautz WE, Schröder T, Dannenberg KA, März M, Hölzer H, Ahlers O, et al. Shame in Medical Education: a randomized study of the Acquisition of intimate Examination skills and its effect on subsequent performance. Teach Learn Med. 2017;29(2):196–206.

Hayat AA, Salehi A, Kojuri J. Medical student’s academic performance: the role of academic emotions and motivation. J Adv Med Educ Professionalism. 2018;6(4):168–75.

Deeks JJ, Riley RD, Higgins JP. Combining Results Using Meta-Analysis. Systematic Reviews in Health Research: Meta‐Analysis in Context. 2022:159 – 84.

Aboalshamat K, Hou X-Y, Strodl E. The impact of a self-development coaching programme on medical and dental students’ psychological health and academic performance: a randomised controlled trial. BMC Med Educ. 2015;15:134.

Jamil H, Alakkari M, Al-Mahini MS, Alsayid M, Al Jandali O. The impact of anxiety and depression on academic performance: a cross-sectional study among medical students in Syria. Avicenna J Med. 2022;12(03):111–9.

Mirawdali S, Morrissey H, Ball P. Academic anxiety and its effects on academic performance. 2018.

Al-Qaisy LM. The relation of depression and anxiety in academic achievement among group of university students. Int J Psychol Couns. 2011;3(5):96–100.

Cheng DR, Poon F, Nguyen TT, Woodman RJ, Parker JD. Stigma and perception of psychological distress and depression in Australian-trained medical students: results from an inter-state medical school survey. Psychiatry Res. 2013;209(3):684–90.

Lee M, Larson R. The Korean ‘examination hell’: long hours of studying, distress, and depression. J Youth Adolesc. 2000;29(2):249–71.

Ali SK. 861 – Social phobia among medical students. Eur Psychiatry. 2013;28:1.

Bonna AS, Sarwar M, Md Nasrullah A, Bin Razzak S, Chowdhury KS, Rahman SR. Exam anxiety among medical students in Dhaka City and its Associated Factors-A cross-sectional study. Asian J Med Health. 2022;20(11):20–30.

Kim K-J. Factors associated with medical student test anxiety in objective structured clinical examinations: a preliminary study. Int J Med Educ. 2016;7:424–7.

Von der Embse N, Jester D, Roy D, Post J. Test anxiety effects, predictors, and correlates: a 30-year meta-analytic review. J Affect Disord. 2018;227:483–93.

Song J, Chang L, Zhou R. Test anxiety impairs filtering ability in visual working memory: evidence from event-related potentials. J Affect Disord. 2021;292:700–7.

Theobald M, Breitwieser J, Brod G. Test anxiety does not predict exam performance when knowledge is controlled for: strong evidence against the interference hypothesis of test anxiety. Psychol Sci. 2022;33(12):2073–83.

Lowe PA. Examination of test anxiety in samples of Australian and US Higher Education Students. High Educ Stud. 2019;9(4):33–43.

Kavanagh BE, Ziino SA, Mesagno C. A comparative investigation of test anxiety, coping strategies and perfectionism between Australian and United States students. North Am J Psychol. 2016;18(3).

Mihăilescu AI, Diaconescu LV, Ciobanu AM, Donisan T, Mihailescu C. The impact of anxiety and depression on academic performance in undergraduate medical students. Eur Psychiatry. 2016;33:S341–2.

Terrizzi JA Jr, Shook NJ. On the origin of shame: does shame emerge from an evolved disease-avoidance architecture? Front Behav Neurosci. 2020;14:19.

Epstein RM. Mindful practice. JAMA. 1999;282(9):833–9.

Hauer KE, Teherani A, Dechet A, Aagaard EM. Medical students’ perceptions of mentoring: a focus-group analysis. Med Teach. 2005;27(8):732–4.

Kalet A, Pugnaire MP, Cole-Kelly K, Janicik R, Ferrara E, Schwartz MD, et al. Teaching communication in clinical clerkships: models from the macy initiative in health communications. Acad Med. 2004;79(6):511–20.

Gregor A. Examination anxiety: live with it, control it or make it work for you? School Psychol Int. 2005;26(5):617–35.

Download references

Acknowledgements

I would like to thank Lancaster university library for helping me to search the literature and to find the appropriate databases and thanks to Lancaster university to prove access to several softwares.

No funding.

Author information

Authors and affiliations.

King Abdulaziz University, Jeddah, Saudi Arabia

Nora Alshareef

Lancaster University, Lancaster, UK

Nora Alshareef, Ian Fletcher & Sabir Giga

You can also search for this author in PubMed   Google Scholar

Contributions

NA made substantial contributions throughout the systematic review process and was actively involved in writing and revising the manuscript. NA’s responsible for the design of the study, through the acquisition, analysis, and interpretation of data, to the drafting and substantive revision of the manuscript. NA has approved the submitted version and is personally accountable for her contributions, ensuring the accuracy and integrity of the work. IF was instrumental in screening the literature, extracting data, and conducting the quality assessment of the included studies. Additionally, IF played a crucial role in revising the results and discussion sections of the manuscript, ensuring that the interpretation of data was both accurate and insightful. IF has approved the submitted version and has agreed to be personally accountable for his contributions, particularly in terms of the accuracy and integrity of the parts of the work he was directly involved in. SG contributed significantly to the selection of papers and data extraction, demonstrating critical expertise in resolving disagreements among authors. SG’s involvement was crucial in revising the entire content of the manuscript, enhancing its coherence and alignment with the study’s objectives. SG has also approved the submitted version and is personally accountable for his contributions, committed to upholding the integrity of the entire work.

Corresponding author

Correspondence to Nora Alshareef .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Consent of publication was obtained from the other authors.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Alshareef, N., Fletcher, I. & Giga, S. The role of emotions in academic performance of undergraduate medical students: a narrative review. BMC Med Educ 24 , 907 (2024). https://doi.org/10.1186/s12909-024-05894-1

Download citation

Received : 08 March 2024

Accepted : 12 August 2024

Published : 23 August 2024

DOI : https://doi.org/10.1186/s12909-024-05894-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical students
• Academic performance

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

Our websites may use cookies to personalize and enhance your experience. By continuing without changing your cookie settings, you agree to this collection. For more information, please see our University Websites Privacy Notice .

UConn Today

August 27, 2024 | Courtney Chandler - UConn Health

NIDCR-funded Study Finds Females Have Lower Salivary Flow Than Males

Findings key for patient risk assessment and treatment management strategies

(Adobe Stock)

Reduced salivary flow, or hyposalivation, can cause an increased risk for tooth decay and other mouth conditions. Measuring salivary flow is important to guide risk assessment and management strategies when treating patients with oral health diseases. Typically, the same standard normal values are used for both females and males in interpreting results of salivary flow testing.

A recent publication led by Dr. Rajesh Lalla, professor and associate dean for research at the UConn School of Dental Medicine, took a deeper look at differences in salivary flow between female and male patients before and after radiation therapy for head and neck cancer. This treatment commonly leads to a significant reduction in salivary flow and complaints of dry mouth.

The research team reported results from their landmark OraRad study, which enrolled 572 patients across six clinical sites. OraRad is a prospective cohort study funded by the National Institute of Dental and Craniofacial Research (NIDCR) to document and study the risk factors for oral complications after radiation therapy.

In the study, the researchers found that females have a significantly lower stimulated salivary flow than males both before and after radiation therapy—which can put females at a higher risk for oral disease in situations where salivary flow is compromised.

The finding of lower salivary flow in females even before the start of radiation therapy led the researchers to examine reports of prior studies. They found that other studies have reported lower salivary flow in females in various populations and age groups, supporting the conclusion that this difference is a generalized finding that is not limited to head and neck cancer patients. The reason for this difference between the sexes has not been extensively studied, but some data suggest that it could be related to the size of the salivary glands, which produce saliva.

“The results of our study, in combination with the prior literature, demonstrate that females and males have significantly different ranges of normal salivary flow,” says Lalla. “This should be taken into account when testing salivary flow in clinical practice and research. These findings also suggest that because females as a group have lower normal salivary flow, they may be at higher risk of reaching critically low levels, in situations where normal saliva production is reduced.”

According to Lalla, the difference in salivary flow between the sexes has not been well-appreciated.

The findings in this study advance both research and clinical practice, as they will enable a more accurate assessment, test interpretation, and clinical management of female patients with dry mouth due to various reasons.

“Having a very dry mouth can greatly affect quality of life and increase risk for several oral diseases,” Lalla says.

The OraRad study is funded by the NIDCR through grant U01DE022939 (Program Officer Dr. Dena Fischer). The Principal Investigators are Drs. Michael Brennan (Atrium Health/Wake Forest University) and Lalla. Additional study sites include Brigham and Women’s Hospital/Harvard University, University of Pennsylvania, New York University, University of North Carolina at Chapel Hill, and University of Minnesota.

Recent Articles

August 28, 2024

Governor Lamont Visits UConn Health to Celebrate Connecticut’s Innovative Collaborations Reducing the State’s Carbon Footprint

Read the article

With Record Success, UConn Research Looks to Make Profound Impact

Neag School Welcomes New Visiting Professor