research papers on primary care

• CLASSIFIEDS
• Advanced search

American Board of Family Medicine

Advanced Search

The Importance of Primary Care Research in Understanding Health Inequities in the United States

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• Info & Metrics

Eliminating health and health care inequities is a longstanding goal of multiple United States health agencies, but overwhelming scientific evidence suggests that health and health care inequities persist in the United States, despite decades of research and initiatives to alleviate them. Because of its comprehensiveness, studying health inequities in the context of primary care allows for the use of multiple paradigms and methodologic approaches to understanding almost any state of health, disease, social challenge, or societal circumstance a patient or group of patients might face. We argue in this special communication that the many features/advantages of primary care research have valuable contributions to make in reducing health inequity, and scientists, journals, and funders should increase the incorporation of primary care approaches and findings into their portfolios to better understand and end health inequity.

• Delivery of Health Care
• Health Care Disparities
• Primary Health Care

Health inequities are differences in health status or the distribution of health resources between different population groups, arising from the social conditions in which people are born, grow, live, work and age. 1 Eliminating health and health care inequities is a longstanding stated goal of multiple United States health agencies, but overwhelming evidence suggests that these inequities persist in the United States, despite decades of research and initiatives to alleviate them. This stasis has led to calls for advancement in health inequities research methods and content by several US federal organizations. In 2012, the National Institutes of Health (NIH) convened a summit calling for a broadening of approaches to address health inequities, 2 and the National Institute of Minority Health and Health Disparities (NIMHD) has led visioning exercises to identify health inequity research priority areas. 3 , 4 While these renewed calls are needed, there are still gaps to better study health inequity. Overall, US health inequities research has been frequently described as a subdiscipline of public health research, 5 and major federal health inequities initiatives have relied on surveys initially developed around the mid-20 th century. 6 While a survey-based, public health approach benefits understanding region and society-wide trends and intervention efforts to reduce inequities, definitive progress on fully understanding and eliminating health inequities remains unfulfilled. An essential avenue for understanding and addressing health care inequities may be to more directly observe how vulnerable populations interact with the US health care system. Primary care providers are the front door to this system-even in a nation without universal primary care access- to which a wide swath of the United States, including vulnerable populations, access at multiple points throughout their life. 7 , 8 The addition of primary care research perspectives, approaches, and data into health inequities research may be a crucial step toward understanding, improving, and ultimately helping end health inequity in the United States.

• The What and Why of Primary Care Research

Primary care is first contact health care that is comprehensive, continuous, and coordinated. 9 Primary care research is research done in the primary care environment, 10 therefore, involving primary care patients, practitioners, perspectives, and priorities. Because of its comprehensiveness, studying health inequities in the context of primary care allows for the use of multiple paradigms and methodologic approaches to understanding almost any state of health, disease, social challenge, or societal circumstance patients might face. Further, while most research methods can be used in primary care, some methods such as pragmatic trials, 11 , 12 dissemination and implementation research, 13 and patient-investigator partnerships 14 are especially appropriate for primary care settings. Primary care delivery will not solve inequity alone, but observational and interventional research in the primary care setting is an essential and overlooked piece of the science to understand and reduce health inequity. Research in the primary care setting is a window that displays disease and health care and a wide representation of the issues relevant to inequity: the experience of violence, poverty, addiction, racism, cultural factors, and disadvantage, among others, throughout a lifetime. 7 , 8 , 15 , 16 The beneficial relationships forged in primary care 17 , 18 may, in part, start to mitigate the effects of violence perpetrated by researchers in the past. 19 There have been calls to examine inequities over the life course, 20 and primary care disciplines, especially family medicine, are well-positioned to do this given their comprehensiveness in scope.

The Reach of Primary Care for Health Inequities Research

For the researcher interested in health inequities research, a context-specific discipline might elicit sampling concerns: does the US primary care environment contain enough patients experiencing inequities to produce meaningful understanding on these issues? Is not studying those in the US primary care environment just the study of care quality for a subpopulation with unlimited access to resources and all the health care they need? Are vulnerable people—with poor access to services and resources—represented in a context that requires access a priori ? Historically, in the United States, these questions may have resulted in caution in evaluating health inequities in primary care settings, but this is rapidly changing. Even in a society that does not have universal health care coverage, a large proportion of the population does have contact with primary care providers; in national surveys, more than 85% of US individuals, across demographic groups, have at least some usual source of care (doctor's office or clinic/health center—not the emergency department). 21 Specifically, vulnerable and marginalized populations do see primary care providers, especially in the nation's network of community health centers (CHCs). CHCs (clinics receiving federal funding to provide comprehensive primary care) serve ∼30 million patients in the United States, approximately 10% of the country, regardless of citizenship, income, insurance status, language spoken, or other socioeconomic criteria, and especially serve low-income patients and racial/ethnic minorities. 8 Whether a patient accesses a CHC or not, numerous primary care networks, many of them now interconnected, widely represent those who might experience health inequities. For instance, primary care practices nationwide are increasingly part of data-connected networks – research networks, networks with shared administrative resources, and networks that share electronic health records and their functionalities for innovation and data aggregation. 22 , 23 These networks join the existing core resource of practice-based research networks (PBRNs) in primary care. 24 Though large connected primary care networks (data networks and PBRNs) may not have the representativeness of national surveys, they contain large patient samples with richer information on objectively measured health outcomes, care utilization, and increasingly, robust social determinants of health data. 25 All this is routinely collected in primary care clinics, which is challenging to collect or subject to recall bias in public health surveys. Amid calls for the integration of social care and the evaluation of social determinants of health into health care, 26 , 27 and calls for multi-level and “complex system analysis reflective of real-world settings” 4 to better understand inequity, these reports have missed an opportunity to explicitly recommend primary care research as a viable and necessary response to these calls. The primary care setting sits at the nexus of complex system factors, is already in the “real world” and therefore may have enhanced external validity, is where most social needs are witnessed in health care, and is where research into these aims is likely to be most effective. In addition, primary care data are already multi-level and routinely collected: multiple visit observations for a patient over time, patients nested within providers, providers nested within clinics, and clinics nested in neighborhoods, cities, and states. 22 , 25

• Recommendations to Improve Health Inequity Research

Researchers interested in US health inequities should consider primary care settings as a crucial avenue for understanding the full picture of health inequity and developing real-world interventions to end this inequity. The published opportunities of the NIMHD Health Disparities Science Visioning Initiative 3 all rely on studying the primary care environment. Still, primary care is not explicitly mentioned in this list. We would continue the call for an enhanced partnership between primary care and public health in a manner that leverages the research strengths of both fields to take advantage of these opportunities optimally. This outcome would mean a concerted and longitudinal integration of national US survey data with primary care-related datasets to even more fully capture the exposures, experiences, and care of those most at risk for poor health outcomes. Second, it would mean sustained collaboration in developing and testing scalable health-related interventions that span boundaries: boundaries between regions, care settings, and between “community” and “health care” settings. In the long-term, funding agencies and health systems could invest even more in primary care centered networks to continue building data sources that have the potential to aggregate significant data on the longitudinal experience and outcomes of vulnerable populations over the entire life course. While Congress has designated the Agency for Health Care Research and Quality (AHRQ) as the “principal source of funding for primary care research,” the AHRQ's 2021 budget was 0.5% of the NIH's budget, 28 , 29 and a very small proportion of the NIH budget is awarded to disciplines in primary care research. 30 In response to all these issues, we make the following recommendations:

Funding agencies in the United States should increasingly fund research projects that utilize broad primary care settings to study health inequity.

Journal editorial boards should recognize the importance, scientific merit, and enhanced external validity of utilizing primary care settings in health inequity research. They should prioritize the inclusion of primary care researchers—especially those with experience in health equity research— on board rosters.

Researchers should consider multi-level, etiologic, and complex system analyses 4 and understand that primary care sits at a nexus of multi-level investigations into health inequity (primary care is the bridge between biology, behavior, health care, and community); researchers should utilize the existing multi-level data in primary settings and networks for observational and intervention studies.

Primary care providers treat and health inequities affect every organ, every system, every malady, in every family, and every community. Primary care researchers, along with public health researchers, may bring about understanding and intervention to end health inequity in the United States together.

• Acknowledgments

The authors would acknowledge our home institutions and the patients and staff of the OCHIN Practice-Based Research Network, who support our work in general.

This article was externally peer reviewed.

Funding: National Institute on Aging and National Institute on Minority Health and Health Disparities.

Conflict of interest: None.

To see this article online, please go to: http://jabfm.org/content/33/5/849.full .

• Received for publication February 12, 2021.
• Revision received April 12, 2021.
• Accepted for publication April 13, 2021.
• 1. ↵ World Health Organization . 10 Facts on Health Inequities and their Causes . Available from: https://www.who.int/features/factfiles/health_inequities/en/ . Published 2017 . Accessed December 2019 .
• Dankwa-Mullan I ,
• Williams K ,
• Wasserman J ,
• Palmer RC ,
• Ibrahim SA ,
• Pérez-Stable EJ
• Satcher D ,
• Higginbotham EJ
• 6. ↵ National Center for Health Statistics . Healthy People 2020 Data Issues 2015 . Available from: https://www.cdc.gov/nchs/healthy_people/hp2020/hp2020_data_issues.htm . Published November 6, 2015 . Accessed November 1, 2020 .
• Stange KC ,
• Flocke SA ,
• Miller WL ,
• Crabtree BF ,
• Zyzanski SJ
• 8. ↵ National Association of Community Health Centers . Community Health Center Chartbook . Available from: http://www.nachc.org/wp-content/uploads/2019/01/Community-Health-Center-Chartbook-FINAL-1.28.19.pdf . Published 2019 . Accessed November 1, 2020 .
• 9. ↵ World Health Organization . Primary Health Care . Available from: https://www.who.int/news-room/fact-sheets/detail/primary-health-care . Published April 1, 2021 . Accessed June 15, 2021 .
• Starfield B
• Carroll JK ,
• Dickinson LM ,
• Eichner B ,
• Michaels LAC ,
• Heintzman J ,
• Crosson J ,
• Likumahuwa S ,
• Warren NT ,
• Gaudino JA Jr . ,
• Likumahuwa-Ackman S ,
• Gillespie CF ,
• Bradley B ,
• Cottrell EK ,
• Hendricks M ,
• Dambrun K ,
• Blumenthal DME ,
• Nutting PA ,
• Goodwin MA ,
• Zyzanski SJ ,
• 19. ↵ Office of Environment, Health, Safety and Security, US Department of Energy . Chapter 12: the iodine 131 experiment in Alaska . In: Final Report of the Advisory Committee on Human Radiation Experiments . Washington, DC : US Government Printing Office ; 1995 . (Stock no. 061-000-008489) Available from: https://bioethicsarchive.georgetown.edu/achre/final/chap12_4.html . Accessed March 29, 2021 .
• Gilman SE ,
• Geronimus AT
• 21. ↵ Centers for Disease Control and Prevention . Early Release of Selected Estimates Based on Data From the National Health Interview Survey, January–September 2017 . Available from: https://www.cdc.gov/nchs/data/nhis/earlyrelease/EarlyRelease201803.pdf . Published March, 2021 . Accessed November 1, 2020 .
• Cottrell E ,
• 23. ↵ Kaiser Permanente Center for Health Research . Data & Analytics Expertise . Available from: https://research.kpchr.org/About/Capabilities/Data-Analytics-and-Expertise . Published 2018 . Accessed November 1, 2020 .
• 24. ↵ Agency for Healthcare Research and Quality . Practice-Based Research Networks . Available from: https://pbrn.ahrq.gov/about . Accessed March 25, 2021 .
• Bazemore AW ,
• 26. ↵ Lancet Editorial Staff . No health care without social care . The Lancet 2019 ; 394 : 1206 . OpenUrl
• 27. ↵ National Academy of Medicine . Integrating Social Care into the Delivery of Health Care: Moving Upstream to Improve the Nation's Health . Available from: http://nationalacademies.org/hmd/Reports/2019/integrating-social-care-into-the-delivery-of-health-care . Published September 25, 2019 . Accessed November 1, 2020 .
• 28. ↵ American Institute of Physics . Final FY21 Appropriations: National Institutes of Health . Available from: https://www.aip.org/fyi/2021/final-fy21-appropriations-national-institutes-health2021 . Published November 19, 2020 . Accessed March 25, 2021 .
• 29. ↵ Agency for Healthcare Research and Quality . Budget Estimates for Appropriations Committees, Fiscal Year 202 . Available from: https://www.ahrq.gov/cpi/about/mission/budget/2021/index.html . Published February 2021 . Accessed March 25, 2021 .
• Cameron BJ ,

In this issue

• Table of Contents
• Table of Contents (PDF)
• Cover (PDF)
• Index by author
• Back Matter (PDF)
• Front Matter (PDF)

Thank you for your interest in spreading the word on American Board of Family Medicine.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager

• Tweet Widget
• Facebook Like
• Google Plus One

Jump to section

Related articles.

• No related articles found.
• Google Scholar

Cited By...

• Family Medicine Research on Health Equity, Addiction, and Eating Breakfast--Just for Starters

More in this TOC Section

• What Complexity Science Predicts About the Potential of Artificial Intelligence/Machine Learning to Improve Primary Care
• An Exploration of Professionalism in Everyday Practice
• Thinking “Green” When Treating “Pink Puffers” and “Blue Bloaters”—Reducing Carbon Footprint When Prescribing Inhalers

Similar Articles

• Open access
• Published: 13 December 2023

Features and development trends of primary care research conducted by practice-based research networks from 1991 to 2023: a scoping review protocol

• Yang Wang 1 , 2 ,
• Xinyang Cao 3 ,
• Zhijie Xu 4 &
• Hai Fang 2  

Systematic Reviews volume  12 , Article number:  229 ( 2023 ) Cite this article

793 Accesses

Metrics details

Practice-based research networks (PBRNs) have been recognized as essential laboratories and mechanisms for developing primary care research. This scoping review aims to examine and map the features and development trends of productivity, research categories, and methods in original primary care research conducted by global PBRNs between 1991 and 2023.

We have assembled an interdisciplinary team that will undertake this scoping review, following the framework developed by Arksey and O’Malley. Targeted literature includes original primary care research conducted by PBRNs, published from January 1, 1991, to December 31, 2023. An integrated search strategy will gather publications from 3 electronic databases (PubMed, Web of Science, and Embase), 16 major primary health care journals, and 364 relevant organizations. Two experienced researchers will independently screen the titles, keywords, and abstracts of all references and extract data regarding eight key elements. Disagreements between the reviewers will be resolved through group discussions, moderated by a third reviewer. Articles to be included will (1) be conducted in the primary care context, (2) be led by PBRNs, (3) provide a full report of original research, and (4) be published in a peer-reviewed journal between the aforementioned dates in any language. Exclusions encompass reviews, letters, commentaries, case reports, and conference papers. Final data will be displayed using tables and charts according to different conceptual categories.

This scoping review is one of the initial attempts to delineate the development trends and features of primary care research conducted by PBRNs. This study will provide reference information for researchers in countries/regions that are building their research infrastructure and capacity in general practice, family medicine, and primary care.

Systematic review registration

Registered in OSF on July 25, 2022 ( https://osf.io/zgv9c ).

Peer Review reports

Enhancing primary health care (PHC) is the most crucial approach to improving people’s health and social well-being and achieving health-related sustainable development goals [ 1 ]. In the 14 key levers supporting the successful implementation of PHC proposed by the World Health Organization (WHO), the role of developing PHC-oriented research is to support all the other levers by creating and providing knowledge, evidence, and intelligence. Therefore, it has unique features: covering all components of PHC, cooperating with a broad range of stakeholders, and pragmatically addressing important issues for the local community (“Act local, think global”) [ 2 , 3 ].

Primary care research is defined as “research done in a primary care context” [ 4 ] and involves five research types (basic research, clinical research, health service research, health system research, and educational research) [ 5 ]. It is a “bottom-up” multidisciplinary research area that views primary care facilities and communities as the fundamental units and places the research focus on them [ 6 ]. The collaboration between primary care practitioners and researchers is crucial for developing primary care research, as this can integrate science and practice and facilitate mutual reinforcement [ 7 , 8 ].

Over the last half-century, a key approach to achieving this collaboration has been practice-based research networks (PBRNs) [ 8 ]. The Agency for Healthcare Research and Quality (AHRQ) defines this as “a group of ambulatory practices devoted principally to the primary care of patients and affiliated in their mission to investigate questions related to community-based practice and to improve the quality of primary care” [ 9 ]. It highlights PBRNs’ three core features as a research laboratory: engagement in primary care research based on clear and stable purpose, structure, and resources [ 10 , 11 ]; components including multiple primary care practices favorable to data collection [ 12 ]; and a focus on answering research questions derived from practice or conducting research, which is important for practice, especially regarding translating evidence into practice [ 13 ]. In some countries and regions, it is also named “primary care research network” [ 12 ].

The value of PBRNs for developing primary care research and related health science research has been widely recognized by the global academic community in recent decades [ 12 , 14 , 15 , 16 ]. Since the initiation of health system reforms in 2009, which focused on augmenting human and financial resources for primary care, China has witnessed a rapid expansion of primary care and general practice [ 17 ]. Against this backdrop, a common concern in the local academic community in recent years is how to cultivate primary care research in China, a middle- to low-income country with a distinct health system backdrop, in order to better foster local primary care practices [ 18 ]. Consequently, creating a comprehensive map that chronicles the historical development trends in this domain over the past 30 years would serve as a macro-level reference for health and research policymakers and administrators, both in China and in countries with similar circumstances. Such a reference would provide them a panoramic view of the distinctive history and evolution of research derived from PBRNs.

In our previous study, we found that the number of studies recorded in the AHRQ’s literature database of PBRNs has been increasing rapidly since the 1990s [ 19 ]. The number of registered PBRNs and published primary care studies also increased during the same period [ 8 , 16 , 20 , 21 ]. The spread of registered PBRNs across the world may be reflective of the change in the productivity of PBRNs. This trend is also seen in countries outside of North America, such as the UK, the Netherlands, and Australia [ 22 , 23 ]. In addition, some changes to primary care research methods during this period may also reflect the evolution in the features of the studies conducted by PBRNs [ 24 , 25 , 26 , 27 ]. Therefore, we would like to further examine and map the global features and development trends of productivity, research categories, and methods in original primary care research conducted by PBRNs between 1991 and 2020. In bibliometrics, productivity is usually approximate to the number of available scientific publications, which is a key indicator for measuring the research output of a researcher, institution, or region/country [ 28 ], and has been used to assess the development of primary care research [ 20 , 21 ].

We identified the innovative value of our study after conducting a preliminary search in PubMed, Web of Science, Embase, Cochrane Database, and Open Science Framework.

Study design

We plan to conduct this work following the recommended procedures developed by Arksey and O’Malley [ 29 ]. The procedure is a common and established approach for designing and conducting a scoping review and involves five stages: (1) identifying the research question, (2) identifying relevant studies, (3) selecting studies, (4) charting the data, and (5) collating, summarizing, and reporting results. Therefore, we assembled an interdisciplinary team (including YW and HF, primary health care researchers; XC, scientific editors of a primary care academic journal; and ZX, an academic general practitioner) to conduct this scoping review. Furthermore, we will report our results according to the PRISMA Extension for Scoping Reviews (PRISMA-ScR) checklist [ 30 ]. The scoping review protocol is being registered in the Open Science Framework database ( https://osf.io/zgv9c ).

Stage 1: Identifying the research question

This study aims to explore the following research question: “What are the features and development trends of productivity, research type, and methods of original primary care research that were conducted by PBRNs and published between 1991 and 2023?” We plan to initiate our search in January 2024. In this work, we restricted the meaning of “primary care research” to “research done in a primary care context,” based on Starfield’s definition [ 4 ]. As the meaning of “primary care context” may vary in different countries and regions, we will refer to the reference book on global primary care research published by the World Organization of Family Doctors (WONCA), which describes the primary care context in different regions [ 31 ]. In addition, we will also refer to the definitions proposed by major local academic associations. For example, in the USA, the American Academy of Family Physicians defines “primary care” as “health care services by physicians and their health care teams” and limited “primary care physician” to “a specialist in family medicine, general internal medicine, or general pediatrics” [ 32 ]. Further, according to the AHRQ definition, we defined “research conducted by PBRNs” as articles that meet any of the following conditions: (1) clearly labeled as a study by PBRNs; (2) the first author is affiliated to PBRNs (that is, PBRN researchers must have made major contributions); and (3) the contribution of PBRNs have been reported in the papers’ introduction or methods sections [ 9 ].

Stage 2: Identifying relevant studies

In our previous study, we recognized that adding more databases to the three main databases (PubMed, Web of Science, and Embase) does not improve the efficiency of obtaining target publications [ 19 ]. Moreover, the accuracy of the papers collected in AHRQ’s PBRN literature database that were defined as research conducted by PBRNs was less than 50% [ 19 ]. Therefore, we plan to search the target publications through three different approaches and then combine them.

Electronic database

We will conduct a comprehensive search of original articles published from January 1, 1991, to December 31, 2020, in PubMed, Web of Science, and Embase through a search strategy developed in previous work [ 33 , 34 ] (Supplement Table 1 ).

Hand-searching of key journals

We will manually search 16 major primary health care journals indexed in the Journal Citation Report. Based on our initial search, it is possible to find published original research conducted by PBRNs in them (Supplement Table 2 ).

Relevant organizations

We will search PubMed for each PBRN’s name as an affiliation according to a PBRNs list (Supplement Table 3 ) that we compiled—which was based on the PBRNs mentioned in three important PBRNs-related information sources [ 6 , 35 , 36 ]—and three studies conducted by researchers experienced with PBRNs [ 19 , 34 , 37 ]. We reviewed the original list and removed PBRNs that did not meet our research aim; for example, we removed all non-primary care PBRNs from AHRQ’s PBRN registry website.

Stage 3: Selecting studies

After the search, all the information regarding the included articles will be imported to EndNote X9.2 software (Clarivate Analytics, Philadelphia, USA, 2019). We will remove duplicates using the software functions. The revision and study selection will then be performed by importing the literature information into Rayyan, a web and mobile app for conducting systematic reviews that improves screening efficiency with its semi-automation function [ 38 ]. Two researchers with expertise in reviewing academic papers (YW, XC) will screen the titles, keywords, and abstracts of the first 300 references to improve the eligibility criteria and ascertain that they are consistently understood through weekly comparisons and group discussions. A follow-up screening of approximately 10,000–15,000 articles will begin when the consistency between the two reviewers is greater than 90%. If the two reviewers fail to reach a consensus, the third researcher (ZX) will moderate specific group discussions to resolve discrepancies. Finally, we will use a PRISMA flow diagram to demonstrate the study selection process. Articles that meet the following criteria will be included:

Conducted in the primary care context

Conducted by PBRNs, including the three cases we elaborated on in Stage 1

A full report of original research

Published in a peer-reviewed journal between January 1, 1991, and December 31, 2023

Published in any language

We will exclude reviews, letters, commentary, case reports, and conference papers. Given that the main mechanism of PBRNs in supporting primary care research is to generate research from primary care practice and translate or implement evidence into practice, we will include only original research in this review and will exclude non-research papers as well as reviews. We excluded review papers because the fundamental purpose of review studies is to “gather research, getting rid of rubbish and summarizing the best of what remains” [ 39 ], which essentially makes them secondary and information studies that can be conducted by any type of institution. Thus, they are not closely related to the unique mechanism between PBRNs and primary care research.

Stage 4: Charting the data

We plan to extract nine elements from each article according to the proposed data extraction form (Table 1 ). Of these elements, six (article title, publication year, journal’s name, first authors’ name, first authors’ affiliation, and country/region) can be extracted from the bibliometric record. We will clean and classify them with the support of Openrefine, an AI-based software that supports the efficient cleaning and transformation of bibliometric data [ 40 ]. In addition, two reviewers (YW and XC) will independently classify each included article by research category and research method according to the taxonomy of primary care research developed by WONCA and the research methods list summarized from the “appropriate research methodology” section of Research Agenda for General Practice/Family Medicine and Primary Health Care in Europe [ 3 ]. They may also add new but relevant classifications based on subsequent findings outside the aforementioned frameworks. They will first classify 50 papers, and after the raw agreement is higher than 90%, they will classify the remaining papers.

Stage 5: Collating, summarizing, and reporting results

The data will be presented in tables and charts depending on the conceptual categories. Tables and charts will present the distribution of years, countries/regions, type of first authors’ affiliations, research categories, and methods of the identified studies. Narrative summaries will be used to provide additional explanatory information based on the collected data. We also plan to use descriptive analysis and comparative statistics (including chi-squared test and nonparametric tests) to find possible common features and the development trend of a group of studies during a stable period.

Based on our knowledge, this scoping review is one of the initial attempts to delineate the development trends and features of primary care research conducted by PBRNs. In it, we will generate a systematic search strategy to combine the information obtained from multiple approaches to ensure the comprehensiveness of the search. This study will provide reference information for researchers in countries/regions that are building their research infrastructure and capacity in general practice, family medicine, and primary care.

The findings of this study will present a macroscopic outline of how pioneer researchers in primary care, within various national and regional research environments and institutions, choose distinct principal research domains and methodologies, utilizing PBRNs as a fundamental base to conduct primary care research work. By offering a holistic view of the PBRN-grounded primary care research landscape across diverse regions and research settings, our work aspires to identify and illuminate the varied strategies, challenges, and successes encountered by researchers globally. Thus, this review not only encapsulates the historical and current trajectories of primary care research but also aims to equip researchers, practitioners, and policymakers with insights and benchmarks that can inform the design, initiation, and optimization of future PBRN studies and initiatives in different sociocultural and institutional contexts.

As the main contribution of PBRNs to primary care research is generating practice-based knowledge, this study will include only original articles and exclude reviews. This scoping review may miss some studies published in non-English journals that are not indexed in the Journal Citation Report.

Availability of data and materials

Not applicable.

Abbreviations

• Practice-based research networks
• Primary health care

World Health Organization

Agency for Healthcare Research and Quality

PRISMA Extension for Scoping Reviews

World Organization of Family Doctors

World Health Organization. Declaration of Astana. Global conference on primary health care. Astana, Kazakhstan, 25–26 October 2018. WHO/HIS/SDS/2018.61.

World Health Organization, United Nations Children’s Fund. Operational framework for primary health care: transforming vision into action WHO. 2020. Available from: https://apps.who.int/iris/bitstream/handle/10665/337641/9789240017832-eng.pdf . Accessed November 24, 2022.

van Royen P, Beyer M, Chevallier P, Eilat-Tsanani S, Lionis C, Peremans L, et al. Series: the research agenda for general practice/family medicine and primary health care in Europe. Part 5: needs and implications for future research and policy. Eur J Gen Pract. 2010;16:244–8. https://doi.org/10.3109/13814788.2010.528385 . PMID:21073268.

Article   PubMed   Google Scholar  

Starfield B. A framework for primary care research. J Fam Pract. 1996;42:181–5.

CAS   PubMed   Google Scholar  

Beasley JW, Starfield B, van Weel C, Rosser WW, Haq CL. Global health and primary care research. J Am Board Fam Med. 2007;20:518–26. https://doi.org/10.3122/jabfm.2007.06.070172 . (PMID:17954858).

Beasley J, Bazemore A, Mash B. The nature of primary care research. In: Goodyear-Smith F, Mash B, editors. International perspectives on primary care research. London: CRC Press, Taylor & Francis Group; 2016. p. 11–3.

Google Scholar  

Nutting PA, Green LA. Practice-based research networks: reuniting practice and research around the problems most of the people have most of the time. J Fam Pract. 1994;38:335–6 (PMID:8163956).

Green LA, Hickner J. A short history of primary care practice-based research networks: from concept to essential research laboratories. J Am Board Fam Med. 2006;19:1–10. https://doi.org/10.3122/jabfm.19.1.1 . (PMID:16491999).

Agency for Healthcare Research and Quality. About PBRNs. 2021. https://www.ahrq.gov/ncepcr/research-transform-primary-care/pbrn/index.html . Accessed 6 Mar 2022.

Green LA, White LL, Barry HC, Nease DE, Hudson BL. Infrastructure requirements for practice-based research networks. Ann Fam Med. 2005;3:S5-11. https://doi.org/10.1370/afm.299,PMID:15928219PMCID:PMC1466956 .

Article   PubMed   PubMed Central   Google Scholar  

Davis MM, Keller S, DeVoe JE, Cohen DJ. Characteristics and lessons learned from practice-based research networks (PBRNs) in the United States. J Healthc Leadersh. 2012;4:107–16. https://doi.org/10.2147/JHL.S16441,PMID:26213481PMCID:PMC4512302 .

van Weel C, Rosser WW. Improving health care globally: a critical review of the necessity of family medicine research and recommendations to build research capacity. Ann Fam Med. 2004;26:S5-16. https://doi.org/10.1370/afm.194,PMID:15655089PMCID:PMC1466768 .

Article   Google Scholar  

Nutting PA, Beasley JW, Werner JJ. Practice-based research networks answer primary care questions. JAMA. 1999;281(8):686–8. https://doi.org/10.1001/jama.281.8.686 . (PMID:10052423).

Article   CAS   PubMed   Google Scholar  

Nutting PA. Practice-based research networks: building the infrastructure of primary care research. J Fam Pract. 1996;42:199–203 (PMID:8606312).

Westfall JM, Mold J, Fagnan L. Practice-based research –‘Blue Highways’ on the NIH roadmap. JAMA. 2007;297(4):403–6. https://doi.org/10.1001/jama.297.4.403 . (PMID:17244837).

Hickner J, Green LA. Practice-based research networks (PBRNs) in the United States: growing and still going after all these years. J Am Board Fam Med. 2015;28:541–5. https://doi.org/10.3122/jabfm.2015.05.150227 . (PMID:26355124).

Meng Q, Mills A, Wang L, Han Q. What can we learn from China’s health system reform? BMJ. 2019;19(365):l2349. https://doi.org/10.1136/bmj.l2349.PMID:31217222,PMCID:PMC6598719 .

Fu Q, Jin H, Yu D. Development status and strategy of research capability of general practice and community health in China: based on the data of Chinese journals in general practice from 2001 to 2020. Chin Gen Pract. 2022;25(34):4252. https://doi.org/10.12114/j.issn.1007-9572.2022.0705(Chinese) .

Wang Y, Xu ZJ, Li L, Kuang L, Xu Y, Han J. The development of practice-based research networks in North America in early stage: a literature review from historical perspective. Chin Gen Pract. 2021;24(28):3525–42. https://doi.org/10.12114/j.issn.1007-9572.2021.00.291(Chinese) .

Glanville J, Kendrick T, McNally R, Campbell J, Hobbs FR. Research output on primary care in Australia, Canada, Germany, the Netherlands, the United Kingdom, and the United States: bibliometric analysis. BMJ. 2011;342:d1028. https://doi.org/10.1136/bmj.d1028,PMID:21385804PMCID:PMC3050436 .

Hajjar F, Saint-Lary O, Cadwallader JS, Chauvin P, Boutet A, Steinecker M, et al. Development of primary care research in North America, Europe, and Australia from 1974 to 2017. Ann Fam Med. 2019;17(1):49–51. https://doi.org/10.1370/afm.2328PMID:30670396PMCID:PMC6342604 .

Fleming DM. The role of research networks in primary care: based on a presentation at WONCA Dublin in June 1998. Eur J Gen Pract. 1998;4(3):96–9.

Pirotta M, Temple-Smith M. Practice-based research networks. Aust Fam Physician. 2017;46(10):793–5.

PubMed   Google Scholar  

Tapp H, White L, Steuerwald M, Dulin M. Use of community-based participatory research in primary care to improve healthcare outcomes and disparities in care. J Comp Eff Res. 2013;2:405–19. https://doi.org/10.2217/cer.13.45PMID:24236682PMCID:PMC4042315 .

Holtrop JS, Rabin BA, Glasgow RE. Dissemination and implementation science in primary care research and practice: contributions and opportunities. J Am Board Fam Med. 2018;31:466–78. https://doi.org/10.3122/jabfm.2018.03.170259 . (PMID:29743229).

Plsek PE, Greenhalgh T. Complexity science: the challenge of complexity in health care. BMJ. 2001;323:625–8. https://doi.org/10.1136/bmj.323.7313.625,PMID:11557716PMCID:PMC1121189 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Creswell JW, Hirose M. Mixed methods and survey research in family medicine and community health. Fam Med Community Health. 2019;28(7):e000086. https://doi.org/10.1136/fmch-2018-000086,PMID:32148709PMCID:PMC6910743 .

Haustein S, Larivière V. The use of bibliometrics for assessing research: possibilities, limitations and adverse effects. In: Welpe I, Wollersheim J, Ringelhan S, Osterloh M, editors. Incentives and performance. Cham: Springer; 2015. p. 121–39.

Chapter   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32. https://doi.org/10.1080/1364557032000119616 .

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169:467–73.

Goodyear-Smith F, Mash B, editors. International perspectives on primary care research. London: CRC Press, Taylor & Francis Group; 2017.

American Academy of Family Physicians. Primary care. https://www.aafp.org/about/policies/all/primary-care.html . Accessed 6 Mar 2022.

Gill PJ, Roberts NW, Wang KY, Heneghan C. Development of a search filter for identifying studies completed in primary care. Fam Pract. 2014;31:739–45. https://doi.org/10.1093/fampra/cmu066 . (Epub 2014 Oct 18 PMID:25326923).

Dania A, Nagykaldi Z, Haaranen A, Muris JW, Evans PH, Mäntyselkä P, et al. A review of 50-years of international literature on the internal environment of building practice-based research networks (PBRNs). J Am Board Fam Med. 2021;34:762–97. https://doi.org/10.3122/jabfm.2021.04.200595 . (PMID:34312269).

Gunn JM. Should Australia develop primary care research networks? Med J Aust. 2002;15(177):63–6. https://doi.org/10.5694/j.1326-5377.2002.tb04667.x . (PMID:12098340).

Agency for Healthcare Research and Quality. PBRN registry. https://www.ahrq.gov/ncepcr/communities/pbrn-registry/index.html . Accessed 6 Mar 2022.

Tierney WM, Oppenheimer CC, Hudson BL, Benz J, Finn A, Hickner JM, et al. A national survey of primary care practice-based research networks. Ann Fam Med. 2007;5:242–50. https://doi.org/10.1370/afm.699,PMID:17548852,PMCID:PMC1886484 .

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev 20165:210. https://doi.org/10.1186/s13643-016-0384-4 PMID:27919275 PMCID:PMC5139140.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26(2):91–108.

Ham K. Openrefine. 2.5 version. http://openrefine.org . free, open-source tool for cleaning and transforming data. J Med Libr Assoc 2013;101:233.

Download references

Acknowledgements

We appreciate Dr. Sheyu Li and Dr. Chenxi Liu for reviewing and advising on the initial version of the study design.

Author information

Authors and affiliations.

School of Public Health, Peking University, Beijing, 100191, China

China Center for Health Development Studies, Peking University, Beijing, China

Yang Wang & Hai Fang

Chinese General Practice Publishing House, Co., Ltd, Handan, China

Xinyang Cao

Department of General Practice, The Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

YW developed the first version of the protocol and search strategies. XC, ZX, and HF critically revised the protocol, search strategy, and manuscript for important intellectual content. All authors approved the final version to be published and agreed to be accountable for all aspects of the work.

Corresponding author

Correspondence to Hai Fang .

Ethics declarations

Ethics approval and consent to participate.

This study does not require ethical approval as human participants, and original, unpublished data were not involved. Patients and the public were not involved in the preparation of this protocol and will not be involved in the final scoping review.

Consent for publication

We plan to submit this study as a conference paper to the North American Primary Care Research Group 2024 annual meeting (USA) and as a review paper to a primary care academic journal.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Search strategy for PubMed, WOS, and Embase. Table S2. List of 16 primary care and family medicine journals. Table 3. List of PBRNs.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wang, Y., Cao, X., Xu, Z. et al. Features and development trends of primary care research conducted by practice-based research networks from 1991 to 2023: a scoping review protocol. Syst Rev 12 , 229 (2023). https://doi.org/10.1186/s13643-023-02395-y

Download citation

Received : 05 April 2023

Accepted : 17 November 2023

Published : 13 December 2023

DOI : https://doi.org/10.1186/s13643-023-02395-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Primary care
• General practice
• Family medicine

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Primary health care quality indicators: An umbrella review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations MEDCIDS–Department of Community Medicine, Information and Health Decision Sciences, Faculty of Medicine, University of Porto, Porto, Portugal, CINTESIS–Centre for Health Technology and Services Research, Porto, Portugal

Roles Data curation, Formal analysis, Investigation, Writing – original draft, Writing – review & editing

Affiliation USF Camélias, ACeS Grande Porto VII (ARS Norte)–Vila Nova de Gaia, Portugal

Roles Data curation, Formal analysis, Investigation, Writing – review & editing

Roles Data curation, Formal analysis, Investigation

Affiliation MEDCIDS–Department of Community Medicine, Information and Health Decision Sciences, Faculty of Medicine, University of Porto, Porto, Portugal

Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

Affiliations MEDCIDS–Department of Community Medicine, Information and Health Decision Sciences, Faculty of Medicine, University of Porto, Porto, Portugal, CINTESIS–Centre for Health Technology and Services Research, Porto, Portugal, Public Health Unit, ACeS Grande Porto VIII (ARS Norte)–Espinho/Gaia, Portugal

Roles Conceptualization, Writing – review & editing

Roles Formal analysis, Validation, Writing – review & editing

Roles Conceptualization, Formal analysis, Methodology, Resources, Validation, Visualization

Roles Conceptualization, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

• André Ramalho, 
• Pedro Castro, 
• Manuel Gonçalves-Pinho, 
• Juliana Teixeira, 
• João Vasco Santos, 
• João Viana, 
• Mariana Lobo, 
• Paulo Santos, 
• Alberto Freitas

• Published: August 16, 2019
• https://doi.org/10.1371/journal.pone.0220888
• Reader Comments

Nowadays, evaluating the quality of health services, especially in primary health care (PHC), is increasingly important. In a historical perspective, the Department of Health (United Kingdom) developed and proposed a range of indicators in 1998, and lately several health, social and political organizations have defined and implemented different sets of PHC quality indicators. Some systematic reviews in PHC quality indicators are reported but only in specific contexts and conditions. The aim of this study is to characterize and provide a list of indicators discussed in the literature to support managers and clinicians in decision-making processes, through an umbrella review on PHC quality indicators. The methodology was performed according to PRISMA Statement. Indicators from 33 eligible systematic reviews were categorized according to the dimensions of care, function, type of care, domains and condition contexts. Of a total of 727 indicators or groups of indicators, 74.5% (n = 542) were classified in process category and 89.5% (n = 537) with chronic type of care (n = 428; 58.8%) and effective domain (n = 423; 58.1%) with the most frequent values in categorizations by dimensions. The results of this overview of reviews are valuable and imply the need for future research and practice regarding primary health care quality indicators in the most varied conditions and contexts to generate new discussions about their use, comparison and implementation.

Citation: Ramalho A, Castro P, Gonçalves-Pinho M, Teixeira J, Santos JV, Viana J, et al. (2019) Primary health care quality indicators: An umbrella review. PLoS ONE 14(8): e0220888. https://doi.org/10.1371/journal.pone.0220888

Editor: Wisit Cheungpasitporn, University of Mississippi Medical Center, UNITED STATES

Received: March 18, 2019; Accepted: July 22, 2019; Published: August 16, 2019

Copyright: © 2019 Ramalho et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files, and are from studies indexed and available in four databases (Medline, Scopus, ISI-WOS and CINAHL via Ebsco Host).

Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was financed by FEDER - Fundo Europeu de Desenvolvimento Regional funds through the COMPETE 2020 - Operacional Programme for Competitiveness and Internationalisation (POCI), and by Portuguese funds through FCT - Fundação para a Ciência e a Tecnologia in the framework of the project POCI-01-0145-FEDER-030766 (“1st.IndiQare - Quality indicators in primary health care: validation and implementation of quality indicators as an assessment and comparison tool”).

Competing interests: The authors have declared that no competing interests exist.

Introduction

Primary health care (PHC) is defined by the World Health Organization (WHO) as the “essential health care based on scientifically sound and socially acceptable methods and technology, which make universal health care accessible to all individuals and families in a community. It is through their full participation and at a cost that the community and the country can afford to maintain at every stage of their development in the spirit of self-reliance and self-determination" [ 1 ]. Some studies suggest that health systems with better financial and clinical results are those with a greater focus on PHC, thus enhancing the sustainability of the entire health system [ 2 – 4 ]. This depends on providing high quality primary health care, hence raising the need to develop methods for quality assessment and monitoring [ 5 ]. One of these methods is the use of quality indicators—a quantitative measure of the activities, that can assist as a guideline for quality monitoring and evaluation of relevant patient care and support services [ 6 , 7 , 8 ].

Quality of care was defined by the Institute of Medicine (IOM) in 1999 as the degree to which health services increase the likelihood of desired outcomes and are consistent with current professional knowledge [ 9 ]. The evaluation of the degree of quality of care is done through indicators, a set of measures that assist health care quality monitoring and evaluation in several areas, such as governance, management, assistance and support [ 10 , 11 ]. The importance of indicators is given by the fact that they allow for signalling opportunities of improvement, and controlling compliance with the best existing clinical practices, through quantitative parameters (planning, organizational, clinical) aiming better processes and outcomes [ 12 , 13 ].

Studies of how quality can be assessed were conducted by Donabedian and Fleming, who categorized the information from which inferences can be drawn on the topic, in three categories: structure, process and outcome [ 14 ]. The “three-part” assessment approach performed by the authors is only possible because a good structure increases the probability of a good health care processes, and good processes increase the probability of good outcomes [ 14 ]. Importantly, for a process to be a valid measure of quality, it must be closely related to a result that people care about [ 12 ]. It is also worth remembering that we often find factors that interfere with patients' survival and health-disease dynamics, and in these cases, it may be useful for outcome measures to be adjusted for other factors (such as lifestyle, disease) to control confounders that may affect the analysis of outcome indicators [ 10 ]. The development and selection of indicators must meet requirements for use, such as validity, reliability, relevance, pertinence, applicability, data availability, minimum bias, and moreover based on the best evidence available [ 15 , 16 ].

For historical contextualization only, the National Health Service Executive and the Department of Health in United Kingdom (UK)—pioneers in this area—proposed a range of indicators in 1998, many of which would apply to primary health care groups [ 17 ]. The interest in assessing the quality of primary health care services has increased, especially after 2004, when the Quality and Outcomes Framework (QOF) was introduced in the UK [ 18 – 20 ]. After the development of the QOF, some pay-for-performance systems have been developed over the years. These were based on the concept of allocative efficiency: “the optimal use of resources to achieve the intended outcomes” [ 21 ]. As such, financial incentive schemes are being used for PHC units worldwide and professionals, representing a way of rewarding improvements in productivity and/or adaptation to better quality healthcare provision [ 22 ].

Lately, several health, social and political organizations such as World Health Organization (WHO), Organization for Economic Cooperation and Development (OECD), European Commission and the Agency for Research and Quality of Health Care (AHRQ), have defined and implemented different sets of quality indicators for primary care [ 23 – 25 ]. There are several studies proposing PHC quality indicators in different countries, which have led to some systematic reviews revealing substantial geographical variability regarding quality of primary care services [ 26 ]. Identifying papers referring to PHC quality assessment projects, these systematic reviews reported that the number and content of indicators and their domains varied among studies. Moreover, they demonstrated that the lack of standardization of collection tools across projects would lead to invalid comparisons [ 27 – 31 ].

Considering the importance of understanding the PHC context, identifying and measuring quality indicators are essential factors for the achievement of high-quality care [ 32 ]. Some systematic reviews related to the topic are available in the literature but with focus on specific contexts, making it necessary to synthesize and understand the reality of these indicators in a broader scope. The aim of this umbrella review is not an exercise for a meta-review, but rather to identify systematic reviews of studies on quality indicators (QI) for PHC to provide a list of selected indicators considered in systematic reviews.

An umbrella review was conducted, to collect and extract data from all systematic review studies uncovering PHC quality indicators. The methodology was performed according to PRISMA Statement [ 33 ] ( Fig 1 and S1 File ). All the phases were performed by two independent reviewers with a third as a tie-breaker, considering the eligibility criteria. Composing PICO, participants were the primary care systems and the intervention to be analysed is the implementation of quality indicators. The comparator was the categories such as context, dimension, type and domain of care, and the main outcome was the primary health care quality indicators to present a summary list of the indicators used in PHC. The protocol was registered at PROSPERO [ 34 , 35 ] with number CRD42019124170 ( S2 File ).

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0220888.g001

Search strategy

Identification phase..

From an initial set of studies, a search expression was defined and calibrated [ 36 , 37 ] through test rounds for each and combined terms in electronic databases. The search database eligible for calibration was MEDLINE. There were no restrictions on publication period or language. We considered studies from inception until 20 th December 2018, the date when the search was performed. After the calibration, the most relevant search expression ( S3 File ) was used in four databases (MEDLINE, Web of Science, Scopus and CINAHL plus via EBSCOhost). The terms related to systematic reviews were chosen using information of balance between sensitivity and specificity terms, available in the literature [ 38 – 42 ].

Study selection

Eligibility criteria..

Included studies 1) are systematic reviews regardless of their objective or nature (including studies that have used a formal systematic review as their starting point) 2) have a primary health care scope and 3) aim at quality indicators assessment or development. We excluded studies that 1) did not have an abstract in the screening phase or 2) which, in the eligibility phase, did not have the full-text version available, even after direct contact with the author.

Screening phase.

Once we obtained all the articles, duplicate between databases were identified and excluded using Endnote. From 2817 articles, a total of 1480 remained after removing the duplicates and were evaluated in the screening phase (reading of title and abstracts) by two independent reviewers and a third as a tie-breaker.

Eligibility phase.

Full-texts of all the included articles were extracted (n = 33). As it was planned to contact the corresponding author if the full text of the article was not available, we used the ResearchGate website to extract full text articles, or to contact the authors for the articles that were not available. All eligible articles were assessed in full text format. The eligibility criteria were reapplied by two independent reviewers and a third as a tie-breaker, and the reference lists of each eligible article were scrutinized for any omitted studies.

Quality assessment and risk of bias.

The evaluation of the quality and risk of bias of the eligible systematic reviews was carried out by evaluation through AMSTAR-2 tool [ 43 ]. The disagreement between the reviewers was solved by consensus in an agreement meeting by three reviewers. The AMSTAR-2 tool was considered for the definition of quality classification, fulfilling the systematic review research model. Articles that meet AMSTAR-2 requirements have been classified as "HIGH"; those that did not meet up to 2 relevant requirements were classified as "MODERATE", and those with more than 2 requirements not appraised were classified as "LOW". This quality assessment was carried out in order to understand how the studies were conducted and how the indicators were selected. However, none of the selected articles were excluded based on this assessment because the objective of this umbrella review does not include results from implementation of indicators, only a list of indicators implemented. The AMSTAR-2 items #11 and #12 were not applicable to the studies.

Data collection process.

In first stage, a standard data extraction form was created, and general data extracted from each study included the following characteristics: article title, name of first author, publication type, country of origin, year of publication and indicators identified in the studies. Three reviewers independently extracted the data. Differences in data extracted was resolved by consensus method.

A second stage consisted in abstracting information regarding quality indicators using the primary studies in the systematic reviews included. This was necessary since some indicators identified in the systematic reviews lacked a proper description. Finally, indicators duplicated were identified by the reviewers involved in the first and second stage of data extraction and excluded through consensus.

Synthesis analysis.

Analysis of the indicators were carried independently by two reviewers and third as a tie-breaker, who categorized the indicators presented in the systematic reviews included, according to five classifications frameworks: Context reflects the WHO ICPC-2 chapters categorization (General and Unspecified; Blood, Blood Forming Organs and Immune Mechanism; Digestive; Eye; Ear; Cardiovascular; Musculoskeletal; Neurological; Psychological; Respiratory; Skin; Endocrine/Metabolic and Nutritional; Urological; Pregnancy, Childbearing, Family Planning; Female Genital; Male Genital; Social Problems) [ 44 ]; the dimensions of care was defined based on the framework proposed by Donabedian to assess quality of healthcare (structure, process and outcome)[ 10 , 14 ], type of care reflects whether an indicator is associated with acute, chronic, or preventive care [ 10 , 45 ]; function of care conveys information about the purpose of health care (screening and prevention, diagnosis, treatment, follow up and continuity) [ 10 , 45 ] and domains and domain of health care quality was defined based on the framework proposed by National Academy of Medicine (NAM)(former Institute of Medicine) in 2001 (safe, effective, efficient, timely, patient-centred, equitable)[ 9 ].

Frequencies were computed based on these frameworks to analyse and summarize the information extracted, in two perspectives: Indicators by Context and Dimensions of care; and Type, Function and Domain by Dimensions of care.

Search and study selection

The identification phase results returned 2817 articles (being 419 MEDLINE, 1452 Scopus, 567 ISI-WOS, 379 CINAHL via EBSCOhost). After removal of duplicate articles our research started with 1480 articles. Title and abstract were scrutinized for relevance based on inclusion and exclusion criteria. From a total of 1401 excluded articles, 1332 did not meet the eligibility criteria and 69 had no abstract available. The eligibility phase started with 79 articles that were read in their full-text versions, checking for the eligibility criteria. The studies identified by that involved RAND methodology, their inclusion in the umbrella review was justified since the methods included an initial systematic review prior the implementation of a panel discussion for validating appropriateness of indicators. Since the goal was to be as inclusive/comprehensive as possible, these studies were also included. In the perspective of the authors, the exclusion of these studies could compromise comprehensiveness of the umbrella review. The excluded studies (n = 46) did not have a full text version available or did not meet the eligibility criteria. Thirty-three articles were selected [ 29 , 30 , 46 – 77 ], for qualitative analysis and for the quality and risk of bias assessment. ( Fig 1 )

The Quality and Risk of Bias Assessment was carried out using the AMSTAR-2 assessment tool [ 45 ]. This assessment performed by the reviewers classified the confidence rate of each review as "low" (n = 14), “moderate” (n = 17) or “high” (n = 3) ( S4 File ).

Among the studies with low overall confidence rate, the main points of non-compliance with the requirements were, the non-performance of adequate studies selection with no extraction in duplicate (at least two independent reviewers); studies presented the quantity of excluded articles but without proper justification; not considering risk of bias (RoB) in individual studies when interpreting / discussing the results of the review; not using a satisfactory technique to assess RoB in individual studies that were included in the review and did not provide a satisfactory explanation for, or discussion of any heterogeneity observed in the results.

Study characteristics.

The 33 articles in this umbrella review included articles from Canada (n = 5), Spain (n = 5) and the United Kingdom (n = 5), among other countries ( Fig 2 ).

https://doi.org/10.1371/journal.pone.0220888.g002

Although the diversity of countries where the systematic reviews were performed, all articles were evaluated in the English version, except article #16 (Spanish).

The reviews comprised a total of 1406 included primary studies and 21 national guidelines, The databases used to search for these articles were the most varied, with the most used databases: MEDLINE (100%) and EMBASE (70%) ( Table 1 ). Seven hundred and twenty seven (n = 727) indicators were extracted from the systematic reviews and primary studies in the reviews (Supplementary Material S1 Appendix ) .

https://doi.org/10.1371/journal.pone.0220888.t001

The dimension of care with the highest number of indicators by context was process (n = 548, 74.5%), followed by outcome (n = 146, 20.0%) and structure (n = 46, 6.0%). The frequency of indicators among the classification by dimension of care and condition contexts is shown in Table 2 . When analysed by dimension of care and condition context, the indicator totals within each dimension (columns) could not be added up because there were indicators (n = 13) that participate in more than one context category within each dimension of care. The total number of indicators analysed was the denominator of the percentage in parentheses and refers to the total number of indicators in the extraction list (n = 727) indicated in the heading. The same is observed in context totals (lines). The ranking of the highest number of indicators found were classified in the categories A—general and non-specific followed by the K—Circulatory System categories specific, P–Psychological and R—Respiratory System. The categories B—Blood, hematopoietic and lymphatic organs, H—Ears and Z—Social Issues, had no indicators presented in the included studies.

https://doi.org/10.1371/journal.pone.0220888.t002

Among the indicators of structure (n = 45), the indicators with the most frequent type of care were those classified in all three categories—Acute, Chronic and Preventive (n = 34, 45.3%), e.g. Professional profiles; Primary care expenditures; Availability of primary care services. Those of specific category of type of care were less frequent ( Table 3 ).

https://doi.org/10.1371/journal.pone.0220888.t003

Structure indicators were more commonly assigned to more than three functions of care (n = 35, 77.7%) (Diagnosis, Screening and Prevention, Follow-up and continuity, Treatment), eg Availability: Number of physicians per unit of population; Availability: Number of hospital beds per unit of population; Technical efficiency.

Most structure indicators were associated with the effective domain of health care quality (n = 22, 48.8%) e.g. Governance: (From) centralization of primary care management and service development; Integration of primary care in the health care system; Appropriate technology in primary care. No structure indicators was associated with the safe domain of health care quality.

Among the indicators of process (n = 542), Chronic care was the most frequent type of care observed (n = 355, 65.5%), e.g. Comorbid psychiatric conditions and response to treatment; Follow-up contacts during treatment episode after initial evaluation; Comprehensive diabetes care: HbA1c testing. Preventive care (n = 88, 16.2%) and all types of care (n = 80, 14.7%) shared similar frequencies ( Table 4 ).

https://doi.org/10.1371/journal.pone.0220888.t004

Treatment was the most frequent function of care of the process indicators (n = 254, 46.8%), e.g Tranquilisers prescribed: % of the recommended; Possible contraindications should be taken into account when antibiotics are prescribed; Co-prescription of itraconazole with simvastatin, or with atorvastatin at a dose ≥80mg. Screening and prevention and Follow up and continuity were also common, associated with 111 indicators each. Examples of Screening and Prevention indicators are: Pap smear rate; Urinary incontinence during initial dementia evaluation; Preventive care Immunizable conditions; Medical attention for nephropathy; and of Follow up and Continuity: Follow up by the same clinician; Plan for follow up care explained and scheduled; Extra pyramidal effects monitoring; Percentage of patients with asthma and measures of variability or reversibility recorded. Most process indicators were also associated with the effective domain of health care quality (n = 310, 57.2%) e.g. Follow-up contacts during treatment episode after initial evaluation; Coordinated care; Asthma: Percentage of children with follow-up from the same doctor for at least 80% of their visits. Also a common domain of health care quality in the listing was Safe (n = 152, 28%), e.g. Detection of Falls; Polyfarmacy; Systemic Lupus Erythematosus: Discussion about teratogenic risks of medication.

Among the indicators of Outcome (n = 140), Chronic care was the most frequent type of care observed (n = 67, 47.8%), e.g. Absenteeism from Work/School for Asthma; Proportion with increased BMI / abdominal waist line; Prevention of pressure ulcers in patients included in the chronic dependent patients care program; Duration of untreated psychosis. The frequency of indicators regarding acute care only (n = 51, 36.4%) and preventive care (n = 44, 31.4%) were similar ( Table 5 ).

https://doi.org/10.1371/journal.pone.0220888.t005

Treatment was the most frequent function of care within outcome indicators (n = 57, 40.7%), e.g Sedation side effects; Number of deaths in seven days between those whose calls were handled by doctors or nurses. Screening and Prevention (n = 25, 17.8%) and Diagnosis (n = 22, 15.7%) followed with similar number of outcome indicators. Examples of Screening and Prevention indicators are: Quality of maternal and child health care: maternal mortality rates; Quality of health promotion: Smoking rate; Preventive care: Low birth weight rate.

Finally, most outcome indicators were associated with the effective domain of health care quality (n = 91, 65%) e.g. Potentially preventable hospitalisation clinical indicator of Chronic Obstructive Pulmonary Disease; Comorbid psychiatric conditions and response to treatment. Also, a common health care quality domain in the listing was Patient-centered (n = 26, 18.5%), e.g. Patients with multiple chronic conditions and medications attended in primary care; Patient Quality of Life; Patient satisfaction with the family physician/specialist coordination of care.

Primary health care (PHC) is where the patient's first contact with the health system occurs and comprises a range of actions which includes many dimensions, domains, and contexts [ 14 ]. Due to these characteristics, it becomes important to evaluate and monitor the quality of primary care [ 78 – 80 ]. It is established that primary care can lead to better health outcomes, lower costs, and greater equity in health [ 81 ] and this can be achieved by using QIs, a set of objective measures with clinical evidence [ 6 , 82 – 83 ] that can represent an acceptable standard of care across a specific patient population [ 84 ].

As the aim of this umbrella review of systematic reviews was to search indexed literature, in order to find a setlist of QI useful for monitoring quality in PHC, our study shows interesting answers to what was proposed, identifying 33 systematic reviews of studies on quality indicators in primary health care and providing a list of selected indicators considered in the included reviews. The study resulted in 727 quality indicators, which were later categorized by context, dimension, type of care, function and domain.

Context of care was classified according to the International Classification of Primary Care (ICPC-2), which is recommended by the World Organization of Family Doctors (WONCA) for codification in this level of care.[ 44 ]. Although practical and useful for primary care, this classification represents a simplification and attempt at uniformization with other classification systems such as the International Classification of Diseases (ICD-11), which is not achieved completely [ 85 ]. Furthermore, since ICPC-2 is a classification based primarily in the location of the symptoms or disease, the authors could not define the context for 100 indicators, since they relate mostly to organizational measures not contemplated in adopted system. “Not defined” was the fourth most common context, representing 13.8% of the total of indicators found.

The majority of the indicators belong to the context category “A–General and Unspecified” (n = 112, 15.4%), which may reflect an attempt at creating indicators applicable to a wide range of procedures and contexts. Circulatory, psychological, respiratory, musculoskeletal and endocrine/metabolic diseases are the next most frequent contexts, indicating also a bigger concern for areas which are more prevalent in primary care (see Table 2 ) .

Most of the indicators found by the authors were related to the dimension of Process (n = 542, 74.5% of total). As defined by Donabedian, this dimension focuses in what is actually done, such as patient’s procedures in seeking care and practitioner’s activities while providing it [ 14 ]. Since QIs represent an opportunity for improvement in areas where quality standards are not met, process indicators may help implementing better procedures and guidelines, resulting in better health care. Outcome dimension was the second most frequent dimension (n = 140, 19.2%); since healthcare outcomes depend on the care provided, these indicators evaluate the result of the course of action of PHC professionals, unlike process indicators which evaluate a single aspect of care.

Type of care

Type of care was classified as acute, chronic or preventive, with “Chronic” being the most frequent. Indicators focused on chronic care are very helpful, since family doctors follow their patients longitudinally for many years, monitoring and managing the chronic diseases they develop throughout their lives [ 79 ]. The management and control of chronic conditions/diseases in the population is one of the main focuses of the activities of primary health care, being also the most studied and evaluated by the QIs, as our study demonstrates. Indicators such as control of prescriptions and monitoring of diseases such as asthma, COPD, hypertension and diabetes, as well as indicators of ambulatory care sensitive conditions that can generate avoidable hospitalizations are part of the list of indicators presented [ 30 ].

Indicators relating to “Treatment” were the most frequent, followed by “Screening and Prevention” and “Follow-up and Continuity”. Once again, the results mirror important aspects of PHC. The consideration of the patient as a whole and the approach of disease in a holistic perspective imply that the healthcare provider must consider indications, potential adverse effects and comorbidities of each patient before elaborating a treatment plan [ 79 ]. Within outcome indicators, most these were focused on treatment, contributing to the evaluation of its complications and preventable hospitalizations, once again alerting providers to re-evaluate their patients and review therapeutic options.

Regarding “Screening and Prevention”, the prevention of disease as well as early diagnosis are the main focus of this level of care [ 1 ]; the development of screening programs for oncological conditions and adequate follow-up for prevention of complications contribute to better health care in this aspect.

“Effective” was the most common domain among the three dimensions of care. Indicators under this domain focus on the capacitation of PHC providers and their articulation with secondary care. Since the effectiveness of a health system depends on the quality of its primary care [ 29 , 86 ], it would be expected that this would be an area of interest.

Other domains such as “Patient-centered” or “Safety” were also commonly evaluated through QIs, demonstrating once again the concern for a holistic approach of PHC.

Limitations

Although there is a significant amount of literature on health quality indicators, some of them are not directly linked to PHC, making it difficult to extrapolate the conclusions of the QI that are applied mainly to the secondary and tertiary levels of attention. Most articles published on QIs in PHC tend to choose very limited and specific areas of health care, without a generic approach to PHC as a whole. The uniqueness and heterogeneity found in these studies show the importance of comprehensive systematic reviews on PHC.

Systematic reviews included in this paper selected primary studies using slightly different methodological assessment and statistical pooling; some of these articles did not discriminate how many primary studies were included in the analysis. The use of different databases in each systematic review and different methods for choosing search terms, calibration and specificity of the search expressions must be considered when interpreting the results.

The authors of this article have searched the primary studies included in each systematic review in order to obtain a list of PHC quality indicators. The lack of a uniform method to collect and present the QIs among the included reviews limited the ability to withdraw complete information from every paper. As an example, most studies were missing information regarding the numerator, denominator and calculation method for each QI.

Conclusions

This is, to the best of our knowledge, the first umbrella review focusing on QIs for primary healthcare in a border scope. We present a final list of indicators ( S1 Appendix supplementary material ) from eligible systematic reviews summarizing the indicators available in the literature, allowing us to understand which areas of primary care are better covered by these measures. The results of our umbrella review are valuable and imply the need for future research and practice regarding quality indicators, as a great opportunity for further studies to test the acceptability, feasibility, reliability, comparison tools and validity of those indicators, while also checking for problems with their implementation to PHC, with adequate information and registration systems. It also provides a ready way for clinicians, managers and health decision makers to gain a clear understanding of the most evidence-based publications related to PHC quality indicators.

Supporting information

S1 file. prisma statement checklist..

https://doi.org/10.1371/journal.pone.0220888.s001

S2 File. PROSPERO protocol register.

https://doi.org/10.1371/journal.pone.0220888.s002

S3 File. Search expression query.

https://doi.org/10.1371/journal.pone.0220888.s003

S4 File. Quality and risk of bias assessment.

https://doi.org/10.1371/journal.pone.0220888.s004

S1 Appendix. Supplementary material–indicators list.

https://doi.org/10.1371/journal.pone.0220888.s005

• 1. World Health Organization. Declaration of Alma-Ata. Adopted at the International Conference on Primary Health Care, Alma-Ata, 6–12 September 1978, USSR.
• View Article
• PubMed/NCBI
• Google Scholar
• 9. Institute of Medicine (US) The National Roundtable on Health Care Quality; Donaldson MS, editor. Measuring the Quality of Health Care: A Statement by The National Roundtable on Health Care Quality. Washington (DC): National Academies Press (US); 1999. Available from: https://www.ncbi.nlm.nih.gov/books/NBK230819/ https://doi.org/10.17226/6418
• 12. Donaldson MS. Measuring the quality of health care. Washington, DC: National Academy Press; 1999. p. 3
• 16. Guide to Inpatient Quality Indicators:Quality of Care in Hospitals–Volume, Mortality, and Utilization. AHRQ Quality Indicators; 2007. Available at https://www.ahrq.gov/downloads/pub/inpatqi/iqi_guide.pdf
• 18. Department of Health. Rewarding quality and outcomes. In: Department of Health. The new GMS contract. London: The Stationery Office, 2003: 17–24.
• 19. NHS Digital Quality and Outcomes Framework, Achievement, prevalence and exceptions data—2017–18 [PAS] https://digital.nhs.uk/data-and-information/publications/statistical/quality-and-outcomes-framework-achievement-prevalence-and-exceptions-data accessed 22 November 2018
• 24. WHO. Library Cataloguing-in-Publication Data Quality of care: a process for making strategic choices in health systems. 2006 accessed 22 November 2018. http://www.who.int/management/quality/assurance/QualityCare_B.Def.pdf
• 25. Agency for Healthcare Research and Quality. Guide to the Prevention Quality Indictors. Rockville, MD: AHRQ; 2017. https://www.qualityindicators.ahrq.gov/Archive/PQI_TechSpec_ICD10_v70.aspxAccessed 22 November 2018.
• 32. Canadian Institute for Health Information. Primary Health Care in Canada: A Chartbook of Selected Indicator Results 2016. Ottawa, ON: CIHI; 2016.
• 39. McMaster Univesity Health Information Research Unit—HIRU ~ Search Strategies for MEDLINE in Ovid Syntax and the PubMed translation. https://hiru.mcmaster.ca/hiru/HIRU_Hedges_MEDLINE_Strategies.aspxacessed in 05th dec 2018.
• 44. WONCA–World Organization of Family Doctors. An introduction to the international classification of primary care. Version 2" at http://www.ph3c.org/PH3C/docs/27/000098/0000054.pdf Accessed on 05 Jul 2019.
• 78. Foundation Canadian Health Services Research Foundation. Contexts and Models in Primary Healthcare and their Impact on Interprofessional Relationships. Ottawa, 2012. Available at https://www.cfhi-fcass.ca/Libraries/Commissioned_Research_Reports/ScottLagendyk-April2012-E.sflb.ashx
• 79. WONCA Europe, 2011. The European Definition of General Practice / Family Medicine available at http://www.woncaeurope.org/sites/default/files/documents/Definition%203rd%20ed%202011%20with%20revised%20wonca%20tree.pdf
• 82. Canadian Institute of Health Information, Enhancing the Primary Health Care Data Collection Infrastructure in Canada Report 2 –Pan-Canadian Primary Health Care Indicator Development Project. 2006
• 83. Manitoba Primary Care Quality Indicators Guide ver. 3.0 –Accessed at https://www.gov.mb.ca/health/primarycare/providers/pin/docs/mpcqig.pdf on Feb 11th 2019

• Open access
• Published: 03 June 2022

Effectiveness of registered nurses on patient outcomes in primary care: a systematic review

• Julia Lukewich 1 ,
• Ruth Martin-Misener 2 ,
• Allison A. Norful 3 ,
• Marie-Eve Poitras 4 ,
• Denise Bryant-Lukosius 5 ,
• Shabnam Asghari 6 ,
• Emily Gard Marshall 7 ,
• Maria Mathews 8 ,
• Michelle Swab 9 ,
• Dana Ryan 1 &
• Joan Tranmer 10  

BMC Health Services Research volume  22 , Article number:  740 ( 2022 ) Cite this article

12k Accesses

17 Citations

12 Altmetric

Metrics details

A Correction to this article was published on 22 June 2022

This article has been updated

Globally, registered nurses (RNs) are increasingly working in primary care interdisciplinary teams. Although existing literature provides some information about the contributions of RNs towards outcomes of care, further evidence on RN workforce contributions, specifically towards patient-level outcomes, is needed. This study synthesized evidence regarding the effectiveness of RNs on patient outcomes in primary care.

A systematic review was conducted in accordance with Joanna Briggs Institute methodology. A comprehensive search of databases (CINAHL, MEDLINE Complete, PsycINFO, Embase) was performed using applicable subject headings and keywords. Additional literature was identified through grey literature searches (ProQuest Dissertations and Theses, MedNar, Google Scholar, websites, reference lists of included articles). Quantitative studies measuring the effectiveness of a RN-led intervention (i.e., any care/activity performed by a primary care RN) that reported related outcomes were included. Articles were screened independently by two researchers and assessed for bias using the Integrated Quality Criteria for Review of Multiple Study Designs tool. A narrative synthesis was undertaken due to the heterogeneity in study designs, RN-led interventions, and outcome measures across included studies.

Forty-six patient outcomes were identified across 23 studies. Outcomes were categorized in accordance with the PaRIS Conceptual Framework (patient-reported experience measures, patient-reported outcome measures, health behaviours) and an additional category added by the research team (biomarkers). Primary care RN-led interventions resulted in improvements within each outcome category, specifically with respect to weight loss, pelvic floor muscle strength and endurance, blood pressure and glycemic control, exercise self-efficacy, social activity, improved diet and physical activity levels, and reduced tobacco use. Patients reported high levels of satisfaction with RN-led care.

Conclusions

This review provides evidence regarding the effectiveness of RNs on patient outcomes in primary care, specifically with respect to satisfaction, enablement, quality of life, self-efficacy, and improvements in health behaviours. Ongoing evaluation that accounts for primary care RNs’ unique scope of practice and emphasizes the patient experience is necessary to optimize the delivery of patient-centered primary care.

Protocol registration ID

PROSPERO: International Prospective Register of Systematic Reviews. 2018. ID=CRD42 018090767 .

Peer Review reports

Primary care is the foundation of a highly functioning health care system and provides comprehensive, patient-centered care that considers the needs and experiences of the individual patient, their families, and the well-being of the broader community [ 1 ]. Primary care providers are the first contact and principal point of continuing care for patients within the health care system, and coordinate other specialist care and services that patients may need [ 1 , 2 ]. The delivery of primary care occurs across varied settings but is most frequently provided in a clinic and, increasingly, by interprofessional teams that may consist of family physicians, registered nurses (RNs), nurse practitioners, pharmacists, and other health professionals. In primary care settings, RNs function as generalists and provide a broad range of patient services across the lifespan, including preventative screening, health education and promotion, chronic disease prevention and management, and acute episodic care [ 3 , 4 , 5 , 6 ]. Specifically, family physicians and RNs represent a key collaborative relationship within these teams, contributing to strengthened primary care delivery and improvements in the comprehensiveness, efficiency, and value of care for patients [ 7 , 8 , 9 ]. Internationally, nurses are increasingly embedded in primary care settings and are recognized as the most prominent non-physician contributor to primary care teams, although the scope and speed of implementation in this area differs across countries [ 10 , 11 ]. Primary care nursing in Australia is the fastest growing employment sector, with 63% of general practices employing a primary care nurse (and 82% of this group representing RNs) [ 12 , 13 ]. The World Health Organization’s report [ 14 ] on the state of the world’s nursing workforce emphasizes the need to strengthen the integration of RNs into primary care, as well as the need for further research to evaluate their impact. Global workforce data are unavailable given the variability in scope of practice and role terminology, and the lack of available information across countries. A recent review of the international literature identified that titles used to refer to RNs in primary care vary across countries [ 15 ]. For instance, titles for this role in Canada are ‘family practice nurse’ and ‘primary care nurse’, whereas in Australia, the United Kingdom, and Netherlands this title is known as ‘general practice nurse’ [ 15 ]. For the purpose of this manuscript, ‘primary care RN’ will be used throughout.

Most research in this area to date has focused on describing the roles and activities of primary care RNs. A systematic review conducted by Norful et al. [ 5 ] synthesized 18 studies from eight countries related to primary care RNs and identified assessment, monitoring, and follow-up of patients with chronic diseases as fundamental roles of the primary care RN. In contrast, there have been a number of reviews conducted on the effectiveness of nurse practitioners in primary care [ 16 , 17 , 18 ]. It is imperative that primary care RNs also begin to demonstrate their contributions to patient care within this setting. Research examining RN effectiveness has primarily been conducted within the acute care and long-term care settings and focused on staffing, role enactment, and work environment. Within these settings, there is substantial evidence demonstrating the positive effects of RN staffing on improving care and reducing adverse outcomes for hospitalized patients [ 24 , 25 ].

Furthermore, select countries including Australia, Canada, New Zealand, and the United Kingdom have developed national standards of practice or competencies to define the scope and depth of practice for primary care RNs [ 4 , 19 , 20 , 21 , 22 , 23 ]. National competencies for primary care RNs were recently published in Canada [ 7 ]. These competencies articulate the unique scope of practice and contributions to patient care for primary care RNs across six overarching domains, namely, (1) Professionalism, (2) Clinical Practice, (3) Communication, (4) Collaboration and Partnership, (5) Quality Assurance, Evaluation and Research, and (6) Leadership.

Theoretical foundation

Determining effectiveness normally requires an examination of an intervention (e.g., primary care nursing) on a particular outcome. Incorporation of the patient perspective offers a more complete understanding of the challenges patients face within our healthcare system, especially those with long-term chronic diseases. Measuring the patient experience, which is a strong predictor of quality and value of care, should be done systematically [ 26 ]. The Organization for Economic Cooperation and Development (OECD) Patient Reported Indicator Surveys (PaRIS) Conceptual Framework was developed through a comprehensive process involving extensive international collaborations and provides a roadmap and survey tools (i.e., patient and provider questionnaires) to focus the evaluation of health care interventions on patient-reported metrics [ 27 ]. This framework provides a fuller evaluation of performance by complimenting other metrics (e.g., system/cost outcomes), while also focusing attention on the needs of the patient. The main domains of the framework include: patient reported experience measures (PREMs), patient reported outcome measures (PROMs), and health behaviours (e.g., physical activity, diet, tobacco use, alcohol use). Within primary care, the PaRIS Framework can serve as a guide for routine collection of these outcomes to facilitate quality improvement and patient-centered care [ 27 ]. A growing body of research in this area has adapted the use of this model to serve as an organizational and methodological framework. For example, multiple studies have used this framework as a method of investigating the suitability and feasibility of questionnaire and survey instruments when addressing patient perspectives [ 28 , 29 ] or in the evaluation of health-related quality of life measures from the patient’s perspective [ 30 ]. A recently published systematic review that explored the opportunities and challenges of routine collection of PREMs and PROMs data for melanoma care within primary care settings found that these measures can address important care gaps and facilitate research and assessment [ 31 ]. Similarly, a study employing qualitative methods found that the use of patient-reported measures by practitioners enhanced patients’ ability to self-manage, communicate, engage, and reflect during consultations [ 32 ]. A recent environmental scan of the PROMs landscape was conducted within Canada and internationally, indicating a lack of standardized programs for routine collection and reporting of patient outcomes. Consequently, the need for enhanced PROMs information has been identified as an area of high priority [ 33 ].

Although existing literature provides some information about the contributions of primary care RNs towards outcomes of care, a systematic review synthesizing the effectiveness of the primary care RN workforce is needed. Prior to beginning the study, the Cochrane Database of Systematic Reviews, the Joanna Briggs Institute (JBI) Library of Systematic Reviews, and the Prospective Register of Systematic Reviews (PROSPERO) were searched and no existing registered protocols or previous systematic reviews on this topic were identified. Evaluating PREMs, PROMs, and health behaviours, as well as other patient-level outcomes, is necessary to accurately demonstrate the contribution of primary care RNs, hold them accountable for their care, and generate evidence to inform decisions and policies that impact their implementation and optimization [ 34 , 35 ]. Therefore, the purpose of this systematic review is to summarize evidence examining primary care RNs’ impact on patient outcomes, including physiologic changes (via biomarkers), PREMs, PROMs, and health behaviours.

A systematic review of effectiveness was conducted using JBI Systematic Review methodology [ 36 ] and findings were reported in accordance with the 2009 (and where possible, the 2021) Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) framework [ 37 , 38 ]. Covidence software was used to manage and organize the literature [ 39 ] and enable a team approach for study and data review. The protocol for this systematic review is registered on PROSPERO (registration ID CRD42018090767). This paper presents a summary of findings from studies that report on patient outcomes, including biomarkers, PREMs, PROMs, and health behaviours. A full description of the methods and findings from studies that measured care delivery and system outcomes are reported in the companion paper “Effectiveness of Registered Nurses on System Outcomes in Primary Care: A Systematic Review” [ 40 ].

Search strategy

The search strategy aimed to include both published and unpublished literature. Following a limited search in CINAHL and Medline that identified optimal search terms, two members of the research team performed a comprehensive search of relevant electronic databases (see Supplementary File 1). Grey literature was identified using ProQuest Dissertations and Theses, MedNar, Google Scholar, the websites of relevant nursing organizations (e.g., International Council of Nurses, Community Health Nurses of Canada), and reference lists of included articles. There were no location or publication date restrictions on search criteria. Studies published in any year up to and including the date of article retrieval (January, 2022) were considered. Ongoing searches for grey literature included studies with publication dates up to January, 2022.

Inclusion criteria

Studies considered for inclusion reported on any quantitative study published in English with outcomes that directly measured, or were related to, an intervention attributable to a primary care RN. Only studies focused on RNs or equivalent (e.g., practice nurse, general nurse) [ 15 ] were included; if the RN designation was unclear or could not be determined based on the region of publication, the study was excluded. Studies that involved primary care RNs who underwent considerable advanced/focused training or those that exclusively examined structural variables were excluded. Full details regarding inclusion criteria are published in the companion paper [ 40 ].

Reviewers included two study authors (DR and JL) and two trained research assistants (AR and OP). All identified titles and abstracts were screened independently by two reviewers for potential study eligibility. Two reviewers independently screened full-text articles for relevance, applying pre-established eligibility criteria. Any disagreements were resolved through discussion, or by a third reviewer.

Risk of bias

The risk of bias and quality of each study was assessed using the Integrated Quality Criteria for Review of Multiple Study Designs  (ICROMS) tool (see scoring matrix located in Supplementary File 2) [ 41 ]. All full-text articles that met eligibility criteria were appraised for quality by two independent reviewers. All studies that met inclusion/exclusion criteria also met the minimum ICROMS score to be included in the review.

Data extraction and synthesis

All eligible full-text studies underwent data extraction using a tool pre-designed and tested by the research team and based on the Cochrane Public Health Group Data Extraction Template [ 42 ]. Data extracted from the articles included: country and year of publication, study aim and design, description of primary care setting, patient sample sizes and demographics, details of study intervention and primary care RN involvement/role, outcome measures used to evaluate these interventions, and study results. To address the broad range of terms and descriptors used across included studies (e.g., traditional care, standard care, basic support, care delivered by anyone other than a primary care RN) and to provide clarity in the presentation of our results, we refer to all control groups as “usual care” or the “comparator group”. Outcomes were grouped in accordance with the OECD PaRIS Conceptual Framework Classification [ 27 ] into one of three categories defined by this model (i.e., PREMS, PROMs, health behaviours), and an additional category added by the research team (i.e., biomarkers) (see Table 1 ). Biomarkers consist of outcomes related to changes in patient health status as measured by clinical assessment (e.g., hemoglobin A1c [HbA1c] values, blood pressure, body weight). PREMs are defined as patient experience indicators related to health care access, autonomy in care, and overall satisfaction with care received, and are often assessed through self-report questionnaires or population-based surveys [ 27 ]. These outcomes can be summarized further based on patient experiences surrounding access (e.g., first point of contact), comprehensiveness of care, self-management support, trust, and overall perceived quality of care. PROMs are described as outcomes relating to a patient’s self-reported physical, mental, and social health status and can be categorized as either generic or condition-specific and applied to a broad patient population [ 27 ]. Outcomes identified on this level can be further categorized into functional status (e.g., disability, physical, mental, social function), symptoms, and health-related quality of life. The remaining outcomes were categorized according to the health behaviors classification, which includes lifestyle behaviors and actions that can contribute to a patient’s overall health status (e.g., physical activity, smoking status, dietary intake) [ 27 ]. Due to the diversity of included designs, interventions, and outcomes across studies, narrative synthesis was used to present study findings.

Figure  1 presents a PRISMA diagram outlining the results of the literature search.

PRISMA Diagram of Literature Search. *This paper reports on studies that measured patient outcomes. Findings from studies that measured care delivery and system outcomes are reported in the companion paper "Effectiveness of registered nurses on system outcomes in primary care: a systematic review" [ 40 ]

Study characteristics

Of the 29 articles included in the final review, 23 reported on patient outcomes (included in the present analysis) [ 40 ]. Table 2  presents a detailed summary of the study characteristics for each of these articles. Studies were published between the years 1996–2021 and conducted in the United Kingdom ( n  = 9), United States ( n  = 6), Australia ( n  = 5), and New Zealand ( n  = 3). Study designs included randomized controlled trials ( n  = 9), observational ( n  = 8) (e.g., survey, secondary data analysis), cohort ( n  = 1), non-controlled ( n  = 2) and controlled before-after ( n  = 1), and two studies with mixed-methods designs that combined a non-controlled before-after with a non-randomized controlled trial ( n  = 1) or with an observational design ( n  = 1). Sample sizes ranged from 81–2850 patients. Quality scores, as assessed by the ICROMS tool, varied between studies. Four studies were scored at the minimum threshold for their study design [ 46 , 56 , 61 , 65 ], six studies scored 1–2 points above threshold [ 44 , 45 , 48 , 49 , 53 , 57 ], and thirteen studies exceeded the minimum cut-off score by 3 or more points [ 43 , 47 , 50 , 51 , 52 , 54 , 55 , 58 , 59 , 60 , 62 , 63 , 64 ].

Overview of RN interventions

The nature of interventions examined in this review differed across studies. The most common interventions were related to chronic disease prevention and management, specifically, case management or targeted chronic disease management care (e.g., diabetes, obesity, hypertension, hypocholesteremia) ( n  = 7) [ 43 , 44 , 45 , 46 , 47 , 48 , 49 ] and primary and secondary preventative care for patients at risk of chronic disease (e.g. prediabetes, coronary heart disease) ( n  = 3) [ 50 , 51 , 52 ]. Other studies examined primary care RN-delivered smoking cessation support ( n  = 4) [ 53 , 54 , 55 , 56 ], back pain education and management [ 57 ], pelvic floor muscle training [ 58 ], consultations aimed at increasing patient physical activity levels [ 59 , 60 ], and a telecare program for patients with diagnosed depression [ 61 ]. Four studies examined the impact of RN care in general (at an organizational-level); three of which focused on consultations with patients in general practice [ 62 , 63 , 64 ] and another which examined the impacts of a nurse-operated telephone consultation/triage service [ 65 ].

In thirteen studies, primary care RNs carried out the intervention independently without the support of other staff/providers [ 45 , 46 , 49 , 51 , 53 , 54 , 55 , 57 , 58 , 59 , 60 , 62 , 63 ], and in 10 studies, they carried out the intervention interdependently, in collaboration with health care providers (e.g., physicians, clinical pharmacy specialists [CPS], dieticians) or members of the research team (e.g., trial nurse facilitator) [ 43 , 44 , 47 , 48 , 50 , 52 , 56 , 61 , 64 , 65 ]. Three of these 10 studies involved evaluating RNs at the general practice-level and therefore are assumed to be evaluating an interdependent role involving support of other health care providers [ 47 , 61 , 64 ]. The presence and type of comparator group also differed across study designs. Specifically, five of the included studies compared a nurse-led intervention to the same intervention led by other health care providers [ 46 , 52 , 54 , 55 , 58 ]. Other studies compared nurse-led interventions with that of ‘usual care’ not associated with nurse involvement ( n  = 4) [ 43 , 50 , 57 , 60 ], or with ‘usual care’ that was associated with reduced or alternative levels of nurse involvement ( n  = 5) [ 44 , 49 , 53 , 56 , 59 ]. The remaining studies examined the effectiveness of a primary care RN-delivered intervention on specific outcomes of care using an observational or before-after design ( n  = 5) [ 48 , 51 , 61 , 62 , 65 ], or did not contain a specific intervention, but rather, examined the impact of varying roles and practice characteristics of the primary care RN in general practice ( n  = 4) [ 45 , 47 , 63 , 64 ].

Overview of outcomes

A total of 46 patient outcomes were identified across included studies (Table 3 ). Physiologic disease control outcomes, which were measured via biomarkers, included quality of care for diabetes (e.g., HbA1c, fasting blood glucose) [ 43 , 44 , 50 , 51 ], obesity (e.g., body mass index [BMI], waist circumference) [ 44 , 47 , 50 , 51 , 59 , 60 ], pelvic floor strength and endurance [ 58 ], hypercholesterolemia (e.g., total cholesterol) [ 49 ], and hypertension (e.g., blood pressure) [ 43 , 44 , 48 , 50 , 51 ]. Patient experience outcomes identified under the PREMs category included patient satisfaction with access to care (RN versus physician as first point of contact) [ 64 ], quality of self-management support (e.g., smoking cessation counseling, chronic disease services) [ 56 , 62 ], comfort/trust with primary care RN roles [ 45 ], and overall satisfaction or perceived quality of care with provider consultations, treatment, or advice/support received [ 45 , 51 , 55 , 57 , 63 , 65 ]. Patient reported outcomes identified within the PROMs category consisted of physical and social functional status [ 43 , 57 ], level of disability (e.g., activity levels, bed rest, work loss) [ 57 , 61 ], changes in self-reported anxiety, depression, or pain [ 59 , 60 , 61 ], adverse health events (e.g., falls, fractures, severe hypoglycemia) [ 43 , 59 , 60 ], and health-related qualify of life (e.g., physical activity, social activity) [ 43 , 46 , 51 , 52 , 60 ]. Lastly, outcomes grouped under the health behaviors classification included reduction and/or cessation of tobacco use [ 51 , 53 , 54 , 55 , 56 ], changes to physical activity (e.g., level of aerobic exercise, daily step count) [ 51 , 57 , 59 , 60 ], and improvements in dietary intake [ 49 ].

Physiologic disease control via biomarkers

Ten studies measured clinical patient outcomes when comparing interventions that involved primary care RNs to that of usual care or an intervention delivered by a comparator group. Clinical biomarkers included those for diabetes (HbA1c, fasting blood glucose), obesity (BMI, total fat mass), hypertension (blood pressure), and cardiovascular risk (total cholesterol). Of the ten studies, four examined diabetic control. After one year, Aubert et al [ 43 ]. reported significant differences in HbA1c values; patients in the primary care RN case management group had a larger mean reduction (-1.7 percentage points) over 12 months in comparison to usual care (-0.6 percentage points) (difference -1.1, 95% CI: -1.62 to 0.58; p  < 0.001). Additionally, patients in the intervention group had a greater decrease in fasting blood glucose than the usual care group (-48.3 mg/dL versus -14.5 mg/dL; difference -33.8, 95% CI: -56.12 to 11.48;  p  = 0.003). Bellary et al [ 44 ]. found a small but non-significant reduction in HbA1c among their patient sample after two years. One additional study that conducted a retrospective data analysis of clinical outcome data from patients attending an independently RN-led primary care clinic, did not detect significant changes in HbA1c between initial intake at baseline and follow-up visits at various intervals (reported as 3 months to “several years” depending on the individual) [ 51 ].

Seven studies examined obesity-related outcomes such as BMI, weight, and total fat mass. In their adjusted regression models, Karnon et al [ 47 ]. reported that high level involvement of primary care RNs in the provision of obesity-related clinical activities (in comparison to low level involvement) yielded significantly larger mean reductions in BMI (mean difference -1.10, 95% CI: -0.45 to -1.76; p  = 0.001) after one year, however, there were no significant improvements in terms of the proportion of patients losing weight (mean difference 0.09, 95% CI: -0.07 to 0.25;  p  = 0.259). Coppell et al [ 50 ]. found a significant weight reduction (-1.3 kg) in the primary care RN-led prediabetes intervention arm compared to usual care (gained 0.8 kg) (2.2 kg difference; p  < 0.001). Mean BMI and waist circumference also decreased in the intervention arm compared to an increase in the control group, however, these differences were not significant. Likewise, a third study reported that fat mass was slightly reduced at 12 months, but differences between the intervention and control groups were equivalent when the primary care RN group was compared to both postal intervention ( p  = 0.54) and usual care ( p  = 0.30) [ 60 ]. There were no significant reductions in BMI or waist circumference in the remaining four studies [ 43 , 44 , 51 , 59 ].

Five studies investigated the impact of enhanced nurse involvement in primary care delivery on blood pressure. Bellary et al [ 44 ]. reported significant differences between groups in diastolic blood pressure (-1.91, 95% CI: -2.88 to -0.94 mm Hg; p  = 0.0001) and mean arterial pressure (1.36, 95% CI: -2.49 to -0.23 mm Hg; p  = 0.018), favoring the intervention (additional time spent with a primary care RN). In a second study conducted by O’Neill et al. in which the RN independently assessed blood pressure and collaborated with either a CPS or physician in hypertension case management, [ 48 ] there was a greater decrease in systolic blood pressure in patients who received care from CPS and primary care RN teams (-14 mm Hg) compared to patients receiving care from physician-directed primary care RNs (-10 mm Hg) ( p  = 0.04), however, there were no significant changes in diastolic blood pressure between groups. The remaining three studies found no significant changes in blood pressure when comparing a primary care RN-led intervention to that of usual care [ 43 , 50 ] or from initial baseline to follow-up [ 51 ].

Total cholesterol was measured in four studies. Pine et al  [ 49 ]. reported that the mean total cholesterol level decreased by 0.29 mmol/L (11.2 mg/dL) (4.3%) from the initial physician visit to the first primary care RN visit. Following five counseling sessions by a primary care RN, the mean total cholesterol levels of all patients decreased (-0.14 mmol/L; p  = 0.4). However, during the follow-up comparison study, there were no significant differences in total cholesterol improvement between the nurse-counseling intervention group and the comparison patients, and total cholesterol levels in both groups improved significantly ( p  = 0.002). The remaining three studies reported equivalent results in regards to total cholesterol reduction [ 43 , 44 , 50 ].

Patient experience outcomes via PREMS

Nine articles reported on patient experience outcomes via PREMs: overall perceived quality of care [ 45 , 51 , 55 , 63 , 65 ], self-management support [ 56 , 62 ], access (first point of primary care contact) [ 64 ], comprehensiveness [ 57 ], and trust [ 45 ].

In regard to overall perceived quality of care, Halcomb and colleagues [ 45 ] found that Australian patients were very satisfied and comfortable with chronic disease care delivered by a primary care RN. This was particularly true for patients with diabetes who reported being almost three times more comfortable (38% versus 14%, p  = 0.016) with their encounter than patients who consulted for other chronic health conditions. A similar study in New Zealand also revealed high satisfaction with primary care RN-delivered services overall, with increased satisfaction associated with an increased number of visits (i.e., those who had more than four previous visits to the primary care RN) after controlling for demographic factors [ 63 ]. Longer consultation time with a primary care RN resulted in higher patient satisfaction (OR = 2.50, 95% CI: 1.43 to 4.35; p  < 0.01) and patient enablement (OR = 2.55, 95% CI: 1.45 to 4.50; p < 0.01) than shorter consultation time [ 62 ]. Moreover, patients who attended practices where primary care RNs worked with broad scopes of practice and high levels of autonomy were more satisfied (OR = 1.76, 95% CI: 1.09 to 2.82; p  = 0.04) and more enabled (OR = 2.56, 95% CI: 1.40 to 4.68; p  < 0.01) than patients who attended practices where nurses worked with more limited scopes of practice and lower levels of autonomy [ 62 ].

Patients also reported improved health, better understanding of disease diagnosis, medication, and treatment plan, and more motivation for self-management as a result of primary care RN-led lifestyle clinics focused on diabetes, smoking cessation, women’s health, cardiovascular risk, respiratory/asthma, and diet/nutrition [ 51 ]. Furthermore, patients reported positive experiences with primary care RN-led telephone consultations for acute illness [ 65 ], back pain education [ 57 ], and smoking cessation support [ 55 , 56 ]. For instance, Cherkin et al. [ 57 ] reported higher satisfaction ( p  < 0.01) and higher perceived knowledge ( p  < 0.001) for patients who received a primary care RN-led educational intervention for back pain than those in the usual care group. Nearly all patients (98%, n  = 385) in an Australian study [ 56 ] that examined smoking cessation behavioral support from a primary care RN rated the support provided as helpful (19%) or very helpful (79%) and indicated that they may have been more successful with smoking cessation if they had been able to have more sessions with the RN. With respect to access to care, a study by Caldow et al [ 64 ]. found that patients expressed satisfaction and preference with primary care RN versus physician consultations for minor illness as first point of contact if this resulted in a reduced waiting time, suggesting that patients would be accepting of an expanded nursing role in primary care.

Patient reported outcomes via PROMs

Patient reported outcome measures via PROMs were examined across eight studies and included health-related quality of life [ 43 , 46 , 51 , 52 , 60 ], symptoms [ 59 , 60 , 61 ], self-efficacy [ 60 ], and functional status [ 57 ].

Health-related quality of life, as measured through patient self-report, was assessed in five studies [ 43 , 46 , 51 , 52 , 60 ]. In a 12-month randomized controlled trial conducted by Aubert et al., [ 43 ] a primary care RN-led case management model of adult diabetes care was compared with that of usual care in a primary care setting. Health-related quality of life was assessed by a validated questionnaire developed by the Centers for Disease Control and Prevention for the Behavioral Risk Factor Surveillance System to assess patient perception of health status across four domains. The results demonstrated an improved perception of health status in both groups, with patients in the intervention group more than twice as likely to report improvement in health status score (mean change = 0.47) as those in the usual care group (mean change = 0.20) ( difference= 0.27; 95% CI: -0.03 to 0.57; p  = 0.02). In contrast, the other four studies [ 46 , 51 , 52 , 60 ] examining health-related quality of life did not report significant differences in regards to these outcomes, including two cluster randomized controlled trials [ 46 ,  52 , 60 ]. One study assessed the effectiveness of three different methods of secondary prevention care of coronary heart disease (recall to a primary care RN; recall to a physician; audit and feedback) [ 52 ], while the other compared the efficacy of a primary care RN-supported physical activity intervention to that of usual care [ 60 ]. Both studies reported equivalent scores between groups on all dimensions of patient self-reported quality of life measurements. Lastly, an observational study by Marshall et al [ 51 ]. used patient satisfaction surveys to assess perceptions of Nurse-Led Healthy Lifestyle Clinics (NLHLC) in New Zealand. Using scores from the Dartmouth Primary Care Cooperative (COOP) Information charts [ 66 ], it was noted that there were no statistically significant differences in the COOP dimensions related to self-perceived quality of life from first clinic visit to last clinic visit. However, significant improvements were noted in relation to the COOP variables related to patient-perceived social activity (mean difference = -0.20; p  = 0.049), change in health (mean difference = -0.42; p  = 0.001), and overall health (mean difference = -0.21; p  = 0.025).

Patient self-reported symptoms were measured in three studies [ 59 , 60 , 61 ] and included outcomes related to both mental and physical health (anxiety, depression, pain), as well as the occurrence of adverse health events (injuries, fractures, cardiovascular events, deaths, and deterioration of any pre-existing health problems). In an observational study of a nurse telecare intervention for adults with depression in the United States, Pearson et al [ 61 ]. found a significant improvement in mean scores on the SF-12 Mental Functioning Scale between baseline (mean = 29.9) and 6-months post-intervention (mean = 48.2) ( p  < 0.0001). During the same time interval, significant differences were noted on the Hamilton Depression Rating scale (14.6 to 6.5; p  < 0.001), as well as the mean scores on the Work Limitations Questionnaire (70.4 to 87.2; p  < 0.001), which both represent an improvement in functioning. Paired t-test results for the difference in mean scores on all three instruments were statistically significant ( p  = 0.0001) and the majority of patients (63%) experienced at least a 50% reduction in the Hamilton Depression Rating score at 6-months. The remaining studies to examine self-reported symptoms as an outcome were two randomized controlled studies that measured the effects of a primary care RN-delivered intervention on patient physical activity [ 59 , 60 ]. Both studies assessed changes to patient self-reported levels of depression, anxiety, and pain and incidents of adverse health events. The results of both studies reported no statistically significant between-group differences in mean scores of either symptom at 3- or 12-months post-intervention. Additionally, while total number of adverse events did not differ between groups for either study, Harris et al [ 60 ]. found a significant reduction in cardiovascular events among the intervention group over the 12 month period ( p  = 0.04).

Patient self-efficacy was examined in a three-arm cluster randomized controlled study conducted by Harris et al [ 60 ]., in which patient-reported levels of exercise self-efficacy were examined at 3- and 12-months following a physical activity intervention. Exercise self-efficacy in this study was characterized by a patient’s willingness to set goals, create action plans, engage in self-monitoring, and seek out social support, and are directly related to long-term physical activity adherence. Findings indicated that exercise self-efficacy was significantly increased in both intervention groups at 3-months for postal group (pedometer delivered by mail) versus control (Effect Size [ES] = 1.1, 95% CI: 0.2 to 2.0; p  = 0.01), primary care RN group versus control (ES = 2.3, 95% CI: 1.4 to 3.2; p  < 0.001), and primary care RN group versus postal group (ES = 1.2, 95% CI: 0.3 to 2.1; p  = 0.01). For primary care RN group versus control group, the difference remained significant at the 12-month follow-up (ES = 1.2, 95% CI: 0.3 to 2.2; p  = 0.01), but not for the postal group versus control ( p = 0.2) or the primary care RN group versus postal group ( p  = 0.22).

The sole study to evaluate functional status was a randomized controlled trial conducted by Cherkin et al [ 57 ]. comparing usual care, usual care plus an educational booklet, and usual care plus an educational session with a primary care RN and an educational booklet to improve outcomes of low back pain in primary care. None of the interventions had a statistically significant effect on functional status, including days of limited activity, bed rest, or work loss resulting from back pain one week after the intervention or at any subsequent follow-up.

Health behaviors

Of all the studies included ( n  = 23), nine considered health behavior outcomes. Among the studies examining the impact of a primary care RN-led intervention on health behaviors, it was found that tobacco use was the most documented health behavior ( n  = 5) [ 51 , 53 , 54 , 55 , 56 ]. Tobacco use was examined by looking at both abstinence from smoking, as well as daily reductions or changes to smoking behaviour, and this was measured at multiple follow-up periods throughout the duration of the intervention. All five studies demonstrated positive changes in smoking-related health behaviors following either an independent [ 51 , 53 , 54 , 55 ] or interdependent [ 56 ] primary care RN intervention. For example, Byers et al [ 54 ]. compared a primary care RN-led intervention with a physician-led intervention to support smoking cessation. The results show that support provided by the primary care RN was equivalent to that provided by the comparator group (29.1% versus 18.2% quit rate, respectively; p  = 0.077) in supporting the patient to quit smoking. Marshall et al [ 51 ]. looked at primary care RN-led healthy habits lifestyle clinics for patients with or at risk of chronic disease within targeted populations with known health inequalities. Following the intervention, 94% of patients reported having a better understanding of their diagnosis, medication and treatement plan, and an increase in motivation to self-manage their health needs. Other studies have examined the impact of nursing interventions on patient-reported levels of physical activity [ 57 , 59 , 60 ]. Only one study considered adherence to healthy eating as a health outcome following a RN-led intervention in primary care [ 49 ]. In this study, Pine and colleagues evaluated the effect of a nursing intervention to support cholesterol lowering for patients diagnosed with hypercholesterolemia. To do this, primary care RNs provided a total of five counseling visits focused on nutritional education and physical activity (1-month after referral, and at 3-, 5-, 7-, and 12-months) to 82 patients with total cholesterol higher than 6.21 mmol/l. Intervention patients were already following a diet consistent with the program at baseline, however, the mean score for Section 1 of the Eating Pattern Assessment Tool (questions related to foods with serum cholesterol-raising potential) improved from 23.4 at the first visit to 20.4 at the final visit ( p  < 0.001).

This systematic review presents a comprehensive synthesis of literature examining the impact of primary care RNs on patient outcomes. The findings suggest that outcomes resulting from care provided by primary care RNs are comparable and complementary to care provided by other primary care providers, specifically with respect to chronic disease prevention and management, smoking cessation, and wellness counseling. This review supports that primary care RNs deliver appropriate and high-quality patient care. There was a high level of patient satisfaction reported regarding experiences with RN-led care. Patients appear to be comfortable with RNs providing primary care services and taking on expanded roles in primary care. This is consistent with findings from other studies that have examined patient satisfaction and comfort with RN roles in primary care practices across multiple countries [ 67 , 68 , 69 , 70 ]. Our findings are aligned with existing evidence that has linked patient experiences of care to the level of autonomy and scope of practice of the RN in the clinical setting. A recently published study from Canada evaluated patient experiences in primary care organizations and determined that patient-reported experience was significantly enhanced in clinics in which RNs systematically followed patients and used their scope of practice to their full potential [ 71 ].

Patient experience is a strong indicator of patient-perceived quality of care and fundamental to achieving desired patient outcomes for a range of physical and mental health domains [ 72 , 73 , 74 , 75 , 76 ]. The recently developed PaRIS Framework served as a valuable tool for organizing patient-reported outcomes, however, it did not capture all patient outcomes identified within the studies included in this review. For example, clinical biomarkers, such as HbA1c and fasting blood glucose (used as a measure of diabetes care quality), were not considered in the PaRIS Framework and therefore added by the study authors during the analysis phase as an additional patient outcome category. Similarly, studies evaluating RN interventions in primary care did not consider many components of the PaRIS Framework, such as delivery system design (e.g., clinic remuneration, remote consultations), individual and sociodemographic factors (e.g., patient demographic characteristics), and health and health care capabilities, and only measured select components from the patient-reported experiences of care, health behaviours, and patient-reported outcome domains. For instance, many articles did not provide information regarding RN characteristics, such as level of education, years of experience, or specific roles/tasks that they performed in-clinic prior to the intervention that might have impacted outcomes observed. In addition, conceptual definitions of outcomes within included studies may vary and not align with meanings as defined within the PaRIS Framework. A taxonomy for RN outcomes may be useful and should be considered in future revisions and applications of the OECD PaRIS Framework.

The included studies evaluated a variety of primary care RN interventions but did not capture all roles that encompass their broad scope of practice. Commonly offered services by primary care RNs that have yet to be comprehensively evaluated include prenatal and well-baby care, therapeutic interventions (e.g., wound care, treatment of infections), preventative care (e.g., immunizations, health promotion and education) and care coordination (e.g., nursing surveillance, system navigation). Although self-management supports (e.g., smoking cessation, physical activity, diabetes, nutrition, pain management, healthy lifestyle promotion, chronic disease prevention) were examined in a few studies [ 43 , 44 , 49 , 50 , 51 , 53 , 54 , 55 , 56 , 57 , 59 , 60 ], further evaluation of self-management and behaviour support interventions (included within the PaRIS Framework) offered by RNs in primary care is needed. Moreover, recently developed competencies identify that, in addition to clinical practice activities centered around patient care provision, primary care RNs engage in a wide range of non-clinical roles, such as leadership, research, and interprofessional collaboration. These non-clinical domains of practice for primary care RNs require further understanding and evaluation.

There is a general apprehension among some medical practitioners that if RNs assume more responsibilities or enhanced roles within primary care settings, high-quality care and patient safety will be compromised [ 77 , 78 ]. Findings from this study show that patient care is equivalent to that of “usual care” and in many cases, produced better patient outcomes when the intervention was provided by a primary care RN. This aligns with literature in the acute care and long-term care settings [ 24 , 25 ]. The findings from our review call into question concerns that RN-provided care increases risk or reduces quality of care and equally, lends support towards the efficacy of primary care RN care provision on improvements to patient outcomes. Additionally, primary care RNs are required to practice within their legislated and regulated scope of practice, regardless of the practice setting or types of clinical roles performed [ 79 , 80 ].

Generally, there are methodological challenges associated with examining the contributions of a specific health care provider within the context of a team [ 81 , 82 , 83 ]. As the focus of practice and research moves towards interdisciplinary teams, it is increasingly difficult to isolate and evaluate the impact of primary care RN interventions. In addition, the roles of primary care RNs and team compositions vary across practice settings. Although this review exclusively examined studies in which an intervention was delivered by a RN, many studies were excluded because of unclear terminology surrounding the nursing designation/role (i.e., unable to discern whether nurse in study was a RN or equivalent). Furthermore, the review included only nine randomized controlled trials, which provide the strongest level of evidence, that specifically compared RN-led interventions to care delivered by other health care professionals and/or usual care [ 43 , 52 , 53 , 57 , 58 , 59 , 60 , 84 , 85 ]. Comparator groups in studies varied considerably, impacting the ability to make comparisons across studies and limiting the generalizability of findings from this study. Despite these challenges, this review provides preliminary evidence on patient outcomes used to evaluate a variety of different RN interventions in a multidimensional health care environment. The findings from this study, coupled with an existing framework (e.g., OECD PaRIS Framework) serve as a tool to map roles and activities to outcomes and guide future evaluation of primary care RNs. Overall, as the presence of RNs in primary care increases globally, further evaluation research implementing control/comparison groups into study design and controlling for confounding factors (e.g., nurse characteristics) is needed to strengthen the evidence related to the effectiveness of RNs in primary care.

Strengths and limitations

This review provides preliminary evidence regarding the effectiveness of RNs on patient outcomes in primary care. Traditional means of measuring the effectiveness of care provision in the healthcare sector have focused mainly on the use of clinical data. A strength of this systematic review is its patient-oriented approach that assesses health outcomes from the patient perspective [ 27 , 86 ]. Additional strengths of this systematic review include the application of a comprehensive search strategy and use of the PRISMA checklist in the screening process. However, despite utilizing a comprehensive search strategy, it is possible that not all relevant studies were retrieved and included in this review. Furthermore, although we conducted an appraisal using an established quality assessment tool (i.e., ICROMS), this tool presented certain challenges. For instance, although a strength of this tool is that it offered criteria to assist with the process of assigning quality scores, there is a degree of subjectivity involved in the appraisal process. In addition, the minimum cut-off scores varied across study designs and therefore, made comparisons of the quality between different types of studies difficult. Similarly, the score itself is difficult to interpret without an understanding of the tool and design matrix (limitations of articles are summarized in Supplementary Table 3). The ICROMS tool was also not designed to specifically appraise mixed methods or observational designs. The lack of consistent and available data regarding terminology used to describe RNs, or equivalent nursing titles, across countries limited the ability to include studies published in certain regions. Only studies published in the English language were included, which may limit generalizability to certain countries. High-quality research employing robust study designs (e.g., randomized controlled trials) need to be conducted to further understand the impact of RNs on patient outcomes in primary care.

Primary care RNs can provide patient care that is comparable and complementary to that of other primary care providers, specifically with respect to patient satisfaction, enablement, self-reported quality of life, self-efficacy, and improvements in health behaviours. This review provides preliminary evidence regarding the effectiveness of RNs on patient outcomes in primary care. Findings are applicable to researchers and other stakeholders engaged in primary care reform and can inform further integration and optimization of this role, as well as contribute to future research.

Availability of data and materials

All data generated or analysed during this study are included in this published article.

Change history

22 june 2022.

A Correction to this paper has been published: https://doi.org/10.1186/s12913-022-08204-x

Abbreviations

Registered Nurses

Patient Reported Indicator Survey

Patient-reported experience measures

Patient-reported outcome measures

Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

Prospective Register of Systematic Reviews

Integrated Quality Criteria for Review of Multiple Study Designs

Hemoglobin A1c

Body mass index

Clinical Pharmacy Specialist

Primary Care Cooperative

Nurse-Led Healthy Lifestyle Clinic

Organization for Economic Cooperation and Development

Effect Size

National Academies of Sciences, Engineering, and Medicine. Implementing high-quality primary care: rebuilding the foundation of health care. The National Academies Press. 2021. https://www.nap.edu/read/25983 . Accessed 08 Nov 2021.

Muldoon LK, Hogg WE, Levitt M. Primary care (PC) and primary health care (PHC): what’s the difference? Can J Pub Health. 2006;87(5):409–11.

Article   Google Scholar  

Halcomb E, Stephens M, Bryce J, Foley E, Ashley C. Nursing competency standards in primary health care: an integrative review. J Clin Nurs. 2016;25:1193–205.

Lukewich J, Allard M, Ashley L, Aubrey-Bassler K, Bryant-Lukosius D, Klassen T, et al. National competencies for registered nurses in primary care: a Delphi study. West J Nurs Res. 2020;42(12):1078–87. https://doi.org/10.1177/0193945920935590 .

Article   PubMed   PubMed Central   Google Scholar  

Norful A, Martsolf G, de Jacq K, Poghosyan L. Utilization of registered nurses in primary care teams: a systematic review. Int J Nurs Stud. 2017;74:15–23. https://doi.org/10.1016/j.ijnurstu.2017.05.013 .

Poitras ME, Chouinard MC, Gallagher F, Fortin M. Nursing activities for patients with chronic disease in primary care settings: a practice analysis. Nurs Res. 2018;67(1):35–42. https://doi.org/10.1097/NNR.0000000000000253 .

Article   PubMed   Google Scholar  

College of Family Physicians of Canada. Patient-centred primary care in Canada: bring it on home. CFPC. 2009. https://www.cfpc.ca/CFPC/media/Resources/Health-Care-Delivery/Bring-it-on-Home-FINAL-ENGLISH.pdf . Accessed 09 Nov 2021.

Schottenfeld L, Petersen D, Peikes D, et al. Creating patient-centered team-based primary care. Agency for Healthcare Research and Quality. 2016. https://pcmh.ahrq.gov/sites/default/files/attachments/creating-patient-centered-team-based-primary-care-white-paper.pdf . Accessed 08 Nov 2021.

Smolowitz J, Speakman E, Wojnar D, Whelan EM, Ulrich S, Hayes C, et al. Role of the registered nurse in primary health care: meeting health care needs in the 21st century. Nurs Outlook. 2015;63(2):130–6. https://doi.org/10.1016/j.outlook.2014.08.004 .

Organization for Economic Cooperation and Development (OECD). Health at a glance 2019: OECD Indicators. OECD Publishing. 2019. https://www.oecd-ilibrary.org/docserver/4dd50c09-en.pdf?expires=1631157370&id=id&accname=guest&checksum=AA4126CD8BEAEF8E2B96A614027E7B0D . Accessed 08 Nov 2021.

Freund T, Everett C, Griffiths P, Hudon C, Naccarella L, Laurant M. Skill mix, roles and remuneration in the primary care workforce: who are the healthcare professionals in the primary care teams across the world? Int J Nurs Stud. 2015;52:727–43. https://doi.org/10.1016/j.ijnurstu.2014.11.014 .

Australian Institute of Health and Welfare. A profile of primary health care nurses. Australian Government. 2020. https://www.aihw.gov.au/reports/primary-health-care/a-profile-of-primary-care-nurses/contents/primary-health-care-nurses . Accessed 08 Nov 2021.

Australian Primary Health Care Association . General practice nursing. APNA. 2021 https://www.apna.asn.au/profession/what-is-primary-health-care-nursing/general-practice-nursing . Accessed 08 Nov 2021.

World Health Organization. State of the world's nursing: investing in education, jobs and leadership. WHO. 2020. https://www.who.int/publications/i/item/9789240003279 . Accessed 07 Nov 2021.

Barrett C, Ryan D, Poitras ME, Norful AA, Martin-Misener R, Tranmer J, et al. Job titles and education requirements of registered nurses in primary care: an international document analysis. Int J Nurs Stud Adv. 2021:e1–35. doi: https://doi.org/10.1016/j.ijnsa.2021.100044 .

Canadian Nurses Association. Effectiveness of registered nurses and nurse practitioners in supporting chronic disease self-management. CNA. 2012.

Horrocks S, Anderson E, Salisbury C. Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors. BMJ. 2002;324(7341):819–23.

Martin-Misener R, Harbman P, Donald F, Reid K, Kilpatrick K, Carter N, et al. The cost-effectiveness of nurse practitioners in primary and specialised ambulatory care: a systematic review. BMJ Open. 2015;5(6): e007167. https://doi.org/10.1136/bmjopen-2014-007167 .

Australian Nursing and Midwifery Federation. National practice standards for nurses in general practice. Australian Nursing and Midwifery Federation- Federal Office. 2014. http://www.anmf.org.au/documents/National_Practice_Standards_for_Nurses_in_General_Practice.pdf . Accessed 08 Nov 2021.

Canadian Family Practice Nurses Association. National competencies for registered nurses in primary care. CFPNA. 2019. https://www.cfpna.ca/copy-of-resources-1 . Accessed 08 Nov 2021.

Mid-Central District Health Board, New Zealand Nurses Organisation, The New Zealand College of Primary Health Care Nurses. Aotearoa New Zealand primary health care nursing standards of practice. New Zealand Nurses Organization. 2019. https://www.nzno.org.nz/Portals/0/publications/Primary%20Health%20Care%20Nursing%20Standards%20of%20Practice%202019.pdf?ver=XYUZI2v-cpVH28Oy1rhfdw%3d%3d . Accessed 08 Nov 2021.

The Queen’s Nursing Institute. The QNI standards of education and practice for nurses new to general practice nursing. QNI. 2020. https://www.qni.org.uk/wp-content/uploads/2020/05/Standards-of-Education-and-Practice-for-Nurses-New-to-General-Practice-Nursing-1.pdf . Accessed 08 Nov 2021.

The Queen’s Nursing Institute, The Queen’s Nursing Institute Scotland. The QNI/QNIS voluntary standards for general practice nursing education and practice. QNI & QNIS. 2017. https://www.qni.org.uk/wp-content/uploads/2016/09/GPN-Voluntary-Standards-for-Web.pdf . Accessed 09 Nov 2021.

Kane RL, Shamliyan TA, Mueller C, Duval S, Wilt TJ. The association of registered nurse staffing levels and patient outcomes: systematic review and meta-analysis. Med Care. 2007;45(12):1195–204.

Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse-staffing levels and the quality of care in hospitals. N Engl J Med. 2002;346(22):1715–22. https://doi.org/10.1056/NEJMsa012247 .

Berkowitz B. The patient experience and patient satisfaction: measurement of a complex dynamic. OJIN. 2016;21(1): e1. https://doi.org/10.3912/ojin.vol21no01man01 .

Organization for Economic Cooperation and Development. Measuring what matters: the patient-reported indicator surveys. OECD Publishing. 2019. https://ec.europa.eu/health/sites/default/files/systems_performance_assessment/docs/pagoda_en.pdf . Accessed 08 Nov 2021.

Gerrard G, Jones R, Hierons RJ. How did we do? An investigation into the suitability of patient questionnaires (PREMs and PROMs) in three primary care oral surgery practices. Br Dent J. 2017;223(1):27–32.

Article   CAS   Google Scholar  

Depla AL, Ernst-Smelt HE, Poels M, Crombag NM, Franx A, Bekker MN. A feasibility study of implementing a patient-centered outcome set for pregnancy and childbirth. Health Sci Rep. 2020;3(3): e168. https://doi.org/10.1002/hsr2.168 .

Aiyegbusi OL, Kyte D, Cockwell P, Anderson N, Calvert M. A patient-centered approach to measuring quality in kidney care: patient-reported outcome measures and patient-reported experience measures. Curr Opin Nephrol Hypertens. 2017;26(6):442–9. https://doi.org/10.1097/MNH.0000000000000357 .

Blood Z, Tran A, Caleo L, Saw R, Dieng M, Shackleton M, et al. Implementation of patient-reported outcome measures and patient-reported experience measures in melanoma clinical quality registries: a systematic review. BMJ Open. 2021;11(2): e040751. https://doi.org/10.1136/bmjopen-2020-040751 .

Wheat H, Horrell J, Valderas JM, Close J, Fosh B, Lloyd H. Can practitioners use patient reported measures to enhance person centred coordinated care in practice? A qualitative study. Health Qual Life Outcomes. 2018;16(1): e223. https://doi.org/10.1186/s12955-018-1045-1 .

Canadian Institutes for Health Information. PROMs background document. CIHI. 2015. https://www.cihi.ca/sites/default/files/proms_background_may21_en-web_0.pdf . Accessed 09 Nov 2021.

Dubois CA, D’Amour D, Pomey MP, Girard F, Brault I. Conceptualizing performance of nursing care as a prerequisite for better measurement: a systematic and interpretive review. BMC Nurs. 2013;12:7. https://doi.org/10.1186/1472-6955-12-7 .

Sidani S, Irvine D. A conceptual framework for evaluating the nurse practitioner role in acute care settings. J Adv Nurs. 1999;30(1):58–66. https://doi.org/10.1046/j.1365-2648.1999.01049.x .

Article   CAS   PubMed   Google Scholar  

Aromataris E, Munn Z, eds . Joanna Briggs Institute reviewer's manual. The Joanna Briggs Institute. 2017. https://reviewersmanual.joannabriggs.org/ . Accessed 09 Nov 2021.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1136/bmj.b2535 .

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71. https://doi.org/10.1136/bmj.n71 .

Covidence. Covidence systematic review software [computer program]. Veritas Health Innovation. n.d. www.covidence.org

Lukewich J, Ryan D, Asghari S, Marshall EG, Mathews M, Swab M, et al. Effectiveness of registered nurses on system outcomes in primary care: a systematic review. BMC Health Services. 2022;22:440. https://doi.org/10.1186/s12913-022-07662-7

Zingg W, Castro-Sanchez E, Secci FV, Edwards R, Drumright LN, Sevdalis N, et al. Innovative tools for quality assessment: integrated quality criteria for review of multiple study designs (ICROMS). Pub Health. 2016;133:19–37. https://doi.org/10.1016/j.puhe.2015.10.012 .

Cochrane Public Health Group. Data extraction and assessment template. Cochrane. 2011.

Aubert RE, Herman WH, Waters J, Moore W, Sutton D, Peterson BL, et al. Nurse case management to improve glycemic control in diabetic patients in a health maintenance organization. A randomized, controlled trial. Ann Intern Med. 1998;129(8):605–12.

Bellary S, O’Hare JP, Raymond NT, Gumber A, Mughal S, Szczepura A, et al. Enhanced diabetes care to patients of south Asian ethnic origin (the United Kingdom Asian Diabetes Study): a cluster randomised controlled trial. Lancet. 2008;371(9626):1769–76. https://doi.org/10.1016/S0140-6736(08)60764-3 .

Halcomb EJ, Salamonson Y, Cook A. Satisfaction and comfort with nursing in Australian general practice. Collegian. 2015;22(2):199–205.

Iles RA, Eley DS, Hegney DG, Patterson E, Young J, Del Mar C, et al. Revenue effects of practice nurse-led care for chronic diseases. Aust Health Rev. 2014;38(4):363–9. https://doi.org/10.1071/AH13171 .

Karnon J, Ali Afzali HH, Gray J, Holton C, Banham D, Beilby J. A risk adjusted cost-effectiveness analysis of alternative models of nurse involvement in obesity management in primary care. Obesity. 2013;21(3):472–9. https://doi.org/10.1002/oby.20100 .

O’Neill JL, Cunningham TL, Wiitala WL, Bartley EP. Collaborative hypertension case management by registered nurses and clinical pharmacy specialists within the Patient Aligned Care Teams (PACT) model. J Gen Intern Med. 2014;29(2):S675–81. https://doi.org/10.1007/s11606-014-2774-4 .

Pine DA, Madlon-Kay DJ, Sauser M. Effectiveness of a nurse-based intervention in a community practice on patients’ dietary fat intake and total serum cholesterol level. Arch Fam Med. 1997;6:129–34.

Coppell KJ, Abel SL, Freer T, Gray A, Sharp K, Norton JK, et al. The effectiveness of a primary care nursing-led dietary intervention for prediabetes: a mixed methods pilot study. BMC Fam Pract. 2017;18(1):106. https://doi.org/10.1186/s12875-017-0671-8 .

Marshall B, Floyd S, Forrest R. Clinical outcomes and patients’ perceptions of nurse-led healthy lifestyle clinics. J Prim Health Care. 2011;3(1):48–52.

Moher M, Yudkin P, Wright L, Turner R, Fuller A, Schofield T, et al. Cluster randomised controlled trial to compare three methods of promoting secondary prevention of coronary heart disease in primary care. BMJ. 2001;322(7298):1338. https://doi.org/10.1136/bmj.322.7298.1338 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Aveyard P, Brown K, Saunders C, Alexander A, Johnstone E, Munafò MR, et al. Weekly versus basic smoking cessation support in primary care: a randomised controlled trial. Thorax. 2007;62(10):898–903. https://doi.org/10.1136/thx.2006.071837 .

Byers MA, Wright P, Tilford JM, Nemeth LS, Matthews E, Mitchell A. Comparing smoking cessation outcomes in nurse-led and physician-led primary care visits. J Nurs Care Qual. 2018;33(3):272–8. https://doi.org/10.1097/NCQ.0000000000000296 .

Faulkner K, Sutton S, Jamison J, Sloan M, Boase S, Naughton F. Are nurses and auxiliary healthcare workers equally effective in delivering smoking cessation support in primary care? Nicotine Tob Res. 2016;18(5):1054–60. https://doi.org/10.1093/ntr/ntv206 .

Zwar NA, Richmond RL, Forlonge G, Hasan I. Feasibility and effectiveness of nurse-delivered smoking cessation counselling combined with nicotine replacement in Australian general practice. Drug Alcohol Rev. 2011;30(6):583–8. https://doi.org/10.1111/j.1465-3362.2010.00243.x .

Cherkin DC, Deyo RA, Street JH, Hunt M, Barlow W. Pitfalls of patient education. Limited success of a program for back pain in primary care. Spine. 1996;21(3):345–5. https://doi.org/10.1097/00007632-199602010-00019.

Waterfield A, Waterfield M, Campbell J, Freeman R. Can effective supervised pelvic floor muscle training be provided by primary care nurses? A randomized controlled trial. Int Urogynecol J. 2021;32(10):2717–25.

Harris T, Kerry SM, Victor CR, Ekelund U, Woodcock A, Iliffe S, et al. A primary care nurse-delivered walking intervention in older adults: PACE (pedometer accelerometer consultation evaluation)-Lift cluster randomised controlled trial. PLoS Med. 2015;12(2): e1001783. https://doi.org/10.1371/journal.pmed.1001783 .

Harris T, Kerry SM, Limb ES, Victor CR, Iliffe S, Ussher M, et al. Effect of a primary care walking intervention with and without nurse support on physical activity levels in 45- to 75-Year-olds: the pedometer and consultation evaluation (PACE-UP) cluster randomised clinical trial. PLoS Med. 2017;14(1): e1002210. https://doi.org/10.1371/journal.pmed.1002210 .

Pearson B, Katz S, Soucie V, Hunkeler E, Meresman J, Rooney T, et al. Evidence-based care for depression in Maine: dissemination of the Kaiser Permanente Nurse Telecare Program. Psych Quarterly. 2003;74(1):91–102. https://doi.org/10.1023/a:1021149923867 .

Desborough J, Bagheri N, Banfield M, Mills J, Phillips C, Korda R. The impact of general practice nursing care on patient satisfaction and enablement in Australia: a mixed methods study. Int J Nurs Stud. 2016;64:108–19. https://doi.org/10.1016/j.ijnurstu.2016.10.004 .

Halcomb E, Davies D, Salamonson Y. Consumer satisfaction with practice nursing: a cross-sectional survey in New Zealand general practice. Aust J Prim Health. 2015;21(3):347–53. https://doi.org/10.1071/PY13176 .

Caldow J, Bond C, Ryan M, Campbell NC, San Miguel F, Kiger A, et al. Treatment of minor illness in primary care: a national survey of patient satisfaction, attitudes and preferences regarding a wider nursing role. Health Expect. 2007;10(1):30–45. https://doi.org/10.1111/j.1369-7625.2006.00422.x .

Gallagher M, Huddart T, Henderson B. Telephone triage of acute illness by a practice nurse in general practice: outcomes of care. Br J Gen Pract. 1998;48(429):1141–5.

CAS   PubMed   PubMed Central   Google Scholar  

Nelson E, Wasson J, Kirk J, et al. Assessment of function in routine clinical practice: description of the COOP chart method and preliminary findings. J Chronic Dis. 1987;40:55–63.

Fernandez JM, Ariza-Cardiel G, Rodriguez-Martinez G, Gayo-Milla M, Martinez-Gil M, Alzola-Martin C, et al. Satisfaction with primary care nursing: use of measurement tools and explanatory factors. Rev Calid Asist. 2015;30(2):86–94. https://doi.org/10.1016/j.cali.2015.01.007 .

Randall S, Crawford T, Currie J, River J, Betihavas V. Impact of community-based nurse-led clinics on patient outcomes, patient satisfaction, patient access and cost effectiveness: a systematic review. Int J Nurs Stud. 2017;73:24–33.

Karaca A, Durna Z. Patient satisfaction with the quality of nursing care. Nurs Open. 2019;6(2):535–45. https://doi.org/10.1002/nop2.237 .

Nunu WN, Munyewende PO. Patient satisfaction with nurse-delivered primary health care services in Free State and Guateng provinces South Africa: a comparative study. Afr J Prim Health Care Fam Med. 2017;28(9):e1-8.

Google Scholar  

Da Silva, Brault I, Pineault R, Chouinard MC, Prud’homme A, D’Amour D. Nursing practice in primary care and patients’ experience of care. J Prim Care Community Health. 2018;9:1–7. https://doi.org/10.1177/2150131917747186 .

Veroff D, Marr A, Wennberg D. (2013), Enhanced support for shared decision-making reduced costs of care for patients with preference-sensitive conditions. Health Aff. 2013;32(2):285–93. https://doi.org/10.1377/hlthaff.2011.0941 .

Basch E. Patient-reported outcomes – harnessing patients’ voices to improve clinical care. N Engl J Med. 2017;376(2):105–8. https://doi.org/10.1056/NEJMp1611252 .

Basch E, Deal AM, Dueck AC. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318(2):197–8. https://doi.org/10.1001/jama.2017.7156 .

Greenhalgh J, Dalkin S, Gibbons E. How do aggregated patient-reported outcome measures data stimulate health care improvement? A realist synthesis. J Health Serv Res Policy. 2018;23(1):57–65. https://doi.org/10.1177/1355819617740925 .

Calvert M, O’Connor D, Basch E. Harnessing the patient voice in real-world evidence: the essential role of patient-reported outcomes. Nat Rev Drug Discov. 2019;18(10):731–2. https://doi.org/10.1038/d41573-019-00088-7 .

American Medical Association. AMA successfully fights scope of practice expansions that threaten patient safety. American Medical Association. 2020. https://www.ama-assn.org/practice-management/scope-practice/ama-successfully-fights-scope-practice-expansions-threaten . Accessed 08 Nov 2021.

Robeznieks A. Scope of practice: how the AMA fights for patient safety. American Medical Association. 2020. https://www.ama-assn.org/practice-management/scope-practice/scope-practice-how-ama-fights-patient-safety . Accessed 09 Nov 2021.

Canadian Nurses Association. Framework for the practice of registered nurses in Canada. CNA. 2015. https://www.caccn.ca/files/RN%20Framework%20Revision%20V2%2014_11_13.pdf . Accessed 09 Nov 2021.

International Council of Nurses. Scope of practice and decision-making framework toolkit: ICN regulation series. ICN. 2010. https://www.icn.ch/sites/default/files/inline-files/2010_ICN%20Scope%20of%20Nursing%20and%20Decision%20making%20Toolkit_eng.pdf . Accessed 09 Nov 2021.

Bottorff JL, Kelly M, Young J. Methodological issues in researching palliative care nursing practice. Can J Nurs Res. 2005;37(2):50–68.

PubMed   Google Scholar  

Lopatina E, Donald F, DiCenso A, Martin-Misener R, Kilpatrick K, Bryant-Lukosius D, et al. Economic evaluation of nurse practitioner and clinical nurse specialist roles: a methodological review. Int J Nurs Stud. 2017;72:71–82. https://doi.org/10.1016/j.ijnurstu.2017.04.012 .

Vincelette C, Thivierge-Southidara M, Rochefort CM. Conceptual and methodological challenges of studies examining the determinants and outcomes of omitted nursing care: a narrative review of the literature. Int J Nurs Stud. 2019;100: 103403. https://doi.org/10.1016/j.ijnurstu.2019.103403 .

Katz DA, Brown RB, Muehlenbruch DR, Fiore MC, Baker TB, AHRQ Smoking Cessation Guideline Study Group. Implementing guidelines for smoking cessation: comparing the efforts of nurses and medical assistants. Am J Prev Med. 2004;27(5):411–6. https://doi.org/10.1016/j.amepre.2004.07.015 .

Low N, McCarthy A, Roberts TE, Huengsberg M, Sanford E, Sterne JAC, et al. Partner notification of chlamydia infection in primary care: randomised controlled trial and analysis of resource use. BMJ. 2006;332(7532):14–9. https://doi.org/10.1136/bmj.38678.405370.7C .

Canadian Institutes for Health Information. Patient-centred measuring and reporting in Canada: launching the discussion toward a future state. CIHI. 2017. https://www.cihi.ca/sites/default/files/document/visioning-day-paper-en-web.pdf . Accessed 09 Nov 2021.

Download references

Acknowledgements

We acknowledge the following research assistants for the contributions to the systematic review screening, appraisal and data extraction process: Richard Buote, Ashley Joyce, Olivia Parsons, and Arifur Rahman.

This research was supported by funding received from Memorial University and the Department of Health & Community Services, Government of Newfoundland & Labrador.

Author information

Authors and affiliations.

Faculty of Nursing, Memorial University, 300 Prince Phillip Drive, St. John’s, NL, A1B 3V, Canada

Julia Lukewich & Dana Ryan

School of Nursing, Dalhousie University, 5869 University Ave. St, Halifax, NS, B3H 4R2, Canada

Ruth Martin-Misener

School of Nursing, Columbia University, 630 West 168th Street, New York, NY, 10032, USA

Allison A. Norful

Département de Médecine de Famille Et Médecine d’urgence, Université de Sherbrooke, 2500 Boulevard de l’Université, Sherbrooke, QC, J1K 2R1, Canada

Marie-Eve Poitras

School of Nursing, McMaster University, 1280 Main St W, Hamilton, ON, L8S 4L8, Canada

Denise Bryant-Lukosius

Department of Family Medicine, Memorial University, 300 Prince Phillip Drive, St. John’s, NL, A1B 3V6, Canada

Shabnam Asghari

Department of Family Medicine Primary Care Research Unit, Dalhousie University, 1465 Brenton Street, Suite 402, Halifax, NS, B3J 3T4, Canada

Emily Gard Marshall

Department of Family Medicine, Schulich School of Medicine & Dentistry, University of Western, 1151 Richmond Street, OntarioLondon, ON, N6A 5C1, Canada

Maria Mathews

Health Sciences Library, Faculty of Medicine, Memorial University, 300 Prince Phillip Drive, St. John’s, NL, A1B 3V6, Canada

Michelle Swab

School of Nursing, Queen’s University, 92 Barrie Street, Kingston, ON, K7L 3N6, Canada

Joan Tranmer

You can also search for this author in PubMed   Google Scholar

Contributions

JL conceived of the larger project, obtained grant funding, supervised data collection and screening, interpreted and synthesized results, and drafted and revised the manuscript; DBL, RMM, AN, MEP interpreted results and drafted and revised the manuscript; MS carried out the initial search strategy, assisted with adherence to PRISMA guidelines, interpreted results, and revised the manuscript; SA, EGM, MM, JT interpreted results and revised the manuscript; DR screened, appraised and extracted data, and assisted with interpretation of results and the drafting/revising of the manuscript. All authors approved the final draft.

Corresponding author

Correspondence to Julia Lukewich .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: a typesetting error in the third paragraph of the ‘Patient experience outcomes via PREMS’ subsection under the ‘Results’ was corrected.

Supplementary Information

Additional file 1., additional file 2., additional file 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lukewich, J., Martin-Misener, R., Norful, A.A. et al. Effectiveness of registered nurses on patient outcomes in primary care: a systematic review. BMC Health Serv Res 22 , 740 (2022). https://doi.org/10.1186/s12913-022-07866-x

Download citation

Received : 10 November 2021

Accepted : 03 March 2022

Published : 03 June 2022

DOI : https://doi.org/10.1186/s12913-022-07866-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Effectiveness
• Primary healthcare
• Registered nurse
• Primary care nursing
• Systematic review
• Patient measures

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Primary Care

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Last »
• Family Medicine Follow Following
• Health Professional Education Follow Following
• Medical Oncology Follow Following
• Non-formal Education Follow Following
• Primary Health Care Follow Following
• Mobile Media Follow Following
• General Surgery Follow Following
• Chronic illness Follow Following
• Gastroenterology Follow Following
• History of Medicine and the Body Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Journals
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Getting started in primary care research: choosing among six practical research approaches

https://doi.org/ 10.1136/fmch-2018-000042

While many primary care practitioners want to conduct research, many also struggle with getting started. This article’s purpose is to assist emerging researchers in identifying a topic of interest, to try the ‘fit’ of feasible research approaches and commit to a research approach. The article addresses six objectives: (1) identify how important primary care research comes from clinical stories; (2) recognise how clinical stories become the source of research topics; (3) discern how the research process resembles the care of patients; (4) distinguish the essential features of six research approaches feasible for primary care researchers; (5) evaluate the fit of the six research approaches featured in this special issue; and (6) develop a list of steps that need to be taken to implement primary care research projects. Using ‘HPV (human papilloma) vaccination’ as a hypothetical topic, the article illustrates how an emerging researcher can complete the worksheets. Using the HPV topic, a worksheet illustration shows how to complete the worksheets, and examples from the literature illustrate how actual studies have used six feasible research approaches for primary care: (1) survey research, (2) semistructured qualitative interviews, (3) curriculum development, (4) continuous quality improvement, (5) clinical policy analysis and (6) case study research. The worksheet exercises support choosing a feasible research approach for emerging researchers. Emerging researchers using these exercises can identify a topic, choose a research strategy aligned with the researcher’s interest, create a study title, develop a list of the next steps, and be well positioned to implement an original research project

• Significance statement

While many family medicine and community health practitioners lack formal training in research, they would like to engage in research. A resource that helps aspiring researchers move from a clinical topic of interest to choosing a practical research approach has been lacking. This article provides guidance for aspiring researchers to identify topics of interest from clinical stories. The article then provides a worksheet activity to support choice of a project by considering six research approaches: survey, semistructured interviews, curriculum development, quality improvement, health policy analysis and case study research. By comparing each according to the problem, information needed, intervention involved, implementation strategy, and how to evaluate findings and take the next steps, aspiring researchers can choose a practical research approach.

• Introduction

Primary care practitioners can and need to engage in research. 1 2 The purpose of this article is to assist emerging researchers in primary care, that is, students, residents, fellows and practitioners who are motivated to conduct primary care research, but lack guidance as to how to proceed. The paper addresses this challenge by demonstrating how research questions come from clinical stories, illustrating six feasible research approaches for emerging researchers, developing a title and planning the next steps. This article helps aspiring researchers consider different research approaches. Through the use of worksheets and examples, the main outcome of this article is to identify a topic and choose an appropriate research approach.

Best research questions come from clinical stories

Emerging researchers often say, “I want to do research, but I don’t know where to start.” Clinical stories provide the best starting point for developing research plans. A story is the narration of an incident or a series of events that becomes the source of research questions. Clinical stories reflect issues that a primary care provider sees, but does not understand or that do not make sense. Clinical stories can be anything that causes one to stop and think, or something with multiple options for treatment with an elusive best choice. Identifying a story of personal relevance will best facilitate and sustain completing a research project.

Where do clinical stories arise?

Clinical stories can arise out of relationships with patients, office staff or other colleagues including subspecialists. Stories often arise from specific settings or transitions between settings, for example, ambulatory, hospital, long-term institution or home care settings. They may arise during training of students, residents or fellows, as questions emerge about clinical care or the best approach to training. Sometimes, stories come from circumstances that recur repeatedly over time or from observations regarding a series of patients. Stories also give rise to anomalies that become research questions.

Activity for getting started: identifying a relevant clinical story

Table 1 illustrates a clinical story that could became the source of a research project with real-world applicability. The topic of the case is human papillomavirus (HPV) vaccination, and comes from my work as a bilingual (English and Japanese) family physician working in a clinic serving many Japanese and other Asian families. 3 Table 2 is for readers to complete in response to the clinical story. To get started, use table 1 as a reference and complete in table 2 the following three items: (1) researcher name(s), (2) research topic and (3) a clinical story.

A clinical story

Consider the case of a 46-year-old obese Asian woman who visits the office frequently and often with different practitioners. Her medical history includes recurrent visits for chest pain, hypertension, hypercholesterolaemia, type 2 diabetes, and suspected anxiety and depression. Two previous cardiologists who evaluated her concluded she had non-cardiac chest pain. She says they told her that her "heart was fine" and advised her "not to worry". The patient has other physical complaints, including intermittent aches, fatigue, headaches and food allergies. On questioning, she is adamant as in previous visits that she has no problems with anxiety or depression. She agrees to laboratory testing and regular follow-up, but asks for another referral to cardiology as the visit comes to closure. While the details vary, visits with patients such as this one occur frequently in primary care.

Research design starts with the story and the question

A number of questions arise from the story of this patient. How can primary care practitioners optimally manage the care of patients with multiple chronic problems? How can they efficiently recognise common mental health problems like anxiety and depression? What is the acceptability of a diagnosis of anxiety and depression for patients, and how might it differ among different cultural groups? How might it differ between men and women? What are the best ways to follow patients with chronic diseases over time? Concerns such as these that arise from this story can be framed as clinically meaningful research questions.

Activity: getting started by making a list

Developing a list of interesting clinical stories or problems that occur during patient care provides a great starting point for conducting research. Table 3 provides an organisational structure for recording important details. This structure serves as a reminder of the importance of stories. Completing the table helps distinguish whether the identified problems resonate as issues of real or passing interest. The list can be completed electronically, in a computer or handheld device, or in a paper notepad. The list also functions to stimulate ideas that can be later discussed with a colleague, for example, practice colleague, resident, fellow or advisor, in order to reflect about the personal meaning, clinical value and practical significance of the issues, and potential for research.

How to recognise a good clinical story

Recognising a good clinical story requires an inquisitive and receptive mind when problems arise that lack good answers. What are the ways to recognise good clinical stories? As an early third-year resident, I had a 52-year-old female patient new to me who came for well-woman check-up in my family medicine clinic ( Table 3 ). On review of her chart, I noted she had a history of obesity, hypertension and uterine fibroids. I also noted that she was status-post total hysterectomy due to her fibroids. I was puzzled both if I should do a Pap smear, and if so, how to do it. I consulted with my faculty preceptor. “That is a great question!” she responded. In looking at the literature, I found that recommendations on whether and how to do a Pap smear in this setting were notably vague, absent or conflicting. This story led to the question: What is the utility of obtaining Pap smears for women who have undergone total hysterectomy for benign reasons? I used a clinical policy analysis to explore this question and published an article in response. 4 Later, with the support of a small grant, I conducted a cost-effectiveness analysis on the topic as well. 5

Recording details of clinical stories

Let us now review how the case above served as a starting point for developing a research question, and how we can replicate the process of question development and choice of method using other stories. Look at table 3 and note the steps to follow. Then retrieve and use your list of clinical stories to generate a research question. Using full sentences if possible, complete the other sections of the table. Record when the event occurred, especially specific date in the event of a need to review records. Indicate who was involved, for example, specific patient, learner or administrative personnel. Record in detail what happened. Indicate where the event occurred, for example, ambulatory setting, hospital and so on. Record why the care, incident or phenomenon was perceived to be a problem. Describe how the problem was responded to, and as relevant, the outcome of the response/intervention. Use the comment section to record any other information that might be important, including ideas or other reminders that might be helpful at a later time, for example, others involved, follow-up issues and so on.

Analogies between clinical practice and conducting clinical, management and education research

The five major steps of the approach to research enquiry are remarkably similar to the care of patients (see table 4 ). Practitioners new to research may found this analogy helpful for thinking about the research process.

‘Trying On’ feasible research approaches

Having identified topics of interest, emerging researchers often become stuck at the fourth step in table 4 . As they are uncertain about what approach to use, it is hard to identify and settle on a strategy for intervention and/or research. I advise ‘trying on’ a variety of options. This means considering the various research approaches, thinking through the work involved and the expected outcomes. Think of the analogy of trying on different clothes to find out what is the best ‘fit’. This article now focuses on ‘trying on’ the six different approaches featured in this special issue of FMCH .

Activity for ‘Trying On’ the six featured approaches

Using table 1 as a reference, complete section 4 of table 2 . Trying on all approaches will provide the most thorough opportunity for choosing. To fully benefit from the exercise, try on at least two. This exercise will help narrow down a decision about one or two preferred approach(es) so as to focus reading on the candidate methodological approaches. The following discussions and the full papers on these topics in this special issue can be used as references.

Feasible research approaches for primary care

The research approaches addressed in this special issue of FMCH include (1) survey research, (2) qualitative interviews, (3) curriculum development, (4) continuous quality improvement, (5) clinical policy analysis and (6) case study research. To illustrate the six approaches, this article features examples for each methodological approach applied to the topic of HPV vaccination ( table 5 ). For example, suppose a resident physician consulted with a faculty about an interest in doing HPV vaccine research. The illustration in table 1 , section 4 suggests how the topic could be pursued with each type of project. Below, examples from the literature illustrate the application of each. While some examples were conducted nationally, the same research could be conducted in an emerging researcher’s own community, clinical practice (especially if serving a unique population) or in one’s residency. By addressing a gap in the literature, the findings could yield a scholarly publication. Table 1 , section 4 illustrates how I might conduct this research and can guide completion of table 2 , section 4.

Survey research

Researchers who engage in survey research may begin by identifying information for inclusion in the survey through qualitative procedures. 6 Surveys involve a series of questions delivered to a target population. When engaging in survey research, using a previous survey that has been demonstrated to be reliable and valid can be very practical. Unfortunately, existing surveys do not address the issues of interest to primary care researchers. Emerging researchers can create a survey, but should be methodical and patient. Experienced survey methodologists go through extensive revisions, cognitive testing and pilot testing. Surveys often include closed and open-ended questions. 7 Responses are summarised and the findings examined. Analysis of data frequently involves a comparison between one or more groups for differences. Table 5 illustrates how Allison et al 8 used survey research to assess primary care physicians’ attitudes about HPV vaccination.

Semistructured qualitative interviews

While often attributed to Einstein, William Bruce Cameron may have been the first to articulate the concept of ‘Not everything that can be counted counts, and not everything that counts can be counted’. 9 This point illustrates the importance of qualitative research for understanding how things happen, why they happen and the perspectives of particular groups. A key approach to qualitative research is the semistructured qualitative interview. 6 Research appropriate for semistructured qualitative interviews often involves investigating perspectives, experiences and processes as they relate to a particular phenomenon ( table 4 ). While less familiar to many medical researchers, qualitative research has a robust history in numerous fields in the social sciences. 10 Table 5 illustrates how Hughes et al 11 explored the process of HPV vaccine decision making in paediatric primary care using semistructured qualitative interviews.

Curriculum development

Primary care educators often gravitate to research about curriculum development as an opportunity to advance educational practice according to best evidence. 12 Systematic change in curriculum grounded in evidence involves the same five major steps illustrated in table 4 and follows a research approach by (1) identifying a deficit in performance, (2) assessing current educational efforts, (3) developing a curriculum, (4) implementing the curriculum and (5) evaluating the impact of the curriculum change and modifying. Table 5 illustrates how the Centers for Disease Control and Prevention developed an online curriculum to promote interest in HPV vaccination. 11

Quality improvement

In administration of a clinic, primary care practitioners may be positioned or requested to examine current practice approaches and optimise them. 13 Continuous quality improvement provides a framework for improving practice performance ( table 4 ): (1) identifying the problem or unintended outcome of clinical care or practice management; (2) assessing the factors involved; (3) developing an alternative strategy to the current approach; (4) implementing the alternative strategy; and (5) evaluating the effects of the alternative strategy and modifying it, and thus resuming the cycle. Table 5 illustrates how the Quality through Technology and Innovation in Pediatrics organisation developed a quality improvement effort to support practices to increase HPV vaccination rates. 14

Clinical policy analysis

A clinical policy analysis process can apply locally or globally. 15 That is, as new recommendations or evidence emerges, a solo-practice physician or physician group will need to decide whether to follow recommendations. Alternatively, as illustrated above under the 'How to recognise a good clinical story' section, a seemingly simple problem can have implications for national policy. Policy work can also be conceptualised under the five steps of table 4 . This process involves (1) identifying relevant policies or guidelines addressing the health issue; (2) assessing the development of the policy and consulting relevant literature; (3) analysing the policy based on a framework; (4) disseminating the policy, for example through publication, or perhaps locally within an organisation or even within a practice; and (5) evaluating the impact of the policy recommendation, for example, was there uptake by one or more organisations. Table 5 illustrates how Shapiro et al 16 used health policy analysis to examine the rationale for vaccinating male adolescents in Canada.

Case study research

Primary care practitioners may be engaged with unique populations in their communities, work in international projects, develop novel teaching approaches or develop unique clinical practices. These activities and many others can be studied or evaluated using a case study approach. 17 Case studies may use qualitative or mixed-methods data collection. 18 A case study involves forming a research question about a particular phenomenon (the case), collecting and analysing the information, and developing a deep understanding about the phenomenon. There are many different types of case studies (eg, single and multiple) and purposes of case studies. 19 Broadly speaking, case study research may illustrate a unique phenomenon, an intrinsic case, or be used to illustrate a general or typical problem, an instrumental case. 20 As in table 4 , the five major steps have similarities to the other approaches: (1) identifying a phenomenon of study and bounding it; (2) searching for informative literature about the phenomenon; (3) assessing if the case is sufficiently unique and informative for a novel addition or expansion of the literature; (4) collecting information about the case using existing and or new data; and (5) analysing the collected data in response to the question, drawing conclusions and identifying the contribution as part of dissemination. Table 5 illustrates how Aujo et al 21 used case study research to disprove a cultural resistance to uptake of HPV vaccination stemming from concern that receiving HPV vaccination leads to early sexual debut among Ugandan girls.

Activity: choosing among the selected research approaches

Having ‘tried on’ different research approaches by completing table 2 , section 4, the next step involves reading more about the research approaches you have selected. This special issue of FMCH contains ‘how to do it’ articles for each of the six research approaches listed. For the top choices, read more about the approaches and consider reading more indepth references listed in each article. Consider reading the full articles summarised to learn how other authors have used specific approaches. Discussing top choices with a research partner, mentor or advisor may help with choosing an approach. Of course, the choice to use another approach may be relevant should none of these six have sufficient appeal or fit.

Composing the project title

Having identified a research topic grounded in clinical care, and an approach that is practical, feasible and a good fit for the researcher’s interest, the next step involves writing a title for the project. Effective titles generally have four elements: the population, the topic, the location of the research and the methodology. 22 Table 6 provides formulas for writing a project title and an example for each type of project. Using table 1 , section 5 and table 5 as references, compose a title for your study and record this in section 5 of table 2 . In reality, the initial title often gets changed and may require further editing.

Activity: identifying the next steps

Using table 1 , section 6 as an example, complete table 2 , section 6. This section can be completed based on any number of concerns or ideas. Perhaps conversing with a research partner, mentor or someone else can move the project along. Would further review of the literature move the project forward? A closer reading of the companion reference articles in this special edition of FMCH may clarify the commitment to one or another of the approaches. Would duplicating a previous study with a different population serve as an option? Most important, record which of the approaches look to be the most fun or appealing. This factor cannot be underscored in terms of the prognosis for completion of the project. Additionally, reading the articles in this issue on analytical approaches for quantitative and qualitative data provides guidance about how to approach analysing the data. 23 24

As research needed in primary care settings differs from other clinical settings, it becomes apparent that family doctors need to conduct the research. This article encourages developing meaningful research questions from clinical stories, considering different approaches for investigating the phenomenon of interest and choosing an approach. A title also can help give a proposal ‘life’ by embodying the essence of the project. For the next step, aspiring researchers may wish to turn to the other articles in this special issue of FMCH to fully develop research questions and research plans appropriate for the research approach chosen to pursue the identified topic.

• Supplementary files
• Publication history

• Open access
• Published: 09 August 2024

Primary health care coverage in Portugal: the promise of a general practitioner for all

• Eduardo Costa 1 ,
• Joana Pestana 2 &
• Pedro Pita Barros 2  

Human Resources for Health volume  22 , Article number:  55 ( 2024 ) Cite this article

95 Accesses

Metrics details

Primary care is an essential pillar of health systems. Many countries have implemented different policies to improve access to primary care. However, persistent challenges remain. This paper offers a critical analysis of the evolution of primary care coverage in Portugal, focusing on the number of patients without an assigned general practitioner (GP).

We collected and analyzed publicly available data from 2009 to 2023 to decompose primary care coverage in three components: the number of patients enrolled in primary care units (demand-side effect), the number of GPs measured in full-time equivalent (supply-side effect), and the average number of patients on each GP’s list (patient-to-GP ratio, capturing a productivity effect). We provide national and local level estimates for these three components.

Between 2009 and 2023, there was an overall decline in the number of patients enrolled in primary health care units. Concurrently, there was also a net decrease of GPs measured in full-time equivalent. Additionally, there was a progressive reduction in the average number of patients on each GP’s list. The rise in the number of patients without an assigned GP is attributed not only to a reduction in the number of physicians, but also to a decrease in the patient load per doctor.

Conclusions

Hiring additional GPs may not suffice to enhance coverage. Achieving higher coverage may imply revisiting patient load per doctor or considering alternative care models. Understanding the challenges related to GP coverage is critical for improving the efficiency of primary care.

Peer Review reports

Introduction

Currently, health systems face multiple challenges related to shortage of health professionals, increasing demand and limited budgets. An efficient primary care system is a necessary condition to deliver high-quality care [ 1 ]. However, achieving universally primary care depends not only on organizational adaptability of health systems, but also on the availability of human resources.

This paper provides a critical analysis of the evolution of primary care coverage in Portugal. We compile publicly available data to decompose primary care coverage in three effects: the variation in the number of enrolled patients in primary care (demand-side), employed physicians measured in full-time equivalent (supply-side), and physicians’ list (patient-to-GP ratio, capturing a productivity effect). We then discuss the challenges related to achieving universal coverage.

The worldwide shortage of health professionals prevents health systems from attaining universal health coverage goals. In the US, estimates point towards a deficit of 139 160 physician professionals by 2030 [ 2 ]. Similarly, a shortage of nearly 400 000 physicians and of nearly 2.5 million nurses is forecasted by 2030 in OECD countries [ 3 ]. The 2022 WHO report on health and care workforce in Europe points out that countries struggle to attract and retain health professionals [ 4 ]. Worldwide, a shortage of up to 15 million health workers is predicted by the end of the decade [ 5 ]. Furthermore, the uneven distribution of physicians across regions results in underserved areas [ 6 ].

Shortages in the healthcare workforce can be attributed to a myriad of factors, with various challenges emerging both on demand and supply sides. On the demand-side, the aging population in many regions, coupled with increasing prevalence of chronic diseases, has contributed to the increase in healthcare needs [ 7 ]. Projections for the United States indicate a need for an additional 10 000 physicians in primary care until 2025 to adequately address the challenges associated with population aging [ 8 ]. Moreover, rising population expectations for healthcare services, coupled with increased access to information and personalized treatments, have intensified the demand for health professionals [ 9 ]. The unprecedented challenges posed by the COVID-19 pandemic have only served to exacerbate the workload on these professionals.

On the supply-side, the declining attractiveness of health professions is influenced by several factors. Physicians tend to prefer urban and suburban areas, leaving rural and underserved populations with inadequate healthcare access and potentially exacerbating health inequities [ 6 ]. The insufficient investment in state-of-the-art facilities and advanced technology hampers the efficiency of healthcare services. Specifically, primary care relies on the establishment of robust patient–doctor relationships and proves to remain particularly labor-intensive despite the existence of health-related apps and technical support programs for physicians [ 10 ]. Deteriorating working conditions further contribute to shortages, as health professionals grapple with environments that may not adequately support their well-being and productivity. This ranges from stagnant remunerations in the context of escalating living costs, to rigid job positions which limit flexibility and creates a financial disincentive. This lack of flexibility can make it challenging to tailor employment contracts to health professionals’ preferences, which increasingly value autonomy [ 11 ].

In primary care, a key challenge for policy-makers is the recruitment of general practitioners (GP). The inadequacy of resources, or their maldistribution, further hinders the achievement of this goal. Some reforms have been implemented to improve the relative attractiveness of primary care [ 12 ]. These include efforts to reduce physicians' administrative workload, automate routine tasks such as health record management, and provide improved access to essential skills and support. However, many of these reforms have often fallen short of their intended objectives [ 13 ].

In recent years, the Portuguese National Health Service (NHS) has undergone multiple reforms to enhance access to primary care. The NHS provides publicly financed health care to the population, even though private health care services are also available. Over the last decades, political parties have promised to assign a GP to every resident in Portugal. This has been both a frequent electoral pledge, as well as a milestone included in multiple government programs, at least since the beginning of the century. Despite the variety of policies implemented and increased funding, the goal of a GP for all residents has not yet been achieved.

In a country with a population of around 10.3 million inhabitants, 16.5% of patients in primary care had no assigned GP in 2023. These patients have significant access constraints to primary care. For these patients, access is privileged only for acute conditions and contingent on physicians’ limited availability. The lack of access to primary care may force patients to search for private health care services or, within the NHS network, to enter hospital care through the emergency department. The proportion of patients without an assigned GP in 2023 was slightly above 2009 levels (15.8%). However, it was significantly above 2019 levels, where primary care coverage was at its maximum level—with only 7.3% of patients without an assigned GP.

A key factor that influences the number of patients without GP is related with the ability of the NHS to retain primary care doctors with attractive working conditions [ 14 , 15 ]. However, such ability is challenged by many different factors. First, despite policies implemented by the NHS to attract GPs to underserved areas, evidence suggests that GPs tend to concentrate in locations with more favorable working conditions [ 16 ]. Second, the dual practice context of the Portuguese health system imposes additional nuances. In fact, in 2023, a total of 8 856 GPs were licensed to practice in Portugal, with 6 934 (78%) of those being employed in the NHS (either in full-time or part-time) [ 17 ]. Although precise data are not available, a survey revealed that two-thirds of physicians did not work full-time in the public sector, and more than half also worked in private practice [ 18 ]. Third, this situation may also be exacerbated by the impending retirements, since 1 489 GPs (17%) were aged 50–65 in 2022 [ 17 ].

Moreover, migration of physicians to other countries may impose further challenges to the Health System, including in primary care. Although no specific data exists regarding GPs, research among medical residents and junior doctors suggests that 55–65% were considering the possibility of moving abroad in the near future [ 19 , 20 ]. In 2023, the Portuguese Physicians Association claimed that over 450 physicians manifested their intention to migrate (around 2 000 physicians in the period between 2019 and 2023) [ 21 , 22 ]. Although this represents less than 1% of doctors registered in Portugal (63 053), it corresponds to roughly 22% of new medical residents in 2023 (2 044) [ 23 ].

The shortage of GPs in the Portuguese NHS primary care network can be proxied by the number of job openings defined by the Health Ministry, since the hiring for primary care is centralized and happens periodically, once or twice a year. In the last three hiring seasons, between 900 and 1 000 vacancies were opened [ 24 , 25 ]. However, applications were substantially lower: in December 2023, there were only 114 hired physicians for a total of 924 vacant positions [ 25 ].

This is the first study that describes the primary care coverage in Portugal over more than one decade, from a pre-financial crisis and austerity measures period to a period post the COVID-19 pandemic. Some of the issues faced by the Portuguese NHS primary care workforce are shared by many other health systems, particularly in Europe. Hence, improving the discussion on their root causes and potential solutions is key to generating actionable contributions to other health systems struggling with similar challenges, as discussed by the editorial for the special collection on the medical workforce crisis in primary care in Europe [ 26 ].

Institutional background

The Portuguese NHS, a publicly funded healthcare system, offers comprehensive and universal healthcare coverage to the population [ 27 ], although private healthcare also exists. Nonetheless, the NHS represents roughly two-thirds of overall public health expenditure [ 28 ].

Within the Portuguese NHS, primary care services are delivered through public health centers, where GPs, also known as family doctors, work in teams of professionals and serve as gatekeepers. GPs operate on a patient list basis, and while all residents have the option to enroll in a primary care unit of their catchment area, this does not guarantee assignment to a GP. Residents can opt out of the NHS primary care network and seek care in the private sector through voluntary health insurance or out-of-pocket payments. Nonetheless, almost all residents are enrolled in a primary care unit, either in a GP list or awaiting vacancy in any GP list of the practice where they are enrolled. In 2023, 10.5 million patients were enrolled in primary care units, exceeding the 10.3 inhabitants according to the 2021 census, which reflects additional patients such as migrants.

In 2023, of the enrolled patients in primary care, 16.5% were not assigned to a GP. Consequently, these patients face significant constraints in accessing the NHS—especially when considering the GP’s gatekeeping role. While they can request acute same-day appointments at their primary care units, availability is contingent on the physician's schedule, which often has limited openings.

In 2005, a primary care reform was launched with the goal of improving quality of care. Primary care practices underwent administrative consolidation with the creation of regional groups of primary care practices (ACES) [ 27 ]. Under this reform, a new organizational and financing model was also rolled out: the Family Health Units (FHU). A team of physicians, subject to prior assessment and budgetary availability, could voluntarily apply to transition their practice into a FHU unit [ 29 ]. These teams are contracted to provide care for a specific geographically defined population, with an average of 1,900 patients assigned to each physician. In these units, physicians, nurses and administrative staff receive an individual monetary incentive linked to their performance, which in the case of physicians can represent up to 60% of their salaries. A transitional model (FHU-A) with pay-for-performance incentives attributed to the team was created, for units transitioning between the traditional model (Personalized Health Care Unit, PHCU) to the full FHU model with individual incentives and more stringent performance targets (FHU-B).

Those physicians who choose not to establish or join FHU would continue to operate in practices following the prior model (PHCU). In the PHCU model, while also having a fixed list of patients, physicians are paid a fixed salary. Moreover, as FHU are being established almost exclusively with patients with a GP, patients without a GP tend to concentrate in the remaining PHCU units. This implies that physicians in PHCU not only manage patients in their list but are also responsible for providing care to patients without an assigned GP, enrolled in their practices.

The roll-out of FHU was slow and subject to budget constraints. In 2007, there were 81 FHUs (12.9% of the practices). By the end of 2023, the number of FHUs had increased to 619 (68.2%), employing 4 107 GPs (76.1%) and covering 7.2 million patients (69.5%). Among these patients, only 3.65% were not included in a GP list. In the traditional model (PHCU), 1 290 GPs (23.9%) were providing care to 30.5% of all enrolled patients, of which 40.5% were not assigned to a GP [ 30 ].

This paper presents descriptive evidence on GP coverage for Portuguese residents, examining its evolution over time and decomposing it into supply, demand and productivity effects. Data for this study were gathered from several official sources, specifically the Access Report from the Central Administration of the Health System [ 31 ], the NHS Transparency database [ 14 ], and the primary healthcare monitoring dashboards (BI-CSP) [ 30 ].

The key variables of interest include (i) the number of enrolled patients in primary care practices (demand-side effect); (ii) the number of GPs in primary care practices measured in full-time equivalents (supply-side effect); and (iii) the number of patients without an assigned GP.

This information was collected from 2009 to 2023 for each group of primary care practices (AceS), consisting of 55 units (74 units from 2009 to 2012). Further contextual variables were also collected to conduct secondary analysis. Data were aggregated both at the level of five regional health administrations (ARS) and at the national level. Only mainland Portugal was considered.

To ensure accuracy and prevent potential double-counting of physicians working in multiple practices simultaneously, we adopted the full-time equivalent (FTE) method to calculate the weighted number of GPs. Not adjusting the number of GPs to full-time equivalent would impose a bias in our results, namely in terms of the patient-to-GP ratio. According to the data from December 2023, there were 6 934 GPs working in public primary care practices, which corresponds to 5 395 GPs FTE [ 30 ]. This means that in each ACES the number of GPs FTE corresponds to 83.9% of the total number of GPs. To calculate the FTE, the effective working hours were compared versus the working hours of the 40 h per week of the current normal working period for Specialist and Intern Physicians. Throughout the paper, mentions to the number of GPs refer to GPs measured in FTE.

The analytical approach is twofold. First, we provide an overview of GP coverage, detailing the observed trends. Second, we conduct a decomposition of GP coverage, distinguishing between demand, supply and productivity effects at the national and local level.

The calculation of the number of patients without an assigned GP ( \({m}_{j}\) ) in each group of primary care practices (AceS) ( j) is determined by several factors. Firstly, it depends on the total number of patients enrolled at each group of primary care practices ( \({n}_{j}\) ). An increase in the number of patients enrolled increases the difficulty of providing coverage for all. Secondly, it is contingent on the number of full-time equivalent physicians working at each group of primary care practices ( \(\sum {GP}_{i}\) ). The recruitment of additional physicians enhances overall coverage. Thirdly, it relies on the number of patients each physician ( i ) has on her own list ( \({L}_{i}\) ). Larger patient lists per physician contribute to increased coverage, assuming all other variables remain constant. Therefore, the estimation of patients without an assigned GP was derived from the following expression:

This expression can be simplified considering the average number of patients with GP per doctor (patient-to-GP ratio) in each group of primary care practices ( \({L}_{j}\) ). This represents the ratio between patients with an assigned GP and the number of doctors. Thus, the previous expression can be written has a function of variables collected for each group of primary care practices j :

One can use the previous expression to decompose the change in the number of patients without GP into different effects. The following expression represents the decomposition of the change on the number of patients without an assigned GP between two periods (0 and 1) in three effects—each one within brackets: change in the number of enrolled patients, change in the number of physicians, and change in the patient-to-GP ratio (the average list size per physician):

Descriptive evidence

Universal coverage of primary care, by granting a GP to each resident, has been a political objective since the onset of the NHS. Figure  1 provides the proportion of patients enrolled in primary care practices with an assigned GP between 2009 and 2023 for each of the five administrative regions.

Patients enrolled in a primary care practice with an assigned GP per administrative region (% of enrolled patients in primary care units; 2009–2023)

In 2023, 83% of the population nationwide had an assigned GP. This was close to the historical minimum of 82% coverage verified in 2011, and well below the 93% maximum coverage registered in 2019. One can see that, even though sizeable differences are identified across regions, there is a common pattern among them. Nationwide, GP coverage had a small decline between 2009 and 2011, followed by an almost continuous recovery until 2019. However, 2019 was a turning point since the coverage of GPs has declined steadily since then. Table A1 and Fig. A1, in the appendix, provide further details on differences across local groups of primary care practices.

GP coverage rate depends on several factors. Two key variables of interest include the number of enrolled patients in primary care units (demand-side), and the number of GPs measured in full-time equivalent (supply-side). Figures A2 and A3, available in the appendix, display the evolution of those two variables. Between 2009 and 2023, the number of enrolled patients in primary care units decreased by 7%, while the number of GPs FTE also decreased by 5%.

The proportion of patients without GP is dynamic and heterogeneous across time and regions. Figure  2 displays the relationship between the change in patients without GP (2023 vs 2012) and the proportion of patients without GP in 2023. We are considering long-run trends over this period, even though the effects are not linear over time, as discussed below.

Relation between the change in patients without GP (2023 vs 2012) and the proportion of individuals without GP in 2023 (% of enrolled patients by groups of primary care units)

In the Lisboa region, for instance, there is significant variability in the proportion of patients without GP, even though most groups of primary care practices (ACES) in this region saw an increase in the proportion of patients without GP. The same does not happen for practices in the Norte region, were most ACES displayed higher coverage rates and large improvements in coverage.

Other factor that may play a key role is related with the average number of patients assigned to a GP per doctor, referred to as the patient-to-GP ratio. This does not represent the ratio of enrolled patients per GP, but instead the ratio of enrolled patients with GP per GP. Thus, this ratio reflects the average size of a GP’s list, and can be interpreted as a productivity variable for physicians.

The correlation between this variable and other healthcare and demographic variables is investigated in Table A2 , available in the appendix, although no strong correlation was identified. Results suggest a negligible positive correlation with the proportion of patients without an assigned GP. Moreover, a moderate negative correlation exists with both the proportion of elderly individuals and diabetes prevalence. Additionally, a weak negative correlation is observed with population density. This issue was further investigated through scatter plots (appendix A4 to A6).

Decomposition

As explained before, one can decompose the change in the number of patients without GP in these three effects (plus a crossed effect, allocated to the list dimension). The following figure describes the role of each factor (Fig. 3 ).

Contribution of each effect (enrolled patients, GP, patient-to-GP ratio) to the change in the number of patients without GP between 2009 and 2023

Over time, the reduction in the number of enrolled patients in primary care units—capturing a demand-side effect—led to a reduction in the number of patients without GP. This contributed to a reduction of approximately 800 thousand patients without GP. If everything else would remain constant, the number of patients without GP would be reduced by 44% due to the reduction in the number of enrolled patients.

However, during the same period, the number of full-time equivalent GPs has also decreased from 5 650 FTE to 5 395 FTE in 2023 (a 5% reduction)—representing a supply-side effect. By itself, such lack of GPs contributes to an increase in over 400 thousand patients without GP. This corresponds to a 23% increase in the number of patients without GP.

Surprisingly, and often ignored in public discussion, is the impact of the patient-to-GP ratio, which captures a productivity dimension. During this period, the average number of patients assigned to a GP per GP has declined. This implies that more doctors would be required to take care of the same number of patients. In fact, we estimate that the reduction in the patient-to-GP ratio has contributed to an increase of over 300 thousand patients without GP (18%).

This effect may be explained by different factors. However, since our estimates are based on aggregate data, it is not possible to disentangle the relative importance of each potential mechanism. In fact, the reduction in the patient-to-GP ratio may be linked either with an actual reduction of the number of patients in physicians’ lists (for example due to the increased complexity of patients [ 32 ]), or by an exit of physicians with larger lists—which reduces the average list size. Further details on these mechanisms are provided in the discussion section.

Overall, even though the reduction in the number of enrolled patients contributes to alleviate the pressure on the coverage rate, the reduction in the number of GPs and in the patient-to-GP ratio had a reverse effect on GP coverage.

These aggregate effects are estimated over a relatively long period of time. However, these three effects may have different contributions throughout time as presented in Fig.  4 . Up until 2016, the reduction in the number of enrolled patients (dark blue bar) contributed to an improvement in the coverage rate. To some extent, this may be associated with administrative measures implemented in 2012 and 2013 to remove non-users from primary care. However, since 2017, additional patients have been enrolled in the system. This had the opposite effect, negatively impacting the coverage rate.

Yearly decomposition of the change in the number of patients without GP (2009–2023). “Total” represents the absolute early changes on the number of patients without GP

Up until 2014, there was a reduction in the number of GPs (light blue bar) which led to a reduction of coverage rates. This situation was partially reversed in the following years until 2018. However, in 2019, and particularly in 2021 and 2022, the change in the number of GPs had a large negative effect in the coverage rate.

Finally, since 2016, and except for the 2019, 2021 and 2022, there was a negative effect on the coverage rate due to a deterioration on the patient-to-GP ratio (grey bar).

There is also heterogeneity across groups of primary care practices (ACES), as displayed by Fig.  5 . This plot represents, for each ACES, the relative contribution of each of the three effects to the change in the number of patients without GP.

Relative contribution of each effect to the change in the number of patients without GP per group of primary care practice (2012–2023, columns add to 100% (left axis); groups of primary care practices ordered from larger increases in the % of patients without GP to larger reductions in the % of patients without GP (right axis))

One can observe some trends which are common to most—but not all—groups. First, most ACES saw a reduction in the number of enrolled patients. Everything else constant, this demand-side effect contributes to reducing the number of patients without GP. Only 15 ACES (27%) saw an increase in the number of enrolled patients.

Second, most ACES were able to hire more doctors, measured in full-time equivalents (supply-side effect). Everything else constant, hiring additional doctors contributes to alleviate the number of patients without GP. Only 6 ACES (11%) saw a negative impact from this supply-side effect.

Third, there was a reduction in patient-to-GP ratio across most ACES. Everything else constant, this reduction contributes to increase the number of patients without GP. Only 1 ACES (2%) saw a positive contribution from this productivity effect.

It is also interesting to note that the negative contribution from the change in patient-to-GP ratio was smaller in ACES that were able to achieve sizeable reductions in the proportion of patients without GP (closer to the right-hand side of the plot). These practices were able to attract and recruit new doctors, compared with practices that saw sizeable increases in patients without GP.

From 2009 to 2023, there has been a slight decline in the number of patients without a GP, decreasing from 1.8 million to 1.7 million patients. However, within this period, significant variations occurred, with a pronounced increase of patients without GP in recent years. The variation in GP coverage may be attributed to three different factors, as highlighted by the previous section.

Between 2009 and 2023, there was an overall decrease in the number of enrollees in primary care (demand-side effect)—a reduction of approximately 793 thousand patients (-7%). During the same period, there was also a net outflow of GPs. These reductions in the number of doctors contributed to an increase in the number of patients without GP by around 416 thousand patients (supply-side effect). Additionally, there was a progressive reduction in the overall number of patients with a GP per physician (patient-to-GP ratio, capturing a productivity effect). This effect was sufficient to counteract the decline in the number of enrolled patients.

Enrolled patients in primary care (demand-side effect)

The initial decline in the number of patients enrolled in primary care, followed by a subsequent increase from 2016 onward, can be attributed to various factors. Demographic shifts and migratory patterns, marked by low birth rates [ 33 ], increased mortality, and stagnant immigration, suggest a foreseeable reduction in enrolled patients in the medium term. Conversely, changes in socio-economic conditions may prompt residents to seek NHS care due to deteriorating health and economic circumstances that hinder the affordability of private healthcare.

Overall, there was a 2.1% population decline from the 2011 to 2021 census. Interestingly, the pandemic may have prompted individuals not previously registered in primary care to seek enrollment, driven by the need for COVID-19 vaccination. Additionally, the recent increase in migration and telecommuting may further contribute to an increase in the number of users.

Moreover, administrative measures implemented in the context of the financial crisis removed non-users from primary care. In fact, between 2012 and 2013, the number of enrolled patients in primary care was reduced by over 800 thousand, largely due to such reset on administrative records. The fact that the number of enrolled patients in primary care is still above the overall Portuguese population, suggests the need to further improve the accuracy of existing official registries.

Number of full-time equivalent GPs in primary care (supply-side effect)

The mathematical decomposition reveals that the hiring of GPs, in 2015 to 2018, 2020, and 2023, significantly contributed to reducing the number of patients without a GP. However, these new hires were insufficient to assign a doctor to all patients. This may be related to where new hires are being placed. If these were concentrated in practices with a significant shortage of doctors, such as those with small teams at risk of closure due to a lack of professionals, then these hirings, while important for meeting the respective population's needs, did little to resolve the "chronic" lists of patients without a GP in large practices serving vast populations in suburban regions, for example.

Historically spatial maldistribution persists in Portugal despite increased physician supply [ 34 ]. The maps presented in the appendix (Fig. A1) highlight the dynamic evolution in the ratio of patients lacking a designated GP, within each local group of practices (ACES). This ratio ranged between 0 and 46% across different local areas. Studying the patterns within each local group reveals geographical disparities, despite temporal fluctuations. Some ACES consistently grapple with a higher prevalence of patients lacking access to a GP relative to their population size, compared to others. However, our findings indicate only a weak correlation (correlation coefficient = 0.05) between the ratio of patients without a GP and the number of doctors per district. This suggests that the maldistribution of physicians may not be the primary factor contributing to the observed disparities between regions.

Additionally, it is crucial to understand which practices attracted applications. A well-known problem in many countries, including Portugal, is the retention of doctors in remote or underserved areas. Isolated strategies have been implemented, including incentives for medical students to choose family medicine or work in underserved regions, and recruiting foreign physicians through bilateral agreements [ 35 ]. The incentive scheme introduced in 2015 to attract physicians to underserved areas [ 16 ] aimed to enhance practice attractiveness. However, its effectiveness in retaining professionals remains unknown. Potential barriers to hiring doctors include the decreasing real remuneration for physicians in Portugal, with a 21% decrease from 2010 to 2021 [ 23 , 36 ], leading many professionals to seek better-paid work opportunities abroad [ 37 ]. Some authors have also pointed to the insufficient number of places in general and family medicine specialty training. In 2014 around 30% of all vacancies in specialized training for doctors are in family medicine [ 38 ].

Patient-to-GP ratio (productivity effect)

An intriguing finding from this study is associated with the effect of the average number of patients with an assigned GP per GP. The fluctuation in this patient-to-GP ratio, especially in 2016 to 2018, 2020, and 2023, indicates that smaller patient lists might have contributed to the increase in the number of patients without a doctor.

The size of GP patient lists has been the subject of much debate in Portugal. In 2007 the concept of weighted units was introduced, where younger and elder age groups weigh more at the GP list since they represent an increased workload. A weighting factor 1.5 was applied for children aged 0 to 6 years, 2 for adults between 65 and 74 years and a weighting factor 2.5 for adults aged 75 and over years old. In 2007, the minimum list size by each GP was defined as 1,917 weighted units, which corresponded to an average of 1,550 patients in the list [ 39 ]. The State Budget Law for 2021 [ 40 ] limited patient lists to a maximum of 1,917 patient-weighted units for new GP. While this measure creates the right incentives for providing better care, it does not contribute to reducing the number of patients without a GP. The average panel size lies around 1 700 patients per GP, but many extreme situations exist where GP lists are composed of less than 1 000 patients and more than 2 000 patients [ 41 ]. This panel size is in line with other countries rules (Denmark 1 600, England between 1 807 and 2 686, Norway up to 2 500 patients).

On the one hand, migratory movements and demographic changes suggest an increase on doctors' lists of the elderly population and migrant pregnant women [ 41 ]. Since risk adjustment was based solely on the age of the population on their lists, the surge in the population with greater care and assistance needs, such as the elderly, is reflected in the number of patients weighted units. In fact, the number of weighted units per enrolled patient has been increasing over time. Our data show that, in 2009, there were 1.24 weighted units per enrolled patient, which increased by 5% to 1.32 in 2023. This implies that additional GPs—at least 200—are required to cover the same overall number of patients, just because of demographic changes.

On the other hand, shifting distribution of doctors between organizational models may also play a role. Doctors in Personalized Health Care Units (PHCU) typically have lists with fewer patients but follow more patients without a doctor, while those in Family Health Units (FHU) manage larger lists but provide limited services to the population without a doctor. Thus, if the recent hiring of GPs has placed more professionals in PHCU at the expense of FHU, this will mathematically reduce the average number of patients per doctor. Similarly, exit of physicians with larger lists may also contribute to reduce the patient-to-GP ratio. This is particularly relevant if older physicians that retire have larger patient lists than younger new hires.

Moreover, it is essential to ascertain whether the geographic location chosen by doctors for practice imposes limitations on expanding care to more patients. If doctors are hired in sparsely populated areas, the number of patients to fill their lists is limited.

Overall, this research suggests that the patient-to-GP ratio is a key variable, often ignored in public discussion which tends to focus on demand (enrolled patients) and supply factors (number of GPs). In the context of the workforce crisis in primary care faced by numerous countries [ 26 ], it is important that policy-makers consider the impact of such factor on attaining universal coverage of primary health care. Policies to improve the way such ratio is defined—and exploring alternatives implemented by different countries—may contribute to mitigate the negative effects from the current GP crisis in Europe.

Limitations and further research

While this descriptive study sheds light on the challenges of the expansion of primary care coverage, there are certain limitations that warrant consideration. The level of detail and causal analysis are limited considering the aggregate nature of the data used in this paper. In fact, the unit of observation is each group of primary care practices (ACES), while differences may also occur between GPs working in the same ACES. Further studies could delve into the dynamics of team composition within practices. Another aspect of the current organizational changes to the FHU model coupled with other demographic changes is their impact on the patient list composition.

This study provides a comprehensive analysis of GP coverage dynamics in Portugal. We decompose the variation in the number of patients without a GP in three factors.

Between 2009 and 2023, there was an overall decrease in the number of enrollees in primary care (demand-side effect). During the same period, there was also a net outflow of GP, measured in full-time equivalents (supply-side effect). Additionally, there was a progressive reduction in the patient-to-GP ratio (productivity effect). This effect of a gradual reduction in the number of patients in each doctor’s list was sufficient to counteract the decline in the number of enrolled patients, contributing to increasing the number of patients without GP in recent years.

The study highlights the complexity of managing human resources allocation to achieve optimal health care coverage. A proper understanding of the challenges regarding GP coverage is critical to enhance the efficiency of primary care services. In particular, the study emphasizes the importance of strategic planning, efficient recruitment practices, and ongoing evaluation of the effectiveness of initiatives designed to improve GP coverage.

Availability of data and materials

Data analyzed in this study are available in the Portal da Transparência SNS, available at https://www.sns.gov.pt/transparencia/ , in the BI-CSP dashboards, available at https://bicsp.min-saude.pt/pt/biufs/Paginas/default.aspx , and in the ACSS Report on Access to Health Care 2022, available at https://www.acss.min-saude.pt/wp-content/uploads/2022/09/Relat%C3%B3rio-de-Acesso-2021.pdf

Kringos D, Boerma W, Bourgueil Y, Cartier T, Dedeu T, Hasvold T, Groenewegen P. The strength of primary care in Europe: an international comparative study. Br J Gen Pract. 2013;63(616):742–50. https://doi.org/10.3399/bjgp13X674422 .

Article   Google Scholar  

Zhang X, Lin D, Pforsich H, Lin VW. Physician workforce in the United States of America: forecasting nationwide shortages. Hum Resour Health. 2020;18(1):1–9.

Article   CAS   Google Scholar  

Scheffler RM, Arnold DR. Projecting shortages and surpluses of doctors and nurses in the OECD: what looms ahead. Health Econ Policy Law. 2019;14(2):274–90.

Article   PubMed   Google Scholar  

World Health Organization. Health and care workforce in Europe: time to act. Geneva: WHO; 2022.

Google Scholar  

Liu JX, Goryakin Y, Maeda A, Bruckner T, Scheffler R. Global health workforce labor market projections for 2030. Hum Res Health. 2017;15(1):11. https://doi.org/10.1186/s12960-017-0187-2 .

Lorkowski J, Jugowicz A. Shortage of physicians: a critical review. Medical Research and Innovation, 2021; 57–62.

Reinhardt UE. Does the aging of the population really drive the demand for health care? Health Aff. 2003;22(6):27–39.

Petterson SM, Liaw WR, Phillips RL, Rabin DL, Meyers DS, Bazemore AW. Projecting US primary care physician workforce needs: 2010–2025. Annals Family Med. 2012;10(6):503–9.

Janson AL, Moen A, Fuglerud KS. Design of the capable health empowerment tool: Citizens’ needs and expectations. Stud Health Technol Inf. 2020;270:926–30.

Boulos MNK, Wheeler S, Tavares C, Jones R. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng Online. 2011;10(1):1–14.

Pestana J, Frutuoso J, Costa E, Fonseca F. Heterogeneity in physician’s job preferences in a dual practice context—evidence from a DCE. Soc Sci Med. 2024;343: 116551.

Sinnott C, Dorban-Hall B, Dixon-Woods M. Tackling the crisis in general practice. BMJ. 2023;381:p966. https://doi.org/10.1136/bmj.p966 .

Article   PubMed   PubMed Central   Google Scholar  

Verma P, Ford JA, Stuart A, Howe A, Everington S, Steel N. A systematic review of strategies to recruit and retain primary care doctors. BMC Health Serv Res. 2016. https://doi.org/10.1186/s12913-016-1370-1 .

Chen-Xu J, Miranda Castilho B, Moura Fernandes B, Silva Gonçalves D, Ferreira A, Gonçalves AC, Ferreira Vieira M, Silva AM, Borges F, Paes MM. Medical residency in Portugal: a cross-sectional study on the working conditions. Front Health Serv. 2023;5(3):1190357.

European Observatory on Health Systems and Policies. 2023. Portugal: Country Health Profile 2023.

da Saúde M. (2015). Decree-Law 101/2015. Lisboa, Portugal. https://diariodarepublica.pt/dr/detalhe/decreto-lei/101-2015-67401245

dos Médicos O. 2023. Estatísticas de médicos inscritos na Ordem dos Médicos. Accessed 18 June 2024.

Ferreira M, Lopes A, Guimarães M, Barros H. The medical career and the key factors driving the exodus of doctors from the national health service in Portugal. Acta Med Port. 2018;31:483–8.

Martins MJ, Laíns I, Brochado B, Oliveira-Santos M, Teixeira PP, Brandão M, et al. Career satisfaction of medical residents in Portugal. Acta Med Port. 2015;28:209–21.

Ramos P, Alves H. Migration intentions among Portuguese junior doctors: results from a survey. Health Policy. 2017;121:1208–14. https://doi.org/10.1016/j.healthpol.2017.09.016 .

Ordem dos Médicos. Emigração de médicos disparou em 2021. 5 May 2022. https://ordemdosmedicos.pt/emigracao-de-medicos-disparou-em-2021/ . Assessed 18 Jun 2024

Público. Mais de 450 médicos pediram declaração para emigrar este ano. Público. 18 Dez 2023. https://www.publico.pt/2023/12/18/sociedade/noticia/450-medicos-pediram-declaracao-emigrar-ano-2074068 . Assessed 18 June 2024

Barros, P. B., Costa, E. (2022). Recursos Humanos em Saúde. Social Equity Initiative. Nova school of business and economics. https://www.novasbe.unl.pt/Portals/0/Files/Social%20Equity%20Initiative/Nova%20SBE_KC%20Health_Recursos%20Humanos_2022.pdf . Assessed 18 June 2024.

Caeiro T. Concurso para medicos de família—mais de 80% das vagas ficaram por preencher na região de Lisboa. Observador. [Internet]. 26 May 2023. https://observador.pt/2023/05/26/concurso-para-medicos-de-familia-mais-de-80-das-vagas-ficaram-por-preencher-na-regiao-de-lisboa/ . Assessed 18 Jun 2024.

Maia A. Concurso para novos médicos de família teve 143 candidatos admitidos. Havia vaga para 900. Público. 5 Jan 2024. Assessed in 18 June 2024. https://www.publico.pt/2024/01/05/sociedade/noticia/concurso-novos-medicos-familia-143-candidatos-admitidos-vaga-900-2075844

Russo G, Perelman J, Zapata T, Šantrić-Milićević M. The layered crisis of the primary care medical workforce in the European region: what evidence do we need to identify causes and solutions? Hum Resour Health. 2023;21(1):55.

Simões JA, Augusto GF, Fronteira I, Hernández-Quevedo C. Portugal: health system review. Health Syst Transit. 2017;19(2):1–184.

Instituto Nacional de Estatísitca. Conta Satélite da Saúde. 2023.

Dimitrovová K, Perelman J, Serrano-Alarcón M. Effect of a national primary care reform on avoidable hospital admissions (2000–2015): a difference-in-difference analysis. Soc Sci Med. 2020;252:112908. https://doi.org/10.1016/j.socscimed.2020.112908 .

BI-CSP dashboards. Database. https://bicsp.min-saude.pt/pt/biufs/Paginas/default.aspx . Accessed 5 Feb 2024.

ACSS (2022) Relatório Anual de Acesso a Cuidados de Saúde nos Estabelecimentos do SNS e Entidades Convencionadas de 2021, https://www.acss.min-saude.pt/wp-content/uploads/2022/09/Relat%C3%B3rio-de-Acesso-2021.pdf

Maricoto T, Nogueira R. A new formula for managing family doctors’ patient list in Portugal. Cien Saude Colet. 2021;26(suppl 1):2449–58. https://doi.org/10.1590/1413-81232021266.1.40852020 .

Statistics Portugal, Demographic indicators, Total fertility rate (No.) by Place of residence; available at http://smi.ine.pt/Indicador/Detalhes/12127?LANG=EN ; visited in February 2024.

Correia I, Veiga P. Geographic distribution of physicians in Portugal. Eur J Health Econ. 2010;11(4):383–93. https://doi.org/10.1007/s10198-009-0208-8 .

de Oliveira APC, Dussault G, Craveiro I. Challenges and strategies to improve the availability and geographic accessibility of physicians in Portugal. Hum Res Health. 2017;15(1):24. https://doi.org/10.1186/s12960-017-0194-3 .

OECD. OECD economic surveys: Portugal 2023. Org Econ Co-Oper Dev. 2023. https://doi.org/10.1787/2b8ee40a-en .

Ferreira PL, Raposo V, Tavares AI, Correia T. Drivers for emigration among healthcare professionals: testing an analytical model in a primary healthcare setting. Health Policy. 2020;124(7):751–7.

Costa A, Pedro R, Magalhães S. Medicina Geral e Familiar em Portugal [General and family medicine in Portugal]. Revista Brasileira de Medicina de Família e Comunidade. 2014;9(32):295–9. https://doi.org/10.5712/rbmfc9(32)935 .

Ministério da Saúde. Decree-Law 298/2007. Lisboa, Portugal. 2007. https://diariodarepublica.pt/dr/detalhe/decreto-lei/298-2007-640665

Orçamento do Estado para 2021, Lei n.º 75-B/2020, Diário da República n.º 253/2020, 1º Suplemento, Série I de 2020–12–31. https://diariodarepublica.pt/dr/legislacao-consolidada/lei/2020-152803680 . Visited in February 2024.

Pinto D. Impact of ageing of the population on the list size of family physicians. Revista Portuguesa de Medicina Geral e Familiar. 2014;30(5):338–9.

Download references

This work is financed by Portuguese funds through the FCT—Foundation for Science and Technology, I.P., under the project UIDB/00097/2020 (CEGIST). This work was funded by Fundação para a Ciência e a Tecnologia (UIDB/00124/2020, UIDP/00124/2020 and Social Sciences DataLab - PINFRA/22209/2016), POR Lisboa and POR Norte (Social Sciences DataLab, PINFRA/22209/2016).

Author information

Authors and affiliations.

CEGIST - Centre for Management Studies, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001, Lisbon, Portugal

Eduardo Costa

Nova School of Business and Economics, NOVA University Lisboa, Campus de Carcavelos, Rua da Holanda 1, 2775-405, Carcavelos, Portugal

Joana Pestana & Pedro Pita Barros

You can also search for this author in PubMed   Google Scholar

Contributions

EC: conceptualization, methodology, formal analysis, visualization, writing—original draft, writing—review and editing; JP: formal analysis, data curation, visualization, writing—original draft, writing—review and editing; PB: conceptualization, methodology, writing—review and editing.

Corresponding author

Correspondence to Eduardo Costa .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Tables 1 and 2 and Figs. 6 , 7 , 8 , 9 , 10 and 11 here.

Distribution of patients without GP across regions (% of enrolled patients in primary care)

Change in the volume of enrolled patients in primary care units (2009 = 100; 2009—2023)

Change in the number of GPs in primary care units (2009 = 100; 2009—2023)

Relationship between the variation in the average number of patients per GP in each ACES (horizontal axis) with changes in the proportion of elderly patients (vertical axis) (2012 versus 2023; %)

Relationship between the variation in the average number of patients per GP in each ACES (horizontal axis) with changes in the prevalence of diabetic patients (vertical axis) (2012 versus 2023; %)

Relationship between the variation in the average number of patients per GP in each ACES (horizontal axis) with population density (vertical axis) (2012 versus 2023; %)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Costa, E., Pestana, J. & Barros, P.P. Primary health care coverage in Portugal: the promise of a general practitioner for all. Hum Resour Health 22 , 55 (2024). https://doi.org/10.1186/s12960-024-00936-7

Download citation

Received : 28 March 2024

Accepted : 26 June 2024

Published : 09 August 2024

DOI : https://doi.org/10.1186/s12960-024-00936-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Primary care
• General practice
• Patient lists
• Family doctor
• Portuguese NHS

Human Resources for Health

ISSN: 1478-4491

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Introduction
• Conclusions
• Article Information

Outcomes are estimated from bivariate and multivariable generalized estimating equation models. aOR, indicates adjusted odds ratio; GAD-7, Generalized Anxiety Disorder 7-item scale; PHQ-9, Patient Health Questionnaire 9-item scale; whiskers, 95% CIs.

eTable 1. Survey Instruments

eTable 2. Prevalence of Exposure Over Time

eTable 3. Prevalence of Outcomes Over Time by Exposure Group

eTable 4. E-Value Calculation for Association Between Puberty Blockers or Gender-Affirming Hormones and Mental Health Outcomes

eTable 5. Examining Association Between Puberty Blockers or Gender-Affirming Hormones and Mental Health Outcomes Separately

eTable 6. Bivariate Model Restricted to Youths Ages 13 to 17 Years

eTable 7. Multivariable Model Restricted to 90 Youths Ages 13 to 17 Years

eTable 8. Sensitivity Analyses using Patient Health Questionnaire 8-item Scale Score of 10 or Greater for Moderate to Severe Depression

eFigure 1. Schematic of Generalized Estimating Equation Model

eFigure 2. Association Between Receipt of Gender-Affirming Hormones or Puberty Blockers and Mental Health Outcomes

eReferences

• Medical Groups Defend Patient-Physician Relationship and Access to Adolescent Gender-Affirming Care JAMA Medical News & Perspectives April 19, 2022 This Medical News article discusses physicians’ advocacy to protect patients and the patient-physician relationship amid efforts by politicians to limit access or criminalize gender-affirming care. Bridget M. Kuehn, MSJ
• As Laws Restricting Health Care Surge, Some US Physicians Choose Between Fight or Flight JAMA Medical News & Perspectives June 13, 2023 In this Medical News article, 13 physicians and health care experts spoke with JAMA about the increasing efforts to criminalize evidence-based medical care in the US. Melissa Suran, PhD, MSJ
• Data Errors in eTables 2 and 3 JAMA Network Open Correction July 26, 2022
• Improving Mental Health Among Transgender and Gender-Diverse Youth JAMA Network Open Invited Commentary February 25, 2022 Brett Dolotina, BS; Jack L. Turban, MD, MHS

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Tordoff DM , Wanta JW , Collin A , Stepney C , Inwards-Breland DJ , Ahrens K. Mental Health Outcomes in Transgender and Nonbinary Youths Receiving Gender-Affirming Care. JAMA Netw Open. 2022;5(2):e220978. doi:10.1001/jamanetworkopen.2022.0978

Manage citations:

© 2024

• Permissions

Mental Health Outcomes in Transgender and Nonbinary Youths Receiving Gender-Affirming Care

• 1 Department of Epidemiology, University of Washington, Seattle
• 2 Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle
• 3 School of Medicine, University of Washington, Seattle
• 4 Department of Psychiatry and Behavioral Medicine, Department of Adolescent and Young Adult Medicine, Seattle Children’s Hospital, Seattle, Washington
• 5 University of California, San Diego School of Medicine, Rady Children's Hospital
• 6 Division of Adolescent Medicine, Department of Pediatrics, Seattle Children’s Hospital, Seattle, Washington
• Invited Commentary Improving Mental Health Among Transgender and Gender-Diverse Youth Brett Dolotina, BS; Jack L. Turban, MD, MHS JAMA Network Open
• Medical News & Perspectives Medical Groups Defend Patient-Physician Relationship and Access to Adolescent Gender-Affirming Care Bridget M. Kuehn, MSJ JAMA
• Medical News & Perspectives As Laws Restricting Health Care Surge, Some US Physicians Choose Between Fight or Flight Melissa Suran, PhD, MSJ JAMA
• Correction Data Errors in eTables 2 and 3 JAMA Network Open

Question   Is gender-affirming care for transgender and nonbinary (TNB) youths associated with changes in depression, anxiety, and suicidality?

Findings   In this prospective cohort of 104 TNB youths aged 13 to 20 years, receipt of gender-affirming care, including puberty blockers and gender-affirming hormones, was associated with 60% lower odds of moderate or severe depression and 73% lower odds of suicidality over a 12-month follow-up.

Meaning   This study found that access to gender-affirming care was associated with mitigation of mental health disparities among TNB youths over 1 year; given this population's high rates of adverse mental health outcomes, these data suggest that access to pharmacological interventions may be associated with improved mental health among TNB youths over a short period.

Importance   Transgender and nonbinary (TNB) youths are disproportionately burdened by poor mental health outcomes owing to decreased social support and increased stigma and discrimination. Although gender-affirming care is associated with decreased long-term adverse mental health outcomes among these youths, less is known about its association with mental health immediately after initiation of care.

Objective   To investigate changes in mental health over the first year of receiving gender-affirming care and whether initiation of puberty blockers (PBs) and gender-affirming hormones (GAHs) was associated with changes in depression, anxiety, and suicidality.

Design, Setting, and Participants   This prospective observational cohort study was conducted at an urban multidisciplinary gender clinic among TNB adolescents and young adults seeking gender-affirming care from August 2017 to June 2018. Data were analyzed from August 2020 through November 2021.

Exposures   Time since enrollment and receipt of PBs or GAHs.

Main Outcomes and Measures   Mental health outcomes of interest were assessed via the Patient Health Questionnaire 9-item (PHQ-9) and Generalized Anxiety Disorder 7-item (GAD-7) scales, which were dichotomized into measures of moderate or severe depression and anxiety (ie, scores ≥10), respectively. Any self-report of self-harm or suicidal thoughts over the previous 2 weeks was assessed using PHQ-9 question 9. Generalized estimating equations were used to assess change from baseline in each outcome at 3, 6, and 12 months of follow-up. Bivariate and multivariable logistic models were estimated to examine temporal trends and investigate associations between receipt of PBs or GAHs and each outcome.

Results   Among 104 youths aged 13 to 20 years (mean [SD] age, 15.8 [1.6] years) who participated in the study, there were 63 transmasculine individuals (60.6%), 27 transfeminine individuals (26.0%), 10 nonbinary or gender fluid individuals (9.6%), and 4 youths who responded “I don’t know” or did not respond to the gender identity question (3.8%). At baseline, 59 individuals (56.7%) had moderate to severe depression, 52 individuals (50.0%) had moderate to severe anxiety, and 45 individuals (43.3%) reported self-harm or suicidal thoughts. By the end of the study, 69 youths (66.3%) had received PBs, GAHs, or both interventions, while 35 youths had not received either intervention (33.7%). After adjustment for temporal trends and potential confounders, we observed 60% lower odds of depression (adjusted odds ratio [aOR], 0.40; 95% CI, 0.17-0.95) and 73% lower odds of suicidality (aOR, 0.27; 95% CI, 0.11-0.65) among youths who had initiated PBs or GAHs compared with youths who had not. There was no association between PBs or GAHs and anxiety (aOR, 1.01; 95% CI, 0.41, 2.51).

Conclusions and Relevance   This study found that gender-affirming medical interventions were associated with lower odds of depression and suicidality over 12 months. These data add to existing evidence suggesting that gender-affirming care may be associated with improved well-being among TNB youths over a short period, which is important given mental health disparities experienced by this population, particularly the high levels of self-harm and suicide.

Transgender and nonbinary (TNB) youths are disproportionately burdened by poor mental health outcomes, including depression, anxiety, and suicidal ideation and attempts. 1 - 5 These disparities are likely owing to high levels of social rejection, such as a lack of support from parents 6 , 7 and bullying, 6 , 8 , 9 and increased stigma and discrimination experienced by TNB youths. Multidisciplinary care centers have emerged across the country to address the health care needs of TNB youths, which include access to medical gender-affirming interventions, such as puberty blockers (PBs) and gender-affirming hormones (GAHs). 10 These centers coordinate care and help youths and their families address barriers to care, such as lack of insurance coverage 11 and travel times. 12 Gender-affirming care is associated with decreased rates of long-term adverse outcomes among TNB youths. Specifically, PBs, GAHs, and gender-affirming surgeries have all been found to be independently associated with decreased rates of depression, anxiety, and other adverse mental health outcomes. 13 - 16 Access to these interventions is also associated with a decreased lifetime incidence of suicidal ideation among adults who had access to PBs during adolescence. 17 Conversely, TNB youths who present to care later in adolescence or young adulthood experience more adverse mental health outcomes. 18 Despite this robust evidence base, legislation criminalizing and thus limiting access to gender-affirming medical care for minors is increasing. 19 , 20

Less is known about the association of gender-affirming care with mental health outcomes immediately after initiation of care. Several studies published from 2015 to 2020 found that receipt of PBs or GAHs was associated with improved psychological functioning 21 and body satisfaction, 22 as well as decreased depression 23 and suicidality 24 within a 1-year period. Initiation of gender-affirming care may be associated with improved short-term mental health owing to validation of gender identity and clinical staff support. Conversely, prerequisite mental health evaluations, often perceived as pathologizing by TNB youths, and initiation of GAHs may present new stressors that may be associated with exacerbation of mental health symptoms early in care, such as experiences of discrimination associated with more frequent points of engagement in a largely cisnormative health care system (eg, interactions with nonaffirming pharmacists to obtain laboratory tests, syringes, and medications). 25 Given the high risk of suicidality among TNB adolescents, there is a pressing need to better characterize mental health trends for TNB youths early in gender-affirming care. This study aimed to investigate changes in mental health among TNB youths enrolled in an urban multidisciplinary gender clinic over the first 12 months of receiving care. We also sought to investigate whether initiation of PBs or GAHs was associated with depression, anxiety, and suicidality.

This cohort study received approval from the Seattle Children’s Hospital Institutional Review Board. For youths younger than age 18 years, caregiver consent and youth assent was obtained. For youths ages 18 years and older, youth consent alone was obtained. The 12-month assessment was funded via a different mechanism than other survey time points; thus, participants were reconsented for the 12-month survey. The study follows the Strengthening the Reporting of Observational Studies in Epidemiology ( STROBE ) reporting guideline.

We conducted a prospective observational cohort study of TNB youths seeking care at Seattle Children’s Gender Clinic, an urban multidisciplinary gender clinic. After a referral is placed or a patient self-refers, new patients, their caregivers, or patients with their caregivers are scheduled for a 1-hour phone intake with a care navigator who is a licensed clinical social worker. Patients are then scheduled for an appointment at the clinic with a medical provider.

All patients who completed the phone intake and in-person appointment between August 2017 and June 2018 were recruited for this study. Participants completed baseline surveys within 24 hours of their first appointment and were invited to complete follow-up surveys at 3, 6, and 12 months. Youth surveys were used to assess most variables in this study; caregiver surveys were used to assess caregiver income. Participation and completion of study surveys had no bearing on prescribing of PBs or GAHs.

We assessed 3 internalizing mental health outcomes: depression, generalized anxiety, and suicidality. Depression was assessed using the Patient Health Questionnaire 9-item scale (PHQ-9), and anxiety was assessed using the Generalized Anxiety Disorder 7-item scale (GAD-7). We dichotomized PHQ-9 and GAD-7 scores into measures of moderate or severe depression and anxiety (ie, scores ≥10). 26 , 27 Self-harm and suicidal thoughts were assessed using PHQ-9 question 9 (eTable 1 in the Supplement ).

Participants self-reported if they had ever received GAHs, including estrogen or testosterone, or PBs (eg, gonadotropin-releasing hormone analogues) on each survey. We conducted a medical record review to capture prescription of androgen blockers (eg, spironolactone) and medications for menstrual suppression or contraception (ie, medroxyprogesterone acetate or levonorgestrel-releasing intrauterine device) during the study period.

We a priori considered potential confounders hypothesized to be associated with our exposures and outcomes of interest based on theory and prior research. Self-reported gender was ascertained on each survey using a 2-step question that asked participants about their current gender and their sex assigned at birth. If a participant’s self-reported gender changed across surveys, we used the gender reported most frequently by a participant (3 individuals identified as transmasculine at baseline and as nonbinary on all follow-up surveys). We collected data on self-reported race and ethnicity (available response options were Arab or Middle Eastern; Asian; Black or African American; Latinx; Native American, American Indian, or Alaskan Native or Native Hawaiian; Pacific Islander; and White), age, caregiver income, and insurance type. Race and ethnicity were assessed as potential covariates owing to known barriers to accessing gender-affirming care among transgender youth who are members of minority racial and ethnic groups. For descriptive statistics, Asian and Pacific Islander groups were combined owing to small population numbers. We included a baseline variable reflecting receipt of ongoing mental health therapy other than for the purpose of a mental health assessment to receive a gender dysphoria diagnosis. We included a self-report variable reflecting whether youths felt their gender identity or expression was a source of tension with their parents or guardians. Substance use included any alcohol, marijuana, or other drug use in the past year. Resilience was measured by the Connor-Davidson Resilience Scale (CD-RISC) 10-item score developed to measure change in an individual’s state resilience over time. 28 Resilience scores were dichotomized into high (ie, ≥median) and low (ie, <median). Prior studies of young adults in the US reported mean CD-RISC scores ranging from 27.2 to 30.1. 29 , 30

We used generalized estimating equations to assess change in outcomes from baseline at each follow-up point (eFigure 1 in the Supplement ). We used a logit link function to estimate adjusted odds ratio (aOR) for the association between variables and each mental health outcome. We initially estimated bivariate associations between potential confounders and mental health outcomes. Multivariable models included variables that were statistically significant in bivariate models. For all outcomes and models, statistical significance was defined as 95% CIs that did not contain 1.00. Reported P values are based on 2-sided Wald test statistics.

Model 1 examined temporal trends in mental health outcomes, with time (ie, baseline, 3, 6, and 12 months) modeled as a categorical variable. Model 2 estimated the association between receipt of PBs or GAHs and mental health outcomes adjusted for temporal trends and potential confounders. Receipt of PBs or GAHs was modeled as a composite binary time-varying exposure that compared mean outcomes between participants who had initiated PBs or GAHs and those who had not across all time points (eTable 2 in the Supplement ). All models used an independent working correlation structure and robust standard errors to account for the time-varying exposure variable.

We performed several sensitivity analyses. Because our data were from an observational cohort, we first considered the degree to which they were sensitive to unmeasured confounding. To do this, we calculated the E-value for the association between PBs or GAHs and mental health outcomes in model 2. The E-value is defined as the minimum strength of association that a confounder would need to have with both exposure and outcome to completely explain away their association (eTable 4 in the Supplement ). 31 Second, we performed sensitivity analyses on several subsets of youths. We separately examined the association of PBs and GAHs with outcomes of interest, although we a priori did not anticipate being powered to detect statistically significant outcomes owing to our small sample size and the relatively low proportion of youths who accessed PBs. We also conducted sensitivity analyses using the Patient Health Questionnaire 8-item scale (PHQ-8), in which the PHQ-9 question 9 regarding self-harm or suicidal thoughts was removed, given that we analyzed this item as a separate outcome. Lastly, we restricted our analysis to minor youths ages 13 to 17 years because they were subject to different laws and policies related to consent and prerequisite mental health assessments. We used R statistical software version 3.6.2 (R Project for Statistical Computing) to conduct all analyses. Data were analyzed from August 2020 through November 2021.

A total of 169 youths were screened for eligibility during the study period, among whom 161 eligible youths were approached. Nine youths or caregivers declined participation, and 39 youths did not complete consent or assent or did not complete the baseline survey, leaving a sample of 113 youths (70.2% of approached youths). We excluded 9 youths aged younger than 13 years from the analysis because they received different depression and anxiety screeners. Our final sample included 104 youths ages 13 to 20 years (mean [SD] age, 15.8 [1.6] years). Of these individuals, 84 youths (80.8%), 84 youths, and 65 youths (62.5%) completed surveys at 3, 6, and 12 months, respectively.

Our cohort included 63 transmasculine youths (60.6%), 27 transfeminine youths (26.0%), 10 nonbinary or gender fluid youths (9.6%), and 4 youths who responded “I don’t know” or did not respond to the gender identity question on all completed questionnaires (3.8%) ( Table 1 ). There were 4 Asian or Pacific Islander youths (3.8%), 3 Black or African American youths (2.9%); 9 Latinx youths (8.7%); 6 Native American, American Indian, or Alaskan Native or Native Hawaiian youths (5.8%); 67 White youths (64.4%); and 9 youths who reported more than 1 race or ethnicity (8.7%). Race and ethnicity data were missing for 6 youth (5.8%).

At baseline, 7 youths had ever received PBs or GAHs (including 1 youth who received PBs, 4 youths who received GAHs, and 2 youths who received both PBs and GAHs). By the end of the study, 69 youths (66.3%) had received PBs or GAHs (including 50 youths who received GAHs only [48.1%], 5 youths who received PBs only [4.8%], and 14 youths who received PBs and GAHs [13.5%]), while 35 youths had not received either PBs or GAHs (33.7%) (eTable 3 in the Supplement ). Among 33 participants assigned male sex at birth, 17 individuals (51.5%) had received androgen blockers, and among 71 participants assigned female sex at birth, 25 individuals (35.2%) had received menstrual suppression or contraceptives by the end of the study.

A large proportion of youths reported depressive and anxious symptoms at baseline. Specifically, 59 individuals (56.7%) had baseline PHQ-9 scores of 10 or more, suggesting moderate to severe depression; there were 22 participants (21.2%) scoring in the moderate range, 11 participants (10.6%) in the moderately severe range, and 26 participants (25.0%) in the severe range. Similarly, half of participants had a GAD-7 score suggestive of moderate to severe anxiety at baseline (52 individuals [50.0%]), including 20 participants (19.2%) scored in the moderate range, and 32 participants (30.8%) scored in the severe range. There were 45 youths (43.3%) who reported self-harm or suicidal thoughts in the prior 2 weeks. At baseline, 65 youths (62.5%) were receiving ongoing mental health therapy, 36 youths (34.6%) reported tension with their caregivers about their gender identity or expression, and 34 youths (32.7%) reported any substance use in the prior year. Lastly, we observed a wide range of resilience scores (median [range], 22.5 [1-38], with higher scores equaling more resiliency). There were no statistically significant differences in baseline characteristics by gender.

In bivariate models, substance use was associated with all mental health outcomes ( Table 2 ). Youths who reported any substance use were 4-fold as likely to have PHQ-9 scores of moderate to severe depression (aOR, 4.38; 95% CI, 2.10-9.16) and 2-fold as likely to have GAD-7 scores of moderate to severe anxiety (aOR, 2.07; 95% CI, 1.04-4.11) or report thoughts of self-harm or suicide in the prior 2 weeks (aOR, 2.06; 95% CI, 1.08-3.93). High resilience scores (ie, ≥median), compared with low resilience scores (ie, <median), were associated with lower odds of moderate or severe anxiety (aOR, 0.51; 95% CI, 0.26-0.999).

There were no statistically significant temporal trends in the bivariate model or model 1 ( Table 2 and Table 3 ). However, among all participants, odds of moderate to severe depression increased at 3 months of follow-up relative to baseline (aOR, 2.12; 95% CI, 0.98-4.60), which was not a significant increase, and returned to baseline levels at months 6 and 12 ( Figure ) prior to adjusting for receipt of PBs or GAHs.

We also examined the association between receipt of PBs or GAHs and mental health outcomes in bivariate and multivariable models (eFigure 2 in the Supplement ). After adjusting for temporal trends and potential confounders ( Table 4 ), we observed that youths who had initiated PBs or GAHs had 60% lower odds of moderate to severe depression (aOR, 0.40; 95% CI, 0.17-0.95) and 73% lower odds of self-harm or suicidal thoughts (aOR, 0.27; 95% CI, 0.11-0.65) compared with youths who had not yet initiated PBs or GAHs. There was no association between receipt of PBs or GAHs and moderate to severe anxiety (aOR, 1.01; 95% CI, 0.41-2.51). After adjusting for time-varying exposure of PBs or GAHs in model 2 ( Table 4 ), we observed statistically significant increases in moderate to severe depression among youths who had not received PBs or GAHs by 3 months of follow-up (aOR, 3.22; 95% CI, 1.37-7.56). A similar trend was observed for self-harm or suicidal thoughts among youths who had not received PBs or GAHs by 6 months of follow-up (aOR, 2.76; 95% CI, 1.22-6.26). Lastly, we estimated E-values of 2.56 and 3.25 for the association between receiving PGs or GAHs and moderate to severe depression and suicidality, respectively (eTable 4 in the Supplement ). Sensitivity analyses obtained comparable results and are presented in eTables 5 through 8 in the Supplement .

In this prospective clinical cohort study of TNB youths, we observed high rates of moderate to severe depression and anxiety, as well as suicidal thoughts. Receipt of gender-affirming interventions, specifically PBs or GAHs, was associated with 60% lower odds of moderate to severe depressive symptoms and 73% lower odds of self-harm or suicidal thoughts during the first year of multidisciplinary gender care. Among youths who did not initiate PBs or GAHs, we observed that depressive symptoms and suicidality were 2-fold to 3-fold higher than baseline levels at 3 and 6 months of follow-up, respectively. Our study results suggest that risks of depression and suicidality may be mitigated with receipt of gender-affirming medications in the context of a multidisciplinary care clinic over the relatively short time frame of 1 year.

Our findings are consistent with those of prior studies finding that TNB adolescents are at increased risk of depression, anxiety, and suicidality 1 , 11 , 32 and studies finding long-term and short-term improvements in mental health outcomes among TNB individuals who receive gender-affirming medical interventions. 14 , 21 - 24 , 33 , 34 Surprisingly, we observed no association with anxiety scores. A recent cohort study of TNB youths in Dallas, Texas, found that total anxiety symptoms improved over a longer follow-up of 11 to 18 months; however, similar to our study, the authors did not observe statistically significant improvements in generalized anxiety. 22 This suggests that anxiety symptoms may take longer to improve after the initiation of gender-affirming care. In addition, Olson et al 35 found that prepubertal TNB children who socially transitioned did not have increased rates of depression symptoms but did have increased rates of anxiety symptoms compared with children who were cisgender. Although social transition and access to gender-affirming medical care do not always go hand in hand, it is noteworthy that access to gender-affirming medical care and supported social transition appear to be associated with decreased depression and suicidality more than anxiety symptoms.

Time trends were not significant in our study; however, it is important to note that we observed a transient and nonsignificant worsening in mental health outcomes in the first several months of care among all participants and that these outcomes subsequently returned to baseline by 12 months. This is consistent with findings from a 2020 study 36 in an academic medical center in the northwestern US that observed no change in TNB adolescents’ GAD-7 or PHQ-9 scores from intake to first follow-up appointment, which occurred a mean of 4.7 months apart. Given that receipt of PBs or GAHs was associated with protection against depression and suicidality in our study, it could be that delays in receipt of medications is associated with initially exacerbated mental health symptoms that subsequently improve. It is also possible that mental health improvements associated with receiving these interventions may have a delayed onset, given the delay in physical changes after starting GAHs.

Few of our hypothesized confounders were associated with mental health outcomes in this sample, most notably receipt of ongoing mental health therapy and caregiver support; however, this is not surprising given that these variables were colinear with baseline mental health, which we adjusted for in all models. Substance use was the only variable associated with all mental health outcomes. In addition, youths with high baseline resilience scores were half as likely to experience moderate to severe anxiety as those with low scores. This finding suggests that substance use and resilience may be additional modifiable factors that could be addressed through multidisciplinary gender-affirming care. We recommend more granular assessment of substance use and resilience to better understand support needs (for substance use) and effective support strategies (for resilience) for TNB youths in future research.

This study has a number of strengths. This is one of the first studies to quantify a short-term transient increase in depressive symptoms experienced by TNB youths after initiating gender-affirming care, a phenomenon observed clinically by some of the authors and described in qualitative research. 37 Although we are unable to make causal statements owing to the observational design of the study, the strength of associations between gender-affirming medications and depression and suicidality, with large aOR values, and sensitivity analyses that suggest that these findings are robust to moderate levels of unmeasured confounding. Specifically, E-values calculated for this study suggest that the observed associations could be explained away only by an unmeasured confounder that was associated with both PBs and GAHs and the outcomes of interest by a risk ratio of 2-fold to 3-fold each, above and beyond the measured confounders, but that weaker confounding could not do so. 31

Our findings should be interpreted in light of the following limitations. This was a clinical sample of TNB youths, and there was likely selection bias toward youths with supportive caregivers who had resources to access a gender-affirming care clinic. Family support and access to care are associated with protection against poor mental health outcomes, and thus actual rates of depression, anxiety, and suicidality in nonclinical samples of TNB youths may differ. Youths who are unable to access gender-affirming care owing to a lack of family support or resources require particular emphasis in future research and advocacy. Our sample also primarily included White and transmasculine youths, limiting the generalizability of our findings. In addition, the need to reapproach participants for consent and assent for the 12-month survey likely contributed to attrition at this time point. There may also be residual confounding because we were unable to include a variable reflecting receipt of psychotropic medications that could be associated with depression, anxiety, and self-harm and suicidal thought outcomes. Additionally, we used symptom-based measures of depression, anxiety, and suicidality; further studies should include diagnostic evaluations by mental health practitioners to track depression, anxiety, gender dysphoria, suicidal ideation, and suicide attempts during gender care. 2

Our study provides quantitative evidence that access to PBs or GAHs in a multidisciplinary gender-affirming setting was associated with mental health improvements among TNB youths over a relatively short time frame of 1 year. The associations with the highest aORs were with decreased suicidality, which is important given the mental health disparities experienced by this population, particularly the high levels of self-harm and suicide. Our findings have important policy implications, suggesting that the recent wave of legislation restricting access to gender-affirming care 19 may have significant negative outcomes in the well-being of TNB youths. 20 Beyond the need to address antitransgender legislation, there is an additional need for medical systems and insurance providers to decrease barriers and expand access to gender-affirming care.

Accepted for Publication: January 10, 2022.

Published: February 25, 2022. doi:10.1001/jamanetworkopen.2022.0978

Correction: This article was corrected on July 26, 2022, to fix minor errors in the numbers of patients in eTables 2 and 3 in the Supplement.

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2022 Tordoff DM et al. JAMA Network Open .

Corresponding Author: Diana M. Tordoff, MPH, Department of Epidemiology, University of Washington, UW Box 351619, Seattle, WA 98195 ( [email protected] ).

Author Contributions : Diana Tordoff had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Diana Tordoff and Dr Wanta are joint first authors. Drs Inwards-Breland and Ahrens are joint senior authors.

Concept and design: Collin, Stepney, Inwards-Breland, Ahrens.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Tordoff, Wanta, Collin, Stepney, Inwards-Breland.

Critical revision of the manuscript for important intellectual content: Wanta, Collin, Stepney, Inwards-Breland, Ahrens.

Statistical analysis: Tordoff.

Obtained funding: Inwards-Breland, Ahrens.

Administrative, technical, or material support: Ahrens.

Supervision: Wanta, Inwards-Breland, Ahrens.

Conflict of Interest Disclosures: Diana Tordoff reported receiving grants from the National Institutes of Health National Institute of Allergy and Infectious Diseases unrelated to the present work and outside the submitted work. No other disclosures were reported.

Funding/Support: This study was supported Seattle Children’s Center for Diversity and Health Equity and the Pacific Hospital Preservation Development Authority.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

• Share full article

Advertisement

Supported by

A Blood Test Accurately Diagnosed Alzheimer’s 90% of the Time, Study Finds

It was much more accurate than primary care doctors using cognitive tests and CT scans. The findings could speed the quest for an affordable and accessible way to diagnose patients with memory problems.

By Pam Belluck

Scientists have made another major stride toward the long-sought goal of diagnosing Alzheimer’s disease with a simple blood test . On Sunday, a team of researchers reported that a blood test was significantly more accurate than doctors’ interpretation of cognitive tests and CT scans in signaling the condition.

The study , published Sunday in the journal JAMA, found that about 90 percent of the time the blood test correctly identified whether patients with memory problems had Alzheimer’s. Dementia specialists using standard methods that did not include expensive PET scans or invasive spinal taps were accurate 73 percent of the time, while primary care doctors using those methods got it right only 61 percent of the time.

“Not too long ago measuring pathology in the brain of a living human was considered just impossible,” said Dr. Jason Karlawish, a co-director of the Penn Memory Center at the University of Pennsylvania who was not involved in the research. “This study adds to the revolution that has occurred in our ability to measure what’s going on in the brain of living humans.”

The results, presented Sunday at the Alzheimer’s Association International Conference in Philadelphia, are the latest milestone in the search for affordable and accessible ways to diagnose Alzheimer’s, a disease that afflicts nearly seven million Americans and over 32 million people worldwide. Medical experts say the findings bring the field closer to a day when people might receive routine blood tests for cognitive impairment as part of primary care checkups, similar to the way they receive cholesterol tests.

“Now, we screen people with mammograms and PSA or prostate exams and other things to look for very early signs of cancer,” said Dr. Adam Boxer, a neurologist at the University of California, San Francisco, who was not involved in the study. “And I think we’re going to be doing the same thing for Alzheimer’s disease and hopefully other forms of neurodegeneration.”

In recent years, several blood tests have been developed for Alzheimer’s. They are currently used mostly to screen participants in clinical trials and by some specialists like Dr. Boxer to help pinpoint if a patient’s dementia is caused by Alzheimer’s or another condition.

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

• Type 2 Diabetes
• Heart Disease
• Digestive Health
• Multiple Sclerosis
• Diet & Nutrition
• Supplements
• Health Insurance
• Public Health
• Patient Rights
• Caregivers & Loved Ones
• End of Life Concerns
• Health News
• Thyroid Test Analyzer
• Doctor Discussion Guides
• Hemoglobin A1c Test Analyzer
• Lipid Test Analyzer
• Complete Blood Count (CBC) Analyzer
• What to Buy
• Editorial Process
• Meet Our Medical Expert Board

Is Urgent Care Replacing Primary Care Physicians?

Jeff Greenberg / Getty Images

Key Takeaways

• A recent survey showed that many Americans trust convenient care clinics or urgent care to provide routine healthcare services that primary care providers typically perform.
• Convenient care clinics are easily accessible to individuals with or without an established primary care provider.
• While convenient care clinics fill gaps in primary care, they are not a substitute for regularly seeing a primary care provider.

A recent survey by RegisteredNursing.org showed that many Americans are turning to convenient care clinics and urgent care for healthcare services typically performed by a primary care provider.

For an overwhelming majority of the 1,004 respondents (77%), the reason why boils down to the immediate availability of care. More than half of respondents (54%) simply want faster service.

The survey results come at a time when primary care providers are in short supply. Many primary care providers (PCPs) have a full load of patients each day, so they may not have open appointments when patients are sick and need to be seen on the same day. They may not be taking new patients at all. One in four survey participants said they don’t have a PCP.

While convenient care clinics focus on the problem at hand, primary care providers say they consider a patient’s health from a broader perspective.

“It’s important to have someone who knows you as a person,” Lori Solomon, MD, MPH , Chair and Clinical Associate Professor of‌ Family and Community Medicine at New York Medical College in Valhalla, New York, told Verywell.

What Are the Alternatives to Primary Care?

Convenient care clinics, often found within popular community locations such as grocery stores or pharmacies, provide walk-in and same-day appointments for routine healthcare. CVS MinuteClinic is one example. Services may include physical exams, vaccines, minor illness or injury care, STI screening, and counseling for chronic diseases like high blood pressure, diabetes, obesity, and depression.

Urgent care centers, on the other hand, can see patients who need same-day care but do not need to be in the emergency room.

Anyone experiencing chest pain, shortness of breath, stroke symptoms, loss of consciousness, or a severe injury should go to the emergency room.

According to the survey, the most common reasons for convenient or urgent care clinic visits include illness or infections, injuries or pain, medical tests, screenings, physical exams, and vaccines.

More Convenient, But Not Always More Preferred

Over half of the respondents say they’ve gone to a walk-in clinic, while 82% have gone to urgent care.

The draw isn’t just appointment availability, but also ease of access. Urgent care clinics alone are growing at a pace of over 7% per year in the U.S.—at the end of 2019, there were 11,481 urgent care centers nationwide. By the end of 2022, that figure hit 14,075.

“People are mobile, and when they are traveling, they are going to see whoever is most readily available wherever they are,” Caryn Bowden, FNP-C , a pediatric nurse practitioner with Wilmington Health in North Carolina, told Verywell.

Meanwhile, primary care providers are dwindling because of reasons ranging from too many patients and too much administrative documentation to low compensation compared to other specialties.

“The American Academy of Family Physicians is committed to increasing the number of family physicians in the United States and ensuring patients have a usual source of care,”  Sarah Sams, MD , family physician and board member of the American Academy of Family Physicians, told Verywell. “We know from research that more than 1 in 10 U.S. children don’t have a primary doctor, and more than 1 in 4 adults don’t have one either.  There has been a nearly 40% jump in the share of U.S. children without a usual source of care over the last decade. This is concerning because we know that patients with a primary care physician have higher rates of preventive care, have improved communication with their care team, and often experience better health outcomes.”

Two-thirds of survey respondents said they would prefer to see their regular doctor over a convenient care or urgent care practitioner. It’s just not always realistic.

The good news is that the quality of care doesn’t seem to suffer. When compared to their primary care provider, 73% of survey respondents felt they received equal or better quality care from a walk-in clinic.

If you go to a convenient care center, you will likely be seen by a nurse practitioner or physician assistant instead of a physician. Each state individually determines regulations regarding nurse practitioner and physician assistant practice. While physicians have more extensive training than nurse practitioners and physician assistants, most communities would be unable to meet their population’s healthcare needs if they relied solely on physicians.

Why You Should See a Primary Care Provider If Possible

A doctor who knows your medical history will more likely accurately diagnose your condition, especially if your problem has been ongoing. An urgent care visit will rule out conditions that need immediate attention, but long-term health issues require more in-depth evaluation.

If you must visit urgent care or a walk-in clinic, you should schedule a follow-up appointment with your primary care provider afterward.

“Primary care providers have ongoing relationships with their patients over the years. That is key to maintaining health and keeping patients out of the hospital,” Solomon said. “Primary care providers are going to get a comprehensive picture and do a complete workup. Turning to a walk-in clinic may be quicker, but in the long run, seeing your primary care provider saves more time and energy.”

Primary care visits are also an opportunity to check that you are up-to-date on other medical tests or care you may need.

“When you utilize quick care, you’re missing out on a lot of preventative care, like mammograms and vaccines, that keep you healthy over the long term,” Sams said.

Some primary care providers practice within larger hospital systems or healthcare networks. If possible, when you need urgent care, visit a walk-in facility within the same organization as your primary care provider. They often use electronic medical records, and your primary care provider can review the notes from your walk-in visit.

“We have a walk-in clinic housed within our office. It is staffed during office hours, and at times, extended hours. Patients can walk in and be seen by one of their doctors’s partners, who will have access to their medical records and history,” Sams said of her primary care practice. “The provider who sees the patient can message or even text the primary care provider during or after the visit.”

How to Find a Reliable Primary Care Provider

Almost one-third of the U.S. population does not have a primary care provider, according to the National Association of Community Health Centers . These individuals may be uninsured, live in a community with few primary care providers, or be healthy and not see the need for routine medical care.

If you are looking for a primary care provider, Bowden recommends selecting a convenient care clinic that is affiliated with a larger hospital or healthcare network.

“Those providers can say, ‘You need a follow-up, let’s get you into one of our providers.’ Stand-alone clinics don’t have those peer-to-peer relationships,” she said. Plus, walk-in clinics that are part of larger healthcare systems often use electronic medical records that other providers within the network can view, so a new primary care provider can view your previous convenient or urgent care visits.

If you try a new primary care provider and don’t have a great experience, don’t be afraid to seek out a new one for your next visit. A solid doctor-patient relationship ultimately depends on whether the personalities and philosophies of each person align.

“People avoid going to their physician if they don’t feel like it’s a good match,” Sams said.

What This Means For You

If you do not have a primary care provider, ask trusted family members or friends for a referral. It is best to see a primary care provider for routine care, but if you need non-emergency medical care and cannot get into your primary care provider, you can receive safe and effective care from walk-in clinics.

Urgent Care Association. The essential nature of urgent care in the healthcare ecosystem post-COVID-19 .

Gaffney A, Woolhandler S, Cai C, et al. Medical documentation burden among US office-based physicians in 2019: a national study . JAMA Intern Med. 2022;182(5):564–566. doi:10.1001/jamainternmed.2022.0372

Robert Graham Center. The health of U.S. primary care: 2024 scorecard report .

By Cyra-Lea Drummond, BSN, RN Drummond is a registered nurse and a writer specializing in heart health, cardiac care, pediatric health, and more.

This paper is in the following e-collection/theme issue:

Published on 13.8.2024 in Vol 26 (2024)

Effect of a Mobile Health–Based Remote Interaction Management Intervention on the Quality of Life and Self-Management Behavior of Patients With Low Anterior Resection Syndrome: Randomized Controlled Trial

Authors of this article:

Original Paper

• Peng Zhou 1, 2 * , MNS   ; 
• Hui Li 3 * , MSN   ; 
• Xueying Pang 4 * , MSN   ; 
• Ting Wang 4 , MSN   ; 
• Yan Wang 2 , MSN   ; 
• Hongye He 4 , MSN   ; 
• Dongmei Zhuang 2 , MSN   ; 
• Furong Zhu 2 , MNS   ; 
• Rui Zhu 1 , MSN   ; 
• Shaohua Hu 1 , PhD  

1 Department of Nursing, the First Affiliated Hospital of Anhui Medical University, Hefei, China

2 School of Nursing, Anhui Medical University, Hefei, China

3 College of Traditional Chinese Medicine, Bozhou University, Bozhou, China

4 Department of Gastrointestinal Surgery, the First Affiliated Hospital of Anhui Medical University, Hefei, China

*these authors contributed equally

Corresponding Author:

Shaohua Hu, PhD

Department of Nursing

the First Affiliated Hospital of Anhui Medical University

218 Jixi Road

Hefei, 230009

Phone: 86 62922005

Email: [email protected]

Background: People who undergo sphincter-preserving surgery have high rates of anorectal functional disturbances, known as low anterior resection syndrome (LARS). LARS negatively affects patients’ quality of life (QoL) and increases their need for self-management behaviors. Therefore, approaches to enhance self-management behavior and QoL are vital.

Objective: This study aims to assess the effectiveness of a remote digital management intervention designed to enhance the QoL and self-management behavior of patients with LARS.

Methods: From July 2022 to May 2023, we conducted a single-blinded randomized controlled trial and recruited 120 patients with LARS in a tertiary hospital in Hefei, China. All patients were randomly assigned to the intervention group (using the “e-bowel safety” applet and monthly motivational interviewing) or the control group (usual care and an information booklet). Our team provided a 3-month intervention and followed up with all patients for an additional 3 months. The primary outcome was patient QoL measured using the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30. The secondary outcomes were evaluated using the Bowel Symptoms Self-Management Behaviors Questionnaire, LARS score, and Perceived Social Support Scale. Data collection occurred at study enrollment, the end of the 3-month intervention, and the 3-month follow-up. Generalized estimating equations were used to analyze changes in all outcome variables.

Results: In the end, 111 patients completed the study. In the intervention group, 5 patients withdrew; 4 patients withdrew in the control group. Patients in the intervention group had significantly larger improvements in the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 total score (mean difference 11.51; 95% CI 10.68-12.35; Cohen d =1.73) and Bowel Symptoms Self-Management Behaviors Questionnaire total score (mean difference 8.80; 95% CI 8.28-9.32; Cohen d =1.94) than those in the control group. This improvement effect remained stable at 3-month follow-up (mean difference 14.47; 95% CI 13.65-15.30; Cohen d =1.58 and mean difference 8.85; 95% CI 8.25-9.42; Cohen d =2.23). The LARS score total score had significantly larger decreases after intervention (mean difference –3.28; 95% CI –4.03 to –2.54; Cohen d =–0.39) and at 3-month follow-up (mean difference –6.69; 95% CI –7.45 to –5.93; Cohen d =–0.69). The Perceived Social Support Scale total score had significantly larger improvements after intervention (mean difference 0.47; 95% CI 0.22-0.71; Cohen d =1.81).

Conclusions: Our preliminary findings suggest that the mobile health–based remote interaction management intervention significantly enhanced the self-management behaviors and QoL of patients with LARS, and the effect was sustained. Mobile health–based remote interventions become an effective method to improve health outcomes for many patients with LARS.

Trial Registration: Chinese Clinical Trial Registry ChiCTR2200061317; https://tinyurl.com/tmmvpq3

Introduction

The Global Cancer Statistics 2020 showed that colorectal cancer ranks third in incidence of malignant tumors and second in cause of death worldwide [ 1 ]. Colorectal cancer incidence is also on the rise in China, with rectal cancer accounting for 60% of cases and middle and lower rectal cancers being the most common [ 2 ]. With the advancement of medical technology, optimal management of middle and lower rectal cancers increasingly favors sphincter-preserving surgery (SPS) [ 3 ]. This operation preserves anal function and avoids the inconvenience and pressure caused by permanent colostomy [ 4 ]. However, 70%-90% of patients after SPS struggle with long-term anorectal functional disturbances called low anterior resection syndrome (LARS) [ 5 , 6 ].

The presence of LARS has a severe adverse effect on the quality of life (QoL) of patients [ 7 ]. Postoperative LARS induces a spectrum of adverse physical and psychological effects in patients; for example, up to 50% of patients with LARS report toilet dependence during rehabilitation [ 8 , 9 ], 36% of patients experience pain, and approximately 13% of patients report high psychological distress [ 10 , 11 ]. Furthermore, LARS can restrict a patient’s social life, leading to further impact on their QoL [ 12 ]. Recently, longitudinal studies have found that patients’ QoL is still affected by LARS even 15 years after surgery [ 13 ]. Research has shown that patients can improve their QoL through methods, such as pelvic floor muscle exercises and dietary adjustments during home care; however, the effectiveness of these methods is limited by patients’ lack of knowledge of LARS and rehabilitation guidance [ 14 , 15 ].

Owing to the frequent occurrence of LARS in patients post discharge, patients must have a high level of self-management behavior [ 16 ]. However, in China, the majority of patients have a passive response to LARS, and their self-management behavior is at a low level [ 17 ]. Enhancing self-management awareness and providing information on supportive care can improve the self-management behavior of patients with LARS [ 18 ]. Research has demonstrated that motivational interviewing (MI) enhances self-management awareness and supports behavioral change [ 19 ].

Therefore, to improve patients’ QoL and self-management behaviors, providing supportive care information to patients is crucial. A qualitative exploration of patients with LARS’s perspectives on information needs revealed that timely symptom management measures are critical during home-based rehabilitation [ 20 ]. However, it is difficult to maintain continuity and instantaneity with existing management measures [ 21 , 22 ]. Owing to current advances in mobile technology, mobile health (mHealth) has been widely considered a means of patient health management, which can improve the effects of symptoms and assist patients in timely access to the required information [ 23 , 24 ].

To date, remote follow-up tools for patients with LARS have yielded promising results [ 25 ]. For patients with LARS, mHealth-based remote interventions may become an effective method to assist them in improving symptoms. However, mHealth intervention measures constructed for patients with LARS are rare. Most studies have only completed the development and pilot research of remote intervention programs, leading to insufficient data on the effectiveness of remote interventions in improving patient health outcomes [ 26 , 27 ]. WeChat (Tencent Corp) is China’s most frequently used instant messaging and social media application [ 28 ]. Evidence suggests that WeChat-based mHealth interventions effectively improve health outcomes in various health conditions [ 29 , 30 ].

This study aimed to assess the effectiveness of a remote digital management intervention designed for patients with LARS. The effectiveness of the intervention measure is determined by improvement in QoL, self-management behaviors, gastrointestinal symptoms, and social support. We hypothesized that the remote digital management intervention can effectively improve the health outcomes of patients with LARS.

Study Design

This study was conducted from July 15, 2022, to March 15, 2023, in Hefei, China. Our team provided a 3-month intervention and followed up with all patients for an additional 3 months. The intervention group used the “e-bowel safety” applet and received monthly MI. The control group received the usual care and was provided with a handbook containing information related to LARS. The CONSORT (Consolidated Standards of Reporting Trials) checklist is in Multimedia Appendix 1 .

Ethical Considerations

This randomized controlled trial (RCT) was approved by the ethics committee of the First Affiliated Hospital of Anhui Medical University (PJ2022-07-53) and registered on the Chinese Clinical Trial Registry (ChiCTR2200061317). All data were identified with a code number to ensure the confidentiality of the subjects’ data. No compensation was provided to participants.

Participants

The patients were recruited from a tertiary hospital in Hefei, Anhui Province, China. Patients were eligible to participate in our study if they met the following criteria: age older than 18 years, a diagnosis of rectal cancer, underwent SPS, LARS scores ≥21, ostomy closure surgery performed at least 3 months prior, the ability to read and write text, and proficiency in using WeChat. Patients with chronic gastrointestinal conditions, prior or current mental health disorders, cognitive impairments, communication disorders, or those who have participated in other clinical studies are ineligible for participation in this research. When patients meeting the recruitment criteria appeared in the hospital database, the system sent recruitment information to these patients with the approval of doctors not directly involved in the research design.

In this study, the sample size was determined based on the QoL. Previous research has shown that the QoL for patients with rectal cancer is 77±19 [ 31 ]. In an RCT using the EORTC QLQ-C30, a difference of 10 points is considered clinically significant [ 32 ]. With a two-sided test level of 0.05 and 80% test efficacy, each group requires a sample size of 45. Accounting for a 20% dropout rate, 112 patients are needed.

Intervention

Our previous study provided a comprehensive description of the intervention protocol [ 33 ]. The patients in the intervention group used the “e-bowel safety” applet for 3 months. They were required to check in on the applet daily and record their daily gastrointestinal symptoms. Our “e-bowel safety” applet comprises 4 main sections: a rehabilitation plan, LARS knowledge, web-based consultation, and patient stories. The rehabilitation plan module involves the collaborative development of home dietary and exercise plans by patients and researchers. The applet features intelligent reminders to monitor daily plan completion and provide prompts. After completing the rehabilitation plan, patients must fill out a daily health diary, and researchers dynamically adjust the rehabilitation plan based on patients’ feedback and physical condition. The LARS knowledge module offers evidence-based information on LARS and symptom management strategies. The web-based consultation module provides patients with an opportunity to interact with health care professionals, offering personalized guidance and feedback. The patient stories module allows patients to share symptom management experiences or engage with other patients, with all published content subject to researcher approval. Additionally, an incentive system has been designed to encourage participation. For instance, patients earn points by sharing personal stories or comments, which can later be exchanged for rewards after accumulating a certain number of points.

Moreover, our team members conducted monthly MIs with patients. MIs were led by 4 researchers with expertise in health coaching and disease management, including 1 clinical psychologist (Shangxin Zhang) and 3 registered nurses (TW, HH, and Ling Fang). The researchers engage with patients via WeChat for 30-60 minutes per call. The aim of MIs is to assist patients in setting rehabilitation goals, reinforcing self-management awareness, and promoting health behavior changes. The content of MIs is based on the interview guide determined by the research team, which guides the conversation from the initial session to explore the participant’s motivation to identify the facilitating factors and barriers to achieving their health goals. The interview guide is outlined in Multimedia Appendix 2 .

Patients in the control group received the usual care and were provided with a handbook containing information related to LARS. At the same time, our team members followed up with patients, using the same timing and frequency as the MI intervention group.

Randomization and Masking

This study was a single-blind, two-arm RCT. After obtaining consent from eligible patients, assistants who were not involved in the study randomly assigned them to the intervention and control groups at a 1:1 ratio. The randomization process was performed by the assistants and anonymized envelopes were used with block randomization, including block sizes randomly varying between 4 (2:2) and 6 (3:3). The research assistants (Ping Ni and Ai Wang) who collected the data were unaware of the patient assignments throughout the study. Patients used the QR codes provided by the research team to access the “e-bowel safety” applet, effectively reducing contamination between the 2 groups. Patients were blinded to their group assignments throughout the entire research process.

Quality Control and Participant Retention

Several strategies were used to ensure quality control and participant retention. Our “e-bowel safety” applet can monitor patients’ plan execution and provide reminders, which ensures the daily plans are followed strictly by patients. Before the formal intervention, we conducted a pilot experiment and gathered participant feedback to enhance our plan. The specific results are included in Multimedia Appendix 3 . Furthermore, patients received consistent guidance from our research assistants (Ping Ni and Ai Wang) when they had questions about the questionnaire content. Before the start of the study, all research assistants must undergo training and assessment on the use of all questionnaires by research team members. Only research assistants who pass the assessment can participate in data collection. Additionally, team members regularly check the progress of research assistants’ work to ensure that they are following the questionnaire collection process, identifying issues promptly, and making corrections.

Outcome Measures

The patients’ demographic and clinical information were obtained from the hospital database. Data were collected from patients using scales for their QoL, social support, self-management behaviors, and LARS scores at different time periods (0, 3, and 6 months). The research assistants (Ping Ni and Ai Wang) who collected the data assisted patients in completing questionnaires over the phone or through direct personal interaction.

Primary Outcome: QoL

The EORTC-QLQ-C30 (European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30) was used to measure QoL. This questionnaire comprises 30 items divided into 15 dimensions, including 1 dimension for QoL, 5 dimensions for functionality, 3 dimensions for symptoms, and 6 dimensions for additional symptoms. All dimension scores were linearly transformed to a scale of 0-100 points. Elevated scores on the 5 functionality dimensions and the QoL dimension were linked to improved functional status, whereas the reverse pattern was observed for the symptom dimensions and additional symptom dimensions. The Cronbach α coefficient ranged from 0.764 to 0.809 [ 34 ].

Secondary Outcome

Self-management.

The self-management behavior of patients was assessed by the Bowel Symptoms Self-Management Behaviors Questionnaire (BSSBQ). This questionnaire comprises 24 items divided into 5 functional scales, with each item scored on a scale of 0 (never) to 7 (always). Higher scores indicate better bowel symptom self-management behavior. The Cronbach α coefficient was 0.81 [ 17 ].

Bowel Function

The LARS score consists of 5 items, with a total score ranging from 0 to 42. Patients’ gastrointestinal symptoms are classified into no LARS, minor LARS, and major LARS based on the total score. The LARS score is a validated instrument for assessing bowel symptoms. The Cronbach α coefficient was 0.767 [ 35 ].

Social Support

The Perceived Social Support Scale (PSSS) consists of 12 items, with each item scored on a scale of 1 (extreme disagreement) to 7 (strong consent). The total scores ranged from 12 to 84. The higher the score, the stronger the perceived social support by the patient. This scale is widely used to assess the level of social support among patients in China. The Cronbach α coefficient of this scale was 0.899 [ 36 ].

Feasibility

The feasibility of intervention was assessed through the completion status of MI sessions and the adherence to health diary entries. The 3-month intervention corresponds to 3 MI sessions and 84 days of health diary entries.

Statistical Methods

All data were analyzed using SPSS Statistics (version 23.0; IBM Corp). An intention-to-treat analysis was performed in this study. We used the last observed values of the patients to replace missing data. Chi-square analysis was used to analyze the remaining demographic characteristics, and a 2-tailed independent sample t test was used to analyze the age and tumor height. Descriptive data were computed, including means with SD, medians with ranges, and frequencies with proportions where appropriate. The statistical significance was established at P <.05 (2-tailed test). Generalized estimating equations were used to analyze changes in QoL, self-management behaviors, LARS, and social support scores at different time points. The calculation of effect sizes was performed using Cohen d for the mean differences at various time periods.

Participant Characteristics

Initially, 60 patients were recruited in the control and intervention groups. During the study, 9 patients dropped out (dropout rate 7.5%). In the intervention group, 5 patients withdrew from the study, including 2 patients who received a reostomy because of an anastomotic fistula and 3 patients whose condition worsened. In the control group, 4 patients dropped out, including 2 patients whose condition worsened and 2 patients who refused to continue the intervention because of the side effects of chemotherapy. No statistically significant differences were observed between the patients who dropped out and those who completed all evaluations ( P =.17). Figure 1 shows the CONSORT flowchart of this study. Table 1 demonstrates no statistically significant differences in the demographics and clinical information between the control and intervention groups at baseline.

a LAR: low anterior resection.

b TaTME: transanal total mesorectal excision.

c EORTC-QLQ-C30: European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30.

d BSSBQ: Bowel Symptoms Self-Management Behaviors Questionnaire.

e LARS: Low anterior resection syndrome.

f PSSS: Perceived Social Support Scale.

Main Evaluation Indexes

Table 2 shows that the patients’ QoL improved for both groups. Patients in the intervention group demonstrated greater improvements in the EORTC-QLQ-C30 total score than those in the control group after intervention (mean difference 11.51; 95% CI 10.68-12.35; Cohen d =1.73). Furthermore, this improvement effect remained stable at 3-month follow-up (mean difference 14.47; 95% CI 13.65-15.30; Cohen d =1.58). Table 3 shows that the EORTC-QLQ-C30 total score in both groups exhibited a trend of change over the 6-month period ( P <.001). Differences were observed between the 2 groups and the interaction between group and time. A subgroup analysis was conducted on patients receiving preoperative chemotherapy versus postoperative chemotherapy. Among the 49 patients in the intervention group and 51 in the control group undergoing postoperative chemotherapy, a nominally significant improvement in the change from baseline in the EORTC-QLQ-C30 total score at 3 months was observed compared to the control group (difference of 4.42; P <.001). However, this effect was not seen in patients receiving preoperative chemotherapy. The specific results are included in Multimedia Appendix 4 .

a GEE: Generalized estimating equations.

b Difference in mean change from baseline to endpoint between the groups.

d Baseline.

e N/A: Not applicable.

f After the intervention.

g 3-month follow-up.

h BSSBQ: Bowel Symptoms Self-Management Behavior Questionnaire.

i LARS: Low anterior resection syndrome score.

j PSSS: Perceived Social Support Scale.

a EORTC-QLQ-C30: European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30.

b BSSBQ: Bowel Symptoms Self-Management Behaviors Questionnaire.

c LARS: Low anterior resection syndrome.

d PSSS: Perceived Social Support Scale.

Secondary Evaluation Indexes

Table 2 shows that the patients’ self-management behavior was enhanced for both groups. The BSSBQ total score had significantly larger improvements after intervention (mean difference 8.80; 95% CI 8.28-9.32; Cohen d =1.94) and at 3-month follow-up (mean difference 8.85; 95% CI 8.25-9.42; Cohen d =2.23) between groups. The BSSBQ total score showed statistically significant time effects ( P <.001; Table 3 ).

The LARS score total score had significantly larger decreases after intervention (mean difference –3.28; 95% CI –4.03 to –2.54; Cohen d =–0.39) and at 3-month follow-up (mean difference –6.69; 95% CI –7.45 to –5.93; Cohen d =–0.69). Table 3 shows that the LARS score total score in both groups exhibited a trend of change over the 6-month period. The intergroup effect exhibits homogeneity ( P =.05).

The PSSS total score had significantly larger improvements after intervention (mean difference 0.47; 95% CI 0.22-0.71; Cohen d =1.81); however, the improvement in this effect did not persist at 3-month follow-up (mean difference 0.23; 95% CI –0.20 to 0.45; P =.07; Table 2 ). Table 3 shows that the PSSS total score in both groups exhibited a trend of change over the 6-month period.

Among the 55 patients who completed the intervention, 45 patients completed 3 MI sessions on time, 7 patients postponed 1 MI session because of scheduling conflicts, and 3 patients only completed 2 MI sessions. The mean number of attended MI sessions was 2.95 (SD 0.23). Additionally, 40 patients completed 84 health diary entries, while the remaining 11 patients did not submit completed entries or fulfill the required entries. The mean number of days of health diary entries was 82.87 (SD 3.15). We invited patients from the intervention group to complete a survey to evaluate their perceptions of the intervention's usability. In the end, 49 people completed the survey. The specific results are included in Appendix 5.

Principal Findings

To the best of our knowledge, the “e-bowel safety” applet is the first mobile app developed for patients with LARS in China. This study offers a valuable reference point for future initiatives in mHealth interventions for patients with LARS. A mHealth-based intervention was found to be feasible and effective in helping patients with LARS relieve bowel dysfunction, improve their self-management behavior, and improve their QoL compared to usual care.

This study found that the EORTC-QLQ-C30 total score of the intervention group increased significantly more than that of the control group after the intervention, indicating that the mHealth-based remote interaction could improve the QoL of patients with LARS. These results can be attributed to multiple factors. First, uncontrollable changes in intestinal function, concerns about prognosis, and fear of the future make patients with LARS feel uncertain [ 37 ]. A sense of uncertainty influences a patient’s QoL [ 38 ]. Patients using the “e-bowel safety” applet can provide timely feedback on their problems to the medical staff and obtain solutions, which can effectively reduce the uncertainty of patients during home rehabilitation. Second, decreased bowel dysfunction severity positively affected the QoL [ 39 ]. Third, peer support reportedly enhances cancer adaptation and QoL [ 40 ]. The patients’ stories module offers a channel for communication and emotional support among patients with LARS. In this section, patients can share their experiences related to disease management or self-management and receive responses from their peers through comments.

As expected, the BSSBQ total score in the intervention group after the intervention was significantly higher than that in the control group. The findings supported our hypothesis that health-based remote interaction can enhance the self-management behavior of patients with LARS. After the intervention, the results of enhanced self-management behavior were consistent with a previous face-to-face 6-month self-management program study for LARS, which may indicate that mHealth-based remote interaction may yield intervention effects on self-management behavior similar to those observed in face-to-face interventions [ 41 ]. However, a more significant effect was observed at 3-month follow-up. This may be because monthly motivational interviews help patients adopt positive health behaviors and improve their self-management awareness [ 42 ]. Moreover, current web-based self-management information on LARS is overly intricate for patients, and the information fails to meet the patient’s needs [ 43 ]. The strength of our “e-bowel safety” applet is the credibility of the information provided and medical consultation from experts, which can meet the information needs of patients. Finally, our team members created an individualized self-management plan for each participant in the intervention group and reminded them to follow the plans on the applet, which ensured that the patients developed good habits.

Consistent with previous studies [ 41 ], this study found that the intervention group demonstrated a more significant decline in the LARS score than the control group. The LARS score also showed significant time effects, indicating that the patient’s bowel dysfunction changed significantly during the 6-month period. This may be because our team members guided patients in rehabilitation exercises and diet adjustments, which have been proven effective in improving bowel dysfunction [ 44 - 46 ]. Meanwhile, the severity of bowel dysfunction decreased over time [ 13 ].

Unlike those of previous studies, our findings indicated that mHealth-based remote interaction management intervention could improve the social support levels in the short term; however, sustaining a stable long-term effect on social support was not realized [ 47 ]. The patients in the study might have used the “e-bowel safety” applet only for 3 months, and the impact of the intervention on social support may not yield a residual advantage at 3-month follow-up. Furthermore, most patients’ physical and social functions gradually stabilized at 6 months. Our “e-bowel safety” applet focuses on intensive support for symptom management and lacks support knowledge for patients when symptoms plateau, which should be refined in future studies to achieve long-term effects.

In this study, MI was used to stimulate behavioral change and maintenance. The dual intervention of mHealth and MI promotes effective engagement and motivation for health behavior changes. Nearly all the patients (55/60) successfully completed the 3-month intervention and the follow-up during the intervention process, signifying that the mHealth-based remote interaction management intervention is feasible and acceptable. In addition, none of the patients in the intervention group experienced adverse consequences caused by the intervention, indicating that the intervention was safe.

Limitations

This study has some limitations. First, this study enrolled patients from a tertiary hospital in China, which restricts the generalizability of our results. In the future, we will recruit patients from more hospitals to confirm our research findings. Second, patients were subjected to a limited 3-month follow-up period, thereby restricting our assessment of the enduring effects of the mHealth-based remote interaction management intervention on self-management behavior and QoL. Finally, patients were required to use WeChat and smartphones, which presents the potential for selection bias.

Conclusions

The mHealth-based remote interaction management intervention effectively enhanced the self-management behavior and QoL of patients with LARS, and the impact remained consistent during the 3-month follow-up. Bowel dysfunction also significantly improved throughout the entire research process. This study suggests that mHealth intervention could provide an effective and new option for many patients with LARS. Multicenter studies are necessary to establish the generalizability and effectiveness of these interventions.

Acknowledgments

This work was supported by the 2021 Anhui Higher Education Institutions Provincial Quality Engineering Project (grant 2021jyxm0718) and the Scientific Research and Cultivation project of the School of Nursing, Anhui Medical University (grant hlqm12023055).

Conflicts of Interest

None declared.

CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth) checklist (version 1.6.1).

The interview guide of motivational interviewing.

Results of pilot experiment.

The results of subgroup analysis.

Comments and attitudes towards intervention of intervention group.

• Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(3):209-249. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cao W, Chen H, Yu Y, Li N, Chen W. Changing profiles of cancer burden worldwide and in China: a secondary analysis of the global cancer statistics 2020. Chin Med J (Engl). 2021;134(7):783-791. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zheng K, Hu Q, Yu G, Zhou L, Yao Y, Zhou Y, et al. Trends of sphincter-preserving surgeries for low lying rectal cancer: a 20-year experience in China. Front Oncol. 2022;12:996866. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Langenfeld SJ, Davis BR, Vogel JD, Davids JS, Temple LKF, Cologne KG, et al. The American society of colon and rectal surgeons clinical practice guidelines for the management of rectal cancer 2023 supplement. Dis Colon Rectum. 2024;67(1):18-31. [ CrossRef ] [ Medline ]
• Pappou EP, Temple LK, Patil S, Smith JJ, Wei IH, Nash GM, et al. Quality of life and function after rectal cancer surgery with and without sphincter preservation. Front Oncol. 2022;12:944843. [ CrossRef ] [ Medline ]
• Keane C, Fearnhead NS, Bordeianou LG, Christensen P, Basany EE, Laurberg S, et al. International consensus definition of low anterior resection syndrome. Dis Colon Rectum. 2020;63(3):274-284. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pieniowski EHA, Nordenvall C, Palmer G, Johar A, Tumlin Ekelund S, Lagergren P, et al. Prevalence of low anterior resection syndrome and impact on quality of life after rectal cancer surgery: population-based study. BJS Open. 2020;4(5):935-942. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sturiale A, Martellucci J, Zurli L, Vaccaro C, Brusciano L, Limongelli P, et al. Long-term functional follow-up after anterior rectal resection for cancer. Int J Colorectal Dis. 2017;32(1):83-88. [ CrossRef ] [ Medline ]
• van der Heijden JAG, Thomas G, Caers F, van Dijk WA, Slooter GD, Maaskant-Braat AJG. What you should know about the low anterior resection syndrome—clinical recommendations from a patient perspective. Eur J Surg Oncol. 2018;44(9):1331-1337. [ CrossRef ] [ Medline ]
• Qaderi SM, van der Heijden JAG, Verhoeven RHA, de Wilt JHW, Custers JAE, PLCRC study group. Trajectories of health-related quality of life and psychological distress in patients with colorectal cancer: a population-based study. Eur J Cancer. 2021;158:144-155. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moon J, Garfinkle R, Zelkowitz P, Dell'Aniello S, Vasilevsky C, Brassard P, et al. Incidence and factors associated with mental health disorders in patients with rectal cancer post-restorative proctectomy. Dis Colon Rectum. 2023;66(9):1203-1211. [ CrossRef ] [ Medline ]
• Bohlok A, Mercier C, Bouazza F, Galdon MG, Moretti L, Donckier V, et al. The burden of low anterior resection syndrome on quality of life in patients with mid or low rectal cancer. Support Care Cancer. 2020;28(3):1199-1206. [ CrossRef ] [ Medline ]
• Pieniowski EHA, Palmer GJ, Juul T, Lagergren P, Johar A, Emmertsen KJ, et al. Low anterior resection syndrome and quality of life after sphincter-sparing rectal cancer surgery: a long-term longitudinal follow-up. Dis Colon Rectum. 2019;62(1):14-20. [ CrossRef ] [ Medline ]
• Eyl RE, Xie K, Koch-Gallenkamp L, Brenner H, Arndt V. Quality of life and physical activity in long-term (≥5 years post-diagnosis) colorectal cancer survivors—systematic review. Health Qual Life Outcomes. 2018;16(1):112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sun V, Wendel CS, Demark-Wahnefried W, Grant M, McMullen CK, Bulkley JE, et al. Diet and behavior modifications by long-term rectal cancer survivors to manage bowel dysfunction-associated symptoms. Nutr Cancer. 2019;71(1):89-99. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lai X, Wong FKY, Ching SSY. Review of bowel dysfunction of rectal cancer patients during the first five years after sphincter-preserving surgery: a population in need of nursing attention. Eur J Oncol Nurs. 2013;17(5):681-692. [ CrossRef ] [ Medline ]
• Hou X, Pang D, Lu Q, Yang P, Jin S. Bowel dysfunction and self-management for bowel symptoms after sphincter-preserving surgery: a cross-sectional survey of Chinese rectal cancer patients. Cancer Nurs. 2017;40(4):E9-E16. [ CrossRef ] [ Medline ]
• Pape E, Pattyn P, Van Hecke A, Somers N, Van de Putte D, Ceelen W, et al. Impact of low anterior resection syndrome (LARS) on the quality of life and treatment options of LARS—a cross sectional study. Eur J Oncol Nurs. 2021;50:101878. [ CrossRef ] [ Medline ]
• Frost H, Campbell P, Maxwell M, O'Carroll RE, Dombrowski SU, Williams B, et al. Effectiveness of motivational interviewing on adult behaviour change in health and social care settings: a systematic review of reviews. PLoS One. 2018;13(10):e0204890. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pape E, Decoene E, Debrauwere M, Van Nieuwenhove Y, Pattyn P, Feryn T, et al. Information and counselling needs of patients with major low anterior resection syndrome: a qualitative study. J Clin Nurs. 2023;32(7-8):1240-1250. [ CrossRef ] [ Medline ]
• Bazzell A, Madsen LT, Dains J. Clinical management of bowel dysfunction after low anterior resection for rectal cancer. J Adv Pract Oncol. 2016;7(6):618-629. [ FREE Full text ] [ Medline ]
• Sakr A, Sauri F, Alessa M, Zakarnah E, Alawfi H, Torky R, et al. Assessment and management of low anterior resection syndrome after sphincter preserving surgery for rectal cancer. Chin Med J (Engl). 2020;133(15):1824-1833. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Keutzer L, Wicha SG, Simonsson US. Mobile health apps for improvement of tuberculosis treatment: descriptive review. JMIR Mhealth Uhealth. 2020;8(4):e17246. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Saeidzadeh S, Kamalumpundi V, Chi N, Nair R, Gilbertson-White S. Web and mobile-based symptom management interventions for physical symptoms of people with advanced cancer: a systematic review and meta-analysis. Palliat Med. 2021;35(6):1020-1038. [ CrossRef ] [ Medline ]
• Liu F, Guo P, Su X, Cui M, Jiang J, Wang S, et al. A novel remote follow-up tool based on an instant messaging/social media app for the management of patients with low anterior resection syndrome: pilot prospective self-control study. JMIR Mhealth Uhealth. 2021;9(3):e22647. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moon J, Monton O, Smith A, Garfinkle R, Zhao K, Zelkowitz P, et al. Interactive online informational and peer support application for patients with low anterior resection syndrome: patient survey and protocol for a multicentre randomized controlled trial. Colorectal Dis. 2021;23(5):1248-1257. [ CrossRef ] [ Medline ]
• Olivia M, Allister S, Jeongyoon M, Marie D, Richard G, Carol-Ann V, et al. An online educational and supportive care application for rectal cancer survivors with low anterior resection syndrome: a mixed methods pilot study. Colorectal Dis. 2023;25(9):1812-1820. [ CrossRef ] [ Medline ]
• Montag C, Becker B, Gan C. The multipurpose application WeChat: a review on recent research. Front Psychol. 2018;9:2247. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bao Y, Wang C, Xu H, Lai Y, Yan Y, Ma Y, et al. Effects of an mHealth intervention for pulmonary tuberculosis self-management based on the integrated theory of health behavior change: randomized controlled trial. JMIR Public Health Surveill. 2022;8(7):e34277. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guo Y, Hong YA, Cai W, Li L, Hao Y, Qiao J, et al. Effect of a WeChat-based intervention (Run4Love) on depressive symptoms among people living with HIV in China: a randomized controlled trial. J Med Internet Res. 2020;22(2):e16715. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Battersby NJ, Juul T, Christensen P, Janjua AZ, Branagan G, Emmertsen KJ, et al. Predicting the risk of bowel-related quality-of-life impairment after restorative resection for rectal cancer: a multicenter cross-sectional study. Dis Colon Rectum. 2016;59(4):270-280. [ CrossRef ] [ Medline ]
• Cocks K, King MT, Velikova G, Martyn St-James M, Fayers PM, Brown JM. Evidence-based guidelines for determination of sample size and interpretation of the European organisation for the research and treatment of cancer quality of life questionnaire core 30. J Clin Oncol. 2011;29(1):89-96. [ CrossRef ] [ Medline ]
• Li H, Zhou P, Pang X, Wang T, Yin D, Fu M, et al. Mobile health-based remote interaction management intervention for patients with low anterior resection syndrome: study protocol for a randomised controlled trial. BMJ Open. 2022;12(12):e066046. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ, et al. The European organization for research and treatment of cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993;85(5):365-376. [ CrossRef ] [ Medline ]
• Emmertsen KJ, Laurberg S. Low anterior resection syndrome score: development and validation of a symptom-based scoring system for bowel dysfunction after low anterior resection for rectal cancer. Ann Surg. 2012;255(5):922-928. [ CrossRef ] [ Medline ]
• Zimet GD, Powell SS, Farley GK, Werkman S, Berkoff KA. Psychometric characteristics of the multidimensional scale of perceived social support. J Pers Assess. 1990;55(3-4):610-617. [ CrossRef ] [ Medline ]
• Yin L, Fan L, Tan R, Yang G, Jiang F, Zhang C, et al. Bowel symptoms and self-care strategies of survivors in the process of restoration after low anterior resection of rectal cancer. BMC Surg. 2018;18(1):35. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Giammanco MD, Gitto L. Coping, uncertainty and health-related quality of life as determinants of anxiety and depression on a sample of hospitalized cardiac patients in Southern Italy. Qual Life Res. 2016;25(11):2941-2956. [ CrossRef ] [ Medline ]
• Emmertsen K, Laurberg S, Rectal Cancer Function Study Group. Impact of bowel dysfunction on quality of life after sphincter-preserving resection for rectal cancer. Br J Surg. 2013;100(10):1377-1387. [ CrossRef ] [ Medline ]
• Brodar KE, Carlisle V, Tang PY, Fisher EB. Identification and characterization of peer support for cancer prevention and care: a practice review. J Cancer Educ. 2022;37(3):645-654. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li C, Li Z, Lu Q, Zhou Y, Qin X, Wu A, et al. The effectiveness of a self-management program of bowel dysfunction in patients with mid and low rectal cancer after sphincter-preserving surgery: a pilot randomized controlled trial. Cancer Nurs. 2023;46(1):67-76. [ CrossRef ] [ Medline ]
• Seven M, Reid A, Abban S, Madziar C, Faro JM. Motivational interviewing interventions aiming to improve health behaviors among cancer survivors: a systematic scoping review. J Cancer Surviv. 2023;17(3):795-804. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lightner A. Assessing the readability, quality and accuracy of online health information for patients with low anterior resection syndrome following surgery for rectal cancer, by Garfinkle et al. Colorectal Dis. 2019;21(5):500. [ CrossRef ]
• Yuanyuan W, Shiyin H, Lei H, Ding D. Pelvic floor muscle exercises alleviate symptoms and improve mental health and rectal function in patients with low anterior resection syndrome. Front Oncol. 2023;13:1168807. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sun V, Crane TE, Slack SD, Yung A, Wright S, Sentovich S, et al. Rationale, development, and design of the altering intake, managing symptoms (AIMS) dietary intervention for bowel dysfunction in rectal cancer survivors. Contemp Clin Trials. 2018;68:61-66. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sun V, Crane TE, Freylersythe S, Slack SD, Yung A, Krouse RS, et al. Altering intake and managing symptoms: feasibility of a diet modification intervention for post-treatment bowel dysfunction in rectal cancer. Clin J Oncol Nurs. 2022;26(3):283-292. [ CrossRef ] [ Medline ]
• Zhu J, Ebert L, Liu X, Wei D, Chan SW. Mobile breast cancer e-Support program for Chinese women with breast cancer undergoing chemotherapy (part 2): multicenter randomized controlled trial. JMIR Mhealth Uhealth. 2018;6(4):e104. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 24.10.23; peer-reviewed by V Sun, C Thomson; comments to author 13.03.24; revised version received 07.05.24; accepted 03.06.24; published 13.08.24.

©Peng Zhou, Hui Li, Xueying Pang, Ting Wang, Yan Wang, Hongye He, Dongmei Zhuang, Furong Zhu, Rui Zhu, Shaohua Hu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 13.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Giving to Mass General Brigham
• Search the Site Site-wide search 0 items available in list reset search Search Close Search

Research Spotlight: Generative AI “Drift” and “Nondeterminism” Inconsistences are Important Considerations in Healthcare Applications

• Genetic Conditions
• share on facebook
• share on linkedIn
• share on X, formerly known as Twitter
• share by email
• print article

Samuel (Sandy) Aronson, ALM, MA,  executive director of IT and AI Solutions for Mass General Brigham Personalized Medicine and senior director of IT and AI Solutions for the  Accelerator for Clinical Transformation , is the corresponding author of a paper published in NEJM AI that looked at whether generative AI could hold promise for improving scientific literature review of variants in clinical genetic testing. Their findings could have a wide impact beyond this use case.  

How would you summarize your study for a lay audience?

We tested whether generative AI can be used to identify whether scientific articles contain information that can help geneticists determine whether genetic variants are harmful to patients. While testing this work, we identified inconsistencies in generative AI that could present a risk for patients if not adequately addressed. We suggest forms of testing and monitoring that could improve safety.

What question were you investigating?

We investigated whether generative AI can be used to determine: 1) whether a scientific article contains evidence about a variant that could help a geneticist’s assessment of a genetic variant and 2) whether any evidence found about the variant supports a benign, pathogenic, intermediate or inconclusive conclusion.

What methods or approach did you use?

We tested a generative AI strategy based on GPT-4 using a labeled dataset of 72 articles and compared generative AI to assessments from expert geneticists.

What did you find?

Generative AI performed relatively well, but more improvement is needed for most use cases. However, as we ran our tests repeatedly, we observed a phenomenon we deemed important: running the same test dataset repeatedly produced different results. Through repeated running of the test set over time, we characterized the variability. We found that both drift (changes in model performance over time) and nondeterminism (inconsistency between consecutive runs) were present. We developed visualizations that demonstrate the nature of these problems.

What are the implications?

If a clinical tool developer is not aware that large language models can exhibit significant drift and nondeterminism, they may run their test set once and use the results to determine whether their tool can be introduced into practice. This could be unsafe.

What are the next steps?

Our results show that it could be important to run a test set multiple times to demonstrate the degree of variability (nondeterminism) present. Our results also show that it is important to monitor for changes in performance (drift) over time.          

Authorship: In addition to Aronson, Mass General Brigham authors include Kalotina Machini, Jiyeon Shin, Pranav Sriraman, Emma R. Henricks, Charlotte J. Mailly, Angie J. Nottage, Sami S. Amr, Michael Oates, and Matthew S. Lebo. Additional authors include Sean Hamill.

Paper cited: Aronson SJ et al. “Integrating GPT-4 Models into a Genetic Variant Assessment Clinical Workflow: Assessing Performance, Nondeterminism, and Drift in Classifying Functional Evidence from Literature” NEJM AI DOI: 10.1056/AIcs2400245

Disclosures: Aronson, Shin, Mailly, and Oates report research grants and similar funding via Brigham and Women’s Hospital from Better Therapeutics, Boehringer Ingelheim, Eli Lilly, Milestone Pharmaceuticals, NovoNordisk, and PICORI.  Aronson, Oates, Machini, Henricks, and Lebo report NIH funding through Mass General Brigham. Aronson reports serving as a paid consultant for Nest Genomics.

Contributor

Related research about artificial intelligence.

published on Jul 8, 2024

published on Jul 1, 2024

published on Jun 17, 2024

published on May 23, 2024

published on May 9, 2024

published on May 6, 2024

published on Apr 24, 2024

published on Apr 18, 2024

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Br J Gen Pract
• v.72(716); March 2022

Academic primary care: challenges and opportunities

The rise of academic general practice in the UK has been remarkable: from the first professor of general practice, Richard Scott in 1963, and a small academic discipline; to a vibrant community reflected through the Society for Academic Primary Care (SAPC), and a renewed commitment to world-leading research through the fourth round of investment in the National Institute for Health Research (NIHR) School for Primary Care Research (SPCR) in England. 1

Just as academic general care has evolved, the clinical services provided by health and care professionals in this setting have expanded. General practice is embedded within diverse primary care teams comprising of various professionals: GPs, advanced nurse practitioners, physician associates, clinical pharmacists, first contact practitioners, social workers, and paramedics to name some. In 2010, Howie warned for the need of a contemporary identity, for the scientific discipline of general practice to continue to make significant contributions moving forwards. 2

CURRENT CONTEXT

Renamed as academic primary care to reflect the expansion of the clinical setting and intellectual work involved in the discipline; academic primary care delivers the scholarship (research and teaching) of community-based clinical practice: needed for excellent primary care. Primary care research is delivered in community settings mostly led by academic university departments, and primary care teaching occurs across university, NHS, and third-sector settings.

With the growth in the number of medical students, exposure and training in primary care remains essential in the training of tomorrow’s clinicians, and in encouraging future GP careers. The quality of primary care research is world leading; 3 , 4 but the capacity of primary care academics, including GPs, remains of concern. With only a small increase in UK senior academic GPs from 224 full time equivalents (FTE) in 2015 to 254 FTE in 2020, senior academic GPs comprise just 8% of the UK clinical academic workforce. 5 Academic GPs are a vital component of academic primary care because they bring clinical conceptualisation and context for greater impact on patients, practice, and the NHS.

But when general practice is fighting additional pressures from COVID-19, on a backdrop of chronic underinvestment, rising patient and service demand, and recruitment concerns, why is academic primary care important? Hobbs recently outlined why research matters to GPs, such as answering the ‘what to do’ questions. 6 Academic primary care can generate solutions to the current and future difficulties in practice, turning challenges into opportunities.

However, is the current identity of academic primary care one that service practitioners can resonate with and be inspired by? Initiatives like WiseGP (championing clinical scholarship in daily general practice; https://www.wisegp.co.uk ), the NIHR Clinical Research Networks (CRN), and The Scottish Deep End Project 7 aim to address this by connecting, promoting, and developing the academic skills of GPs and primary care practitioners in practice, and should be credited for a bottom-up grassroots approach.

COVID-19 RESPONSE

The contribution of academic primary care during COVID-19 is not to be underestimated. The research response can be underlined by the adaptive platform PRINCIPLE randomised controlled trial (RCT), which is directly influencing the management of COVID-19 in the community, and the first study to describe people’s experiences of living with long COVID. 8 , 9 The Royal College of General Practitioners (RCGP) Research Surveillance Centre, funded by UK Health Security Agency, which collects and monitors data from practices has been extended to facilitate the weekly surveillance of COVID-19 cases recorded in electronic primary care records, and for virology and serology testing. 10 Primary care researchers continue to contribute evidence on service delivery during the pandemic and on the management of COVID-19 (for example, NIHR-funded PANORAMIC platform RCT testing novel antivirals for early COVID-19 treatment). 11 The role of educators has been crucial: primary care educators and the RCGP have rapidly innovated to provide new educational opportunities, including on the art of remote consulting.

There are four main challenges for academic primary care moving forward. First is the need for credibility and buy in from service practitioners, primary and secondary care teams, policymakers, patients, and the public. This will facilitate the implementation of evidence and completion of research driven by patients and stakeholders. Second is the urgent need for further and sustained multi-professional capacity in academic primary care to facilitate innovative educational programmes and address the growing complexity of patient presentations and changes in health policy within a fast-changing primary care context. Even though progress has been made, we still lack good quality evidence in many areas of practice, such as mental illness and chronic pain, and increasing the numbers of senior GP and non-clinical academics will be critical in tackling this.

Third is the provision of primary care research and infrastructure funding, targeting areas of greater need, to enable robust evidence grounded in a primary care context to be suitable for adoption in practice to improve patient outcomes in the community, enhance NHS efficiency savings, and reduce treatments costs across primary and secondary care systems. Finally, it is crucial for integrated care systems to foster cross-sector working across public health and social care to provide holistic and realistic solutions to the problems that matter most to patients and their families.

OPPORTUNITIES

A barrier to attracting new GPs into practice is that general practice is perceived to lack prestige and challenge. 12 The importance of role models is well documented. 13 The SAPC are leading to address this, and initiatives like Primary care Academic CollaboraTive (a UK network of interested primary care professionals who collectively encourage, design, and undertake research to improve patient care) alongside more visible GP educators and departments of primary care will help. University departments need to showcase and engage their work with the public, host student/trainee symposia, have readily accessible mentorship, and advertise supervision on student projects to highlight, inspire, and advocate the intellectual challenge of general practice. The NIHR CRN primary care specialty group, incubator for primary care, 14 and the Associate Principal Investigator Scheme are avenues to boost engagement and involvement in research with frontline NHS practitioners.

There are research funding schemes that offer GPs and primary care colleagues the ability to address evidence gaps around multimorbidity, complex mental health presentations, and workforce pressures, while supporting capacity building. These include the RCGP Scientific Foundation Board, NIHR and NIHR SPCR Fellowships, Medical Research Council (MRC) Clinical Research Training Fellowships, and the Wellcome Clinical PhD Programme for Primary Care Clinicians, which support Masters and PhD level training for the next generation of primary care research leaders. These entry points into an academic career are important but they need to be accessible to all primary care professionals. In the Netherlands there are options of doctoral research training in vocational training schemes for GP registrars linked to an academic university department: this could be a template for Health Education England (HEE) and UK academic departments to help grow early career academic capacity, promote research in practice settings, and improve credibility and retention. 15

FUTURE IDENTITY

Academic primary care needs to be visible and accessible: striving for broad inclusion and diversity in the discipline. This should be a collective effort from heads of academic departments and medical schools, funders like MRC and NIHR, the RCGP, HEE, and SAPC: each have a key role to play. A positive, welcoming, and forward-thinking culture within academic primary care needs to be promoted and protected. Teams should harness the expertise of a wide range of professionals and people with lived experience to meet the growing challenges of primary care. There will be new significant trials for NHS primary care moving forward: academic primary care, if strengthened and valued, can make important contributions in navigating these.

Faraz Mughal is funded by a NIHR Doctoral Fellowship (reference: NIHR300957). Christian D Mallen is funded by the NIHR Applied Research Collaboration West Midlands and NIHR SPCR. Keele’s School of Medicine has received funding from Bristol Myers Squibb for a non-pharmacological atrial fibrillation screening trial. The views expressed in this article are those of the authors and not necessarily those of the NHS, NIHR, or the Department of Health and Social Care.

Freely submitted; externally peer reviewed.

Competing interests

Faraz Mughal was Royal College of General Practitioners (RCGP) Clinical Fellow in Mental Health from 2015–2020 and is a RCGP Clinical Advisor and Mental Health Representative. Helen Atherton sits on the RCGP Scientific Foundation Board. Joanne Reeve leads WiseGP: a joint RCGP/Society for Academic Primary Care initiative, which has received NIHR SPCR funding; chairs the Society for Academic Primary Care Heads of Departments group, and is panel Chair for the NIHR In-Practice Fellowship scheme. Christian D Mallen is Director of the NIHR SPCR, Wellcome Clinical PhD Programme for Primary Care Clinicians, and was past Chair of the NIHR primary care incubator.