Twenty core entrustable professional activities for hematology residency training: a consensus proposition Open Access

Medical education transitioned into outcome-based approaches, focusing on medical competencies in the second part of the 20th century, with competency-based medical education (CBME) setting the pace of curricular conception worldwide. Identifying core abilities required by graduates and selecting assessment tools to measure progress are crucial steps in planning CBME.¹ 

Since 2000, assorted competency frameworks have been delineated to facilitate CBME use. For instance, Accreditation Council for Graduate Medical Education² and Canadian Medical Education Directives for Specialists³ framework proposals have influenced the curriculum designs of many medical schools.

Subsequently, in 2005 ten Cate idealized the concept of entrustable professional activities (EPAs),⁴ which have enhanced the process of assessing competencies in daily practice as we observe qualified students performing tasks inherent to a specific profession in the work environment. EPAs require knowledge, skills, and attitudes to be performed independently within a period of time, and reflect ≥1 competencies suitable for student training.⁵ Guidance on the preparation and implementation of EPAs have been published and established which parts of an EPA should be ideally submitted to a broader validation process.^6-8 

Advisory boards guided by the Next Accreditation System⁹ in the United States and the Royal College of Physicians and Surgeons of Canada⁷ have integrated EPAs propositions to complement the evaluation system in undergraduate and postgraduate training (medical residencies and fellowships). Regarding hematology residency, both the United States and Canada have adopted EPAs in the field of oncohematology, hematopathology, and apheresis medicine.^10-13 In Brazil, the National Commission of Medical Residency (CNRM)¹⁴ regulates hematology residency programs, mixing theoretical-practical internships based on competency approaches. In fact, the CNRM has newly defined EPAs as an option for assessing medical residents' progression through the residency course, although core hematological EPAs have still not been specified.¹⁵ 

Although assessment systems based on competencies/milestones and EPAs have already been developed in other countries such as the United States and Canada, we believe that the importance of developing regional versions of EPAs stems from local differences between countries. For example, in Brazil, hematology and hemotherapy residency programs (HHRs) are a single program that encompasses a 2-year training in both benign/classical and malignant disorders of hematopoietic and lymphoid systems, in addition to transfusion medicine. Therefore, it was essential to select practical activities that could accommodate all these topics and be applied to each Brazilian state's reality.

Given the breadth and complexity of such programs, translating conceptual proposals of CBME into daily practice can be an arduous work, that could be mediated by the implementation of EPAs. Herein, we developed a list of essential competencies and core EPAs developed by consensus-based methods applicable to HHRs.

The strategy was designed by 3 authors (A.C.N.B., B.K.L.D., and E.V.D.P.) henceforth referred to as principal investigators (PIs). All 3 PIs are part of 1 of the oldest HHRs in Brazil, based on a public academic institution.¹⁶ We first performed bibliographical research on the application of EPAs in the field of hematology and transfusion medicine, searching PubMed, and SciELO for original academic articles, and Google for legal regulatory documents on medical education and hematology practice in Brazil. We selected scientific publications related to CBME, EPAs, and their application in medical residency. Of note, until the first submission of this article, no records of the development of EPAs in hematology in Brazil were found.

Based on this literature review, on regulatory documents, and on the group’s previous experience in both undergraduate and medical residency teaching in the field of hematology, a list of essential competencies was drafted by the PIs and submitted to a review board. Likewise, after validation of this list, a set of beginning and end-of-training core EPAs for HHRs was validated using a modified Delphi¹⁷ technique in 2 rounds. The flowchart in Figure 1 shows the step-by-step methodologic workup.

Ideally, the final formulation of competencies and EPAs should involve the contribution of a diverse range of collaborators in the field of interest. According to the latest data on medical demographics in Brazil,¹⁸ 250 medical doctors attend the 52 2-year hematology residency programs in Brazil, which offer 180 new positions every year. The list of collaborators for our study derived from heads or preceptors of these programs. We invited 15 professionals to review the list of competencies and 69 professionals to compose the panel of collaborators for validating EPAs (modified Delphi). These expert panels comprised professionals from all Brazilian regions and members of various hematology residency programs nationwide, although most are located in the southeastern region of Brazil, which holds most HHRs.¹⁹ 

In addition to residency teaching, many of the professionals work in public and private health care services in various fields of hematology (benign/classical hematology, oncohematology, and transfusion medicine) and at different places (general and school hospitals, outpatient clinics, and private medical offices). For phase 1, we directly contacted these specialists, whereas for phase 2, the invitation was mediated by the Brazilian Association of Hematology, Hemotherapy and Cellular Therapy, which granted permission to our group to communicate with residency preceptors via mail, explaining our proposal and asking for cooperation. Of note, we have obtained formal consent from all panel experts before phase 1 (defining competencies) and phase 2 of research (elaborating and validating EPAs).

Based on the revised literature^20-24 and on regulatory documents on medical education in Brazil,¹⁴ PIs first produced a list of subcompetencies divided into major domains of 6 competencies (patient care, medical knowledge, progressive learning and professional development, communication, ethics and professionalism, and health systems management), encompassing fundamental features of medical education. Subsequently, we prepared a document consisting of a description of competencies, subcompetencies, and their elements, which was sent by regular mail and by email to each panel member (n = 15) with a 3-month time window for analysis.

The panel’s task was to clarify wording, avoid ambiguities, and remove items considered less relevant. The document was formatted in a way that a comment box was available right next to each competency description and its elements to facilitate feedback. Both versions of the document (digital and printed) contained detailed instructions about the task. All responses were then analyzed by the PIs and, after a revision based on expert panel feedback, a final version of the essential competencies list was obtained.

A set of core EPAs was drafted by the PIs based on: (1) scientific literature^6-8,10,11,25; (2) the list of essential competencies validated by our expert panel; and (3) regulatory documents on medical education and hematology practice in Brazil. A total of 23 EPAs were elaborated, divided in 3 categories: (1) caring for the hematological patient, (2) communication and teamwork, and (3) diagnostic support. This initial draft contained the EPA title and its description. Of note, these are the 2 items for which consensus is a requirement according to recent guiding documents on EPA development.^6-8 

A modified Delphi technique was used to validate these EPAs. The 69-member expert panel, recruited as described earlier, received an invitation email with a link to a REDCap-embedded digital copy of the first questionnaire containing the title and an individual description of the 23 EPAs, along with 2 statements for each EPA. Statement 1 was: “The task described by this EPA is essential for hematology practice,” aimed to address the panel perception on the relevance of each topic in a training program for hematologists. Statement 2 was: “The task described by this EPA can be assessed in a practice scenario during a residency program,” aimed to address a key element of an EPA, which is the feasibility of assessment during HHRs. Agreement with each statement was registered using a 4-point Likert scale (strongly disagree to strongly agree) per statement. The document included a detailed description of the task, including a concise description of the concept of EPAs. A pilot study was conducted with 3 researchers to refine language and presentation of the questionnaire.

The content validation of the scale was supported by the scale-level content validity index calculation averaging method (S-CVI/Ave) and the content validity index for item of scale (I-CVI),^26-29 in addition to general descriptive statistics (median with interquartile range [IQR]). The S-CVI/Ave calculation was used to analyze the responses globally and whether the level required for the agreement was reached, whereas I-CVI served to evaluate the representativeness of each item in terms of essentiality and possibility of evaluation. To determine S-CVI/Ave, the average I-CVI value was calculated, which was obtained by dividing the number of experts scoring 3 or 4 on the scale by the total number of experts. Then, the S-CVI/Ave was achieved from the sum of all I-CVI divided by the total number of items on the scale.²⁷ With greater rigor of analysis, S-CVI/Ave of ≥0.9 is recommended as a cutoff point for the “overall agreement” to be established, although 0.8 is the cutoff point most accepted in many studies.^27,30 When there are ≥6 experts involved, the I-CVI accepted as representative must be at least 0.83.²⁶ For our study, we defined that if any statement had an I-CVI of <0.70, it would be discarded; between 0.70 and 0.90 it would be reviewed in a second round; and ≥0.90 it would be validated without need of further review.

General descriptive statistics were also determined as median and IQR for each item of the scale. We also defined that items with median of <3 and IQR of >1 were to be submitted to an additional round of validation. In consequence, “Internal content validity” was established only when all criteria were present (S-CVI/Ave ≥ 0.9; I-CVI ≥ 0.9; median ≥ 3; and IQR ≤ 1). In the second Delphi round, participants were first exposed to a digital document summarizing the responses of round 1, displayed in colored graphs for each EPA, as well as an explanation of the EPAs that had not reached consensus. Next, the same statements and Likert scales were presented. The design of the validation process is detailed in Figure 2.

After each round, data were extracted from the REDCap platform and decoded in the form of spreadsheets for Excel and Statistical Package for the Social Science version 20.0 programs. Duplicate results were excluded. Descriptive statistics were calculated using Statistical Package for the Social Science version 20.0.

The study was approved by the ethics committee of the School of Medical Sciences of University of Campinas, Campinas, SP, Brazil (no. 55475021.3.0000.5404), and conducted between January 2022 and June 2024.

In phase 1, 7 of 15 invited experts agreed to participate (47.7%), yielding a 10-member panel including the PIs. In phase 2, 29 of 69 invited experts (42%) accepted to collaborate and answered the first round Delphi, with 21 of those remaining for the second round. Demographic and career background information of both panels are shown in Table 1. All panelists developed teaching activities in residency programs. Of note, the proportion of panelists who had received formal training in medical education was 40% and 44.8%, in panels 1 and 2, respectively.

A draft document containing 39 competencies was reviewed by a 10-member panel yielding a final list of 39 subcompetencies distributed in 6 major domains (Table 2). Feedback from panel member consisted of exclusion of ambiguous and redundant items and language adaptation so that competencies were more aligned with hematology practice in Brazil.

EPAs were drafted by the PIs based on the validated list of competencies, considering previously described rules for EPA design.^6-8 We specifically valued the relevance of each task to hematology practice, training requirements established by CNRM (the national accreditation body for HHRs), and feasibility of assessment.

An initial list of EPAs containing the title and a description that encompassed the items that are normally included in the “specifications and limitations” section of an EPA was sent to the 29-member expert panel who reviewed the 23 predefined EPAs using a 4-point Likert scale. All collaborators who agreed to participate in the survey evaluated the 2 statements made for each EPA. S-CVI/Ave for statement 1 (essential for practice) was 0.97, and for statement 2 (feasible for assessment) was 0.94. The calculated I-CVI was ≥0.9 (ie, above the validation threshold) in 20 (86%) of the items for statement 1, and in 18 (78%) of items for statement 2. All 23 EPAs analyzed met the validation criteria of median of ≥3 and IQR of ≤1. Furthermore, the essentiality and evaluation scale had S-CVI/Ave of ≥0.9. Consensus agreement and descriptive statistics are shown in Figure 3 and Table 3, respectively.

Seven EPAs had I-CVI for 1 or both of assessed statements below the validation threshold, thus requiring a second validation round. These EPAs covered the following topics: (1) identification and monitoring of complications from hematopoietic stem cell transplantation (HSCT); (2) providing systematic follow-up for patients with chronic hematological disorders; (3) communication skills with family members and the work team; (4) recording and reporting clinical cases; (5) teamwork; (6) carrying out intrathecal chemotherapy; and (7) mobilization of stem cells for transplantation.

In the second round, all medians remained within the criteria for validation, but 3 items did not reach I-CVI of ≥0.9, and 1 item achieved IQR of ≥1. S-CVI/Ave for scale 1 (essential for practice) also did not reach the consensus level of 0.9. EPAs that did not reach consensus validation criteria were those related to: (1) identification and monitoring of complications from HSCT; (4) carrying out intrathecal chemotherapy; and (7) mobilization of stem cells for transplantation.

After the final analyses of the last Delphi round, we evaluated all comments provided in the comment box and rewrote some of the EPAs to improve understanding. In addition, based on consensus analysis, we synthesized our previous list of 23 EPAs in 20 EPAs (Table 4).

The final EPA list was based on previously established EPA sections and included: (1) a title and (2) specifications and limitations. As stated in a previous publication, these are the essential elements that require validation. The full content of the 20 EPAs can be downloaded from our institution public repository for use and adaptation, based on local specificities, provided that this work is referenced (supplemental list of final EPAs).

To facilitate the use of these EPAs by trainers and trainees, we developed a set of “Alert signs” for each EPA (Table 5), consisting of attitudes that, when identified, should prompt immediate feedback and action from trainers, aimed to improve learning and protect patients. Alert signs represent attitudes that are below the lowest threshold for progression in hematology and hemotherapy residency training programs and were included to facilitate the understanding of EPAs during assessment practice. Alert signs were developed by a consensus between the 3 PIs.

With the emergence of innovative teaching approaches, the choice of tools to transpose the CBME theoretical framework onto medical education practices represents a critical strategic decision in medical education. The proposal to use EPAs to achieve this objective is an attractive option and, recently, has been pointed out as a mandatory strategy in medical residency in Brazil.¹⁵ The main contribution of this article was the development and validation of a list of 20 core EPAs that can be used and/or adapted in hematology residency programs, based on an interactive methodology, moderated by expert opinions.

As postulated by ten Cate,^4,5,7 observable, measurable, and specific tasks of a given profession performed in a work environment define EPAs. When performing such tasks, trainees demonstrate multiple skills, allowing assessment of the process, and the actual implementation of the tenets of CBME in residency programs.

Developing and validating EPAs among experts from any country, and particularly in one of continental dimensions and significant regional differences, is a very challenging enterprise. Our strategy departed from initial input by 3 authors actively involved with residency training, and that had been also exposed to CBME and EPA-related scientific literature. Contents produced by this group was validated by larger groups of hematologists, presenting more diverse geographic and demographic profiles, which were specifically assembled for the purpose of this validation. This was made possible by the use of a country-wide list of residency program coordinators shared by our national hematology association that guaranteed that our invitation could reach all residency programs in Brazil. We believe that the response rates of >45% in Delphi rounds in such context are suitable for the validation that we aimed for.³¹ 

Of note, among panel members that actually contributed to validation, more than half worked in both public and private health services and had at least 10 years of experience in the area, which are characteristics consistent with the profile of hematology practice in Brazil. The relative enrichment of our sample with hematologists who still work in academia is, in our opinion, a strength of our study, in that these are professionals that are more likely to have been exposed to concepts such as CBME and EPAs. With regard to the geographic distribution, it somehow mirrors the distribution of hematologists in Brazil. Of note, it has been shown that one of the most important things to consider when forming the group is the “experience” that each involved stakeholder can contribute with.³² 

Following the guidelines presented in an International Association for Medical Education guide to develop and implement EPAs,⁶ we started with the identification of competencies followed by the definition and validation of possible core EPAs for the hematologist, to build an EPA-based competency-oriented workplace curriculum. Initially, we revised the list of competencies already made available by the Ministry of Health¹⁴ using a panel-based strategy. Collaborating panelists reviewed this list of competencies in detail and suggested improvements focused mainly on clarity and coherence with the language aligned with CBME concepts.

After this step we ended up with a comprehensive, objective, and contemporary list of competencies that could, in our opinion, be useful for hematology residencies in Brazil and in other parts of the world. This list was the source document for the development of provisional EPA titles and descriptions that were used in the second validation step. Authors were careful to write EPAs that were neither too short nor too long, correlated to the clinical setting and ranged in number from 20 to 34, as recommended.³³ 

We then proceeded to a “modified Delphi” strategy to promote interaction with a larger panel of specialists. The modification consisted in removing the open-ended questions stage, common to classic Delphi. The anonymity inherent to the Delphi methodology^34,35 was important to guarantee more freedom in judging the statements. Aside from anonymity, enabling the interaction of geographically distant professionals was opportune, considering the pandemic faced at the time the research began. In the second Delphi round, we had a loss rate in relation to the first round of 29.5%, although our sample remained above the threshold of 10 to 15 participants reported to be adequate to form a homogeneous Delphi panel.³⁶ 

During the EPA validation process, 7 EPAs did not reach consensus criteria and were submitted to round 2, with 3 of them being excluded from the final list. Validated EPAs comprised the main activities that represent the practice of hematologists in Brazil, covering in-hospital and outpatient care in the fields of benign/classical and malignant hematology and transfusion medicine.

Of 3 excluded EPAs, 2 were related to HSCT. “Recognizing and handling patients who have undergone HSCT and evolved into complications” as well as “mobilizing and collecting stem cells for autologous stem cell transplantation” did not obtain I-CVI of ≥0.9 in at least one of the statements under evaluation. Analogous to the training program for EPAs defined by The Royal College of Physicians and Surgeons of Canada,¹⁰ we validated the EPA that encompasses appropriate indication of HSCT. Of note, EPAs similar to those excluded in Brazil were validated in Canada. One of the explanations may be linked to the fact that the skills to perform HSCT in Brazil are developed in a subspecialization course in HSCT that is separate from most regular HHRs, which may have led panelists to value only HSCT indication but not its clinical management as part of the set of skills required for hematologists. In agreement with our national competency matrix,¹⁴ we maintained the subcompetencies related to HSCT in our matrix, although part of it could not be specifically connected to an EPA.

“Performing intrathecal chemotherapy” was another excluded EPA. Many hospital services have now dedicated specialists that perform intrathecal infusion of medications, which may partly justify the rejection of this EPA. Of note, skills related to performing invasive procedures other than bone marrow biopsies were also considered of little importance to clinical practice in a survey to a cohort of former hematology residents in Brazil.³⁷ 

The development of alert signs was an innovation of our study, aimed to further facilitate the translation of EPA content to assessment practices, based on the assumption that it is frequently easier to identify behaviors/performance in the extreme of a spectrum than to gauge the nuances across this spectrum.^38,39 Identification of behaviors below these minimal thresholds should allow trainers to identify the need for early educational interventions, which potentially benefits all stakeholders in the education process.⁴⁰ We also believe that this strategy contributes to connect teaching and assessment to patient safety, which is an increased recognized value in medical education.⁴¹ 

Our study has limitations that need to be acknowledged. First, our panel did not include residents, which could have contributed with a different perspective to the process. Second, we did not validate alert signs. Additional validation and implementation studies are warranted to address both limitations.

In conclusion, our study provides 2 important tools that can contribute to hematology training: a competence matrix and a list of 20 core EPAs for hematology and transfusion medicine programs. Moreover, for each EPA we compiled a list of threshold behaviors that should alert trainers about the need to more cautiously assess residents in their training process. Although validated by Brazilian specialists, these tools can be used as a starting point or adapted to hematology residencies in other regions, allowing the implementation and consolidation of CBME framework into the formation of our future hematology workforce.

The authors thank Paulo Roberto Soares for the technical support, and the residency training committee of the Brazilian Association of Hematology and Hemotherapy for the collaboration. The study was funded by a FAPESP (Sao Paulo Research Foundation) grant 2022/13216-0.

Contribution: A.C.N.B., B.K.L.D., and E.V.D.P. designed the study and analyzed the data; A.C.N.B. and E.V.D.P. drafted the manuscript and organized and managed the group of investigators; A.C.N.B. created survey tools, obtained the data, and organized data sets; the members of the C&EPA in Hematology Group participated in the process that resulted in competencies and EPAs as part of discussion panels and providing feedback about study design; and all coauthors revised and approved the manuscript.

Conflict-of-interest disclosure: The authors declare no competing financial interests.

A complete list of the members of the C&EPA in Hematology Group appears in the Appendix.

Correspondence: Erich Vinicius De Paula, University of Campinas, School of Medical Sciences. Rua Tessalia Vieira de Camargo 126, Campinas, SP 13083-887, Brazil; email: erich@unicamp.br.

The members of the C&EPA in Hematology Group are Alessandra Aparecida Paz, Clinical Hospital for Porto Alegre (HCPA), Porto Alegre, RS, Brazil; Alessandro de Moura Almeida, Federal University of Bahia (UFBA), BA, Brazil; Ana Cristina Fenili, Nossa Senhora da Conceição Hospital, Porto Alegre, RS, Brazil; Ana Paula de Azambuja, Federal University of Paraná (UFPR), Curitiba, PR, Brazil; Bruno Deltreggia Benites, University of Campinas (UNICAMP), Campinas, SP, Brazil; Carolina Kassab Wroclawski, Sírio-Libanês Hospital, Sao Paulo, SP, Brazil; Cássio Lins Gil de Farias, D’or Oncology Group, SP, Brazil; Daniele de Andrade Reckziegel, Brasília Blood Center, Brasilia, DF, Brazil; David Cavalcanti Ferreira, Federal University of Santa Catarina (UFSC), Florianopolis, SC, Brazil; Denys Eiti Fujimoto, Santa Casa de Misericórdia Hospital, Sao Paulo, SP, Brazil; Frederico Lisboa Nogueira, Federal University of Minas Gerais, Belo Horizonte, MG, Brazil; Giovanna Steffenello, UFSC, Florianopolis, SC, Brazil; Graziela Toledo Costa Mayrink, Federal University of Juiz de Fora, Juiz de Fora, MG, Brazil; Marco Aurelio Salvino de Araujo, UFBA, Salvador, BA, Brazil; Paula de Melo Campos, UNICAMP, Campinas, SP, Brazil; Rafael de Sá Vasconcelos, Brasília Cancer Center, DF, Brazil; Sheila Soares Silva, Federal University of Triângulo Mineiro, Uberaba, MG, Brazil; Silvia Nathalia Bueno Lopes, UFPR, Curitiba, PR, Brazil; Tahiane de Brum Soares, HCPA, Porto Alegre, RS, Brazil; Vera Lúcia de Piratininga Figueiredo, Servidor Público Estadual Hospital, Sao Paulo, SP, Brazil; Virgínia Guerra Moreira, Santa Casa de Misericórdia de Belo Horizonte, Belo Horizonte, MG, Brazil; and Yana de Sousa Rabelo, Federal University of Goiás, Goiania, GO, Brazil.