This is the first in a series of blog posts that describes how different groups and specialties are using myTIPreport. This series will attempt to highlight what’s currently possible with myTIPreport as well as plans for the future. While curriculum content is specialty-specific, the features and reporting capabilities described can generally be re-used by any specialty.

Introduction

In late 2018, we partnered with Dr. Glenn Woodworth (Regional Anesthesia and Acute Pain Medicine Fellowship Director at OHSU; Anesthesiology Toolbox Editor-in-Chief) and his team to implement app-based, point-of-care assessments for the EPAs his team was developing for Anesthesiology Residencies. The goals of this project were too:

1. Allow faculty members to use the myTIPreport app and website to provide formative feedback at the point-of-care to residents on the set of EPAs developed.
2. Allow the data gathered to automatically inform milestone level calculations.
3. Allow other Anesthesiology residencies to re-use what is built without needing to re-invent the wheel. Everything built should be compiled into a customizable template that anyone can use.

Once developed, we aided their team in launching a multi-site study to evaluate the feasibility of implementation using a mobile app like myTIPreport and the utility for competency and milestone management. In this article, we’ll:

1. Demonstrate how these EPA assessments can be used in real-time and at the point-of-care using the myTIPreport app.
2. Walk through the EPAs that were developed and discuss how the data gathered using them automatically maps to milestones.
3. Show how the data is compiled and can be accessed in reports for resident-discussions and CCC reporting.

In addition to EPAs, through this partnership we’ve also developed the ability to perform Procedural Skills point-of-care assessments for Anesthesiology residencies. That content and tooling will be described in another post.

Evaluating EPAs at the Point-Of-Care

One of the goals of Dr. Woodworth’s team was to allow for faculty members to provide feedback on every encounter they had with a resident that mapped to one of their EPAs (see full list of EPAs here). That presents interesting challenges given the demands that are already placed on residents and faculty members. Fortunately, one of our main focuses over the past few years has been to make the myTIPreport both simple and intuitive which gave us a good base to start from. In addition to that though, this goal also means that the evaluation itself needs to be short, yet gather enough information in order for the feedback to be meaningful and actionable. Ultimately, the evaluation form developed was made up of these three questions:

Difficulty of Case

This question allows the faculty member to designate whether a case was Simple or Complex. We’ll talk more about how this data is used later on.

Overall Entrustment Level

This question asks the faculty member to assess how much supervision they had to provide the resident while performing care. It allows the faculty member to choose one of the following options from a version of the Modified Ottawa Rating Scale that is commonly used when assessing Medical School EPAs:

• I Did Activity: Supervisor did the activity, trainee observed or assisted.
• Direct Supervision: Supervisor talked trainee through activity (constant or near constant supervision, requires physical presence of the supervisor).
• Reactive Supervision: Supervisor directed trainee from time to time (supervisor does not need to be constantly observing, trainee often requires consultation).
• Available If Needed: Supervisor was available just in case (reactive supervision, infrequent).
• Independent Practice: Trainee ready for independent practice

Comments

This is an optional question with the following prompt: “Please explain entrustment level choice. Identify behaviors preventing independent practice, if any.”

Altogether, these make for a lightweight assessment when using the app:

When combined with myTIPreport’s existing capabilities, this evaluation can be initiated in 4 different ways:

1. Faculty Initiated - Either in-person, or after parting ways with the resident, the faculty member can perform the evaluation on their device. Once feedback is submitted the resident is notified via a push notification.
2. Resident Initiated, In-Person - Allows the resident to drive an in-person feedback encounter with the faculty member from the resident’s phone.
3. Resident Requested - After the case, the resident can submit a request for feedback on a specific EPA from a faculty member. This triggers a push notification to the faculty member to submit feedback.
4. Admin Assigned - Program Directors and Coordinators can assign faculty members evaluations to perform on specific EPAs for a given resident.

Mapping EPAs to Milestones

As a reminder, one of the primary goals of this partnership was to allow the assessments gathered using the EPAs to automatically inform milestone reports and level calculations. To enable this, Dr. Woodworth and his team identified the individual milestone elements that applied to both Simple and Complex versions of each EPA, for each entrustment level rating. For example, a rating of Independent Practice for a Complex Airway Management indicates that the resident demonstrated dozens of milestones across 22 of the milestone sub-competencies. Dr. Woodworth’s team is in the process of writing a paper on the process they used to develop the EPAs and these mappings.

Next, we built this EPA to Milestone mapping logic into the myTIPreport backend. Thus, when an EPA assessment is submitted, this data immediately flows to milestone reports as well. Altogether, this means that we can generate milestone sub-competency levels for a resident based solely on the formative feedback they’ve received via EPAs, and we can further use this information to show their performance for individual milestone elements within a sub-competency.

Reporting

We’ve been working closely with Dr. Woodworth and other participants in the pilot to continue to improve our reports. As part of this effort, we’ve enabled and improved a number of scenarios.

In both the phone app and the website, a resident’s current level in each of the EPAs is front-and-center when navigating the list of EPAs:

Administrators have the option of allowing faculty members to see this for residents as well (or keeping it hidden to prevent possible bias).

Resident Dashboard and Graduation Targets

The new Resident Dashboard is available to both the administrators (Program Director, Program Coordinators) and the residents themselves. On this view, you can easily see the resident’s progress across EPAs, Milestones and any other evaluations that exist in your myTIPreport account.

In addition to developing the EPAs and mapping them to milestones, Dr. Woodworth’s group also defined Graduation Targets for each of the EPAs. These graduation targets show up as lines on the progress bars for each EPA. With these, the resident dashboard allows you to quickly identify the EPAs and Milestones where the resident has achieved the graduation target, and those where she has more work to do.

This view also allows ordering by progress to quickly identify areas of strengths and weaknesses.

For any EPA, you can drill in to see growth over time and the individual evaluation scores. The graphs map entrustment level on the y-axis and time on the x-axis. Each node has the initials of the faculty member who submitted the evaluation to make it easy to identify possible outliers from individual faculty members. Finally, clicking any of the nodes will open the evaluation in question.

Class Comparisons

The new Competency Dashboards allow an administrator to compare entrustment levels for a specific EPA or Milestone for residents in a given class or across all classes.

You can view a full walk-through of the reporting features from Dr. Woodworth himself here:

Easy to Adopt and Customize

Everything we’ve built here is templated and reusable. Any Anesthesiology Residency that is interested in what we’ve built here can sign up for an account and have all of these evaluations, mappings, and reports available from day-one with no work. Most settings are customizable, so if a given residency wants to change a certain graduation target or disable an EPA, they can do that easily within the admin experience. Additionally, Dr. Woodworth’s team has developed a number of training materials to make it easier to roll this out in your program.

New to myTIPreport? The features described in this post just begin to scratch the surface of what myTIPreport offers. We have a whole host of other features such as Leaderboards and Citizenship Tracking that encourage participation, MedHub Integration and Rotation Evaluations.

Outside of Anesthesiology, all of the reports and workflows we’ve built as part of this effort can be reused by any specialty. If you’re interested in trying any of this out, you can sign up for a free trial or reach out to us at [email protected].

Next Steps in Anesthesia

Neither the myTIPreport team and Dr. Woodworth’s team is done with this project. We’ve got a number of items lined up to make these tools even better:

1. Dr. Woodworth’s team is writing research papers on how the EPAs and milestone mappings were developed.
2. We’ve already developed and implemented a similar module for assessing Procedural Skills and will be blogging about that soon.
3. EPAs and their mappings do not cover all milestones simply because some milestones don’t apply to EPAs. To cover those, Dr. Woodworth’s team is looking to develop a set of OSCEs and other special assessments to cover things that aren’t seen in the clinical settings, as well as things like Journal Club and M&M Presentations. When those are ready, we’ll build modules for those as well.
4. Similar content and modules for Anesthesiology Fellowships.

We’d like to thank Dr. Woodworth and his team for this fantastic partnership.

If you have any questions or are interested in partnering with us to develop something for your specialty, don’t hesitate to reach out to us at [email protected].

Reference

Full List of EPAs

• Acute and Chronic Pain
  • Management of Acute Pain Out of the OR
  • Management of Chronic Pain
  • Regional Anesthesia Perioperative Care
• Adult OR Cases
  • Abdominal Aortic Surgery Perioperative Care
  • Cardiac Surgery Perioperative Care
  • General Operating Room Case Perioperative Care
  • Intracranial Procedures Perioperative Care
  • Major Trauma Perioperative Care
  • Out of OR Perioperative Care
  • Thoracic Surgery Perioperative Care
• Core Activity
  • Airway Management
  • PACU Management
  • Preop Assessment and Optimization
  • Transfer of care
• Critical Care Critical Care of Non-OR Patients
• Obstetrics
  • Cesarean Section Perioperative Care
  • Labor Analgesia
  • Pregnant Patient Undergoing Non-Obstetric Surgery
• Pediatrics
  • PEDS Anesthetic Perioperative Care
  • PEDS Neonatal Anesthetic Perioperative Care