Understanding Level Calculations

One of the benefits of myTIPreport is that at evaluation time, a teacher doesn't need to think about the traditional level system. Instead, they are only asked to indicate whether or not they have observed a learner being able to perform a given milestone. With that data, myTIPreport is then able to calculate the milestone level they have achieved for ACGME reporting purposes. This article will walk through how that calculation is made.

Basics

First, let's talk through some of the basics of the views.

In the summary view of the left-hand side, you'll see each of the subcompetencies along with some statistics. Each entry has a progress bar that tells you the current level of the learner for the subcompetency along with how many evaluations they've received for it.

The insights page has more granular data on each of the milestones and also gives insight into how the levels are calculated. Let's walk through a few examples:

First, some basics of the view:

• The top right corner shows the time period for the data that's being analyzed. You can change that time period by clicking on that control. Since milestones are cumulative over a learner's time at the program, the start date of the range is ignored but the end date is respected. This behavior can be changed if desired.
• The number of evaluations is listed in the first "button" at the top of the page. If you want to drill into the evaluations, you can do so by clicking that button.

Next you'll see a list of the milestones for this subcompetency and some stats about each of them. Here's how to interpret those numbers:

1. The first number (after "Demonstrated"), is the number of times that a learner has demonstrated a milestone.
2. The second number is the number of times that a learner has had the opportunity to demonstrate a milestone.

In the above picture, you can see that the learner demonstrated the level 1 milestones in the one evaluation she received. You can also see that she had the opportunity to demonstrate all milestones (and wasn't able to).

Level Calculations

How is this data then taken to give a single, overall level number? There are three rules:

1. In order to achieve a milestone, a learner must demonstrate that milestone at least 50% of the time. In the following example, the learner has achieved the first 4 milestones (first 2 levels) because she has demonstrated the first two 3 of 3 times and the second 2 of 3 times. Both of those ratios are greater than 50%. The fifth milestone has only been demonstrated 1 of 3 times so that milestone has not been achieved (similarly with the rest of the milestones and levels).
   
   
2. In order to achieve a level, a learner must have achieved all of the milestones within a level. If a learner has achieved some but not all of the milestones, they will be granted a half-level. In the most recent example, the learner has achieved level 1.5 because they've achieved all milestones in level 1 and 2 of the 3 milestones in level 2.
3. The final rule is that in order to achieve a level, a learner must have achieved all previous levels. That is, a learner cannot achieve level 3 until she has achieved all of the milestones in level 2. Here is an example of that:
   
   

In this example, even though the learner has achieved all of the milestones in level 3, she has not achieved all of the milestones in level 2. This will result in a level score of 1.5 since she has not achieved all of the milestones in level 2, even though she has achieved those in level 3.

Number of Evaluations Considered

By default, only the last 10 opportunities to demonstrate a capability are considered. This helps prevent a large number of failures on advanced milestones early in someone's career plaguing them later. When this limit is hit, it will be displayed like this:

If you want to change the number of evaluations considered when calculating a level, you can do so by following the directions here.

We understand that level calculations can be a bit confusing at first. If you're still unclear about anything explained here, feel free to ask clarifying questions at anytime by emailing [email protected].