myTIPreport

Understanding Level Calculations

One of the benefits of myTIPreport is that at evaluation time, a teacher doesn't need to think about the traditional level system. Instead, they are only asked to indicate whether or not they have observed a learner being able to perform a given milestone. With that data, myTIPreport is then able to calculate the milestone level they have achieved for ACGME reporting purposes. This article will walk through how that calculation is made.

Basics

First, let's talk through some of the basics of the views.

In the summary view of the left-hand side, you'll see each of the sub-competencies along with some statistics. The Evals column indicates the number of evaluations the given learner has received for the given sub-competency. The Level column indicates the level that learner has achieved in the given sub-competency.

The insights page has more granular data on each of the milestones and also gives insight into how the levels are calculated. Let's walk through a few examples:

First, some basics of the view. The top of the page lists the name of the sub-competency along with the number of evaluations from the past 6 months. In this case, you can see that there has only been one evaluation. After that, each milestone and the level it is associated with is listed along with some numbers. Those numbers should be read as follows:

  1. The numerator, or first number, is the number of times that a learner has demonstrated a milestone.
  2. The denominator, or second number, is the number of times that a learner has had the opportunity to demonstrate a milestone.

In the above picture, you can see that the learner demonstrated only the milestones from the first level in the one evaluation she received yet she had the opportunity to demonstrate them all (and just wasn't able to).

Level Calculations

How is this data then taken to give a single, overall level number? There are three rules:

  1. In order to achieve a milestone, a learner must demonstrate that milestone at least 50% of the time. In the following example, the learner has achieved the first 4 milestones because she has demonstrated the first two 3 of 3 times and the second 2 of 3 times. Both of those ratios are greater than 50%. The fifth milestone has only been demonstrated 1 of 3 times so that milestone has not been achieved (similarly with the rest of the milestones).
  1. In order to achieve a level, a learner must have achieved all of the milestones within a level. If a learner has achieved some, but not all of the milestones, they will be granted a half-level. In the most recent example, the learner has achieved level 1.5 because they've achieved all milestones in level 1 and 2 of the 3 milestones in level 2.

  2. The final rule is that in order to achieve a level, a learner must have achieved all previous levels. That is, a learner cannot achieve level 3 until she has achieved all of the milestones in level 2. Here is an example of that:

In this example, even though the learner has achieved all of the milestones in level 3, she has not achieved all of the milestones in level 2. This will result in a level score of 1.5 since she has not achieved all of the milestones in level 2, even though she has achieved those in level 3.

We understand that level calculations can be a bit confusing at first. If you're still unclear about anything explained here, feel free to ask clarifying questions at anytime by emailing support@mytipreport.org.