Page 103 - JRCERT Update Articles

JRCERT Update

Data Analysis: Closing the Loop in Assessment

data will not be beneficial, and no amount of analysis significantly from the normal, expected data sets they
will provide useful information. could be considered erroneous. An example might be
the rubric scores for a communications assignment. If,
Data Collection and Analysis for example, the metric historically has yielded an aver-
It is an assessment best practice to gather data for age in the range of 3.0 to 3.5 on a 5-point scale, and this
SLOs from several sources; this practice is known as year’s cohort average score was 4.9, the educator likely
triangulation. A laboratory performance exam that would want to do some investigation and pose ques-
demonstrates critical thinking, a clinical proficiency tions to determine whether the raw data was collected,
exam that requires critical thinking to complete the scored, and entered appropriately. An outlier also might
exam, and an employer evaluation of the graduate’s be linked to a modification in the educational process,
ability to think critically would be an excellent example such as the implementation of a new textbook, use of a
of triangulation. These multiple data points should new faculty member, employment of a new method of
provide a great deal of information over time to ensure instruction, or student performance that falls signifi-
that students have achieved the objective set by the cantly above or below the cohort. The above examples
program, or point to contradictory information that emphasize the importance of evaluating the data in
requires further investigation. context and incorporating qualitative information with
Data analysis is not merely the restatement of the the raw data.
numbers collected, but rather is a determination of While analyzing the data, it is important to maintain
what the numbers represent. Box 2 identifies com- a global and a detailed perspective. The global perspec-
mon areas to be mindful of when analyzing data from tive includes reviewing qualitative information that
student learning assessment. Data analysis begins explains the raw data. It also involves stepping back
2
with a review of “cleansed” data—data that has been and looking at the data over time, a practice known
purged of outliers. Outliers are data points that veer so as longitudinal trend analysis. Rarely should educators

Box 2
Common Data Analysis Errors 2
Using meaningless Be sure to measure student learning. A course grade that includes points for attendance does not indicate the
metrics and tools extent to which a student has learned and applied effective critical thinking skills.
Data overload Too many metrics is a common error; sometimes less is more. Only collect data that provides insight into
student success. Unnecessary data collection often leads to frustration, and thorough analysis does not occur
because assessment becomes compliance reporting.
Not cleaning up Always assume the data is inaccurate at first. Once you get familiar with it, you will start to feel when something
messy data is not right.
Ignoring outliers Outliers in data can indicate that something is wrong, such as a process not working. Investigate outliers in the
data to make sure nothing is seriously wrong.
Fixating on outliers Although outliers should not be ignored, do not focus on them and ignore everything else. For example, do
not make program changes based on a single poor data point.
Not watching metrics Qualitative data is important to make sure the quantitative data are analyzed in context. For example, scores
in context might be lower than normal because of recent curriculum revisions.
Not using trend data Look at data results over time and cohorts. Do not make significant changes based on poor data results from
a single year or cohort. Conversely, do not continue to monitor for several cycles or fail to develop action plans
when data reveal decreasing results.
Assessing new action When closing the loop in the assessment cycle, it is vital to continue to measure and assess the effectiveness of
plans any action plan or changes that have been implemented.

546 RADIOLOGIC TECHNOLOGY, May/June 2017, Volume 88, Number 5
Reprinted with permission from the American Society of Radiologic Technologists for educational purposes. ©2019. All rights reserved.