For this project, I am attempting to use text analysis software to identify
any possible differences in the “My Goals” statement written by “successful” students
vs. “unsuccessful” students in their ePortfolios. It might be useful if such
goal statements, written at the outset of students’ careers at the college,
were to prove predictive of future success or lack thereof, perhaps enabling
the college to provide greater support to students less likely to succeed.
For the purposes of this project, we gathered 2841 goal
statement “pages” that were created by students between 9/16/2010 and 6/15/201
from our Digication ePortfolio system. We wanted statements written closer to
the beginning of students’ time here, so we limited selection to students with
0-20 credits (which yielded 1065). Of these 1065, only 511 had actual text in
them.
This data was sent to the office of Institutional
Research to identify two groups: students who have been successful (defined as
graduated or transferred early) vs. unsuccessful (defined as no longer in
attendance, but neither graduated nor transferred early). The complete statistics generated for
each students included the number of semesters enrolled starting in Fall 2010,
final GPA, whether graduated (1), whether transferred (1), whether graduated or
transferred (1), and credits attempted, credits earned and GPA for Fall 2010
and Spring 2011.
Professor Dragan then provided massive assistance by creating an “interface”
called ePortfolio Explorer that allowed the data base to be searched and easily
loaded into text analysis software.
The goal statements (combined into 1 document for successful students vs. 1
document for unsuccessful students) were then run through Voyant.
OUTCOMES
There is 1 document in this corpus
with a total of 73,081 words and 5,993 unique words.
Most frequent words in the corpus: career (574), college
(495), work (358), goals (323), degree
(319).
Unsuccessful (=neither graduated nor transferred):
There is 1 document in this corpus
with a total of 73,020 words and 5,997 unique words.
Most frequent words in the corpus: career (569), college
(558), goals (470), want (365), like
(300).
DISCUSSION
At this point, I haven't found any particular insights, and am not quite sure how to pursue the analysis this further (any thoughts would be welcome!). For both successful and unsuccessful students, 3 of the top five words were career, college and goals. While degree was in the top five for successful students (323 occurrences), it was mentioned fairly often by unsuccessful students as well (9th with 248 occurrences). The other 2 top five words for unsuccessful students, want and like, also appear in the top 10 words for successful students. So maybe what I can say at this point is that both successful and unsuccessful students are entering college with goals that are articulated in similar ways (though the individual goals themselves differ widely).
The premise of you project is so promising, and it's hard not to want some instant "wow" conclusion once you've put in all that data. I think we've all learned that one of the commonalities in Digital and Traditional Humanities is that there is always a lot more research to be done - a larger sample, a different way of framing the questions, or alternate sources of material to be analyzed.
ReplyDeleteI think your concluding point is important- that there is a commonality amongst all the students who are starting out, and that maybe the data reflects the success of them actually enrolling and beginning college. It would be great- but difficult or impossible- to have a goals essay when they exited out- whether with degree, transfer or neither. That might offer more clues.
I enjoyed your post!
I agree with Dahlia that this project is so promising. I had a thought, as I often do, while looking at the word frequencies. For the "unsuccessful" students, "want" was one of the top five most frequently occurring words. As a rhetorician, I place great value in word choice as a means to understand meaning and intent. That said, it strikes me that these students "want" certain results, but might lack any clearly identified means of achieving them, while the "successful" students incorporate "work" as one of their most frequently occurring words. Wanting a degree and working for a degree seem to me to be very different in terms of motivation and willingness to do what is necessary to achieve one's goals. Moreover, I am struck by the use of "like" in so many of the mission statements for the "unsuccessful" students. What does that mean? Do these students need or want to like school, like their classes, like their professors...? Or do they want to find a job or career that they like? This is such an ambiguous term (as is "want") when compared with the very specific terms used by the "successful" students. Those students used words like "degree," which is very specific and, in many ways, represents their understanding of the steps necessary to achieve their goals, be it an Associate's, Bachelor's, Master's, or Doctoral degree. By contrast, the "unsuccessful" students mention "career" and "goals," but not the steps necessary to achieve them.
ReplyDeleteThese are just my preliminary thoughts. I don't know what instructions students are provided as they draft their mission statements; perhaps more specificity in the instructions might lead to more focuses mission statements that might in turn yield more insights when word frequency analyses are performed. Or perhaps the word frequency analysis might be combined with another analysis of grades earned in the core, general education courses (I'm thinking of the Pathways Required Core here--composition, math, and science) in order to determine if the successful students earn higher grades in these core courses than do the unsuccessful students.
As somebody who is working on the team developing the New to College Seminar for Humanities/Liberal Arts students, I would be interested in developing strategies for identifying at-risk or potential "unsuccessful" students in order to help them refocus their goals or at least understand what is necessary to become "successful." I think your project is a step in that direction.