Brittany Ann Kos

Grade-a-thons and Divide-and-Conquer: Effective Assessment at Scale

2017 – ASEE – Full Paper

This complete evidence-based practice paper will describe our successful grading and assessment practices of a large freshmen engineering course. In the Fall of 2016 we taught “Introduction to Engineering”, a course designed to help students transition from high school to college and learn strategies to help them become successful engineering students. Over 70% of the students had not yet declared an engineering major but had intentions to transfer to an engineering major the following spring semester. This was a 1-credit hour, online and in-person hybrid class, technologically managed by a Learning Management Software (LMS).

Over 700 students enrolled in the course, and our instructional team consisted of one Instructor, one graduate TA, and two undergraduate TAs. This paper reports evidence-based practice of two assessment methods, Divide-and-Conquer and Grade-a-thons, that we used to successfully evaluate a large-enrollment course with small grading staff. The coursework was divided into two types of assignments: weekly homework and a final report.

The design of the course was based on content that had been previously implemented at this large, midwestern institution, as well as best practices learned from introduction to engineering courses at other institutions. In particular, the final project was based on Ray Landis’ work (Landis 2013). The weekly assignments were 1-page essay assignments. We asked students to reflect on the course’s assigned in-person activities, reading, and videos and to create a personal plan that would set themselves up to becoming a successful engineering student. To the best of our understanding, this is the largest implementation of “Design Your Successful Engineering Path” that has been able to grade final reports at this scale.

Weekly assignments were assessed with Divide-and-Conquer style grading. Student assignments were divided by last name into three even sections; each TA was responsible for one section and performed a combination of “hand-grading” and “mass-grading”. Hand-grading involved looking at each student's assignment, making an assessment based on a rubric, writing comments, and assigning a grade. Mass-grading consisted of giving all students full credit for the assignment. Every week TAs would hand-grade between 80-100 assignments and mass-grade the remaining 130-150 assignments. They would change which students were selected for hand-grading every week, so no student went more than two weeks without being hand-graded. This strategy allowed three TAs to give select students meaningful feedback on their assignments and monitor student progress. The TAs would first complete the hand-grading, to monitor the quality of student work for that assignment. Through this hand-grading assessment, the TAs found that there was a consistent rate of 90-95% of students turning in completed work that followed all requirements, earning the students full-credit. Due to the overall quality of student submissions, we justified mass-grading: giving the remaining students full-credit would be a reasonable strategy, since an overwhelming majority of submissions would have been assigned full-credit with the hand-grading strategy.

The second type of assignment that required assessment was a final report. This report was an accumulation of the previous weekly assignments, where students were expected to create a cohesive strategy to becoming a successful student engineer. These reports were 9-11 pages in length, not including appendices. Since this assignment was significantly longer than the weekly assignments and required that every student be hand-graded, we created a “Grade-a-thon” event that enabled us to grade all student reports in two eight-hour sessions. The grade-a-thon augmented practices seen in hackathons and standardized AP grading assessment. We hired an additional 16 Graders for this event, paying each Grader $100 for the entire day. The grade-a-thon was hosted on a Saturday from 9-5 in a large conference and workspace on campus. Before the event, the TAs graded four student reports to create a standardized grading practice. At the beginning of the event, we performed a calibration session where the Graders could see these examples and practice grading reports themselves; through the calibration session, we created regulated grading practices for all grading staff. We catered lunch and provided snacks for the grading team as well as organized breaks and activities every two hours. This prevented burn-out and kept all graders engaged in grading. In the end, this format allowed all students to receive individual assessment and feedback, while simultaneously expediting grading time and at a relatively low financial cost to the university.

We developed the Divide-and-Conquer and Grade-a-thon strategies as alternatives to automated assessment or hiring full-time appointment-based TAs. Our experiences with this introduction to engineering course show us that we can achieve effective assessment for a large enrollment course with a small instructional staff.

Read More...

Kos, B. A., Miller, S.. 2017. Grade-a-thons and Divide-and-Conquer: Effective Assessment at Scale. ASEE '17: American Society of Engineering Education 124th Annual Conference & Exposition, (Columbus, OH, USA, 2017).

[Full Paper PDF]   [ASEE Link]   [Conference Presentation PDF]

Computer Science Principles: Impacting Student Motivation & Learning Within and Beyond the Classroom.

2016 – ICER – Full Paper

The Computer Science (CS) Principles framework seeks to broaden student participation and diversity in the field by focusing on the creative and social aspects of computing. As the pilot effort undergoes its early execution phases, this research contributes to the theoretical and practical application of CS Principles. We investigated the impact of CS Principles on student motivation and learning outcomes and sought to determine if the pedagogy created any lasting change on student perceptions of CS as a field of practice.

We report a case study of how CS Principles created an effective framework for introducing undergraduate students to the fundamentals of computer science. We discuss how Self-Determination Theory instantiates Self-Directed Learning, Constructionist, and Connectivist learning theories, which can be used to inform the pedagogical framework. Quantitative and qualitative measures were used to assess the impact of CS Principles on student motivation and learning outcomes, followed by an additional surveying of students one year after the completion of the course.

Results indicate that CS Principles facilitated positive programming experiences for students, helped increase learning interest and improve attitudes of CS as a field of study, positively changed perceptions of CS as a creative practice, and also encouraged students to continue learning CS after the course had finished. In particular, many non-majority students in the course self-reported to having positive changes and attitudes about CS explicitly because of the course. These finding suggest that CS Principles is a step in the right direction for creating more engaging and compelling curricula to diverse groups of students, especially those with minimal experience and exposure in the field. We discuss opportunities for future work using the selected theoretical framework for CS Principles.

Read More...

Behnke, K. A., Kos, B. A., Bennett, J. (2016). Computer Science Principles: Impacting Student Motivation & Learning Within and Beyond the Classroom. ICER '16 Proceedings of the 2016 ACM Conference on International Computing Education Research, (Melbourne, AUS, 2016), 171-180.

STEM Careers Infographic Project (SCIP): Teaching Media-Based Computational Thinking Practices

2015 – SIGSCE – Extended Abstract

The STEM Career Infographic Project (SCIP) was a 4-week exploratory project deployed in an 8th grade classroom at Mountain Vista Middle School (MVMS). SCIP was poised to address the growing focus on STEM fields at MVMS and within the school district. We piloted SCIP in Spring 2014 with six science classes or about 180 students. SCIP allowed for students to explore their own STEM interests, while simultaneously engaging in the 6 Computational Thinking Practices (CTP) outlined by the College Board.

Students were required to research a STEM career in-depth, then report on their careers using infographics (CTP #2: Creating Computational Artifacts and CTP #3: Abstracting). We used free and online programs to create the infographics; this provided the students the opportunity to learn software they were not previously exposed to and to explore new communication tools (CTP #1: Connecting Computing and CTP #2: Analyzing Problems and Artifacts). SCIP also provided many occasions for the students to work together by sharing career information or helping each other with the software (CTP #6: Collaborating). At the end of the project the students presented their infographics in front of the class and taught their classmates about their career (CTP #5: Communicating).

The project was incredibly successful. The students had a positive affect through the duration of the project and many also expressed an extreme level of interest in doing similar projects in the future. We will be repeating this project in Spring 2015, with a few adaptations and formal evaluation scheme.

Read More...

Kos, B. A., Sims, E. (2015). STEM Careers Infographic Project (SCIP): Teaching Media-Based Computational Thinking Practices. SIGCSE ‘15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education, (Kansas City, MO, USA, 2015), 681.

[Extended Abstract PDF]   [ACM Link]   [Conference Poster PDF]

Infographics: The New 5-Paragraph Essay

2014 – RMCWiC – Full Paper – Best paper

The STEM Career Infographic Project (SCIP) was a 5-week exploratory project deployed in an 8th grade classroom at Mountain Vista Middle School (MVMS) in the spring of 2014. Students were required to research a STEM career in-depth, then report on their careers using infographics, in lieu of a standard 5-paragraph essay. SCIP was broken down into 9 days of instruction: introduction, research, three days of design lecture, three work days, and a final presentation day. The students were in the lab working on their infographics every day. We observed that infographics were better suited than traditional essays in areas that involved creativity and visual appeal, limited writing for ESL (English as a Second Language) students, fostering and appealing to student’s interests, and overall student enjoyment. Some of the negative obstacles we encountered revolved around limitations of free and online software, addressing the learning curve of technology, and altering student’s expectations of reporting tools. Overall, we considered SCIP a success because of the positive affect we recognized in the students through the duration of the project.

Read More...

Kos, B. A., Sims, E. (2014). Infographics: The New 5-Paragraph Essay. 2014 Rocky Mountain Celebration of Women in Computing, (Laramie, WY, USA, 2014).