Using paper-and-pencil solutions to assess problem solving skill

BACKGROUND Problem solving practice involving paper-and-pencil solutions is a common activity in undergraduate engineering training. This paper addresses three questions: What indicators of problem-solving skill are available in paper-and-pencil solutions, beyond a simple accuracy measure? Can these indicators distinguish between problem solvers in terms of their skill level? How can these indicators be used to improve the effectiveness of instructor feedback to problem-solving exercises? PURPOSE (HYPOTHESIS) Based on the published literature on cognition and problem solving, several indicators were hypothesized to signal knowledge and skill: an accurate free-body diagram, specification of assumptions, accurate expansion of key principles, strategic (forward) inferences, and checking equations and solutions. DESIGN/METHOD A mixed experimental methodology was applied that combined qualitative and quantitative analyses. The qualitative data was comprised of paper-and-pencil solutions and video recordings of participants while they solved problems. Frequency counts of skill indicators and grader-assigned scores comprised the quantitative data. RESULTS Statistical tests confirmed that the hypothesized indicators were associated with individuals' level of skill. However, reliable evidence for these indicators was only partially found in the paper-and-pencil solutions and could only be firmly established by including data from the video recordings. CONCLUSIONS The results show that paper-and-pencil solutions can be diagnostic of the strengths and weaknesses of problem-solvers. However, to provide useful and reliable feedback to students, these solutions must be supplemented with additional input, possibly through regular structured sampling throughout a course, or through more extensive use of computer resources.