Driving an automobile is a complex, dynamic, task requiring that a driver not only constructs both accurate perceptions and accurate interpretations about information affecting their situational skill, current conditions, and the surrounding traffic but also to process this information quickly. This study evaluated the use of a neural network to model driver visual scanning pattern data, as a classification strategy when a driver faces cognitive, emotional, and cognitive-manual secondary tasks. Visual scanning data were collected from 24 participants (12 male and 12 female) during an approximately two-hour drive. The drive consisted of segments having varying distraction characteristics. The simulated viewing environment of the driver was divided into nine sections to aid the analysis. Movement and fixations of the eyes were tracked into and out of each of these sections; these data were used in a neural network to model driver visual scanning responses to each situation. The model was able to differentiate performance for each gender and distraction type successfully, indicating the need to consider gender when developing interventions for combatting driver inattention.