This study investigated behavioral characteristics of users with visual impairments and tested effect of factors regarding the layout of virtual environments (VEs). Various three-dimensional (3D) VEs were simulated with two different factors: number of objects and layout type (random, symmetric). Using a Novint Falcon haptic device, users with visual impairments were required to complete an object recognition task in 3D VEs with different levels of number of object and layout. The characteristics of their movements (speed, applied force, location, direction, etc.) were recorded, and participants evaluated perceived difficulty after they completed each trial. We analyzed their recorded movements and their rating on perceived difficulty. Results showed that 1) number of objects in 3D VE had significant impact on visually impaired users' behavior; 2) different layout had not showed significant influence on their movement; 3) increased number of objects in 3D VE made the task more difficult; 4) visualized results implied that different users had significant different behavior preference in the same 3D VE. It is expected that the results of this study can improve behavioral understanding of users with visual impairments and guidance for assistive technology development for users with visual impairments.