Moving Target Defense (MTD) is one of the game changing ideas introduced to strengthen defenders and weaken adversaries in a networked environment. One of the well-known strategies is to change randomly and more frequently the network IP addresses of critical clients so that the network locations of the clients are less recognizable. The goal is therefore to minimize the time that the location of a host computer is exposed to attackers. Several moving target defense mechanisms have been introduced to demonstrate the effectiveness of IP address-based MTD strategies. To the best of our knowledge, however, there is no study with its focus on investigating the impact of both the IP address change rate and the host diversity on the success of the defense and in turn the failure of attacks. We present the results of an analysis performed on simulating a simplified attack scenario against hosts on a network. The paper investigates the influence of host IP address change rate and host complexity on the success rate of attacks. The results show that both the host IP address change rate and the attack crafting time have significant impact on making attacks unsuccessful.