LiDAR becomes a promising sensor in intelligent transportation for traffic data collection. Unlike the onboard LiDAR sensors which only focus on the surroundings of the vehicle, we set up LiDAR sensors at the infrastructure side to cover a wider detection range. In order to extract real-time movement status of road users, background filtering is the initial step to reduce a large amount of unrelated information. Since the LiDAR sensors are fixed at roadside, the detected background objects at a specific location are unchanged, which is different from dynamic backgrounds detected by onboard LiDAR sensors in a moving vehicle. This paper proposed an innovative real-time Azimuth-Height background filtering method for roadside LiDAR systems by comparing the heights between raw LiDAR data and background objects based on laser channels and azimuth angles. A case study showed more than 98% background data were successfully filtered and target (non-background) data were kept well. The proposed filtering process is performed while parsing original data stream and only target data are stored, thus the speed of processing LiDAR data in real time is increased, and computer memory is saved. The roadside LiDAR sensing systems provide a new approach to extracting pedestrian and vehicle trajectories.