We present results from the first multi-epoch X-ray/IR fast-photometry campaign on the black hole transient GX 339-4, during its 2015 outburst decay. We studied the evolution of the power spectral densities finding strong differences between the two bands. The X-ray power spectral density follows standard patterns of evolution, plausibly reflecting changes in the accretion flow. The IR power spectral density instead evolves very slowly, with a high-frequency break consistent with remaining constant at 0.63 ± 0.03 Hz throughout the campaign. We discuss this result in the context of the currently available models for the IR emission in black hole transients. While all models will need to be tested quantitatively against this unexpected constraint, we show that an IR-emitting relativistic jet that filters out the short-timescale fluctuations injected from the accretion inflow appears as the most plausible scenario.