The hindered rotor transitions of H2 adsorbed in the chemically related and prototypical porous metal-organic frameworks IRMOF-1, IRMOF-8, IRMOF-11, and MOF-177 were studied by inelastic neutron scattering to gain information on the specifics of H2 binding in this class of adsorbents. Remarkably sharp and complex spectra of these materials signify a diversity of well-defined binding sites. Similarities in the spectral features as a function of H2 loading and correlations with recent crystallographic studies were used to assign transitions ranging in rotational barrier from <0.04 to 0.6 kcal/mol as corresponding to localized adsorption sites on the organic and inorganic components of these frameworks. We find that binding of H2 at the inorganic cluster sites is affected by the nature of the organic link and is strongest in IRMOF-11 in accord with our adsorption isotherm data. The sites on the organic link have lower binding energies, but a much greater capacity for increases in H2 loading, which demonstrates their importance for hydrogen uptake by these materials.