The Probabilistic Matrix Factorization (PMF) model has been widely studied for recommender systems, which outperform previous models with a solid probabilistic explanation. To further improve its accuracy by using social information, researchers attempt to combine the PMF model with social network graphs by adding social terms. However, existing works on social terms do not provide theoretical explanations to make the models well understood. The lack of explanations limits further improvement of prediction accuracy. Hence, in this paper we provide our explanation and propose a unified covariance framework to solve this problem. Our explanation, including regularization terms, factorization terms and an ensemble of them, reveals how most social terms work from a probabilistic view. Our framework shows that those terms could be optimized in a direct way compatible to PMF. We find out that accuracy improvements for existing works on regularization terms rely more on personalized properties, and that social information for factorization terms is helpful but not always necessary.