Due to the growing availability of huge amounts of data of different types and the growing capabilities to analyze these data, the expectations of big data applications are high. In this paper, we argue that the usability of big data in the social domain is far from trivial. If the outcomes of big data are wrongly interpreted, this may shape the development of our society in a wrong direction. Therefore, care should be taken of a proper interpretation of big data outcomes and its applications in real-life. To support such an interpretation, we distinguish three major building blocks in big data, the data as input for analyses, the algorithms to analyze the data, and the models as output of the analyses. We show that each of the building blocks entail different complications for a proper interpretation of big data outcomes in practice. Therefore, well thought-through strategies are required for using big data outcomes in a responsible way. We discuss a framework for such strategies.