The porous MIL-47 material shows a selective adsorption behavior for para-, ortho-, and meta-isomers of xylenes, making the material a serious candidate for separation applications. The origin of the selectivity lies in the differences in interactions (energetic) and confining (entropic). This paper investigates the xylene–framework interactions and the xylene–xylene interactions with quantum mechanical calculations, using a dispersion-corrected density functional and periodic boundary conditions to describe the crystal. First, the strength and geometrical characteristics of the optimal xylene–xylene interactions are quantified by studying the pure and mixed pairs in gas phase. An extended set of initial structures is created and optimized to sample as many relative orientations and distances as possible. Next, the pairs are brought in the pores of MIL-47. The interaction with the terephthalic linkers and other xylenes increases the stacking energy in gas phase (−31.7 kJ/mol per pair) by roughly a factor four in the fully loaded state (−58.3 kJ/mol per xylene). Our decomposition of the adsorption energy shows various trends in the contributing xylene–xylene interactions. The absence of a significant difference in energetics between the isomers indicates that entropic effects must be mainly responsible for the separation behavior.