The similarities between Hartree-Fock (HF) theory and the density matrix renormalization group (DMRG) are explored. Both methods can be formulated as the variational optimization of a wave-function Ansatz. Linearization of the time-dependent variational principle near a variational minimum allows to derive the random phase approximation (RPA). We show that the nonredundant parameterization of the matrix product state (MPS) tangent space [J. Haegeman, J. I. Cirac, T. J. Osborne, I. Pižorn, H. Verschelde and F. Verstraete, Phys. Rev. Lett. 107, 070601 (2011)] leads to the Thouless theorem for MPS, i.e., an explicit nonredundant parameterization of the entire MPS manifold, starting from a specific MPS reference. Excitation operators are identified, which extends the analogy between HF and DMRG to the Tamm-Dancoff approximation (TDA), the configuration interaction (CI) expansion, and coupled cluster theory. For a small one-dimensional Hubbard chain, we use a CI-MPS Ansatz with single and double excitations to improve on the ground state and to calculate low-lying excitation energies. For a symmetry-broken ground state of this model, we show that RPA-MPS allows to retrieve the Goldstone mode. We also discuss calculations of the RPA-MPS correlation energy. With the long-range quantum chemical Pariser-Parr-Pople Hamiltonian, low-lying TDA-MPS and RPA-MPS excitation energies for polyenes are obtained.