Comparing Solid State DFT Codes, Basis Sets and Potentials

This web page offers all necessary information to determine the Δ-value between two solid state DFT codes within the PBE formalism. Δ is defined as the root-mean-square energy difference between the equations of state of the two codes, averaged over all crystals in a purely elemental benchmark set. This quantity can act as an accuracy-based guideline when selecting a solid state DFT code for a specific task. A README has been provided in the zip-file (see below), as well as the required input and script files. In addition, the code comparison database has been implemented in ASE. Further information and a discussion of results can be found in the following papers:

Kurt Lejaeghere, Veronique Van Speybroeck, Guido Van Oost and Stefaan Cottenier, Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals, Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014). (Open Access)

Kurt Lejaeghere, Gustav Bihlmayer, Torbjörn Björkman, Peter Blaha, Stefan Blügel, Volker Blum, Damien Caliste, Ivano E. Castelli, Stewart J. Clark, Andrea Dal Corso, Stefano de Gironcoli, Thierry Deutsch, John Kay Dewhurst, Igor Di Marco, Claudia Draxl, Marcin Dułak, Olle Eriksson, José A. Flores-Livas, Kevin F. Garrity, Luigi Genovese, Paolo Giannozzi, Matteo Giantomassi, Stefan Goedecker, Xavier Gonze, Oscar Grånäs, E. K. U. Gross, Andris Gulans, François Gygi, D. R. Hamann, Phil J. Hasnip, N. A. W. Holzwarth, Diana Iuşan, Dominik B. Jochym, François Jollet, Daniel Jones, Georg Kresse, Klaus Koepernik, Emine Küçükbenli, Yaroslav O. Kvashnin, Inka L. M. Locht, Sven Lubeck, Martijn Marsman, Nicola Marzari, Ulrike Nitzsche, Lars Nordström, Taisuke Ozaki, Lorenzo Paulatto, Chris J. Pickard, Ward Poelmans, Matt I. J. Probert, Keith Refson, Manuel Richter, Gian-Marco Rignanese, Santanu Saha, Matthias Scheffler, Martin Schlipf, Karlheinz Schwarz, Sangeeta Sharma, Francesca Tavazza, Patrik Thunström, Alexandre Tkatchenko, Marc Torrent, David Vanderbilt, Michiel J. van Setten, Veronique Van Speybroeck, John M. Wills, Jonathan R. Yates, Guo-Xu Zhang, Stefaan Cottenier, Reproducibility in density functional theory calculations of solids, Science 351 (6280), aad3000 (2016).
Click here to freely access a copy of the manuscript: abstract, full text, pdf.

All codes that have been assessed up until now, are mentioned in the following table. Where available, the full computatonal data and settings can be accessed by means of the file icon on the right. Please click the row of the code that you wish to see as a reference (WIEN2k is the default). Note that clicking a hyperlink takes you to the corresponding website rather than reorder the table.

Code developers and/or experts are invited to report the Δ-value of their code to us. We will try to keep this list up to date.

CodeVersionBasisElectron treatmentΔ-valueReferences & Data
WIEN2k13.1LAPW/APW+loall-electron meV/atomS. Cottenier [16]
ABINIT7.10.2plane wavesGPAW PAW 0.9 (80 Ha cut-off) meV/atomASE [2,16]
RSPt1672LMTOall-electron meV/atomRSPt [6,16]
OpenMX3.7pseudo-atomic orbitalsMorrison-Bylander-Kleinman norm-conserving (2013) meV/atomOpenMX [4,16]
GPAW0.10.0plane wavesPAW 0.9 meV/atomASE [2,16]
VASP5.2.2plane wavesPAW 2007 meV/atomK. Lejaeghere et al. [1,16]
VASP5.2.12plane wavesPAW 2012 meV/atomK. Lejaeghere [16]
VASP5.2.12plane wavesPAW 2012 GW-ready meV/atomK. Lejaeghere
VASP5.2.12plane wavesPAW 2015 (5.4) meV/atomK. Lejaeghere
VASP5.2.12plane wavesPAW 2015 GW-ready (5.4) meV/atomK. Lejaeghere [16]
ABINIT7.5.3plane wavesPAW JTH meV/atomF. Jollet et al. [3,16]
GPAW0.8.0grid-basedPAW 0.6 meV/atomK. Lejaeghere et al. [1,16]
Quantum ESPRESSO5.0.2plane wavesPAW PSLibrary 0.3.1 meV/atomKüçükbenli et al. [5,16]
Quantum ESPRESSO5.1plane wavesGBRV 1.2 ultrasoft meV/atomASE [2,16]
Dacapo2.7.16plane wavesVanderbilt ultrasoft version 2 meV/atomASE [2,16]
ABINIT7.6.4plane wavesTroullier-Martins norm-conserving (FHI) meV/atomASE [2,16]
CASTEP9.0plane wavesOTFG CASTEP 9.0 meV/atomCASTEP [7,16]
CASTEP8.0plane wavesOTFG CASTEP 7.0 meV/atomCASTEP [7,16]
CASTEP8.0plane wavesOTFG Materials Studio meV/atomCASTEP [7]
CASTEP8.0plane wavesVanderbilt ultrasoft meV/atomCASTEP [7,16]
ABINIT7.7.3plane wavesPAW JTH v0.2 meV/atomF. Jollet and M. Torrent [16]
ABINIT7.8.2plane wavesPAW JTH v1.0 meV/atomF. Jollet and M. Torrent
ABINIT7.2.0plane wavesPAW GBRV 1.0 (v1.01 for O and N) meV/atomF. Jollet et al. [3]
ABINIT7.10.2plane wavesPAW GBRV 1.2 meV/atomASE [2,16]
FPLO14.00default local orbitalsall-electron meV/atomFPLO [8,16]
FPLO14.00enhanced local orbitalsall-electron meV/atomFPLO [8,16]
FPLO14.00enhanced local orbitals + fixed compact support radiusall-electron meV/atomFPLO [8,16]
FLEUR0.26LAPW (+lo)all-electron meV/atomFLEUR [9,16]
FHI-aims081213tight numerical orbitalsall-electron (relativistic atomic_zora scalar) meV/atomASE [2,16] (data)
FHI-aims081213light numerical orbitalsall-electron (relativistic atomic_zora scalar) meV/atomASE [2] (data)
FHI-aims081213tier2 numerical orbitalsall-electron (relativistic atomic_zora scalar) meV/atomASE [2,16] (data)
FHI-aims081213tier2 numerical orbitalsall-electron (relativistic zora scalar 1e-12) meV/atomASE [2] (data)
Excitingdevelopment versionLAPW+xloall-electron meV/atomExciting [10,16] (data)
BigDFT1.7.6Daubechies waveletsHGHk-semicore and NLCC 2013 norm-conserving meV/atomBigDFT [11]
BigDFT1.7.6Daubechies waveletsHGHk-semicore and NLCC 2015 norm-conserving meV/atomBigDFT [11,16]
ABINIT7.10.2plane wavesHGHk norm-conserving meV/atomASE [2]
ABINIT7.10.2plane wavesHGHk norm-conserving, semicore if available meV/atomASE [2,16]
Quantum ESPRESSO5.1plane wavesGBRV 1.4 ultrasoft meV/atomASE [2,16]
CASTEP9.0plane wavesGBRV 1.4 ultrasoft meV/atomCASTEP [7,16]
CASTEP17.2.1plane wavesGBRV 1.5 ultrasoft meV/atomCASTEP [7]
Quantum ESPRESSO5.1plane wavesSchlipf-Gygi ONCVPSP 2015-01-24 norm-conserving meV/atomASE [2,16]
GPAW0.10.0grid-basedPAW 0.9 meV/atomASE [2]
Quantum ESPRESSO5.1plane wavesPAW PSLibrary 1.0.0 meV/atomQuantumESPRESSO [12,16]
Quantum ESPRESSO5.1plane wavesSSSP Efficiency (mixed NC/US/PAW potential library) meV/atomQuantumESPRESSO [12] (data)
Quantum ESPRESSO5.1plane wavesSSSP Accuracy (mixed NC/US/PAW potential library) meV/atomQuantumESPRESSO [12,16] (data)
ABINIT7.11.8plane wavespseudo_dojo_ONCVPSP 0.1 norm-conserving meV/atomABINIT [13,16]
Elk3.1.5APW+loall-electron meV/atomElk [14,16]
CASTEP9.0plane wavesSchlipf-Gygi ONCVPSP 2015-05-20 norm-conserving meV/atomCASTEP [7,16]
CASTEP9.0plane wavesSchlipf-Gygi ONCVPSP 2015-01-24 norm-conserving meV/atomCASTEP [7,16]
ATK2015pseudo-atomic orbitalsMorrison-Bylander-Kleinman norm-conserving (2013) meV/atomATK/QuantumWise [15]
ATK2016LCAO HighSchlipf-Gygi ONCVPSP 2015-05-20 norm-conserving meV/atomATK/QuantumWise [15]
Octopus8.0grid-basedSchlipf-Gygi ONCVPSP 2015-01-24 norm-conserving meV/atomOctopus [17]
Questaal7.14.1LMTOall-electron meV/atomQuestaal [18]
CASTEP19.1.1plane wavesOTFG CASTEP 19.1 meV/atomCASTEP [7,16]
CASTEP19.1.1plane wavesOTFG ONCVPSP norm-conserving 19.1 meV/atomCASTEP [7,16]
CASTEP19.1.1plane wavesOTFG ultrasoft for high-throughput QC5 meV/atomCASTEP [7,16]
H
He
Li
Be
B
C
N
O
F
Ne
Na
Mg
Al
Si
P
S
Cl
Ar
K
Ca
Sc
Ti
V
Cr
Mn
Fe
Co
Ni
Cu
Zn
Ga
Ge
As
Se
Br
Kr
Rb
Sr
Y
Zr
Nb
Mo
Tc
Ru
Rh
Pd
Ag
Cd
In
Sn
Sb
Te
I
Xe
Cs
Ba
Hf
Ta
W
Re
Os
Ir
Pt
Au
Hg
Tl
Pb
Bi
Po
At
Rn
Fr
Ra
La
Ce
Pr
Nd
Pm
Sm
Eu
Gd
Tb
Dy
Ho
Er
Tm
Yb
Lu
Ac
Th
Pa
U
Np
Pu
Am
Cm
Bk
Cf
Es
Fm
Md
No
Lr
Alkali metal
Alkaline earth metal
Transition metal
Other metal
Metalloid
Nonmetal
Halogen
Noblegas
Lanthanide
Actinide

Hovering over the Δ-value for a particular code displays the standard deviation over all elements and clicking it displays an overview of the deviations for each individual element (in meV/atom). You can also calculate Δ for a limited number of elements, using the tool "Select Elements" above. The equation of state data for each code are included in the Delta calculation package that can be downloaded at the bottom of this page ('history' archive).

For questions or to report a Δ-value, please contact Kurt.Lejaeghere@UGent.be or Stefaan.Cottenier@UGent.be.  

Updates

March 25 2016:

The results of this large-scale collaboration have now appeared in Science. Thanks to all contributors!

You can now access the computational settings and results for each code, by clicking the file icon to the right of each code.

September 24 2015:

The Delta package download now also contains primCIFs.tar.gz, an archive with the primitive unit cells of the 71 investigated crystals. Thanks to Andris Gulans for verifying these files!

September 14 2015:

At the Psi-k 2015 meeting, the main results of the Δ-project were presented, and 18 of the many collaborators met on stage. See the video (left) for a 15-minute summary about the interpretation of the data set on this page. The group picture (right) shows from left to right: (front row) Nicola Marzari, Phil Hasnip, Stefaan Cottenier, Andris Gulans, Michiel van Setten, Sven Lubeck, Kay Dewhurst; (back row) Santanu Saha, François Gygi, Kurt Lejaeghere, Gian-Marco Rignanese, Peter Blaha, Kevin Garrity, Marc Torrent, Matt Probert, Gustav Bihlmayer, Keith Refson, José Flores-Livas.

May 9 2014:
Based on feedback from developers of codes en potentials, we have thoroughly modified the Δ-concept (version 3.0). You can now select any code to serve as a reference (by clicking on it in the table below), rather than only WIEN2k. The formula for Δ has also been symmetrized and is now the same for code 1 compared to code 2 as for code 2 compared to code 1. Finally, the WIEN2k data have been reassessed with rigorous accuracy settings and very small muffin-tin radii.

Future plans

We aim to extend the Δ-benchmark to multicomponent materials and more complex properties.

References

[1] K. Lejaeghere, V. Van Speybroeck, G. Van Oost and S. Cottenier, "Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals", Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014).
(Open Access)
[2] Marcin Dułak by means of ASE (2012-2015).
[3] F. Jollet, M. Torrent and N. Holzwarth, "Generation of Projector Augmented-Wave atomic data: A 71 element validated table in the XML format", Computer Physics Communications 185, 1246-1254 (2014).
[4] Taisuke Ozaki by means of OpenMX (2013).
[5] E. Küçükbenli et al., "Projector augmented-wave and all-electron calculations a cross the periodic table: a comparison of structural and energetic properties", arXiv:1404.3015 (2014).
[6] Torbjörn Björkman and John M. Wills by means of RSPt (2014).
[7] Chris Pickard and Keith Refson by means of CASTEP (2014-2019).
[8] Klaus Koepernik by means of FPLO (2014).
[9] Gustav Bihlmayer by means of FLEUR (2015).
[10] Andris Gulans and Sven Lubeck by means of Exciting (2015).
[11] Damien Caliste, Thierry Deutsch, Luigi Genovese and Santanu Saha by means of BigDFT (2015).
[12] Ivano Eligio Castelli by means of QuantumESPRESSO (2015).
[13] Matteo Giantomassi and Michiel J. van Setten by means of ABINIT (2015).
[14] José A. Flores Livas and John Kay Dewhurst by means of Elk (2015).
[15] Troels Markussen by means of ATK/QuantumWise (2015).
[16] K. Lejaeghere et al., "Reproducibility in density functional theory calculations of solids", Science 351 (6280), aad3000 (2016).
[17] Nicolas Tancogne-Dejean by means of Octopus (2018).
[18] Jerome Jackson, Dimitar Pashov and Mark van Schilfgaarde by means of Questaal (2019).

All values are in meV.

H
He
Li
Be
B
C
N
O
F
Ne
Na
Mg
Al
Si
P
S
Cl
Ar
K
Ca
Sc
Ti
V
Cr
Mn
Fe
Co
Ni
Cu
Zn
Ga
Ge
As
Se
Br
Kr
Rb
Sr
Y
Zr
Nb
Mo
Tc
Ru
Rh
Pd
Ag
Cd
In
Sn
Sb
Te
I
Xe
Cs
Ba
Hf
Ta
W
Re
Os
Ir
Pt
Au
Hg
Tl
Pb
Bi
Po
At
Rn
Fr
Ra
La
Ce
Pr
Nd
Pm
Sm
Eu
Gd
Tb
Dy
Ho
Er
Tm
Yb
Lu
Ac
Th
Pa
U
Np
Pu
Am
Cm
Bk
Cf
Es
Fm
Md
No
Lr