This web page offers all necessary information to determine the Δ-value between two solid state DFT codes within the PBE formalism. Δ is defined as the root-mean-square energy difference between the equations of state of the two codes, averaged over all crystals in a purely elemental benchmark set. This quantity can act as an accuracy-based guideline when selecting a solid state DFT code for a specific task. A README has been provided in the zip-file (see below), as well as the required input and script files. In addition, the code comparison database has been implemented in ASE. Further information and a discussion of results can be found in the following papers:
Kurt Lejaeghere, Veronique Van Speybroeck, Guido Van Oost and Stefaan Cottenier, Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals, Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014). (Open Access)
Kurt Lejaeghere, Gustav Bihlmayer, Torbjörn Björkman, Peter Blaha, Stefan Blügel, Volker Blum, Damien Caliste, Ivano E. Castelli, Stewart J. Clark, Andrea Dal Corso, Stefano de Gironcoli, Thierry Deutsch, John Kay Dewhurst, Igor Di Marco, Claudia Draxl, Marcin Dułak, Olle Eriksson, José A. Flores-Livas, Kevin F. Garrity, Luigi Genovese, Paolo Giannozzi, Matteo Giantomassi, Stefan Goedecker, Xavier Gonze, Oscar Grånäs, E. K. U. Gross, Andris Gulans, François Gygi, D. R. Hamann, Phil J. Hasnip, N. A. W. Holzwarth, Diana Iuşan, Dominik B. Jochym, François Jollet, Daniel Jones, Georg Kresse, Klaus Koepernik, Emine Küçükbenli, Yaroslav O. Kvashnin, Inka L. M. Locht, Sven Lubeck, Martijn Marsman, Nicola Marzari, Ulrike Nitzsche, Lars Nordström, Taisuke Ozaki, Lorenzo Paulatto, Chris J. Pickard, Ward Poelmans, Matt I. J. Probert, Keith Refson, Manuel Richter, Gian-Marco Rignanese, Santanu Saha, Matthias Scheffler, Martin Schlipf, Karlheinz Schwarz, Sangeeta Sharma, Francesca Tavazza, Patrik Thunström, Alexandre Tkatchenko, Marc Torrent, David Vanderbilt, Michiel J. van Setten, Veronique Van Speybroeck, John M. Wills, Jonathan R. Yates, Guo-Xu Zhang, Stefaan Cottenier, Reproducibility in density functional theory calculations of solids, Science 351 (6280), aad3000 (2016).
Click here to freely access a copy of the manuscript: abstract, full text, pdf.
All codes that have been assessed up until now, are mentioned in the following table. Where available, the full computatonal data and settings can be accessed by means of the file icon on the right. Please click the row of the code that you wish to see as a reference (WIEN2k is the default). Note that clicking a hyperlink takes you to the corresponding website rather than reorder the table.
Code developers and/or experts are invited to report the Δ-value of their code to us. We will try to keep this list up to date.
| Code | Version | Basis | Electron treatment | Δ-value | References & Data |
|---|---|---|---|---|---|
| WIEN2k | 13.1 | LAPW/APW+lo | all-electron | meV/atom | S. Cottenier [16] |
| ABINIT | 7.10.2 | plane waves | GPAW PAW 0.9 (80 Ha cut-off) | meV/atom | ASE [2,16] |
| RSPt | 1672 | LMTO | all-electron | meV/atom | RSPt [6,16] |
| OpenMX | 3.7 | pseudo-atomic orbitals | Morrison-Bylander-Kleinman norm-conserving (2013) | meV/atom | OpenMX [4,16] |
| GPAW | 0.10.0 | plane waves | PAW 0.9 | meV/atom | ASE [2,16] |
| VASP | 5.2.2 | plane waves | PAW 2007 | meV/atom | K. Lejaeghere et al. [1,16] |
| VASP | 5.2.12 | plane waves | PAW 2012 | meV/atom | K. Lejaeghere [16] |
| VASP | 5.2.12 | plane waves | PAW 2012 GW-ready | meV/atom | K. Lejaeghere |
| VASP | 5.2.12 | plane waves | PAW 2015 (5.4) | meV/atom | K. Lejaeghere |
| VASP | 5.2.12 | plane waves | PAW 2015 GW-ready (5.4) | meV/atom | K. Lejaeghere [16] |
| ABINIT | 7.5.3 | plane waves | PAW JTH | meV/atom | F. Jollet et al. [3,16] |
| GPAW | 0.8.0 | grid-based | PAW 0.6 | meV/atom | K. Lejaeghere et al. [1,16] |
| Quantum ESPRESSO | 5.0.2 | plane waves | PAW PSLibrary 0.3.1 | meV/atom | Küçükbenli et al. [5,16] |
| Quantum ESPRESSO | 5.1 | plane waves | GBRV 1.2 ultrasoft | meV/atom | ASE [2,16] |
| Dacapo | 2.7.16 | plane waves | Vanderbilt ultrasoft version 2 | meV/atom | ASE [2,16] |
| ABINIT | 7.6.4 | plane waves | Troullier-Martins norm-conserving (FHI) | meV/atom | ASE [2,16] |
| CASTEP | 9.0 | plane waves | OTFG CASTEP 9.0 | meV/atom | CASTEP [7,16] |
| CASTEP | 8.0 | plane waves | OTFG CASTEP 7.0 | meV/atom | CASTEP [7,16] |
| CASTEP | 8.0 | plane waves | OTFG Materials Studio | meV/atom | CASTEP [7] |
| CASTEP | 8.0 | plane waves | Vanderbilt ultrasoft | meV/atom | CASTEP [7,16] |
| ABINIT | 7.7.3 | plane waves | PAW JTH v0.2 | meV/atom | F. Jollet and M. Torrent [16] |
| ABINIT | 7.8.2 | plane waves | PAW JTH v1.0 | meV/atom | F. Jollet and M. Torrent |
| ABINIT | 7.2.0 | plane waves | PAW GBRV 1.0 (v1.01 for O and N) | meV/atom | F. Jollet et al. [3] |
| ABINIT | 7.10.2 | plane waves | PAW GBRV 1.2 | meV/atom | ASE [2,16] |
| FPLO | 14.00 | default local orbitals | all-electron | meV/atom | FPLO [8,16] |
| FPLO | 14.00 | enhanced local orbitals | all-electron | meV/atom | FPLO [8,16] |
| FPLO | 14.00 | enhanced local orbitals + fixed compact support radius | all-electron | meV/atom | FPLO [8,16] |
| FLEUR | 0.26 | LAPW (+lo) | all-electron | meV/atom | FLEUR [9,16] |
| FHI-aims | 081213 | tight numerical orbitals | all-electron (relativistic atomic_zora scalar) | meV/atom | ASE [2,16] (data) |
| FHI-aims | 081213 | light numerical orbitals | all-electron (relativistic atomic_zora scalar) | meV/atom | ASE [2] (data) |
| FHI-aims | 081213 | tier2 numerical orbitals | all-electron (relativistic atomic_zora scalar) | meV/atom | ASE [2,16] (data) |
| FHI-aims | 081213 | tier2 numerical orbitals | all-electron (relativistic zora scalar 1e-12) | meV/atom | ASE [2] (data) |
| Exciting | development version | LAPW+xlo | all-electron | meV/atom | Exciting [10,16] (data) |
| BigDFT | 1.7.6 | Daubechies wavelets | HGHk-semicore and NLCC 2013 norm-conserving | meV/atom | BigDFT [11] |
| BigDFT | 1.7.6 | Daubechies wavelets | HGHk-semicore and NLCC 2015 norm-conserving | meV/atom | BigDFT [11,16] |
| ABINIT | 7.10.2 | plane waves | HGHk norm-conserving | meV/atom | ASE [2] |
| ABINIT | 7.10.2 | plane waves | HGHk norm-conserving, semicore if available | meV/atom | ASE [2,16] |
| Quantum ESPRESSO | 5.1 | plane waves | GBRV 1.4 ultrasoft | meV/atom | ASE [2,16] |
| CASTEP | 9.0 | plane waves | GBRV 1.4 ultrasoft | meV/atom | CASTEP [7,16] |
| CASTEP | 17.2.1 | plane waves | GBRV 1.5 ultrasoft | meV/atom | CASTEP [7] |
| Quantum ESPRESSO | 5.1 | plane waves | Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving | meV/atom | ASE [2,16] |
| GPAW | 0.10.0 | grid-based | PAW 0.9 | meV/atom | ASE [2] |
| Quantum ESPRESSO | 5.1 | plane waves | PAW PSLibrary 1.0.0 | meV/atom | QuantumESPRESSO [12,16] |
| Quantum ESPRESSO | 5.1 | plane waves | SSSP Efficiency (mixed NC/US/PAW potential library) | meV/atom | QuantumESPRESSO [12] (data) |
| Quantum ESPRESSO | 5.1 | plane waves | SSSP Accuracy (mixed NC/US/PAW potential library) | meV/atom | QuantumESPRESSO [12,16] (data) |
| ABINIT | 7.11.8 | plane waves | pseudo_dojo_ONCVPSP 0.1 norm-conserving | meV/atom | ABINIT [13,16] |
| Elk | 3.1.5 | APW+lo | all-electron | meV/atom | Elk [14,16] |
| CASTEP | 9.0 | plane waves | Schlipf-Gygi ONCVPSP 2015-05-20 norm-conserving | meV/atom | CASTEP [7,16] |
| CASTEP | 9.0 | plane waves | Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving | meV/atom | CASTEP [7,16] |
| ATK | 2015 | pseudo-atomic orbitals | Morrison-Bylander-Kleinman norm-conserving (2013) | meV/atom | ATK/QuantumWise [15] |
| ATK | 2016 | LCAO High | Schlipf-Gygi ONCVPSP 2015-05-20 norm-conserving | meV/atom | ATK/QuantumWise [15] |
| Octopus | 8.0 | grid-based | Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving | meV/atom | Octopus [17] |
| Questaal | 7.14.1 | LMTO | all-electron | meV/atom | Questaal [18] |
| CASTEP | 19.1.1 | plane waves | OTFG CASTEP 19.1 | meV/atom | CASTEP [7,16] |
| CASTEP | 19.1.1 | plane waves | OTFG ONCVPSP norm-conserving 19.1 | meV/atom | CASTEP [7,16] |
| CASTEP | 19.1.1 | plane waves | OTFG ultrasoft for high-throughput QC5 | meV/atom | CASTEP [7,16] |
Hovering over the Δ-value for a particular code displays the standard deviation over all elements and clicking it displays an overview of the deviations for each individual element (in meV/atom). You can also calculate Δ for a limited number of elements, using the tool "Select Elements" above. The equation of state data for each code are included in the Delta calculation package that can be downloaded at the bottom of this page ('history' archive).
For questions or to report a Δ-value, please contact Kurt.Lejaeghere@UGent.be or Stefaan.Cottenier@UGent.be.
Updates
March 25 2016:
The results of this large-scale collaboration have now appeared in Science. Thanks to all contributors!
You can now access the computational settings and results for each code, by clicking the file icon to the right of each code.
September 24 2015:
The Delta package download now also contains primCIFs.tar.gz, an archive with the primitive unit cells of the 71 investigated crystals. Thanks to Andris Gulans for verifying these files!
September 14 2015:
 |
At the Psi-k 2015 meeting, the main results of the Δ-project were presented, and 18 of the many collaborators met on stage. See the video (left) for a 15-minute summary about the interpretation of the data set on this page. The group picture (right) shows from left to right: (front row) Nicola Marzari, Phil Hasnip, Stefaan Cottenier, Andris Gulans, Michiel van Setten, Sven Lubeck, Kay Dewhurst; (back row) Santanu Saha, François Gygi, Kurt Lejaeghere, Gian-Marco Rignanese, Peter Blaha, Kevin Garrity, Marc Torrent, Matt Probert, Gustav Bihlmayer, Keith Refson, José Flores-Livas.
May 9 2014:
Based on feedback from developers of codes en potentials, we have thoroughly modified the Δ-concept (version 3.0). You can now select any code to serve as a reference (by clicking on it in the table below), rather than only WIEN2k. The formula for Δ has also been symmetrized and is now the same for code 1 compared to code 2 as for code 2 compared to code 1. Finally, the WIEN2k data have been reassessed with rigorous accuracy settings and very small muffin-tin radii.
Future plans
We aim to extend the Δ-benchmark to multicomponent materials and more complex properties.
References
[1] K. Lejaeghere, V. Van Speybroeck, G. Van Oost and S. Cottenier, "Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals", Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014).
(Open Access)
[2] Marcin Dułak by means of ASE (2012-2015).
[3] F. Jollet, M. Torrent and N. Holzwarth, "Generation of Projector Augmented-Wave atomic data: A 71 element validated table in the XML format", Computer Physics Communications 185, 1246-1254 (2014).
[4] Taisuke Ozaki by means of OpenMX (2013).
[5] E. Küçükbenli et al., "Projector augmented-wave and all-electron calculations a cross the periodic table: a comparison of structural and energetic properties", arXiv:1404.3015 (2014).
[6] Torbjörn Björkman and John M. Wills by means of RSPt (2014).
[7] Chris Pickard and Keith Refson by means of CASTEP (2014-2019).
[8] Klaus Koepernik by means of FPLO (2014).
[9] Gustav Bihlmayer by means of FLEUR (2015).
[10] Andris Gulans and Sven Lubeck by means of Exciting (2015).
[11] Damien Caliste, Thierry Deutsch, Luigi Genovese and Santanu Saha by means of BigDFT (2015).
[12] Ivano Eligio Castelli by means of QuantumESPRESSO (2015).
[13] Matteo Giantomassi and Michiel J. van Setten by means of ABINIT (2015).
[14] José A. Flores Livas and John Kay Dewhurst by means of Elk (2015).
[15] Troels Markussen by means of ATK/QuantumWise (2015).
[16] K. Lejaeghere et al., "Reproducibility in density functional theory calculations of solids", Science 351 (6280), aad3000 (2016).
[17] Nicolas Tancogne-Dejean by means of Octopus (2018).
[18] Jerome Jackson, Dimitar Pashov and Mark van Schilfgaarde by means of Questaal (2019).
All values are in meV.
