Skip to content

GitLab

Last edited by Matthias Redies
Page history

Home

Developer's Wiki for FLEUR

Please note that this is not the point to put user documentation, which should go to the FLEUR-homepage instead. Use this Wiki for information relevant for FLEUR developers only.

Doxygen

We use doxygen to create the documentation of the source. This can be found here.

Coverage

The automatic tests of FLEUR cover only part of the source. Here you find the analysis.

Useful info for developers

Using fleur with the HDF5 library and debugging it with valgrind

HDF5 has to be built with the same compiler that is also used to compile fleur. If adapted the following commands can be used to compile a HDF5 library for fleur:

  • 'FC=/usr/local/intel/impi/4.0.3.008/intel64/bin/mpiifort CC=/usr/local/intel/impi/4.0.3.008/intel64/bin/mpiicc CXX=/usr/local/intel/impi/4.0.3.008/intel64/bin/mpiicc ./configure --enable-fortran --enable-fortran2003 --enable-parallel --enable-using-memchecker --enable-clear-file-buffers'
  • 'make'
  • 'make install'
  • 'make check' (optional)

Note:

  • The paths have to be adjusted such that that compiler is used which is also used to compile fleur.
  • The parallel switch is not needed for every calculation: Only for parallel calculations in which HDF5 is also used for the eigenvector IO.
  • The last two command line switches in the configure command turn on initializations of irrelevant array parts in HDF5. If valgrind is not needed it is probably the better choice to leave them away. If left away valgrind will complain about missing initializations in the HDF5 library.
  • valgrind gives partially strange behavior if used together with the intel compiler. It would be better to use it together with gfortran.
  • At the moment HDF5 is needed in version 1.8.. Usage of version 1.10. yields some problems.

Furthermore to configure and start fleur with HDF5 the following has to be done:

  • In your .bashrc the HDF5 library has to be added to the LD_LIBRARY_PATH. This implies a line like 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/hdf5/current/hdf5/lib'
  • configure fleur with some line like 'CMAKE_Fortran_FLAGS="-I~/hdf5/current/hdf5/include" FLEUR_LIBRARIES="-L~/hdf5/current/hdf5/lib;-lhdf5_fortran;-lhdf5" ./fleur/configure.sh IFF'

Compiling Fleur on Jureca

To compile Fleur on Jureca the following modules need to be loaded:

module load intel-para CMake HDF5 libxml2/.2.9.7 ELPA/2017.11.001-hybrid

which is then configured with

./configure.sh JURECA_INTEL

or

./configure.sh -external libxc JURECA_INTEL

if libXC should be supported aswell.

Compilling Fleur on Jureca Booster

To compile on the booster you need to first switch to the KNL environment

module purge
ml Architecture/KNL
module load intel-para CMake HDF5 libxml2/.2.9.7 ELPA/2017.11.001-hybrid

Then you can compile as discrebed above.

Running Fleur on Jureca Booster

To run on the Boosters you need to switch the architecture in each job script aswell

#!/bin/bash -x
#SBATCH --nodes=1
##SBATCH --ntasks=10
##SBATCH --ntasks-per-node=10
#SBATCH --cpus-per-task=68
#SBATCH --output=mpi-%j.out
#SBATCH --error=mpi-%j.err
#SBATCH --time=1:00:00
#SBATCH --partition=develbooster
#SBATCH --gres=mem96
##SBATCH --mail-user=your.name@fz-juelich.de
##SBATCH --mail-type=END
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module purge
ml Architecture/KNL
module load intel-para CMake HDF5 libxml2/.2.9.7 ELPA/2017.11.001-hybrid
/work/ias-1/s.rost/fleur_booster/fleur/build/fleur

Building Fleur on the GPUs (JURECA-DC Cluster)

module load NVHPC/21.1-GCC-9.3.0 HDF5 ParaStationMPI imkl CMake
FC=nvfortran ./configure.sh  -gpu acc:cc80

Profiling Fleur on the GPUs (JURECA-DC Cluster)

  • add start and stop markers
call nvtxStartRange("name",integer_colour_code)
call smith_you_want_to_measure()
call nvtxEndRange
  • include the nvtx module
use nvtx

  • compile the code with a flag "-lnvToolsExt" added

  • run with the profiler

nsys profile --trace=nvtx,cuda ./exe

##Running on iffslurm Iffslurm has a very strange configuration. Therefore if you set --cpus-per-task=64 you'll get 64 logical & 32 physical cores. So you need to set --cpus-per-task=128 to get all cores and then OMP_NUM_THREADS=64 to use each core only once. For example:

#!/bin/bash
#SBATCH --job-name=job
#SBATCH --nodes=1                    # Run all processes on a single node    
#SBATCH --ntasks=1                   # Run a single task    
#SBATCH --cpus-per-task=128           # Number of CPU cores per task
#SBATCH --time=24:00:00              # Time limit hrs:min:sec
#SBATCH --output=slurm-%j.log     # Standard output and error log
#SBATCH -p th1-2020-64
 
export OMP_NUM_THREADS=64
export OMP_PROC_BIND=spread
export I_MPI_PIN=enable 

ulimit -c unlimited
ulimit -s unlimited

source compiler-select intel-fi
 
srun ~/fleur/build/fleur_MPI -trace

FLEUR on ifflslurm

On ifflslurm there are precompiled binaires in /local/th1/DFT for each release/