Weird memory usage with second variation SOC and ev parallelization
Summary
I often encounter calculations on iffslurm, where Fleur runs out of memory after 10 or 20 iterations. I discovered that this happens more often with second variation SOC, so I decided to make a few tests. I used the input below with second variation SOC with and without eigenvalue parallelization NdCo5_evpara.job NdCo5_purekpara.job and looked at the memory usage by patching the data for a console output from a run with -debugtime together. I also performed a run with first variation SOC and eigenvalue parallelization.
In these files I plot the memory usage versus the index of occurence in the console output (not time !!!) for the three cases
mem_eigenvaluepara.pdf mem_kpara.pdf mem_nocoeigenvaluepara.pdf
You can clearly make out the Diagonalization and cdngen parts of the timers as the periodic oscillations in the usage (I think they have roughly the same number of timerstarts/stops)
As you can see in mem_eigenvaluepara.pdf the memory usage increases dramatically over multiple iterations. It more than doubles in ~10 iterations. It can be split up into two contributions, which I confirmed with the pure k-parallelization run and the first variation test with ev parallelization
- With ev parallelization some tasks use drastically more memory and the memory usage increases in the diagonalization part
- With second variation SOC there is an increase in memory usage in the cdngen part (you can clearly see that this is not present in
mem_nocoeigenvaluepara.pdf)
Note: I had to reduce the parameters a bit for the noco run. I can also perform a noco run with pure k-parallelization. This should not have any of these problems in the issue
Input and output file:
Compute environment
iffslurm
I'm not sure where this comes from, but in my opinion the memory usage should not increase this much between iterations