Transition CCPP framework from ccpp-prebuild to ccpp-capgen v1 - #695
Conversation
…dim-names to capgen call
Changes to get the SCM tests to run: * clm_lake was included in a SDF, but not used. In the Cap, there is a (new?) check to see if the active conditions are met before calling the scheme. This was causing a runtime error. CLM Lake returns if not turned on, so having this new runtime check beforehand triggered the failure before entering the scheme. I removed the unused CLM Lake from the SDF and it worked fine. * Remove GFS_v16_debug SDFs from test list. (I see these were removed from the CCPP SCM?) * Add snippet to ccpp/CMakeLists to build only requested SDFs (i.e., -DCCPP_SUITES). * Uncomment and test 32-bit and Intel oneAPI in CMakeLists.
…; fix metadata-fortran inconsistencies, fix GitHub actions take 1
|
@climbfuji Looks like most of the tests are B4B, the only exception are the GNU release (optimized) tests. Could this be from compilation flag changes (e.g., Adding -fPIC)? The differences appear to be rounding errors for the GFS based tests, whereas the HRRR_gf and WoFs tests show non-negligible differences. For example: Full set of difference plots are stored as artifacts here |
Thanks for doing this comparison. The NEPTUNE runs are also not b4b identical. I think this is predominantly because the caps look different and the compiler optimizes the code differently. But for the two suites you mentioned, what do they have in common that the other suites don't have? I am certain that the much stricter validator and code generator helped identify and fix a few actual bugs. |
|
@dustinswales @climbfuji GNU release RTs for the SCM have been changing results basically every PR. It seems like something is just not reproducible or otherwise fishy with the environment. I wouldn't base any decisions on whether GNU release tests are b4b or not. |
@grantfirl Thanks. I thought it was an odd result. In any event... I've seen enough to warrant trying capgen-ng in the UFS. Any objections @climbfuji? |
|
No objections at all, please go ahead. I can also help out if you share a working branch and/or a checkout on Ursa, for example. I expect we'll have to fiddle a bit with the physics called from the dycore ... |
|
@dustinswales @grantfirl Looks like we'll have a clean pass through all CI tests in a few minutes. |
Ok, good to know! The coding agent I've been using had started using them more, so I guessed incorrectly. I'll change the PR back to the original macros, I can see the logic in using them for variables that aren't used |
* Removing Intel -qoverride-limits flag and setting release build to use -O2 optimization, instead of the default -O3, decrease the -j4 compile time from 94 minutes to 2 * ran dos2unix on this file to remove ^M end-of-line characters * Suite info cleanup, suite has been removed
scrasmussen
left a comment
scrasmussen
left a comment
There was a problem hiding this comment.
Everything building and looking good to me!
|
@scrasmussen @hertneky Have you all tested this PR anywhere besides the GitHub CI servers? I'm having trouble running locally on my Mac. I keep running into MPI broken pipe errors when trying to use the run script. I don't know if it has to do with my environment or something else. I'm still able to run the top of main just fine, though. |
@grantfirl I've been able to run bomex and twpice cases on Derecho. I also tested with the But that was on Derecho, I'm starting to test it locally on my Linux box right now and can also test on a Mac. How are you setting up the environment to build? Using a Dockerfile, conda env, something else? |
On my local Mac, I've installed spack-stack using the instructions that they provided, although I've been stuck at 1.9.1 or something. I tried to update to 2+ in the past 6 months but ran into issues. I should probably try to update again and retry. Now that Ursa is back, I'll try to run there too. I'd just really like to be able to run locally since, historically, I've enjoyed doing development on my Mac with the SCM. It shouldn't hold up this PR, though. If we can test successfully on Derecho and Ursa, that is good enough, and I can troubleshoot my Mac after this is merged. |
@grantfirl I tried to build the spack-stack on my mac and it wasn't going as smoothly as I liked so I just quickly updated the Dockerfiles we already have so it's at least easy to build and run the SCM locally. I submitted those changes in #713. Not a long term spack solution and probably doesn't match your local workflow (the image doesn't have dev IDEs, etc) but thought I'd at least throw it out there |
|
@scrasmussen @dustinswales I'm able to compile and run now on Ursa. It turns out that just checking out this branch in my existing clone on Ursa was a bad idea. I failed to remove the contents of the previous run directory and there were some weird linking problems created when using the updated code and run script. After removing everything in the run directory and trying to re-run, I was successful running on Ursa, which is good enough to merge, I think. I did notice that one helpful function is now no longer part of CMakeLists.txt, but I created an issue to address this in a follow-up. See #714. |
I built/ran RTs successfully with GNU on Ursa, but for intel, while it built fine, I get a run error @grantfirl @scrasmussen Did either of you test with the ursa_intel build? I thought I would test both compilers since it's a big change to the build structure. Derecho/gnu is also good. I didn't test derecho/intel yet as the build is in the queue. |
@hertneky I built and succesfully with intel on Ursa. BTW, @scrasmussen @hertneky @climbfuji @dustinswales , I'm creating a new physics PR with the physics changes coming directly from ufs/dev. It should be a superset of the existing ccpp-physics PR into scm/dev and it should supercede it, since it started from the same place but included more changes from @dustinswales . |
@grantfirl Great - For intel, I will trust your test then and chalk it up to something wrong on my end. |
hertneky
left a comment
hertneky
left a comment
There was a problem hiding this comment.
We should update the UG for the cmake changes. In scm/doc/TechGuide/chap_quick.rst there are 3 instances of cmake ../src that need to be changed to cmake ../..
Ursa and Derecho SCM RTs ran successfully for me on GNU.
@grantfirl I refreshed the build a 2nd time and then ursa-intel ran fine. Not sure what it was. |
I think that these should now be fixed. |
Derecho intel has now also run scm rts successfully. |
|
All, I reran RTs on Ursa using both compilers after updating the PR branches from main and ufs/dev. There were some runtime errors, but I don't think that they're related to this PR, exactly. They may also be one-off type errors that could be fixed with requesting more memory. For all tests, I ran cmake/make on the login node and ran the RTs (using the full case length, not 10% as is done during CI RTs) on compute nodes. All RRTMGP tests ran into insufficient memory errors in release mode, both for ifx and gfortran. The TEMPO tests ran into a segmentation fault later in the run. This one is weird since the same tests run on CI, albeit only to 10% of the runtime. I think that all RT failures can be investigated after this PR is merged. It has been tested to the hilt in this model, UFS, and NEPTUNE, so I don't have any doubts as to the efficacy of the capgen-ng caps. I think we shouldn't delay this any longer to investigate minor RT issues on Ursa. |
|
I agree, and I am thankful for all your efforts. With so many moving parts and different branches, it is hard to stay on top! Please let me know if and if yes, how, I should update the PR description (or feel free to do it yourself). Thanks. |
I wouldn't worry about the PR description. There is plenty of documentation of everything that went on in the comments. Phenomenal work to get the capgen rewrite done and working across so many hosts! |
5 participants
Description
This PR updates the CCPP framework code generator from ccpp-prebuild to ccpp-capgen v1. The large update of the ccpp-framework code also requires changes in ccpp-physics and ccpp-scm.
The most important changes in CCPP-SCM are:
scm/binviacmake ../...type = controlmetadata tables.ccpp_*in the SCM host model code).Dependencies
Testing