Refactor Clean and Init tasks, move linking jobs to run in the background #303
Conversation
associated task. Remove init task for RTPP
associated main scripts
decrease the number of jobs to be submitted. Move ExternalAnalysis and LinkWarmBackground to run in the background
|
good job! @ibanos90 |
|
Now it works for scenarios/3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC.yaml. Many thanks @ibanos90 ! |
Thanks a lot for the thorough review @junmeiban! |
…ound (#303) This PR refactors some of the tasks. The goal is to reduce the number of jobs to be submitted, hence, computing and waiting time for the each cycle. `Clean` tasks are removed and the content is moved to their associated application tasks (e.g., `CleanHofX` content is moved to `HofX.csh`). `InitVariationals.csh` and `InitEnKF.csh` scripts for the YAML and model stage preparation stage are also removed and their content is moved to `PrepJEDI.csh` (only one preparation task is submitted). For RTPP, `Clean` and `Init` tasks and both removed and merged with `RTPP.csh`. Here, the `LinkExternalAnalysis` and `LinkWarmStartBackgrounds` tasks are set to run in the background instead of Casper. These tasks use very little memory and wall-time, then running them in the background should be fine (but let's monitor that). None - [ ] 3dvar_OIE120km_WarmStart - [ ] 3denvar_OIE120km_IAU_WarmStart - [x] 3dvar_OIE120km_ColdStart - [ ] 3dvar_O30kmIE60km_ColdStart - [x] 3denvar_O30kmIE60km_WarmStart - [x] eda_OIE120km_WarmStart - [x] getkf_OIE120km_WarmStart - [x] 4dhybrid_OIE120km_WarmStart - [ ] ForecastFromGFSAnalysesMPT - [x] 3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC
| # ================================ | ||
| if ("$retainObsFeedback" != True) then | ||
| set member = 1 | ||
| while ( $member <= ${nMembers} ) |
There was a problem hiding this comment.
I think this looks incorrect. I'm seeing Variational[1-5] fail intermittently, while it is trying to write output files.
I see this fail most often when running test/testinput/eda_OIE120km_WarmStart.yaml
When this is running for any given member, it will remove the output for all the other members, so output files for other members can be deleted while they are being written.
Instead of looping through all of the members in a loop it should only remove files for it's own member number, ArgMember
There was a problem hiding this comment.
Hi @jim-p-w, thanks for reporting this issue. When I tested it for this PR it was working correctly, but I will take a look.
There was a problem hiding this comment.
Indeed, the task will fail for the eda task in the case that the jobs end at different times, the files should be removed for each member independently without the loop. Thanks again for reporting this!
Description
This PR refactors some of the tasks. The goal is to reduce the number of jobs to be submitted, hence, computing and waiting time for the each cycle.
Cleantasks are removed and the content is moved to their associated application tasks (e.g.,CleanHofXcontent is moved toHofX.csh).InitVariationals.cshandInitEnKF.cshscripts for the YAML and model stage preparation stage are also removed and their content is moved toPrepJEDI.csh(only one preparation task is submitted). For RTPP,CleanandInittasks and both removed and merged withRTPP.csh. Here, theLinkExternalAnalysisandLinkWarmStartBackgroundstasks are set to run in the background instead of Casper. These tasks use very little memory and wall-time, then running them in the background should be fine (but let's monitor that).Issue closed
None
Tests completed
Tier 1:
Tier 2: