allow staging executable output - #5371
Conversation
spxiwh
left a comment
spxiwh
left a comment
There was a problem hiding this comment.
Thanks Alex. This makes sense, just one change request to allow this to be set at the top-level pegasus-profile section (so be set for all jobs) and a question about whether stage_out and directory are redundant in the File class.
| directory=self.executable.out_dir, tags=all_tags, | ||
| use_tmp_subdirs=use_tmp_subdirs) | ||
| use_tmp_subdirs=use_tmp_subdirs, | ||
| stage_out=self.executable.stage_out) |
There was a problem hiding this comment.
There are some cases that don't use new_output_file_opt, but this would catch most examples.
There was a problem hiding this comment.
... It does seem weird to have File take both directory and stage_out though.
Do we actually need two options here? Can we not just use stage_out for all cases and remove directory. Then pycbc|stage-out becomes pycbc|override-stage-out (or similar) and can take local paths or any sort of transfer protocol that pegasus might understand.
There was a problem hiding this comment.
@spxiwh That's a good idea. It does seem duplicated now when it really didn't need to be.
| self.stage_out = None | ||
| if cp.has_option_tags('pegasus_profile-%s' % name, 'pycbc|stage-out', tags): | ||
| self.stage_out = cp.get_opt_tags('pegasus_profile-%s' % name, | ||
| 'pycbc|stage-out', tags) |
There was a problem hiding this comment.
I would prefer that this be changed to:
if cp.has_option_tags('pegasus_profile', 'pycbc|stage-out', tags + [name]):
to allow one to also set this at the top-level (for all jobs) if desired.
However, I think I never made cp.has_option_tags work for cases like [pegasus_profile-tag1-tag2] which might be needed here. .... Actually looking at the code, I think I did do that at some point, but the documentation of has_option_tags didn't get updated.
| if stage_out: | ||
| self.storage_path = file_url | ||
| else: | ||
| self.storage_path = urllib.parse.urlsplit(file_url).path |
There was a problem hiding this comment.
Can we always use the file_url as the storage path and not need the conditional here?
There was a problem hiding this comment.
Yes, I think you are right. The bottom path is what we had before, and that causes problems if the protocol isn't understoo (it strips out the protocol leaving the bare file name), however, I think just the file_url should also work on the local file system, so we probably never needed this line to begin with.
2 participants
This add a new option you can set in the pegasus profile for a job type that tells it to stage its output to some other location rather than to the standard local file layout. Note this just affects the final storage location, not scratch or working space.
Example config
[pegasus_profile-inspiral]
pycbc|stage-out = scp://gravity-dev2/home/ahnitz/transfer_test
This is useful if you have a remote archival result location and locally storage on a cluster limited.