Questions/guidance testing slurm support

[MEllis-github]

I'm looking forward to taking the slurm support for a test drive!

Questions

• With respect to requirements, does the current support via the REST API and/or the future CLI support assume a container manager (docker / podman / ...) is installed?
• What would you recommend for an initial smoke test?
• @DC-Base are you the POC for these questions, are there others?

[michaelzhiluo]

The Slurm PR is in #4. Currently, Slurm PR supports Rest API; however, the CLI version is being implemented. Both ways to poll the Slurm Cluster assume that there is an exsiting container manager - this compatibility layer currently supports Docker, ContainerD, Singularity, and Podman. (correct me if wrong, @DC-Base )

We haven't tested custom installation for Slurm on any machine, but Daron has an easy Slurm setup script: https://github.com/dc-base/slurm-auto-script

[michaelzhiluo]

Also @AlexCuadron will begin testing Slurm after the skyctl exec PR.

[daronjchang]

Current compatibility layer supports Docker, ContainerD, and Singularity. Podman and PodmanHPC are planned to be added before the PR is finalized.

Current testing is done on locally using REST.
CLI is still in progress and being tested on the Berkeley Watson cluster with no container manager, and locally on a smaller scale with the container managers.

[daronjchang]

The easiest way to do a smoke test is using the REST API on a local machine. Most package managers already compile slurm with the REST enabled, and an additional package with Slurmrestd just needs to be installed.

For setting up Slurmrestd, I recommend a local single node configuration listening on a unix socket, then trying to curl the endpoint located at 'http://localhost:8080/slurm/{open_api_ver}/ping'
Using:
curl --unix-socket {socket_path} 'http://localhost:8080/slurm/{open_api_ver}/ping'

The simplest smoke test is pulling the dockerhub hello-world image and checking the job output log.

[MEllis-github]

Good info., thanks all!

I have (user level) access to several slurm clusters, none of which have the REST plugin, but one at least has a container manager. I can stand-up a VM or a local configuration, but assumed that wouldn't be helpful, perhaps prematurely.

[daronjchang]

The local configuration can be used to test out the REST version. I expect to push out CLI version by the end of the week.

…

On Tue, Mar 19, 2024 at 14:41 MEllis-github ***@***.***> wrote: Good info., thanks all! I have (user level) access to several slurm clusters, none of which have the REST plugin, but one at least has a container manager. I can stand-up a VM or a local configuration, but assumed that wouldn't be helpful, perhaps prematurely. — Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APU2ZTN46KBRRVKR2T5WPMDYZCIFRAVCNFSM6AAAAABE4ICEVOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNBVGE4TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dmatch01]

@MEllis-github I suggest to wait for the CLI since it will take some time to get the right folks to install the REST plugin and other associated requirements.

[MEllis-github]

@DC-Base any updates/recommendations on when to start testing the slurm CLI ? Thanks in advance!

[MEllis-github]

Thanks for the discussion on status and local smoke testing above!

Next questions are

• Is documentation for slurm support (e.g. quickstart, examples, etc.) being created and if so where? I've been looking in the slurm docs and main branches so far, in README.md, docs/, and skyflow/cli respectively.
• What is the first step to discover/attach a slurm cluster assuming it is remote and authenticated? The slurm cluster I am trying to attach is not listed in my skyctl get clusters and skyctl create... --manager slurm... otherwise looks like it might trigger creation (versus attachment only).

$ skyctl create cluster --manager slurm --help
Usage: skyctl create cluster [OPTIONS] NAME

  Attaches a new cluster.

Options:
  --manager TEXT       Cluster manager type (e.g. k8, slurm).  [default: k8;
                       required]
  --cpus TEXT          Number of vCPUs per node (e.g. 1, 1+).
  --memory TEXT        Amount of memory each instance must have in GB. (e.g.
                       32, 32+).
  --disk_size INTEGER  OS disk size in GBs
  --accelerators TEXT  Type and number of GPU accelerators to use
  --ports TEXT         Ports to open on the cluster
  --num_nodes INTEGER  Number of SkyPilot nodes to allocate to the cluster
                       [default: 1]
  --cloud TEXT
  --region TEXT
  --attached           True if cluster is already created and needs to be
                       attached to Skyflow.
  --help               Show this message and exit.

[daronjchang]

I pushed an initial quick start to /examples/slurm-configuration/slurm_README.md

I will merge Michael's doc branch into mine and move it over soon.

[MEllis-github]

Awesome, thanks, I will check it out!

[MEllis-github]

@DC-Base I created an issue here to double check some assumptions and debug an issue getting started: #113. Depending on the info., we can change it from a question-issue to a documentation-issue or whatever next step is appropriate. I can help move answers to documentation as appropriate.

CC: @dmatch01