From 057baa451982b33ac1a668173fefa6f965402e36 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Sat, 19 Aug 2023 16:02:45 -0700 Subject: [PATCH 01/12] Added compute doc, adding other operations docs (e.g. backups) --- docs/compute.rst | 193 ++++++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 1 + docs/operations.rst | 4 +- 3 files changed, 197 insertions(+), 1 deletion(-) create mode 100644 docs/compute.rst diff --git a/docs/compute.rst b/docs/compute.rst new file mode 100644 index 00000000..59dae2f2 --- /dev/null +++ b/docs/compute.rst @@ -0,0 +1,193 @@ +.. _compute: + +####### +Compute +####### + +In order to actually execute ``Transformation``\s to obtain free energy estimates, you must deploy compute services to resources suitable for executing these types of calculations. +This document details how to do this on several different types of compute resources. + +There currently exists a single implementation of an ``alchemiscale`` compute service: the py:class:`~alchemiscale.compute.service.SynchronousComputeService`. +Other variants will likely be created in the future, optimized for different use cases. +This documentation will expand over time as these variants become available; for now, it assumes use of this variant. + +In all cases, you will need to define a configuration file for your compute services to consume on startup. +A template for this file can be found here; replace ``$ALCHEMISCALE_VERSION`` with the version tag, e.g. ``v0.1.4``, you have deployed for your server:: + + https://raw.githubusercontent.com/openforcefield/alchemiscale/$ALCHEMISCALE_VERSION/devtools/configs/synchronous-compute-settings.yaml + + +*********** +Single-Host +*********** + +To deploy a compute service (or multiple services) to a single host, we recommend one of two routes. +* installing all dependencies in a ``conda``/``mamba`` environment +* running the services as Docker containers, with all dependencies baked in + + +.. _compute_conda: + +Deploying with conda/mamba +========================== + +To deploy via ``conda``/``mamba``, first create an environment (we recommend ``mamba`` for its performance):: + + mamba env create -n alchemiscale-compute-$ALCHEMISCALE_VERSION -f https://raw.githubusercontent.com/openforcefield/alchemiscale/$ALCHEMISCALE_VERSION/devtools/conda-envs/alchemiscale-compute.yml + +Once created, activate the environment in your current shell:: + + conda activate alchemiscale-compute-$ALCHEMISCALE_VERSION + +Then start a compute service, assuming your configuration file is in the current working directory, with:: + + alchemiscale compute synchronous -c synchronous-compute-settings.yaml + + +.. _compute_docker: + +Deploying with Docker +===================== + +Assuming your configuration file is in the current working directory, to deploy with Docker, you might use:: + + docker run --gpus all --rm -v $(pwd):/mnt ghcr.io/openforcefield/alchemiscale-compute:$ALCHEMISCALE_VERSION compute synchronous -c /mnt/synchronous-compute-settings.yaml + + +See the `official Docker documentation on GPU use`_ for details on how to specify individual GPUs for each container you launch. +It may also make sense to apply constraints to the number of CPUs available to each container to avoid oversubscription. + + +*********** +HPC Cluster +*********** + +To deploy compute services to an HPC cluster, we recommend submitting them as individual jobs to the HPC cluster's scheduler. +Different clusters feature different schedulers (e.g. SLURM, LSF, TORQUE/PBS, etc.), and vary widely in their hardware and queue configurations. +You will need to tailor your specific approach to the constraints of the cluster you are targeting. + +The following is an example of the *content* of a script submitted to an HPC cluster. +We have left off the top matter that is specific to the queueing system, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) should be tailored to those presented by the queueing system. +Note that for this case we've made use of a ``conda``/``mamba``-based deployment, detailed above in :ref:`deployment_conda`:: + + # don't limit stack size + ulimit -s unlimited + + # make scratch space + mkdir -p /scratch/${USER}/${JOBID}-${JOBINDEX} + + # activate environment + conda activate alchemiscale-compute-$ALCHEMISCALE_VERSION + + # create a YAML file with specific substitutions + # each service in this job can share the same config + envsubst < settings.yaml > configs/settings.${JOBID}-${JOBINDEX}.yaml + + # start up a single service + alchemiscale compute synchronous -c configs/settings.${LSB_JOBID}-${LSB_JOBINDEX}.yaml + + # remove scratch space + rm -r /scratch/${USER}/${JOBID}-${JOBINDEX} + + +The ``envsubst`` line in particular will make a config specific to this job, with environment variable substitutions. +A subset of options used in the config file are given below:: + + --- + # options for service initialization + init: + + # Filesystem path to use for `ProtocolDAG` `shared` space. + shared_basedir: "/scratch/${USER}/${LSB_JOBID}-${LSB_JOBINDEX}/shared" + + # Filesystem path to use for `ProtocolUnit` `scratch` space. + scratch_basedir: "/scratch/${USER}/${LSB_JOBID}-${LSB_JOBINDEX}/scratch" + + # Path to file for logging output; if not set, logging will only go to + # STDOUT. + logfile: /home/${USER}/logs/service.${JOBID}.log + + # options for service execution + start: + + # Max number of Tasks to execute before exiting. If `null`, the service will + # have no task limit. + max_tasks: 1 + + # Max number of seconds to run before exiting. If `null`, the service will + # have no time limit. + max_time: 300 + + +For HPC job-based execution, we recommend limiting the number of ``Task``\s the compute service executes to a small number, preferrably 1, and setting a time limit beyond which the compute service will shut down. +With this configuration, when a compute service comes up and claims a ``Task``, it will have nearly the full walltime of its job to execute it. +Any compute service that fails to claim a ``Task`` will shut itself down, and the job will exit, avoiding waste and a scenario where a ``Task`` is claimed without enough walltime left on the job to complete it. + + +****************** +Kubernetes Cluster +****************** + +To deploy compute services to a Kubernetes ("k8s") cluster, we make use of a similar approach to deployment with Docker detailed above in :ref:`deployment_docker`. +We define a k8s `Deployment`_ featuring a single container spec as the file ``compute-services.yaml``:: + + apiVersion: apps/v1 + kind: Deployment + metadata: + name: alchemiscale-synchronouscompute + labels: + app: alchemiscale-synchronouscompute + spec: + replicas: 1 + selector: + matchLabels: + app: alchemiscale-synchronouscompute + template: + metadata: + labels: + app: alchemiscale-synchronouscompute + spec: + containers: + - name: alchemiscale-synchronous-container + image: ghcr.io/openforcefield/alchemiscale-compute:$ALCHEMISCALE_VERSION + args: ["compute", "synchronous", "-c", "/mnt/settings/synchronous-compute-settings.yaml"] + resources: + limits: + cpu: 2 + memory: 12Gi + ephemeral-storage: 48Gi + nvidia.com/gpu: 1 + requests: + cpu: 2 + memory: 12Gi + ephemeral-storage: 48Gi + volumeMounts: + - name: alchemiscale-compute-settings-yaml + mountPath: "/mnt/settings" + readOnly: true + env: + - name: OPENMM_CPU_THREADS + value: "2" + volumes: + - name: alchemiscale-compute-settings-yaml + secret: + secretName: alchemiscale-compute-settings-yaml + + +This assumes our configuration file has been defined as a *secret* in the cluster. +Assuming the file is in the current working directory, we can add it as a secret with:: + + kubectl create secret generic alchemiscale-compute-settings-yaml --from-file=synchronous-compute-settings.yaml + + +The we can then deploy the compute services with:: + + kubectl apply -f compute-services.yaml + +To scale up the number of compute services, increase the number of ``replicas`` to the number desired, and re-run the ``kubectl apply`` command above. + +A more complete example of this type of deployment can be found in `alchemiscale-k8s`_. + + +.. _Deployment: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ +.. _alchemiscale-k8s: https://github.com/datryllic/alchemiscale-k8s/tree/main/compute diff --git a/docs/index.rst b/docs/index.rst index c2ffb934..8364b3cd 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -30,6 +30,7 @@ in particular the `OpenForceField`_ and `OpenFreeEnergy`_ ecosystems. ./overview ./user_guide ./deployment + ./compute ./operations ./API_docs diff --git a/docs/operations.rst b/docs/operations.rst index 825197fd..8e34fd07 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -9,7 +9,7 @@ Add Users To add a new user identity, you will generally use the ``alchemiscale`` CLI:: - $ export NEO4J_URL=bolt://7687 + $ export NEO4J_URL=bolt://:7687 $ export NEO4J_USER= $ export NEO4J_PASS= $ @@ -51,3 +51,5 @@ The important bits here are: Backups ******* +Performing regular backups of the state store is an important component for any production deployment of ``alchemiscale``. +To From 5d949e1fd20a6f667c1f4b237304c59ebce03bf0 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Sat, 19 Aug 2023 16:10:19 -0700 Subject: [PATCH 02/12] Added missing link --- docs/compute.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/compute.rst b/docs/compute.rst index 59dae2f2..e082a8b3 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -58,6 +58,8 @@ See the `official Docker documentation on GPU use`_ for details on how to specif It may also make sense to apply constraints to the number of CPUs available to each container to avoid oversubscription. +.. _official Docker documentation on GPU use: https://docs.docker.com/config/containers/resource_constraints/#gpu + *********** HPC Cluster *********** From fb119547bb2440d84a30bd289565160498de5bc7 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Sat, 19 Aug 2023 16:17:23 -0700 Subject: [PATCH 03/12] Fixed wrong refs --- docs/compute.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index e082a8b3..d811b8db 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -70,7 +70,7 @@ You will need to tailor your specific approach to the constraints of the cluster The following is an example of the *content* of a script submitted to an HPC cluster. We have left off the top matter that is specific to the queueing system, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) should be tailored to those presented by the queueing system. -Note that for this case we've made use of a ``conda``/``mamba``-based deployment, detailed above in :ref:`deployment_conda`:: +Note that for this case we've made use of a ``conda``/``mamba``-based deployment, detailed above in :ref:`compute_conda`:: # don't limit stack size ulimit -s unlimited @@ -130,7 +130,7 @@ Any compute service that fails to claim a ``Task`` will shut itself down, and th Kubernetes Cluster ****************** -To deploy compute services to a Kubernetes ("k8s") cluster, we make use of a similar approach to deployment with Docker detailed above in :ref:`deployment_docker`. +To deploy compute services to a Kubernetes ("k8s") cluster, we make use of a similar approach to deployment with Docker detailed above in :ref:`compute_docker`. We define a k8s `Deployment`_ featuring a single container spec as the file ``compute-services.yaml``:: apiVersion: apps/v1 From eeeafa9e75882b0995eb7345a0504b5f30f7aa18 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Tue, 22 Aug 2023 22:43:23 -0700 Subject: [PATCH 04/12] Added backup docs, reference to kubernetes deployment --- docs/deployment.rst | 21 ++++++++++++++++----- docs/operations.rst | 28 ++++++++++++++++++++++++++-- 2 files changed, 42 insertions(+), 7 deletions(-) diff --git a/docs/deployment.rst b/docs/deployment.rst index fe6c162a..9a7d5246 100644 --- a/docs/deployment.rst +++ b/docs/deployment.rst @@ -14,7 +14,7 @@ Only Linux is supported as a platform for deploying ``alchemiscale`` services; W .. _deploy-docker-compose: ****************************************** -Single-Host Deployment with docker-compose +Single-host deployment with docker-compose ****************************************** An alchemiscale "server" deployment consists of a ``neo4j`` database (the "state store"), a client API endpoint, a compute API endpoint, and a reverse proxy (``traefik``). @@ -27,7 +27,7 @@ The "server" also requires an object store; see :ref:`deploy-object-store`. .. _deploy-docker-compose-instructions: -Deployment Instructions +Deployment instructions ======================= Install `docker compose `_. @@ -97,11 +97,22 @@ Once connected to the instance, run the following commands:: Now the instance has all of the dependencies required for ``docker-compose``-based deployment (:ref:`deploy-docker-compose-instructions`) +.. _deploy-kubernetes: + +************************************************* +Kubernetes-based deployment with alchemiscale-k8s +************************************************* + +To deploy ``alchemiscale`` to a Kubernetes cluster, review the resources defined and detailed in `alchemiscale-k8s`_. + +.. _alchemiscale-k8s: https://github.com/datryllic/alchemiscale-k8s/tree/main/compute + + .. _deploy-object-store: -************ -Object Store -************ +************************** +Setting up an object store +************************** An "object store" is also needed for a complete server deployment. Currently, the only supported object store is AWS S3. diff --git a/docs/operations.rst b/docs/operations.rst index 8e34fd07..7c0e2ca9 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -51,5 +51,29 @@ The important bits here are: Backups ******* -Performing regular backups of the state store is an important component for any production deployment of ``alchemiscale``. -To +Performing regular backups of the state store is an important operational component for any production deployment of ``alchemiscale``. +To do this, **first shut down the ``neo4j`` service so that no database processes are currently running**. + +The instructions below assume a Docker-based deployment, perhaps via ``docker-compose`` as in :ref:`deploy-docker-compose`. +The same general principles apply to any deployment type, however. + +Creating a database dump +======================== + +**With the ``neo4j`` service shut down**, navigate to the directory containing your database data, set ``$BACKUPS_DIR`` to the absolute path of your choice, then run:: + + docker run --rm -v $(pwd):/var/lib/neo4j/data -v ${BACKUPS_DIR}:/tmp --entrypoint /bin/bash neo4j:4.4 neo4j-admin dump --to /tmp/neo4j-$(date -I).dump + +This will create a new database dump in the ``$BACKUPS_DIR`` directory. + + +Restoring from a database dump +============================== + +To later restore from a database dump, navigate to the directory containing your current database data, and clear or move the current data to another location (Neo4J will not restore to a database that is already initialized). + +**With the ``neo4j`` service shut down**, choose ``$DUMP_DATE``, then run:: + + docker run --rm -v $(pwd):/var/lib/neo4j/data -v ${BACKUPS_DIR}:/tmp --entrypoint /bin/bash neo4j:4.4 neo4j-admin load --from /tmp/neo4j-${DUMP_DATE}.dump + +Automating the backup process to perform regular backups without human intervention for your deployment is ideal, but out of scope for this document. From e4051395d9d5bacc215d81aaba0b1eb030032fab Mon Sep 17 00:00:00 2001 From: David Dotson Date: Tue, 22 Aug 2023 22:51:41 -0700 Subject: [PATCH 05/12] Readability improvements, consistency --- docs/compute.rst | 6 +++--- docs/deployment.rst | 2 +- docs/operations.rst | 20 +++++++++++++++----- 3 files changed, 19 insertions(+), 9 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index d811b8db..274814ca 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -18,7 +18,7 @@ A template for this file can be found here; replace ``$ALCHEMISCALE_VERSION`` wi *********** -Single-Host +Single-host *********** To deploy a compute service (or multiple services) to a single host, we recommend one of two routes. @@ -61,7 +61,7 @@ It may also make sense to apply constraints to the number of CPUs available to e .. _official Docker documentation on GPU use: https://docs.docker.com/config/containers/resource_constraints/#gpu *********** -HPC Cluster +HPC cluster *********** To deploy compute services to an HPC cluster, we recommend submitting them as individual jobs to the HPC cluster's scheduler. @@ -127,7 +127,7 @@ Any compute service that fails to claim a ``Task`` will shut itself down, and th ****************** -Kubernetes Cluster +Kubernetes cluster ****************** To deploy compute services to a Kubernetes ("k8s") cluster, we make use of a similar approach to deployment with Docker detailed above in :ref:`compute_docker`. diff --git a/docs/deployment.rst b/docs/deployment.rst index 9a7d5246..c6efb46b 100644 --- a/docs/deployment.rst +++ b/docs/deployment.rst @@ -105,7 +105,7 @@ Kubernetes-based deployment with alchemiscale-k8s To deploy ``alchemiscale`` to a Kubernetes cluster, review the resources defined and detailed in `alchemiscale-k8s`_. -.. _alchemiscale-k8s: https://github.com/datryllic/alchemiscale-k8s/tree/main/compute +.. _alchemiscale-k8s: https://github.com/datryllic/alchemiscale-k8s .. _deploy-object-store: diff --git a/docs/operations.rst b/docs/operations.rst index 7c0e2ca9..60da3539 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -3,7 +3,7 @@ Operations ########## ********* -Add Users +Add users ********* To add a new user identity, you will generally use the ``alchemiscale`` CLI:: @@ -60,9 +60,14 @@ The same general principles apply to any deployment type, however. Creating a database dump ======================== -**With the ``neo4j`` service shut down**, navigate to the directory containing your database data, set ``$BACKUPS_DIR`` to the absolute path of your choice, then run:: +**With the neo4j service shut down**, navigate to the directory containing your database data, set ``$BACKUPS_DIR`` to the absolute path of your choice, then run:: - docker run --rm -v $(pwd):/var/lib/neo4j/data -v ${BACKUPS_DIR}:/tmp --entrypoint /bin/bash neo4j:4.4 neo4j-admin dump --to /tmp/neo4j-$(date -I).dump + docker run --rm \ + -v $(pwd):/var/lib/neo4j/data \ + -v ${BACKUPS_DIR}:/tmp \ + --entrypoint /bin/bash \ + neo4j:4.4 \ + neo4j-admin dump --to /tmp/neo4j-$(date -I).dump This will create a new database dump in the ``$BACKUPS_DIR`` directory. @@ -72,8 +77,13 @@ Restoring from a database dump To later restore from a database dump, navigate to the directory containing your current database data, and clear or move the current data to another location (Neo4J will not restore to a database that is already initialized). -**With the ``neo4j`` service shut down**, choose ``$DUMP_DATE``, then run:: +**With the neo4j service shut down**, choose ``$DUMP_DATE``, then run:: - docker run --rm -v $(pwd):/var/lib/neo4j/data -v ${BACKUPS_DIR}:/tmp --entrypoint /bin/bash neo4j:4.4 neo4j-admin load --from /tmp/neo4j-${DUMP_DATE}.dump + docker run --rm \ + -v $(pwd):/var/lib/neo4j/data \ + -v ${BACKUPS_DIR}:/tmp \ + --entrypoint /bin/bash \ + neo4j:4.4 \ + neo4j-admin load --from /tmp/neo4j-${DUMP_DATE}.dump Automating the backup process to perform regular backups without human intervention for your deployment is ideal, but out of scope for this document. From 6bc609a0b618c31e208d5071310500e25bb156ec Mon Sep 17 00:00:00 2001 From: "David L. Dotson" Date: Wed, 23 Aug 2023 12:07:01 -0700 Subject: [PATCH 06/12] Update docs/compute.rst Co-authored-by: Mike Henry <11765982+mikemhenry@users.noreply.github.com> --- docs/compute.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compute.rst b/docs/compute.rst index 274814ca..000b5878 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -69,7 +69,7 @@ Different clusters feature different schedulers (e.g. SLURM, LSF, TORQUE/PBS, et You will need to tailor your specific approach to the constraints of the cluster you are targeting. The following is an example of the *content* of a script submitted to an HPC cluster. -We have left off the top matter that is specific to the queueing system, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) should be tailored to those presented by the queueing system. +We have omitted queuing system specific options, flags, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) which should be tailored to those presented by the queuing system. Note that for this case we've made use of a ``conda``/``mamba``-based deployment, detailed above in :ref:`compute_conda`:: # don't limit stack size From 779037f64d74cc50e8161a572d7d0f56f532a1f0 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Wed, 23 Aug 2023 12:07:50 -0700 Subject: [PATCH 07/12] Small doc fix --- docs/compute.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compute.rst b/docs/compute.rst index 000b5878..7feb9555 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -69,7 +69,7 @@ Different clusters feature different schedulers (e.g. SLURM, LSF, TORQUE/PBS, et You will need to tailor your specific approach to the constraints of the cluster you are targeting. The following is an example of the *content* of a script submitted to an HPC cluster. -We have omitted queuing system specific options, flags, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) which should be tailored to those presented by the queuing system. +We have omitted queuing system-specific options and flags, and certain environment variables (e.g. ``JOBID``, ``JOBINDEX``) should be tailored to those presented by the queuing system. Note that for this case we've made use of a ``conda``/``mamba``-based deployment, detailed above in :ref:`compute_conda`:: # don't limit stack size From 8d8f70b222748c0fa5c523492ee334d374a63ae7 Mon Sep 17 00:00:00 2001 From: "David L. Dotson" Date: Wed, 23 Aug 2023 12:33:20 -0700 Subject: [PATCH 08/12] Apply suggestions from code review Co-authored-by: Mike Henry <11765982+mikemhenry@users.noreply.github.com> --- docs/compute.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index 7feb9555..cb2eb9e0 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -75,8 +75,9 @@ Note that for this case we've made use of a ``conda``/``mamba``-based deployment # don't limit stack size ulimit -s unlimited - # make scratch space - mkdir -p /scratch/${USER}/${JOBID}-${JOBINDEX} + # make scratch space (path will be HPC system dependent) + ALCHEM_SCRATCH=/scratch/${USER}/${JOBID}-${JOBINDEX} + mkdir -p $ALCHEM_SCRATCH # activate environment conda activate alchemiscale-compute-$ALCHEMISCALE_VERSION @@ -89,7 +90,7 @@ Note that for this case we've made use of a ``conda``/``mamba``-based deployment alchemiscale compute synchronous -c configs/settings.${LSB_JOBID}-${LSB_JOBINDEX}.yaml # remove scratch space - rm -r /scratch/${USER}/${JOBID}-${JOBINDEX} + rm -r $ALCHEM_SCRATCH The ``envsubst`` line in particular will make a config specific to this job, with environment variable substitutions. From ad58b0676709956e77bbbf4a5be12807d14301fb Mon Sep 17 00:00:00 2001 From: David Dotson Date: Wed, 23 Aug 2023 17:48:45 -0700 Subject: [PATCH 09/12] Added note on Neo4j version for backups --- docs/operations.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/operations.rst b/docs/operations.rst index 60da3539..aebda236 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -52,7 +52,7 @@ Backups ******* Performing regular backups of the state store is an important operational component for any production deployment of ``alchemiscale``. -To do this, **first shut down the ``neo4j`` service so that no database processes are currently running**. +To do this, **first shut down the Neo4j service so that no database processes are currently running**. The instructions below assume a Docker-based deployment, perhaps via ``docker-compose`` as in :ref:`deploy-docker-compose`. The same general principles apply to any deployment type, however. @@ -60,13 +60,13 @@ The same general principles apply to any deployment type, however. Creating a database dump ======================== -**With the neo4j service shut down**, navigate to the directory containing your database data, set ``$BACKUPS_DIR`` to the absolute path of your choice, then run:: +**With the Neo4j service shut down**, navigate to the directory containing your database data, set ``$BACKUPS_DIR`` to the absolute path of your choice and ``$NEO4J_VERSION`` to the version of Neo4j you are using, then run:: docker run --rm \ -v $(pwd):/var/lib/neo4j/data \ -v ${BACKUPS_DIR}:/tmp \ --entrypoint /bin/bash \ - neo4j:4.4 \ + neo4j:${NEO4J_VERSION} \ neo4j-admin dump --to /tmp/neo4j-$(date -I).dump This will create a new database dump in the ``$BACKUPS_DIR`` directory. @@ -75,15 +75,15 @@ This will create a new database dump in the ``$BACKUPS_DIR`` directory. Restoring from a database dump ============================== -To later restore from a database dump, navigate to the directory containing your current database data, and clear or move the current data to another location (Neo4J will not restore to a database that is already initialized). +To later restore from a database dump, navigate to the directory containing your current database data, and clear or move the current data to another location (Neo4j will not restore to a database that is already initialized). -**With the neo4j service shut down**, choose ``$DUMP_DATE``, then run:: +**With the Neo4j service shut down**, choose ``$DUMP_DATE`` and set ``$NEO4J_VERSION`` to the version of Neo4j you are using, then run:: docker run --rm \ -v $(pwd):/var/lib/neo4j/data \ -v ${BACKUPS_DIR}:/tmp \ --entrypoint /bin/bash \ - neo4j:4.4 \ + neo4j:${NEO4J_VERSION} \ neo4j-admin load --from /tmp/neo4j-${DUMP_DATE}.dump Automating the backup process to perform regular backups without human intervention for your deployment is ideal, but out of scope for this document. From 16af29d526a1455bafd6c7c86a9e7610dd2c9804 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Wed, 23 Aug 2023 17:50:04 -0700 Subject: [PATCH 10/12] Avoided abbreviation in script --- docs/compute.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index cb2eb9e0..e0cec7d4 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -76,8 +76,8 @@ Note that for this case we've made use of a ``conda``/``mamba``-based deployment ulimit -s unlimited # make scratch space (path will be HPC system dependent) - ALCHEM_SCRATCH=/scratch/${USER}/${JOBID}-${JOBINDEX} - mkdir -p $ALCHEM_SCRATCH + ALCHEMISCALE_SCRATCH=/scratch/${USER}/${JOBID}-${JOBINDEX} + mkdir -p $ALCHEMISCALE_SCRATCH # activate environment conda activate alchemiscale-compute-$ALCHEMISCALE_VERSION @@ -90,7 +90,7 @@ Note that for this case we've made use of a ``conda``/``mamba``-based deployment alchemiscale compute synchronous -c configs/settings.${LSB_JOBID}-${LSB_JOBINDEX}.yaml # remove scratch space - rm -r $ALCHEM_SCRATCH + rm -r $ALCHEMISCALE_SCRATCH The ``envsubst`` line in particular will make a config specific to this job, with environment variable substitutions. From 41ccbdf069f9fe2d840498a4645f958874189af9 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Wed, 23 Aug 2023 18:06:21 -0700 Subject: [PATCH 11/12] Small RST fixes, bash flow --- docs/compute.rst | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index e0cec7d4..23948860 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -7,7 +7,7 @@ Compute In order to actually execute ``Transformation``\s to obtain free energy estimates, you must deploy compute services to resources suitable for executing these types of calculations. This document details how to do this on several different types of compute resources. -There currently exists a single implementation of an ``alchemiscale`` compute service: the py:class:`~alchemiscale.compute.service.SynchronousComputeService`. +There currently exists a single implementation of an ``alchemiscale`` compute service: the :py:class:`~alchemiscale.compute.service.SynchronousComputeService`. Other variants will likely be created in the future, optimized for different use cases. This documentation will expand over time as these variants become available; for now, it assumes use of this variant. @@ -21,7 +21,8 @@ A template for this file can be found here; replace ``$ALCHEMISCALE_VERSION`` wi Single-host *********** -To deploy a compute service (or multiple services) to a single host, we recommend one of two routes. +To deploy a compute service (or multiple services) to a single host, we recommend one of two routes: + * installing all dependencies in a ``conda``/``mamba`` environment * running the services as Docker containers, with all dependencies baked in @@ -33,7 +34,8 @@ Deploying with conda/mamba To deploy via ``conda``/``mamba``, first create an environment (we recommend ``mamba`` for its performance):: - mamba env create -n alchemiscale-compute-$ALCHEMISCALE_VERSION -f https://raw.githubusercontent.com/openforcefield/alchemiscale/$ALCHEMISCALE_VERSION/devtools/conda-envs/alchemiscale-compute.yml + mamba env create -n alchemiscale-compute-$ALCHEMISCALE_VERSION \ + -f https://raw.githubusercontent.com/openforcefield/alchemiscale/$ALCHEMISCALE_VERSION/devtools/conda-envs/alchemiscale-compute.yml Once created, activate the environment in your current shell:: @@ -51,7 +53,10 @@ Deploying with Docker Assuming your configuration file is in the current working directory, to deploy with Docker, you might use:: - docker run --gpus all --rm -v $(pwd):/mnt ghcr.io/openforcefield/alchemiscale-compute:$ALCHEMISCALE_VERSION compute synchronous -c /mnt/synchronous-compute-settings.yaml + docker run --gpus all \ + --rm \ + -v $(pwd):/mnt ghcr.io/openforcefield/alchemiscale-compute:$ALCHEMISCALE_VERSION \ + compute synchronous -c /mnt/synchronous-compute-settings.yaml See the `official Docker documentation on GPU use`_ for details on how to specify individual GPUs for each container you launch. From b0147150c81a907eaa2a234112747817d319b368 Mon Sep 17 00:00:00 2001 From: David Dotson Date: Wed, 23 Aug 2023 18:16:50 -0700 Subject: [PATCH 12/12] More readability fixes --- docs/compute.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/compute.rst b/docs/compute.rst index 23948860..c5825f9b 100644 --- a/docs/compute.rst +++ b/docs/compute.rst @@ -92,7 +92,7 @@ Note that for this case we've made use of a ``conda``/``mamba``-based deployment envsubst < settings.yaml > configs/settings.${JOBID}-${JOBINDEX}.yaml # start up a single service - alchemiscale compute synchronous -c configs/settings.${LSB_JOBID}-${LSB_JOBINDEX}.yaml + alchemiscale compute synchronous -c configs/settings.${JOBID}-${JOBINDEX}.yaml # remove scratch space rm -r $ALCHEMISCALE_SCRATCH @@ -106,10 +106,10 @@ A subset of options used in the config file are given below:: init: # Filesystem path to use for `ProtocolDAG` `shared` space. - shared_basedir: "/scratch/${USER}/${LSB_JOBID}-${LSB_JOBINDEX}/shared" + shared_basedir: "/scratch/${USER}/${JOBID}-${JOBINDEX}/shared" # Filesystem path to use for `ProtocolUnit` `scratch` space. - scratch_basedir: "/scratch/${USER}/${LSB_JOBID}-${LSB_JOBINDEX}/scratch" + scratch_basedir: "/scratch/${USER}/${JOBID}-${JOBINDEX}/scratch" # Path to file for logging output; if not set, logging will only go to # STDOUT. @@ -185,14 +185,15 @@ We define a k8s `Deployment`_ featuring a single container spec as the file ``co This assumes our configuration file has been defined as a *secret* in the cluster. Assuming the file is in the current working directory, we can add it as a secret with:: - kubectl create secret generic alchemiscale-compute-settings-yaml --from-file=synchronous-compute-settings.yaml + kubectl create secret generic alchemiscale-compute-settings-yaml \ + --from-file=synchronous-compute-settings.yaml -The we can then deploy the compute services with:: +Then we can then deploy the compute services with:: kubectl apply -f compute-services.yaml -To scale up the number of compute services, increase the number of ``replicas`` to the number desired, and re-run the ``kubectl apply`` command above. +To scale up the number of compute services on the cluster, increase ``replicas`` to the number desired, and re-run the ``kubectl apply`` command above. A more complete example of this type of deployment can be found in `alchemiscale-k8s`_.