This document explains how GenoRing is architected and how to contribute to and develop GenoRing modules.
- Directory structure of GenoRing
- Structure of a GenoRing module
- Implementing a GenoRing module
- GenoRing core API
- GenoRing Perl API documentation
- Tips and developer features
Directories followed by (W) should be writable while those followed by (R) could be write-protected.
-
.(W) (i.e. root directory): This directory contains the main "genoring.pl" script that will generate the "docker-compose.yml" and "modules.yml" files. "docker-compose.yml" is automatically generated and should not be manually modified as any modification could be lost when "genoring.pl" regenerates that file. It is used by "docker compose" to start GenoRing. "modules.yml" contains information about modules used by GenoRing and their state. It is only used by GenoRing and could be edited. -
.git(W) This directory is optional and contains Git (version control) files. Using a version-controlled directory may help to keep your eventual code alterations in case of GenoRing upgrades. -
env(W): Contains GenoRing module environment variable values. This directory is automatically generated by genoring.pl at (module) installation time. The contained environment files come from modules' "env" directories and can be altered but might be overridden by genoring.pl if a corresponding module is reinstalled or reconfigured. As environment files contain sensitive data (ie. passwords), genoring.pl restricts their access to their owner (0600, and 0700 for the directory) at each run if needed. The same applies to thesysbackupsdirectory and to the generateddocker-compose.ymlfile (which may contain environment values). This is not managed with a native Windows Perl (ACL) and a warning is issued if the file system does not support Unix permissions. -
sysbackups(W): Contains GenoRing framework backups performed by "genoring.pl backup". Each directory contains a backup. The directory name is the backup name. This directory is not a Docker volume and is not accessible from containers, which prevents them from altering GenoRing backups. It is kept by "genoring.pl reset". The (empty) directory is provided by the repository so its access rights can be adjusted right after cloning, as backups may contain sensitive data (ie. database dumps and settings). -
modules(R or W): Contains available GenoRing modules as well as a module template. Each module has its own directory. If a module has a "src" directory, the "src" sub-directories should be writable, especially when using ARM architectures as the "Dockerfile" may be automatically generated there by "genoring.pl". Make it writable if you want to be able to automatically download new modules. -
modules/TEMPLATE(R): Contains a GenoRing module template. -
modules/<some_module>(R): Contains a GenoRing module. See "Structure of a GenoRing module" for details. -
modules/<some_module>/src/<some_service>(W): Contains a module service sources to compile/generate the corresponding Docker container. -
volumes(W): Contains exposed Docker volumes used by GenoRing containers. Files can be modified but often require root privileges as they are generated by Docker. -
volumes/backups(W): Contains containers internal backups (ie. CMS or database backups made by containers' own tools such as "genoring backup" in the "genoring" container). It is mounted in containers as "/backups". GenoRing framework backups are not stored here but in thesysbackupsdirectory. -
volumes/data(W): Contains (biological) data for modules, in a structured manner. -
volumes/db(W): Contains database data. -
volumes/drupal(W): Contains Drupal system. -
volumes/drupal/config(W): Contains exported Drupal config (for config backups). -
volumes/drupal/php(W): Contains PHP configuration file "php.ini" which can be customized. Any modification made is synchronized when GenoRing is (re)started. -
volumes/drupal/private(W): Contains Drupal private files with restricted access. -
volumes/drupal/vendor(W): Contains libraries provided by composer. -
volumes/drupal/web(W): Contains main Drupal files. -
volumes/offline(W): Contains the "offline.html" file that is displayed when GenoRing is offline. It can be customized. -
volumes/proxy(W): Contains web server config. -
volumes/proxy/nginx(W): Contains nginx configuration file (genoring-fpm.conf) used by the default "genoring-proxy" service as well as module configurations to include at the "http" level (volumes/proxy/nginx/includes) or in the main "GenoRing" server definition (volumes/proxy/nginx/genoring). -
volumes/proxy/httpd(W): Contains Apache HTTPd configuration file (genoring-httpd.conf) used by the "genoring-proxy-httpd" service ("httpd" alternative of the "genoring" module) as well as module configurations to include at the server level (volumes/proxy/httpd/includes) or in the main "GenoRing" virtual host definition (volumes/proxy/httpd/genoring).
A GenoRing module template can be found in the GenoRing "modules/TEMPLATE" directory. A GenoRing module is a directory with the following structure:
-
the directory name must only contain lowercase alpha-numeric characters and underscores and must start with a letter.
-
"README.md": a README file explaining the purpose of the module and how it works.
-
"TOEXTERNAL.md": a file explaining how to turn Docker service containers into "external" services handled either by the server hosting GenoRing or other available servers.
-
"env": a directory containing environment files used by the module that will be used by GenoRing when the module is enabled, to ask the admin to set the environment variables to configure the module and its services. Each environment file can hold multiple environment variables, one by line, following the format "VARIABLE_NAME=variable value" preceded by comments documenting the variable usage. Some @tags should also be present to explain how the variable is used:
- "SET": means it is recommended to customize the variable.
- "OPT": means the variable can be customized or left as is.
- "INS": means the variable is used at installation.
- "RUN": means the variable is used at runtime.
Ex.:
# - Drupal admin account name # @tags: OPT INS DRUPAL_USER=genoringIt is important to document variables with SET or OPT so they can be managed by the GenoRing script at installation time: the first line of the comment block of a variable should contain the short variable description in one line. The next line should be an empty comment line or only contain dashes ('-'). The next comment lines should contain the complete description with explanations on how to fill the value. The comment block should then contain a "@default" annotation followed by a space and the default value (could be empty) and the "@tags" annotation stating the use of the variable. See "modules/genoring/env/genoring.env" for more examples. Note: enabled Docker Compose profiles can be provided to a container through an environment variable defined as
COMPOSE_PROFILES=${COMPOSE_PROFILES}. See "services" note for details on Docker Compose profiles. -
"services": a set of YAML files containing the Docker definition of each service provided by the module. See https://docs.docker.com/compose/compose-file/05-services/ for details. The definition should not include the "services:" element nor an element name for the service and should not include a "container_name" parameter as it will be automatically set by GenoRing using the YAML file name. The indentation must not include extra-spaces for readability as they will be automatically managed by the GenoRing script. To avoid conflicts between modules, service names should be prefixed by "genoring-" followed by their module name followed by a dash, unless a service could be shared amongst multiple modules. Note: 5 profiles are managed by GenoRing and can be used in the service definitions (i.e. "profiles: [...]") to limit the use of a service to a given context:
- "prod": set for production site.
- "staging": set for staging site.
- "dev": set for development site.
- "backend": only enabled for back-end operations like module installation, update and uninstallation.
- "offline": only enabled when the site is offline. Without specific profiles, the service is always loaded (if the module is enabled). Note: The "services" directory may contain a "alt" sub-directory with alternative services that can be used to replace the default ones. An "alt.yml" file contains alternative settings managed by GenoRing. Each alternative is defined by a (machine) name as YAML element key and contains a description string ("description" key), a list of substituted services (structured as current service name as key and new alternative service name as value under the "substitute" key), a list of new services to add ("add" key) and a list of services to remove ("remove" key). The added and removed service names are lists; every added or substituted service must have a definition in "services/alt/". Enabling or disabling an alternative updates the module's selected alternative in "config.yml" and regenerates "docker-compose.yml". When the module is enabled, standard service "disable_.sh" and "enable_.sh" hooks are run for services removed and added by the change. The operation creates a backup and restores the previous service configuration if a hook or configuration step fails. The owning module remains enabled; only its service set changes. Note: it is also possible to override or merge service definitions to other existing services. See TEMPLATE services documentation for details.
-
"volumes": a set of YAML files corresponding to named volumes shared across containers. These are not to be confused with volumes that can be defined in the services above. A module service can mount a shared volume in its "volumes" section but the point is that this shared volume can also be mounted by services in other modules for data sharing. See https://docs.docker.com/compose/compose-file/07-volumes/ for details. Since a same shared volume may be defined in multiple modules, the GenoRing script will ensure they use the same definition using the volume's version number provided as comment in the volume's YAML file header (i.e. "# v1.0"). The GenoRing script will only keep the latest sub-version definition (i.e. if there is a 1.0 and a 1.1 definition, it will use the 1.1 definition) or the last definition met and will raise an error in case of different major versions and abort module activation (i.e. 1.0 vs 2.0). Note: a 'genoring-' prefix is required to avoid collisions with other volumes managed by Docker.
-
"hooks": a directory holding hook scripts. Hook scripts may be available or not for a given action and can be Perl scripts for local system actions or shell scripts (or others) that should be run in module containers. "local hooks" are usually called when all containers are down (except for the "state.pl" hook) while "container hooks" are launched on running containers when the system is fully loaded. Those scripts use special names to be triggered on specific events.
- "backend_<service_name>.sh" will be triggered when GenoRing is started in "backend" mode.
- "backup.pl" will be called on the local system when a backup is created to let the module manage its backup. The first argument provided is the backup machine name.
- "backup_<service_name>.sh" will be called on the corresponding container when a backup is created to let the module manage/generate its backup. The first argument provided is the backup machine name.
- "disable.pl" will be called on the local system when the module is disabled.
- "disable_<service_name>.sh" will be called on the corresponding container to remove the modifications made by the "enable_<service_name>.sh" script.
- "enable.pl" will be called on the local system when the module is installed or enabled.
- "enable_<service_name>.sh" will be called on the corresponding container to alter other containers when needed (e.g. adding menu items for the module features, pre-configuring Drupal modules, etc.).
- "init.pl" will be called on the local system when the module is installed (and enabled).
- "offline_<service_name>.sh" will be triggered when GenoRing is started in "offline" mode.
- "online_<service_name>.sh" will be triggered when GenoRing is started normally (online mode).
- "requirements.pl" will be called on the local system to check a module requirements when a module is installed. If the "requirements.pl" returns a non-zero exit code, it means some requirements are not met. Message output will be displayed to help the user know what is missing.
- "restore.pl" will be called on the local system when a backup should be restored. The first argument provided is the backup machine name.
- "restore_<service_name>.sh" will be called on the corresponding container when a backup should be restored to let the module manage/generate its backup restoration. The first argument provided is the backup machine name.
- "start.pl" will be called on the local system when the GenoRing is started.
- "state.pl" will be called on the local system when the status of the module is needed. When this script is not provided, the system will rely on Docker container state. However, in some cases like for the "genoring" service, the container might be running but may not be fully initialized. Therefore, such a script is needed to get the real container state. The script will just output a line with the container state. The ready state string to use is 'running'. See the corresponding template hook script for more details.
- "stop.pl" will be called on the local system when the GenoRing is stopped.
- "uninstall.pl" will be called on the local system when the module is disabled and uninstalled. The module will be disabled first and "disable.pl" will be called by the system before "uninstall.pl".
- "update.pl" will be called on the local system when what is managed by the module (tools, data, etc.) needs to/should be updated.
- "update_<service_name>.sh" will be called on the corresponding container to perform updates on things managed by the module (tools, data, etc.).
- "upgrade.pl" will be called on the local system when the GenoRing module is updated to a newer version. The first parameter is the previous version of the module that needs to be upgraded to the currently installed newer version.
- "upgrade_<service_name>.sh" will be called on the corresponding container to perform upgrade of a GenoRing module. The first parameter is the previous version of the module that needs to be upgraded to the currently installed newer version. Note: When running shell scripts inside containers, the GenoRing "modules" directory is copied in the container as "/genoring/modules" and allows access to module's files if needed (like the module "res" directory for instance). Note: Hook scripts may be called more than one time after a site installation. It is up to the script to not perform several times a same operation if it has already been done.
-
"src": if the module uses custom containers, their sources will be provided there in sub-directories corresponding to service names (i.e. YAML file names without the ".yml" extensions). Service names should always be prefixed with "genoring-" to avoid conflicts with other non-GenoRing Docker containers and must correspond to a module service name as defined in the module "services" directory. The "src" directory must contain at least a "Dockerfile", a "Dockerfile.amd64" or a "Dockerfile.arm" file. "Dockerfile", if present, is assumed to be designed for amd64 architectures. A "Dockerfile.arm" may be provided but if not, it can be automatically generated by GenoRing at compilation time using the amd64 version. Therefore, it is recommended to only provide a "Dockerfile" and only provide others for specific needs that can't be covered automatically.
-
"overrides": a module may declare Docker image overrides in the optional "overrides" mapping of its module YAML file. Each key identifies a target service (prefer the "genoring-" form; the service name, or its current image name, is also accepted) and must have a matching "services/overrides/.dockerfile". Override Dockerfiles use the target service's image in their "FROM" instruction; GenoRing replaces it with the preceding image in the override chain. Files used by "COPY" must be in the "services/overrides" directory, which is the Docker build context. Overrides are applied only when both the module and the targeted service are active. An optional "alternatives" list limits an override to the selected alternative name; use the module name in that list to select the module's default services. When multiple modules override one service, GenoRing builds them in module-name order. It starts from the original service image (built from local sources when needed, or the public image), applies each override and tags the final image "genoring-override-:", where "" is a short hash of the service, its owner module, its base image and the ordered list of override modules. The original service image is never modified, so instances on the same host with different override sets use distinct images, while instances with the same override set share one image. The final image carries Docker labels ("org.genoring.override.signature" and "org.genoring.override.base-image-id") that record the override content (module versions and Dockerfile digests) and the base image identifier. An existing image whose labels still match is reused instead of rebuilt, even if another instance built it. The image is written into the generated "docker-compose.yml" as a local-only image. The active override module list and effective image are persisted under "service_overrides" in "config.yml". Override images that are no longer used are not removed automatically, since other instances may still use them. Overrides are built when the target service is compiled, for example with
./genoring.pl build genoring genoring; enabling or disabling an override module does not itself rebuild the target image. Rebuild the target service after changing its active override modules or Dockerfiles. For a public-image service, the same build command applies its overrides without rebuilding the public base image (use "-no-cache" to force the override rebuild). Note: these overrides a related to the Docker container content, but it is also possible to override the definition of service using the container. See "services" item and TEMPLATE services documentation for details. -
"res": if the module needs additional directory and files, that could be mounted in containers for instance, they should be put in this resources directory. Note: Any resource that could be edited by the administrator for customization should be copied in a sub-directory of the "./volumes/" directory at installation time.
Use material provided in modules/TEMPLATE.
When using other genoring services in a module, find the service server using
${COMPOSE_PROJECT_NAME}-SERVICE_NAME where SERVICE_NAME is the name of the
service. For instance, if you want to access the JBrowse service, you would use
${COMPOSE_PROJECT_NAME}-jbrowse. This is necessary to support multiple
GenoRing instances running on a same system.
The proxy service of the "genoring" module is either "genoring-proxy" (nginx, default) or "genoring-proxy-httpd" (Apache HTTPd, "httpd" alternative). Modules that rely on the proxy should support both:
- provide their proxy configuration for both servers ("res/nginx/" and "res/httpd/" directories) and install (or remove) both configurations from their local hooks, in "volumes/proxy/nginx/" and "volumes/proxy/httpd/" sub-directories, so the alternative can be switched at any time. Apache HTTPd configurations included in the "genoring" sub-directory are included in both the HTTP and the HTTPS virtual hosts of the main site;
- provide container hooks and service merge files for both services (ie.
enable_genoring-proxy.shandenable_genoring-proxy-httpd.sh,genoring-proxy.merge.ymlandgenoring-proxy-httpd.merge.yml); - declare dependencies on both services (ie.
"REQUIRES genoring genoring-proxy OR genoring genoring-proxy-httpd").
Other services can reach the proxy using the "genoring-proxy" host name in both cases (it is a network alias of the "genoring-proxy-httpd" service). The Perl API provides GetProxyServer() ("nginx" or "httpd") and GetProxyService() to adapt local hooks to the proxy in use.
"GenoRing core module" refers to the "genoring" module in the "modules" directory. It is responsible for running the CMS — by default Drupal. The core module uses 3 containers: one for the database, one for the HTTP proxy and one holding the CMS code. The main container is the one holding the CMS and referred to as the "genoring" container. That container holds a script named "genoring" that is responsible for "running" the CMS as well as performing a set of commands on that CMS.
Using such a script makes it possible to replace the Drupal CMS by any other CMS as long as the commands of the "genoring" script are implemented. For interactions with the "genoring" container, other modules must (should) only rely on that script rather than directly issuing a complex series of shell commands.
For instance, to install a new Drupal module on the Drupal CMS, it could be achieved using "composer require drupal/some_module" and then enable it using "drush pm-enable some_module". But if somebody wants to implement a replacement GenoRing core module using a different CMS, the "composer" and "drush" scripts will not be available and the module hooks calling them will fail. However, if a "genoring" script is used to handle generic tasks such as installing a CMS module on the main "genoring" container, it becomes possible for a core replacement to implement such a script, handle the commands and adapt them as needed. Now, it is up to the core replacement developer to handle or not the installation of a "some_module" module on the replacement CMS.
Since things can get very complicated to translate commands between CMS, a more generic approach is used: the concept of "recipe". A "recipe" is a directory holding whatever is needed to integrate a module to a CMS. Each implementation of the core genoring script is responsible for handling its own recipe. For instance, the default implementation supporting Drupal will find its own recipe files in the "drupal" subdirectory of the recipe directory provided to either install_recipe or uninstall_recipe commands. An alternative CMS implementation could have its own kind of recipe structure in its own subdirectory, as well as supporting and adapting part or all of the "drupal" recipe to its own CMS.
The commands supported by the main "genoring" container script "genoring" are:
-
help: displays a help message explaining the supported commands.
-
start: starts the CMS in foreground mode. It is the endpoint command to use for the container.
-
install_recipe <recipe_directory> [expand_env]: the recipe directory must contain a subdirectory with the name of the CMS (lower cased with underscores) it is designed for. For instance, for Drupal, it must have a "drupal" subdirectory containing a Drupal recipe. "recipe_directory" should be a relative path from genoring root with no trailing slash. Usually, recipes should be stored in a module "res/recipes/_recipe" directory. It is important, to avoid conflicting recipes, that the last subdirectory uses the module name or a name starting with the module name and ends with "_recipe". For example: "modules/brapimapper/res/recipes/brapimapper_recipe". In that example, the Drupal recipe for the Brapi Mapper module will be located in "modules/brapimapper/res/recipes/brapimapper_recipe/drupal". The recipe is a set of files describing things to do to the CMS to integrate a module. For Drupal CMS, it is based on Drupal Recipes. When you provide a Drupal recipe for your GenoRing module, in your "composer.json", use a package name "genoring/_recipe" (starting with "genoring/" and followed by a slash and the last subdirectory name). If "expand_env" is set to "1", environment variables (of the form
${ENV_VAR_NAME}) in all recipe files are expanded. If set to a string, the string is expected to contain environment variable names separated by commas (no spaces), and only those environment variables will be replaced. Note: only the "genoring" core module and the installed module environment files are loaded during the module installation process. Note: if you want to use public recipes (i.e. published on composer repos), you may get them usingcomposer require drupal/<recipe_name>in the module enable hook, followed bygenoring install_recipe drupal/<recipe_name>. -
uninstall_recipe <recipe_directory>: the recipe directory must contain a subdirectory with the name of the CMS (lower cased with underscores) it is designed for. For instance, for Drupal, it must have a "drupal" subdirectory containing a Drupal recipe. The recipe is a set of files describing things to do to the CMS to integrate a module. For Drupal CMS, it is based on Drupal Recipes.
-
backup [backup_name]: generates a CMS site archive using the given name. Default name should include the date in the format 'backup_YYYYMMDD_HHIISS' where YYYY is the year, MM the month, DD the day of the month, HH current hour (24H format), II current minute and SS current second.
-
restore <backup_name>: restores a CMS site archive from the given backup name.
-
command [arguments...]: runs a shell command. Using
genoring command ...rather than the shell command directly allows genoring core replacement to handle and replace some specific commands such as "drush" commands or "composer" commands. -
update: performs the CMS and extensions updates. Handles site backup and restore in case of update failure.
-
offline: sets the site offline and displays a maintenance message.
-
online: sets the site back online (after an "offline" command).
The Perl API functions (in perllib/) are documented in POD, just above each
function. Each function documentation uses the following structure:
=pod
=head2 FunctionName
B<Description>: What the function does.
B<ArgsCount>: 1-3
=over 4
=item $first: (string) (R)
Description of the first parameter.
=item $second: (hash ref) (U)
Description of the second parameter.
=item $third: (bool) (O)
Description of the third parameter.
=back
B<Return>: (array ref)
Description of the returned value.
=cutEach parameter type is followed by a qualifier:
- (R): required parameter.
- (U): optional parameter, but if it is not provided, it must be set to
undefto keep the order of the following parameters. Example:FunctionName('a', undef, 1). - (O): optional parameter that can be omitted. Only the last parameters can be omitted.
B<ArgsCount> gives the number of arguments the function accepts: the number
of required (R) parameters, followed by the total number of parameters when
some are optional (ie. "1-3"). Functions without parameter use "0" and
functions accepting a variable list of arguments use "n" as the maximum (ie.
"1-n"). Such a list is documented as a list parameter (ie.
=item @modules: (list of strings) (R)).
This section documents non "public usage" commands that may help in the development process.
If you use custom host names that are not resolved by public DNS but are set in
your local /etc/hosts file, Docker containers will not be able to resolve
those hosts as well. To have them resolved in GenoRing, you should add them to
an extra-hosts.yml file in the root of GenoRing (where config.yml is).
Each entry must be on one line in the format (yaml hash):
<hostname>: <IP address>
Ex.:
myhost: 10.10.0.8
tartempion: 192.168.0.42
Note: no spaces at the beginning of the line and the space after ":" is important. Comment lines starting with '#' are supported.
If you try to run directly "docker compose up -d" from the shell, it will fail because GenoRing sets a couple of environment variables that are required by Docker Compose to run a GenoRing project. If you want to be able to run the command above, there is a workaround: you can ask GenoRing to provide a local shell prompt with the environment variables set. Use the command:
perl genoring.pl shell -local [-cmd=<COMMAND>].
The "-cmd" flag is optional and can be used if you need to run a single command or a different shell than "bash" (for instance on a Windows system where "bash" is not available). The environment variable "GENORING_SHELL" is set to one until you exit the shell session (which will bring you back to your original shell without GenoRing environment variables).
If you need to test hooks, there are 2 supported commands:
perl genoring.pl localhooks <HOOK_NAME> <MODULE> [OTHER_ARGS]
and
perl genoring.pl containerhooks <HOOK_NAME> <MODULE> <INCLUDE_RELATED_MODULES> [OTHER_ARGS].
Parameters:
- HOOK_NAME: the hook name.
- MODULE: the target module.
- INCLUDE_RELATED_MODULES: not set, or set to 0 if you do not want to have hooks in other modules matching the hook name and the target module to be executed as well.
- [OTHER_ARGS]: a single string containing additional space-separated arguments to pass to the hook script.
If GenoRing fails to setup properly, it is possible to get a dump of the installation process for investigation:
perl genoring.pl diag > genoring_install.log
This will only capture the first time the system tried to setup. To restart capturing new logs, you will need to remove the file "genoring_setup.log" either from your file system in the "volumes" directory ("./volumes/drupal/genoring_setup.log") or from the "genoring" Docker container ("/opt/drupal/genoring_setup.log" in case of non-exposed volumes).
To run automated tests on the GenoRing framework, go into the "tests" directory and simply run "./test.pl". It will run all the tests of the "t" directory. To run a specific test, just run "perl t/the_test.t". Tests are long to run because of the GenoRing test instance setup time.
Tests are run into temporary GenoRing instances stored in the "tests/temptests" directory. Test logs are available in the "tests/temptests/logs" directory.