Changes between Version 28 and Version 29 of HPC_deploy

Timestamp:

2015-11-19T01:46:17+01:00 (10 years ago)

Author:

Pieter Neerincx

Comment:

--

HPC_deploy

@@ -9 +9 @@
 Responsibility for deploying and maintaining software is distributed over two teams:
 * Deployment of ''system'' software (packages from the repos of the Linux distros we use) is handled by our sys admins and beyond the scope of this SOP.
-* Deployment of bioinformatics software is handled by the bioinformaticians from the ''depad'' group.
-
-If you want to join the ''depad'' group, please [wiki:Contact contact the helpdesk]. 
+* Deployment of bioinformatics software is handled by the bioinformaticians from the ''depad'' group.[[BR]]
+  If you want to join the ''depad'' group, please [wiki:Contact contact the helpdesk].
+
 The depad group uses [https://hpcugent.github.io/easybuild/ EasyBuild], which in turn uses !EasyConfigs as recipes to enforce consistent, reproducible installations.
 In a nutshell an !EasyConfig deployment recipe can handle the following steps:
@@ -24 +24 @@
 1. Generate a module file for use with a module system to configure the environment at runtime.
 
-The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the [wiki:HPC_storage#Software Storage SOP].
+The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the [wiki:HPC_storage#Software Storage SOP]. We use the [https://www.tacc.utexas.edu/research-development/tacc-projects/lmod Lua based module system (Lmod)] to make software transparently available on all machines at runtime. Details on how to install and configure !EasyBuild, Lmod and our environment sync script on a new cluster can be found in our [https://github.com/molgenis/depad-utils/blob/master/hpc-2.x/README.md depad-utils GitHub repo].
+
 !EasyBuild comes with !EasyConfigs for many of our apps of interest ''out of the box''; These files are stored
 * On our machines in a sub sub sub directory of where !EasyBuild was installed - Latest stable release we deployed.[[BR]]
@@ -115 +116 @@
 }}}
 
+=== Specifying the default version of an app (optional) ===
+
+When you load an app into your environment with the {{{module load}}} command without specifying explicitly which version you want to use, Lmod will load the highest version number. This is usually fine, but if you installed a new version that is not yet well tested, you may want to explicitly configure an older version as the default. This can simply be accomplished by creating a **relative** symlink named default. Our module files are located at {{{/apps/modules/}}}. For example let's look at our NGS_DNA analysis pipeline. It is part of the ''bio'' collection of apps, so the the module files are in:
+{{{
+$> ls -ahl /apps/modules/bio/NGS_DNA/
+total 20K
+drwxrwsr-x  2 umcg-gvdvries  umcg-depad 4.0K Nov 12 15:28 .
+drwxrwsr-x 64 umcg-pneerincx umcg-depad 4.0K Nov 18 17:43 ..
+lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Aug 25 10:52 3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 -> /apps/modules/all/NGS_DNA/3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
+lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Nov  4 09:04 3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 -> /apps/modules/all/NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
+lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Nov 12 15:28 3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45 -> /apps/modules/all/NGS_DNA/3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45
+lrwxrwxrwx  1 umcg-pneerincx umcg-depad   45 Sep 24 16:22 default -> 3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
+}}}
+Note that all module files for all apps are in {{{/apps/modules/all/}}} and we only see symlinks in {{{/apps/modules/bio/}}}. Version 3.2.1 is the latest, but 3.1.2 has been designated as default using a symlink. The symlink must be a relative one pointing to file or another symlink in the same directory. We can check with if the symlink was recognised using:
+{{{
+$> module avail NGS_DNA
+
+-------------------------- /apps/modules/bio --------------------------
+   NGS_DNA/3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
+   NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 (D)
+   NGS_DNA/3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45
+
+  Where:
+   (D):  Default Module
+
+$> module load NGS_DNA
+$> module list
+
+Currently Loaded Modules:
+  1) Java/1.7.0_80 
+  2) Molgenis-Compute/v15.04.1-Java-1.7.0_80 
+  3) NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
+
+}}}
+
+=== Deprecating a previously installed older version of an app ===
+
+You can inform users that (a certain version of) an app is deprecated and will be removed in the near future by creating a custom message when an app is added to the environment with {{{module load}}}. Add your custom message to {{{/apps/modules/modules.admin}}}. For example with this modules.admin:
+{{{
+#
+# The Lmod admin file consists of "key: value" pairs terminated with a blank line:
+#
+#    moduleName/version:  message
+#    <blank line>
+#
+# Or
+#
+#    Full/PATH/to/Modulefile: message
+#    <blank line>
+#
+# The message can be as many lines as you like and must be terminated with a blank line.
+#
+# Currently used by picard/1.102-Java-1.7.0_80
+R/3.1.2-goolf-1.7.20: Deprecated incomplete installation. Will be removed in the near future.
+}}}
+loading this specific version of ''R'' will now result in:
+{{{
+$> module load R/3.1.2-goolf-1.7.20
+--------------------------------------------------------------------------
+There are messages associated with the following module(s):
+--------------------------------------------------------------------------
+R/3.1.2-goolf-1.7.20:
+   Deprecated incomplete installation. Will be removed in the near future.
+--------------------------------------------------------------------------
+}}}
+
 === Updating the Lmod caches and syncing installed software to nodes === 
 
@@ -141 +208 @@
 === Q: !EasyBuild fails to download the source code. How can I continue the installation process? ===
 A: First check if the location where !EasyBuild tries to download the source code is still up-to-date. If not update the !EasyConfig. If the location is correct, but !EasyBuild cannot access this location directly for example because it is blocked by our firewall or because it requires authentication, you can try to download the source manually and put it in the cache directory for the app in {{{/apps/sources/[a-z]/NameOfTheApp/}}}. When !EasyBuild finds the cached source code it will skip the download step and continue.
-
-
-
- 
-