Changes between Version 37 and Version 38 of HPC_deploy

Timestamp:

2016-12-23T15:10:45+01:00 (8 years ago)

Author:

Pieter Neerincx

Comment:

--

HPC_deploy

@@ -81 +81 @@
 The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the [wiki:HPC_storage#Software Storage SOP]. We use the [https://www.tacc.utexas.edu/research-development/tacc-projects/lmod Lua based module system (Lmod)] to make software transparently available on all machines at runtime. Details on how to install and configure !EasyBuild, Lmod and our environment sync script on a new cluster can be found in our [https://github.com/molgenis/depad-utils/blob/master/hpc-2.x/README.md depad-utils GitHub repo].
 
-!EasyBuild comes with !EasyConfigs for many of our apps of interest ''out of the box''; These files are stored
-* On our machines in a sub sub sub directory of where !EasyBuild was installed - Latest stable release we deployed.[[BR]]
-  The easiest way to find this directory is to load !EasyBuild and search for an app like this:
+==== Locations of !EasyConfig files to deploy software with !EasyBuild ====
+1. !EasyBuild comes with !EasyConfigs for many of our apps of interest ''out of the box''; These files are stored
+   * On our machines in a sub sub sub directory of where !EasyBuild was installed.[[BR]]
+     The easiest way to find this directory is to load !EasyBuild and search for an app with {{{-S}}} like this:
 {{{
 $> module load EasyBuild
@@ -106 +107 @@
 == temporary directory /tmp/eb-JHpvCj has been removed.
 }}}
-* On [https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs] - Latest and greatest from the source.
-* On our machines in {{{/apps/sources/EasyBuild/custom/}}} - Bleeding edge home made !EasyConfigs that were not yet committed to and pull-merged into the !GitHub repos. 
+   * On [https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs] for the latest and greatest from the source.
+1. Custom EasyConfigs not yet pull-merged into the !GitHub repos from [http://www.ugent.be/hpc/en HPC Ugent] are stored in a fork from this repo at:
+{{{
+https://github.com/molgenis/easybuild-easyconfigs.git
+}}}
+   If it ain't on GitHub your EasyConfig does not exist and software centrally deployed in /apps without EasyConfig on GitHub is subject to removal without warning!
+   We use the standard way of working with GitHub repo's, so:
+   1. Create an online fork @ GitHub
+   2. Clone your own online GitHub fork in for example your home dir on the cluster
+   3. When adding/modifying EasyConfigs commit followed by a push to your own fork.
+      Never push to the blessed master @ https://github.com/molgenis/easybuild-easyconfigs.git!
+   4. Create a pull request to get your changes into the blessed master.
+      Creating a pull request to get your changes directly into the main repo from UGent is off course also an option,
+      but may require more patience to have your pull request processed.
+   5. We do plan to create pull requests from the github.com/molgenis to the source @ github.com/hpcugent ...
+   In pseudo code:
+{{{
+#
+# After creating your online fork @ GitHub, login on a cluster
+# and clone your GitHub repo supplemented with our custom EasyConfigs.
+#
+$> cd ${HOME}
+$> mkdir git
+$> cd git
+$> git clone https://github.com/${your_github_account}/easybuild-easyconfigs.git
+$> cd easybuild-easyconfigs
+$> git remote add blessed https://github.com/molgenis/easybuild-easyconfigs.git
+$> git remote set-url --push blessed non.existing.domain
+
+#
+# Now you can deploy a tool; for example version v16.11.1 of our cluster-utils with EasyBuild
+#
+$> ml EasyBuild
+$> eb --robot \
+      --robot-paths="${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/:" \
+      --software=cluster-utils,v16.11.1
+
+#
+# Software which needs to be compiled and requires a toolchain can be deployed with:
+#
+$> ml EasyBuild
+$> eb --robot \
+     --robot-paths="${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/:" \
+     --software=${name},${version} \
+     --toolchain=foss,2015b
+}}}
+   Note that the **foss 2015b** toolchain is still our default. We will most likely skip 2016a and move to 2016b during scheduled maintenance of summer 2017.
+
+==== Creating a new !EasyConfig ====
 
 If there is no existing !EasyConfig, we have to create one ourselves.
@@ -151 +199 @@
 ==== 0. Make sure perms are correct ====
 
-Either use umask before you start:
+use umask **before** you start:
 {{{
 umask 0002
 }}}
-or use
-{{{
-chmod g+rwX,o+rX,o-w /apps/software/.....   /apps/modules/.....
-}}}
-after deploying, but **before** [#UpdateCacheAndSync running the hpc-environment-sync.bash script]!
 
 ==== 1. Find existing !EasyConfig or create a new one ====
- * To search for existing !EasyConfigs in both the default robot search path as well as in our dir for custom !EasyConfigs:
+ * To search for existing !EasyConfigs in both the default robot search path as well as in your fork of our custom !EasyConfigs git repo cloned into your home dir:
 {{{
 $> module load EasyBuild
 $> module list
-$> eb --robot-paths=:/apps/sources/EasyBuild/custom/ -S MySearchTerm
+$> eb --robot-paths="${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/:" -S MySearchTerm
 }}}
  * When no suitable !EasyConfig is present, create your own:
 {{{
-$> cd /apps/sources/EasyBuild/custom/
-$> touch MyEasyConfig-1.2.3.eb
-}}}
-   Make sure to name the *.eb file exactly the same as the name of the eventually installed ''module'' and its ''version''.
+$> cd ${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/
+$> touch m/MyEasyConfig-1.2.3.eb
+}}}
+   Make sure to name the *.eb file exactly the same as the name of the eventually installed ''module'' and its ''version''. When a toolchain is used this must be included in the file name. E.g.:
+{{{
+$> touch m/MyEasyConfig-1.2.3-foss-2015b.eb
+}}}
 
 ==== 2. Installing the software ====
-Enable the ''robot'' option for automatic dependency resolution and **append** our dir for custom !EasyConfigs to the search path to make !EasyBuild search for deps in first the default and second our custom dir in that order of precedence.
+Enable the ''robot'' option for automatic dependency resolution and **prepend** our custom !EasyConfigs repo to the search path to make !EasyBuild search for deps in our custom !EasyConfigs repo first.
 {{{
 $> module load EasyBuild
 $> module list
-$> eb --robot --robot-paths=:/apps/sources/EasyBuild/custom/ /path/to/MyEasyConfig-1.2.3.eb
+$> eb --robot \
+      --robot-paths="${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/:" \
+      --software=MyEasyConfig,1.2.3 \
+      --toolchain=foss,2015b
 }}}
 
@@ -270 +319 @@
 A: By default !EasyBuild will refuse to overwrite an existing installation. Modules that have been deployed and are used for production should never be modified: deploy a new version instead. During debugging/testing it may be necessary to overwrite a module though; I that case you can force install using ''-f'' like this:
 {{{
-$> eb -f --robot --robot-paths=:/apps/sources/EasyBuild/custom/ /path/to/MyEasyConfig-1.2.3.eb
+$> eb -f --robot --robot-paths="${HOME}/git/easybuild-easyconfigs/easybuild/easyconfigs/:" --software=MyEasyConfig,1.2.3 --toolchain=foss,2015b
 }}}
 ==== Q: I've updated the source code for an app, but when I try to re-deploy with !EasyBuild nothing changes; What is wrong? ====