[[TracNav(xQTL)]] = [wiki:xQTL xQTL workbench] - Setup job submission = TODO! content here === Configure your local system to accept and run jobs === * Use ''Settings'' -> ''Install R packages'' * !BioConductor package, manual install?? * Curl install?? === Configure your computer cluster to accept and run jobs === * You have a LINUX/UNIX cluster, then: * Install OpenPBS (torque+maoi) * Install [http://www.r-project.org/ R] * Your scheduler (master/head..) node must a SSH keyboard interactive login mechanism * Try to log in to the cluster from your application server. This will accept the SSH key and ensure your firewall isn't blocking outgoing SSH. === Curl install === Inside R: install.packages("RCurl",lib="~/libs") might result in... `installing *source* package ‘RCurl’ ...`[[BR]] `checking for curl-config... no`[[BR]] `Cannot find curl-config`[[BR]] `ERROR: configuration failed for package ‘RCurl’`[[BR]] Fix by... sudo apt-get install libcurl3-openssl-dev (SuSE: zypper in libcurl-devel) inside, try again `*** installing help indices`[[BR]] `** building package indices ...`[[BR]] `* DONE (RCurl)`[[BR]] === Install R packages plugin === * Press 'Install' for R/qtl (package 'qtl') Console: `Going to install: qtl`[[BR]] `EXECUTING: echo install.packages\(\"qtl\",repos=\"http://cran.xl-mirror.nl/\",lib=\"/Users/joeri/libs\"\)\;q\(\"no\"\) > /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315818859357251000.R`[[BR]] `EXECUTING: R CMD BATCH /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315818859357251000.R`[[BR]] `exit value = 127`[[BR]] `Instalation of qtl failed`[[BR]] Means you first need to install R. After this is done, try again. Now it should read `Going to install: qtl`[[BR]] `EXECUTING: echo install.packages\(\"qtl\",repos=\"http://cran.xl-mirror.nl/\",lib=\"/Users/joeri/libs\"\)\;q\(\"no\"\) > /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315820155628099000.R`[[BR]] `EXECUTING: R CMD BATCH /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315820155628099000.R`[[BR]] `Done installing qtl`[[BR]] After which the plugin checks whether the libraries are installed by looking at USER_HOME/libs. If a directory is not found, it is assumed that it is not installed. `checking dir: /Users/joeri/libs/bitops`[[BR]] `checking dir: /Users/joeri/libs/qtl`[[BR]] `checking dir: /Users/joeri/libs/RCurl`[[BR]] `bitopsDir exists`[[BR]] `qtlDir exists`[[BR]] `rcurlDir exists`[[BR]] = OLD STUFF = = Installation = == Overview == * Configure your local system to accept and run jobs * Configure your computer cluster to accept and run jobs == Running on Tomcat == * NOTE: The Tomcat user is the 'special' user under which Tomcat is running and accessing the system. (ie. tomcat6) * The Tomcat user must own the Tomcat home directory to install dependencies. (ie. /user/share/tomcat6) * NOTE: If libraries are installed owned by root, they cannot be overwritten by Tomcat, meaning you could be stuck with an old version. If problems occur, delete the libraries and reinstall using the Dependency Manager plugin. * The Tomcat user must own the Tomcat java working directory to be able to execute commands. (ie. /var/lib/tomcat6, use the '''tomcattmpdir''' helper servlet to find out) * If 'RCurl' won't install under UNIX, try using the commandline as root and install the following package: 'libcurl3-openssl-dev' == Configuration for Windows == * Install RTools [http://www.murdoch-sutherland.com/Rtools/index.html RTools] * Install Curl for Windows, use a generic binary. Put the install dir on your PATH. [http://curl.haxx.se/download.html cURL] * ... === Continuation === * Optional: if your cluster uses a module system to load programs, load this set: {{{ module initadd torque module initadd maui module initadd R }}} * Then log out from the cluster, then log back in. * Create a libs folder for the dependencies using {{{mkdir libs}}} * Start R and install a few things: {{{ install.packages("RCurl",lib="~/libs") install.packages("qtl",lib="~/libs") install.packages("snow",lib="~/libs") }}} * Install our custom 'ClusterJobs' library by performing a 'checkout' and R install. From your home folder, do: {{{ wget -r -l2 http://www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/ClusterJobs/ cd www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/ R CMD INSTALL ClusterJobs --library=~/libs }}} * Make sure the firewall of your receiver (server that is running XGAP+Cluster+R/qtl) is not blocking port 8080 so the cluster can report back progress and save your results. = other old stuff = === Adding new !ComputationResources === It is possible to add new types of ComputationResource, places where jobs can be executed. The Java interface ComputationResource is located at /xgap_1_4_distro/handwritten/java/plugins/cluster/interfaces/ComputationResource.java in your checked-out project and can be implemented. In the folder /xgap_1_4_distro/handwritten/java/plugins/cluster/implementations/ you can find two implementations that we currently provide: [http://www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/implementations/LocalComputationResource.java LocalComputationResource.java] and [http://www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/implementations/ClusterComputationResource.java ClusterComputationResource.java]. They implement the following functions: {{{ public boolean installDependencies() }}} Install the needed R-packages on this ComputationResource. {{{ public boolean cleanupJob(int jobIs) }}} Remove job files and generated data from this ComputationResource. {{{ public String executeCommand(Command command) }}} Execute a system command on this ComputationResource. {{{ public List executeCommands(List commands) }}} Execute multiple system commands. Not a wrapper for executeCommand, because it can behave differently. At the moment, extending the system is not trivial because there are many ComputationResource specific parts in the core workings of '''ClusterPlugin.java'''. In further development, we will gradually make this more generic and easily extendable.