Document in preparation: this note will be removed when the final version is approved.

This note describes installation of tranSMART version 19.0, released on 21 April 2020 (installed on Ubuntu 18.04)

Discussion on upgrading to 19.0 from an earlier version will be added soon. As the database has significant updates to bring it into line with i2b2, and ETL procedures have significant improvements in speed we recommend a fresh installation of the database and reloading of studies.

Overview

These instructions are based on the assumption that you are installing tranSMART version 19.0 on Ubuntu 18.04. They draw heavily on previous versions of the release notes, see the credits section of this document for details.

The instructions come in two flavors (1) instructions for an all-in-one script and (2) more detailed step-by-step instructions. The step-by-step instructions are useful when you need to see what is going on and possibly correct problems. They also will serve as a helpful guide for installation on other platforms.

As an overview, the install process implements these five steps. For the step-by-step instructions, follow each step in order, reading each step completely. Expect the process to take several hours (depending on the speed of your internet connection, the power of your machine, and the speed of your machine's memory and disk). You can also see these steps in the comments of the all-in-one script.

1. You will need the set up the necessary support tools (some basic unix command-line tools, PostgreSQL, Grails, Tomcat, R, RServe, and SOLR).
2. You will need to set the database (PostgreSQL is used in these instructions - tranSMART also runs on Oracle), and load the database with the core control data for transmart.
3. You will need to download and copy the release war files to tomcat.
4. You will need to start all the parts of the system
5. And you will need to load some number of datasets.

In addition to the browser-based web-application, there is also a set of R-Based modules that connect to the application's REST interface; these can be used to connect to the database, search for and select data, and to perform analyses.

There is also a Sanity Checklist for a fully functional system, to make sure that everything was correctly installed and it running properly. This may help you pinpoint any problems. 

Getting Help

These instructions attempt to be complete, however, unexpected problems can occur. If you are stuck, there are several sources for help, which can be found on the Getting Support page.

When email or posting a question or a request for help, please describe the situation with as much detail as possible: What were you trying to do? What actions were you taking or had you taken before the error occurred? What was the last command, choice, or action that you entered? What was the error message or last output on the terminal? Remember the person you are in communication with can always envision what you are seeing. Help them to do so.

If you know anyone in the community, especially someone who has recently installed tranSMART, contact them with your question. Often they will have seen the same problem.

Finally, if you learn anything that should be added to these instructions page feed it foward: please leave a comment (at the end)!

Hardware and OS Requirements

The requirements vary based on your specific needs. Since these instructions are aimed at producing a demonstration system for review and evaluation, we are assuming very modest requirements, for example, no more than 100 subjects, a couple of studies with genomic data, and fewer than 10 users; in this case, you can combine the app server with the database server into a single VM or a standalone server with at least the following specs:

• CPU: Dual core
• Memory: 8GB (8GB is usually sufficient, smaller systems may have problems installing R and running larger analysis workflows, depending on fine tuning of postgres)
• Storage: 50 GB (for more studies increase the size of the disk storage)
• OS: linux (these instructions assume: Ubuntu 18.04 with testing planned for Ubuntu 20.04 when that is released).

How to setup a VM on your desktop for evaluation

1. Install VirtualBox (Or VMWare)
2. Create a new VM with at least the above specifications on memory and disk space.
3. Download the ISO file for the linux operation system you want (these instructions assume: Ubuntu 18.04)
4. When booting the VM select the ISO as the disk to boot from.
5. Walk through the instructions of the ISO installation program.

All-in-one Install Script, plus Data-Loading

The "all-in-one" instructions require the minimum of human interactions. In these instructions, there are three major steps. First a set of instructions guides you in setting up a Scripts folder on the home directory of an account with sudo privileges. In the second step, you run the install script that will install all of the support command line tools, create the required support directories, load the required toolsets for PostgreSQL, R and Rserve, load the required R packages, load and start SOLR and the SOLR web interface, and load the transmart war files into Tomcat 8. In addition, the second step sets up the transmart database. In the third step, you can run a script that loads some demo datasets. Optionally, by editing a configuration file, you can select additional datasets to load.

Upon completion of the install you can (optionally) run a set of checking scripts that will check to make sure everything is installed correctly and running properly.

In an additional supplementary process, when the full release is made available you can use the full PGP keys to verify the downloaded install artifacts. This process is described in this additional document. There you will see instructions for downloading tranSMART Foundation PGP keys and using them to verify the digital signature of the downloaded artifacts (that is, the zip files of the install scripts, transmart-data, transmart-etl, and transmart-manual plus the application WAR files transmart.war, and gwava.war).

Make sure the machine or VM you are installing on is according to the Hardware and OS Requirements as specified above.

Set up a transmart account

We assume that you have set up an account with sudo privileges to use in these instructions. For sake of illustration we will use an account named transmart. If you have not already set up such an account, you do so with the following commands (assuming, in this case, that you are logged in with the initial admin account, ubuntu).

# set the password for your new user 'transmart'
# use defaults for the other values if you wish
sudo adduser transmart

# add the user transmart to the sudo group
sudo adduser transmart sudo
 
# log out and log back in with your new user

Create a working directory foir installation. By default, this is called 'transmart'

cd
mkdir -p transmart
cd transmart

Set up the install script

To download the Scripts zip file and install its folder, run the following commands in the home directory of the account that you are going to use to run transmart (e.g in our case, the account named transmart).

sudo apt-get install -y curl
sudo apt-get install -y unzip 

# download the installation script
curl http://library.transmartfoundation.org/release/release19_0_0_artifacts/Scripts-release-19.0.zip -o Scripts-release-19.0.zip
# unzip and rename the Scripts folder
unzip Scripts-release-19.0.zip

Run the install script

Next, we run the install script. This script, if successful, installs tranSMART. It the script fails, read the last message, the error message, carefully; it will usually suggest a fix for the problem. Then, fix the problem and rerun the script. The script is robust with respect to being rerun; and will usually run to completion. To recover error messages, review the install.log file in the install user's home directory. 

In the case that the error seems to have left things in an unrecoverable state, delete the folder ~/transmart (and its contents) and restart the install script as below. This will simply reload all the tranSMART artifacts, without having to reload the command-line tools.

Note: during this process, you will be prompted for the account's password, by the 'sudo' command, at several points along the way.

./Scripts/install-ubuntu18/InstallTransmart.sh 2>&1 | tee ~/install.log

Loading datasets

After completing the installation procedure above, you have an 'empty' instance of tranSMART. Now, you need to load data, either your own or some of the Curated Data Repository sets provided by the Foundation. In these instructions we will illustrate loading data using scripts that access the curated datasets. We will use the script in the load datasets code block, below, to load datasets into the tranSMART database. These instructions describe how to load a representative sample of curated datasets. However, they can be easily generalized; you can choose which datasets you wish loaded (the select datasets code block).

For complete details of the loadable datasets, and additional details on the loading process, see the instructions at the start of the Curated Data Repository wiki page.

These two files are in the Scripts/install-ubuntu directory are involved in this process: 

• datasetsList.txt - the list of possible datasets to load, and
• load_datasets.sh - the script to load the datasets.

By default, a representative sample of the datasets are loaded. Specifically, these datasets are loaded, by default: 
EtriksGSE2125, PfizerGSE22138, RanchoGSE11903, RanchoGSE4382, SanofiGSE38642 . The Curated Data Repository sets wiki page has details of the source and content of these datasets.

Optionally, you can edit the list of datasets to load. Using a text-file editor, like vim, as in the details instructions below, edit the file datasetsList.txt, to uncomment the lines indicating of the datasets that you wish to load and/or comment out the lines of the datasets already loaded or not to be loaded. Please note, it is generally not a good idea to try and load a dataset twice. For each dataset in that list, you can find a description on the  Curated Data Repository sets wiki page. The detailed command sequence, below, includes the loading of vim (a visual version of vi) and the editing of the list of datasets. If you are unsure about this editing process you can run the script without editing the file. Doing so will load the 5 representative datasets, above.

# Note, you may use any text edit, to edit datasetList.txt; herein, we use vi
# this next line will run the install that makes available the vi command used below
sudo apt-get install -y vim
 
# Note: in the 'cd' command, the assumption is that the Scripts directory 
# is in the 'transmart' directory of the current account
# if not, adjust accordingly
cd $HOME/transmart

vi ./Scripts/install-ubuntu18/datasetList.txt

Once you are in the (vi) editor, you can use the following commands:

• i - to switch into "insert" mode (the tag --INSERT-- will appear at the bottom of the screen)

• the arrow keys to move from line to line or in position within a line
• the 'delete' key to delete the character to the left of the cursor
• (when in insert mode) any character key to type that character (for example #, at the start of a line, to comment out that line)
• the ESC (escape) key to get out of insert mode
• ZZ  - (two upper-case z characters - when not in insert mode) to exit and save
• :q!  - (when not in insert mode) to exit immediately without saving

Edit the list of datasets by adding the # character at the start of a line to comment out those datasets that you do not which to load and deleting the # character at the front of the lines corresponding to the datasets that you which to load.

Once you have set up, in datasetsList.txt, the list of datasets you wish to load, then, run the script, load_datasets.sh, as indicated, here. 

# Note: in the 'cd' command, the assumption is that the Scripts directory 
# is in the transmart directory of the current account
# if not, adjust accordingly
cd $HOME/transmart

./Scripts/install-ubuntu/load_datasets.sh 

With this, the install is complete (with datasets loaded). Open a browser to http://localhost:8080/transmart/ where you should see the tranSMART Web Application's login page. On the Analyze tab, in the tree-interface on the left, you will see the list of datasets loaded. The default login is admin/admin, and you can change the password of that account at the Admin Page in the interface.

Is the system running?

Finally, at any time, you can test to see if the install was successful, that is if all the "moving parts" were installed and are running, by using the following commands:

# Note: in the 'cd' command, the assumption is that the Scripts directory 
# is in the transmart directory of the current account
# if not, adjust accordingly
cd $HOME/transmart
 
cd Scripts/install-ubuntu/checks/
./checkAll.sh 2>&1 2>&1 | tee ~/checks.log

Review the log file, checks.log, in your home directory. For details see the section, below, on using the web application, towards the end of this document. To restart the needed tool interfaces (after a reboot, for example) see the section, below on Running SOLR, Rserve, and Tomcat.

Step-by-step Instructions

The rest of this document gives instructions for a step by step series of commands (which are also implemented by the all-in-one script) to better illustrate what is going on and give installers of transmart more opportunity to tune or modify the instructions to particular needs. In addition, seeing the detail of the install process may help you in debugging problems. Finally, it is instructive to go through the process of the install by yourself, in you have the time and inclination. 

Verifying tranSMART release artifacts

In these instructions, you will be download the tranSMART release artifacts from the tranSMART library. The current release is tranSMART release 19.0, and it is visible in the library at http://library.transmartfoundation.org/release/release19_0_0.html. As you can see from that page, the artifacts are presented as either zip or war files and each has a PGP signed "detached signature", and files containing the MD5 hash and a SHA1 hash. These can be used to verify the file being download. The process for verification is covered separately. 

Installing Support Tools

The support tools are, generally, unix command line tools. In the instructions that follow, we assume that your installing on the latest LTS version of Ubuntu (these instructions were developed and tested on Ubuntu 14.04) and are using the command line interface. Since the final product of these instructions is run in the browser, it is assumed that you have loaded (or are using) the desktop version of Ubuntu (or that of your target OS).

Setup

Following tools and frameworks are required prior to installing and running TranSMART. These instructions use Ubuntu's apt-get command to install the supporting tools for the subsequent install of tranSMART. We assume that you are installing on a clean Ubuntu OS. To build Ubuntu OS in a VM, see the search for "installing Ubuntu on a VM".

Also, at least in this context, where you are setting up tranSMART for an introduction and exploration, it is a good idea to set up the initial (root) user as the user that will run the application. You can choose the name and password when you do the install, for example the user we use in these instructions is: name = transmart, password = transmart.

Basics

In this case, for the Ubuntu OS, there is a predefined method for installing rsync, curl, tar, unzip, Java (JDK), php, and PostgreSQL; a make file in the transmart-data archive will load these required tools. However, we will install curl separately, as below, to load and extract transmart-data; so that we can run the script. The command line steps below will do the following:

• Create a folder, transmart, to contain the app, scripts, and data

• Within that folder, using curl, download https://library.transmartfoundation.org/release/release_19_0_0_artifacts/transmart-data-release-19.0.zip. 
• Expand that zip file into ~/transmart/transmart-data
• Within that folder run the initial setup commands (these instructions are basically from the file ~/transmart/transmart-data/README.md)
• Run additional commands to install ant and maven

mkdir ~/transmart
cd ~/transmart
sudo apt-get install -y curl
curl http://library.transmartfoundation.org/release/release19_0_0_artifacts/transmart-data-release-19.0.zip -o transmart-data-release-19.0.zip
unzip transmart-data-release-19.0.zip
cd transmart-data
sudo make -C env ubuntu_deps_root
sudo make -C env install_ubuntu_packages18
make -C env ubuntu_deps_regular18
make -C env groovy
sudo apt-get install -y ant
sudo apt-get install -y maven

In the following instructions, it will be necessary to edit configuration files. We are using the visually enhanced version of vi called vim.  Feel free to use another editing tool. If you wish to closely follow the editing instructions here, download and install vim:

sudo apt-get install -y vim

For more detailed information on basic OS commands see this "expand" box.

Install Tomcat

You will need a framework in which to run the transmart war files. Most commonly, for tranSMART exploratory installs, we use Tomcat8 (we have not tested on version 9). For more general notes on installing Tomcat in other OS environments, visit the apache tomcat site.

sudo apt-get install -y tomcat8 

The above install will also start tomcat, but we need to shut it down for now; we will start it later in this process.

sudo systemctl stop tomcat8

To install R and Rserve type the following (this will take several minutes):

cd ~/transmart/transmart-data
cp vars.sample vars
. ./vars
make -C R install_packages

NOTE: There is a problem with the fact that R, as installed above, is not on the PATH System variable. The following three commands are a work-around to fix this problem. Eventually, it will have to be fixed in the scripts. These commands add the path for R to the PATH variable (for all processes); this is done by creating an (new) file Rpath.sh and adding it to the /etc/profile.d/ directory.

cd
echo "export PATH=/home/$USER/transmart/transmart-data/R/root/bin:$PATH" > Rpath.sh
sudo mv Rpath.sh /etc/profile.d/

Make sure you use the full path for the R installation - the path to the R executable.

This change will take place when you next start the machine, login, or start a new terminal window or shell. To have them take effect immediately you must:

source /etc/profile.d/Rpath.sh

Check and make sure that you have the correct version of R on your PATH:

R --version
R CMD Rserve --version

The R version number should be 4.0.0 although it has been reported that other versions work

The Rserve version number should be 1.8-7 or higher. Earlier versions (including the latest version typically installed by R) have a bug which can crash some tranSMART workflows with a type error. To avoid this, the tranSMART R install picks up the latest Rserve from Rforge.net.

If errors occur in advanced workflows, check that the R version is working.

The R server writes by default to file /var/log/tomcat8/rserve-transmart.log and runs as user tomcat8.

Install Rserve service

Create the Rserve system service using sudo:

sudo TABLESPACES=$TABLESPACES RSERVE_USER="tomcat8" make -C R install_rserve_unit

Install SOLR

SOLR is a search engine that uses Lucene to build fast searches. In our case, it is built on top of Hibernate so that each search targets a pseudo-table generated by a data-base query. It is used in multiple places in tranSMART. This process runs in java. So, it should be (mostly) independent of OS type.

In these instructions it is installed as a stand-alone web application using configuration files distributed in transmart-data.

See the documentation for a full explanation of other ways in which it can be installed.

Install SOLR by using the transmart-data command line 'make', create the solr service using sudo

cd ~/transmart/transmart-data
. ./vars
make -C solr solr_home
sudo TABLESPACES=$TABLESPACES SOLR_USER="$USER" make -C solr install_solr_unit

This downloads and installs solr, then creates a service to launch solr for tranSMART.

The transmartApp Web Application

Set up a basic database that supports login and contains five studies. Set up the configuration file. Set up the war files. Start Rserve, SOLR, and Tomcat.

Setting up a new database and sample data

We use the make files in transmart-data to create a new database. These processes should run independent of OS type,

cd ~/transmart/transmart-data
. ./vars
make -j4 postgres

Create the tranSMART Configuration Files

To create the configuration files for the tranSMART web application and the connection to the database:

cd ~/transmart/transmart-data
. ./vars
make -C config install

Unfortunately, these instructions set up the configuration files in the configuration directory for the user transmart in the directory /home/$USER/.grails/transmartConfig, but they need instead to be in the configuration directory of the user tomcat8 which is in the directory /var/lib/tomcat8/.grails/transmartConfig on Ubuntu 18.04. So, we copy them:

sudo mkdir -p /var/lib/tomcat8/.grails/transmartConfig/
sudo cp /home/$USER/.grails/transmartConfig/*.groovy /var/lib/tomcat8/.grails/transmartConfig/
sudo chown -R tomcat8.tomcat8 /var/lib/tomcat8/.grails/transmartConfig/

Setting Up The Web Application

From the transmart-library web site, download the application's war files: transmartApp-release-19.0.war and gwava-release-19.0.war. We will copy then to a staging directory at ~/transmart/war-files . And then, install then in tomcat's webapps directory as transmart.war and gwava.war. Specifically,

cd ~/transmart
mkdir war-files
cd war-files
curl http://library.transmartfoundation.org/release/release19_0_0_artifacts/transmartApp-release-19.0.war --output transmart.war
curl http://library.transmartfoundation.org/release/release19_0_0_artifacts/gwava-release-19.0.war--output gwava.war
sudo cp *.war /var/lib/tomcat8/webapps/

Setting Up The Online Manual

From the transmart-library web site, download the HTML pages for the tranSMART manual and then, install then in tomcat's webapps directory as transmartmanual. Specifically,

cd ~/transmart
curl http://library.transmartfoundation.org/release/release19_0_0_artifacts/transmart-manual-release-19.0.zip --output transmartmanual.zip
unzip transmartmanual.zip
sudo mv transmartmanual /var/lib/tomcat8/webapps/
sudo chown -R tomcat8.tomcat8 /var/lib/tomcat8/webapps/transmartmanual

Loading Data into the Database

The description of loading data is covered on in two separate locations, see the Load Datasets section, above, in the all-in-one instructions, and/or the data loading instructions on the DataCuration page.

Running SOLR, Rserve, and Tomcat

These instructions will need to be run (checked) each time you start up. 

SOLR

Since SOLR runs in Java, these instructions are easily translated into other OS types. Check to see that SOLR is running with:

ps aux | grep start.jar 

If it is not found, start it with:

sudo systemctl start solr

This last statement rebuilds all the indexes (should be done after each database load; and with SOLR running as above). You will also need to rebuild the index if you do any editing on the browse page in the tranSMART web application; browse page editing is not covered in these notes.

cd ~/transmart/transmart-data
. ./vars 
make -C solr browse_full_import rwg_full_import sample_full_import

Rserve

Check to see that Rserve is running with:

 ps aux | grep Rserve

If it is not found, start it with:

sudo systemctl start rserve

NOTE: Rserve must be running under the user tomcat8 which is the initial default for the service definition used above. It is a serious security flaw to run Rserve as a privileged user.

Tomcat

Check to see that Tomcat8 is running with:

 sudo systemctl status tomcat8

If is is running, restart it with:

 sudo systemctl restart tomcat8

If is is not running, start it with:

 sudo systemctl start tomcat8

Using the Web Application

Browse to http://localhost:8080/transmart ; log in with username = admin, password = admin ; click on the Analyze tab.

You should see one study "changed".

Drag it to the selection box and click on "Summary Statistics": this will show a summary of statistics for that study.

In this case, transmart is up and running. 

Notes on PostgreSQL

ls /var/lib/postgresql/tablespaces

sudo sytstemctl start postgresql
sudo systemctl stop postgresql
sudo systemctl status postgresql