Introduction
The CTMM TraIT project recently added the Cell Line Use Case (CLUC) to tranSMART. The CLUC is a collection of data on colorectal and prostate cell lines from an exceptionally broad set of platforms, as shown in the table below.
This diverse set is used to:
Standardize data formats and data processing pipelines from the four domain
Test the integration pipelines within the TraIT translational toolset
By incorporating the same platforms as used for ongoing research projects, this cell line set gives a representative test set comparable to real patient data, without the legal burden of handling personal data. The TraIT Cell Line Use Case transmart-ready files are available under the CC0 license for download here.
Please use the following citation when making use of this dataset: Bierkens, Mariska & Bijlard, Jochem "The TraIT cell line use case." Manuscript in preparation. More information can also be found on the Bio-IT World Poster "Multi-omics data analysis in tranSMART using the Cell Line Use Case dataset".
Table of contents
General remarks
Note that folder structure is very important in the upload process, make sure to structure your data in the correct way (figure 1). For more detailed information about the data type you wish to load please refer to the section dedicated to that specific data type.
It is important to setup the batchdb.properties file to provide transmart-batch with the location and login information needed to load the data. A detailed explanation on the properties file can be found here.
For the tutorial the assumption is made that the data is loaded into a local database with default settings, meaning that the database is located on the same machine that has the data folders and the ETL pipeline scripts.
Important note: As transmart-batch currently does not have a pipeline for VCF data this data typ will have to be loaded with Kettle.
Setting up transmart-batch and general documentation
For the complete documentation on transmart-batch please look here.
To use transmart-batch with 16.1 or 16.2 you can use the V1.0 release. To use the latest version please clone the git repository and build transmart-batch:
git clone https://github.com/thehyve/transmart-batch.git cd transmart-batch ./gradlew capsule
After building you should see transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar
Batchdb.properties file
The properties file contains information as the location of the database, the username and password that are used to upload the data to the database. The properties is build up of four lines indicating which database is being used, either PostgreSQL or Oracle, the location of the database and the user.
PostgreSQL
batch.jdbc.driver=org.postgresql.Driver
batch.jdbc.url=jdbc:postgresql://localhost:5432/transmart
batch.jdbc.user=tm_cz
batch.jdbc.password=tm_cz
Oracle
batch.jdbc.driver=oracle.jdbc.driver.OracleDriver
batch.jdbc.url=jdbc:oracle:thin:@localhost:1521:ORCL
batch.jdbc.user=tm_cz
batch.jdbc.password=tm_cz
Data structure and loading the data
In order to load the data properly the scripts need to know were the data is located, in order to achieve this the data structure is more of less set. In the data (available here) the only thing you have to do is extract the files and you are ready to load. The following figure gives an overview of the data types and the way the folder structure is build up. More details about particular datatypes can be found in there respective sections.
Getting the data to the server
If you want to upload the data to a server you first need to get the data on the server. The easiest way to do this is by opening a terminal window and connect to the server:
ssh username@serverAddress
When the connection is made open a new terminal window (do not close the window where you connected to the server) and navigate to the study you want to copy. From the folder the study is located in run the following command:
scp -r study_name username@serverAddress:~(default, folder on server to put the data, ~ is your home folder)
Loading the data
To load the data transmart-batch needs three files.
batchdb.properties file
data to be loaded params file, this can be the data type or the annotation platform params file
Gene expression data
Before data can be loaded into tranSMART, the platform used to generate the data must be loaded. The annotation/platform files are located in annotation folders (see image to the right) and have their own params files to load the annotation data.
Microarray data
mRNA array
Annotation data:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/mrna_annotation.params
Measured data
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/expression.params
miRNA
Requires transmart-data and kettle
Next to platform data miRNA also needs a dictionary before data can be used in advanced analysis. The dictionary maps the small miRNA parts to genes which allows tranSMART to use the miRNA in the advanced analysis work flow. With a clean tranSMART installation only the gene dictionary is loaded, so it could be the miRNA dictionary is not loaded for your instance of tranSMART. To load the dictionary run the following command from the transmart-data/ folder:
make -C data/postgres/ load_mirna_dictionary
if the dictionary is already present running this command will just return that the dictionary is already loaded.
Agilent miRNA microarray
The annotation/platform information is loaded together with the data. The image on the right shows the file structure for this to work.
miRNA microarray annotation and data
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/mirna_annotation.params <path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/mirna.params
in the miRNA folder there are 4 files that are needed for the upload to be successful:
RNAseq data
For the RNAseq data the folder structure is different from the other data types. The platform annotation files for the HiSeq2000 datasets are directly in HiSeq2000 folder and not nested with each dataset. This is done to reduce redundancy, all six different datasets use the same platform meaning it only has to be loaded once. The annotations should be loaded before the rest of the data.
Loading annotation data
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/rnaseq_annotation.params
Illumina GA II RNAseq
The RNAseq data contains (sequence) read counts for transcripts, i.e. it is a measurement for the relative abundance of transcripts.
Loading data
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/rnaseq.params
The folder should contain the following three files:
Illumina HiSeq2000 RNAseq
Similar to GA II RNAseq but more sub folders, each subfolder contains 1 sample. Each sample sub folder has the same three files as displayed above. To load all the datasets you will have to run the data loading command six times, once for each subfolder.
Copy Number Variation ('aCGH') data
For the array CGH the data is available from two different microarrays and one low coverage sequencing, 180k and 224k Agilent microarrays and qDNAseq respectively. The data has been processed on gene and region level generating a total of six data sets to load. The figure on the right shows the folder structure of the data. The platform annotation needs to be loaded before the actual data. Note that for this datatype the parameter files are called 'cnv.params' instead of 'acgh.params'.
Loading the annotation:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/cnv_annotation.params
Loading the data:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/cnv.params
All of the folders containing data should have the following three files:
Proteomics
Requires transmart-data and kettle
Next to platform data Proteomics also needs a dictionary before data can be used in advanced analysis. The dictionary maps the proteins to genes which allows tranSMART to use the proteins in the advanced analysis work flow. With a clean tranSMART installation only the gene dictionary is loaded, so it could be the protein dictionary is not loaded for your instance of tranSMART. To load the dictionary run the following command from the transmart-data/ folder:
make -C data/postgres/ load_proteomics_dictionary
LFQ and MS/MS
Protein quantities
In the proteomics folder you will find a annotation folder and two data folders. To load the proteomics data first load the annotation file, which is the same for both data sets. Afterwards you can load the LFQ and MSMS datasets which are different representations of the same data (Label Free Quantification vs Mass spec)
Loading the annotation:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/proteomics_annotation.params
Loading the data:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/proteomics.params
In the proteomics folder there are three files that are needed for the upload to be successful:
Metadata tags
Now that all data, apart from VCF, has been loaded it is possible to load the metadata for the Cell-line use case. The loading is similar to loading clinical data and can be done by running the following command:
<path_to>/transmart-batch/build/lib/transmart-batch-1.1-SNAPSHOT-capsule.jar -c <path_to>/batchdb.properties -p <path_to>/tags.params
Note: This load job will give feedback that not all the tags have been loaded, this is due to the VCF data not being loaded yet and will be fixed in the VCF section when you read the VCF tags section.
Small Genomic Variants ('VCF')
For this transmart-data with the old pipeline is used as there is no pipeline to load VCF data in transmart-batch. To use this pipeline you need to add the VCF data to the transmart-data folder as shown in the image to the right. To setup transmart-data please look here.
In the Cell-line use case study folder there is a folder called run-with-kettle, in this folder the VCF data is located. This folder also has its own study.params file as the format of this parameter file changed when migrating from kettle to transmart-batch loading. A nice way to add the data to your transmart-data folder is to add a symbolic link:
ln -s <path_to>/run-with-kettle <path_to>/transmart-data/samples/studies/Cell-line
After executing this command you should now see a folder called 'Cell-line' in transmart-data/samples/studies. The Cell-line folder should contain the study.params and the vcf folder.
There a total of 8 datasets available obtained from 3 different platforms. To load all of them at once go into the Cell-line/vcf directory and run
bash load.sh
Note: make sure you have sourced the vars file from transmart-data or this will not work.
As these datasets are quite large compared to the other data types loading may take some time.
Note: uploading a vcf dataset twice will result in undefined behaviour, because the "old" dataset is not removed.
Complete Genomics DNAseq & Illumina GAII RNAseq both have one vcf file to load while Illumina HiSeq2000 RNAseq contains the remanning 6. Each folder with a vcf file should have the following three files
IMPORTANT!
As the VCF pipeline has a separate way of handling study IDs you will probably see greyed out VCF nodes in the database. To make these available you need to connect to your database with pgadmin, psql or Oracle developer and execute the following SQL query:
insert into i2b2metadata.i2b2_secure
(c_hlevel,c_fullname,
c_name,c_synonym_cd,
C_VISUALATTRIBUTES,C_BASECODE,
C_FACTTABLECOLUMN,C_TABLENAME,
C_COLUMNNAME,C_COLUMNDATATYPE,
C_OPERATOR,C_DIMCODE,
C_COMMENT,C_TOOLTIP,
UPDATE_DATE,DOWNLOAD_DATE,IMPORT_DATE,
SOURCESYSTEM_CD,I2B2_ID,
M_APPLIED_PATH,SECURE_OBJ_TOKEN)
select
c_hlevel,c_fullname,
c_name,c_synonym_cd,
C_VISUALATTRIBUTES,C_BASECODE,
C_FACTTABLECOLUMN,C_TABLENAME,
C_COLUMNNAME,C_COLUMNDATATYPE,
C_OPERATOR,C_DIMCODE,
C_COMMENT,C_TOOLTIP,
UPDATE_DATE,DOWNLOAD_DATE,IMPORT_DATE,
SOURCESYSTEM_CD,null,
M_APPLIED_PATH,'EXP:PUBLIC'
from
i2b2metadata.i2b2
where
sourcesystem_cd like 'TraIT-Cell-line_%';
Adding additional meta data tags to the VCF nodes:
Because of the study ID quirk mentioned above the transmart-batch pipeline is unable to load metadata on the actual high dimensional nodes. In the Cell-line_use_case_V1.1/tags folder there is a file called vcf_tags.txt, in this file there is a SQL query that can be used to insert the final metadata into the database.
Setting up transmart-data
To upload the Cell-line use case data you need the transmart-data folder, which can be found here:
git clone https://github.com/transmart/transmart-data
transmart-data contains a collection of scripts and the basic folder structure you need to upload the data. After downloading the transmart-data folder from github you need to update the ETL-pipeline by running the following command from the transmart-data folder:
make -C env update_etl_git make -C env data-integration
After the transmart-ETL update is done there should now be two folders in /transmart-data/env called tranSMART-ETL and data-integration.
The next step is to configure the vars file in transmart-data, there is a sample file called vars.sample, make a copy of this and name it vars.
The vars file contains information for both oracle and postgres databases, as we are using postgres so only the following parameters must be set correctly:
When you are done setting up the vars file, in the transmart-data folder run:
source vars
