# GSEA - Gene Set Enrichment Analysis - in R studio

I have been trying to conduct a classic GSEA using the broad institute software and can not figure out how to compile it in R for the life of me. I am aware that there exists other packages in R that are almost the same, such as fgsea however, I would like to use this one first and then compare it with these other methods.

I have already generated a ranked list for my contrast of interest and save it as a .rnk file. Here is what I tried to do:

gseaCallH <-
paste("java -Xmx2G -cp C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea2-2.2.4.jar xtools.gsea.GseaPreranked -param_file C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/GSEAPrerankedParameters.txt -rnk C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea_treatFU_min.rnk -gmx C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/h.all.v6.2.symbols.gmt -rpt_label gsea_treatFU_min.rnk -out C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/C7",sep="")

testGSEA <- system(command = gseaCallH, intern = TRUE, ignore.stderr = TRUE)


When I run the system command I get the following error: had status 1

At first I thought maybe it had something to do with JAVA not being properly installed however, I have installed it and still nothing. Maybe the installation was not done correctly? I have tried running it directly from cmd (I'm running windows 10) and got the following error:

Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap


Can anyone advise on how to proceed?

• Are you sure you have the specs to run the application? Did you check the Java version?
– llrs
May 2 '19 at 7:07
• BTW, do you have the rJava package installed? It requires to configure R to be able to use Java (basically running R CMD javareconf) maybe it helps too. Because the way you run it now from R is sending it back to the command line, so you should get the error from the command line
– llrs
May 2 '19 at 7:24
• If by specs you mean the software then yes, I have downloaded the desktop version of the application as well May 2 '19 at 13:21
• Yes, then it might be a problem for their support system... Good luck
– llrs
May 2 '19 at 13:24
• OK... Thank you anyway! May 2 '19 at 13:35

## 2 Answers

I have answered my own question however, it is specific to windows users.. If you get the same error from the cmd as I did, you must manually put in environment variables related to Java. Do as the user Mohamed did in this SO post: https://stackoverflow.com/questions/9303889/error-occurred-during-initialization-of-vm-could-not-reserve-enough-space-for

Following this, you must avoid using paths such as C:/absolute_path and instead use /absolute_path, if you do the former, you will get the exception: java.net.MalformedURLException: unknown protocol: c

In the end, after manually entering the environment variables and running the following command: java -Xmx2G -cp /Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea2-2.2.4.jar xtools.gsea.GseaPreranked -param_file /Users/omk/Desktop/BIOINFORMATIC/MsigDB/GSEAPrerankedParameters.txt -rnk /Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea_treatFU_min.rnk -gmx /Users/omk/Desktop/BIOINFORMATIC/MsigDB/h.all.v6.2.symbols.gmt -rpt_label gsea_treatFU_min.rnk -out /Users/omk/Desktop/BIOINFORMATIC/MsigDB/C7 it has worked.

Where the command can be executed directly from the command line or in R using the system function.

Note: A '#' separated by a space must be present at the beginning of your .rnk file for the program to work. Therefore, .rnk files must have the following format:

# Genes Coeffecients TSPAN6 0.2345676 etc. for every gene.

Are you calling java with sufficient memory? Try increasing -Xmx2G to -Xmx8G, provided your PC has 8 GB RAM.

• Good call. I tried this and got the exception Invalid maximum heap size: -Xmx8G May 2 '19 at 14:22
• a quick google search suggests this might be due to 32 bit java vs 64 bit java? May 3 '19 at 7:36
• So it actually had to do with the environment variables for Java in my computer. I had to manually insert some of them - _JAVA_OPTIONS and JAVA_HOME. Here is a SO post that was helpful: stackoverflow.com/questions/9303889/… Unfortunately I'm still getting an error - a different one though - malformed url exception, unknown protocol C. So I'm getting closer but still no luck.. May 3 '19 at 14:00