I have been trying to conduct a classic GSEA using the broad institute software and can not figure out how to compile it in R for the life of me. I am aware that there exists other packages in R that are almost the same, such as fgsea however, I would like to use this one first and then compare it with these other methods.

I have already generated a ranked list for my contrast of interest and save it as a .rnk file. Here is what I tried to do:

gseaCallH <- 
  paste("java -Xmx2G -cp C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea2-2.2.4.jar xtools.gsea.GseaPreranked -param_file C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/GSEAPrerankedParameters.txt -rnk C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea_treatFU_min.rnk -gmx C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/h.all.v6.2.symbols.gmt -rpt_label gsea_treatFU_min.rnk -out C:/Users/omk/Desktop/BIOINFORMATIC/MsigDB/C7",sep="")

testGSEA <- system(command = gseaCallH, intern = TRUE, ignore.stderr = TRUE)

When I run the system command I get the following error: had status 1

At first I thought maybe it had something to do with JAVA not being properly installed however, I have installed it and still nothing. Maybe the installation was not done correctly? I have tried running it directly from cmd (I'm running windows 10) and got the following error:

Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap

Can anyone advise on how to proceed?

  • $\begingroup$ Are you sure you have the specs to run the application? Did you check the Java version? $\endgroup$
    – llrs
    May 2, 2019 at 7:07
  • $\begingroup$ BTW, do you have the rJava package installed? It requires to configure R to be able to use Java (basically running R CMD javareconf) maybe it helps too. Because the way you run it now from R is sending it back to the command line, so you should get the error from the command line $\endgroup$
    – llrs
    May 2, 2019 at 7:24
  • $\begingroup$ If by specs you mean the software then yes, I have downloaded the desktop version of the application as well $\endgroup$
    – h3ab74
    May 2, 2019 at 13:21
  • $\begingroup$ Yes, then it might be a problem for their support system... Good luck $\endgroup$
    – llrs
    May 2, 2019 at 13:24

2 Answers 2


I have answered my own question however, it is specific to windows users.. If you get the same error from the cmd as I did, you must manually put in environment variables related to Java. Do as the user Mohamed did in this SO post: https://stackoverflow.com/questions/9303889/error-occurred-during-initialization-of-vm-could-not-reserve-enough-space-for

Following this, you must avoid using paths such as C:/absolute_path and instead use /absolute_path, if you do the former, you will get the exception: java.net.MalformedURLException: unknown protocol: c

In the end, after manually entering the environment variables and running the following command: java -Xmx2G -cp /Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea2-2.2.4.jar xtools.gsea.GseaPreranked -param_file /Users/omk/Desktop/BIOINFORMATIC/MsigDB/GSEAPrerankedParameters.txt -rnk /Users/omk/Desktop/BIOINFORMATIC/MsigDB/gsea_treatFU_min.rnk -gmx /Users/omk/Desktop/BIOINFORMATIC/MsigDB/h.all.v6.2.symbols.gmt -rpt_label gsea_treatFU_min.rnk -out /Users/omk/Desktop/BIOINFORMATIC/MsigDB/C7 it has worked.

Where the command can be executed directly from the command line or in R using the system function.

Note: A '#' separated by a space must be present at the beginning of your .rnk file for the program to work. Therefore, .rnk files must have the following format:

# Genes Coeffecients TSPAN6 0.2345676 etc. for every gene.


Are you calling java with sufficient memory? Try increasing -Xmx2G to -Xmx8G, provided your PC has 8 GB RAM.

  • $\begingroup$ Good call. I tried this and got the exception Invalid maximum heap size: -Xmx8G $\endgroup$
    – h3ab74
    May 2, 2019 at 14:22
  • $\begingroup$ a quick google search suggests this might be due to 32 bit java vs 64 bit java? $\endgroup$
    – Pallie
    May 3, 2019 at 7:36
  • $\begingroup$ So it actually had to do with the environment variables for Java in my computer. I had to manually insert some of them - _JAVA_OPTIONS and JAVA_HOME. Here is a SO post that was helpful: stackoverflow.com/questions/9303889/… Unfortunately I'm still getting an error - a different one though - malformed url exception, unknown protocol C. So I'm getting closer but still no luck.. $\endgroup$
    – h3ab74
    May 3, 2019 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.