I have merged two huge .gen files with roughly 500k samples in total - around 3 TB.
I wish to run an IBD (identity by descent) check to identify if there are identical participants in the two cohorts before doing any further analyses.
In the past I have simply used the --genome argument in PLINK and it would work fine however, in this instance the file is so large that doing such would cause the following error to appear: Error: Out of memory. The --memory flag may be helpful.
I then opted to use PLINK v2 as it has a --parallel argument that, apparently, can be used alongside --make-king and/or --make-king-table to conduct this IBD check. I have had some trouble understanding the documentation surrounding these aforementioned arguments and would like some clarification as to how to ideally use them. Please assume that I am running on a multi-node, 24 core cluster.
Here is the current code I am running:
plink2 --gen testo.gen \
--sample testo.sample \
--make-king \
--make-king-table \
--parallel 1 24
And here is what the PLINK documentation says about the arguments for this command:
--parallel <1-based current job index> <total job pieces>
Now, I have failed to understand what the first argument does, even after playing around with the parameters a little and extensively searching the web for an answer.
What would changing this "1-based current job index" do? What does it mean? And for "total job pieces", will the adjustment of this parameter alter how many cores the jobs are distributed to or is it just an identifier as to how many compartmentalized sets are being made?
If not, is there any way to specify which nodes you would like to run a job on and specify the number of cores as well? If yes, could you please provide an example?
Thanks.
