0
$\begingroup$

I have 2 .fa sequenes like:

>some description of first...
ACTGACTACGTACGACTACGATCTGACTACACGTAGATAGACTAGTCACTACG
ACTGACTACGTACGACTGACTACATGACTGAC...
> some description of second...
ATCGATAGCTACGATCGATAGTAGATAG....

This file is 461mb (1 chromosome of homo sapiens & 1 chromosome of pan troglodytes) and I want to align it. I am using mafft.

mafft --globalpair --maxiterate 1000 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa

And it returns me

generating a scoring matrix for nucleotide (dist=200) ... done
All-to-all alignment.
/usr/bin/mafft: line 2719:  6030 Killed                  "$prefix/tbfast" _ -u $unalignlevel $localparam -C $numthreads $seqtype $model -g $pgexp -f $pggop -Q $spfactor -h $pgaof -A $usenaivepairscore $focusarg _ -+ $iterate -W $minimumweight -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg $outnum $addarg $add2ndhalfarg -C $numthreadstb $rnaopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f "-"$gop -Q $spfactor -h $aof $param_fft $localparam $algopt $treealg $scoreoutarg $focusarg < infile > /dev/null 2>> "$progressfile"

Why? How could I align it for further comparing?

EDIT 1

MAFFT working cool, when I tried to use it on test sequence file

>test1
ATCGATCGATGACTACGATCGACTACGTACG
ACTAGCTAGCTACGCATAGTAGATCGA
ATCGACTAGCTCAGATCGACTAGATCGATCGT
>test2
TTATCCACTGATCGACTCAGATCGACTCAGAC
ATCGATCAGCTACGATGACTACGTAC
ACTGACTAGCTAGACTGACTAGCTACGTCA

and after MAFFT mafft test.fa > output.fa it returned

test1 ------atcgatcgat-------------gactacgatcgactacgtacgactagctagc tacgcatagtagatcgaatcgactagctcagatcgactagatcgatcgt test2 ttatccactgatcgactcagatcgactcagacatcgatcagctac---------gatgac tacgtac----------actgactagct-agactgactagctacgtca-

So it works.

I can use all 4 at the example. Now about my large file:

alex@alex:~/Desktop/test/BOTH/homo_sapiens_pan_troglodytes$ mafft --thread 4 --maxiterate 2 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
nthread = 4
nthreadpair = 4
nthreadtb = 4
ppenalty_ex = 0
reallocating...
done.
reallocating...
done.
reallocating...
    done.
reallocating...
done.
reallocating...
...
stacksize: 8192 kb
reallocating...
done.
reallocating...
done.
reallocating...
done.
...
generating a scoring matrix for nucleotide (dist=200) ... done
Gap Penalty = -1.53, +0.00, +0.00



Making a distance matrix ..
    1 / 2 (thread    1)
done.

Constructing a UPGMA tree (efffree=1) ... 
    0 / 2
done.

Progressive alignment 1/1... 

len1=230480994, len2=223796828, Switching to the memsave mode
STEP     1 / 1 (thread    0)m/usr/bin/mafft: line 2719:  3192 Killed                  "$prefix/disttbfast" -q $npickup -E $cycledisttbfast -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt $outnum $addarg $add2ndhalfarg -C $numthreads-$numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -g $gexp -f "-"$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg $anchoropt < infile > pre 2>> "$progressfile"

Doesnt matter I do not enter --thread 4 or not, --maxiterate 2 or --maxiterate 1000, it still shows me same error. It takes around 2 minute. Ram usage when running at the first minute is 6656mb, after 1 min 11028mb and then it out of ram and error..

PS

discussion moved here

$\endgroup$
4
  • $\begingroup$ What machine are you running this on, particularly RAM capacity? Just on a remote off chance are there any "core" dumps, e.g. weirdly large files that suddenly 'appear', especially if they have the name 'core' :-) ? $\endgroup$
    – M__
    Jun 20 '20 at 12:46
  • $\begingroup$ I am running this on Ubuntu 20.04, CPU(s): 4, RAM 12 GB, Intel(R) Core(TM) i3-3220.. Do not think that this file could be so large for 12gb ram.. $\endgroup$
    – Alexey
    Jun 20 '20 at 13:00
  • $\begingroup$ I'd do a full diagnostic and post back, because the first stage is obviously working. E.g. have you successfully got MAFFT working on any other data set? Are all processors being used? Whats the RAM usage when running? Do any other long processes/calculations suddenly quit (e.g. hardware bug)? How long before the error pops up. What happens when you greatly lower the maxiterations? $\endgroup$
    – M__
    Jun 20 '20 at 13:11
  • $\begingroup$ @Michael please see my edit 1. But I still want to compare them $\endgroup$
    – Alexey
    Jun 20 '20 at 13:38
1
$\begingroup$

--thread 4 is essential, --maxiterate just try and get it working, not getting it to work optimally. Just make sure there are no other processes running on the machine when --thread 4 is operating, e.g. you are using another machine as your desktop

The gist is as you spread the calculation across your processors (4) and simplify the calculation (lowering the iterations), you are getting further into the calculation, for example now you are getting into gap penalties.

If you are agreed then I would try performing the alignment using just one chromosome of the human genome against the chimp genome and I suspect that would work. Then take it from there.

The objective is to get the calculation working, for example I wouldn't trust a result with max. iterations of 2, but once you have it working you can then start refining the parameters towards optimal results. The i3 processor isn't perfect in context.

Using --thread 4 is super important and you want to increase maxiterate towards the default if the machine supports it.

I suspect the solution is to perform the calculation on an 8+-core machine and set --thread 8 (or higher depending on the number of cores on the machine), at least it will get further into the calculation. With 4-cores and an i3 (diagnostics of RAM usage would be nice) in any case it just doesn't seem to like it. The alternative is to perform this across the genome via sampling the chromosomes on your current i3.


Just to explain why this works. When you have a RAM intensive job and you spread it across cores the RAM overhead massively reduces for each core that is added. Don't ask me how this works in the hardware mecahanics of RAM distribution per core. The reason I know this is because of emperically parallelising Bayesian calculation using each chain on a different core.

$\endgroup$
3
  • $\begingroup$ I am actually comparing all chromosomes of human vs all chromosomes of chimp. I doesn't clearly understand what do you mean. You chould try it yourself I can wait $\endgroup$
    – Alexey
    Jun 20 '20 at 14:48
  • $\begingroup$ Hi @Alexey you have to get it working first. Only when it is working can you move towards the calculation you want. This is not the finished product just a means of verifying it will work. One issue is the i3 processor in the machine. I wouldn't trust a result with max iterations at 2, but once it works you would increase that parameter. $\endgroup$
    – M__
    Jun 20 '20 at 14:56
  • $\begingroup$ Basically to cut to the chase, I think your machine is underpowered for the calculation you are performing. $\endgroup$
    – M__
    Jun 20 '20 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.