1
$\begingroup$

I have a data frame where the row names are the species names, and each column can either be an Amino-Acid character or "-".

I wish to write it in a fasta format. Simple example of the format is:

>UniRef90_A0A1V4QIJ3
------------------------------------------------------------
--------------------------MDISTFSP-----PVAG-----------VE----
------ITAF-------------V------------------------------------
LN--------G-------------------------------------------------
----EKIPA-----------------------------------------IVLSKD----
---LNPFL------------------Q-ELLDQEPAC-----------------------
------------EHDIGTCYG------------------------DA-------------
--------------------------------------------------PC-----IS-
-----------------------GPDAWNHSGFQC----Q---------VC---------
------------GLVKSWNL-------------------VGDFVVYRQE-----------
>UniRef90_A0A0L9V5F8
------------------------------------------------------------
RFHTLFRNEYGHLRVLQRFDQRSKQIQNLENYRLVEFKSKPNTLLLPHHADADFLLVVLN
GRALLTLVNP-------------D---------------------------------G--
RDSYILEQGHA-------------------------------------------------
----QKIPAGTIFFLVNPNDNENLRI------------------------IKIATPINNP
HRFQDFFL-------------SSTEAQQSYLQG--FS-----------------------
----------------KNVLEASFDSEFKEINRVLFGEEGQQQQGEESQQEGVIVELERE
QIRELIKHA-----------------------------------------KSSSRRSLS-
-----------------------SQ-----DEPFNLRNRKP--------IY--SN-----
------------KF-GRWYEITPEKNP------------Q--------------------

I can do it using the following for loop:

for (n in row.names(df)){
    ss = do.call(paste, c(as.list(mu_df[1, ]), sep = ""))
    write.fasta(ss,
                  names=n,
                  as.string=TRUE,
                  open="a",
                  new_file_path)
}

Is there a faster way of writing it? I have to write millions of those files and I wish to optimize it.

In order to create the data frame:

structure(list(V1 = c("-", "-"), V2 = c("-", "-"), V3 = c("-", 
"-"), V4 = c("-", "-"), V5 = c("-", "-"), V6 = c("-", "-"), V7 = c("-", 
"-"), V8 = c("-", "-"), V9 = c("-", "-"), V10 = c("-", "-"), 
    V11 = c("-", "-"), V12 = c("-", "-"), V13 = c("-", "-"), 
    V14 = c("-", "-"), V15 = c("-", "-"), V16 = c("-", "-"), 
    V17 = c("-", "-"), V18 = c("-", "-"), V19 = c("-", "-"), 
    V20 = c("-", "-"), V21 = c("-", "-"), V22 = c("-", "-"), 
    V23 = c("-", "-"), V24 = c("-", "-"), V25 = c("-", "-"), 
    V26 = c("-", "-"), V27 = c("-", "-"), V28 = c("-", "-"), 
    V29 = c("-", "-"), V30 = c("-", "-"), V31 = c("-", "-"), 
    V32 = c("-", "-"), V33 = c("-", "-"), V34 = c("-", "-"), 
    V35 = c("-", "-"), V36 = c("-", "-"), V37 = c("-", "-"), 
    V38 = c("-", "-"), V39 = c("-", "-"), V40 = c("-", "-"), 
    V41 = c("-", "-"), V42 = c("-", "-"), V43 = c("-", "-"), 
    V44 = c("-", "-"), V45 = c("-", "-"), V46 = c("-", "-"), 
    V47 = c("-", "-"), V48 = c("-", "-"), V49 = c("-", "-"), 
    V50 = c("-", "-"), V51 = c("-", "-"), V52 = c("-", "-"), 
    V53 = c("-", "-"), V54 = c("-", "-"), V55 = c("-", "-"), 
    V56 = c("-", "-"), V57 = c("-", "-"), V58 = c("-", "-"), 
    V59 = c("-", "-"), V60 = c("-", "-"), V61 = c("-", "R"), 
    V62 = c("-", "F"), V63 = c("-", "H"), V64 = c("-", "T"), 
    V65 = c("-", "L"), V66 = c("-", "F"), V67 = c("-", "R"), 
    V68 = c("-", "N"), V69 = c("-", "E"), V70 = c("-", "Y"), 
    V71 = c("-", "G"), V72 = c("-", "H"), V73 = c("-", "L"), 
    V74 = c("-", "R"), V75 = c("-", "V"), V76 = c("-", "L"), 
    V77 = c("-", "Q"), V78 = c("-", "R"), V79 = c("-", "F"), 
    V80 = c("-", "D"), V81 = c("-", "Q"), V82 = c("-", "R"), 
    V83 = c("-", "S"), V84 = c("-", "K"), V85 = c("-", "Q"), 
    V86 = c("-", "I"), V87 = c("M", "Q"), V88 = c("D", "N"), 
    V89 = c("I", "L"), V90 = c("S", "E"), V91 = c("T", "N"), 
    V92 = c("F", "Y"), V93 = c("S", "R"), V94 = c("P", "L"), 
    V95 = c("-", "V"), V96 = c("-", "E"), V97 = c("-", "F"), 
    V98 = c("-", "K"), V99 = c("-", "S"), V100 = c("P", "K"), 
    V101 = c("V", "P"), V102 = c("A", "N"), V103 = c("G", "T"
    ), V104 = c("-", "L"), V105 = c("-", "L"), V106 = c("-", 
    "L"), V107 = c("-", "P"), V108 = c("-", "H"), V109 = c("-", 
    "H"), V110 = c("-", "A"), V111 = c("-", "D"), V112 = c("-", 
    "A"), V113 = c("-", "D"), V114 = c("-", "F"), V115 = c("V", 
    "L"), V116 = c("E", "L"), V117 = c("-", "V"), V118 = c("-", 
    "V"), V119 = c("-", "L"), V120 = c("-", "N"), V121 = c("-", 
    "G"), V122 = c("-", "R"), V123 = c("-", "A"), V124 = c("-", 
    "L"), V125 = c("-", "L"), V126 = c("-", "T"), V127 = c("I", 
    "L"), V128 = c("T", "V"), V129 = c("A", "N"), V130 = c("F", 
    "P"), V131 = c("-", "-"), V132 = c("-", "-"), V133 = c("-", 
    "-"), V134 = c("-", "-"), V135 = c("-", "-"), V136 = c("-", 
    "-"), V137 = c("-", "-"), V138 = c("-", "-"), V139 = c("-", 
    "-"), V140 = c("-", "-"), V141 = c("-", "-"), V142 = c("-", 
    "-"), V143 = c("-", "-"), V144 = c("V", "D"), V145 = c("-", 
    "-"), V146 = c("-", "-"), V147 = c("-", "-"), V148 = c("-", 
    "-"), V149 = c("-", "-"), V150 = c("-", "-"), V151 = c("-", 
    "-"), V152 = c("-", "-"), V153 = c("-", "-"), V154 = c("-", 
    "-"), V155 = c("-", "-"), V156 = c("-", "-"), V157 = c("-", 
    "-"), V158 = c("-", "-"), V159 = c("-", "-"), V160 = c("-", 
    "-"), V161 = c("-", "-"), V162 = c("-", "-"), V163 = c("-", 
    "-"), V164 = c("-", "-"), V165 = c("-", "-"), V166 = c("-", 
    "-"), V167 = c("-", "-"), V168 = c("-", "-"), V169 = c("-", 
    "-"), V170 = c("-", "-"), V171 = c("-", "-"), V172 = c("-", 
    "-"), V173 = c("-", "-"), V174 = c("-", "-"), V175 = c("-", 
    "-"), V176 = c("-", "-"), V177 = c("-", "-"), V178 = c("-", 
    "G"), V179 = c("-", "-"), V180 = c("-", "-"), V181 = c("L", 
    "R"), V182 = c("N", "D"), V183 = c("-", "S"), V184 = c("-", 
    "Y"), V185 = c("-", "I"), V186 = c("-", "L"), V187 = c("-", 
    "E"), V188 = c("-", "Q"), V189 = c("-", "G"), V190 = c("-", 
    "H"), V191 = c("G", "A"), V192 = c("-", "-"), V193 = c("-", 
    "-"), V194 = c("-", "-"), V195 = c("-", "-"), V196 = c("-", 
    "-"), V197 = c("-", "-"), V198 = c("-", "-"), V199 = c("-", 
    "-"), V200 = c("-", "-"), V201 = c("-", "-"), V202 = c("-", 
    "-"), V203 = c("-", "-"), V204 = c("-", "-"), V205 = c("-", 
    "-"), V206 = c("-", "-"), V207 = c("-", "-"), V208 = c("-", 
    "-"), V209 = c("-", "-"), V210 = c("-", "-"), V211 = c("-", 
    "-"), V212 = c("-", "-"), V213 = c("-", "-"), V214 = c("-", 
    "-"), V215 = c("-", "-"), V216 = c("-", "-"), V217 = c("-", 
    "-"), V218 = c("-", "-"), V219 = c("-", "-"), V220 = c("-", 
    "-"), V221 = c("-", "-"), V222 = c("-", "-"), V223 = c("-", 
    "-"), V224 = c("-", "-"), V225 = c("-", "-"), V226 = c("-", 
    "-"), V227 = c("-", "-"), V228 = c("-", "-"), V229 = c("-", 
    "-"), V230 = c("-", "-"), V231 = c("-", "-"), V232 = c("-", 
    "-"), V233 = c("-", "-"), V234 = c("-", "-"), V235 = c("-", 
    "-"), V236 = c("-", "-"), V237 = c("-", "-"), V238 = c("-", 
    "-"), V239 = c("-", "-"), V240 = c("-", "-"), V241 = c("-", 
    "-"), V242 = c("-", "-"), V243 = c("-", "-"), V244 = c("-", 
    "-"), V245 = c("E", "Q"), V246 = c("K", "K"), V247 = c("I", 
    "I"), V248 = c("P", "P"), V249 = c("A", "A"), V250 = c("-", 
    "G"), V251 = c("-", "T"), V252 = c("-", "I"), V253 = c("-", 
    "F"), V254 = c("-", "F"), V255 = c("-", "L"), V256 = c("-", 
    "V"), V257 = c("-", "N"), V258 = c("-", "P"), V259 = c("-", 
    "N"), V260 = c("-", "D"), V261 = c("-", "N"), V262 = c("-", 
    "E"), V263 = c("-", "N"), V264 = c("-", "L"), V265 = c("-", 
    "R"), V266 = c("-", "I"), V267 = c("-", "-"), V268 = c("-", 
    "-"), V269 = c("-", "-"), V270 = c("-", "-"), V271 = c("-", 
    "-"), V272 = c("-", "-"), V273 = c("-", "-"), V274 = c("-", 
    "-"), V275 = c("-", "-"), V276 = c("-", "-"), V277 = c("-", 
    "-"), V278 = c("-", "-"), V279 = c("-", "-"), V280 = c("-", 
    "-"), V281 = c("-", "-"), V282 = c("-", "-"), V283 = c("-", 
    "-"), V284 = c("-", "-"), V285 = c("-", "-"), V286 = c("-", 
    "-"), V287 = c("-", "-"), V288 = c("-", "-"), V289 = c("-", 
    "-"), V290 = c("-", "-"), V291 = c("I", "I"), V292 = c("V", 
    "K"), V293 = c("L", "I"), V294 = c("S", "A"), V295 = c("K", 
    "T"), V296 = c("D", "P"), V297 = c("-", "I"), V298 = c("-", 
    "N"), V299 = c("-", "N"), V300 = c("-", "P"), V301 = c("-", 
    "H"), V302 = c("-", "R"), V303 = c("-", "F"), V304 = c("L", 
    "Q"), V305 = c("N", "D"), V306 = c("P", "F"), V307 = c("F", 
    "F"), V308 = c("L", "L"), V309 = c("-", "-"), V310 = c("-", 
    "-"), V311 = c("-", "-"), V312 = c("-", "-"), V313 = c("-", 
    "-"), V314 = c("-", "-"), V315 = c("-", "-"), V316 = c("-", 
    "-"), V317 = c("-", "-"), V318 = c("-", "-"), V319 = c("-", 
    "-"), V320 = c("-", "-"), V321 = c("-", "-"), V322 = c("-", 
    "S"), V323 = c("-", "S"), V324 = c("-", "T"), V325 = c("-", 
    "E"), V326 = c("-", "A"), V327 = c("Q", "Q"), V328 = c("-", 
    "Q"), V329 = c("E", "S"), V330 = c("L", "Y"), V331 = c("L", 
    "L"), V332 = c("D", "Q"), V333 = c("Q", "G"), V334 = c("E", 
    "-"), V335 = c("P", "-"), V336 = c("A", "F"), V337 = c("C", 
    "S"), V338 = c("-", "-"), V339 = c("-", "-"), V340 = c("-", 
    "-"), V341 = c("-", "-"), V342 = c("-", "-"), V343 = c("-", 
    "-"), V344 = c("-", "-"), V345 = c("-", "-"), V346 = c("-", 
    "-"), V347 = c("-", "-"), V348 = c("-", "-"), V349 = c("-", 
    "-"), V350 = c("-", "-"), V351 = c("-", "-"), V352 = c("-", 
    "-"), V353 = c("-", "-"), V354 = c("-", "-"), V355 = c("-", 
    "-"), V356 = c("-", "-"), V357 = c("-", "-"), V358 = c("-", 
    "-"), V359 = c("-", "-"), V360 = c("-", "-"), V361 = c("-", 
    "-"), V362 = c("-", "-"), V363 = c("-", "-"), V364 = c("-", 
    "-"), V365 = c("-", "-"), V366 = c("-", "-"), V367 = c("-", 
    "-"), V368 = c("-", "-"), V369 = c("-", "-"), V370 = c("-", 
    "-"), V371 = c("-", "-"), V372 = c("-", "-"), V373 = c("E", 
    "-"), V374 = c("H", "-"), V375 = c("D", "-"), V376 = c("I", 
    "-"), V377 = c("G", "K"), V378 = c("T", "N"), V379 = c("C", 
    "V"), V380 = c("Y", "L"), V381 = c("G", "E"), V382 = c("-", 
    "A"), V383 = c("-", "S"), V384 = c("-", "F"), V385 = c("-", 
    "D"), V386 = c("-", "S"), V387 = c("-", "E"), V388 = c("-", 
    "F"), V389 = c("-", "K"), V390 = c("-", "E"), V391 = c("-", 
    "I"), V392 = c("-", "N"), V393 = c("-", "R"), V394 = c("-", 
    "V"), V395 = c("-", "L"), V396 = c("-", "F"), V397 = c("-", 
    "G"), V398 = c("-", "E"), V399 = c("-", "E"), V400 = c("-", 
    "G"), V401 = c("-", "Q"), V402 = c("-", "Q"), V403 = c("-", 
    "Q"), V404 = c("-", "Q"), V405 = c("-", "G"), V406 = c("D", 
    "E"), V407 = c("A", "E"), V408 = c("-", "S"), V409 = c("-", 
    "Q"), V410 = c("-", "Q"), V411 = c("-", "E"), V412 = c("-", 
    "G"), V413 = c("-", "V"), V414 = c("-", "I"), V415 = c("-", 
    "V"), V416 = c("-", "E"), V417 = c("-", "L"), V418 = c("-", 
    "E"), V419 = c("-", "R"), V420 = c("-", "E"), V421 = c("-", 
    "Q"), V422 = c("-", "I"), V423 = c("-", "R"), V424 = c("-", 
    "E"), V425 = c("-", "L"), V426 = c("-", "I"), V427 = c("-", 
    "K"), V428 = c("-", "H"), V429 = c("-", "A"), V430 = c("-", 
    "-"), V431 = c("-", "-"), V432 = c("-", "-"), V433 = c("-", 
    "-"), V434 = c("-", "-"), V435 = c("-", "-"), V436 = c("-", 
    "-"), V437 = c("-", "-"), V438 = c("-", "-"), V439 = c("-", 
    "-"), V440 = c("-", "-"), V441 = c("-", "-"), V442 = c("-", 
    "-"), V443 = c("-", "-"), V444 = c("-", "-"), V445 = c("-", 
    "-"), V446 = c("-", "-"), V447 = c("-", "-"), V448 = c("-", 
    "-"), V449 = c("-", "-"), V450 = c("-", "-"), V451 = c("-", 
    "-"), V452 = c("-", "-"), V453 = c("-", "-"), V454 = c("-", 
    "-"), V455 = c("-", "-"), V456 = c("-", "-"), V457 = c("-", 
    "-"), V458 = c("-", "-"), V459 = c("-", "-"), V460 = c("-", 
    "-"), V461 = c("-", "-"), V462 = c("-", "-"), V463 = c("-", 
    "-"), V464 = c("-", "-"), V465 = c("-", "-"), V466 = c("-", 
    "-"), V467 = c("-", "-"), V468 = c("-", "-"), V469 = c("-", 
    "-"), V470 = c("-", "-"), V471 = c("P", "K"), V472 = c("C", 
    "S"), V473 = c("-", "S"), V474 = c("-", "S"), V475 = c("-", 
    "R"), V476 = c("-", "R"), V477 = c("-", "S"), V478 = c("I", 
    "L"), V479 = c("S", "S"), V480 = c("-", "-"), V481 = c("-", 
    "-"), V482 = c("-", "-"), V483 = c("-", "-"), V484 = c("-", 
    "-"), V485 = c("-", "-"), V486 = c("-", "-"), V487 = c("-", 
    "-"), V488 = c("-", "-"), V489 = c("-", "-"), V490 = c("-", 
    "-"), V491 = c("-", "-"), V492 = c("-", "-"), V493 = c("-", 
    "-"), V494 = c("-", "-"), V495 = c("-", "-"), V496 = c("-", 
    "-"), V497 = c("-", "-"), V498 = c("-", "-"), V499 = c("-", 
    "-"), V500 = c("-", "-"), V501 = c("-", "-"), V502 = c("-", 
    "-"), V503 = c("-", "-"), V504 = c("G", "S"), V505 = c("P", 
    "Q"), V506 = c("D", "-"), V507 = c("A", "-"), V508 = c("W", 
    "-"), V509 = c("N", "-"), V510 = c("H", "-"), V511 = c("S", 
    "D"), V512 = c("G", "E"), V513 = c("F", "P"), V514 = c("Q", 
    "F"), V515 = c("C", "N"), V516 = c("-", "L"), V517 = c("-", 
    "R"), V518 = c("-", "N"), V519 = c("-", "R"), V520 = c("Q", 
    "K"), V521 = c("-", "P"), V522 = c("-", "-"), V523 = c("-", 
    "-"), V524 = c("-", "-"), V525 = c("-", "-"), V526 = c("-", 
    "-"), V527 = c("-", "-"), V528 = c("-", "-"), V529 = c("-", 
    "-"), V530 = c("V", "I"), V531 = c("C", "Y"), V532 = c("-", 
    "-"), V533 = c("-", "-"), V534 = c("-", "S"), V535 = c("-", 
    "N"), V536 = c("-", "-"), V537 = c("-", "-"), V538 = c("-", 
    "-"), V539 = c("-", "-"), V540 = c("-", "-"), V541 = c("-", 
    "-"), V542 = c("-", "-"), V543 = c("-", "-"), V544 = c("-", 
    "-"), V545 = c("-", "-"), V546 = c("-", "-"), V547 = c("-", 
    "-"), V548 = c("-", "-"), V549 = c("-", "-"), V550 = c("-", 
    "-"), V551 = c("-", "-"), V552 = c("-", "-"), V553 = c("G", 
    "K"), V554 = c("L", "F"), V555 = c("V", "-"), V556 = c("K", 
    "G"), V557 = c("S", "R"), V558 = c("W", "W"), V559 = c("N", 
    "Y"), V560 = c("L", "E"), V561 = c("-", "I"), V562 = c("-", 
    "T"), V563 = c("-", "P"), V564 = c("-", "E"), V565 = c("-", 
    "K"), V566 = c("-", "N"), V567 = c("-", "P"), V568 = c("-", 
    "-"), V569 = c("-", "-"), V570 = c("-", "-"), V571 = c("-", 
    "-"), V572 = c("-", "-"), V573 = c("-", "-"), V574 = c("-", 
    "-"), V575 = c("-", "-"), V576 = c("-", "-"), V577 = c("-", 
    "-"), V578 = c("-", "-"), V579 = c("-", "-"), V580 = c("V", 
    "Q"), V581 = c("G", "-"), V582 = c("D", "-"), V583 = c("F", 
    "-"), V584 = c("V", "-"), V585 = c("V", "-"), V586 = c("Y", 
    "-"), V587 = c("R", "-"), V588 = c("Q", "-"), V589 = c("E", 
    "-"), V590 = c("-", "-"), V591 = c("-", "-"), V592 = c("-", 
    "-"), V593 = c("-", "-"), V594 = c("-", "-"), V595 = c("-", 
    "-"), V596 = c("-", "-"), V597 = c("-", "-"), V598 = c("-", 
    "-"), V599 = c("-", "-"), V600 = c("-", "-")), class = "data.frame", row.names = c("UniRef90_A0A1V4QIJ3", 
"UniRef90_A0A0L9V5F8"))
$\endgroup$
3
$\begingroup$

Update

It seems that part of the desired behavior is to write the sequences in the alignment separately. I am not sure that I understand why it is necessary to write them separately. Generally it is inadvisable to write many small files rather than one big file, based purely on issues of performance; you are likely to hit the I/O bound on performance very quickly, and that will be much slower.

One solution that has been suggested previously for this kind of problem is to write the whole alignment to a single file, and then post-process that filed in a second step to extract the sequences of interest. This could be as trivial as a samtools faidx call from the shell, or you could work with it in seqinr as suggested in the previous answer.

Of course, it seems like it would be simpler for most purposes to just have the big alignment file and use e.g. samtools indexing / faidx / biopython / whatever to extract the sequences of interest. But it seems that that is specifically not desired in this case.

Original answer

I would strongly suggest using the seqinr package, which is optimized for solving exactly these problems. See e.g. this similar question.

You will be able to read in an alignment (or whatever), cast it to as.matrix.alignment, do whatever your analysis is, then cast it back to as.alignment, which can then be passed to write.fasta to write the files.

Whenever possible it is preferable to work with such libraries, which handle these utility functions well, fast, and easy.

$\endgroup$
6
  • $\begingroup$ Agree, but because I wish to write a lot smaller fasta based on the original one, I was wondering if there might be faster way. $\endgroup$
    – David
    Dec 25 '20 at 13:13
  • $\begingroup$ @DavidS I see, it had not occurred to me that anyone would want many small "aligned" fasta files. I have updated the answer with one suggestion, but I think that this is just going to be slow based on file I/O no matter what if you want to write large numbers of files. Possibly someone has something clever though. $\endgroup$ Dec 27 '20 at 0:06
  • $\begingroup$ So I start from a big MSA, but I wish to do some sort of "sub-sampling" of it to create a smaller files which will still holds valuable information to use further in a deep learning model. $\endgroup$
    – David
    Dec 27 '20 at 6:50
  • $\begingroup$ @DavidS Have you considered directly subsampling from the alignment as an in-memory data structure? I think that will definitely be faster than writing files and then reading them in again for the model. If the issue is access to the data structure across different processes, you could write the full alignment as an RData file using save(), and then load() it into each process. (assuming the DL model is in R; if it's in e.g. Python, I am sure that analogous alignment handling is possible there, but that's a different question.) $\endgroup$ Dec 27 '20 at 19:19
  • $\begingroup$ I considered it, but the problem is that by just sampling from the MSA your might sample species that are too similar to the reference. So I apply some logic to select k samples which are farther apart by some criteria. $\endgroup$
    – David
    Dec 31 '20 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.