13
$\begingroup$

I want some templates of different file formats that I can use to test my scripts and identify possible bugs in my code.

For example, consider nucleotide FASTA, a simple but often abused format, I would want templates to capture regular and irregular formats, like I have seen all of these:

1) Single line sequence

>1
ATG

2) Multi-line sequence

>1
AT
G

3) Upper and lower case letters in sequence

>1
Atg

4) Ns and Xs (and possibly other letters) in sequence

>1
ANnxX

5) Unusual headers (sometimes non-ASCI characters, need to consider the encoding)

>ATG >汉字
ATG

6) Whitespace between records

>1
ATG

>2
ATG

7) Duplicated headers

>1
ATG
>1
ATC

8) Empty headers or sequences (valid FASTA?)

>
>

9) No new line '\n' character on last line (can mess-up file concatenation)

>1
A# < no new-line here

10) Different newline characters depending on the OS

>1
A# \r\n vs \n

etc.

There should be separate templates for nucleotide and protein FASTA, and separate ones for aligned FASTA.

It would ideally include other aspects too, like different compression formats (such as .gz, .bzip2) and different file extensions (such as .fa, .fasta).

I have never seen resources that provide templates covering these, but I think it would be useful. Of course I could build my own templates but it would take time to capture all the likely variations of the formats, particularly for more complex file formats.

Note, I am not just interested in FASTA format, it was an example.

Also note, I know about tools (such as BioPython) that should handle many formats well, but they may also have bugs. Anyway, in practice sometimes I end up parsing files myself directly because I don't want the overhead or dependency of an external package.

EDIT: Please don't answer this question to say you don't know of any such resources, me neither, hence the question. bli's helpful answer shows that there is at least one test suite that could be used as a starting point. I know that it is normally easy to look up the specification of any particular file format.

Improve this question
$\endgroup$
5
  • $\begingroup$ Actually, the FASTA format definition is very simple. It only has two constraints and the second is often ignored: i) header lines must start with > and can contain anything they like except a \n. ii) sequence lines should be 60 characters per line. Nothing else is relevant, not the number of sequence lines, not the characters in the header, not the extension (which is usually irrelevant outside the Windows world anyway), there is no constraint on what characters a sequence can have (hence no difference between protein and nucleotide fasta specifications). $\endgroup$
    – terdon ♦
    Jun 2, 2017 at 14:21
  • 1
    $\begingroup$ Sure, the specification is simple, but as you say in practice it's not always followed and when you parse a file all these variations becomes relevant. I have had many tools trip up over strange FASTA headers or sequences and clearly it's important for the tools to distinguish nucleotides from amino acids. You may glob files with just 1 particular suffix for example. $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 14:26
  • $\begingroup$ If the tools choke on that, it's the tools that are ignoring the standard. The FASTA standard is one of the very few that is actually clear and easy to understand. It is simply a very free format. And yes, of course you can glob on extensions. I, for example, tend to name my protein files .pep and nt .fa but there is no standard for this. All your tools need to do is i) take the entire line after a > as a sequence name and ii) everything else as the sequence. That's all, and that's all it take to be fasta. $\endgroup$
    – terdon ♦
    Jun 2, 2017 at 14:33
  • 1
    $\begingroup$ Sure, the tools may ignore the standard, and that's why I think it's good to have varied templates to test the tools. And it's not always simple, e.g. if you use FASTA headers to name subsequent files, special characters can be a nuisance. Anyway, as I said FASTA was just an example, the logic extends for other more complex formats $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 14:39
  • $\begingroup$ You mention "I am not just interested in FASTA format, it was an example", but bear in mind that It's best to keep questions as specific as possible. $\endgroup$
    – gringer ♦
    Jun 2, 2017 at 20:27

6 Answers 6

Reset to default
2
$\begingroup$

BioJulia maintains BioFmtSpecimens. From the README:

Bioinformatics is rife with formats and parsers for those formats. These parsers don't always agree on the definitions of these formats, since many lack any sort of formal standard.

This repository aims to consolidate a collection of format specimens to create a unified set of data with which to test software against. Testing against the same cases is a first step towards agreeing on the details and edge cases of a format.

Improve this answer
$\endgroup$
7
$\begingroup$

You mention Biopython, which contains tests: https://github.com/biopython/biopython/tree/master/Tests.

Some of the tests consist in reading files present in the folders listed in the above link. These files could be a starting point for a database of test files. Whenever one comes across a test case not covered with these files, one could construct a new test file and contribute it to Biopython, along with a test, or at least file an issue: https://github.com/biopython/biopython/issues.

That would be a way to contribute to Biopython while constituting a database of test files.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, I like this idea; I wonder if BioPerl and other projects also contain similar test suites that could be used $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 17:00
  • 1
    $\begingroup$ A "me too" here: I've sometimes used the example or test data for BioPython as test data for other programs. You at least have the assurance of it being valid. You might find some similar useful data with the Galaxy tools. $\endgroup$
    – agapow
    Jul 25, 2017 at 9:42
5
$\begingroup$

Not that I am aware. It is best to go with format specifications when coding.

Also it may be good to look at the example files that come together with various tools performing file conversions and handling. E.g.

Improve this answer
$\endgroup$
2
$\begingroup$

As far as I know, there is no single repository that collects all of the common data formats used in bioinformatics. Typically, you have to go to the source to find the specifications for each format. There are a few places that collect descriptions of file formats, though:

  • IGV File Formats, coveres all of the formats usable in the Broad Institute's Integrative Genomics Viewer software (which is a lot).
  • NCI File Formats, mostly formats used by TGCA (including MAF and VCF).
  • UCSC Genomics, covers BED, MAF, and a few others.
  • GenePattern, covers many of the file formats related to microarray data.
  • GSEA, Broad Gene Set Enrichment Analysis documentation.
Improve this answer
$\endgroup$
1
$\begingroup$

No. At least none that I've heard of and I doubt there ever will be. There is no central repository for formats and each tool, community, field etc have their own.

The best you can do is look up the official standard for each format and hope they include examples. Having a truly comprehensive collection of all possible variations is basically impossible for horrible complex formats like VCF. Just consider the simple fact that it allows user-defined INFO fields with arbitrary contents.

So the best you can do is make sure your scripts conform with the standard and hope your input does as well.

Improve this answer
$\endgroup$
5
  • $\begingroup$ There could be a central repository though with templates for the most common formats. I don't see why not? It can't be completely comprehensive of course. To me this seems like a good idea... $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 14:51
  • 1
    $\begingroup$ @Chris_Rands as a general rule, there are sadly very few centralized anything in biology and that's a general issue with the field. That said, I admit I don't really see the use for this. What you need is the standard. Multiple examples of the standard being applied in different ways doesn't seem all that helpful to me; at least not if the standard is well defined (which is not always the case). Come to think of it, I can't think of any such repository of format examples in general, not only in biology. $\endgroup$
    – terdon ♦
    Jun 2, 2017 at 14:56
  • 1
    $\begingroup$ Thanks, well I agree about the lack of standardisation/centerlisation. The main use for me would be test cases/unit testing. Knowing the specification is one thing, but spotting all the edge cases can be tricky without testing $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 15:02
  • 1
    $\begingroup$ @Chris_Rands s/tricky/impossible :) I've now been working in industry after a decade and a half in academia and you wouldn't believe the crap we've been sent as "vcf files". There's no way to cover all the edge cases, some people will give you an excel file with a .vcf extension and expect it to work :( $\endgroup$
    – terdon ♦
    Jun 2, 2017 at 15:11
  • 1
    $\begingroup$ I think a github repo with examples of edge cases for each format is a great idea - everytime any of us came across a new way some one had done something werid with a format we could issue a PR on repo. $\endgroup$
    – Ian Sudbery
    Oct 16, 2018 at 12:55
0
$\begingroup$

These may not be exactly what your are looking for, but they do contain a wide range of formats with examples. If you want more, you might try searching other major databases hosting other types of into e.g. Uniprot, PDB, NCBI.

https://genome.ucsc.edu/FAQ/FAQformat.html

http://www.ensembl.org/info/website/upload/bed.html

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, I'm familiar with UCSC and Ensembl and their format descriptions but that's not really what I meant by templates $\endgroup$
    – Chris_Rands
    Jun 2, 2017 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.