6
$\begingroup$

FCS is a patented data format used for storing flow cytometry data. The most recent version is FCS3.1. There is some documentation on the format, but there is no information on how to read these files. There are some R packages and a MATLAB code to read an FCS file, but I am looking for standard libraries developed either by the FCS consortium or any other group. I also wish to know if FCS is a subset of an existing standard data format that can be read by a standard library using any programming language.

Finally, I would want to convert these files to an easily readable format like HDF5.

$\endgroup$
  • $\begingroup$ I find strange you ask for standard libraries but you exclude the existing standard tools. $\endgroup$ – llrs Jun 30 '17 at 14:19
  • $\begingroup$ @Llopis the OP isn't rejecting standard tools, on the contrary, he is looking for standard libraries. $\endgroup$ – terdon Jun 30 '17 at 14:28
  • $\begingroup$ @terdon I am not sure how standard libraries are decided but a R/Bioconductor package or a MATLAB one seems to me quite standard (Except if the FCS consortium provides one of course). $\endgroup$ – llrs Jun 30 '17 at 14:32
  • $\begingroup$ @Llopis By standard, I mean is there an OS/language independent interface that allows reading of these files. More importantly, I want to know how FCS files can be read by any general language. I am expecting something like a pseudocode, for this purpose. $\endgroup$ – WYSIWYG Jul 1 '17 at 6:28
  • 2
    $\begingroup$ I’m still not really sure what you want: the documentation you link to is as close to a general pseudocode as you will get: it contains a detailed, concise specification of the format. That’s the same information that a pseudocode would convey, and it’s the conventional way in which data formats are described (pseudocode is virtually never used for this). That, in addition to the existence of libraries for common analysis languages (R, Python, Matlab), should be sufficient to work with the format. $\endgroup$ – Konrad Rudolph Jul 1 '17 at 12:13
4
$\begingroup$

A few years ago, I wrote a python script to convert FCS files into tab-separated format. It was far from handling all the possibilities that the format description offers, but at least it worked for some of the files produced on one of our machine: http://www.igh.cnrs.fr/equip/Seitz/en_equipe-programmes.html

The format documentation I found enabled decoding (see section 3 of the pdf you mention), but it requires reading data in binary mode.

The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So a first phase is to parse this header, following the description given in the documentation of the format. The information extracted from the header tells where to find the data and how it is encoded, still according to rules described in the format documentation.

In case this may be useful, and for the record, here is the code from the above-mentioned script (after stripping comments, some of which are merely copied from the format documentation, and adding a few ones):

#!/usr/bin/env python
"""This script tries to read FCS flow cytometry data.
Format parsing inspired by information found here:
http://isac-net.org/Resources-for-Cytometrists/Data-Standards/Data-File-Standards/Flow-Cytometry-Data-File-Format-Standards.aspx
"""

import re
# To decode binary-encoded data
import struct
import sys

class Parameter(object):
    """This object represents one of the parameter types that are present in a DATA segment of a FCS file."""
    __slots__ = ("p_name", "p_bits", "p_range", "p_ampl", "parser")
    def __init__(self, p_name, p_bits, p_range, p_ampl):
        self.p_name = p_name
        self.p_bits = p_bits
        self.p_range = p_range
        self.p_ampl = p_ampl
        # Function for parsing a value of the parameter in the data segment
        self.parser = None

##############################################
# Here starts the parsing of the header part #
# which tells where the other parts are.     #
##############################################

f = open(sys.argv[1], "rb")
# The format name is encoded in 6 letters
# An ASCII letter is coded with one octet
file_format = "".join([f.read(1) for __ in range(6)])
sys.stdout.write("Format: %s\n" % file_format)
# The format descriptions reserves 4 octets that we skip
skip = f.read(4)
# 8 octet chunks encode the start and end positions
# of different parts of the data
text_start = int(f.read(8).strip(" "))
text_end = int(f.read(8).strip(" "))
data_start = int(f.read(8).strip(" "))
data_end = int(f.read(8).strip(" "))
analysis_start = int(f.read(8).strip(" "))
analysis_end = int(f.read(8).strip(" "))

if (analysis_start and analysis_end):
    sys.stderr.write("Cannot deal with ANALYSIS segment of an FCS file.\n")

####################################################
# Here starts the parsing of the "TEXT" portion    #
# which describes how the data proper is organized #
####################################################
f.seek(text_start)
# The first character in the primary TEXT segment is the ASCII delimiter character.
sep = f.read(1)
if sep not in ["_", "@"]:
    alt_sep = "_@_"
elif sep not in ["_", "|"]:
    alt_sep = "_|_"
else:
    assert sep not in ["+", "|"]
    alt_sep = "+|+"
text_segment = f.read(text_end - text_start)

fields = text_segment.split(sep)

info = {}

i = 0
while i < len(fields) - 1:
    key = fields[i]
    i += 1
    val = fields[i]
    i += 1
    # Keywords are case insensitive, they may be written in a file in lower case, upper case, or a
    # mixture of the two. However, an FCS file reader must ignore keyword case. A keyword value may
    # be in lower case, upper case or a mixture of the two. Keyword values are case sensitive.
    info[key.upper()] = val
print "%s events were detected." % info["$TOT"]
print "Each event is characterized by %s parameters" % info["$PAR"]

if info["$NEXTDATA"] != "0":
    sys.stderr.write("Some other data exist in the file but hasn't been parsed.\n")

# L - List mode. For each event, the value of each parameter is stored in the order in which the
# parameters are described. The number of bits reserved for parameter 1 is described using the
# $P1B keyword. There can be only one set of list mode data per data set. The $DATATYPE
# keyword describes the data format. This is the most versatile mode for the storage of flow
# cytometry data because mode C and mode U data can be created from mode L data.
assert info["$MODE"] == "L"

parameters = []

# indices of the parameters
p_indices = range(1, int(info["$PAR"]) + 1)
for i in p_indices:
    p_name = info["$P%dN" % i]
    p_bits =  info["$P%dB" % i]
    p_range = info["$P%dR" % i]
    p_ampl =  info["$P%dE" % i]
    parameters.append(Parameter(p_name, p_bits, p_range, p_ampl))

sys.stdout.write("The parameters are:\n%s\n" % "\t".join([par.p_name for par in parameters]))

# How are 32 bit words organized
if info["$BYTEORD"] == "4,3,2,1":
    endianness = ">"
else:
    endianness = "<"
    assert info["$BYTEORD"] == "1,2,3,4"

    # I stripped a long comment which is just a copy of the documentation
# Type of data:
if info["$DATATYPE"] == "I":
    for par in parameters:
        nb_bits = int(par.p_bits)
        assert nb_bits % 8 == 0
        nb_bytes = nb_bits / 8
        # Determine format string for unpacking (see https://docs.python.org/2/library/struct.html)
        if nb_bytes == 1:
            c_type = "B" # unsigned char
        elif nb_bytes == 2:
            c_type = "H" # unsigned short
        elif nb_bytes == 4:
            c_type = "L" # unsigned long
        elif nb_bytes == 8:
            c_type = "Q" # unsigned long long
        else:
            raise ValueError, "Number of bytes (%d) not valid for an integer (see https://docs.python.org/2/library/struct.html#byte-order-size-and-alignment)." % nb_bytes
        fmt = "%s%s" % (endianness, c_type)
        p_range = int(par.p_range)
        def parser(data):
            value = struct.unpack(fmt, data.read(nb_bytes))[0]
            try:
                assert value < p_range
            except AssertionError:
                print "Value %s higher than %d" % (str(value), p_range)
            return value
        par.parser = parser
    pass
else:
    raise NotImplementedError, "Only the parsing of integer value has been implemented so far."


out_file = open(sys.argv[2], "w")
out_file.write("#amplification_types\t" + "\t".join([par.p_ampl for par in parameters]) + "\n")
out_file.write("parameters\t" + "\t".join([par.p_name for par in parameters]) + "\n")
i = 1
##############################################
# Here starts the parsing of the data proper #
##############################################
f.seek(data_start)
while f.tell() < data_end:
    values = []
    for par in parameters:
        values.append(par.parser(f))
    out_file.write("%d\t" % i + "\t".join(map(str, values)) + "\n")
    i += 1
out_file.close()
f.close()
$\endgroup$
  • $\begingroup$ The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So I begin by parsing this header, following the description given in the documentation of the format. The information extracted from the header tells me where to find the data and how it is encoded, still according to rules described in the format documentation. I'm not good at writing pseudocode, but I can try to add comments in the python code. $\endgroup$ – bli Jul 1 '17 at 8:24
4
$\begingroup$

I maintain several Python libraries for interacting with flow cytometry data (FCS files). The most recent of which, FlowKit, provides functionality to export the event data as a NumPy array, Pandas DataFrame, or CSV text. Using the h5py library, it's trivial to export as an HDF5 file

$\endgroup$
3
$\begingroup$

R/Bioconductor has a number of different flow cytometry processing packages. One place to start for looking at cytometry data from a high level would be openCyto (or its vignette), which is a large set of tools for basic extraction and analysis of FCS files.

I have looked in the past at the FCS files as an R structure using flowCore. Loading a single FCS file is fairly straightforward and follows a familiar R pattern:

file.name <- "/dir/file.fcs"
x <- read.FCS(file.name, transformation=FALSE)
summary(x)

Asking about converting FCS files into an "easily readable format like HDF5" doesn't seem like the right question. HDF5 is a container format and shares a lot of similarity to file systems. I've found it best to keep FCS files as they are, as it is a compact, standardised binary format.

$\endgroup$
  • $\begingroup$ HDF5 is not a major concern (I just like it). I am still curious about how something like read.FCS works. I believe if FCS is a standard format then there should be standard (non-proprietary) libraries to read/write FCS files. As of now I'm forced to use R or Matlab (and still there are no standard libraries here too. I guess there are just different packages/programs made by different groups/individuals; no single standard). What if I want to analyse a FCS file on C, Fortran or Python (etc)? $\endgroup$ – WYSIWYG Jul 1 '17 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.