Extract nanopore read ID & start times from fastq file

Question

I have a fastq file from minION (albacore) that contains information on the read ID and the start time of the read. I want to extract these two bits of information into a single csv file.

I've been trying to figure out a grep/awk/sed solution, but without success.

E.g.

@93a12f52-95e5-40c7-8c3e-70bf94ed0720 runid=17838b1d08f30a031bf60afabb146a8b0fba7486 read=12217 ch=492 start_time=2017-07-04T06:42:43Z
CTATTGTCCCCTGCCGCTGCCCCTCCTGCTACGCCCCACTGCCTCACCAGCCGTTACGGTCGCCCCCCATCGCATGCCTTTACACACACACTTCTTTACACATGCTATCTTCCC
+
""*)&$-.,-(#"&'$%''+16#"$##&)%%/"+*(*(&#"&'%1"+)#)""%$$#&&'1%"'8>MJ<#'&%'.2'.$(&#'()'&&%'$('%"%%%..$#"&"#+&,*$%"#"
@ff37e422-a25f-404c-8314-ef1733f9c30c runid=17838b1d08f30a031bf60afabb146a8b0fba7486 read=8432 ch=200 start_time=2017-07-04T06:56:41Z
CGATGGCCGTATGCTTTTGTTATGAAGCGAAAAGCTGCTCGCTTCTCTAGATAATAATGATGTGGCGAAAACGCTATGCGATTCGTTGACATACATGATGGCGGATTTATCTACCACTTTGTGGCATGCTTTTCTCGCCAGATAATGGAATGTTTCTCTGGCGGTAATGGATAGTATCAAATCTCACTAGCCCATTCTATAAGCGCATCCGCATGCACTAGTTCTTGATTCGATCGTCCTCTAGCATGTTCGAAGAAGATAGCATTCACTATCATCATCGCCTCAGGTAAGTTTATTCGGTTGGGGCGTGTGAAGGCAAACACCTTGTTGTCCAGTAAGTTTTCAGTTACTATAACTTAAAGTCGCACATGAATCTAGTCTCCTATTCCCCACCCATGATCCCACTCACACATTTCTACAGAGATGTGGTTAGAAATTTTCATATTAGGTCAGCTTTGACTCAATAAGACATAATTCTTCACTGAATGACTTTTTAAGAACCACCAGGACCAGAGAGAACCAAGAGAGTGGTACCTCTTAAAACACAATAAAGTGATTCAGCCTTAGCCATTGGATTCTGGAGGACCTTGAACCATGTGGGAAGCAGCTCAGGGTGGCCATGTACTATACTGGCGGGTAAGCTTCTGGAGTGCTAGGTTCTTTTTGTCTTTTCTTAAGCATTGCCGCCAGTTGATTGGGTTTTGAACATAAAATAATGCGCCACCAGCAATTCCAGATTTGTTCCTACGGGATAGATTTGTTCAGTTCTAGCATTATGCTTCACTAACCAGATGCGGGCCCTAAGTCCTTCACTTGGAATATTGGATTGGATCATGAGAATATTCTGTCTGAAGCTCGTCATTAATTTTGTTACAAAATAGAGCTTTTTGACTGGAAGTACCACCATACGTGTTCTCAAACTTCAGCATTTTTAGAACTTCCCACGGCATCTTGACCCTTTTCACAGCATGGATAGTCAGGCAGCAGTGAACTTTGTGACTCTTTAATGCCTTCACTTTTCTCTCAGTTTCCCCGCCTTGCGTTATCTTTACTCGTCTTGGGACTTTTATCCCAATGCCAGCCTTCTACCCTGAGACCTCAGTGGGTCATCATCCCAGCCCGGGACATCTCATCCCATCATTTATGGGCTGTTGTGTTTTTTTCAAAACCTAGCCCTCTCAGGAGGAGGAGGAGTGGGAGTCAGTTCAGTGAGGAGGATTAGGATGATCTGAAATGTAAGCACATATAAGCGAAGCACTTATTTTGGGTTGGGTCCTCACGGTGGACATAAGATCGCCTTATGTGTTTAGTAAGCCATTCCTAGCTCTCAATGGCGTGATTACATAGAAGCGTGAGGGATCAGTCCTATGGAAGACTAGGAAGTAAATGAACAAAATATATTAACCATAGAAGTCTCATGGGTCGCTGTAGCCAAAAGATTAACACTTTTGACTACATTGTGGTTTTAGGCATTGAAACAAAAACTTTGAGTCTCCTAAACAAATGAATGGAAAATAGTAGCGAACTTCGATTCCTAACATTAAATCTAGAAATAGCAAGTTAGTTTAAAGACTTTATTTAGCTTTGCTTGCTATAATGAAAACCTTGCCTCCCGGTCGGGGCCATTGTGCCTGAAGCTAGCTTATTGTCTCCTCGAGCTCCCAGCTTCAGCAACTCCTTTTGAAGCGTTTGTCTCAGCTTGGATCTTCAGCAGCTCTTGGTGGCTCTTTTGAGCTAGCTCCTCTGAGATCTTGTATTTGGTAGGTCGCTTAGTCATAGTACTTTTCTTTTAACACCCTTCAGCTCTACGATTACATTTGGTTTTGTGGATATCATAATGATTGATGTGAAGATACATTGTACATGTG
+
"#""#"##"'"&%&?JP7+80)'&*+&&(,.>3&*(#%(')*&'*#(-$)"&,63;?844&&#$'3+.>@9;-/259...&:"$"/*"#*#%(.&%)+,/76/+'4B93:;70.1)'1:)*#&()(+'03589/++&)'$-:;@,B5(9+.JAU;7-+&,1212.+329NPSDHRHH6),&)%)+).-*$,<1(.+&$$2&,-0+.&'//"4(%"+#,/)<56%*%*$<4*/;DI+6>((-%*).+--;2-'04-062<@-<>6CP)#.0/+,#(*2+#"$$-2'5(%%&'5+)$$%8+0;>.@9*(,((,*-2/49G9/1/3./.0CQ93E+/-D-'67J*)($-2+'*/&*)+%&'092<C9/5()$&%(.,+-'**)'2/.(//6<59-5-)"**40:A4+834)-(*+;K0-,&)+%)(#-&+&&))+$>9.6<?A;3'/85$/--..)/3DP99HJ9."2;I?@C/5%.4<0+0+&$-05C46208AFQE80"%356+02*9I73//(-*/9706192*/)%)(&'($1749C*KKA*/#"/*)5($'#+7@=$%)/&+%##*&&;.),#(4.,2/'-G03%""#,(2)./,%%1;7#+%$'(03'5)+&.%1)&(%'#*,$3#''-3,),$"),'+,,*$%'%$33ILADPPY@JW\O2-8'9[Y@--*.,""$%$"%#"65',0/4(6.*43B1.#&)+),(6BI729=9&#$.)#,(,(/-286D&),'C;P=)'-'#$)*91*$%(.3LH31,($/(+)'$$*#+'((('*)00:7C1E24(%+$&-,$'++)<)'**"&''"(*(.14++$5&*,%?.F1<10"&%$,)&&*(&$%$/.""+3"#"#"*%5*->87FQ>AKY<9855,&(*@>HAZRQ)'%$"--&/#%.+,/A9F8-1'+"6--*34P-9;=997;7<SUU8((-+.:784/%).&)#*'(*/%7H(1JI-+&+)&9.,/N)"%"&))22/&,)%)'+*/17:8$''(*.*..(/.(%+*&22(+)7GM99A@9)0*1%/470((&((%(937'15084#,($"&'%"'4.,+5<7($*0+*291$*&$%%('+/)'.1-(-.7=K/'%"'*)03<:/<=1(>.((-9-'+>&&&%';,)69V<'1:>FNXA+01:7*#"$'$.%&4BQT7/&03++)'&'*3E.+0(*6..*<.)9*)))'%#(*.&&'"*-IU=&,-))-.,->'G--6..90067%(,0/*%9)&#'""#$,5H/$/#-/&2(.0.+;>9--7.3@+2(/4.*0&,-&'/*%0-**-#,/2&-$*)'+#"*)-*##&$3):C:D6H+',./2?<0>)%#&&3::9;8$#%#('$#$',,,*-346@=.%&&(&%('*+&*&1)'&-..#'+2*)/6499=-2*,0,82,$)($:2%*##%$,.5=-5/((#)2./('($@=4,)%1+176:AHQT,(,)0(()@2'+(18G?($)(*'.)&&+*12SD7,18I2)()*+C:01ETZIPDUOB+.55K7B:A5)*/('')/-1=7E<IGR=43@A68;K/:8.'--)BGEE5-0/5+,2''DXBLW7)#DHTC5AL82+--,&#);+).,'3)/.336/&'&(,+8E346-/%1-)1*$+),@-&-%'$+)5.&&%&#(,2&03&/((,(-5%%)?7I4*').+/7+&***845(*/.)(.0))<?YK3=GHH2&*)%$%#%KLY<9&&+,8;/2*./;+))-)/03$&*$+0'36<J+5/+&+(HGFFM57,59)-,0)/:**"%$&#$'&$%#)*%33VX=@FOL00(''4IGIYOG0=1,,#(*$)-0CFM?8+31E8)(+)(%'"'/""%+3/,1;+7-*''&-%98.$$/'&)&+&(&&"

Should produce

93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Answer 1

awk '{if(NR%4==1) print $1, $5}' file.fastq | sed -e "s/ start_time=/, /" -e "s/^@//"

The awk command gets the first of every 4 lines, printing the first and fifth "word". sed is then used to strip the initial @ and replace start_time= with ,. The output on your example file is:

93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Answer 2

Since the string start_time will only appear on the header line, or else you don't have a valid fastq file, you can simply do:

$ perl -ne '/^@(\S+).*start_time=(.*)/ && print "$1, $2\n"' file.fastq 
93a12f52-95e5-40c7-8c3e-70bf94ed0720,2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c,2017-07-04T06:56:41Z

Alternatively, since you mentioned awk and sed:

$ awk -v OFS=", " '/start_time/{print $1,$NF}' file.fastq | sed 's/start_time=//'
@93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
@ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Or, doing the whole thing in awk:

$ awk 'sub(/start_time=/,""){print $1", "$NF}' file.fastq 
@93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
@ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

And if the @ annoy you:

$ awk 'sub(/^@/,"") && sub(/ .*start_time=/,", ")' file.fastq 
93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

And in sed:

$ sed -n 's/^@\([^ ]*\).*start_time=\(.*\)/\1, \2/p' file.fastq 
93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Or, if your sed supports it:

$ sed -En 's/^@(\S+).*start_time=(.*)/\1, \2/p' file.fastq 
93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Finally, since grep can't do replacements, to use it you would have to do something like:

$ grep -oP '^@\K\S+|start_time=\K.*' file.fastq | paste - - 
93a12f52-95e5-40c7-8c3e-70bf94ed0720    2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c    2017-07-04T06:56:41Z

And to get the commas:

$ grep -oP '^@\K\S+|start_time=\K.*' file.fastq | paste - - | sed 's/\t/, /'
93a12f52-95e5-40c7-8c3e-70bf94ed0720, 2017-07-04T06:42:43Z
ff37e422-a25f-404c-8314-ef1733f9c30c, 2017-07-04T06:56:41Z

Answer 3

Albacore produces a sequencing_summary.txt file (actually TSV, not CSV) in the same directory as the workspace folder that might have the data that you want in it. Here are the fields present in that file:

• filename
• read_id
• run_id
• channel
• start_time
• duration
• num_events
• template_start
• num_events_template
• template_duration
• num_called_template
• sequence_length_template
• mean_qscore_template
• strand_score_template
• complement_start
• num_events_complement
• complement_duration
• num_called_complement
• sequence_length_complement
• mean_qscore_complement
• strand_score_complement
• sequence_length_2d
• mean_qscore_2d