I implemented pandas "Intervals" and ... it should be a few lines, clearly there are limitations. For non-overlapping data it is very cool however. It will work for overlapping data, BUT if the data you are using as the interval data is overlapping, it falls over. It could work if an independent (non-overlapping) interval was constructed.
Anyway, the point of the excercise was to keep it all in pandas, "start_stop" should not be needed. This can't be done via Intervals, you've got to drop out, and that will slow it down and harder to parse.
The query contig df3.iloc[1,]
, is still a bug (hence not printed), should be [x,] where x is the value_counts hit, but this is just a pilot.
The script would be fine if it the df was wrangled using groupby
rather than value_counts
because value_counts
trashes important data, which is what I'm trying to recover via df3.iloc[1,]
. I like pandas elegant solutions, but you know another day another dollar.
Note: I changed 200 to 199 otherwise there's too much overlap.
Overlap Output (not labeled very well)
start stop
index
(1.0, 100.0] 0 0
(100.0, 199.0] 1 0
(150.0, 500.0] 1 1
(900.0, 950.0] 0 1
import pandas as pd
def tableit (dict):
df = pd.DataFrame(dict)
table = pd.pivot_table(df, values=['start', 'stop'], columns=['chromosome'], index=['feature', 'misc', 'misc2'], fill_value = '-')
return table
def start_stop(point, table2, bin):
df3 = pd.cut(table2[point,'chr1'], bin[0]).value_counts()
df3 = df3.to_frame(point).reset_index()
for x in bin:
if not (x == bin[0]):
df3_tmp = pd.cut(table2[point,'chr1'], x).value_counts()
df3_tmp = df3_tmp.to_frame(point).reset_index()
df3 = pd.concat([df3, df3_tmp]
else:
continue
df3.set_index('index', inplace=True)
return df3, df3.iloc[1,] # [1,] is a bug and the query locus needs correctly reading from df3_tmp
def main():
dict1 = {'chromosome':['chr1','chr1','chr1','chr1','chr2'], 'start':[1, 100, 150, 900, 1], 'stop':[100, 199, 500, 950, 100], 'feature':['feature1', 'feature2', 'feature3', 'feature4', 'feature4'], 'misc':[0, 0, 0, 0, 0], 'misc2':['+','+','-','+','+']}
dict2 = {'chromosome':['chr1','chr1'], 'start':[155, 800], 'stop':[200, 901], 'feature':['feature5', 'feature6'], 'misc':[0, 0], 'misc2':['-','+']}
table1 = tableit(dict1)
table2 = tableit(dict2)
bin = [[i, j] for i, j in zip(table1['start', 'chr1'], table1['stop', 'chr1'])]
df_start, locus = start_stop('start', table2, bin)
# locus.name gets to the data
df_stop, _ = start_stop('stop', table2, bin)
df_merge = pd.merge(df_start, df_stop, on="index", how='outer')
print (df_merge)
if __name__ == "__main__":
main()
The following would have been better,
table_join = pd.concat([table1, table2]).fillna('-')
and then sort them.. perhaps including a new column to specify the original dataframe and screen for the overlap with table_join.iloc[i, column] - table_join.iloc[1-i, other_column].
The key step in all approaches is to convert from "long format" to "wide format" and thats the most important thing I've done in all code and makes pandas work.
Output Table_join
start stop
chromosome chr1 chr2 chr1 chr2
feature misc misc2
feature1 0 + 1.0 - 100.0 -
feature2 0 + 100.0 - 199.0 -
feature3 0 - 150.0 - 500.0 -
feature4 0 + 900.0 1 950.0 100
feature5 0 - 155.0 - 200.0 -
feature6 0 + 800.0 - 901.0 -