Using column 2 of one file to match with two columns of another file, and append

Question

I have file 1 like following:

---

1   15776220    15776240    GTGACCAGCAGGTGTCTCTG    16855676    16855696    CTGTCCAGCAGAGGGCGGTG    

And file 2 as following

---

1   15776231    2   5008    G:5002  A:6 1   16855677    2   5008    A:5003  C:5

I am trying to write a code(but failing!) to do the following:

if $2 of file2 comes in between $2 and $3 of file1 (between the interval of $2 and $3) and if $8 of file2 comes in between $5 and $6 of file1 (between the interval of $5 and $6), then the output would be $1,$2,$3,$4 from file1 and $2,$3,$4,$5,$6 from file2 and $5,$6,$7 from file1 and $8,$9,$10,$11,$12 from file 2, all in one line. So something like this:

1   15776220    15776240    GTGACCAGCAGGTGTCTCTG    15776231    2   5008    G:5002  A:6 16855676    16855696    CTGTCCAGCAGAGGGCGGTG    16855677    2   5008    A:5003  C:5

Something like bedtools does but I am not able to use bedtools one this since I don't have two columns in file2. Is there any way to make bedtools work on this?

(There could be many lines from file2 intersecting between the same line in file1,so that line could be repeated)

Answer 1

Pipe a modified form of the second file to BEDOPS bedmap and the first file, then pipe that result to cut out desired columns:

$ awk -v OFS='\t' '{ print $1, $2, ($2+1), $3, $4, $5, $6, $7, $8, $9, $10; }' second.txt | bedmap --echo --echo-map first.bed - | cut -f1-4,8,9,11- - > answer.bed

This should run pretty quickly and use very little memory, as it uses Unix streams on (presumably) sorted BED data.

You'll want to test the column indices in my answer by running this on your test inputs, but that's an easy adjustment for the larger dataset, in case I missed a column.

If you're not sure about sort order, pipe to sort-bed:

$ sort-bed first.unsorted.bed > first.bed
$ awk -v OFS='\t' '{ print $1, $2, ($2+1), $3, $4, $5, $6, $7, $8, $9, $10; }' second.txt | sort-bed - | bedmap --echo --echo-map first.bed - | cut -f1-4,8,9,11- - > answer.bed

Use streams where you can. It's a different way of thinking but can pay huge dividends.

Answer 2

Here's one approach in Perl. Note that I have not tested this extensively, and I am not confident it works. Please test first and let me know if there are problems.

#!/usr/bin/env perl
use strict;
use warnings;
use 5.10.0;
my (@file1,%file2);
open(my $f1, '-|',"sort -nk2 $ARGV[0]");
while (<$f1>) {
  chomp;
  my @fields = split(/\s+/);
  push @file1, [ @fields ];
}
close($f1);

open(my $f2, "sort -nk2 $ARGV[1] |");
line:while (<$f2>) {
  chomp;
  my @fields = split(/\s+/);
  foreach my $file1Line (@file1) {
    my $file1Start1 = $file1Line->[1];
    my $file1End1 = $file1Line->[2];
    my $file1Start2 = $file1Line->[4];
    my $file1End2 = $file1Line->[5];
    my $file2Start1 = $fields[1];
    my $file2End1 = $fields[2];
    my $file2Start2 = $fields[7];
    my $file2End2 = $fields[8];
    my @file1Fields = @$file1Line;
    if ($file1Start1 < $file2Start1 &&
        $file1End1 > $file2End1 &&
        $file1Start2 < $file2Start2 &&
        $file1End2 > $file2End2) {
      say join "\t", @{$file1Line}[0..3], @fields[1..5], @{$file1Line}[4..6], @fields[7..11];
    }
    elsif ($file1Start1 > $file2Start1) {
      last;
    }
  }
}
close($f2);    

Save the script as foo.pl somewhere in your $PATH, make it executable and run as:

$ foo.pl file1.bed file2.bed 
1   15776220    15776240    GTGACCAGCAGGTGTCTCTG    15776231    2   5008    G:5002  A:6 16855676    16855696    CTGTCCAGCAGAGGGCGGTG    16855677    2   5008    A:5003  C:5