Sort both files by chromosome and position and read each file one line by line.
Either you iterate to the next line in one file, or go the next in the other, or there's a match that you print out and then iterate both lines.
This is not necessarily an awk approach, but is doable with Perl, Python, or any other scripting language.
In pseudocode:
Sub getCommonLines()
Sort FileA
Sort FileB
Get lineA from FileA
Get lineB from FileB
While not EOF in FileA or EOF in FileB:
If lineA > lineB:
Get lineB from FileB
Elseif lineB > lineA:
Get lineA from FileA
Else:
Print lineA or lineB
Get lineA from FileA
Get lineB from FileB
End Sub
The use of the >
operator is based on lexicographical and numerical ordering provided from sorting files A and B, and comparison should be done only on the fields of interest. If you compare other fields (esp. from VCF files) you may get other fields that give you the wrong or unexpected comparison result.
The term EOF
means end-of-file, which means just what it says — you're at the end of the file and there are no more lines to compare between files, so your subroutine is done.
Sorting has a time cost, but memory overhead for modern sorting algorithms is very minimal. You can sort via process substitution, but sorting to an intermediate file can be useful if you have other things you want to do with them, where sorted input is valuable.
This would be a good approach for very large files that you do not edit or update often (or at all), for which a hash table (or "dictionary", or "associative array", all the same terms for the same thing) is impractical because of its large memory overhead.
You can also use a tool like comm
with sorted files, as described in another answer/comment. This does basically what my answer describes without you having to understand or implement things. You must remember to provide sorted input, however, as with other Unix tools that do line comparison (uniq
, etc.).
Despite this answer being a really long "comment", it is useful to have a basic understanding of how things work under the hood, so that when you use these tools, you know what's going on and why you need to provide inputs in a certain way. It's easy to get a million one-liners off of Stack Exchange sites, but understanding is valuable beyond gold.