I wrote a command-line (C++14) tool called
subset which is up on Github: https://github.com/alexpreynolds/subset
This should be reasonably memory efficient and fast. The
subset tool does not store input lines in a table, but instead streams through the file once, storing a 4 or 8k buffer chunk of the input file (depending on OS).
It stores line numbers in an array, but eight bytes per integer * 100k is 800kB, for that use case — not very much memory there.
There's an O(nlogn) sort penalty on the line number array, but again this list will be much smaller than the query file, and integer sorting is fairly optimized, so the hit should be small.
If your line number list is already sorted, I could add an option to skip sorting; let me know if that would be useful.
The filtering step walks through the line number array and input file linearly, printing lines where there are index matches, and skipping over the rest.
subset will quit early in parsing the input file if there are no more line numbers to query. So this feature is especially useful for speeding up filtering of very large query files. (If your query file has 1M rows, say, and your last line number of interest is 12345, there's no reason to read through the rest of the file.)
You can grab, build and install it like so:
$ git clone https://github.com/alexpreynolds/subset.git
$ cd subset
$ cp subset /usr/local/bin
Once the binary is in your path, there are a couple ways to use it.
For example, you can specify a start index and length value. The following grabs seven lines starting with the 33rd line (32 as a 0-indexed value):
$ subset --prefix-with-indices -s 32 -n 7 -i query.txt > answer.txt
Or you can specify a text file containing line numbers, each on a separate line. The following reads in a file called
line-numbers.txt and uses that to filter
$ subset --prefix-with-indices -l line-numbers.txt -i query.txt > answer.txt
The indices in
line-numbers.txt should be positive, 0-indexed integers. The list of numbers does not need to be sorted, as
subset will sort the list of numbers for you. This is so that an efficient single pass through the input/query file can be done.
You can leave out
--prefix-with-indices to leave out the debug prefix. This is there so that you can do a sanity check on the result.
test/makefile tests demonstrate options and usages for the two types of filtering.