It sounds like you are saying your file is already sorted? Is this correct?
RDKit Mae supplier is an iterator as is the gzip reader, so does not load all in memory, so that is the best option given that loading all 1e6 into memory crashes your system. What is the fraction of unreadable compounds in RDKit? And does it matter? These are likely failing sanitisation —the
sanitize=False flag. If for some reason none can be ignored, circumventing this is hard. The
removeHs=True flag is rarely/never the cause even though the description of several aromatics require protons, e.g. an indole ring is
c1ccc2c(c1)cc[nH]2, the proton removal happens after loading. If you have an identifier and a list of SMILES you can correct dodgy bond orders etc. But making an unsanitisable molecule sanitary is tricky and rarely worth it as the output molecule is likely rubbish anyway.
Plus loading in RDKit will allow you do some additional calculations, such as ligand efficiency (best for sorting anyway), partition coefficient etc etc. As opposed to simply slicing your sorted file.
Alternatively, I believe Open Babel does not load all into memory and has a
splitinto argument to make the file smaller and can convert gzipped Mae file to SDF.