I don't think it will be possible to do what you ask, right now with current knowledge. Selecting variants relevant to cancer risk is still an open problem and usually requires quite a lot of human intervention.
You can use different measures of population frequency to filter common variants with the assumption that frequent variants won't be pathogenic.
You can use multiple in-silico predictors of pathogenicity to try to remove "benign" or "tolerated" variants, but predictors are far from perfect and you have to take into account that many of them are essentially measuring the same, so when combining their outputs you may be biasing your analysis towards nucleotide conservation, etc...
You can use in-silico predictors of splicing effects. These are even farther from perfect and return very noisy data.
You can take into account where the variants lie in the gene. In many genes, variants in the last few nucleotides won't be pathogenic, but may be important in others. In some genes only variants in some domains have an effect, while not in others.
You have to take into account if only one or both alleles of certain genes are affected. For that you may need phasing information. And obviously the list of genes.
Some genes have an impact on cancer risk when they are "deactivated" (or broken somehow) (tumor supressors) while others have an effect when their function is improved (oncogenes).
And for many many genes you'll simply not know if they have any effect in cancer risk at all.
You can get data from different databases and annotate your genomes to remove many variants but in the end you won't have very high quality data to enter to your ML algorithms. Maybe this will work for your project, but take into account that this "first step" of data cleaning is not easy or obvious with current biological knowledge.