# Lifting from hg38 to hg37

I am having strange issues lifting this dataset to hg37 from hg38. I am using the HGDP WGS data from ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/. The data set split by chromosome, so I first converted them to BED then merged them to a single file using plink1.9. However, when trying to lift to hg37/19 using either liftOver or CrossMap v0.5.1 I get strange errors. I tried both the chain from ensemble.org and from UCSC, I tried them both compressed and uncompressed. I am starting to think I need to try the conversion on the VCF files but these are so large I would prefer not to duplicate them unnecessarily.

# Data

The data looks fine. If we look at the BIM:

user@desktop:/media/luks8tb1/data/genomics/HGDP$$head hgdp_hg38_chrpos.bim 1 1:10153 0 10153 G A 1 1:10163 0 10163 C T 1 1:10180 0 10180 C T 1 1:10250 0 10250 C A 1 1:10257 0 10257 C A 1 1:10291 0 10291 T C 1 1:10297 0 10297 T C 1 1:10327 0 10327 C T 1 1:10330 0 10330 A C 1 1:10333 0 10333 T C user@desktop:/media/luks8tb1/data/genomics/HGDP$$ tail hgdp_hg38_chrpos.bim
Y   Y:56887228  0   56887228    C   T
Y   Y:56887285  0   56887285    C   A
Y   Y:56887316  0   56887316    A   G
Y   Y:56887463  0   56887463    A   C
Y   Y:56887491  0   56887491    A   C
Y   Y:56887503  0   56887503    T   G
Y   Y:56887583  0   56887583    G   A
Y   Y:56887631  0   56887631    T   C
Y   Y:56887837  0   56887837    A   G
Y   Y:56887844  0   56887844    C   T


The data are in chrpos variant names here but I had the same result using the original rsid.

# Chains

Chains look fine, and I am able to read them into R without problems:

user@desktop:/media/luks8tb1/data/genomics/HGDP$head GRCh38_to_GRCh37.chain chain 1 1 248956422 + 10000 297968 1 249250621 + 10000 267719 2 167417 80249 50000 40302 chain 1 1 248956422 + 347968 501617 1 249250621 - 248779253 248932902 3 153649 chain 1 1 248956422 + 585988 12949384 1 249250621 + 521368 13009210 4 1044707 1 0 3709 3 0 user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail GRCh38_to_GRCh37.chain

chain 1 chrY 57227415 + 56821509 57217415 Y 59373566 + 58967656 59363566 11183
85168 0 1
78580 0 1
63496 0 1
3158 1 0
30382 0 1
54147 0 1
80974


user@desktop:/media/luks8tb1/data/genomics/HGDP$CrossMap.py bed GRCh38_to_GRCh37.chain hgdp_hg38_chrpos.bed hgdp_hg37_chrpos.bed @ 2020-08-25 20:04:26: Read the chain file: GRCh38_to_GRCh37.chain Traceback (most recent call last): File "/home/user/.local/bin/CrossMap.py", line 166, in <module> crossmap_bed_file(mapTree, in_file, out_file) File "/home/user/.local/lib/python3.6/site-packages/cmmodule/mapbed.py", line 32, in crossmap_bed_file for line in ireader.reader(inbed): File "/home/user/.local/lib/python3.6/site-packages/cmmodule/ireader.py", line 27, in reader yield l.decode('utf8').strip().replace("\r", "") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 20: invalid continuation byte  # liftOver user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.bed hg38ToHg19.over.chain hgdp_hg37.bed hgdp_hg37_errors.bed
Mapping coordinates
Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed)


# Trying the PED file with liftOver

I converted BED file to PED with plink1.9:

plink --bfile hgdp_hg38 --recode --out hgdp_hg38


Then:

user@desktop:/media/luks8tb1/data/genomics/HGDP\$ liftOver hgdp_hg38.ped hg38ToHg19.over.chain hgdp_hg37.ped hgdp_hg37_errors.ped
Mapping coordinates
invalid unsigned integer: "HGDP00001"


CrossMap does not support PED.

BED format can refer to the ".txt" file format that stores chromosomes and positions of genomic features as well as some other things like name. bed format can also refer to the plink format, which belongs to bed, bim, fam trio. The plink bed format stores all the genetic variants, but the program needs to refer to bim file for their positions.
• Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed) According to this, they are not. Aug 26 '20 at 7:38