1
$\begingroup$

I was curious to give DNA BERT a try. This is a BERT (Bidirectional Encoder Representations from Transformers) model that was trained on short (k=3,4,5, or 6) k-mers of DNA. Overall, It is exciting work since it bridges NLP approaches and genetics data.

I span up a pytroch3.6 machine on SageMaker with a GPU and tried to assess the prediction of its pre-trained DNABERT model:

Preparing the data:

!git clone https://github.com/jerryji1993/DNABERT
%cd DNABERT
!python -m pip install --editable .
%cd examples
!python -m pip install -r requirements.txt
!curl -L https://northwestern.app.box.com/shared/static/g8m974tr86h0pvnpymxq84f1yxlhnvbi --output a6-new-12w-0.zip
!unzip a6-new-12w-0.zip
!cp ./sample_data/ft/prom-core/6/dev.tsv /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0
  1. Running prediction based on the pre-trained data:
!python run_finetune.py \
   --model_type=dna \
   --tokenizer_name=dna6 \
   --model_name_or_path=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0 \
   --task_name=dnaprom \
   --do_predict \
   --data_dir=/home/ec2-user/SageMaker/DNABERT/examples/sample_data/ft/prom-core/6  \
   --max_seq_length=75 \
   --per_gpu_pred_batch_size=128   \
   --output_dir=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out \
   --predict_dir=/home/ec2-user/SageMaker/DNABERT/examples/result \
   --n_process=4

Output:

11/09/2020 04:35:57 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   loading configuration file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/config.json
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": "dnaprom",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "num_rnn_layer": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "rnn": "lstm",
  "rnn_dropout": 0.0,
  "rnn_hidden": 768,
  "split": 10,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 4101
}

============================================================
<class 'transformers.tokenization_dna.DNATokenizer'>
11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 acquired on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /home/ec2-user/.cache/torch/transformers/tmpa9hkn2q1
Downloading: 28.7kB [00:00, 12.5MB/s]                                           
11/09/2020 04:35:57 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.file_utils -   creating metadata file for /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 released on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/pytorch_model.bin

However, the result directory is empty and I am not sure what did I miss?

!ls -lh /home/ec2-user/SageMaker/DNABERT/examples/result
total 0


!ls -l  /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out
total 0

EDIT: I have tried also to run this on another ec2 machine and apparently it gets a segmentation fault, so my conclusion is that it's unstable. I will train a RoBERTAa model or similar by myself:

SageMaker doesn't propagate the segmentation fault to the notebook which is another weird thing about it.

11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 acquired on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpwpxfkxlm
Downloading: 28.7kB [00:00, 20.2MB/s]
11/12/2020 14:01:18 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.file_utils -   creating metadata file for /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 released on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/DNABERT/examples/6-new-12w-0/pytorch_model.bin
Segmentation fault
Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I wonder if the data science stackexchange would end up being better for this question. I'd want to remove Michaels answer first though, which would be a shame. Could you maybe just repost this over there and provide a link back for context? $\endgroup$
    – Devon Ryan
    Commented Nov 12, 2020 at 7:58

1 Answer 1

Reset to default
1
$\begingroup$

I don't why this RNN k-mer Pytorch model is not producing an output. I do know that,

 "rnn_dropout": 0.0,

... is bad news for any ANN (artificial neural network) let alone LSTM RNN, which is a pretty complex model.

Firstly, you have to assign a meaningful drop out rate and 0.5 is okay.

If dropout rate = 0 as an output then the model hasn't worked. A dropout layer is essential to present overfitting. Its a foundational part of deep learning. No-one emperically knows quite why it works, there's a few approximate good guesses, but a dropout is essential.

The other thing to note is what does "n_process" mean? If you are parallelising a GPU across 4-cores, okay its got a lot of layers but even so thats alot of cores. It ain't a CPU and I'd get it working on a single GPU core first.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I see your points. However, this should work presumably out of the box. I have tried to use google colab too, but it was too complicated there compared to SageMaker. $\endgroup$
    – 0x90
    Commented Nov 10, 2020 at 1:54
  • $\begingroup$ Yep thats Google! Its deep learning is always geeky :-) I'm surprised to hear this about colab because it is supposed to be set up for DL, but can vouch 100% thats what Google developers do. Tensorflow (Google) vs PyTorch (FaceBook) ... is similar, PyTorch is much less geekified. $\endgroup$
    – M__ ♦
    Commented Nov 10, 2020 at 2:00
  • $\begingroup$ I was intending to give DNABERT a try as it was published on GitHub. Sadly the developers haven’t replied back to me. Feel free to give it a try and see if it works for you as-is. $\endgroup$
    – 0x90
    Commented Nov 10, 2020 at 2:02
  • $\begingroup$ I don't know, because I don't know how they are vectorising their k-mers. In my opinion converting DNA to numbers for DL is key to the results. You are using a solid model with a crazy number of layers (12 is alot). Anyway, it was fun to chat. $\endgroup$
    – M__ ♦
    Commented Nov 10, 2020 at 2:06
  • 1
    $\begingroup$ Thanks @0x90. Sorry I said 'crazy' what I meant was the number of layers was alot. This is a good thing and was (perhaps not now) the basic method for increasing the accuracy of the model in any field of DL. This approach was famously deployed by one big player (was it Google?) to win an image training competition with the other big players. Its just in bioinformatics its the most layers I've seen in any model. $\endgroup$
    – M__ ♦
    Commented Nov 10, 2020 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.