3
$\begingroup$

Hi I am trying to get the cancer ontologies (obo_id and label) from EBI-OLS. Earlier I have used the below code to get the obo_id terms and labels

descendants = requests.get(
    "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants"
)

As there are more than 1 page so I created total_pages

total_pages = descendants.json()["page"]["totalElements"] // 1000 + 1

cancer_id = []

for i in range(total_pages):
    ## Check the number of id's present in 'totalElements'
    des = {"page": i, "size": descendants.json()["page"]["totalElements"]}
    ## Add the number of 'totalElements' and 'pages' in the new python request for descendants
    cancer_efo = requests.get(
        "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants",
        params=des,
    )
    ## Get the number of obo_id's present in the cancer_efo.json
    obo = [obj["obo_id"] for obj in cancer_efo.json()["_embedded"]["terms"]]
    ## Get the number of cancer labels present in the cancer_efo.json
    label = [obj["label"] for obj in cancer_efo.json()["_embedded"]["terms"]]
    data = zip(obo, label)
    cancer_id.extend(data)

The above code worked pretty well with one link Now the issue is that I want to get ontologies for 37 descendent links and maximum links have more than 1 page. I want to get one file (cancer_id) containing all obo_id's and labels from these all 37 links.

I made a separate file called list.txt (Right now I am copying only 3 here)

list.txt

https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3171/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3457/descendants

I tried the below code but it is not working

import requests

with open ('list.txt', 'r') as fin:
    links = fin.readlines()

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

data = list(map(requestIt, links))

obo = obj["obo_id"] for obj in data

The error is

obo = obj["obo_id"] for obj in data
                        ^
SyntaxError: invalid syntax
Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
1
$\begingroup$

Okay, its done. What you are looking for is as follows:

import requests
from pathlib import Path

# Test code    
def get_all_keys(d):
    for key, value in d.items():
        yield key
        if isinstance(value, dict):
            yield from get_all_keys(value)

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

if __name__ == '__main__':

    path = Path('/PathToDir/test', 'link.html')
    with open (path, 'r') as fin:
        links = fin.readlines()
        
        
    data = list(map(requestIt, links))

    # Test code
    for x in get_all_keys(data[0]['_embedded']['terms'][0]):
        print(x)

    # Answer code
    for i,d in enumerate(data):
        print (f'Link {i + 1} ') 
        for obo_id in d['_embedded']['terms']:
            print (obo_id['obo_id'])

Output

Link 1 NCIT:C8263 NCIT:C9163 NCIT:C5630

Link 2 NCIT:C7175 NCIT:C68700 NCIT:C68696 NCIT:C68697 NCIT:C68698 NCIT:C68699 NCIT:C9155 NCIT:C9288 NCIT:C9287 NCIT:C9019 NCIT:C9018 NCIT:C36055 NCIT:C36058 NCIT:C36057 NCIT:C36056 NCIT:C38377 NCIT:C122688 NCIT:C156718 NCIT:C156719 NCIT:C169107

Link 3 NCIT:C7056 NCIT:C131911 NCIT:C165799 NCIT:C68684 NCIT:C4998 NCIT:C5089 NCIT:C5095 NCIT:C8154 NCIT:C8604 NCIT:C127840 NCIT:C8489 NCIT:C8852 NCIT:C69144 NCIT:C115349 NCIT:C115367 NCIT:C165240 NCIT:C8926 NCIT:C160230 NCIT:C160231 NCIT:C182035

Where Link 1 is ...

https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants

... and so on.

If the code is listed # Test code that is what is used to interrogate the data structure. This doesn't contribute to your answer.

However, given the way I've done it, it's important to understand the data structure. Thus output of # Test code is at the base of this post. If you need another part of the data structure you simply switch id_odo to whatever else you are looking for (at the base of this post).

There will be an easy to use OOP method based out the data object. I did't want to spend time investigating the object ... the question is specifically about obo_id so it was easier to go for that.

Test code output

iri
lang
description
synonyms
annotation
Contributing_Source
Display_Name
Legacy Concept Name
Neoplastic_Status
Preferred_Name
Semantic_Type
UMLS_CUI
code
label
ontology_name
ontology_prefix
ontology_iri
is_obsolete
term_replaced_by
is_defining_ontology
has_children
is_root
short_form
obo_id  # <- its here
in_subset
obo_definition_citation
obo_xref
obo_synonym
is_preferred_root
_links
self
href
parents
href
ancestors
href
hierarchicalParents
href
hierarchicalAncestors
href
jstree
href
graph
href
Improve this answer
$\endgroup$
2
$\begingroup$

@Megha The requestIt(link) function returns a list of dictionary objects. So your "data" variable is a list of dictionary objects.

Every item on this list (a dict) has a key named "_embedded" whose value is also a dictionary. This dictionary (the _embedded) has a key named "terms" which is a list of dictionaries. The obo_ids you need are inside the dictionaries of the "terms" list.

The problem with your code was that you were trying to access the "obo_id" key on the top level dictionary.

So a simple solution would be like this:

with open ('urls.txt', 'r') as fin:
    links = fin.readlines()
resps = [requestIt(link) for link in links] # this one holds all the data
obo_ids = []
for resp in resps:                          # for every response we got
    terms = resp['_embedded']['terms']      # choose the terms list
    for term in terms:                      # for every dict in this list
        obo_ids.append(term['obo_id'])      # extract obo_id and append it to our answer
print(obo_ids)                              # do stuff with obo_ids

You should probably perform some exception handling inside the requestIt() function to handle broken links etc.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome @punxx and this looks cool $\endgroup$
    – M__ ♦
    Commented Mar 9, 2023 at 17:24
1
$\begingroup$

@Megha, I have not looked through the code in too much detail, however.

obo = obj["obo_id"] for obj in data

This isn't formal Python code. The correction would simply be to place this in a comprehension ...

obo = [obj["obo_id"] for obj in data]

... unless you want an iterator.

I think there is a deep issue with obj["obo_id"]. If the syntax still throws an exception could you provide a print out of either data please, or else ....

obj for obj in data:
    print (obj)
    print (obj.keys)

The data data structure for the output of get.requests followed by .json() is kinda key here

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the answer. Got the same error with obo = [obj["obo_id"] for obj in data]. Then I tried obj for obj in data: print (obj) print (obj.keys). Got an error `obj for obj in data: ^ SyntaxError: invalid syntax $\endgroup$
    – Megha
    Commented Mar 7, 2023 at 10:08
  • $\begingroup$ Is there any way I can run the code (the cancer_id [ ] one)that worked perfectly for one link in a for loop for all the links? $\endgroup$
    – Megha
    Commented Mar 7, 2023 at 10:10
  • 1
    $\begingroup$ Thanks @Megha. Did you get a print out for print (obj)? Then an error for print (obj.keys)? The idea is to check the output before the error. $\endgroup$
    – M__ ♦
    Commented Mar 7, 2023 at 13:46
  • 1
    $\begingroup$ Hi @Megha over the weekend I will replicate your code and investigate, the object, json, dictionary ... (whatever it is) output. I should get I definitive answer, but I'm afraid I can't do that during this week due to work stuff. $\endgroup$
    – M__ ♦
    Commented Mar 7, 2023 at 15:44
  • 1
    $\begingroup$ @Megha cancer_all.json() is an object so its behaviour is very different. obj["obo_id"] is not going to work because you'll need to reference the instance. It's possible something like str(obj)['repo_url'] might work. I'll need to check and examine all instances etc ... $\endgroup$
    – M__ ♦
    Commented Mar 8, 2023 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.