0
$\begingroup$

I'm working with UK Biobank data. I made a cohort with the Cohort Browser and tried to extract the data with this code df = participant.retrieve_fields(names=field_names, filter_sql = cohort.sql, coding_values = "replace", engine=dxdata.connect()).

This is part of the text of the error: Py4JJavaError: An error occurred while calling 04100.showString. org.apache.spark. SparkException: Exception thrown in awaitResult:. The full error message is cut off.

However, if I remove the filter - df = participant.retrieve_fields(names=field_names, coding_values = "replace", engine=dxdata.connect()) – there is no issue.

What kind of error is this and why would filtering for a cohort help?

Improve this question
$\endgroup$
3
  • $\begingroup$ Thanks for the question and the edit. This is Python API for a Biobank - looks very interesting. Just to clarify the implication is you can connect to Biobank ok, but when you add a filter this stops the operation. However the code that connects okay has not been shown. Is that all correct? $\endgroup$
    – M__ ♦
    Jul 22 at 1:36
  • $\begingroup$ Yes, that's correct $\endgroup$
    – eclarette
    Jul 23 at 3:23
  • $\begingroup$ Okay, it's a bit hard without knowing the underlying code. $\endgroup$
    – M__ ♦
    Jul 23 at 4:27

2 Answers 2

Reset to default
1
$\begingroup$

It looks like the issue comes from one of the fields I was filtering for. When I remove that filter the code works again. Still need to investigate why that happened.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could you kindly expand on the answer please, specifically which field? $\endgroup$
    – M__ ♦
    Jul 24 at 17:34
  • 1
    $\begingroup$ It really isn't clear. Possibly something in self-reported data, field 20002. I ended up using a workaround by filtering for fields separately and joining them with df.union(df2) $\endgroup$
    – eclarette
    Jul 25 at 19:23
0
$\begingroup$

Generally remote biological servers, for example UCSC genome server, will not perform SQL queries for a free account. They'll allow downloads, but not queries. That is probably the same reason here.

The reason is simply these are often huge databases and queries can take up a lot of their computational resources, particularly if everyone is doing them. I don't know BioBank UK, but suggest its common practice.

If thats true the only difference with Biobank and UCSC is the UCSC server refusals are stated very clearly, "this operation ain't permitted for free accounts" (paraphrasing). What might be happening in an API error is not being picked up and relayed to you. Depends how the API is written so it's just reporting a timeout.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The account is paid, not free $\endgroup$
    – eclarette
    Jul 23 at 22:11
  • $\begingroup$ @eclarette If you login directly you will receive a more informative message than going through an API, i.e. perform the SQL query there. The API isn't reporting the error message. $\endgroup$
    – M__ ♦
    Jul 23 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.