1
$\begingroup$

I'm trying to download pdf from pubmed central using selenium webdriver in python. The code is provided below which downloads the pdf fine but, it does not quit automatically. i.e. it is sort of stuck executing browser.get(href) line in download_pmc_pdf(). What could be the reason?

import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
ncbi_url = "https://www.ncbi.nlm.nih.gov/"

download_dir = "pdf_downloaded"

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

browser = webdriver.Firefox(options=options)

###############################################################################
def download_pmc_pdf(pmcid_key):
    pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
        
        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                browser.get(href)
                break  # exit the loop as we have found the PDF link
        print("line beofre browser.quit()")
        browser.quit()  # quit the browser outside the loop
    except Exception as e:
        print(e)


    
###############################################################################
pmcid = "2788555"            
download_pmc_pdf(pmcid)
browser.quit()
Improve this question
$\endgroup$
2
  • $\begingroup$ This might be better-suited to stackoverflow since it's primarily a selenium question rather than a bioinformatics question, but for what it's worth, I see the same behavior when I try your script. (I've used selenium a bit before but never tried get for anything other than HTML.) I don't suppose the e-utils support PDF paper download, do they? $\endgroup$
    – Jesse
    Jul 14 at 14:25
  • $\begingroup$ To the best of my knowledge, E-utils primarily supports text, not PDF downloads. For some PMC-listed articles, the XML, mediline records lack full text, hence the need for browser automation. Though e-utils is my preferred choice, this situation demands an alternative. Nonetheless, the issue is now resolved $\endgroup$
    – Wasim Aftab
    Jul 14 at 15:50

2 Answers 2

Reset to default
2
$\begingroup$

I had to execute a JavaScript code that opens the pdf link in a new window/tab. After that, just closed this newly opened window/tab and continued with the script execution. Here is the updated code:

import os
import time
import csv

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ncbi_url = "https://www.ncbi.nlm.nih.gov/"

download_dir = "pdf_downloaded"

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

###############################################################################
def download_pmc_pdf(pmcid_key):
    browser = webdriver.Firefox(options=options)
    pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))

        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                # Open the pdf link in a new window
                browser.execute_script(f'window.open("{href}","_blank");')
                break

        time.sleep(15)  # delay for 10 seconds
        # After the file is downloaded:
        with open(f'{download_dir}/file_mapping.csv', 'a', newline='') as file:
            writer = csv.writer(file)
            writer.writerow([pmcid_key, os.path.basename(href)])
        ## Switch to the new window and close it 
        ## (this is a hack to deal with bowser freezing upon downloading)
        browser.switch_to.window(browser.window_handles[-1])
        browser.close()

    except Exception as e:
        print(e)
    finally:
        browser.quit()  # quit the browser outside the loop    
###############################################################################
pmcid = "2788555"   
download_pmc_pdf(pmcid)
Improve this answer
$\endgroup$
1
$\begingroup$

Pubmed Central stores pdfs for all available open access articles on a public-facing ftp server (https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/). You can query the database using the OA Web Service API, e.g. with cURL

# for pubmed id "10026098"
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098
<OA><responseDate>2023-07-17 01:39:13</responseDate><request id="PMC10026098">https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098</request><records returned-count="2" total-count="2"><record id="PMC10026098" citation="Ther Adv Med Oncol. 2023 Mar 18; 15:17588359231161409" license="CC BY-NC" retracted="no"><link format="tgz" updated="2023-03-23 12:10:30" href="ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/83/eb/PMC10026098.tar.gz" /><link format="pdf" updated="2023-03-20 09:34:33" href="ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf" /></record></records></OA>

# get the pdf link from the API (ftp storage)
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf

# download the pdf
pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
curl -O "$f"

# download many pdfs from a list of ids
cat list_of_ids.txt
10026098
9341183

while read -r id_from_list
do
    pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
    curl -O "$pdf"
done < list_of_ids.txt

This approach is preferable to selenium scraping for many reasons, and if you have any problems the NCBI provides documentation (https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart) and offers support (you can email them at [email protected]).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your time! Unfortunately, it did not work, returning the error curl: (23) Failed writing received data to disk/application Furthermore, I've noticed a potential issue. What if the article listed on PubMed is not available on PMC but on other platforms, such as the publisher's website? For example, PMID:37230009 $\endgroup$
    – Wasim Aftab
    Jul 17 at 13:05
  • $\begingroup$ You're welcome; just trying to help. That cURL error is very likely related to your curl install and file permissions (e.g. askubuntu.com/questions/1356327/…); you could try using wget instead, e.g. wget -O - https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 2> /dev/null | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p' should work. And as for articles on Pubmed that aren't on PMC, your question specifies 'articles on PMC', so that is what my answer addresses. If you want to download pdfs from any source, that is a totally different question. $\endgroup$
    – jared_mamrot
    Jul 17 at 23:15
  • $\begingroup$ Your solution is crashing because pdf="$(curl ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"(ftp.*.pdf)\".*/\1/p')" is empty for the pmcids that I used: 8968431, 8570779, 8290288, 8254498. But it worked for the pmcid that you provided. It seems pdfs corresponding to my pmcids are not present on the public-facing ftp server because when I tried your code with some of the pdfs that are listed there it worked. So, for my use case, browser automation is the only way so far $\endgroup$
    – Wasim Aftab
    Jul 19 at 14:41
  • $\begingroup$ Ohh! Yep - I see the problem - only Open Access articles are accessible via the API (<OA><responseDate>2023-07-19 18:55:39</responseDate><request>https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=8968431</request><error code="idIsNotOpenAccess">identifier 'PMC8968431' is not Open Access</error></OA>). Thanks for taking the time to check! $\endgroup$
    – jared_mamrot
    Jul 19 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.