Download all pubmed central article ids that have a keyword appearing in their titles/abstracts and also falling between two dates

Question

I am using the following code to download all pubmed central article ids that have a keyword appearing in their titles/abstracts and also falling between two dates. The code seems to work when I select database = "pubmed" but when I switch to database = "pmc" I get empty list. For example, using the start and end dates as mentioned in the code I get 4 PubMed ids: ['36902449', '36880658', '36583446', '35239250'] and when I check manually, the first one among them (which is '36902449') is also present in PMC database, of course, the id is different (10003631) but same article. The code is expected to fetch that id. I'd appreciate it if someone could point out what I'm missing here.

import os
import datetime
from Bio import Entrez

# Set the email address to identify yourself to NCBI
Entrez.email = "example@gmail.com"

# Set the date range for the search
start_date = datetime.datetime(2023, 3, 1)
end_date = datetime.datetime(2023, 3, 22)

# Format the dates as strings in the correct format
start_date_str = start_date.strftime("%Y/%m/%d")
end_date_str = end_date.strftime("%Y/%m/%d")

# Build the search query with the date range
keyword = "dosage compensation"
# database = "pubmed"
database = "pmc"

# Use esearch to fetch the list of matching article IDs
if database == "pubmed":
    search_term = f'("{start_date_str}"[Date - Publication] : "{end_date_str}"[Date - Publication]) AND {keyword}[Title/Abstract]'
    handle = Entrez.esearch(db=database,
                            term=search_term,
                            retmax=100,
                            api_key = os.environ['ENTREZ_API_KEY'])
elif database == "pmc":
    search_term = f'("{start_date_str}"[Publication Date] : "{end_date_str}"[Publication Date]) AND {keyword}[Title/Abstract] AND open_access[filter]'
    handle = Entrez.esearch(db=database,
                        term=search_term,
                        retmax=1000,
                        api_key = os.environ['ENTREZ_API_KEY'])

record = Entrez.read(handle)
id_list = record['IdList']
handle.close()

# Print the number of matching articles
print(f"Found {len(id_list)} articles published between {start_date_str} and {end_date_str} in {database}")
# print(id_list)

Answer 1

There seems to be a problem with using the [Title/Abstract] field. What you have looks correct and should produce the expected result, since it should capture articles where the search term appears in either the title or in the abstract, or in both. But it does not. Strangely, you get the expected result using {keyword}[Abstract].

A workaround could be to OR the fields with:

f'("{keyword}"[Title] OR "{keyword}"[Abstract])'

Perhaps something like:

search_term = ' AND '.join([
    f'("{start_date_str}"[Publication Date] : "{end_date_str}"[Publication Date])',
    f'("{keyword}"[Title] OR "{keyword}"[Abstract])',
    f'(open_access[filter])',
])