Skip to content

About

Scripts for ingesting PubMed and bioRxiv publications into the legacy GCID Drupal 7 site

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GCID Publications Ingestion

Adds new PubMed and bioRxiv publications to the legacy TMC-GCID Drupal 7 site (gcid-old) without touching the broken Biblio module UI.

Problem

The Drupal server is air-gapped (no outbound HTTP/HTTPS). The Biblio module's built-in PubMed lookup, DOI lookup, and all other automated import paths require outbound access that does not exist. There is also no RIS/BibTeX import UI. Direct database writes are too risky (Biblio spans multiple tables with serialized fields).

Solution

Fetch metadata locally on the MacBook, normalize it to JSON, transfer via SCP, then inject into Drupal using a Drush PHP script that calls node_save().

Pipeline

rt38697/1564067/n1552291.tsv   ← authoritative publication list (March 2026)
         │
         ▼  MacBook
scripts/fetch_metadata.py      ← fetch PubMed XML via eUtils; bioRxiv via API
         │
         ▼
articles.json                  ← one record per article, normalized
         │
    scp to gcid-old:/space1/hale/
         │
         ▼  gcid-old (via SSH)
scripts/load_biblio.php        ← Drush script; creates biblio nodes via node_save()

Prerequisites

Requirement Value
MacBook Python 3.14+, uv, SSH access to gcid-old
gcid-old Drupal 7.97, Drush 8.4.9 at /usr/local/bin/drush
Drupal root /space1/websites/apache-hosts/gcid
Writable dir /space1/hale/

The SSH host gcid-old must be defined in ~/.ssh/config.

Input

rt38697/1564067/n1552291.tsv — the March 2026 authoritative publication list. Supersedes rt38697/1551492/n1539821.tsv (February 2026).

TSV columns: Title_of_the_paper, Authors, Year_published, PMID, Status, Journal_Published, Date_of_Publication/AOP, final_publication_Pubmed_url, bioRixV_URL, Date_of_Publication_on_bioRixV.

The Status column drives the Biblio type:

TSV Status biblio_type_name biblio_type integer
Published Journal Article 102
pre-print Unpublished 124

Step 1 — Fetch metadata (MacBook)

# Original TSV (default):
uv run scripts/fetch_metadata.py

# Addendum TSV (e.g. from a follow-up email):
uv run scripts/fetch_metadata.py inputs/message-2026-04-28a.tsv articles.json

Reads the TSV, fetches each article from PubMed eUtils (PMID rows) or bioRxiv API (pre-print rows), and writes articles.json to the project root.

Optional positional arguments: [tsv_path [output_path]]. Defaults to rt38697/1564067/n1552291.tsv and articles.json.

Do not modify original RT ticket TSV files. For addendum publications, create a new TSV under inputs/ with the same column headers and pass it as an argument.

Edge cases handled:

  • PMIDs may have a PMID: prefix or \xa0 (Excel non-breaking space) — both stripped automatically.
  • <ArticleTitle> elements may contain inline markup (e.g. <i> for species names); the script uses itertext() to capture the full text.
  • If the bioRxiv API is unavailable for a pre-print, the script falls back to TSV title/author/year data and logs a WARNING (does not abort).
  • A row with TSV Status = pre-print but a valid PMID (published to PubMed before the TSV was updated) gets PubMed metadata with biblio_type_name overridden to Unpublished.
  • The bioRxiv DOI may appear in either bioRixV_URL or final_publication_Pubmed_url; both columns are checked.

Step 2 — Transfer files (MacBook → gcid-old)

scp articles.json gcid-old:/space1/hale/articles.json
scp scripts/load_biblio.php gcid-old:/space1/hale/load_biblio.php

Step 3 — Load into Drupal (gcid-old)

ssh gcid-old '/usr/local/bin/drush --root=/space1/websites/apache-hosts/gcid \
  php-script /space1/hale/load_biblio.php'

The loader is fully idempotent. For each article it queries {biblio_pubmed} by PMID (or {biblio} by DOI for pre-prints without a PMID) and skips any that already exist. Safe to re-run.

Expected output for a clean run:

CREATED [nid=174]: PMID:39007612 "Using Human Intestinal Organoids..."
...
Done. Created: 13  Skipped: 0

Expected output for a re-run (idempotency check):

SKIP [already exists nid=174]: PMID:39007612 "Using Human Intestinal Organoids..."
...
Done. Created: 0  Skipped: 13

Scripts

Script Runs on Purpose
scripts/fetch_metadata.py MacBook Reads TSV → fetches PubMed/bioRxiv → writes articles.json
scripts/load_biblio.php gcid-old Reads articles.json → creates biblio nodes via Drush
scripts/verify_nodes.php gcid-old Spot-check: prints status and title for nids 174–186
scripts/spot_check.php gcid-old Detailed dump of 4 sample nodes for validation

Reference — Biblio field mapping

All fields are direct properties on the $node object (not Field API arrays). Full schema in docs/biblio_schema.md.

JSON field Drupal property Table
title $node->title {node}
biblio_type_name $node->biblio_type (int) {biblio}
year $node->biblio_year {biblio}
journal $node->biblio_secondary_title {biblio}
volume $node->biblio_volume {biblio}
issue $node->biblio_issue {biblio}
pages $node->biblio_pages {biblio}
doi $node->biblio_doi {biblio}
issn $node->biblio_issn {biblio}
date_str $node->biblio_date {biblio}
abstract $node->biblio_abst_e {biblio}
pubmed_url $node->biblio_custom1 {biblio}
pmid $node->biblio_pubmed_id {biblio_pubmed}
pmcid $node->biblio_pmcid {biblio_pubmed}
authors $node->biblio_contributors {biblio_contributor}

Note: biblio_pubmed_md5 (in {biblio_pubmed}) has no database default and must be set before node_save(). The loader sets it to md5('').

Results (April 2026)

14 articles ingested total:

nid PMID / DOI Type Source
174 PMID 39007612 Journal Article March 2026 TSV
175 PMID 39389204 Journal Article March 2026 TSV
176 PMID 40594045 Journal Article March 2026 TSV
177 PMID 39820858 Journal Article March 2026 TSV
178 PMID 39652359 Journal Article March 2026 TSV
179 PMID 40228901 Journal Article March 2026 TSV
180 PMID 40445752 Journal Article March 2026 TSV
181 PMID 39522694 Journal Article March 2026 TSV
182 DOI 10.1101/2025.08.06.668939 Unpublished March 2026 TSV
183 PMID 40501838 Unpublished March 2026 TSV
184 PMID 39386636 Unpublished March 2026 TSV
185 PMID 40463537 Unpublished March 2026 TSV
186 PMID 41637508 Unpublished March 2026 TSV
187 PMID 41676723 Unpublished inputs/message-2026-04-28a.tsv

Drush php-script gotchas

Two non-obvious behaviors when using drush php-script:

  1. File-level variables are not PHP globals. Variables defined at the top of the script (outside any function) are not accessible via global $var inside functions. Use define() for constants or define values inside the function that uses them.

  2. $argv contains Drush's own flags. When invoked as drush --root=... php-script foo.php, the --root= flag occupies $argv[1] inside the script. Do not use $argv for script arguments; use define() constants instead.

Related documents

  • docs/biblio_schema.md — complete field schema discovered from the live site
  • docs/publication_types.md — full list of Biblio type names and integers
  • CLAUDE.md — execution environment, constraints, and rejected approaches
  • plan.md — phased execution checklist (all phases complete except browser spot-check of the public publications page)

About

Scripts for ingesting PubMed and bioRxiv publications into the legacy GCID Drupal 7 site

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages