Adds new PubMed and bioRxiv publications to the legacy TMC-GCID Drupal 7 site
(gcid-old) without touching the broken Biblio module UI.
The Drupal server is air-gapped (no outbound HTTP/HTTPS). The Biblio module's built-in PubMed lookup, DOI lookup, and all other automated import paths require outbound access that does not exist. There is also no RIS/BibTeX import UI. Direct database writes are too risky (Biblio spans multiple tables with serialized fields).
Fetch metadata locally on the MacBook, normalize it to JSON, transfer via SCP,
then inject into Drupal using a Drush PHP script that calls node_save().
rt38697/1564067/n1552291.tsv ← authoritative publication list (March 2026)
│
▼ MacBook
scripts/fetch_metadata.py ← fetch PubMed XML via eUtils; bioRxiv via API
│
▼
articles.json ← one record per article, normalized
│
scp to gcid-old:/space1/hale/
│
▼ gcid-old (via SSH)
scripts/load_biblio.php ← Drush script; creates biblio nodes via node_save()
| Requirement | Value |
|---|---|
| MacBook | Python 3.14+, uv, SSH access to gcid-old |
| gcid-old | Drupal 7.97, Drush 8.4.9 at /usr/local/bin/drush |
| Drupal root | /space1/websites/apache-hosts/gcid |
| Writable dir | /space1/hale/ |
The SSH host gcid-old must be defined in ~/.ssh/config.
rt38697/1564067/n1552291.tsv — the March 2026 authoritative publication list.
Supersedes rt38697/1551492/n1539821.tsv (February 2026).
TSV columns: Title_of_the_paper, Authors, Year_published, PMID,
Status, Journal_Published, Date_of_Publication/AOP,
final_publication_Pubmed_url, bioRixV_URL, Date_of_Publication_on_bioRixV.
The Status column drives the Biblio type:
| TSV Status | biblio_type_name |
biblio_type integer |
|---|---|---|
Published |
Journal Article | 102 |
pre-print |
Unpublished | 124 |
# Original TSV (default):
uv run scripts/fetch_metadata.py
# Addendum TSV (e.g. from a follow-up email):
uv run scripts/fetch_metadata.py inputs/message-2026-04-28a.tsv articles.jsonReads the TSV, fetches each article from PubMed eUtils (PMID rows) or bioRxiv
API (pre-print rows), and writes articles.json to the project root.
Optional positional arguments: [tsv_path [output_path]]. Defaults to
rt38697/1564067/n1552291.tsv and articles.json.
Do not modify original RT ticket TSV files. For addendum publications, create
a new TSV under inputs/ with the same column headers and pass it as an
argument.
Edge cases handled:
- PMIDs may have a
PMID:prefix or\xa0(Excel non-breaking space) — both stripped automatically. <ArticleTitle>elements may contain inline markup (e.g.<i>for species names); the script usesitertext()to capture the full text.- If the bioRxiv API is unavailable for a pre-print, the script falls back to TSV title/author/year data and logs a WARNING (does not abort).
- A row with TSV
Status = pre-printbut a valid PMID (published to PubMed before the TSV was updated) gets PubMed metadata withbiblio_type_nameoverridden toUnpublished. - The bioRxiv DOI may appear in either
bioRixV_URLorfinal_publication_Pubmed_url; both columns are checked.
scp articles.json gcid-old:/space1/hale/articles.json
scp scripts/load_biblio.php gcid-old:/space1/hale/load_biblio.phpssh gcid-old '/usr/local/bin/drush --root=/space1/websites/apache-hosts/gcid \
php-script /space1/hale/load_biblio.php'The loader is fully idempotent. For each article it queries {biblio_pubmed}
by PMID (or {biblio} by DOI for pre-prints without a PMID) and skips any
that already exist. Safe to re-run.
Expected output for a clean run:
CREATED [nid=174]: PMID:39007612 "Using Human Intestinal Organoids..."
...
Done. Created: 13 Skipped: 0
Expected output for a re-run (idempotency check):
SKIP [already exists nid=174]: PMID:39007612 "Using Human Intestinal Organoids..."
...
Done. Created: 0 Skipped: 13
| Script | Runs on | Purpose |
|---|---|---|
scripts/fetch_metadata.py |
MacBook | Reads TSV → fetches PubMed/bioRxiv → writes articles.json |
scripts/load_biblio.php |
gcid-old | Reads articles.json → creates biblio nodes via Drush |
scripts/verify_nodes.php |
gcid-old | Spot-check: prints status and title for nids 174–186 |
scripts/spot_check.php |
gcid-old | Detailed dump of 4 sample nodes for validation |
All fields are direct properties on the $node object (not Field API arrays).
Full schema in docs/biblio_schema.md.
| JSON field | Drupal property | Table |
|---|---|---|
title |
$node->title |
{node} |
biblio_type_name |
$node->biblio_type (int) |
{biblio} |
year |
$node->biblio_year |
{biblio} |
journal |
$node->biblio_secondary_title |
{biblio} |
volume |
$node->biblio_volume |
{biblio} |
issue |
$node->biblio_issue |
{biblio} |
pages |
$node->biblio_pages |
{biblio} |
doi |
$node->biblio_doi |
{biblio} |
issn |
$node->biblio_issn |
{biblio} |
date_str |
$node->biblio_date |
{biblio} |
abstract |
$node->biblio_abst_e |
{biblio} |
pubmed_url |
$node->biblio_custom1 |
{biblio} |
pmid |
$node->biblio_pubmed_id |
{biblio_pubmed} |
pmcid |
$node->biblio_pmcid |
{biblio_pubmed} |
authors |
$node->biblio_contributors |
{biblio_contributor} |
Note: biblio_pubmed_md5 (in {biblio_pubmed}) has no database default
and must be set before node_save(). The loader sets it to md5('').
14 articles ingested total:
| nid | PMID / DOI | Type | Source |
|---|---|---|---|
| 174 | PMID 39007612 | Journal Article | March 2026 TSV |
| 175 | PMID 39389204 | Journal Article | March 2026 TSV |
| 176 | PMID 40594045 | Journal Article | March 2026 TSV |
| 177 | PMID 39820858 | Journal Article | March 2026 TSV |
| 178 | PMID 39652359 | Journal Article | March 2026 TSV |
| 179 | PMID 40228901 | Journal Article | March 2026 TSV |
| 180 | PMID 40445752 | Journal Article | March 2026 TSV |
| 181 | PMID 39522694 | Journal Article | March 2026 TSV |
| 182 | DOI 10.1101/2025.08.06.668939 | Unpublished | March 2026 TSV |
| 183 | PMID 40501838 | Unpublished | March 2026 TSV |
| 184 | PMID 39386636 | Unpublished | March 2026 TSV |
| 185 | PMID 40463537 | Unpublished | March 2026 TSV |
| 186 | PMID 41637508 | Unpublished | March 2026 TSV |
| 187 | PMID 41676723 | Unpublished | inputs/message-2026-04-28a.tsv |
Two non-obvious behaviors when using drush php-script:
-
File-level variables are not PHP globals. Variables defined at the top of the script (outside any function) are not accessible via
global $varinside functions. Usedefine()for constants or define values inside the function that uses them. -
$argvcontains Drush's own flags. When invoked asdrush --root=... php-script foo.php, the--root=flag occupies$argv[1]inside the script. Do not use$argvfor script arguments; usedefine()constants instead.
docs/biblio_schema.md— complete field schema discovered from the live sitedocs/publication_types.md— full list of Biblio type names and integersCLAUDE.md— execution environment, constraints, and rejected approachesplan.md— phased execution checklist (all phases complete except browser spot-check of the public publications page)