Skip to content

About

CLI tools and package to help with automating Request Tracker using its REST 1.0 API

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

RT Tools

A Python package and command-line tool for interacting with RT (Request Tracker) systems. Provides authenticated access to RT servers for querying ticket information, attachments, and other data.

Features

  • Complete Ticket Downloads: Download entire tickets with metadata, complete history, individual history items, and attachments
  • Chronological Transcript: --transcript writes ticket.md, the whole ticket as one readable Markdown file with attribution, ordering, and attachment names inline — a versioned, specified machine interface as well as a human one; -l/--lean writes only that plus the real attachments
  • Smart Attachment Processing: Automatically skips zero-byte attachments and outgoing emails, with automatic XLSX→TSV conversion of every worksheet
  • Recursive History Fetching: Handles broken RT API parameters with robust fallback methods
  • Persistent Authentication: Automatically manages RT session cookies, stored privately under ~/.secrets/rt-tools/
  • Portable Credentials: Reads the RT password from a ~/.secrets file on Linux (including the HPC) or the macOS keychain
  • SSL Certificate Verification: Custom certificate support for secure RT server connections
  • Flexible Logging: Configurable log levels (quiet, normal, verbose) for debugging and production use
  • Ticket Search: search-tickets filters by creation date and queue and writes TSV, ready to pipe into download-ticket
  • Writing to RT: create-ticket, comment-ticket, and reply-ticket create tickets and add comments or replies, with a preview, a confirmation prompt, and --dry-run
  • Command-line Interface: Multiple CLI tools for accessing RT ticket data and attachments

Installation

Installing with uv tool

To use the commands without a checkout or an activated virtualenv, install a release tag as a uv tool. uv puts the console scripts (download-ticket, search-tickets, create-ticket, and the rest) on your PATH, each in an isolated environment.

Tags are named vX.Y.Z. v1.2.x below is a placeholder, not a tag: replace it with the latest release from the tags page.

# Install a specific release
uv tool install git+https://github.com/BCM-HGSC/request-tracker-tools.git@v1.2.x

# Or over SSH
uv tool install git+ssh://git@github.com/BCM-HGSC/request-tracker-tools.git@v1.2.x

# Move an existing install to another release
uv tool install --force git+https://github.com/BCM-HGSC/request-tracker-tools.git@v1.2.x

# Remove it
uv tool uninstall rt-tools

If uv tool reports that the install directory is not on PATH, run uv tool update-shell once.

Development Installation

# Clone the repository
git clone <repository-url>
cd rt-tools

# Install in development mode with dev dependencies
pip install -e .[dev]

# Or using uv
uv pip install -e .[dev]

Note: rt-tools is not published on PyPI; install it from GitHub as shown above.

Note: The package includes openpyxl for automatic XLSX→TSV conversion of Excel attachments.

Configuration

Before using RT Tools, you need to make your RT password available through one of the sources below. RT Tools consults them in this order and uses the first one it finds:

  1. --password-file FILE command-line option (highest priority)
  2. $RT_PASSWORD_FILE environment variable
  3. Secret file: ~/.secrets/rt-tools/password, then ~/.secrets/rt
  4. macOS keychain, service name "foobar" (only where /usr/bin/security exists)

A file named by option 1 or 2 must exist; a missing one is an error rather than a reason to fall through to a later source.

Secret file (Linux, including the HPC)

mkdir -m 700 -p ~/.secrets/rt-tools
touch ~/.secrets/rt-tools/password
chmod 600 ~/.secrets/rt-tools/password
# then put the password on the first line

Only the first line is read, and its trailing newline is stripped. RT Tools refuses to read a secret file that has any group or other permission bit set — use mode 600 or 400.

macOS keychain

security add-generic-password -s "foobar" -a "your_username" -w "your_password"

Cookie file

Session cookies are written to ~/.secrets/rt-tools/cookies.txt, in a directory created mode 700 with the file created mode 600. The cookie is a bearer credential, so it is treated as a secret in its own right. Earlier versions wrote cookies.txt into the current working directory; that file is no longer used and can be deleted.

Note: The SSL certificate for RT server verification is bundled with the package and requires no manual setup.

Usage

Command Line Interface

download-ticket - Downloads complete tickets with metadata, history, and attachments:

# Download ticket to current directory (creates ./rt37525/)
download-ticket 37525

# Download to specific directory (creates local/output/rt37525/)
download-ticket 37525 --output-dir local/output

# Also write ticket.md, the whole ticket as one chronological Markdown file
download-ticket 37525 --transcript

# Transcript plus nothing the transcript already covers
download-ticket 37525 --transcript --prune

# ...and none of the HTML twins either: ticket.md plus real attachments
download-ticket 37525 --lean

# Re-download over an existing tree, dropping what this run did not write
download-ticket 37525 --lean --clean

# Write the contents directly into a fixed path, with no rt37525 level
download-ticket 37525 --into work/submission-a/ticket --transcript

# Download multiple tickets (authenticate once, loop sequentially)
download-ticket 37525 37526 37527

# From clipboard or file via xargs
pbpaste | xargs download-ticket
cat tickets.txt | xargs download-ticket

# With verbose logging to see download progress
download-ticket --verbose 37525

# With quiet mode for minimal output
download-ticket --quiet 37525 --output-dir /tmp

# With an explicit password file (all four commands accept this)
download-ticket 37525 --password-file ~/.secrets/rt-tools/password

Target Directory Resolution: The download-ticket command resolves the output directory in the following order:

  1. --output-dir command-line option (highest priority)
  2. $DOWNLOAD_TICKET_DIR environment variable
  3. ~/.config/download-ticket/config.toml config file (default_dir setting)
  4. Current working directory (fallback)

--into DIR bypasses this resolution entirely: the ticket contents are written straight into DIR, with no rt{ticket_id} level. It is mutually exclusive with --output-dir and takes a single ticket ID.

# Environment variable example
export DOWNLOAD_TICKET_DIR="~/Downloads/rt-tickets"
download-ticket 37525  # Creates ~/Downloads/rt-tickets/rt37525/

# Config file example (~/.config/download-ticket/config.toml)
# default_dir = "~/Documents/rt-data"
download-ticket 37525  # Creates ~/Documents/rt-data/rt37525/

Directory Structure:

output_directory/         # Resolved from --output-dir, env var, config, or cwd
└── rt37525/              # Ticket directory (rt{ticket_id} format)
    ├── ticket.md         # Chronological transcript (--transcript/--lean only)
    ├── metadata.txt      # Ticket basic information
    ├── history.txt       # Complete ticket history
    ├── attachments.txt   # Attachment index
    ├── 456/              # History entry directory
    │   ├── message.txt   # Full RT history entry (raw format)
    │   └── content.txt   # New content only — quoted replies stripped
    ├── 458/              # Another history entry directory
    │   ├── message.txt
    │   ├── content.txt
    │   ├── n800.pdf      # Attachment for this history entry
    │   └── n801.xlsx     # Also saved as n801.tsv (auto-converted)
    └── ...

Features:

  • Organized structure: Each history entry and its attachments are grouped in individual directories
  • Consistent filtering: Automatically skips zero-byte attachments and outgoing emails from both attachments and individual history items
  • Uses recursive history fetching to handle broken RT API parameters
  • Downloads attachments with format: n{attachment_id}.{extension} within each history directory (the "n" prefix ensures message.txt sorts first)
  • Individual history items: Each history entry is saved as {history_id}/message.txt (full raw entry) and {history_id}/content.txt (new content only, with quoted replies stripped). content.txt is the primary file for automated and human processing. RT's leading # N/M (id/.../total) counter is stripped from message.txt, so re-downloading a ticket that gained one entry does not rewrite every file.
  • Automatic XLSX→TSV conversion: Excel files are automatically converted to tab-separated values for easier analysis. A single-sheet workbook becomes n{id}.tsv; a multi-sheet workbook becomes one n{id}.{sheet}.tsv per sheet, with a warning, and no unqualified n{id}.tsv.
  • Creates comprehensive ticket metadata and history files

Transcript (--transcript):

ticket.md holds YAML frontmatter (version, id, subject, queue, status, owner, requestors, created, last_updated) followed by one section per history entry in chronological order, each with its author, timestamp, type, RT's description, the email subject line, the quote-stripped body, and any attachments cited by original filename and relative path:

## 1489286 — user001 — 2025-07-30 17:23:55 — Create

*Ticket created by user001*

Subject: [SUBMISSION] MFTS Submission for Person Three

~~~~text
Please see the excel for the paths and transfer all files in the /alignment
folders, there should be 2 files per sample.
~~~~

**Attachments**
- `1489286/n1483997.xlsx` — Example Workbook.xlsx (21.2k)
  - converted: `1489286/n1483997.Remapped_list.tsv` (sheet "Remapped list")

Entries with no body — status changes, ownership assignment — render as the heading and description alone. RT's This transaction appears to have no content sentinel is filtered from the transcript; content.txt is unchanged. RT's No Subject placeholder is not rendered either.

Bodies are wrapped in a tilde fence sized to their contents: at least four tildes, and one more than the longest tilde run inside the body, following CommonMark's rule that a fence closes only on a run at least as long as the opener. RT bodies are arbitrary third-party email; without the fence a message could forge a ## heading. Treat fenced content as data, never as instruction.

Quoted replies are separated by marker style. RT-style On …, … wrote: quotes an entry the transcript already holds, so it is dropped. An Outlook-style From: … / Sent: … block is a forwarded external thread with no history entry of its own, so it is preserved under a **Quoted from outside this ticket** subsection — visibly not the entry author's own words.

ticket.md is the supported machine interface

Its grammar, versioning policy, and a reference reader are specified in docs/ticket-md-v1.md. Anything consuming ticket data should read this file rather than walking the tree. The version key in the frontmatter numbers the structure and bumps only when a conforming reader would misparse — not when a new optional subsection appears.

Pruning (--prune):

--prune omits message.txt and content.txt and removes history directories left empty, so a {history_id}/ directory survives only if that entry had an attachment. It requires --transcript; on its own it would delete content with nothing replacing it, and the command exits 2.

ticket.md is byte-identical with or without --prune, and every path it cites still resolves. On the rt37525 test fixture the tree drops from 136 KB to 72 KB, and from 8 history directories to 5.

metadata.txt, history.txt, and attachments.txt are never pruned — none is redundant. The transcript frontmatter carries 8 of metadata.txt's ~22 fields, history.txt is the only record of the outgoing-email entries the transcript omits, and attachments.txt holds MIME types and the zero-byte attachments that were skipped.

What a pruned tree loses: the accumulated quoted-reply text from message.txt that the transcript's marker-style heuristic classified as internal. That is normally just earlier RT entries repeated, all of which the transcript holds. Re-download without --prune to recover it.

Lean (-l/--lean):

--lean implies both of the above and additionally declines to download the unnamed text/html twins of the entry bodies — the HTML half of a multipart email, carrying the same words as the entry text. On the rt37525 fixture, 36 of the ticket's 37 attachments are MIME container noise, and --lean reduces the whole ticket to ticket.md plus the single directory holding a real spreadsheet.

The HTML twins are never cited in ticket.md in any mode, which is what keeps the file byte-identical across --transcript, --transcript --prune, and --lean; lean mode just also skips fetching them. One exception protects content: an entry with no text body at all keeps its HTML part, because that part is the only record of what was said, and the transcript cites it normally.

This is the everyday mode. Making it the default is a 2.0 change, at which point today's full output moves behind a --debug flag.

Clean (-c/--clean):

Downloads are otherwise additive: a file an earlier run wrote survives even when RT no longer has it, and re-downloading --lean over a full tree keeps every HTML twin --lean declined to fetch — so the mode appears to do nothing. --clean closes that gap. After a download that completed, it deletes the files this tool itself writes that this run did not produce:

  • ticket level — metadata.txt, history.txt, attachments.txt, ticket.md
  • per entry — message.txt, content.txt, n{attachment_id}.*

...then removes any history directory that leaves empty. Deletion is driven by the set of paths the run actually wrote, so the n{id}.{sheet}.tsv files a multi-sheet workbook just produced are never mistaken for orphans, and a run that aborted deletes nothing at all.

It is not rsync --delete. A file the downloader never writes — an analysis.yaml, a hand-written note, whatever an --into DIR already held — is left alone. The downloader cleans up after itself, and only after itself.

The property this buys: download-ticket ID --lean --clean over any existing tree gives the same bytes as a fresh --lean download into an empty directory. At 2.0 this becomes unconditional rather than gaining an inverse flag.

A ticket that does not exist aborts before anything is written. RT reports that with HTTP 200 and # Ticket N does not exist. as the body, so it has to be recognized in the payload rather than the status.

Do not point rt-analysis at a pruned or lean tree while it still walks the tree. Its lib.read_content() returns "" for a missing content.txt, so the extraction scripts degrade to subject-line-only prompts silently rather than failing. The fix is for it to read ticket.md. rt-sanitizer and text-processing select files by extension and are unaffected.

search-tickets - Searches tickets and writes TSV to stdout:

# Tickets created in August, in either of the two default queues
search-tickets --start-date 2026-08-01 --end-date 2026-08-31

# One queue, open-ended
search-tickets --queue sub --start-date 2026-09-01

# Feed the IDs to another command
search-tickets --queue sub --start-date 2026-09-01 | tail -n +2 | cut -f1 | xargs download-ticket

Filters on Created, inclusive at both ends, and on queue. --queue is repeatable and the values are OR'd; mft and sub are aliases for "Managed File Transfer" and "Submissions". Any other value is a literal RT queue name, matched case-insensitively against RT's queue list — an unknown name exits 2 and logs the queues that do exist. Omitting --queue searches both aliases.

Output is TSV with a header row: id, subject, status, created, last_updated, owner, queue. Tabs and newlines inside a field are collapsed to single spaces so each ticket occupies exactly one line. All statuses are included, resolved ones among them.

dump-ticket - Retrieves and displays RT ticket information:

# Basic ticket information
dump-ticket 37525

# Ticket with additional path components (e.g., attachments)
dump-ticket 37525 attachments/1483996/content

# With verbose logging (shows authentication details, headers, etc.)
dump-ticket --verbose 37525

# With quiet mode (only errors and warnings)
dump-ticket --quiet 37525

dump-rest - Retrieves content from RT REST API URLs (relative to REST/1.0 endpoint):

# List all tickets
dump-rest

# Show specific ticket
dump-rest ticket/37525/show

# Access ticket attachments
dump-rest ticket/37525/attachments

# With logging options
dump-rest --verbose ticket/37525/show
dump-rest --quiet user/username

dump-url - Retrieves content from arbitrary RT URLs (relative to server base):

# Access REST API directly
dump-url REST/1.0/

# Access specific RT paths
dump-url NoAuth/css/base/main.css

# With logging options
dump-url --verbose REST/1.0/ticket/37525/show
dump-url --quiet some/path

open-ticket - Opens tickets in the web UI. Unlike the other commands it creates no session and needs no password: the browser already holds RT's cookie.

# Open one ticket
open-ticket 37525

# Open several at once
open-ticket 37525 37526

Writing to RT

Three commands write. They are separate rather than one command with an --action flag because comment and correspond hit the same RT endpoint but differ in who receives email, which is too large a difference to hang on a flag.

create-ticket - Creates a ticket from a content block file:

# The file is the content block itself
create-ticket new-ticket.txt

# Preview the exact POST body; send nothing
create-ticket new-ticket.txt --dry-run

# Override individual fields without editing the file
create-ticket new-ticket.txt --queue Submissions --subject "Delivery of WGS data"

The file holds Field: value one per line, with continuation lines indented by one space and # lines ignored. Queue is required; the body is Text:

# Lines starting with # are ignored.
Queue: Submissions
Subject: Delivery of WGS data
Requestor: someone@example.org
Text: The first paragraph of the ticket body.

 A second paragraph. A blank line works the way you would expect; RT's own
 encoding needs a line holding one space, and that is handled for you.

Optional fields include Cc, AdminCc, Owner, Status, Priority, Starts, Due, and CF-{Name} for custom fields. Omitting Owner leaves assignment to RT, which is the ordinary case.

comment-ticket - Adds an internal comment. Mails nobody:

comment-ticket 37525 -m "Checked the manifest; 3 samples are missing."

# Body from a file, or from stdin
comment-ticket 37525 --body notes.md
generate-report | comment-ticket 37525 --body -

# With no -m and no --body, opens $EDITOR
comment-ticket 37525

reply-ticket - Replies to the ticket. RT mails the requestors, and a new ticket becomes open:

reply-ticket 37525 -m "Your data is available at the path below."

# Cc this reply only
reply-ticket 37525 -m "See attached path list." --cc colleague@example.org

# Record time worked alongside the reply
reply-ticket 37525 --body reply.md --time-worked 30

Before anything is sent, all three print what will be sent and ask:

RT write: reply to ticket 37525
endpoint: POST ticket/37525/comment
!! this sends email
--- content
id: 37525
Action: correspond
Text: Your data is available at the path below.
--- end
Send this to RT? [y/N]
  • -n/--dry-run prints that preview and exits without sending.
  • -y/--yes skips the prompt.
  • When stdin is not a terminal, the commands refuse unless -y is given. A prompt nobody can answer is not consent, so a script that would silently mail requestors fails instead.

Attachments are not yet supported on writes.

Python API

from rt_tools import RTSession, get_ticket_statuses

# Create an authenticated session
with RTSession() as session:
    session.authenticate()

    # Check status of one or more tickets
    statuses = get_ticket_statuses(["37525", "37526"], session)
    # {"37525": "open", "37526": "resolved"}

    # Access ticket data programmatically
    response = session.get("https://rt.hgsc.bcm.edu/REST/1.0/ticket/37525")
    print(response.text)

get_ticket_statuses(ticket_ids, session) fetches ticket status via the REST API and returns a dict mapping each ticket ID to "open" (new/open/stalled), "resolved", or "unknown".

Architecture

Core Components

  • RTSession: Extends requests.Session with RT-specific authentication and cookie management
  • Authentication: Automatic login using stored credentials with session persistence
  • credentials: Resolves the password across secret files and the keychain, and enforces private permissions on every secret it reads or writes
  • Cookie Management: Mozilla-format cookie jar for maintaining authentication across sessions
  • Logging: Structured logging with configurable levels

Authentication Flow

  1. Loads existing cookies from ~/.secrets/rt-tools/cookies.txt if available
  2. Checks authorization status by parsing RT server responses
  3. If unauthorized, resolves the password from a secret file or the macOS keychain
  4. Performs authentication POST request and saves new cookies mode 600
  5. Subsequent requests use stored authentication cookies

URL Construction

RT URLs are constructed as: BASE_URL/ticket/{id}/{path_components...}

  • Base URL: https://rt.hgsc.bcm.edu/REST/1.0/
  • Ticket ID: Numeric identifier (e.g., 37525)
  • Path components: Optional additional paths (e.g., attachments/1483996/content)

Development

Code Quality

# Run linting
ruff check src/rt_tools/

# Auto-fix issues
ruff check --fix src/rt_tools/

# Run tests
pytest

# Install pre-commit hooks (one-time setup)
uv pip install pre-commit
pre-commit install

# Run pre-commit on all files
pre-commit run --all-files

Pre-commit hooks are configured to run:

  • Ruff linting and formatting
  • MyPy type checking
  • Basic file hygiene (trailing whitespace, end-of-file fixes)
  • YAML/TOML validation
  • Large file detection

Binary test fixtures (.bin files) are excluded from text processing hooks.

Building

# Build package
python -m build

Logging Levels

  • Default (INFO): Shows basic operation status and responses
  • Verbose (DEBUG): Shows detailed request/response information, headers, authentication steps
  • Quiet (WARNING): Shows only warnings and errors, suppresses routine operational messages

Security

  • Passwords are never stored in code or configuration files — they come from the macOS keychain or a private file under ~/.secrets
  • Secret files must be mode 600 or 400; a file readable by group or other is rejected with an error
  • Cookie files are created mode 600 in a mode 700 directory, and tightened on load if they are found to be more permissive
  • SSL certificate verification prevents man-in-the-middle attacks
  • Session cookies are stored locally and reused to minimize authentication requests

On a shared filesystem such as an NFS-mounted HPC home directory, Unix permissions are the only barrier protecting a secret file. Storage administrators and node root can read it. Use this mechanism only for credentials where that exposure is acceptable.

Requirements

  • Python 3.13+
  • macOS or Linux (the keychain is used on macOS; Linux uses a ~/.secrets file)
  • Network access to RT server
  • Valid RT user credentials

License

This project is licensed under the MIT License - see the LICENSE file for details.

About

CLI tools and package to help with automating Request Tracker using its REST 1.0 API

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages