Add import-mbox and import-emlx commands for local email import (#103)

## Summary

- Add `import-mbox` command for importing MBOX files (including zipped
archives from HEY.com, Google Takeout, etc.)
- Add `import-emlx` command for importing Apple Mail `.emlx` mailbox
directory trees
- Both importers share a common ingestion path (`IngestRawMessage`) with
checkpoint/resume, attachment extraction, and FTS5 indexing
- Fix invalid UTF-8 in Message-IDs, email addresses, and domains during
ingestion so DuckDB Parquet cache builds succeed
- Extend `repair-encoding` to cover
`conversations.source_conversation_id`, `participants.email_address`,
and `participants.domain`

## What's in this PR

### `import-mbox`
- New CLI: `msgvault import-mbox <identifier> <export-file>`
- Accepts `.mbox/.mbx` or a `.zip` containing `.mbox` files
- Records imported mail under a configurable `--source-type` (example:
`--source-type hey`)
- Optional `--label` to apply a label to newly imported messages
- Interrupt-safe imports with checkpoints and resume (default), with
`--no-resume` to start fresh
- Deterministic ZIP extraction to `dataDir/imports/mbox/<sha256(zip)>`
with a `.done` sentinel and stable ordering
- Streaming MBOX reader (`internal/mbox`) with mboxrd-style unescaping

### `import-emlx`
- New CLI: `msgvault import-emlx <identifier> <path>`
- Discovers `.mbox`/`.imapmbox` directories in Apple Mail directory
trees
- Parses `.emlx` format (byte-count prefix + MIME + XML plist metadata)
- Derives labels from directory paths, stripping `.mbox`/`.imapmbox`
suffixes
- Deduplicates identical messages across mailboxes (one message,
multiple labels)
- Checkpoint/resume with the same patterns as `import-mbox`

### Shared ingestion (`internal/importer`)
- Extracted `IngestRawMessage` so both importers share the same parsing,
recipient storage, attachment, and FTS5 indexing path
- UTF-8 sanitization on all address fields (Message-ID, email, domain,
display name) to prevent DuckDB errors during Parquet cache builds
- `repair-encoding` extended with three new fields for fixing existing
bad data

### Bug fixes and refactoring
- Fix attachment cache unbounded growth (key by path+size+hash)
- Fix `sourceMsgID` stability (content-based instead of
file-discriminator-based)
- Fix `MarkMessageDeletedByGmailID` to respect permanent flag
- Extract mbox zip handling into `internal/importer/mboxzip` package
- Various refactoring in `main.go`, `export_eml.go`,
`store_attachment.go`, `messages.go`

## Test plan

- [x] `make test` passes
- [x] `go vet ./...` passes
- [x] End-to-end CLI tests for both `import-mbox` and `import-emlx`
- [x] Unit tests for MBOX reader, EMLX parser, mailbox discovery,
ingestion UTF-8 sanitization
- [ ] Manual test: `./msgvault import-mbox you@hey.com
/path/to/hey-export.zip --source-type hey --label hey`
- [ ] Manual test: `./msgvault import-emlx me@icloud.com
~/Library/Mail/V10`
- [ ] Manual test: `./msgvault repair-encoding` followed by `./msgvault
build-cache --full-rebuild`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Wes McKinney <wesmckinn+git@gmail.com>

Author: 3 people

.githooks/pre-commit

@@ -18,9 +18,9 @@ if [ -n "$UNFORMATTED" ]; then
     exit 1
 fi
 
-# Run linter
+# Run linter (only check new issues vs HEAD)
 echo "Running linter..."
-if ! golangci-lint run ./... 2>&1; then
+if ! golangci-lint run --new-from-rev=HEAD ./... 2>&1; then
     echo ""
     echo "Lint failed. Fix errors before committing."
     exit 1

.gitignore

@@ -23,6 +23,7 @@ oauth_client*.json
 # Local development state
 .beads/
 .githooks/post-commit
+.githooks/post-rewrite
 .mcp.json
 
 # Python

README.md

@@ -15,11 +15,12 @@ Archive a lifetime of email. Analytics and search in milliseconds, entirely offl
 
 Your messages are yours. Decades of correspondence, attachments, and history shouldn't be locked behind a web interface or an API. msgvault downloads a complete local copy and then everything runs offline. Search, analytics, and the MCP server all work against local data with no network access required.
 
-Currently supports Gmail, with WhatsApp and other messaging platforms planned.
+Currently supports Gmail sync, plus offline imports from MBOX exports (including HEY.com).
 
 ## Features
 
 - **Full Gmail backup**: raw MIME, attachments, labels, and metadata
+- **MBOX import**: import email exports from providers without IMAP/POP access (e.g. HEY.com)
 - **Interactive TUI**: drill-down analytics over your entire message history, powered by DuckDB over Parquet
 - **Full-text search**: FTS5 with Gmail-like query syntax (`from:`, `has:attachment`, date ranges)
 - **MCP server**: access your full archive at the speed of thought in Claude Desktop and other MCP-capable AI agents
@@ -86,12 +87,22 @@ msgvault tui
 | `stats` | Show archive statistics |
 | `verify EMAIL` | Verify archive integrity against Gmail |
 | `export-eml` | Export a message as `.eml` |
+| `import-mbox` | Import email from an MBOX export (HEY has no IMAP/POP) |
 | `build-cache` | Rebuild the Parquet analytics cache |
 | `repair-encoding` | Fix UTF-8 encoding issues |
 | `list-senders` / `list-domains` / `list-labels` | Explore metadata |
 
 See the [CLI Reference](https://msgvault.io/cli-reference/) for full details.
 
+## Importing HEY.com
+
+HEY exports mail as MBOX (often delivered as a `.zip`). Import it using `import-mbox`:
+
+```bash
+msgvault init-db
+msgvault import-mbox you@hey.com /path/to/hey-export.zip --source-type hey --label hey
+```
+
 ## Configuration
 
 All data lives in `~/.msgvault/` by default (override with `MSGVAULT_HOME`).

SECURITY.md

@@ -27,6 +27,7 @@ If you discover a security vulnerability in msgvault, please report it responsib
 **File permissions:**
 - OAuth token files created with 0600 permissions (owner read/write only)
 - Config directory (`~/.msgvault/`) should be 0700
+- Attachment storage directory (`~/.msgvault/attachments/`) is created with 0700; attachment files are 0600
 - Cross-platform support including Windows DACL
 
 **SQL injection prevention:**

cmd/msgvault/cmd/export_eml.go

@@ -2,15 +2,20 @@ package cmd
 
 import (
 	"fmt"
-	"os"
 	"strconv"
+	"strings"
 
 	"github.com/spf13/cobra"
 	"github.com/wesm/msgvault/internal/fileutil"
 	"github.com/wesm/msgvault/internal/query"
 	"github.com/wesm/msgvault/internal/store"
 )
 
+const (
+	stdoutSentinel = "-"
+	emlFileMode    = 0o600
+)
+
 var (
 	exportEMLOutput string
 )
@@ -29,75 +34,88 @@ Examples:
   msgvault export-eml 18f0abc123def -o important.eml`,
 	Args: cobra.ExactArgs(1),
 	RunE: func(cmd *cobra.Command, args []string) error {
-		idStr := args[0]
+		return runExportEML(cmd, args[0], exportEMLOutput)
+	},
+}
+
+type resolvedMessage struct {
+	ID              int64
+	SourceMessageID string
+}
 
-		// Open database
-		dbPath := cfg.DatabaseDSN()
-		s, err := store.Open(dbPath)
+func resolveMessage(engine *query.SQLiteEngine, cmd *cobra.Command, messageRef string) (resolvedMessage, error) {
+	if id, err := strconv.ParseInt(messageRef, 10, 64); err == nil {
+		msg, err := engine.GetMessage(cmd.Context(), id)
 		if err != nil {
-			return fmt.Errorf("open database: %w", err)
+			return resolvedMessage{}, fmt.Errorf("get message: %w", err)
 		}
-		defer s.Close()
-
-		// Create query engine to look up the message
-		engine := query.NewSQLiteEngine(s.DB())
-
-		// Try to parse as numeric ID first
-		var msgID int64
-		var sourceMessageID string
-		if id, err := strconv.ParseInt(idStr, 10, 64); err == nil {
-			msgID = id
-			// Get source message ID for output filename
-			msg, err := engine.GetMessage(cmd.Context(), id)
-			if err != nil {
-				return fmt.Errorf("get message: %w", err)
-			}
-			if msg == nil {
-				return fmt.Errorf("message not found: %s", idStr)
-			}
-			sourceMessageID = msg.SourceMessageID
-		} else {
-			// It's a source message ID (Gmail ID)
-			sourceMessageID = idStr
-			msg, err := engine.GetMessageBySourceID(cmd.Context(), idStr)
-			if err != nil {
-				return fmt.Errorf("get message: %w", err)
-			}
-			if msg == nil {
-				return fmt.Errorf("message not found: %s", idStr)
-			}
-			msgID = msg.ID
+		if msg == nil {
+			return resolvedMessage{}, fmt.Errorf("message not found: %s", messageRef)
 		}
+		return resolvedMessage{ID: id, SourceMessageID: msg.SourceMessageID}, nil
+	}
 
-		// Get the raw MIME data
-		rawData, err := s.GetMessageRaw(msgID)
-		if err != nil {
-			return fmt.Errorf("get raw message data: %w (message may not have raw data stored)", err)
-		}
+	msg, err := engine.GetMessageBySourceID(cmd.Context(), messageRef)
+	if err != nil {
+		return resolvedMessage{}, fmt.Errorf("get message: %w", err)
+	}
+	if msg == nil {
+		return resolvedMessage{}, fmt.Errorf("message not found: %s", messageRef)
+	}
+	return resolvedMessage{ID: msg.ID, SourceMessageID: msg.SourceMessageID}, nil
+}
 
-		// Determine output filename
-		outputPath := exportEMLOutput
-		if outputPath == "" {
-			outputPath = fmt.Sprintf("%s.eml", sourceMessageID)
+func sanitizeEMLFilename(sourceMessageID string) string {
+	safe := strings.Map(func(r rune) rune {
+		if r == '/' || r == '\\' || r == '\x00' {
+			return '_'
 		}
+		return r
+	}, sourceMessageID)
+	if safe == "" {
+		safe = "message"
+	}
+	return safe + ".eml"
+}
 
-		// Write to file or stdout
-		if outputPath == "-" {
-			_, err = os.Stdout.Write(rawData)
-			return err
-		}
+func runExportEML(cmd *cobra.Command, messageRef, outputPath string) error {
+	dbPath := cfg.DatabaseDSN()
+	s, err := store.Open(dbPath)
+	if err != nil {
+		return fmt.Errorf("open database: %w", err)
+	}
+	defer s.Close()
 
-		err = fileutil.SecureWriteFile(outputPath, rawData, 0600) // Restricted permissions for email content
-		if err != nil {
-			return fmt.Errorf("write file: %w", err)
-		}
+	engine := query.NewSQLiteEngine(s.DB())
 
-		fmt.Printf("Exported message to: %s (%d bytes)\n", outputPath, len(rawData))
-		return nil
-	},
+	resolved, err := resolveMessage(engine, cmd, messageRef)
+	if err != nil {
+		return err
+	}
+
+	rawData, err := s.GetMessageRaw(resolved.ID)
+	if err != nil {
+		return fmt.Errorf("get raw message data: %w (message may not have raw data stored)", err)
+	}
+
+	if outputPath == "" {
+		outputPath = sanitizeEMLFilename(resolved.SourceMessageID)
+	}
+
+	if outputPath == stdoutSentinel {
+		_, err = cmd.OutOrStdout().Write(rawData)
+		return err
+	}
+
+	if err := fileutil.SecureWriteFile(outputPath, rawData, emlFileMode); err != nil {
+		return fmt.Errorf("write file: %w", err)
+	}
+
+	cmd.Printf("Exported message to: %s (%d bytes)\n", outputPath, len(rawData))
+	return nil
 }
 
 func init() {
 	rootCmd.AddCommand(exportEMLCmd)
-	exportEMLCmd.Flags().StringVarP(&exportEMLOutput, "output", "o", "", "Output file path (default: <gmail_id>.eml, use - for stdout)")
+	exportEMLCmd.Flags().StringVarP(&exportEMLOutput, "output", "o", "", "Output file path (default: <source_message_id>.eml, use - for stdout)")
 }