Skip to content

feat: support custom run dir, tmpfs #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
flotwig wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: support custom run dir, tmpfs #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ docker run --detach \
| SHARED_RSYNC_THREADS | shared:rsync_threads | | `20` |
| WARRIOR_ID | warrior_id | | |
| CONCURRENT_ITEMS | concurrent_items | | `3` |
| WARRIOR_RUN_DIR | | | `/home/warrior` |

## Other Ways to Run

Expand Down Expand Up @@ -132,3 +133,15 @@ First edit the `docker-compose.yml` file with any configuration keys (as describ
cd examples
docker compose up -d
```

#### Run using tmpfs

The Warrior uses disk as scratch space. You may want to use a `tmpfs` mount to [prevent unnecessary wear and tear](https://github.com/ArchiveTeam/warrior-dockerfile/issues/83) if you have RAM to spare. Docker Compose example:

```
warrior:
environment:
WARRIOR_RUN_DIR: /warrior-tmp
tmpfs:
- /warrior-tmp:exec,mode=777,uid=1000,gid=1000
```
16 changes: 13 additions & 3 deletions start.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,18 @@
import json
import os
import subprocess
import shutil

CONFIG_FILE = 'projects/config.json'
DEFAULT_RUN_DIR = '/home/warrior'
WARRIOR_RUN_DIR = os.getenv('WARRIOR_RUN_DIR', DEFAULT_RUN_DIR)
CONFIG_FILE = os.path.join(WARRIOR_RUN_DIR, 'projects/config.json')

# copy data/projects from image to the custom run dir
if WARRIOR_RUN_DIR != DEFAULT_RUN_DIR:
for dir_name in ['data', 'projects']:
dst = os.path.join(WARRIOR_RUN_DIR, dir_name)
src = os.path.join(DEFAULT_RUN_DIR, dir_name)
shutil.copytree(src, dst)
Copy link
Copy Markdown

@damian-anlauf damian-anlauf Jul 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The container currently crashes on startup, if the WARRIOR_RUN_DIR is not empty.
Adding dirs_exist_ok=True to shutil.copytree would fix the issue.

Copy link
Copy Markdown

@Jalle19 Jalle19 Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed:

Traceback (most recent call last):
  File "/home/warrior/start.py", line 15, in <module>
    shutil.copytree(src, dst)
  File "/usr/local/lib/python3.9/shutil.py", line 568, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/local/lib/python3.9/shutil.py", line 467, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/scratch/data'

Copy link
Copy Markdown
Author

@flotwig flotwig Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#90 (comment)


try:
with open(CONFIG_FILE, 'r') as fp:
Expand All @@ -24,9 +34,9 @@
[
"run-warrior3",
"--projects-dir",
"/home/warrior/projects",
os.path.join(WARRIOR_RUN_DIR, "projects"),
"--data-dir",
"/home/warrior/data",
os.path.join(WARRIOR_RUN_DIR, "data"),
"--warrior-hq",
"https://warriorhq.archiveteam.org",
"--port",
Expand Down