Full-stack project for collecting articles from different sources.
- SQLAlchemy is now configured with the async provider (
AsyncEngine+AsyncSession) - FastAPI routes and crawl service now use async DB access
- Scheduler was moved to
AsyncIOScheduler - Alert keywords are no longer attached to a specific source
- Keywords are stored globally in
monitored_keywordsand can be managed from the dashboard
- Backend: Python, FastAPI, SQLAlchemy, Alembic, PostgreSQL, APScheduler, Elasticsearch
- Frontend: Vue 3, TypeScript, Vuetify, Pinia, Vue Router, Axios
- Infra: Docker Compose
docker compose up --buildServices:
- Backend: http://localhost:8000
- Frontend: http://localhost:5173
- Elasticsearch: http://localhost:9200
- Postgres: localhost:5432
cd backend
python -m venv .venv
source .venv/bin/activate # Linux/macOS
pip install -r requirements.txt
cp .env.example .env-dev
APP_MODE=dev alembic upgrade head
APP_MODE=dev uvicorn app.main:app --reloadThe app uses JWT access tokens, rotating refresh tokens, role-based access control, and row ownership. Start with these docs:
sequenceDiagram
participant UI as Vue frontend
participant API as FastAPI backend
participant DB as PostgreSQL
UI->>API: POST /api/auth/login (email, password)
API->>DB: Load user, roles, permissions
API->>DB: Store refresh token jti as ACTIVE
API-->>UI: access_token + refresh_token
UI->>API: Request with Authorization bearer access_token
API->>API: Decode JWT and load current user
API->>DB: Load AuthorizationContext
API-->>UI: Protected response
API-->>UI: 401 when access token expires
UI->>API: POST /api/auth/refresh
API->>DB: Rotate refresh token
API-->>UI: New access_token + refresh_token
The frontend retries non-auth requests once after a successful refresh. Because refresh tokens rotate, concurrent refresh attempts are collapsed into a single in-flight request.
Core backend env vars live in backend/.env.example:
DATABASE_URL=postgresql+asyncpg://postgres:postgres@postgres:5432/news_monitor
ELASTICSEARCH_URL=http://elasticsearch:9200
CORS_ORIGINS=http://localhost:5173
DEFAULT_KEYWORDS=Auth-related settings have development defaults in backend/app/core/config.py, but set them explicitly for shared environments:
SECURITY_KEY=change-me
ALGORITHM=HS256
ACCESS_TTL_MINUTES=660
REFRESH_TTL_MINUTES=10080
ISSUER=News CrawlerSECURITY_KEY must be a real secret outside local development.
When APP_MODE is not prod, seed data is created during app startup through app.db.seed_data.seed_data(). The default local admin credentials are:
- Email/login username:
admin@user.com - Username:
admin - Password:
Admin123!
The seeded admin role bypasses permission checks. The seeded manager and user roles start without permissions, and new registrations do not receive a role automatically, so assign roles from /admin/users.
For an existing database, run seeding before applying the owner backfill migration because existing rows are assigned to the seeded admin user:
cd backend
alembic upgrade headPermissions are named resource:action:scope; examples: source:read:own, article:read:any. Resources, actions, and scopes are validated in backend/app/core/permission_types.py, and the admin UI list is backed by frontend/src/stores/admin.ts.
To add a new permission category:
- Add or confirm the resource/action/scope enum values in
backend/app/core/permission_types.py. - Add the resource/action pair to
backend/app/services/resource_actions/resource_actions.pyso the catalog exposes it. - Protect the route with
RequiredPermissionsAndOwnership("resource:action:scope", mode=PermissionMode.ANY, resource_type=...)when ownership checks are needed. - Use
filter_owned_resources()or equivalent repository filtering for list/search reads. - Add tests for allowed, forbidden, and owner-mismatch cases.
- Create or assign the permission from
/admin/roles.
cd frontend
npm install
npm run devHelm chart: k8s/news-crawler
The chart deploys:
- frontend
- backend
- PostgreSQL
- Elasticsearch
- optional Kibana
Important deployment note:
- keep
backend.replicaCount=1unless you split the scheduler out of the API process, because APScheduler starts inside the backend application lifecycle and would run once per replica
Render locally:
helm template news-crawler ./k8s/news-crawlerAKS workflow:
- GitHub Actions workflow:
.github/workflows/deploy-aks.yml - required repository variable placeholders:
AZURE_RESOURCE_GROUP,AZURE_AKS_CLUSTER_NAME,AZURE_CONTAINER_REGISTRY,AKS_INGRESS_HOST - required repository secret placeholders:
AZURE_CREDENTIALS,POSTGRES_PASSWORD
This initial version uses straightforward HTML parsing:
- the homepage is crawled to discover article links
- article pages are parsed for title, date, body, author, and tags when available
- duplicate articles are avoided by
source_id + external_idandurl
The parser is intentionally heuristic so you can adapt it quickly if the site layout changes.