feat(seo): add robots.txt

[prudentbird]

Summary

Adds a generated robots.txt (src/app/robots.ts) so crawlers get explicit guidance.

• Allows all public pages.
• Disallows /api/, /admin, and /user (gated/authenticated areas) to save crawl budget.
• Points to https://retailytics.ajared.ng/sitemap.xml (added in the next PR, feat(seo): add sitemap.xml #108).

Auth pages (/login, /register, /forgot-password) are intentionally left crawlable so their noindex,follow meta is honored.

Done when (issue #107)

• robots.txt is served, allows public pages, blocks API and gated/authenticated routes, and points to the sitemap.

Validation

• npm run typecheck ✅
• npm run build ✅ — /robots.txt route generated.

Notes

Stacked on #112 (targets feat/seo-112-footer-legal-pages).

Closes #107

[claude]

Claude finished @prudentbird's task —— View job

---

PR Review: feat(seo): add robots.txt

Reviewing this PR for code quality, best practices, potential bugs, performance considerations, security concerns, and test coverage.

Todo List:

• Read repository guidelines (CLAUDE.md) - No CLAUDE.md found
• Analyze the robots.ts implementation
• Check code quality and Next.js best practices
• Review security considerations
• Assess performance implications
• Evaluate test coverage needs
• Provide comprehensive feedback

🟢 Overall Assessment

This is a well-implemented robots.txt solution that follows Next.js App Router conventions correctly. The code is clean, follows TypeScript best practices, and addresses the SEO requirements effectively.

✅ Strengths

Code Quality & Best Practices:

• Proper use of Next.js 16+ MetadataRoute.Robots type from next/types
• Clean, readable TypeScript implementation
• Follows Next.js App Router file convention (robots.ts in app directory)
• Good separation of concerns by importing site config from ~/lib/site
• Consistent with existing patterns (similar to opengraph-image.tsx, twitter-image.tsx)

SEO Implementation:

• ✅ Allows all public pages with allow: '/'
• ✅ Properly blocks sensitive areas: /api/, /admin, /user
• ✅ Strategic decision to leave auth pages crawlable for noindex,follow meta handling
• ✅ Includes sitemap reference and host declaration
• ✅ Dynamic URL generation using site configuration

🟡 Minor Considerations

Security & Best Practices:

• The implementation is secure - no sensitive information exposed
• Dynamic URL generation from siteConfig.url is appropriate
• Disallow rules properly protect API routes and admin areas

Performance:

• Minimal performance impact - static route generation at build time
• Could benefit from additional file-level cache headers, but Next.js handles this automatically for metadata routes

Code Robustness:

• frontend/src/lib/site.ts:6 - The URL trimming logic (replace(/\/$/, '')) is good for consistency
• Consider if trailing slashes in disallow rules might be needed: /api/ vs /api (current implementation is correct for blocking subdirectories)

🔍 Testing Considerations

Current State: No dedicated tests for robots.txt generation
Recommendation: While not critical for this simple implementation, consider:

// Potential test structure (if testing is added later)
describe('robots.txt', () => {
  it('should block sensitive routes', () => {
    const robots = robotsFunction();
    expect(robots.rules.disallow).toContain('/api/');
  });
});

However, since this is a simple metadata route with no complex logic, the validation mentioned in the PR description (npm run build ✅) is sufficient.

🎯 Recommendations

1. Documentation: The PR description is excellent and covers the implementation well
2. Future Enhancement: When sitemap.xml is implemented (feat(seo): add sitemap.xml #108), verify the robots.txt correctly points to it
3. Monitoring: Consider adding robots.txt to your SEO monitoring once deployed

📝 Code-Specific Feedback

frontend/src/app/robots.ts:6-10 - The rules object structure is optimal for this use case. The single userAgent rule with array disallow is more maintainable than multiple rules.

frontend/src/app/robots.ts:11-12 - Good use of the site configuration for dynamic URLs. This ensures consistency across environments.

✨ Final Verdict

Approve - This PR is ready for merge. The implementation is solid, follows Next.js best practices, and effectively addresses the SEO requirements. The strategic decisions around crawlability are well-reasoned and documented.

The code quality is high, security considerations are properly addressed, and the performance impact is minimal. No blocking issues identified.

---