[Feature Request]: AAdd an evaluation command

[danduh]

Do you need to file an issue?

• I have searched the existing issues and this feature is not already filed.
• My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
• I believe this is a legitimate feature request, not just a question. If this is a question, please use the Discussions area.

Is your feature request related to a problem? Please describe.

My proposal is to add a graphrag evaluate command. The user would provide a file with questions and expected answers.

The settings file should also allow the user to define an LLM judge. This model would review the generated answers and measure:

• Answer quality and accuracy.
• Whether answers are based on the source data.
• Whether citations are correct.
• Response time and cost.
• Differences between search methods.

This command would also be useful in CI/CD pipelines. It could run after an update or full reindex to check that the process worked correctly and that new data was indexed as
expected. If the results fall below a defined score, the pipeline could fail before the new index is released.

Describe the solution you'd like

No response

Additional context

No response