Promptfoo: Test Your Prompts and Models
Promptfoo is a tool for testing prompts and models against a list of test cases to ensure quality and prevent regressions.
Prompt engineering is often a game of "trial and error." Promptfoo turns this into a systematic engineering discipline by allowing you to test your prompts against a comprehensive list of test cases, just like unit tests for code.
CLI-Based Testing
Promptfoo runs from the command line, allowing you to quickly compare the outputs of multiple prompts or multiple models (e.g., GPT-4 vs. Claude 3) side-by-side. It provides a visual matrix that makes it obvious where a prompt is failing or where one model outperforms another.
Automated Evaluation Metrics
Beyond visual inspection, Promptfoo supports automated assertions. You can check for the presence of specific keywords, use LLMs to grade the "helpfulness" of the response, or even check for security vulnerabilities, ensuring that your AI responses always meet your quality standards.