Initial commit: llmqt LLM Query Tester
Single-file Python CLI to batch-test multiple LLM models with predefined queries. Supports YAML/JSON config, reasoning detection (<think> tags and reasoning_content field), per-query token/speed stats, and graceful API error handling. Install with `pip install -e .` to get the `llmqt` command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
14
example_test.yaml
Normal file
14
example_test.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
# llmqt example config
|
||||
# Run with: llmqt example_system_prompt.md example_test.yaml
|
||||
# Outputs will be written to ./example_test/<model_name>.md
|
||||
|
||||
# List of models to test. Any OpenAI-compatible model name works.
|
||||
models:
|
||||
- gpt-4o-mini
|
||||
- gpt-4o
|
||||
|
||||
# List of queries to send to each model (in order).
|
||||
queries:
|
||||
- "What is the capital of France?"
|
||||
- "Explain the difference between TCP and UDP in simple terms."
|
||||
- "Write a Python function that checks if a number is prime."
|
||||
Reference in New Issue
Block a user