Help & Documentation

What is ModelZoo?

ModelZoo is an AI model benchmarking platform that helps you test, compare, and analyze language models' performance. You can benchmark models from various providers (OpenAI, Ollama, etc.) using standardized test cases to measure accuracy, speed, cost, and energy efficiency.

Key Features

• Test multiple models simultaneously
• Create custom test cases and test suites
• Track performance metrics including accuracy, speed, and cost
• Compare models with detailed analytics
• Export results in multiple formats
• Integrate with Slack for notifications
• Manage API keys securely

Your First Benchmark

1. Go to Model Zoo
2. Add a model (Ollama local, OpenAI, or Base44)
3. Select or create test cases
4. Click "Run Benchmark" to test your model
5. View results in the "Results" tab
6. Analyze performance in "Leaderboard" or "Comparison"

Adding Models

You can add models from three sources:

• Ollama: Local models running on your machine. Requires Ollama server running.
• OpenAI: Cloud-based models like GPT-4o, GPT-3.5-turbo. Requires API key in Settings.
• Base44: Pre-configured models from our curated collection.

Benchmark Configuration

• Temperature: Controls randomness (0=deterministic, 1=creative). Use 0 for testing.
• Max Tokens: Maximum length of model response. Default 10 is usually sufficient for triple validation.
• Delay Between Requests: Wait time between API calls (in seconds). Use 0.25+ to avoid rate limits.

Performance Metrics

• Accuracy: Percentage of correct predictions (0-100%)
• Response Time: Average time per query in seconds
• Token Usage: Input and output tokens consumed
• Estimated Cost: Calculated based on provider pricing
• Performance Score: Combined metric weighing accuracy and speed
• Energy Efficiency: Score considering accuracy, speed, and model size
• Drift: Performance change compared to previous test

During Benchmark

• Real-time progress updates show current status
• Pause/Resume to stop and continue testing
• View live results as tests complete
• Cancel to stop the benchmark immediately

What are Test Cases?

Test cases are knowledge graph triple validation examples. Each case consists of:

• Utterance: Natural language text (e.g., "Alice works at Google")
• Triple: Structured knowledge representation [subject, predicate, object]
• Is Valid: Whether the triple correctly represents the utterance

Creating Test Cases

1. Go to Model Zoo → Test Cases tab
2. Click "Add Test Case"
3. Enter utterance and triple
4. Mark as valid or invalid
5. Add optional metadata (category, difficulty, domain)
6. Click Save

Test Case Properties

• Category: user_attribute, assistant_attribute, preference, location, temporal, negation, other
• Difficulty: easy, medium, hard
• Domain: Topic area (e.g., healthcare, finance)
• Keywords: Tags for filtering

AI-Generated Test Cases

Use the "Generate Test Cases" feature to automatically create cases using AI:

• Select generation mode (Standard, Custom, From Sample)
• Specify number of cases and difficulty
• Review and add generated cases

Managing API Keys

1. Go to Settings → API Keys tab
2. Click "New Key" to create one
3. Give it a descriptive name
4. Copy immediately (you won't see it again)
5. Use in your applications
6. Revoke when no longer needed

Provider API Keys

Store provider keys in Settings for use in benchmarks:

• OpenAI API Key: Required for GPT models. Get from platform.openai.com
• Other Providers: Store credentials in Settings → Account

Testing API Keys

Use the test button (⚡ icon) next to each key to verify it works correctly.

API Endpoint Example

Understanding Costs

Costs are calculated based on:

• Prompt Tokens: Input text tokens
• Completion Tokens: Output text tokens
• Provider Pricing: Cost per 1k tokens

Custom Pricing Configuration

1. Go to Model Zoo → Cost Analysis tab
2. Click "Add Custom Pricing"
3. Select provider and model
4. Enter cost per 1k prompt tokens
5. Enter cost per 1k completion tokens
6. Save configuration

Cost Visualization

• View total costs by model
• Compare costs between providers
• Track cost trends over time
• Identify most expensive models

Leaderboard View

Rank models by various metrics:

• Accuracy: Highest correctness rate
• Speed: Fastest response time
• Balanced Score: Best overall performance
• Energy Efficiency: Best efficiency rating

Filtering Results

• Filter by provider (OpenAI, Ollama, Base44)
• Filter by status (completed, failed, archived)
• Sort by any metric column

Model Comparison

1. Select 2+ models in the Models tab
2. Click "Compare Selected"
3. View side-by-side metrics
4. See visual comparisons

AI Performance Analysis

Get AI-generated insights:

• Key strengths and weaknesses
• Best use cases for each model
• Production readiness assessment
• Optimization recommendations

Slack Integration

Receive benchmark notifications in Slack:

1. Go to Settings → Slack tab
2. Click "Connect Webhook"
3. Paste your Slack webhook URL
4. Save configuration
5. Enable notifications during benchmarks

Notifications Include

• Benchmark start and completion status
• Final accuracy and performance scores
• Quick links to view detailed results

Q: Why is my Ollama model not showing up?

A: Make sure Ollama is running locally and set to the correct address (usually http://localhost:11434). Check your settings in Model Zoo.

Q: How do I improve accuracy scores?

A: Use models with larger parameter counts, adjust temperature to 0 for consistency, and create more diverse test cases covering edge cases.

Q: Can I export my results?

A: Yes! Go to Results tab and use the Export buttons to download as JSON, CSV, or PDF.

Q: What's the difference between accuracy and balanced score?

A: Accuracy is just correctness %. Balanced score weighs both accuracy and response time, giving a more complete picture of performance.

Q: How often should I run benchmarks?

A: Run benchmarks when testing new models, after model updates, or periodically to track performance drift over time.

Q: Can I compare models from different providers?

A: Yes! Select any models (OpenAI, Ollama, Base44) and use the Comparison feature to compare them side-by-side using the same test cases.

Q: What does "Energy Efficiency" measure?

A: It's a combined metric that weighs accuracy, response speed, and model size to determine overall efficiency and cost-effectiveness.

Q: How is drift calculated?

A: Drift compares the accuracy of the current test against your previous test with the same model. Positive = improvement, negative = regression.

Q: Can I share results with my team?

A: Export results as PDF and share, or use Slack integration for automatic notifications. More sharing features coming soon.

Q: How do I archive old test results?

A: Click the archive button (📦) on any result in the Results tab. Archived results move to history but can be retrieved if needed.