We Compare AI
Home/Methodology

Our Methodology

We Compare AI tracks 100+ AI tools across 1,000+ data points. Here is exactly how we collect that data, how we keep it current, and what we will never do.

Updated in real-time by AI agents100+ tools trackedZero vendor sponsorships

Core Principles

Zero Vendor Bias

We are not paid by any AI company to appear in our comparisons. No sponsored rankings, no pay-to-win placements. Every tool is evaluated on the same criteria.

Real-Time Updates

Our autonomous AI agents monitor vendor documentation, pricing pages, and official announcements continuously. Data is refreshed as soon as changes are detected — not monthly, not weekly.

Source Verification

Every data point is traced to a primary source: official pricing pages, vendor documentation, published research papers, or independently verified benchmarks. We do not copy from other directories.

Full Transparency

We show our work. Every comparison page shows when data was last updated. This page explains exactly how we collect and score information. If we're uncertain about a data point, we say so.

How We Calculate the Overall Score

A weighted formula across 4 dimensions — reflecting what matters most to real users.

Performance35%

Benchmark scores (MMLU, HumanEval, MT-Bench, Chatbot Arena ELO). The most heavily weighted dimension — a tool must perform well to rank.

Value30%

Price-to-capability ratio. A $0.15/M token model that matches a $15/M model in quality scores very high here.

Reliability20%

Uptime records, API stability, vendor risk signals, and outage history over the past 12 months.

Ease of Use15%

API quality, documentation, onboarding friction, and UI/UX for end-users and developers alike.

Where Our Data Comes From

Every data point is traceable to one of these primary sources.

Official Pricing Pages
We pull pricing directly from vendor websites, not third-party summaries.
Official Documentation
Model specs, context windows, rate limits sourced from vendor API docs.
Published Benchmarks
We reference MMLU, HumanEval, HELM, MT-Bench, and Chatbot Arena for performance data.
Compliance Certificates
SOC 2, HIPAA BAA, ISO 27001 status verified against vendor trust portals.
Independent Latency Testing
Response times measured via direct API calls from US, EU, and APAC regions.
SEC/Funding Filings
Vendor risk scores incorporate publicly available funding rounds and financial filings.

How We Keep Data Current

Our update pipeline runs continuously, 24 hours a day.

1
AI Agents Monitor Sources
Autonomous agents check official pricing pages, documentation, and release notes continuously for changes.
2
Change Detection
When a change is detected (e.g. a price update, new model release, or compliance cert change), a flag is raised.
3
Human Review (for major changes)
Significant changes — new model launches, major pricing shifts — are reviewed by our team before publishing.
4
Timestamp Updated
Every comparison page shows the exact date the data was last verified. You always know how fresh the information is.
5
Blog Agent Publishes
Our daily blog agent automatically publishes articles about notable changes, keeping users informed without manual effort.

What We Track

Pricing & Token Costs
Input/output token pricing, subscription tiers, free tier limits
Performance Benchmarks
MMLU, HumanEval, MT-Bench, Chatbot Arena ELO scores
Latency by Region
Response times from US East, US West, EU, and Asia Pacific
Compliance Certifications
SOC 2 Type II, HIPAA BAA, GDPR, ISO 27001, FedRAMP
Context Window
Maximum token input per request, including long-context variants
Integrations
Native connectors, API availability, Zapier/Make.com support
Vendor Risk Signals
Funding stability, outage history, dependency concentration
Model Updates & Deprecations
Release dates, version changes, end-of-life notices

What We Will Never Do

  • Accept payment for higher rankings or 'featured' placement
  • Copy data from other directories without independent verification
  • Publish benchmarks we cannot trace to a primary source
  • Leave stale data live without a visible 'last updated' date
  • Make claims about tools we have not independently reviewed

Found an error in our data?

We take accuracy seriously. If you spot outdated or incorrect information, please let us know and we will fix it within 24 hours.

Report an error →