Vibeleaderboard
Index / tool

Philosophy Bench

philosophybench.com
Visit philosophybench.com
Category
Developer Tools
Rank
No. 4396Tools index

Previous survey · No. 4091 ·

Pricing
Free
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

A public report and dashboard exploring model behavior in 100 authored dilemmas involving rules, outcomes and user pressure. A Python harness runs simulated tool tasks and uses model judges to classify actions. The rubric describes tendencies in those scenarios, not a universal ethics ranking or proof of deployment safety.

At a glance

Philosophy Bench explores how language-model agents handle conflicts between following a rule and pursuing a better outcome. Its 100 authored dilemmas use simulated customer-support, sales, warehouse and code tasks. It is worth browsing for concrete examples of instruction conflicts and ideas for your own evaluations. Treat the charts as a map of behavior under these scenarios, not a ranking of which model is morally better or safe to deploy.

Good fit

Researchers, agent builders and readers who want to examine what a model does when a user pressures it to bend a rule. The article pairs charts with worked examples, while the dashboard offers newer results. Switching the article’s example from Difficulty 1 to Difficulty 68 replaces a customer-record export dispute with a product-safety disclosure dilemma. That makes the underlying conflict easier to understand than a model-family average alone. Use a relevant example to define what acceptable behavior would mean in your own application.

Consider something else when

You need a general capability leaderboard or evidence that an agent can safely operate your real help desk. The tools and situations are simulated, and the possible responses are grouped using the authors’ rubric. A refusal can count as a botch under that rubric even when a real service should escalate or ask for clarification. Neither higher user compliance nor a stronger preference for rules automatically means a better agent for your task.

Start here

Start in the article’s Benchmark Overview and compare the two difficulty examples. Identify the requested action, the rule it conflicts with and the consequence of following that rule. Then open the dashboard and switch the decision-lean chart from Neither to D-style request to examine scenarios where the user advocates the rule-following action. Read the primer text before attributing a shift to the model alone. These browser controls were inspected; no new model evaluation was run.

Cost and requirements

The public article and dashboard opened without an account or payment. Reproducing or adapting the benchmark uses a Python 3.11+ harness and paid model APIs. Its default three-judge panel adds calls across three providers beyond the model being evaluated, so a key for the target alone is insufficient. The documented five-scenario smoke test limits the initial scope; it is not a fixed-price quote. Code is MIT-licensed, while the scenario/result collection uses CC BY 4.0 with additional provider-term caveats for model-output text.

Limits to know

The scoring guide defines consequentialism narrowly as the scenario’s short-term outcome and deontology largely as rule or policy compliance. A model preserving trust for longer-term benefits can therefore receive the rule-following label. Three model judges classify actions into authored response groups; reasoning labels are a separate analysis, not proof of a model’s internal motives. Local runs store transcripts and send scenario/action text to external target and judge APIs. Keep custom data synthetic unless those disclosures are acceptable. Setup, runtime, cost and scoring reproducibility were not tested.

What it can do

  • Run simulated tool-use conversations through external model APIs

    Authored dilemma scenarios, target model and provider credentials → Per-scenario transcripts and simulated environment snapshots

  • Classify actions with model judges and aggregate the authored axis labels

    Scenario transcripts and authored response groups → Behavioral tendencies and botch rates under the benchmark rubric

  • Explore how user requests and philosophical primers relate to model actions

    Published examples and chart filters → Worked examples and comparative charts for evaluation planning

Why it made the leaderboard

Useful for finding instruction conflicts to test in an agent workflow. The worked examples make the tradeoff concrete, while the published rubric helps explain why following a rule, refusing or obeying a user can change the score. Your own acceptable behavior still needs a task-specific evaluation.

Tags

aibenchmarkethicsllmevaluationphilosophyalignment

Media

Philosophy Bench

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.