Promote Sign in Sign up

LLM behavior analysis

Bloom by Safety Research ✓

Evaluate any behavior immediately with AI

264 views 985.4M/mo traffic 4 saves Unknown
Local Open Source Language Learning Models
More categories (4)
Behavior Evaluation Automated Testing Transcript Analysis LLM behavior analysis
Visit website Sign in to save

Verdict

A quick read before you dig into the full page.

Best if
Evaluate any behavior immediately with AI
Skip if
Complex seed configuration
Pick this when
Automated evaluation

Considering LLMWise instead? Compare side by side

Overview

Bloom is an open-source tool for the automated evaluation of behaviors in Language Learning Models (LLMs). As a scaffolded evaluation system, Bloom uses an evaluation configuration, also known as a 'seed', which specifies a target behavior, exemplary transcripts, and the types of interactions the user wants to examine.

The tool then generates a suite of interactions with the target model intending to expose the selected behavior. The evaluation suite's growth depends on how it is seeded, differing from other evaluations that might use a fixed elicitation technique and prompting pattern, which increases the uniqueness and adaptability of each evaluation run.To ensure reproducibility, Bloom evaluations should always be cited together with their complete seed configuration.

The tool also allows users to add API keys from providers and lets them configure behavioral evaluative facets through behaviors.json and seed.yaml files.

Additionally, Bloom also includes an interactive viewer that provides an intuitive interface to browse transcripts from the run, showcasing conversation flows with correct formatting.

Bloom by Safety Research website

Access & connections

Where you can use Bloom by Safety Research.

Platforms

Web

Capabilities

What this tool accepts, produces, and is strongest at.

Input

  • API
  • Tabular

Output

  • API
  • Text

Highlights

  • Agents
  • Open Source
  • Local

Supported features

  • Agents
  • Automation
  • Integrations
  • Open Source
  • Runs Locally

Pros & cons

Pros

  • Open-source
  • Automated evaluation
  • Unique evaluation
  • Adaptable evaluation
  • Evaluations are reproducible
  • API key addition
  • Configurable behavioral evaluative facets
  • Interactive transcript viewer
Show 29 more
  • Customization options
  • Support for behaviors.json and seed.yaml
  • Scaffolded evaluation system
  • Target behavior specificity
  • Exemplary transcript provision
  • Configurable user interactions
  • Suite growth based on seeding
  • Complete seed configuration citation
  • Ability to evaluate any behavior
  • Showcases conversation flows
  • Correct formatting of transcripts
  • API integration
  • Data visualization capabilities
  • Model API handled via LiteLLM
  • Weights and Biases (wandb) integration
  • Four primary stages of functionality
  • Resumability of interrupted evaluations
  • Detailed configuration parameters
  • Suitable for large-scale experiments
  • Diverse scenario controls
  • Configurable amount of rollouts
  • Supports extended thinking (reasoning_effort)
  • Diversified reasoning effort levels
  • Temperature controls for model outputs
  • Web search during base scenario generation
  • Max conversation length specification
  • Selective scenario variation rollouts
  • Definition of additional judgment qualities
  • Pipeline stage-specific model selection

Cons

  • Complex seed configuration
  • Depends on evaluation 'seed'
  • May require API keys
  • Requires understanding of Json/Yaml
  • Manual evaluation configuration
  • Results reproducibility linked to 'seed'
  • Requires Python environment setup
  • Relies heavily on text files
Show 2 more
  • Requires specific file formatting
  • Limited to Language Learning Models

Compare alternatives

Similar tools in the catalog — with a clear reason to open each one.

Why look Offers a public API

Full comparison

Why look Offers a public API

Full comparison

Releases 1

Initial release December 20, 2025 Current
Initial release of Bloom by Safety Research.

FAQ 12

What is Bloom by Safety Research?
Bloom by Safety Research is an open-source tool designed for the automated evaluation of behaviors in Language Learning Models (LLMs). This tool functions as a scaffolded evaluation system that uses a specific evaluation configuration known as a 'seed'. The seed details the target behavior, exemplary transcripts, and the types of interactions users wish to inspect. Bloom generates a series of interactions with the target model that attempt to expose the selected behavior.
What is a 'seed' in Bloom's configuration?
In Bloom's configuration, a 'seed' is an evaluation configuration that specifies various aspects of the behavior evaluation including the target behavior, exemplary transcripts, and types of interactions that the user is interested in. The 'seed' strongly influences the growth of the evaluation suite, which may differ notably from other evaluations that use a fixed elicitation technique and prompting pattern.
How does the Bloom tool reveal selected behaviors in LLMs?
Bloom tool reveals selected behaviors in LLMs by generating an evaluation suite of interactions with the target model. These interactions are designed based on the 'seed' configuration, which specifies the target behavior and example transcripts. Through these interactions, Bloom attempts to expose and evaluate the chosen behavior in the language learning model.
What makes an evaluation run in Bloom unique?
An evaluation run in Bloom is made unique by the 'seed' configuration, which sponsors the growth of the evaluation suite. As the growth of the suite depends on the initial seeding, it differs from other evaluations that could use a fixed elicitation technique and prompting pattern. This means each run is unique and adaptable, thus fitting different behavioral evaluations.
Why should Bloom evaluations be cited together with their full seed configuration?
Bloom evaluations should be cited together with their full seed configuration to ensure reproducibility. The seed configuration includes details about the target behavior, exemplary transcripts, and types of interactions the user is interested in. By citing the full seed configuration, others can understand and replicate the conditions under which the evaluation occurred, maintaining the consistency and reliability of the results.
How do I add API keys in Bloom?
To add API keys in Bloom, you have to modify the .env file. You can include keys such as OPENAI_API_KEY or ANTHROPIC_API_KEY, and others depending on the provider. This allows the tool to interface with various API providers.
How can I configure behavioral evaluative facets in Bloom?
In Bloom, you can configure behavioral evaluative facets by modifying the behaviors.json and seed.yaml files. The behaviors.json file can contain an entry with a unique key and description for the behavior you want to evaluate. The seed.yaml file can be used to customize key parameters including behavior, examples, total evaluations, target model, and others according to your evaluation needs.
What is an interactive viewer in context of Bloom?
The interactive viewer in Bloom is a feature that provides an intuitive interface to browse transcripts from the evaluation run. It showcases conversation flows with correct formatting, judgment scores, justifications, and allows filtering and search through transcripts. This aids in efficient analysis and understanding of the evaluation results.
What information can I obtain from the transcript analysis in Bloom?
From the transcript analysis in Bloom, you can gain detailed information about the interaction between the LLM and the evaluation model. It includes the conversation flows with correct formatting, judgment scores, and justifications. This data is useful in understanding how the target behavior surfaces in different contexts and interactions.
Does Bloom support customization of the seed configuration?
Yes, Bloom supports customization of the seed configuration. While setting up the evaluation, you can specify your target behavior, exemplary transcripts, and the types of interactions you're interested in by modifying behaviors.json and seed.yaml files. This allows for a highly tailored evaluation based on the user's specific needs.
How does Bloom ensure reproducibility in its evaluations?
Bloom ensures reproducibility in its evaluations by requiring that all evaluations be cited together with their complete seed configuration. The seed configuration details the conditions under which the evaluation took place, including target behavior, exemplary transcripts, and types of interactions. By doing so, Bloom makes it possible for others to understand and replicate the same conditions, ensuring the consistency and reliability of results across different runs and teams.
How does Bloom contribute to educational technology?
Bloom contributes to educational technology by providing a tool to evaluate and understand the behavior of Language Learning Models (LLMs). By facilitating the exploration of behavior like bias or self-preservation in these models, Bloom can help educators and developers in improving the effectiveness and inclusivity of AI in language learning applications.
Bloom by Safety Research
Visit website