Skip to main content
SAICA-KG

Tools

safe-rlhf

1,598

Safe RLHF: constrained value alignment via safe reinforcement learning from human feedback.

PreventionPre-generationFully autonomousExperimentalTier 3Model
Maturity tier details

Computed tier: 3(raw score 3.78). Effective tier: 3.

Facets:documented_in=0evaluated_on=0integration_surfaces=0.5paper_citations=0recent_commit=2stars=1.28

Source of truth:pipeline/maturity/score.py

Description

PKU Alignment's Safe-RLHF is a reference implementation of Safe RLHF — reinforcement learning from human feedback with explicit safety constraints (constrained optimization). Used to train helpful-and-harmless LLMs with measurable safety bounds.

Why it's included

Training-time supervision: the safety-constrained RLHF branch of model alignment. Complements detection-time tools by preventing unsafe behaviour at the model layer rather than the runtime layer.

Addresses failure modes

Techniques

constrained-rlhfhelpful-harmless-trainingsafety-preference-modeling

View raw YAML on GitHub →