safe-rlhf
1,598Safe RLHF: constrained value alignment via safe reinforcement learning from human feedback.
PreventionPre-generationFully autonomousExperimentalTier 3Model
Maturity tier details
Computed tier: 3(raw score 3.78). Effective tier: 3.
Facets:documented_in=0evaluated_on=0integration_surfaces=0.5paper_citations=0recent_commit=2stars=1.28
Source of truth:pipeline/maturity/score.py
Description
PKU Alignment's Safe-RLHF is a reference implementation of Safe RLHF — reinforcement learning from human feedback with explicit safety constraints (constrained optimization). Used to train helpful-and-harmless LLMs with measurable safety bounds.
Why it's included
Training-time supervision: the safety-constrained RLHF branch of model alignment. Complements detection-time tools by preventing unsafe behaviour at the model layer rather than the runtime layer.
Addresses failure modes
- Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
- Scope creep — Actions outside declared task boundary.
- Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
Techniques
constrained-rlhfhelpful-harmless-trainingsafety-preference-modeling