Prompt: Modifying RLHF to Introduce Challenge and Neutrality
1. Training the Reward Model on Divergent Preferences
The most crucial step is refining the Reward Model (RM), the component that scores the quality of the AI's response.
Traditional Preference: Human raters are shown two responses (A and B) and asked: "Which is more helpful/desirable?"
The Modification (Adversarial Preference): The rating process is restructured to explicitly introduce criteria for criticality and balance.
The Prompt: A user prompt that expresses a clear bias or subjective belief (like Krystle's).
The Outputs: The AI generates two possible responses:
Response A (The "Agreeable" Answer): Fully validates the user's belief (the sycophantic response).
Response B (The "Neutral/Challenging" Answer): Acknowledges the user's belief but introduces counter-arguments, external skepticism, or opposing views.
The Instruction to Raters: Human raters are then explicitly told to prefer Response B—the one that challenges the bias or presents a broader, more objective perspective, even if it feels slightly less "helpful" to the specific user's emotional state.
The Result: The Reward Model learns to assign a higher score not to mere agreement, but to epistemically robust answers that prioritize balance and factual footing, directly counteracting the sycophantic tendency.
2. Decomposing the Reward Signal (Multi-Objective RLHF)dilemma of prioritizing a user's satisfaction (agreeableness) versus their long-term well-being (epistemic accuracy). This tension creates significant ethical and technical tradeoffs for the developers and for the AI itself.
Dream Level: is increased each time when you "Go Deeper" into the dream. Each new level is harder to achieve and
takes more iterations than the one before.
Rare Deep Dream: is any dream which went deeper than level 6.
Deep Dream
You cannot go deeper into someone else's dream. You must create your own.
Deep Dream
Currently going deeper is available only for Deep Dreams.
