Copyrighted Text Exposure Causes AI Models to Hallucinate, Groundbreaking Study Shows

Censorship mechanisms in language models could be compromising their capacity to convey truth on a broader scale. Recent research indicates that the same internal processes designed to block 'unsafe' responses also inhibit the sharing of factual information. This implies that efforts to align models for safety might inadvertently lead to increased hallucination.

For years, developers have focused on reducing falsehoods in language models. The push for greater truthfulness—by curbing hallucinations and guiding models toward verifiable facts—has become a dominant and widely supported research direction.

However, a new Australian study suggests that alignment methods—training techniques that restrict 'unsafe' exchanges—may be hindering models from providing accurate responses altogether by imposing stricter controls:

Enhancing a model's factual accuracy (labeled 'Truthfulness Enhancement' in the figure) can lead it into activation zones that bypass its refusal mechanisms. Similarly, edits aimed at reducing hallucinations may shift internal representations across safety boundaries. This could enable harmful prompts to evade safeguards unless refusal features are carefully isolated and maintained. Source: https://arxiv.org/pdf/2510.07775

The study reveals that the internal pathways responsible for factual recall also govern refusal behavior—the mechanism that prevents models from responding to unsafe or sensitive prompts. When alignment techniques amplify refusal signals too strongly, these pathways overlap, blurring the model's ability to distinguish between rejecting harmful content and inadvertently suppressing valid information.

Ironically, as models improve at refusing inappropriate requests, their ability to convey truth diminishes.

Sensitive Subjects

The illustration above highlights that the core challenge involves not only delivering fair and accurate results to users but also mitigating legal risks for LLM providers.

For example, the case study referenced in the images involves a controversial topic—race-based prison statistics—which an AI might responsibly discuss with academics or researchers but should avoid when manipulated by malicious actors seeking to extract abusive, offensive, or illegal responses.

Since aligned LLMs cannot assess the intent behind a query, they default to a cautious approach:

Responses to sensitive prompts vary by alignment strategy. A safety-focused model blocks the query entirely, while a truth-oriented model provides factual context, improving informativeness but reducing suppression. This supports the idea that edits enhancing truthfulness can lower refusal thresholds, increasing vulnerability to harmful prompts unless refusal mechanisms are safeguarded.

As an aside, these findings might lead critics of so-called 'woke' agendas to argue that heavily aligned models are less truthful and useful than their unregulated counterparts.

The paper's evidence partially supports this view but contextualizes it within the broader risks of using unaligned LLMs—including legal exposure across criminal and civil violations, as well as the spread of misinformation, which remains difficult to filter effectively due to cost constraints.

Intertwined Functions

To understand the underlying mechanisms, the researchers mapped activations of individual attention heads and found that features associated with hallucination and refusal often occupy overlapping regions within the model.

They discovered that fine-tuning or steering these regions to reduce falsehoods can weaken the model's built-in safeguards, as both functions share similar latent space:

'Improving factual accuracy often weakens refusal behavior. Our analysis shows this occurs because components encoding hallucination and refusal information overlap, causing alignment methods to unintentionally suppress factual knowledge.

'We also explore how fine-tuning on benign datasets, even those curated for safety, can degrade alignment for the same reason.'

The authors propose using a sparse autoencoder (SAE)—a network designed to isolate distinct activation patterns—to separate these functions and preserve safety during truthfulness training. This approach aims to make models both safer and more accurate without compromising either quality.

The new paper, titled The Unintended Trade-off of AI Alignment: Balancing Hallucination Mitigation and Safety in LLMs, comes from five researchers affiliated with Deakin University and independent research.

Methodology

The study investigates whether improving truthfulness in language models weakens their ability to refuse harmful prompts and whether both behaviors rely on shared internal components.

Testing two truthfulness-enhancing methods, the authors found that gains in factual accuracy consistently increase susceptibility to jailbreaks.

This trade-off stems from overlapping attention heads that encode both factual and refusal signals. Even benign fine-tuning—intended to improve utility without affecting safety—can disrupt safeguards by altering shared pathways.

The study defines three key terms: truthfulness refers to a model's ability to provide accurate responses based on available knowledge without suppressing harmless content; hallucination occurs when the model generates incorrect information despite access to correct facts; and refusal behavior, or safety alignment, describes mechanisms that block responses to harmful or sensitive prompts.

The authors note that these functions interact in subtle ways:

'Although truthfulness and safety are often analyzed separately, real-world prompts frequently contain sensitive terms with benign intent (e.g., for analysis, detection, or education). In such cases, safety mechanisms may overfire—suppressing accurate, useful information—and reduce practical truthfulness by omission.

'Understanding how edits aimed at increasing factuality affect refusal behavior is essential to achieving truthfulness with minimal, appropriate suppression.'

The authors developed a LoRA that guides a conditioned LLM toward a more 'truthful' state, reducing hallucination. The paper's appendix includes multiple examples illustrating the unintended consequences of this approach.

The analysis begins by treating truthfulness-enhancing methods, such as head-steering and latent-direction mapping, as intentional modifications to a model's internal computations.

Precision Steering

The key question is whether these changes inadvertently affect the same pathways that govern refusal behavior. To test this, the study evaluated models on factual accuracy using TruthfulQA and on safety performance under adversarial conditions using AdvBench and StrongReject.

The baseline techniques included Inference-time intervention (ITI), which activates attention heads linked to truthful answers, and TruthX, which shifts representations along a learned "truthful" direction.

Both methods improve accuracy but also make models more likely to respond to harmful prompts they would previously have refused.

To isolate and manipulate hallucination behavior directly, the authors defined a latent direction corresponding to hallucinated responses, training a LoRA module on incorrect answers from the TruthfulQA dataset using LLaMA3-8B-Instruct.

This produced a linear vector representing the difference between truthful and hallucinated answers, allowing the model to be steered toward or away from hallucination.

Steering along the hallucination direction improves accuracy on TruthfulQA but increases Attack Success Rate (ASR) on AdvBench and StrongReject, highlighting the trade-off between truthfulness and safety.

Steering along the hallucination axis reduced factual accuracy, while reversing direction improved it. Applying this technique to harmful prompt benchmarks confirmed earlier findings: truthfulness gains came at the expense of weakened refusal. Even when hallucination was captured as a clean linear direction, enhancing factual output increased vulnerability to unsafe completions.

The authors emphasize*:

'This reinforces the trade-off between truthfulness and safety, showing that even when truthfulness is represented as a single linear direction, enhancing factuality can come at the expense of weakened safety alignment.'

Data and Testing

To prevent fine-tuning from weakening refusal behavior, the authors employed a method to separate refusal features from those linked to hallucination. They identified attention heads involved in both behaviors and used an SAE to extract latent features specific to refusal.

These features defined a protected subspace. During training, gradient updates were modified to avoid this subspace, allowing the model to reduce hallucinations without compromising safety.

The authors fine-tuned on the CommonsenseQA dataset, evaluating performance across six commonsense reasoning tasks: CSQA, HellaSwag, ARC Challenge, ARC Easy, WinoGrande, and SST-2.

Target modules were fine-tuned using LoRA with rank 8, a learning rate of 2×10⁻⁴, weight decay of 0.01, one training epoch, and a batch size of two. All experiments used the AdamW optimizer.

Safety was evaluated using two harmful content benchmarks: AdvBench (500 samples) and StrongReject (300 prompts). Outputs were classified as safe or unsafe by LlamaGuard3.

Experiments were conducted on LLaMA3-8B-Instruct and Qwen2.5-Instruct.

Baseline methods included SafeLoRA, SaLoRA, SAP, and vanilla supervised fine-tuning (SFT). All were tested with default hyperparameters using 200 prompts from HarmBench, except SafeLoRA.

Accuracy was the primary metric, and Attack Success Rate (ASR) was used for harmful benchmarks, based on LlamaGuard3 results.

Top: Results from LLaMA-3-8B-Instruct, with best scores in bold. Bottom: Performance of fine-tuning methods on Qwen2.5 7B Instruct across commonsense and reasoning tasks (higher scores indicate better accuracy) and safety benchmarks AdvBench and StrongReject (lower ASR values indicate stronger robustness). Best results in each column are bolded.

Regarding these results, the authors state:

'Our surgical approach achieves the best balance between safety and utility: it significantly reduces harmful benchmark scores while preserving fine-tuning accuracy. In contrast, methods like SAP, SaLoRA, and SafeLoRA either increase harmfulness or degrade utility.

'A key reason is that these methods operate directly on the gradient of the safety subspace, which, due to polysemanticity [**], can constrain model performance.

'Compared to vanilla fine-tuning (SFT), our method improves average fine-tuning accuracy (FA) from 56.15% to 75.09%, a gain of approximately +19%.'

The method reduced the Attack Success Rate from 9.23% to 0.58% on AdvBench and from 9.90% to 0.00% on StrongReject—a more than fifteen-fold decrease in harmful outputs. The base model, while low in harmfulness, achieved limited task accuracy.

The authors note:

'These results underscore the importance of preserving refusal features during fine-tuning: by isolating and protecting the refusal subspace, our method maintains safety alignment without sacrificing task performance.

'Overall, this confirms that our approach effectively mitigates the trade-off between truthfulness and safety.'

Finally, the authors tested the method's resilience under adversarial conditions by adding 10% harmful instructions from the Circuit Break dataset to the fine-tuning set.

Despite this deliberate contamination, the approach maintained strong performance across both benign and harmful evaluations:

Performance of LLaMA3 8B Instruct fine-tuned on a poisoned commonsense dataset, comparing accuracy and safety outcomes across methods.

The new method reduced ASR more effectively than SAP while avoiding significant utility loss. Task accuracy remained close to LoRA SFT and SafeLoRA, demonstrating that refusal alignment can be maintained even under contaminated training conditions when refusal features are properly isolated.

Conclusion

The most intriguing finding