[dajiaohuang]

Summary

• LlamaHFTokenizer.tokenize accepted add_bos but ignored it, so chat-format callers could add duplicate BOS tokens or omit a requested BOS.
• Honor the flag while preserving an explicit BOS in input text and other tokenizer-added special tokens.

Validation

• Focused synthetic pytest: 4 passed using Python 3.13 and a task-local package stub to load the tokenizer without a native llama.cpp library or model.
• Ruff check and format check passed. The full test suite/native build was not run; no model weights were downloaded and no inference or external AI service was used.

Changelog

• The entry is deferred because active PR fix: resize detokenization buffer for long token pieces #2388 also modifies CHANGELOG.md; this PR avoids that overlapping path. Changelog follow-up remains with C1 once the path is clear.