SentencepiecesTokenizer.ids2tokens ignores token_list - root cause of the paraformer-en gibberish in #3040 #3780

Summary
iic/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020 is not mispackaged. The model predicts the right token ids; the text goes wrong at decoding. SentencepiecesTokenizer.ids2tokens ignores the loaded token_list (tokens.json) and decodes the ids with the BPE model's own id order. For this model the two orders differ, so every word is replaced by an unrelated one. Decoding the same ids through tokens.json gives correct English.

Environment
FunASR 1.4.16 (same code on current main), torch 2.14.1, Python 3.11, macOS arm64, CPU.

Root cause
funasr/tokenizer/sentencepiece_tokenizer.py:

def ids2tokens(self, *args, **kwargs):
    return self.decode(*args, **kwargs)   # -> self.sp.DecodeIds(ids)

Paraformer inference calls tokenizer.ids2tokens(token_int). The ids index tokens.json (10020 entries: <blank>, <s>, </s>, ~, <, ...), but sp.DecodeIds reads them as bpe.model ids (10000 pieces: <unk>, <s>, </s>, ▁the, s, ...). Weights load with "All keys matched successfully" and the decoder output size (10020) equals len(tokens.json).

Repro

from funasr import AutoModel
m = AutoModel(model="iic/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020", device="cpu")
tk = m.kwargs["tokenizer"]
cap = {}
orig = type(tk).ids2tokens
def spy(self, ids, **kw):
    cap["ids"] = list(ids)
    return orig(self, ids, **kw)
type(tk).ids2tokens = spy
print(m.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav")[0]["text"])
print(tk.sp.DecodePieces([tk.token_list[i] for i in cap["ids"]]))

Output on the official sample:

procedure telegram impatience phuket sleeping danielot pro
he tried to think how it could be

Line 1 is the gibberish reported in #3040; line 2 is the correct transcript from the same ids.

Suggested fix

def ids2tokens(self, integers, **kwargs):
    if self.token_list is not None:
        return [self.token_list[i] for i in integers]
    return self.decode(integers, **kwargs)

Models whose tokens.json already follows bpe.model order decode the same either way.