Add a dedicated documentation entry for tokenize.TokenInfo #160036

Documentation

tokenize() and generate_tokens() return TokenInfo objects, but TokenInfo has no class entry in the documentation. Its fields and exact_type property are described inside the tokenize() documentation entry, which makes them hard to find by name or link to directly.

Consider moving those descriptions into a dedicated TokenInfo entry with individual attribute entries, and link to it from tokenize() and generate_tokens(). This would make the existing documentation easier to find without duplicating it.

cc @pablogsal @lysnikolaou

Linked PRs

• gh-160036: Document tokenize.TokenInfo separately #160037