[Core] NumPy String support in NumSharp

Overview

Add NumPy-compatible string dtype support to NumSharp, enabling text data processing with full NumPy 2.x behavioral parity.

Problem

NumSharp cannot store strings in arrays, perform vectorized string operations, or interoperate with NumPy string data. This blocks text-heavy ML/data science workflows, pandas-like DataFrames, and full .npy file compatibility.

Reference

docs/plans/NUMPY_STRING_TYPES.md - 1959-line specification from NumPy v2.4.2 source analysis.

Proposal

Core Type System

Add NPTypeCode.Bytes for fixed-width byte strings (S, NPY_STRING = 18)
Add NPTypeCode.Unicode for fixed-width Unicode strings (U, NPY_UNICODE = 19)
Add NPTypeCode.VString for variable-length UTF-8 strings (T, NPY_VSTRING = 2056)
Keep NPTypeCode.Char as its own type (maps to U1 or S1 internally for string operations)
Add np.bytes_ scalar type (wraps byte[])
Add np.str_ scalar type (wraps string or int[] for UTF-32)
Add np.character base class for string scalar hierarchy
Add np.flexible base class (parent of character and void)

Dtype Parsing and Construction

Memory Layout - bytes_ (S)

Store as contiguous byte buffer, 1 byte per character
Itemsize = exact byte count per element
Null-pad shorter strings to itemsize
Truncate longer strings to itemsize
No byte order (always | prefix)
Strip trailing nulls on scalar access

Memory Layout - str_ (U)

Store as UTF-32/UCS-4, 4 bytes per code point (npy_ucs4 = uint)
Itemsize = character count x 4
Support little-endian (<) and big-endian (>) byte orders
Null-pad shorter strings (null = 0x00000000)
Truncate longer strings to character count
Implement byte swapping for cross-platform .npy files

Memory Layout - StringDType (T)

16-byte header per element (64-bit) or 8-byte (32-bit)
Implement small-string optimization (15 bytes or less inline on 64-bit, 7 or less on 32-bit)
Implement arena allocator for medium strings (255 bytes or less)
Implement heap allocation for large/mutated strings
Store UTF-8 encoded content
Implement flag byte: MISSING (0x80), INITIALIZED (0x40), OUTSIDE_ARENA (0x20), LONG (0x10)
Thread-safe allocator access (mutex/lock)
Preserve embedded null bytes (unlike legacy types)

StringDType NA/Missing Support

Support StringDType(na_object=None) - None as NA
Support StringDType(na_object=np.nan) - NaN as NA
Support StringDType(coerce=True/False) - coercion control
NA values sort to end
np.isnan() identifies NaN-like NA
Raise on comparison with non-NaN NA (e.g., None)

Array Creation

Type Conversions and Casting

Comparison Operations

== element-wise equality (returns bool array)
!= element-wise inequality
<, <=, >, >= lexicographic comparison
np.equal, np.not_equal, np.less, np.less_equal, np.greater, np.greater_equal
Comparison broadcasts correctly
bytes_ and str_ cannot be compared directly (TypeError)
StringDType can compare with str_ and object (type promotion)

String Concatenation and Repetition

np.strings.add(a, b) - concatenate strings
np.strings.multiply(a, n) - repeat string n times
multiply(s, 0) or multiply(s, -1) results in empty string
multiply(s, sys.maxsize) results in OverflowError
Broadcasting: ["a", "b"] + "x" results in ["ax", "bx"]

String Length

np.strings.str_len(arr) - returns int64 array
Length = code points, not bytes (emoji = 1)
Empty string results in 0

String Searching

np.strings.find(a, sub, start=0, end=None) - first occurrence, -1 if not found
np.strings.rfind(a, sub, start=0, end=None) - last occurrence
np.strings.index(a, sub, ...) - like find but raises ValueError
np.strings.rindex(a, sub, ...) - like rfind but raises ValueError
np.strings.count(a, sub, start=0, end=None) - count occurrences
np.strings.startswith(a, prefix, start=0, end=None) - bool array
np.strings.endswith(a, suffix, start=0, end=None) - bool array

Case Operations

np.strings.upper(arr) - uppercase
np.strings.lower(arr) - lowercase
np.strings.capitalize(arr) - first char upper, rest lower
np.strings.title(arr) - titlecase each word
np.strings.swapcase(arr) - swap case

Character Classification

Whitespace Operations

np.strings.strip(arr, chars=None) - strip both ends
np.strings.lstrip(arr, chars=None) - strip left
np.strings.rstrip(arr, chars=None) - strip right
np.strings.center(arr, width, fillchar=" ") - center-align
np.strings.ljust(arr, width, fillchar=" ") - left-align
np.strings.rjust(arr, width, fillchar=" ") - right-align
np.strings.zfill(arr, width) - zero-fill numeric strings
np.strings.expandtabs(arr, tabsize=8) - expand tabs

Splitting and Partitioning

np.strings.partition(arr, sep) - returns tuple of 3 arrays (before, sep, after)
np.strings.rpartition(arr, sep) - partition from right
np.char.split(arr, sep, maxsplit) - returns object array of lists
np.char.rsplit(arr, sep, maxsplit) - split from right
np.char.splitlines(arr, keepends=False) - split on line boundaries
np.char.join(sep, seq) - join sequences

Replacement and Slicing

np.strings.replace(arr, old, new, count=-1) - replace occurrences
np.strings.slice(arr, start=0, stop=None, step=1) - substring extraction
np.strings.slice(arr, None, None, -1) - reverse string

Encoding/Decoding

np.strings.encode(arr, encoding="utf-8", errors="strict") - str to bytes
np.strings.decode(arr, encoding="utf-8", errors="strict") - bytes to str
Error modes: "strict", "ignore", "replace"

Formatting and Translation

np.strings.mod(format_arr, values) - printf-style formatting
np.strings.translate(arr, table, deletechars=None) - character translation

Array Functions with String Support

Structured Arrays

String fields in structured dtypes: [("name", "U20"), ("id", "i4")]
StringDType in structured arrays
Nested structured types with string fields
Record arrays (np.rec.array) with attribute access

File I/O

np.save()/np.load() for bytes_, str_, StringDType (.npy format)
np.savez()/np.load() for string arrays (.npz archives)
np.loadtxt(dtype="U") - load text as strings
np.savetxt(fmt="%s") - save strings to text
np.genfromtxt(dtype="U") - flexible text loading
arr.tofile()/np.fromfile(dtype="S10") - binary I/O (fixed-width only)
Memory mapping for fixed-width strings: np.memmap(dtype="U100")
StringDType cannot be memory-mapped (variable length)

Boolean Conversion and Truthiness

arr.astype(bool) - empty = False, non-empty = True
np.nonzero(arr) returns indices of non-empty strings
StringDType NA is truthy for nonzero (but identifiable via np.isnan)

Edge Cases

Empty string arrays: np.array([""], dtype="S") results in dtype="|S1"
Zero-size dtype: np.dtype("S0").itemsize results in 0
Embedded null bytes preserved in storage
Null bytes may terminate on scalar access (bytes_)
Unicode surrogates: valid in UTF-32, may fail converting to UTF-8
Combining characters: "e\u0301" is 2 code points (displays as 1)
No automatic Unicode normalization
Very large strings: respect NPY_MAX_STRING_SIZE
Overflow detection in multiply and add

Legacy Support

np.char.chararray - deprecated subclass (minimal support if needed)
chararray automatic whitespace stripping behavior
chararray instance methods (upper, lower, etc.)
np.char.* functions as aliases to np.strings.*

Type Checking

np.issubdtype(arr.dtype, np.bytes_) - True for S
np.issubdtype(arr.dtype, np.str_) - True for U
np.issubdtype(arr.dtype, np.character) - True for S or U
np.issubdtype(arr.dtype, np.flexible) - True for S, U, V

Evidence

Reference document: docs/plans/NUMPY_STRING_TYPES.md
NumPy source: src/numpy/ (v2.4.2)
String dtypes required for: pandas interop, scikit-learn text vectorizers, NLP pipelines, CSV/text file processing

Scope / Non-goals

In scope: Core string dtypes, string operations API, file I/O, structured arrays, type conversions

Out of scope:

Regular expressions (use .NET Regex)
Locale-aware collation/sorting
NLP-specific operations (tokenization, stemming)
Full chararray compatibility (deprecated in NumPy)

Current String Handling in NumSharp

NPTypeCode Enum

Code	Value	C# Type	Size	Description
`NPTypeCode.Char`	4	`char`	2 bytes (but reported as 1)	Single UTF-16 code unit
`NPTypeCode.String`	18	`string`	IntPtr.Size	Reference type - limited support

Key Files

File	Purpose
`Backends/NPTypeCode.cs`	Type enum, conversions, size info
`Backends/NDArray.String.cs`	String helper methods
`Creation/np.array.string.cs`	`np.array(string[])` implementation
`Creation/np.asarray.cs`	`np.asarray(string)` implementation
`Creation/np.dtype.cs`	Dtype parsing (`'S'`, `'U'` -> `char`)
`APIs/np.save.cs`	.npy file I/O with string support

Current Capabilities

1. Single String -> Char Array

// Works: converts string to char[]
var arr = NDArray.FromString("hello");  // Shape: (5,), dtype: char
var str = NDArray.AsString(arr);         // "hello"

2. Char Array Operations

// Works: create char arrays directly
var arr = np.array('h', 'e', 'l', 'l', 'o');  // Shape: (5,), dtype: char
arr.GetString();  // "hello"
arr.SetString("world");

3. String Array -> Encoded Format (Hack)

// np.array(string[]) uses a special encoding format
var arr = np.array(new[] { "Hello,", " SciSharp!" });
// Stored as: "2 6 10:Hello, SciSharp!" (metadata + content)
// Actual dtype: char, Shape: (28,)
NDArray.AsStringArray(arr);  // Decodes back to string[]

4. np.save/np.load String Support

// Saves string arrays to .npy with dtype "|S{maxlen}"
// Each string is written as fixed-width bytes (ASCII only)
np.save("file.npy", new[] { "hello", "world" });

Dtype Parsing

np.dtype("S")     -> typeof(char)  // Not bytes!
np.dtype("S10")   -> typeof(char)  // Ignores size
np.dtype("U")     -> typeof(char)  // Not unicode!
np.dtype("U10")   -> typeof(char)  // Ignores size
np.dtype("char")  -> typeof(char)
np.dtype("string")-> typeof(char)
np.dtype("String")-> typeof(String)  // NPTypeCode.String

NPY_TYPECHAR Mapping

NPY_STRINGLTR ('S')  -> NPTypeCode.Char
NPY_STRINGLTR2 ('a') -> NPTypeCode.Char  
NPY_UNICODELTR ('U') -> NPTypeCode.Char
NPY_CHARLTR ('c')    -> NPTypeCode.Char (via NPY_UBYTELTR)

Limitations

No real string dtype: NPTypeCode.String exists but is barely usable - most operations throw or fall back to char
No fixed-width strings: Can't create dtype='U10' with 10 characters per element
No Unicode storage: Everything maps to C# char (UTF-16), not NumPy's UTF-32
No byte strings: 'S' dtype doesn't store bytes, just chars
Encoding hack: np.array(string[]) uses a metadata format, not real string storage
Size ignored: 'S10', 'U10' parsing ignores the size specifier
ASCII-only I/O: np.save writes strings as single-byte ASCII
No string operations: No np.strings.* or np.char.* functions

Type Priority / Group

NPTypeCode.Char   -> Group: 0, Priority: 0  (same as Byte)
NPTypeCode.String -> Group: 0, Priority: 0  (same as Byte)

Size Reporting (Bug?)

NPTypeCode.Char.SizeOf()   -> 1  // Should be 2 for C# char!
NPTypeCode.String.SizeOf() -> 1  // "because it is a char basically"

This is a far cry from NumPy's string support. The current implementation is essentially "store chars and provide helper methods to convert to/from strings."

Current-state compatibility matrix (NumPy 2.4.2 vs NumSharp now)

Mapping of every string capability against the current implementation, to scope this issue. Verified live against NumPy 2.4.2 and ground-truthed in source (nditer @ f7be5f7e). Saved in-repo at docs/numpy/STRING_COMPAT_MATRIX.md, next to the spec.

Companion to: NUMPY_STRING_TYPES.md (full spec) · tracks #592.

Legend: ✅ present & compatible · 🟡 partial / hack / divergent · ❌ missing or throws

Two spec corrections found while ground-truthing against live NumPy 2.4.2 (the spec doc is otherwise accurate):

np.dtype('c') is not deprecated/raising — it returns dtype('S1'). (NumSharp currently throws on 'c'.)

S == U does not raise — ==/!= return element-wise [False…]; only ordering comparisons (<,>) raise.

A. Type system / dtypes

Capability	NumPy 2.4.2	NumSharp — now
`bytes_` `'S'` (enum 18)	fixed-width, 1 B/char, null-padded	no `NPTypeCode.Bytes`; `dtype('S')` throws `NotSupportedException`	❌
`str_` `'U'` (enum 19)	fixed-width, 4 B/char (UTF-32)	no `NPTypeCode.Unicode`; `dtype('U')` throws	❌
`StringDType` `'T'` (enum 2056)	variable UTF-8 + NA	no type; `dtype('T')` throws ("cannot parse")	❌
`'c'` char dtype	= `S1` (1 byte)	`dtype('c')` throws (explicitly in `_unsupported_numpy_codes`)	❌
`NPTypeCode.Char`	n/a	exists = C# `char` (UTF-16, 2 B, reported size 1)	🟡
`NPTypeCode.String` (18)	n/a (NumSharp-only)	in enum, maps `typeof(string)`, `SizeOf=1`, barely wired	🟡
`object` dtype holding strings	yes	no object dtype	❌

B. Dtype parsing & properties

Capability	NumPy 2.4.2	NumSharp — now
`dtype('S'/'S10'/'\|S10')`	bytes_	throws	❌
`dtype('U'/'U10'/'<U10'/'>U10')`	str_ (+byteorder)	throws	❌
`dtype('T')`, `dtype('a')`	StringDType / legacy-S	throws	❌
`dtype('string')` / `dtype('char')`	n/a	→ `typeof(string)` / `typeof(char)` (NumSharp-only aliases)	🟡
tuple form `dtype((np.str_,5))`	yes	not supported	❌
`.char` / `.kind` = `S`/`U`/`T`	yes	`DType.kind` hardcodes `'S'` for Char & String; wrong for real strings	🟡
`.itemsize` (exact bytes)	`S10`→10, `U10`→40, `T`→16	n/a for strings	❌
`.name` (`bytes80`/`str320`/`StringDType128`)	yes	`"string"` only	❌
`.str` (`\|S10`/`<U10`/`\|T16`)	yes	none	❌
`.type` (scalar class)	yes	none	❌

C. Scalars & type hierarchy

Capability	NumPy 2.4.2	NumSharp — now
`np.bytes_` / `np.str_` scalar types	yes	none	❌
`np.character` / `np.flexible` bases	yes	none	❌
`issubdtype(d, np.character/flexible/bytes_/str_)`	yes	not applicable (types absent)	❌

D. Array creation

Capability	NumPy 2.4.2	NumSharp — now
`np.array([...], dtype='S'/'U'/'T')`	auto-size, truncate, null-pad	throws (dtype)	❌
`np.array(string[])`	real `<U` array	metadata hack: packs to char vector `"N l1 l2:contents"`	🟡
from Python `b'...'` bytes	yes	no bytes path	❌
`np.zeros/empty/full(dtype='U10')`	yes	throws	❌
`new NDArray(typeof(string), shape)`	n/a	throws `NotSupportedException` (issue #341)	❌
single `string` → char vector (`FromString`, implicit op)	n/a	works (shape `(n,)` char)	✅
`GetString`/`SetString`/`GetStringAt` on char vectors	n/a	works	✅

E. Memory layout

Capability	NumPy 2.4.2	NumSharp — now
Fixed-width null-padded buffer (S)	yes	none	❌
UTF-32/UCS-4 + byte-order (U)	yes	none	❌
16 B header + arena + small-string-opt + NA flags (T)	yes	none	❌
Contiguous UTF-16 chars	n/a	the only thing we store	🟡

F. Casting / conversion

Capability	NumPy 2.4.2	NumSharp — now
num → str (`'1'`,`'1.5'`,`'nan'`,`'inf'`)	yes (`<U21` for int64)	none	❌
str → num (`'22'`→22)	yes	none	❌
bool → str (`'True'`/`'False'`)	yes	none	❌
str → bool (empty=False, `'False'`→True)	yes	none	❌
`S↔U↔T`, width truncation, casting safety	yes	none	❌
`datetime64 ↔ str`	yes	none (no datetime64)	❌

G. Comparisons

Capability	NumPy 2.4.2	NumSharp — now
`==`/`!=` element-wise on strings	bool array	char vectors compare as uint16 numbers, not strings	🟡
`<`,`<=`,`>`,`>=` lexicographic	yes	numeric on char only	❌
broadcasting in comparisons	yes	n/a for strings	❌
`S` vs `U` mismatch (`==`→all-False, `<`→raises)	yes	n/a	❌

H. String operations — `np.strings` (46 ufuncs) & `np.char`

Family	NumPy 2.4.2	NumSharp — now
`add` (concat), `multiply` (repeat)	yes	entire namespace absent	❌
`str_len` (code points)	yes	—	❌
`find/rfind/index/rindex/count/startswith/endswith`	yes	—	❌
case: `upper/lower/capitalize/title/swapcase`	yes	—	❌
classify: `isalpha/isdigit/isalnum/isspace/islower/isupper/istitle/isnumeric/isdecimal`	yes	—	❌
whitespace/pad: `strip/lstrip/rstrip/center/ljust/rjust/zfill/expandtabs`	yes	—	❌
`replace`, `slice`	yes	—	❌
`partition/rpartition`	yes	—	❌
`split/rsplit/splitlines/join` (object arrays)	yes	—	❌
`encode/decode`, `mod`, `translate`	yes	—	❌
`np.char.*` legacy + `chararray`	yes (deprecated)	—	❌

I. Array functions with string support

Capability	NumPy 2.4.2	NumSharp — now
`sort`/`argsort` (lexicographic)	yes	`np.sort` missing entirely; `argsort` numeric-only	❌
`argmax`/`argmin` (lexicographic)	yes	numeric-only	❌
`searchsorted`, `unique`	yes	numeric-only	❌
`concatenate`/`stack`/`vstack`/`hstack`/`dstack`	yes	n/a for strings	❌
`where`/`take`/`nonzero`/`any`/`all` (empty=False)	yes	n/a for strings	❌

J. File I/O & structured arrays

Capability	NumPy 2.4.2	NumSharp — now
`np.save` string array → `.npy`	`\|S`/`<U`/`\|T` preserved	managed `string[]` only → `\|S{maxlen}`, ASCII-lossy `(byte)s[i]`, no `U`/`T`	🟡
`np.load` `.npy` string	dtype preserved as NDArray	reads `\|S` → managed `string[]` (null-terminated ASCII); no `U`/`T`; not a string-dtype NDArray	🟡
`savez`/`npz`, `loadtxt`/`savetxt`/`genfromtxt`, `tofile`/`fromfile`, `memmap`	yes	none for strings	❌
structured dtypes w/ string fields, `recarray`	yes	no structured dtypes	❌

Tally (≈70 capabilities)

✅ ~3 — single-string ↔ char-vector helpers
🟡 ~9 — char fakery, string[] packing hack, ASCII .npy round-trip, fake dtype props
❌ ~58 — all three real dtypes, every np.strings/np.char op, casting, comparisons, structured/text I/O

Bottom line

NumSharp today has no real string dtype — only a char (UTF-16) vector dressed up with helper methods, one brittle string[]→char packing hack, and an ASCII-only .npy string[] path that bypasses the NDArray dtype system entirely. The two "🟡 file I/O" rows are the only places a multi-string array survives a round trip, and only for ASCII.

Generated from NumSharp nditer @ f7be5f7e and NumPy v2.4.2 live verification.

Add NumPy-NumSharp string compatibility matrix

[Core] NumPy String support in NumSharp #592

Description