Interface TokenTextSplitterParams

interface TokenTextSplitterParams {
    allowedSpecial: string[] | "all";
    chunkOverlap: number;
    chunkSize: number;
    disallowedSpecial: string[] | "all";
    encodingName: TiktokenEncoding;
    keepSeparator: boolean;
    lengthFunction?: ((text: string) => number) | ((text: string) => Promise<number>);
}

Hierarchy (view full)

Implemented by

Properties

allowedSpecial chunkOverlap chunkSize disallowedSpecial encodingName keepSeparator lengthFunction?

Properties

allowedSpecial

allowedSpecial: string[] | "all"

chunkOverlap

chunkOverlap: number

chunkSize

chunkSize: number

disallowedSpecial

disallowedSpecial: string[] | "all"

encodingName

encodingName: TiktokenEncoding

keepSeparator

keepSeparator: boolean

OptionallengthFunction

lengthFunction?: ((text: string) => number) | ((text: string) => Promise<number>)

Settings

Member Visibility

On This Page

Properties