tokenize
English
Etymology
token + -ize
Pronunciation
- (General American) IPA(key): /ˈtoʊ.kən.aɪz/
Verb
tokenize (third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized)
- (transitive, computing) To reduce to a set of tokens by lexical analysis.
- (transitive, computing) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.
Derived terms
- tokenizable
- tokenizer
Related terms
- tokenization