| tokenize {styler} | R Documentation |
utils::getParseData() is used to obtain a flat parse table from text.
tokenize(text)
text |
The text to parse. |
Apart from the columns provided by utils::getParseData(), the following
columns are added:
A column "short" with the first five characters of "text".
A column "pos_id" for (positional id) which can be used for sorting (because "id" cannot be used in general). Note that the nth value of this column corresponds to n as long as no tokens are inserted.
A column "child" that contains nests.
A flat parse table