The entrance point into Lexer API is {@link org.netbeans.api.lexer.TokenHierarchy} class with its static methods that provide its instance for the given input source.
{@link org.netbeans.api.lexer.TokenHierarchy#tokenSequence()} allows to walk through the tokens that were created for the given input.
{@link org.netbeans.api.lexer.Token} models a single token recognized in the input. It carries a token identification {@link org.netbeans.api.lexer.TokenId} returned by {@link org.netbeans.api.lexer.Token#id()} and token text represented as {@link java.lang.CharSequence} returned by {@link org.netbeans.api.lexer.Token#text()}.
{@link org.netbeans.api.lexer.Language} represents all the token ids for a single language.
Each token id may belong to one or more token categories that allow to better operate tokens of the same type (e.g. keywords or operators).