Tokenizer Class Reference

Splits up a source string into Tokens based on the given delimiter(s). More...

#include <Tokenizer.h>

Public Member Functions
	Tokenizer ()
	Creates a Tokenizer with no initial settings.
	Tokenizer (const std::string &raw)
	Creates a Tokenizer and tokenizes the given raw string over spaces. More...
	Tokenizer (const std::string &raw, const char delim)
	Creates a Tokenizer and tokenizes the given raw string over the given delimiter. More...
	Tokenizer (const std::string &raw, const std::string &delims)
	Creates a Tokenizer and tokenizes the given raw string over the given delimiters. More...
	Tokenizer (const Token &raw)
	Creates a Tokenizer and tokenizes the contents of the given token over spaces. More...
	Tokenizer (const Token &raw, const char delim)
	Creates a Tokenizer and tokenizes the contents of the given token over the given delimiter. More...
	Tokenizer (const Token &raw, const std::string &delims)
	Creates a Tokenizer and tokenizes the contents of the given token over the given delimiters. More...
void	setSource (const std::string &raw)
	Sets the source string to tokenize and tokenizes it. More...
void	setDelims (const std::string &delims)
	Sets the delimiters to tokenize over. More...
const std::vector< Token > &	getTokens () const
	Gets a reference to a vector of all of the tokens that were found in the source string. More...
const Token &	getToken (const size_t pos) const
	Gets the Token at the given position in the token vector. More...
size_t	indexOf (const Token &token, const size_t start=0) const
	Finds the index of the given Token in the token vector. More...

Detailed Description

Splits up a source string into Tokens based on the given delimiter(s).

This is used under the hood by iash to parse the user's command input into UserCommands to be used by iash and Commands.

Since

0.5

Constructor & Destructor Documentation

Tokenizer() [1/6]

Tokenizer::Tokenizer

(

const std::string &

raw

)

Creates a Tokenizer and tokenizes the given raw string over spaces.

This does not tokenize over tabs, newlines, and carriage returns, use Tokenizer::Tokenizer(const std::string&, const std::string&) instead.

Parameters

raw	the string to tokenize

Tokenizer() [2/6]

Tokenizer::Tokenizer	(	const std::string &	raw,
		const char	delim
	)

Creates a Tokenizer and tokenizes the given raw string over the given delimiter.

Parameters

raw	the string to tokenize
delim	the delimiter to tokenize over

Tokenizer() [3/6]

Tokenizer::Tokenizer	(	const std::string &	raw,
		const std::string &	delims
	)

Creates a Tokenizer and tokenizes the given raw string over the given delimiters.

Parameters

raw	the string to tokenize
delims	the delimiters to tokenize over as a string. Each character in the string is interpreted as a delimiter

Tokenizer() [4/6]

Tokenizer::Tokenizer

(

const Token &

raw

)

Creates a Tokenizer and tokenizes the contents of the given token over spaces.

This does not tokenize over tabs, newlines, and carriage returns, use Tokenizer::Tokenizer(const Token&, const std::string&) instead.

Parameters

raw	the Token to tokenize further

Tokenizer() [5/6]

Tokenizer::Tokenizer	(	const Token &	raw,
		const char	delim
	)

Creates a Tokenizer and tokenizes the contents of the given token over the given delimiter.

Parameters

raw	the Token to tokenize further
delim	the delimiter to tokenize over

Tokenizer() [6/6]

Tokenizer::Tokenizer	(	const Token &	raw,
		const std::string &	delims
	)

Creates a Tokenizer and tokenizes the contents of the given token over the given delimiters.

Parameters

raw	the Token to tokenize further
delims	the delimiters to tokenize over as a string. Each character in the string is interpreted as a delimiter

Member Function Documentation

getToken()

const Token & Tokenizer::getToken

(

const size_t

pos

)

const

Gets the Token at the given position in the token vector.

Parameters

pos	the position of the Token

Returns

a const reference to the Token

getTokens()

const std::vector< Token > & Tokenizer::getTokens

(

)

const

Gets a reference to a vector of all of the tokens that were found in the source string.

Returns

an immutable vector reference of all tokens that were found

indexOf()

size_t Tokenizer::indexOf	(	const Token &	token,
		const size_t	start = 0
	)		const

Finds the index of the given Token in the token vector.

Parameters

token	the Token to search for
start	the position to start searching from

setDelims()

void Tokenizer::setDelims

(

const std::string &

delims

)

Sets the delimiters to tokenize over.

Does not retokenize the source.

setSource()

void Tokenizer::setSource

(

const std::string &

raw

)

Sets the source string to tokenize and tokenizes it.

Parameters

raw	the string to tokenize

---

The documentation for this class was generated from the following files:

• Tokenizer.h
• Tokenizer.cpp