class Drizzle::Lexer

Create a Lexer instance using a String. This string will be split on newline characters and turned into an array of lines, as if it had come from a File.

Since this should only be called from the REPL, the file_name will be "<stdin>".

Create a Lexer instance using a File instance.

This will replace the filename and load the lines in to use as input.

Generate the next Token instance from the given input.

This method first tries the current_char of the Lexer against all of the single character Tokens in Drizzle. If it does not match, it then attempts to build up identifiers / keywords or numbers, depending on what the character is.

Check what the next character on the current line is, without updating all the pointers

Builds up a possible identifier from the source.

This method is run whenever #get_next_token comes across a letter in the input.

Update the char_num and read_char_num pointers, as well as the current_line and current_char values. If the end of the current line has been reached, move to the next line (if exists). If the end of the input has been reached, set the current char to be the Char::ZERO.

Builds up an integer number from the source code.

The interpreter book only handles integer numbers when I move to creating an ANTLR parser then Drizzle will have other number types too :D The number is returned as a String still, to keep Token implementation simple

Once a comment is found in the source, skip the lexer to the next line.

In Drizzle, there are currently only single line comments, so we can just skip the line.

Skip whitespace characters in the input as it is unnecessary to turn them into Tokens for Drizzle

Whitespace characters include spaces, newlines, tabs, carriage returns, etc.

Determine whether a given character is allowed to be used as part of an identifier name.

Since the #read_identifier method is only run when the #get_next_token finds a letter, we can safely allow numbers in this as well.