(handle_newline, check_long_token, skip_escaped_newlines,
unterminated): New functions.
(ACCEPT_CHAR, SAVE_STATE, RESTORE_STATE): New macros.
(parse_identifier): Was parse_name, new implementation.
(skip_line_comment, skip_block_comment, skip_whitespace,
parse_number, parse_string, trigraph_ok, save_comment,
adjust_column, _cpp_get_line): New implementations.
(lex_token): New function. Lexes a token at a time, looking
forwards. Contains most of the guts of the old lex_line.
(lex_line): New implementation, using lex_token to obtain
individual tokens.
(cpp_scan_buffer): Use the token's line, not the list's line.
* cpplib.c (_cpp_check_directive, _cpp_check_linemarker):
New implementations.
(do_assert): Don't bother setting the answer's list's line.
(cpp_push_buffer): Initialise new pfile and read_ahead members
of struct cpp_buffer.
* cpplib.h (cppchar_t): New typedef.
(struct cpp_buffer): read_ahead, pfile and col_adjust are
new members.
(struct lexer_state): New structure that determines the state
and behaviour of the lexer.
(IN_DIRECTIVE, KNOWN_DIRECTIVE): New macros.
(struct cpp_reader): New member "state". Rename
multiline_string_line and multiline_string_column. Delete
col_adjust, in_lex_line members.
(CPP_BUF_COLUMN): Update.