Add operator lexing

[adamhutchings]

No description provided.

[adamhutchings]

Do not want to merge yet until token type identification is fixed.

[TheCrystalKeeper]

Seems like it doesn't break anything, but I'm convinced that the code

        ungetc(c, l->fp);
        t->contents[pos - 1] = '\0';
        t->type = ttype_from_string(t->contents);
        t->length = pos;
        return 0;

doesn't need to be duplicated inside of lex()

[adamhutchings]

I could #define a macro to do that, or goto an end block which would be called finalize_token or something similar. What do you think?

[TheCrystalKeeper]

Try adding a return 1 to those places where you want to void the values of t. Then inside of lex() if it is determined to be 1, set t appropriately. I think that makes it just clearer.

[JakeRoggenbuck]

A few lines is okay imo