refactor: move lib/llm/src/tokens.rs to lib/tokens

The crate `lib/tokens` was not really used and contained old duplicate
code. Looks like the origin intention was to split tokens functionality
out of `dynamo-runtime`, so let's complete it.

To avoid too many changes, keep a redirect with `pub use dynamo_tokens::*;`.

Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>

Author: da-x

Cargo.lock

@@ -44,6 +44,7 @@ harness = false
 [dependencies]
 # repo
 dynamo-runtime = { workspace = true }
+dynamo-tokens = { workspace = true }
 
 # workspace
 anyhow = { workspace = true }