Output Processing

[hexthedev]

OpenAI spits out some funny outputs with lots of // and stuff. It would be nice to add a processing layer to the output with an interface so that output processing can be customized.

Adding this functionality, or at least the interfaces, would facilitate the output of the REST adapter layer to the consuming layer.

[hexthedev]

@syedbdusa

[abethke]

In case its useful to anyone this is what I did to compensate for the junk in the responses.

        string raw = comp.Result.choices[0].text;
        raw = raw.Replace("\\\\", "\\");
        raw = raw.Replace("\\", "\\u");
        raw = System.Text.RegularExpressions.Regex.Unescape(raw);

[hexthedev]

Awesome, thanks for this.

I was actually thinking of starting up a new repo called the "Unity OpenAI Toolkit", or something along those lines, as a Unity package that that will depend on the API wrapper, but provide some of these higher level tools.

If you have any other ideas, please let me know.

[abethke]

I revised my code after extensive testing showed some Unicode parsing errors triggering an Argument exception when trying to regrex decode the unicode. Here is the revised parsing to 'clean' the GPT response string.

        result = result.Replace("\\\\00a0", "");
        result = result.Replace("\\\\2019", "'");
        result = result.Replace("\\\"", "\"");
        result = result.Replace("\\\\", "\"");

[abethke]

Ok here's what I've figured out... It's UTF16/32 that's been escaped some how.

All of the \\2019 (and similar) are actually supposed to be \u2019 which Unity then reads and displays correctly; in this case as an apostrophe.

Having said that I've been struggling on and off for weeks trying to get this to reparse on my side properly and am close to giving up and just adding a ton of Replace commands for specific characters that are showing up commonly.

[abethke]

Here's the full and ultimate fix...

    string ConvertHexIntToAsciiString(string in_string)
    {
        try
        {
            int testValue = int.Parse(in_string);
            int intAscii = Convert.ToInt32(testValue.ToString(), 16);
            char resultChar = (char)intAscii;
            return resultChar.ToString();
        }
        catch (Exception e)
        {
            return in_string;
        }
    }
    string CleanUnicodeJunk(string in_string)
    {
        // clean the leading junk which I've never identified? Capitalization unicode?
        in_string = in_string.Replace("\\\\00a0", "");

        string junkTag = "\\\\";
        while (in_string.Contains(junkTag))
        {
            int index = in_string.IndexOf(junkTag);
            string hexString = in_string.Substring(index + 2, 4);
            string unicodeAscii = ConvertHexIntToAsciiString(hexString);
            in_string = in_string.Replace(junkTag + hexString, unicodeAscii);
        }
        return in_string;
    }