Java Function Host failing with Class Cast Exception

[TurtleBeach]

Check for a solution in the Azure portal

For issues in production, please check for a solution to common issues in the Azure portal before opening a bug. In the Azure portal, navigate to your function app => Platform features => Diagnose and solve problems and the relevant dashboards before opening your issue.

Investigative information

Java Function App started failing for a function that is IN PRODUCTION and has been running for months
Result: Failure
Exception: ClassCastException: com.microsoft.azure.functions.shaded.com.google.gson.internal.LinkedTreeMap cannot be cast to com.google.gson.internal.LinkedTreeMap
Stack: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
at com.microsoft.azure.functions.worker.broker.JavaMethodExecutorImpl.execute(JavaMethodExecutorImpl.java:54)
at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:57)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)
at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: com.microsoft.azure.functions.shaded.com.google.gson.internal.LinkedTreeMap cannot be cast to com.google.gson.internal.LinkedTreeMap
at ai.com.neuralstudio.functions.GetJobPrice.httpHandler(GetJobPrice.java:410)

Please provide the following:

• Timestamp:
  2020-08-14 15:57:31

• Function App version:

• Function App name:

• Function name(s) (as appropriate):

• Invocation ID: (Failed, Id=880b2f47-1eaa-48fa-b1fe-3ebe9d776728, Duration=23ms

• Region: EAST US

Repro steps

NO REPRO - LOOK AT THE FUNCTION LOGS!

This is ridiculous. I should not have to debug / change my code for this kind of failure. What did MS do to the Java function host???

Provide the steps required to reproduce the problem:

Expected behavior

Provide a description of the expected behavior.

Actual behavior

Provide a description of the actual behavior observed.

Known workarounds

Provide a description of any known workarounds.

Related information

Provide any related information

• Programming language used
• Links to source
• Bindings used

[TurtleBeach]

Additional information - I was able to work around this by pinning the function to 2.0.14192.

1. WHERE does the ...microsoft.azure.functions.shaded.. garbage come from? I complained over a year ago about having to deal with internal Google classes as a result of Microsoft deciding to use gson, which also added a huge amount of unused code to Java functions. But I fixed my functions to deal with the Google class. Now MS is messing it up? IN A PRODUCTION RUNTIME???
2. WHY does MS deploy Java runtimes BEFORE they are available as a standard update to VS Code?? I get messages to update the core tools, and I do so within days. BUT THE JAVA Runtime was NOT updated with the most recent core tools update. I have also asked how to force that update, and have NEVER gotten a straight answer.

I am embarrassed to admit that I actually pay Microsoft for the privilege of using Azure functions.
Incredible.

[amamounelsayed]

@TurtleBeach We are sorry for any inconvenience. We are investigating the app to check what may caused this issue.

[amamounelsayed]

@TurtleBeach Can you please share sample of the code that created the issue?

[TurtleBeach]

The JSON payload signature is … HttpRequestMessage<HashMap<String, Object>> request, … .

The payload comprises two or more inner HashMap<String, Object> components (e.g., but obviously not the exact content):
{ "User":{ "A":"string", "B": true }, "key2":{ "C": 42, "D": false } }
The "outer" object is the expected HashMap. The structure is such that it "should" be interpreted as containing two key - value pairs in which the key is a string and the value is another object which in turn should be yet another HashMap.

The outer object is retrieved (after checking that body is not null, and is in fact a HashMap) using:
HashMap<String, Object> orderJSON = (HashMap<String, Object>) body;

At the point where I retrieve the "inner" objects, there is a comment in my code:

// 2018-11-19 if we get here extract the User Information Map
// NOTE that it must be cast as a LinkedTreeMap due to Gson

Line 410 (immediately beneath the comment) that fails with Runtime 2.0.14248 is very simple:
LinkedTreeMap<String, Object> userInfo = (LinkedTreeMap<String, Object>)orderJSON.get("User");

With Runtime 2.0.14192 (and, I believe 2.0.14230 - I am not willing to dig into the inscrutable MS documentation to try to figure out when 2.0.14230 was deployed, but I think the function was running with that deployment) there was no problem with that line and the function worked.

NOTE THE DATE of the comment - it was November 2018, and I complained then about MS deciding to use Gson and changing internal representations - because prior to that the "inner" objects were treated as HashMaps, as I and my original code expected, based on the structure of the payload! In any case the function has worked since November 2018 - with several logical changes and additions, but ALWAYS getting the internal objects as LinkedHashMaps.

Hence, my extreme frustration when almost 2 years later, with Java functions at GA level for quite some time, MS makes yet another inexplicable internal change.

Even more frustrating, as I noted above, my development environment (VS Code) "appears" to be the latest (core tools 2.7.2796). YET when I run the function locally, I am using Runtime version 2.0.14192.0. So I can't even reasonably debug, because I cannot find how to force an update to the latest Runtime.

I repeat my question from above: in what Universe, other than the MS Universe, is it rational or appropriate to deploy code to production before the code is available to the tool chain?? Acknowledging that sometimes I miss things that are staring me in the face, I spent an hour yesterday trying to find out how to force VS Code to get the latest Runtime. Nada. Zilch. Zip.

So in addition to explaining why the runtime treatment of JSON changed, would some one please explicitly post how to force VS Code to get the latest Java runtime.

[amamounelsayed]

@TurtleBeach Thank you so much for your detailed explanation for the issue. It is in our top priority to handle this case in java worker and will keep the thread update it.
Regarding the VScode and the tooling question we will follow up with your questions and get back to you.

Thank you so much for your time

[amamounelsayed]

@TurtleBeach we continue to investigate the issue.

We implemented the same code you shared with the interface Map and it worked instead of using LinkedTreeMap https://github.com/google/gson/blob/master/gson/src/main/java/com/google/gson/internal/LinkedTreeMap.java.

Here is the test case,
HashMap<String, Object> orderJSON = request.getBody();
Map<String, Object> userInfo = (Map<String, Object>) orderJSON.get("User");
String a = (String)userInfo.get("A");
boolean b = (Boolean) userInfo.get("B");

Tested example: { "User":{ "A":"string", "B": true }, "key2":{ "C": 42, "D": false } }

By this way the function code will not be tightly coupled with the internal implementation of java worker.

One other idea that it may help, if the function would like to have full control on the implementation, instead of using the HashMap passed as the object type, just pass String then parse the json string to any format the function may requires.

For the second part part of your question, why we shad, we had a pin issue that has all the details, the objective is the start to decouple worker and customer code jar dependencies,
#381

For the code release, VScode uses the coretools, we release the core tools code after releasing the runtime. We will discuss with the team your feedback, really appreciate it.

On different note,there is future announcement please be aware of,
#388

[TurtleBeach]

@amamounelsayed Thank you for your response. Yes, using a Map interface will work, and I will update my code to use it. But that still begs the question of why (going back to 2018) it was necessary to change to a library that used and exposed a non-standard class. I try to avoid as much as possible (a) anything Google (difficult, yes) and (b) anything not inherent in Java (acknowledging that Map is a Java interface, but HashMap is also a well documented and tested Java class; back when I started this trek there was no reason to expect the mapping of a JSON payload to a simple HashMap would change).

I look forward to the response about updating runtimes in VS Code, and I have one more basic question about core tools:

I have been unable to find a clear, unambiguous statement about whether core tools 2 and 3 (or, more specifically, V2 and V3 of Functions) can coexist on the same (physical) machine. Yes, I could spend an hour or two or whatever it takes to empirically find out, but why can't that basic information be published right where Function versions, and how to install them, are described? With all due modesty, I have debugged production software using front panel switches on LSI 11 and other computers located in a foreign (to me) country with no support from anywhere - so yes, I can pretty much do anything eventually. But it irks me when I have to waste time and effort 'discovering' very basic capabilities and restrictions of tools in 2021, when the people who build the tools could answer such to me basic, and important to developers, questions immediately.

[amamounelsayed]

@TurtleBeach Thank you so much for your feedback, the team is always working on enhancing the product.

Answering your questions related to VScode, VS code by default uses the core tools package installed via NPM. We will discuss with the team your feedback about can we update coretools first before runtime. Azure/azure-functions-core-tools#2162

For core tools versions, Core tools are built as self contained packages. You can have multiple versions of core tools running on the same physical machine. You can download the versions of core tools you want to run from here the core tools release page https://github.com/Azure/azure-functions-core-tools/releases.

For any further details for core tools https://github.com/Azure/azure-functions-core-tools
Also VScode https://github.com/microsoft/vscode-azurefunctions/

Thank you so much!

[amamounelsayed]

@TurtleBeach I am happy to share that we released the core tools. Thank you so much!

Core tools have been release
v3 3.0.2852 or higher
v2 2.7.2855 or higher

https://github.com/Azure/azure-functions-core-tools/releases