Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout exceptions with no discernible cause in java scenarios #3081

Closed
paulbatum opened this issue Jun 29, 2018 · 4 comments
Closed

Timeout exceptions with no discernible cause in java scenarios #3081

paulbatum opened this issue Jun 29, 2018 · 4 comments

Comments

@paulbatum
Copy link
Member

I had a case reported where a Java function app was in a bad state (functions not executing successfully). The error logs show timeouts on the invocation path:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.GetJobProgress ---> System.AggregateException : One or more errors occurred. (The operation has timed out.) (The operation has timed out.) (The operation has timed out.) ---> The operation has timed out.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Rpc\WorkerLanguageInvoker.cs : 73
<snip>

At other points of time, the application appears to be working OK.

Unfortunately the logs provide no further insight into what was going wrong.

Query showing some failures:

//wawseus
FunctionsLogs
| where PreciseTimeStamp > datetime(2018-06-28 17:00)
| where PreciseTimeStamp < datetime(2018-06-28 20:00)
| where HostInstanceId == "2b969b05-f405-43e6-9299-58951dad963a"

Query showing successful executions at a similar timeframe:

FunctionsLogs
| where PreciseTimeStamp > datetime(2018-06-28 17:00)
| where PreciseTimeStamp < datetime(2018-06-28 20:00)
| where AppName =~ "run-price-functions-20171122104322436"
| where HostInstanceId == "ed17854e-b4ce-4344-93e5-c81049c847bc"

One possibility occurs to me - the logs show the following:
Start Process: D:\Program Files\Java\zulu8.23.0.3-jdk8.0.144-win_x64\bin\java

I know there were some recent changes around delayed acquisition of java to help with disk space issues. Is it possible that this is happening and causing delays that result in timeout errors?

@paulbatum paulbatum changed the title Timeout exceptions with no discernible cause in out of proc scenarios Timeout exceptions with no discernible cause in java scenarios Jun 29, 2018
@brettsam brettsam added this to the Triaged milestone Jul 6, 2018
@paulbatum paulbatum modified the milestones: Triaged, Functions Sprint 28 Jul 10, 2018
@pragnagopa
Copy link
Member

cc: @shrishrirang

@paulbatum
Copy link
Member Author

I have recieved further confirmation that there have been changes to how Java is made available in app service and these do involve lazy acquisition that can take significant periods of time. I am following up internally with the right teams to try to get this addressed. I strongly suspect this is the cause of the timeouts.

@paulbatum
Copy link
Member Author

java specific issue, so I'm removing the tracking label for V2 GA.

@pragnagopa
Copy link
Member

fix checked in #3405

@ghost ghost locked as resolved and limited conversation to collaborators Jan 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants