Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use provided TextWriter or TraceWriter make webjob crash #1042

Closed
EmmanuelEngel opened this issue Mar 6, 2017 · 10 comments
Closed

Use provided TextWriter or TraceWriter make webjob crash #1042

EmmanuelEngel opened this issue Mar 6, 2017 · 10 comments

Comments

@EmmanuelEngel
Copy link

Simple webjob that use provided logger crash. Exception is not catchable.

Repro steps

Provide the steps required to reproduce the problem

  1. Set up project with this simple code into Program.cs

using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using System.Threading;
using Microsoft.Azure.WebJobs;
using System;
using Microsoft.Azure.WebJobs.Host;

namespace WebJob
{
public class Functions
{
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.

    public static object locker = new object();

    [NoAutomaticTrigger]
    //public static async Task ProcessQueueMessage(TextWriter log)
    public static async Task ProcessQueueMessage(TraceWriter log)
    {
        try
        {
            var w = log;
            w.Level = System.Diagnostics.TraceLevel.Verbose;
            //using (TextWriter w = TextWriter.Synchronized(log))
            {
                List<Task> tasks = new List<Task>();
                int count = 0;
                for (int i = 0; i < 500; ++i)
                {
                    int k = i;
                    Task pending = Task.Run(() =>
                    {
                        for (int j = 0; j < 500; ++j)
                        {
                            try
                            {
                                //w.WriteLine($"[{Interlocked.Increment(ref count)}][{k}] : {j}");
                                w.Info($"[{Interlocked.Increment(ref count)}][{k}] : {j}");
                            }
                            catch (Exception)
                            {                                 
                            }
                        }
                    });
                    tasks.Add(pending);
                }
                await Task.WhenAll(tasks);
            }
        } catch (Exception)
        {
            // nop
        }
    }
}

}

  1. Step B

Deploy it on a web job and start it

Expected behavior

No crash

Actual behavior

Crash:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ProcessQueueMessage ---> System.IndexOutOfRangeException: Index was outside the bounds of the array. at System.Text.StringBuilder.get_Chars(Int32 index) at Microsoft.Azure.WebJobs.Host.Bindings.TextWriterTraceAdapter.Write(Char value) at System.IO.TextWriter.Write(Char[] buffer, Int32 index, Int32 count) at System.IO.TextWriter.WriteLine(String value) at WebJob.Functions.<>c__DisplayClass0_2.b__0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at WebJob.Functions.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__2c.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__13.MoveNext() --- End of inner exception stack trace --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__1.MoveNext()

Known workarounds

None if you want to use WebJob logger (and kudu)

Related information

Look like #675. Definitely NOT closed.

  • Package version

Microsoft.Azure.WebJobs.Core :2.0.0
Newtonsoft.Json : 9.0.1
WindowsAzure.Storage: 8.1.1

  • Links to source

Seen below
@brettsam
Copy link
Member

brettsam commented Mar 6, 2017

Thanks for filing this -- I want to focus on the TraceWriter issue you're seeing because:

I suspect that you have a different stack when using TraceWriter -- the stack you've included has TextWriterTraceAdapter in it, which only happens if you log with the TextWriter. Would you mind retrying this with TraceWriter and paste that error stack into here? I've taken your code and run it in a WebJob and it worked as expected.

@EmmanuelEngel
Copy link
Author

EmmanuelEngel commented Mar 6, 2017 via email

@brettsam
Copy link
Member

brettsam commented Mar 6, 2017

I still see TextWriterTraceAdapter in that stack, so something doesn't look right. Is this for the version that uses TraceWriter?

That being said, I still can't seem to repro this even with TextWriter. If I can do that, it means I've set things up correctly, so I'd like to focus on that first.

A couple more questions that may help me set this up:

  1. You are running this manually via the Dashboard, right? Does it happen with a single function run? Or do you need to run a bunch of them?
  2. Where do you find this exception trace? In the output for the function in the dashboard?
  3. Just to double-check: you are running on an S1 instance?

@mathewc mathewc changed the title Use provided TextWrtier or TraceWriter make webjob crash Use provided TextWriter or TraceWriter make webjob crash Mar 6, 2017
@EmmanuelEngel
Copy link
Author

EmmanuelEngel commented Mar 6, 2017 via email

@EmmanuelEngel
Copy link
Author

EmmanuelEngel commented Mar 6, 2017 via email

@brettsam brettsam mentioned this issue Mar 6, 2017
Merged
@mathewc
Copy link
Member

mathewc commented Mar 7, 2017

This can be closed now, right? Perhaps we can get @EmmanuelEngel to try the patched version and verify the fix.

@brettsam
Copy link
Member

brettsam commented Mar 7, 2017

Yes, this fixes #982 as well.

@brettsam brettsam closed this as completed Mar 7, 2017
@brettsam
Copy link
Member

brettsam commented Mar 7, 2017

@EmmanuelEngel -- I've updated our myget feed with this latest package if you'd like to try it out. TextWriter should now work.

You can find info on getting our latest builds here: https://github.com/Azure/azure-webjobs-sdk/wiki/%22Nightly%22-Builds

@EmmanuelEngel
Copy link
Author

EmmanuelEngel commented Mar 7, 2017 via email

@mathewc
Copy link
Member

mathewc commented Mar 7, 2017

Thanks - if things look good, we'll try to get a 2.0.1 release with this out quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mathewc @brettsam @EmmanuelEngel