Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard does not show exception for retries #375

Closed
DavidLievrouw opened this issue Nov 25, 2023 · 3 comments
Closed

Dashboard does not show exception for retries #375

DavidLievrouw opened this issue Nov 25, 2023 · 3 comments

Comments

@DavidLievrouw
Copy link
Contributor

I have noticed an inconsistency between Hangfire.Mongo and other storages, like in-memory, and Redis (Hangfire.Pro):
When a job is retried, the encountered exception is not shown in the dashboard, when using Hangfire.Mongo.

When using any of the other storages I tested, the exceptions are shown on every retry.

When I use the ASP.NET Core sample, and make a job fail, I get this.

DefaultBehaviour

Any of the others do this (notice the Failed-state with exception information):

DesiredBehaviour

I traced the issue down to the MongoWriteOnlyTransaction class. It only saves the latest StateHistory item to the database, instead of all the ones that were encountered, during handling of all the configured state filters.

I don't know if this behavior is the desired behavior. It might be for performance reasons. But in any case, to change it, I did the following:

MongoJobUpdates.cs

/// <summary>
/// Updates for a specific job
/// </summary>
public class MongoJobUpdates
{
    /// <summary>
    /// Set updates
    /// </summary>
    public BsonDocument Set { get; } = new();

    /// <summary>
    /// Push updates
    /// </summary>
    public List<BsonDocument> Pushes { get; } = new();

    /// <summary>
    /// Creates a UpdateOneModel with a filter for the given job id
    /// </summary>
    /// <param name="jobId"></param>
    /// <returns></returns>
    public UpdateOneModel<BsonDocument> CreateUpdateModel(string jobId)
    {
        var filter = new BsonDocument("_id", ObjectId.Parse(jobId));
        var update = new BsonDocument();
        if (Set.Any())
        {
            update["$set"] = Set;
        }

        if (Pushes.Any())
        {
            var pushByElement = Pushes
                .SelectMany(p => p)
                .GroupBy(elem => elem.Name)
                .Select(g => new BsonElement(g.Key, new BsonDocument("$each", new BsonArray(g.Select(e => e.Value)))));

            update["$push"] = new BsonDocument(pushByElement
                .ToDictionary(p => p.Name, p => p.Value));
        }


        var updateModel = new UpdateOneModel<BsonDocument>(filter, update);
        return updateModel;
    }
}

MongoWriteOnlyTransaction.cs

 public override void SetJobState(string jobId, IState state)
        {
            var stateDto = new StateDto
            {
                Name = state.Name,
                Reason = state.Reason,
                CreatedAt = DateTime.UtcNow,
                Data = state.SerializeData()
            }.Serialize();

            var updates = GetOrAddJobUpdates(jobId);
            updates.Set[nameof(JobDto.StateName)] = state.Name;
            updates.Pushes.Add(new BsonDocument(new Dictionary<string, object>
            {
                { nameof(JobDto.StateHistory), stateDto }
            }));
        }

        public override void AddJobState(string jobId, IState state)
        {
            var stateDto = new StateDto
            {
                Name = state.Name,
                Reason = state.Reason,
                CreatedAt = DateTime.UtcNow,
                Data = state.SerializeData()
            }.Serialize();

            var updates = GetOrAddJobUpdates(jobId);
            updates.Pushes.Add(new BsonDocument(new Dictionary<string, object>
            {
                { nameof(JobDto.StateHistory), stateDto }
            }));
        }

Now the exceptions are shown for every retry.

Should I create a PR, or is the current behavior as intended?

Thanks for your efforts.

@gottscj
Copy link
Owner

gottscj commented Nov 27, 2023

@DavidLievrouw,

Thank you for the feedback! I can do the required updates if you like. Im also looking into an optimization another place in the MongoWriteOnlyTransaction anyways.

Thanks!

gottscj added a commit that referenced this issue Nov 28, 2023
fix Dashboard does not show exception for retries (#375)
fix Slow operations reported by Atlas (#374)
gottscj added a commit that referenced this issue Nov 28, 2023
* optimize trimming list and show retry info in dashboard

fix Dashboard does not show exception for retries (#375)
fix Slow operations reported by Atlas (#374)

* update changelog and version
@gottscj
Copy link
Owner

gottscj commented Nov 28, 2023

@DavidLievrouw,

I added your changes. Please let me know if it works as you expect.

Thanks for the help!

@DavidLievrouw
Copy link
Contributor Author

Perfect! Thank you.

@gottscj gottscj closed this as completed Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants