Poor Benchmark Performance

[DanielStout5]

I'm evaluating various C# templating libraries. I saw that the Cottle library had a benchmark that included RazorLight, so I added RazorEngineCore to it.

The results are very bad, so I'm thinking there must be something wrong with how I wrote the benchmark.

Here are some of the results:

As you can see, RazorEngineCore is ~321 times slower than the next slowest library.

Here's the implementation:

const string content = @"
<ul id='products'>
@foreach (var product in Model.Products)
{
  <li>
    <h2>@product.Name</h2>
    <p>@product.Description.Substring(0, System.Math.Min(product.Description.Length, 15)) - Only @product.Price.ToString(""f1"", System.Globalization.CultureInfo.CreateSpecificCulture(""en-US""))$</p>
  </li>
}
</ul>";

var context = new { Products = CompareEngine.Products };

return () =>
{
    var engine = new RazorEngine();
    return () => engine.Compile(content).Run(context);
};

For comparison, here's the code for the RazorLight benchmark:

const string content = @"
<ul id='products'>
@foreach (var product in Model.Products)
{
  <li>
    <h2>@product.Name</h2>
    <p>@product.Description.Substring(0, System.Math.Min(product.Description.Length, 15)) - Only @product.Price.ToString(""f1"", System.Globalization.CultureInfo.CreateSpecificCulture(""en-US""))$</p>
</li>
}
</ul>";

var context = new { CompareEngine.Products };

return () =>
{
    var engine = new RazorLightEngineBuilder().UseProject(new StringRazorLightProject(content)).Build();
    return () => engine.CompileRenderAsync(string.Empty, context).Result;
};

[DanielStout5]

I tried changing the benchmark like so:

return () =>
{
    var engine = new RazorEngine();
    var template = engine.Compile(content);
    return () => template.Run(context);
};

Which compiles the template as part of the setup step instead of the render step. The results are still very bad - but it just moved most of the time from the render to the setup step. Note that the time is now in microseconds instead of nanoseconds like the previous screenshots:

In that run, RazorEngineCore's setup (as in, compiling the template) was 469 times slower than the next slowest (RazorLight).

It does show that the slow part is just the template compilation, not the execution.

It looks like this library might actually be hundreds of times slower than RazorLight (which is already not a fast templating library). My guess is that it's because this library is actually writing temporary files on the filesystem rather than doing everything in-memory.

[adoconnection]

Hi Daniel!
Thank you for running performance tests.

The result depends of what you acually want to measure.
First of all lets classify engines. they are either compile or evaluate document, then, they either do internal caching or not.
For instance compilable template will take long to compile, but will be faster to run compared to evaluated template.

Based on that, what was the operation you were about to measure?

[DanielStout5]

First of all lets classify engines. they are either compile or evaluate document, then, they either do internal caching or not.

All the engines in this benchmark appear to do some kind of compilation so far as I can tell. They all create a template (sometimes called a document) which is then later passed a model to render the final string result.

I also don't believe any internal caching is at play here, since each benchmark execution runs fresh (e.g. RazorLight creates a new engine builder and project for each run)

Here's the file with all the benchmarks in my fork of the Cottle repo: https://github.com/DanielStout5/cottle/blob/master/src/Cottle.Benchmark/Inputs/CompareEngine.cs

Please let me know if there's a way to use RazorEngineCore with better performance! I find your library has a better API than the only real alternative I can find (RazorLight)

For instance compilable template will take long to compile, but will be faster to run compared to evaluated template.

Yes, but RazorEngineCore is not even the fastest to render, despite having a compile step that is literally hundreds of times slower than the already-not-speedy RazorLight

Based on that, what was the operation you were about to measure?

See the benchmark code for more details, but what we're measuring is 1. Compiling the template and 2. Rendering the string from it

[adoconnection]

RazorLight creates a new engine builder and project for each run

it still does internal caching.
Calling CompileRenderAsync will eventually call CompileTemplateAsync which will try use cache
https://github.com/toddams/RazorLight/blob/22381db30ca3c61b4b1e4004e1536407dd138e49/src/RazorLight/EngineHandler.cs#L50

You measure compile/parse + render which is gives you incorrect results.

To get right values you should compile once

IRazorEngine razorEngine = new RazorEngine();
IRazorEngineCompiledTemplate template = razorEngine.Compile("Hello @Model.Name");

and then measure Run method

string result = template.Run(new
{
    Name = "Alexander"
});

[DanielStout5]

Are you certain that's how RazorLight works?

I took a quick look and in the Build method of RazorLightEngineBuilder, it creates a new EngineHandler (https://github.com/toddams/RazorLight/blob/22381db30ca3c61b4b1e4004e1536407dd138e49/src/RazorLight/RazorLightEngineBuilder.cs) which has its own cache.

So my impression is that each run of the benchmark Razorlight will have a separate cache and won't benefit from previous executions.

[adoconnection]

I am. There is no magic :)

[adoconnection]

have a look at codeproject article to have a sense whats going on under the hood:
https://www.codeproject.com/Articles/5260233/Building-String-Razor-Template-Engine-with-Bare-Ha

neither me or RazorLight cant make RazorEngine work faster of slower, and the same for compiling source code.
so the performance would be quite the same +-
However RazorLight is more abstract way of using RazorEngine.

full story: #58 😸

[DanielStout5]

What about this?

In RazorEngineCore you do:

string fileName = string.IsNullOrWhiteSpace(options.TemplateFilename) ? Path.GetRandomFileName() : options.TemplateFilename;
RazorSourceDocument document = RazorSourceDocument.Create(templateSource, fileName);

I.e. it's creating a file in the filesystem

Whereas the equivalent use of RazorSourceDocument in RazorLight is this:

RazorSourceDocument source = RazorSourceDocument.ReadFrom(stream, projectItem.Key);

(And note that in the above benchmark, it uses TextSourceRazorProjectItem, not the file system project item - so stream above comes from new MemoryStream(Encoding.UTF8.GetBytes(_content));)

So I don't think the issue is just that RazorLight is caching here and that's why it does better on the benchmark - it looks to me like RazorEngineCore will be reading and writing to the filesystem whereas RazorLight will just be using data held in memory. That alone seems like it could explain the benchmark performance.

This is the benchmark code for RazorLight:

var engine = new RazorLightEngineBuilder().UseProject(new StringRazorLightProject(content)).Build();
return () => engine.CompileRenderAsync(string.Empty, context).Result;

It's not calling UseMemoryCachingProvider, so my impression is that it won't cache the templates. I'll dig into that a bit further to confirm - I haven't looked much at the RazorLight source code.

Update: I wrote this test in the RazorLight repo:

[Fact]
public async Task CompileTextSource()
{
	var content = "Hi @Model.Name";
	var model = new { Name = "Bob" };
	var eng = new RazorLightEngineBuilder().UseProject(new TextRazorLightProject(content)).Build();
	var str1 = await eng.CompileRenderAsync("", model);

	var eng2 = new RazorLightEngineBuilder().UseProject(new TextRazorLightProject(content)).Build();
	var str2 = await eng.CompileRenderAsync("", model);
}

private class TextRazorLightProject : RazorLightProject
{
	private readonly string content;
	public TextRazorLightProject(string content)
	{
		this.content = content;
	}

	public override Task<IEnumerable<RazorLightProjectItem>> GetImportsAsync(string templateKey)
	{
		return Task.FromResult(Enumerable.Empty<RazorLightProjectItem>());

	}

	public override Task<RazorLightProjectItem> GetItemAsync(string templateKey)
	{
		return Task.FromResult(new TextSourceRazorProjectItem(templateKey, content) as RazorLightProjectItem);
	}
}

Which is how the Cottle benchmark works. I stepped through the code compiled that same template multiple times and I can confirm RazorLight is NOT caching the template with the above code (because IsCachingEnabled returns false) which means it really is hundreds of times faster than RazorEngineCore. My guess is that it's because of the lack of writing to the filesystem which I described above.

[adoconnection]

https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.razor.language.razorsourcedocument.readfrom?view=aspnetcore-6.0#microsoft-aspnetcore-razor-language-razorsourcedocument-readfrom(system-io-stream-system-string)

[adoconnection]

Im not happy that you can not spend a minute to double check your results and rather start arguing, so im out, sorry

[DanielStout5]

Wow, not sure where you got the vibe that I'm arguing! I'm just trying to understand the results of the benchmark and to help you improve this library. No hostility intended, I assure you. I appreciate all the work you put into this free-licensed library and I'm aware you have no obligation to listen to anything I'm saying here!

I see what you mean about RazorSourceDocument.Create and RazorSourceDocument.ReadFrom being roughly equivalent (Create is directly using the string, ReadFrom has the string being inside a stream) so that's probably not the difference.

If you read the second half of my comment above, you'll see that the benchmark for RazorLight is not using the caching you indicated earlier (i.e. IsCachingEnabled is false in the benchmark). Do you have any other ideas for why the benchmark for RazorLight performs so much better than the one for RazorEngineCore? Is there some other caching that RazorLight does even when IsCachingEnabled is false which RazorEngineCore doesn't do?

[adoconnection]

Hi, dont get me wrong, im friendly person.
I appreciate your effort but its rather me teacheing you then you helping me :) Im happy to share my experience but from my point of view you are rushing in to fast. Give it a break, take a deep breath, keep things simple. If you abandon cool benchmark frameworks and do side by side comparison you will instantly see how things going 👍

using RazorLight;
using RazorEngineCore;

Console.WriteLine("Hello, World!");

DateTime dateTime1;
DateTime dateTime2;
DateTime dateTime3;
DateTime dateTime4;
DateTime dateTime5;

IList<Product> products = new List<Product>
{
    new Product{ Id = 1, Name = "AAA"},
    new Product{ Id = 2, Name = "BBB"},
    new Product{ Id = 3, Name = "CCC"},
};

string template = @"
Hello @Model.Name

@foreach(var product in @Model.Products)
{
    <div>@product.Id / @product.Name</div>
}

";

while (true)
{
    Console.WriteLine("====================================");
    Console.WriteLine("Razor Light");

    dateTime1 = DateTime.Now;
    var engine = new RazorLightEngineBuilder().UseProject(new StringRazorLightProject(template)).Build();
    string result11 = engine.CompileRenderAsync(string.Empty, new { Name = "Test", Products = products }).Result;
    dateTime3 = DateTime.Now;
    string result12 = engine.CompileRenderAsync(string.Empty, new { Name = "Test", Products = products }).Result;
    dateTime4 = DateTime.Now;
    string result13 = engine.CompileRenderAsync(string.Empty, new { Name = "Test", Products = products }).Result;
    dateTime5 = DateTime.Now;

    Console.WriteLine("- Total: " + (dateTime5 - dateTime1).TotalMilliseconds);
    Console.WriteLine("- Run 1: " + (dateTime3 - dateTime1).TotalMilliseconds);
    Console.WriteLine("- Run 2: " + (dateTime4 - dateTime3).TotalMilliseconds);
    Console.WriteLine("- Run 3: " + (dateTime5 - dateTime4).TotalMilliseconds);
    Console.WriteLine("");

    Console.WriteLine("Razor Engine Core");

    dateTime1 = DateTime.Now;
    IRazorEngine razorEngine = new RazorEngine();
    var template2 = razorEngine.Compile(template);
    dateTime2 = DateTime.Now;
    var result21 = template2.Run(new { Name = "Test", Products = products });
    dateTime3 = DateTime.Now;
    var result22 = template2.Run(new { Name = "Test", Products = products });
    dateTime4 = DateTime.Now;
    var result23 = template2.Run(new { Name = "Test", Products = products });
    dateTime5 = DateTime.Now;

    Console.WriteLine("- Total: " + (dateTime5 - dateTime1).TotalMilliseconds);
    Console.WriteLine("- Compile: " + (dateTime2 - dateTime1).TotalMilliseconds);
    Console.WriteLine("- Run 1: " + (dateTime3 - dateTime2).TotalMilliseconds);
    Console.WriteLine("- Run 2: " + (dateTime4 - dateTime3).TotalMilliseconds);
    Console.WriteLine("- Run 3: " + (dateTime5 - dateTime4).TotalMilliseconds);

    Console.WriteLine("");
    Console.WriteLine("Press any key");
    Console.ReadKey();  
}

1. cold start - whatever engine will take that time for the first run, so its not RazorLight's fault
   2, 3 are actual compile and render phases of RazorLight and RazorEngineCore

[DanielStout5]

I don't think an informal benchmark like that is an accurate representation of performance - like you say, when you set it up like that, the order of execution matters (and that doesn't account for GC, JIT optimizations, etc). That's why benchmark frameworks like BenchmarkDotNet (https://github.com/dotnet/BenchmarkDotNet) are used.

But no worries - I won't take up any more of your time. Best of luck with your library!

[seppo498573908457]

I've been testing this library out for some time related my work. I notice that you have many sync'ed functions and async versions for them too. I have not run strict tests, but when looking at the code for RazorEngineTemplateBase.cs I can see that you are marking synchronously executing code as Async and the sync-version just wraps around it. There is no code in that file that will/can execute asynchronously. That is against C# coding conventions, if the code is not really async, do not add unnecessary wrappers. It will only complicate (slow down) things when the callers are choose the "wrong" version. If you must have both versions (to satisfy an interface or other contracts) I'd suggest you turn it around. Instead of:

public virtual string Result()
{
    return ResultAsync().GetAwaiter().GetResult();
}
public virtual Task<string> ResultAsync()
{
    return Task.FromResult<string>(this.stringBuilder.ToString());
}

do:

public virtual string Result()
{
    return this.stringBuilder.ToString();
}
public virtual Task<string> ResultAsync()
{
    return Task.FromResult<string>(Result());
}

And for void-returners, instead of:

public void EndWriteAttribute()
{
    EndWriteAttributeAsync().GetAwaiter().GetResult();
}
public virtual Task EndWriteAttributeAsync()
{
    this.stringBuilder.Append(this.attributeSuffix);
    this.attributeSuffix = null;
    return Task.CompletedTask;
}

do:

public virtual void EndWriteAttribute()
{
    this.stringBuilder.Append(this.attributeSuffix);
    this.attributeSuffix = null;
}
public virtual Task EndWriteAttributeAsync()
{
    EndWriteAttribute();
    return Task.CompletedTask;
}

In the syncronized call you save unnecessary waiters. Async versions will be the same. Also marking them all as virtual would save trouble when people inherit/override. The same pattern repeats in some other files too and the effect can add up measureably.