List of performance regressions caused by switching to ICU

[adamsitnik]

Before 5.0, we were using ICU only on Unix systems. In 5.0 we have decided to use it on Windows by default as well.

This is something that we have done in order to have the same behavior of string-related globalization APIs on every OS.

However, this particular change has affected the performance characteristics of many frequently used methods. Some of them have regressed, some have improved.

Recently we have reported a lot of 5.0 regressions related to that. Since we have done this on purpose and we are most probably not going to revert the switch, I am opening this issue to track the list of all known regressions. When the list becomes complete, we are most probably going to update the 5.0 release docs and close the issue and label it as wont fix.

Please feel free to edit the list.

Known changes:

• System.Memory.ReadOnlySpan.IndexOfString 5 regressions, 5 improvements
• System.Globalization.Tests.StringHash 2 regressions, 4 improvements
• System.Globalization.Tests.StringEquality: 8 regressions, 14 improvements
• System.Globalization.Tests.StringSearch: 30 regressions, 32 improvements
• System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse: 1 regression, 5 improvements

cc @danmosemsft @tarekgh @billwert @DrewScoggins @GrabYourPitchforks @jkotas @safern

[ghost]

Tagging subscribers to this area: @tarekgh, @safern, @krwq
See info in area-owners.md if you want to be subscribed.

[adamsitnik]

System.Memory.ReadOnlySpan.IndexOfString 5 regressions, 5 improvements

Method	input	value	comparisonType	3.1 Mean	5.0 ICU	5.0 NLS
IndexOfString	AAAAA5AAAA	5	InvariantCulture	79.07 ns	35.41 ns	72.05 ns
IndexOfString	AAAAAAAAAAAA(...)AAAAAAAAAAAA [1000]	X	Ordinal	39.72 ns	38.89 ns	38.25 ns
IndexOfString	AAAAAAAAAAAA(...)AAAAAAAAAAAA [100]	x	InvariantCultureIgnoreCase	300.45 ns	214.39 ns	294.59 ns
IndexOfString	AAAAAAAAAAAA(...)AAAAAAAAAAAA [100]	x	OrdinalIgnoreCase	245.94 ns	802.33 ns	240.42 ns
IndexOfString	ABCDE	c	InvariantCultureIgnoreCase	71.81 ns	30.26 ns	63.83 ns
IndexOfString	Hello Worldb(...)allylong!xyz [186]	w	OrdinalIgnoreCase	45.80 ns	102.63 ns	40.30 ns
IndexOfString	Hello Worldb(...)allylong!xyz [187]	~	Ordinal	28.52 ns	22.28 ns	21.03 ns
IndexOfString	Hello Worldbb(...)bbbbbbbbbbba! [47]	y	Ordinal	24.15 ns	14.09 ns	13.57 ns
IndexOfString	More Test's	Tests	OrdinalIgnoreCase	56.38 ns	108.93 ns	50.48 ns
IndexOfString	StrIng	string	OrdinalIgnoreCase	38.79 ns	55.01 ns	33.86 ns
IndexOfString	foobardzsdzs	rddzs	InvariantCulture	101.46 ns	44.93 ns	94.90 ns
IndexOfString	string1	string2	InvariantCulture	91.45 ns	37.14 ns	80.95 ns
IndexOfString	?	?	InvariantCulture	108.70 ns	4,069.11 ns	105.36 ns
IndexOfString	????????????(...)???????????? [100]	?	Ordinal	25.08 ns	16.88 ns	15.48 ns
IndexOfString	????????????(...)???????????? [1000]	x	Ordinal	39.70 ns	39.19 ns	37.99 ns

Comment: InvariantCulture and InvariantCultureIgnoreCase have improved, while OrdinalIgnoreCase has regressed. We could implement OrdinalIgnoreCase path in managed code if it ever becomes a problem

[danmoseley]

Thank you @adamsitnik good idea I'm glad you created this.

[tarekgh]

I have this PR opened for addressing all ordinal operations #40910

[tarekgh]

@adamsitnik are you going to close all other bugs complaining about ICU perf against this one?

[adamsitnik]

are you going to close all other bugs complaining about ICU perf against this one?

That's my plan. I will try to do my best to do it for as many as I can.

[tarekgh]

Also, I want to be clear about the expectation here. I don't think we can fix all perf here as we are limited by calling ICU. we can look at how we can improve it but I am not expecting to get the perf to the point where we used to call NLS. So, it will be good to decide which items in this list is a blocker. The only one was the Ordinal cases which I am addressing in the attached PR. I am not aware of any other blocking scenario. We'll look more of course on other scenarios anyway but I am not sure how much we can do before 5.0 release.

[adamsitnik]

System.Globalization.Tests.StringHash 2 regressions, 4 improvements

Method	Count	Options	3.1 Mean	5.0 ICU Mean	5.0 NLS Mean
GetHashCode	128	(, IgnoreCase)	1,806.07 ns	1,806.0 ns	1,806.10 ns
GetHashCode	128	(, None)	1,823.38 ns	2,284.0 ns	1,818.16 ns
GetHashCode	128	(en-US, IgnoreCase)	1,831.84 ns	1,816.0 ns	1,790.90 ns
GetHashCode	128	(en-US, None)	1,820.91 ns	2,296.8 ns	1,815.97 ns
GetHashCode	128	(en-US, Ordinal)	99.33 ns	100.0 ns	99.61 ns
GetHashCode	128	(en-US, OrdinalIgnoreCase)	123.93 ns	122.6 ns	121.49 ns
GetHashCode	131072	(, IgnoreCase)	2,207,798.50 ns	1,773,953.8 ns	2,193,202.55 ns
GetHashCode	131072	(, None)	2,209,727.34 ns	1,893,550.0 ns	2,187,722.71 ns
GetHashCode	131072	(en-US, IgnoreCase)	2,196,075.39 ns	1,781,730.1 ns	2,190,508.15 ns
GetHashCode	131072	(en-US, None)	2,214,228.87 ns	1,899,934.3 ns	2,185,224.74 ns
GetHashCode	131072	(en-US, Ordinal)	96,294.65 ns	96,513.7 ns	96,551.91 ns
GetHashCode	131072	(en-US, OrdinalIgnoreCase)	113,393.49 ns	112,940.4 ns	116,436.80 ns

Comment: there are two regressions for Count=128 but four improvements for Count=131072

[adamsitnik]

System.Globalization.Tests.StringEquality: 8 regressions, 14 improvements

Method	Count	Options	3.1 Mean	5.0 ICU Mean	5.0 NLS Mean
Compare_Same	1024	(, IgnoreCase)	996.299 ns	575.414 ns	997.39 ns
Compare_Same_Upper	1024	(, IgnoreCase)	3,373.889 ns	6,798.371 ns	3,356.57 ns
Compare_DifferentFirstChar	1024	(, IgnoreCase)	43.730 ns	37.378 ns	42.73 ns
Compare_Same	1024	(, None)	1,003.408 ns	572.528 ns	1,003.25 ns
Compare_Same_Upper	1024	(, None)	4,123.875 ns	6,747.893 ns	4,033.30 ns
Compare_DifferentFirstChar	1024	(, None)	41.883 ns	34.904 ns	43.04 ns
Compare_Same	1024	(en-US, IgnoreCase)	992.625 ns	564.239 ns	1,012.86 ns
Compare_Same_Upper	1024	(en-US, IgnoreCase)	3,334.161 ns	6,789.031 ns	3,336.49 ns
Compare_DifferentFirstChar	1024	(en-US, IgnoreCase)	41.821 ns	36.251 ns	41.92 ns
Compare_Same	1024	(en-US, IgnoreNonSpace)	1,667.106 ns	564.521 ns	1,671.78 ns
Compare_Same_Upper	1024	(en-US, IgnoreNonSpace)	9,547.973 ns	2,103.148 ns	9,599.69 ns
Compare_DifferentFirstChar	1024	(en-US, IgnoreNonSpace)	66.929 ns	35.786 ns	59.21 ns
Compare_Same	1024	(en-US, IgnoreSymbols)	1,676.295 ns	569.468 ns	1,679.39 ns
Compare_Same_Upper	1024	(en-US, IgnoreSymbols)	9,100.891 ns	11,605.391 ns	9,075.81 ns
Compare_DifferentFirstChar	1024	(en-US, IgnoreSymbols)	8,219.648 ns	21,916.551 ns	8,311.14 ns
Compare_Same	1024	(en-US, None)	988.500 ns	572.970 ns	1,005.15 ns
Compare_Same_Upper	1024	(en-US, None)	4,040.998 ns	6,740.818 ns	4,007.38 ns
Compare_DifferentFirstChar	1024	(en-US, None)	41.931 ns	35.349 ns	42.87 ns
Compare_Same	1024	(en-US, Ordinal)	82.818 ns	67.527 ns	68.47 ns
Compare_Same_Upper	1024	(en-US, Ordinal)	13.907 ns	14.796 ns	14.82 ns
Compare_DifferentFirstChar	1024	(en-US, Ordinal)	5.432 ns	9.946 ns	10.27 ns
Compare_Same	1024	(en-US, OrdinalIgnoreCase)	815.592 ns	822.866 ns	834.45 ns
Compare_Same_Upper	1024	(en-US, OrdinalIgnoreCase)	1,225.463 ns	1,218.922 ns	1,253.77 ns
Compare_DifferentFirstChar	1024	(en-US, OrdinalIgnoreCase)	10.517 ns	11.977 ns	11.30 ns
Compare_Same	1024	(pl-PL, None)	21,378.839 ns	572.036 ns	21,617.36 ns
Compare_Same_Upper	1024	(pl-PL, None)	22,136.295 ns	17,518.080 ns	22,375.84 ns
Compare_DifferentFirstChar	1024	(pl-PL, None)	61.794 ns	37.698 ns	62.93 ns

Comment: ICU seems to be faster when inputs are the same or the first character is different but slows down when we are comparing lowercase and uppercase versions of the same words

[adamsitnik]

System.Globalization.Tests.StringSearch: 30 regressions, 32 improvements

Method	Options	3.1 Mean	5.0 ICU Mean	5.0 NLS Mean
IsPrefix_FirstHalf	(, IgnoreCase, False)	279.556 ns	204.122 ns	270.881 ns
IsPrefix_DifferentFirstChar	(, IgnoreCase, False)	58.887 ns	18.055 ns	53.431 ns
IsSuffix_SecondHalf	(, IgnoreCase, False)	264.358 ns	191.353 ns	262.566 ns
IsSuffix_DifferentLastChar	(, IgnoreCase, False)	61.441 ns	18.846 ns	56.044 ns
IndexOf_Word_NotFound	(, IgnoreCase, False)	802.657 ns	628.962 ns	798.926 ns
LastIndexOf_Word_NotFound	(, IgnoreCase, False)	975.995 ns	620.031 ns	965.195 ns
IsPrefix_FirstHalf	(, IgnoreCase, True)	1,012.810 ns	3,677.834 ns	1,019.808 ns
IsPrefix_DifferentFirstChar	(, IgnoreCase, True)	72.985 ns	853.547 ns	72.039 ns
IsSuffix_SecondHalf	(, IgnoreCase, True)	2,552.799 ns	7,467.664 ns	2,575.991 ns
IsSuffix_DifferentLastChar	(, IgnoreCase, True)	5,376.895 ns	1,288.707 ns	5,329.497 ns
IndexOf_Word_NotFound	(, IgnoreCase, True)	3,931.968 ns	11,275.663 ns	3,831.631 ns
LastIndexOf_Word_NotFound	(, IgnoreCase, True)	3,142.703 ns	17,463.707 ns	3,042.517 ns
IsPrefix_FirstHalf	(, None, False)	211.472 ns	173.105 ns	201.325 ns
IsPrefix_DifferentFirstChar	(, None, False)	56.298 ns	17.402 ns	53.396 ns
IsSuffix_SecondHalf	(, None, False)	182.540 ns	174.536 ns	179.486 ns
IsSuffix_DifferentLastChar	(, None, False)	58.275 ns	18.166 ns	55.200 ns
IndexOf_Word_NotFound	(, None, False)	699.751 ns	500.508 ns	678.148 ns
LastIndexOf_Word_NotFound	(, None, False)	850.232 ns	502.581 ns	845.351 ns
IsPrefix_FirstHalf	(, None, True)	1,052.320 ns	3,636.952 ns	1,021.712 ns
IsPrefix_DifferentFirstChar	(, None, True)	73.568 ns	848.922 ns	72.912 ns
IsSuffix_SecondHalf	(, None, True)	2,595.612 ns	7,554.091 ns	2,540.565 ns
IsSuffix_DifferentLastChar	(, None, True)	5,306.733 ns	1,317.628 ns	5,348.958 ns
IndexOf_Word_NotFound	(, None, True)	3,935.667 ns	11,177.835 ns	3,803.958 ns
LastIndexOf_Word_NotFound	(, None, True)	3,083.463 ns	16,993.198 ns	3,029.761 ns
IsPrefix_FirstHalf	(en-US, IgnoreCase, False)	270.134 ns	205.387 ns	273.552 ns
IsPrefix_DifferentFirstChar	(en-US, IgnoreCase, False)	57.763 ns	18.110 ns	54.154 ns
IsSuffix_SecondHalf	(en-US, IgnoreCase, False)	258.893 ns	190.355 ns	260.361 ns
IsSuffix_DifferentLastChar	(en-US, IgnoreCase, False)	58.943 ns	19.272 ns	56.319 ns
IndexOf_Word_NotFound	(en-US, IgnoreCase, False)	807.236 ns	622.869 ns	798.647 ns
LastIndexOf_Word_NotFound	(en-US, IgnoreCase, False)	975.083 ns	623.089 ns	955.210 ns
IsPrefix_FirstHalf	(en-US, IgnoreCase, True)	1,018.976 ns	3,666.344 ns	1,026.411 ns
IsPrefix_DifferentFirstChar	(en-US, IgnoreCase, True)	76.465 ns	887.811 ns	72.117 ns
IsSuffix_SecondHalf	(en-US, IgnoreCase, True)	2,563.798 ns	7,529.526 ns	2,565.718 ns
IsSuffix_DifferentLastChar	(en-US, IgnoreCase, True)	5,305.949 ns	1,304.493 ns	5,346.831 ns
IndexOf_Word_NotFound	(en-US, IgnoreCase, True)	3,892.140 ns	11,776.383 ns	3,926.784 ns
LastIndexOf_Word_NotFound	(en-US, IgnoreCase, True)	3,090.867 ns	17,430.155 ns	3,072.910 ns
IsPrefix_FirstHalf	(en-US, IgnoreNonSpace, False)	1,020.808 ns	178.017 ns	1,017.612 ns
IsPrefix_DifferentFirstChar	(en-US, IgnoreNonSpace, False)	67.336 ns	18.067 ns	65.861 ns
IsSuffix_SecondHalf	(en-US, IgnoreNonSpace, False)	2,552.693 ns	178.425 ns	2,544.768 ns
IsSuffix_DifferentLastChar	(en-US, IgnoreNonSpace, False)	4,921.510 ns	18.199 ns	4,858.353 ns
IndexOf_Word_NotFound	(en-US, IgnoreNonSpace, False)	3,906.457 ns	506.498 ns	3,792.084 ns
LastIndexOf_Word_NotFound	(en-US, IgnoreNonSpace, False)	3,076.625 ns	507.002 ns	3,048.544 ns
IsPrefix_FirstHalf	(en-US, IgnoreSymbols, False)	1,022.075 ns	16,888.924 ns	1,020.346 ns
IsPrefix_DifferentFirstChar	(en-US, IgnoreSymbols, False)	70.421 ns	29,267.090 ns	66.428 ns
IsSuffix_SecondHalf	(en-US, IgnoreSymbols, False)	3,437.616 ns	19,397.341 ns	3,411.783 ns
IsSuffix_DifferentLastChar	(en-US, IgnoreSymbols, False)	5,308.903 ns	34,515.843 ns	5,333.271 ns
IndexOf_Word_NotFound	(en-US, IgnoreSymbols, False)	3,310.485 ns	11,414.777 ns	3,285.185 ns
LastIndexOf_Word_NotFound	(en-US, IgnoreSymbols, False)	3,468.180 ns	16,640.286 ns	3,393.434 ns
IsPrefix_FirstHalf	(en-US, None, False)	205.249 ns	176.915 ns	200.000 ns
IsPrefix_DifferentFirstChar	(en-US, None, False)	59.057 ns	17.779 ns	53.980 ns
IsSuffix_SecondHalf	(en-US, None, False)	184.039 ns	177.072 ns	181.258 ns
IsSuffix_DifferentLastChar	(en-US, None, False)	59.873 ns	18.232 ns	55.897 ns
IndexOf_Word_NotFound	(en-US, None, False)	688.002 ns	504.985 ns	681.283 ns
LastIndexOf_Word_NotFound	(en-US, None, False)	841.277 ns	506.640 ns	859.611 ns
IsPrefix_FirstHalf	(en-US, None, True)	1,026.332 ns	3,676.392 ns	1,129.406 ns
IsPrefix_DifferentFirstChar	(en-US, None, True)	72.958 ns	860.499 ns	73.269 ns
IsSuffix_SecondHalf	(en-US, None, True)	2,555.954 ns	7,625.414 ns	2,556.183 ns
IsSuffix_DifferentLastChar	(en-US, None, True)	5,320.978 ns	1,349.628 ns	5,373.210 ns
IndexOf_Word_NotFound	(en-US, None, True)	3,783.346 ns	11,472.314 ns	3,884.931 ns
LastIndexOf_Word_NotFound	(en-US, None, True)	3,080.818 ns	17,512.610 ns	3,077.301 ns
IsPrefix_FirstHalf	(en-US, Ordinal, False)	12.480 ns	14.469 ns	12.584 ns
IsPrefix_DifferentFirstChar	(en-US, Ordinal, False)	6.229 ns	8.496 ns	8.925 ns
IsSuffix_SecondHalf	(en-US, Ordinal, False)	14.058 ns	12.034 ns	14.248 ns
IsSuffix_DifferentLastChar	(en-US, Ordinal, False)	19.645 ns	17.446 ns	16.225 ns
IndexOf_Word_NotFound	(en-US, Ordinal, False)	39.451 ns	37.728 ns	39.207 ns
LastIndexOf_Word_NotFound	(en-US, Ordinal, False)	137.367 ns	97.667 ns	98.362 ns
IsPrefix_FirstHalf	(en-US, OrdinalIgnoreCase, False)	73.307 ns	82.618 ns	83.575 ns
IsPrefix_DifferentFirstChar	(en-US, OrdinalIgnoreCase, False)	9.844 ns	9.585 ns	9.731 ns
IsSuffix_SecondHalf	(en-US, OrdinalIgnoreCase, False)	109.128 ns	83.181 ns	80.728 ns
IsSuffix_DifferentLastChar	(en-US, OrdinalIgnoreCase, False)	203.785 ns	149.656 ns	141.373 ns
IndexOf_Word_NotFound	(en-US, OrdinalIgnoreCase, False)	701.811 ns	3,125.640 ns	705.609 ns
LastIndexOf_Word_NotFound	(en-US, OrdinalIgnoreCase, False)	617.823 ns	3,161.417 ns	621.380 ns
IsPrefix_FirstHalf	(pl-PL, None, False)	2,594.625 ns	5,704.863 ns	2,558.898 ns
IsPrefix_DifferentFirstChar	(pl-PL, None, False)	101.082 ns	857.463 ns	98.650 ns
IsSuffix_SecondHalf	(pl-PL, None, False)	8,633.560 ns	8,435.046 ns	8,586.963 ns
IsSuffix_DifferentLastChar	(pl-PL, None, False)	19,423.848 ns	1,216.271 ns	17,376.208 ns
IndexOf_Word_NotFound	(pl-PL, None, False)	9,534.684 ns	14,329.673 ns	9,455.980 ns
LastIndexOf_Word_NotFound	(pl-PL, None, False)	8,689.332 ns	18,014.702 ns	8,657.101 ns

[adamsitnik]

System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse: 1 big regression for ja culture, 5 improvements

Method	culturestring	3.1 Mean	5.0 ICU	5.0 NLS
ToStringHebrewIsrael	?	540.1 ns	463.3 ns	518.3 ns
ToString		254.0 ns	248.9 ns	245.7 ns
Parse		466.9 ns	423.7 ns	424.7 ns
ToString	da	247.5 ns	234.0 ns	241.1 ns
Parse	da	551.2 ns	449.3 ns	515.6 ns
ToString	fr	251.5 ns	246.4 ns	244.3 ns
Parse	fr	466.5 ns	424.7 ns	440.2 ns
ToString	ja	258.1 ns	243.5 ns	245.4 ns
Parse	ja	482.1 ns	5,078.9 ns	456.9 ns

Comment: this is a known ICU issue: #31273

[adamsitnik]

are you going to close all other bugs complaining about ICU perf against this one?

@tarekgh I've gone through all System.Memory and System.Globalization issues with performance tag and updated the list. It should be complete now.

[danmoseley]

@Symbai I see you've thumbs-down. Could you share your concerns?

[Symbai]

@danmosemsft Its not about adamsitnik work of collecting the regressions which I find very helpful. Its about the fact that switching to ICU has introduced MAJOR regressions (+4,500ns, ~ 4x slower) while the "improvements" are usually around ~2ns which I bet is only some noise and running them a couple of times will show there are no improvements at all.

I'm not into ICU and I dont know why this change was made at all but seeing all the performance improvements in .NET Core the last versions and now such a big regression on hot code path that is literally being used anywhere, makes me really wonder what in god's name can be that much useful about ICU that it worth THIS regression. I've thumbs-down because I disagree on this ICU change and I disagree on statements which say that "we might not fix some of these regressions". Especially without telling people that switching to .NET 5 will make their code much more slower without a benefit (there might be a benefit in ICU, but I bet it won't affect most people... unlike this performance regression).

[tarekgh]

@Symbai you have the option to switch back to use NLS if you want to.

ICU is the future direction in general for the .NET and Windows too. ICU will give the opportunity to have a consistency between OS's and OS versions. The benefit of using ICU is really worth it. ICU will give opportunity to the apps to customize the globalization behavior too.

[danmoseley]

Also note that this is already what is used on Linux and Mac, and increasingly used within Windows OS itself. So we are aligning with the industry here. If and where it is slow - we all benefit from making it faster. The .NET team have contributed bug reports and performance improvements to libicu in the past, and I expect we will do so again.

[krwq]

As a side note, ICU is open source and we can contribute to make hot paths faster

[danmoseley]

also cc @lewing (although I guess his focus is just size)

[iSazonov]

For information only. PowerShell 7.1 was released on .Net 5.0 and we already got performance feedback PowerShell/PowerShell#14087

[tarekgh]

@iSazonov did you have chance to investigate more and know where is the regression is coming from?

Also, could you please measure the same scenario on .NET 6.0? we have some more optimization there it may help?

[lewing]

@danmosemsft we care about performance a lot too, we just had disproportionately large (small?) size needs here. @tarekgh great work on all this.

PGO of the unmanaged code looks complicated in our case because we'd need a way for our compiler/toolchain to consume the the profile.

[iSazonov]

@iSazonov did you have chance to investigate more and know where is the regression is coming from?

@tarekgh I believe it is an expected regression after moving to ICU. I shared the feedback to demonstrate what users can see in real scenarios.

Also, could you please measure the same scenario on .NET 6.0? we have some more optimization there it may help?

PowerShell repository is still on .Net 5.0. I guess PowerShell MSFT team will move to .Net 6.0 after .Net 6 Preview1 will be available to them. Then we could compare.

If users see such a noticeable regression, I would expect your latest improvements to be ported to .Net 5.0.

As for PGO, I think this could be a good step towards reaping benefits quickly with minimal cost. I assume you already have these statistics from MSFT services. But in general it is obvious which functions are the most used.

More important is how this performance is achieved. I see that there is a desire to make Runtime more compact. This is important for scenarios like WASM. But in other cases, this is not critical. If hardware intrinsics use double and triple implementations, why not do the same for strings? - a compact but slower implementation for WASM and etc. and a faster but more capacious implementation for other scenarios - it is not problem for PowerShell script execution to have additional 10 buffers by 20 Kb or even by 1 Mb.
I can speculate that no one will understand why the Binq service is 20% slower just for the sake of WASM running in the browser. Based on the same test set WASM could be get more compact runtime by means of conditional compiling to use only compact implementations.

[tarekgh]

I believe it is an expected regression after moving to ICU. I shared the feedback to demonstrate what users can see in real scenarios.

@iSazonov I am still unclear what exactly regressed in the user scenario? did you investigate PowerShell and looked where the regression is happening? what calls to .NET is regressed?

[adamsitnik]

the referenced issue (PowerShell/PowerShell#14087) mentions:

(...) on many lines of text to remove lines matching a criteria in a 85MB text file

So it could be string.IndexOf, string.Contains, string.Remove, string.Replace or even RegEx. Let's wait for the user to provide more details

A script of mine that reliably took 26 or 27 minutes to run in 7.0 now takes 40 minutes in 7.1.

It should take less than a minute to handle 85 MB text file. We might be able to provide some recommendations as well.

[adamsitnik]

I've looked into the PowerShell trace files and it looks like the regression is not related to ICU, but Regex.Replace (cc @stephentoub). For more details please see PowerShell/PowerShell#14087 (comment)

We should expect a new issue to be created soon by PowerShell contributors (PowerShell/PowerShell#14087 (comment))

[tarekgh]

To tell, @jefgen thankfully is in doing some optimization work from the ICU side

unicode-org/icu#1471
unicode-org/icu#1473

You can see the perf numbers in the ticket https://unicode-org.atlassian.net/browse/ICU-21388

[adamsitnik]

@jefgen very nice improvements! did Windows Team consider using PGO (Profile Guided Optimization) to improve ICU perf?

[jefgen]

@jefgen very nice improvements!

Thanks!
FWIW, I had a few more ideas that might help to improve the performance more, but I haven't had time to investigate them as of yet.

did Windows Team consider using PGO (Profile Guided Optimization) to improve ICU perf?

Yes the version of ICU that ships as part of the OS with Windows (icu.dll) is indeed using PGO. However, IIUC, I think that the "training data" used for it is from scenarios for the old/legacy Edge browser. I wonder if it might be possible to somehow use training data from .NET Core instead? (I'm not sure if that is possible though, and/or how it might work).

That said, the pre-built Windows binary downloads for the public ICU don't use PGO though.
The Nuget package that our team produces for ICU (with data changes for Microsoft products) also doesn't currently use PGO either. (Both of these would/could be used for the .NET app-local ICU feature).

TBH, I'm not super familiar with PGO though, so I wonder if it might be good to sync up to discuss more sometime perhaps?.

[iSazonov]

I measured PowerShell startup scenario. Command line is pwsh -c exit. PowerShell processes args[] and for the command line it calls once string.ToLowerInvariant() for args[0] that is "c". This takes 26 ms. I guess it is slow ICU initialization.
SpeedScope file:
PerfViewData-7.2-Startup-AI-DT.speedscope.json.zip

Update: I found another slow ICU initialization in the scenario - we mistakenly called string.StartsWith() with a default StringComparison.CurrentCulture - it calls slow icu!ucol_openElements. I replaced with with Ordinal but
the fact remains - ICU is initialized too slow for PowerSHell startup scenario.

[tarekgh]

To update the current status I have done the latest measurement comparing the 3.1 (the Base) against 6.0 (the diff).

Usually I experience some instability in the results of IsPrefix and IsSuffix on my machine so maybe the numbers of these tests are not accurate.

Considering 6.0 faster in some scenarios and slower in other scenarios, we may need to look more at the slower scenarios. But in general, I think we are in much better state even we can do more.

Slower	diff/base	Base Median (ns)	Diff Median (ns)	Modality
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	355.35	63.98	22735.21
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (,	11.89	60.06	714.32
System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja)	11.81	435.96	5150.00
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (,	11.59	59.34	687.94
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	10.59	71.06	752.27
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Igno	9.79	1230.14	12044.71
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	9.58	71.05	680.59
System.Memory.ReadOnlySpan.IndexOfString(input: "?", value: "?", comparisonType:	9.35	89.84	839.52
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (pl	6.92	97.22	672.96
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	4.77	6048.56	28822.60
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	3.97	3387.45	13432.09
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ign	3.85	3875.09	14903.18
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, Ig	3.80	3425.80	13008.36
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	3.79	3406.47	12902.87
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, No	3.79	3375.25	12779.77
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCas	3.26	1225.06	3989.05
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	3.25	3735.02	12154.61
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, Tru	3.20	1231.01	3936.81
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Igno	3.20	1249.21	3991.98
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None	3.17	1246.25	3944.60
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCa	2.19	2962.57	6493.42
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, Tr	2.18	2985.69	6514.83
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Non	2.17	2994.42	6483.37
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ign	2.16	3016.11	6524.22
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	2.06	9793.37	20199.81
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	2.01	5.04	10.12
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (pl-PL, None	1.93	2702.20	5217.48
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.74	3055.79	5309.93
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.72	3064.04	5259.70
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (pl-P	1.67	8432.65	14106.59	several?
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, I	1.63	3512.51	5714.36
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, Fal	1.43	143.80	205.91
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.42	3804.85	5415.19
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, Fa	1.41	183.05	257.96
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.39	3860.38	5362.28
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None	1.38	150.76	207.55
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	1.36	6.29	8.53
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, I	1.35	4352.17	5872.62	several?
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, Ignore	1.33	4349.01	5804.45
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Non	1.33	192.87	256.04
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, N	1.31	4342.61	5683.86
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None,	1.30	4424.18	5730.22
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.18	12.26	14.51
System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (, None))	1.18	1592.80	1882.03
System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, N	1.15	1637.91	1881.80
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	1.11	8.05	8.98
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCa	1.08	214.00	231.22

Faster	base/diff	Base Median (ns)	Diff Median (ns)	Modality
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	327.82	5636.99	17.20
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (pl	21.58	21715.82	1006.19
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (pl-	17.60	17759.59	1009.28
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ign	11.52	2976.54	258.40
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, I	9.97	4352.38	436.54
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	7.90	3339.68	422.59
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Igno	5.94	1227.92	206.88
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, N	5.76	6038.00	1048.36
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	5.70	6012.40	1055.24
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, I	5.66	6028.64	1065.38
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	5.31	6000.28	1130.70
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	3.97	10432.14	2630.57
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	3.97	64.39	16.24
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	3.88	64.89	16.73
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	3.27	53.30	16.32
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	3.18	51.48	16.19
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en	3.10	51.88	16.72
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, N	2.59	43.52	16.79
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (,	2.59	41.51	16.04
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, I	2.58	43.56	16.89
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (,	2.57	43.39	16.87
System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", com	2.48	76.93	30.99
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	2.38	108.67	45.67
System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs",	2.32	86.74	37.46
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	2.18	78.59	36.06
System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", compar	2.11	63.04	29.82
System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonT	2.07	53.78	25.93
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	2.07	878.20	423.61
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, No	2.07	873.42	422.79
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	2.01	1082.01	539.23
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, Ig	2.00	1090.20	544.92
System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", compa	1.98	43.01	21.69
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None,	1.81	771.12	426.00
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, N	1.79	770.64	431.25
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Igno	1.77	274.01	155.07
System.Memory.ReadOnlySpan.IndexOfString(input: "More Test's", value: "Tests", c	1.75	56.81	32.51
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	1.74	62.12	35.63
System.Memory.ReadOnlySpan.IndexOfString(input: "???????????????????????????????	1.71	23.46	13.70
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCas	1.68	263.07	156.53
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	1.65	1234.22	747.17
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, O	1.65	827.37	501.93
System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbcbbbbb	1.61	21.75	13.49
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	1.60	884.82	553.21
System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA	1.56	280.66	179.49
System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbbbbbbb	1.54	42.92	27.95
System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (, Ign	1.52	2208814.73	1456858.52
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	1.51	173.44	114.64
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ord	1.51	91.77	60.81
System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US	1.50	2185948.21	1458638.64
System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbbbbbbb	1.45	26.75	18.43
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, Ignore	1.42	768.69	541.95
System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US	1.39	2175261.61	1562373.13
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.39	22249.54	16009.44
System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (, Non	1.38	2154764.29	1563706.25
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	1.34	1344.50	1004.85
System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA	1.33	38.36	28.86
System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: da)	1.32	507.79	385.35
System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA	1.31	233.70	178.34
System.Memory.ReadOnlySpan.IndexOfString(input: "???????????????????????????????	1.31	37.81	28.91
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	1.31	1325.58	1014.02
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (,	1.30	1299.60	1003.48
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	1.29	1301.69	1010.57
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	1.28	45.46	35.50
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, I	1.28	769.79	601.59
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en	1.27	1285.98	1011.62
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (,	1.25	1261.57	1009.91
System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024	1.24	43.21	34.91
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, O	1.23	38.85	31.67	several?
System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-U	1.22	129.74	106.45
System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: fr)	1.21	437.41	360.07
System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (pl-PL, N	1.19	9184.99	7697.78
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ign	1.18	272.30	230.99
System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: )	1.17	425.10	363.77
System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, I	1.13	1665.63	1470.75
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (pl-PL, Non	1.13	8582.96	7609.07
System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (, Ignore	1.10	1652.53	1503.49
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-	1.09	17.00	15.56
System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Option	1.08	10493.72	9745.46
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ord	1.07	13.45	12.63
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Ordi	1.06	11.74	11.10	several?
System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, O	1.04	101.90	97.68

[L2]

@adamsitnik @tarekgh Friendly ping to see if there's any progress with resolving these ICU related regressions, since it's now the default library.

Seeing this on my end, for example:

Slower	diff/base	NLS - Base Median (ns)	ICU - Diff Median (ns)	Modality
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreSymbols, False))	407.94	54.02	22036.56
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, True))	12.13	57.79	700.76
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, True))	11.94	57.98	691.99
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, True))	11.79	57.83	681.78
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, True))	11.68	58.88	688.01
System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (pl-PL, None, False))	8.70	79.92	695.66

Thanks

[tarekgh]

@L2 We are not actively looking at this at the current time for other high-priority work. Using options like IgnoreSymbols is expected to have more cost because it is done with the slow path. Could you please talk more about your scenario to try to see if there is any workaround for now?

[L2]

Thanks, @tarekgh . It was mainly due to the difference seen for these benchmarks when comparing the previous NLS library to the now default ICU library. I've tried setting DOTNET_SYSTEM_GLOBALIZATION_USENLS and can confirm this gives back the original perf.

Currently on my system (win10 x64 with latest updates), the icu.dll located in Windows/System32 reports being version 64.2. Looking over at the ICU github repo, this release is from Apr 17, 2019 (https://github.com/unicode-org/icu/releases?page=3) and I think built with an older MSVC2017 (https://htmlpreview.github.io/?https://github.com/unicode-org/icu/blob/release-64-2/icu4c/readme.html).

Do you know if there's any plans to roll out an updated icu.dll via windows updates anytime soon?

Thanks

[tarekgh]

I think Windows 10 should have a higher version of ICU than what you have listed.

@jefgen could you please comment on @L2 question #40942 (comment)?

[jefgen]

The story is unfortunately a bit complicated...
The version of ICU that is part of the Windows OneCore base was updated to a newer version, version 68.2. However, Windows 10 is still based on the older OneCore release, so this means that Windows 10 didn't get the updated version of ICU. The Windows 11 release was built on the newer OneCore bits, so it got the updated version of ICU.
(You can see that the Windows 10 build number is still ~1904x, so the last few releases are all based on the same underlying OneCore bits).

What this means is that Windows 10 is still on ICU version 64.2, while Windows 11 is on ICU version 68.2.

roll out an updated icu.dll via windows updates anytime soon?

I don't think there are currently any plans to push an updated version of ICU via Windows update at this time.

built with an older MSVC2017

AFAIK, the OS binaries aren't built using the public compiler. However, I'm not sure off-hand what exact version of the cl.exe compiler is used though.

[tarekgh]

Thanks @jefgen for the details.

@L2 you still have the option to use the ICU app-local feature to use the latest ICU version if you want.

  <ItemGroup>
        <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.6" />
        <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="68.2.0.6" />
  </ItemGroup>