Skip to content

fix: Provide more generic API for the capacity limit parsing#20372

Open
erenavsarogullari wants to merge 1 commit intoapache:mainfrom
erenavsarogullari:capacity_limit
Open

fix: Provide more generic API for the capacity limit parsing#20372
erenavsarogullari wants to merge 1 commit intoapache:mainfrom
erenavsarogullari:capacity_limit

Conversation

@erenavsarogullari
Copy link
Member

@erenavsarogullari erenavsarogullari commented Feb 15, 2026

Which issue does this PR close?

Rationale for this change

Currently, datafusion.runtime.max_temp_directory_size is a disk based config but when it is set as invalid limit or invalid unit, memory limit is mentioned in error message. This seems to introduce inconsistency between related runtime config and error message like disk vs memory limit. This error message can be updated more generic by setting problematic config name and covering both memory and disk based capacity/size settings.

Current:

statement error DataFusion error: Error during planning: Failed to parse number from memory limit 'invalid_size'
SET datafusion.runtime.max_temp_directory_size = 'invalid_size'

statement error DataFusion error: Error during planning: Unsupported unit 'B' in memory limit '1024B'
SET datafusion.runtime.max_temp_directory_size = '1024B'

New:

statement error DataFusion error: Error during planning: Failed to parse number from 'datafusion.runtime.max_temp_directory_size', limit 'invalid_size'
SET datafusion.runtime.max_temp_directory_size = 'invalid_size'

statement error DataFusion error: Error during planning: Unsupported unit 'B' in 'datafusion.runtime.max_temp_directory_size', limit '1024B'. Unit must be one of: 'K', 'M', 'G'
SET datafusion.runtime.max_temp_directory_size = '1024B'

What changes are included in this PR?

Following improvements are being offered:

  1. Setting problematic config name in error message to cover all use-cases (e.g: both memory and disk based capacity/size settings),
  2. Allowed units are also added in error message,
  3. SessionContext.parse_memory_limit() => SessionContext.parse_capacity_limit() function signature is also being renamed to cover both memory and disk based capacity/size settings. This is applied to both SessionContext and Benchmark Utils,
  4. Being added new UT cases to cover these negative use-cases in terms of above changes.

Are these changes tested?

Yes and adding new UT cases

Are there any user-facing changes?

Yes, new more detailed and generic error messages are exposed to end-users.

@github-actions github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Feb 15, 2026
@erenavsarogullari erenavsarogullari changed the title Provide more generic API for the capacity limits Provide more generic API for the capacity limit parsing Feb 15, 2026
@erenavsarogullari erenavsarogullari changed the title Provide more generic API for the capacity limit parsing fix: Provide more generic API for the capacity limit parsing Feb 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide more generic API for the capacity limit parsing

1 participant