Custom fields

[vitalik]

Hi

do you have any plans on upgrading custom fields ?

class PostCode(str):

    @classmethod
    def __get_validators__(cls):
           yield cls.validate
    
    @classmethod
    def validate(cls, v):
       ...

[Kludex]

Can you show me some expected input / expected transformations?

[vitalik]

@Kludex well hard to tell... I'm just comparing docs for v1 and v2 - that does not look like a trivial case:

class PostCodeAnnotation:


    @classmethod
    def __get_pydantic_core_schema__(
        cls, _source_type: Any, _handler: GetCoreSchemaHandler
    ) -> core_schema.CoreSchema:
        return core_schema.no_info_after_validator_function(
            cls.validate,
            core_schema.str_schema(),
        )

    @classmethod
    def __get_pydantic_json_schema__(
        cls, schema: core_schema.CoreSchema, handler: GetJsonSchemaHandler
    ) -> JsonSchemaValue:
        json_schema = handler(schema)
        json_schema.update(
            # simplified regex here for brevity, see the wikipedia link above
            pattern='^[A-Z]{1,2}[0-9][A-Z0-9]? ?[0-9][A-Z]{2}$',
            # some example postcodes
            examples=['SP11 9DG', 'W1J 7BU'],
        )
        return json_schema

    @classmethod
    def validate(cls, v: str):
        m = post_code_regex.fullmatch(v.upper())
        if m:
            return f'{m.group(1)} {m.group(2)}'
        else:
            raise PydanticCustomError('postcode', 'invalid postcode format')


PostCode = Annotated[str, PostCodeAnnotation]

[Kludex]

What would be the input for this expected code?

[vitalik]

it's from official docs:

• custom field in v1
• custom field in v2

[Kludex]

For the __modify_schema__, we could:

1. Find the FunctionDef named __modify_schema__, and save the second argument name (e.g. field_schema).
2. If field_schema.update is found, then add field_schema = handler(schema) in the first line after the FunctionDef, and return field_schema in the last line.
3. If field_schema.update is not found, add a TODO note telling to update manually.

For the __get_validator__, I think is a bit more complicated... Would it be safe to assume core_schema.no_info_after_validator_function on every code source? 🤔

[vitalik]

I think it is not auto solvable problem as in __get_validators__ you can return all sort of things and have multiple validators which does not look like a case for __modify_schema__ ...

@samuelcolvin @dmontagu @hramezani maybe pydantic should have some backwards(deprecated) compatible approach to custom fields ?

overall to me custom fields feels very low-level implementation - I am not able to remember it without looking at examples :)

Maybe there should exist some simpler approach for 80+% of use cases, like:

class PostCode(str):

    @classmethod
    def __pydantic_validate__(cls, v):
           if v not in UK_DATABASE_CALL:
                   raise ValidationError
           return v

which should atomatically add needed __modify_schema__ guts

[adriangb]

There is a simpler approach for 80% of the use cases:

from annotated_types import Predicate  # or `pydantic.PlainValidator`, etc.

PostalCode = Annotated[str, Predicate(lambda v: True if v in UK_DATABASE_CALL else False)]

Hence why this section comes before the __get_pydantic_core_schema__ section.

For what it's worth to get multiple validators in __get_pydantic_core_schema__ you can either use a chain validator:

from typing import Any, Callable, Dict, Iterable, List

from pydantic_core import CoreSchema, core_schema

from pydantic import (
    TypeAdapter,
    GetCoreSchemaHandler,
    GetJsonSchemaHandler,
)
from pydantic.json_schema import JsonSchemaValue


class PostalCode(str):
    @classmethod
    def __get_pydantic_core_schema__(
        cls, _source_type: Any, _handler: GetCoreSchemaHandler
    ) -> core_schema.CoreSchema:
        val_func_schemas: List[CoreSchema] = []
        for validation_function in getattr(cls, '__get_validators__', lambda: ())():
            val_func_schemas.append(core_schema.no_info_plain_validator_function(validation_function))
        return core_schema.chain_schema(val_func_schemas)

    @classmethod
    def __get_pydantic_json_schema__(
        cls, schema: core_schema.CoreSchema, handler: GetJsonSchemaHandler
    ) -> JsonSchemaValue:
        json_schema = {'type': 'string'}
        modify_schema = getattr(cls, '__modify_schema__', None)
        if modify_schema is not None:
            json_schema = modify_schema(json_schema) or json_schema
        return json_schema

    @classmethod
    def __get_validators__(cls) -> Iterable[Callable[..., Any]]:
        yield cls.validate1
        yield cls.validate2

    @classmethod
    def validate1(cls, v: str) -> str:
        return v * 2

    @classmethod
    def validate2(cls, v: str) -> str:
        return v[:5]

    @classmethod
    def __modify_schema__(cls, field_schema: Dict[str, Any]) -> Dict[str, Any]:
        # __modify_schema__ should mutate the dict it receives in place,
        # the returned value will be ignored
        field_schema.update(
            # simplified regex here for brevity, see the wikipedia link above
            pattern='^[A-Z]{1,2}[0-9][A-Z0-9]? ?[0-9][A-Z]{2}$',
            # some example postcodes
            examples=['SP11 9DG', 'w1j7bu'],
        )
        return field_schema


ta = TypeAdapter(PostalCode)
assert ta.validate_python('abc') == 'abcab'
assert ta.json_schema() == {'examples': ['SP11 9DG', 'w1j7bu'], 'pattern': '^[A-Z]{1,2}[0-9][A-Z0-9]? ?[0-9][A-Z]{2}$', 'type': 'string'}

I guess bump-pydantic could insert some version of this automatically. The one thing I don't think is trivial to figure out automatically is the json_schema = {'type': 'string'}. I think Pydantic V1 just brute forced this by iterating through the bases and trying to generate a schema for each base and just returning the first one, which is obviously supper buggy but I guess works in very simple cases. The thing is those very simple cases are now better served by the Annotated[str, ...] pattern so we don't want to encourage it.

[vitalik]

@adriangb

There is a simpler approach for 80% of the use cases:

from annotated_types import Predicate  # or `pydantic.PlainValidator`, etc.

PostalCode = Annotated[str, Predicate(lambda v: True if v in UK_DATABASE_CALL else False)]

but this is not something you can automate with a bump-pydantic tool (too complex)

I think the best solution would be to introduce some meta class into pydantic codebase that will help transition from v1 to v2

class V1CustomField:
    ... magic with metaclasses ...



# then finally all you will have to do to migrate v1 to v2 is to add extra parent class:

class PostCode(str, V1CustomField): # <--- !!!

    @classmethod
    def __get_validators__(cls):
           yield cls.validate
    
    @classmethod
    def validate(cls, v):
       ...

[adriangb]

Then we have to carry that metaclass around until V3. And in V3 we'll need a compatibility metaclass for the compatibility metaclass, right?

[vitalik]

Then we have to carry that metaclass around until V3. And in V3 we'll need a compatibility metaclass for the compatibility metaclass, right?

I feel some sarcasm here :) but no - you make it with deprecation warning and remove in v3

there are already bunch of deprecated code - https://github.com/pydantic/pydantic/tree/main/pydantic/deprecated

[adriangb]

It’s definitely something to consider. We can always add it in the next minor release.