[BUG][codegen] Nullable references, nullable primitive types and complex types

[sebastien-rosset]

Bug Report Checklist

• Have you provided a full/minimal spec to reproduce the issue?
• Have you validated the input using an OpenAPI validator (example)?
• What's the version of OpenAPI Generator used?
• Have you search for related issues/PRs?
• What's the actual output vs expected output?
• [Optional] Bounty to sponsor the fix (example)

Description

There is inconsistent support for nullable types across language generators. Some language generators (e.g. "go") are lenient and allow null values even if the OAS spec does explicitly list nullable types. Other language generators (e.g. Python) do strict validation and raise an Exception if the server returns a null value that wasn't explicitly allowed in the OAS spec.

To complicate matters, there are multiple ways to specify nullable values in the OAS spec, and there are changes between 3.0 and 3.1:

1. In OAS 3.1, the type property now allows "null" as a type, alone or within a type array. This is the standard way to allow null values in JSON Schema.

type: ['null', 'string']
type: 'null'

2. The "nullable" attribute is marked deprecated in 3.1. There are multiple discussion threads about the issues with the "nullable" attribute.

• Clarify Semantics of nullable in OpenAPI
• Reference objects don't combine well with “nullable
• nullable vs type arrays, JSON Schema compatibility
• For discussion only; support JSON Schema Draft 6 in 3.1

openapi-generator version

4.2.3

OpenAPI declaration file content or url

Suppose an OAS spec defines #/definitions/Address of type: object:

Address:
  type: object

In OAS 3.0, Address can be marked as nullable:

Address:
  type: object
  nullable: true

Then other types can reference "Address", such as an "Order" type that has a "billTo" property, and "billTo" can have a null value:

Order:
  type: object
  properties:
    billTo:
      $ref: "#/definitions/Address"

One issue is that at least one language generator (python-experimental) is ignoring the "nullable" attribute. I.e. in the case of Python, when the server returns a null value for "billTo", the client raises an exception because it did not expect a null value.
On the other hand, the "go-experimental" code generator accepts null values, even if "nullable: true" wasn't specified in the OAS spec.

Another way to specify a nullable property is to use the vendor extension "x-nullable". A grep shows "x-nullable" is still used in a few OpenAPITool files. Is this still needed? I think this is a remnant of pre-3.0 OAS, before the "nullable" attribute existed.

In addition, "nullable" brings other issues that are described in Clarify Semantics of nullable in OpenAPI, hence it looks like the OAS TSR is deprecating "nullable" in 3.1.

Another problem with the above YAML is that it's not possible to selectively specify which properties of type "address" are nullable or not. For example, suppose the "Order" type has two properties, "shipTo" and "billTo":

Order:
  type: object
  properties:
    shipTo:
      $ref: "#/definitions/Address"
    billTo:
      $ref: "#/definitions/Address"

Let's assume for the sake of this argument that "shipTo" must be set to a non-null value, and "billTo" can have a null value. Using "nullable" inside "Address" would not work, or at least the same "Address" type could not be reused for both the "shipTo" and "billTo" properties.

Order:
  type: object
  properties:
    shipTo:
      $ref: "#/definitions/Address"
    billTo:
      $ref: "#/definitions/Address"

With the above, both "shipTo" and "billTo" can have a null value, which is not what was intended.

To address this issue, one may be tempted to use the nullable attribute as a sibling of $ref. Instead of specifying the nullable attribute in "Address", "nullable" would be specified in the property, as shown below.

Order:
  type: object
  properties:
    shipTo:
      $ref: "#/definitions/Address"
      nullable: false
    billTo:
      $ref: "#/definitions/Address"
      nullable: true

However, the above is not supported in 3.0, the spec explicitly states all sibling attributes of "$ref" are ignored, and that seems to be ignored by the OpenApiTools code generators as well.

Note: the OAS TSR is considering (but nothing is decided) supporting siblings attributes of "$ref" and the associated merge semantic. However it's unlikely this will apply to "nullable" because "nullable" is deprecated anyway.

One way to workaround the above issue is to define two types, "Address" and "OptionalAddress":

OptionalAddress:
  type: object
  nullable: true
  properties:
    zipCode: 
      type: string
    city: 
      type: string

Address:
  allOf:
   - $ref: "#/definitions/OptionalAddress"
  type: object
  #Setting nullable to false is really a no-op, but makes the intent more explicit.
  nullable: false 

Then the "Order" type can be changed as below:

Order:
  type: object
  properties:
    shipTo:
      $ref: "#/definitions/Address"
    billTo:
      $ref: "#/definitions/OptionalAddress"

However this has two problems: the OAS spec becomes verbose (maybe the OAS ends up specifying XYZ types and OptionalXYZ type for everything), and anyway "nullable: false" is a no-op.

Maybe a bigger issue is that "nullable" is being deprecated in 3.1, so authors should stop using nullable and use other constructs to specify nullable properties.

Besides the "nullable" attribute, another way to specify nullable properties is to use "oneOf":

Order:
  type: object
  properties:
    billTo:
      oneOf:
      - type: 'null'
      - $ref: "#/definitions/Address"
    shipTo:
      $ref: "#/definitions/Address"

This seems to be the standard, future-proof way to specify nullable properties for complex types.
Note: type: 'null' is not supported in OAS 3.0.x. It is being introduced in OAS 3.1.

From the perspective of the OpenAPITool code generator, there are a few problems:

1. Support for 'oneOf' is incomplete and sometimes not supported at all in the OpenAPITools language generators.
2. The generated code could be very verbose, and casual SDK users will be surprised by the amount of complexity to access what they thought was just a simple value. In this particular case, it would be nice if the generated code has a simple Get() method that returns a pointer to the instance, and that pointer can be null. I'm oversimplifying, but I'm contrasting this to a fully expanded "oneOf" construct which can be very verbose in the generated code. This may require special case handling in codegen.

It would be really nice if the generated code is simple, and looks the same whether it was written using "oneOf" or nullable. However, this does not seem very simple to achieve because the OAS spec could have more than oneOf types, such as below:

Order:
  type: object
  properties:
    billTo:
      oneOf:
      - type: 'null'
      - $ref: "#/definitions/Address"
      - $ref: "#/definitions/BitcoinUrl"

Finally, for properties that are of a primitive type, and the type should accept a null value, it's possible to use type arrays in OAS 3.1. This is the equivalent of "oneOf" for primitive types:

Address:
  type: object
  properties:
    addressLine1:
      type: ['null', 'string']

which is equivalent to:

Address:
  type: object
  properties:
    addressLine1:
      oneOf:
      - type: 'null'
      - type: 'string'

Command line used for generation

Steps to reproduce

Related issues/PRs

Suggest a fix

[sebastien-rosset]

Summary of the proposal:

1. Support type: 'null'. It is defined in JSON schema and got added to OAS 3.1. It also seems to be implemented by other OAS tools.
2. Support type arrays for primitive types, such as type: ['null', 'string']
3. Support the following construct:

OptionalAddress:
      oneOf:
      - type: 'null'
      - $ref: "#/definitions/Address"

I am primarily interested in Java, Go-experimental and python-experimental, so I will focus on these, but the same constructs apply to every language.

[pekunicki]

It's also the tricky thing in C# 8.0 where there is a semantic separation for nullable and non-nullable reference types. Please see the following bug.
NSwag should support C# 8.0

[TomWebbio]

Hey,

I was the one that originally sent in this request a few months ago. I was wondering what the status of this ticket is, and if there is any chance of this being fixed in the foreseeable future.

When trying to generate a client for the following JSON it still gives errors:

"partner": { "allOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] }

The generated code:

'partner': UserMvoDTO & objectFromJSON(json['partner']),

Do you think there is any way this will be fixed soon? I imagine a lot of people are using NestJS to generate their swagger.

[sebastien-rosset]

Hey,

I was the one that originally sent in this request a few months ago.

Which request? Can you send a link?

I was wondering what the status of this ticket is, and if there is any chance of this being fixed in the foreseeable future.

When trying to generate a client for the following JSON it still gives errors:

Which error specifically?

"partner": { "allOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] }

I don't think this JSON achieves what you intended. Unless I am mistaken, I think what you intended is the value of partner is either a UserMvoDTO (which presumably is not nullable) or the null value.
allOf means that your object must pass all of the validation constraints. I.e. it must pass the validation for UserMvoDTO (hence it cannot be null) AND it must pass validation for "nullable": true. For the second constraint, the value can be free-form and it is nullable, but it does not matter because the first constraint specifies the value cannot be null.

I think you should be using oneOf.

The generated code:

'partner': UserMvoDTO & objectFromJSON(json['partner']),

Do you think there is any way this will be fixed soon? I imagine a lot of people are using NestJS to generate their swagger.

It would be good to understand the specific error. One aspect is to understand whether the issue is specific to a language generator (e.g. C#, go, java...) or if it is language independent (codegen). If it is language-specific, you should open an issue specific to that language (and maybe you have already done it).

[TomWebbio]

Which request? Can you send a link?

I made the request in your slack channel and sadly don't have a link. After some discussion by you guys you created this particular issue to deal with the problem.

I think you should be using oneOf.

Yes that does make sense, you are correct in your assumption of what we are trying to achieve. However, we use the NestJS framework and this is what the code generates. We will try to look into why the JSON is being generated like that, as that seems to be a NestJS framework issue. Thanks for pointing me in the right direction.

It would be good to understand the specific error. One aspect is to understand whether the issue is specific to a language generator (e.g. C#, go, java...) or if it is language independent (codegen). If it is language-specific, you should open an issue specific to that language (and maybe you have already done it).

I have not done that yet, as I specified the language and frameworks we were using before this issue was created and I assumed it was taken into account when you guys created the issue. We will first look into the NestJS problem, and if generating the correct JSON fixes the issue I won't have to open another issue for this repo.

Thanks for your reply, I'll update you on whether we fix the problem.

[TomWebbio]

We did some testing by manually changing the JSON to oneOf, but sadly now it gives a different issue.
The JSON:

"partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] } },

The generated code:

partner: UserMvoDTO | object;

As you can see, now it's | object, while it should be | null.

Perhaps the "nullable": true is incorrect? This seems to be according to specifications and the way NestJS uses it in it's output.

Is this something you can help me with here, or should I make a specific issue for Typescript/Nodejs?

[pekunicki]

Hi Both,

Thanks for the engagement.
I've dived into that and it seems that many generators handle the nullable reference object with oneOf, i.e.

"partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] } },

In my opinion, it's correct, from C# perspective at least, so you can handle as a specific issue for Typescript/Nodejs.

However, do you really think it's the right thing? oneOf should be used for handling the inheritance, so I don't fully get why the nullable prop cannot be used the same way as with the primitive types?

e.g.

    partner:
      $ref: "#/components/schemas/UserMvoDTO"
      nullable: true

Do you know if there is any update on that?

Note: the OAS TSR is considering (but nothing is decided) supporting siblings attributes of "$ref" and the associated merge semantic. However it's unlikely this will apply to "nullable" because "nullable" is deprecated anyway.

[sebastien-rosset]

We did some testing by manually changing the JSON to oneOf, but sadly now it gives a different issue.
The JSON:

"partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] } },

The generated code:

partner: UserMvoDTO | object;

As you can see, now it's | object, while it should be | null.

No, because 'Nullable' is not the same thing as the null value. If on the other hand, you had the OAS document below, it would be exactly UserMvoDTO or the null value.

 "partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "type": "null" } ] } },`

Note that "type": "null" is introduced in OAS 3.1.

But that's not what you wrote. What you wrote is "nullable": true, without any other OAS attribute. That means it can literally be any JSON document. The following documents will pass the validation for nullable: true

{ "partner": null }
{ "partner": "foo" }
{ "partner": { "name": "foo", "address": "600 Pennsylvania Ave NW" } }
{ "partner": [ {"abc": "def" }, "foo", 123 ] }

Perhaps the "nullable": true is incorrect? This seems to be according to specifications and the way NestJS uses it in it's output.

Is this something you can help me with here, or should I make a specific issue for Typescript/Nodejs?

[sebastien-rosset]

Hi Both,

Thanks for the engagement.
I've dived into that and it seems that many generators handle the nullable reference object with oneOf, i.e.

"partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "nullable": true } ] } },

In my opinion, it's correct, from C# perspective at least, so you can handle as a specific issue for Typescript/Nodejs.

However, do you really think it's the right thing? oneOf should be used for handling the inheritance, so I don't fully get why the nullable prop cannot be used the same way as with the primitive types?

e.g.

    partner:
      $ref: "#/components/schemas/UserMvoDTO"
      nullable: true

I think you may be confusing "nullable": true with type: 'null'.

Do you know if there is any update on that?

Note: the OAS TSR is considering (but nothing is decided) supporting siblings attributes of "$ref" and the associated merge semantic. However it's unlikely this will apply to "nullable" because "nullable" is deprecated anyway.

[pekunicki]

@sebastien-rosset I don't think, it's confusing with type: 'null'. According to the OpenApi Specs, the Nullable param can coexist together with type: 'null'. See more details below.

OAI/OpenAPI-Specification@3cb92bd

More discussion:
OAI/OpenAPI-Specification#2050

[TomWebbio]

Alright, well at this point I've tried both of your solutions but neither of them generated valid code.

When I use the following JSON:
"partner": { "oneOf": [ { "$ref": "#/components/schemas/UserMvoDTO" }, { "type": "null" } ] }

I get the following code:
partner: UserMvoDTO | ModelNull;

And then in the return value the following code has errors:

partner: UserMvoDTO | ModelNullFromJSON(json["partner"])

At this point I'm not sure what I should do. The JSON we had originally was what NestJS generated using the Swagger plugin. NestJS is a pretty popular framework, so while I can believe they've made mistakes generating the Swagger JSON, it seems strange that we're the only one with this issue.

Before making an issue at the NestJS repository, I'd like to at least have a working example of what the JSON should look like. The requirements are pretty simple in my opinion, I want a nullable custom type. It works for Nullable primitive types, just not for custom types.

If you could be so kind as to give me an example of a nullable custom type that actually works in your generator for Typescript, I'd be very much obliged.

Thanks for the effort you guys have already put in.

[sebastien-rosset]

I tend to focus on generators for Java, go, powershell and Python, and I will say upfront I don't know about the generators for NetJS. I can tell you what my experience has been for Java, go, powershell and Python, maybe this will be applicable to NetJS. I have used OpenAPITools to generate SDKs for Java, go, powershell and Python. It is an awesome tool, and the community is great. That being said, it is an open-source community, so when facing an issue, I knew we would have to dig into the code to understand the issues, and in some cases submit pull requests.

[sebastien-rosset]

@sebastien-rosset I don't think, it's confusing with type: 'null'. According to the OpenApi Specs, the Nullable param can coexist together with type: 'null'. See more details below.

OAI/OpenAPI-Specification@3cb92bd

More discussion:
OAI/OpenAPI-Specification#2050

Yes, I wasn't saying they cannot coexist. I was saying that if you write "nullable": true, you should not be surprised to see the following code being generated:

partner: UserMvoDTO | object;

That's because nulllable: true means it can be any arbitrary JSON document.

[sebastien-rosset]

Also, when I wrote this issue, I was thinking about language-independent modeling, i.e. codegen. I did merge #5290 to support this, so I think I can close this issue. Then I opened specific issues and PRs for go, Java and python. It appears the issue you are reporting is specific to a particular language, so it would make sense to open a separate issue, and ideally submit a pull request.

[TomWebbio]

Alright, thanks for your feedback and help. I will see if I can report a language specific issue.

[Jaesin]

Thanks for such a great summary of the problem @sebastien-rosset .

nullable has been removed from OAS v3.1 Ref: OAI/OpenAPI-Specification#2244

[spacether]

This is working in the python client generator in v6.2.0 and onward. Verified by these tests:

• https://github.com/OpenAPITools/openapi-generator/blob/master/samples/openapi3/client/petstore/python/tests_manual/test_nullable_string.py
• https://github.com/OpenAPITools/openapi-generator/blob/master/samples/openapi3/client/petstore/python/tests_manual/test_fruit.py#L114