Pulumi not able to generate code with convert when importing TF helm chart from awslabs/data-on-eks examples

[stooj]

What happened?

I'm trying to convert some TF/Helm from awslabs/data-on-eks.

I run the pulumi convert --from terraform --language python --out pulumistyle --verbose 6 inside data-on-eks/ai-ml/jupyterhub and get a code generation error.

Error is (full output is at bottom of this issue comment):

================================================================================
The Pulumi CLI encountered a code generation error. This is a bug!
We would appreciate a report: https://github.com/pulumi/pulumi/issues/
Please provide all of the below text in your report.
================================================================================
Pulumi Version:   v3.146.0
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.

Example

Using pulumi/pulumi v3.146.0

2. Clone awslabs/data-on-eks repo: git clone https://github.com/awslabs/data-on-eks.git
3. cd to the ai-ml/jupyterhub directory
4. Try to convert the code: pulumi convert --from terraform --language python --out pulumi

Output of pulumi about

Output of pulumi about:

CLI
Version      3.146.0
Go Version   go1.23.4
Go Compiler  gc

Host
OS       nixos
Version  24.11 (Vicuna)
Arch     x86_64

Backend
Name           pulumi.com
URL            https://app.pulumi.com
User           Unknown
Organizations
Token type     personal

Pulumi locates its logs in /tmp by default
warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub). If you have not created a project yet, use `pulumi new` to do so: no project file found
warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub). If you have not created a project yet, use `pulumi new` to do so: no project file found

Since there is no pulumi project yet, here's the output of pulumi plugin ls:

NAME        KIND       VERSION  SIZE    INSTALLED       LAST USED
archive     resource   0.3.0    45 MB   17 minutes ago  3 minutes ago
aws         resource   6.66.3   716 MB  17 minutes ago  3 minutes ago
cloudinit   resource   1.4.9    45 MB   14 minutes ago  3 minutes ago
kubernetes  resource   4.21.0   149 MB  16 minutes ago  3 minutes ago
random      resource   4.17.0   45 MB   16 minutes ago  3 minutes ago
terraform   converter  1.0.23   40 MB   17 minutes ago  3 minutes ago
time        resource   0.1.1    61 MB   16 minutes ago  3 minutes ago
tls         resource   5.1.0    46 MB   14 minutes ago  3 minutes ago

Additional context

Complete output of the error (I've removed many duplicate lines from the verbose warnings I got before the error was thrown):

warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/addons.tf:277,37-91: Function not yet implemented; Function try not yet implemented, see pulumi/pulumi-converter-terraform#16
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/addons.tf:267,15-280,7: Function not yet implemented; Function templatefile not yet implemented, see pulumi/pulumi-converter-terraform#192
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/addons.tf:289,44-55: Terraform input not yet implemented; path
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/addons.tf:289,28-88: Function not yet implemented; Function templatefile not yet implemented, see pulumi/pulumi-converter-terraform#192
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/jupyterhub.tf:111,29-40: Terraform input not yet implemented; path
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/jupyterhub.tf:128,29-40: Terraform input not yet implemented; path
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:36,12-68: Function not yet implemented; Function slice not yet implemented, see pulumi/pulumi-converter-terraform#65
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:127,58-132: Function not yet implemented; Function zipmap not yet implemented, see pulumi/pulumi-converter-terraform#65 (catch all bug)
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:127,20-190: Function not yet implemented; Function compact not yet implemented, see pulumi/pulumi-converter-terraform#65
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:151,14-154,9: Function not yet implemented; Function merge not yet implemented, see pulumi/pulumi-converter-terraform#65
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:58,54-128: Function not yet implemented; Function zipmap not yet implemented, see pulumi/pulumi-converter-terraform#65 (catch all bug)
warning: /home/stooj/code/pulumi/stooj/data-on-eks/ai-ml/jupyterhub/main.tf:58,16-186: Function not yet implemented; Function compact not yet implemented, see pulumi/pulumi-converter-terraform#65
Converting to python...
================================================================================
The Pulumi CLI encountered a code generation error. This is a bug!
We would appreciate a report: https://github.com/pulumi/pulumi/issues/
Please provide all of the below text in your report.
================================================================================
Pulumi Version:   v3.146.0
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: addons.pp:317,1-1: Unterminated template string; No closing marker was found for the string.
error: jupyterhub.pp:132,1-1: Unterminated template string; No closing marker was found for the string.
error: could not generate output program

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[stooj]

I downgraded my converter to check:

This issue was introduced between v1.0.19 and v1.0.20 of pulumi-converter-terraform

[justinvp]

I took a closer look into this. When converting with v1.0.19, I don't get the Unterminated template string error, but the generated Python project is empty. @stooj, were you seeing actual code in the Python project when converting with v1.0.19?

If I do a comparison converting to PCL using v1.0.19 and v1.0.23, the v1.0.23 is able to convert much more than v1.0.19, and because it's converting way more, it ends up hitting a new error that we haven't seen before:

Here's a comparison:

v1.0.19

[~/Documents/2025/01/22/convertestdataoneks/data-on-eks/ai-ml/jupyterhub (main)]$ pulumi convert --from terraform --language pcl --out pulumipcl-1.0.19
[~/Documents/2025/01/22/convertestdataoneks/data-on-eks/ai-ml/jupyterhub (main)]$ tree pulumipcl-1.0.19
pulumipcl-1.0.19
├── eks-blueprints-addons_1.19.0
│   ├── eks-blueprints-addon_1.1.1
│   │   ├── main.pp
│   │   ├── outputs.pp
│   │   └── variables.pp
│   └── sqs_4.0.1
│       ├── main.pp
│       ├── outputs.pp
│       └── variables.pp
└── iam_5.52.2
    └── modules
        └── iam-role-for-service-accounts-eks
            ├── main.pp
            ├── outputs.pp
            ├── policies.pp
            └── variables.pp

7 directories, 10 files

v1.0.23

[~/Documents/2025/01/22/convertestdataoneks/data-on-eks/ai-ml/jupyterhub (main)]$ pulumi convert --from terraform --language pcl --out pulumipcl-1.0.23
[~/Documents/2025/01/22/convertestdataoneks/data-on-eks/ai-ml/jupyterhub (main)]$ tree pulumipcl-1.0.23
pulumipcl-1.0.23
├── Pulumi.yaml
├── addons.pp
├── cognito.pp
├── eks-blueprints-addons_1.19.0
│   ├── eks-blueprints-addon_1.1.1
│   │   ├── main.pp
│   │   ├── outputs.pp
│   │   └── variables.pp
│   ├── helm.pp
│   ├── main.pp
│   ├── outputs.pp
│   ├── sqs_4.0.1
│   │   ├── main.pp
│   │   ├── outputs.pp
│   │   └── variables.pp
│   └── variables.pp
├── eks-data-addons_1.33.0
│   ├── airflow.pp
│   ├── aws-efa-k8s-device-plugin.pp
│   ├── aws-neuron-device-plugin.pp
│   ├── cnpg-operator.pp
│   ├── dask-hub.pp
│   ├── dask-operator.pp
│   ├── emr-flink-operator.pp
│   ├── emr-spark-operator.pp
│   ├── flink-operator.pp
│   ├── irsa
│   │   ├── main.pp
│   │   ├── outputs.pp
│   │   └── variables.pp
│   ├── jupyterhub.pp
│   ├── karpenter-resources.pp
│   ├── kubecost.pp
│   ├── kuberay-operator.pp
│   ├── main.pp
│   ├── mlflow-tracking.pp
│   ├── nvidia-device-plugin.pp
│   ├── nvidia-gpu-operator.pp
│   ├── nvidia-triton-server.pp
│   ├── outputs.pp
│   ├── pinot.pp                                                                                                                      │   ├── qdrant.pp
│   ├── spark-history-server.pp
│   ├── spark-operator.pp
│   ├── strimzi-kafka-operator.pp
│   ├── superset.pp
│   ├── trino.pp                                                                                                                      │   ├── variables.pp
│   ├── volcano.pp
│   └── yunikorn.pp                                                                                                                   ├── eks_19.21.0
│   ├── kms_2.1.0                                                                                                                     │   │   ├── main.pp
│   │   ├── outputs.pp                                                                                                                │   │   └── variables.pp
│   ├── main.pp                                                                                                                       │   ├── modules
│   │   ├── _user_data                                                                                                                │   │   │   ├── main.pp
│   │   │   ├── outputs.pp                                                                                                            │   │   │   └── variables.pp
│   │   ├── eks-managed-node-group                                                                                                    │   │   │   ├── main.pp
│   │   │   ├── outputs.pp                                                                                                            │   │   │   └── variables.pp
│   │   ├── fargate-profile                                                                                                           │   │   │   ├── main.pp
│   │   │   ├── outputs.pp                                                                                                            │   │   │   └── variables.pp
│   │   └── self-managed-node-group                                                                                                   │   │       ├── main.pp
│   │       ├── outputs.pp                                                                                                            │   │       └── variables.pp
│   ├── node_groups.pp
│   ├── outputs.pp
│   └── variables.pp
├── iam_5.52.2
│   └── modules
│       └── iam-role-for-service-accounts-eks
│           ├── main.pp
│           ├── outputs.pp
│           ├── policies.pp
│           └── variables.pp
├── jupyterhub.pp
├── main.pp
├── outputs.pp
├── variables.pp
├── vpc.pp
└── vpc_5.17.0
    ├── main.pp
    ├── outputs.pp
    ├── variables.pp
    └── vpc-flow-logs.pp

17 directories, 77 files

The actual error looks similar to an issue the K8s converter was hitting where multi-line strings weren't being emitted in PCL correctly. (See pulumi/pulumi-converter-kubernetes#10 and pulumi/pulumi-converter-kubernetes#5). Should be easy enough for us to find the problem in the generation PCL and reduce it down to a minimal repro.

Since this looks like a converter bug, I'll move it to the converter repo

[stooj]

Sorry, my english was a little ambiguous.

This issue was introduced between v1.0.19 and v1.0.20 of pulumi-converter-terraform

So the converter finished without error in v1.0.19 (I think you saw that too). I got the Unterminated template string in v1.0.20 and above.

[stooj]

Just adding to this:

• Convert to go or python code: error with empty --out directory
• Convert to pcl: error with lots of treats in the --out directory.

I don't think converting to pcl is a normal thing for end-users, so they will see empty directories when they try convert.

Hope that helps