Import-Vaporshell causes ConvertFrom-Json to Fail on PS4.0

[RegEM]

Hi great project.
I am stuck though running Import-Vaporshell though. I get this error, and don't know how to fix it.
Running this: $template = Import-Vaporshell .\awstemplate.json
Tried full path and in quotes etc, but haven't figured out the secret.
Please help me.

"ConvertFrom-Json : Invalid object passed in, ':' or '}' expected. (1): {
At line:36 char:24

• $tempObj = $temp | ConvertFrom-Json -Verbose:$false
  

  • CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
  • FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand"

[scrthq]

I'm glad you dig! From the error, it looks like the template JSON is malformed?

Import-Vaporshell does a few things:

1. Gets the content of the JSON file at that path, then converts from JSON into a PSObject
2. Adds various script methods to the object to allow adding/removing resources,outputs,parameters,etc.
3. Adds the custom object type "Vaporshell.Template" for use during validation when exporting it.

It looks like it's hanging on step one there.

Try this and see if you get a similar error:

Get-Content .\awstemplate.json | ConvertFrom-Json

If you're getting the same thing, check out the template you're importing from to ensure is correctly formatted and not missing something like a closing brace or colon

[RegEM]

Thx.
The templates I am trying to import, are exported from vaposhell.
I am validating too, "VERBOSE: The template was validated successfully!"
Here is one example. Not sure how to post json.
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "CFN - tcam utility functions", "Transform": "AWS::Serverless-2016-10-31", "Resources": { "sometbl": { "Type": "AWS::Serverless::SimpleTable" }, "tcamUpdateMonthstable": { "Type": "AWS::Serverless::Function", "Properties": { "Handler": "handler.updateSomestable", "Runtime": "nodejs6.10", "CodeUri": "s3://somebucket/update_table.zip", "Policies": "AmazonDynamoDBReadOnlyAccess", "Environment": { "Variables": { "TABLE_NAME": { "Ref": "sometbl" } } } } } } }
I tried your suggestion. No improvement.

[scrthq]

Weird! Checking this out now!

[RegEM]

I see this reference, but I'm not that conversant in PS.
https://stackoverflow.com/questions/18243707/powershell-convertfrom-json-error

[scrthq]

If I wrap that JSON example in single quotes and pipe it directly to ConvertFrom-Json, it works fine for me.

'{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "CFN - tcam utility functions", "Transform": "AWS::Serverless-2016-10-31", "Resources": { "sometbl": { "Type": "AWS::Ser
verless::SimpleTable" }, "tcamUpdateMonthstable": { "Type": "AWS::Serverless::Function", "Properties": { "Handler": "handler.updateSomestable", "Runtime": "nodejs6.10", "CodeUri": "s3://so
mebucket/update_table.zip", "Policies": "AmazonDynamoDBReadOnlyAccess", "Environment": { "Variables": { "TABLE_NAME": { "Ref": "sometbl" } } } } } } }' | ConvertFrom-Json

AWSTemplateFormatVersion Description                  Transform                  Resources
------------------------ -----------                  ---------                  ---------
2010-09-09               CFN - tcam utility functions AWS::Serverless-2016-10-31 @{sometbl=; tcamUpdateMonthstable=}

What version of Powershell are you using? Easiest would be to paste the output of $PSVersionTable here

[RegEM]

Name Value

---

PSVersion 4.0
WSManStackVersion 3.0
SerializationVersion 1.1.0.1
CLRVersion 4.0.30319.36366
BuildVersion 6.3.9600.17400
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
PSRemotingProtocolVersion 2.2

[RegEM]

As you did, direct in quotes works for me too.

[scrthq]

Alright... Can you try using the full path instead of `.\awstemplate.json'? Checking if it's not resolving the path correctly

[RegEM]

I tried already, but will again.

[RegEM]

Tried,
PS C:\Users\rr\Documents\Vaporshell> Get-Content "C:\Users\rr\Documents\Vaporshell\template.json" | ConvertFrom-Json and without quotes. No luck.

[scrthq]

would you be able to attach that template.json here so I can check it out? if it contains anything sensitive, don't stress on it. There's something odd about the json itself though if it's not even working when piped into ConvertFrom-Json

[RegEM]

So you know, the template is a bit more complicated, but does validate, and is accepted in quotes on the command line. Hence the different name.
I recently bought a new laptop. I will move to that, maybe later today, and confirm it is just a problem on the old one. Will probably be ok.
Nice to fix this issue, but I suggest not expending too much effort. Not unless you get other reporters.

[RegEM]

Renamed with .txt, as couldn't post otherwise.

[scrthq]

Weird! it's working fine for me importing straight off that file:

I ♥ PS>Import-Vaporshell -Path "C:\Users\nferrell\Downloads\template.json.txt"

AWSTemplateFormatVersion Description                  Transform                  Resources
------------------------ -----------                  ---------                  ---------
2010-09-09               CFN - tcam utility functions AWS::Serverless-2016-10-31 @{tcamMonthstbl=; tcamDaystbl=; tcamHourstbl=; tcamUpdateMonthstable=}

[scrthq]

didn't see anything wrong with the file either when I checked it out in VSCode, not linting issues, etc... I'd suspect something local to that computer, interested in hearing about success / etc on your other one!

[RegEM]

Well thanks a lot for checking. If you think of anything let me know. I've been limping this laptop along for a while. Time to move on...

[scrthq]

last attempt: can you try getting rid of the pipeline and seeing if the following works for you on that PC?

ConvertFrom-Json -InputObject ([System.IO.File]::ReadAllText("C:\Users\rr\Documents\Vaporshell\template.json"))

[RegEM]

Yes. that works.

[scrthq]

Cool! Alright, I should have version 2.0 released by this weekend. I'll swap the way that Import-Vaporshell is using ConvertFrom-Json to that. It looks like pipeline issues with how I built the function.

For the record, I was testing in PS5.1 and 6.0.0-beta5, I did not have a PS4 machine to test on at the moment.

I appreciate submitting this! I'm going to keep it open as a bug and close it out once the new version is released. Cheers!

[RegEM]

Awsome. Glad to help. I look forward to v2. Have a great day!

[scrthq]

Cheers, you too!

[RegEM]

Nate, Import-Vaporshell works on my new Win10 laptop.

Unfortunately having some other config issue. Fails on both validatingtemplate, and awscli didn't seem to install correctly. Not sure if powershellget is an issue. Need to setup the pc a bit. Will let you know if it is an issue. Prob. be a few days.

I'll add that I jumped over to the new laptop because of can't use/do transform error on the old one. Probably an old version of awscli or something.

[scrthq]

What version of Vaporshell are you using on your Win10 laptop? The latest version (1.2.3) actually is leveraging the AWS SDK for template validation, not the AWS CLI. I would love to see the error you are getting when trying to validate template as well!

[RegEM]

ok. seems likely I installed cli instead of sdk. should have time this morning to check, the vaporshell was latest, afaik.

[scrthq]

The SDK is actually bundled with the module now (v.1.2.3), so you shouldn't need to install anything extra =]... Send me the error you're getting if you can!

[RegEM]

ok. that makes more sense to me. I was just installing node etc. (I needed to anyways).

I believe I have just had to re-install cfn-flip, so I'm not sure if my install permissions are allowing the install to be done correctly. Anyways, after getting rid of a cfn-flip error, I am back to the following: (Sorry I am not properly setup on the laptop, so this is just summarizing the error)

I get,

1. "exception calling .ctor with 1 arg... "No region endpoint or serviceurl configured"
   At line 19 char 9, $req = ....
2. some "Write-S3Object not recognized, I think that is on the Export
3. some New-CFN-Stack not recognized

[RegEM]

New-CFNStack error is of course mine, as I am running from a ps1 script

[RegEM]

The Writes3 appears to be from my script too. I have just installed AWSPowershell, but the setup is still not quite correct

[RegEM]

I dunno, my credentials are not persisting properly on Win10.

[scrthq]

ah, I get ya! looks like you need to add your region to your credentials file.

You can set/update it with Set-VSCredentials:

NAME
    Set-VSCredentials

SYNTAX
    Set-VSCredentials [[-ProfileName] <string>] [-AccessKey <string>] [-SecretKey <string>] [-Region {APNortheast1 | APNortheast2 | APSouth1 | APSoutheast1 | APSoutheast2 | CACentral1 |
    CNNorth1 | EUCentral1 | EUWest1 | EUWest2 | SAEast1 | USEast1 | USEast2 | USGovCloudWest1 | USWest1 | USWest2}]  [<CommonParameters>]

[RegEM]

OK. I'll look at that. I did manage to eliminate error 3 by using Set-DefaultAWSRegion. I'll add that the command was different in earlier versions, I think.

[RegEM]

Ran Set-VSCredentials, which seems to have fixed Vaporshell, as it now gets right through to the validation.

I still have my local PS errors. Doesn't find Write-S3Object for example, but can from command line. I'll figure it out.

Error 3 has changed to the same error I was getting on teh old laptop, New-CFNStack : Create Stack cannot be used with templates containg transforms"

[scrthq]

For sure! The updates coming in v2 outside of small fixes are all AWS SDK wrappers, which the AWSPowerShell module already does; i.e. the following would work the same:

Set-AWSCredential -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -StoreAs myCredentials

Set-VSCredentials -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -ProfileName myCredentials

The main difference between AWSPowershell and Vaporshell is that you can also specify the -Region with Set-VSCredentials. Also, Vaporshell comes with both .Net45 and .NetCore SDK's and will load the appropriate assemblies on module import depending on the Powershell version, meaning your Vaporshell scripts would work whether you're running them in Powershell 4/5 on Windows or Powershell 6 on Ubuntu. You would need to import AWSPowershell.NetCore instead of AWSPowershell when working in CoreCLR.

---

For Serverless (or any template with Transforms), I believe you would need to create a Change Set, review, then execute it instead of deploying the stack straight away since it has to process the Transforms first: https://github.com/awslabs/serverless-application-model/blob/master/HOWTO.md#deploying-to-aws-cloudformation

[RegEM]

Ok. Thanks for the great explanation. I believe I even understand most of it.
I was hoping it was a bit easier re: the transforms, as my starting point was your API backend example. I just tagged on a few elements. A deployment example or two would of course be fantastic.
I will go back to the old laptop now with more confidence. Too much going on to move to the new one right now anyways.

[RegEM]

hmmm, I thought you just asked me to send you the error. Can do, if you still want it? Disregard, Github is acting weird on me.

[RegEM]

I changed my deployment from New-CFNStack type to "aws cloudformation deploy --template-file 'https://tcam-vapo-fn.s3.amazonaws.com/template.yaml' --stack-name 'CFNtcamUtilities' --capabilities CAPABILITY_IAM"
And it worked fine, creating the change set on the fly. It didn't quite finish properly, but I just hit return to get the success text.
Thanks for pointing me in the right direction.

[scrthq]

Awesome! With Vaporshell 2.0, you'd be able to do something similar to the following to create the change set then execute it:

New-VSChangeSet -TemplateUrl 'https://tcam-vapo-fn.s3.amazonaws.com/template.yaml' -Capabilities CAPABILITY_IAM -StackName CFNtcamUtilities

Invoke-VSChangeSetExecution -StackName CFNtcamUtilities

[RegEM]

cool. I'm just trying to knock off some super simple lambda functions and iot rules, and this should help me do that. I hope you have a lot of success with this.

[scrthq]

Thank you!! Definitely a passion project right now, hopefully it becomes something essential to a lot of CFN gurus 🥇

[scrthq]

@RegEM -- Just noticed something odd about calling the SDK from Powershell 3/4 that doesn't happen in 5 that is likely causing your -ValidateTemplate flag to fail (and will also likely kick an error when you try to use Set-VSCredentials). Assuming the error you get looks a bit like this?

Method invocation failed because [Amazon.Runtime.StoredProfileAWSCredentials] does not contain a method named 'new'.

I've fixed it already in the new version and should hopefully have that out by this weekend!

[RegEM]

thanks.
I don't recall any fail on Set-VSCredentials, at all.
I think it was just that the command Initialize-AWSDefaults was replaced with Set-AWSCredential. So my shell commands were not authorized properly on the new box is all.
So I believe I could successfully run Vaporshell, including the validation, once I used either Set-AWS, or Set-VS.

[scrthq]

Gotcha! Well, it was at least something I encountered when testing in both PS v3 and v4 on a Win7 VM, so I felt it worth the mention. Regardless, I'm glad it was working fine for you and it'll be patched to work universally ASAP!

[RegEM]

Thanks Nate.
I'm looking forward to trying to import my stack that I have been working on, done with serverless. It's not super complicated, so I think it will be a nice test. I'm hoping to abandon serverless and do everything in vaporshell.
I'm starting over now and learning how to setup a proper dev environment. Or at least my guess on what will be good. So plenty to keep me busy while you fine tune v2..

[scrthq]

The only thing left now is to update documentation and deploy v2.0.0! Here's a GIF of a quick template build (S3 bucket only) to stack deployment (with New-VSStack) to stack tear down (with Remove-VSStack):

[scrthq]

v2.0.0 out now! https://www.powershellgallery.com/packages/Vaporshell/2.0.0