expired tokens corrupt things

[marafa-sugarcrm]

if the aws tokens expire while running the import, aws2tf will continue to run but the state vs the files link will be messed up.
attempting to clean up the tf files as well as the state will not help

[marafa-sugarcrm]

related to #53

[awsandy]

If your working in Cloud9 - turn off the temporary credentials in the settings, the default instance role is sufficient for aws2tf.

This is a common thing with Terraform you need credentials that last as long as the provisioning run (or aws2tf run)

[marafa-sugarcrm]

revisiting.
i use aws sso from cli. it expires after 8 hours as per aws/aws-cli#7538. when scanning a large aws account, those 8 hours can be insufficient

can aws2tf handle this expiration in a better manner?

Found Error: │ Error: reading Organization (o-4oizb4q5g1) accounts: AccessDeniedException: You don't have permissions to access this resource. .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Invalid character .... (pass for now)
Found Error: │ Error: Invalid character .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Attribute redefined .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Argument or block definition required .... (pass for now)
Found Error: │ Error: Unsupported argument .... (pass for now)
Found Error: │ Error: Unsupported argument .... (pass for now)
Found Error: │ Error: Unsupported argument .... (pass for now)

[awsandy]

So I'd suggest:

1. aws2tf can be "restarted" from where it left off:

on the second run use ./aws2tf.sh ........ -f yes -c yes

You have to use both of these the -f yes does a "fast forward" ie all the resource types that were successfully imported are quickly skipped.

The -c yes is a "continue" flag - it uses the statefile form the previous run (doesn't delete it) and jumps over resources for which it already has a *.tf file

2. A port to python and terraform v5's method of importing resources is well underway - this new python based code only makes a few calls to terraform (as opposed to the 100's/1000's calls currently being made and subsequently is dramatically faster.

You will be able to transition to this new version by setting an environment variable - and if the current code base is ported and the environment variable is set aws2tf will call the new python version for each terraform type supported/ported.

I'll issue new docs on how to do this next week.

Over time the existing bash shell version of the code will be phased out and replaced with the much faster python version.

[marafa-sugarcrm]

revisiting
i came up with a script

    supported_resources=$(${HOME}/git/aws2tf/aws2tf.sh -t | grep -A 1000  -E '^ *-t <type>' |grep -v type)

    for resource in ${supported_resources}
    do 
        dir="${HOME}/git/aws2tf/generated/tf.${AWS_ACCOUNT_ID}_${AWS_DEFAULT_REGION}"
        if ! [ -d ${dir}_${resource} ]
        then 
            msg 4 "aws2tf: resource= $resource"
            if ${HOME}/bin/aws2tf -p ${AWS_PROFILE} -r ${AWS_DEFAULT_REGION} -t ${resource}
            then 
                mv ${dir} ${dir}_${resource}
            else 
                msg 3 "Failed to run aws2tf succesfully"
            fi
        else
            msg 2 "Skipping ${resource}"
        fi
    done

however it is dependent on the list generated by aws2tf itself. is this the complete list?

[awsandy]

It is probably not complete - I'll look at what it takes to make it complete for you.

[awsandy]

As the Python version is a lot faster - the expired token problem should be less prevalent