AWS::QuickSight::Template - AWS::QuickSight::* #274

pgarbe · 2019-11-22T15:03:41Z

1. Title

AWS::QuickSight::*

2. Scope of request

There'se no QuickSight resources available, although the scope of the API has recently been extended.

3. Expected behavior

Create DataSets, Analysis, Dashboards and Reportings via CloudFormation.

5. Helpful Links to speed up research and evaluation

https://docs.aws.amazon.com/quicksight/latest/APIReference/API_Operations.html

6. Category (required)

Analytics (Athena, EMR, Glue,...)

AndrewCi · 2020-03-03T15:05:17Z

Is there any sort of timeline or update on when additional CloudFormation support for QuickSight will be available?

humanzz · 2020-06-05T13:24:43Z

Hi there... any updates on this?

matthewcarlisle · 2020-07-08T16:05:41Z

Bueller?

rianwouters · 2020-08-20T13:52:27Z

It's quite unclear to me why AWS keeps on building specific resources.
If you would only standardize a bit on the format of the HTTP API of the services, a custom resource could easily be build to have cloudformation support readily available as soon as the Service`s HTTP API / SDK support is available.
See started this already almost 2 years ago, but it was especially difficult because there is hardly any standardisation in the SDK interfaces. See https://github.com/rianwouters/cfn-customresources

benzaam · 2020-10-26T12:56:32Z

Hi ! Any update on this? This would come handy, certainly with creating templates. This is currently a pain to do with the AWS CLI...

moltar · 2021-01-17T02:46:44Z

@rohits-spec do you have any insight as to how soon this is coming? We are planning our deployment of QS right now, and we need to know whether to use work arounds or if CF will be supported any day now?

Will this take:

week
month
quarter

?

Thanks! ❤️

aidansteele · 2021-01-17T02:48:55Z

Quicksight now has CloudFormation support for:

AWS::QuickSight::Analysis
AWS::QuickSight::Dashboard
AWS::QuickSight::Template
AWS::QuickSight::Theme

I haven't actually tried them yet to see how complete they are, but they arrived yesterday!

moltar · 2021-01-17T04:07:43Z

Ah cool.

I was hoping there'd be support for actually deploying the QS service itself.

It's annoying to do manual setup in staging, dev and testing.

Especially when all other parts are already automated with CDK.

rohits-spec · 2021-02-10T22:53:25Z

Closing this issue as Quicksight onboarded CloudFormation on 1/14.

otakusid · 2021-02-11T08:41:01Z

Any chances that samples with CloudFormation resources configuration will be added to the documentation? It will be cool to have an example of an Analysis setup with some visualizations configuration

pgarbe · 2021-02-16T14:22:58Z

@rohits-spec I'd not close it. It's not usable at the moment.

RemjeyLiu · 2021-02-16T18:28:18Z

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization
Namespace
Users
Groups
Group Memberships
Data Sources
Data Sets
Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

rohits-spec · 2021-02-16T19:44:21Z

Reopening this issue due to the concerns raised here by community members. I will work with Quicksight team and provide an update here.

austinlasseter · 2021-03-08T14:47:11Z

I agree with the commenters above. We need robust CloudFormation support for QuickSight. Re-building the same dashboards from scratch in dev, test and prod using nothing but the GUI... really kinda sucks.

danieljamesscott · 2021-03-08T14:51:26Z

Probably not quite the right place to ask for this for it, but API support for configuring VPC Connections to QuickSight would be nice. https://docs.aws.amazon.com/quicksight/latest/user/vpc-creating-a-connection-in-quicksight.html

Then adding CloudFormation support for it too would be wonderful!

moltar · 2021-03-08T14:52:33Z

Ha, @danieljamesscott was just looking for the exact same thing today. I guess there is no way, eh?

danieljamesscott · 2021-03-08T15:00:25Z

Ha, @danieljamesscott was just looking for the exact same thing today. I guess there is no way, eh?

Not that I've found. 😢

gtfuhr · 2021-03-17T12:58:39Z

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization

Namespace

Users

Groups

Group Memberships

Data Sources

Data Sets

Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

Is there any estimate for the deployment of the DataSets, DataSources, and Permissions QuickSight/CloudFormation features?

Lanayx · 2021-04-06T12:35:22Z

Can anybody provide any examples of using these new CloudFormation or CDK constructs? The documentation is pretty useless

mmuller88 · 2021-04-27T08:43:28Z

Here we go. Will do a blog post soon and create those CDK custom constructs
dataset.txt
datasource.txt

accrisoftteam · 2021-05-05T15:40:04Z

I recently attempted to use AWS::QuickSight::DataSource and ran into an Internal Error when attempting to use RdsParameters. We have worked around the issue thanks to AWS Support (use MySqlParameters instead) but wanted to mention it here as Support said it is reported as non-functional.

mmuller88 · 2021-05-06T13:43:47Z

Here we go.
Post: https://martinmueller.dev/qs-constructs-eng/
GitHub Repo: https://github.com/mmuller88/cdk-quicksight-constructs

karthikns16 · 2021-10-27T21:28:36Z

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization

Namespace

Users

Groups

Group Memberships

Data Sources

Data Sets

Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

Please give this capability ASAP.

otakusid · 2021-11-11T20:39:16Z

no no no, there is still no way to describe details of analysis and there are no samples of resources usage in documentation

amukasyan · 2021-11-11T20:48:27Z

I strongly disagree with closing. Unless documentation has been greatly updated and all above concerns are addressed in a blog post.

luchees · 2021-12-02T07:18:49Z

Is there any timeline on fixing the circular dependency between Analysis and Template?

alerof · 2021-12-06T09:00:36Z

Has anybody created AWS::QuickSight::DataSet for ATHENA DataSource ?

I'm stuck with
"Model validation failed (#: extraneous key [DataSourceArn] is not permitted)" (RequestToken: 61292db1-xxx, HandlerErrorCode: InvalidRequest)

I have

  QSDataSource:
    Type: AWS::QuickSight::DataSource
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSourceId: 'QSDataSource-XXX'
      Permissions:
        - Actions:
            - 'quicksight:DescribeDataSource'
            - 'quicksight:DescribeDataSourcePermissions'
            - 'quicksight:PassDataSource'
            - 'quicksight:UpdateDataSource'
            - 'quicksight:DeleteDataSource'
            - 'quicksight:UpdateDataSourcePermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      Name: !Sub 'ApiGw-${Environment}'
      SslProperties:
        DisableSsl: false
      Type: ATHENA
	  
  QSDataSet:
    DependsOn: QSDataSource
    Type: AWS::QuickSight::DataSet
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSetId: 'QSDataSet-XXX'
      ImportMode: SPICE
      Name: ApiGwShort
      Permissions:
        - Actions:
            - 'quicksight:CreateDataSet'
            - 'quicksight:DeleteDataSet'
            - 'quicksight:DescribeDataSet'
            - 'quicksight:DescribeDataSetPermissions'
            - 'quicksight:PassDataSet'
            - 'quicksight:UpdateDataSet'
            - 'quicksight:UpdateDataSetPermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      PhysicalTableMap:
        RelationalTable:
          DataSourceArn: !Sub 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:datasource/QSDataSource-XXX'
          InputColumns:
            - servicename
            - dt
            - responsesize
          Name: apigw_short
          Schema: !Sub 'logs-db-${Environment}'

moltar · 2021-12-14T13:12:31Z

How do you create a new DataSource? It is asking for dataSourceId, and if I provide one, it errors out with:

Resource of type 'AWS::QuickSight::DataSource' with identifier 'Foo' was not found.

If I don't provide it, then it errors out with:

1 validation error detected: Value null at 'dataSourceId' failed to satisfy constraint: Member must not be null

So what's the correct approach?

I've tried the following values:

AB
b5172c44-86fa-49b2-bd8f-9679260dc378 (Random UUID)
null

moltar · 2021-12-15T05:53:06Z

Huh, so strange. The error:

Resource of type 'AWS::QuickSight::DataSource' with identifier 'My-DataSource-Name' was not found.

Is returned when specifying VpcConnectionProperties.VpcConnectionArn value that does not exist.

So weird and misleading.

moltar · 2021-12-15T05:58:01Z

Probably not quite the right place to ask for this for it, but API support for configuring VPC Connections to QuickSight would be nice. https://docs.aws.amazon.com/quicksight/latest/user/vpc-creating-a-connection-in-quicksight.html

Then adding CloudFormation support for it too would be wonderful!

This is really a huge blocker.

We can codify everything: application, RDS, security groups, VPC.

And we can codify a data source, a data set, and a dashboard.

But the bridge between these two is missing, and thereby it breaks everything, especially when using things like AWS CDK (infra-as-code).

vjancich · 2021-12-27T14:29:41Z

Has anybody created AWS::QuickSight::DataSet for ATHENA DataSource ?

I'm stuck with "Model validation failed (#: extraneous key [DataSourceArn] is not permitted)" (RequestToken: 61292db1-xxx, HandlerErrorCode: InvalidRequest)

I have

  QSDataSource:
    Type: AWS::QuickSight::DataSource
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSourceId: 'QSDataSource-XXX'
      Permissions:
        - Actions:
            - 'quicksight:DescribeDataSource'
            - 'quicksight:DescribeDataSourcePermissions'
            - 'quicksight:PassDataSource'
            - 'quicksight:UpdateDataSource'
            - 'quicksight:DeleteDataSource'
            - 'quicksight:UpdateDataSourcePermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      Name: !Sub 'ApiGw-${Environment}'
      SslProperties:
        DisableSsl: false
      Type: ATHENA
	  
  QSDataSet:
    DependsOn: QSDataSource
    Type: AWS::QuickSight::DataSet
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSetId: 'QSDataSet-XXX'
      ImportMode: SPICE
      Name: ApiGwShort
      Permissions:
        - Actions:
            - 'quicksight:CreateDataSet'
            - 'quicksight:DeleteDataSet'
            - 'quicksight:DescribeDataSet'
            - 'quicksight:DescribeDataSetPermissions'
            - 'quicksight:PassDataSet'
            - 'quicksight:UpdateDataSet'
            - 'quicksight:UpdateDataSetPermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      PhysicalTableMap:
        RelationalTable:
          DataSourceArn: !Sub 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:datasource/QSDataSource-XXX'
          InputColumns:
            - servicename
            - dt
            - responsesize
          Name: apigw_short
          Schema: !Sub 'logs-db-${Environment}'

I has the same issue @alerof, had to contact AWS support. PhysicalTableMap in DataSet in CF should look like this. It works for me know

      PhysicalTableMap:
        table:
          RelationalTable:
            Name: yourName
            DataSourceArn: !GetAtt DataSource.Arn
            InputColumns:
              - Name: column
                Type: STRING
            Schema: yourSchema

denys-tyshetskyy · 2022-01-18T01:43:32Z

Hi, I am trying to join 2 logical tables with Aurora being a datasource. Getting error: Unsupported join on clause "ptr-process-type = process-type-id"
Code looks like this:

QSProcessDataSet:
    Type: AWS::QuickSight::DataSet
    Properties:
      DataSetId: !Join
        - ''
        - - QS-Process-DataSet
          - !Ref Suffix
      Name: !Join
        - ''
        - - QS-Process-DataSet
          - !Ref Suffix
      AwsAccountId: !Ref AWS::AccountId
      ImportMode: SPICE
      PhysicalTableMap:
          PhysicalTable1:
            RelationalTable:
              DataSourceArn: !GetAtt QSDataSource.Arn
              InputColumns:
              - Name: id
                Type: INTEGER
              - Name: ptr_process_type
                Type: INTEGER
              Name: process
          PhysicalTable2:
            RelationalTable:
              DataSourceArn: !GetAtt QSDataSource.Arn
              InputColumns:
              - Name: id
                Type: INTEGER
              Name: process_type
      LogicalTableMap:
          LogicalTable1:
            Alias: QS-Process-Table
            DataTransforms:
            - RenameColumnOperation:
                ColumnName: ptr_process_type
                NewColumnName: ptr-process-type
            Source:
              PhysicalTableId: PhysicalTable1
          LogicalTable2:
            Alias: QS-ProcessType-Table
            DataTransforms:
            - RenameColumnOperation:
                ColumnName: id
                NewColumnName: process-type-id
            Source:
              PhysicalTableId: PhysicalTable2
          LogicalTable3:
            Alias: QS-ProcessProcessTypeJoin-Table
            Source:
              JoinInstruction: 
                LeftOperand: LogicalTable1
                RightOperand: LogicalTable2
                RightJoinKeyProperties:
                   UniqueKey: true
                OnClause: ptr-process-type = process-type-id
                Type: LEFT

Am I missing something?

Solved: I had to change the order of the tables in the join

denys-tyshetskyy · 2022-01-18T21:01:13Z

Is there a way to join two datasets in cloudformation? When I am attempting to join LogicalTable from second dataset with LogicalTable from the first dataset, the cloudformation is unable to recognize the name of the LogicalTable from another dataset.
Invalid request provided: Operands in JoinInstruction must refer to logical table id

gunarkroeger · 2022-03-16T16:41:37Z

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I am also running into this issue and agree that this should be re-opened.

It's not possible creating this with API/CLI nor CFN and CDK. Seems to be not only CFN who has a circular dependency.

I created this ticket in aws-cdk for this specific problem: aws/aws-cdk#19212

michuer · 2022-04-19T13:29:09Z

In my brief experience trying to use QuickSight with CloudFormation, I've already seen a number of problems:

As mentioned above by @nathanagez, the documentation is out of sync with reality. The Required field frequently does not reflect its true value (e.g. here and here)

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I was expecting a way to describe the details of an analysis rather than just defining its related data sets (i.e. adding sheets, adding visuals to those sheets, etc.). Not being able to describe these things will require me to edit them through the UI in production, which somewhat defeats the purpose of using IAC. Like @mjdean1994, my organization really frowns on having to click through a UI to configure things

@rohits-spec I'd also suggest that this ticket is reopened, because the CloudFormation support does not seem usable in its current state.
CC @jthomerson

Agreed this issue should be re-opened.

Hey, i can help you out with the circular dependency part.

create a Template using an "Manually created Analysis" as a Source entity.

Then Create one more template and analysis using Step1 template as Source entity.

i am creating template in Step2 to create Dashboards using that template.

References : https://devops.learnquicksight.online/quicksight-via-cloudformation.html you can download the cloudformation sample from here and they are using a publicly created template's arn, so you can describe to get the code of that.

Thankyou

Thank you for this workaround, it help with circular dependency problem

sidpremkumar · 2022-05-31T16:49:15Z

This is a huge problem for us, is there any update on when / if ever this will be fixed?

gunarkroeger · 2022-07-22T11:28:12Z

@zjinmei please move this out of shipped. As many before have pointed out, there is currently no way to automate the template to dashboard creation with CDK.

Problem 1: Circular dependency
it is not possible to create a template without an analysis or an analysis without a template. The only way to workaround it is to manually create an analysis, which is not a proper solution for CD/CI

Problem 2: Changes are not diff-able
if there are some changes done, we cannot do proper versioning with the code because the analysis is not really defined by cdk. Also, although we can define QuickSight dashboards on alpha, beta and prod stages, any changes will not be propagated because the pipeline does not detect that there were any changes.

Proposed solution:
As mentioned before, we need to be able to create the template fully with CDK, with JSON fields to define the visuals, titles, etc.

AkifRafique · 2022-07-22T17:53:21Z

As there still unresolved issues changing status back to researching.

Disciple153 · 2022-10-25T21:47:46Z

I wrote a solution to this problem with Python, and I am working on a PR to add functionality to CDK. Until QuickSight resources can be created using pure CDK, I think the best solution is to provide import functionality, and enable functionality to create copies of those imported resources. Once QuickSight resources can be created using pure CDK, that functionality can be added as well.

drewmoore · 2023-01-06T12:05:59Z

Also, cloudformation doesn't support tag-based row level security on datasets. This feature is essential for anonymous embedding, which is essential for my company's use case.

seanlogan-wh · 2023-01-19T22:25:44Z

It does not seem possible to reference a template version when creating a dashboard. As part of the pipeline I update the template from an existing template to create a new version. The dashboard references the template but it does not seem possible to pull the latest version from the template without a custom resource

  Dashboard:
    Type: AWS::QuickSight::Dashboard
    Properties:
      AwsAccountId: !Ref AWS::AccountId
      DashboardId: dashboard-id
      Name: dashboard-name
      SourceEntity:
        SourceTemplate:
        # Can't reference the template from CloudFormation above, it defaults to version 1 and can't use !GetAtt Version on Template
          Arn: !GetAtt Template.Version. VersionNumber or !GetAtt Template.Version (not currently supported)
          DataSetReferences:
            - DataSetPlaceholder: Placeholder
              DataSetArn: !GetAtt DataSet.Arn

I receive an error

Requested attribute Version.VersionNumber must be a readonly property
or
Attribute 'Version' does not exist
Looking at the cloudformation documentation for template it lists version as a return value

gunarkroeger · 2023-03-07T14:21:56Z

I see it is now possible to create templates using json and the CLI commands: https://docs.aws.amazon.com/cli/latest/reference/quicksight/create-template.html

@AkifRafique, when can we expect this to be available for CDK too?

TheDanBlanco · 2023-09-25T22:47:13Z

hey folks, being entirely honest, this is a hard thread to follow as a non-quicksights user/expert. i found some comments on the cdk thread that leads me to believe this is solved. can anyone confirm?

ericzbeard · 2024-03-20T21:11:52Z

Since the original request on this issue has been completed, I'm going to close this issue and recommend that any outstanding bugs or coverage gaps in the QuickSight resources be reported on new issues. It is very difficult to isolate individual problems that need to be addressed from the long chain of comments here. While many of the gaps have been covered, we recognize that there are still some missing pieces to fully automating a CI/CD solution for this service. Please report each of them separately, and we'll get them assigned and prioritized.

TheDanBlanco added the analytics Athena, EMR, CloudSearch, Elasticsearch, Kinesis, etc. label Nov 26, 2019

craigataws added this to the cov milestone Jul 21, 2020

rohits-spec closed this as completed Feb 10, 2021

rohits-spec reopened this Feb 16, 2021

WaelA changed the title AWS::QuickSight::* AWS::QuickSight::Template - AWS::QuickSight::* Oct 26, 2021

cfn-github-issues-bot closed this as completed Nov 11, 2021

zjinmei added the enhancement New feature or request label Nov 11, 2021

zjinmei reopened this Nov 12, 2021

peterwoodworth mentioned this issue Jun 21, 2022

QuickSight: Proper Analysis or Template CDK Construct aws/aws-cdk#19212

Open

2 tasks

ericzbeard closed this as completed Mar 20, 2024

AWS::QuickSight::Template - AWS::QuickSight::* #274

AWS::QuickSight::Template - AWS::QuickSight::* #274

Comments

pgarbe commented Nov 22, 2019

1. Title

2. Scope of request

3. Expected behavior

5. Helpful Links to speed up research and evaluation

6. Category (required)

AndrewCi commented Mar 3, 2020

humanzz commented Jun 5, 2020

matthewcarlisle commented Jul 8, 2020

rianwouters commented Aug 20, 2020

benzaam commented Oct 26, 2020

moltar commented Jan 17, 2021

aidansteele commented Jan 17, 2021

moltar commented Jan 17, 2021

rohits-spec commented Feb 10, 2021

otakusid commented Feb 11, 2021

pgarbe commented Feb 16, 2021

RemjeyLiu commented Feb 16, 2021

rohits-spec commented Feb 16, 2021

austinlasseter commented Mar 8, 2021

danieljamesscott commented Mar 8, 2021

moltar commented Mar 8, 2021

danieljamesscott commented Mar 8, 2021

gtfuhr commented Mar 17, 2021

Lanayx commented Apr 6, 2021

mmuller88 commented Apr 27, 2021

accrisoftteam commented May 5, 2021

mmuller88 commented May 6, 2021

karthikns16 commented Oct 27, 2021

otakusid commented Nov 11, 2021

amukasyan commented Nov 11, 2021 via email • edited Loading

luchees commented Dec 2, 2021

alerof commented Dec 6, 2021

moltar commented Dec 14, 2021

moltar commented Dec 15, 2021

moltar commented Dec 15, 2021

vjancich commented Dec 27, 2021 • edited Loading

denys-tyshetskyy commented Jan 18, 2022 • edited Loading

denys-tyshetskyy commented Jan 18, 2022

gunarkroeger commented Mar 16, 2022

michuer commented Apr 19, 2022

sidpremkumar commented May 31, 2022

gunarkroeger commented Jul 22, 2022

AkifRafique commented Jul 22, 2022

Disciple153 commented Oct 25, 2022

drewmoore commented Jan 6, 2023

seanlogan-wh commented Jan 19, 2023

gunarkroeger commented Mar 7, 2023

TheDanBlanco commented Sep 25, 2023

ericzbeard commented Mar 20, 2024

amukasyan commented Nov 11, 2021 via email •

edited

Loading

vjancich commented Dec 27, 2021 •

edited

Loading

denys-tyshetskyy commented Jan 18, 2022 •

edited

Loading