Add Adaptive Cards support for interactive bot elements

[cbcoutinho]

Summary

This issue proposes adding support for Adaptive Cards to enable rich, interactive bot experiences in Nextcloud Talk. This will allow bots to create sophisticated forms, buttons, layouts, and data displays beyond simple text messages and reactions.

I've created a PR that introduces Adaptive Cards as a POC in #16506, with example cards showcasing some of the functionality.

Problem Statement

Currently, Nextcloud Talk bots are limited to:

• Sending/receiving text messages
• Adding/removing emoji reactions
• Creating polls

This restricts bots from providing interactive experiences like:

• Form-based data collection (feedback, registrations, settings)
• Multi-step workflows with dynamic branching
• Quick action buttons (confirmations, menu selections)
• Rich content displays (tables, formatted layouts, images)
• Date/time selection interfaces

Proposed Solution

Adopt Adaptive Cards as the standard for interactive bot elements. Adaptive Cards is an open-source, platform-agnostic framework with:

• JSON-based declarative schema
• Industry adoption (Microsoft Teams, Outlook, Webex)
• Rich ecosystem (visual designer, SDKs, samples)
• Comprehensive element library (20+ elements, 6 input types)
• Built-in accessibility support

Full details in ADR 0001

Phase 1 & 2 Implementation (This PR)

Phase 1: Basic Card Rendering

Frontend:

• Integration of adaptivecards npm package
• AdaptiveCardRenderer.vue component wrapping the SDK
• Support for core elements: TextBlock, Image, Container, ColumnSet
• Support for basic inputs: Input.Text, Input.ChoiceSet, Input.Toggle
• Support for actions: Action.Submit, Action.OpenUrl

Backend:

• New adaptivecards capability in capabilities endpoint
• Extended bot message schema to support card objects
• Cards embedded in bot messages via parameters.object.type: "adaptivecard"

Bot Message Format:

{
    "message": "Please fill out this form",
    "parameters": {
        "object": {
            "type": "adaptivecard",
            "id": "card-abc123",
            "card": {
                "type": "AdaptiveCard",
                "version": "1.5",
                "body": [
                    {
                        "type": "TextBlock",
                        "text": "What's your feedback?",
                        "weight": "Bolder"
                    },
                    {
                        "type": "Input.Text",
                        "id": "feedback",
                        "isMultiline": true,
                        "placeholder": "Enter your feedback..."
                    }
                ],
                "actions": [
                    {
                        "type": "Action.Submit",
                        "title": "Submit"
                    }
                ]
            }
        }
    }
}

Phase 2: Action Handling

User Interactions:

• Intercept Action.Submit in renderer
• Collect all Input.* values from the card
• Send webhook to bot with Activity Streams 2.0 payload

Webhook Payload (on submit):

{
    "type": "adaptivecard_submit",
    "actor": {
        "id": "users/alice",
        "name": "Alice"
    },
    "target": {
        "id": "12345",
        "name": "Project Discussion"
    },
    "card": {
        "id": "card-abc123",
        "values": {
            "feedback": "Great feature!"
        },
        "data": {}
    }
}

Security:

• Action.OpenUrl with domain whitelisting
• URL confirmation dialogs for external links
• SSRF protection (blocks javascript:, data: URIs)
• Server-side input validation and sanitization
• XSS prevention with content escaping

Backwards Compatibility:

• Feature gated by adaptivecards capability
• Older clients show fallback: "🤖 Bot Name sent an interactive card (view in updated client)"
• Includes deeplink to web UI

Example Use Cases

1. Meeting Scheduler Bot

{
    "type": "AdaptiveCard",
    "version": "1.5",
    "body": [
        {
            "type": "TextBlock",
            "text": "📅 Schedule Your 1:1 Meeting",
            "size": "Large",
            "weight": "Bolder"
        },
        {
            "type": "Input.Date",
            "id": "date",
            "label": "Preferred date"
        },
        {
            "type": "Input.Time",
            "id": "time",
            "label": "Preferred time"
        }
    ],
    "actions": [
        {
            "type": "Action.Submit",
            "title": "Schedule"
        }
    ]
}

2. Incident Response Bot

{
    "type": "AdaptiveCard",
    "version": "1.5",
    "body": [
        {
            "type": "TextBlock",
            "text": "🚨 Production Alert",
            "color": "Attention",
            "weight": "Bolder"
        },
        {
            "type": "FactSet",
            "facts": [
                {"title": "Service", "value": "API Gateway"},
                {"title": "Severity", "value": "High"},
                {"title": "Error Rate", "value": "23%"}
            ]
        },
        {
            "type": "Input.ChoiceSet",
            "id": "action",
            "label": "Response",
            "choices": [
                {"title": "Acknowledge", "value": "ack"},
                {"title": "Escalate", "value": "escalate"},
                {"title": "Resolve", "value": "resolve"}
            ]
        }
    ],
    "actions": [
        {
            "type": "Action.Submit",
            "title": "Submit"
        }
    ]
}

Future Phases (Not in This PR)

Phase 3: Card Updates & Nextcloud Extensions

• Bot API for updating cards: PUT /ocs/v2.php/apps/spreed/api/v1/bot/{token}/card/{cardId}
• Real-time card updates via WebSocket
• Custom Nextcloud actions: x-nextcloud.startCall, x-nextcloud.shareFile, etc.
• Collaborative mode (show live responses like polls)

Phase 4: Advanced Features

• Action.Execute for custom verbs
• Action.ToggleVisibility
• Data binding with ${variable} expressions
• Conditional rendering with $when rules
• Full Adaptive Cards 1.5 spec compliance

Benefits

1. Rich Interactive Experiences - Forms, buttons, layouts, data displays
2. Developer Familiarity - Reuse knowledge from Teams/Outlook bot development
3. Excellent Tooling - Visual designer, extensive samples, documentation
4. Future-Proof - Ongoing spec development by community
5. Cross-Platform Potential - Portable across federated instances
6. Reduced Maintenance - Spec evolution handled by Adaptive Cards community
7. Accessibility - Built-in ARIA support
8. Faster Development - ~13 weeks total vs. 16+ weeks for proprietary solution

Trade-offs

• Bundle Size: ~150KB added (adaptivecards package, minified)
• External Dependency: Relies on Microsoft-maintained open standard
• Learning Curve: Team and bot developers learn new schema
• Limited Customization: Styling constrained by SDK (ensures consistency)

Testing

The implementation includes:

• Unit tests for AdaptiveCardRenderer component
• Integration tests for bot message handling
• Security tests for URL validation and XSS prevention
• Example cards in docs/adaptive-cards-examples.md

Documentation

• ADR 0001: Full architectural decision with evaluation criteria
• Bot Documentation: Updated docs/bots.md with Adaptive Cards usage
• Examples: Comprehensive examples in docs/adaptive-cards-examples.md

References

• Adaptive Cards Official Site
• Adaptive Cards Designer
• Schema Explorer
• JavaScript SDK Documentation
• ADR 0001

Related Issues

• Bot Overlay API - Display bot messages in call view #16114 Bot Overlay API

[nickvergessen]

This is a bit problematic for our mobile apps.
https://github.com/microsoft/AdaptiveCards/?tab=readme-ov-file#end-user-license-agreement-for-our-binary-packages

Consumption of the AdaptiveCards binary packages are subject to the Microsoft EULA (End User License Agreement). Please see the relevant terms as listed below:
…

• Android/iOS

There is no way that we will ask mobile users to accept Microsoft's EULA

---

We like the general idea of having something like that in place, but maybe apart from "Microsoft Universe" and "Custom for Nextcloud" there is something else existing for other such cases that could be reused?

[cbcoutinho]

This is a great point - I wasn't aware of how deep the "Microsoft Universe" went with this product or the related EULA. I was under the impression that the MIT license was enough, so I might need some help assessing the other options.

When I first looked into this, Adaptive Cards was the easiest to get to a POC while being aesthetically pleasing. I haven't given the others a try but might be worth it depending on interest in this issue. For my particular use-case I want a way to send structured output to/from a bot, for which the existing message API is not sufficient.

There are some other contenders in this space that generalize on this idea:

• Slack Bot Kit
• JSON Forms

---

Here's a little side-by-side of Slack Block Kit vs JSON Forms vs Adaptive Cards at first glance. For my specific use-case of bots requesting structured output, the routing via Adaptive Cards was relatively straight forward. Doing something similar with one of the other two would require special handling of e.g. rendering, media and actions.

Bot Use Case	Block Kit	JSON Forms	Adaptive
Action buttons	✅ Has them	❌ No built-in	✅ Has
Rich media (images/video)	✅ Has them	❌ None	✅ Has
Tables	✅ Table block	❌ None	⚠️ FactSet
File uploads	✅ File input	❌ None	❌
Array controls	⚠️ Manual	✅ Built-in	❌
Conditional logic	❌ App-side	✅ Rules	✅ $
Vue renderer	❌ Need custom	✅ Official	⚠️ Framework-agnostic
PHP library	✅ Official	❌ JS only	❌ JS
Clean licensing	✅ MIT	✅ MIT	❌ EULA

[nickvergessen]

Reopening this issue as we still like to have something like this in the future. Doesn't mean you have to do it or anything. You can also unsubscribe, but the issue contains good information.

From your last table it seems json forms is the best candidate