Memealyzer

Meme + Analyzer = Memealyzer

Memealyzer determines if a meme's sentiment is positive, negative, or neutral.

For example, given this meme:

Memealyzer will extract the text, analyize the sentiment of that text, and then change the border color to red for negative sentiment, green for positive, and yellow for neutral.

Memealyzer is an app built to demonstrate some the latest and greatest Azure tech to dev, debug, and deploy microservice applications, including:

• Azure SDKs - to interact Azure services
• Bicep & Azure CLI - to provision Azure resources
• Project Tye to dev, debug, and deploy the solution
• Blazor - for a responsive client-side app built with .NET/C#.
• Azure Functions - to connect message queues with SignalR
• Azure SignalR Service - to send messages to the web app

Architecture

This is the current .NET architecture - we are in the process of developing for other languages and architectures as well.

Azure Resources

The following Azure resources are used in this application:

1. Resource Group
2. Storage Account
3. Cognitive Services Form Recognizer
4. Cognitive Services Text Analytics
5. Cosmos DB
6. Key Vault
7. Azure Kubernetes Service
8. Application Insights
9. Azure SignalR Service
10. Azure Functions
11. Azure Service Bus

You can select the Data and Messaging Providers via environment variables. See the "Environment Variables" section below for more information.

Quickstart

We follow a 4 step process to go from nothing to a completely deployed Azure solution.

1. Start Dev Environment with VS Code Remote Container (GitHub Codespaces coming soon)
2. Provision Azure Resources with Bicep
3. Run Locally with Tye
4. Deploy to Azure with Tye or GitHub Actions

You will see {BASENAME} throughout this document. Replace it with any unique text that you'd like, such as memealyzer007. That will be used as the basename for all your Azure resources, i.e. If you use memealyzer007, then the resource group will be named memealyzer007rg.

We use the Azure CLI to perform resource deployment and configuration. The scripts below will automatically prompt you to login to the Azure CLI and set your active Azure subscription. You can set the AZURE_SUBSCRIPTION_ID environment variable in the .env file if you don't want to be prompted every time you run these scripts.

1. Start Dev Environment

The fastest way to get to get the Memealyzer dev machine setup is to use the Codespaces Dev Container which includes all of your development dependencies.

1. Install VS Code
2. Install VS Code - Remote Containers Extension
3. Open a Linux shell, such as bash or WSL bash
4. Clone the repo:
   • git clone https://github.com/jongio/memealyzer
5. Open the repo in VS Code code memealyzer
6. Hit F1, then select "Remote Containers - Open Folder in Container"
7. Select the memealyzer root folder and click the Open button.

It can take a while to open the Dev Container the first time, but subsequent opens will load much faster.

2. Provision Azure Resources

This will provision all the required Azure resources with Bicep.

• ./provision.sh {BASENAME}

It may take up to 30 minutes to provision all resources.

You can deprovision at any time with ./deprovision.sh {BASENAME}

3. Run Locally

Now that our Azure Resources are provisioned, we'll now use Tye to run our application locally.

1. ./run.sh {BASENAME}
2. Open the Tye dashboard: http://localhost:8000 and ensure that all services are running.

   The Azure Function takes a few minutes to start, so if you get an error while loading the app, then wait a few minutes before trying again.

3. Open http://localhost:1080
4. You can add memes by clicking on the "+" button
5. You can start the memestream by clicking on the "∞" button. This will add a new meme every 5 seconds.

If you get build errors, then CD to ./src/net and run ./clean.sh to clean up the .NET dirs.

If the app doesn't start correctly, then stop using CTRL+C and re-run the run.sh command. We are researching why this happens.

4. Deploy to Azure

Now that we've provisioned our resources and tested locally, it is time to deploy our code to Azure with Tye.

1. ./deploy.sh {BASENAME}
2. Click on the link that is outputted to the console - this address will be unique to your deployment.

Please also take a look at the GitHub Action in the .github/workflows folder to see how we are auto-deploying this to an environment.

VS Code Workspace

Execute the following command to open the VS Code workspace that has all the projects you need to get started:

code memealyzer-net.code-workspace

or

code-insiders memealyzer-net.code-workspace if you are using VS Code Insiders build.

Manual Dev Machine Setup

We recommend using a VS Code Dev Container, but you can setup on bare metal with these steps.

1. Terminal - WSL2, Zsh, GitBash, PowerShell, not Windows Command prompt as this application uses bash script files.
2. Azure CLI - v2.20.0+
3. Git
4. VS Code
5. Docker
6. Kubectl
7. .NET SDK - 5.0
8. Azure Functions Core Tools - v3.0.2881 minimum
9. Project Tye
10. Bicep - az bicep install
11. Playwright - Test Automation - Install Playwright via npm and run this to install dependencies: npx playwright install-deps

Troubleshooting

1. Error while loading WebApp

If you see this error:

A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found in './src/net/WebApp/bin/Debug/net5.0/'.

Then stop running the soltion and run it again. We do not know why this occasionally happens.

2. Error while building or running

If you see this error:

/opt/dotnet/5.0.100/sdk/5.0.100/Microsoft.Common.CurrentVersion.targets(4354,5): error MSB3021: Unable to copy file "obj/Debug/netstandard2.1/Lib.Model.dll" to "bin/Debug/netstandard2.1/Lib.Model.dll". Access to the path is denied. [/home/codespace/workspace/src/net/Libs/Lib.Model/Lib.Model.csproj]

Option 1: Run this from within the devcontainer to take ownership of the files: sudo chown -R codespace ~/workspace/src Option 2: CD to ./src/net and run ./clean.sh to clean up the .NET dirs.

Configuration

Data Provider

You can configure which store the app uses to persist your image metadata, either Cosmos DB or Azure Table Storage.

1. Open .env file that is in the root of this project.
2. Find or add the AZURE_STORAGE_TYPE setting
3. Set it to one of the following values:
   • COSMOS_SQL - This will instruct the app to use Cosmos DB.
   • STORAGE_TABLE - This will instruct the app to use Azure Storage Tables.

Cosmos Emulator

You can now use the Cosmos Emulator instead of Azure Cosmos. Follow these steps to enable the emulator, which will tell the ./run.sh and ./debug.sh scripts to run the cosmos-emulator container. See, ./scripts/cosmos.sh for details.

1. Open .env file that is in the root of this project.
2. Scroll to bottom and uncomment USE_COSMOS_EMULATOR=true
3. Use the ./run.sh and other commands as you normally would and the Cosmos Emulator will be used.

Troubleshooting

1. You may see this error message when starting the emulator: Error: failed to start containers: cosmos-emulator, which is likely a false-negative and can be ignored.

2. If you see this error while trying to run the app, then remove the container using docker tools and re-start memealyzer.

   The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

Azure Storage Emulation (Azurite)

You can locally emulate all three Azure Storage services: Blobs, Queues, and Tables with Azurite.

To emulate all three of them, all you need to do is set the USE_AZURITE environment variable in the .env file to true. You can also enable or disable each individual service emulation with the following settings:

• USE_AZURITE - Enables emulation for all three.
• USE_AZURITE_BLOB - Enables emulation for blobs only.
• USE_AZURITE_QUEUE - Enables emulation for queues only.
• USE_AZURITE_TABLE - Enables emulation for tables only.

Ngrok Tunnel to Azurite

When Azurite is enabled, an Ngrok tunnel is created to the blob and queue endpoints. This is done because:

1. The Form Recognizer client requires a publically addressable endpoint to read the blob from
2. The Azure Function Queue binding requires a queue that is also public.

See the Lib.Tunnel project for more information.

Ngrok is the only currently supported tunneling client, but we may consider adding alternatives in the future.

By default Ngrok has very aggressive throttling for the free account and it will often throttle requests from the Storage Queue Function. For this reason, do not use the "Meme Stream" feature and do not aggresively add memes while using Azurite/Ngrok. For this reason we have added the AZURE_STORAGE_QUEUE_MAX_POLLING_INTERVAL config setting that allows you to specify how long the function should wait before polling the queue for new messages. The default is set to 10 seconds, but this still will not prevent you from being throttled if you add a lot of memes at once.

If you have a paid account, then you can specify your auth token in the NGROK_AUTHTOKEN environment variable in this format --authtoken yourtoken.

Note that Azurite and Ngrok run over HTTP, not HTTPS so if you adopt this pattern in your application, then ensure you are not transporting any sensitive data in your application.

Messaging Provider

You can configure which messaging service you want to use, either Service Bus Queue or Azure Storage Queue.

1. Open .env file that is in the root of this project.
2. Find or add the AZURE_MESSAGING_TYPE setting
3. Set it to one of the following values:
   • SERVICE_BUS_QUEUE - This will instruct the app to use Service Bus Queue.
   • STORAGE_QUEUE - This will instruct the app to use Azure Storage Queue.

Border Style

You can configure the image border style with the Azure App Configuration service. It will default to solid, but you can change it to any valid CSS border style. You can either do this in the Azure Portal, or via the Azure CLI with this command:

az appconfig kv set -y -n {BASENAME}appconfig --key borderStyle --value dashed

After you change the setting, reload the WebApp to see the new style take effect.

Environment Variables

You can add override any of the following environment variables to suit your needs. Memealyzer chooses smart defaults that match what is created when you deploy the app.

Name	Default Value	Allowed Values
BASENAME	This will be populated from the first argument you pass to scripts, such as ./run.sh meme01
AZURE_SUBSCRIPTION_ID	The subscription that you want to use for your app.
AZURE_COSMOS_ENDPOINT	https://${BASENAME}cosmosaccount.documents.azure.com:443
AZURE_COSMOS_KEY	Default value in .env files
USE_COSMOS_EMULATOR	Set to true to enable Cosmos Emulator	false
AZURE_COSMOS_DB	memealyzer
AZURE_COSMOS_COLLECTION	images
AZURE_COSMOS_KEY_SECRET_NAME	CosmosKey
AZURE_COSMOS_THROUGHPUT	400
USE_AZURITE	Set this to true to use Azurite for Tables, Blobs, and Queue	false
USE_AZURITE_BLOB	Set this to true to use Azurite for Blob	false
USE_AZURITE_QUEUE	Set this to true to use Azurite for Queue	false
USE_AZURITE_TABLE	Set this to true to use Azurite for Table	false
AZURITE_ACCOUNT_KEY	devstoreaccount1
AZURITE_CONNECTION_STRING	Default value in Config.cs
AZURITE_PROXY_CONNECTION_STRING	The Azurite connection string with the Tunnel endpoints.
NGROK_URI	Default value is read at runtime from localhost:4040
AZURE_FORM_RECOGNIZER_ENDPOINT	https://${BASENAME}fr.cognitiveservices.azure.com/
AZURE_KEYVAULT_ENDPOINT	https://${BASENAME}kv.vault.azure.net/
AZURE_STORAGE_ACCOUNT_NAME	${BASENAME}storage
AZURE_STORAGE_BLOB_ACCOUNT_NAME	${BASENAME}storage
AZURE_STORAGE_QUEUE_ACCOUNT_NAME	${BASENAME}storage
AZURE_STORAGE_TABLE_ACCOUNT_NAME	${BASENAME}storage
AZURE_STORAGE_BLOB_ENDPOINT	https://${BASENAME}storage.blob.core.windows.net/
AZURE_STORAGE_QUEUE_ENDPOINT	https://${BASENAME}storage.queue.core.windows.net/
AZURE_STORAGE_TABLE_ENDPOINT	https://${BASENAME}storage.table.core.windows.net/
AZURE_STORAGE_QUEUE_MAX_POLLING_INTERVAL	"00:00:10"
AZURE_TEXT_ANALYTICS_ENDPOINT	https://${BASENAME}ta.cognitiveservices.azure.com/
AZURE_APP_CONFIG_ENDPOINT	https://${BASENAME}appconfig.azconfig.io
AZURE_CONTAINER_REGISTRY_SERVER	${BASENAME}acr.azurecr.io
AZURE_SERVICE_BUS_NAMESPACE	{BaseName}sb.servicebus.windows.net
AZURE_STORAGE_BLOB_CONTAINER_NAME	blobs
AZURE_MESSAGES_QUEUE_NAME	messages
AZURE_STORAGE_QUEUE_MSG_COUNT	10
AZURE_STORAGE_QUEUE_RECEIVE_SLEEP	1 second
AZURE_STORAGE_TABLE_NAME	images
AZURE_STORAGE_TYPE	COSMOS_SQL	COSMOS_SQL, STORAGE_TABLE
AZURE_MESSAGING_TYPE	SERVICE_BUS_QUEUE	SERVICE_BUS_QUEUE, STORAGE_QUEUE
AZURE_CLIENT_SYNC_QUEUE_NAME	sync
IMAGE_ENDPOINT	https://meme-api.herokuapp.com/gimme/wholesomememes
TUNNEL_TYPE	NGROK	NGROK
NGROK_AUTHTOKEN	Needs to have --authtoken and then your token