Repository Archived and Moved

This repository has been archived and its contents have been moved to a new location.

You can now find the content-watcher-service repository within the AmplicaLabs/gateway repository at the following path:

./services/content-watcher

For any future updates or contributions, please refer to the new location. Thank you!

---

Link to the new repository: AmplicaLabs/gateway

Content Watcher

📗 Table of Contents

• 📖 About the Project
• 🔍 Arch Map
• 🛠 Built With
  • Tech Stack
  • Key Features
• 🚀 Live OpenAPI Docs
• 💻 Getting Started
  • Prerequisites
  • Environment Variables
  • Setup
  • Install
  • Usage
  • Swagger
  • Queue Management
  • Run tests
  • Linting
  • Formatting
• 🤝 Contributing
• ⭐️ Show your support
• 🙏 Acknowledgements
• ❓FAQ
• 📝 License

📖 Content Watcher Service

The Content Watcher Service is part of the "Social Gateway" collection of services that provides a familiar callback API to retrieve content and publishing announcements from the Frequency chain. The service handles all of the necessary blockchain interaction and allows clients to interact using a familiar, web2-friendly interface.

🔭 Arch Maps

The Content Watcher Service provides a webhook system for receiving notice of new content. Even with a crawl request, the results are not retreived, but pushed to the webhook.

(back to top)

🛠 Built With

Tech Stack

Server

• NestJS
• Node.js
• TypeScript

Data store

• Redis (ioredis)

Frameworks and Libraries

• BullMQ (NestJS BullMQ Integration)
• Joi
• Axios

Polkadot and DSNP Integration

• Polkadot API (@polkadot/api)
• DSNP (Activity Content, Frequency Schemas, ParquetJS)

Testing

• Jest
  • ts-jest
  • Supertest
  • ioredis-mock

Formatting

• Prettier
• TypeScript ESLint

Build and Deployment

• Dotenv
• Docker
• Docker Compose
• Nest CLI

Key Features

API

• Parse DSNP Messages on Frequency
• Send content to registered webhooks
• Simple Schema and MSA Id based filtering

Scanner API

• Start, Stop, and restart scanning

(back to top)

🚀 Live Docs

• Live Docs
• API Documentation
• GitHub

(back to top)

💻 Getting Started

This guide is tailored for developers working in the code base for the Content Watcher Service itself. For a more tutorial tailored more for developers wanting to deploy the Content Watcher Service as part of the broader Social Gateway in order to develop their own Social Gateway app, visit Live Docs.

To prepare and run a local instance of the Content Watcher Service for local development, follow the guide below.

Prerequisites

In order to run this project you need:

• Nodejs
• Docker or Docker-compatible container system for running Gateway Services
  • (note, Docker is not strictly required; all of the services described below may be installed or built & run locally, but that is outside the scope of this guide)

Environment Variables

Use the provided env.template file to create an initial environment for the application, and edit as desired. Additional documentation on the complete set of environment variables is provided in the ENVIRONMENT.md file.

1. For running locally, copy to the template file to .env and update as needed.

cp env.template .env

2. Configure the environment variable values according to your environment. - Docker: .env.docker.dev - Local: .env

Setup

Clone this repository to your desired folder:

Example commands:

  git clone git@github.com:AmplicaLabs/content-watcher-service.git
  cd content-watcher-service

Install

Install NPM Dependencies:

  npm install

Usage

To run the project, execute the following command:

1. Start the required auxiliary services

Frequency node, Redis, IPFS

docker compose up -d frequency redis ipfs

2. [Optional] Start the publishing services

Content Publishing Service

docker compose up -d content-publishing-service-api content-publishing-service-worker

3. Start the application services

Each of the application services may be run either under Docker or bare-metal, depending on your preferred development workflow.

Running locally

npm run start:api:dev

-- or --

Running under Docker

docker compose up [-d] content-watcher-service

4. [Optional] Setup a basic publisher

Setup provider and users for the publishing service.

npm run local:init

5. [Optional] Webhook registration

Start a simple webhook that will just echo out responses to the console.

npm run local:webhook

5. [Optional] Trigger Content

Publishes some random content through the Content Publishing Service in Docker. Can be run more than once.

npm run local:publish

5. Check the job in BullUI, to monitor job progress based on defined tests.

Swagger UI

Check out the Swagger UI hosted on the app instance at http://localhost:3000/api/docs/swagger to view the API documentation and submit requests to the service.

Queue Management

You may also view and manage the application's queue at http://localhost:3000/queues.

For the Content Publishing Service Queue go to http://localhost:3001/queues.

Run unit tests

To run unit tests, run the following command:

  npm test

Run e2e tests:

1. Execute the following make command to deploy the entire stack:

   make test-services-start

2. Run the following make command to execute the content watcher tests:

   make test-e2e

3. Alternatively, create a .env file, run npm run start:api to start the content watcher as a standalone service, register a webhook with the content watcher using swagger, and try the following scenarios:

• Reset Scanner: This action will reset the scanner to start from the beginning of the chain or whichever block is chosen to start with. Upon successful parsing, a respective announcement will be made to the webhook.
• Put a Search Request: This action will put a search request on the queue. The request requires a start block and end block. Upon successful parsing, a respective announcement will be made to the webhook.

Linting:

  npm run lint

Auto-format:

  npm run format

(back to top)

🤝 Contributing

Contributions, issues, and feature requests are welcome!

• Contributing Guidelines
• Open Issues

(back to top)

⭐️ Show your support

If you would like to explore contributing bug fixes or enhancements, issues with the label good-first-issue can be a good place to start.

(back to top)

🙏 Acknowledgements

Thank you to Frequency for assistance and documentation making this possible.

❓FAQ

• Can I use this service in my production social app?

  • Yes. All the Gateway Services are intended to be ready-to-use out of the box as part of the fabric of your own social media app using DSNP on Frequency.

• Does this service index content?

  • No. This can be used by your own indexing service to get access to the content, but the service is intentially limited to getting the content and further customization is open to you.

• Does this service filter content?

  • No. This can be used by your own content filtering service to get new content and then have your custom service process them.

(back to top)

📝 License

This project is Apache 2.0 licensed.

(back to top)