Integrate Open-Fn with OpenSPP

[jeremi]

What is OpenFn?

Digital Public Good for workflow automation–a platform.

Video link explaining how OpenFN works

Official documentation

OpenFn v1 iPaaS will be used to integrate with OpenSPP.

• Write the Adaptor
• Prepare a demo flow

registrant_id is from: https://github.com/OpenSPP/openspp-registry/issues/247

Implement the methods:

• getGroup(registrant_id)
• getIndividual(registrant_id)
• searchGroup(domain)
• searchIndividual(domain)
• getGroupMembers(registrant_id)
• getServicePoint(name)
• getPrograms(program_id) Add a field program_id to each program. It is a string auto generated with the format: PROG_{SEQ} openspp-program#88
• getPrograms()
• getEnrolledPrograms(registrant_id)
• enroll(registrant_id, program_id)
• unenroll(registrant_id, program_id)
• createChangeRequest(group_id, individual_id, cr_type, data)

[nhatnm0612]

@jeremi https://community.openfn.org/t/testing-new-adaptor-on-local-environment-with-lightning/331/2
there is no way to test local adaptor on local Lightning running, confirmed by OpenFN employee. I gonna switch to the only way is writing automated tests and running that.

[jeremi]

ok

[dasunhegoda]

Left to be done,

• Publishing of the adaptor - create a PR and request them to accept it. @nhatnm0612 will create the PR today.

[jeremi]

How can I test it manually?

[nhatnm0612]

@jeremi we can not, only testing using unittest code via pnpm, and in term of unittest, I mean mocking test response from Odoo as well.

[dasunhegoda]

@nhatnm0612 will raise the PR today.

[jeremi]

but @nhatnm0612 can we install it locally and run it (not automated tests)?

[nhatnm0612]

but @nhatnm0612 can we install it locally and run it (not automated tests)?

@jeremi no, it is confirmed by OpenFN employee

[dasunhegoda]

@nhatnm0612 will setup OpenFN on local to be utilized with OpenSPP repo.

[dasunhegoda]

@nhatnm0612 has raised a PR in the OpenFN repo. OpenFn/adaptors#409

[dasunhegoda]

The methods mentioned in the description have been implemented.

[dasunhegoda]

@nhatnm0612 will list down the steps to setup on local.

[nhatnm0612]

steps for setting up local OpenFN:

1. Requirements:

• Using Unix like Os [Ubuntu] / MacOs with Apple Silicon chips [harder version].
• Having basic knowledge of nodejs & Unix like file system.

2. Pre-installed:

• pnpm & asdf
• elixir, erlang, nodejs, k6 & phoenix framework for elixir [these will be installed via asdf virtual machine] doc - keep it in mind that I am running local OpenFN Lightning in my MacOS since the docker version does not run on Mac M1
• fixing your local credentials to the Postgres DB to OpenFN Lightning at: https://github.com/OpenFn/Lightning/blob/c057048b56242613d05748a53b352ad0473fd552/config/dev.exs#L4

After Pre-installed step, basically, you can boost up OpenFn Lightning with this command:
iex -S mix phx.server

3. Installing new adaptor locally:

• Using mix [elixir] & ecto [elixir] to install dependencies, create DB & migrate
• Copy schemas from new adaptor to be the new schema for OpenSPP login:
  New schema: https://github.com/nhatnm0612/adaptors/blob/add-openspp-adaptor/packages/openspp/configuration-schema.json
  Schemas location: [OpenFNLightningRepo/priv/schemas]
  After this step, boost up OpenFN Lightning and you should see new schema for login.
• Fixing the code of pulling adaptor at: https://github.com/OpenFn/Lightning/blob/c057048b56242613d05748a53b352ad0473fd552/lib/lightning/adaptor_registry.ex#L224

  def fetch() do
    start = DateTime.utc_now()
    Logger.debug("Fetching adaptors from NPM.")

    openfn_adapters =
      Npm.user_packages("openfn")
      |> Enum.map(fn {name, _} -> name end)
      |> Enum.filter(fn name ->
        Regex.match?(~r/@openfn\/language-\w+/, name)
      end)
      |> Enum.reject(fn name ->
        name in @excluded_adaptors
      end)
      |> Task.async_stream(
        &fetch_npm_details/1,
        ordered: false,
        max_concurrency: 10,
        timeout: @timeout
      )
      |> Stream.map(fn {:ok, detail} -> detail end)
      |> Enum.to_list()
    result = openfn_adapters ++ [
      %{
        latest: "1.0.0",
        name: "openspp",
        repo: "git+https://github.com/nhatnm0612/adaptors/tree/add-openspp-adaptor",
        versions: [%{version: "1.0.0"}]
      }
    ]
    diff = DateTime.utc_now() |> DateTime.diff(start, :millisecond)
    # IO.inspect(result)
    Logger.debug(fn -> "Finished fetching adaptors in #{diff}ms." end)
    result
  end

[nhatnm0612]

@dasunhegoda due to requests from OpenFN side, I need a rectangle image [png, 512*190px] of OpenSPP for the adaptor logos. [link]. Also, they asked me to create test jobs with OpenSPP sandbox server's credentials [link]

I made some changes last weekend and all the feedbacks are positive now except these 2.

[dasunhegoda]

@nhatnm0612 : You can use the logo on our web and resize it as per requirement. If additional help is required, Lit usually helps us with this. We can use the openspp demo env as the sandbox.

[jeremi]

@nhatnm0612 ask Lit for it. He will prepare it for you.

[jeremi]

Can you add the methods:

• createIndividual(data)
• updateIndividual(individual_id, data)
• createGroup(data)
• updateGroup(group_id, data)
• addToGroup(group_id, individual_id, role)
• removeFromGroup(group_id, individual_id)

[nhatnm0612]

link to slack message

• getServicePoint(name) -> getServicePoint(id) using new field in create unique id on areas & service points for integrate with OpenFN #273
• searchServicePoint(domain)
• getArea(id) using new field in create unique id on areas & service points for integrate with OpenFN #273
• searchArea(domain)
• correct docstring for old methods
• create docstring with multi examples