ETS error when running Vault without mix

[monooso]

Hello,

I'm running into an issue when attempting to copy existing data to encrypted fields on fly.io (it works fine locally).

The error is as follows:

* 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 4.0.1) :ets.lookup(MyApp.Vault.Config, :config)
    lib/cloak/vault.ex:285: Cloak.Vault.read_config/1
    (allegro 0.1.0) lib/cloak/vault.ex:224: MyApp.Vault.encrypt/1
    (allegro 0.1.0) lib/cloak_ecto/type.ex:37: MyApp.Encrypted.Binary.dump/1
    (ecto 3.8.4) lib/ecto/type.ex:938: Ecto.Type.process_dumpers/3
    (ecto 3.8.4) lib/ecto/repo/schema.ex:996: Ecto.Repo.Schema.dump_field!/6
    (ecto 3.8.4) lib/ecto/repo/schema.ex:1009: anonymous fn/6 in Ecto.Repo.Schema.dump_fields!/5
    nofile:1: (file)

I read through the troubleshooting guide, which mentions a similar-sounding issue, but I don't believe it's the same problem; as I understand it, Fly compiles the application during deployment.

For the sake of completeness, this is the data migration code. We call it from a function in MyApp.Releases (mix is not available on Fly).

defmodule MyApp.StudentEncryption do
  @moduledoc "Functions to help with encrypting existing data."

  alias MyAppSchemas.Student
  alias Ecto.Changeset
  alias Ecto.Multi

  @doc """
  Encrypt the student data.
  """
  def encrypt(repo) do
    Multi.new()
    |> Multi.put(:students, repo.all(Student))
    |> Multi.merge(&encrypt_students/1)
    |> repo.transaction()
  end

  defp encrypt_students(%{students: students}) do
    Enum.reduce(students, Multi.new(), fn student, multi ->
      Multi.update(
        multi,
        {:student, student.id},
        Changeset.change(student, %{encrypted_name: student.name})
      )
    end)
  end
end

Do you have any suggestions on how to debug this problem? We're completely stumped.

Thanks,
Stephen

[dsincl12]

@monooso did you find a solution to this? I have the exact same issue :(

[brandonjoyce]

I'm having a similar issue when stopping and restarting my application. I suspect it's because I have my vault not being started early enough in the supervision tree. I'm going to try moving it up and see if that fixes it.

EDIT: That didn't seem to do it. Maybe there's some kind of Application.ensure_started call needed? Or maybe I'm barking up the wrong tree altogether.

[KristerV]

i'm trying to migrate data like:

1. fetch record
2. decrypt content
3. DateTime.to_date()
4. encrypt
5. update

pretty simple usecase, but getting the same error. i understand Vault is a Genserver, but starting it manually doesn't help.

the guides/how_to/encrypt_existing_data.md page suggest to use IEX or a mix task for this so do i understand correctly that unless you're using a custom server (with elixir installed) you can't use this package in production? pretty major oversight, great package otherwise.

Solution (edit)

actually I just figured it out. you start the Vault with MyApp.Vault.start_link() and the ETS table error goes away.

real solution is to either support this officially or document it somewhere.

I've documented the workaround in danielberkompas/cloak_ecto#47.

[danielberkompas]

This happens when you don't wait for the vault process to start before calling functions on it. The load_app function in release.ex doesn't ensure all the processes are actually running.

start_link/0 will block until the process is started, which is why that workaround works.

In this case, the right approach would be to define a start_app function in your release.ex file that ensures the application is started before running any vault functions:

def start_app do
  load_app
  Application.ensure_all_started(:my_app)  
end

Then call start_app before calling any vault functions.

https://hexdocs.pm/phoenix/releases.html#ecto-migrations-and-custom-commands

I've added a helpful error message for this in #126.