[Feature] - Update credential filters to v4 (#5949)

* update cred filters to v4

* ignore v3 key if v4 is there already

* add comment + unit test

* update hub state instead of replacing

* remove unused file

* fix unittests

* ruff

* add comment

* remove lc

* black

* comment

* comment

* comment

* comment

* send warning if v3 keys found

* tradier v3

* keep v3 intact

* better

* unit test + minor fixes

* cleaner

* comment

* sorted

* remove unecessary class

* remove ultima from v3tov4 map

* unit test v3tov4 map

* unit test v3tov4 map

* unit test v3tov4 map

* providers.json

* ^

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

openbb_platform/assets/providers.json

@@ -0,0 +1,58 @@
+[
+ {
+ "name": "Alpha Vantage",
+ "credentials": ["alpha_vantage_api_key"],
+ "v3_credentials": ["API_KEY_ALPHAVANTAGE"],
+ "link": "https://www.alphavantage.co/support/#api-key",
+ "instructions": "Go to: https://www.alphavantage.co/support/#api-key\n\n![AlphaVantage](https://user-images.githubusercontent.com/46355364/207820936-46c2ba00-81ff-4cd3-98a4-4fa44412996f.png)\n\nFill out the form, pass Captcha, and click on, \"GET FREE API KEY\"."
+ },
+ {
+ "name": "BizToc",
+ "credentials": ["biztoc_api_key"],
+ "v3_credentials": ["API_BIZTOC_TOKEN"],
+ "link": "https://biztoc.com",
+ "instructions": "The BizToc API is hosted on RapidAPI. To set up, go to: https://rapidapi.com/thma/api/biztoc.\n\n![biztoc0](https://github.com/marban/OpenBBTerminal/assets/18151143/04cdd423-f65e-4ad8-ad5a-4a59b0f5ddda)\n\nIn the top right, select 'Sign Up'. After answering some questions, you will be prompted to select one of their plans.\n\n![biztoc1](https://github.com/marban/OpenBBTerminal/assets/18151143/9f3b72ea-ded7-48c5-aa33-bec5c0de8422)\n\nAfter signing up, navigate back to https://rapidapi.com/thma/api/biztoc. If you are logged in, you will see a header called X-RapidAPI-Key.\n\n![biztoc2](https://github.com/marban/OpenBBTerminal/assets/18151143/0f3b6c91-07e0-447a-90cd-a9e23522929f)"
+ },
+ {
+ "name": "FRED",
+ "credentials": ["fred_api_key"],
+ "v3_credentials": ["API_FRED_KEY"],
+ "link": "https://fred.stlouisfed.org/docs/api/api_key.html",
+ "instructions": "Go to: https://fred.stlouisfed.org\n\n![FRED](https://user-images.githubusercontent.com/46355364/207827137-d143ba4c-72cb-467d-a7f4-5cc27c597aec.png)\n\nClick on, \"My Account\", create a new account or sign in with Google:\n\n![FRED](https://user-images.githubusercontent.com/46355364/207827011-65cdd501-27e3-436f-bd9d-b0d8381d46a7.png)\n\nAfter completing the sign-up, go to \"My Account\", and select \"API Keys\". Then, click on, \"Request API Key\".\n\n![FRED](https://user-images.githubusercontent.com/46355364/207827577-c869f989-4ef4-4949-ab57-6f3931f2ae9d.png)\n\nFill in the box for information about the use-case for FRED, and by clicking, \"Request API key\", at the bottom of the page, the API key will be issued.\n\n![FRED](https://user-images.githubusercontent.com/46355364/207828032-0a32d3b8-1378-4db2-9064-aa1eb2111632.png)"
+ },
+ {
+ "name": "Financial Modeling Prep",
+ "credentials": ["fmp_api_key"],
+ "v3_credentials": ["API_KEY_FINANCIALMODELINGPREP"],
+ "link": "https://financialmodelingprep.com/developer",
+ "instructions": "Go to: https://site.financialmodelingprep.com/developer/docs\n\n![FinancialModelingPrep](https://user-images.githubusercontent.com/46355364/207821920-64553d05-d461-4984-b0fe-be0368c71186.png)\n\nClick on, \"Get my API KEY here\", and sign up for a free account.\n\n![FinancialModelingPrep](https://user-images.githubusercontent.com/46355364/207822184-a723092e-ef42-4f87-8c55-db150f09741b.png)\n\nWith an account created, sign in and navigate to the Dashboard, which shows the assigned token. by pressing the \"Dashboard\" button which will show the API key.\n\n![FinancialModelingPrep](https://user-images.githubusercontent.com/46355364/207823170-dd8191db-e125-44e5-b4f3-2df0e115c91d.png)"
+ },
+ {
+ "name": "Intrinio",
+ "credentials": ["intrinio_api_key"],
+ "v3_credentials": ["API_INTRINIO_KEY"],
+ "link": "https://intrinio.com/",
+ "instructions": "Go to: https://intrinio.com/starter-plan\n\n![Intrinio](https://user-images.githubusercontent.com/85772166/219207556-fcfee614-59f1-46ae-bff4-c63dd2f6991d.png)\n\nAn API key will be issued with a subscription. Find the token value within the account dashboard."
+ },
+ {
+ "name": "Polygon",
+ "credentials": ["polygon_api_key"],
+ "v3_credentials": ["API_POLYGON_KEY"],
+ "link": "https://polygon.io",
+ "instructions": "Go to: https://polygon.io\n\n![Polygon](https://user-images.githubusercontent.com/46355364/207825623-fcd7f0a3-131a-4294-808c-754c13e38e2a.png)\n\nClick on, \"Get your Free API Key\".\n\n![Polygon](https://user-images.githubusercontent.com/46355364/207825952-ca5540ec-6ed2-4cef-a0ed-bb50b813932c.png)\n\nAfter signing up, the API Key is found at the bottom of the account dashboard page.\n\n![Polygon](https://user-images.githubusercontent.com/46355364/207826258-b1f318fa-fd9c-41d9-bf5c-fe16722e6601.png)"
+ },
+ {
+ "name": "Nasdaq",
+ "credentials": ["nasdaq_api_key"],
+ "v3_credentials": ["API_KEY_QUANDL"],
+ "link": "https://www.quandl.com/tools/api",
+ "instructions": "Go to: https://www.quandl.com\n\n![Quandl](https://user-images.githubusercontent.com/46355364/207823899-208a3952-f557-4b73-aee6-64ac00faedb7.png)\n\nClick on, \"Sign Up\", and register a new account.\n\n![Quandl](https://user-images.githubusercontent.com/46355364/207824214-4b6b2b74-e709-4ed4-adf2-14803e6f3568.png)\n\nFollow the sign-up instructions, and upon completion the API key will be assigned.\n\n![Quandl](https://user-images.githubusercontent.com/46355364/207824664-3c82befb-9c69-42df-8a82-510d85c19a97.png)"
+ },
+ {
+ "name": "Tradier",
+ "credentials": ["tradier_api_key"],
+ "v3_credentials": ["API_TRADIER_TOKEN"],
+ "link": "https://tradier.com/products/market-data-api",
+ "instructions": "Go to: https://documentation.tradier.com\n\n![Tradier](https://user-images.githubusercontent.com/46355364/207829178-a8bba770-f2ea-4480-b28e-efd81cf30980.png)\n\nClick on, \"Open Account\", to start the sign-up process. After the account has been setup, navigate to [Tradier Broker Dash](https://dash.tradier.com/login?redirect=settings.api) and create the application. Request a sandbox access token."
+ }
+]

openbb_platform/core/openbb_core/app/model/hub/hub_user_settings.py

@@ -1,16 +1,15 @@
 """Hub user settings model."""
 
-# from typing import Dict, List, Union
+from typing import Dict, Optional
 
-from openbb_core.app.model.hub.features_keys import FeaturesKeys
 from pydantic import BaseModel, ConfigDict, Field
 
 
 class HubUserSettings(BaseModel):
- """User settings for the Hub."""
+ """Hub user settings model."""
 
  # features_settings: Dict[str, str]
- features_keys: FeaturesKeys = Field(default_factory=FeaturesKeys)
+ features_keys: Dict[str, Optional[str]] = Field(default_factory=dict)
  # features_sources: Dict[str, List[str]]
  # features_terminal_style: Dict[str, Union[str, Dict[str, str]]]
 

openbb_platform/core/openbb_core/app/service/hub_service.py

@@ -1,27 +1,38 @@
 """Hub manager class."""
 
 from typing import Optional
+from warnings import warn
 
 from fastapi import HTTPException
 from jose import JWTError
 from jose.exceptions import ExpiredSignatureError
 from jose.jwt import decode, get_unverified_header
 from openbb_core.app.model.abstract.error import OpenBBError
 from openbb_core.app.model.credentials import Credentials
-from openbb_core.app.model.hub.features_keys import FeaturesKeys
 from openbb_core.app.model.hub.hub_session import HubSession
 from openbb_core.app.model.hub.hub_user_settings import HubUserSettings
 from openbb_core.app.model.profile import Profile
 from openbb_core.app.model.user_settings import UserSettings
 from openbb_core.env import Env
-from pydantic import SecretStr
 from requests import get, post, put
 
 
 class HubService:
  """Hub service class."""
 
  TIMEOUT = 10
+ # Mapping of V3 keys to V4 keys for backward compatibility
+ V3TOV4 = {
+ "API_KEY_ALPHAVANTAGE": "alpha_vantage_api_key",
+ "API_BIZTOC_TOKEN": "biztoc_api_key",
+ "API_FRED_KEY": "fred_api_key",
+ "API_KEY_FINANCIALMODELINGPREP": "fmp_api_key",
+ "API_INTRINIO_KEY": "intrinio_api_key",
+ "API_POLYGON_KEY": "polygon_api_key",
+ "API_KEY_QUANDL": "nasdaq_api_key",
+ "API_TRADIER_TOKEN": "tradier_api_key",
+ }
+ V4TOV3 = {v: k for k, v in V3TOV4.items()}
 
  def __init__(
  self,
@@ -31,6 +42,7 @@ def __init__(
  """Initialize Hub service."""
  self._base_url = base_url or Env().HUB_BACKEND
  self._session = session
+ self._hub_user_settings: Optional[HubUserSettings] = None
 
  @property
  def base_url(self) -> str:
@@ -81,13 +93,10 @@ def push(self, user_settings: UserSettings) -> bool:
  def pull(self) -> UserSettings:
  """Pull user settings from Hub."""
  if self._session:
- hub_user_settings = self._get_user_settings(self._session)
- if hub_user_settings:
- profile = Profile(
- hub_session=self._session,
- )
- credentials = self.hub2platform(hub_user_settings)
- return UserSettings(profile=profile, credentials=credentials)
+ self._hub_user_settings = self._get_user_settings(self._session)
+ profile = Profile(hub_session=self._session)
+ credentials = self.hub2platform(self._hub_user_settings)
+ return UserSettings(profile=profile, credentials=credentials)
  raise OpenBBError(
  "No session found. Login or provide a 'HubSession' on initialization."
  )
@@ -216,42 +225,38 @@ def _put_user_settings(
  detail = response.json().get("detail", None)
  raise HTTPException(status_code, detail)
 
- @classmethod
- def hub2platform(cls, settings: HubUserSettings) -> Credentials:
+ def hub2platform(self, settings: HubUserSettings) -> Credentials:
  """Convert Hub user settings to Platform models."""
- # TODO: Hub is getting credentials from `all_api_keys.json`, which uses
- # the terminal names conventions. We need to update it to use a file that
- # lives here in the platform and maps the credential names between the two.
- credentials = Credentials(
- alpha_vantage_api_key=settings.features_keys.API_KEY_ALPHAVANTAGE,
- biztoc_api_key=settings.features_keys.API_BIZTOC_TOKEN,
- fred_api_key=settings.features_keys.API_FRED_KEY,
- fmp_api_key=settings.features_keys.API_KEY_FINANCIALMODELINGPREP,
- intrinio_api_key=settings.features_keys.API_INTRINIO_KEY,
- polygon_api_key=settings.features_keys.API_POLYGON_KEY,
- nasdaq_api_key=settings.features_keys.API_KEY_QUANDL,
- )
- return credentials
-
- @classmethod
- def platform2hub(cls, credentials: Credentials) -> HubUserSettings:
+ if any(k in settings.features_keys for k in self.V3TOV4):
+ deprecated = {
+ k: v for k, v in self.V3TOV4.items() if k in settings.features_keys
+ }
+ msg = ""
+ for k, v in deprecated.items():
+ msg += f"\n'{k}' -> '{v}', "
+ msg = msg.strip(", ")
+ warn(
+ message=f"\nDeprecated v3 credentials found.\n{msg}"
+ "\n\nYou can update them at https://my.openbb.co/app/platform/credentials.",
+ )
+ # We give priority to v4 keys over v3 keys if both are present
+ hub_credentials = {
+ self.V3TOV4.get(k, k): settings.features_keys.get(self.V3TOV4.get(k, k), v)
+ for k, v in settings.features_keys.items()
+ }
+ return Credentials(**hub_credentials)
+
+ def platform2hub(self, credentials: Credentials) -> HubUserSettings:
  """Convert Platform models to Hub user settings."""
-
- def get_cred(cred: str) -> Optional[str]:
- secret_str: Optional[SecretStr] = getattr(credentials, cred, None)
- return secret_str.get_secret_value() if secret_str else None
-
- features_keys = FeaturesKeys(
- API_BIZTOC_TOKEN=get_cred("biztoc_api_key"),
- API_FRED_KEY=get_cred("fred_api_key"),
- API_INTRINIO_KEY=get_cred("intrinio_api_key"),
- API_KEY_ALPHAVANTAGE=get_cred("alpha_vantage_api_key"),
- API_KEY_FINANCIALMODELINGPREP=get_cred("fmp_api_key"),
- API_POLYGON_KEY=get_cred("polygon_api_key"),
- API_KEY_QUANDL=get_cred("nasdaq_api_key"),
- )
- hub_user_settings = HubUserSettings(features_keys=features_keys)
- return hub_user_settings
+ # Dump mode json ensures SecretStr values are serialized as strings
+ credentials = credentials.model_dump(mode="json", exclude_none=True)
+ settings = self._hub_user_settings or HubUserSettings()
+ for v4_k, v in sorted(credentials.items()):
+ v3_k = self.V4TOV3.get(v4_k, None)
+ # If v3 key was there, we keep it
+ k = v3_k if v3_k in settings.features_keys else v4_k
+ settings.features_keys[k] = v
+ return settings
 
  @staticmethod
  def check_token_expiration(token: str) -> None:

openbb_platform/core/openbb_core/app/static/account.py

@@ -34,6 +34,7 @@ def __init__(self, base_app: "BaseApp"):
  self._openbb_directory = (
  base_app._command_runner.system_settings.openbb_directory
  )
+ self._hub_service: Optional[HubService] = None
 
  def __repr__(self) -> str:
  """Human readable representation of the object."""
@@ -120,18 +121,20 @@ def login(
  Optional[UserSettings]
  User settings: profile, credentials, preferences
  """
- hs = self._create_hub_service(email, password, pat)
- incoming = hs.pull()
+ self._hub_service = self._create_hub_service(email, password, pat)
+ incoming = self._hub_service.pull()
  updated: UserSettings = UserService.update_default(incoming)
  self._base_app._command_runner.user_settings = updated
  if remember_me:
  Path(self._openbb_directory).mkdir(parents=False, exist_ok=True)
  session_file = Path(self._openbb_directory, self.SESSION_FILE)
  with open(session_file, "w") as f:
- if not hs.session:
+ if not self._hub_service.session:
  raise OpenBBError("Not connected to hub.")
 
- json.dump(hs.session.model_dump(mode="json"), f, indent=4)
+ json.dump(
+ self._hub_service.session.model_dump(mode="json"), f, indent=4
+ )
 
  if return_settings:
  return self._base_app._command_runner.user_settings
@@ -151,14 +154,12 @@ def save(self, return_settings: bool = False) -> Optional[UserSettings]:
  Optional[UserSettings]
  User settings: profile, credentials, preferences
  """
- hub_session = self._base_app._command_runner.user_settings.profile.hub_session
- if not hub_session:
+ if not self._hub_service:
  UserService.write_default_user_settings(
  self._base_app._command_runner.user_settings
  )
  else:
- hs = HubService(hub_session)
- hs.push(self._base_app._command_runner.user_settings)
+ self._hub_service.push(self._base_app._command_runner.user_settings)
 
  if return_settings:
  return self._base_app._command_runner.user_settings
@@ -178,14 +179,12 @@ def refresh(self, return_settings: bool = False) -> Optional[UserSettings]:
  Optional[UserSettings]
  User settings: profile, credentials, preferences
  """
- hub_session = self._base_app._command_runner.user_settings.profile.hub_session
- if not hub_session:
+ if not self._hub_service:
  self._base_app._command_runner.user_settings = (
  UserService.read_default_user_settings()
  )
  else:
- hs = HubService(hub_session)
- incoming = hs.pull()
+ incoming = self._hub_service.pull()
  updated: UserSettings = UserService.update_default(incoming)
  updated.id = self._base_app._command_runner.user_settings.id
  self._base_app._command_runner.user_settings = updated
@@ -208,12 +207,10 @@ def logout(self, return_settings: bool = False) -> Optional[UserSettings]:
  Optional[UserSettings]
  User settings: profile, credentials, preferences
  """
- hub_session = self._base_app._command_runner.user_settings.profile.hub_session
- if not hub_session:
+ if not self._hub_service:
  raise OpenBBError("Not connected to hub.")
 
- hs = HubService(hub_session)
- hs.disconnect()
+ self._hub_service.disconnect()
 
  session_file = Path(self._openbb_directory, self.SESSION_FILE)
  if session_file.exists():

openbb_platform/core/openbb_core/app/static/app_factory.py

@@ -62,10 +62,7 @@ def reference(self) -> Dict[str, Dict]:
 def create_app(extensions: Optional[E] = None) -> Type[BaseApp]:
  """Create the app."""
 
- class EmptyClass:
- pass
-
- class App(BaseApp, extensions or EmptyClass): # type: ignore
+ class App(BaseApp, extensions or object): # type: ignore[misc]
  def __repr__(self) -> str:
  # pylint: disable=E1101
  ext_doc = extensions.__doc__ if extensions else ""