Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Сlient module #140

Merged
merged 27 commits into from
Jan 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
7d1c22b
Add a client module
pomadchin Aug 20, 2020
c6562af
Add Query from Franklin
pomadchin Aug 20, 2020
7630412
Upd up to the main branch
pomadchin Dec 23, 2020
31a0bf0
Add item and collection creation endpoints
pomadchin Dec 23, 2020
c732bae
Move to sttp client to have some layer of abstraction over the client…
pomadchin Dec 23, 2020
be07703
Add ClientJS
pomadchin Dec 23, 2020
739912d
Formatting
pomadchin Dec 23, 2020
7a275a2
Generate STAC Client specs
pomadchin Dec 24, 2020
ed8fda8
Code cleanup
pomadchin Dec 24, 2020
46d9c57
Configure sbt
pomadchin Dec 24, 2020
0469396
Close sttp client in tests
pomadchin Dec 24, 2020
8821a2f
Tune CircleCI
pomadchin Dec 24, 2020
f060644
Tune SBT
pomadchin Dec 24, 2020
c44cffc
Move StacClient into a shared dir
pomadchin Dec 24, 2020
1d8a5f8
Add more tests
pomadchin Dec 24, 2020
0647db7
Remove log4cats from specs
pomadchin Dec 24, 2020
2733d69
Move BaseSttpStacClient into shared folder
pomadchin Dec 24, 2020
7377a3f
Try to limit JS mem usage
pomadchin Dec 24, 2020
990ad20
Remove unnecessary dep
pomadchin Dec 24, 2020
d587f46
Make JS tests more tiny
pomadchin Dec 24, 2020
fdee543
Give 2g to the JVM
pomadchin Dec 24, 2020
7f6abce
polygons => multipolygons
pomadchin Dec 24, 2020
1d83880
Address review comments
pomadchin Dec 24, 2020
baa5c6c
Simplify and unify client specs
pomadchin Dec 25, 2020
30123e3
Add convenient types
pomadchin Dec 25, 2020
6d2ab01
Consolidate tests
pomadchin Dec 25, 2020
c708009
Rename BasetSttpClient => SttpStacClientF; modify pagination token co…
pomadchin Dec 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@ project/plugins/project/
/project/.sbtboot
/project/.boot/
/project/.ivy/
/.sbtopts

# Molecule
.molecule/
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [Unreleased]
### Added
- Сlient module [#140](https://github.com/azavea/stac4s/pull/140)

### Fixed
- Repaired build.sbt configuration to get sonatype publication to cooperate [#186](https://github.com/azavea/stac4s/pull/186)

Expand Down
102 changes: 65 additions & 37 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ lazy val commonSettings = Seq(
else
git.gitDescribedVersion.value.get
},
scalaVersion := "2.12.11",
scalaVersion := "2.12.12",
cancelable in Global := true,
scalafmtOnCompile := true,
scapegoatVersion in ThisBuild := Versions.ScapegoatVersion,
scapegoatVersion in ThisBuild := Versions.Scapegoat,
scapegoatDisabledInspections := Seq("ObjectNames", "EmptyCaseClass"),
unusedCompileDependenciesFilter -= moduleFilter("com.sksamuel.scapegoat", "scalac-scapegoat-plugin"),
addCompilerPlugin("org.typelevel" %% "kind-projector" % "0.11.2" cross CrossVersion.full),
Expand Down Expand Up @@ -105,17 +105,17 @@ lazy val credentialSettings = Seq(

val coreDependenciesJVM = Seq(
"org.locationtech.jts" % "jts-core" % Versions.Jts,
"org.locationtech.geotrellis" %% "geotrellis-vector" % Versions.GeoTrellisVersion
"org.locationtech.geotrellis" %% "geotrellis-vector" % Versions.GeoTrellis
)

val testingDependenciesJVM = Seq(
"org.locationtech.geotrellis" %% "geotrellis-vector" % Versions.GeoTrellisVersion,
"org.locationtech.geotrellis" %% "geotrellis-vector" % Versions.GeoTrellis,
"org.locationtech.jts" % "jts-core" % Versions.Jts
)

val testRunnerDependenciesJVM = Seq(
"io.circe" %% "circe-testing" % Versions.CirceVersion % Test,
"org.scalatest" %% "scalatest" % Versions.ScalatestVersion % Test,
"io.circe" %% "circe-testing" % Versions.Circe % Test,
"org.scalatest" %% "scalatest" % Versions.Scalatest % Test,
"org.scalatestplus" %% "scalacheck-1-14" % Versions.ScalatestPlusScalacheck % Test
)

Expand All @@ -125,57 +125,56 @@ lazy val root = project
.settings(commonSettings)
.settings(publishSettings)
.settings(noPublishSettings)
.aggregate(coreJS, coreJVM, testingJS, testingJVM, coreTestJS, coreTestJVM)
.aggregate(coreJS, coreJVM, testingJS, testingJVM, coreTestJS, coreTestJVM, clientJS, clientJVM)

lazy val core = crossProject(JSPlatform, JVMPlatform)
.in(file("modules/core"))
.settings(commonSettings)
.settings(publishSettings)
.settings({
libraryDependencies ++= Seq(
"com.beachape" %%% "enumeratum" % Versions.EnumeratumVersion,
"com.beachape" %%% "enumeratum-circe" % Versions.EnumeratumVersion,
"com.chuusai" %%% "shapeless" % Versions.ShapelessVersion,
"eu.timepit" %%% "refined" % Versions.RefinedVersion,
"io.circe" %%% "circe-core" % Versions.CirceVersion,
"io.circe" %%% "circe-generic" % Versions.CirceVersion,
"io.circe" %%% "circe-parser" % Versions.CirceVersion,
"io.circe" %%% "circe-refined" % Versions.CirceVersion,
"org.typelevel" %%% "cats-core" % Versions.CatsVersion,
"org.typelevel" %%% "cats-kernel" % Versions.CatsVersion
"com.beachape" %%% "enumeratum" % Versions.Enumeratum,
"com.beachape" %%% "enumeratum-circe" % Versions.Enumeratum,
"com.chuusai" %%% "shapeless" % Versions.Shapeless,
"eu.timepit" %%% "refined" % Versions.Refined,
"io.circe" %%% "circe-core" % Versions.Circe,
"io.circe" %%% "circe-generic" % Versions.Circe,
"io.circe" %%% "circe-parser" % Versions.Circe,
"io.circe" %%% "circe-refined" % Versions.Circe,
"org.typelevel" %%% "cats-core" % Versions.Cats,
"org.typelevel" %%% "cats-kernel" % Versions.Cats
)
})
.jvmSettings(
libraryDependencies ++= coreDependenciesJVM
)
.jsSettings(
libraryDependencies ++= Seq(
"io.github.cquiroz" %% "scala-java-time" % "2.1.0"
)
)
.jvmSettings(libraryDependencies ++= coreDependenciesJVM)
.jsSettings(libraryDependencies += "io.github.cquiroz" %%% "scala-java-time" % "2.1.0")

lazy val coreJVM = core.jvm
lazy val coreJS = core.js

lazy val testing = (crossProject(JSPlatform, JVMPlatform))
lazy val testing = crossProject(JSPlatform, JVMPlatform)
.in(file("modules/testing"))
.dependsOn(core)
.settings(commonSettings)
.settings(publishSettings)
.settings(
libraryDependencies ++= Seq(
"com.beachape" %%% "enumeratum" % Versions.EnumeratumVersion,
"com.beachape" %%% "enumeratum-scalacheck" % Versions.EnumeratumVersion,
"com.chuusai" %%% "shapeless" % Versions.ShapelessVersion,
"eu.timepit" %%% "refined-scalacheck" % Versions.RefinedVersion,
"eu.timepit" %%% "refined" % Versions.RefinedVersion,
"io.chrisdavenport" %%% "cats-scalacheck" % Versions.ScalacheckCatsVersion,
"io.circe" %%% "circe-core" % Versions.CirceVersion,
"org.scalacheck" %%% "scalacheck" % Versions.ScalacheckVersion,
"org.typelevel" %%% "cats-core" % Versions.CatsVersion
"com.beachape" %%% "enumeratum" % Versions.Enumeratum,
"com.beachape" %%% "enumeratum-scalacheck" % Versions.Enumeratum,
"com.chuusai" %%% "shapeless" % Versions.Shapeless,
"eu.timepit" %%% "refined-scalacheck" % Versions.Refined,
"eu.timepit" %%% "refined" % Versions.Refined,
"io.chrisdavenport" %%% "cats-scalacheck" % Versions.ScalacheckCats,
"io.circe" %%% "circe-core" % Versions.Circe,
"org.scalacheck" %%% "scalacheck" % Versions.Scalacheck,
"org.typelevel" %%% "cats-core" % Versions.Cats
)
)
.jvmSettings(libraryDependencies ++= testingDependenciesJVM)
.jsSettings(
libraryDependencies ++= Seq(
"io.github.cquiroz" %%% "scala-java-time" % "2.1.0" % Test
)
)

lazy val testingJVM = testing.jvm
lazy val testingJS = testing.js
Expand All @@ -187,8 +186,8 @@ lazy val coreTest = crossProject(JSPlatform, JVMPlatform)
.settings(noPublishSettings)
.settings(
libraryDependencies ++= Seq(
"io.circe" %%% "circe-testing" % Versions.CirceVersion % Test,
"org.scalatest" %%% "scalatest" % Versions.ScalatestVersion % Test,
"io.circe" %%% "circe-testing" % Versions.Circe % Test,
"org.scalatest" %%% "scalatest" % Versions.Scalatest % Test,
"org.scalatestplus" %%% "scalacheck-1-14" % Versions.ScalatestPlusScalacheck % Test
)
)
Expand All @@ -200,3 +199,32 @@ lazy val coreTest = crossProject(JSPlatform, JVMPlatform)

lazy val coreTestJVM = coreTest.jvm
lazy val coreTestJS = coreTest.js
lazy val coreTestRef = LocalProject("modules/core-test")

lazy val client = crossProject(JSPlatform, JVMPlatform)
.in(file("modules/client"))
.dependsOn(core, testing % Test)
.settings(commonSettings)
.settings(publishSettings)
.settings(
libraryDependencies ++= Seq(
"io.circe" %%% "circe-core" % Versions.Circe,
"io.circe" %%% "circe-generic" % Versions.Circe,
"io.circe" %%% "circe-refined" % Versions.Circe,
"io.circe" %%% "circe-parser" % Versions.Circe,
"com.chuusai" %%% "shapeless" % Versions.Shapeless,
"eu.timepit" %%% "refined" % Versions.Refined,
"org.typelevel" %%% "cats-core" % Versions.Cats,
"com.softwaremill.sttp.client3" %%% "core" % Versions.Sttp,
"com.softwaremill.sttp.client3" %%% "circe" % Versions.Sttp,
"com.softwaremill.sttp.client3" %%% "json-common" % Versions.Sttp,
"com.softwaremill.sttp.model" %%% "core" % Versions.SttpModel,
"com.softwaremill.sttp.shared" %%% "core" % Versions.SttpShared,
"org.scalatest" %%% "scalatest" % Versions.Scalatest % Test
)
)
.jsSettings(libraryDependencies += "io.github.cquiroz" %%% "scala-java-time" % "2.1.0")
.jvmSettings(libraryDependencies ++= coreDependenciesJVM)

lazy val clientJVM = client.jvm
lazy val clientJS = client.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package com.azavea.stac4s.api.client

import com.azavea.stac4s.Bbox
import com.azavea.stac4s.api.client.utils.ClientCodecs
import com.azavea.stac4s.geometry.Geometry
import com.azavea.stac4s.types.TemporalExtent

import eu.timepit.refined.types.numeric.NonNegInt
import io.circe._
import io.circe.generic.semiauto._
import io.circe.refined._

case class SearchFilters(
bbox: Option[Bbox] = None,
datetime: Option[TemporalExtent] = None,
intersects: Option[Geometry] = None,
collections: List[String] = Nil,
items: List[String] = Nil,
limit: Option[NonNegInt] = None,
query: Map[String, List[Query]] = Map.empty,
next: Option[PaginationToken] = None
)

object SearchFilters extends ClientCodecs {

implicit val searchFilterDecoder: Decoder[SearchFilters] = { c =>
for {
bbox <- c.downField("bbox").as[Option[Bbox]]
datetime <- c.downField("datetime").as[Option[TemporalExtent]]
intersects <- c.downField("intersects").as[Option[Geometry]]
collectionsOption <- c.downField("collections").as[Option[List[String]]]
itemsOption <- c.downField("items").as[Option[List[String]]]
limit <- c.downField("limit").as[Option[NonNegInt]]
query <- c.get[Option[Map[String, List[Query]]]]("query")
paginationToken <- c.get[Option[PaginationToken]]("next")
} yield {
SearchFilters(
bbox,
datetime,
intersects,
collectionsOption.getOrElse(Nil),
itemsOption.getOrElse(Nil),
limit,
query getOrElse Map.empty,
paginationToken
)
}
}

implicit val searchFilterEncoder: Encoder[SearchFilters] = deriveEncoder
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package com.azavea.stac4s.api.client

import cats.MonadError
import sttp.client3.SttpBackend
import sttp.model.Uri

object SttpStacClient {

def apply[F[_]: MonadError[*[_], Throwable]](
client: SttpBackend[F, Any],
baseUri: Uri
): SttpStacClient[F] =
SttpStacClientF.instance[F, SearchFilters](client, baseUri)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package com.azavea.stac4s.api

package object client {
type SttpStacClient[F[_]] = SttpStacClientF.Aux[F, SearchFilters]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
package com.azavea.stac4s.api.client

import com.azavea.stac4s.testing.JsInstances

import sttp.client3.UriContext

class SttpStacClientSpec extends SttpStacClientFSpec with JsInstances {
lazy val client = SttpStacClient(backend, uri"http://localhost:9090")
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package com.azavea.stac4s.api.client

import com.azavea.stac4s.Bbox
import com.azavea.stac4s.api.client.utils.ClientCodecs
import com.azavea.stac4s.types.TemporalExtent

import eu.timepit.refined.types.numeric.NonNegInt
import geotrellis.vector.{io => _, _}
import io.circe._
import io.circe.generic.semiauto._
import io.circe.refined._

case class SearchFilters(
bbox: Option[Bbox] = None,
datetime: Option[TemporalExtent] = None,
intersects: Option[Geometry] = None,
collections: List[String] = Nil,
items: List[String] = Nil,
limit: Option[NonNegInt] = None,
query: Map[String, List[Query]] = Map.empty,
next: Option[PaginationToken] = None
)

object SearchFilters extends ClientCodecs {

implicit val searchFilterDecoder: Decoder[SearchFilters] = { c =>
for {
bbox <- c.downField("bbox").as[Option[Bbox]]
datetime <- c.downField("datetime").as[Option[TemporalExtent]]
intersects <- c.downField("intersects").as[Option[Geometry]]
collectionsOption <- c.downField("collections").as[Option[List[String]]]
itemsOption <- c.downField("items").as[Option[List[String]]]
limit <- c.downField("limit").as[Option[NonNegInt]]
query <- c.get[Option[Map[String, List[Query]]]]("query")
paginationToken <- c.get[Option[PaginationToken]]("next")
} yield {
SearchFilters(
bbox,
datetime,
intersects,
collectionsOption.getOrElse(Nil),
itemsOption.getOrElse(Nil),
limit,
query getOrElse Map.empty,
paginationToken
)
}
}

implicit val searchFilterEncoder: Encoder[SearchFilters] = deriveEncoder
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package com.azavea.stac4s.api.client

import cats.MonadError
import sttp.client3.SttpBackend
import sttp.model.Uri

object SttpStacClient {

def apply[F[_]: MonadError[*[_], Throwable]](
client: SttpBackend[F, Any],
baseUri: Uri
): SttpStacClient[F] =
SttpStacClientF.instance[F, SearchFilters](client, baseUri)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package com.azavea.stac4s.api

package object client {
type SttpStacClient[F[_]] = SttpStacClientF.Aux[F, SearchFilters]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
package com.azavea.stac4s.api.client

import com.azavea.stac4s.testing.JvmInstances

import sttp.client3.UriContext

class SttpStacClientSpec extends SttpStacClientFSpec with JvmInstances {
lazy val client = SttpStacClient(backend, uri"http://localhost:9090")
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package com.azavea.stac4s.api.client

import cats.syntax.either._
import eu.timepit.refined.types.numeric.PosInt
import io.circe
import io.circe.generic.semiauto._
import io.circe.parser.parse
import io.circe.refined._
import io.circe.syntax._
import io.circe.{Decoder, Encoder}

import java.time.Instant
import java.util.Base64

final case class PaginationToken(timestampAtLeast: Instant, serialIdGreaterThan: PosInt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct for Franklin's implementation of pagination, but only for Franklin's -- the particular pagination strategy is left open to implementers. So there are a few things that are a bit weird as a result --

  • client side search filters can't assume that the value in next will be a json PaginationToken, since it will just be an opaque string by the time it reaches the client. This means this is probably wrong for any other STAC API implementation.
  • the thing Franklin actually sends back in this field is a base64 encoding of the pagination token, so decoding it probably won't work client side (not sure if you had a chance to test that)
  • the thing Franklin expects is for the client to have a hold of the opaque token, so it does a base64 decode to get the json out

https://github.com/azavea/franklin/blob/f5be8ddf48661c5bc43cbd22cb7277e961641803/application/src/main/scala/com/azavea/franklin/api/schemas/package.scala#L84-L85

So I think this shouldn't work

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say this is fine; since it is a default implementation (i.e. if smth doesn't work user can always define theirs own STACClient with a redefined search filters: see https://github.com/azavea/stac4s/blob/7f6abceeebea59dfc9a605891c73fc121116610b/modules/client/jvm/src/main/scala/com/azavea/stac4s/api/client/SttpStacClient.scala#L9-L13 - they only need to define their own version of SearchFilters), if smth won't work in the future with other STAC Services I think we should handle it later (?).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened an issue for that #198

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really talking about streaming here though -- I don't think this pagination encoding will work, and the tests included here don't exercise pagination. Since the pagination implementation here isn't exercised and I think won't work, I think it should just be omitted until someone is working on #198

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops forgot to push not tested codecs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jisantuc added it into the issue description #198


/** Circe codecs should encode token into a base64 string
* https://github.com/azavea/franklin/blob/f5be8ddf48661c5bc43cbd22cb7277e961641803/application/src/main/scala/com/azavea/franklin/api/schemas/package.scala#L84-L85
*/
object PaginationToken {
val b64Encoder = Base64.getEncoder
val b64Decoder = Base64.getDecoder

val defaultDecoder: Decoder[PaginationToken] = deriveDecoder
val defaultEncoder: Encoder[PaginationToken] = deriveEncoder

def encPaginationToken(token: PaginationToken): String = b64Encoder.encodeToString(
token.asJson(defaultEncoder).noSpaces.getBytes
)

def decPaginationToken(encoded: String): Either[circe.Error, PaginationToken] = {
val jsonString = new String(b64Decoder.decode(encoded))
for {
js <- parse(jsonString)
decoded <- js.as[PaginationToken](defaultDecoder)
} yield decoded
}

implicit val dec: Decoder[PaginationToken] =
Decoder.decodeString.emap(str => decPaginationToken(str).leftMap(_.getMessage))

implicit val enc: Encoder[PaginationToken] = { encPaginationToken(_).asJson }
}
Loading