TypeDB 3.0.0-alpha-8

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-8

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-8

New Features

• Is constraint

  We implement the is constraint support in queries:

  match
      $x isa company;
      $y isa person;
      $z isa person;
      not { $y is $z; };  # <---
      $r1 isa employment, links ($x, $y);
      $r2 isa employment, links ($x, $z);
      select $x, $y, $z;

• Introduce 3.0 undefine queries
  We introduce undefine queries with the new "targeted" syntax to enable safe and transparent schema concept undefinitions.

If you want to undefine the whole type, you can just say:

undefine 
  person;

If you want to undefine a capability from somewhere, use undefine **capability** from **somewhere**.
It consistently works with owns (person is preserved, only the owns is undefined):

undefine
  owns name from person;

annotations of owns (person owns name is preserved, only @regex(...) (you don't need to specify the regex's argument) is undefined):

undefine
  @regex from person owns name;

and any other capability, even specialisation:

undefine
  as parent from fathership relates father;

Want to undefine multiple concepts in one go? Worry not!

undefine
  person;
  relates employee from employment;
  @regex from name;
  @values from email;

The error messages in all definition queries (define, redefine, and undefine) were enhanced, and the respective query/language BDD tests were introduced in the CI with concept unit tests.

Additional fixes:

• answers of match capability queries like match $x owns $y, match $x relates $y, and $match $x plays $y now include transitive capabilities;
• define lets you write multiple declarations of an undefined type, specifying its kind anywhere, even in the last mention of this type;
• failures in schema queries for schema transactions no longer lead to freezes of the following opened transactions;
• no more crashes on long string attributes deletion with existing has edges;

Bugs Fixed

• Fix outputs from earlier disjunction branches affecting later branches
  Fixes outputs from earlier disjunction branches affecting later branches

• Ensure sorted join executor respects selected outputs

  Before this change, an intersection executor (which is used for any number of sorted iterators, not necessarily a join) would write into the output row all items in the constituent executors, so long as they have a row position assigned. That would happen even if those values were not selected for future steps, which prevented deduplication of answers from being handled correctly.

Code Refactors

• Simplify attribute encoding

  We remove separate prefixes for attribute instances and instead introduce a value type prefix after the attribute type encoding.

  Encoding before:

  [String-Attribute-Instance][<type-id>][<value>][<length>]
  

  After:

  [Attribute-Instance][<type-id>][String][<value>][<length>]
  

  Note: this change breaks backwards compatibility

• Refactor after basic functions
  Refactors the executor to be have more simple, flat instructions than less, complex ones.

Other Improvements

• Carry correct types into type list executor

  In cases where the list of types is dependent on the context (e.g. {$t label T;} or {$t label U;};, the type list executor (role name, explicit label, or kind) should only be producing the types in the list defined by the constraint. That was handled correctly when the type constraint was used as a post-check, but incorrectly during direct execution.

• Update spec.md

• Introduce 3.0 value and as match constraints
  We introduce value match constraint for querying for attribute types of specific value types:

  match
    $lo value long;
  

  We introduce as match constraint for querying for role types specialisation:

  match
    $relation relates $role as parentship:child;
  

  which is equivalent to:

  match
    $relation relates $role;
    $role sub parentship:child;
  

  but is useful for shortening your queries, making it similar to define definitions, and in case of anonymous variables:

  match
    $relation relates $_ as parentship:child;
  

• Optimise comparators

  We optimise the execution of queries containing comparators (>/</==/...) where these can be used to restrict the data search space. This PR sets up general infrastructure for using restrictions when searching the data, but uses it only for optimising ISA lookups - follow up work will optimise HAS operations as well.

• Add type annotation to fix build in circleci

• Update spec: fix list insert, add value pattern, add as! pattern

• Bugfix: do not reuse cartesian product iterators between inputs

  We can't reuse the iterators created from the previous input row, as they would iterate over different tuples. Now, when we take a new input, we reset the cartesian iterators.

• Update spec.md (fix sssnake expresssions)

• Green match BDD redux

  We resolve the remaining flaky failures in match BDDs.

• Update banner.png

• Update many unit & integration tests & enable in CI
  Updates many unit & integration tests & enables them in CI

TypeDB 3.0.0-alpha-7

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-7

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-7

New Features

• Expression support

  We add expression execution to match queries:

  match
      $person_1 isa person, has age $age_1;
      $person_2 isa person, has age $age_2;
      $age_2 == $age_1 + 2;

• Basic streaming functions

  Implement function execution for non-recursive stream functions which return streams only.
  Recursive functions & Non-stream functions will throw unimplemented.

  These can currently only be used from the preamble. Sample:

             with
              fun get_ages($p_arg: person) -> { age }:
              match
                  $p_arg has age $age_return;
              return {$age_return};
  
              match
                  $p isa person;
                  $z in get_ages($p);

• Implement tabling machinery for cyclic functions

  Introduces machinery needed to support finding the fixed-point of cyclic function calls. Cyclic functions now run, and return an incomplete set of results followed by an error. It is possible that the planner chooses a plan.

Bugs Fixed

• Minor query planner bug fixes

  1. Fix the issue where an iterator would attempt to use a variable before assigning to it.
  2. Let role name constraint behave as a post-check.

Code Refactors

• Resolve warnings

Other Improvements

• Update protocol to 3.0.0-alpha-7 release

• 3.0 Rename Ok.Empty to Ok.Done. Add query_type to the Done response

  We add a query_type field to all the Query.InitialRes.Ok protobuf messages to support retrieval of this information for any QueryAnswer on the client side.
  Additionally, the Ok.Empty message was renamed to Ok.Done.

• Fix and add to CI concept BDD tests

  We fix data commit error collection and checks and add concept package BDD tests to CI.

TypeDB 3.0.0-alpha-6

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-6

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-6

New Features

• Partial disjunction support

  We introduce partial support for disjunctions in queries:

  match
      $person isa person;
      { $person has name $_; } or { $person has age $_; };

• Fetch execution

  We implement fetch execution, given an executable pipeline that may contain a Fetch terminal stage.

  Note: we have commented out match-return subqueries, fetching of expressions ({ "val" : $x + 1 }), and fetching of function outputs ("val": mean_salary($person)), as these require function-body evaluation under the hood - this is not yet implemented.

Bugs Fixed

• Fix document answers streaming for fetch
  We fix document answers streaming for fetch in order to complete the first version of fetch queries execution.

Code Refactors

• Function compilation preparation & trivial plan implementation
  Prepares higher level packages for compiling functions.

• Fetch annotation, compilation, and executables

  We implement Fetch annotation, compilation and executable building, rearchitecting the rest of the compiler to allow for tree-shaped nesting of queries (eg. functions or fetch sub-pipelines).

• Fetch annotations framework

  We implement further refactoring, which pull Fetch into Annotations and Executables, without implementing any sub-methods yet.

Other Improvements

• Add database name validation for database creation
  We add validation of names for created databases. Now, all database names should be valid TypeQL identifiers.

• Enable connection BDDs in CI
  We update some of the steps for BDDs to match the updated definitions.
  Additionally, we add connection BDDs to the factory CI to make sure that this piece of the system is safe and stable.

• Refactor compiler

  We refactor the compiler to standardize naming, removing the usage of 'program', and introducing 'executable' as the second stage of compilation.

• Refactor pipeline annotations

  We implement the next step of Fetch implementation, which allows us to embed Annotated pipelines into Fetch sub-queries and into functions. We comment out the code paths related to type inference for functions, since functions are now enhanced to include pipelines.

TypeDB 2.29.1

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.29.1

Server + Console: Distributions for 2.29.1

New Features

• Update docker image to have both x86_64 and arm64 versions
  Publish arm64 docker images. Previously, docker would fall-back to running the x86_64 images, which sometimes resulted in a JVM crash loop.

TypeDB 2.29.0

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.29.0

Server + Console: Distributions for 2.29.0

New Features

• Update TypeDB Console version
  Update TypeDB Console to a version with an updated password-update command which allows specifying the password directly.

Code Refactors

• Cleaner logs when transaction close forcibly terminates reasoner
  We update the reasoner to explicitly check whether the cause of termination was a transaction close or an exception, and only log in the latter case.

Other Improvements

• Update APT maintainer to TypeDB Community

  The maintainer field of our APT package is now TypeDB Community community@typedb.com.

• Fix large queries freezing server with new thread-pool for transaction requests
  We introduce a separate thread-pool for servicing transaction requests. This avoids the case where multiple long running queries can occupy all the threads in the pool, and causing the server to be unable to process other requests. Critically, this allows session close calls to terminate any such transactions.

TypeDB 3.0.0-alpha-5

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-5

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-5

New Features

• Negation

  We add support for negations in match queries:

  match $x isa person, has name $a; not { $a == "Jeff"; };

  Note: unlike in 2.x, TypeDB 3.x supports nesting negations, which leads to a form of "for all" operator:

  match
      $super isa set;
      $sub isa set;
      not { # all elements of $sub are also elements of $super:
          (item: $element, set: $sub) isa set-membership;  # there is no element in $sub...
          not { (item: $element, set: $super) isa set-membership; }; # ... that isn't also an element of $super
      };

• Require implementation

  Implement the 'require' clause:

  match
  ...
  require $x, $y, $z;
  

  Will filter the match output stream to ensure that the variable $x, $y, $z are all non-empty variables (if they are optional).

• Let match stages accept inputs

  We now support match stages in the middle of a pipeline, which enables queries like this:

  insert
      $org isa organisation;
  match
      $p isa person, has age 10;
  insert
      (group: $org, member: $p) isa membership;
  

Other Improvements

• Add console script as assembly test

• Add BDD test for pipeline modifiers
  Add BDD test for pipeline modifiers; Fixes some bugs uncovered in the process.

• Fix failing type-inference tests

• Fetch part I

  We implement the initial architecture and refactoring required for the Fetch implementation, including new IR data structures and translation steps.

• TypeDB 3 - Specification

  Added specification of the TypeDB 3.0 database system behaviour.

TypeDB 3.0.0-alpha-4

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-4

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-4

New Features

• Include console in 3.0 distribution
  Includes console in 3.0 distribution. It can perform database management & run queries.

  Boot up server and console using the same binary, just like TypeDB 2.x:

  ./typedb server
  

  and

  ./typedb console
  

• Introduce reduce stages for pipelines
  Introduce reduce stages for pipelines to enable aggregates and group-aggregates.
  Aggregate operators available: count, sum max, min, mean, median, std

  Calculating mean duration of a site's visitors:

  match
    $page isa page, has url == "typedb.com";
    (page: $page) isa page-visit, has duration $duration;
  reduce $mean_visitor = mean($duration);
  
  // returns [ $mean_visitor ]
  

  Counting number of page visits grouped per website

  match 
    $page isa page;
    (page: $page, visitor: $visitor) isa page-visit;
  reduce $site_visitors = count($visitor) within $page; 
  
  // return [ $page, $site_visitors ] for each page
  

• Add support for ISO timezones; finalise datetime & duration support

Bugs Fixed

• Fix greedy planner: variables are produced immediately
• Fix QueryStreamTransmitter to not lose Continue signals and hang streaming operations
• Fix reboot bugs, update logging

Other Improvements

• Fix datetime-tz encode: send datetime as naive_utc instead of naive_local

• Fix datetimes: encode in seconds instead of millis

• Add query type to the query rows header. Implement connection/database bdd steps
  We update the server to server the updated protocol and return the type of the executed query as a part of the query row stream answer.
  We also fix the server's database manipulation code and implement additional bdd steps to pass the connection/database bdd tests.

• Fix within-transaction query concurrency

  We update the in-transaction query behaviour to result in a predictable and well-defined behaviour. In any applicable transaction, the following rules hold:

  1. Read queries are able to run concurrency and lazily, without limitation
  2. Write queries always execute eagerly, but answers are sent back lazily
  3. Schema queries and write queries interrupt all running read queries and finished write queries that are lazily sending answers

  As a user, we see the following - reads after writes or schema queries are just fine:

  let transaction = transaction(...);
  let writes_iter = transaction.query("insert $x isa person;").await.unwrap().into_rows();
  let reads_iter = transaction.query("match $x isa person;").await.unwrap().into_rows();
  
  // both are readable:
  writes_iter.next().await.unwrap();
  reads_iter.next().await.unwrap();
  

  In the other order, we will get an interrupt:

  let transaction = transaction(...);
  let reads_iter = transaction.query("match $x isa person;").await.unwrap().into_rows();
  let writes_iter = transaction.query("insert $x isa person;").await.unwrap().into_rows();
  
  // only writes are still available
  writes_iter.next().await.unwrap();
  assert!(reads_iter.next().await.is_err());
  

TypeDB 3.0.0-alpha-1

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-1

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-1

TypeDB 3.0 Alpha Announcement

You've been increasingly asking for early access to the new TypeDB written in Rust, and today we're thrilled to announce: we're rolling out 3.0 Alpha releases!

We're incredibly excited with the direction TypeDB and TypeQL have taken in the last months, and hope you will be too. Some top highlights:

⚡️ Multi-stage query pipelines. Expressive Functions in place of Rules. Constraints. Lists. Structs. Many additional data types. It's way too much to pack into an email, so the curious reader can see the finer details in The TypeDB 3.0 Roadmap.
🚀 Server rewritten in Rust. Built for safety and performance, the Rust language shares many of TypeDB's core principles. It's gaining popularity along many fronts, seeing increasing adoption, for example, in the development of the Linux kernel. Expect new performance and scalability profiles.
⚙️ Driver API that gets rid of Sessions and simplifies all query methods down to a simple transaction.query("...") interface. No more Concept API.

Consider TypeDB 3.0 Alpha releases to be a "prototype". Do not use it in production systems yet (please!). The storage layer is subject to change as we optimize and stabilize, so there won't be compatibility. Do experiment and try the new features. Get a feel for the new constraint language. Try creating complex DB workflows with pipelines. And tell us what's missing and bugs you've found.

TypeDB 3.0-alpha releases will be rolling approximately every week, bringing new features incrementally until the 3.0 roadmap is complete. That will then initiate the 3.0-beta phase, where we'll process your bug reports, stabilise and optimize performance, until the first full 3.0 release!

In the meantime, we'll be posting new Driver API snippets in our Discord Chat Server.

We're sure you'll love the new language, server, and drivers as much as we do, and look forward to hearing from everyone in the community.

3.0.0-alpha-1

New features:

1. Query pipelines: You can chain an arbitrary sequence of clauses together into a query pipeline - think of your queries as streams or iterators in your favorite query language.
2. Streamlined delete syntax: Delete syntax is inverted compared to insert or match syntax. Instead of writing delete $x has id $id;, which is confusing as to whether it deletes $x, $id or the ownership between them, you now write: delete $id of $x; to delete the ownership, or delete $x; to delete $x!
3. Cardinality constraints: TypeDB 3.0 ships with cardinality restrictions from the get go -- using the @card(x..y) syntax to restrict how many times an attribute can be owned, role can be played, or role can be related. By default, attribute ownership and relation players are @card(0,1)!
4. Value constraints: With the new @values(...) annotation, you can restrict attribute ownerships to specific values. Similarly, the @range(<lower>..<upper>) can restrict attributes' numerical values to a specific range.
5. New value types: There are several new built-in value types: dates, timezoned-datetime, durations, and fixed-decimal numbers.
6. Error traces: TypeDB 3.0 returns error stack traces, which will show you the cause of your error through all the operations that failed as a result, improving debuggability and readability.

Features from TypeDB 2.x that that will be coming soon:

1. Fetch & aggregates
2. Server configuration and data migration (to help you bring your 2.x data)
3. TypeDB Console

TypeDB 2.28.3

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.28.3

Server + Console: Distributions for 2.28.3

New Features

Bugs Fixed

Code Refactors

Other Improvements

• Add development-mode for local and CI set ups

  We add a new server's config parameter called development-mode.enable. This parameter can be used to separate real "release" application runs from our testing.

  This parameter can influence many different sides of the server (logging settings, networking behavior?.., anything we want).development-mode.enable is an optional parameter, and the absence of it in the config is equivalent to development-mode.enable=false. It is expected to be set to true for all the config.yml files used by local environments, tests, and snapshots.

  We add a new Bazel build flag --//server:config=[development/release] that helps choose the server configuration for all the build jobs, addressing the newly added development-mode.enable option.

  Example of a release configuration build:

  bazel build --//server:config=release //server:assemble-mac-x86_64-zip
  

  Example of a development configuration build:

  bazel build //server:assemble-mac-x86_64-zip
  bazel build --//server:config=development //server:assemble-mac-x86_64-zip
  

• Update diagnostics service to version 1
  We introduce an updated version of diagnostics sent from a TypeDB server.

  1. The updated diagnostics data contains more information about the server resources and details for each separate database. More details can be found in the examples below.
  2. For the JSON reporting, we calculated diffs between the current timestamp and the sinceTimestamp (the previous hour when the data had to be sent: it's updated even if we had errors sending the data for simplicity). For the Prometheus data, we send raw counts as Prometheus calculates diffs based on its queries and expects raw diagnostics from our side.
  3. For the JSON monitoring, we show only the incrementing counters from the start of the server just as for the Prometheus diagnostics data (also available through the monitoring page). This way, the content is different from the reporting data.
  4. The schema and data diagnostics about each specific database are sent only from the primary replica of a deployment at the moment of the diagnostics collection. The connection peak values diagnostics regarding a database are still reported by a non-primary replica if the database exists or there were established transactions within the last hour before the database had been deleted.
  5. If the statistics reporting is turned off in the config, we send a totally safe part of the diagnostics data once to notify the server about the moment when the diagnostics were turned off. No user data is shared in this snapshot. This action is performed only if the server is up for 1 hour (to avoid our CI tests report data), and only if the server has not successfully sent such a snapshot after turning the statistics reporting off the last time. If there is an error in sending this snapshot, the server will try again after a restart (no extra logic here).

TypeDB 2.28.3-rc0

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.28.3-rc0

Server + Console: Distributions for 2.28.3-rc0

New Features

Bugs Fixed

Code Refactors

Other Improvements

• Add development-mode for local and CI set ups

We add a new server's config parameter called development-mode.enable. This parameter can be used to separate real "release" application runs from our testing.

This parameter can influence many different sides of the server (logging settings, networking behavior?.., anything we want).development-mode.enable is an optional parameter, and the absence of it in the config is equivalent to development-mode.enable=false. It is expected to be set to true for all the config.yml files used by local environments, tests, and snapshots.

We add a new Bazel build flag --//server:config=[development/release] that helps choose the server configuration for all the build jobs, addressing the newly added development-mode.enable option.

Example of a release configuration build:

bazel build --//server:config=release //server:assemble-mac-x86_64-zip

Example of a development configuration build:

bazel build //server:assemble-mac-x86_64-zip
bazel build --//server:config=development //server:assemble-mac-x86_64-zip

• We remove deployment-id from the TypeDB config as it makes sense only for Cloud servers.