Improve IncrementalReleaseFilter and Aggregation Run Finding

Integrations/python/deephaven/doc/io/deephaven/engine/rowset/impl/rsp/RspArray.json

@@ -80,6 +80,6 @@
   "valuesToString": ":return: java.lang.String"
  },
  "path": "io.deephaven.engine.rowset.impl.rsp.RspArray",
- "text": "A set representation for long values using Regular Space Partitioning (RSP) of the long space in \"blocks\" of (2^16)\n elements.\n \n\n Modeled heavily after roaringbitmap.RoaringArray (keeping API method names and semantics as much as possible), with\n modifications for:\n \n\n* Full \"unsigned long\" 64 bit range (as opposed to 32 bit in RoaringArray).\n* Spans of all bits set (\"AllSet\") that can be arbitrarily big (ie, not constrained to 2^16 = RB Container\n size).\n\n\n The handling of unsigned values follows RB; ie, key values are compared/sorted as unsigned longs.\n \n\n Definitions:\n \n\n* A \"block\" is a particular interval [n*2^16, (n+1)*2^16 - 1] of the long domain.\n* A \"span\" is a partition of the domain consisting of one or more consecutive blocks; a span is a subset of\n the domain represented by an interval [n*2^16, (n+m)*2^16 - 1], m >= 1.\n * Full blocks are blocks whose domain are fully contained in the set, ie, the set contains every possible value in\n the block's interval (as a bitmap, it would be \"all ones\").\n* Spans of full blocks are represented by a single \"full blocks span\" object (just a Long) which knows how many\n 2^16 ranges it has (it's \"full blocks span len\" (\"flen\") is the number of full blocks in the span).\n* Individual blocks that are not completely full are stored in an RB Container; their \"full blocks span len\" is\n zero.\n\n\n Our internal representation uses two parallel arrays:\n \n\n* a long[] spanInfos array that contains the information for the offset to the values in the span,\n which we call the span's \"key\". For instance, a full block span that represents all the long values in [65536,\n 196607] has as its key the value 65536.\n* an Object[] spans array that contains the actual spans. At the most basic level, a span can be\n either a full block span or a container of values (but there is nuance in exactly how to represent them, see\n below).\n\n\n We use several optimizations to reduce memory utilization for sparse sets. Details follow.\n \n\n The long[] spanInfos and Object[] spans data members of this class are used, combined, to\n represent the offset (key) and span values in the set, against that offset. The two arrays are used, together, as\n parallel arrays and the information for a given conceptual span is contained in both of them for the same\n corresponding rowSet i.\n \n\n There are two basic cases for a span: it is either a full blocks span, containing a >=1 number of full blocks, or it\n is a container, containing individual values in the particular 2^16 block corresponding to the span's key.\n \n\n There are four ways total that these two cases can be represented between the long in the `spanInfos` array and the\n Object in the `spans` array. Given a span at position `i`:\n \n\n* If the corresponding Object spans[i] is of type Long, then the long spanInfos[i] value\n is the key for the span (with its lower 16 bits as zero), and the Long value represents how many full blocks are\n present. Example, the set [ 0, 2^50 - 1 ] is represented as spanInfo==0 and span==Long(2^34).\n* As an optimization to conserve memory, if the Object spans[i] is the Object reference with value\n FULL_BLOCK_SPAN_MARKER (a singleton and final marker Object defined statically in this file). then the\n upper 48 bits of the long spanInfo[i] value represent the key for the span, and the lower 16 bits of the\n long spanInfo[i] value represent the full block span length. Example, the set [ 65536, 196607 ] is\n represented by spanInfo==65538 and span==FULL_BLOCK_SPAN_MARKER (note\n 196607 == 65536*3 - 1, so the set is 2 full blocks, and 65538 == 65536 | 2.\n* If the corresponding Object spans[i] is null, then the long spanInfos[i] represents the\n single value present in the span (note in this case, its upper 16 bits still corresponds to its key). Example, the\n set { 65537 } is represented by spanInfo==65537 and span==null.\n* If the corresponding Object spans[i] is of type short[] or of type\n Container, then it represents a container of multiple values in a single block (but not all of the\n possible values in the block, since in that case it would be a full block span as above). In this case the higher 48\n bits of its corresponding spanInfo represent the key for the span. Depending on the actual type of span there are two\n subcases:\n\n \n* If spans[i] is of type Container, then the values in the roaringbitmaps container\n object are part of the set, considered against its key offset. The key is represented in the higher 48 bits of its\n corresponding spaninfo. The lower 16 bits of spanInfo are zero in this case. Example, the set [ 100,000-100,010,\n 100,020-100,030 ] is represented by spaInfo==65536,\n span==RunContainer({34464-34474, 34484-34494})\n* If spans[i] is of type short[], then an ArrayContainer with the\n short[] contents needs to be reconstructed. The lower 16 bits of the spanInfo value are used to\n represent the other data members of ArrayContainer. This case exists as an optimization to reduce memory utilization\n for sparse blocks. For details of this reconstruction please see the code for the definition of the SpanView class\n below.\n\n\n\n\n Notes:\n \n\n* Our version of RB Container supports a \"shared\" boolean flag that is used to implement copy-on-write (COW)\n semantics and allow operation results to share containers in COW fashion.\n* We extended the Container class hierarchy to include specializations for empty, single value, single range, and\n two values containers. These are immutable; empty is used only as a way to return empty results, and are never actual\n stored in the spans array. For details, please see the Container class definition and derived class hierarchy.",
+ "text": "A set representation for long values using Regular Space Partitioning (RSP) of the long space in \"blocks\" of (2^16)\n elements.\n \n\n Modeled heavily after roaringbitmap.RoaringArray (keeping API method names and semantics as much as possible), with\n modifications for:\n \n\n* Full \"unsigned long\" 64 bit range (as opposed to 32 bit in RoaringArray).\n* Spans of all bits set (\"AllSet\") that can be arbitrarily big (ie, not constrained to 2^16 = RB Container\n size).\n\n\n The handling of unsigned values follows RB; ie, key values are compared/sorted as unsigned longs.\n \n\n Definitions:\n \n\n* A \"block\" is a particular interval [n*2^16, (n+1)*2^16 - 1] of the long domain.\n* A \"span\" is a partition of the domain consisting of one or more consecutive blocks; a span is a subset of\n the domain represented by an interval [n*2^16, (n+m)*2^16 - 1], m >= 1.\n * Full blocks are blocks whose domain are fully contained in the set, ie, the set contains every possible value in\n the block's interval (as a bitmap, it would be \"all ones\").\n* Spans of full blocks are represented by a single \"full blocks span\" object (just a Long) which knows how many\n 2^16 ranges it has (it's \"full blocks span len\" (\"flen\") is the number of full blocks in the span).\n* Individual blocks that are not completely full are stored in an RB Container; their \"full blocks span len\" is\n zero.\n\n\n Our internal representation uses two parallel arrays:\n \n\n* a long[] spanInfos array that contains the information for the offset to the values in the span,\n which we call the span's \"key\". For instance, a full block span that represents all the long values in [65536,\n 196607] has as its key the value 65536.\n* an Object[] spans array that contains the actual spans. At the most basic level, a span can be\n either a full block span or a container of values (but there is nuance in exactly how to represent them, see\n below).\n\n\n We use several optimizations to reduce memory utilization for sparse sets. Details follow.\n \n\n The long[] spanInfos and Object[] spans data members of this class are used, combined, to\n represent the offset (key) and span values in the set, against that offset. The two arrays are used, together, as\n parallel arrays and the information for a given conceptual span is contained in both of them for the same\n corresponding rowSet i.\n \n\n There are two basic cases for a span: it is either a full blocks span, containing a >=1 number of full blocks, or it\n is a container, containing individual values in the particular 2^16 block corresponding to the span's key.\n \n\n There are four ways total that these two cases can be represented between the long in the `spanInfos` array and the\n Object in the `spans` array. Given a span at position `i`:\n \n\n* If the corresponding Object spans[i] is of type Long, then the long spanInfos[i] value\n is the key for the span (with its lower 16 bits as zero), and the Long value represents how many full blocks are\n present. Example, the set [ 0, 2^50 - 1 ] is represented as spanInfo==0 and span==Long(2^34).\n* As an optimization to conserve memory, if the Object spans[i] is the Object reference with value\n FULL_BLOCK_SPAN_MARKER (a singleton and final marker Object defined statically in this file). then the\n upper 48 bits of the long spanInfo[i] value represent the key for the span, and the lower 16 bits of the\n long spanInfo[i] value represent the full block span length. Example, the set [ 65536, 196607 ] is\n represented by spanInfo==65538 and span==FULL_BLOCK_SPAN_MARKER (note\n 196607 == 65536*3 - 1, so the set is 2 full blocks, and 65538 == 65536 | 2.\n* If the corresponding Object spans[i] is null, then the long spanInfos[i] represents the\n single value present in the span (note in this case, its upper 16 bits still corresponds to its key). Example, the\n set { 65537 } is represented by spanInfo==65537 and span==null.\n* If the corresponding Object spans[i] is of type short[] or of type\n Container, then it represents a container of multiple values in a single block (but not all of the\n possible values in the block, since in that case it would be a full block span as above). In this case the higher 48\n bits of its corresponding spanInfo represent the key for the span. Depending on the actual type of span there are two\n subcases:\n\n \n* If spans[i] is of type Container, then the values in the roaringbitmaps container\n object are part of the set, considered against its key offset. The key is represented in the higher 48 bits of its\n corresponding spaninfo. The lower 16 bits of spanInfo are zero in this case. Example, the set [ 100,000-100,010,\n 100,020-100,030 ] is represented by spaInfo==65536,\n span==RunContainer({34464-34474, 34484-34494})\n* If spans[i] is of type short[], then an ArrayContainer with the\n short[] contents needs to be reconstructed. The lower 16 bits of the spanInfo value are used to\n represent the other data members of ArrayContainer. This case exists as an optimization to reduce memory utilization\n for sparse blocks. For details of this reconstruction please see the code for the definition of the SpanView class\n below.\n\n\n\n\n Notes:\n \n\n* Our version of RB Container supports a \"shared\" boolean flag that is used to implement copy-on-write (COW)\n semantics and allow operation results to share containers in COW fashion.\n* We extended the Container class hierarchy to include specializations for empty, single value, single range, and\n two values containers. These are immutable; empty and singleton are used only as a way to return empty and singleton\n results, and are never actually stored in the spans array. For details, please see the Container class definition and\n derived class hierarchy.",
  "typeName": "class"
 }

Integrations/python/deephaven/doc/io/deephaven/engine/table/impl/by/AggregationContextFactory.json

@@ -1,7 +1,6 @@
 {
  "className": "io.deephaven.engine.table.impl.by.AggregationContextFactory",
  "methods": {
-  "allowKeyOnlySubstitution": "Should we allow substitution with a KeyOnlyAggregationFactory (e.g. selectDistinct) when there are only\n key columns? Instances whose operators could have side effects or are already KeyOnlyAggregationFactory\n should return false.\n\n:return: (boolean) Whether to allow a KeyOnlyAggregationFactory to be substituted for this when there are only key\n         columns",
   "makeAggregationContext": "Make an AggregationContext for this aggregation.\n\n:param table: (io.deephaven.engine.table.Table) - The source Table to aggregate\n:param groupByColumns: (java.lang.String...) - The key column names\n:return: (io.deephaven.engine.table.impl.by.AggregationContext) A new or safely reusable AggregationContext"
  },
  "path": "io.deephaven.engine.table.impl.by.AggregationContextFactory",

Integrations/python/deephaven/doc/io/deephaven/engine/table/impl/by/FirstOrLastByAggregationFactory.json

@@ -1,7 +1,6 @@
 {
  "className": "io.deephaven.engine.table.impl.by.FirstOrLastByAggregationFactory",
  "methods": {
-  "allowKeyOnlySubstitution": "Should we allow substitution with a KeyOnlyAggregationFactory (e.g. selectDistinct) when there are only\n key columns? Instances whose operators could have side effects or are already KeyOnlyAggregationFactory\n should return false.\n\n:return: (boolean) Whether to allow a KeyOnlyAggregationFactory to be substituted for this when there are only key\n         columns",
   "makeAggregationContext": "Make an AggregationContext for this aggregation.\n\n:param table: (io.deephaven.engine.table.Table) - The source Table to aggregate\n:param groupByColumns: (java.lang.String...) - The key column names\n:return: (io.deephaven.engine.table.impl.by.AggregationContext) A new or safely reusable AggregationContext",
   "toString": ":return: java.lang.String"
  },

Integrations/python/deephaven/doc/io/deephaven/engine/table/impl/by/FormulaAggregationFactory.json

@@ -1,7 +1,6 @@
 {
  "className": "io.deephaven.engine.table.impl.by.FormulaAggregationFactory",
  "methods": {
-  "allowKeyOnlySubstitution": "Should we allow substitution with a KeyOnlyAggregationFactory (e.g. selectDistinct) when there are only\n key columns? Instances whose operators could have side effects or are already KeyOnlyAggregationFactory\n should return false.\n\n:return: (boolean) Whether to allow a KeyOnlyAggregationFactory to be substituted for this when there are only key\n         columns",
   "applyToAllBy": "*Overload 1*  \n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param formula: java.lang.String\n  :param columnParamName: java.lang.String\n  :param groupByColumnNames: java.lang.String...\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 2*  \n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param formula: java.lang.String\n  :param columnParamName: java.lang.String\n  :param groupByColumns: io.deephaven.engine.table.impl.select.SelectColumn[]\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 3*  \n  :param aggregationControl: io.deephaven.engine.table.impl.by.AggregationControl\n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param formula: java.lang.String\n  :param columnParamName: java.lang.String\n  :param groupByColumnNames: java.lang.String...\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 4*  \n  :param aggregationControl: io.deephaven.engine.table.impl.by.AggregationControl\n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param formula: java.lang.String\n  :param columnParamName: java.lang.String\n  :param groupByColumns: io.deephaven.engine.table.impl.select.SelectColumn[]\n  :return: io.deephaven.engine.table.impl.QueryTable",
   "makeAggregationContext": "Make an AggregationContext for this aggregation.\n\n:param inputTable: (io.deephaven.engine.table.Table) - The source Table to aggregate\n:param groupByColumnNames: (java.lang.String...) - The key column names\n:return: (io.deephaven.engine.table.impl.by.AggregationContext) A new or safely reusable AggregationContext",
   "toString": ":return: java.lang.String"

Integrations/python/deephaven/doc/io/deephaven/engine/table/impl/by/FreezeByAggregationFactory.json

@@ -1,7 +1,6 @@
 {
  "className": "io.deephaven.engine.table.impl.by.FreezeByAggregationFactory",
  "methods": {
-  "allowKeyOnlySubstitution": "Should we allow substitution with a KeyOnlyAggregationFactory (e.g. selectDistinct) when there are only\n key columns? Instances whose operators could have side effects or are already KeyOnlyAggregationFactory\n should return false.\n\n:return: (boolean) Whether to allow a KeyOnlyAggregationFactory to be substituted for this when there are only key\n         columns",
   "makeAggregationContext": "Make an AggregationContext for this aggregation.\n\n:param table: (io.deephaven.engine.table.Table) - The source Table to aggregate\n:param groupByColumns: (java.lang.String...) - The key column names\n:return: (io.deephaven.engine.table.impl.by.AggregationContext) A new or safely reusable AggregationContext",
   "toString": ":return: java.lang.String"
  },

Integrations/python/deephaven/doc/io/deephaven/engine/table/impl/by/GroupByAggregationFactory.json

@@ -1,7 +1,6 @@
 {
  "className": "io.deephaven.engine.table.impl.by.GroupByAggregationFactory",
  "methods": {
-  "allowKeyOnlySubstitution": "Should we allow substitution with a KeyOnlyAggregationFactory (e.g. selectDistinct) when there are only\n key columns? Instances whose operators could have side effects or are already KeyOnlyAggregationFactory\n should return false.\n\n:return: (boolean) Whether to allow a KeyOnlyAggregationFactory to be substituted for this when there are only key\n         columns",
   "by": "*Overload 1*  \n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param groupByColumnNames: java.lang.String...\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 2*  \n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param groupByColumns: io.deephaven.engine.table.impl.select.SelectColumn[]\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 3*  \n  :param aggregationControl: io.deephaven.engine.table.impl.by.AggregationControl\n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param groupByColumnNames: java.lang.String...\n  :return: io.deephaven.engine.table.impl.QueryTable\n  \n*Overload 4*  \n  :param aggregationControl: io.deephaven.engine.table.impl.by.AggregationControl\n  :param inputTable: io.deephaven.engine.table.impl.QueryTable\n  :param groupByColumns: io.deephaven.engine.table.impl.select.SelectColumn[]\n  :return: io.deephaven.engine.table.impl.QueryTable",
   "getInstance": ":return: io.deephaven.engine.table.impl.by.AggregationContextFactory",
   "makeAggregationContext": "Make an AggregationContext for this aggregation.\n\n:param inputTable: (io.deephaven.engine.table.Table) - The source Table to aggregate\n:param groupByColumnNames: (java.lang.String...) - The key column names\n:return: (io.deephaven.engine.table.impl.by.AggregationContext) A new or safely reusable AggregationContext",