diff --git a/docs/changelog.md b/docs/changelog.md index 92339f176d..a926b48983 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -204,6 +204,9 @@ use `query.batch.has-step-mode = none` as replacement for `query.batch-property- In case previous behavior is desired, use `query.batch.properties-mode = required_properties_only` for `query.fast-property = false` or use `query.batch.properties-mode = all_properties` for `query.fast-property = true`. +`label` step now uses pre-fetching strategy by default. Use `query.batch.label-step-mode = none` to disable pre-fetching +optimization for `label` step. + [Batch processing](https://docs.janusgraph.org/operations/batch-processing/) allows JanusGraph to fetch a batch of vertices from the storage backend together instead of requesting each vertex individually which leads to a high number of backend queries. diff --git a/docs/configs/janusgraph-cfg.md b/docs/configs/janusgraph-cfg.md index b3e73edd8a..288e7bb1e1 100644 --- a/docs/configs/janusgraph-cfg.md +++ b/docs/configs/janusgraph-cfg.md @@ -373,6 +373,7 @@ pre-fetched together in the same multi-query. In case the next step is one of the properties access steps with unspecified scope of property keys then this mode behaves same as `all_properties`.
- `required_and_next_properties_or_all` - Prefetch the same properties as with `required_and_next_properties`, but in case the next step is not `values`, `properties,` `valueMap`, `elementMap`, or `propertyMap` then acts like `all_properties`.
- `none` - Skips `has` step batch properties pre-fetch optimization.
| String | required_and_next_properties | MASKABLE | +| query.batch.label-step-mode | Labels pre-fetching mode for `label()` step. Used only when `query.batch.enabled` is `true`.
Supported modes:
- `all` - Pre-fetch labels for all vertices in a batch.
- `none` - Skips vertex labels pre-fetching optimization.
| String | all | MASKABLE | | query.batch.limited | Configure a maximum batch size for queries against the storage backend. This can be used to ensure responsiveness if batches tend to grow very large. The used batch size is equivalent to the barrier size of a preceding `barrier()` step. If a step has no preceding `barrier()`, the default barrier of TinkerPop will be inserted. This option only takes effect if `query.batch.enabled` is `true`. | Boolean | true | MASKABLE | | query.batch.limited-size | Default batch size (barrier() step size) for queries. This size is applied only for cases where `LazyBarrierStrategy` strategy didn't apply `barrier` step and where user didn't apply barrier step either. This option is used only when `query.batch.limited` is `true`. Notice, value `2147483647` is considered to be unlimited. | Integer | 2500 | MASKABLE | | query.batch.properties-mode | Properties pre-fetching mode for `values`, `properties`, `valueMap`, `propertyMap`, `elementMap` steps. Used only when `query.batch.enabled` is `true`.
Supported modes:
- `all_properties` - Pre-fetch all vertex properties on non-singular property access (fetches all vertex properties in a single slice query). On single property access this mode behaves the same as `required_properties_only` mode.
- `required_properties_only` - Pre-fetch necessary vertex properties only (uses a separate slice query per each required property)
- `none` - Skips vertex properties pre-fetching optimization.
| String | required_properties_only | MASKABLE | diff --git a/docs/operations/batch-processing.md b/docs/operations/batch-processing.md index 2eabddb825..29a225f9ab 100644 --- a/docs/operations/batch-processing.md +++ b/docs/operations/batch-processing.md @@ -159,7 +159,7 @@ Batched query processing takes into account two types of steps: 1. Batch compatible step. This is the step which will execute batch requests. Currently, the list of such steps is the next: `out()`, `in()`, `both()`, `inE()`, `outE()`, `bothE()`, `has()`, `values()`, `properties()`, `valueMap()`, - `propertyMap()`, `elementMap()`. + `propertyMap()`, `elementMap()`, `label()`. 2. Parent step. This is a parent step which has local traversals with the same start. Such parent steps also implement the interface `TraversalParent`. There are many such steps, but as for an example those could be: `and(...)`, `or(...)`, `not(...)`, `order().by(...)`, `project("valueA", "valueB", "valueC").by(...).by(...).by(...)`, `union(..., ..., ...)`, @@ -330,3 +330,4 @@ access when direct vertex properties are requested (for example `vertex.properti See configuration option `query.batch.has-step-mode` to control properties pre-fetching behaviour for `has` step. See configuration option `query.batch.properties-mode` to control properties pre-fetching behaviour for `values`, `properties`, `valueMap`, `propertyMap`, and `elementMap` steps. +See configuration option `query.batch.label-step-mode` to control labels pre-fetching behaviour for `label` step. diff --git a/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java b/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java index 682617486b..0c217d1f35 100644 --- a/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java +++ b/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java @@ -139,6 +139,7 @@ import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertiesStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertyMapStep; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode; +import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryLabelStepStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryStrategyRepeatStepMode; import org.janusgraph.graphdb.transaction.StandardJanusGraphTx; @@ -217,6 +218,7 @@ import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.HAS_STEP_BATCH_MODE; import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.IDS_STORE_NAME; import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.INITIAL_JANUSGRAPH_VERSION; +import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.LABEL_STEP_BATCH_MODE; import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.LIMITED_BATCH; import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.LIMITED_BATCH_SIZE; import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.LOG_BACKEND; @@ -5008,6 +5010,7 @@ public void testLimitBatchSizeForMultiQuery() { testLimitBatchSizeForHasStep(numV, barrierSize, limit, bs, cs); testLimitBatchSizeForPropertySteps(numV, barrierSize, limit, cs); + testLimitBatchSizeForLabelStep(numV, barrierSize, limit, bs, cs); // test batching for `out()` profile = testLimitedBatch(() -> graph.traversal().V(bs).barrier(barrierSize).out()); @@ -6166,6 +6169,45 @@ private void testLimitBatchSizeForMultiQueryOfConnectiveSteps(JanusGraphVertex[] assertEquals((int) Math.ceil((double) Math.min(bs.length, limit) / barrierSize), countOptimizationQueries(profile.getMetrics())); } + private void testLimitBatchSizeForLabelStep(int numV, int barrierSize, int limit, JanusGraphVertex[] bs, JanusGraphVertex[] cs) { + + TraversalMetrics profile; + + // test batching for `label()` + profile = testLimitedBatch(() -> graph.traversal().V(cs).barrier(barrierSize).label(), + option(USE_MULTIQUERY), true, option(LIMITED_BATCH), true, + option(LABEL_STEP_BATCH_MODE), MultiQueryLabelStepStrategyMode.ALL.getConfigName()); + assertEquals(3, countBackendQueriesOfSize(barrierSize, profile.getMetrics())); + assertEquals(1, countBackendQueriesOfSize((numV - 3 * barrierSize), profile.getMetrics())); + + // test batching for `label()` with default labels + profile = testLimitedBatch(() -> graph.traversal().V(bs).barrier(barrierSize).label(), + option(USE_MULTIQUERY), true, option(LIMITED_BATCH), true, + option(LABEL_STEP_BATCH_MODE), MultiQueryLabelStepStrategyMode.ALL.getConfigName()); + assertEquals( Math.ceil(bs.length / (double) barrierSize), countOptimizationQueries(profile.getMetrics())); + + // test batching for `label()` in a parent step + profile = testLimitedBatch(() -> graph.traversal().V(cs).barrier(barrierSize).union(__.label()), + option(USE_MULTIQUERY), true, option(LIMITED_BATCH), true, + option(LABEL_STEP_BATCH_MODE), MultiQueryLabelStepStrategyMode.ALL.getConfigName()); + assertEquals(3, countBackendQueriesOfSize(barrierSize, profile.getMetrics())); + assertEquals(1, countBackendQueriesOfSize((numV - 3 * barrierSize), profile.getMetrics())); + + // test batching for `label()` with `limit`. In this case TinkerPop optimizes the usage by moving limit before + // `label` step. + profile = testLimitedBatch(() -> graph.traversal().V(cs).label().limit(limit), + option(USE_MULTIQUERY), true, option(LIMITED_BATCH), true, + option(LABEL_STEP_BATCH_MODE), MultiQueryLabelStepStrategyMode.ALL.getConfigName()); + assertEquals(1, countBackendQueriesOfSize(limit, profile.getMetrics())); + + // test disabled batching for `label()` + profile = testLimitedBatch(() -> graph.traversal().V(cs).barrier(barrierSize).label(), + option(USE_MULTIQUERY), true, option(LIMITED_BATCH), true, + option(LABEL_STEP_BATCH_MODE), MultiQueryLabelStepStrategyMode.NONE.getConfigName()); + assertEquals(0, countBackendQueriesOfSize(barrierSize, profile.getMetrics())); + assertEquals(0, countOptimizationQueries(profile.getMetrics())); + } + @Test public void testMultiSliceDBCachedRequests(){ clopen(option(DB_CACHE), false); diff --git a/janusgraph-benchmark/src/main/java/org/janusgraph/CQLMultiQueryBenchmark.java b/janusgraph-benchmark/src/main/java/org/janusgraph/CQLMultiQueryBenchmark.java index c0a6efc89d..a809aa8592 100644 --- a/janusgraph-benchmark/src/main/java/org/janusgraph/CQLMultiQueryBenchmark.java +++ b/janusgraph-benchmark/src/main/java/org/janusgraph/CQLMultiQueryBenchmark.java @@ -198,4 +198,13 @@ public List getAdjacentVerticesLocalCounts() { tx.rollback(); return result; } + + @Benchmark + public List getLabels() { + JanusGraphTransaction tx = graph.buildTransaction() + .start(); + List result = tx.traversal().V().has("name", "inner").label().toList(); + tx.rollback(); + return result; + } } diff --git a/janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java b/janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java index dba638eb91..697de011f5 100644 --- a/janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java +++ b/janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java @@ -16,6 +16,7 @@ import org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode; +import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryLabelStepStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode; import java.time.Instant; @@ -167,6 +168,15 @@ public interface TransactionBuilder { */ TransactionBuilder setPropertiesStrategyMode(MultiQueryPropertiesStrategyMode propertiesStrategyMode); + /** + * Sets `label` step strategy mode. + *

+ * Doesn't have any effect if multi-query was disabled via config `query.batch.enabled = false`. + * + * @return Object with the set labels strategy mode settings + */ + TransactionBuilder setLabelsStepStrategyMode(MultiQueryLabelStepStrategyMode labelStepStrategyMode); + /** * Sets the group name for this transaction which provides a way for gathering * reporting on multiple transactions into one group. diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java index c18be46205..0d8c408a3e 100644 --- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java @@ -58,6 +58,7 @@ import org.janusgraph.graphdb.query.index.BruteForceIndexSelectionStrategy; import org.janusgraph.graphdb.query.index.IndexSelectionStrategy; import org.janusgraph.graphdb.query.index.ThresholdBasedIndexSelectionStrategy; +import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryLabelStepStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryStrategyRepeatStepMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode; @@ -376,6 +377,15 @@ public class GraphDatabaseConfiguration { MultiQueryPropertiesStrategyMode.NONE.getConfigName()), ConfigOption.Type.MASKABLE, MultiQueryPropertiesStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigName()); + public static final ConfigOption LABEL_STEP_BATCH_MODE = new ConfigOption<>(QUERY_BATCH_NS,"label-step-mode", + String.format("Labels pre-fetching mode for `label()` step. Used only when `"+USE_MULTIQUERY.toStringWithoutRoot()+"` is `true`.
" + + "Supported modes:
" + + "- `%s` - Pre-fetch labels for all vertices in a batch.
" + + "- `%s` - Skips vertex labels pre-fetching optimization.
", + MultiQueryLabelStepStrategyMode.ALL.getConfigName(), + MultiQueryLabelStepStrategyMode.NONE.getConfigName()), + ConfigOption.Type.MASKABLE, MultiQueryLabelStepStrategyMode.ALL.getConfigName()); + // ################ SCHEMA ####################### // ################################################ @@ -1348,6 +1358,7 @@ public boolean apply(@Nullable String s) { private String unknownIndexKeyName; private MultiQueryHasStepStrategyMode hasStepStrategyMode; private MultiQueryPropertiesStrategyMode propertiesStrategyMode; + private MultiQueryLabelStepStrategyMode labelStepStrategyMode; private StoreFeatures storeFeatures = null; @@ -1470,6 +1481,10 @@ public MultiQueryPropertiesStrategyMode propertiesStrategyMode() { return propertiesStrategyMode; } + public MultiQueryLabelStepStrategyMode labelStepStrategyMode() { + return labelStepStrategyMode; + } + public boolean adjustQueryLimit() { return adjustQueryLimit; } @@ -1599,6 +1614,7 @@ private void preLoadConfiguration() { repeatStepMode = selectExactConfig(REPEAT_STEP_BATCH_MODE, MultiQueryStrategyRepeatStepMode.values()); hasStepStrategyMode = selectExactConfig(HAS_STEP_BATCH_MODE, MultiQueryHasStepStrategyMode.values()); propertiesStrategyMode = selectExactConfig(PROPERTIES_BATCH_MODE, MultiQueryPropertiesStrategyMode.values()); + labelStepStrategyMode = selectExactConfig(LABEL_STEP_BATCH_MODE, MultiQueryLabelStepStrategyMode.values()); indexSelectionStrategy = Backend.getImplementationClass(configuration, configuration.get(INDEX_SELECT_STRATEGY), REGISTERED_INDEX_SELECTION_STRATEGIES); diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/step/JanusGraphLabelStep.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/step/JanusGraphLabelStep.java new file mode 100644 index 0000000000..b6f30adc12 --- /dev/null +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/step/JanusGraphLabelStep.java @@ -0,0 +1,114 @@ +// Copyright 2023 JanusGraph Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.janusgraph.graphdb.tinkerpop.optimize.step; + +import org.apache.tinkerpop.gremlin.process.traversal.Traverser; +import org.apache.tinkerpop.gremlin.process.traversal.step.Profiling; +import org.apache.tinkerpop.gremlin.process.traversal.step.map.LabelStep; +import org.apache.tinkerpop.gremlin.process.traversal.util.MutableMetrics; +import org.apache.tinkerpop.gremlin.structure.Element; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.janusgraph.core.BaseVertexQuery; +import org.janusgraph.graphdb.query.profile.QueryProfiler; +import org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder; +import org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryUtil; +import org.janusgraph.graphdb.tinkerpop.optimize.step.fetcher.LabelStepBatchFetcher; +import org.janusgraph.graphdb.tinkerpop.profile.TP3ProfileWrapper; +import org.janusgraph.graphdb.util.CopyStepUtil; +import org.janusgraph.graphdb.util.JanusGraphTraverserUtil; + +/** + * This class extends the default TinkerPop's {@link LabelStep} and adds vertices multi-query optimization to this step. + *

+ * Before this step is evaluated it usually receives multiple future vertices which might be processed next with this step. + * This step stores all these vertices which might be needed later for evaluation and whenever this step receives the + * vertex for evaluation which wasn't preFetched previously it sends multi-query for a batch of vertices to fetch their + * labels. + *

+ * This step optimizes only access to Vertex properties and skips optimization for any other Element. + */ +public class JanusGraphLabelStep extends LabelStep implements Profiling, MultiQueriable { + + private boolean useMultiQuery = false; + private QueryProfiler queryProfiler = QueryProfiler.NO_OP; + private int batchSize = Integer.MAX_VALUE; + private LabelStepBatchFetcher labelStepBatchFetcher; + + public JanusGraphLabelStep(LabelStep originalStep){ + super(originalStep.getTraversal()); + CopyStepUtil.copyAbstractStepModifiableFields(originalStep, this); + + if (originalStep instanceof JanusGraphLabelStep) { + JanusGraphLabelStep originalJanusGraphLabelStep = (JanusGraphLabelStep) originalStep; + setBatchSize(originalJanusGraphLabelStep.batchSize); + setUseMultiQuery(originalJanusGraphLabelStep.useMultiQuery); + } + } + + @Override + protected String map(final Traverser.Admin traverser) { + if (useMultiQuery && traverser.get() instanceof Vertex) { + return labelStepBatchFetcher.fetchData(getTraversal(), (Vertex) traverser.get(), JanusGraphTraverserUtil.getLoops(traverser)); + } + return super.map(traverser); + } + + @Override + public void setUseMultiQuery(boolean useMultiQuery) { + this.useMultiQuery = useMultiQuery; + if(this.useMultiQuery && labelStepBatchFetcher == null){ + labelStepBatchFetcher = new LabelStepBatchFetcher(this::makeLabelsQuery, batchSize); + } + } + + private Q makeLabelsQuery(Q query) { + return (Q) BasicVertexCentricQueryUtil.withLabelVertices((BasicVertexCentricQueryBuilder) query) + .profiler(queryProfiler); + } + + @Override + public void setBatchSize(int batchSize) { + this.batchSize = batchSize; + if(labelStepBatchFetcher != null){ + labelStepBatchFetcher.setBatchSize(batchSize); + } + } + + @Override + public void registerFirstNewLoopFutureVertexForPrefetching(Vertex futureVertex, int futureVertexTraverserLoop) { + if(useMultiQuery){ + labelStepBatchFetcher.registerFirstNewLoopFutureVertexForPrefetching(futureVertex); + } + } + + @Override + public void registerSameLoopFutureVertexForPrefetching(Vertex futureVertex, int futureVertexTraverserLoop) { + if(useMultiQuery){ + labelStepBatchFetcher.registerCurrentLoopFutureVertexForPrefetching(futureVertex, futureVertexTraverserLoop); + } + } + + @Override + public void registerNextLoopFutureVertexForPrefetching(Vertex futureVertex, int futureVertexTraverserLoop) { + if(useMultiQuery){ + labelStepBatchFetcher.registerNextLoopFutureVertexForPrefetching(futureVertex, futureVertexTraverserLoop); + } + } + + @Override + public void setMetrics(MutableMetrics metrics) { + queryProfiler = new TP3ProfileWrapper(metrics); + } +} diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphLocalQueryOptimizerStrategy.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphLocalQueryOptimizerStrategy.java index 3b22522ce8..8e3a56dc03 100644 --- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphLocalQueryOptimizerStrategy.java +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphLocalQueryOptimizerStrategy.java @@ -20,6 +20,7 @@ import org.apache.tinkerpop.gremlin.process.traversal.step.branch.LocalStep; import org.apache.tinkerpop.gremlin.process.traversal.step.filter.RangeGlobalStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.ElementMapStep; +import org.apache.tinkerpop.gremlin.process.traversal.step.map.LabelStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertiesStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertyMapStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.VertexStep; @@ -31,6 +32,7 @@ import org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphTraversalUtil; import org.janusgraph.graphdb.tinkerpop.optimize.step.HasStepFolder; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphElementMapStep; +import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphLabelStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertiesStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertyMapStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphVertexStep; @@ -67,20 +69,24 @@ public void apply(final Traversal.Admin traversal) { final Optional tx = JanusGraphTraversalUtil.getJanusGraphTx(traversal); final MultiQueryPropertiesStrategyMode propertiesStrategyMode; + final MultiQueryLabelStepStrategyMode labelStepStrategyMode; final int txVertexCacheSize; if(tx.isPresent()){ TransactionConfiguration txConfig = tx.get().getConfiguration(); txVertexCacheSize = txConfig.getVertexCacheSize(); propertiesStrategyMode = txConfig.getPropertiesStrategyMode(); + labelStepStrategyMode = txConfig.getLabelStepStrategyMode(); } else { GraphDatabaseConfiguration graphConfig = janusGraph.getConfiguration(); txVertexCacheSize = graphConfig.getTxVertexCacheSize(); propertiesStrategyMode = graphConfig.propertiesStrategyMode(); + labelStepStrategyMode = graphConfig.labelStepStrategyMode(); } applyJanusGraphVertexSteps(traversal); applyJanusGraphPropertiesSteps(traversal, txVertexCacheSize, propertiesStrategyMode); + applyJanusGraphLabelSteps(traversal, labelStepStrategyMode); inspectLocalTraversals(traversal, txVertexCacheSize, propertiesStrategyMode); } @@ -184,6 +190,19 @@ private void inspectLocalTraversals(final Traversal.Admin traversal, int t }); } + private void applyJanusGraphLabelSteps(Traversal.Admin traversal, MultiQueryLabelStepStrategyMode labelStepStrategyMode){ + if(MultiQueryLabelStepStrategyMode.NONE.equals(labelStepStrategyMode)){ + return; + } + TraversalHelper.getStepsOfAssignableClass(LabelStep.class, traversal).forEach(originalStep -> { + if(originalStep instanceof JanusGraphLabelStep){ + return; + } + final JanusGraphLabelStep janusGraphLabelStep = new JanusGraphLabelStep(originalStep); + TraversalHelper.replaceStep(originalStep, janusGraphLabelStep, originalStep.getTraversal()); + }); + } + private static void unfoldLocalTraversal(final Traversal.Admin traversal, LocalStep localStep, Traversal.Admin localTraversal, MultiQueriable vertexStep) { diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/MultiQueryLabelStepStrategyMode.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/MultiQueryLabelStepStrategyMode.java new file mode 100644 index 0000000000..a3a4438a9e --- /dev/null +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/MultiQueryLabelStepStrategyMode.java @@ -0,0 +1,42 @@ +// Copyright 2023 JanusGraph Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.janusgraph.graphdb.tinkerpop.optimize.strategy; + +import org.janusgraph.graphdb.configuration.ConfigName; + +public enum MultiQueryLabelStepStrategyMode implements ConfigName { + + /** + * Prefetch labels for all vertices in a batch. + */ + ALL("all"), + + /** + * Skips `label` step pre-fetch optimization. + */ + NONE("none") + ; + + private final String configurationOptionName; + + MultiQueryLabelStepStrategyMode(String configurationOptionName){ + this.configurationOptionName = configurationOptionName; + } + + @Override + public String getConfigName() { + return configurationOptionName; + } +} diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardTransactionBuilder.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardTransactionBuilder.java index 787c71d103..88fdfb3813 100644 --- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardTransactionBuilder.java +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardTransactionBuilder.java @@ -29,6 +29,7 @@ import org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration; import org.janusgraph.graphdb.database.StandardJanusGraph; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode; +import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryLabelStepStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode; import java.time.Instant; @@ -90,6 +91,7 @@ public class StandardTransactionBuilder implements TransactionConfiguration, Tra private MultiQueryHasStepStrategyMode hasStepStrategyMode; private MultiQueryPropertiesStrategyMode propertiesStrategyMode; + private MultiQueryLabelStepStrategyMode labelStepStrategyMode; private final boolean forceIndexUsage; @@ -115,6 +117,7 @@ private StandardTransactionBuilder(GraphDatabaseConfiguration graphConfig, Stand this.multiQuery = graphConfig.useMultiQuery(); this.hasStepStrategyMode = graphConfig.hasStepStrategyMode(); this.propertiesStrategyMode = graphConfig.propertiesStrategyMode(); + this.labelStepStrategyMode = graphConfig.labelStepStrategyMode(); this.writableCustomOptions = writableCustomOptions; if(customOptions == null){ this.customOptions = new MergedConfiguration(writableCustomOptions, graphConfig.getConfiguration()); @@ -243,6 +246,12 @@ public TransactionBuilder setPropertiesStrategyMode(MultiQueryPropertiesStrategy return this; } + @Override + public TransactionBuilder setLabelsStepStrategyMode(MultiQueryLabelStepStrategyMode labelStepStrategyMode) { + this.labelStepStrategyMode = labelStepStrategyMode; + return this; + } + @Override public void setCommitTime(Instant time) { throw new UnsupportedOperationException("Use setCommitTime(long,TimeUnit)"); @@ -290,7 +299,7 @@ propertyPrefetching, multiQuery, singleThreaded, threadBound, getTimestampProvid indexCacheWeight, getVertexCacheSize(), getDirtyVertexSize(), logIdentifier, restrictedPartitions, groupName, defaultSchemaMaker, hasDisabledSchemaConstraints, skipDBCacheRead, hasStepStrategyMode, propertiesStrategyMode, - customOptions); + labelStepStrategyMode, customOptions); return graph.newTransaction(immutable); } @@ -417,6 +426,11 @@ public MultiQueryPropertiesStrategyMode getPropertiesStrategyMode() { return propertiesStrategyMode; } + @Override + public MultiQueryLabelStepStrategyMode getLabelStepStrategyMode() { + return labelStepStrategyMode; + } + @Override public String getGroupName() { return groupName; @@ -478,6 +492,7 @@ private static class ImmutableTxCfg implements TransactionConfiguration { private boolean hasDisabledSchemaConstraints = true; private MultiQueryHasStepStrategyMode hasStepStrategyMode; private MultiQueryPropertiesStrategyMode propertiesStrategyMode; + private MultiQueryLabelStepStrategyMode labelStepStrategyMode; private final BaseTransactionConfig handleConfig; @@ -499,6 +514,7 @@ public ImmutableTxCfg(boolean isReadOnly, boolean skipDBCacheRead, MultiQueryHasStepStrategyMode hasStepStrategyMode, MultiQueryPropertiesStrategyMode propertiesStrategyMode, + MultiQueryLabelStepStrategyMode labelStepStrategyMode, Configuration customOptions) { this.isReadOnly = isReadOnly; this.hasEnabledBatchLoading = hasEnabledBatchLoading; @@ -523,6 +539,7 @@ public ImmutableTxCfg(boolean isReadOnly, this.skipDBCacheRead = skipDBCacheRead; this.hasStepStrategyMode = hasStepStrategyMode; this.propertiesStrategyMode = propertiesStrategyMode; + this.labelStepStrategyMode = labelStepStrategyMode; this.handleConfig = new StandardBaseTransactionConfig.Builder() .commitTime(commitTime) .timestampProvider(times) @@ -650,6 +667,11 @@ public MultiQueryPropertiesStrategyMode getPropertiesStrategyMode() { return propertiesStrategyMode; } + @Override + public MultiQueryLabelStepStrategyMode getLabelStepStrategyMode() { + return labelStepStrategyMode; + } + @Override public Instant getCommitTime() { return handleConfig.getCommitTime(); diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/TransactionConfiguration.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/TransactionConfiguration.java index 1b582c119d..5959b974ab 100644 --- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/TransactionConfiguration.java +++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/TransactionConfiguration.java @@ -17,6 +17,7 @@ import org.janusgraph.core.schema.DefaultSchemaMaker; import org.janusgraph.diskstorage.BaseTransactionConfig; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode; +import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryLabelStepStrategyMode; import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode; /** @@ -208,4 +209,9 @@ public interface TransactionConfiguration extends BaseTransactionConfig { */ MultiQueryPropertiesStrategyMode getPropertiesStrategyMode(); + /** + * @return Label step strategy mode used for the transaction. Can be configured via config `query.batch.label-step-mode`. + */ + MultiQueryLabelStepStrategyMode getLabelStepStrategyMode(); + } diff --git a/janusgraph-test/src/test/java/org/janusgraph/graphdb/tinkerpop/optimize/JanusGraphStepStrategyTest.java b/janusgraph-test/src/test/java/org/janusgraph/graphdb/tinkerpop/optimize/JanusGraphStepStrategyTest.java index 213b244498..8316af3fdd 100644 --- a/janusgraph-test/src/test/java/org/janusgraph/graphdb/tinkerpop/optimize/JanusGraphStepStrategyTest.java +++ b/janusgraph-test/src/test/java/org/janusgraph/graphdb/tinkerpop/optimize/JanusGraphStepStrategyTest.java @@ -30,6 +30,7 @@ import org.apache.tinkerpop.gremlin.process.traversal.step.filter.RangeGlobalStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.ElementMapStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep; +import org.apache.tinkerpop.gremlin.process.traversal.step.map.LabelStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertiesStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertyMapStep; import org.apache.tinkerpop.gremlin.process.traversal.step.map.VertexStep; @@ -56,6 +57,7 @@ import org.janusgraph.graphdb.tinkerpop.optimize.step.HasStepFolder; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphElementMapStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphHasStep; +import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphLabelStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphMultiQueryStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertiesStep; import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphPropertyMapStep; @@ -187,6 +189,10 @@ private void applyMultiQueryTraversalSteps(Traversal.Admin traversal) { JanusGraphElementMapStep janusGraphElementMapStep = new JanusGraphElementMapStep<>(elementMapStep, true, true); TraversalHelper.replaceStep(elementMapStep, janusGraphElementMapStep, elementMapStep.getTraversal()); }); + TraversalHelper.getStepsOfAssignableClassRecursively(LabelStep.class, traversal).forEach(labelStep -> { + JanusGraphLabelStep janusGraphLabelStep = new JanusGraphLabelStep<>(labelStep); + TraversalHelper.replaceStep(labelStep, janusGraphLabelStep, labelStep.getTraversal()); + }); } @SuppressWarnings({ "unchecked", "rawtypes" }) @@ -465,6 +471,9 @@ private static Stream generateMultiQueryTestParameters() { // Need `JanusGraphMultiQuerySteps` before `properties` step which is used after other steps arguments(g.V().properties("weight"), g_V().is(MQ_STEP).barrier(defaultBarrierSize).properties("weight"), otherStrategies), + // Need `JanusGraphLabelStep` to use instead of `LabelStep` + arguments(g.V().label(), + g_V().is(MQ_STEP).barrier(defaultBarrierSize).label(), otherStrategies), }); } }