Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Near cache #180

Draft
wants to merge 87 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
66e6c03
POC initial commit
dhirupandey Jun 14, 2024
8085d1d
Changes to get Get request to work
dhirupandey Jun 18, 2024
14e1a5e
Add another implementation that uses async handling of responses
dhirupandey Jun 26, 2024
50563ae
How to check if server is running v1gRPC version
dhirupandey Jul 9, 2024
51636ae
Basic stuff now working
dhirupandey Jul 24, 2024
2d51742
Added support for clear, destroy, truncate
dhirupandey Jul 25, 2024
a84b3a7
Reorganized code and added support for get_or_default, put_if_absent
dhirupandey Jul 26, 2024
09c8af0
Added support for getAll
dhirupandey Jul 30, 2024
a903ff2
Added support for putAll
dhirupandey Jul 31, 2024
980d9cc
Cleanup and added support for remove,replace and remove_mapping
dhirupandey Aug 1, 2024
103057d
Added support for replace_mapping, contains_key, contains_value and i…
dhirupandey Aug 13, 2024
4138943
Added support for size, invoke, invoke_all, aggregate
dhirupandey Aug 15, 2024
6d8e1d9
Code cleanup and added support for add_index, remove_index
dhirupandey Aug 29, 2024
ca729af
Test Code cleanup
dhirupandey Aug 29, 2024
7ed18d6
Merge branch 'main' into v1-poc
rlubke Sep 4, 2024
8134313
- Cleanup pass to fix validation errors. Several errors remain as th…
rlubke Sep 6, 2024
92f4347
- Code cleanup
rlubke Sep 9, 2024
761b9fc
Code cleanup
dhirupandey Sep 11, 2024
0e1bb56
- Code cleanup
rlubke Sep 11, 2024
228368b
- checkpoint
rlubke Sep 20, 2024
be64693
Implementation and tests for DocumentChunk
dhirupandey Sep 20, 2024
a30a9f4
Fix validation errors
dhirupandey Sep 20, 2024
5b75e83
Implement paging
rlubke Oct 1, 2024
e36d027
Fix test execution issue. Not all tests pass yet; still need to add s…
rlubke Oct 2, 2024
d4120b0
Update event management to use dispatcher functionality.
rlubke Oct 2, 2024
87b66b1
Remove unnecessary variable.
rlubke Oct 2, 2024
542b442
Implemented BaseQueryResult, BinaryQueryResult and added tests for th…
dhirupandey Oct 3, 2024
b9f7a40
Fixing code analysis failures
dhirupandey Oct 3, 2024
eacdc11
Re-work protocol handshake a bit to resolve obtaining multiple maps/c…
rlubke Oct 3, 2024
15061a1
Added e2e test for SimilaritySearch
dhirupandey Oct 4, 2024
501dd2f
Checkpoint:
rlubke Oct 11, 2024
0d9a62a
Ensure stream write in handshake is guarded by catch.
rlubke Oct 11, 2024
3f6f7ee
Fix cd/ci; use install instead of autoupdate with pre-commit
rlubke Oct 11, 2024
1f6ae7e
Fix req timeout test on github.
rlubke Oct 11, 2024
6c26881
Fix req timeout test on github.
rlubke Oct 11, 2024
93645fc
Attempt to work around probably grpc bug.
rlubke Oct 11, 2024
e823d2b
Fix TLS options test for github.
rlubke Oct 11, 2024
d47e867
Document workaround for gRPC issue.
rlubke Oct 11, 2024
8f58d71
Add Python 3.12.x to the test matrix.
rlubke Oct 11, 2024
6d5eedb
Ensure clean environment for all env tests.
rlubke Oct 11, 2024
3859876
Add Python 3.13 to test matrix.
rlubke Oct 11, 2024
04c00c3
.
rlubke Oct 11, 2024
1acaea2
Reverting changes.
rlubke Oct 11, 2024
d73b0d2
Update to latest grpc.
rlubke Oct 11, 2024
3f7f82d
Remove 3.13 from matrix for the time being.
rlubke Oct 14, 2024
260c5a2
Checkpoint:
rlubke Oct 16, 2024
aa3ca71
Python 3.8 issues with latest checkin.
rlubke Oct 16, 2024
14d411e
Revert changes to pom that were not to be checked in.
rlubke Oct 16, 2024
c4cb2f2
Revert changes to Makefile that were not to be checked in.
rlubke Oct 16, 2024
f511179
Update V0 client to support request_timeout context manager.
rlubke Oct 16, 2024
266a20d
Add request timeout support for stream and paged stream dispatchers a…
rlubke Oct 17, 2024
c506d9b
Skip unary timeout test until proxy issue is resolved.
rlubke Oct 17, 2024
edac2b1
Merge main -> v1-poc
rlubke Oct 18, 2024
0055d77
Added e2e test for SimilaritySearch using DocumentChunk
dhirupandey Oct 23, 2024
c9dd40b
Added implementation and serialization tests for BinaryQuantIndex
dhirupandey Oct 25, 2024
d153469
Added e2e test for BinaryQuantIndex
dhirupandey Oct 29, 2024
7a74f07
Added documentation for classes - part1
dhirupandey Oct 30, 2024
d5878d8
Merge branch 'main' into ai-work
dhirupandey Oct 30, 2024
5704eb4
Documentation added - part2
dhirupandey Oct 30, 2024
b02cf08
Minor style change.
rlubke Oct 30, 2024
6737638
Added documentation for DocuementChunk
dhirupandey Oct 31, 2024
c93874d
Merge branch 'v1-poc' into ai-work
dhirupandey Oct 31, 2024
3d107e3
Merged with v1-poc
dhirupandey Oct 31, 2024
ea3b485
Fix typo
dhirupandey Oct 31, 2024
ed96669
Remove custom error types.
rlubke Nov 1, 2024
f66922e
Updates to timeout tests.
rlubke Nov 1, 2024
af39f4f
Merge branch 'v1-poc' into ai-work
dhirupandey Nov 1, 2024
697078a
Comment out running of ai tests
dhirupandey Nov 1, 2024
ec4ac80
Add support for logging of request/responses controlled by the COHERE…
rlubke Nov 5, 2024
723cf32
Merge branch 'main' into v1-poc
rlubke Nov 20, 2024
830384d
Add Python 3.13.x to test matrix.
rlubke Nov 20, 2024
65caac2
Add Python 3.13.x to test matrix.
rlubke Nov 20, 2024
af35eba
Remove 3.8.x from test matrix as it is out of support.
rlubke Nov 27, 2024
b5acfb0
Diagnostics
rlubke Nov 27, 2024
58cdfea
Diagnostics
rlubke Nov 27, 2024
1db52d7
Fix timeout tests.
rlubke Nov 27, 2024
e71c988
Fix timeout tests.
rlubke Nov 27, 2024
657829f
NearCache checkpoint
rlubke Dec 4, 2024
591a8f8
NearCache checkpoint:
rlubke Dec 4, 2024
a8573d4
NearCache checkpoint:
rlubke Dec 4, 2024
312479b
NearCache checkpoint:
rlubke Dec 4, 2024
20d8420
NearCache checkpoint:
rlubke Dec 4, 2024
271728e
NearCache checkpoint:
rlubke Dec 4, 2024
09646d9
NearCache checkpoint:
rlubke Dec 10, 2024
5697e00
NearCache checkpoint:
rlubke Dec 10, 2024
a9ec9db
NearCache checkpoint
rlubke Dec 12, 2024
4b0ab7e
NearCache checkpoint:
rlubke Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,11 @@ omit =
src/coherence/messages_pb2_grpc.py
src/coherence/services_pb2.py
src/coherence/services_pb2_grpc.py
src/coherence/cache_service_messages_v1_pb2.py
src/coherence/cache_service_messages_v1_pb2_grpc.py
src/coherence/common_messages_v1_pb2.py
src/coherence/common_messages_v1_pb2_grpc.py
src/coherence/proxy_service_messages_v1_pb2.py
src/coherence/proxy_service_messages_v1_pb2_grpc.py
src/coherence/proxy_service_v1_pb2.py
src/coherence/proxy_service_v1_pb2_grpc.py
8 changes: 4 additions & 4 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,19 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8.x", "3.9.x", "3.10.x", "3.11.x"]
poetry-version: ["1.5.0"]
python-version: ["3.9.x", "3.10.x", "3.11.x", "3.12.x", "3.13.x"]
poetry-version: ["1.8.4"]
os: [ubuntu-latest]
coherenceVersion:
- 24.03
- 24.09
- 22.06.10
base-image:
- gcr.io/distroless/java17-debian11
profile:
- ",-jakarta,javax"
- ",jakarta,-javax"
exclude:
- coherenceVersion: 24.03
- coherenceVersion: 24.09
profile: ",-jakarta,javax"
- coherenceVersion: 22.06.10
profile: ",jakarta,-javax"
Expand Down
75 changes: 64 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ COHERENCE_WKA1 ?= server1
COHERENCE_WKA2 ?= server1
CLUSTER_PORT ?= 7574
# Profiles to include for building
PROFILES ?= ",-jakarta,javax"
COHERENCE_BASE_IMAGE ?= gcr.io/distroless/java17-debian11
PROFILES ?= ",jakarta,-javax"
COHERENCE_BASE_IMAGE ?= gcr.io/distroless/java17-debian12

# ----------------------------------------------------------------------------------------------------------------------
# Set the location of various build tools
Expand All @@ -66,6 +66,27 @@ CURRDIR := $(shell pwd)
COMPOSE:=$(shell type -p docker-compose || echo docker compose)
$(info COMPOSE = $(COMPOSE))

# ----------------------------------------------------------------------------------------------------------------------
# List of unit tests
# ----------------------------------------------------------------------------------------------------------------------
UNIT_TESTS := tests/unit/test_cache_options.py \
tests/unit/test_local_cache.py \
tests/unit/test_environment.py \
tests/unit/test_serialization.py \
tests/unit/test_extractors.py

# ----------------------------------------------------------------------------------------------------------------------
# List of E2E tests
# ----------------------------------------------------------------------------------------------------------------------
E2E_TESTS := tests/e2e/test_session.py \
tests/e2e/test_client.py \
tests/e2e/test_events.py \
tests/e2e/test_filters.py \
tests/e2e/test_processors.py \
tests/e2e/test_aggregators.py \
tests/e2e/test_near_caching.py \
# tests/e2e/test_ai.py \

# ----------------------------------------------------------------------------------------------------------------------
# Clean-up all of the build artifacts
# ----------------------------------------------------------------------------------------------------------------------
Expand Down Expand Up @@ -116,21 +137,53 @@ generate-proto: ## Generate Proto Files
sed -e 's/import messages_pb2 as messages__pb2/import coherence.messages_pb2 as messages__pb2/' \
< $(CURRDIR)/src/coherence/services_pb2_grpc.py > $(CURRDIR)/src/coherence/services_pb2_grpc.py.out
mv $(CURRDIR)/src/coherence/services_pb2_grpc.py.out $(CURRDIR)/src/coherence/services_pb2_grpc.py
curl -o $(PROTO_DIR)/proxy_service_v1.proto \
https://raw.githubusercontent.com/oracle/coherence/$(COHERENCE_VERSION)/prj/coherence-grpc/src/main/proto/proxy_service_v1.proto
curl -o $(PROTO_DIR)/proxy_service_messages_v1.proto \
https://raw.githubusercontent.com/oracle/coherence/$(COHERENCE_VERSION)/prj/coherence-grpc/src/main/proto/proxy_service_messages_v1.proto
curl -o $(PROTO_DIR)/common_messages_v1.proto \
https://raw.githubusercontent.com/oracle/coherence/$(COHERENCE_VERSION)/prj/coherence-grpc/src/main/proto/common_messages_v1.proto
curl -o $(PROTO_DIR)/cache_service_messages_v1.proto \
https://raw.githubusercontent.com/oracle/coherence/$(COHERENCE_VERSION)/prj/coherence-grpc/src/main/proto/cache_service_messages_v1.proto
python -m grpc_tools.protoc --proto_path=$(CURRDIR)/etc/proto --pyi_out=$(CURRDIR)/src/coherence --python_out=$(CURRDIR)/src/coherence \
--grpc_python_out=$(CURRDIR)/src/coherence \
$(CURRDIR)/etc/proto/proxy_service_v1.proto \
$(CURRDIR)/etc/proto/proxy_service_messages_v1.proto \
$(CURRDIR)/etc/proto/common_messages_v1.proto \
$(CURRDIR)/etc/proto/cache_service_messages_v1.proto
sed -e 's/import proxy_service_messages_v1_pb2 as proxy__service__messages__v1__pb2/import coherence.proxy_service_messages_v1_pb2 as proxy__service__messages__v1__pb2/' \
< $(CURRDIR)/src/coherence/proxy_service_v1_pb2.py > $(CURRDIR)/src/coherence/proxy_service_v1_pb2.py.out
mv $(CURRDIR)/src/coherence/proxy_service_v1_pb2.py.out $(CURRDIR)/src/coherence/proxy_service_v1_pb2.py
sed -e 's/import common_messages_v1_pb2 as common__messages__v1__pb2/import coherence.common_messages_v1_pb2 as common__messages__v1__pb2/' \
< $(CURRDIR)/src/coherence/proxy_service_messages_v1_pb2.py > $(CURRDIR)/src/coherence/proxy_service_messages_v1_pb2.py.out
mv $(CURRDIR)/src/coherence/proxy_service_messages_v1_pb2.py.out $(CURRDIR)/src/coherence/proxy_service_messages_v1_pb2.py
sed -e 's/import proxy_service_messages_v1_pb2 as proxy__service__messages__v1__pb2/import coherence.proxy_service_messages_v1_pb2 as proxy__service__messages__v1__pb2/' \
< $(CURRDIR)/src/coherence/proxy_service_v1_pb2_grpc.py > $(CURRDIR)/src/coherence/proxy_service_v1_pb2_grpc.py.out
mv $(CURRDIR)/src/coherence/proxy_service_v1_pb2_grpc.py.out $(CURRDIR)/src/coherence/proxy_service_v1_pb2_grpc.py
sed -e 's/import common_messages_v1_pb2 as common__messages__v1__pb2/import coherence.common_messages_v1_pb2 as common__messages__v1__pb2/' \
< $(CURRDIR)/src/coherence/cache_service_messages_v1_pb2.py > $(CURRDIR)/src/coherence/cache_service_messages_v1_pb2.py.out
mv $(CURRDIR)/src/coherence/cache_service_messages_v1_pb2.py.out $(CURRDIR)/src/coherence/cache_service_messages_v1_pb2.py

# ----------------------------------------------------------------------------------------------------------------------
# Run tests with code coverage
# ----------------------------------------------------------------------------------------------------------------------
.PHONY: test
test: ##
pytest -W error --cov src/coherence --cov-report=term --cov-report=html \
tests/test_serialization.py \
tests/test_extractors.py \
tests/test_session.py \
tests/test_client.py \
tests/test_events.py \
tests/test_filters.py \
tests/test_processors.py \
tests/test_aggregators.py \
pytest -W error --cov src/coherence --cov-report=term --cov-report=html $(UNIT_TESTS) $(E2E_TESTS)

# ----------------------------------------------------------------------------------------------------------------------
# Run unit tests with code coverage
# ----------------------------------------------------------------------------------------------------------------------
.PHONY: test-unit
test-unit: ##
pytest -W error --cov src/coherence --cov-report=term --cov-report=html $(UNIT_TESTS)

# ----------------------------------------------------------------------------------------------------------------------
# Run e2e tests with code coverage
# ----------------------------------------------------------------------------------------------------------------------
.PHONY: test-e2e
test-e2e: ##
pytest -W error --cov src/coherence --cov-report=term --cov-report=html $(E2E_TESTS)

# ----------------------------------------------------------------------------------------------------------------------
# Run standards validation across project
Expand Down
Loading
Loading