[BUG] opensearch-php 2.4.0 regressed Point In Time search

[TimWolla]

What is the bug?

After updating to 2.4.0, the searchByPointInTime() in the example usage of the USER_GUIDE.md no longer worked:

https://github.com/opensearch-project/opensearch-php/blob/main/USER_GUIDE.md#example-usage

How can one reproduce the bug?

Execute the example in USER_GUIDE.md. Specifically I'm starting OpenSearch in Docker like this:

docker run -it --rm -p 9200:9200 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=CorrectHorseBatteryStaple1" opensearchproject/opensearch

and then run the following script:

Click to expand

<?php

require('vendor/autoload.php');

define('INDEX_NAME', 'test_elastic_index_name' . time());


class MyOpenSearchClass
{

    protected ?\OpenSearch\Client $client;
    protected $existingID = 1668504743;
    protected $deleteID = 1668504743;
    protected $bulkIds = [];


    public function __construct()
    {
        // Simple Setup
        $this->client = OpenSearch\ClientBuilder::fromConfig([
            'hosts' => [
                'https://localhost:9200'
            ],
	    'basicAuthentication' => ['admin', 'CorrectHorseBatteryStaple1'],
	    'SSLVerification' => false,
            'retries' => 2,
            'handler' => OpenSearch\ClientBuilder::multiHandler()
        ]);

        // OR via Builder
        // $this->client = (new \OpenSearch\ClientBuilder())
        //     ->setHosts(['https://localhost:9200'])
        //     ->setBasicAuthentication('admin', 'admin') // For testing only. Don't store credentials in code.
        //     // or, if using AWS SigV4 authentication:
        //     ->setSigV4Region('us-east-2')
        //     ->setSigV4CredentialProvider(true)
        //     ->setSSLVerification(false) // For testing only. Use certificate for validation
        //     ->build();
    }


    // Create an index with non-default settings.
    public function createIndex()
    {
        $this->client->indices()->create([
            'index' => INDEX_NAME,
            'body' => [
                'settings' => [
                    'index' => [
                        'number_of_shards' => 4
                    ]
                ]
            ]
        ]);
    }

    public function info()
    {
        // Print OpenSearch version information on console.
        var_dump($this->client->info());
    }

    // Create a document
    public function create()
    {
        $time = time();
        $this->existingID = $time;
        $this->deleteID = $time . '_uniq';


        // Create a document passing the id
        $this->client->create([
            'id' => $time,
            'index' => INDEX_NAME,
            'body' => $this->getData($time)
        ]);

        // Create a document passing the id
        $this->client->create([
            'id' => $this->deleteID,
            'index' => INDEX_NAME,
            'body' => $this->getData($time)
        ]);

        // Create a document without passing the id (will be generated automatically)
        $this->client->create([
            'index' => INDEX_NAME,
            'body' => $this->getData($time + 1)
        ]);

        //This should throw an exception because ID already exists
        // $this->client->create([
        //     'id' => $this->existingID,
        //     'index' => INDEX_NAME,
        //     'body' => $this->getData($this->existingID)
        // ]);
    }

    public function update()
    {
        $this->client->update([
            'id' => $this->existingID,
            'index' => INDEX_NAME,
            'body' => [
                //data must be wrapped in 'doc' object
                'doc' => ['name' => 'updated']
            ]
        ]);
    }

    public function bulk()
    {
        $bulkData = [];
        $time = time();
        for ($i = 0; $i < 20; $i++) {
            $id = ($time + $i) . rand(10, 200);
            $bulkData[] = [
                'index' => [
                    '_index' => INDEX_NAME,
                    '_id' => $id,
                ]
            ];
            $this->bulkIds[] = $id;
            $bulkData[] = $this->getData($time + $i);
        }
        //will not throw exception! check $response for error
        $response = $this->client->bulk([
            //default index
            'index' => INDEX_NAME,
            'body' => $bulkData
        ]);

        //give elastic a little time to create before update
        sleep(2);

        // bulk update
        for ($i = 0; $i < 15; $i++) {
            $bulkData[] = [
                'update' => [
                    '_index' => INDEX_NAME,
                    '_id' => $this->bulkIds[$i],
                ]
            ];
            $bulkData[] = [
                'doc' => [
                    'name' => 'bulk updated'
                ]
            ];
        }

        //will not throw exception! check $response for error
        $response = $this->client->bulk([
            //default index
            'index' => INDEX_NAME,
            'body' => $bulkData
        ]);
    }
    public function deleteByQuery(string $query)
    {
        if ($query == '') {
            return;
        }
        $this->client->deleteByQuery([
            'index' => INDEX_NAME,
            'q' => $query
        ]);
    }

    // Delete a single document
    public function deleteByID()
    {
        $this->client->delete([
            'id' => $this->deleteID,
            'index' => INDEX_NAME,
        ]);
    }

    public function search()
    {
        $docs = $this->client->search([
            //index to search in or '_all' for all indices
            'index' => INDEX_NAME,
            'size' => 1000,
            'body' => [
                'query' => [
                    'prefix' => [
                        'name' => 'wrecking'
                    ]
                ]
            ]
        ]);
        var_dump($docs['hits']['total']['value'] > 0);

        // Search for it
        $docs = $this->client->search([
            'index' => INDEX_NAME,
            'body' => [
                'size' => 5,
                'query' => [
                    'multi_match' => [
                        'query' => 'miller',
                        'fields' => ['title^2', 'director']
                    ]
                ]
            ]
        ]);
        var_dump($docs['hits']['total']['value'] > 0);
    }

    // Write queries in SQL
    public function searchUsingSQL()
    {
        $docs = $this->client->sql()->query([
          'query' => "SELECT * FROM " . INDEX_NAME . " WHERE name = 'wrecking'",
          'format' => 'json'
        ]);
        var_dump($docs['hits']['total']['value'] > 0);
    }

    public function getMultipleDocsByIDs()
    {
        $docs = $this->client->search([
            //index to search in or '_all' for all indices
            'index' => INDEX_NAME,
            'body' => [
                'query' => [
                    'ids' => [
                        'values' => $this->bulkIds
                    ]
                ]
            ]
        ]);
        var_dump($docs['hits']['total']['value'] > 0);
    }

    public function getOneByID()
    {
        $docs = $this->client->search([
            //index to search in or '_all' for all indices
            'index' => INDEX_NAME,
            'size' => 1,
            'body' => [
                'query' => [
                    'bool' => [
                        'filter' => [
                            'term' => [
                                '_id' => $this->existingID
                            ]
                        ]
                    ]
                ]
            ]
        ]);
        var_dump($docs['hits']['total']['value'] > 0);
    }

    public function searchByPointInTime()
    {
        $result = $this->client->createPointInTime([
            'index' => INDEX_NAME,
            'keep_alive' => '10m'
        ]);
        $pitId = $result['pit_id'];

        // Get first page of results in Point-in-Time
        $result = $this->client->search([
            'body' => [
                'pit' => [
                    'id' => $pitId,
                    'keep_alive' => '10m',
                ],
                'size' => 10, // normally you would do 10000
                'query' => [
                    'match_all' => (object)[]
                ],
                'sort' => '_id',
            ]
        ]);
        var_dump($result['hits']['total']['value'] > 0);

        $last = end($result['hits']['hits']);
        $lastSort = $last['sort'] ?? null;

        // Get next page of results in Point-in-Time
        $result = $this->client->search([
            'body' => [
                'pit' => [
                    'id' => $pitId,
                    'keep_alive' => '10m',
                ],
                'search_after' => $lastSort,
                'size' => 10, // normally you would do 10000
                'query' => [
                    'match_all' => (object)[]
                ],
                'sort' => '_id',
            ]
        ]);
        var_dump($result['hits']['total']['value'] > 0);

        // Close Point-in-Time
        $result = $this->client->deletePointInTime([
            'body' => [
              'pit_id' => $pitId,
            ]
        ]);
        var_dump($result['pits'][0]['successful']);
    }

    // Delete index
    public function deleteByIndex()
    {
        $this->client->indices()->delete([
            'index' => INDEX_NAME
        ]);
    }

    //simple data to index
    public function getData($time = -1)
    {
        if ($time == -1) {
            $time = time();
        }
        return [
            'name' => date('c', $time) . " - i came in like a wrecking ball",
            'time' => $time,
            'date' => date('c', $time)
        ];
    }
}

try {

    $e = new MyOpenSearchClass();
    $e->info();
    $e->createIndex();
    $e->create();
    //give elastic a little time to create before update
    sleep(2);
    $e->update();
    $e->bulk();
    $e->getOneByID();
    $e->getMultipleDocsByIDs();
    $e->search();
    $e->searchUsingSQL();
    $e->searchByPointInTime();
    $e->deleteByQuery('');
    $e->deleteByID();
    $e->deleteByIndex();
} catch (\Throwable $th) {
    echo 'uncaught error ' . $th->getMessage() . "\n";
}

I'm receiving the following output:

array(5) {
  ["name"]=>
  string(12) "93c9ae71a202"
  ["cluster_name"]=>
  string(14) "docker-cluster"
  ["cluster_uuid"]=>
  string(22) "v9dOxBHORdy9du7MlMdLSQ"
  ["version"]=>
  array(9) {
    ["distribution"]=>
    string(10) "opensearch"
    ["number"]=>
    string(6) "2.18.0"
    ["build_type"]=>
    string(3) "tar"
    ["build_hash"]=>
    string(40) "99a9a81da366173b0c2b963b26ea92e15ef34547"
    ["build_date"]=>
    string(30) "2024-10-31T19:08:39.157471098Z"
    ["build_snapshot"]=>
    bool(false)
    ["lucene_version"]=>
    string(6) "9.12.0"
    ["minimum_wire_compatibility_version"]=>
    string(6) "7.10.0"
    ["minimum_index_compatibility_version"]=>
    string(5) "7.0.0"
  }
  ["tagline"]=>
  string(47) "The OpenSearch Project: https://opensearch.org/"
}
bool(true)
bool(true)
bool(true)
bool(false)
bool(false)
uncaught error {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: [indices] cannot be used with point in time;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: [indices] cannot be used with point in time;"},"status":400}

What is the expected behavior?

I expected the following output:

array(5) {
  ["name"]=>
  string(12) "93c9ae71a202"
  ["cluster_name"]=>
  string(14) "docker-cluster"
  ["cluster_uuid"]=>
  string(22) "v9dOxBHORdy9du7MlMdLSQ"
  ["version"]=>
  array(9) {
    ["distribution"]=>
    string(10) "opensearch"
    ["number"]=>
    string(6) "2.18.0"
    ["build_type"]=>
    string(3) "tar"
    ["build_hash"]=>
    string(40) "99a9a81da366173b0c2b963b26ea92e15ef34547"
    ["build_date"]=>
    string(30) "2024-10-31T19:08:39.157471098Z"
    ["build_snapshot"]=>
    bool(false)
    ["lucene_version"]=>
    string(6) "9.12.0"
    ["minimum_wire_compatibility_version"]=>
    string(6) "7.10.0"
    ["minimum_index_compatibility_version"]=>
    string(5) "7.0.0"
  }
  ["tagline"]=>
  string(47) "The OpenSearch Project: https://opensearch.org/"
}
bool(true)
bool(true)
bool(true)
bool(false)
bool(false)
bool(true)
bool(true)
bool(true)

What is your host/environment?

Ubuntu 24.04, running PHP 8.3 against OpenSearch 2.18.0 running in Docker.

Do you have any screenshots?

n/a

Do you have any additional context?

git bisect says:

f110a17 is the first bad commit

commit f110a1752a09f31f4be5a16fefbf7d01962f753c
Author: Kim Pepper <kim@pepper.id.au>
Date:   Sat Nov 16 05:22:36 2024 +1100

    Replace callable with Endpoint factory (#237)
    
    * Replace endpoint callable with EndpointFactory
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Add BC for deprecated  property
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Address feedback
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Remove trait and fix bc
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Addressed feedback
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Address feedback
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Added changelog and deprecation trigger
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Update changelog
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Generate API
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    * Pass  factory to closure
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>
    
    ---------
    
    Signed-off-by: Kim Pepper <kim@pepper.id.au>

 CHANGELOG.md                                       |   2 +
 src/OpenSearch/Client.php                          | 336 +++++++++++----------
 src/OpenSearch/ClientBuilder.php                   |  52 ++--
 src/OpenSearch/EndpointFactory.php                 |  55 ++++
 src/OpenSearch/EndpointFactoryInterface.php        |  21 ++
 src/OpenSearch/EndpointInterface.php               |  81 +++++
 src/OpenSearch/LegacyEndpointFactory.php           |  39 +++
 src/OpenSearch/Namespaces/AbstractNamespace.php    |  19 +-
 .../Namespaces/AsynchronousSearchNamespace.php     |  16 +-
 src/OpenSearch/Namespaces/CatNamespace.php         | 124 ++++----
 src/OpenSearch/Namespaces/ClusterNamespace.php     | 100 +++---
 .../Namespaces/DanglingIndicesNamespace.php        |  14 +-
 .../Namespaces/FlowFrameworkNamespace.php          |  40 +--
 src/OpenSearch/Namespaces/IndicesNamespace.php     | 333 ++++++++++----------
 src/OpenSearch/Namespaces/IngestNamespace.php      |  20 +-
 src/OpenSearch/Namespaces/InsightsNamespace.php    |   4 +-
 src/OpenSearch/Namespaces/IsmNamespace.php         |  60 ++--
 src/OpenSearch/Namespaces/KnnNamespace.php         |  34 +--
 src/OpenSearch/Namespaces/ListNamespace.php        |  12 +-
 src/OpenSearch/Namespaces/MlNamespace.php          |  70 ++---
 src/OpenSearch/Namespaces/NodesNamespace.php       |  22 +-
 .../Namespaces/NotificationsNamespace.php          |  36 +--
 .../Namespaces/ObservabilityNamespace.php          |  28 +-
 src/OpenSearch/Namespaces/PplNamespace.php         |  16 +-
 src/OpenSearch/Namespaces/QueryNamespace.php       |  20 +-
 src/OpenSearch/Namespaces/RemoteStoreNamespace.php |   4 +-
 src/OpenSearch/Namespaces/RollupsNamespace.php     |  24 +-
 .../Namespaces/SearchPipelineNamespace.php         |  12 +-
 src/OpenSearch/Namespaces/SecurityNamespace.php    | 314 +++++++++----------
 src/OpenSearch/Namespaces/SnapshotNamespace.php    |  46 +--
 src/OpenSearch/Namespaces/SqlNamespace.php         |  30 +-
 src/OpenSearch/Namespaces/TasksNamespace.php       |  12 +-
 src/OpenSearch/Namespaces/TransformsNamespace.php  |  32 +-
 src/OpenSearch/Namespaces/WlmNamespace.php         |  28 +-
 tests/ClientTest.php                               |  16 +-
 tests/Namespaces/MachineLearningNamespaceTest.php  | 137 ++++-----
 tests/Namespaces/SqlNamespaceTest.php              |  26 +-
 util/EndpointProxies/createProxy.php               |   5 +-
 .../indices/refreshSearchAnalyzersProxy.php        |   3 +-
 util/EndpointProxies/ml/createConnectorProxy.php   |   3 +-
 util/EndpointProxies/ml/deleteConnectorProxy.php   |   3 +-
 util/EndpointProxies/ml/deployModelProxy.php       |   3 +-
 util/EndpointProxies/ml/getConnectorProxy.php      |   3 +-
 util/EndpointProxies/ml/getConnectorsProxy.php     |   3 +-
 util/EndpointProxies/ml/getModelGroupsProxy.php    |   3 +-
 util/EndpointProxies/ml/getModelProxy.php          |   3 +-
 util/EndpointProxies/ml/predictProxy.php           |   3 +-
 util/EndpointProxies/ml/undeployModelProxy.php     |   3 +-
 util/EndpointProxies/ml/updateModelGroupProxy.php  |   3 +-
 .../security/changePasswordProxy.php               |   3 +-
 .../security/createActionGroupProxy.php            |   5 +-
 .../security/createRoleMappingProxy.php            |   5 +-
 util/EndpointProxies/security/createRoleProxy.php  |   3 +-
 .../EndpointProxies/security/createTenantProxy.php |   3 +-
 util/EndpointProxies/security/createUserProxy.php  |   3 +-
 .../security/getActionGroupsProxy.php              |   5 +-
 .../security/getDistinguishedNamesProxy.php        |   5 +-
 .../security/getRoleMappingsProxy.php              |   5 +-
 util/EndpointProxies/security/getRolesProxy.php    |   5 +-
 util/EndpointProxies/security/getTenantsProxy.php  |   5 +-
 util/EndpointProxies/security/getUsersProxy.php    |   4 +-
 .../security/patchActionGroupsProxy.php            |   5 +-
 .../security/patchRoleMappingsProxy.php            |   5 +-
 util/EndpointProxies/security/patchRolesProxy.php  |   5 +-
 .../EndpointProxies/security/patchTenantsProxy.php |   5 +-
 util/EndpointProxies/security/patchUsersProxy.php  |   5 +-
 util/EndpointProxies/sql/closeCursorProxy.php      |   4 +-
 util/EndpointProxies/sql/explainProxy.php          |   4 +-
 util/EndpointProxies/sql/queryProxy.php            |   4 +-
 util/NamespaceEndpoint.php                         |   3 +-
 util/template/client-class                         |  34 ++-
 util/template/endpoint-function                    |   5 +-
 util/template/endpoint-function-bool               |   5 +-
 util/template/new-namespace                        |   2 +-
 74 files changed, 1302 insertions(+), 1108 deletions(-)
 create mode 100644 src/OpenSearch/EndpointFactory.php
 create mode 100644 src/OpenSearch/EndpointFactoryInterface.php
 create mode 100644 src/OpenSearch/EndpointInterface.php
 create mode 100644 src/OpenSearch/LegacyEndpointFactory.php

Full log:

# status: waiting for both good and bad commits
# bad: [d35ea28b2ce991b9e4bd338833ffa613f19bef5e] Clean up imports and whitespace (#282)
git bisect bad d35ea28b2ce991b9e4bd338833ffa613f19bef5e
# status: waiting for good commit(s), bad commit known
# good: [1866e6ee95c15036038d6c95a5c54c6fe648de36] fix: php 8.4 deprecations (#226)
git bisect good 1866e6ee95c15036038d6c95a5c54c6fe648de36
# bad: [70e71ffe98409d7345aac371c7cd5ed72a6731eb] Added optional headers to the AWS SigningDecorator. (#253)
git bisect bad 70e71ffe98409d7345aac371c7cd5ed72a6731eb
# good: [6b08b1699c1b83cae89724dc0f306872cf3298a7] Updated opensearch-php to reflect the latest OpenSearch API spec (2024-11-14) (#238)
git bisect good 6b08b1699c1b83cae89724dc0f306872cf3298a7
# bad: [887df5e8d1b0513fd36ca66cc626ef379bda51f9] Adds a test for the signing client decorator (#252)
git bisect bad 887df5e8d1b0513fd36ca66cc626ef379bda51f9
# bad: [d7508383d2b824a2aaae4feb0a22a10a3b8cb1f4] Updated opensearch-php to reflect the latest OpenSearch API spec (2024-12-15) (#241)
git bisect bad d7508383d2b824a2aaae4feb0a22a10a3b8cb1f4
# bad: [f110a1752a09f31f4be5a16fefbf7d01962f753c] Replace callable with Endpoint factory (#237)
git bisect bad f110a1752a09f31f4be5a16fefbf7d01962f753c
# first bad commit: [f110a1752a09f31f4be5a16fefbf7d01962f753c] Replace callable with Endpoint factory (#237)

[dblock]

@TimWolla Thanks for bisecting the change, that was a big part of the PSR rewrite. Try turning this into a unit/integration test instead, let's narrow down why it doesn't work?

[TimWolla]

Try turning this into a unit/integration test instead, let's narrow down why it doesn't work?

If this was a request to me doing that, unfortunately I have to decline: I don't actually use the “Point in Time” search, but it was part of an integration test of mine (based on the example in the USER_GUIDE.md) that started failing. I've already marked it as expected failure and can't spend more time debugging a +1302-1108 lines commit, but at least wanted to make you aware of the regression with as much information I can easily provide.

[dblock]

Thanks for reporting it @TimWolla!