Roadmap to version 2

[sstok]

TL;DR: Version 1.x will become an LTS version and 2.0 will not be backward compatible, requires at least PHP 7 and Symfony 3.

For RollerworksSearch 2 I propose the following (rigorous) changes:

Support for Symfony 2 and PHP 5 is dropped, 1.x becomes an LTS version
with support until Nov 2018 (bug only fixes).

The main goal is to get rid of old architecture problems and embrace
good quality code and new features. All deprecations are removed.
DX and performance is a pre!

The new API is whenever possible introduced
(adding new methods to an interface is allowed).

2.0 will introduce a number of BC breaking changes that are
not possible with a BC layer. Mainly the usage of PHP 7 scalar type hints,
and strict return-types.

Rules:

• No big 'change all the things' pull request!
• At the time of starting the 2.0 no new features are accepted
  for the 1.x branch;
• The 1.x branch will be supported until November 2018;
• The introduction of the new API in 1.x makes it easier
  to upgrade, it's not a smooth transition as 2.0 has other breaking changes;

First stage:

• Introduce the new API (with build-in BC layer)
  • The BC promise may be broken with new methods and properties.
• Update the documentation to use the new API
• Switch components to the new the new API
• Release the final minor version of the 1.x branch
• Update external extensions to use the new API
  • The Symfony Validator receives a new ValueValidator to support the new interface

Second stage:

• Remove the old deprecated API
• Switch to PHP 7 scalar type-hints
• Switch to PHP 7 strict return types
• Release 2.0-beta1
• Update external extensions to use the new PHP 7 compatible API
  • Release each completely changed extension with 2.0-beta1 to start
    testing

Final stage (approximately 3 months after last beta release)

• Release 2.0 stable

FieldSet

The FieldSet is immutable and cannot be changed after creation.
Use a FieldSetBuilder to keep the building process open.

FieldSet configuration in app-config is removed in favor of a FieldSet registry
with tags; The FieldSet name is removed, and for view it's replaced with a 'blockPrefix'.

A new view class is introduced: FieldSetView which holds all the SearchField
view classes. The client-side will need to handle the form-widget rendering.

Input and values

The ValuesGroup, ValuesBag and InvalidValue are moved to the Value
namespace and made immutable. Calling a mutator method returns a new object
with the changes applied. Existing objects remain unchanged.

As a small bonus it will be possible to pass multiple values in a single
call, to prevent to much object creation.

And it will be possible to add custom value objects. Each value object needs to
implement the new Rollerworks\Component\Search\Value\ValueHolder interface.
The ValueHolder extends the SPL Serializable interface.

Simple (excluded) values don't require an object anymore,
reduces memory usage (significantly)!

Alias/Label resolving

Deprecate the FieldAliasResolver with no alternative, this is only
used by the FilterQuery processor. Make the FilterQuery use the field label,
to build an internal alias resolver.

Deprecate FieldLabelResolverInterface in favor of a Symfony translator
extension for auto-configuring a label (run the field-name to a translator).

And add the translation_domain option for translation domain (default is search).
Option is always present, but requires a processor to be effective.

View and exported-normalized representation

A ValueHolder does not contains a view representation, use the
field type view-transformer to get the human readable format.

If the value needs to be unlocalized (normalized) use the field type
normalizer-transformer. This is a new feature that FINALLY adds support
for REST API end-points!.

Note: There is only one normalizer-transformer and view-transformer allowed
per field, use a chain transformer if multiple are needed.

Method changes for ValuesBag

All specific methods will be removed, instead the class has only the following
methods for values: get, has, remove, add. Which support all classes implementing
the ValueHolder interface.

The simple (excluded) value methods will remain (as they have no interface).

Getting a value happens by using the FQCN, so there is no risk of receiving
the wrong object 👍

Improved error handling

Input is transformed (and validated using a validator implementation)
by the input processor.

In case of a transformation or validation failure the value is stored in an InvalidValue object.
At the end of the process, an InvalidSearchConditionException is thrown
with the origin input and a list of all the invalid values (InvalidValue[]).

When the process was successful it returns a SearchCondition with transformed values.

Using an exception makes strict return-type possible
and ensures failure don't go unnoticed.

InvalidValue

The InvalidValue class holds the same information as the current implementation,
but the path value is highly dependent on processors format.

For example the path for an FilterQuery is groups[0].fields['name']['value-position']
as there is no structure (only values at a specific position).

While the array process provides groups[0].fields['field-name']['ranges'][0]
with a complete structure path.

SmartQuery could provide something as simple as a value position (eg. 1).

Nested optimizers (feature)

Add optimizers that operate on nested values, eg. a value
that is already present at a lower group level or a range
that in a deeper level that overlaps with a lower level range.

Each (nested) group has a separate valuesBag (cloned), which is used to look
for overlap from higher hierarchy groups. In practice this works the same as the
current algorithm but instead of using only the current level, the higher levels
are tracked as they are passed trough (tracked descending traversal).

To Resolve and other stuff

• Rewrite Input processor internal logic;
  • Keep more information in private properties, and reset after processing;
• Replace [Input|Exporter]Factory in favor of [Input|Exporter]ProcessorLoader;
• ApiPlatform support (separate package)
• PSR-7 SearchProcessor (separate package)
• ConditionBuilder UI (separate package, try to make agnostic or provide implementation
  for each framework separate)
• Better documentation? More user/developer focused and less internal stuff
  • Explain the components (minimum), information flow
• Deprecate SearchConditionInterface, and use the class only
• Deprecate FieldConfigInterface::getModelRef{Class|Property}()
• Add {set|get}ViewTransformer() (deprecate other transformer methods)
• Add {set|get}ModelTransformer()
• Rename FieldConfigInterface::getDataLocked() to FieldConfigInterface::isLocked()

Processor

Make processors stateless - use a SearchPayload object to store information;

• New package rollerworks/search-request-processor
• SearchProcessor::processRequest(ServerRequest $request, ProcessorConfig $config, 'export-format'): SearchPayload;
• Use PRS-7 for the request information;
• SearchPayload can be stored in a cache as serialized by SearchCode

SearchPayload (immutable, serializable):

• SearchCondition (read-only property)
• Messages (errors messages) (read-only property)
• SearchCode (read-only property)
• ExportedCondition (cached result of the exported condition) (read-only property)
• valid()
• isChanged()

Api Platform support (separate package)

Focused on Api Platform 2.0.

Provide condition as array, only process when there is no cached
result available. Caching is handled by persistence (short-lived).
The condition is always optimized and stored by the original input-hash.

It's also possible to use a POST, which will return eg. an "error" context
or redirects to the correct result (with an optimized condition).

When there is an error an "error"-context result is returned
with the related HTTP status-code.

Provide a rate limiter for Search operations, n new conditions
per time-frame by IP/Auth token.

[sstok]

I hope with these changes to also solve #1, other planned features will be added in the 1.x branch when possible 👍

[sstok]

Updated with some new idea's and notes. Proposed changes are much better now, and less 'change all the things'.

[sstok]

Scratched the immutable values idea, it's to much complexity for little gain.

[sstok]

New change: Remove model mapping completely, only Doctrine ORM uses this and implementation and its a bit complex to use and understand. I prefer explicit configuration of fields to Entity#Property mapping for the WhereBuilder.

[sstok]

TL;DR: Version 1.x becomes an LTS version and 2.0 will not be backward compatible,
requires at least PHP 7 (7.1) and Symfony 3.

Support for Symfony 2 and PHP 5 is dropped, 1.x becomes an LTS version
with support until Nov 2018 (bug only fixes).

The main goal is to get rid of old architecture problems and embrace
good quality code and new features. All deprecations are removed.
DX and performance is a pre!

Because keeping compatibility while embracing a completely new API
is not possible, the 2.0 is a complete rewrite with no adapters
for older versions. Mainly the usage of PHP 7 scalar type hints,
and strict return-types.

Rules:

• No big 'change all the things' pull request, one combined change per pr!
• At the time of starting the 2.0 no new features are accepted
  for the 1.x branch;
• The 1.x branch will be supported until November 2018;

To allow for enough development time, this release follows
an ALPHA, BETA, RC, stable path. It's a complete rewrite!

FieldSet

• The FieldSet is immutable and cannot be changed after creation.
  Use a FieldSetBuilder to keep the building process open.
• FieldSet configuration in app-config is removed in favor of a FieldSet registry
  with FCQN loading and customer loaders (same implementation as Datagrid component).
• The FieldSet name is used only for validating a supported FieldSet configuration is provided.
• The View uses a 'blockPrefix'.
• A new view class is introduced: FieldSetView which holds all the SearchField
  view classes. The client-side will need to handle the form-widget rendering.

Input and values (DONE)

• The ValuesGroup, ValuesBag and InvalidValue are moved to the Value
  namespace and made immutable. (DONE)
• Allow to use custom value objects. Each value object needs to
  implement the new Rollerworks\Component\Search\Value\ValueHolder interface.
  The ValueHolder extends the SPL Serializable interface. (DONE)

Simple (excluded) values don't require an object anymore,
to reduce memory usage (significantly)!

Alias/Label resolving (DONE)

Remove the FieldAliasResolver and FieldLabelResolver, this is only
used by the FilterQuery processor. Make the FilterQuery use the field label,
to build an internal alias resolver.

And add the translation_domain option for translation domain (default is search).
Option is always present, but requires a processor to be effective.

View and exported-normalized representation (DONE 🤘 )

• A ValueHolder does not contain a view representation, use the
  field's view-transformer to get a human readable format.

• A ValueHolder holds a normalized value (eg. an object).
  The value is exportable by the Field's norm-transformer.

  This is a new feature that FINALLY adds support
  for REST API end-points!.

Note: There is only one normalize-transformer and view-transformer allowed
per field, use a chain transformer if multiple are needed.

Method changes for ValuesBag (DONE)

All specific methods will be removed, instead the class has only the following
methods for values: get, has, remove, add.

Which support all classes implementing the ValueHolder interface.
Getting a value happens by using the FQCN, so there is no risk of receiving
the wrong object 👍

The simple (excluded) value methods will remain (as they have no interface).

Improved error handling (DONE)

Input is transformed (and validated using a validator implementation)
by the input processor.

In case of a transformation or validation failure the value is stored in an
InvalidValue object. At the end of the process, an InvalidSearchConditionException
is thrown with the origin input and a list of all the invalid values (InvalidValue[]).

When the process was successful it returns a SearchCondition with
transformed values.

Using an exception makes strict return-type possible
and ensures failure don't go unnoticed.

InvalidValue (DONE)

The InvalidValue class holds the same information as the current implementation,
but the path value is highly dependent on processors format.

For example the path for an FilterQuery is groups[0].fields['name']['value-position']
as there is no structure (only values at a specific position).

While the array process provides groups[0].fields['field-name']['ranges'][0]
with a complete structure path.

SmartQuery could provide something as simple as a value position (eg. 1).

Nested optimizers (feature) (REJECTED, maybe later)

Add optimizers that operate on nested values, eg. a value
that is already present at a lower group level or a range
that in a deeper level that overlaps with a lower level range.

Each (nested) group has a separate valuesBag (cloned), which is used to look
for overlap from higher hierarchy groups. In practice this works the same as the
current algorithm but instead of using only the current level, the higher levels
are tracked as they are passed trough (tracked descending traversal).

To Resolve and other stuff

• Replace [Input|Exporter]Factory in favor of [Input|Exporter]ProcessorLoader;
• ApiPlatform support (separate package)
• PSR-7 SearchProcessor (separate package)
• ConditionBuilder UI (separate package, try to make agnostic or provide implementation
  for each framework separate)
• Better documentation? More user/developer focused and less internal stuff
  • Explain the components (minimum), information flow
• Deprecate SearchConditionInterface, and use the class only
• Remove FieldConfigInterface::getModelRef{Class|Property}() (DONE)
• Add {set|get}ViewTransformer() (deprecate other transformer methods) (DONE)
• Add {set|get}NormTransformer() (DONE)
• Rename FieldConfigInterface::getDataLocked() to FieldConfigInterface::isLocked() (DONE)

Processor (DONE)

Make processors stateless - use a SearchPayload object to store information;

• New package rollerworks/search-request-processor
• SearchProcessor::processRequest(ServerRequest $request, ProcessorConfig $config, 'export-format'): SearchPayload;
• Use PRS-7 for the request information;
• SearchPayload can be stored in a cache as serialized by SearchCode

SearchPayload (immutable, serializable):

• SearchCondition (read-only property)
• Messages (errors messages) (read-only property)
• SearchCode (read-only property)
• ExportedCondition (cached result of the exported condition) (read-only property)
• isValid()
• isChanged()

Api Platform support (separate package)

Focused on Api Platform 2.0.

Provide condition as array, only process when there is no cached
result available. Caching is handled by persistence (short-lived).
The condition is always optimized and stored by the original input-hash.

It's also possible to use a POST, which will return eg. an "error" context
or redirects to the correct result (with an optimized condition).

When there is an error an "error"-context result is returned
with the related HTTP status-code.

Provide a rate limiter for Search operations, n new conditions
per time-frame by IP/Auth token.

[sstok]

Note to self: Remove group-id and level-count from exceptions, always communicate the full path to processing methods. Don't keep this as properties as there is much risk of getting this wrong.

Edit. Do keep the current level as property, as this doesn't affect other flows.

[sstok]

New idea for the Error handling.

Throw only InvalidSearchConditionException and add any processing exceptions to the errors-list. So in the end you can have failed transformation for a value error, and a Syntax/values-overflow error as as last message (because processing stops then).

This solves a number of problems:

• Complex exception handling (only one exception to check for)
• No more loss of other error messages (other errors are still reported)
• No more need to "parse" the exception message.

DONE.

[sstok]

So far so good, Doctrine ORM is not as easy as I'd hoped, but I will get it done before the deadline (I hope).

[sstok]

Doctrine ORM integration is as good as done, the Symfony integration bundle needs to be updated.

As I have not had any big changes in the way Doctrine DBAL/ORM is used I'm thinking about moving these back the main bundle, and enabling them automatically when the dependencies are met.
This will make the maintenance less troubling and installation easier 😁

[sstok]

🎉 We have an v2.0.0-ALPHA1 release for the project!