Skip to content

Class with @RedisHash and byte[] property is turned into Map<String, Byte> #1981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bergerst opened this issue Feb 22, 2021 · 5 comments
Closed
Assignees
Labels
type: bug A general bug

Comments

@bergerst
Copy link
Contributor

bergerst commented Feb 22, 2021

I discovered that if you define a class with @RedisHash and a byte[] property, one key-value pair is stored for each byte.

Here's some example code:

@RedisHash("myclass")
@lombok.Data
public class MyClass {
  @Id
  private String id;
  private byte[] property;

  public MyClass(String id) {
    this.id = id;
    property = new byte[4000];
    Random rd = new Random();
    rd.nextBytes(property);
  }
}

When an object of this class is stored in Redis using a CrudRepository<MyClass, String>, the Redis content will look like this:

property.[3018] = 50
property.[298] = 156
property.[1892] = 2
...

When you then use findById on the repository, Bucket.extract() will be called by MappingRedisConverter.readCollectionOrArray() for each key, leading to about 16 million String.startsWith() calls.

In my opinion, the byte[] should be stored as a String value, not as a Map<String, Byte>.

Replacing the byte[] with a String reduced the run time of my webservice operation from 1200 ms to 45 ms.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Feb 22, 2021
@christophstrobl christophstrobl added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged labels Feb 23, 2021
@bergerst
Copy link
Contributor Author

bergerst commented Feb 23, 2021

Generally speaking, the performance of MappingRedisConverter.readCollectionOrArray() can be improved in all cases when large collections or arrays are stored.

Do you want me to create a PR for that part? It may not be the optimal solution, but it does the job much better.

A second change will be needed in order to read byte[] properties at once. How it's done right now:

  1. Read byte as byte[]
  2. Convert byte[] to Byte
  3. Add Byte to Collection<Byte>
  4. Convert Collection<Byte> to byte[]
  5. Set property with byte[]

@christophstrobl
Copy link
Member

We'd be happy to review a PR.
Please follow the contribution guidelines and make sure all tests are passing. The above change seems break some in MappingRedisConverterUnitTests.

bergerst added a commit to bergerst/spring-data-redis that referenced this issue Feb 23, 2021
By using a TreeMap, entries starting with a certain prefix can be found
much faster than before.

Related tickets spring-projects#1981
@bergerst
Copy link
Contributor Author

You're right, I didn't understand the full logic of the existing implementation.

The unit tests pass for my submitted PR.

@mp911de mp911de self-assigned this Mar 3, 2021
@mp911de
Copy link
Member

mp911de commented Mar 3, 2021

It would make sense to consider byte[] as a native simple type as the underlying data structure for Strings, numbers, etc. is a byte[]. Following the path opens a discussion about backward compatibility. Our code should be able to read byte arrays in the current format and directly from the hash value.

I'd be generally fine with tweaking the representation as of the next feature release. Any objections @christophstrobl?

@christophstrobl
Copy link
Member

fine with me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants