Support for Point property type.

[apexJCL]

Are there any plans on adding support for Neo4J spatial functions and data types?

[aanastasiou]

Hi there

I would be interested in looking into this, at least as far as the data types themselves are concerned. Did you have a specific use case in mind? Would you also be interested in querying Points via neomodel or would serialisation / deserialisation be alright as the first step?

[apexJCL]

Serialization/Deserialization would be enough as first step, I can always use cypher directly for querying and later inflate the nodes by myself.

Mostly I'd been wanting to use the geospatial capabilities of neo4j given a collection of places I am storing.

[aanastasiou]

Just to let you know, while I was planning to have a go at this during the weekend, I was side-tracked by this which causes a severe error on the main project I am working on.

If you would like to have a go at implementing this functionality for neomodel, you would be looking in this file and specifically, you would be creating a brand new Property. According to the documentation, the minimum information required to describe a point is x,y,z,crs and although you could leave the validation up to Neo4j, I think that it would be worth doing validation internally using pyproj, at least for WGS84 points, to make sure that what you pass to the database is valid. The rest of the properties that depend on the selection of CRS could be implemented as "ghost" properties on Property's __getattr__.

All the best

[apexJCL]

Alright, I'll give it a go, thanks!

[aanastasiou]

Hi, I have started working on this serialising and de-serialising Shapely Points so that Points are a little bit more usable out of the box. But it would still be useful if you could do some testing once I have a (presentable) first version ready (?).

[apexJCL]

Sounds good to me, I was actually checking out the deflating of the Point, as I see that the neo4j driver returns a Point instance, but Shapely really seems more useful as you say.

Let me know when it's ready for testing, I'll update my fork and give it a go, thanks in advance.

[aanastasiou]

Well, that Point would leave you with an extra iteration to convert points to whatever else you want to do with them. Dealing with Shapely points saves you that conversion and then from Shapely you have a shorter "bridge" to more complex geospatial processing libraries. So you could be doing "Is point from database in some area defined by an SHP file" type of queries, using neomodel with (almost) no additional code.

[aanastasiou]

@apexJCL, have a look at this repo and specifically branch feature_spatial_datatypes.

Two things to note about this:

1. I did not ask you if you were on Python2.7 or Python3. During my write up, I have assumed Python3 throughout but once we are satisfied that this is addressing the functionality required at this point, then I am going to go ahead and:

   • Make sure it is functional in Pythonn2.7
   • Add tests

2. I have not included any additional documentation with this but neomodel/spatial_properties.py is laden with information.

   • Basically, when specifying a PointProperty in your application's models you have to set AT LEAST its CRS, like so location = neomodel.PointProperty(crs="cartesian") (or whatever CRS is suitable for you).
   • Then after that, PointProperty expects neomodel.NeomodelPoint as the datatype. This is basically a shapely point, which is also CRS aware and does one or two other things as well. But the key thing to remember here is that NeomodelPoints are immutable (just like shapely.geometry.Point are). This might be something that needs revision in the future but for the moment, if you want to update a PointProperty you need to create a completely new NeomodelPoint. A copy constructor however is available.

Please have a look at this (and maybe you too @robinedwards (?)) and let me know if it is down the right track.

All the best

[apexJCL]

Alright, I'll give it a go!

About my env, yeah, I'm using Python 3.7.1, so no problem at all in that part.
Immutable NeomodelPoint sounds fine by me, I still need to see if it represents any drawbacks.

[aanastasiou]

For 3.7.1 it should work out of the box. Just a note, if you don't have shapely installed, the datatype and property will not be available. So, to even access these objects you need to have shapely installed. It produces a warning in stderr for the moment, which you might miss.

[apexJCL]

Yeah, I installed the package from the git repo and when I launched the server, I saw the message, installed Shapely and now it's working.

I was able to create a new Place and assign it a Point, then I checked on neo4j browser and it was ok, but then, I tried to retrieve the Place and there was an error inflating the Node:

'point': POINT(-100.813475 20.528036), 'place_id': 'ChIJuVNr6ke6LIQRZl0cnFsmrDM', 'longitude': -100.813475}> of class 'Place': Expected POINT defined over wgs-84, got <class 'neo4j.v1.types.spatial.WGS84Point'

I think that these srid's are mixed

Because in Neo4j docs:

• WGS-84-3D: 4979
• WGS-84: 4326

Content related to issue #378

Also, I think it may be related, but I have a StructuredNode called User but when neomodel tries to inflate it using the registry, it says that there's no objects that match the label.

The StructuredNode definition is like the following:

class User(StructuredNode):
    uid = UniqueIdProperty()
    username = StringProperty(unique_index=True, required=False)
    email = EmailProperty(unique_index=True)
    name = StringProperty()
    last_name = StringProperty()
    password = StringProperty()
    profile_picture_filename = StringProperty()
    facebook_id = StringProperty()
    last_login = DateTimeProperty(default_now=True)
    firebase_device_id = neomodel.StringProperty(required=True)

    _password = None

    @property
    def profile_picture_url(self):
        return default_storage.url("/profiles/{filename}".format(filename=self.profile_picture_filename))

    @property
    def is_anonymous(self):
        """
        Always return False. This is a way of comparing User objects to
        anonymous users.
        """
        return False

    @property
    def is_authenticated(self):
        """
        Always return True. This is a way to tell if the user has been
        authenticated in templates.
        """
        return True

    def set_password(self, raw_password):
        """
        Define la contraseña del usuario, generando un hash para el mismo.

        :param raw_password:
        :return:
        """
        self.password = make_password(raw_password)
        self._password = raw_password

    def check_password(self, raw_password):
        """
        Return a boolean of whether the raw_password was correct. Handles
        hashing formats behind the scenes.
        """

        def setter(raw_password):
            self.set_password(raw_password)
            # Password hash upgrades shouldn't be considered password changes.
            self._password = None
            self.save()

        return check_password(raw_password, self.password, setter)

    def set_unusable_password(self):
        # Set a value that will never be a valid hash
        self.password = make_password(None)

    def has_usable_password(self):
        """
        Return False if set_unusable_password() has been called for this user.
        """
        return is_password_usable(self.password)

    @property
    def incoming_friend_requests(self):
       """ommited"""
       pass

    @classmethod
    def normalize_username(cls, username):
        return unicodedata.normalize('NFKC', username) if isinstance(username, str) else username

[aanastasiou]

@apexJCL Thanks for letting me know, left a note about this and will correct once i am back, you are probably right about the mix up with the srids, in the meantime, please let me know of anything else and will do another iteration.

All the best

[aanastasiou]

@apexJCL For the problem mentioned in your edit:

Can you please run :schema in the neo4j browser and let me know if you see any indexes / constraints matching those expected by your class hierarchy?

If :schema reports "no indexes / no constraints", then, can you please run neomodel_install_labels from the command line as per this.

[apexJCL]

Schema on neo4j was loaded, but ran install_labels anyways and the problem persisted.

It may be related to the population of the registry, because when I run a shell and load a node, it works, but when running my server and prompting for an auth request, the registry seems empty.

This is the stracktrace:

Collapsed data referring to issue #378

Traceback (most recent call last):
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/util.py", line 150, in _object_resolution
    resolved_object = self._NODE_CLASS_REGISTRY[frozenset(a_result_attribute[1].labels)].inflate(
KeyError: frozenset({'User'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/carlos/<redacted>/src/core/auth/middleware.py", line 20, in __call__
    user = authenticate(request=request)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
    user = backend.authenticate(request, **credentials)
  File "/home/carlos/<redacted>/src/core/auth/backends.py", line 22, in authenticate
    user = User.nodes.get_or_none(uid=data['user_id'])
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/match.py", line 564, in get_or_none
    return self.get(**kwargs)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/match.py", line 548, in get
    result = self._get(limit=2, **kwargs)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/match.py", line 539, in _get
    return self.query_cls(self).build_ast()._execute()
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/match.py", line 445, in _execute
    results, _ = db.cypher_query(query, self._query_params, resolve_objects=True)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/util.py", line 32, in wrapper
    return func(self, *args, **kwargs)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/util.py", line 202, in cypher_query
    results = self._object_resolution(results)
  File "/home/carlos/.virtualenvs/src-giY_RzAA/lib/python3.7/site-packages/neomodel/util.py", line 162, in _object_resolution
    raise ModelDefinitionMismatch(a_result_attribute[1], self._NODE_CLASS_REGISTRY)
neomodel.exceptions.ModelDefinitionMismatch: Node with labels User does not resolve to any of the known objects

It seems that the registry isn't being populated at the time of the request, but we can track this on another issue, I'll investigate further to check if it's something related to Django or Graphene (part of my current stack).

[aanastasiou]

Indeed, please see this too, might be related.

[apexJCL]

Yeah, it's related to that, because I'm still running 3.2.9 on the original version of my server (without using your repo/branch)

edit: specified the server's neomodel version is running in production.

[aanastasiou]

@apexJCL not if you used the repo I shared with you because it's a branch on the latest version which is 3.3.0. But as you say, let's keep this thread for POINT and yeah, you are more than welcome to contribute to the other one for the registry problem too.

[aanastasiou]

@apexJCL Please see this. I have corrected some details here and there, including the srid discrepancy you flagged, provided proper tests and the module is now also compatible with python2.7.

If you don't find any major problems with the functionality of PointProperty then I am going to queue these changes for the next release.

I now have a better idea of what may be causing #378 too, for a possible temporary fix please see this link.

All the best

[apexJCL]

PointProperty seems to be working as expected, points are being loaded/saved/removed.

Regarding #378 , I'll give it a go with the temporary fix.

Thanks!

[aanastasiou]

@apexJCL Just a quick update regarding #378, please see here.

[aanastasiou]

@apexJCL Please note, NeomodelPoint, PointProperty will most likely appear under neomodel.contrib.spatial_properties. For more information please see here

[apexJCL]

Awesome, thanks!