SERAC activities filtering does not work

[mgerbaux]

filtering xreports on activities returns no result : https://www.camptocamp.org/xreports?act=skitouring
maybe linked with the reorganization of activities dedicated to SERAC (c0f7e60 ?)

[ataraxido]

@momomaniac @cbeauchesne on my local dev API I get a PGSQL error :

psycopg2.ProgrammingError: column xreports.event_activity does not exist

And also elasticsearch, when searching xreports, returns

"activities": [
"ice_climbing"
],

instead of

"event_activity": "ice_climbing"

Could you tell me if the dev is iso and the issue is also in production or if it's the dev which isn't up to date and the migration I should roll ?

[brunobesson]

I am not working on the backend side. But having activities as an array rather than a single one is a good sign that some things were missed with last SERAC migration.
@momomaniac ?

[ataraxido]

The thing is that I found the migration script of this field, that's why I'm wondering if it's only on my dev or also in production ; cc @cbeauchesne

[cbeauchesne]

It's also in prod. But my guess is that the elastic search part of that change has not be done

[ataraxido]

Ah yes, indeed, what I saw is the sql migration
But if it has been ran in production you shouldn't get this on production environment, right ?

psycopg2.ProgrammingError: column xreports.event_activity does not exist

[cbeauchesne]

Yes, strange, I expect sawing an elasticsearch error Oo

Is it the same error on prod, on a your dev env ?

[ataraxido]

Sorry @cbeauchesne but I didn't understand your message.

The file we are talking : https://github.com/c2corg/v6_api/blob/master/alembic_migration/versions/06d2a35e39c8_improve_serac_database.py ; haven't been ran ?

[cbeauchesne]

Pretty sure it has ran, it's automatic.

I need to make some tests, I'll try to find some timme this week

[ataraxido]

Any news on this @cbeauchesne ?

[momomaniac]

Hello @ataraxido , nice to see you on board, I've not been around a lot recently.

I checked the db on demo and prod, they both do include the SERAC modifications:

demo:

# alembic history
06d2a35e39c8 -> bece9007ab83 (head), Add documents tags
85a5ed3c76a8 -> 06d2a35e39c8, Improve SERAC database
83956b269661 -> 85a5ed3c76a8, Add index on AssociationLog
24f8da659c78 -> 83956b269661, Add rate limiting structure
etc.

prod:

# alembic history                                                                           
85a5ed3c76a8 -> 06d2a35e39c8 (head), Improve SERAC database
83956b269661 -> 85a5ed3c76a8, Add index on AssociationLog
etc.

so I'm also quite surprised about the sqlalchemy error.
I'll do some further checks and try to find out what's going on.

[momomaniac]

for your local dev environment, you should check your docker-compose logs for alembic errors, maybe some db migration has gone wrong locally.

you can check the alembic version on your composition with:

docker-compose exec api alembic current
docker-compose exec api alembic history

or you could try to force update the DB locally:

docker-compose exec api alembic upgrade head

then you might see useful error messages.

[momomaniac]

I checked the filters on the demo server,
https://api.demov6.camptocamp.org/xreports?act=skitouring
it seems to work all right, but it does not work on the prod server
https://api.camptocamp.org/xreports?act=skitouring

actually the elastic search mapping is up to date on demo, but not on prod (prod still has the old index definitions with field activities instead of event_activity)
So the ES mapping has not been updated correctly when putting the feature in prod. I found ES migration scripts in https://github.com/c2corg/v6_api/tree/master/es_migration, but I don't know if this is still up to date... Or is there any automatic way to update the ES mappings ???

[brunobesson]

I did not update the syncer for the update following your work I wasn't aware of the need. So let's make the next update (which will update the syncer) and see if everything gets solved.

[brunobesson]

OK, the syncer seems ok now. But only new activities are taken into account.
Would @momomaniac or @cbeauchesne be able to write the code to run once the query to index all previous serac reports?
I might be able to do it, but as I do not write any python this could take a lot of useless time...

[momomaniac]

hello Bruno, yes, I'll have a quick look

…

On Mon, Jan 4, 2021 at 9:26 AM Bruno Besson ***@***.***> wrote: OK, the syncer seems ok now. But only new activities are taken into account. Would @momomaniac <https://github.com/momomaniac> or @cbeauchesne <https://github.com/cbeauchesne> be able to write the code to run once the query to index all previous serac reports? I might be able to do it, but as I do not write any python this could take a lot of useless time... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#909 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANY5RCSG2QAYY37KKGQ3FYTSYF3TBANCNFSM4TTYA7KQ> .

[ataraxido]

Any news ? :)

[loicperrin]

Up ! @momomaniac

[brunobesson]

Up ! @momomaniac

[momomaniac]

oups, toujours aussi désolé. ce WE je ne pourrai pas regarder, j'essaierai de trouver un peu de temps la semaine prochaine.

…

On Thu, Apr 1, 2021 at 2:50 PM Bruno Besson ***@***.***> wrote: Up ! @momomaniac <https://github.com/momomaniac> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#909 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANY5RCWZJWEK4EQ3KLQBICTTGRTZDANCNFSM4TTYA7KQ> .

[ataraxido]

C'est pour la blague hein ;)

[loicperrin]

Up @momomaniac ! Si tu as un peu de temps, ce serait cool de débloquer ce sujet.