Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Django 2.0 support #344

Open
fijter opened this issue Nov 16, 2017 · 30 comments
Open

Django 2.0 support #344

fijter opened this issue Nov 16, 2017 · 30 comments

Comments

@fijter
Copy link

fijter commented Nov 16, 2017

It won't take long before Django 2.0 is released which is currently in beta. I've tried out HVAD with it and it doesn't seem compatible, at least not the admin part; I seem to be getting this error on a list overview:

_clone() got an unexpected keyword argument '_forced_unique_fields'

It would be nice if I could continue using HVAD after upgrading to 2.0 once it gets released.

@spectras
Copy link
Collaborator

spectras commented Nov 16, 2017

Hello!

You are perfectly right, it's not compatible right now.
Due to lack of time, I only work on hvad to keep it compatible those days. I did not follow Django release cycle lately, if Django 2.0 is due soon, it's high time I put compatibility on my todo list.

No ETA yet, but I push this to the top of my list. Thanks for reminding me!

@claudep
Copy link

claudep commented Nov 30, 2017

For information, this is due to this Django commit: django/django@66933a6
So I guess _chain should be used instead of _clone for Django >= 2.0

@spectras
Copy link
Collaborator

spectras commented Dec 1, 2017

Okay, it seems we reached a turning point in hvad history.

  • On the one hand, there have been many changes in Django 2.0 internals. They break a lot if things in current hvad codebase.
  • On the other hand, hvad2, while still incomplete, works well with Django 2.0. This is because it uses a completely different approach from hvad1.

Thus the dilemma I now face: should I spend lengthy hours fixing hvad1 compatibility, or should I release hvad2 despite it being incomplete?

Would you be willing to try hvad2 in its current state? You may install it through pip install https://github.com/KristianOellegaard/django-hvad/archive/dev/hvad2.tar.gz. It requires Django 2.0 and python 3.

Most notable changes from hvad 1 are:

  • AUTOLOAD_TRANSLATIONS setting is now False by default.
  • Instanciating a translatable model now always creates a translation, unless explicitly told not to.
  • Accessing the language_code attribute never loads a translation. It returns None if no translation is loaded.
  • A new introspection API has replaced the incoherent set of methods and functions:
    • instance.translations.active replaces get_cached_translation(instance).
    • instance.translations.all_languages() replaces instance.get_available_languages().
    • getattr(instance.translations.active, name) replaces instance.safe_translation_getter(name).
    • lazy_translation_getter is gone.
  • Under the hood, the queries generated by hvad2 have changed to use a true JOIN clause with the joining conditions directly embedded. This dramatically improves performance, but causes issues when using several separate filter clauses on the same table. This is the part that still needs work, though it's only triggered by edge cases.

@spectras
Copy link
Collaborator

I have been looking some more, the ramifications of Django 2 changes run even deeper than I thought.

This means given the little time I have, making hvad1 work with Django2 is close to unfeasible without some external contribution. So I guess I'm heading towards this approach:

  • Consider hvad 1.8 the final hvad1 release, supporting Django1 until Django 1.11 LTS expires.
  • Introduce hvad 2.0 starting from Django 2.0.

I am worried because the change is somewhat disruptive to existing code bases.
I hoped it could be introduced in a softer manner, slowly phasing out deprecated methods and gathering feedback on the new API instead of forcing it onto user projects.

@claudep
Copy link

claudep commented Jan 15, 2018

I guess you're going to merge the hvad2 branch into master soon? Are they blocker to make a pre-release?

@spectras
Copy link
Collaborator

spectras commented Jan 15, 2018

Hello,

It's a good question, I am still worried it is not as polished I wished it to be. Yet I don't have time to work on it at the moment. Perhaps a good compromise would be releasing it as is and tagging it as an alpha release.

As for points that still need work:

Won't make it to v2: most of #249. Those were the actual reasons the refactor was initiated for, so it's a pity they won't make it. But the refactor in itself brings many improvements to the core, most notably the change to actual join filters instead of regular filters.

At least this will lay the ground that will allow those things to be added incrementally at a later point.

@claudep
Copy link

claudep commented Jan 15, 2018

Yes, I think that with a pre-release you'll more likely get the expected reports for corner cases…

@claudep
Copy link

claudep commented Mar 27, 2018

I tried the dev/hvad2 branch with Django 2.0, but still got an error:
TypeError: _clone() got an unexpected keyword argument '_local_field_names'

Any idea?

@elenaoat
Copy link

Hello, I am getting an assertion error
assert django.VERSION < (1, 10)
on django-hvad-1.8.0 with django 1.10
So I guess not even this version of Django is supported?

@rasca
Copy link
Contributor

rasca commented Sep 23, 2018

Hi @spectras . I'm updating a big app to the latest Django.

We are deciding which way to take for translations given that hvad doesn't support Dango>=2 yet.

Could you please explain us what the current status is? Both for the master branch and the hvad2 branch. We might put in the work to finish hvad2 but we need some guidance. #273 talks about some design decisions needed and corner cases. Where can I read about these?

@tback
Copy link

tback commented Oct 6, 2018

Hey @rasca: I fixed the tests for django 2.1 postgres on https://github.com/tback/django-hvad/tree/dev/hvad2 . I don't know if that works or where it will take me, I just thought you might be interested in any work that is going on.

@chakmear
Copy link

Hi @spectras I just started updating several apps of mine to Django 2, most of them using hvad, all of them with a lot of translated data but I'm getting to the same dead end as others, hvad not supporting Django 2... Can you tell us about the current status? And is an update to "hvadDjango2Compatible" possible without loosing data? Thank you very much!

@spectras
Copy link
Collaborator

spectras commented Oct 26, 2018

Hello everyone,

I am really sorry hvad has gotten out of focus lately. As a matter of fact, maintaining hvad as django changes is quite some work, and I got discouraged by the sheer amount of things that broke with Django 2.0. Especially as I had hvad almost working with the beta release, and multiple breaking changes were added right before the final release, at a time I had no free time at all.

Now, I understand many large and smaller projects depend on hvad and I don't want hvad to be yet another argument against open-source dependencies. Here is what I intend to do:

  • I will devote a fair share of my free time to hvad in the next few weeks.
  • By mid-November, say 18th, depending on my findings, either have hvad running on Django 2.1, or publish a clear end-of-life.

Thanks for bearing the lack of news. I will look what people posted here and the other issues, especially @tback 's passing tests, which look really promising.

Also, if someone is interested in stepping up and becoming an official hvad maintainer, there is definitely a gap to fill :)

@spectras
Copy link
Collaborator

spectras commented Nov 18, 2018

Good news, I have a version that passes all tests on Django 2.1 on mysql, postgres and sqlite.

As it turns out, the errors were scary, but they stemmed from a smaller-than-expected number of roots. Always frustrating when so many hours end up in so little actual changes in the code.

As hvad2.0 introduces breaking changes in the API, I need to write up on those.
Especially as no solution was found for a few problematic corner cases identified in #273, so in the meantime they will need to be thoroughly documented. There is also some cleaning to do.

So it is unlikely I can release today, but expect it within the upcoming week.

@bahattincinic
Copy link

@spectras When will you release new stable version? I saw that you released a beta version. But I'm not sure that is it works correctly. Do you think should I wait for the stable version?

@bahattincinic
Copy link

bahattincinic commented Dec 30, 2018

I tested the beta version. When I try to use python manage.py dumpdata with the translatable model. I got the following error;

CommandError: Unable to serialize database: 'Model' object has no attribute '_hvad_query'

i think, it related to the Django serializers. i got the same error.

from django.core import serializers
serializers.serialize(queryset=Model.objects.all(), format='json')

Very Hacky Solution:

https://github.com/django/django/blob/master/django/core/serializers/python.py#L59

if you add following code in the else condition it works;

if field.name == '_hvad_query':
    return

@spectras
Copy link
Collaborator

spectras commented Jan 2, 2019

Hello, the beta version is a release, I keep it beta precisely so people can test it and provide feedback, just like you did. I'll mark it as stable if it goes without issues for long enough.

By the way, if you could repost that exact message as a separate issue, it would be great!

@bahattincinic
Copy link

bahattincinic commented Feb 1, 2019

Also, I faced with a different bug.

When I try to add a new model, I got the following error while generating migrations.

python manage.py makemigrations
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/core/management/commands/makemigrations.py", line 170, in handle
    migration_name=self.migration_name,
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 44, in changes
    changes = self._detect_changes(convert_apps, graph)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 160, in _detect_changes
    self.generate_renamed_models()
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 462, in generate_renamed_models
    model_fields_def = self.only_relation_agnostic_fields(model_state.fields)
  File "/Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 100, in only_relation_agnostic_fields
    del deconstruction[2]['to']
KeyError: 'to'

Debugging Data:

> /Users/bahattincinic/Projects/personal/env/lib/python3.7/site-packages/django/db/migrations/autodetector.py(101)only_relation_agnostic_fields()
    100                 import ipdb; ipdb.set_trace()
--> 101                 del deconstruction[2]['to']
    102             fields_def.append(deconstruction)

ipdb> deconstruction
('hvad.fields.SingleTranslationObject', ['cms.Menu', 'cms.MenuTranslation'], {})

Django Version:

Django==2.1.4

@chrisflink
Copy link

I found that same bug when rebuilding all migrations from scratch. No solution but a workaround:

  1. Remove language fields from your model(s)
  2. Run python manage.py makemigrations
  3. Add language fields to your models
  4. Run python manage.py makemigrations again

This worked for me, hope it helps others too. I found no time for further debugging. Maybe good to know that in the models I use RichTextFields and Taggit tags, there might be a compatibility issue because I found this error on the taggit project: jazzband/django-taggit#206

Hope this helps debugging and thanks for HVAD!

@TheDeadOne
Copy link

Python 3.6.3
Django 2.1.7
django-hvad 2.0.0
django-cms 3.6.0

Traceback:

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\exception.py" in inner
  34.             response = get_response(request)

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in wrapper
  604.                 return self.admin_site.admin_view(view)(*args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\views\decorators\cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\sites.py" in inner
  223.             return view(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapper
  45.         return bound_method(*args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in changelist_view
  1675.             cl = self.get_changelist_instance(request)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in get_changelist_instance
  742.             sortable_by,

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\views\main.py" in __init__
  45.         self.root_queryset = model_admin.get_queryset(request)

File "D:\Projects\some-project\env3\lib\site-packages\hvad\admin.py" in get_queryset
  347.             qs = qs.order_by(*ordering)

File "D:\Projects\some-project\env3\lib\site-packages\hvad\manager.py" in order_by
  647.         return super(TranslationQueryset, self).order_by(*fieldnames)

File "D:\Projects\some-project\env3\lib\site-packages\django\db\models\query.py" in order_by
  1024.         obj = self._chain()

File "D:\Projects\some-project\env3\lib\site-packages\django\db\models\query.py" in _chain
  1163.         obj = self._clone()

File "D:\Projects\some-project\env3\lib\site-packages\hvad\manager.py" in _clone
  201.         return super(TranslationQueryset, self)._clone(**kwargs)

Exception Type: TypeError at /admin/news/news/
Exception Value: _clone() got an unexpected keyword argument 'shared_model'

and

Traceback:

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\exception.py" in inner
  34.             response = get_response(request)

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "D:\Projects\some-project\env3\lib\site-packages\django\core\handlers\base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in wrapper
  604.                 return self.admin_site.admin_view(view)(*args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\views\decorators\cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\sites.py" in inner
  223.             return view(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in add_view
  1637.         return self.changeform_view(request, None, form_url, extra_context)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapper
  45.         return bound_method(*args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in changeform_view
  1525.             return self._changeform_view(request, object_id, form_url, extra_context)

File "D:\Projects\some-project\env3\lib\site-packages\django\contrib\admin\options.py" in _changeform_view
  1554.         ModelForm = self.get_form(request, obj, change=not add)

File "D:\Projects\some-project\env3\lib\site-packages\hvad\admin.py" in get_form
  172.         return translatable_modelform_factory(language, self.model, **defaults)

File "D:\Projects\some-project\env3\lib\site-packages\hvad\forms.py" in translatable_modelform_factory
  229.     klass = modelform_factory(model, form, *args, **kwargs)

Exception Type: TypeError at /admin/news/news/add/
Exception Value: modelform_factory() got an unexpected keyword argument 'change'

@mohammedhammoud
Copy link

Any progress on this one?

@kcleong
Copy link

kcleong commented Jun 6, 2019

Removing the change value from the defaults dictionary in hvad.admin.TranslatableAdmin.get_form fixes the modelform_factory() got an unexpected keyword argument change'` exception. See for example this commit: PythonUnited@9ffd352

I'm not sure what the change value does, so use this 'fix' at your own risk. Using this fix I got hvad working with Django 2.1.9.

In Django 2.1 the modelform_factory[1] method does not include the change argument, however in Django 2.0[2] I also do not see the change argument.

Where can I find the latest repo/branch for Django 2? The dev/hvad2 has been last updated 2 years ago. I've been using my own fork with fixes for Django 2.

  1. https://github.com/django/django/blob/2.1.9/django/forms/models.py#L473-L476
  2. https://github.com/django/django/blob/2.0/django/forms/models.py#L471-L474

@DmytroLitvinov
Copy link

Hello @kcleong .
This is the latest branch with compatibility for Django>2

Could you please share link to your own fork?

@kcleong
Copy link

kcleong commented Jun 6, 2019

@DmytroLitvinov, thanks for the pointer to the Django 2.x branch. This is the fork I am using PythonUnited/django-hvad

@silau2005
Copy link

I like django-hvad a lot, it works amazingly with RESTful API but the recent Django 2.0 upgrading in our project really brings some operation issues.
I would like to share some tips here, for people with simple Django use case, you might considering a short term alternative django-parler, it was inspiring by hvad thus compitable with hvad as well, especially on ORM/model part.

For others use a lot of RESTful API and want to stay with hvad's awesome serializer, try to avoid using
.language and get_translation_aware_manager. then you are fine. example here

this is a temporary fix till the official hvad 2.0, cheers

@maltebeckmann
Copy link

maltebeckmann commented Jan 19, 2020

@silau2005 Your article saved me. What a beautiful short-term solution. 多谢你!真没想到django-hvad到现在还没更新过。

@mroobert
Copy link

mroobert commented Jun 9, 2020

@silau2005 the link is broken... and I would need so much to see how I can avoid using .language call... Can you point me to another link or something.. :D thx!

@silau2005
Copy link

@silau2005 the link is broken... and I would need so much to see how I can avoid using .language call... Can you point me to another link or something.. :D thx!

@mroobert I just fixed the link, you might check it again. Wish it helps. Let me know your result. cheers

@mroobert
Copy link

mroobert commented Jun 10, 2020

@silau2005 thx <3, now the link is working

One more question... when you use this type of code to avoid the .language call :

def retrieve(self, request, pk=None):
        lang = self.request.query_params.get('lang', DEFAULT_LANGUAGE)
        queryset = LocalizedString.objects.filter(namespace__title=pk)
        queryTranslations = LocalizedString.translations.field.model.objects.filter(language_code=lang, master__in=queryset)
        serializer = LocalizedStringSerializer(queryTranslations, many=True)
        strings = {d['key']: d['value'] for d in serializer.data}
        return Response(strings)

and after that you pass the queryTranslations to the serializer:

serializer = LocalizedStringSerializer(queryTranslations, many=True)
strings = {d['key']: d['value'] for d in serializer.data}
return Response(strings)

This is triggering an error in serializer.data ===> /hvad/utils.py", line 70, in load_translation
trans_model = instance._meta.translations_model
AttributeError: 'Options' object has no attribute 'translations_model'

It is this the right way of using the serializer if you're avoiding .language call.. or ?

The serializer:

class LocalizedStringSerializer(TranslatableModelSerializer):
    key = serializers.SerializerMethodField()
    value = serializers.SerializerMethodField()

    def get_key(self, obj):
        return obj.title

    def get_value(self, obj):
        return obj.value

    class Meta:
        model = LocalizedString
        fields = ('key', 'value')

The model:

class LocalizedString(TranslatableModel):
    title = models.CharField(max_length=255, unique=True)
    translations = TranslatedFields(
        value=models.TextField('description'),
    )
    # placeholder = PlaceholderField('localizable_string_placeholder')
    namespace = models.ForeignKey(StringsNamespace, related_name='strings', null=True, on_delete=models.CASCADE)
    is_translatable = models.BooleanField(default=True, verbose_name=_("Translatable"))

    panels = [
        MultiFieldPanel(
            [
                FieldPanel('title'),
                FieldPanel('namespace', widget=forms.Select),
                FieldPanel('is_translatable', help_text=_(
                    "Makes this string translatable. Adds a non translatable comment to the exported string if "
                    "unchecked"
                )),
            ],
            heading=_('Localized String')
        )
    ]

    def get_translated(self, language_code):
        translated = type(self).objects.language(language_code).filter(pk=self.pk)
        if translated.exists():
            return translated.first()
        return super(LocalizedString, self).translate(language_code)

@silau2005
Copy link

This is triggering an error in serializer.data ===> /hvad/utils.py", line 70, in load_translation
trans_model = instance._meta.translations_model
AttributeError: 'Options' object has no attribute 'translations_model'

I will talk a little bit about this exception
django-hvad adds translations_model (pointing to Master model by foreign key master) behind the scenes for every TranslatableModel

LocalizedString.translations.field.model.objects.filter(language_code=lang, master__in=queryset)
is actually equivalent to
translations_model.objects.filter(language_code=lang, master__in=queryset)

and the serializer you are using is LocalizedStringSerializer, while the model/queryTranslations you put into the serializer is from translations_model

so they don't match each other

my suggestion is to modify

queryset = LocalizedString.objects.filter(namespace__title=pk)
queryTranslations = LocalizedString.translations.field.model.objects.filter(language_code=lang, master__in=queryset)
#serializer = LocalizedStringSerializer(queryTranslations, many=True)
#strings = {d['key']: d['value'] for d in serializer.data}
#v1 = queryset.objects.values()
#v2 = queryTranslations.objects.values()
#fields = queryset.model._meta.fields
#fieldsTranslations = queryTranslations..model._meta.fields

you could construct the dictionary from queryset and queryTranslations
getting dictionary by queryset.objects.values()
change the prefix if needed
and queryset.model._meta.fields might help to shorten the result

checking would be needed if the result should be a single dictionary instead of multiple dict from the querysey

wish the information helps, I don't have time to do deep investigation, good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests