dbtoyaml failure #182

excavador · 2018-02-21T21:07:19Z

python 3.6
osx

➜ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pgdbconn (0.8.0)
pip (9.0.1)
psycopg2 (2.7.4)
Pyrseas (0.8.0)
PyYAML (3.12)
setuptools (38.5.1)
wheel (0.30.0)

Traceback (most recent call last):
  File "/Users/oleg/XXX/pyrseas/bin/dbtoyaml", line 11, in <module>
    sys.exit(main())
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbtoyaml.py", line 49, in main
    dbmap = db.to_map()
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/database.py", line 487, in to_map
    dbmap.update(self.db.schemas.to_map(self.db, opts))
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 355, in to_map
    schemas.update(self[sch].to_map(db, self, opts))
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 104, in to_map
    schobjs.append((obj, obj.to_map(db, dbschemas, opts)))
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/table.py", line 525, in to_map
    tbls[self.foreign_keys[k.name].ref_table].column_names()))
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 486, in to_map
    dct = super(ForeignKey, self).to_map(db)
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/__init__.py", line 368, in to_map
    deps -= self.get_implied_deps(db)
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 564, in get_implied_deps
    idx = self._find_referenced_index(db, self._references)
  File "/Users/oleg/XXX/pyrseas/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 576, in _find_referenced_index
    if uc.columns == self.ref_cols:
AttributeError: 'dict' object has no attribute 'columns'

The text was updated successfully, but these errors were encountered:

excavador · 2018-02-21T21:20:42Z

I do not have simple repro case, but I inserted pretty-print and found following:

pyrseas/dbobject/constraint.py

578       for uc in ref_table.unique_constraints.values(): # my code
579             print(uc) # my code
580         for uc in list(ref_table.unique_constraints.values()):
581             if uc.columns == self.ref_cols:
582                 return uc

There is single value:

{'columns': ['some_column_name']}

i.e. you have dict instead of object.

Probably it helps.

jmafc · 2018-02-21T21:40:16Z

I've reproduced the problem. The fix is to replace the if uc.columns by if uc["columns"]. Later today I'll make sure nothing else gets broken by the change.

excavador · 2018-02-21T21:41:12Z

@jmafc thank you very much for quick response!

jmafc · 2018-02-21T22:19:43Z

It needs more work (your other issue causes another error with this change), but it's solvable.

jmafc · 2018-02-22T22:37:21Z

This appears to be a difference in behavior between Python 2 and 3. Although your traceback shows 3.6, I'm wondering if you may be running it from Python 2 (in issue #184, it seems evident that is the case because under Python 3, we're able to show the view definition formatted, not as a single string).

excavador · 2018-02-22T22:40:07Z

➜ pip3.6 install pyrseas
Collecting pyrseas
  Using cached Pyrseas-0.8.0-py2.py3-none-any.whl
Collecting PyYAML>=3.10 (from pyrseas)
Collecting pgdbconn>=0.8.0 (from pyrseas)
Collecting psycopg2>=2.5 (from pyrseas)
  Using cached psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: PyYAML, psycopg2, pgdbconn, pyrseas
Successfully installed PyYAML-3.12 pgdbconn-0.8.0 psycopg2-2.7.4 pyrseas-0.8.0
➜ dbtoyaml -H 127.0.0.1 -U postgres my_db_name
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/dbtoyaml", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbtoyaml.py", line 49, in main
    dbmap = db.to_map()
  File "/usr/local/lib/python3.6/site-packages/pyrseas/database.py", line 487, in to_map
    dbmap.update(self.db.schemas.to_map(self.db, opts))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 355, in to_map
    schemas.update(self[sch].to_map(db, self, opts))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 104, in to_map
    schobjs.append((obj, obj.to_map(db, dbschemas, opts)))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/table.py", line 525, in to_map
    tbls[self.foreign_keys[k.name].ref_table].column_names()))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 486, in to_map
    dct = super(ForeignKey, self).to_map(db)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/__init__.py", line 368, in to_map
    deps -= self.get_implied_deps(db)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 564, in get_implied_deps
    idx = self._find_referenced_index(db, self._references)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 576, in _find_referenced_index
    if uc.columns == self.ref_cols:
AttributeError: 'dict' object has no attribute 'columns'

excavador · 2018-02-22T22:40:44Z

➜ python3.6 /usr/local/bin/dbtoyaml -H 127.0.0.1 -U postgres my_db_name
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/dbtoyaml", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbtoyaml.py", line 49, in main
    dbmap = db.to_map()
  File "/usr/local/lib/python3.6/site-packages/pyrseas/database.py", line 487, in to_map
    dbmap.update(self.db.schemas.to_map(self.db, opts))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 355, in to_map
    schemas.update(self[sch].to_map(db, self, opts))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/schema.py", line 104, in to_map
    schobjs.append((obj, obj.to_map(db, dbschemas, opts)))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/table.py", line 525, in to_map
    tbls[self.foreign_keys[k.name].ref_table].column_names()))
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 486, in to_map
    dct = super(ForeignKey, self).to_map(db)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/__init__.py", line 368, in to_map
    deps -= self.get_implied_deps(db)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 564, in get_implied_deps
    idx = self._find_referenced_index(db, self._references)
  File "/usr/local/lib/python3.6/site-packages/pyrseas/dbobject/constraint.py", line 576, in _find_referenced_index
    if uc.columns == self.ref_cols:
AttributeError: 'dict' object has no attribute 'columns'

excavador · 2018-02-22T22:41:56Z

➜ head -n 2 /usr/local/bin/dbtoyaml
#!/usr/local/opt/python3/bin/python3.6

➜ /usr/local/opt/python3/bin/python3.6 -V
Python 3.6.4

excavador · 2018-02-22T22:42:08Z

@jmafc seems like it's python 3.6

excavador · 2018-02-22T22:47:10Z

➜ python2.7 /usr/local/bin/dbtoyaml -H 127.0.0.1 -U postgres my_db_name
Traceback (most recent call last):
  File "/usr/local/bin/dbtoyaml", line 7, in <module>
    from pyrseas.dbtoyaml import main
ImportError: No module named pyrseas.dbtoyaml

➜ pip2.7 install pyrseas
Collecting pyrseas
  Using cached Pyrseas-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: psycopg2>=2.5 in /usr/local/lib/python2.7/site-packages (from pyrseas)
Collecting pgdbconn>=0.8.0 (from pyrseas)
  Using cached pgdbconn-0.8.0.tar.gz
Requirement already satisfied: PyYAML>=3.10 in /usr/local/lib/python2.7/site-packages (from pyrseas)
Building wheels for collected packages: pgdbconn
  Running setup.py bdist_wheel for pgdbconn ... done
  Stored in directory: /Users/oleg/Library/Caches/pip/wheels/46/b2/c1/c36d224736917e247daf3e9aed80dd477d994ac1768773cd01
Successfully built pgdbconn
Installing collected packages: pgdbconn, pyrseas
Successfully installed pgdbconn-0.8.0 pyrseas-0.8.0

➜ head -n 2 /usr/local/bin/dbtoyaml
#!/usr/local/opt/python/bin/python2.7

➜ python2.7 /usr/local/bin/dbtoyaml -H 127.0.0.1 -U postgres my_db_name
Traceback (most recent call last):
  File "/usr/local/bin/dbtoyaml", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbtoyaml.py", line 49, in main
    dbmap = db.to_map()
  File "/usr/local/lib/python2.7/site-packages/pyrseas/database.py", line 487, in to_map
    dbmap.update(self.db.schemas.to_map(self.db, opts))
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/schema.py", line 355, in to_map
    schemas.update(self[sch].to_map(db, self, opts))
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/schema.py", line 104, in to_map
    schobjs.append((obj, obj.to_map(db, dbschemas, opts)))
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/table.py", line 525, in to_map
    tbls[self.foreign_keys[k.name].ref_table].column_names()))
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/constraint.py", line 486, in to_map
    dct = super(ForeignKey, self).to_map(db)
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/__init__.py", line 368, in to_map
    deps -= self.get_implied_deps(db)
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/constraint.py", line 564, in get_implied_deps
    idx = self._find_referenced_index(db, self._references)
  File "/usr/local/lib/python2.7/site-packages/pyrseas/dbobject/constraint.py", line 576, in _find_referenced_index
    if uc.columns == self.ref_cols:
AttributeError: 'dict' object has no attribute 'columns'

exactly the same problem if I'm running under python2.7

jmafc · 2018-02-23T03:09:58Z

I'm getting different behavior under 2.7 and 3.6, depending on whether I used my own test (which didn't have a function or view but just two tables) or your test case for the other two issues. It seems that under 2.7 (with my test), one table gets mapped before the other and under 3.6, it's reversed (since the "maps" are dicts, this is not unusual). The problem is the Table.to_map code calls the UniqueConstraint.to_map code to get the column names (from column numbers) and what was a Python object gets converted to a dict in the process. We'll probably have to normalize the column names independently, like we already do for Constraint.alter.

excavador · 2018-02-23T11:07:27Z

@jmafc behavior depends from dist key traversal order.
On my local machine I have the same behavior, you have the different, but problem the same.
Both observation (my python 2.7/3.6 and your) address to "undefined dict key order traversal"

jmafc · 2018-02-28T02:27:55Z

This is taking a little longer than expected. The problem is when ForeignKey.get_implied_deps calls ForeignKey._find_referenced_index. By definition, the latter cannot return a dict, because the item is going to be added to a set. What has to be returned is a Python object, but this is happening in the context of several to_map methods: the table, the foreign keys and the unique constraints and some of the output dicts are apparently being confused with the objects, because the dicts have a very similar structure.

excavador · 2018-02-28T02:28:59Z

@jmafc thank you for keep me updated!

excavador · 2018-03-02T00:02:28Z

@jmafc verified on my case, works fine, thank you!
Would be great also receive new release in pypi :)

jmafc · 2018-03-02T01:03:27Z

Yes, I suspected you would say that. We don't really have a maintenance release policy, but in the past we typically waited until we had three or more fixes in the maintenance branch and also until someone asked "could we have this or that fix?" Right now, this is the only change in r0.8, and although there are other file changes, the real fix--if I'm not mistaken--is in the two lines in DbObject.to_map. So for now, I'll wait for a while.

excavador · 2018-03-02T03:55:13Z

No problem.
I can use pip install git+https://github.com/perseas/Pyrseas for a while.
Thank you!

jmafc · 2018-03-02T04:12:21Z

Please be aware that using the top level directory gives you, by default, the master branch (which now also includes many non-user tested changes for #176. If you want just what will become v0.8.1, you'll probably want to install from https://github.com/perseas/Pyrseas/tree/r0.8

jmafc added the dbtoyaml label Feb 21, 2018

jmafc closed this as completed in cfc7f95 Feb 28, 2018

jmafc mentioned this issue Dec 21, 2021

Faster deepcopy #237

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dbtoyaml failure #182

dbtoyaml failure #182

excavador commented Feb 21, 2018 •

edited

Loading

excavador commented Feb 21, 2018 •

edited

Loading

jmafc commented Feb 21, 2018

excavador commented Feb 21, 2018

jmafc commented Feb 21, 2018

jmafc commented Feb 22, 2018

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018 •

edited

Loading

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018

jmafc commented Feb 23, 2018

excavador commented Feb 23, 2018 •

edited

Loading

jmafc commented Feb 28, 2018

excavador commented Feb 28, 2018

excavador commented Mar 2, 2018 •

edited

Loading

jmafc commented Mar 2, 2018

excavador commented Mar 2, 2018 •

edited

Loading

jmafc commented Mar 2, 2018

dbtoyaml failure #182

dbtoyaml failure #182

Comments

excavador commented Feb 21, 2018 • edited Loading

excavador commented Feb 21, 2018 • edited Loading

jmafc commented Feb 21, 2018

excavador commented Feb 21, 2018

jmafc commented Feb 21, 2018

jmafc commented Feb 22, 2018

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018 • edited Loading

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018

excavador commented Feb 22, 2018

jmafc commented Feb 23, 2018

excavador commented Feb 23, 2018 • edited Loading

jmafc commented Feb 28, 2018

excavador commented Feb 28, 2018

excavador commented Mar 2, 2018 • edited Loading

jmafc commented Mar 2, 2018

excavador commented Mar 2, 2018 • edited Loading

jmafc commented Mar 2, 2018

excavador commented Feb 21, 2018 •

edited

Loading

excavador commented Feb 21, 2018 •

edited

Loading

excavador commented Feb 22, 2018 •

edited

Loading

excavador commented Feb 23, 2018 •

edited

Loading

excavador commented Mar 2, 2018 •

edited

Loading

excavador commented Mar 2, 2018 •

edited

Loading