Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add ability to create datasets using the gbq module #11121

Closed
wants to merge 1 commit into from
Closed

ENH: Add ability to create datasets using the gbq module #11121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions doc/source/api.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,7 @@ Google BigQuery

read_gbq
to_gbq
generate_bq_schema
create_table
delete_table
table_exists


.. currentmodule:: pandas

Expand Down
50 changes: 7 additions & 43 deletions doc/source/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4005,14 +4005,10 @@ The key functions are:
.. currentmodule:: pandas.io.gbq

.. autosummary::
:toctree: generated/
:toctree: generated/

read_gbq
to_gbq
generate_bq_schema
create_table
delete_table
table_exists
read_gbq
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisvandenbossche do these have to be indented?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's OK (they just have to be indented consistenly, but does not matter how many spaces)

to_gbq

.. currentmodule:: pandas

Expand Down Expand Up @@ -4078,8 +4074,7 @@ Assume we want to write a DataFrame ``df`` into a BigQuery table using :func:`~p

.. note::

If the destination table does not exist, a new table will be created. The
destination dataset id must already exist in order for a new table to be created.
The destination table and destination dataset will automatically be created if they do not already exist.

The ``if_exists`` argument can be used to dictate whether to ``'fail'``, ``'replace'``
or ``'append'`` if the destination table already exists. The default value is ``'fail'``.
Expand Down Expand Up @@ -4146,19 +4141,13 @@ For example:
often as the service seems to be changing and evolving. BiqQuery is best for analyzing large
sets of data quickly, but it is not a direct replacement for a transactional database.


Creating BigQuery Tables
''''''''''''''''''''''''

As of 0.17.0, the gbq module has a function :func:`~pandas.io.gbq.create_table` which allows users
to create a table in BigQuery. The only requirement is that the dataset must already exist.
The schema may be generated from a pandas DataFrame using the :func:`~pandas.io.gbq.generate_bq_schema` function below.

For example:

.. code-block:: python
.. warning::

gbq.create_table('my_dataset.my_table', schema, projectid)
As of 0.17, the function :func:`~pandas.io.gbq.generate_bq_schema` has been deprecated and will be
removed in a future version.

As of 0.15.2, the gbq module has a function :func:`~pandas.io.gbq.generate_bq_schema` which will
produce the dictionary representation schema of the specified pandas DataFrame.
Expand All @@ -4174,31 +4163,6 @@ produce the dictionary representation schema of the specified pandas DataFrame.
{'name': 'my_int64', 'type': 'INTEGER'},
{'name': 'my_string', 'type': 'STRING'}]}

Deleting BigQuery Tables
''''''''''''''''''''''''

As of 0.17.0, the gbq module has a function :func:`~pandas.io.gbq.delete_table` which allows users to delete a table
in Google BigQuery.

For example:

.. code-block:: python

gbq.delete_table('my_dataset.my_table', projectid)

The following function can be used to check whether a table exists prior to calling ``table_exists``:

:func:`~pandas.io.gbq.table_exists`.

The return value will be of type boolean.

For example:

.. code-block:: python

In [12]: gbq.table_exists('my_dataset.my_table', projectid)
Out[12]: True

.. note::

If you delete and re-create a BigQuery table with the same name, but different table schema,
Expand Down
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.17.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,8 +328,8 @@ has been changed to make this keyword unnecessary - the change is shown below.
Google BigQuery Enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Added ability to automatically create a table using the :func:`pandas.io.gbq.to_gbq` function if destination table does not exist. (:issue:`8325`).
- Added ability to automatically create a dataset using the :func:`pandas.io.gbq.to_gbq` function if destination dataset does not exist. (:issue:`11121`).
- Added ability to replace an existing table and schema when calling the :func:`pandas.io.gbq.to_gbq` function via the ``if_exists`` argument. See the :ref:`docs <io.bigquery>` for more details (:issue:`8325`).
- Added the following functions to the gbq module: :func:`pandas.io.gbq.table_exists`, :func:`pandas.io.gbq.create_table`, and :func:`pandas.io.gbq.delete_table`. See the :ref:`docs <io.bigquery>` for more details (:issue:`8325`).
- ``InvalidColumnOrder`` and ``InvalidPageToken`` in the gbq module will raise ``ValueError`` instead of ``IOError``.

.. _whatsnew_0170.enhancements.other:
Expand Down
Loading