Skip to content

Conversation

@nailo2c
Copy link
Contributor

@nailo2c nailo2c commented May 13, 2025

Closes: #33076

This PR adds a create_collection() function to the MongoHook which refers to pymongo.database.Database.create_collection.

Before:

mongo_hook = MongoHook(conn_id="mongo_default")
mongo_conn = mongo_hook.get_conn()
mongo_db = mongo_conn["db_name"]

# create time collection here
mongo_db.create_collection("time_test", timeseries= {
    "timeField": "timestamp",
    "metaField": "metadata",
    "granularity": "hours"
})

# use the collection
mongo_collection = mongo_db["time_test"]

# add document
mongo_collection.insert_one(
  {
    "metadata": { "sensorId": 5578, "type": "temperature" },
    "timestamp": dt.utcnow(),
    "temp": 12
  }
)

After:

mongo_hook = MongoHook(conn_id="mongo_default")

# create (or get) the time-series collection in one call
mongo_collection = mongo_hook.create_collection(
    name="time_test",
    timeseries={
        "timeField": "timestamp",
        "metaField": "metadata",
        "granularity": "hours",
    },
)

# add document
mongo_collection.insert_one(
    {
        "metadata": {"sensorId": 5578, "type": "temperature"},
        "timestamp": dt.utcnow(),
        "temp": 12,
    }
)

I’ve completed an end-to-end test on my local Airflow instance, and everything worked as expected.
截圖 2025-05-12 下午7 28 06

@nailo2c nailo2c force-pushed the feature/add-create_collection-to-MongoHook branch from 381326d to 1a5a9e0 Compare May 13, 2025 04:51
@potiuk potiuk force-pushed the feature/add-create_collection-to-MongoHook branch from 1a5a9e0 to 91fa706 Compare May 13, 2025 10:15
@potiuk potiuk closed this May 13, 2025
@potiuk potiuk reopened this May 13, 2025
@potiuk
Copy link
Member

potiuk commented May 13, 2025

Closed / reopened to rebuild after rebase to see if static check still fails

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@eladkal eladkal changed the title feat: add create_collection function with unit tests feat: add create_collection function to MongoHook May 20, 2025
@potiuk potiuk merged commit d58a0f2 into apache:main May 22, 2025
64 checks passed
dadonnelly316 pushed a commit to dadonnelly316/airflow that referenced this pull request May 26, 2025
* feat: add create_collection function with unit tests

* rm comments

* fix static checks

* make the type of `create_kwargs` clearer

Co-authored-by: Wei Lee <weilee.rx@gmail.com>

* modfiy variable name: create_if_exists -> return_if_exists

* move import CollectionInvalid to top

---------

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
* feat: add create_collection function with unit tests

* rm comments

* fix static checks

* make the type of `create_kwargs` clearer

Co-authored-by: Wei Lee <weilee.rx@gmail.com>

* modfiy variable name: create_if_exists -> return_if_exists

* move import CollectionInvalid to top

---------

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[MongoHook] support creation of time series collections

3 participants