Monitor client mounts in iml-device

[ip1981]

Example:

chroma=> select * from chroma_core_lustreclientmount;
 id |       state_modified_at       |  state  | immutable_state | not_deleted | mountpoint | content_type_id | filesystem | host_id
----+-------------------------------+---------+-----------------+-------------+------------+-----------------+---------------+---------
 16 | 2020-05-18 16:01:10.768484+00 | mounted | f               | t           | /mnt/fs    |              35 |             fs |       5
 17 | 2020-05-18 16:01:10.768484+00 | mounted | f               | t           | /mnt/fs2   |              35 |             fs2 |       5

may 19 10:18:29 adm.local iml-device[13087]: May 19 10:18:29.751 DEBUG iml_device: Client mounts at c1.local: [("fs2", Some("/mnt/fs2"))]
may 19 10:18:29 adm.local iml-device[13087]: May 19 10:18:29.756 DEBUG iml_device: Inserted ChromaCoreLustreclientmount { id: 17, state_modified_at: 2020-05-18T16:01:10.768484Z, state: "mounted", immutable_state: false, not_deleted: Some(true), mountpoint: Some("/mnt/fs2"), content_type_id: Some(35), filesystem_id: 2, host_id: 5 }
may 19 10:18:29 adm.local iml-device[13087]: May 19 10:18:29.759 DEBUG iml_device: Updated [ChromaCoreLustreclientmount { id: 16, state_modified_at: 2020-05-19T10:18:29.751826Z, state: "unmounted", immutable_state: false, not_deleted: Some(true), mountpoint: None, content_type_id: Some(35), filesystem_id: 1, host_id: 5 }]

chroma=> select * from chroma_core_lustreclientmount;
 id |       state_modified_at       |   state   | immutable_state | not_deleted | mountpoint | content_type_id | filesystem | host_id
----+-------------------------------+-----------+-----------------+-------------+------------+-----------------+---------------+---------
 17 | 2020-05-18 16:01:10.768484+00 | mounted   | f               | t           | /mnt/fs2   |              35 |             fs2 |       5
 16 | 2020-05-19 10:18:29.751826+00 | unmounted | f               | t           |            |              35 |             fs |       5

Closes #1800, #1801.

Signed-off-by: Igor Pashev pashev.igor@gmail.com

---

This change is 

[jgrund]

Instead of selecting over filesystems and using a crossjoin with unrelated tables, and doing a query per incoming mount,

you should select over the lustre client mount table (which will allow for matching inner joins to fs and host tables) and do the whole thing in a single batch upsert query.

[ip1981]

you should select over the lustre client mount table

How can I query an empty table?

[jgrund]

How can I query an empty table?

Possibly empty, and you'd be doing an upsert.

example:

integrated-manager-for-lustre/iml-orm/src/sfa.rs

Lines 260 to 274 in 6374408

	pub fn batch_upsert(xs: Vec<Self>) -> impl Executable {
	diesel::insert_into(Self::all())
	.values(xs)
	.on_conflict(sd::index)
	.do_update()
	.set((
	sd::child_health_state.eq(excluded(sd::child_health_state)),
	sd::failed.eq(excluded(sd::failed)),
	sd::health_state_reason.eq(excluded(sd::health_state_reason)),
	sd::health_state.eq(excluded(sd::health_state)),
	sd::member_index.eq(excluded(sd::member_index)),
	sd::member_state.eq(excluded(sd::member_state)),
	sd::enclosure_index.eq(excluded(sd::enclosure_index)),
	))
	}

[ip1981]

I need filesystem id and host id, which I can't know without querying other tables first, can I? I only have host name and filesystem name.

[ip1981]

It is still possible in this case to do upsert in a batch, but in raw SQL :)

[jgrund]

It is still possible in this case to do upsert in a batch, but in raw SQL :)

I think it's possible within the query builder as well

[jgrund]

Here's a high-level way to do the batch upsert on mounted clients (I checked that this compiles):

// This content_type_id query can be done once at startup, doesn't need to be inside the loop.
let content_type_id = djct::table
            .select(djct::id)
            .filter(djct::model.eq("lustreclientmount"))
            .first_async::<i32>(&pool)
            .await
            .optional()?;

        let host_id: Option<i32> = mgthost::table
            .select(mgthost::id)
            .filter(mgthost::fqdn.eq(host.to_string()))
            .first_async::<i32>(&pool)
            .await
            .optional()?;

        let host_id = match host_id {
            Some(x) => x,
            None => continue,
        };

        let xs: Vec<_> = lustre_mounts
            .into_iter()
            .map(|(fs, tg)| {
                (
                    clmnt::host_id.eq(host_id),
                    clmnt::filesystem_id.eq(1), // hardcoded because this needs to be filesystem_name instead
                    clmnt::mountpoint.eq(tg),
                    clmnt::state.eq("mounted"),
                    clmnt::state_modified_at.eq(chrono::offset::Utc::now()),
                    clmnt::immutable_state.eq(false),
                    clmnt::not_deleted.eq(true),
                    clmnt::content_type_id.eq(content_type_id),
                )
            })
            .collect();

        diesel::insert_into(clmnt::table)
            .values(xs)
            .on_conflict((clmnt::host_id, clmnt::filesystem_id, clmnt::not_deleted))
            .do_update()
            .set((
                clmnt::mountpoint.eq(excluded(clmnt::mountpoint)),
                clmnt::state.eq(excluded(clmnt::state)),
                clmnt::state_modified_at.eq(state_modified_at),
            ))
            .execute_async(&pool)
            .await?;

I didn't implement upsert for unmounted clients, but it should be mostly similar to this.

[ip1981]

I needs some refactoring.

[ip1981]

Refactored and made it ready for filesystem foreign key removal.

[ip1981]

On the clippy warning "redundant closure found" => rust-lang/rust-clippy#3791

[jgrund]

On the clippy warning "redundant closure found"

It makes sense since vec![] expands to Vec::new() in rust 1.44 (rust-lang/rust#70632).

Workaround is to use Vec::new constructor points-free e.g:

.or_insert_with(Vec::new)

[jgrund]

This won't pass because the python agent needs updates as well (I'll take care of this).

[jgrund]

https://github.com/whamcloud/iml-agent/pull/150