Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CT-2049: Add CommandCompleted event #7180

Merged
merged 6 commits into from
Mar 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20230316-123305.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add CommandCompleted event, and fire it upon completion of every command
time: 2023-03-16T12:33:05.696752-07:00
custom:
Author: QMalcolm
Issue: "6878"
37 changes: 35 additions & 2 deletions core/dbt/cli/requires.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,22 @@
from dbt.config import RuntimeConfig
from dbt.config.runtime import load_project, load_profile, UnsetProfile
from dbt.events.functions import setup_event_logger, fire_event, LOG_VERSION
from dbt.events.types import MainReportVersion, MainReportArgs, MainTrackingUserState
from dbt.events.types import (
CommandCompleted,
MainReportVersion,
MainReportArgs,
MainTrackingUserState,
)
from dbt.exceptions import DbtProjectError
from dbt.parser.manifest import ManifestLoader, write_manifest
from dbt.profiler import profiler
from dbt.tracking import active_user, initialize_from_flags, track_run
from dbt.utils import cast_dict_to_dict_of_strings

from click import Context
import datetime
from functools import update_wrapper
import time


def preflight(func):
Expand Down Expand Up @@ -53,7 +60,33 @@ def wrapper(*args, **kwargs):
# Adapter management
ctx.with_resource(adapter_management())

return func(*args, **kwargs)
start_func = time.perf_counter()

try:
(results, success) = func(*args, **kwargs)

fire_event(
CommandCompleted(
command=ctx.command_path,
success=success,
completed_at=datetime.datetime.utcnow(),
elapsed=time.perf_counter() - start_func,
)
)
# Bare except because we really do want to catch ALL exceptions,
# i.e. we want to fire this event in ALL cases.
except: # noqa
fire_event(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to fire another event when we run into issues, listed here, we actually have a issue tracking it here. @jtcohen6 maybe we can just resolve the other issue also in this PR(even though it is not scheduled), how you feel?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally okay doing it separately to keep each PR smaller.

Copy link
Contributor Author

@QMalcolm QMalcolm Mar 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we should definitely work to fire that event again (and handle the fail fast exceptions better). I'm not opposed to taking on that work. However, I'd personally prefer to make it a separate PR to keep the PR process as smooth as possible.

Copy link
Contributor

@jtcohen6 jtcohen6 Mar 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree - let's do it in a separate PR! Even though that issue is not scheduled for the current sprint, we definitely want it before the v1.5 RC - we have it tracked in #7162

CommandCompleted(
command=ctx.command_path,
success=False,
completed_at=datetime.datetime.utcnow(),
elapsed=time.perf_counter() - start_func,
)
)
raise

return (results, success)

return update_wrapper(wrapper, func)

Expand Down
16 changes: 16 additions & 0 deletions core/dbt/events/proto_types.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions core/dbt/events/types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -1747,6 +1747,19 @@ message NoNodesSelectedMsg {
NoNodesSelected data = 2;
}

// Q039
message CommandCompleted {
string command = 1;
bool success = 2;
google.protobuf.Timestamp completed_at = 3;
float elapsed = 4;
}

message CommandCompletedMsg {
EventInfo info = 1;
CommandCompleted data = 2;
}

// W - Node testing

// Skipped W001
Expand Down
10 changes: 10 additions & 0 deletions core/dbt/events/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -1827,6 +1827,16 @@ def message(self) -> str:
return "No nodes selected!"


@dataclass
class CommandCompleted(DebugLevel, pt.CommandCompleted):
def code(self):
return "Q039"

def message(self) -> str:
status = "succeeded" if self.success else "failed"
return f"Command `{self.command}` {status} at {self.completed_at} after {self.elapsed:0.2f} seconds"


# =======================================================
# W - Node testing
# =======================================================
Expand Down
2 changes: 2 additions & 0 deletions tests/unit/test_events.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import re
from typing import TypeVar

from datetime import datetime
from dbt.contracts.results import TimingInfo
from dbt.events import AdapterLogger, test_types, types
from dbt.events.base_types import (
Expand Down Expand Up @@ -326,6 +327,7 @@ def test_event_codes(self):
types.NoNodesSelected(),
types.DepsUnpinned(revision="", git=""),
types.NoNodesForSelectionCriteria(spec_raw=""),
types.CommandCompleted(command="", success=True, elapsed=0.1, completed_at=datetime.utcnow()),
# W - Node testing ======================
types.CatchableExceptionOnRun(exc=""),
types.InternalErrorOnRun(build_path="", exc=""),
Expand Down