Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[UX] remove stacktrace for pipe and ssh info #1324

Merged
merged 5 commits into from
Oct 30, 2022
Merged

[UX] remove stacktrace for pipe and ssh info #1324

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4859e1d
UX: remove stacktrace for pipe and ssh info
Michaelvll Oct 29, 2022
db54960
Add comment
Michaelvll Oct 29, 2022
e2ce234
Avoid ray output in the logs
Michaelvll Oct 29, 2022
883e9cf
format
Michaelvll Oct 29, 2022
cb7641a
revert ssh quiet option
Michaelvll Oct 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions sky/backends/backend_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1996,19 +1996,23 @@ def kill_children_processes():
# Handle ctrl-c
def interrupt_handler(signum, frame):
del signum, frame
logger.warning(f'{colorama.Style.DIM}The job will keep '
f'running after Ctrl-C.{colorama.Style.RESET_ALL}')
kill_children_processes()
# Avoid using logger here, as it will print the stack trace for broken
# pipe, when the output is piped to another program.
print(f'{colorama.Style.DIM}Tip: The job will keep '
f'running after Ctrl-C.{colorama.Style.RESET_ALL}')
with ux_utils.print_exception_no_traceback():
raise KeyboardInterrupt(exceptions.KEYBOARD_INTERRUPT_CODE)


# Handle ctrl-z
def stop_handler(signum, frame):
del signum, frame
logger.warning(f'{colorama.Style.DIM}The job will keep '
f'running after Ctrl-Z.{colorama.Style.RESET_ALL}')
kill_children_processes()
# Avoid using logger here, as it will print the stack trace for broken
# pipe, when the output is piped to another program.
print(f'{colorama.Style.DIM}Tip: The job will keep '
f'running after Ctrl-Z.{colorama.Style.RESET_ALL}')
with ux_utils.print_exception_no_traceback():
raise KeyboardInterrupt(exceptions.SIGTSTP_CODE)

Expand Down
15 changes: 10 additions & 5 deletions sky/skylet/log_lib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,7 @@ def tail_logs(job_owner: str,
time.sleep(_SKY_LOG_WAITING_GAP_SECONDS)
status = job_lib.update_job_status(job_owner, [job_id], silent=True)[0]

start_stream_at = 'INFO: Tip: use Ctrl-C to exit log'
if follow and status in [
job_lib.JobStatus.RUNNING, job_lib.JobStatus.PENDING
]:
Expand All @@ -425,15 +426,19 @@ def tail_logs(job_owner: str,
with open(log_path, 'r', newline='') as log_file:
# Using `_follow` instead of `tail -f` to streaming the whole
# log and creating a new process for tail.
for line in _follow_job_logs(
log_file,
job_id=job_id,
start_streaming_at='INFO: Tip: use Ctrl-C to exit log'):
for line in _follow_job_logs(log_file,
job_id=job_id,
start_streaming_at=start_stream_at):
print(line, end='', flush=True)
else:
try:
start_stream = False
with open(log_path, 'r') as f:
print(f.read())
for line in f.readlines():
if start_stream_at in line:
start_stream = True
if start_stream:
print(line, end='', flush=True)
Comment on lines +435 to +441
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! Is the purpose of this change to hide the output that comes before INFO: Tip: use Ctrl-C? I can still see it:

(base) romilb@romilbx1yoga:/mnt/d/Romil/Berkeley/Research/sky-experiments/examples$ sky cancel myclus 2
Cancelling jobs (2) on cluster 'myclus'...
(base) romilb@romilbx1yoga:/mnt/d/Romil/Berkeley/Research/sky-experiments/examples$ sky logs myclus 2
Tailing logs of job 2 on cluster 'myclus'...
I 10-30 02:32:35 log_lib.py:385] Start streaming logs for job 2.
2022-10-30 02:19:35,045 INFO worker.py:1227 -- Using address 172.31.35.172:6379 set in the environment variable RAY_ADDRESS
2022-10-30 02:19:35,045 INFO worker.py:1337 -- Connecting to existing Ray cluster at address: 172.31.35.172:6379...
2022-10-30 02:19:35,050 INFO worker.py:1513 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265 
INFO: Tip: use Ctrl-C to exit log streaming (task will not be killed).
INFO: Waiting for task resources on 1 node. This will block if the cluster is full.
INFO: All task resources reserved.
INFO: Reserved IPs: ['172.31.35.172']
(task pid=26142) hi

Is this expected?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, that is not expected. Could you check the skypilot on your cluster to see if it has the latest update? If the cluster is launched before this PR, it may require a sky launch to update the skypilot on that cluster.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh yes this was a cluster launched from master branch. Re-launching from this branch fixed it. Thanks!

except FileNotFoundError:
print(f'{colorama.Fore.RED}ERROR: Logs for job {job_id} (status:'
f' {status.value}) does not exist.{colorama.Style.RESET_ALL}')