Clean up badly-dedented strings #3034
Description
In #2967, we ran Black over the whole codebase. Black doesn’t change the
AST, so in particular it doesn’t change multi-line strings. We ran
docformatter once to fix doc comments, but there are other cases where
multi-line strings should be reindented appropriately.
The following script (needs Python 3.8 for fixes to .col_offset) finds
strings with wonky indentation:
Script to find badly-dedented strings
import ast
import glob
def main():
for filename in glob.glob("tensorboard/**/*.py", recursive=True):
with open(filename) as infile:
source = infile.read()
sourcelines = source.splitlines()
module = ast.parse(source, filename=filename)
strings = [x for x in ast.walk(module) if isinstance(x, ast.Str)]
for node in strings:
base = _indent_of(sourcelines[node.lineno - 1])
if node.lineno == node.end_lineno:
# Rule out false positives on single-line strings with
# literal "\n" escape sequences.
continue
lines = node.s.splitlines()
if len(lines) <= 1:
continue
if any(_indent_of(x) < base for x in lines[1:] if x.strip()):
print("%s:%d:%s" % (filename, node.lineno, _preview(lines)))
def _indent_of(line):
n = 0
for c in line:
if c != " ":
break
n += 1
return n
def _preview(lines):
joined = " ".join(x.strip() for x in lines if x.strip())
ellipsis = "..."
maxlen = 40
if len(joined) <= maxlen:
return joined
else:
return joined[: maxlen - len(ellipsis)] + ellipsis
if __name__ == "__main__":
main()Script output (as of 74214c3)
tensorboard/notebook.py:93:Deprecated: use `%load_ext tensorboar...
tensorboard/notebook.py:346:(async () => { const url = await goog...
tensorboard/notebook.py:238:Unexpected result from `manager.start...
tensorboard/plugin_util_test.py:38:# _Hello_, **world!** Check out [my w...
tensorboard/plugin_util_test.py:40:<h1><em>Hello</em>, <strong>world!</s...
tensorboard/plugin_util_test.py:135:<!-- WARNING: discarded 36 null bytes...
tensorboard/plugin_util_test.py:47:Here is some data: TensorBoard usage ...
tensorboard/plugin_util_test.py:59:<p>Here is some data:</p> <table> <th...
tensorboard/manager_e2e_test.py:331:#!/bin/sh printf >&2 'tensorbad: fata...
tensorboard/manager_e2e_test.py:224:#!/bin/sh printf >&2 'fatal: somethin...
tensorboard/manager_e2e_test.py:253:#!/bin/sh printf >&2 'info: something...
tensorboard/manager_e2e_test.py:280:#!/bin/sh rm -r %s exit 22
tensorboard/manager_e2e_test.py:305:#!/bin/sh printf >%s '%%s' "$$" print...
tensorboard/backend/application.py:433:<!DOCTYPE html> <head><base href="plu...
tensorboard/backend/event_processing/sqlite_writer.py:76:INSERT INTO USERS (user_id, user_name...
tensorboard/backend/event_processing/sqlite_writer.py:93:SELECT experiment_id FROM Experiments...
tensorboard/backend/event_processing/sqlite_writer.py:106:INSERT INTO Experiments ( user_id, ex...
tensorboard/backend/event_processing/sqlite_writer.py:133:SELECT run_id FROM Runs WHERE experim...
tensorboard/backend/event_processing/sqlite_writer.py:146:INSERT INTO Runs ( experiment_id, run...
tensorboard/backend/event_processing/sqlite_writer.py:190:INSERT INTO Tags ( run_id, tag_id, ta...
tensorboard/backend/event_processing/sqlite_writer.py:239:INSERT OR REPLACE INTO Tensors ( seri...
tensorboard/backend/event_processing/db_import_multiplexer_test.py:58:SELECT Runs.run_name FROM Runs ORDER ...
tensorboard/backend/event_processing/db_import_multiplexer_test.py:70:SELECT Experiments.experiment_name FR...
tensorboard/backend/event_processing/reservoir.py:28:A map-to-arrays container, with deter...
tensorboard/uploader/uploader_main.py:331:This will delete the experiment on ht...
tensorboard/uploader/uploader_main.py:379:This will list all experiments that y...
tensorboard/uploader/uploader_main.py:435:This will upload your TensorBoard log...
tensorboard/uploader/uploader_main.py:482:This will download all your experimen...
tensorboard/uploader/uploader_main.py:517:Uploader has failed because of a time...
tensorboard/scripts/generate_testdata.py:38:The directory where serialized data w...
tensorboard/scripts/generate_testdata.py:45:Whether to remove and overwrite TARGE...
tensorboard/tools/import_google_fonts.py:213:}, generated_rule_name = "files", ext...
tensorboard/tools/import_google_fonts.py:187:filegroup_external( name = "%s", lice...
tensorboard/tools/import_google_fonts.py:198:# %s (%s) "%s": [ "%s", "%s", ],
tensorboard/tools/diagnose_tensorboard.py:220:Conflicting package installations fou...
tensorboard/tools/diagnose_tensorboard.py:347:The ".tensorboard-info" directory was...
tensorboard/tools/diagnose_tensorboard.py:395:Your current directory contains a `te...
tensorboard/tools/diagnose_tensorboard.py:405:Your Python path contains a `tensorbo...
tensorboard/tools/diagnose_tensorboard.py:490:Please try each suggestion enumerated...
tensorboard/tools/diagnose_tensorboard.py:502:No action items identified. Please co...
tensorboard/tools/diagnose_tensorboard.py:303:Your computer's hostname, %r, contain...
tensorboard/tools/diagnose_tensorboard.py:315:Python can't read your computer's hos...
tensorboard/plugins/image/images_plugin.py:85:SELECT 1 FROM Tags WHERE Tags.plugin_...
tensorboard/plugins/image/images_plugin.py:107:SELECT Runs.run_name, Tags.tag_name, ...
tensorboard/plugins/image/images_plugin.py:211:SELECT computed_time, step, CAST (T0....
tensorboard/plugins/image/images_plugin.py:322:SELECT data FROM TensorStrings WHERE ...
tensorboard/plugins/image/images_demo.py:159:Demonstration of forming a Gaussian b...
tensorboard/plugins/image/images_demo.py:267:Demonstration of [Sobel edge detectio...
tensorboard/plugins/beholder/visualizer.py:185:input: unprocessed numpy arrays. retu...
tensorboard/plugins/beholder/im_util.py:37:input: unscaled sections. returns: se...
tensorboard/plugins/text/text_plugin_test.py:135:<table> <tbody> <tr> <td>one</td> <td...
tensorboard/plugins/text/text_plugin_test.py:152:<table> <thead> <tr> <th>c1</th> <th>...
tensorboard/plugins/text/text_plugin_test.py:180:<table> <tbody> <tr> <td>one</td> </t...
tensorboard/plugins/text/text_plugin_test.py:204:<table> <thead> <tr> <th>X</th> </tr>...
tensorboard/plugins/text/text_plugin_test.py:303:<table> <tbody> <tr> <td><p>foo</p></...
tensorboard/plugins/text/text_plugin_test.py:319:<table> <tbody> <tr> <td><p>foo</p></...
tensorboard/plugins/text/text_plugin_test.py:112:<table> <tbody> <tr> <td><p>one</p></...
tensorboard/plugins/text/text_plugin_test.py:346:<table> <tbody> <tr> <td><p>foo</p></...
tensorboard/plugins/graph/keras_util_test.py:39:node { name: "sequential/dense" attr ...
tensorboard/plugins/graph/keras_util_test.py:115:node { name: "model/functional_input"...
tensorboard/plugins/graph/keras_util_test.py:189:node { name: "model/cycle_input" attr...
tensorboard/plugins/graph/keras_util_test.py:266:node { name: "model/lstm_input" attr ...
tensorboard/plugins/graph/keras_util_test.py:306:node { name: "sequential_2/sequential...
tensorboard/plugins/graph/keras_util_test.py:409:node { name: "model/main_input" attr ...
tensorboard/plugins/graph/keras_util_test.py:566:node { name: "model_1/sub_func_input_...
tensorboard/plugins/graph/keras_util_test.py:713:node { name: "model/func_seq_input" a...
tensorboard/plugins/graph/keras_util_test.py:794:node { name: "sequential/model/func_s...
tensorboard/plugins/graph/graph_util_test.py:25:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:113:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:330:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:688:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:61:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:85:node { name: "A" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:139:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:163:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:187:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:211:node { name: "X" op: "Input" device: ...
tensorboard/plugins/graph/graph_util_test.py:247:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:265:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:289:node { name: "X" op: "Input" } node {...
tensorboard/plugins/graph/graph_util_test.py:307:node { name: "W" op: "Input" device: ...
tensorboard/plugins/graph/graph_util_test.py:375:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:403:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:456:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:484:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:542:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:583:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:619:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:647:library { function { signature { name...
tensorboard/plugins/graph/graph_util_test.py:703:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:716:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:739:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:752:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:780:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:797:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:818:library { gradient { function_name: "...
tensorboard/plugins/graph/graph_util_test.py:831:library { gradient { function_name: "...
tensorboard/plugins/graph/keras_util.py:49:Walks the nested keras layer configur...
tensorboard/plugins/core/core_plugin.py:239:SELECT experiment_id, experiment_name...
tensorboard/plugins/core/core_plugin.py:274:SELECT Runs.run_id, Runs.run_name, Ru...
tensorboard/plugins/core/core_plugin.py:332:Directory where TensorBoard will look...
tensorboard/plugins/core/core_plugin.py:347:Like `--logdir`, but with special int...
tensorboard/plugins/core/core_plugin.py:365:What host to listen to (default: loca...
tensorboard/plugins/core/core_plugin.py:375:Serve on all public interfaces. This ...
tensorboard/plugins/core/core_plugin.py:403:Whether to purge data that may have b...
tensorboard/plugins/core/core_plugin.py:415:[experimental] sets SQL database URI ...
tensorboard/plugins/core/core_plugin.py:424:[experimental] enables DB read-and-im...
tensorboard/plugins/core/core_plugin.py:434:Prints digests of event files to comm...
tensorboard/plugins/core/core_plugin.py:471:The particular event file to query fo...
tensorboard/plugins/core/core_plugin.py:482:An optional, relative prefix to the p...
tensorboard/plugins/core/core_plugin.py:506:The max number of threads that Tensor...
tensorboard/plugins/core/core_plugin.py:518:How often the backend should load mor...
tensorboard/plugins/core/core_plugin.py:531:[experimental] The mechanism to use f...
tensorboard/plugins/core/core_plugin.py:547:[experimental] If true, this enables ...
tensorboard/plugins/core/core_plugin.py:563:[experimental] Configures the age thr...
tensorboard/plugins/core/core_plugin.py:581:[experimental] Whether to use generic...
tensorboard/plugins/core/core_plugin.py:592:An optional comma separated list of p...
tensorboard/plugins/core/core_plugin.py:193:SELECT run_name, started_time IS NULL...
tensorboard/plugins/core/core_plugin.py:387:Port to serve TensorBoard on. Pass 0 ...
tensorboard/plugins/hparams/summary_v2_test.py:89:hparams { key: "learning_rate" value ...
tensorboard/plugins/hparams/summary_v2_test.py:308:time_created_secs: 1555624767.0 hpara...
tensorboard/plugins/hparams/summary_v2.py:540:Args: tag: The tag name of the scalar...
tensorboard/plugins/hparams/keras_test.py:112:start_time_secs: 1234.5 group_name: "...
tensorboard/plugins/hparams/keras_test.py:134:end_time_secs: 6789.0 status: STATUS_...
tensorboard/plugins/hparams/backend_context_test.py:101:description: 'Test experiment' metric...
tensorboard/plugins/hparams/backend_context_test.py:119:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:126:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:133:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:140:hparam_infos: { name: 'batch_size' ty...
tensorboard/plugins/hparams/backend_context_test.py:176:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:182:hparams:[ {key: 'lr' value: {number_v...
tensorboard/plugins/hparams/backend_context_test.py:188:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:194:hparam_infos: { name: 'batch_size' ty...
tensorboard/plugins/hparams/backend_context_test.py:238:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:244:hparams:[ {key: 'lr' value: {number_v...
tensorboard/plugins/hparams/backend_context_test.py:250:hparams:[ {key: 'batch_size' value: {...
tensorboard/plugins/hparams/backend_context_test.py:256:hparam_infos: { name: 'batch_size' ty...
tensorboard/plugins/hparams/list_metric_evals_test.py:60:session_name: '/this/is/a/session' me...
tensorboard/plugins/hparams/list_session_groups_test.py:333:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:503:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:513:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:530:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:543:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:571:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:599:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:627:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:655:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:683:start_index: 0 slice_size: 3 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:344:session_groups { name: "group_1" hpar...
tensorboard/plugins/hparams/list_session_groups_test.py:556:name { tag: "current_temp" } value: 5...
tensorboard/plugins/hparams/list_session_groups_test.py:563:name { tag: "delta_temp" } value: -18...
tensorboard/plugins/hparams/list_session_groups_test.py:584:name { tag: "current_temp" } value: 1...
tensorboard/plugins/hparams/list_session_groups_test.py:591:name { tag: "delta_temp" } value: 1.5...
tensorboard/plugins/hparams/list_session_groups_test.py:612:name { tag: "current_temp" } value: 1...
tensorboard/plugins/hparams/list_session_groups_test.py:619:name { tag: "delta_temp" } value: 150...
tensorboard/plugins/hparams/list_session_groups_test.py:640:name { tag: "current_temp" } value: 1...
tensorboard/plugins/hparams/list_session_groups_test.py:647:name { tag: "delta_temp" } value: 150...
tensorboard/plugins/hparams/list_session_groups_test.py:668:name { tag: "current_temp" } value: 1...
tensorboard/plugins/hparams/list_session_groups_test.py:675:name { tag: "delta_temp" } value: 1.5...
tensorboard/plugins/hparams/list_session_groups_test.py:696:name { tag: "current_temp" } value: 5...
tensorboard/plugins/hparams/list_session_groups_test.py:703:name { tag: "delta_temp" } value: -18...
tensorboard/plugins/hparams/list_session_groups_test.py:712:start_index: 1 slice_size: 1 allowed_...
tensorboard/plugins/hparams/list_session_groups_test.py:726:col_params: { metric: { tag: 'optiona...
tensorboard/plugins/hparams/list_session_groups_test.py:744:col_params: { hparam: 'string_hparam'...
tensorboard/plugins/hparams/list_session_groups_test.py:761:col_params: { hparam: 'string_hparam'...
tensorboard/plugins/hparams/list_session_groups_test.py:779:col_params: { hparam: 'initial_temp' ...
tensorboard/plugins/hparams/list_session_groups_test.py:797:col_params: { metric: { tag: 'current...
tensorboard/plugins/hparams/list_session_groups_test.py:816:col_params: { metric: { tag: 'current...
tensorboard/plugins/hparams/list_session_groups_test.py:839:col_params: { hparam: 'optional_strin...
tensorboard/plugins/hparams/list_session_groups_test.py:856:col_params: { hparam: 'optional_strin...
tensorboard/plugins/hparams/list_session_groups_test.py:874:col_params: { metric: { tag: 'optiona...
tensorboard/plugins/hparams/list_session_groups_test.py:893:col_params: { metric: { tag: 'delta_t...
tensorboard/plugins/hparams/list_session_groups_test.py:909:col_params: { hparam: 'string_hparam'...
tensorboard/plugins/hparams/list_session_groups_test.py:926:col_params: { hparam: 'string_hparam'...
tensorboard/plugins/hparams/list_session_groups_test.py:944:col_params: { hparam: 'bool_hparam' o...
tensorboard/plugins/hparams/list_session_groups_test.py:965:col_params: { hparam: 'bool_hparam' o...
tensorboard/plugins/hparams/list_session_groups_test.py:987:col_params: { metric: { tag: 'optiona...
tensorboard/plugins/hparams/list_session_groups_test.py:1004:col_params: { metric: { tag: 'optiona...
tensorboard/plugins/hparams/list_session_groups_test.py:1021:col_params: { hparam: 'optional_strin...
tensorboard/plugins/hparams/list_session_groups_test.py:1038:col_params: { hparam: 'optional_strin...
tensorboard/plugins/hparams/list_session_groups_test.py:65:description: 'Test experiment' user: ...
tensorboard/plugins/hparams/list_session_groups_test.py:92:hparams:{ key: 'initial_temp' value: ...
tensorboard/plugins/hparams/list_session_groups_test.py:105:status: STATUS_SUCCESS end_time_secs:...
tensorboard/plugins/hparams/list_session_groups_test.py:114:hparams:{ key: 'initial_temp' value: ...
tensorboard/plugins/hparams/list_session_groups_test.py:127:status: STATUS_SUCCESS end_time_secs:...
tensorboard/plugins/hparams/list_session_groups_test.py:136:hparams:{ key: 'initial_temp' value: ...
tensorboard/plugins/hparams/list_session_groups_test.py:149:status: STATUS_FAILURE end_time_secs:...
tensorboard/plugins/hparams/list_session_groups_test.py:158:hparams:{ key: 'initial_temp' value: ...
tensorboard/plugins/hparams/list_session_groups_test.py:174:status: STATUS_UNKNOWN end_time_secs:...
tensorboard/plugins/hparams/list_session_groups_test.py:183:hparams:{ key: 'initial_temp' value: ...
tensorboard/plugins/hparams/list_session_groups_test.py:196:status: STATUS_SUCCESS end_time_secs:...
tensorboard/plugins/interactive_inference/witwidget/notebook/base.py:149:<link rel="import" href="/nbextension...
tensorboard/plugins/interactive_inference/witwidget/notebook/colab/wit.py:281:<link rel="import" href="/nbextension...
tensorboard/plugins/interactive_inference/witwidget/notebook/colab/wit.py:294:(new BroadcastChannel('{channel_name}...
tensorboard/plugins/profile/profile_plugin_loader.py:37:IP address of "master tpu", used for ...
tensorboard/plugins/profile/trace_events_json_test.py:41:devices { key: 2 value { name: 'D2' d...
tensorboard/plugins/debugger/debugger_plugin_loader.py:79:The port at which the non-interactive...
tensorboard/plugins/debugger/debugger_plugin_loader.py:94:The port at which the interactive deb...
tensorboard/plugins/debugger/debugger_plugin_loader.py:138:To use the debugger plugin, you need ...
tensorboard/plugins/histogram/histograms_plugin.py:88:SELECT 1 FROM Tags WHERE Tags.plugin_...
tensorboard/plugins/histogram/histograms_plugin.py:127:SELECT Tags.tag_name, Tags.display_na...
tensorboard/plugins/histogram/histograms_plugin.py:215:SELECT tag_id FROM Tags JOIN Runs USI...
tensorboard/plugins/histogram/histograms_plugin.py:242:SELECT MIN(step) AS step, computed_ti...
tensorboard/plugins/custom_scalar/custom_scalars_plugin.py:267:Fetches the custom layout specified b...
tensorboard/plugins/scalar/scalars_plugin.py:86:SELECT 1 FROM Tags WHERE Tags.plugin_...
tensorboard/plugins/scalar/scalars_plugin.py:130:SELECT Tags.tag_name, Tags.display_na...
tensorboard/plugins/scalar/scalars_plugin.py:195:SELECT Tensors.step, Tensors.computed...
tensorboard/plugins/pr_curve/pr_curves_plugin.py:214:SELECT Tags.tag_name, Tags.display_na...
tensorboard/plugins/pr_curve/pr_curves_plugin.py:284:SELECT TagPickingTable.run_name, Tens...
tensorboard/plugins/pr_curve/pr_curves_plugin.py:374:SELECT 1 FROM Tags WHERE Tags.plugin_...
tensorboard/plugins/pr_curve/pr_curves_plugin.py:99:SELECT Runs.run_name, Tensors.step, T...
Call a match a true positive if it can be safely reindented without
affecting the meaning of the program, or at least if such changes are
okay (e.g., a user-facing message is wrapped slightly differently). Call
a match a false positive if we don’t want to fix it up. Then, at a
glance, we have:
- True positives:
- SQL queries
- Protobuf pbtxt encodings, mostly in tests
- HTML, mostly in text
- Any strings that get
textwrap.dedented - A few docstrings that
docformatterdidn’t fix up (e.g.,
because because they weren’t well-formed in the first place)
- False positives:
- Flag definitions in
core_plugin.py, which have always been
aligned to column 0 intentionally - Maybe some others that I missed
- Flag definitions in