Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEST][FLAKY] test_auo_scheduler_layout_rewrite #7548

Closed
tqchen opened this issue Feb 28, 2021 · 7 comments · Fixed by #7580
Closed

[TEST][FLAKY] test_auo_scheduler_layout_rewrite #7548

tqchen opened this issue Feb 28, 2021 · 7 comments · Fixed by #7580

Comments

@tqchen
Copy link
Member

tqchen commented Feb 28, 2021

=================================== FAILURES ===================================
__________ test_correctness_layout_rewrite_rewrite_for_preTransformed __________

    @tvm.testing.requires_llvm
    def test_correctness_layout_rewrite_rewrite_for_preTransformed():
        N = 128
        target = tvm.target.Target("llvm")
        task = auto_scheduler.SearchTask(func=matmul_auto_scheduler_test, args=(N, N, N), target=target)
        dag = task.compute_dag
    
        with tempfile.NamedTemporaryFile() as fp:
            log_file = fp.name
    
            search_policy = auto_scheduler.SketchPolicy(task)
    
            measure_ctx = auto_scheduler.LocalRPCMeasureContext()
            tuning_options = auto_scheduler.TuningOptions(
                num_measure_trials=2,
                runner=measure_ctx.runner,
                verbose=2,
                measure_callbacks=[auto_scheduler.RecordToFile(log_file)],
            )
            task.tune(tuning_options, search_policy=search_policy)
            inp, _ = auto_scheduler.load_best_record(log_file, task.workload_key, target)
            s, bufs = dag.apply_steps_from_state(
>               inp.state, layout_rewrite=auto_scheduler.LayoutRewriteOption.REWRITE_FOR_PRE_TRANSFORMED
            )
E           AttributeError: 'NoneType' object has no attribute 'state'

tests/python/unittest/test_auto_scheduler_layout_rewrite.py:89: AttributeError
=============================== warnings summary ===============================

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/625/pipeline

@tqchen
Copy link
Member Author

tqchen commented Feb 28, 2021

cc @comaniac @merrymercy

@comaniac
Copy link
Contributor

This happens when no valid schedules can be applied (load_best_record returns None in this case). Maybe in the i386 machine it's easy to get timeout and results in this error. We could consider either of the following:

A. Only perform this test on x86.
B. Improve this test by mocking a valid record.

@merrymercy which one do you prefer or do you have other suggestions?

@masahi
Copy link
Member

masahi commented Feb 28, 2021

I remember we had this problem before @jcf94

@merrymercy
Copy link
Member

merrymercy commented Mar 1, 2021

We can use this tuning option to make sure we always get a valid schedule. It will try at most 100 schedules but stops immediately when we get a valid one.

tune_option = auto_scheduler.TuningOptions(
num_measure_trials=100,
num_measures_per_round=2,
early_stopping=1,

@comaniac
Copy link
Contributor

comaniac commented Mar 1, 2021

I'm afraid that this would make the CI time longer especially on slower machines such as i386.

@jcf94
Copy link
Contributor

jcf94 commented Mar 3, 2021

How about to set a smaller N?

@comaniac
Copy link
Contributor

comaniac commented Mar 3, 2021

Combining the above ideas, I'll file a PR to

  1. Reduce the workload size (i.e., N) from 128x128 to 16x16.
  2. Enlarge trial number to 100 with early_stopping=1.

@comaniac comaniac linked a pull request Mar 3, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants