Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip RWKV test in past CI #24204

Merged
merged 2 commits into from
Jun 12, 2023
Merged

Skip RWKV test in past CI #24204

merged 2 commits into from
Jun 12, 2023

Conversation

ydshieh
Copy link
Collaborator

@ydshieh ydshieh commented Jun 12, 2023

What does this PR do?

In the past CI with torch 13.1 (and older), RWKV tests fail.

  • The first failure is

    test_model_parallelism
    
    (line 114)  RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k,   &alpha, a, lda, b, ldb, &beta, c, ldc)
    • this affects 1x more subsequential tests
    • skip this particular test will reduce a lot the number of failures.
  • The docker env. of that past CI uses a base docker image shipped with cuda 11.6

  • In a docker image with cuda 11.8 but installing torch==1.13+cu116, no failure.

Let's not take too much time on identifying what exactly cause the failure here, and just skip RWKV tests on past CI with torch < 2.0.

The goal is just to have a clean past CI report.

@ydshieh ydshieh requested a review from sgugger June 12, 2023 15:53
@ydshieh ydshieh marked this pull request as ready for review June 12, 2023 15:54
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me!

@ydshieh ydshieh merged commit 4fe9716 into main Jun 12, 2023
@ydshieh ydshieh deleted the skip_rwkv_in_past branch June 12, 2023 16:14
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants