Crashes (SIGABRT) Under Test Automation #539
Comments
|
Hi, couple of questions:
|
|
Updated above w/ answers- also copied here for convenience, thank you @andyundso.
Running against 2017 for Linux in a Docker container via Docker Desktop for Mac. SQL Server Authentication. I don't think I'm having an authentication problem as my app (and tests) are able to connect to the database and execute queries.
By "test automation", I simply meant the execution of the test suite for this project. As part of this upgrade process, I will get the suite running in an automated CI system, either w/ updates to our existing, but dated Jenkins CI installation, or in GitHub Actions. I have not done this yet.
I will try this against our Rails 5.2 branch, assuming the newer TinyTDS is backwards compatible both w/ Rails 5.2 (should be, I think), and Ruby 2.5.x (??).
This is on an Intel Mac. |
|
Disclaimer: I'm unfamiliar with the C code in I also own an Intel Mac, the same OS version as yours, the same FreeTDS version. I maintain two Rails projects, both with RSpec, but one runs Ruby 3.0, the other 3.1. I checked if I also ran into these issues when using the version of In general, from what I read in the code, the
Is there some consistency when you hit these errors? For example, I assume you run your test suite in random order. Once it crashes and you re-run the test suite with the same seed, does it crash again with the same error? As mentioned, it would be interesting to know if you also hit this error when updating |
|
similar error when in rails console on Ruby 3.1.2, Rails 7.0, TinyTDS 2.1.5, freetds v1.3.20 on M1 mac running against various MS Sql Server servers e.g. Microsoft SQL Server 2016 (SP3-CU1-GDR) (KB5021128) - 13.0.7024.30 (X64) usually after I've made a bunch of queries and have had the console open a while, but only on exit |
|
I was able to reliably trigger this with a single spec example and it's beginning to form some type of working theory for me. This is an excerpt from a Rails controller spec: Test output: One other important detail is that my application is a multi-tenant app, where the current tenant is a per-request concept that can be switched based on criteria like the hostname requested. One of the things that tenant switching did* is switch the database connection with the Apartment gem, which we had* forked to support the Something must be happening lifetime of these database connections when this spec is run. The database connection being used by the Rails server handling the *Did and *had b/c we've actually removed Apartment and the tenant switching functionality that used it to change the database connections. The tenancy concept is still in place, and requests still switch tenants as before, but we aren't manipulating the database connections during the tenant switch any longer. |
|
@clintmiller have you tried reproducing bundling against main? there may be an unreleased fix |
|
I was originally on the search for something else, but I managed to bring up the issue as well: I added this test to the it 'raises error when query cannot be sent' do
client = new_connection
assert_client_works(client)
thread1 = Thread.new do
client.execute("WaitFor Delay '00:00:10'")
end
assert thread1.alive?
thread2 = Thread.new do
assert_raises(TinyTds::Error) { client.execute("WaitFor Delay '00:00:02'") }
end
thread1.join
thread2.join
endbut from here it gets complicated - I would have to compile Ruby in Debug mode to get meaningful pointers when the coredump is generated. and even then I am wondering how my example test applies to your test suite failure. while writing the comment I had another idea. what works fine is the following: it "is another thing" do
client = new_connection
client.execute("SELECT 1 as [one]")
client.execute("SELECT 1 as [one]")
endthis properly generates an error: but this test code: it "is another thing with wait" do
client = new_connection
client.execute("WaitFor Delay '00:00:10'")
client.execute("SELECT 1 as [one]")
endgenerates this error:
I have no clue yet what this all means and if it is even a lead following, but I have to leave the office for today and needed to write down my exploration so far :) |
|
had some more time today. bare with me, another long comment. so my primary goal was to dissallow executing if (cwrap->userdata->dbsql_sent && !dbdead(cwrap->client) && !cwrap->userdata->closed) {
rb_raise(cTinyTdsError, "Trying to send another query before consuming results. Please call \"do\", \"each\" or \"insert\" first.");
}basically, first,
there was one testing failing on this line with the error message I introduced earlier: id_constraint_name = @client.execute("EXEC sp_helpindex [datatypes]").find { |row| row['index_keys'] == 'id' }['index_name']looking through the code, I assume ... but
I am not sure how I feel about result = client.execute("SELECT 1 as [one]; SELECT 2 as [two]; SELECT 3 as [three]")
result.to_a
# => [{"one"=>1}, {"two"=>2}, {"three"=>3}]
# exit early out of the iterator
result = client.execute("SELECT 1 as [one]; SELECT 2 as [two]; SELECT 3 as [three]")
result.each(cache_rows: true) { |r| break if r.key?("two") }
# => nil
result.to_a
# => [{"one"=>1}]
# for comparison: here is how a regular array behaves
numbers = [1, 2, 3, 4, 5]
numbers.each { |n| break if n == 3 }
# => nil
numbers.to_a
# => [1, 2, 3, 4, 5]I looked into the Then there are again, not sure where this all leads. but I think if we can fix this |
|
new week, new long comment. but this should hopefully be the last one before the PR, as I am quite clear on what to do next. I went over my idea from last week (show an error when the user tries to run another
But I do not like this API. Needing to cancel a result when fetching something with Additionally, what I already showed earlier is that the results are incomplete when the cache by tiny_tds is activated (by default) and you return out of the iterator early: # exit early out of the iterator
result = client.execute("SELECT 1 as [one]; SELECT 2 as [two]; SELECT 3 as [three]")
result.each(cache_rows: true) { |r| break if r.key?("two") }
# => nil
result.to_a
# => [{"one"=>1}]You can also force this into the other direction by deactivating caching and getting incomplete results on the seond result = client.execute("SELECT 1 as [one]; SELECT 2 as [two]; SELECT 3 as [three]")
# =>
# #<TinyTds::Result:0x000073c52bfa0848
# ...
result.each(cache_rows: false) { |r| puts r.inspect }
# {"one"=>1}
# {"two"=>2}
# {"three"=>3}
# => []
result.each(cache_rows: false) { |r| puts r.inspect }
# nil
# => []This is due to the fact as from Ultimately, there would be ways to fix this in this C code that it behaves correctly with
What advantages would give that?
I wrote this paragraph earlier in the afternoon and although it no longer fits into the original comment, I did not want to discard it. I double-checked in the ruby pg gem and the trilogy gem how they handle fetching rows. |
Can they still be read from TinyTds::Client.default_query_options by default? Also, any thoughts on this statement in the readme?
|
|
Hi @clintmiller, do you maybe have a chance to try my proposed fix in #563? You once provided a spec where you could reproduce the error consistently, but it has been a year, so maybe this is no longer accurate. |
|
@andyundso I'll get it built and see what it does on my test case! |
|
@andyundso Quick update here- this PR seems to fix the specific issue I was running into. I'd probably have to fully roll-out our CI pipeline in GitHub Actions to see if it's really stable for us over a couple of days worth of runs. That will go on my todo list. As an aside, I also have had to update from the MSSQL docker image |
|
@clintmiller If you are using GitHub Actions |
|
Hi @clintmiller , just wanted to check in and ask how the fix is performing. do you less or even no troubles on your CI? |
Environment
Operating System
Intel Mac, x86_64
TinyTDS Version and Information
I'm actually pointing at a post-2.1.5 commit on
tiny_tds:23ed1e4, where #527 was merged.FreeTDS Version
SQL Server
Running against 2017 for Linux in a Docker container via Docker Desktop for Mac. SQL Server Authentication.
Description
In the course of trying to upgrade an application from Ruby 2.5/Rails 5.2.x to Ruby 3.1/Rails 6.1.x, I have noticed a significant increase in automated test failures (we're using
rspec-rails). These failures seem take a couple of forms:1. An
EXC_BAD_ACCESS (SIGABRT).Terminal output:
OS Crash Report:
I'm pretty sure this was a simple unit test, given the running threads in the Ruby process. One thing that immediately catches my eye is that there appear to be two threads in the TinyTDS C code at the same time. I'm also curious about the
nogvl_*functions, but only b/c of their name. I'm not sure I fully understand their purpose.2. An
EXC_CRASH (SIGABRT).Terminal output:
OS Crash Report:
Obviously, this case with more threads is a system/integration test that started a webserver.
Other Notes
It's interesting to note, that the application boots and runs in my development environment w/o any issues. I'm not sure how it will do with a production workload given the crashes I'm seeing in our test suite.
Next Steps
Making a minimal reproduction test-case would be useful, but I'm not quite sure where to start. On Ruby 2.5/Rails 5.2, our test suite rarely (if ever), crashed in this manner. Is there any other additional information I can provide or debug builds of TinyTDS I could use to reveal more helpful information?
The text was updated successfully, but these errors were encountered: