Errors thrown when requesting database target size over 1TB

[hfxdklee]

I'm attempting to create a database target of 1024GB and the creation process fails immediately when attempting to create the customer CSV files. I drop the size below a terabyte and it works as expected.

[toddmuirhead]

What happens when it fails? Do you get any error message. What OS are you running on?

[hfxdklee]

Wow quick response! Thanks! I'm running Win2016 datacenter edition, latest patchset. It looks like it's a bug with the CSV creation program in that it might have some sort of maximum value coded in it. [cid:d0ac21e0-94d1-4590-84af-e99d12c78f1e]

…

________________________________ From: Todd Muirhead <notifications@github.com> Sent: Thursday, March 7, 2019 1:30 PM To: dvdstore/ds3 Cc: David Klee; Author Subject: Re: [dvdstore/ds3] Errors thrown when requesting database target size over 1TB (#4) What happens when it fails? Do you get any error message. What OS are you running on? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#4 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALDoPl8RlRLoTHuCNxRZIPgL-dZtTk5bks5vUWjNgaJpZM4bj656>.

[toddmuirhead]

I was able to chase this down and the limit of 1024 GB is not arbitrary. There is a reason. The max value for an INT is 2,147,483,647. Ratios are used for determining the number of rows when creating a specified size DVDStore database. Specifically for the customers table - a 100GB database will have 200 million rows with each row representing a customer. When you specify a 1024GB size, this translates to 2.024 billion rows. The program that creates the customer row data is limited by the max integer value. To make it simple we stopped at 2.046 rows or max size of 1023GB, but I guess we could have gone up to 2.147 or a size of 1073GB.

Do you have a need to create a larger test database?

[hfxdklee]

Holy cow! Thanks for tracking this down! The short answer is yes, I’d like to use dvdstore to create a test database in the 5-10TB range. We’re doing some stress testing of some new NVMe storage and I want to really hammer it from a SQL Server perspective with a very wide random workload to sample the performance. However, if it’s going to create a significant burden to you to fix this, I definitely understand and wouldn’t worry about it. Would a bigint replacement for the int32 value be a simple adjustment? David From: Todd Muirhead <notifications@github.com> Sent: Friday, March 15, 2019 1:58 PM To: dvdstore/ds3 <ds3@noreply.github.com> Cc: David Klee <dklee@heraflux.com>; Author <author@noreply.github.com> Subject: Re: [dvdstore/ds3] Errors thrown when requesting database target size over 1TB (#4) I was able to chase this down and the limit of 1024 GB is not arbitrary. There is a reason. The max value for an INT is 2,147,483,647. Ratios are used for determining the number of rows when creating a specified size DVDStore database. Specifically for the customers table - a 100GB database will have 200 million rows with each row representing a customer. When you specify a 1024GB size, this translates to 2.024 billion rows. The program that creates the customer row data is limited by the max integer value. To make it simple we stopped at 2.046 rows or max size of 1023GB, but I guess we could have gone up to 2.147 or a size of 1073GB. Do you have a need to create a larger test database? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#4 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALDoPs7mdd-6EkmUXRv4M59mJ_Uf1T9rks5vW-1JgaJpZM4bj656>.

[toddmuirhead]

David - I have fixed this issue with the update that I just uploaded. It was essentially fixing the c program - ds3_create_cust.c so that it uses 'long long int' type instead of just 'int'. This one change results in having to change several other things in the program and also an update to InstallDVDStore.pl to allow for bigger numbers. I was only able to test at a slightly higher value - about 1.4 TB. I've limited the InstallDVDStore.pl script to accept up to 4096 GB.

Let me know how it goes.

Thanks - Todd