Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create dm-cache and test failed #61

Closed
pahome opened this issue May 28, 2020 · 30 comments
Closed

Create dm-cache and test failed #61

pahome opened this issue May 28, 2020 · 30 comments

Comments

@pahome
Copy link

pahome commented May 28, 2020 •
edited
Loading

I create dm-cache with SSD RAID0(/dev/md1) and HDD(/dev/sdd)

sudo dmsetup create ssd-metadata --table '0 122657 linear /dev/md1 0'
sudo dd if=/dev/zero of=/dev/mapper/ssd-metadata
sudo dmsetup create ssd-blocks --table '0 1875282143 linear /dev/md1 122657'
sudo dmsetup create home-cached --table '0 1953525168 cache /dev/mapper/ssd-metadata /dev/mapper/ssd-blocks /dev/sdd 512 1 writeback default 0'

Then write new profile ssd in ~/.dmtest/config:

profile :ssd do
  metadata_dev '/dev/mapper/ssd-metadata'
  data_dev '/dev/mapper/home-cached'
end

And run
rvmsudo dmtest run --suite cache --profile ssd -t FIOTests

But it shows all error:

Warning: can not check `/etc/sudoers` for `secure_path`, falling back to call via `/usr/bin/env`, this breaks rules from `/etc/sudoers`. Run:

    export rvmsudo_secure_path=1

to avoid the warning, put it in shell initialization file to make it persistent.

In case there is no `secure_path` in `/etc/sudoers`. Run:

    export rvmsudo_secure_path=0

to avoid the warning, put it in shell initialization file to make it persistent.
Loaded suite cache
FIOTests
  fio_across_cache_size_mq...#<Test::Unit::Error:0x0000000001c1af60
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_fio_across_cache_size_mq(FIOTests)">
FAIL
  fio_across_cache_size_smq...#<Test::Unit::Error:0x0000000001f971c0
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_fio_across_cache_size_smq(FIOTests)">
FAIL
  fio_on_fast...#<Test::Unit::Error:0x0000000002034f38
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_fio_on_fast(FIOTests)">
FAIL
  fio_on_regions...#<Test::Unit::Error:0x00000000027025b8
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_fio_on_regions(FIOTests)">
FAIL
  fio_on_slow...#<Test::Unit::Error:0x00007fe3a8007ad8
 @exception=#<ArgumentError: wrong number of arguments (given 2, expected 3)>,
 @test_name="test_fio_on_slow(FIOTests)">
FAIL
  origin_same_size_as_ssd_mq...#<Test::Unit::Error:0x000000000276d700
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_origin_same_size_as_ssd_mq(FIOTests)">
FAIL
  origin_same_size_as_ssd_smq...#<Test::Unit::Error:0x00000000027bfde8
 @exception=#<RuntimeError: out of free space>,
 @test_name="test_origin_same_size_as_ssd_smq(FIOTests)">
FAIL

Why it shows out of free space? and other errors?
Is there anything wrong?
@pahome pahome changed the title How to write config for dm-cache? Create dm-cache and test failed May 28, 2020
@jthornber
Copy link
Owner

dmtest sets up caches etc itself, just put the raw block devices in the config (or a slice of them).

The error you're getting is to do with sudo, nothing to do with dmtest.

@pahome
Copy link
Author

pahome commented May 28, 2020

@jthornber
Excuse me, somethin weird still found.

I modified config to contain only raw device
I'm sure /dev/sdb and /dev/sdc are all ssd.

profile :ssd do
  metadata_dev '/dev/sdb'
  data_dev '/dev/sdc'
end

default_profile :ssd

Then run a basic task with rvmsudo(failed without rvmsudo)
rvmsudo dmtest run --suite cache --profile ssd -t BackgroundWritebackTests

It still got something wrong

Warning: can not check `/etc/sudoers` for `secure_path`, falling back to call via `/usr/bin/env`, this breaks rules from `/etc/sudoers`. Run:

    export rvmsudo_secure_path=1

to avoid the warning, put it in shell initialization file to make it persistent.

In case there is no `secure_path` in `/etc/sudoers`. Run:

    export rvmsudo_secure_path=0

to avoid the warning, put it in shell initialization file to make it persistent.
Loaded suite cache
BackgroundWritebackTests
  cache_remains_clean_through_reload_mq_1...16357 dirty blocks
8586 dirty blocks
885 dirty blocks
0 dirty blocks
0 dirty blocks
No input files found for test-dev-184974
PASS
  cache_remains_clean_through_reload_mq_2...#<Test::Unit::Error:0x00007f8aa0005500
 @exception=
  #<ProcessControl::ExitError: command failed: cache_restore  --metadata-version 2 -i metadata.xml -o /dev/mapper/test-dev-518937>,
 @test_name=
  "test_cache_remains_clean_through_reload_mq_2(BackgroundWritebackTests)">
FAIL
  cache_remains_clean_through_reload_smq_1...16362 dirty blocks
8893 dirty blocks
1324 dirty blocks
0 dirty blocks
0 dirty blocks
No input files found for test-dev-992634
PASS
  cache_remains_clean_through_reload_smq_2...#<Test::Unit::Error:0x00000000028a0cd0
 @exception=
  #<ProcessControl::ExitError: command failed: cache_restore  --metadata-version 2 -i metadata.xml -o /dev/mapper/test-dev-378744>,
 @test_name=
  "test_cache_remains_clean_through_reload_smq_2(BackgroundWritebackTests)">
FAIL
  clean_a_cache_with_large_blocks...#<Test::Unit::Error:0x00000000029c7f28
 @exception=
  #<ProcessControl::ExitError: command failed: cache_restore  --metadata-version 2 -i metadata.xml -o /dev/mapper/test-dev-205475>,
 @test_name="test_clean_a_cache_with_large_blocks(BackgroundWritebackTests)">
FAIL
  clean_data_never_gets_written_back_mq_1...No input files found for test-dev-623727
PASS
  clean_data_never_gets_written_back_mq_2...#<Test::Unit::Error:0x0000000002bd3178
 @exception=
  #<ProcessControl::ExitError: command failed: cache_restore  --metadata-version 2 -i metadata.xml -o /dev/mapper/test-dev-997659>,
 @test_name=
  "test_clean_data_never_gets_written_back_mq_2(BackgroundWritebackTests)">
FAIL
...

I'm a newbie so don't know where to debug
Could your teach me why this FAIL happened?

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

Yes

Host:/home/git/device-mapper-test-suite$ which cache_restore 
/usr/sbin/cache_restore
Host:/home/git/device-mapper-test-suite$ ll /usr/sbin/cache_restore 
lrwxrwxrwx 1 root root 11  十  25  2015 /usr/sbin/cache_restore -> pdata_tools*

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020

Found it

...
D, [2020-05-28T16:18:13.978003 #13488] DEBUG -- : executing: 'cache_restore  --metadata-version 2 -i metadata.xml -o /dev/mapper/test-dev-513496'
D, [2020-05-28T16:18:13.980712 #13488] DEBUG -- : stderr:
    cache_restore: unrecognized option '--metadata-version'
    Usage: cache_restore [options]
    Options:
      {-h|--help}
      {-i|--input} <input xml file>
      {-o|--output} <output device or file>
      {-q|--quiet}
      {-V|--version}

      {--debug-override-metadata-version} <integer>
      {--omit-clean-shutdown}

Is that version too old?
Mine OS is ubuntu 16.04

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 

$ cache_restore --version
0.5.6

Do you build from its github directly?

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020

In cache suit BackgroundWritebackTests, some still failed

BackgroundWritebackTests
  cache_remains_clean_through_reload_mq_1    
  cache_remains_clean_through_reload_mq_2    
  cache_remains_clean_through_reload_smq_1    
  cache_remains_clean_through_reload_smq_2    
  clean_a_cache_with_large_blocks    
  clean_data_never_gets_written_back_mq_1    
  clean_data_never_gets_written_back_mq_2    
  clean_data_never_gets_written_back_smq_1    
  clean_data_never_gets_written_back_smq_2    
  dirty_data_always_gets_written_back_mq_1    FAIL
  dirty_data_always_gets_written_back_mq_2    FAIL
  dirty_data_always_gets_written_back_smq_1    FAIL
  dirty_data_always_gets_written_back_smq_2    FAIL

I take a look at dirty_data_always_gets_written_back_mq_1 log:

... ...
D, [2020-05-28T16:41:51.280358 #21050] DEBUG -- : stdout:
    0 937703088 cache 8 58/1024 128 16380/16384 17 29 0 0 8 4 0 1 writeback 2 migration_threshold 1024 mq 10 random_threshold 0 sequential_threshold 0 discard_promote_adjustment 0 read_promote_adjustment 0 write_promote_adjustment 0 rw -
D, [2020-05-28T16:41:51.281044 #21050] DEBUG -- : executing: 'dmsetup remove test-dev-539956'
D, [2020-05-28T16:41:51.366892 #21050] DEBUG -- : executing: 'cache_check /dev/mapper/test-dev-585676'
D, [2020-05-28T16:41:51.438179 #21050] DEBUG -- : stderr:
    examining superblock
    examining mapping array
    examining hint array
    examining discard bitset
D, [2020-05-28T16:41:51.637689 #21050] DEBUG -- : stdout:
    === dm-2 ===
      CPU  0:                   56 events,        3 KiB data
      CPU  1:                    0 events,        0 KiB data
      CPU  2:                16408 events,      770 KiB data
      CPU  3:                    0 events,        0 KiB data
      Total:                 16464 events (dropped 0),      772 KiB data
D, [2020-05-28T16:41:51.642712 #21050] DEBUG -- : executing: 'dmsetup remove test-dev-393569'
D, [2020-05-28T16:41:51.666299 #21050] DEBUG -- : executing: 'dmsetup remove test-dev-464559'
D, [2020-05-28T16:41:51.690019 #21050] DEBUG -- : executing: 'dmsetup remove test-dev-585676'
E, [2020-05-28T16:41:51.720470 #21050] ERROR -- : Failure:
test_dirty_data_always_gets_written_back_mq_1(BackgroundWritebackTests)
    [/usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/tests/cache/background_writeback_tests.rb:83:in `block in dirty_data_always_gets_written_back'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/ensure_elapsed.rb:13:in `ensure_elapsed_time'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/cache_stack.rb:75:in `block in activate_support_devs'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/ensure_elapsed.rb:13:in `ensure_elapsed_time'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/device-mapper/lexical_operators.rb:42:in `block in with_devs'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/prelude.rb:6:in `bracket'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/device-mapper/lexical_operators.rb:37:in `with_devs'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/cache_stack.rb:67:in `activate_support_devs'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/tests/cache/background_writeback_tests.rb:72:in `dirty_data_always_gets_written_back'
     /usr/share/rvm/gems/ruby-2.5.3/gems/device_mapper_test_suite-0.0.1/lib/dmtest/test-utils.rb:25:in `block (2 levels) in define_tests_across']:
<16384> expected but was
<0>.
I, [2020-05-28T16:41:51.720736 #21050]  INFO -- : Peak bufio allocation was 458752

I don't understand what it means

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

What are you trying to do?

I want to try my custom dm-cache driver, so want to find tool to test if driver works
But now I only use dm-cache of upstream kernel 5.4(need to enable dm-cache) in Ubuntu16.04

Unfortunately, I'm not familiar with ruby

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

I checked background_writeback_tests.rb
If fails here:

def dirty_data_always_gets_written_back(policy, metadata_version)
...
    traces, _ = blktrace(s.origin) do
    ...
    assert_equal(cache_size / block_size, filter_writes(traces[0]).size)
  end
end

My cache_size is 2097152, block_size is 128, so cache_size / block_size is 16384
Is that possible to modify that by arguments? or other ways?
Or blktrace return wrong traces?

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

Linux pahome-QW09 5.4.20+ 1 SMP Wed May 27 17:07:17 CST 2020 x86_64 x86_64 x86_64 GNU/Linux

And I also enable dm-cache, dm-verity, dm-dust when make menuconfig.

Can I know your kernel version? I wanna tried .

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020

Are you modifying my existing dm-cache, or writing a new policy for it, or
writing something new from scratch?

No, I don't modified anything about kernel 5.4 and your code.
Just enable dm-cache...etc and reboot. Then dmtest run --suite cache... like above mentioned.

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020

"I want to try my custom dm-cache driver" <- what did you mean by this

In the future, I want to re-write dm-cache for my job.
So I first use upstream dm-cache and test it by device-mapper-test-suite

BTW, I saw cache_size and block_size was defined by function, maybe that's why I always failed.

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

The target of rewrite maybe combine flashcache and dm-cache
So I don't know difference between dm-cache and dm-writecache
And I have no idea to port rewrited dm-cache to upstream.

@pahome
Copy link
Author

pahome commented May 28, 2020

Is that bug?

In ./lib/dmtest/tests/cache/fio_tests.rb

def run_fio(dev, name, file_size) <- 3 arguments
  outfile = AP("fio-#{name}.out")
  size_in_meg = file_size / meg(1)
...
def baseline(dev, name)
  sub_vol_size = meg(1024 + 128)

  vg = TinyVolumeManager::VM.new
  vg.add_allocation_volume(dev)
  vg.add_volume(linear_vol("vol", sub_vol_size))
  with_dev(vg.table("vol")) do |vol|
    run_fio(vol, name) <- 2 arguments
  end
end

function arguments number different there

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020

If I want to test dm-cache in kernel 5.4
What item should I test in cache suite?

BackgroundWritebackTests <-- passed
BurstyWriteTests
DTTests
DiscardTests
FIOTests <-- fail in fio_on_slow and fio_on_fast
GitExtractTests
IOUseTests
InvalidateCBlocksTests
LargeConfigTests
MetadataScalingTests
MetadataVersionTests
NeedsCheckTests
NoCleanShutdownTests
PassthroughTests
PolicySwitchTests
ResizeTests
SMQComparisonTests
SmallConfigTests
ToolsTests
WriteThroughTests

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 28, 2020 •
edited
Loading

It seems fio_on_slow and fio_on_fast of FIOTests will failed cuz of fallocate too much space

This grab the fio_on_slow.log from beginning to bug point:


D, [2020-05-28T18:26:46.799484 #30119] DEBUG -- : executing: 'blockdev --getsz /dev/sdc'
D, [2020-05-28T18:26:46.848807 #30119] DEBUG -- : stdout:
    937703088
D, [2020-05-28T18:26:46.849017 #30119] DEBUG -- : executing: 'dmsetup create test-dev-251980 --notable'
D, [2020-05-28T18:26:46.883216 #30119] DEBUG -- : writing table: <<table:0 2359296 linear /dev/sdc 0>>
D, [2020-05-28T18:26:46.883446 #30119] DEBUG -- : executing: 'dmsetup load test-dev-251980 /tmp/dm-table20200528-30119-ks48qz'
D, [2020-05-28T18:26:46.887144 #30119] DEBUG -- : executing: 'dmsetup resume test-dev-251980'
D, [2020-05-28T18:26:46.891093 #30119] DEBUG -- : executing: 'mkfs.ext4 -F -E lazy_itable_init=1,nodiscard /dev/mapper/test-dev-251980'
D, [2020-05-28T18:26:47.394981 #30119] DEBUG -- : stdout:
    Creating filesystem with 294912 4k blocks and 73728 inodes
    Filesystem UUID: 127858a3-167c-4f06-8da4-b14d440ba866
    Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

    Allocating group tables: 0/9^H^H^H   ^H^H^Hdone
    Writing inode tables: 0/9^H^H^H   ^H^H^Hdone
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: 0/9^H^H^H   ^H^H^Hdone

D, [2020-05-28T18:26:47.395192 #30119] DEBUG -- : stderr:
    mke2fs 1.45.5 (07-Jan-2020)
D, [2020-05-28T18:26:47.395862 #30119] DEBUG -- : executing: 'mount /dev/mapper/test-dev-251980 ./fio_test '
D, [2020-05-28T18:26:47.547895 #30119] DEBUG -- : executing: 'fio /tmp/fio-job20200528-30119-s6xtpg --output=/home/pahome/git/device-mapper-test-suite/fio-slow.out'
D, [2020-05-28T18:26:51.666986 #30119] DEBUG -- : stderr:
    fio: posix_fallocate fails: No space left on device   <------Here
    fio: io_u error on file randrw.0.0: No space left on device: write offset=1201340416, buflen=65536
D, [2020-05-28T18:26:51.667228 #30119] DEBUG -- : command failed with 'pid 4671 exit 1': fio /tmp/fio-job20200528-30119-s6xtpg --output=/home/pahome/git/device-mapper-test-suite/fio-slow.out

Is there any way to avoid this? Like space maximum?

@pahome
Copy link
Author

pahome commented May 28, 2020

Sorry, where to download dt in Ubuntu?
Looks like I download wrong binary
It shows

Host:~/git/qnap_linux_5_4$ dt help
DITrack command-line client, version 0.8

General usage:
	dt <command> [<options>] [<args>]

Available commands:
	act
	cat
	commit, ci
	help
	list, ls
	new
	remove, rm
	status, st
	update, up

Type 'dt help <command>' for help on specific command.

@jthornber
Copy link
Owner

jthornber commented May 28, 2020 via email

@pahome
Copy link
Author

pahome commented May 29, 2020

Do you have any ideas about the fio_on_fast & fio_on_slow issues?

@pahome
Copy link
Author

pahome commented May 29, 2020

When I run task BurstyWriteTests in cache suite, it fails.

Take a look at one of the log smallfile_cache_mq_1.log:

D, [2020-05-29T09:54:31.981131 #26449] DEBUG -- : executing: 'mount -o nouuid /dev/mapper/test-dev-970212 ./smallfile-mount'
D, [2020-05-29T09:54:32.047034 #26449] DEBUG -- : executing: 'python ~/smallfile/smallfile_cli.py --top ./smallfile-mount --fsync Y --file-size-distribution exponential --hash-into-dirs Y --files-per-dir 30 --dirs-per-dir 5 --threads 4 --file-size 64 --operation create --files 10000'
D, [2020-05-29T09:54:32.401764 #26449] DEBUG -- : stderr:
    python: can't open file '/home/pahome/smallfile/smallfile_cli.py': [Errno 2] No such file or directory
D, [2020-05-29T09:54:32.401873 #26449] DEBUG -- : command failed with 'pid 26620 exit 2': python ~/smallfile/smallfile_cli.py --top ./smallfile-mount --fsync Y --file-size-distribution exponential --hash-into-dirs Y --files-per-dir 30 --dirs-per-dir 5 --threads 4 --file-size 64 --operation create --files 10000

Where to find the smallfile_cli.py?

@pahome pahome mentioned this issue Jun 1, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jthornber @pahome