Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cant delete large file #1185

Closed
tnatanael opened this issue May 20, 2019 · 22 comments
Closed

Cant delete large file #1185

tnatanael opened this issue May 20, 2019 · 22 comments
Labels
Question

Comments

@tnatanael
Copy link

Hy guys, i created a simple cluster, with 2 storages, and after uploading a file with 1Gb and running the cluster for 1 week, i am not able to delete this file, the delete operation runs with success but the file persists...

What i tried:
recover-node
recover-disk
recover-consistency

I wonder that when i put the cluster in the production env, with so many files this would be a very annoying bug, so please help me.
@mocchira
Copy link
Member

@tnatanael What you'd have to do for deleting file physically is "compaction-start" as described here: http://leo-project.net/leofs/docs/admin/system_operations/data/#how-to-operate-data-compaction

please check the doc above out for more details.

@tnatanael
Copy link
Author

Tried with leofs-adm compact-start, it says OK, but the file persists, even after waiting the process to finish

@yosukehara
Copy link
Member

Let us know your LeoFS' error log and the state of the large object:

  • The error log is output under LeoStorage's log directory
  • To understand the current state of the object, execute leofs-adm whereis <OBJECT_NAME> command

@tnatanael
Copy link
Author

How can i discover the object name? Is it the filename of the original file?

@tnatanael
Copy link
Author

image

@tnatanael
Copy link
Author

when i run compact and when i try to delete the file using s3 api this error message pops on log

@yosukehara
Copy link
Member

Is it the filename of the original file?

Exactly, leofs-adm whereis <file-path>

@tnatanael
Copy link
Author

I tried with only filename...
leofs-adm whereis 1000mb_1

And bucket + filename
leofs-adm whereis teste-thiago/1000mb_1

The 2 options says
[ERROR] Could not get ring

@tnatanael
Copy link
Author

I am 100% sure that the file was corrupt due to disk failures, but it may need to be cleared by the manual delete or auto by the cluster in some way

@yosukehara
Copy link
Member

I've understood that your LeoFS' RING (routing table) is broken. So let me know the current state of the system. Can you share the result of leofs-adm status and the operation histories to this day?

@tnatanael
Copy link
Author

Sure!!!
image

It is a test cluster, i am simulating a disk failure we experienced in production

How can i get the operations history?

@yosukehara
Copy link
Member

How can i get the operations history?

$ history | gpre leofs-adm

@tnatanael
Copy link
Author

image

Do you want a team viewer session?

@yosukehara
Copy link
Member

image

I've just clearly understood that LeoManager's RING is broken.
I'm going to consider how to recover the system.

@tnatanael
Copy link
Author

Ok... only to state, the cluster is still working, i am uploading and removing new files right now... only this file is undeletable...

@yosukehara
Copy link
Member

TO: @mocchira your opinion will be much appreciated.

@tnatanael
Copy link
Author

Hy guys! Can this ticket be labelled as a bug instead of question?

@yosukehara
Copy link
Member

yosukehara commented May 22, 2019 •
edited
Loading

Please do the following if you understand that we may NOT be able to restore your system completely.
I considered how to recover your LeoManager’s RING as below:

Procedure:

  1. Stop the all nodes of LeoStorage, LeoGateway, and LeoManager
  2. Backup all files of LeoManager nodes (Archive LeoManager‘s directory)
  3. After data backup of LeoManager nodes, Remove Mnesia files under mnesia directory of both LeoManager nodes
  4. Start the LeoManager nodes
  5. Start the LeoStorage nodes
  6. Execute leofs-adm start
  7. Confirm the state of the system with leofs-adm status
  8. Start the LeoGateway node

If you succeeded in doing the procedure, you can execute the data-compaction command.

@yosukehara
Copy link
Member

yosukehara commented May 23, 2019 •
edited
Loading

I'd like to share an example of the procedure of recovering LeoManager's RING as below.

[Example] How To Recover LeoManager's RING

Before recovery:

$ leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.5.0
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash |
                previous ring-hash |
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
 type  |           node           |    state     | rack id |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:10:33 +0900
  S    | storage_1@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:10:33 +0900
  S    | storage_2@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:10:33 +0900
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------

The Procedure of Recovering LeoManager's RING

1. Stop the all nodes

$ ./package/leo_manager_0/bin/leo_manager stop
ok
$ ./package/leo_manager_1/bin/leo_manager stop
ok
$ ./package/leo_gateway_0/bin/leo_gateway stop
ok
$ ./package/leo_storage_0/bin/leo_storage stop
ok
$ ./package/leo_storage_1/bin/leo_storage stop
ok
$ ./package/leo_storage_2/bin/leo_storage stop
ok

2. Archive LeoManager's directories

$ tar czf leo_manager_0_backup.ta.gz ./package/leo_manager_0/
$ tar czf leo_manager_1_backup.ta.gz ./package/leo_manager_1/

$ ls -la | grep backup.ta.gz
-rw-r--r--   1 yosukehara  staff  15435040 May 23 10:12 leo_manager_0_backup.ta.gz
-rw-r--r--   1 yosukehara  staff  15429047 May 23 10:12 leo_manager_1_backup.ta.gz

3. Remove the LeoManager's data directories

## manager_0:
$ rm -rf ./package/leo_manager_0/work/mnesia/*
## manager_1:
$ rm -rf ./package/leo_manager_1/work/mnesia/*

4. Restart the all nodes except LeoGateway's node(s)

$ ./package/leo_manager_0/bin/leo_manager start
$ ./package/leo_manager_1/bin/leo_manager start
$ ./package/leo_storage_0/bin/leo_storage start
$ ./package/leo_storage_1/bin/leo_storage start
$ ./package/leo_storage_2/bin/leo_storage start
$ leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.5.0
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash |
                previous ring-hash |
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
 type  |           node           |    state     | rack id |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | attached     |         |                |                | 2019-05-23 10:14:00 +0900
  S    | storage_1@127.0.0.1      | attached     |         |                |                | 2019-05-23 10:14:03 +0900
  S    | storage_2@127.0.0.1      | attached     |         |                |                | 2019-05-23 10:14:05 +0900
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------

After restarted the all nodes, execute leofs-adm start command:

$ leofs-adm start
Generating RING...
Generated RING
OK  33% - storage_0@127.0.0.1
OK  67% - storage_2@127.0.0.1
OK 100% - storage_1@127.0.0.1
OK

5. Restart LeoGateway node(s)

$ ./package/leo_gateway_0/bin/leo_gateway start

6. Confirm the RING hashes of the all nodes:

$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.5.0
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | d5d667a6
                previous ring-hash | d5d667a6
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
 type  |           node           |    state     | rack id |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:14:16 +0900
  S    | storage_1@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:14:16 +0900
  S    | storage_2@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:14:16 +0900
  G    | gateway_0@127.0.0.1      | running      |         | d5d667a6       | d5d667a6       | 2019-05-23 10:14:32 +0900
-------+--------------------------+--------------+---------+----------------+----------------+----------------------------

The important thing is that the values ​​of current ring and prev ring before recovery and the values after recovery are the same - current ring: d5d667a6, prev ring: d5d667a6.

@tnatanael
Copy link
Author

I'll try this tomorrow and return, but i am wondering why this happens? It appear to be an expected behaviour? Thanks for now!

@tnatanael
Copy link
Author

Sorry for delay... it worked... after that procedure i was able to delete the file...
Thanks!

@akezakky555
Copy link

@yosukehara I tried to do follow your instructions but I found some problem. After I restart all leofs service. All User and Buckets are disappear.

After Remove mnesia folder
leofs02

So I recovery mnesia folder at leo_manager Everything is back. but RING is broken again.
leofs01

Can you suggest us how to fix this problem?

Thanks

@yosukehara yosukehara mentioned this issue Sep 18, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yosukehara @mocchira @tnatanael @akezakky555