Cannot create object/file - Openstack Swift Keystone 3.0 / HTTP response code 500 was returned, slowing down

[vguilleaume]

I'm facing with an issue with S3FS -> S3PROXY connected to an OpenStack Swift - Keystone 3.0

Connection is working as expected.
We see that the system hang when creating file in container swift.

The system create object but do not allow to save it. System can list, read existing contents , but the creating through S3FS - S3proxy do not work. Furthermore , when we tried to try to create one, object is defined, but stay empty and from there we cannot anymore list the object in the container.

We see error message such as :
HTTP response code 500 was returned, slowing down

We can reproduce the problem

Launch the swift proxy ( s3proxy ) with the connector

s3proxy --properties /etc/s3proxy/swiftconnector

Run the mount access

s3fs -f -o url=http://127.0.0.1:8000 -o passwd_file=/etc/passwd-s3fs/swift -o use_path_request_style my_container /mnt/Pool/7/my_container/switchcontainer2/

When creating a new file, i receive a slowing down error message :
at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:76) at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499) at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) 2021-04-26T09:26:36.772Z [INF] curl.cpp:RequestPerform(2347): HTTP response code 500 was returned, slowing down

We analyste the difference between the 2 files, first one good → second one not good

(my_env) [root@test ~]# openstack object show my_container my_file.txt

+----------------+---------------------------------------+
| Field | Value |
+----------------+---------------------------------------+
| account | AUTH_481b9914945142b5b404dd46b70ae1e4 |
| container | my_container |
| content-length | 18 |
| content-type | text/plain; charset=UTF-8 |
| etag | f15e03e6bdcc396aa00e7a76c08d6e54 |
| last-modified | Mon, 26 Apr 2021 07:52:54 GMT |
| object | my_file.txt |
+----------------+---------------------------------------+
(my_env) [root@datamoer-test ~]# openstack object show my_container my_file2.txt
+---------------+--------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+--------------------------------------------------------------------------------------------+
| account | AUTH_481b9914945142b5b404dd46b70ae1e4 |
| container | my_container |
| content-type | text/plain; charset=UTF-8 |
| etag | d41d8cd98f00b204e9800998ecf8427e |
| last-modified | Mon, 26 Apr 2021 09:21:29 GMT |
| object | my_file2.txt |
| properties | Atime='1619428887', Ctime='1619428887', Gid='0', Mode='33188', Mtime='1619428887', Uid='0' |
+---------------+--------------------------------------------------------------------------------------------+

[vguilleaume]

Here is another set of error message i received when trying to list the container
[s3proxy] W 05-04 10:13:01.241 S3Proxy-Jetty-27 o.g.s.o.e.j.server.HttpChannel:396 |::] /my_container/my_file2.txt java.lang.NullPointerException: null at org.gaul.s3proxy.S3ProxyHandler.addMetadataToResponse(S3ProxyHandler.java:2841) at org.gaul.s3proxy.S3ProxyHandler.handleBlobMetadata(S3ProxyHandler.java:1628) at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:703) at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:77) at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499) at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748)

and the ls command is just hanging :

[root@datamoer-test switchcontainer2]# ls
ls: cannot access my_file2.txt: No such file or directory

and after couple of minutes, i have the result :

[root@datamoer-test switchcontainer2]# ls -l
ls: cannot access my_file2.txt: No such file or directory
total 1153
-rw-r----- 1 root root 18 Apr 26 07:52 my_file.txt
?????????? ? ? ? ? ? my_file2.txt
[root@datamoer-test switchcontainer2]#

[gaul]

Could you test with the latest master?

[vguilleaume]

I will do right now

[vguilleaume]

Hi @gaul,

Let me couple of minutes, as i believe i have to rebuild the "project" , if you have somewhere a short description how to rebuild a "image", please let me know. Then i will be faster to test it.

[gaul]

CI should have pushed a new build to Docker Hub. Otherwise you can build locally with mvn package which produces target/s3proxy.

[vguilleaume]

Thanks, i'm preparing a mvn to build it. I keep you updated.

[vguilleaume]

Great news ! This is working perfectly now. Thanks for your quick reaction and support. I can close the ticket.