The AWS Access Key Id you provided does not exist in our records

[iby]

I've been running Riak CS in a docker container as a local alternative to S3 for awhile, it's unspeakably awesome. After it's been up for a few days and restated I start being frustrated with this error.

The AWS Access Key Id you provided does not exist in our records.

It's very inconsistent. Sometimes it happens more often, sometimes it doesn't at all, it's almost feels like when the Riak CS "warms up" this happens less often. Sometimes I get it a lot with Transmit, then when I run some random command with s3cmd, like ls, it works and starts working with Transmit. Sometimes it works with Transmit, but not with the AWS SDK.

I googled this up, the only suggestion on how to solve was found here:

Can you try raising the pb_backlog to 128 in your riak app.config on each
node. It's likely those disconnect errors are left over from the stampede
of connections from the CS connection pool on startup. For one reason or
another the resets don't come through and the hanging disconnected socket
isn't discovered until the next send attempt.

But, when I uncomment the pb_backlog and set any value (10, 20, 64, 128), Riak refuses to start, it just eats all CPU and hangs.

I run everything on a single node, it's doesn't get under pressure, it receives a few files during unit testing, thats it. I remember from the past that it's recommended to have at least 3 nodes, but that feels counter productive in my case.

Is this a bug or is this something that can be fixed by tuning the configuration?

[shino]

Are there any logs of Riak CS or Riak?

[iby]

I've started a new instance a few hours ago, it behaves good right now, can't reproduce this at the moment. Will share the logs when it happens again.

[iby]

Got it… Same thing happens again, sometimes works, sometimes doesn't. Right now it failed to display the bucket contents, after a few dozen retries it seems to be showing the bucket contents finally.

/var/log/riak-cs/error.log
/var/log/riak-cs/crash.log
/var/log/riak-cs/console.log

/var/log/riak/error.log – empty
/var/log/riak/crash.log – empty
/var/log/riak/console.log

[iby]

Just notice, that when first encountered today, the AWS SDK was failing with Access Denied message, but after it started to list bucket contents without failing, other API operations fail with The AWS Access Key Id you provided does not exist in our records.

[iby]

When was trying to upload the file it failed at first, then tried downloading a file, this also failed, but uploading began functioning, downloading still doesn't work.

cat /var/log/rial-cs/error.log
2015-01-24 21:01:00.143 [error] <0.17982.3>@riak_cs_block_server:handle_cast:194 Put <<"mybucket">> <<"user/pianorow.gif">> UUID <<144,20,214,161,186,232,77,68,129,80,82,95,66,193,152,20>> block 0 failed: {error,timeout}
2015-01-24 21:01:00.146 [error] <0.17982.3> gen_server <0.17982.3> terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:00.146 [error] <0.17982.3> CRASH REPORT Process <0.17982.3> with 1 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_server:terminate/6 line 747
2015-01-24 21:01:00.146 [error] <0.431.0> Supervisor riak_cs_put_fsm_sup had child undefined started with {riak_cs_put_fsm,start_link,undefined} at <0.17979.3> exit with reason no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in context child_terminated
2015-01-24 21:01:00.147 [error] <0.123.0> Lager event handler error_logger_lager_h exited with reason {'EXIT',{{badmatch,["/buckets/mybucket/objects/user%2Fpianorow.gif",{error,{exit,{{{badmatch,{error,timeout}},[{riak_cs_block_server,handle_cast,2,[{file,"src/riak_cs_block_server.erl"},{line,198}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_fsm,sync_send_event,[<0.17979.3>,{finalize,undefined},infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_object,...},...]}},...]},...}}
2015-01-24 21:01:00.156 [error] <0.17981.3> gen_fsm <0.17981.3> in state waiting_update_command terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:00.156 [error] <0.17981.3> CRASH REPORT Process <0.17981.3> with 0 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_fsm:terminate/7 line 611
2015-01-24 21:01:20.159 [error] <0.18056.3>@riak_cs_block_server:handle_cast:194 Put <<"mybucket">> <<"user/pianorow.gif">> UUID <<29,221,45,219,49,91,67,25,167,208,37,250,208,111,0,77>> block 0 failed: {error,timeout}
2015-01-24 21:01:20.159 [error] <0.18056.3> gen_server <0.18056.3> terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:20.160 [error] <0.18056.3> CRASH REPORT Process <0.18056.3> with 1 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_server:terminate/6 line 747
2015-01-24 21:01:20.160 [error] <0.431.0> Supervisor riak_cs_put_fsm_sup had child undefined started with {riak_cs_put_fsm,start_link,undefined} at <0.18054.3> exit with reason no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in context child_terminated
2015-01-24 21:01:20.160 [error] <0.123.0> Lager event handler error_logger_lager_h exited with reason {'EXIT',{{badmatch,["/buckets/mybucket/objects/user%2Fpianorow.gif",{error,{exit,{{{badmatch,{error,timeout}},[{riak_cs_block_server,handle_cast,2,[{file,"src/riak_cs_block_server.erl"},{line,198}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_fsm,sync_send_event,[<0.18054.3>,{finalize,undefined},infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_object,...},...]}},...]},...}}
2015-01-24 21:01:20.264 [error] <0.18055.3> gen_fsm <0.18055.3> in state waiting_update_command terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:20.264 [error] <0.18055.3> CRASH REPORT Process <0.18055.3> with 0 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_fsm:terminate/7 line 611
2015-01-24 21:01:40.182 [error] <0.18131.3>@riak_cs_block_server:handle_cast:194 Put <<"mybucket">> <<"user/pianorow.gif">> UUID <<59,193,218,56,87,231,76,199,136,255,168,93,62,126,20,95>> block 0 failed: {error,timeout}
2015-01-24 21:01:40.182 [error] <0.18131.3> gen_server <0.18131.3> terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:40.182 [error] <0.18131.3> CRASH REPORT Process <0.18131.3> with 1 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_server:terminate/6 line 747
2015-01-24 21:01:40.184 [error] <0.123.0> Lager event handler error_logger_lager_h exited with reason {'EXIT',{{badmatch,["/buckets/mybucket/objects/user%2Fpianorow.gif",{error,{exit,{{{badmatch,{error,timeout}},[{riak_cs_block_server,handle_cast,2,[{file,"src/riak_cs_block_server.erl"},{line,198}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_fsm,sync_send_event,[<0.18129.3>,{finalize,undefined},infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_object,...},...]}},...]},...}}
2015-01-24 21:01:40.184 [error] <0.431.0> Supervisor riak_cs_put_fsm_sup had child undefined started with {riak_cs_put_fsm,start_link,undefined} at <0.18129.3> exit with reason no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in context child_terminated
2015-01-24 21:01:40.288 [error] <0.18130.3> gen_fsm <0.18130.3> in state waiting_update_command terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:01:40.288 [error] <0.18130.3> CRASH REPORT Process <0.18130.3> with 0 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_fsm:terminate/7 line 611
2015-01-24 21:02:22.682 [error] <0.18537.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:22.683 [error] <0.18504.3> CRASH REPORT Process <0.18504.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18537.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:22.724 [error] <0.18543.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:22.726 [error] <0.13137.0> CRASH REPORT Process <0.13137.0> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18543.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:22.751 [error] <0.18549.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:22.753 [error] <0.17973.3> CRASH REPORT Process <0.17973.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18549.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:35.410 [error] <0.18331.3>@riak_cs_block_server:handle_cast:194 Put <<"mybucket">> <<"user/pianorow.gif">> UUID <<207,0,62,171,80,92,74,165,132,87,60,152,188,119,223,189>> block 0 failed: {error,timeout}
2015-01-24 21:02:35.410 [error] <0.18331.3> gen_server <0.18331.3> terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:02:35.411 [error] <0.18331.3> CRASH REPORT Process <0.18331.3> with 1 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_server:terminate/6 line 747
2015-01-24 21:02:35.411 [error] <0.123.0> Lager event handler error_logger_lager_h exited with reason {'EXIT',{{badmatch,["/buckets/mybucket/objects/user%2Fpianorow.gif",{error,{exit,{{{badmatch,{error,timeout}},[{riak_cs_block_server,handle_cast,2,[{file,"src/riak_cs_block_server.erl"},{line,198}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_fsm,sync_send_event,[<0.18329.3>,{finalize,undefined},infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_object,...},...]}},...]},...}}
2015-01-24 21:02:53.450 [error] <0.18663.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:53.451 [error] <0.18051.3> CRASH REPORT Process <0.18051.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18663.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:53.467 [error] <0.18669.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:53.468 [error] <0.18510.3> CRASH REPORT Process <0.18510.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18669.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:53.489 [error] <0.18675.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d314a6e985301e2d10150f7fc089fd2d.jpeg">> block# {<<128,173,23,110,130,29,71,116,161,59,47,158,164,113,64,105>>,0}: {error,notfound}
2015-01-24 21:02:53.490 [error] <0.18513.3> CRASH REPORT Process <0.18513.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18675.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:02:55.422 [error] <0.18407.3>@riak_cs_block_server:handle_cast:194 Put <<"mybucket">> <<"user/pianorow.gif">> UUID <<97,92,249,197,242,177,67,151,180,124,117,86,71,173,35,213>> block 0 failed: {error,timeout}
2015-01-24 21:02:55.423 [error] <0.18407.3> gen_server <0.18407.3> terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:02:55.424 [error] <0.18407.3> CRASH REPORT Process <0.18407.3> with 1 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_server:terminate/6 line 747
2015-01-24 21:02:55.424 [error] <0.123.0> Lager event handler error_logger_lager_h exited with reason {'EXIT',{{badmatch,["/buckets/mybucket/objects/user%2Fpianorow.gif",{error,{exit,{{{badmatch,{error,timeout}},[{riak_cs_block_server,handle_cast,2,[{file,"src/riak_cs_block_server.erl"},{line,198}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_fsm,sync_send_event,[<0.18405.3>,{finalize,undefined},infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_object,...},...]}},...]},...}}
2015-01-24 21:02:55.424 [error] <0.431.0> Supervisor riak_cs_put_fsm_sup had child undefined started with {riak_cs_put_fsm,start_link,undefined} at <0.18405.3> exit with reason no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in context child_terminated
2015-01-24 21:02:55.428 [error] <0.18406.3> gen_fsm <0.18406.3> in state waiting_update_command terminated with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198
2015-01-24 21:02:55.429 [error] <0.18406.3> CRASH REPORT Process <0.18406.3> with 0 neighbours exited with reason: no match of right hand value {error,timeout} in riak_cs_block_server:handle_cast/2 line 198 in gen_fsm:terminate/7 line 611
2015-01-24 21:03:49.016 [error] <0.18888.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d6a8ea5f7c0b61121236989de8d68c3f.jpeg">> block# {<<221,53,209,168,57,141,79,89,134,24,57,209,22,93,115,179>>,0}: {error,notfound}
2015-01-24 21:03:49.017 [error] <0.17915.3> CRASH REPORT Process <0.17915.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18888.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:03:49.057 [error] <0.18894.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d6a8ea5f7c0b61121236989de8d68c3f.jpeg">> block# {<<221,53,209,168,57,141,79,89,134,24,57,209,22,93,115,179>>,0}: {error,notfound}
2015-01-24 21:03:49.057 [error] <0.18877.3> CRASH REPORT Process <0.18877.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18894.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214
2015-01-24 21:03:49.088 [error] <0.18900.3>@riak_cs_get_fsm:waiting_chunks:315 riak_cs_get_fsm: Cannot get S3 <<"mybucket">> <<"user/0/temporary/d6a8ea5f7c0b61121236989de8d68c3f.jpeg">> block# {<<221,53,209,168,57,141,79,89,134,24,57,209,22,93,115,179>>,0}: {error,notfound}
2015-01-24 21:03:49.089 [error] <0.18548.3> CRASH REPORT Process <0.18548.3> with 0 neighbours exited with reason: {normal,{gen_fsm,sync_send_event,[<0.18900.3>,get_next_chunk,infinity]}} in gen_fsm:sync_send_event/3 line 214

[shino]

@ianbytchek I don't understand what is the cause, just explain what log says.

• Two lines in CS console.log https://gist.github.com/ianbytchek/ee3e6c7ea05a1e8b2541#file-console-log-L24 indicates user fetch (getting user object from riak by riak-cs) failed by notfound.
• The log just above says block (content of S3 object) retrieval failed by timeout (default 60sec).

It may be nice to try without docker?

[iby]

Thanks @shino, it might be the case of some problem between CS and Riak then. Using the same user I can list all buckets, but can't list the specific bucket – one of the common problems.

The problem seems to be happening when the VM gets unexpectedly switched off, maybe Riak gets cut off in the middle of doing something?

I used to run this on my hardware not that long ago, this never happened, so it would be pointless trying this without docker. My setup is only one node for local development, it's not critical that it fails, I simply start the new container, and it's not what the real life setup would be like. But if you want to chase this I can share the docker file, you can build the image and try this yourself. Doing this in a virtual machine and killing it should give the same result, the instance must contain objects and must be in operation for awhile.

[shino]

The problem seems to be happening when the VM gets unexpectedly switched off, maybe Riak gets cut off in the middle of doing something?

Network connectivity issue may cause the errors.

I will close this issue because it seems not related to riak cs software itself.
If you want to discuss and/or seek workaround, I guess riak-users mailing list is
better place than here.

@ianbytchek Thanks for your interests and reporting 😄

[iby]

@shino cool 😄