Issue receiving credentials with 0.7.4rc0 of aca-py #1724
Comments
|
@etschelp What I tested with the following configuration and steps. I was able to issue to and issue from 0.7.3 agent and also store credentials without any issues. Steps:
|
|
This happens if one of the wallets is of type askar. It does not happen every time, sometimes like every 5th time, sometimes sooner. On one machine running arm aca-py runs emulated and is very slow this happens nearly every time. On other machines it's more randomly. So this looks like a synchronisation issue I guess. Also it happens more often if the auto flags are disabled. |
|
Even easier, if both agents are running 0.7.4rc0 and use wallet type askar this happens nearly every time. Previously this was only true for the V1 protocol, if I switch to V2 with indy payload it breaks every time with: |
|
I wasn't able to trigger any error [within ACA-Py w/ askar --wallet-type] in the following scenarios:
I tried to test this in I also tested this using
|
|
Thats weird. I can still confirm that I can not issue a credential anymore when running 0.7.4-rc0 with askar wallets. The exchange gets stuck on the issuer side when the holder requests the credential after the offer with: The wrong state exception that I mentioned above is the result of that. I tested this on different machines and different setups and everywhere it is the same. |
|
@etschelp Did you test it today after the new |
|
@shaangill025 no changes with the new version still seeing Output of the config endpoint: {
"config": {
"admin.admin_insecure_mode": true,
"admin.enabled": true,
"admin.host": "0.0.0.0",
"admin.port": "8031",
"admin.webhook_urls": [
"http://localhost:8080/log"
],
"admin.admin_client_max_request_size": 1,
"debug.auto_respond_presentation_proposal": true,
"debug.auto_store_credential": true,
"debug.auto_verify_presentation": true,
"auto_disclose_features": true,
"external_plugins": [
"aries_cloudagent.messaging.jsonld"
],
"default_endpoint": "http://x.x.x.x:8888",
"additional_endpoints": [],
"tails_server_base_url": "https://tails.bosch-digital.co",
"tails_server_upload_url": "https://tails.bosch-digital.co",
"revocation.notify": true,
"revocation.monitor_notification": true,
"ledger.genesis_url": "https://indy-test.bosch-digital.de/genesis",
"ledger.keepalive": 5,
"log.level": "info",
"auto_ping_connection": true,
"debug.monitor_ping": true,
"public_invites": true,
"trace.target": "log",
"trace.tag": "",
"trace.label": "Dummy Agent",
"emit_new_didcomm_prefix": true,
"emit_new_didcomm_mime_type": true,
"exch_use_unencrypted_tags": true,
"auto_provision": true,
"transport.inbound_configs": [
[
"http",
"0.0.0.0",
"8030"
]
],
"transport.outbound_configs": [
"http"
],
"transport.enable_undelivered_queue": true,
"default_label": "Dummy Agent",
"transport.max_message_size": 2097152,
"transport.max_outbound_retry": 4,
"transport.ws.heartbeat_interval": 3,
"transport.ws.timeout_interval": 15,
"wallet.name": "wallet_db",
"wallet.storage_type": "postgres_storage",
"wallet.type": "askar",
"wallet.storage_config": "{\"url\":\"bpa-wallet-db1:5432\",\"max_connections\":5}",
"endorser.author": false,
"endorser.endorser": false,
"endorser.auto_endorse": false,
"endorser.auto_write": false,
"endorser.auto_create_rev_reg": false,
"endorser.auto_promote_author_did": false,
"ledger.read_only": false,
"ledger.genesis_transactions": "..."
}
} |
|
@etschelp Based upon the provided config, I tried with the following startup arguments [v0.7.4-rc0, same network and tails server] and was able to issue multiple indy credentials [v1 and v2] without any errors. Also tested issuing with cred-def created and stored using v0.7.3. Can you test with the same config as earlier but with local postgres storage. I have requested @ianco if he could help by confirming this. |
|
@shaangill025 I will look into this next week. I just tested this with the latest state from main, and there it behaves slightly different. Nevertheless I have the impression that this is related to a synch issue between multiple processes in aca-py. When the holder requests the credential I can see in the issuer logs that the exchange goes through and even changes to credential_issued. But it does not stop there after that I see further error logs saying first "wrong state, expected request received but was credential issued", and after that "wrong state expected request_received but was abandoned". |
|
@shaangill025 I have tested this some more and long story short as soon as the 0.7.4rc0 image is started with the askar profile this happens. And I think I have figured out why you can not reproduce this. To reproduce you have to check the events that are fired against the webhook of the controller, and check the exchange state on the issuer side, because the credential appears in the wallet of the holder as issued. What happens is:
I have not found the root cause, but it is somewhere inside askar. |
|
@etschelp as you mentioned on the aca-pug call this morning, I suspect this is related to BPA is responding to web hooks, and some kind of synchronization issue between BPA's response to web hooks, BPA -> Aca-py admin api calls, and stuff aca-py is doing on its own. To solve this we're going to first need to reproduce the exact situation that you're seeing, which means (I think) we're going to have to start up a BPA environment and give it a run. Can you provide some info about your BPA setup? Are you just running with the standard config that's in the BPA repo? We may need to schedule a call where you can walk us through your setup and demo the scenario that is generating these issues ... |
|
@ianco One very simple way to test this is to use GitPod. I have prepared a branch that uses the rc2 with askar you can open it via: https://gitpod.io/#https://github.com/hyperledger-labs/business-partner-agent/pull/755 Apart from that it uses exactly the configuration from the main branch. Mainly:
It takes a while to start up, once its up you can open the remote explorer on the left hand side (second to last icon):
Clicking on open browser, for ports 8081 and 8090 gives you two empty bpa's. The terminal in the original window gives you all the logs. If you are not familiar with the BPA here are all the steps to reproduce:
I hope this helps, if you have more questions we can schedule a call any time this week. |
|
I'm not able to run this. The GitPod instance starts up fine, but the 8090 port toggles between "open (public)" and "not served", until it stops at the latter. As such, I can't do anything with the instances. Am I missing something? The one at 8081 starts up properly. Very cool, BTW! |
|
Thats weird, I recreated the environment several times and every time it comes up nicely. It takes a while for everything to be fully up, and during startup not all ports are available. Anyhow, it was just an idea to make debugging easier. Ianco's comment got me thinking ,so I started playing around with the endpoints and the auto flags. For credential issuance the bpa was using /issue-credential/send I switched that one to /issue-credential/send-offer and voila it works. I believe at least for the v1 api the issue was that the /send endpoint at some point always needed another call to the /issue endpoint otherwise the flow would not run completely, this behaviour has now changed and the additional call is not needed any more hence the state exception. It is strange though that aca-py fails the exchange if the /issue endpoint is called and the exchange state is already on issued. The v2 endpoints still behave the old way, meaning calling /issue after issuance wont fail the exchange. Also what is the difference between those two endpoints? As send-offer with auto_issue=true also would automate the whole flow. I will test this some more and then close the ticket as the other issue with the askar exception mentioned above seems to be fixed with the new version in the rc2. |
Signed-off-by: Philipp Etschel <philipp.etschel@ch.bosch.com>
|
Bummer about GitPod not working. Very weird -- I'd love to see this working. I'll see if I can delete the workspace and try it again. Is there anything in the /send handling that we need to add to detect this situation and give a better error message? Is this a combination of using "/send" (which requires --auto... handling) and the --auto... flags not being set? |
|
In this case it is more an issue with any combination of either --auto flags set, or using autoXXX=true parameters, or the automated rest endpoints and then calling the respective none automated rest endpoint again from the controller. e.g. using /issue-credential/send and then calling /issue manually in the automated flow again. Before 0.7.4 that resulted in an exception and nothing more, now the exchange also fails with an webhook event, which was unexpected. Of course such a combination is 100% a controller issue, but on the other hand the manual call should not change the exchange state. Here the auto flow was already successful and calling /issue resulted in an abandoned state. I have to retest this in other combinations to see if this was triggered by a race condition in the exchange, or a failure in the exchanges state machine, which would be bad. Generally I'm tending to move away from any auto flag/parameter combination anyway as this often yields in unexpected results, mostly between the v1 and v2 api's where they sometimes work and sometimes do not. Basically the issue above was because of such a workaround leftover in the code where I had to add a manual step in an auto flow, because it was not fully auto. Current example is the --auto-verify-presentation flag which stopped working for me with 0.7.4 where I should probably open another ticket :) |
|
Sounds good -- thanks. I'm also a believer in getting rid of the "--auto..." flags as well. I prefer to see generic controllers that implement the "--auto" flows, and that also identify where to add custom business logic to override default processing. |
|
Just my $0.02 on this issue - I feel that However I agree with @swcurran that for a production application it's better to have all the In any case, the Regarding the state checking in each endpoint - and creating problem report messages and putting the exchange into So maybe the exception handling needs to be reviewed across all of the credential and proof exchange endpoints? |
|
@ianco Concerning the state checking I notice two things:
|
I'm experiencing issues receiving credentials while testing with the 0.7.4 release candidate.
I was testing with the following setup:
agent1:
agent2:
agent 2 sends credential offer to agent 1. and if agent 1 wants to accept the offer the exchange gets stuck on agent 1 with:
If agent 1 sends a credential proposal to agent 2 and agent 2 accepts the proposal the exchange gets stuck on agent2 with:
The text was updated successfully, but these errors were encountered: