Balances do not match when importing via bunq api.

[OGKevin]

I am running Firefly III version 4.7.5.3

Description
After running import via bunq api. Account balances do not match.

Steps to reproduce
Run the bunq importer via the API and then check its balance.

Extra info
Please add extra info here, such as OS, browser, and the output from the /debug-page of your Firefly III installation (click the version at the bottom).

Bonus points
Earn bonus points by:

• Add a screenshot

minus 8 thousand ? 😅 All these balances make 0 sense. And none are accurate :(

• Replicate the problem on the demo site https://demo.firefly-iii.org/ (bunq import disabled, maybe use sandbox on demo site ? )

[JC5]

Mmm. Incorrect balances are usually caused by incorrect transactions: transfers becoming expenses and deposits. Might that be the case?

The math is pretty much foolproof but if Firefly III is making garbage out of the transactions from the API, the result will be garbage too.

Does the sandbox still have 1 transaction in it?

[OGKevin]

Yea, sandbox will have 0 transactions you have to create them your self. But thats rather EZ. Just create a lil script that makes 10 batch payments of 100 payments. So you can easily create 1k payments. of random amounts to random sandbox users. alpha, bravo, charlie, delta, echo@bunq.com.

And of-course you would need to topup via a request to suggardaddy@bunq.com.

[JC5]

Alright, that's something I can work on. Are the transactions correct otherwise on your import? No weird stuff in Firefly III?

[tbrasser]

I also had a mismatch in the balances between bunq app and firefly after bunq import. I just changed the initial amount until the value matched the actual value. How far back does the bunq api import from? Because it seems that the initial amount is what's causing the mismatch (in my case anyway).

[OGKevin]

I think it goes back until the beginning of the monetary account. I saw that my first transaction ever was also imported.

[tbrasser]

I've now added all my accounts and I'm now seeing the same behaviour. Could it be a result of the internal bunq to bunq account transactions being doubled? #1564

[JC5]

Yes, that's a possibility. I'm just now building something to fill my own account(s). @OGKevin is there any documentation on sugardaddy and the other accounts? I sent out a payment request, but I'm not sure what to expect.

[OGKevin]

@JC5 https://github.com/bunq/tinker_php/blob/d95d22158115881d7f32bb3bff7db37cdbacee70/src/BunqLib.php#L484-L495 here is an example.

You literally just send a request to that email and then it will be auto accepted. The max per request is 500 tho. If you did that and you still didn't get the money that i would wait until https://together.bunq.com/d/3779-encountered-error-while-retrieving-new-sandbox-apikey is resolved and then try again.

[JC5]

Ah very nice!

I requested 10K, I’m a big spender 😎. I’ll tone it down 😋

[JC5]

I've fixed #1564 and I hope the problem is related to this one. Transfers are no longer stored twice so importing data from bunq should result in perfect results.

[OGKevin]

@JC5 ill pull develop and will let you know.

[OGKevin]

hmm, is develop broken ? when i visit http://firefly.ogkevin.net/accounts/create/asset throws

Whoops! An error occurred.

Unfortunately, this error was not recoverable. Firefly III broke. The error is:

Trying to get property 'id' of non-object
Error

This error occured in file /home/firefly/firefly-iii/app/Http/Controllers/Account/CreateController.php on line 87 with code 0.

[JC5]

Did you migrate the database? There are new account types, I believe the controller is looking for them.

[OGKevin]

firefly@OGKevinVPS:~/firefly-iii$ php artisan migrate
**************************************
*     Application In Production!     *
**************************************

 Do you really wish to run this command? (yes/no) [no]:
 > yes

Nothing to migrate.
firefly@OGKevinVPS:~/firefly-iii$

🤔

[JC5]

🤔

Wait! Add --seed.

[OGKevin]

yea that worked, but now after the import they still dont match.

Im now at -7k. And the other ones are also become less, but still far off. I've also noticed that in the dashboard, you will also see accounts that dont belong to you. As in expense accounts e.g.

However none of these are correct tho, as it the amount of euros.

Tested on commit: 566be8d

[JC5]

Mmm, stranger and stranger. Especially that last one.

Could you see if any transactions in particular are wrong? In the meantime, I'll try to build a more complex sandbox environment.

[OGKevin]

I'll take a look at the transactions. You use the payment endpoint to pull transactions i take it ? I can take a look at the code as well and see if i can spot stuff. But thats kinda hard bc you dont use the git flow with pull requests :P.

[tbrasser]

Seems as if there are extra ibans from bunq that get used when using bunq requests (amongst other things?) While they should be from/to one of your own accounts, they end up as extra accounts that are either your own name or "bunq". Not at home now to test this / look it up, but that's what I saw yesterday.

…

________________________________ From: Kevin Hellemun <notifications@github.com> Sent: Thursday, August 16, 2018 5:42:47 PM To: firefly-iii/firefly-iii Cc: Thomas Brasser; Comment Subject: Re: [firefly-iii/firefly-iii] Balances do not match when importing via bunq api. (#1607) I'll take a look at the transactions. You use the payment endpoint to pull transactions i take it ? I can take a look at the code as well and see if i can spot stuff. But thats kinda hard bc you dont use the git flow with pull requests :P. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1607 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACr6MLb0Xn0V0jd0XeWjwncj4bRuP1FBks5uRZL3gaJpZM4V6Bme>.

[JC5]

I know @OGKevin, I'm old fashioned like that. I'll read up on it 👍

@tbrasser, thanks for the pointer!

[JC5]

I've pushed some fixes and it seems my local instance is pretty much up to date after about ~120 transactions. This is just basic stuff though.

Regardless, Firefly III will go back in time in a loop until all transactions are downloaded and will then start moving forward in time to keep up with new transactions. That should already stabilise the import a bit.

If any of you have the time to verify this with your own accounts I'd be grateful!

[tbrasser]

I'll test the dev branch as soon as I'm back from holidays in 2 weeks! Kutgw!

[OGKevin]

I'll give it a test in a bit as well. Will let you know.

[OGKevin]

Hi there,

Firefly III v4.7.6 ran into an error: Maximum execution time of 30 seconds exceeded.

The error was of type "Symfony\Component\Debug\Exception\FatalErrorException".

The error occured on/at: Wed, 29 Aug 2018 15:16:42 +0000.

This error occured in file "/home/firefly/firefly-iii/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php" on line 315 with code 1.

    The error was encountered by user #2, ogkevin@bunq.com.

The IP address related to this error is: 10.8.0.14
URL is: http://firefly.ogkevin.net/import/job/store/1r4J2pznsxws
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15

The full stacktrace is below. If you think this is a bug in Firefly III, you
can forward this message to thegrumpydictator@gmail.com. This can help fix
the bug you just encountered.

If you prefer, you can also open a new issue here:

https://github.com/firefly-iii/firefly-iii/issues

The full stacktrace is below:

#0 {main}

Beep boop,

The Firefly III Mail Robot

PS: This message was sent because a request from IP 10.8.0.14 triggered it.

Got this nice little email 😁

[OGKevin]

So the import didnt finish 🤔

[OGKevin]

Where do you save the last/newer id ?

I've deleted the entries in the preferences table, but the Import is still continuing from that last save point instead of beginning all over again 🤔

[JC5]

The .env file is overwritten at start, so I suggest using -e APP_DEBUG=true -e APP_LOG_LEVEL=debug.

See for more options this file: https://github.com/firefly-iii/firefly-iii/blob/master/.env.docker

I'll expand the wiki.

[tbrasser]

Got it, I was looking in .env.example instead of .env.docker, so I thought some variables weren't exposed. I'll see to it if I can get to it today, otherwise I'll try again tomorrow.

[JC5]

Sure thing, I really appreciate the time you're spending on this.

[tbrasser]

I have a log in which I did the following:

• start firefly
• remove my profile
• make new profile
• set-up bank-accounts
• set bunq api key
• run import

I will email a sharing link to you (gmail) when it's done importing, please don't share my financial details with anyone (A)

[JC5]

Thanks, I saw it. Will dive into it tonight. I appreciate the trust 👍

[bluewalk]

Ran the import and still incorrect, seems better but now other accounts have +10k whilst another account is in debit whilst it's actually not... Curious of @tbrasser's logs will help you.

[JC5]

For those in the thread. I've pushed improved logging. If you set the log level to "info" (using either the .env file or the -e command line options) you should be able to see a more concise overview of what is happening in the log files.

[tbrasser]

I'll generate improved logs when I'm back at my network. Still no VPN road warrior set-up so I can do these things remotely. Get Outlook for Android On Wed, Oct 31, 2018 at 5:11 PM +0100, "James Cole" <notifications@github.com> wrote: For those in the thread. I've pushed improved logging. If you set the log level to "info" (using either the .env file or the -e command line options) you should be able to see a more concise overview of what is happening in the log files. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JC5]

Let's keep this open for now.

[JC5]

@tbrasser When you have time, I would appreciate some feedback on the import process using the new log structure. If you feel uncomfortable using the develop-branch, I suggest I will revive this thread after the release of 4.7.9.

[OGKevin]

I'll give it a try today if I don't forget 👍

[JC5]

That would be cool. I'm especially curious about the DUO and Belastingdienst related payouts.

[tbrasser]

I'm sorry for the delays here. I'm in the middle of a new job (my first 'real' one) and moving. I'm definitely going to use firefly III with Bunq and will test everything here and provide feedback to get it working. I just can't promise on when I can find the time. But really thanks a lot for all the effort here

[JC5]

No problem, just let me know. The new "saving goal" option is also wrecking havoc with the importer so plenty of things to test 😉

[OGKevin]

Alright, so I ran the import. Balances are a lot better but not totally correct yet.

2 of my accounts, my "savings" and "monthly expenses" accounts now have the correct balance.

My saving goals accounts are a little short approx 10 euros short. While my main account is way of. It says I have -90 euro on my account.

The BelastingDienst payments seem to be alright.

[OGKevin]

I re-ran it tonight, this time included all my accounts. The balances are miss matches now, including the ones that succeeded yesterday :(.

The importer crashes with the following error:

automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | [2018-12-13 21:23:04] production.NOTICE: Done storing. Firefly III has stored 1741 transactions.
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | 2018-12-13 21:23:04,748 DEBG 'apache2' stdout output:
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | [2018-12-13 21:23:04] production.NOTICE: Done storing. Firefly III has stored 1741 transactions.
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    |
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | [2018-12-13 21:23:05] production.ERROR: The import storage routine crashed: Undefined index: zoom_level
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | 2018-12-13 21:23:05,000 DEBG 'apache2' stdout output:
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | [2018-12-13 21:23:05] production.ERROR: The import storage routine crashed: Undefined index: zoom_level
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    |
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | [2018-12-13 21:23:05] production.ERROR: #0 /var/www/firefly-iii/app/Http/Controllers/Import/JobStatusController.php(215): FireflyIII\Http\Controlle$
s\Import\JobStatusController->storeTransactions(Object(FireflyIII\Models\ImportJob))
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | #1 [internal function]: FireflyIII\Http\Controllers\Import\JobStatusController->store(Object(FireflyIII\Models\ImportJob))
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | #2 /var/www/firefly-iii/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array)
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | #3 /var/www/firefly-iii/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction('st$
re', Array)
automation_firefly_iii_app.1.lh7ftyn990u2@automation-prod-s-1vcpu-1gb-ams3-01    | #4 /var/www/firefly-iii/vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Il$
uminate\Routing\Route), Object(FireflyIII\Http\Controllers\Import\JobStatusController), 'store')

[JC5]

I'll do some digging this weekend. I've been working on the API mostly for the last few weeks.

[JC5]

I finally ran into this problem myself. I have two accounts who are off by one euro. Stay tuned.

[JC5]

Storing the same auto-save amount twice in one day fails. The second one is probably thought to be a duplicate.

[JC5]

This is now fixed. So far, no more problems here.