run Gittip #17

[chadwhitacre]

16

[chadwhitacre]

Backup taken and verified.

[chadwhitacre]

Kicked off payday script.

[chadwhitacre]

Payin loop done. Doing payouts.

[chadwhitacre]

pid-2821 thread-140735081639104 (MainThread) Traceback (most recent call last):
pid-2821 thread-140735081639104 (MainThread)   File "/Users/whit537/personal/gittip/www.gittip.com/gittip/cli.py", line 19, in payday
pid-2821 thread-140735081639104 (MainThread)     Payday(db).run()
pid-2821 thread-140735081639104 (MainThread)   File "/Users/whit537/personal/gittip/www.gittip.com/gittip/billing/payday.py", line 111, in run
pid-2821 thread-140735081639104 (MainThread)     self.payout(ts_start, genparticipants(False))
pid-2821 thread-140735081639104 (MainThread)   File "/Users/whit537/personal/gittip/www.gittip.com/gittip/billing/payday.py", line 199, in payout
pid-2821 thread-140735081639104 (MainThread)     self.ach_credit(ts_start, participant, tips, total)
pid-2821 thread-140735081639104 (MainThread)   File "/Users/whit537/personal/gittip/www.gittip.com/gittip/billing/payday.py", line 434, in ach_credit
pid-2821 thread-140735081639104 (MainThread)     assert balance is not None, balance # sanity check
pid-2821 thread-140735081639104 (MainThread) AssertionError

[chadwhitacre]

This is due to people creating accounts during the payin loop, as we saw on #169.

[sigmavirus24]

Perhaps queue up new account creation during the loop?

[chadwhitacre]

We're a victim of our own success here. Gittip is getting some strong traffic at the moment (~160 active as I write). We need to:

1. get payday done
2. still and yet clean up the payday script to not break like this (allow participants to be created during payday #170)

[chadwhitacre]

The good news is that payins are done. Here's my plan for fire-fighting (1):

1. repair the database
2. comment out the payins loop
3. re-run payday (only running payouts loop)

[chadwhitacre]

So what database repair is needed?

[chadwhitacre]

Here's the flow of this bug:

• Someone creates an account during the payin loop. Default values for balance and pending are 0.0 and NULL, respectively.
• 0.0 + NULL = NULL, so during clear_pending_to_balance, balance is set to NULL
• The new user fails the sanity check in the payouts loop

[chadwhitacre]

The repair is to set balance to 0.0 where it is NULL. We have 31 of these. There are 8087 records in participants overall (up from 8070 a few minutes ago).

[chadwhitacre]

=> update participants set balance=0.0 where balance is null;
UPDATE 31

(1): Done.

[chadwhitacre]

Now to satisfy myself that re-running the payouts loop will work, and then to run it.

[chadwhitacre]

I commented out payin() and clear_pending_to_balance() and am rerunning the payday script.

I neglected to also comment out zero_out_pending(). This should mean that all records will have pending 0.0, except those that are added while the payout loop is running. After payout is done I will reset those to NULL.

[chadwhitacre]

Okay, payouts finished, and I reset pending to NULL. Calling it a day on Gittip 17.

[chadwhitacre]

And, for the record, I did remember to uncomment payin() and clear_pending_to_balance(). ;-)

[chadwhitacre]

@sigmavirus24 See #170 for fix for the issue that bit us here.