Payment made, but *not* deducted from user account?

[pjf]

Opening this for FreshDesk 459. We have a user who has reported that they've received funds from Gittip, but these funds were not deducted from their Gittip account. I've confirmed that this has indeed happened.

I very this as a critical thing to investigate before the next payday run, as it may be that gittip is (or has been) leaking money, which is very, very bad.

Attn @patcon and @whit537 at the very least.

[chadwhitacre]

★★★

[seanlinsley]

They gave permission to be identified on the ticket, so this is https://www.gittip.com/NuvolaPlayer/, here on Github as @fenryxo

[seanlinsley]

Looking at the history, it doesn't appear that PayPal MassPay happened for Gittip 101, on May 8th.

[chadwhitacre]

Gittip 101 is #2362.

[chadwhitacre]

Hunch: there was a failure (user error?) in posting MassPay back to Gittip for May 9.

[chadwhitacre]

Let's spot check some other MassPay folks ...

[chadwhitacre]

select * from exchanges
where timestamp > '2014-05-08'::timestamptz and timestamp < '2014-05-10'::timestamptz
order by timestamp desc limit 1000;

Looks like only 6 out of 44 MassPay records (14%) posted back to the database on May 9. :-(

[chadwhitacre]

select * from exchanges where note like 'PayPal%' order by timestamp desc;

Spot check indicates that May 9 was the only time this happened.

[chadwhitacre]

So probably the thing to do is to post those old payments. If we're lucky (hah!) there's enough in everyone's account this week to accommodate (won't hit negative balance constraint, #2389).

[seanlinsley]

I haven't looked at the data, but I would've assumed that anyone who didn't get MassPay during 101 got the extra balance added to MassPay during 102, like it did for NuvolaPlayer.

[chadwhitacre]

@seanlinsley Yeah, but since I haven't run MassPay for this week yet, they'll have accumulated more since last week.

[seanlinsley]

I don't follow. The extra balance would have accrued between 100 and 101, since MassPay failed in 101. Then the next week (last week, 102), they should have gotten their money from both 101 and 102. I don't see how they would've accumulated more between 102 and the current 103.

[chadwhitacre]

Why wouldn't they accumulate more during 103? We haven't flushed 103 to MassPay yet.

[seanlinsley]

It seems that 103's MassPay would just be their normal amount, no?

[chadwhitacre]

Right, so we can deduct the payout we did in 101 but failed to record, and they won't get a payout this week.

[chadwhitacre]

Right?

[chadwhitacre]

Although I'd like to satisfy myself that we actually recorded all the payouts properly last week and other weeks first.

[seanlinsley]

Right, so we can deduct the payout we did in 101 but failed to record (...)

Oh? I was under the assumption that we didn't do MassPay for them at all for 101. But the idea that we simply failed to record them certainly makes more sense 😅

[chadwhitacre]

Yeah, the way MassPay works we:

1. Download info from Gittip.
2. Upload a CSV to PayPal.
3. Post back to Gittip.

It was only step 3 that (partially) failed during 101.

[pjf]

Popping into failure analysis mode, how does step 3 usually happen? I imagine we want a change request somewhere that rings giant bells if it doesn't complete, and potentially a second which allows for the procedure to be restarted safely if it bails out part-way through.

I agree that re-doing the transactions for 101 now makes sense. There is the question as to what should be done if anyone's balance goes negative because of that, but it may be worthwhile to see how many people will actually fall into that state first. If it's only a handful (or less) then we may be able to deal with them on a case-by-case basis.

[chadwhitacre]

Step 3 happens by me running ./bin/masspay.py -p on my laptop. One improvement would be to add a check to that script that queries the database for the number of exchanges just recorded and compares that to the number in the report from PayPal.

[chadwhitacre]

=> \i foo.sql
┌────────────┬───────┐
│    date    │ count │
├────────────┼───────┤
│ 2014-05-15 │    48 │
│ 2014-05-09 │     6 │
│ 2014-05-01 │    41 │
│ 2014-04-24 │    43 │
│ 2014-04-18 │    44 │
│ 2014-04-10 │    45 │
│ 2014-04-04 │    40 │
│ 2014-03-31 │    40 │
│ 2014-03-20 │    35 │
│ 2014-03-13 │    35 │
│ 2014-03-06 │    35 │
│ 2014-02-28 │    36 │
│ 2014-02-20 │    30 │
│ 2014-02-14 │    30 │
│ 2014-02-06 │    25 │
│ 2014-01-31 │    27 │
│ 2014-01-23 │    24 │
│ 2014-01-16 │    22 │
│ 2014-01-09 │    19 │
│ 2014-01-02 │    17 │
│ 2013-12-26 │    17 │
│ 2013-12-20 │    17 │
│ 2013-12-13 │    14 │
│ 2013-12-05 │    13 │
│ 2013-11-28 │    12 │
│ 2013-11-21 │    13 │
│ 2013-11-15 │    12 │
│ 2013-11-09 │     9 │
│ 2013-10-31 │     5 │
│ 2013-10-24 │     4 │
└────────────┴───────┘
(30 rows)

SELECT date, count(date) FROM
(
    SELECT timestamp::date AS date FROM exchanges
    WHERE note LIKE 'PayPal MassPay to%'
) AS foo
GROUP BY date
ORDER BY date DESC

[chadwhitacre]

$ wc -l *.gittip.csv
      45 2014-04-10.output.gittip.csv
      44 2014-04-17.output.gittip.csv
      44 2014-04-18.output.gittip.csv
      43 2014-04-24.output.gittip.csv
      41 2014-05-01.output.gittip.csv
      44 2014-05-09.output.gittip.csv
      48 2014-05-15.output.gittip.csv
     309 total
$

(The rest aren't handy right now.)

[chadwhitacre]

The only counts off in recent history are for 2014-05-09.

The 17th/18th duplicate is because the masspay.py script depends on the current date in the filename so if I run it the day after payday then I cp the file to get the filename right so it gets picked up.

[chadwhitacre]

Okay! I rolled out #2434, so now admins can record exchanges that push balances into the negative. I hand-edited a CSV and ran ./bin/masspay.py -p to record the missing MassPays for the remaining 14 participants. I added an additional note: "Recorded late. See #2417."