run Gratipay 135 #3045

chadwhitacre · 2015-01-01T16:53:34Z

134

chadwhitacre · 2015-01-01T17:03:48Z

Droplet spun up and updated.

chadwhitacre · 2015-01-01T17:34:08Z

47 (37) accounts reviewed.

chadwhitacre · 2015-01-01T17:46:11Z

Backup taken and verified.

chadwhitacre · 2015-01-01T17:50:39Z

Script started.

chadwhitacre · 2015-01-01T17:51:21Z

Thu Jan  1 17:46:52 UTC 2015
pid-1785 thread-139755151517440 (MainThread) Starting a new payday.
pid-1785 thread-139755151517440 (MainThread) Payday started at 2015-01-01 17:47:07.498402+00:00.
pid-1785 thread-139755151517440 (MainThread) Greetings, program! It's PAYDAY!!!!
pid-1785 thread-139755151517440 (MainThread) Prepared the DB.
pid-1785 thread-139755151517440 (MainThread) Traceback (most recent call last):
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 28, in payday
pid-1785 thread-139755151517440 (MainThread)     Payday.start().run()
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 133, in run
pid-1785 thread-139755151517440 (MainThread)     self.payin()
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 157, in payin
pid-1785 thread-139755151517440 (MainThread)     holds = self.create_card_holds(cursor)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 427, in create_card_holds
pid-1785 thread-139755151517440 (MainThread)     holds = self.fetch_card_holds(participant_ids)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 407, in fetch_card_holds
pid-1785 thread-139755151517440 (MainThread)     cancel_card_hold(hold)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/exchanges.py", line 236, in cancel_card_hold
pid-1785 thread-139755151517440 (MainThread)     hold.save()
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/balanced/resources.py", line 177, in save
pid-1785 thread-139755151517440 (MainThread)     resp = method(href, data=attrs)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 460, in put
pid-1785 thread-139755151517440 (MainThread)     return self._op(self.interface.put, uri, data=data, **kwargs)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 512, in _op
pid-1785 thread-139755151517440 (MainThread)     handle_error(ex)
pid-1785 thread-139755151517440 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 489, in handle_error
pid-1785 thread-139755151517440 (MainThread)     raise ex
pid-1785 thread-139755151517440 (MainThread) HTTPError: HTTPError(status='Not Found', category_code='not-found', description='<p>The requested URL was not found on the server.</p><p>If you entered the URL manually please check your spelling and try again.</p> Your request id is OHM43439a0691de11e484ce02b1203540

Looks like we're picking up an old card hold or something?

chadwhitacre · 2015-01-01T18:05:46Z

I have about 30 minutes before I have to leave to pick up @clone1018 @seanlinsley.

chadwhitacre · 2015-01-01T18:06:04Z

So what's the URL we tried to hit and why? And why is it 404?

chadwhitacre · 2015-01-01T18:10:30Z

So CardHold.query.filter(CardHold.f.meta.state == 'new') returned something. What? Why?

chadwhitacre · 2015-01-01T18:17:32Z

$ heroku run python
>>> from gratipay import wireup
>>> wireup.billing(wireup.env())
>>> from balanced import CardHold
>>> holds = list(CardHold.query.filter(CardHold.f.meta.state == 'new'))
>>> len(holds)
568

chadwhitacre · 2015-01-01T18:26:13Z

All of these holds are from last week, and are all status='failed':

>>> list(set([h.created_at[:10] for h in holds]))
['2014-12-25']
>>> list(set([h.status for h in holds]))
['failed']
>>>

chadwhitacre · 2015-01-01T18:26:50Z

Some were updated this week. Presumably those before which ever one was 404.

>>> list(set([h.updated_at[:10] for h in holds]))
['2015-01-01', '2014-12-25']
>>>

chadwhitacre · 2015-01-01T18:27:43Z

At this point I'm in a rush. I say we hotpatch to return silently if canceling a hold 404s.

chadwhitacre · 2015-01-01T18:29:21Z

>>> updated = [h.updated_at[:10] for h in holds]
>>> updated.count('2015-01-01')
2
>>> updated.count('2014-12-25')
566
>>>

chadwhitacre · 2015-01-01T18:41:22Z

Sent my brother and dad to pick up @clone1018 and @seanlinsley.

chadwhitacre · 2015-01-01T18:41:47Z

So I'm in a bit less of a rush. :-)

Changaco · 2015-01-01T18:46:42Z

The problem is that these holds are marked as failed but they don't have a failure_reason, which is what the fetch_card_holds() uses to detect failure.

>>> list(set([h.failure_reason for h in holds]))
[None]

chadwhitacre · 2015-01-01T18:46:54Z

Hypothesis: this is a regression introduced in #3011.

chadwhitacre · 2015-01-01T18:48:04Z

@Changaco Hypothesis is that the Balanced API changed subtly here?

Changaco · 2015-01-01T18:49:45Z

Upstream issue: balanced/balanced-api#461.

chadwhitacre · 2015-01-01T18:49:46Z

Do we perhaps clobber failure_reason somehow when setting state='failed'?

chadwhitacre · 2015-01-01T18:51:05Z

Ah, I see: h.status == 'failed' is core Balanced API, and `h.meta['state'] == 'failed' is Gratipay API.

chadwhitacre · 2015-01-01T18:52:18Z

@Changaco So it seems that we should look for h.status == 'failed' instead of h.failure_reason when propagating to h.meta['state']. Yes?

chadwhitacre · 2015-01-01T18:54:11Z

I'm gonna try that.

Changaco · 2015-01-01T18:55:34Z

@whit537 I think we should look for either one.

chadwhitacre · 2015-01-01T18:55:35Z

Rerunning with this patch:

diff --git a/gratipay/billing/payday.py b/gratipay/billing/payday.py                                    
index 9095491..197f32f 100644                                                                           
--- a/gratipay/billing/payday.py                                                                        
+++ b/gratipay/billing/payday.py                                                                        
@@ -390,7 +390,7 @@ class Payday(object):                                                               
         holds = {}                                                                                     
         for hold in CardHold.query.filter(CardHold.f.meta.state == 'new'):                             
             state = 'new'                                                                              
-            if hold.failure_reason:                                                                    
+            if hold.status == 'failed':                                                                
                 state = 'failed'                                                                       
             elif hold.voided_at:                                                                       
                 state = 'cancelled'

chadwhitacre · 2015-01-01T18:57:37Z

Though I'm still not satisfied as to why we're seeing this now: balanced/balanced-api#461 is from a year ago.

chadwhitacre · 2015-01-01T19:01:25Z

Why do we have 568 holds left over from last week? Is that normal behavior? Do we have a leak of some kind?

chadwhitacre · 2015-01-01T19:02:30Z

Rerunning ...

chadwhitacre · 2015-01-01T19:03:26Z

pid-2086 thread-140048078632704 (MainThread) Picking up with an existing payday.
pid-2086 thread-140048078632704 (MainThread) Payday started at 2015-01-01 17:47:07.498402+00:00.
pid-2086 thread-140048078632704 (MainThread) Greetings, program! It's PAYDAY!!!!
pid-2086 thread-140048078632704 (MainThread) Prepared the DB.
pid-2086 thread-140048078632704 (MainThread) Traceback (most recent call last):
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 28, in payday
pid-2086 thread-140048078632704 (MainThread)     Payday.start().run()
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 133, in run
pid-2086 thread-140048078632704 (MainThread)     self.payin()
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 157, in payin
pid-2086 thread-140048078632704 (MainThread)     holds = self.create_card_holds(cursor)
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 427, in create_card_holds
pid-2086 thread-140048078632704 (MainThread)     holds = self.fetch_card_holds(participant_ids)
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 401, in fetch_card_holds
pid-2086 thread-140048078632704 (MainThread)     hold.save()
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/balanced/resources.py", line 177, in save
pid-2086 thread-140048078632704 (MainThread)     resp = method(href, data=attrs)
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 460, in put
pid-2086 thread-140048078632704 (MainThread)     return self._op(self.interface.put, uri, data=data, **kwargs)
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 512, in _op
pid-2086 thread-140048078632704 (MainThread)     handle_error(ex)
pid-2086 thread-140048078632704 (MainThread)   File "/home/whit537/gratipay.com/env/lib/python2.7/site-packages/wac.py", line 489, in handle_error
pid-2086 thread-140048078632704 (MainThread)     raise ex
pid-2086 thread-140048078632704 (MainThread) HTTPError: HTTPError(status='Not Found', category_code='not-found', description='<p>The requested URL was not found on the server.</p><p>If you entered the URL manually please check your spelling and try again.</p> Your request id is OHMbc180a8491e811e49dde06429171ffad.', status_code=404, category_type='request', request_id='OHMbc180a8491e811e49dde06429171ffad', response=None)

chadwhitacre · 2015-01-01T19:03:44Z

This time it's in the other hold.save() call.

chadwhitacre · 2015-01-01T19:05:15Z

How the heck are we given a hold from CardHold.query.filter that then is 404 when we turn right around and try to save it?

Changaco · 2015-01-01T19:37:28Z

@whit537 I suggest rerunning with fetch_card_holds() bypassed entirely:

--- a/gratipay/billing/payday.py
+++ b/gratipay/billing/payday.py
@@ -424,7 +424,7 @@ class Payday(object):

         # Fetch existing holds
         participant_ids = set(p.id for p in participants)
-        holds = self.fetch_card_holds(participant_ids)
+        holds = {}

         # Create new holds and check amounts of existing ones
         def f(p):

chadwhitacre · 2015-01-01T20:26:42Z

Restarted with ...

diff --git a/gratipay/billing/payday.py b/gratipay/billing/payday.py                                    
index 9095491..307d128 100644                                                                           
--- a/gratipay/billing/payday.py                                                                        
+++ b/gratipay/billing/payday.py                                                                        
@@ -423,8 +423,8 @@ class Payday(object):                                                               
             return {}                                                                                  

         # Fetch existing holds                                                                         
-        participant_ids = set(p.id for p in participants)                                              
-        holds = self.fetch_card_holds(participant_ids)                                                 
+        #participant_ids = set(p.id for p in participants)                                             
+        holds = {} # self.fetch_card_holds(participant_ids)                                            

         # Create new holds and check amounts of existing ones                                          
         def f(p):

chadwhitacre · 2015-01-01T20:38:21Z

Script ran for 11 minutes (0:11:06.399468).

chadwhitacre · 2015-01-01T20:57:07Z

Pinned https://twitter.com/Gratipay/status/550757281953251330.

chadwhitacre · 2015-01-01T21:17:09Z

Logs downloaded and uploaded. Droplet destroyed.

chadwhitacre · 2015-01-01T21:21:44Z

MassPay done and posted back for 76 users.

chadwhitacre · 2015-01-01T21:31:12Z

Escrow shuffled.

chadwhitacre added the ★★★ label Jan 1, 2015

chadwhitacre mentioned this issue Jan 1, 2015

hold.{save,cancel} is 404 immediately after .query? balanced/balanced-api#727

Closed

chadwhitacre closed this as completed Jan 1, 2015

seanlinsley mentioned this issue Jan 8, 2015

run Gratipay 136 #3088

Closed

Changaco mentioned this issue Jan 8, 2015

Fix detection of failed credit card holds #3091

Merged

chadwhitacre added the Payday label Jan 21, 2016

run Gratipay 135 #3045

run Gratipay 135 #3045

Comments

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

Changaco commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

Changaco commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

Changaco commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

Changaco commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015

chadwhitacre commented Jan 1, 2015