HIGH IMPACT: Cache the results on the phone (both platforms)

[shankari]

Our results are currently computed on the server side and sent over to the client at the following times:

• when there are no trip left to confirm
• when the user clicks on the "RESULTS" option in the menu bar

In both these cases, the phone makes a HTTP POST call to the "/compare" endpoint, which returns a HTML page. The page is then displayed in a webview on the appropriate platform.

This was an early attempt by me to work cross-platform, and to:

• make it easier to update the app without having to release a new version every time, and
• to customize the result view for the type of user

This works great, but is very slow, so slow that a lot of times, people just don't wait for it to finish loading. Part of the slowness was due to the time taken to compute the results, so I checked in a fix to pre-compute the results daily on the server side.
issue: https://github.com/e-mission/e-mission-server/issues/23
fix: e-mission/e-mission-server#25

However, even after that fix, the result view load is slow. I suspect that this is due to the time required to retrieve the associated .css and .js files. I had hoped that these would be cached on the phone, so the retrieve time would be negligible, but it looks like even the version check required to determine whether a new version is available takes time, specially since there are multiple of these checks (see snippet below).

Pre-fetching the results to the phone should move this delay to the background and result in a more responsive UI for the user.

The idea is that you would add this additional POST call to the existing remote sync mechanism

• android: https://github.com/e-mission/e-mission-phone/blob/master/android/app/src/main/java/edu/berkeley/eecs/e_mission/data_sync/ConfirmTripsAdapter.java#L80
• ios: https://github.com/e-mission/e-mission-phone/blob/master/iOS/CFC_Tracker/CFC_Tracker/AppDelegate.m#L163
  store the result on the local disk, and then change the display code load from local disk instead.

One thing to watch out for: the returned HTML document currently has relative references to javascript code, i.e.
<script src="clients/choice/front/ionic.bundle.js" charset="utf-8"></script>
so you would need to parse the returned HTML, issue the appropriate GET calls as well and store those in the appropriate locations on the local disk as well.

[neerajbaid]

Assuming you're referring to this screen,

Could we instead simply make this a native using a framework like this(iOS) this(Android)?
Beneficial for a number of reasons:

1. Much reduced network usage: only fetch a JSON rather than entire HTML/CSS/whatever.
2. Vastly improved UI.
3. Remove the need for hacky solutions like parsing HTML, fetching and caching more documents, etc.
4. JSON can simply be cached in NSUserDefaults due to its low size.

To address the reasons you picked the current solution:

1. What sort of OTA updates did you have in mind? Seems like the purpose this chart fills is quite core to the app.
2. I'm sure any sort of personalization could be communicated over JSON and then properly interpreted by the device.

I imagine it would be fairly easy to have the backend simply directly send the data it's using to create that HTML rather than sending the HTML itself. And I don't expect implementing this natively would take much time since we have frameworks doing all the heavy lifting.

[shankari]

1. What sort of OTA updates did you have in mind? Seems like the purpose this chart fills is quite core to the app.

I was going to switch your result screen automagically on the server to show you, but I don't know which email address you have used to register with e-mission. If you tell me the address, I can do it tomorrow. But basically, by flipping a single field in your profile on the server, I can switch your result screen from the data view to the game view.

This is useful for performing research studies since a lot of research involves measurement of people's perceptions and reactions to changes. For example, we used this feature to run a study on whether people prefer data or a game or the ability to choose between the two. This made it much easier to run that kind of A/B testing.

1. I'm sure any sort of personalization could be communicated over JSON and then properly interpreted by the device.

It absolutely can, but at the cost of publishing a new version of the app. This is not a problem for the android version of the app, but for the iOS version, updates have to pass review by the Review Board, which is always really frustrating. For example, the most recent update that we submitted to the App Store was rejected because "we depend on Moves for our functionality".

I tried to argue that:

• there are a ton of other apps using the data from moves and a quick sampling shows several that require you to log in to moves before doing anything
• the app store review process approved at least two earlier versions of the app, both of which also used Moves in the same way
• we extend Moves' functionality to perform classification of motorized modes so we are not duplicating their functionality

But no dice. Basically, at this point, we cannot publish any changes to the iOS app store unless it includes some form of data collection.

And this was a problem because a group in the ITS department wanted to use e-mission to conduct a survey on carshare behavior, and this was our opportunity to get a larger deployment that we could showcase. And they really really wanted a result screen that was unique to their study.

Gautham had the brilliant idea of supporting manual trip entry as the basic functionality and saying that Moves provided automatic trip entry, which was an enhancement. I stayed up all night trying to get that to work in time, and then I had an even more brilliant idea of just returning a different HTML based on the user, which would require NO client updates.

So if the app review process were smoother and more transparent, it would absolutely make sense to change the client locally and send only the data over.

But given the current process, I like the flexibility of changing the part that the outside researchers care about (the result screen) quickly and without any uncertainty.

Thoughts?

[neerajbaid]

Any sort of code change will need to pass App Review (obviously that includes small things, like this HTML caching), so we might as well do the fix correctly. Keep in mind that using any major web views puts you at the mercy of a 10.6 rejection. (Yes, 10.6 is incredibly broad). As you know, App Review can be very inconsistent.

Couldn't we simply add a Boolean in the JSON we sync to the client, telling the device which view (graph vs. game) to show? And we could also sync over the URL of the image we want to show for the game, to minimize client-side hardcoding and maximize future flexibility. The backend could simply manage which stage of the game a user is in (which I'm sure it does right now) and translate that to an image URL. And, of course, we wouldn't break previous versions of the iOS app--it's easy to simply version the API.

The email address I'm using is wdaareg@gmail.com.

Do you expect the researchers' need to change frequently and without notice?

[shankari]

Any sort of code change will need to pass App Review (obviously that includes small things, like this HTML caching),

Agreed. But once we have the HTML caching in place, we don't need client side code changes in order to change the result UI.

Couldn't we simply add a Boolean in the JSON we sync to the client, telling the device which view (graph vs. game) to show?

We could, and that's what happens on the server side (at a very high level). But note the implication that we need to have the code for every single potential result in the client then. With the webview, we can come up with completely new client views (maybe a set of animated images) without having the modify the client view in any way.

we might as well do the fix correctly
I guess it is just not obvious to me that "correctly" means native in this case. Like in most systems, you design for what your goals, requirements and constraints are.

In our case, these are:

• Goal: we want to get reasonably large amounts of actionable policy data. Given our current setting, we believe that the easiest way to accomplish this is through building a platform to run travel behavior studies.
• Constraint: we have one person working on this full-time, and the UI design is not her primary focus or research goal. Additional not-very-technical resources (people who can write python programs and javascript) are available on a very short term basis
• Requirements:
  • Customization: Each study wants the ability to show its own results. My current impression, and Prof. Culler's impression from working in these areas before is that people outside CS care a lot about the UI. But that doesn't mean that they have the skillz to do a great UI design - they really just know javascript. But they care a lot about what information is displayed in the UI.
  • Flexibility: The UI needs to adapt fairly quickly based on feedback from their pre-study.
  • Timeliness: Studies are invariably running late, and grants have timelines within which they need to be used. Having the ability to deploy a study in a pre-determined time frame is, IMHO, crucial.

Let's say we had somebody from the ITS team come by and say that they wanted to deploy a study in the next 2-3 weeks that had a game with some more graphs on it, so that when you clicked the score, it showed you the components of the score. If we were doing native result views, what would you say to them?

[shankari]

Do you expect the researchers' need to change frequently and without notice?

I don't know for sure, since so far, our deployment rate is close to zero. We had one pre-survey and they decided not to use smartphone collection for the full survey. But even for the pre-survey, they had a requirements around the results and the filtering, and they would bring them up at arbitrary times, and they would not make progress on the deployment until they saw the results.

Thanks,
Shankari

[neerajbaid]

This all makes sense. I think it's just very weird for me to work on a mobile app where building the best user-centric product is not a goal, but these are also very different constraints than usual.

I will look further into the best way to execute your caching solution.

[shankari]

Yeah, I think that this is a change is focus from building an app to
building a platform and is hopefully a good learning experience for you :)

At the same time, you have certainly raised my awareness towards looking
beyond the travel diary/travel study world and making whatever I build
accessible to a wider audience by packaging it as a native SDK.

If you are open to it, I'd be interested in brainstorming further about
what the API for such an SDK should look like - it is hard to strike a good
balance between ease of use and flexibility.

On Tue, Feb 10, 2015 at 3:40 PM, Neeraj Baid notifications@github.com
wrote:

This all makes sense. I think it's just very weird for me to work on a
mobile app where building the best user-centric product is not a goal, but
these are also very different constraints than usual.

I will look further into the best way to execute your caching solution.

—
Reply to this email directly or view it on GitHub
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-73808450
.

[jesca]

Is there a strong reason why we are using HttpClient rather than HttpUrlConnection for displaying the result summary? There's a handy platform-provided cache, the HttpResponseCache that seems handy for this issue. However, it doesn't work with HttpClient, only HttpUrlConnection.

See: http://developer.android.com/reference/android/net/http/HttpResponseCache.html

Also, Apache HttpClient should be considered deprecated for anything later than Gingerbread.

[shankari]

If I don't have a comment explaining it, there is no strong reason. Feel free to change to HttpUrlConnection.

I'll look through my previous commit logs (this project was initially hosted internally, and then moved to github without the previous commit history) to confirm that there is no deep reason.

[zacatac]

After working for some time with the HttpResponseCache I learned that POST requests are not meant cached by default. So it seems we have two options, (1) we can change the api s.t. getting the display is done with a GET rather than a POST request. Or (2) we can do some kind of manual caching of the raw html that we receive. That would involve writing that html file to the file system and then checking for staleness manually.

I am not actually sure how this was accomplished on iOS, because I think the problem would be the same in that case. The problem being that each time the view is loaded, a new request must be made because the previous request was not cached.

Of the two options, I think that (1) changing the api call makes more sense. We are only getting data with the /compare call, and there are no server-side side effects, so a GET request makes sense.

[jesca]

A POST request isn't the right way to "get" information. We are retrieving information from the server, so technically we shouldn't even be using a POST request for this task. It also makes sense that POST request can't be cached. I propose that we fix it by going with option 1)

[shankari]

Aha! Great detective work, Zack.

This might be why the cache didn't work for Neeraj. He said that he got the
"Spammer don't hack me message".
https://github.com/e-mission/e-mission-server/blob/master/CFC_WebApp/api/cfc_webapp.py#L71

He probably tried to do a "GET" on the compare URL, which would obviously
trigger that code path since it wasn't one of the supported GET endpoints.

He worked around it by loading the WebView in the background as soon as the
app was launched, instead of waiting for the trips to be confirmed. That is
not a general enough solution, IMHO, since if the user opens the app to see
the results when there are no trips to be confirmed, this wouldn't help.
But we agreed that he would submit a pull request for now, and I'd come up
with a flow diagram for the iOS OAuth code, which is pretty convoluted, and
revisit it.

On Wed, Feb 18, 2015 at 6:43 PM, zacatac notifications@github.com wrote:

After working for some time with the HttpResponseCache I learned that POST
requests are not meant cached by default. So it seems we have two options,
(1) we can change the api s.t. getting the display is done with a GET
rather than a POST request. Or (2) we can do some kind of manual caching of
the raw html that we receive. That would involve writing that html file to
the file system and then checking for staleness manually.

I am not actually sure how this was accomplished on iOS, because I think
the problem would be the same in that case. The problem being that each
time the view is loaded, a new request must be made because the previous
request was not cached.

—
Reply to this email directly or view it on GitHub
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-74990714
.

[zacatac]

@neerajbaid what happens when you close out the results pane by traveling to another screen (e.g. settings) and then come back to the results screen? Aren't you reloading each time you go through a flow like that?

[shankari]

Also, my apologies for not catching the non-cachableness of POST requests during our earlier discussions/design review.

[zacatac]

No worries. This gives us the chance to look at some different approaches. One idea to compensate for this is to have some kind of persistent data structure that stores the data that we receive from the /compare call. That way we can display the data immediately, and then update once the request completes. I suppose this would interfere with the customizability of the views though since only data and not visualizations would be sent back and forth.

[shankari]

(2) we can do some kind of manual caching of the raw html that we receive. That would involve writing that html file to the file system and then checking for staleness manually.

This was what I suggested originally because even with a HTTP cache, the phone still has to contact the server to determine freshness. If we have multiple javascript files, that is multiple liveness checks (client -> server calls) happening in user visible time. Depending on how the cache is implemented, it also means that the results could unusable if the network connection is spotty or absent because the phone won't be able to contact the server. Do you know whether the cache returns existing values if the server is unreachable?

It didn't seem like this would be too much work - all the javascript files are defined upfront in the headers, so you just need to parse the html HEAD tag to recreate the structure on the server and then load from disk.

But I think that it might be too much work at this stage in the project.

[zacatac]

I don’t think that the cache will contact the server to realize freshness. Since the cache is stored phone-side we can specify how long the cached requests stays fresh. That value is then stored at the phone-side cache and checked whenever we make a new request to that same resource, so until it is stale we wouldn’t make external requests. 

I suppose that the trouble with storing the html file locally is that we basically need to reimplement what the httpResponseCache handles for us, which I imagine will be quite a lot of work. At least on iOS Neeraj was saying that this is extremely difficult.

On Wed, Feb 18, 2015 at 7:40 PM, shankari notifications@github.com
wrote:

(2) we can do some kind of manual caching of the raw html that we receive. That would involve writing that html file to the file system and then checking for staleness manually.
This was what I suggested originally because even with a HTTP cache, the phone still has to contact the server to determine freshness. If we have multiple javascript files, that is multiple liveness checks (client -> server calls) happening in user visible time. Depending on how the cache is implemented, it also means that the results could unusable if the network connection is spotty or absent because the phone won't be able to contact the server. Do you know whether the cache returns existing values if the server is unreachable?

It didn't seem like this would be too much work - all the javascript files are defined upfront in the headers, so you just need to parse the html HEAD tag to recreate the structure on the server and then load from disk.

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-74994639

[shankari]

A POST request isn't the right way to "get" information. We are retrieving information from the server, so technically we shouldn't even be using a POST request for this task. It also makes sense that POST request can't be cached. I propose that we fix it by going with option 1)

Good point on GET versus POST. But if you recall our discussion in class, this particular method returns privacy sensitive information, so we want to ensure that it is called with a JWT and I picked POST because we were "sending" a JWT and the response was to retrieve the user specific information. We clearly don't want to send the JWT as a query param - a snooper is going to be able to read that user's information until the JWT expires.

It also ensured that there was a clear separation between methods that returned user specific data and methods that returned public data.

Having said that, there might be a way to pass the JWT in the GET headers.

I have to go have dinner with my family now, but maybe you could spend ~ 10-15 minutes playing around with what would happen if it did become a GET. Change the route on the server side, and try to put the token into a header and see if the cache works then.

If we know that it works technically, we can argue the philosophy later :)

[zacatac]

Sounds good. We will get started on that.

[shankari]

Great. Remember: no token in query params. But I think that the header route should work. Also, you need to change the code that validates the JWT to read it from the headers instead of the post message. @jesca knows exactly where the JWT is validated :)

[shankari]

So my understanding is that using GET with the standard Authentication header doesn't hit the cache, but using a new custom header does? Is that right?

Just trying to understand the landscape for how to fix this...

[zacatac]

Yeah that's correct. 

—
Sent from my iPhone

On Thu, Feb 19, 2015 at 9:10 PM, shankari notifications@github.com
wrote:

So my understanding is that using GET with the standard Authentication header doesn't hit the cache, but using a new custom header does? Is that right?

Just trying to understand the landscape for how to fix this...

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75190615

[zacatac]

I think we will have to just cache the data in our own cache, and then update that data whenever a request completes. 

—
Sent from my iPhone

On Thu, Feb 19, 2015 at 9:10 PM, shankari notifications@github.com
wrote:

So my understanding is that using GET with the standard Authentication header doesn't hit the cache, but using a new custom header does? Is that right?

Just trying to understand the landscape for how to fix this...

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75190615

[shankari]

Let's see what the outcome of the email thread around my email to the security grads group is.
Also, by data, you actually mean data + html/scripts, right?

Because that was what confused Prof. Culler today morning - he thought we were trying to cache only the data, and was suggesting that we could send a script to the client as well. Although it looked like he didn't like javascript as the script very much...

[shankari]

@neerajbaid so I think that your cache didn't work initially because POST requests are not cached. If we switch to GET with custom headers, it seems like the cache should work on iOS as well.

Or is there some other technical reason that makes it hard to use the iOS cache?

Because your solution has the drawback that if a user launches the app just to see the results, maybe if there are no trips, then it doesn't help at all. And we want to make our results interesting enough for people to use, right?

[shankari]

To recap some more of Prof. Culler's suggestions while they are still fresh in my mind.

The main goal here is to display a result screen whose format can:

• change without updating the app, and
• can be created by non-EECS majors (i.e. transportation people)

We can do this by sending data + script from the server to the client. The script need not be javascript (although I will argue that this is the easiest since there is a built-in interpreter in the WebView). But even if we assume that we are going to display HTML + javascript, it does not imply that the server has to return HTML + javascript that you parse to get the files.

Instead, the server could return a custom data structure that listed all the components of the result e.g.

    {
         'javascript_files': ['clients/choice/front/ionic.bundle.js', 'clients/choice/front/ionic-tabs-myapp.js'],
         'html': '<html><head>...</head><body></body></html>'
    }

And the client could parse that data structure easily, store the components in a cache, and then load from disk. It didn't need to parse the HTML directly, with all the complications that it entails.

@neerajbaid, @zacatac, @jesca do you remember something different?

[zacatac]

Yeah that's what I got too. I have never used a mobile app with a UI this customizable, and I imagine there is a reason for that (namely difficulty). But the principle of sending data back and forth along with Resource files seems good. 

Maybe a first step could be to make the api only send the data, and then store the default histogram view on the phone and populate the view with data when it arrives. Then we could build up the customizability aspect from that starting point. 

—
Sent from my iPhone

On Thu, Feb 19, 2015 at 10:50 PM, shankari notifications@github.com
wrote:

To recap some more of Prof. Culler's suggestions while they are still fresh in my mind.
The main goal here is to display a result screen whose format can:

• change without updating the app, and

• can be created by non-EECS majors (i.e. transportation people)
  We can do this by sending data + script from the server to the client. The script need not be javascript (although I will argue that this is the easiest since there is a built-in interpreter in the WebView). But even if we assume that we are going to display HTML + javascript, it does not imply that the server has to return HTML + javascript that you parse to get the files.
  Instead, the server could return a custom data structure that listed all the components of the result e.g.
  {
  'javascript_files': ['clients/choice/front/ionic.bundle.js', 'clients/choice/front/ionic-tabs-myapp.js'],
  'html': '...'
  }
  And the client could parse that data structure easily, store the components in a cache, and then load from disk. It didn't need to parse the HTML directly, with all the complications that it entails.
  @neerajbaid, @zacatac, @jesca do you remember something different?

  Reply to this email directly or view it on GitHub:
  https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75197128

[shankari]

It looks to me like there are a couple of options here:

1. Switch to HTTP GET with a custom header
2. Switch to parsing the HTML page returned from the server and saving the components
3. Switch to returning a custom data structure with the data and the script

First, note that (3) is a generalization of (2) - in (2), the data structure returned is the HTML file. Which one do you want to work on?

I said earlier that this is a high impact change, so I would be willing to help with the implementation, and that offer still stands.

Maybe a first step could be to make the api only send the data, and then store the default histogram view on the phone and populate the view with data when it arrives. Then we could build up the customizability aspect from that starting point.

Well, the result screen is already customizable using the WebView, and includes options for "data" and "game", so reverting to storing the default histogram view on the phone would be a step backwards. I would rather do the HTTP GET with custom header solution than go backwards to a non-customizable view.

[zacatac]

I suppose there is a security reason that HTTP GET with Authorization headers is not cached, but if our tokens are short lived enough (I think you said ~15 minutes), that at least alleviates the security issue a bit.

As of now the phone-side code does exactly what (1) describes. It will be easy to change the server side code to look into the custom header field instead of json. 

So pending some more thought on the design of (2) or the more generalized (3), I can make a pull request for solution (1). Does that sound like a good plan?

On Fri, Feb 20, 2015 at 2:28 PM, shankari notifications@github.com
wrote:

It looks to me like there are a couple of options here:

1. Switch to HTTP GET with a custom header
2. Switch to parsing the HTML page returned from the server and saving the components
3. Switch to returning a custom data structure with the data and the script
   First, note that (3) is a generalization of (2) - the data structure returned is the HTML file. Which one do you want to work on?
   I said earlier that this is a high impact change, so I would be willing to help with the implementation, and that offer still stands.

Maybe a first step could be to make the api only send the data, and then store the default histogram view on the phone and populate the view with data when it arrives. Then we could build up the customizability aspect from that starting point.

Well, the result screen is already customizable using the WebView, and includes options for "data" and "game", so reverting to storing the default histogram view on the phone would be a step backwards. I would rather do the HTTP GET with custom header solution than go backwards to a non-customizable view.

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75331246

[shankari]

Yeah I think that is reasonable.

Can you outline a testing strategy for (1)? In particular, it would be good to not only test that the caching works, but that the invalidation works - that is, if the script does change on the server side, the new results are displayed.

Also, @neerajbaid, once we switch to GET, you won't get the "spammer" message any more. Are there technical reasons why a cache won't work on iOS?

[shankari]

In particular, it would be really nice to have the same solution on both platforms (HTTP cache) if possible, for increased maintainability.

[zacatac]

Do you have a suggested cache timeout? I will set it to 2 hours for now, but this can always be updated.

[shankari]

2 hours is fine - that's the interval at which we currently pull trips from the server.
You could even increase it to 6 hours - the results are currently updated once a day.

[zacatac]

Okay, sounds good.

On Tue, Feb 24, 2015 at 11:22 PM, shankari notifications@github.com
wrote:

2 hours is fine - that's the interval at which we currently pull trips from the server.

You could even increase it to 6 hours - the results are currently updated once a day.

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75916685

[shankari]

@zacatac so what happens once the cache is invalidated? Will there be a long delay at that time, or will the old results still be returned while the new data is fetched in the background?
In other words, assume that the cache is invalidated every 5 minutes. What will happen with this scenario?

• User launches e-mission app.
• Result screen is displayed
• User closes app
• User immediately resumes app (result screen is displayed quickly?)
• User closes app
• User resumes app after 10 mins (does the result screen display quickly or slowly?)

[zacatac]

In that case the result will be displayed slowly. The cache considers the data to be invalid so it won't return the necessary html files. 

There aren't enough knobs on the built in cache to tune it to be best effort. We could however mimic best effort by setting the maxStale value to the max (4weeks) and then manually track the last time the data was updated. Then we can begin requesting updated data and only flush the cache when we know the server has returned the updated data. That would take a bit more work, but it's something to keep in mind. 

—
Sent from my iPhone

On Tue, Feb 24, 2015 at 11:54 PM, shankari notifications@github.com
wrote:

@zacatac so what happens once the cache is invalidated? Will there be a long delay at that time, or will the old results still be returned while the new data is fetched in the background?
In other words, assume that the cache is invalidated every 5 minutes. What will happen with this scenario?

• User launches e-mission app.
• Result screen is displayed
• User closes app
• User immediately resumes app (result screen is displayed quickly?)
• User closes app

- User resumes app after 10 mins (does the result screen display quickly or slowly?)

Reply to this email directly or view it on GitHub:
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-75919429

[shankari]

How about this simpler solution?

Since the background sync on android is pretty reliable, is it possible to do the following on every background sync:

• invalidate the cache right now
• load the data (which will refresh the cache)
• set the cache to be valid for 4 hours

This way, in the normal case, the cache will be refreshed in the background without affecting user-perceived latency. If the background sync is not invoked for a long time, we will fall back to a foreground fetch and a user-perceived latency.

[shankari]

As far as I can deduce from the HttpResponseCache documentation:

• invalidate the cache right now (connection.addRequestProperty("Cache-Control", "max-age=0");)
• open a connection
• GET the data (which will refresh the cache)
• set the cache to be valid for 4 hours (set "max_stale=" + 6 hours)

[shankari]

@zacatac, actually, I don't think that you need the last of those, since you will be using that property by default for the regular calls.

So I think that adding a call to retrieve the data with forced validation

  HttpUrlConnection conn = new URL(AppSettings.getResultUrl(cachedContext).openConnection();
  try {
        conn.addRequestProperty("Cache-Control", "max-age=0");
        InputStream in = new BufferedInputStream(urlConnection.getInputStream());
        // Read the data from the input stream and discard it, ensures that it gets put into the cache
  } finally {
       conn.disconnect();
  }

in the background fetch method
https://github.com/e-mission/e-mission-phone/blob/master/android/app/src/main/java/edu/berkeley/eecs/e_mission/data_sync/ConfirmTripsAdapter.java#L80
should work, although obviously it depends on the details of the cache implementation.

Basically, this will ensure that the cache is refreshed every 2 hours. And the cache expiration is 6 hours. So whenever the user accesses the cache, it will always contain fresh data.

[shankari]

@zacatac, @neerajbaid

I just want to step back and put what we are doing here in the context of the higher-level design. We are not figuring out a way for users to load random web pages more quickly. Instead, we are figuring out a way to sync the results onto the phone in the background so that they are visible to the user instantly. Think "coda paper".

Since our results are currently displayed in HTML, we are able to use a built-in cache for the details of the sync. But we really want to invoke the sync in the background on a periodic basis so that the user is able to view them without any delay.

[neerajbaid]

While theoretically that would be great, you know that there is no reliable way to wake the app up in the background to do long running fetch operations without remote notifications (even then I'm not sure if the system gives us enough time depending on network quality). So, the next best option is loading the webpage earlier so it seems quicker to the user, which is what I'm doing. With backend changes allowing, I will now be able to cache this for x hours.

It was previously my understanding that this was an acceptable solution, doesn't sound like that's the case any more.

[shankari]

There are definitely differences between ios and android that affect that
ideal design, which is one of the reasons that I find iOS programming so
challenging.

I think that we should aim for the ideal design on android where it is
technically possible.

On iOS the background loading + cache is an acceptable compromise.

Shankari
On Feb 25, 2015 9:04 AM, "Neeraj Baid" notifications@github.com wrote:

While theoretically that would be great, you know that there is no
reliable way to wake the app up in the background to do long running fetch
operations without remote notifications (even then I'm not sure if the
system gives us enough time depending on network quality). So, the next
best option is loading the webpage earlier so it seems quicker to the user,
which is what I'm doing. With backend changes allowing, I will now be able
to cache this for x hours.

It was previously my understanding that this was an acceptable solution,
doesn't sound like that's the case any more.

—
Reply to this email directly or view it on GitHub
https://github.com/e-mission/e-mission-phone/issues/21#issuecomment-76003414
.

[zacatac]

Made pull requests for server and phone side