Limiting download size for Android

Discussion about Schedules Direct grabber code and data formats.
Post Reply
DougEdey
Posts: 4
Joined: Fri Jul 29, 2011 7:19 pm

Limiting download size for Android

Post by DougEdey »

Hi,

I'm working on an Android Application for the Schedules Direct listings, however, my understanding is that there is currently no way to limit the data that is downloaded by the SOAP download request (I must point out that I found this extremely difficult to find out to begin with, I cannot find ANY API documentation).

This causes a significant issue with Android since there is only a 16MB heap available for Android Applications, a 24 hour download of data for my cable listings appears to get 8MB of data alone (with standard overhead for classes I get a near instantaneous out of memory).

I am currently using 2 hour downloads of data (downloading it as a string then converting to an input stream for XML parsing is the only apparant way to search the data. Then this is being placed in a SQLite database, so I don't need ALL the information every download request

A few things I've noticed though.

1) Is there a way to NOT download everything for the time period, i.e. just get the straight listings and not the cast list, genres, channel listings
2) I fear that if 1) is not possible I will have to have massive limitations on the app I am developing, like slow downloads over longer periods, which I fear will hit an API limit.
3) Do you have any formal API documentation if there are other ways to grab the data aside from the sole download method I've found.

rmeden
SD Board Member
Posts: 1527
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: Limiting download size for Android

Post by rmeden »

There's the only one "download" method and it only allows start/stop times. There's no way to filter out types without using a separate server, which would be considered "redistribution" and violate our terms.

I'll discuss with the team.. maybe we can set something up like http://schedulesdirect.org/getdata to provide data for mobile apps.

Robert

DougEdey
Posts: 4
Joined: Fri Jul 29, 2011 7:19 pm

Re: Limiting download size for Android

Post by DougEdey »

Things like channels and genres are going to be static right?

So my idea would be to do a

downloadChannels() <- no input since it's just going to be the same unless a lineup changes or channel ID
downloadListings(start, end) <- similar to the current download method
downloadActors(showID) <- may create too much stress if we do it per episode, but maybe per show would do
downloadGenres() <- Again, this should be static

This was I can do the channels and genres on an irregular basis (once a month or forced), the Listings can be done as the current download method is now, not frequently. And actors can be done if the client enables it and when viewing more detailed info (aswell as dumping it into an SQLite DB so we'd only request for incomplete data)

rmeden
SD Board Member
Posts: 1527
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: Limiting download size for Android

Post by rmeden »

I don't think we'll be able to do anything that fancy, at least without a more complicated server setup.

What I was thinking of doing is filtering out most of the tags to reduce the size. You'll still specify just a start/stop time.

Robert

DougEdey
Posts: 4
Joined: Fri Jul 29, 2011 7:19 pm

Re: Limiting download size for Android

Post by DougEdey »

I'm currently exploring the XMLPullParser API, so this could provide me with a workaround, but 24 hours of data provides me with > 500,000 main elements (without going into sub stuff), so I'll see if I have enough time to dump data into a database before the socket connection times out.

I would still like to see a slimlined data grab though, even if it's just a method to grab the listings and nothing else during a time period.

Post Reply