LonelyCache

Hi,

I just wanted to give a quick status on where development is currently at. The back end processing is now able to retrieve pocket queries from geocaching.com and add/update geocaches, find logs, and geocachers to/in the database. To me this is a big milestone since now we have real data (albeit incomplete) in the database to work with. There is still a lot of work left, for example, we have not started work on the front end.

For those of you not famliar with some of the terminology used:

• Front end: Generates the actual web pages that you'll see when visiting the LonelyCache Project web site. The front end will retrieve any data from the back end as needed.
• Back end: The behind the scenes work that gathers data, adds it to the database and keeps it up to date. The back end will also fill data requests from the front end as demanded.

--
Russell

That is excellent news, Corfman Clan! Getting the backend up and running was always going to be the long pole in the tent to get initial operating capability. I am excited to hear about his initial milestone. Do you mind a few questions? You mentioned that the data so far is retrieved with Pocket Queries -- is that how data retrieval is primarily going to work, and are you having to use a Premium Membership to run these queries? Is this how the API delivers data or is interfacing with the API still yet to come? Thanks for all your efforts!

Since I don't have a clue as to what you're talking about, I'll just say THANK YOU.

rocketsciguy wrote:That is excellent news, Corfman Clan! Getting the backend up and running was always going to be the long pole in the tent to get initial operating capability. I am excited to hear about his initial milestone. Do you mind a few questions? You mentioned that the data so far is retrieved with Pocket Queries -- is that how data retrieval is primarily going to work, and are you having to use a Premium Membership to run these queries? Is this how the API delivers data or is interfacing with the API still yet to come? Thanks for all your efforts!

Pocket queries will only be one means of data retrieval. PQs provide the simplest way to load caches into the DB, especially new ones. PQs only deliver the last five logs per cache though, so an API call that retrieves cache logs will also be utilized, among others. The API is utilized to get the list of available PQs and to retrieve them. PQs are a geocaching.com premium member feature and so is the ability to retrieve a meaningful quantity of cache data via the API. For example, a basic member is limited to retrieving full cache data for only three caches per day while a premium member can retrieve full cache data for 6,000 caches per day.

the greenskeeper wrote:Since I don't have a clue as to what you're talking about, I'll just say THANK YOU.

And I'll say you're welcome. So far this has been a fun and compelling project to work on

I think I'll try to set the trend for a bimonthly status update. At this point the data retrieval process is essentially complete and contains the following functionality:

• The ability to add/update geocaches, geocachers, and logs in the LCP database from pocket queries via the Geocaching.com Live API.
• The ability to add/update geocaches, geocachers, and logs in the LCP database directly via the Geocaching.com Live API.
• The ability to detect archived geocaches and remove them and their associated logs from the LCP DB via the Geocaching.com Live API.
• The ability to utilize any number of geocaching.com accounts concurrently for the data collection process.
• All Geocaching.com Live API transaction/cache limits are honored.
• A cache update will be scheduled 7 days after the last cache update. A cache update will synch geocache data and retrieve the latest 30 geocache logs.
• A full cache update will be scheduled 90 days after the last full cache update. A full cache update will synch both geocache data and all geocache logs.

What's left with data retrieval is to add some smarts to how error conditions are handled. For example, if the Geocaching.com Live API is down for maintenance, it is probably best to not continuously attempt to access it.

So far, running the data retrieval processes on my laptop, I have a database with 52,277 geocaches, 96,613 geocachers, and 3,700,621 found logs. From what I'm seeing, I believe my laptop could handle a 100,000 cache DB (about the size of the LonelyCache Project ). I'm looking forward to running this on real web and DB servers .

Again, I want to emphasize that though we've made a lot of progress, there is still a lot more left to do. For example, we still need to generate the statistics and design and create the web pages. Heck, we don't even have a logo yet .

--
Russell

Thanks Russ for all of your work. I can't wait for the site to be up and running and then going back to searching for high-value lonely caches. It will be very interesting to see how some cache point values have changed in the last 6 months.

Thanks for the update looks like things are moving along. I feel like a kid waiting for xmas.

Russell
I am glad to see that you are undertaking this project. I am eagerly awaiting the final product.
Adrenalin junkie

Corfman Clan wrote:So far, running the data retrieval processes on my laptop, I have a database with 52,277 geocaches, 96,613 geocachers, and 3,700,621 found logs. From what I'm seeing, I believe my laptop could handle a 100,000 cache DB (about the size of the LonelyCache Project ). I'm looking forward to running this on real web and DB servers .

Just a quick update to say that the system is now up and running on the host site using the test servers. I've been watching it run for about two hours now and it seems to be working very well.

Corfman Clan wrote:Just a quick update to say that the system is now up and running on the host site using the test servers. I've been watching it run for about two hours now and it seems to be working very well.

Great news as always from Mr. Corfman. Pretty soon you'll be working the frontend design! I'm looking forward to beta testing -- even an unfinished and unpolished site will be very welcome I'm sure by all here!

The last two weeks has seen a fair bit of progress:

• Error and exception handling has been scrutinized and improved dramatically.
• The ability to manaage accounts for data gathering has been implemented.
• The ability to manage worker threads for data gathering has been implemented.
• Statistics gathering for geocaching.com Live API use has been added.
• The system has been brought up and is running on test.lonelycache.com (sorry, the general public does not have access to it)
• Work on the statistics queries is well under way.

I'll let Redfist comment more on the statistics creation when he's able but he's been travelling extensively this month and is currently in Brazil.

--
Russell

Thank you for the update Russell. Always good to hear the progress.

Corfman Clan wrote:
I'll let Redfist comment more on the statistics creation when he's able but he's been travelling extensively this month and is currently in Brazil.

--
Russell

yea it looks like he is real busy seen he found the ape cache lol, wish my company would send me to brazil

Thanks for the update. I can't wait to see how this site looks like when it goes live for the public to see.

I must admit - finding the Brazil APE cache was great!

Now back to working on LCP!

In the last couple weeks, we've moved from working on the back end to some front end work. We've also taken a closer look at how the leader boards will be managed and computed and, I feel, have a good design planned. There has also been some changes with how Groundspeak will allow/limit access to the Live API that will effect the LCP so we'll need to address some concerns with how the data gathering is currently working. The funnest change, is that we now have cache points that are being displayed.

I'll throw out some teasers of the site. It's no longer completely vapor ware but it has no flash at all and the look will be completely changed to something nicer. Right now, the concern has been to display the data, not make it pretty.

The site is not retreiving data from the live geocaching.com site, but from a test site. The test site was a snapshot of geocaching.com taken around mid February, 2011. This means that no finds nor caches have been added for over a year. Please keep that in mind when looking at the snap shots below.

Enjoy,
Russell

This is just AWESOME!

Way to go guys! I really appreciate all the work...especially with all those pesky puzzle caches out there to distract you!

Woo-hoo I made the teaser pix ... that counts for something right?

Nice work!!! Looking forward to when it goes live.