Personalized Restaurant and Bar Recommendations
Introducing MyFriendSuggests.com

July 17, 2007 – 12:00 am

We are please to announce the official launching of MyFriendSuggests.com a social networking site that provides you personalized suggestions for restaurants, bars, clubs, doctors, grocery stores and just about anything else you can think of.  Unlike other sites where you may find yourself digging through 100’s of reviews from people who may be nothing like you MyFriendSuggests.com provides you a much more personalized experience.

 So how does it work?

  • First you build your friend network.  Add and invite your closest friends, we’ll add their friends, and their friend’s friends (and so on) to your network, sort of like 6 degrees of Kevin Bacon.
  • Then rate your favorite places (restauarants, bars, clubs, doctors, etc).  Rating is as simple as clicking the stars next to a places name.  Also you can add new suggestions by clicking Make Suggestion.  The more places you rate the more accurate our suggestions.
  • Using our proprietary formula we will create suggestions based on your ratings, the ratings of people like you and those in your friend network.  These suggestions will be JUST for YOU, based on YOUR preferences.
  • Also while searching throughout the site you’ll be able to easily see which other users are in your friend network and which users have similar tastes to you. 

We’ve also got some other great features such as:

  • Favorite Neighborhoods allows you to quickly see the new recommendations for your favorite neighborhoods as well as who the experts are for that area.
  • Neighborhood Messages allow you to post a question about a given area.  Maybe your new to town and want advice on a gardner or barbershop. 
  • Friend Messaging allows you to directly message anyone (or all) of the people in your friend network to ask them a question directly.
  • Our Newsletter will send you an email letting you know about new suggestions in your favorite neighborhoods AND any personalized suggestions we come up with using our recommendation system.
  • More great features coming soon!

Since we are a new site we need your help filling up our suggestions and ratings so that we can provide the most accurate recommendations possible.  Register today and invite some friends!

 If you have any thoughts or questions feel free to contact us at info at MyFriendSuggests.com

friend network MyFriendSuggests restaurants social networking suggestions Web 2.0

Blackberry GPS/Location API

July 10, 2007 – 1:36 am

I just got finished playing with the blackberry Location API.  I was pleasently suprised how it easy it was to work with.  The key to it is that your phone must be a 4.2 BB OS phone (or upgraded to 4.2 like I did with my8700g) for the Location API to work with BT Pucks.   The two source files for a simple application are pasted below the app is call Spot Finder.

 Basically what it does is allows you to mark and save a location that you are currently in and then later find your way back to that location.  It will give you the distance and direction that you need to go.  Could be useful for remember your parking spot or the location of a bar you need to get back to.  Unfortately in my basic testing its accurate but not that accurate.  It’s not perfect but it is a interesting little application especially for learning the basics of the blackberry Location API. 

 You can download spot finder here. 

Java Source:

/* Spot Finder.java */

package com.apsquared.spotfinder;
import java.util.Enumeration;
import javax.microedition.lcdui.TextField;
import net.rim.device.api.ui.*;
import net.rim.device.api.ui.component.*;
import net.rim.device.api.ui.container.*;
import net.rim.blackberry.api.invoke.*;
import javax.microedition.location.*;
public class SpotFinder extends MyApp {

public static void main(String[] args)
  {
  SpotFinder theApp = new SpotFinder();
  //To make the application enter the event thread and start processing messages, we invoke the enterEventDispatcher method
  theApp.enterEventDispatcher();
  }

MainScreen theMainScreen = null;
  RichTextField spotLocation = null;
  RichTextField currentLocation = null;
  RichTextField introMsg = null;
  RichTextField directions = null;

String storeName = "SPOT";

double spotLat;
  double spotLong;
  double curLat;
  double curLong;

public SpotFinder()
  {
  theMainScreen = createMainScreen();
  pushScreen(theMainScreen);
  }

private MainScreen createMainScreen()
  {

//Create a new screen for the applicatin
  MainScreen screen = new MainScreen();
  screen.addKeyListener(this);
  screen.addTrackwheelListener(this);
  introMsg = new RichTextField("Welcome to Spot Finder!");
  spotLocation = new RichTextField("Current Spot: ");
  currentLocation = new RichTextField("Current Location: ");
  directions = new RichTextField("");

screen.add(introMsg);
  screen.add(spotLocation);
  screen.add(currentLocation);
  screen.add(directions);
  //Add a field to the title region of the screen. We use a simple LabelField here. The ELLIPSIS option truncates
  // the label text with "..." if the text was too long for the space available.
  screen.setTitle(new LabelField("Spot Finder", LabelField.ELLIPSIS | LabelField.USE_ALL_WIDTH));
  return screen;

}

public void setDirections(final String locStr)
  {
  UiApplication.getUiApplication().invokeLater (new Runnable() {
  public void run()
  {
  directions.setText(locStr);
  }});
  }

public void setIntroMsg(final String locStr)
  {
  UiApplication.getUiApplication().invokeLater (new Runnable() {
  public void run()
  {
  introMsg.setText(locStr);
  }});
  }

public void setCurLocation(double lat, double lng, float course, float speed)
  {
  curLat = lat;
  curLong = lng;
  String latLong = new String(curLat+" "+curLong);
  setCurLocation(latLong);

//TODO: check for valid spot
  Coordinates from = new Coordinates(curLat,curLong,0);
  Coordinates to = new Coordinates(spotLat,spotLong,0);
  float dist = from.distance(to) * (float)3.2808399 ;
  //i think calculation should be azimuth - course (+360 if value is neg)
  float dir = from.azimuthTo(to) - course;
  if (dir<0)
  dir+=360;
  setDirections("Distance: "+dist+" ft.\nDirection: "+dir);
  }

public void setCurLocation(final String locStr)
  {
  UiApplication.getUiApplication().invokeLater (new Runnable() {
  public void run()
  {
  currentLocation.setText("Current Location:\n"+locStr);
  }});
  }

public void setSpot(double lat, double lng)
  {
  spotLat = lat;
  spotLong = lng;
  String latLong = new String(spotLat+" "+spotLong);
  setSpot(latLong);
  }

public void setSpot(final String locStr)
  {
  UiApplication.getUiApplication().invokeLater (new Runnable() {
  public void run()
  {
  spotLocation.setText("Current Spot:\n"+locStr);
  }});
  }

public void msgBox(final String msg)
  {
  UiApplication.getUiApplication().invokeLater (new Runnable() {
  public void run()
  {
  Dialog.alert(msg);
  }});
  }

protected void makeMenu(Menu menu, int instance)
  {
  Field focus = UiApplication.getUiApplication().getActiveScreen().getLeafFieldWithFocus();
  if(focus != null) {
  ContextMenu contextMenu = focus.getContextMenu();
  if( !contextMenu.isEmpty()) {
  menu.add(contextMenu);
  menu.addSeparator();
  }
  }

MenuItem markSpot = new MenuItem("Get Current Spot",50,8) {
  public void run()
  {
  markSpot();
  }
  };

MenuItem saveSpot = new MenuItem("Save Current Spot",50,8) {
  public void run()
  {
  saveSpot();
  }
  };

MenuItem loadSpot = new MenuItem("Load Current Spot",50,8) {
  public void run()
  {
  loadSpot();
  }
  };

MenuItem findSpot = new MenuItem("Find Current Spot",50,8) {
  public void run()
  {
  findSpot();
  }
  };

menu.add(markSpot);
  menu.add(saveSpot);
  menu.addSeparator();
  menu.add(loadSpot);
  menu.add(findSpot);
  }

private void saveSpot()
  {
  LandmarkStore lStore = LandmarkStore.getInstance(storeName);
  if (lStore == null)
  {
  try
  {
  LandmarkStore.createLandmarkStore(storeName);
  }
  catch (Exception e)
  {
  msgBox("Error creating store. "+e.toString());
  return;
  }
  }
  try
  {
  lStore = LandmarkStore.getInstance(storeName);
  Enumeration e = lStore.getLandmarks();
  Landmark lMark = null;
  if (e!=null && e.hasMoreElements())
  {
  lMark = (Landmark) e.nextElement();
  lMark.setQualifiedCoordinates(new QualifiedCoordinates(spotLat,spotLong,0,Float.NaN, Float.NaN));
  }
  else
  {
  lMark = new Landmark("SPOT1","SPOT1",new QualifiedCoordinates(spotLat,spotLong,0,Float.NaN, Float.NaN),null);
  }
  lStore.addLandmark(lMark, null);
  msgBox("Spot Saved!");
  }
  catch (Exception e)
  {
  msgBox("Error saving spot. "+e.toString());
  }
  }

private void loadSpot()
  {
  LandmarkStore lStore = LandmarkStore.getInstance(storeName);
  if (lStore == null)
  {
  msgBox("No store to open.");
  return;
  }
  try
  {
  lStore = LandmarkStore.getInstance(storeName);
  Enumeration e = lStore.getLandmarks();
  while (e.hasMoreElements())
  {
  Landmark lMark = (Landmark) e.nextElement();
  setSpot(lMark.getQualifiedCoordinates().getLatitude(),lMark.getQualifiedCoordinates().getLongitude());
  break;
  }
  }
  catch (Exception e)
  {
  msgBox("Error saving spot. "+e.toString());
  }
  }

private void findSpot()
  {
  try
  {
  LocationProvider lPvd = LocationProvider.getInstance(new Criteria());
  lPvd.setLocationListener(new MyLocationListener(this,false), 5, 5, 5);
  }
  catch (Exception e){
  setIntroMsg("Error setting up location listener:\n"+e.toString());
  }
  }

private void markSpot()
  {
  try
  {
  LocationProvider lPvd = LocationProvider.getInstance(new Criteria());
  lPvd.setLocationListener(new MyLocationListener(this,true), 1, 1, 1);
  }
  catch (Exception e)
  {
  setIntroMsg("Error getting location:\n"+e.toString());
  }
  }

//empty - nothing to do on exit
  protected void onExit() {
  }

}

/* MyLocationListener */
package com.apsquared.spotfinder;
import javax.microedition.location.*;

public class MyLocationListener implements LocationListener {

SpotFinder sf;
 boolean onceOnly;

public MyLocationListener(SpotFinder sf, boolean onceOnly)
 {
  this.sf = sf;
  this.onceOnly = onceOnly;
 }

public void locationUpdated(LocationProvider provider,
  Location location)
 {
  QualifiedCoordinates qc = location.getQualifiedCoordinates();
  if (onceOnly)
  {
  sf.setSpot(qc.getLatitude(),qc.getLongitude());
  provider.reset();
  provider.setLocationListener(null, 0, 0, 0);
  }
  else
  {
  sf.setCurLocation(qc.getLatitude(),qc.getLongitude(),location.getCourse(),location.getSpeed());

}
 }

public void providerStateChanged(LocationProvider provider,
  int newState)
 {

}
}

Blackberry GPS java Programming

Introducing APTags a Java API for Tagging

June 16, 2007 – 6:11 am

Tagging pages or items is a big part of the Web2.0 movement.  After doing some mild searching I didn’t find a prebuilt API for working with tags for Java (found some PHP stuff).  So I created one to be used in the next release of our site (www.myfriendsuggests.com).  As we’ve done in the past we decided to make the API available for anyone who wants to use it.  We don’t have a ton of working examples or documentation but the API handles all the DB communication and allows for:

  • Adding tags to items
  • Finding items with a tag (or tags)
  • Finding items tagged by a certain user
  • Generating a tag cloud for a user
  • Generating a tag cloud for an item
  • Other tag queries.

So far this has only been tested with MySQL but if there is interest out there please leave us a comment and we’ll work to verify it for other databases.

 To learn more and download the APTags API click here.

api java tagging tags Web 2.0

Creating a custom recommender using taste

May 29, 2007 – 7:19 pm

Taste is a great framework for collaborative filtering.  We are going to be launching a new recommendation algorithm on our site (MyFriendSuggests.com) in the coming weeks (Stay Tuned!) based on the Taste framework.  Taste provides a User-based and Item-based recommender.  User based recommenders find users that have similiar tastes to you and then use their ratings to predict how you might rate a given item.  Item based recommenders find items that are similar to each others and use those similar items to predict how you might rate a given item.  In our testing we found that a recommender that uses both types of recommenders would be most effective.  Basically we use the following formulat to predict user u’s rating of object x.

P(u,x) = alpha*uRec(u,x) + (1-alpha) * iRec(u,x)

Where alpha is a constant between 0 and 1 (basically weighting the two recommenders) and uRec and iRec are the Taste User and Item based recommenders.

Using the Taste evaluators you can build a simple program to find the bast value of alpha for your application.  Since we still have very sparse data we are leaving the value 0.50 until we have more data to work with.  In the next few days I’ll be posting some more on how I used taste to build our recommender.

collaborative filtering java Programming recommender taste Web 2.0

Book Review: Founders At Work

May 13, 2007 – 3:01 pm

I’m not much of a reader, but I just read a book Founders At Work, by Jessica Livingston.  It’s basically a bunch of interviews done with various people who led some of the biggest startups of the past 10-20 years.  I found it to be a real interesting read especially for someone like me, a ‘techie’ who is very interested in the world of startups (especially web startups).   The book gives some great insight into the early days of some of the web’s most successful startups.  It’s real interesting to learn about how many of these sites were started by accident or started with something else in mind and then evolved into the successes they are today.  I recommend this book if you are interested in starting your own business with some friends and colleagues.

Book Review Marketing Startups Website

Scraping Hotmail for Contacts using JScrape

May 5, 2007 – 3:20 pm

As we’ve seen in my posts for scraping AOL, GMail and Yahoo, each site has its own “tricks” that make it challenging to scrape contact information from.  The final site in this series of posts is for Hotmail.  Hotmail is one of the trickier ones.  As I did with the previous posts I’m going to outline some of the trickier parts of scraping the site.

After posting to Hotmail.com you need to parse all the hidden parameters on the form, you will need to repost those parameters along with the login and passwd for the user.  You also need to pass a parameter PwdPad which is generated by remove X chars from the end of the string “IfYouAreReadingThisYouHaveTooMuchFreeTime” where X is the length of the user’s password.   To determine the URL you need to parse out of the JavaScript the value of the JS variable, g_DO[”hotmail.com”]. 

After posting to the URL you will need to parse some more JS, find the window.location.replace JS and use the URL in that parameter to post your next URL.  In the response you will find a mailbox ID, you can find that by looking for ‘_UM=’ in the response and parsing out the value.  From there you are home free… simply post to:  http://”+host+“/cgi-bin/addresses?”+mbox  (you can get the host by grabbing the attribute using the following code:  String host = get.getRequestHeader(“Host”).getValue(); ).

Well that’s about it.  Hopefully that helps some people out.  If you want to see this in action sign up for an account at MyFriendSuggests.com and use my version of the contact importer (and while your there try our site out and let us know what you think). 

java MyFriendSuggests Scraping Social Marketing Web 2.0

Scraping GMAIL for contacts

May 3, 2007 – 6:58 pm

In our previous posts we’ve looked at how to scrape both Yahoo! and AOL webmail for a list of contacts given a username and password.  This technique can be critical in growing your user base by allowing your users to invite many friends in one quick and easy step.  Our next site that we supported is GMail. 

However, for GMail we did not use our JScrape API but rather just used the G4J API.  It was extremely easy to use and to incorporate into our framework.  I recommend downloading it and testing it, it should only take a few short lines of code, here is what I did:

GMConnector gm = new GMConnector(userID,passwd,1);
gm.connect();
GMContact [] data = gm.getContact(1,
“”);

The last site we will cover is Hotmail which was probably the most challenging of the 4 sites. 

Scraping AOL WebMail for contacts

April 30, 2007 – 9:12 pm

This is the 3rd post in a short series discussing how I built an API to grab contact list information from Yahoo!, AOL, GMail and Hotmail.  In our first post we reviewed the high level approach to scraping sites.  In our second post we went over how to scrape Yahoo! - which is by far the easiest of the 4 sites to scrape.  This post will discuss how to scrape AOL which is much more challenging as it requires some cookie manipulation and some javascript emulation.  The tips below aren’t necessarily the best way to do this but it worked for me.

For working with AOL you need to work with the HttpClient and PostMethod objects, from the Apache Commons HttpClient API, directly.  For all URLs you post to make sure to set User-Agent and set the cookie policy:

post.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
post.setRequestHeader(“User-Agent”,” Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; .NET CLR 2.0.50727)”);
 

Also for each post I set the Referrer attribute to the previous URL. After you post to the first URL you’ll need to process all the hidden variables that are returned and add them to next post.  Also there was a cookie that I seemed to need to manually add, to do so I used the following snippet of code:

Cookie[] c = client.getState().getCookies();
String cStr =
“”;
for (int i = 0 ; i < c.length; i++)
   cStr += c[i].getName()+
“=”+c[i].getValue()+“; “;
cStr+=
“s_cc=true; s_sq=aolsnssignin%2Caolsvc%3D%2526pid%253Dsso%252520%25253A%252520login%2526pidt%253D1%2526oid%253DSign%252520In%2526oidt%253D3%2526ot%253DSUBMIT%2526oi%253D97″;
post.setRequestHeader(“Cookie”,cStr);
This second post should also contain the user name and password.  This is the first part of the login. In the response you’ll find that there is javascript that will forward to a new specific URL, you need to get it dynamically.  I used the following code:

int onLoad = data.indexOf(“<body onLoad”);
int http = data.indexOf(“http:”,onLoad);
int endPos = data.indexOf(‘\”,http);
String newURL = data.substring(http,endPos);

The resulting page ALSO has some JavaScript that you will be required to emulated.  I used the following code to find the new URL:

http = data.indexOf(“gInitBasePath “);
int startPos = data.indexOf(‘\”‘,http);
endPos = data.indexOf(
‘\”‘,startPos+1);
newURL =
“http://webmail.aol.com”+data.substring(startPos+1,endPos);
newURL = newURL.replaceAll(
” “, “%20″);

Your almost there!!  In the response for that last request you need to find the uid returned in one of the cookies.  Just grab all the cookies and parse out the “uid:”.   Last but not least just post to the Address book url (you can do find this by using Fiddler) and pass in the value for the uid for user attribute.  At that point you can use JScrape to process the resulting page and parse out all the email addresses. 

Hopefully these tips help you in creating your own contact importer.

 

java Scraping Social Marketing Web 2.0

Scraping Yahoo! for contacts using JScrape

April 29, 2007 – 10:25 pm

This post builds on my previous post, in which we discuss how to scrape webmail sites for contacts.  Yahoo! is by far the easiest of the sites to scrape (of the major sites).  After you’ve sniffed the URLs used for the login you just need to replace the username and password for the login.  Yahoo! currently does not use any JavaScript tricks or special cookies during the login.  Using JScrape as-is should be sufficient.  The one trick to Yahoo is that it breaks up the address book into seperate pages.  In my solution I dynamically grab these URL’s using the following snippet of code:

public String[] getURLs()
{

 String q = “declare namespace xhtml=\”http://www.w3.org/1999/xhtml\”; \n” +
 ”for $d in //xhtml:ol[@id=’abcnav’]/xhtml:li/xhtml:a \n”+
 ” return <li> { $d/@href/string() } </li> “;

//pScrape is a com.apsquared.jscrape.PageScraper object that has already logged in to the site.
  List l =
pScrape.scrapePageForList(“http://address.yahoo.com/yab/us”, q);
  if (l == null)
   
return null;

  String[] ret = new String[l.size()];
  for (int i = 0; i < l.size() ; i++)
  {
    TinyNodeImpl ti = (TinyNodeImpl)l.get(i);
    ret[i] = new String(ti.getStringValue());
  }
  return ret;
}

Note: this may return null if the user account only has a small # of contacts.

For each url returned you need to scrape the page looking for the contacts.  I used the following XQuery for that scrape:

declare namespace xhtml=\”http://www.w3.org/1999/xhtml\”;
for $d in //xhtml:td[@class=’contactnumbers’]/xhtml:span/xhtml:a
return <li> { data($d) } </li>

That’s about it, as we’ll see in the next few days this is much simpler than many other sites (GMail, Hotmail, AOL) as they require many more tricks to login.

java Programming Scraping Social Marketing Web 2.0 XQuery Yahoo!

Scraping WebMail sites for contacts using JScrape

April 29, 2007 – 9:06 pm

Many new websites, especially those that depend on social networks, are now offering ways to import contacts from various WebMail sites.  I’m not going to go into the ethics of asking a user for their user name and password to a webmail site and scraping the site but I will touch on the technical challenges.  I started by building JScrape, a Java API that makes scraping websites easier.  I then decided to try to scrape contact lists from Yahoo!, GMail, Hotmail and AOL.  I found that each of these sites had their own challenges.  The easiest by far was Yahoo!, so that is what I’ll start with.  I’m not going to provide the exact code but will give you tips that will definetly get you going.

The basic process for all of these sites is:

1) Use a tool (such as Fiddler or Ethereal) to capture the network traffic that occurs when you login to the site.
2) Each site will use different cookies and JS to make logging in more challenging (this is the hard part). 
3) Use the same session and post to the address book page for that site.
4) Use JScrape to parse out the email addresses that you want.  You may need to page through different pages depending on the number of email addresses (and how the site displays the addresses).

Sounds simple eh?  Well step #2 can be quite challengine and frustrating.  I will add a new blog entry for each of the different sites and how to “login” to them, so check back soon. 

java Scraping Social Marketing Technorati Web 2.0 Yahoo!