A few pieces of software I’ve worked on have required me to scrape data from existing websites.  In general the code to do this is ugly.  The way I had been doing it was using the standard java connectivity classes to grab the data from the site and then parsing it using the standard string parsing routines.  This made the parsing ugly to write and even uglier to maintain.  In search of a better way I came across an article that discussed the us of XQuery to scrape HTML.  I took it once step farther and created an entire end to end API for grabbing the data from a website and running the query to return either a simple string or a list of objects.  My API relies on a few other APIs, namely TagSoup, Saxon and Commons-HttpClient.  However with just some simple code you can begin scraping web pages.

Our first release of the API as well as some sample code showing how to scrape a stock price from Yahoo! can be found here.  Please remember this is the first release and it definetly is in an Alpha stage.  Please let us know where you think the documentation and samples can be improved.

http://www.apsquared.net/JScrape.html

 Enjoy!

Share This

The concept of constants in CSS doesn’t seem to really exist.  So I created a JSP page instead of a css file to make life easier, simple concept but saved me lots of time when I decided to give my site a face lift.

How I did it:

I created a file mystyles.jsp and basically just set some java variables at the top then referenced them using the <%= %> syntax in the file.  For example:


<% //basic colors
String white = "#FFFFFF";
String red = "#FF0000";
String blue = "#0000FF";
String black = "#000000";
//base definitions
String primaryColor = "#3B895B"; //color of widgets
String accentColor = "#3B895B";
String font = "Verdana, Arial, Helvetica, sans-serif";
%>body {
font-family: <%=font>;
padding-top: 2px;
margin-top: 0px;
color: <%=black>;
}
I think you get the idea, then I just include the css using:<link rel="stylesheet" type="text/css" href="mystyle.jsp"> 

Share This

Figured this may help someone in the feature, sure helped me.

 If you get an exception in you JSP code, the line # corresponds to the the .java file, however at least in my installation of eclipse, eclipse doesn’t automatically find the file.  I found the java files in:

 <MYWORKSPACE>\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\work\Catalina\localhost\<mycontext>\org\apache\jsp>

If you know of an easier way to find the files let me know! 
 

Share This