The Internet is a cornerstone of modern society. We use it through many everyday tasks such as communicating with others in the form of e-mail, managing finances through Internet banking and e-commerce, and purely just for fun.
The Internet works by making use of a protocol known as Hypertext transfer protocol (HTTP), this protocol underpins the Internet. The development of HTTP was a joint effort by the World Wide Web Consortium and the Internet Engineering Task Force (IETF). The work done by these resulted in the Request for Comments document RFC 2616 (Wikipedia 2007). This document defines the HTTP/1.1 protocol that is in use today.
HTTP uses various request methods in order to obtain information, a brief summary of some of these are;
We can demostrate what information these methods return by using a simple java web browser.
To do this we need to use a few classes in our program, they are;
Create URL object, an example of this can be found in the HttpClient.java program.
// Create new URL object
URL url = new URL(address);
Parse URL example, an example of this can be found in the HttpClient.java program.
// Split a URL into the protocol, host, and filename.
String protocol = url.getProtocol();
String host = url.getHost();
String filename = url.getFile();
// Create socket, passing host and port
Socket socket = new Socket(host, port);
// Get data
Scanner serverOut =
new Scanner(socket.getInputStream());
// Send data
PrintWriter to_server =
new PrintWriter(new OutputStreamWriter
(socket.getOutputStream()));
// Close socket
socket.close();
Examples of this code in action can be found in the HttpClient.java program.
An example of this code can be found in the GetURLInfo.java program.
// url holds variable to be processed
URLConnection connection = url.openConnection();
Now we have a bit of background knoweledge we can take a look at the example programs.
Back to top
You can download and run any of these Java programs (making the assumption that you have some way of compiling them!)
If you have the SDK from Sun Microsystems you can compile them from the command prompt by typing:
javac prog_name.java
This will produce a class file.
You can run the program in a simular way from the command line by typing:
java prog_name
This program is a VERY simple web browser written in the Java language.
It uses the three classes that we have already looked at.
It works by prompting the user for a URL. This must be given in the full format or the java.net.URL class will throw an error.
It strips out three main elements; protocol, port, host, and filename.
The first check is on the protocol, our program will only deal with Http and not any other.
URL url = new URL(path); String host = url.getHost();
String filename = url.getFile();
String protocol = url.getProtocol();
if (!protocol.equals("http")) {
throw new IllegalArgumentException("URL must use 'http:' protocol");
}
Next it checks for the port of the request, if one is not found it uses the default of port 80.
int port = url.getPort();
if (port == -1) {
port = 80;
// if no port, use the default HTTP port
}
Now we have checked the request is as we want it the next step is to open a connection with the server. We do this by making use of the java.net.Socket class to create a new socket and pass it the parameters of the host and port number to be used.
Socket socket = new Socket(host, port);
Once a connection has been established, we send the command to get the data using the desired Http request method. The first example uses the HEAD method.
// request data with method
to_server.println("HEAD" + filename + "HTTP/1.0\n");
// get data from server
Scanner serverOut = new Scanner(socket.getInputStream());
The program then iterates over the returned data outputting its contents to the screen (STDOUT).
while (serverOut.hasNextLine()) {
System.out.println(serverOut.nextLine());
}
When all the data has been delivered the program then closes the connection.
socket.close();
The source code has been fully anotated showing where in the program this all happens.
Firstly we will run it using the HEAD method.
A text file of the full output can be found here
Let's take a look at the responses we get.
Now we will run it again using the GET method, we will look at the differences in the output.
to_server.println("GET" + filename + "HTTP/1.0\n");
A text file of the full output can be found here
As you can see it returns ALL the data about the page, including the HTML tags! A conventional browser (like FireFox) parses this output delivering a formatted page. Any href tags are displayed as hyperlinks on the page.
Java isn't the only language you can do this in.
Here is the code of a simular program written in Perl.
Notice how it performs the same tasks as the Java program, but it only calls one external library to Java's three classes.
As you will notice it also does it in less lines of coding. Argubley more efficient!
As users we don't want to see all this data, we are intrested in the information in the page. By using HTML tags a browser is able
to display a variety of media in a friendly fashion. We are going to modify this program to list the hyperlinks (if any) from a URL
that we request, like a simple search engine.
To do this we have to make a few changes to the original code.
N.B. The URL used for this example will be
http://java.sun.com/docs/books/tutorial/networking/index.html
We will start with a base program, and add some enhancements until we get our basic search engine.
We shall start by jazzing up the user interface. We are all used to graphical user interfaces, so are program should have one too. Java has a rich library of graphical functions in the Swing library. We will use the method showInputDialog of JOptionPane (of the Java API) to create a GUI prompt.
String address = javax.swing.JOptionPane.showInputDialog(
null, "Please enter the FULL URL>");
Now we need to configure the program to access the world wide web, including sites outside of the UWE system.
To do this we need to set up our program to be able to pass through the UWE firewall and proxy server.
This is done by setting some system properties.
System.getProperties().put( "proxySet", "true" );
System.getProperties().put( "proxyHost", "proxysg.uwe.ac.uk" );
We now have access to the outside web.
View source of GetURLInfo #2
A text file of the full output can be found here
Finally we want to filter the results returned by the request to output only the hyperlinks (the href tags).
We can do this by using a powerful feature called regular expressions.
To find out more about using regexs with Java go here.
A regex is basically a pattern matching tool. We will feed it the href tag and ask the program to output the characters that are contained within them.
// define a regex to search for
String regex = "<a href=\"(.*?)\"(.*?)</a>";
Pattern p = Pattern.compile(regex);
// set up a matcher object
Matcher m;
We can then make the program iterate (loop) through the data, checking for our regex, and when it finds it output the characters within it to the page.
while (in.hasNextLine())
{
m = p.matcher(in.nextLine());
while (m.find())
{
System.out.println(" " + m.group(1));
}
}
We now have our basic search engine.
View source of GetURLInfo #3
A text file of the full output can be found here
The Perl program (http.pl) performs a simular job as the last GetURLInfo example. The main difference between the two are a result of the language type. Java is an object orientated programming language, Perl is a procedural language (although you can do OOP with it). This difference represents the amount of lines required to execute the same program. Java is longer because it has to create objects, this takes up lines of code. There are other subtle differences, these include the syntax used for pattern matching.
Back to top