Connecting to Remote Content

Using XMLHttpRequest

XMLHttpRequest is an API for transferring XML between a local script and a remote server via HTTP. It is an integral part of the modern web, and all major browsers support it. Besides XML, it can be used to retrieve data in other formats, for example JSON, HTML and plain text. In this section we'll look into the XML and JSON communication mechanisms.

let url = "http://www.example.com/";
let request = Components.classes["@mozilla.org/xmlextras/xmlhttprequest;1"]
              .createInstance(Components.interfaces.nsIXMLHttpRequest);
request.onload = function(aEvent) {
  window.alert("Response Text: " + aEvent.target.responseText);
};
request.onerror = function(aEvent) {
   window.alert("Error Status: " + aEvent.target.status);
};
request.open("GET", url, true);
request.send(null);

In this example we demonstrate how to make a XMLHttpRequest call in asynchronous mode. You can see that an instance of the XMLHttpRequest class is created and it holds all functionality for making a request. We create this instance using XPCOM instead of the usual way (new XMLHttpRequest()) because this way works both in chrome and non-chrome code.

Following initialization, onload and onerror handlers are registered to a callback function to handle the response returned from the remote server. In both cases aEvent.target is an nsIXMLHttpRequest. In the onload callback function, the responseText parameter contains the server response as text.

If the response is an XML document, the responseXML property will hold an XMLDocument object that can be manipulated using DOM methods. Sometimes the server doesn't specify an XML Content-Type header, which is necessary for the XML parsing to happen automatically. You can use overrideMimeType to force the response to be parsed as XML.

request.overrideMimeType("text/xml"); // do this before sending the request!

The open method takes two required parameters: the HTTP request method and the URL to send the request. The HTTP request method can be "GET", "POST" or "PUT". Sending a POST request requires you to set the content type of the request and to pass the post data to the send() method as below.

request.open("POST", url, true);
request.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
request.send("data=hello&version=2");

The third parameter for the open method specifies whether the request should be handled asynchronously or not. In asynchronous mode code execution continues immediately after the send call. In synchronous mode the code and user interface are blocked while waiting for a response to come back.

Now let's look at the most common types of content you can use to communicate with remote servers.

JSON content

JSON is a very lightweight and simple data representation format, similar to the object representation used in JavaScript. Unlike JavaScript, the JSON format doesn't allow any kind of code that can be run, only data.

JSON used to be risky in terms of security because the favored way of parsing it was to use the JavaScript eval function. Since eval executes any code contained in the string, workarounds had to be devised in order to close security holes. Luckily, Firefox now provides a few alternatives for extension developers. The JSON page explains in detail how to parse JSON data in different versions of Firefox and other applications.

Assume we need to parse the following data:

{"shops": [{"name": "Apple", "code": "A001"}, {"name": "Orange"}], "total": 100}

When the onload callback function is called, the response text is converted into a JS object using the parse method. You can then use this object like any other JavaScript objects in your code.

request.onload = function(aEvent) {
  let text = aEvent.target.responseText;
  let jsObject = JSON.parse(text);
  window.alert(jsObject.shops[1].name); // => "Orange"
  window.alert(jsObject.total);         // => 2;
};

The JavaScript object can also be serialized back with the stringify method.

let string = JSON.stringify(jsObject);

XML content

XML is possibly the most popular data interchange format. Let's assume that the XML returned from remote server is this:

<?xml version="1.0"?>
<data>
  <shops>
    <shop>
      <name>Apple</name>
      <code>A001</code>
    </shop>
    <shop>
      <name>Orange</name>
    </shop>
  </shops>
  <total>2</total>
</data>

When a valid XML response comes back from the remote server, the XML document object can be manipulated using different DOM methods, to display the data in the UI or store it into a local datasource.

request.onload = function(aEvent) {
  let responseXML = aEvent.target.responseXML;
  let rootElement = responseXML.documentElement;
  if (rootElement && "parseerror" != rootElement.tagName) {
    let shopElements = rootElement.getElementsByTagName("shop");
    let totalElement = rootElement.getElementsByTagName("total")[0];
    window.alert(shopElements[1].getElementsByTagName("name")[0].firstChild.nodeValue); // => Orange
    window.alert(totalElement.firstChild.nodeValue);                                     // => 2
  }
};

Using DOM functions is good for simple XML documents, but DOM manipulation code can become too complicated if the documents are more complex. There are a couple of tools you can use to process these documents more efficiently:

Using XPath

XPath stands for XML Path Language, it uses a non-XML syntax that provides a flexible way of addressing (pointing to) different parts of an XML document.

Taken from the XPath page.

You can use XPath to quickly access specific nodes in an XML or HTML document with a simple query mechanism. XPath can also be used to extract information from web pages once they load, along with the page load interception techniques discussed previously.

XPath is very useful for cases when you're receiving large and complex XML files, and you only need some of the data contained in them. Using XPath to parse a complete XML document is probably not a good idea performance-wise.

Using XSLT

XSLT (eXtensible Stylesheet Language Transformations) is another tool used to manipulate XML documents and transform them into other forms of text output, such as HTML, XUL, and so on.

We can not cover all transformations to various output formats, so we'll just look into converting an XML document to XUL.

First you need to create an XSLT stylesheet that acts as a template. This template will transform the XML you receive (in our case, the example XML document above) and convert it into XUL. The XSLT tutorial contains details for building these templates.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xul="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <xsl:template match="/data">
    <xul:vbox>
      <xsl:for-each select="shops/name">
        <xul:hbox>
          <xul:label value="Name:" />
          <xul:label>
            <xsl:value-of select="." />
          </xul:label>
        </xul:hbox>
      </xsl:for-each>
      <xul:hbox>
        <xul:label value="Total:" />
        <xul:label>
          <xsl:value-of select="total" />
        </xul:label>
      </xul:hbox>
    </xul:vbox>
  </xsl:template>
</xsl:stylesheet>

Next you need to read the XSLT stylesheet as a file stream and parse it into a document object. After that, the XSLT stylesheet can be imported into an XSLT processor as shown below. Now, the processor is ready to perform the transformation.

let domParser = Components.classes["@mozilla.org/xmlextras/domparser;1"]
                .createInstance(Components.interfaces.nsIDOMParser);
let fileStream = Components.classes["@mozilla.org/network/file-input-stream;1"]
                .createInstance(Components.interfaces.nsIFileInputStream);
let xsltProcessor = Components.classes["@mozilla.org/document-transformer;1?type=xslt"]
                .createInstance(Components.interfaces.nsIXSLTProcessor);
let xslDocument;
fileStream.init(someXSLFile, -1, 0x01, 0444); // read only
// parse from the XSLT stylesheet file stream
xslDocument = domParser.parseFromStream(
    fileStream, null, fileStream.available(), "text/xml");
// import the XSLT stylesheet to the XSLT processor
xsltProcessor.importStylesheet(xslDocument);

Finally, you can either use nsIXSLTProcessor.transformToDocument() or nsIXSLTProcessor.transformToFragment() methods to transform the XML document. The nsIXSLTProcessor.transformToDocument() method returns a DOM Document with the results of the transformation, whereas, the nsIXSLTProcessor.transformToFragment() method returns a DOM DocumentFragment node. In this example code, the first child of the XUL document is appended to a XUL element after the transformation.

request.onload = function(aEvent) {
  let responseXML = aEvent.target.responseXML;
  let xulNode;
  // transform the XML document to a XUL document
  xulDocument = xsltProcessor.transformToDocument(responseXML);
  // append the XUL node to a XUL element
  xulNode = document.adoptNode(xulDocument.firstChild);
  document.getElementById("foo").appendChild(xulNode);
};

We effectively transformed the XML file into XUL and integrated it into the UI.

Here are a couple of practical situations were you may want to use XSLT:

1. Convert a large XML document directly into XUL.
2. Filter a complex XML file and generate a simpler XML document with only the data you need, so then you can use regular DOM functions to read it.
3. Convert XML into SQL statements. You could use this to generate a script to run on your local database. You would of course need to be very careful about escaping characters and protecting yourself against SQL injection attacks.
4. Convert XML into RDF. This was more useful when RDF was the default storage format. You can still use RDF as an intermediate format, though, and then use templates to generate XUL and display the data.

HTTP debugging

When you start debugging HTTP requests, you may find it hard to know exactly what data was sent, especially with POST data. We recommend you to use extensions like Tamper Data. They help you to track HTTP/HTTPS requests and responses occurring in Firefox.

After installation, you can find a Tamper Data menu item in the menu bar:

• Tools > Tamper Data or
• View > Sidebar > Tamper Data

Once you open the Tamper Data view, all requests and responses will begin to appear in it. You can discover some interesting things about Firefox like this, such as the automatic update URLs for extensions, and the behavior of web applications such as Gmail.

If you click on the "Start Tamper" button, for every request made you will get a popup dialog for tampering with it before it is sent. You can use it to view or even modify the data in a request, and then inspect the result. This can be a lot of work because there is a lot of web activity in a normal Firefox window, so use it sparingly.

A tutorial on Tamper Data can be found here.

This tutorial was kindly donated to Mozilla by Appcoast.