Software - Articles

Advanced Data Capture with Google Gears

April 7, 2009

During some recent consulting, a client needed to enable offline data collection for an existing web application. End users, who, when on assignment, were often out of range of internet services, needed some way to fill in forms on their laptops. Later, when they regained connectivity, those forms needed to be automatically uploaded to a central server for further processing.

For a couple of reasons, we decided that Google Gears would be the best tool for implementing these requirements. First, the online version of the forms had already been developed in HTML, and Gears would allow to simply extend those forms with offline support and use the browser for delivery, rather than to rewrite them for some other platform. Needless to say, having only one version of a form to maintain drives down development time and maintenance costs greatly. In comparison, two another approaches: to provide offline support using Eclipse RCP or Microsoft InfoPath, would have required reimplementation of the offline forms in SWT or an InfoPath-specific XML lingo, respectively.

Secondly, once you've extended forms to include offline support, it's only a short reach to enable advanced features like auto-save and crash recovery into the online forms (if you use GMail, consider how much time and stress it's auto-save drafts have undoubtedly saved you).

Problems

Unfortunately, there was one requirement for the offline forms that proved difficult to implement in Gears. In addition to completing text forms offline, users needed the ability to take pictures (using a web-camera or some other DirectShow device) and associate them with a particular form. For example, if a social worker was making a home visit, he had to be able to attach pictures documenting abusive living conditions to his report.

Ideally, we would launch an applet to capture images, and, using a library like dsj doing so in Java wouldn't be a problem, but saving that image so that it can be accessed from the forms via Gears is tricky, and this short article will show an approach for doing that.

Understanding Gears

Before discussing the solution in depth, let's review the architecture of Gears. Remember that Gears is a browser extension that runs on an end-user's computer, and, therefore, that the context of the subsequent discussion is the user's computer, not a server.

First, every Gears-enabled web application has its own private SQLite database on the client machine, to which the application is free to create, delete, and update both tables and data. In our application, this is where offline forms are stored. The Gears Database API provides detailed information on where these databases are stored, though suffice it to say that the exact location is dependent on both the operating system and browser being used.

Secondly, a single SQLite database manages the LocalServer component for all Gears-enabled websites. The local server hosts managed and unmanaged resource stores for an application, which the application generally uses for caching remote data, specified by URL, for offline use. Applications can only interact with this database through the LocalServer API. One key, oft-misunderstood, feature of the local server is that it automatically intercepts and serves requests for any URLs that it has cached, transparently to the application.

Solution Sketch

Now, coming back to our problem, after the applet captures the image it needs to be pulled in by the offline application and saved to the private database along with the rest of the form data. Unfortunately, a straightforward solution is not workable in Gears for at least three reasons:

  1. There is no way to pass captured image data from an applet to the enclosing HTML page.
  2. Gears has very limited Blob support, so, critically, there is no way to save Blob data into the private database.
  3. If, somehow, issues one and two were both resolved, Gears does not provide a way to set the source of an image to data stored in an application variable (it may be possible to set a data src for an image, but this would require base64 encoding binary data in the browser, which seems out-of-reach with the current APIs).

The trick is to leverage, or rather, commandeer, the LocalServer for our ends. Like the alcon blue butterfly, which fools ants into raising its young as their own, our applet, after capturing an image, deposits it into the stewardship of the LocalServer under the guise of a fictitious, unique URL. By way of collusion, the offline application queries the applet for the URL, and can then not only associate it with a form saved in the database, but can actually define it as the source (i.e., src) for an image. The trusty LocalServer intercepts the request and unwittingly serves up the saved image from its cache.

In the section below, we'll break down the solution, step-by-step.

Detailed Blueprints

After an image is captured by the applet, most of the initial action takes place there, in Java code.

Initialization: Computing Output Locations

Before the applet can begin writing to Gears' databases it first needs to compute their locations. To begin we need the home directory of the Gears installation:

Compute the Gears home directory
    if ("Windows Vista".equals(osName)) {
        if ("Firefox".equals(browserName)) {
            gearsHome = computeFirefoxOutputLocation(new File(userHome 
                      + "\\AppData\\Local\\Mozilla\\Firefox\\Profiles"));
 5      } else
        if ("Explorer".equals(browserName)) {
            gearsHome = new File(userHome 
                      + "\\AppData\\LocalLow\\Google\\Google Gears for "
                      + "Internet Explorer\\");
10      } else
        if ("Chrome".equals(browserName)) {
            gearsHome = new File(userHome 
                      + "\\AppData\\Local\\Google\\Chrome\\User Data\\Default"
                      + "\\Plugin Data\\Google Gears");
15      }
    }
    if ("Windows XP".equals(osName)) {
        if ("Firefox".equals(browserName)) {
            gearsHome = computeFirefoxOutputLocation(new File(userHome 
20                    + "\\Local Settings\\Application Data\\Mozilla"
                      + "\\Firefox\\Profiles"));
        } else
        if ("Explorer".equals(browserName)) {
            gearsHome = new File(userHome 
25                    + "\\Local Settings\\Application Data\\Google\\Google "
                      + "Gears for Internet Explorer");
        } else
        if ("Chrome".equals(browserName)) {
            gearsHome = new File(userHome 
30                    + "\\Local Settings\\Application Data\\Google\\Chrome"
                      + "\\User Data\\Default\\Plugin Data\\Google Gears");
        }
    }

Above, I've included code for Windows XP and Vista for each of Firefox, IE, and Chrome. Computing the exact location for Firefox, which supports multiple browsing profiles, is delegated to the method computeFirefoxOutputLocation, reproduced below:

Compute the output location
    private File computeFirefoxOutputLocation(File file) {
        File result = null;
        for (File f: file.listFiles()) {
            if (f.getName().endsWith(".default")) {
 5              if (result == null)
                    result = f;
                else
                    return null;
            }
10      }
        File[] files = result.listFiles(new FilenameFilter() {
            public boolean accept(File dir, String name) {
                return (name.equals("Google Gears for Firefox"));
            }
15      });
        return files.length != 1 ? null: files[0];
    }

Of all aspects of the solution, this is probably the least desirable because it supports a rather limited set of platforms. Of course, this code could easily be extended for wider support, but a better solution would be for Gears to expose the home directory location in a JavaScript object on the page, where it could be queried by the applet.

In fact, we use a very similar technique to compute the image output location which, in Gears, is dependent on the hostname, protocol, and port number of the application.

Compute the image output location
    String browserName=null, browserVersion=null, hostname=null, 
           port=null, protocol=null;
    try {
        JSObject win = JSObject.getWindow(this);
 5      browserName = String.valueOf(win.eval("getBrowserName()"));
        browserVersion = String.valueOf(win.eval("getBrowserVersion()"));
        hostname = String.valueOf(win.eval("getHostname()"));
        port = String.valueOf(win.eval("getPort()"));
        protocol = String.valueOf(win.eval("getProtocol()"));
10
        securityOriginUrl = protocol + "//" + hostname 
                          + (port.length() == 0 ? "" : ":" + port);
        entriesURL = securityOriginUrl + "/publisher/"; 
        
15      if (protocol.endsWith(":"))
            protocol = protocol.substring(0, protocol.length() - 1);
        if (port.length() == 0 && "http".equals(protocol)) {
            port = "80";
        }
20  } catch (JSException e) {
        
    }

    // Now we need to compute the actual image output location.
25  //
    if (gearsHome != null) {
        localserverDB = new File(gearsHome, "localserver.db");
        
        File f = find(gearsHome, hostname);
30      if (f != null) {
            f = find(f, protocol + "_" + port);
            if (f != null) { 
                f = findOrCreate(f, "com.ahmadsoft.sacwis.frame.Desktop" 
                  + "[publisher]#localserver");
35              if (f != null) {
                    imageOutputLocation = f;
                }
            }
        }
40  }

In lines 4-9 the applet accesses methods on the Window object to get the required information.

Saving the Image

When the image needs to be saved, the applet uses the information computed during initialization.

Image saving overview
   BufferedImage bi = graph.getImage();
    try {
        File outputFile = new File(imageOutputLocation, generateImageFilename(".jpg"));
        ImageIO.write(bi, "jpg", outputFile);
5       registerPayload(outputFile);
        
        JSObject win = JSObject.getWindow(ImageCaptureAppletImpl.this);
        win.setMember("captureUrl", getCapturedImageUrl());
        win.eval("doClose()");
10  } catch (IOException e1) {
        JOptionPane.showMessageDialog(ImageCaptureAppletImpl.this 
            ,e1.getMessage()
            ,"Error writing file" 
            ,JOptionPane.ERROR_MESSAGE);
15  }

The call to generateImageFilename in line 3 generates a unique filename formatted as publisher_{yyyyMMddHHmmssSSS}_{Math.abs(random.nextLong())} and with the extension .jpg.

Line 5 registers the image with Gears so that it becomes "visible" to the local server. In line 8, the URL associated with the image during registration is saved to the main page so that our application can read it later on.

Below, I've reproduced the registerPayload method. There's a lot involved in registering a resource in the Gears LocalServer, and if you're not interested in the details, you can simply copy and paste the code below.

Register the payload
   /**
     * Responsible for registering the payload with the Gears Database after it 
     * has been created.
     * @param outputFile
5    */
    private void registerPayload(File outputFile) {
        int versionId = -1, payloadId = -1, entryId = -1;
        
        try {
10          Class.forName("org.sqlite.JDBC");
            Connection conn = DriverManager.getConnection("jdbc:sqlite:" 
                           + localserverDB.getAbsolutePath());
            java.sql.Statement s = conn.createStatement();
            final String vIdQry = 
15              "SELECT VersionID from VERSIONS WHERE ServerID IN "
              +   "( SELECT ServerID FROM SERVERS WHERE Name=\'publisher\' "
              +   "  AND SecurityOriginUrl=\'" + getSecurityOriginUrl() + "\')";
            ResultSet r = s.executeQuery(vIdQry);
            if (r.next()) { // should not fail.
20              versionId = r.getInt(1);
            } 
            r.close();

            SimpleDateFormat headerFormat = 
25              new SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss z");
            Date now = new Date();
            Date expiration = new Date(Long.MAX_VALUE);
            
            final String httpHeader = 
30          "Date: " + headerFormat.format(now) + "\r\n" + 
            "Last-Modified: " + headerFormat.format(now) + "\r\n" + 
            "Expires: " + headerFormat.format(expiration) + "\r\n" + 
            "Content-Type: img/jpeg\r\n" + 
            "Content-Length: " + outputFile.length() + "\r\n" + 
35          "Server: VX Puffer/3.1\r\n";
            
            final String payQry = "INSERT INTO PAYLOADS "
              + "(CreationDate, Headers, StatusCode, StatusLine) VALUES (?,?,?,?)";
            PreparedStatement payQryStmt = conn.prepareStatement(payQry);
40          payQryStmt.setLong(1, System.currentTimeMillis());
            payQryStmt.setString(2, httpHeader);
            payQryStmt.setInt(3, 200);
            payQryStmt.setString(4, "HTTP/1.1 200 OK");
            int rowsUpdated = payQryStmt.executeUpdate();
45          payQryStmt.close();
            
            r = s.executeQuery("SELECT last_insert_rowid();");
            if (r.next())
                payloadId = r.getInt(1);
50          r.close();

            capturedImageUrl = getEntriesURL() + outputFile.getName();
            final String entQry = "INSERT INTO ENTRIES "
              + "(VersionID, Url, PayloadID, IgnoreQuery) "
55            + "VALUES (" + versionId + ",?," + payloadId + ",0)";
            PreparedStatement entQryStmt = conn.prepareStatement(entQry);
            entQryStmt.setString(1, capturedImageUrl);
            rowsUpdated = entQryStmt.executeUpdate();
            entQryStmt.close();
60
            r = s.executeQuery("SELECT last_insert_rowid();");
            if (r.next())
                entryId = r.getInt(1);
            r.close();
65          
            final String bdQry = "INSERT INTO RESPONSEBODIES (BodyID, FilePath) "
              + "VALUES (" + payloadId + ", ?)";
            PreparedStatement bdQryStmt = conn.prepareStatement(bdQry);
            
70          String filePath = 
            imageOutputLocation.getParentFile().getParentFile().getName() + "\\" + 
            imageOutputLocation.getParentFile().getName() + "\\" + 
            imageOutputLocation.getName() + "\\" + 
            outputFile.getName();
75          
            System.out.println(payloadId + " - " + filePath);
            
            bdQryStmt.setString(1, filePath);
            rowsUpdated = bdQryStmt.executeUpdate();
80          System.out.println(rowsUpdated);
            bdQryStmt.close();
            
            conn.close();
        } catch (ClassNotFoundException e) {
85          e.printStackTrace();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }

The publisher resource store referenced in line 13 is actually initialized by the offline application through the Gears API. Sample code, written using GWT, is below:

Create the resource store
LocalServer server = Factory.getInstance().createLocalServer();
ResourceStore store = server.createStore("publisher");

This concludes a detailed look at the applet-side implementation of the image capture, which involves saving the captured image and registering it with the local server. Next we turn attention to the offline application's side of things.

Working With the Image

After the applet has closed, the offline application associates the captured image with the current form and displays it to the user. (A note to readers: the offline application is developed with GWT, which is why all the code is written in Java even though it runs in the browser.)

Below is the method used to retrieve the captured image's URL:

Retrieve the image URL
private native String getCaptureUrl() /*-{
    return $wnd.captureUrl;
}-*/;

Using this method, saveImage associates the URL with the form.

Associate the image with a form
public void saveImage() {
    String captureUrl = getCaptureUrl();
    ...
    FamilyVisitResource result = DatabaseHelper
        .INSTANCE.create(database, model.getId(), captureUrl);
    
    // notify model listeners.
    //
    for (FamilyVisitForm.ModelChangeListener l: listeners)  
        l.onImageAdded(result);
    ... 
}

Later, when the user opens the image, the URL is loaded from the database into a model bean, and displaying it is a snap:

Display the image
Grid pnlImage = new Grid(4, 1);
pnlImage.setCellPadding(0);
pnlImage.setCellSpacing(0);
pnlImage.setWidth("100%");

{
    Label l = new Label("Captured Image");
    l.addStyleName("as-fv-narrative-lbl");
    pnlImage.setWidget(0, 0, l);
    pnlImage.setWidget(1, 0, new Image(model.getUrl()));
}
formFamilyVisit.add(pnlImage);

Conclusion

After reading this article you've hopefully gained insights into novel uses for Google Gears. I've included a screencast that shows the entire image capture process in an early prototype of the said application. We later enhanced image capture even further with time and GPS stamping, all of which was integrated into our Gears application using the principles outlined above.

Resources

  1. Google Gears is a browser plugin that enables offline browser applications.
  2. Google Web Toolkit supports Java-based browser development.
  3. dsj is a DirectShow wrapper for the Java platform.
  4. A screencast that shows Gears-based offline image capture as described in the article.

Feedback

Please feel free to contact me or leave a comment below.

comments powered by Disqus