fRemote: iPhone to Firefox Remote – Public Prototype Release

The iPhone remote for Firefox, code named fRemote, is a prototype iPhone based remote control for the Firefox browser. The remote works by having a Firefox Add-On and Safari on the iPhone open a non-blocking connection to a Tornado server. Then the two communicate back and forth, as if they were chatting, but sending Javascript commands and responses that make Firefox do different things.

The goal of this project is to create an interface to “elements” on a webpage using a remote control. Remotes are easily created and shared among the community, and can be single or multi-session.

Video Demo

 

Public Prototype Release

You can download the code and run this yourself but please be aware that this is just a prototype and needs a lot of work.

In many cases, you will want to check the code to see what is going on in the various remotes. For example, custom data is hardcoded – the Google Docs remote points to two of my test presentations.

If you are not a developer, this project is not at a stage that would be beneficial to you. You should have knowledge of Javascript and Python if you intend to contribute or hack this

Installing

First, get the Tornado server up and running on your computer. You can find out more about that at:

Now that you have that running, make sure the chat demo that comes with it works for you!

Ok, good, now download the fRemote code at:

There are two pieces of this application:

  • server (this is the Tornado app we will be running)
  • add-on (this is the Firefox Add-On)

We won’t be building an XPI file for the Add-On, rather, we’ll just point Firefox to it to remove the hassle of rebuilding the XPI for every change. To do this, create a text file called fremote@dascgo.com with the content of the file being the location of the “add-on” directory. For instance, on my Mac, the content of the file is:

  • /Users/daniel/webapps/fremote/add-on/

Now, move this file to the Firefox extensions directory. For instance, mine is located at:

  • /Library/Application Support/Mozilla/Extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/

Details about this (and how to do on other OSes) can be found here:

Quit Firefox if it is open but do not relaunch it yet. It will fail if the server is not running (tries to connect and pull .js file from it). So, let’s crank up the server.

Move to the “server” directory in a terminal and start the server with the command:

  • ./start.py

start.py is the server and is a modified version of the chat demo that comes with the Tornado server.

Test that it is running by hitting:

and you should see the app with the default controls. Good.

Starting Up

We should be able to start using the remote now. There are a couple of options for doing this.

First:

  • Launch Firefox (to be controlled)

Then your options are:

  • Launch Safari (or another instance of Firefox) to point to the remote.
  • Use an iPhone simulator such as PhoneGap (http://phonegap.com/download) to point to the remote.
  • Actually use your iPhone by replacing localhost in the url with your computers IP Address.

Which ever you use, point the browser you want to use for your remote to:

Now, in Firefox, turn on the Add-On by going to:

  • Tools > fRemote.

A green bar should appear across the top of Firefox to let you know you are in remote mode.

Using

Let’s test it. The first Remote listed should be Presentation Viewer. Click on that and you should see a list of two presentations: “pres1” and “pres2”.

Click on one of them and Firefox should redirect to the presentation (the presentations are nothing fancy, just examples). Once Firefox loads the page, the remote should change from a list of presentations to controls for the currently selected presentation, which consists of a Next/Back/End button. Try them out and see if the presentation advances.

You can play with the other remotes to get a feel of what they do. I run through each of them in the video above if you need a reference.

Create a New Remote

Adding a new remote, in this prototype, is just a matter of a few steps:

  • Remote’s are JS files, currently stored at /path/to/server/static/dsg_*.js
    • The “dsg” prefix is just my initials, denoting remotes i’ve created.
    • Copy one of these and rename it to whatever you want to call it.
  • Open /path/to/server/templates/remote.html
    • In this file, look for:
      • <script src=”{{ static_url(“jquery.js”) }}” type=”text/javascript”></script>
    • You will see the existing remotes called immediately after that. Add your newly created remote here.
    • There is a $(document).ready call on this page that will load the remotes when the page is ready. Inside that, you’ll see an array of remotes being passed to tools.init. Add your remote id here (info.id – see next section).

Coding a Remote

Remotes are JSON files. They consist of 4 major blocks: init, screens, buttons, and info:

remote_id = {    info: {},    buttons: {},    screens: {},    init: {}  }

The Info Block:

info: {    id:          "dsg_googledocs",    logo:        "/static/logos/googledocs.png",    name:        "Presentation Viewer",    description: "Control and Share Presentations from Google Docs",    url:         "",    shareable:   true,    queueable:   true,  }

The info block is metadata on your remote. The “name”, “description” and “logo” is what is displayed on the list of remotes screen. “shareable” and “queueable” are options for later functionality.

The Buttons Block:

buttons: {    next: {      label: "next",      style: { top: "10", left: "10", width: "287", height: "166", color: "#aff4b2" },      mouseup: {        action: function(){          tools.trigger("#ToolbarNext", "MouseEvents", "mousedown");        },        responses: {          success: function(message){ },          failure: function(message){ }        }      }    }  }

The buttons block will contain all the buttons in your remote, regardless of screen. The formatting for buttons are all generally the same. The label helps to define what button type and how it’s labeled (more on this in a second). The style will place the button in that location on the screen. Buttons can have mouseup and mousedown blocks that contain an action and response handlers. This action function is what is called in either of those states. The responses block for that state will be called once the Add-On in Firefox is finished with the command it was just sent.

An example of how a buttons mouseevent and response works is as follows:

  • a remote has a mousedown action that will tell Firefox to go to a certain webpage
  • once that page has finished loading, Firefox will send back that the action has been completed and call the “success” response for that button’s mousedown
  • In the success function, you might tell the remote to draw a new screen of buttons that relate to the loaded page.

The Screens Block:

screens: {    home: ["presentations"],    presentation: ["next", "previous", "end"]  }

A remote can have multiple screens, which is just a collection of buttons. In the above example, there are two screens. Each item in the screen arrays refer to buttons. So on the home screen, there is a single button called “presentations” (which will refer to the remote.buttons.presentations button). The “presentation” screen contains 3 buttons.

The Init Block:

init: function(){    tools.setup_remote();    tools.draw_screen("home");  }

The init will start up your remote. All remotes will need to call tools.setup_remote(); but you can also do any other setup steps here. In the above example, the only thing we want to do is draw the initial screen of “home”, which, shown above, will draw the one button “presentations” to the screen.

Different Types of Buttons:

Right now, there are three types of buttons:

  • A normal button with a label.
  • An “input button” that takes input from a user (the url field in the Go To URL Remote).
  • A “list” button to create a list of clickable items (The list of presentations in the Presentation Remote).

To identify what type of button you want to create is determined by what you set as the label. In the above example for the “next” button, the label is a non-empty string (label: “next”). The engine will take a non-empty string and know that is should create a normal button with a label in it (the string).

If the label is an empty string, it will know to draw a user input button. Such as the button for entering a URL in the remote that jumps you to different urls.

If the label is an array then it will draw a list, such as the list of presentations in the Presentation remote. This actually needs to be an array or arrays. Example:

label: [    ["pres1", "http://docs.google.com/present/view?id=dfzbxv77_12dpcdh2dc"],    ["pres2", "http://docs.google.com/present/view?id=dfzbxv77_12424mt9gk"]  ]

This will create a scrollable list with two items in it. The first item will have a label of “pres1” and when you click it, the url “http://docs.google.com/present/view?id=dfzbxv77_12dpcdh2dc&#8221; will be passed to the mouseup/down function for that button.

Helpers in the Add-On

Helpers for the Remote (ways to interact with the HTML loaded in Firefox) are located in /path/to/add-on/chrome/content/application.js under fremote.commands.[name_of_command](params). All commands take one param which some split on a pipe character (“|”).

  • redirect(param): Will redirect the browser to a passed url
  • trigger(param): Will trigger a mouse or keyboard event, depending on what is passed in. The param is split on | to generate up to 4 options:
    • [0] The selector to find an element on the page to trigger an event on
    • [1] The kind of event this is: MouseEvents or KeyboardEvent
    • [2] The type of event this is: mousedown/mouseup, keydown/keyup
    • [3] (optional) if keyboard event, this the the keycode to to trigger on.
  • flash(param): This is how to send commands to a flash file (such as in the Grooveshark remote). Params are split into two items:
    • [0] id of the flash file
    • [1] command to send the flash file
    • Calling this, for example, would be tools.flash(“gsliteswf”,”togglePlayback”); which would find the flash with id “gsliteswf” and send it the command “togglePlayback” (which in Grooveshark, toggles the pause/play button)
  • collect(param): Param is a selector that will be used to select a group of dom objects. Then, the page as a whole would be grayed out and the first item of this collection will be highlighted (as seen in the Google search remote).
  • collect_next()/collect_prev(): will jump forward/back to the next item in the collection you gathered with the collect command.
  • fullscreen(param): Param will be a selector that will be used to grab an element and make it take over the whole page (used in the demo to make the flash game fullscreen)

When you call these commands in the JS for the remote, you do not need to combine all params into a single string at that time. When passed from the remote to the Add-On, the params will be joined with a pipe character separating them.

The Add-On should contain core functionality, but remote creators can inject their own helpers if needed, mentioned in the “long term goals” section below.

Long Term Goals

Now that you are running the demo, you might want to start submitting additions to the code. That would be awesome, but I would like to throw out my ideas on what I want to accomplish with this project. By no means is this roadmap set in stone, it’s just the initial direction I am pointed.

Signup:

  • There is a web site where a user goes, registers an account.
  • Once registered:
    • The user will be given a url to point their iPhone to. This page will request a log in and once in, will show the user remotes they have installed.
    • The user will be given the Firefox Add-On to install. On first run of the Add-On, it will require the user to log in.
  • Now, the Add-On and iPhone are associated to a particular user (this is important so commands know who to go to).
  • From the remote, the user can search for remotes to install, such as the Presentation remote.
  • Remotes can accept data for a person and have a limited database schema to store this data.
  • The Presentation remote needs to know the user’s presentations, so they can add them either through the iPhone remote interface, or at the website where they registered (you can manage your remotes there if needed).
  • I add some presentations and refresh my remote and find that they are there and can use them.

That describes general use of the service. In a nutshell, here’s how remotes will be created:

  • The web site will have a remote builder interface so that anyone with an account can create remotes.
  • The interface allows “Screens” of “Buttons” to be build with drag/drop functionality to easily create a remote.
  • After you add a button to a screen, you can set the actions that happen on that button.
  • The Add-On comes with built in helpers like collecting elements (seen in the Google search demo), but remote creators can also script additional functionality that the browser will import when it see the user has selected a remote that needs it.
  • Once a remote it built, it can be published (with versioning). Published remotes are available to all users to install to their account (unpublished just to the creator, for testing or strictly personal use).

Now, sharing. Examples like the Presentation and Books remote would be perfect to share. Imagine someone giving a presentation and people in the back of the room can not see the slideshow. The presenter can turn on “sharing” in a remote, then, anyone wanting to follow along, can “subscribe” to that username in thier Firefox Add-On. So as the presenter advances through the slide show in his browser, anyone that is subscribed with see their browsers taking commands directly from the presenter.

That being said, what are you ideas?

Known Issues

This is a prototype. Hastily written and without long term goals considered. It either needs massive re-factoring or a rewrite. Here are some known issues:

  • Sometimes the server and add-on get tripped up and Firefox and the server should be restarted.
  • Sometimes the selectors seem to fail.
  • Sometimes the response from the Add-On back to the Remote does not work.
  • Various things are logged to the Firefox Error console so you may want to watch that (or use fremote.tools.logger(“message here”) to log your own information).
  • The Add-On creates a green bar at the top of the browser with what looks like buttons and a search field. None of these items are active except the close button.

Other Ideas

  • Have a button in the browser that will trigger recording and it will playback things you do. Like in google reader, I can record the ‘k’ button the first time I switch to a different article. The just sit back and do it from my phone. Easy way to create a remote
  • HTML5 video… try to find a site that has free videos that can be used in html5 and then control it from the remote. find movies from the remote too.
  • Hook into the microphone and let people speak what they want to do?
  • Would like to see a “mouse” button-type that maybe draws a little “joystick” button on the screen and how you move your finger over it, it will move the mouse in the browser.

Contact

I can be reached at dascgo@gmail.com or @dascgo on Twitter.

Advertisements

2 comments

  1. wow, this is really inspiring.How do you think about including the web server directly in the browser? So that it would sort of work like Opera Unite (just to give you an idea… I’m not an Opera user). As far as I know, Firefox addons and Chrome extensions can contain executables.

  2. @Jannes: That’s a cool idea. I’m currently playing with an add-on using websockets to communicate (via a service like http://pusherapp.com/) and will try to use that system for communication between two or more browsers. I’ll post more about this if I get anything working.


Comments are closed.