Building a Javascript HTTP Request Queue

Anyone working with AJAX knows the headache of dealing with HTTP Requests. They can only be done one at a time, and any one called while waiting on another leaves that request blowing in the wind. The only way to successfully handle request that can be fired off at any time, is to use a queuing system to manage the calls. I’m sure there are other recommendations out there, but I always enjoy the challenge of creating my own solutions.

Before I get into the code of my AJAX Request Queue, I want to show you the debug window. I set this up as a div with a few form fields that I can pop open at any time to show me what’s going on in the queue.

	Form Field “queueMsg” – A general message about the state of the queued requests.
	Form Field “queueInCall” – The value of the inCall variable… where or not the HTTP Request line is busy.
	Form Field “queueReadyState” – The value of the readystate variable of the current HTTP Request, if in a call.
	Form Field “queueTest” – The list of calls waiting to be executed. This example shows that there are 2 pending calls, one to get some slides and another to get comments for a particular slide.

The following is the Javascript code for the queue, all contained in a .JS file called createRequestObject, which is linked on pages that need to make HTTP Requests. I’m not going to go into too much detail about the code, just the overall idea.

Setting up initial variables:

// TURN ON DEBUG WINDOW FEEDBACK // I WILL WANT TO DISABLE THIS IN THE LIVE VERSION… var httpTesting = true; // SET MY REQUEST OBJECT var http = createRequestObject(); // VARIABLE TO TRACK IF WE ARE CURRENTLY IN A CALL var inCall = false; // QUEUE FOR CALLS var callToArray = new Array(); // QUEUE FOR FUNCTION TO EXECUTE WHEN CALL COMPLETE var returnToArray = new Array();

Create our Request Object:

function createRequestObject() { var reqObj; var browser = navigator.appName; if(browser == "Microsoft Internet Explorer"){ reqObj = new ActiveXObject("Microsoft.XMLHTTP"); isIE = true; }else{ reqObj = new XMLHttpRequest(); } return reqObj; }

Insert calls to the Queue:

function sendCall(whereTo, returnTo){ // GET THE NEXT ARRAY ITEM AND REMOVE FROM THE ARRAY callToArray.push(whereTo); returnToArray.push(returnTo); }

Watcher function that is called on intervals to set out queued calls based on inCall variable:

function callQueue(){ // IF WE HAVE A WAY OF MONITORING THE QUEUE, UPDATE IT if(httpTesting){ calls = ""; for(i=0;i<callToArray.length;i++){ calls += callToArray[i] + "n"; } document.getElementsByName("queueTest")[0].value = calls; document.getElementsByName("queueInCall")[0].value = inCall; document.getElementsByName("queueReadyState")[0].value = http.readyState; } // CHECK THE QUEUE AND SEND THE NEXT CALL IN LINE if(!inCall && callToArray.length > 0){ // DO WE HAVE ANYTHING IN THE QUEUE? if(callToArray.length > 0){ // WE DO, SO GET THE FIRST ITEM IN THE CALL ARRAY AND REMOVE IT whereTo = callToArray.shift(); returnTo = returnToArray.shift(); // SEND THAT CALL doCall(whereTo, returnTo); }else{ // UPDATE DEBUG QUEUE if(httpTesting){ document.getElementsByName("queueMsg")[0].value = "no items in queue."; } } }else{ // UPDATE DEBUG QUEUE if(httpTesting){ if(inCall){ document.getElementsByName("queueMsg")[0].value = "currently in a call."; }else{ document.getElementsByName("queueMsg")[0].value = "no items in queue."; } } } }

If we are not currently in a call and have a pending request, the call URL is sent here:

function doCall(whereTo, returnTo){ inCall = true; http.open('get', whereTo); // DO WE HAVE A FUNCTION TO CALL ONCE CALL IS COMPLETED? if(returnTo.length > 0){ eval("http.onreadystatechange = "+returnTo); } // SEND CALL http.send(null); }

This is a generic return function intended to just clear the inCall variable when we have a call type that we don’t need/expect a response from:

function hr_inCall(){ if(http.readyState == 4){ inCall = false; } }

And finally, to start the Queue engine, we have:

var queueWatcher = setInterval(callQueue, 100);

All that together successfully runs my Request Queue and gives me the ability to monitor it. A pause button would be a nice addition, but I haven’t had time to implement it. However, it would simply need to clear the queueWatcher interval on pause and restart it on play.

Finally, for any page that needs to make a call, for example, a log in, I would just have the submit button call a function that gathers the data and sends it to the queue, as such:

function checkLogin(){ username = document.getElementsByName("loginEmail")[0].value; password = document.getElementsByName("loginPassword")[0].value; document.getElementById("loginSubmit").innerHTML = "Checking Login..."; sendCall("req/getLogin.php?username=" + username + "&password=" + password, "hr_checkLogin"); }

You will notice when I sendCall, I have “hr_checkLogin” as the second parameter, which, once the call is complete, will call:

function hr_checkLogin(){ if(http.readyState == 4){ inCall = false; response = http.responseText if(response != "0"){ document.getElementById("loginArea").innerHTML = "Login successful. Welcome back. <A href='projects.php'>Projects</a>"; }else{ document.getElementById("loginSubmit").innerHTML = "<font color='red'>Login Failed</font>"; } } }

The important thing is to always set inCall to false once your call is complete, or the next request will never be processed.

Like I said, there may be better solutions out there, but I’m a big fan of building my own, which always helps me to become a better programmer.

Any thoughts?

dID: Fill out any form, any where, with one click

There are countless rants about every web site out there requiring you to create an account. Who isn’t turned off with creating logins, logging in, requesting lost passwords and the like? Yeah, I know… most places I could click the “remember me” button, but I’m a developer, I’m constantly clearing my cache and cookies. Hey, I’m easily forgettable… and I think only once in my professional career have I ever written down a password. It’s a big mess.

Ok, so what about other things that I’m constantly doing out over and over and over again like my address? What if I had a system that had all the data I wanted to put in there, then when I went to any form, on any site, I just clicked a button and my info is there, submitted, and I’m moving on. Yeah, that’s exactly what I wanted… so I thought about it for a few days and then developed for about 2 weeks, calling the project danielID, or dID for short.

dID consisted of three major parts:

• There was the dID web site, where you would ::gulp:: log in and create your profile. It was broken down into various areas such as personal, business, etc. Also, here you could find logs where you’ve sent your data, who is requesting your data, etc.

• The dID Firefox extension. My first attempt at writing an extension, and while I didn’t get far (just enough to get it to work) it was very interesting stuff. Basically, the extension runs in your tool bar and anytime you are on a page with a form, clicking it would analyze the form and fit as much as your info into it that it could.

• The third piece was just a pretty interface, launched by a bookmarklet, which would give you greater control over the forms on a page. It was geared towards volunteers wanting to help the project by aligning data at dID to data at a site. For instance, dID may have you first name mapped to fname and firstname, the site you are on has a form with a first name field, but its named first_name. With this control panel, a volunteer could map that field name to our field for first names.

So, that last bullet also tells you a bit about how the system worked. At the dID site, you fill in as much information about yourself as you like. Each field, like above, fname, is mapped to various possibilities of what that field is called (at other sites). When you are at some other site, on a form page, the form DOM is collected and sent to dID, which is parsed, associated with dID fields, and the real data is sent back, into the proper form fields. Here’s an example:

Above is a mock up web site, with two form fields in it. Also in the shot is the Firefox extension. Currently, it displays two options; Button to fill in the form, or a drop down to select which form you want to focus on. If you don’t focus, dID will try to fill in both.

	We drop down and mouse over a form on the page with the name of “search”.
	The search form highlights to show us which one it is.

	Ok, that wasn’t the form we wanted to fill out, so we select the contact form.
	The contact form highlights

	Now we click on the button to fill in the form. The extension sends the form object back to us, we analyze it, and return associated data.
	Data that we have mapped populates itself into the form.
	Fields that are not mapped are highlighted to the user so that they can fill them out. If the form fills out 100%, it is automatically submitted.

When I was building this, and I saw it work for the first time, I was completely blown away. Don’t you love those moments?

The bookmarklet opened a small version of the dID site. With the popup, you could monitor your information, fill out forms, or contribute to the dID mapping system. For instance, here are a few screenshots:

		Here we see the control panel that is looking at a page that has a form we recognize (noted by the green checkmark). We get a bit of info about the site and form. We also have tabs here so we can easily update our data, prefs, etc.
		Now we have surfed to a page were dID is not familiar with the form. Maybe there is no form; maybe the form is in flash… In any case, if there is a form here that we are not seeing, the user has the option to report it to us.
		Now we are back on a page with a form we “kind of” know about, but someone has reported it as broken.

One other big plan for dID was having a user PIN number, called an idSN, that you can give sites that they would use to get your information from us. They could do away with their forms and request data about you (that you specify is ok for such uses) whenever they needed it. With that, the following ideas were on the drawing board.

• If you want to unregistered from a service: We can’t guarantee a service will remove you from there system, we can block further requests of your data from them.
• If you wanted a site to have access to a block of your information for a limited period of time, you could set up an idSN that would expire – you choose data you wish to give and expire time, idSN is generated…
• Allow site owners to pull info about users that are members of their site. Nothing that is identifiable to a single person, but trends and demographics to make their support of dID worthwhile.
• Users can report abuses from services accessing their data…

There was a ton of other ideas, all very interesting stuff, but eventually I dropped the project. Once I saw that I COULD do (that’s the fun part to me), I decided the security issues involved with it would be over my head. And yeah, cookies, and browser auto-fills do ‘some’ of this, but that wasn’t the point. The point was it sounded like an interest idea and a cool challenge, and I like those!

A sneak peak at my latest personal project

I’m really excited about my latest personal project, however, it’s going over the budget a bit – I try to get a working proof of concept done in about a week. However, this is one that I’ve really wanted to do for a long time, so I’m ok with dedicating a little more effort to it. I don’t want to give too much away, and being mysterious from time to time can be fun, so all I leave you with is a simple screenshot.

Looking Back: The NewsSort Feed Reader

I wanted to go back a little and look at a feed reader I created. It was developed early in 2003 if I remember correctly. At the time, I was an avid read of Techdirt. I still am, but back then, that was about the only news site I read on a daily basis. I was really interested in their Corporate Intelligence product and wanted to create something similar.

NewsSort created a social experience around news. Although it is not as open (socially) as digg.com, NewsSort allowed groups to be created with the group creator managing who had access. The thought was to have a company group where co-workers shared stories important to their industry, and communicating about them, which could include sensitive information that would be private to that group. Sounds web 1.0 doesn’t it?

Homepage

	Site stats
	Login
	Registration

Group Page

	Once you log in, you get a list of groups you are a member of

Create a new group

	You can create your own groups

Group Members

	As the admin of a group, you can manage who has access to it
	Typing in email addresses invites new people to your group

Group News

	Here is the actual news articles coming in based on selected RSS sources
	Each article can be saved to the groups library for later access and commenting
	Each group can have keywords that are set to filter the stories coming in

Selected Feeds for a Group

	These are the feeds you can choose to get news from. Of course, if I was to set this up today, that it would be open to any feed you wish to use
	There was a way to suggest a feed, so that was a start in the right direction

Group Keywords

	As mentioned above, each group can have keywords to that NewsSort could filter out items the group had no interest in. I mentioned this was inspired by Techdirt’s Corporate Intelligence, which uses real people to compile important stories for a company, so this was an attempt to emulate that.

Group Library

	Here is a list of stories added to the group’s library
	Various info of the story is shown here, such as; Date saved, number of comments, importance (as voted by the group) and category assigned

Story with Comments

	The story saved in the library
	The voting system used by the group members
	The comments left by the group members

It was fun to look back at this. There are a lot of projects I started in the early 00’s that I wish I had stuck with (like my online calendar!) and this is definitely one of them. I don’t even recall why I stopped working on this. How do you stay interested in your personal projects?

BTW, I use NewsHutch for collecting my daily news.

My slideshow application for a Web 2.0 Office

A recent post on Techcrunch about Preezo, which looks GREAT by the way, and all the talk lately of Zoho/Google Office had me reminiscing of a slideshow tool I was working on very recently.

The slideshow project was born from a web based courseware development tool I’m developing for work. It started as a concept of what a version 2 would be like; in the current tool, you have to select predefined page layouts, but I want users to place content elements where ever they wished.

Once I had a great div positioning, scaling, content defining system, I thought it would be great as a way to set up slide shows… and from there, in any spare time I had, I developed my prototype slideshow builder.

Slideshows Directory
This is a directory of all your current slideshows. The link will take you to the editor for that slideshow. In the future, there would be controls to allow live sharing of the show (collaborative viewing) and ways to embed the slideshow in a blog or social network.

	Icon to open control panel for adding a new slideshow.

Slideshows Directory w/Control Panel to add a new show
This is the directory page again, showing the “add show” control panel.

	Close/Cancel adding a new slideshow.
	Save Slideshow.

The Slideshow work area
The work area for a slide show. It consists of 3 parts: A list of slides in this show, general comments for this show, and the slide work area.

	Icon to add slides.
	Icon to add comments.
	Icon to add container elements to a slide and the current slide selected name.
	Icons to play slide show, debug (for testing) and to go back to Presentation Directory.

Adding a slide to a show
Clicking the green (+) in the slides area will open the New Slide Control Panel.

	New Slide dialog.
	Slide title.
	Slide type: Normal Slide (Text/Images/Etc.) or RSS Slide (talked about below).
	Save Slide Icon.

Adding a comment to a slide show
Clicking the green (+) in the comment area will open a New Comment Control Panel.

	Slides that have been added.
	Controls for the slides include: Delete, Sorting and Preferences for each Slide.
	A comment window for collaborating on slideshow creation.

A successfully added comment
Displays a comment added to the current slide, or in reference to the entire slideshow if no slide is currently selected. The Slides and Comments area constantly looks for new data and refreshes as needed to make sure two or more people working on a show always see the latest work from their collaborators.

	Comment added at Presentation root level. Comments appear only where they are taken. A comment taken on slide “Introduction” will not appear on slide “Summary”.

The work area for a selected slide
Now that a slide is selected, the slide work area becomes active so we can begin to build our slide.

	Selected slides title appears over the work area.
	New Icons appear for help creating the slideshow.

Preferences Control Panel for a slide
Each slide has a preferences control panel that you can use to define that slide (or set as a global for all slides)

	Slide Preferences have been click for the current slide.
	Preference include: Time, Background and Duplicate Slide with room to add more functionality as needed.

Preferences Control Panel – Time Controls
Accessing the time controls for a slide in the slide control panel.

	Time Preference selected.
	Option for when to change slide: On click, or based on number of seconds.
	Option to specify number of seconds to wait until the slide changes, if option selected.
	Option to apply these settings to all slides.

Preferences Control Panel – Background Controls
Accessing the background (image/color) controls for a slide in the slide control panel.

	Background Preference selected.
	Option to use an HTML color code for background.
	Option to specify an image hosted on the web as the background.
	Option to apply these settings to all slides.

Adding a Container to a slide
The data on a slide is made up of various containers. For example, you may have a container for each bullet point, or for an image.

	Adding a container to the slide.
	Container Name.
	Container Type: Text, Image, Flash, Video.
	Background has been set to the image we linked to above.

A successfully added container
The container shows up onscreen after being added with controls to modify it.

	Our container now appears.
	Option to delete the container.
	Options to edit content of container / drag it elsewhere on screen.
	Option to scale container to desired size.

The container alignment grid
To help align items properly, there is a grid that can be displayed behind all containers.

	We can turn on a grid to help us align our containers.

Editing content (text) of a container
A rich text editor is available for creating the perfect copy for the presentation.

	Our container is aligned where we want it.
	The Edit Container Content button.
	The Edit Container Content Control Panel. This container is a text element, so it brings up the text editor.

A container with it’s content

	Our text content is saved in the container.

Adding an image container

	Adding another container with type set to Image.
	The Edit Container Content button.
	The Edit Container Content Control Panel. This container is an image element, so it brings up the image link area.

Successfully showing an image container

	The image is now loaded into the container.

Scaling container media content

	As the container scales, so does the image.

Container Control Panel
Kind of like Layers in Photoshop, each container on this slide is listed out here.

	The container management (for this slide) button.
	The container management window, displaying all containers and their settings.
	Type.
	Location (from top).
	Location (from left).
	Width.
	Height.
	Edit Icon.

Editing containers from the Control Panel
Each container can be hidden, or its settings edited.

	The image container was hidden.
	The Text container is being edited so all settings become editable.
	Save Icon.

Setting up an RSS based slide
A pretty unique feature, I think, is the ability to use RSS to populate slide content. With this, the presentation could be a live document that is always changing based on current, for instance, sales figures.

	This page is an RSS generated page, as specified when created.
	RSS Icon, grayed out because no feed has been selected for this page.
	RSS Feed Control Panel.
	URL of RSS feed.
	Number of ITEMs from the RSS feed to display – effectively, how many times this slide will loop with different data from the feed.
	Save button.

RSS nodes creating containers
Slide showing all nodes of and item element defined as containers. This examples used a feed from Digg.

	Now that the Feed has been selected and saved, it is checked out and a container is created for each items node. Now you can arrange the feeds data as you wish.

Deleting a container
Example shows how a user would delete a container.

	Nodes (or any container) that you don’t wish to use can be deleted. Confirmation is required.
	The system highlighting the container you are about to delete so no mistakes are made.

RSS Node Containers positioned as needed
With the nodes aligned as wished, the next step would be to set font size, colors, etc for the elements. Unfortunately, that step is still on the to-do list. Also, until the slideshow is ran, placeholder data populates the container, it is also noted what node is related to which container.

	Slide now as all feed elements desired arranged to the users liking.
	The nodes are specified so the user knows what data from the feed is in what container.

Adding a comment with a pointer
To help with collaborative commenting, I thought it would be neat to allow a comment to point to an area onscreen that it was talking about.

	A comment is added to this slide.
	The comment pointer, a tool to help specify what the comment is about, is selected.
	A drag-able, rotate-able arrow is displayed on screen to the person leaving the comment and point to the area of the slide he is talking about.

Seeing the Comment Pointer
When I am looking at a comment left by someone else, and I mouse over it, the pointer becomes visible so I have a better idea of what is being said in the comment.

	When the mouse is over the comment, it highlights and;
	The comment pointer appears.

The slideshow playing (first slide)
The first slide as seen when the slideshow is playing.

The slideshow playing (RSS slide)
The RSS (first item) slide as seen while slideshow is playing. Formatting needs to be applied to the data.

There is still a ton of work to be done on this. However, I am particularly happy about its progress so far. I really like the RSS feed integrating; I don’t believe anyone else has done that with a slideshow.

Anyone want to fund me finishing this?!?