Redirecting to Jobseekers Direct by ZOIS

For a little while there in January 2011 it was a busy few days here at ZOIS central. Quite unexpectedly I've had to field quite a large number of telephone queries from Jobseekers. All of which I've handled with a cheerful but increasingly weary resignation. So, what happened? I thought I'd write a little first-person narrative on this, along the way there's a bit of explanation on how this stuff came about. I've tried to be as ungeeky about this as possible but it's probably best understood by somebody with a bit of a technical background.

As you are, no doubt, familiar with my web-scraping efforts. The original FTP site has now been augmented by a small Zoo of interfaces of various stripes, such as JSON, RSS, automated e-mail and, inevitably, a web one. There are actually two web interfaces, but the interested will wind up with job-details culled from the local database and displayed in a rather bald retro-looking page by some dynamic means. Since the job may have been scraped some time ago, and the vacancy could have been amended or withdrawn, I'd decided to put a link up to allow the display of the original vacancy from the Jobseekers Direct (JSD) web-site. The user could then visit JSD and find the true current state of the vacancy, and if it was still on offer.

A Call Centre, we'll need one if we've network problems again

Surprisingly there's no easy way of doing this, other than opening a JSD page and inviting the user to cut and paste the vacancy reference into their search form. Yes, it's that bad. There's a bit of annoying and, in my opinion, unnecessary state information that needs to be acquired first and input with the query. Happy it is all managed somewhat automatically by a modern browser. The state information expires pretty quickly and thus it's not possible, for example, to bookmark things nor to simply go to the displayed page with a known URL and reference combination.

However, it doesn't say programmer on my CV for nothing. I wrote a little mini-scraper to do the necessary state acquisition under the covers, input the reference, and display the result. All by a simple click on a HTML link. There's a nitty-gritty technical explanation for the technically minded written up as a Technical Note.

This link program generates a fair amount of traffic, relative to it's size, both in-bound and out-bound, not only does one have to acquire state, but there are a number of re-directs, where other pages are acquired and displayed under the covers too. This is the reason why the JSD web-site can seem slow to load sometimes, even in a good modern browser.

So this link program was a Good Thing, and is on my web-site to this day. It doesn't actually generate that much traffic, overall, for the users have already seen the vacancy details held in the Mirror database and visiting the JSD web-site is only a confirmatory act. But then it was discovered by other web-sites.

These web-sites do generalised searches on a private copy of the Mirror database which then get from FTP service. The 'hits' are displayed as a series of links referring back to the original data, not the data in their local database, or the ZOIS database, but in this instance the original vacancy on JSD. To do this they invoke that fancy little link program of mine. After a little Internet rummaging the correct vacancy is display, or depressingly, a notice that it is no longer available.

This was fine, but it did eat a lot of bandwidth out-bound from ZOIS, particularly as some of these sites are quite popular. Well more popular than mine, annoyingly. It was then that I discovered that my Internet Service Provider (ISP) had changed the terms and conditions of my connection, on the quiet, without letting me know. Suddenly as a result of a new introduced 'Fair Use' bandwidth Cap, the bandwidth was limited to 100KBit/s, little better than dialup and modem. Traffic soon swamped that and everything ground to a halt with loads of open connections. Worse, as timeouts occurred, normal users were seeing 'fail' pages with ZOIS contact details all over them. They were clicking-in from third-party web sites thinking they were going to see a Jobseekers Direct job vacancy. What was all this ZOIS stuff? So they 'phoned me up to find out where the vacancy was. It was fun, and not what I envisaged when I started my little pro-bono effort.

I've splattered plenty of apologetic disclaimers around the 'fail' pages, so that should folk land on them by accident they know what's going on.

It has also prompted me to re-write that clever JSD vacancy link program so it caches just about everything. This has eased some of the to-and-froing, but by its very nature, caches have to be treated with caution and refreshed frequently.

As I only keep the funny state information for 15 minutes, and the pages for 24 hours, we're still getting a lot of cache misses. A lot of vacancies that folk want to see are over a day old. By and large, though, it's all working, should the user request a page that somebody else has seen it is delivered promptly from the cache, otherwise it's fetched from the JSD web-site with a minimum amount of pre-cached fuss. The irony is not lost that, as this appears to come from the JSD web-site, the caching makes it a bit slicker looking than it actually is.

What I've done looks a lot like cross-site scripting, so I have to say it's not; even if you, as a sophisticated and eagle eyed user, may notice the funny address in your browsers address-bar. You are not the subject of anything dubious. I do inject a little bit of hardly noticeable disclaimer at the end of the retrieved JSD page, particularly if the reader has come from one of those third-party web-sites I wrote about earlier. They won't have seen the disclaimers on the rest of my site.

Well thanks for reading this far. This page was culled from a number of e-mails; I hope I've calmed them down by judicious editing, for steam was coming out of my ears when I first wrote them. The road to a call centre really is paved with good intentions.

Martin Sullivan, Cockermouth, January 2011.

~Z~

Navigation

This is some navigation, it should appear at the side if the page is wide, and at the bottom if it is not.

Jobcentre Plus Mirror Search

We've two search interfaces to the database. One based on the local Jobcentre Plus office and the other on an ad-hoc search.

Find your local office: 

Example: "chester".

Or, alternatively, try typing this kind of query: something in somewhere. Everything's optional. Experiment!

Find: 

Example: "painters or plumbers in wakefield".

Please send feedback about these searches.

Also of Interest

Juicy morsels from the rest of the site:

• Some site news; it's a sort of infrequently updated journal for the author too.
• Rather specific Technical Notes, the author can still bore people in the Historic Third-person.

About the Author

His name is Martin Sullivan. He's a fifty-something some-time computer programmer living in the quaintly named town of Cockermouth in north-west England.

The Company

ZOIS Limited is basically a vehicle for Martin and his OLTP work, he owns it. You can find out more about it elsewhere on the main site.

Pro-Bono Plug

Martin is currently on a health sabbatical, but is interested in the odd bit of pro-bono work by the way of therapeutic recovery. So, if you've anything that he can tackle on a non-commercial basis from his base in Cockermouth please let him know. Unsurprisingly, he's particularly interest in biology, again, and the application of computing to it.