These happen to be implemented for WordPress already, as a plugin.

That covers a lot of the subscription side of what Hijack would need. There’s more to do; I’m going to write a post describing this in more detail.

Grammatically “apathetic” looks a bit awkward there but I still meant it, though by using the word I was exaggerating your position in an attempt to show that for this app your “programmer’s logic” didn’t fit. While the nicest way to do this would be through standards, this is not in the interest of a user who wants to monitor their favourite forums _now_, and not just a scant few that are modeled on some sort of new protocol (which, as a commenter above mentioned, is kind of already done with Usenet.)

As for the idea that Hijack is a kludge, I dunno. A lot of apps I use feel like they’re messed beyond recognition and could be nicer in an ideal world, but they cope with the problems and I make use of them very effectively - my web browser and my mail client, for one. I really don’t see a future where web forums become abandoned for rich clients, I see them as a minority thing for the quirky power users on quirky platforms like the Mac, and thus I think it’s acceptable that they require some maintenance and not be as perfect and simple as they could.

It’s funny that I’m arguing that for a “My Dream App” competition, but I honestly think it’s far less feasible for a client based on standard forum-provided data to be a good product than one that spends a lot of time trudging through messy forum templates. There’s not enough demand for a forum aggregator for a standard to be adopted, basically, and so I don’t think the examples you cited are directly comparable as that kind of ground-up approach would result in a client for a specification nobody cares about and never will.

It’s not possible to build an “universal scraper” of any sort when there is no standard on what’ll be scraped.

Unless every forum in the world decides to go the route of tagging specific parts of the forum with IDs or with classes this won’t be feasible without hordes of people creating “plug-ins” for each different forum (bear in mind that even generic forum software like phpbb, yabb and bbpress allow for theme customisation, where the theme can be pretty much whatever you want).

I’d dare to say that probably bbpress would be the best candidate for an app of this sort, as it can provide RSS feeds of its forums and posts, and posting can be done through the methods defined in the RSS. But that means the best candidate is the newcomer in the arena. Existing major forum apps would need to change their releases to add a functionality like this and even then older forums that won’t be updated won’t benefit from it.

The route to do this is not to make an app that does scraping, it’s to propose a standard for it and lobby this standard to the major forum programmers (and to bbpress, I insist. WordPress’ weight is too much to ignore and integration with WP itself makes bbpress a probably quick-raiser to the top-five). Along with the protocol an app can be designed that takes advantage of it, but the protocolo needs to exist so other platforms and other forum bases can adopt it as well.

That’s how RSS started, by the way.

Jason Harris, Developer at My Dream App has repeatedly stated that Hijack is is feasible. He suggests to do it not through HTML, but through DOM scraping. For more info, you can check out the Official Feasibility thread in the MDA forums and other threads in the Hijack section: http://mydreamapp.com/forums/viewtopic.php?id=1263

I was about to submit an idea to it, back when it started but after seeing what the people were raving about and what’s being the most successful pitches I realised mine wasn’t as flashy and, indeed, too techy for most of the reviewers.

Several other people did try to do this, and seemingly failed because their mockups weren’t pretty (there isn’t any project right not in the semifinals that isn’t shiny and/or animated). In the end I didn’t even see if anyone had proposed anything like I had thought (some did propose other ideas I had thought of, and were summarily buried to the bottom of the lists).

*sigh*, maybe next time.

Either way, I see from your last response that you’re approaching this from a different perspective. I can appreciate that, and I agree that I’d like to see forum software change to include an appropriate standard. I’m sure it could happen, too, just not sure what would push it forward. You say “today” and “tomorrow”, but in reality it took a couple years to catch on. Forum software could take just as long, maybe it will transition faster, maybe slower. I’ve never created and moderated my own forum, so I don’t really know.

I hope my comments have been helpful. I’m mainly concerned that it seems your article unfairly attacks two ideas (the whole MDA competition, and Hijack) based on their weaknesses, without seeing if those weaknesses were acknowledged or addressed in any way. Another words, I’d hope for more balance. But then again, you’re coming from quite a different perspective, and I’m not a regular reader of your blog, so perhaps I’m missing something vital to the discussion here.

SuitCase: You raise some good points, although I am still not convinced it will work well enough. But perhaps some others have a looser definition of “well enough”.

By “apathetic” I think you mean “antithetical”? I don’t think we’re really at odds in what we want. I’m just looking at it from what I think is a broader perspective: I would rather not have a ton of work go into something that is at heart a kludge.

Remember, today’s “hyper-cool Web _.0 … weird new protocol” often becomes tomorrow’s everyday reality. RSS used to be that weird new protocol. So did CSS, Flash, JavaScript, tables, inline JPEGs. I’m sure when Mosaic came out the Gopher/Archie folks said “forget this foo-foo ivory-tower ‘hypertext’ stuff, we just want something that works with the FTP sites we already have.”

I am immensely in love with the idea of Hijack, I’ve been wishing for a rich client that accesses content from forums for many years and had fantasies about learning how to code myself just so I could try and make it.

Yes, the collection of data will be hard, however I think you are exaggerating the difficulty it would involve. That Dapper thing looks promising enough, but it already seemed mostly feasible to me because some forums use semantic markup (or at least, informative CSS stylings), offer information-rich RSS feeds, or (probably the easiest thing) a default skin\the option to choose the default skin in the user prefs. Imagine - if you had support for the subSilver theme of phpBB alone, you’d probably then support 10% of the forums on the internet. Do the same for the default looks of vBulletin, punbb, Invision and.. heck, I dunno, UBB? Then throw in the ones that can’t be changed, like ezboard and proboards.. And you’ve probably covered 50% or more of all forums, and I bet those “forum definitions” for stuff like subSilver would work with a lot of custom themes as they are almost always heavily based on the default. So even if we take it as “50% can feasibly work”, it’s a useful app already, and all that needs to be accomplished after that is a training mode and\or definition file sharing community, both concepts which are being discussed in the forum for the app.

Regardless of “rarely listen to the user” philosophy, I think your idea that people should start writing protocols and microformats or whatever for forums clearly shows a programmer-oriented perspective that is apathetic to what users like I really want from this application. I don’t care to use an app that works with the five hyper-cool Web 2.0 sites that bothered to implement a weird new protocol. I want something that works with the web as a whole, and I think that’s a lot more feasible than you’re indicating.

I’ll add though that I think “screen scraping” on the web (more accurately it would be called markup scraping) is fundamentally different than the original meaning — trying to infer information from pixels. You already have the text, you’re just trying to scrape the semantics - and markup is supposed to be about semantics.

This is why I think microformats are a good approach because they make the markup richer to let you extract the semantics you want in a reliable way, without the need for out-of-band metadata.