In the course of my Empirical Methods course, I recently read “Let’s go to the whiteboard: how and why software developers use drawings“, a case study conducted by Microsoft Research on MS employees. Of course, the reason for reading it was to get an example of a tangible case study, and the discussion about it in class was from the viewpoint of its characteristics as a case study. However, the study itself raises some interesting issues. In particular, let me quote a paragraph from it:

Many design decisions are made during one-to-one meetings. In many of these meetings developers produced a diagram as the conversation was occurring. Developers might benefit from recording these events. Such a recording could encompass the conversation, the sketch as it evolves, and the deictic references made to the sketch. These recordings could be especially relevant to ad-hoc meetings and design reviews.

Many of the developers interviewed suggested that they desired some sort of “intelligent whiteboard” to augment the drawing process and capture the result in electronic form. Tablet PC’s, which are capable of exactly that, are in wide deployment in Microsoft, yet few developers used them for this purpose. Digital cameras make whiteboard capture trivial, and are ubiquitous, yet we saw little use of them for this purpose. There are many commercial products for digitizing whiteboards, yet we saw no adoption of these technologies. There are many potential explanations for this lack of adoption, such as the perceived cost of their use we found in our surveys. These explanations should be investigated further.

Virtually all developers had whiteboards in their offices, every team had whiteboards in the hallways, and every meeting room had a substantial portion of its wall space dedicated to whiteboards. Were a whiteboard capture system implemented it would ultimately be rolled out to the tens of thousands of whiteboards, with an exorbitant cost of deployment and maintenance. Despite these problems, and lack of adoption of existing capture techniques, the benefits of capturing meetings (including the whiteboard contents as it evolves) could be substantial.

So, we have on the one hand the idea of recording a meeting and capturing the image of a diagram and on the other hand the idea of using a specifically engineered board to draw directly in a digitized format. Of course, serious doubts can be raised for both the economic feasibility of it as well as the actual benefit from doing either. Plus, the people using such diagrams don’t seem to be much interested in actually trying to save them. What’s more, earlier in the paper, a very important distinction is made between such diagrams that have a transient nature and those that are meant for archiving, documentation and so on. These are all very valid concerns.

But, I could not resist imagining an Optical Modeling Recognition system, similar to traditional OCR, if not for no other reason, then just the fun of it. I understand of course that may not even be some novel idea and I can also not think myself getting involved in something like this anytime soon, but the geekiness comes out naturally 🙂 So, here are some quick thoughts I jotted down:

Can we build tools that automatically recognize the conceptual structure of hand-scribbled diagrams?

Can we make such software that from the scribbled diagrams can synthesize some useful model in some more formal form?

I can of course imagine that the Computer Vision people have been dealing with such problems for a while, so I think that there must be more or less standard ways to end up with at least a digitized representation of the structure of such things. But the interesting problem that comes to my mind is weather we can somehow extract the semantics of the diagram, in other words weather we can deduce a model from the diagram. The obstacles to that, that I can think of include the ad hoc nature of the notation employed by people scribbling down such diagrams, the very high level of abstraction employed, the minimal effort invested in making the diagrams concise and so on.

Of course, I  could be just shovelling out nonsense here, but hey,  I took  it out of mind and now I can think of something else 🙂