Eyewire.org expert review

eyewireIt is not so often that some piece of software comes out that gets me as excited as what the folks at eyewire.org are doing. What they are attempting to do is to take one of the most difficult aspects of generating synapse-level “Connectomes” and speed it up by crowd sourcing the labor. By breaking the task into pieces that can be completed in short periods of time and adding some game-like elements, they hope they can dramatically speed up their reconstructions of the human retina connectome. But how useful is it really?

My initial impressions were very positive. It comes across as being a nicely polished product, and much thought has gone into marketing and promotion. It is very easy to get excited about! The idea that anyone with the inkling can sit down for 15 minutes and make some kind of genuine contribution to neuroscience is pretty awesome. As I currently make a living reconstructing connectomes in nematodes, this kind of innovative approach is something that I hope achieves quite a bit of success.

When you first sign up, it is not exactly clear that there will be some tutorials to get you started, and searching the wiki for information was not very efficient. Indeed, there are even some broken links (e.g. THIS ONE) that make the wiki a bit frustrating. But, when you just start playing the game you first go through 6 very nicely designed tutorials that guide you through the tasks you are expected to do. Each of the neurons they are reconstructing have been chopped up into a series of cubes, and the neuron in that cube has been traced using an automated algorithm (likely some derivative of a watershed algorithm or something similar. I couldn’t find any information on exactly HOW they were generating the initial segmentation, as the link in the wiki was empty. The algorithm seems to work pretty well on their data though. By design, it under-represents which pixels in the cube should be a part of the neuron. Your job is to find all of the pieces that the algorithm missed. You do this by clicking on regions of 2D images that you think should be a part of the neuron, and it propagates that region out a certain distance (meaning you don’t actually have to draw the pixels in…). The interface works pretty well and is actually kind of fun to do, which is the idea I guess. Once you have examined the portion of the neuron contained in your cube, you get a “score” based on how what you did compares to what other people have done. You earn more points if your version of the cube nicely matched what other people previously did. It also shows a leaderboard so you can compare how much work you’ve done in a day, week or month to all of the other users.

As I was going through the tutorials, there were several things that frustrated me. As a neuroanatomist, the quality of the data they were reconstructing was, in some ways, disappointing. They use a very special fixation technique which stains all of the regions in between cells, but washes out the contents of cells almost entirely. The result is a kind of binary image that makes computer scientists happy, but would frustrate a biologist. The computer scientist is happy because his/her algorithm will have an easier time following cell membranes. But the biologist will be sad because most of the beauty of the cell has disappeared. You can’t see anything like mitochondria, microtubules, not even synaptic vesicles. In order to nicely identify a synapse, you need to be able to see things like neurotransmitter vesicles or synaptic densities. This is sadly all removed by this staining technique, so when reconstructing this connectome you will never actually SEE a synapse, they must be inferred by the branching shape of the neuron. Thus, at best, you end up with a network of potential connections rather than a network of synaptic connections. The value of this map of potential connections should not be underestimated, but should be considered in context when compared to more completely reconstructed connectomes. This may also frustrate some users who may spend hours of their time tracing neurons with the intention of finding synaptic connections, but never get the opportunity to actually see ANY of them.

There were some other frustrations as well. As a player, your task is to fill in areas that the algorithm failed to find. However, there is another type of mistake that the algorithm can and does make… it might decide to include some pixels that should NOT be included. And you have, as far as I can see, no way of removing these pixels from the reconstruction. At minimum, it seems, there should be some way to mark problem areas so that they can later be manually looked at. Better would be to allow us to “erase” regions that the computer had already filled in.

Another type of error that is just a bit frustrating is that sometimes you click on a small region that very clearly belongs to your neuron, but it propagates those pixels to include larger regions that clearly do NOT belong. Again, it would be nice to be able to paint these in… this isn’t such a big deal, but might be frustrating to the type of obsessive compulsive perfectionist that enjoys doing this sort of thing (I include myself in that category!)

Even in the tutorials, there are some errors in how they assign points.. IN tutorial 6, it said I got 99% right, and indicated in red some pixels that presumably I had gotten wrong. I am pretty darn sure that I was right on that one.

To be fair, I probably am looking at this from a far more critical eye than most (and a critical eye that wants them very badly to succeed!). None of these frustrations don’t really distract from the importance of the initiative. Neither should they truly turn people off from participating. Ultimately, the goal is to identify all of the branches in the neuron, and few erroneous pixels here or there simply don’t matter. I couldn’t find any circumstances where the algorithm had mistakenly labeled an entire branch as part of the neuron when it could not be there (this in itself is a big success!). One should not trivialize the difficulty in obtaining large electron microscopy datasets of the sort they are using for EyeWire, nor should the challenges involved with automated segmentation of images or of deploying such a crowd-sourced science effort be underestimated. If it were easy you would see it done much more often.

One of the things I liked the most about it was how they really did break the task down to where non-experts could contribute to a problem which previously required tremendous amounts of time and expertise. By breaking the task into small chunks, it really does become somewhat game-like and does give the user (rightly so) the feeling that they are doing something important. If I were to add anything to the “game-play” aspects, it would be to implement typical things like the ability to earn badges for accomplishments (maybe for the first thousand or 5 thousand points…) or maybe for you to be able to see a tailored leader board based on just the people you know, maybe by looking at those  in your Facebook network. Some users might also find widgets that could be displayed on a website or social network page defining how much they have contributed to the project. This could serve the purpose of some free advertising as well. A further suggestion would be to keep statistics.. maybe have graphs of your activity.

Overall, it seems that EyeWire is well on its way to being a great success… apparently over 4000 people have contributed to it in the last week alone. It is truly a well implemented and pioneering approach to a very difficult problem that has me and a lot of other people pretty excited, and it this point it is very clear that the project will make a significant and much-needed contribution to our understanding of synaptic wiring. I highly recommend giving it a try.. I like it so much that I’ll probably spend some more time with it even though I have my OWN connectomes to be reconstructing!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s