All posts by chrislintott

Studying the Impact of the Zooniverse

Below is a guest post from a researcher who has been studying the Zooniverse and who just published a paper called ‘Crowdsourced Science: Sociotechnical epistemology in the e-research paradigm’. That being a bit of a mouthful, I asked him to introduce himself and explain – Chris.

My name is David Watson and I’m a data scientist at Queen Mary University of London’s Centre for Translational Bioinformatics. As an MSc student at the Oxford Internet Institute back in 2015, I wrote my thesis on crowdsourcing in the natural sciences. I got in touch with several members of the Zooniverse team, who were kind enough to answer all my questions (I had quite a lot!) and even provide me with an invaluable dataset of aggregated transaction logs from 2014. Combining this information with publication data from a variety of sources, I examined the impact of crowdsourcing on knowledge production across the sciences.

Last week, the philosophy journal Synthese published a (significantly) revised version of my thesis, co-authored by my advisor Prof. Luciano Floridi. We found that Zooniverse projects not only processed far more observations than comparable studies conducted via more traditional methods—about an order of magnitude more data per study on average—but that the resultant papers vastly outperformed others by researchers using conventional means. Employing the formal tools of Bayesian confirmation theory along with statistical evidence from and about Zooniverse, we concluded that crowdsourced science is more reliable, scalable, and connective than alternative methods when certain common criteria are met.

In a sense, this shouldn’t really be news. We’ve known for over 200 years that groups are usually better than individuals at making accurate judgments (thanks, Marie Jean Antoine Nicolas de Caritat, aka Marquis de Condorcet!) The wisdom of crowds has been responsible for major breakthroughs in software development, event forecasting, and knowledge aggregation. Modern science has become increasingly dominated by large scale projects that pool the labour and expertise of vast numbers of researchers.

We were surprised by several things in our research, however. First, the significance of the disparity between the performance of publications by Zooniverse and those by other labs was greater than expected. This plot represents the distribution of citation percentiles by year and data source for articles by both groups. Statistical tests confirm what your eyes already suspect—it ain’t even close.

Influence of Zooniverse Articles

We were also impressed by the networks that appear in Zooniverse projects, which allow users to confer with one another and direct expert attention toward particularly anomalous observations. In several instances this design has resulted in patterns of discovery, in which users flag rare data that go on to become the topic of new projects. This structural innovation indicates a difference not just of degree but of kind between so-called “big science” and crowdsourced e-research.

If you’re curious to learn more about our study of Zooniverse and the site’s implications for sociotechnical epistemology, check out our complete article.

Advertisements

The importance of acknowledgement

Trying to understand the vast proliferation of ‘citizen science’ projects is a Herculean task right now, with projects cropping up all over the place dealing with both online data analysis like that which concerns us here at the Zooniverse and with data collection and observation of the natural world via projects like iNaturalist. As the number of projects increases, so do questions about the effectiveness of these projects, and so does our desire to keep track of the impact all of the effort put into them is having.

These aren’t easy questions to answer, and an attempt to track the use of citizen science in the literature is made by Ria Follett and Vladimir Strezov, two researchers in the Department of Environmental Sciences at Macquarie University, in a recent paper published in the journal PLOS One. They look at papers including the words ‘citizen science’, and includes the surprising result that ‘online’ projects accounted for only 12% of their sample. They explain :

The missing articles dis- cussed discoveries generated using “galaxy zoo” data, rather than acknowledging the contribtions of the citizens who created this data.

This, to me, is pushing a definition to extremes. Every one of the ‘missing’ papers cited has a link to a list of volunteers who contributed; several have volunteers listed on the author list! To claim that we’re not ‘acknowledging the contribtions’ of volunteers because we don’t use the shibboleth ‘citizen science’ is ridiculous. Other Zooniverse projects, such as Planet Hunters, don’t even appear in the study for much the same reason, and it’s sad that a referee didn’t dig deeper into the limited methodology used in the article.

Part of the problem here is the age-old argument about the term ‘citizen science’. It’s not a description most of our volunteers would use of themselves, but rather a term imposed from the academy to describe (loosely!) the growing phenomenon of public participation in public research. In most of our Galaxy Zoo papers, we refer to ‘volunteers’ rather than ‘citizen scientists’ – and we believe strongly in acknowledging the contributions of everyone to a project, whatever term they choose to label themselves with.

Chris

Orchids and Lab Rats

Orchid Observers, the latest Zooniverse project, is perhaps at first glance a project like all the others. If you visit the site, you’ll be asked to sort through records of these amazing and beguiling plants, drawn from the collections of the Natural History Museum and from images provided by orchid fans from across the country. There’s a scientific goal, related to identifying how orchid flowering times are changing across the UK, a potential indicator of the effects of climate change, and we will of course be publishing our results in scientific journals.

5550b1e921d72904e200003b

Yet the project is, we hope, also a pointer to one way of creating a richer experience for Zooniverse volunteers. While other projects, such as iNaturalist, have made great efforts in mobilizing volunteers to carry out data collection, this is the first time we’ve combined that sort of effort with ‘traditional’ Zooniverse data analysis. We hope that those in a position to contribute images of their own will also take part in the online phase of the project, both as classifiers but also sharing their expertise online – if you’re interested, there’s an article in the most recent BSBI News that team member Kath Castillo wrote to encourage that magazine’s audience to get involved in both phases of the project.

BSBI News – published by the Botanical Society of Britain and Ireland, and not as far as I know available online – is a common place for the environmental and naturalist communities to advertise citizen science projects in this way, and so it also serves as a place where people talk about citizen science. The same edition that contains Kath’s article also includes a piece by Kew research associate Richard Bateman chewing over the thorny issue of funding distributed networks of volunteers to participate (and indeed, to coordinate) projects like these. He alludes to the ConSciCom project in which we’re partners, and which has funded the development of both Orchid Observers and another Zooniverse project, Science Gossip, suggesting that we view volunteers as either a freely available source of expertise or, worse, as ‘laboratory rats’.

Neither rings true to me. While the work that gets done in and around Zooniverse projects couldn’t happen without the vast number of hours contributed by volunteers, we’re very conscious of the need to go beyond just passively accepting clicks. We view our volunteers as our collaborators – that’s why they appear on author lists for papers, and why when you take part in a Zooniverse project, then we should take on the responsibility of communicating the results back to you in a form that’s actually useful. The collaboration with the historians in ConSciCom, who study the 19th century – a time when the division between ‘professional’ and ‘citizen’ scientist was much less clear – has been hugely useful in helping us think this through (see, for example, Sally Frampton’s discussion of correspondence in the medical journals of the period). Similarly, it’s been great to work with the Natural History Museum who have a long and distinguished history in working with all sorts of naturalist groups. We’ve been working hard on directly involving volunteers in more than mere clickwork too, and ironically enough, the kind of collaboration with volunteer experts we hope to foster in Orchid Observers is part of the solution.

I hope you enjoy the new project – and as ever, comments and thoughts on how we can improve are welcome, either here or via the project’s own discussion space.

Chris

PS This debate is slightly different, but it reminds me of the discussions we’ve had over the years about whether ‘citizen’ science is actually science, or just mere clickwork. Here are some replies from 2010 and from 2013.

Introducing Darren McRoy – Zooniverse Community Builder!

darren

 

Back in August I wrote about our search for someone we were calling a ‘community builder,’ which I said was ‘the most important job in the Zooniverse.’ The position was created because of the rapid expansion of the project, and the plans we have for the next year or two, which will mean we may be able to create hundreds or thousands of new projects. If the Zooniverse isn’t constrained by the slow process of project-by-project development, then we need to rethink how we choose what is hosted on our platform, what gets promoted—and how we talk about such things. We need, in fact, to try and build a broader Zooniverse community, capable of taking the choice of projects out of our hands. At the same time, we want the tools we use to engage with this community to let everyone have a say, from new classifiers on a single project to those who roam freely across all of our Talk discussion boards.

As many of you will have already discovered, we’ve found someone we can help us with this process — Darren McRoy. Darren is a 2010 graduate of Northwestern University’s Medill School of Journalism. He has worked as a reporter and editor and is an experienced writer and communicator with a strong focus on developing online communities and strategic digital content. One of his first projects will be gathering and compiling the feedback that will inform the upcoming rebuild of the Talk discussion system. He will be a regular presence on the forums, responding to users’ comments and concerns and seeking opportunities to spur additional conversation. He will also be contributing some written content for Zooniverse projects, blogs, websites, etc. when needed, and giving feedback to the development team.

You should see quite a lot of Darren, and we’d like to encourage you to talk to him if you have any questions, comments, concerns, or other feedback about the Zooniverse community. In particular, right now he is seeking feedback about how Talk can be improved to better serve both the science goals and the growing community of contributors and volunteers.

Darren can be reached via email at darren@zooniverse.org or DZM on Talk. Please feel free to contact him — he is looking forward to working with all of you!

Chris

The most important job in the Zooniverse

Job ad

The Zooniverse team has, over the last five years or so, shown signs of growing uncontrollably like some sort of bacterial colony that requires feeding with grant money. The job we’ve just advertised (at Adler Planetarium) might, though, be the most important yet. As those who are eager followers of this blog will know, we’re currently working hard on rebuilding the Zooniverse platform so that it can support many more projects.

If the Zooniverse can get to the point where we’re no longer constrained by the number of projects that can be built, we will need to think about how projects get chosen to appear on the Zooniverse, and about who should make that decision. Our opinion is that you – our community – should be more involved, and to work out how to make that happen we’re looking for what we’re calling a ‘community builder’. As you’ll see from the job description, this isn’t a technical post, but rather we’re looking for someone who knows how to build a community that is capable of awesome things. If that sounds like you, please get in touch.

Chris

PS The post is funded by a new grant from our friends at the Alfred P. Sloan foundation, to whom we’re eternally* grateful.

* – or as near as we can make it

Two more jobs at the Zooniverse

As part of our ongoing expansion of the Oxford Zooniverse team, I’m delighted to announce that there are two new jobs available at Zooniverse HQ in Oxford. We’re looking for developers and scientists who are excited at the prospect of helping us find more planets, keep an eye on more animals and generally make the Zooniverse more awesome.

5202453944_ed67b41917_b

We’re looking for the following kinds of people:

Data Scientist/Hadoopist to help us build up the processing power of the Zooniverse infrastructure

Postdoc in the statistics of citizen science – this might be a scientist with an interest or experience in citizen science, or someone with statistical expertise. In any case we’re looking to take a proper crack at the generic problem of combining classifications to produce consensus.

Both jobs are two year positions, and we’re really excited about expanding the team in Oxford. If you’d like to know more, you can contact me on cjl AT astro.ox.ac.uk or 07808 167288.

Chris

Want to work with the Zooniverse?

As part of a large expansion of the Oxford Zooniverse team, I’m delighted to announce that there are four new jobs available at Zooniverse HQ in Oxford. We’re looking for developers who are excited at the prospect of helping us find more planets, keep an eye on more animals and generally make the Zooniverse more awesome.

5202453944_ed67b41917_b

We’re looking for the following kinds of people:

Infrastructure Engineer
Senior Front-End Developer
Data Scientist/Hadoopist
Senior Application Developer

These jobs mark the start of the next stage in the Zooniverse’s evolution, and we’re really excited about expanding the team in Oxford. If you’d like to know more, you can contact me on cjl AT astro.ox.ac.uk or 07808 167288.

Chris

Not the Premier League : How Zooniverse got blocked by the courts

Anyone browsing the BBC News Technology section last night might have seen an unexpected appearance of a couple of our projects in this story about illegal streaming of Premier League football games. The story started on Saturday with an email from a volunteer pointing out that Virgin Media, a major Internet Service Provider in the UK, were blocking access to Notes From Nature. All is well now, but if you do experience problems please let us know. If you’d like the background, then read on.

Continue reading Not the Premier League : How Zooniverse got blocked by the courts

(Many) Zooniverse Papers Now Open Access

You don’t have to hang around the Zooniverse very long to find out that we’re rather proud of our growing list of publications. We think it’s essential that these papers are available to everyone which is why, for example, we’ve been posting versions of the astronomical papers on arXiv’s Astro-Ph. This is where I get papers I want to read, anyway, but there are advantages to occasionally being able to access the ‘real thing’ – the journal’s own version of the paper.

The doors to the Bodelian library in Oxford are labelled by subject. The one on the left here serves both astronomy and rhetoric. Credit : Jim Linwood

I’m delighted, therefore, to say that Oxford University Press, publishers of the journal we most frequently submit papers to have agreed to make all Zooniverse papers completely free to access. This applies to any Zooniverse paper in Monthly Notices of the Royal Astronomical Society (which is neither monthly nor contains notices of the Royal Astronomical Society), so whether you want to read about bulgeless galaxies, the Solar System’s dust, the supernovae we discovered, Planet Hunters results or Milky Way Project bubbles you can now do so from the journal itself.

Anyone Fancy an Asteroid?

This could be the Asteroid Zoo logo
This might be the Asteroid Zoo logo

Anyone interested in astronomy on the web will be aware of the fabulous success of Planetary Resources’ fundraising effort to build and launch the ARKYD space telescope. They’ve already raised more than a million dollars – helped in part by a cunning plan to let you take a picture of yourself in space – but they’re not stopping there. With three days to go, we’re delighted to announce that they’re going to try and help us help Zooniverse volunteers hunt for potentially hazardous asteroids.

The latest stretch goal is to support the development by the Zooniverse of a citizen science asteroid hunt. If the new target is hit, we’ll build a system that uses more than 3 million images, taken data from the Catalina Sky Survey – the survey responsible for nearly half of the near Earth asteroid discoveries in the last fifteen years. We know there are asteroids out that are waiting to be discovered, and we’re willing to bet that the existing routines used to scan through the survey data didn’t find them all.

Recent discoveries of near-Earth objects; Catalina's the big purple part.
Recent discoveries of near-Earth objects; Catalina’s the big purple part.

Anyone who’s followed the Zooniverse over the last few years knows that we believe in doing projects that make authentic contributions to science, and so I’m especially pleased that the project with Planetary Resources is also focused on improving machine learning solutions to asteroid hunting. Rather like our supernova project, an ideal outcome would be to use the classifications provided by volunteers to improve automated searching and suggest new methods by which machines might take up the strain. In the meantime, though, there are new (small) worlds to find – with your help, we’ll be launching the search for them soon.

I’ve put my money where my mouth is already, and if you can afford it then I hope you’ll follow the link and donate so we can all go asteroid hunting. You can also watch their Kickstarter video to see what they’re trying to do.

PRI-KS-Banner-428x60