We are excited to share with you results from two Zooniverse projects, ‘Etch A Cell – Fat Checker’ and ‘Etch A Cell – Fat Checker Round 2’. Over the course of these two projects, more than 2000 Zooniverse volunteers contributed over 75 thousand annotations!

One of the core aims of these two projects was to enable the design and implementation of machine learning approaches that could automate the annotation of fat droplets in novel data sets, to provide a starting point for other research teams attempting to perform similar tasks.

With this in mind, we have developed multiple machine learning algorithms that can be applied to both 2D and 3D fat droplet data. We describe these models in the blog post below.

Machine learning model	2D or 3D data	Publications to date	Other outputs
PatchGAN	2D	https://ceur-ws.org/Vol-3318/short15.pdf	https://github.com/ramanakumars/patchGAN https://pypi.org/project/patchGAN/
TCuP-GAN	3D	“What to show the volunteers: Selecting Poorly Generalized Data using the TCuPGAN”; Sankar et al., accepted in ECMLPKDD Workshop proceedings.	https://github.com/ramanakumars/TCuPGAN/
UNet/UNet3+/nnUNet	2D		https://huggingface.co/spaces/umn-msi/fatchecker

Machine learning models for the segmentation of fat droplets in 2D data

Patch Generative Adversarial Network (PatchGAN)
Generative Adversarial Networks (GANs) were introduced in 2018 for the purpose of realistic learning of image-level features and have been used for various computer vision related applications. We implemented a pixel-to-pixel translator model called PatchGAN, which learns to convert (or “translate”) an input image to another image form. For example, such a framework can learn to convert a gray-scale image to a colored version.

The “Patch” in PatchGAN signifies its capability to learn image features in different sub-portions of an image (rather than just across an entire image as a whole). In the context of the Etch A Cell – Fat Checker project data, predicting the annotation regions of fat droplets is analogous to PatchGAN’s image-to-image translation task.

We trained the PatchGAN model framework on the ~50K annotations generated by volunteers in Etch A Cell – Fat Checker. Below we show two example images from the Etch A Cell: Fat Checker (left column) along with aggregated annotations provided by the volunteers (middle panel), and their corresponding 2D machine learning model (PatchGAN) predicted annotations.

We found that the PatchGAN typically performed well in learning the subject image to fat-droplet annotation predictions. However, we noticed that the model highlighted some regions potentially missed by the volunteers, as well as instances where it has underestimated some regions that the volunteers have annotated (usually intermediate to small sized droplets).

We made have made this work, our generalized PatchGAN framework, available via an open-source repository at https://github.com/ramanakumars/patchGAN and https://pypi.org/project/patchGAN/. This will allow anyone to easily train the model on a set of images and corresponding masks, or to use the pre-trained model to infer fat droplet annotations on images they have at hand.
 

UNet, UNet3+, and nnUNet
In addition to the above-mentioned PatchGAN network, we have also trained three additional frameworks for the task of fat droplet identification – UNet, UNet3+, and nnUNet.

UNet is a popular deep-learning method used for semantic segmentation within images (e.g., identifying cars/traffic in an image) and has been shown to capture intricate image details and precise object delineation. Its architecture is U-shaped with two parts – an encoder that learns to reduce the input image down a compressed “fingerprint” and a decoder which learns to predict the target image (e.g., fat droplets in the image) based on that compressed fingerprint. Fine-grained image information is shared between the encoder and decoder parts using the so-called “skip connections”. UNet3+ is an upgraded framework built upon the foundational UNet that has been shown to capture both local and global features within medical images.

nnUNet is an user-friendly, efficient, and state-of-the-art deep learning platform to train and fine-tune models for diverse medical imaging tasks. It employs a UNet-based architecture and comes with image pre-processing and post-processing techniques.

We trained these three networks on the data from the Fat Checker 1 project. Below, we show three example subject images along with their corresponding volunteer annotations and different model predictions. Between the three models, nnUNet demonstrated superior performance.

Machine learning models for the segmentation of fat droplets in 3D data

Temporal Cubic PatchGAN (TCuP-GAN
Motivated by the 3D volumetric nature of the Etch A Cell – Fat Checker project data, we also developed a new 3D deep learning method that learns to predict the direct 3D regions corresponding to the fat droplets. To develop this, we built on top of our efforts of our PatchGAN framework and merged it with another computer vision concept called “LongShort-Term Memory Networks (LSTMs)”. Briefly, in recent years, LSTMs have seen tremendous success in learning sequential data (e.g., words and their relationship within a sentence) and they have been used to learn relationships among sequences of images (e.g., movement of a dog in a video).

We have successfully implemented and trained our new TCuP-GAN model on the Etch A Cell – Fat Checker data. Below is an example image cube – containing a collection of 2D image slices that you viewed and annotated – along with the fat droplet annotation prediction of our 3D model. For visual guidance, we show the middle panel where we reduced the transparency of the image cube shown in the left panel, highlighting the fat droplet structures that lie within.

We found that our TCuP-GAN model successfully predicts the 3D fat droplet structures. In doing so, our model also learns realistic (and informative) signatures of lipid droplets within the image. Leveraging this, we are able to ask the model which 2D image slices contain the most confusion between lipid droplets and surrounding regions of the cells when it comes to annotating the fat droplets. Below, we show two example slices where the model was confident about the prediction (i.e., less confusion; top panel) and where the model was significantly confused (red regions in fourth column of the bottom panel). As such, we demonstrated that our model can help find those images in the data set that preferentially require information from the volunteers. This can serve as a potential efficiency step for future research teams to prioritize certain images that require attention from the volunteers.

Integrating Machine Learning Strategies with Citizen Science

Several thousands of annotations collected from the Etch A Cell – Fat Checker project(s) and their use to train various machine learning frameworks have opened up possibilities that can enhance the efficiency and annotation gathering and help accelerate the scientific outcomes for future projects.

While the models we described here all performed reasonably well in learning to predict the fat droplets, there were a substantial number of subjects where they were inaccurate or confused. Emerging “human-in-the-loop” strategies are becoming increasingly useful in these cases – where citizen scientists can help with providing critical information on those subjects that require the most attention. Furthermore, an imperfect machine learning model can provide an initial guess which the citizens can use as a starting point and provide edits, which will greatly greatly reduce the amount of effort needed by individual citizen scientists.

For our next steps, using the data from the Etch A Cell – Fat Checker projects, we are working towards building new infrastructure tools that will enable future projects to leverage both citizen science and machine learning towards solving critical research problems.

Enhancements to the freehand drawing tool

First, we have made upgrades to the existing freehand line drawing tool on Zooniverse. Specifically, users will now be able to edit a drawn shape, undo or redo during any stage of their drawing process, automatically close open shapes, and delete any drawings. Below is a visualization of an example drawing where the middle panel illustrates the editing state (indicated by the open dashed line) and re-drawn/edited shape. The tool box with undo, redo, auto-close, and delete functions is also shown.

A new “correct a machine” framework

We have built a new infrastructure that will enable researchers to upload machine learning (or any other automated method) based outlines in a compatible format to the freehand line tool such that each subject when viewed by a volunteer on a Zooniverse will be shown the pre-loaded machine outlines, which they can edit using the above newly-added functionality. Once volunteers provide their corrected/edited annotations, their responses will be recorded and used by the research teams for their downstream analyses. The figure below shows an example visualization of what a volunteer would see with the new correct a machine workflow. Note, that the green outlines shown on top of the subject image are directly loaded from a machine model prediction and volunteers will be able to interact with them.

From Fat Droplets to Floating Forests

Finally, the volunteer annotations submitted to these two projects will have an impact far beyond the fat droplets identification in biomedical imaging. Inspired by the flexibility of the PatchGAN model, we also carried out a “Transfer Learning” experiment, where we tested if a model trained to identify fat droplets can be used for a different task of identifying kelp beds in satellite imaging. For this, we used the data from another Zooniverse project called Floating Forests.

Through this work, we found that our PatchGAN framework readily works to predict the kelp regions. More interestingly, we found that when our pre-trained model to detect fat droplets within Fat Checker project data was used as a starting point to predict the kelp regions, the resultant model achieved very good accuracies with only a small number of training images (~10-25% of the overall data set size). Below is an example subject along with the volunteer annotated kelp regions and corresponding PatchGAN prediction. The bar chart illustrates how the Etch A Cell – Fat Checker based annotations can help reduce the amount of annotations required to achieve a good accuracy.

In summary: THANK YOU!

In summary, with the help of your participation in the Etch A Cell – Fat Checker and Etch A Cell – Fat Checker Round 2 projects, we have made great strides in processing the data and training various successful machine learning frameworks. We have also made a lot of progress in updating the annotation tools and building new infrastructure towards making the best partnership between humans and machines for science. We are looking forward to launching new projects that use this new infrastructure we have built!

This project is part of the Etch A Cell organisation

‘Etch A Cell – Fat Checker’ and ‘Etch A Cell – Fat Checker round 2’ are part of The Etchiverse: a collection of multiple projects to explore different aspects of cell biology. If you’d like to get involved in some of our other projects and learn more about The Etchiverse, you can find the other Etch A Cell projects on our organisation page here.

Thanks for contributing to Etch A Cell! – Fat Checker!

Introducing Etch a Cell – Demolition Squad

Earlier this year, the team behind the Etch A Cell series of citizen science projects launched their latest project; ‘Etch A Cell – Demolition Squad’. Through this project, researchers based at the Francis Crick Institute (London, UK) have teamed up with Zooniverse volunteers to study ‘lysosomes’.

What are lysosomes?

Lysosomes are balloon-shaped organelles contain a range of enzymes that can break down biological substances including proteins, sugars, and fats. These enzymes allow lysosomes to function as the cell’s digestive system; lysosomes digest and degrade substances from both inside and outside of cells. This versatile functionality allows lysosomes to contribute to a range of biological tasks, including the breakdown of aged cellular components, the destruction of viruses, and the support of digestion during times of hunger. Lysosomes even play a role in the life cycle of cells by facilitating the removal of cells that have reached their expiry date. Etch A Cell – Demolition Squad has the aim of improving our ability to study lysosomes, to enable us to further understand how they contribute to the world of cellular dynamics.

An image from the first data set analysed in Etch A Cell – Demolition Squad.

What’s the aim of the project?

The team behind Etch A Cell work with a variety of research teams to study different aspects of biology using cutting-edge Electron Microscopes. With their remarkable magnification and resolution, these microscopes allow researchers to capture intricate images of tissues, cells, and molecules. These images can be used to provide us with a richer understanding of biology, which can help us understand the biological changes associated with health and disease. Recent developments in electron microscope technology have enabled automatic image collection, leading to a deluge of data. This influx of information is helping propel research forward, however, it has caused a bottleneck in data analysis pipelines – which is why Etch A Cell needs the help of Zooniverse volunteers!

How are Zooniverse volunteers helping?

To study the images generated by our microscopes we typically analyse them by ‘segmenting’ the features we’re interested in, which means to draw around the bits of the cell that we want to examine. In Etch A Cell projects, Zooniverse volunteers help with this segmentation task. In Etch A Cell – Demolition Squad volunteers were asked to help with segmenting lysosomes. These structures can be very difficult to spot, so an imaging technique was used that enables the selective labelling of the lysosomes with a marker that allows their easier identification. You can see an example image below – the marker is shown in the images as pink, so volunteers were asked to draw around the grey blobs labelled with pink. In the image below you can also see some green lines which show how one expert segmenter drew around the lysosomes in this image.

In Etch A Cell – Demolition Squad, volunteers were asked to draw around lysosomes. To make the lysosomes easier to spot they were labelled with a marker, that is shown in the image above as pink. This image also shows lysosome segmentations done by an expert in green.

What are the results so far?

Since Etch A Cell – Demolition Squad was launched in May 2023, this project has received more than 12,000 classifications from hundreds of Zooniverse volunteers! This has allowed all 792 images in the first project data set to be retired. We’ve now started taking a look at the data, and we’re really impressed with the segmentations submitted! Well done to everyone who has contributed!

Here’s a sneak peek at some of the data you’ve produced through your collective efforts:

Briefly, you can see from these images show that Zooniverse volunteers have done fantastic job at this challenging task: the collective volunteer segmentations (shown in green in panel F) correspond really well to the location of lysosomes in the image (shown in pink in image B). With thanks to Francis Crick Institute internship student Fatihat Ajayi for generating these images.

The images above show the raw data (A) and this data overlaid with the pink marker that highlights the lysosomes (B). Image B is the one Zooniverse volunteers would have seen in the project interface. All the volunteer segmentations submitted for this image are shown overlaid in image C. Image D also shows the volunteer segmentations together, but in this image the line drawings have been ‘filled in’ and stacked on top of each other to make it easier to see where the volunteers agree there is a lysosome (the more volunteers who annotated a region, the brighter white it shows). Image E shows where most of the volunteers agreed there were lysosomes in this image, and image F shows how this corresponds to the raw data image (A). From these images you can see how collectively, Zooniverse volunteers did a great job at drawing around the lysosomes.

What’s next for Etch A Cell – Demolition Squad?

In the near future, we will be uploading new batches of data that we will need your help to analyse. These new data sets may look slightly different, but the task off segmenting the lysosomes will remain the same.

This project is part of the Etch A Cell organisation

‘Etch A Cell – Demolition Squad’ is one of multiple projects produced by the Etch A Cell team and their collaborators to explore different aspects of cell biology. If you’d like to get involved in some of our other projects, you can find the other Etch A Cell projects on our organisation page here.

Thanks for contributing to Etch A Cell!

Name: Ramana Sankar

Location: University of California, Berkeley

Tell us about your role within the team:

I’ve been with the Zooniverse team for two years now as a postdoc working with Lucy Fortson at University of Minnesota. My main role was to help with building human-machine interfaces for project teams on Zooniverse (particularly in the avenue of speeding up project completion rates and improving avenues serendipitous discovery), and also with providing data science assistance to projects.

What did you do in your life before the Zooniverse?

I did my Ph.D. at Florida Institute of Technology. My main focus was on studying the formation of thunderstorms on Jupiter. I was (and still am) interested in understanding more about the dynamics of the jovian atmosphere and how the fluid dynamics and chemical processes shape the cloud structures we see when we look at Jupiter. As part of this work, I also tried out a few deep learning approaches to reduce data from the JunoCam instrument and gained some machine learning and data science experience, which drew me to apply for the postdoctoral scholar position with the Zooniverse team at UMN.

What does your typical working day involve?

Most of my day is spent with code. I break my tasks into astrophysical research (focusing on Jupiter’s atmosphere), machine/deep learning (developing new models for the project teams that I am assisting) and working on aggregation utilities for project teams.

How would you describe the Zooniverse in one sentence?

A platform that reduces the barrier of entry for scientific pursuits

Tell us about the first Zooniverse project you were involved with

As part of my introduction to the team, a fellow postdoc at UMN and I was asked to create a test Zooniverse project to get a hang of using the Zooniverse project builder. We built the “Chi Square Kitties” project where we asked people to rate the fits of cats inside various containers. We scraped the web for cat images (there seemed to be a limitless supply) and asked a simple question of whether the cat inside the container was an underfit, overfit or a purr-fect fit!

Of all the discoveries made possible by the Zooniverse, which for you has been the most notable? (and why?)

I love the fact that the Gravity Spy team found a glitch class called “Air compressor” which was noise from a nearby A/C unit. This is the kind of discovery that is incredibly difficult (if not impossible) without having human eyes on the data. This is the kind of discovery which is fueled by Zooniverse!

What’s been your most memorable Zooniverse experience?

On the Jovian Vortex Hunter project, the volunteers came up with very creative names for some of the types of clouds that was observed. Some of my favourite ones are “red-compact-nursery” for subjects which contain very small red cyclones which seem to be forming in between the folded filamentary regions. These creative names (in my view) are actually much more useful than very descriptive scientific jargon, since it helps to create very interpretable labels based on what the feature looks like, rather than the underlying physics.

What are your top three citizen science projects? 

The best things about citizen science is that they cater to a very wide audience. Some projects can be highly accessible and can be done by basically anyone, while others require very technical expertise. There are a ton of amazing projects on Zooniverse which span this experience range, but I will pick some non-Zooniverse projects to highlight some of the reasons why I really love the citsci methodology:

1. JunoCam: the Juno spacecraft almost did not have a camera since it did not fit within the payload mass and budget constraints. Fortunately, a last minute decision was made to put a simple 4-channel camera (previously flight tested on the MSL) and not have a dedicated science team for the instrument. Instead, JunoCam relies on thousands of artists and hobbyists for downloading and processing the raw footage. JunoCam is a story to be told for the way it engages people, particularly artists, in scientific communication, and also how, now, there are several papers written on using the data processed by citizen scientists for new science!

2. PVOL (Planetary Virtual Observatory and Laboratory): This is a website for amateur observers to post their observation of giant planets to a centralized server which can be used by research to provide contextual information to any related studies. PVOL has helped fueled a lot of research into giant planet atmospheres, especially in the context of long cadence studies (longevity of features, periodicity of instabilities) and also in very short term (e.g., meteor impacts, etc.) This is a very useful way to use a very skilled hobby (astrophotography) for an extremely useful cause.

3. RadioJOVE: A project for interested parties to build and operate a simple multi-wavelength radio telescope. This was started as a way to observe Jupiter (and now other sources) in the radio wavelengths. What is amazing about this project is that it teaches a very niche skill (putting together and using a radio telescope) to non-scientists. Through the method of citsci, this project has made great strides in science communication and ways to inspire kids to pursue STEM degrees.

What advice would you give to a researcher considering creating a Zooniverse project?

Volunteers are not simply labelers. They do not replace a complicated algorithm, but instead provide value far beyond a simple classification. The greatest strength of Zooniverse is the fact that volunteers want to be engaged deeply in science, and that needs to be baked into the ethos of any Zooniverse project

How can someone who’s never contributed to a citizen science project get started?

Find your passion in science and search for local resources. For example, if you’re interested in ecology, there are probably conservation groups that could use your help with taking photos of local fauna and/or flaura. If you’re an astronomy enthusiasts, you can talk to the local planetarium, who can provide you with resources and ways to contribute. Finally, you can see if there are public seminars in their local university for departments related to their interest. These can help answer many of the questions you might have about the field.

Where do you hope citizen science and the Zooniverse will be in 10 years time?

I hope that the bridge between science and citizen science narrows further than it currently is. I hope that more research teams can use citizen science not only as a method to analyze and process data, but also as a way to drive science engagement and reduce the barrier of entry to research. I also hope that future spacecraft missions follow in the path laid by JunoCam and have a strong citsci component

Is there anything in the Zooniverse pipeline that you’re particularly excited about?

I’m really interested in how machine learning will play a role in Zooniverse. As the world heads deeper into the use of AI and automation, I am interested in seeing how Zooniverse will adopt these tools while maintaining efficiency, and more importantly, accuracy. Several projects on Zooniverse already use a variety of AI-based tools, so I am looking forward to seeing many more projects make use of these resources.

When not at work, where are we most likely to find you?

Either at home playing video games (catching up on my huge backlog which built up over grad school) or out hiking. I know these are polar opposites, but if I can get over the barrier of pulling away from my PC, I’m always ready to do a 5 mile hike!

Do you have any party tricks or hidden talents?

I learnt to sing Carnatic music, which is a classical South Indian music tradition, and I like to think I can hold a tune (although others can be the judge of that)!